-
Notifications
You must be signed in to change notification settings - Fork 61
feat: add code scanning coverage report and alert dismissal scripts #144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
a2d2490
feat: add script to generate comprehensive code scanning coverage report
joshjohanning 018fb03
feat: add code scanning coverage report as node script
joshjohanning e5b80b3
feat: add archived header to bash script
joshjohanning ce85343
fix(docs): alphabetization
joshjohanning 90bea81
feat: add GitHub App authentication support and update README
joshjohanning 8525270
refactor: rename script
joshjohanning ebcec20
feat(deps): update octokit and related dependencies to latest versions
joshjohanning bb3de7f
fix(docs): clarify repository permissions in README
joshjohanning 46ab062
feat: add support for fetching open alert counts and summary statisti…
joshjohanning b6d1a7e
feat: add API call counting and update alert sub reports to only show…
joshjohanning e3294fd
feat: grab repo language via GraphQL to save on API usage
joshjohanning e82e904
test: add Jest for testing and update package.json for module support
joshjohanning 1274e6c
refactor!: rename optional params for clarity
joshjohanning 0995c3d
feat: improve argument parsing with required value checks
joshjohanning a2bf38f
feat!: remove get-code-scanning-coverage-report.sh script and update …
joshjohanning 2609134
docs: update README
joshjohanning 271c4e5
Update scripts/code-scanning-coverage-report/package.json
joshjohanning eb0e3e4
Update scripts/code-scanning-coverage-report/code-scanning-coverage-r…
joshjohanning 462dfb5
deps: update package-lock.json
joshjohanning 510dbab
feat: add open alerts count to code scanning coverage report
joshjohanning 7442c3d
feat: add configurable stale days option to code scanning coverage re…
joshjohanning 13c2502
feat: add support for processing multiple organizations from a file i…
joshjohanning b7c1138
feat: enable automatic installation ID lookup for multi-org scanning …
joshjohanning d16195b
fix: prevent conflicting usage of --orgs-file and organization argume…
joshjohanning f634d56
test: add authentication functions and tests for GitHub App integrati…
joshjohanning 4ab5d08
feat: add new utility for dismissing alerts by rule pattern
joshjohanning 136f6e1
feat: implementing suggestions from pull request
joshjohanning File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,227 @@ | ||
| # code-scanning-coverage-report | ||
|
|
||
| Generate a comprehensive code scanning coverage report for all repositories in a GitHub organization. | ||
|
|
||
| ## Features | ||
|
|
||
| - Reports CodeQL enablement status, last scan date, and scanned languages | ||
| - Identifies CodeQL-supported languages that are not being scanned | ||
| - Shows open alert counts and analysis errors/warnings | ||
| - Generates actionable sub-reports for remediation | ||
| - Supports parallel API calls for faster processing | ||
| - Works with GitHub.com and GitHub Enterprise Server | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Node.js 18 or later | ||
| - A GitHub token with `repo` scope, or GitHub App credentials | ||
|
|
||
| ## Installation | ||
|
|
||
| ```shell | ||
| cd scripts/code-scanning-coverage-report | ||
| npm install | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| ```shell | ||
| # Set your GitHub token | ||
| export GITHUB_TOKEN=ghp_xxxxxxxxxxxx | ||
|
|
||
| # Note: If you are authenticated with the GitHub CLI, you can use `gh auth token` to get your token: | ||
| # export GITHUB_TOKEN=$(gh auth token) | ||
|
|
||
| # Basic usage - output to stdout | ||
| node code-scanning-coverage-report.js my-org | ||
|
|
||
| # Output to file (also generates sub-reports) | ||
| node code-scanning-coverage-report.js my-org --output report.csv | ||
|
|
||
| # Check a single repository | ||
| node code-scanning-coverage-report.js my-org --repo my-repo | ||
|
|
||
| # Sample 25 random repositories | ||
| node code-scanning-coverage-report.js my-org --sample --output sample.csv | ||
|
|
||
| # Include workflow status column | ||
| node code-scanning-coverage-report.js my-org --check-workflow-status --output report.csv | ||
|
|
||
| # Check for unscanned GitHub Actions workflows | ||
| node code-scanning-coverage-report.js my-org --check-unscanned-actions --output report.csv | ||
|
|
||
| # Use with GitHub Enterprise Server | ||
| export GITHUB_API_URL=https://github.example.com/api/v3 | ||
| node code-scanning-coverage-report.js my-org --output report.csv | ||
|
|
||
| # Adjust concurrency (default: 10) | ||
| node code-scanning-coverage-report.js my-org --concurrency 5 --output report.csv | ||
|
|
||
| # Process multiple organizations from a file | ||
| node code-scanning-coverage-report.js --orgs-file orgs.txt --output report.csv | ||
|
|
||
| # Customize stale threshold (default: 90 days) | ||
| node code-scanning-coverage-report.js my-org --stale-days 60 --output report.csv | ||
| ``` | ||
|
|
||
| ## Options | ||
|
|
||
| | Option | Description | | ||
| |--------|-------------| | ||
| | `--orgs-file <file>` | File containing list of organizations (one per line) | | ||
| | `--output <file>` | Write CSV to file (also generates sub-reports) | | ||
| | `--repo <repo>` | Check a single repository instead of all repos | | ||
| | `--sample` | Sample 25 random repositories | | ||
| | `--fetch-alerts` | Include alert counts in report (increases API usage) | | ||
| | `--check-workflow-status` | Check CodeQL workflow run status (success/failure) | | ||
| | `--check-unscanned-actions` | Check if repos have Actions workflows not being scanned | | ||
| | `--concurrency <n>` | Number of concurrent API calls (default: 10) | | ||
| | `--stale-days <n>` | Days after last scan to consider repo stale (default: 90) | | ||
| | `--help` | Show help message | | ||
|
|
||
| ## API Usage | ||
|
|
||
| Default options use approximately **2 API calls per repository**: | ||
|
|
||
| - With a **Personal Access Token** (5,000 requests/hour): supports up to ~2,500 repos | ||
| - With **GitHub App authentication** (15,000 requests/hour): supports up to ~7,500 repos | ||
|
|
||
| Optional flags increase API usage: | ||
|
|
||
| | Flag | Additional Calls | | ||
| |------|------------------| | ||
| | `--fetch-alerts` | +1 call per repo (paginated for repos with many alerts) | | ||
| | `--check-workflow-status` | +1-2 calls per repo | | ||
| | `--check-unscanned-actions` | +1 call per repo | | ||
|
|
||
| The script displays your current rate limit at startup and total API calls used at completion. | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| Two authentication methods are supported: | ||
|
|
||
| - **Personal Access Token (PAT)**: Simple setup, good for testing or small organizations | ||
| - **GitHub App**: Recommended for production use - provides higher rate limits (5,000 vs 15,000 requests/hour) | ||
|
|
||
| ### Token Authentication | ||
|
|
||
| | Variable | Description | | ||
| |----------|-------------| | ||
| | `GITHUB_TOKEN` | GitHub token with `repo` scope | | ||
| | `GITHUB_API_URL` | API endpoint (defaults to `https://api.github.com`) | | ||
|
|
||
| ### GitHub App Authentication (recommended) | ||
|
|
||
| | Variable | Description | | ||
| |----------|-------------| | ||
| | `GITHUB_APP_ID` | GitHub App ID | | ||
| | `GITHUB_APP_PRIVATE_KEY_PATH` | Path to GitHub App private key file (.pem) | | ||
| | `GITHUB_API_URL` | API endpoint (defaults to `https://api.github.com`) | | ||
|
|
||
| The script automatically looks up the installation ID for each organization being processed. This enables scanning multiple organizations with a single command using `--orgs-file`. | ||
|
|
||
| **Required GitHub App Permissions:** | ||
|
|
||
| Repository permissions: | ||
|
|
||
| | Permission | Access | Required For | | ||
| |------------|--------|--------------| | ||
| | Code scanning alerts | Read | Code scanning status, analyses, and alert counts | | ||
| | Contents | Read | Listing repositories and checking for workflow files | | ||
| | Metadata | Read | Detecting repository languages (this is automatically added) | | ||
| | Actions | Read | Workflow run status (only if using --check-workflow-status) | | ||
|
|
||
| Organization permissions: | ||
|
|
||
| | Permission | Access | Required For | | ||
| |------------|--------|--------------| | ||
| | Administration | Read | Listing all repositories in the organization | | ||
|
|
||
joshjohanning marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| **Note:** The app must be installed on the organization with access to the repositories you want to scan (either "All repositories" or selected repositories). The app can only report on repositories it has been granted access to. | ||
|
|
||
| **Note:** If GitHub App credentials are provided, they take precedence over `GITHUB_TOKEN`. | ||
|
|
||
| ### GitHub App Usage Example | ||
|
|
||
| ```shell | ||
| # Set GitHub App credentials | ||
| export GITHUB_APP_ID=123456 | ||
| export GITHUB_APP_PRIVATE_KEY_PATH=/path/to/private-key.pem | ||
|
|
||
| # Run the report for a single org | ||
| node code-scanning-coverage-report.js my-org --output report.csv | ||
|
|
||
| # Run the report for multiple orgs (installation ID is looked up automatically) | ||
| node code-scanning-coverage-report.js --orgs-file orgs.txt --output report.csv | ||
| ``` | ||
|
|
||
| ## Output Columns | ||
|
|
||
| | Column | Description | | ||
| |--------|-------------| | ||
| | Organization | Organization name | | ||
| | Repository | Repository name | | ||
| | Default Branch | The default branch of the repository | | ||
| | Last Updated | When the repository was last updated | | ||
| | Languages | Languages detected in the repository (semicolon-separated) | | ||
| | CodeQL Enabled | `Yes`, `No Scans`, `Disabled`, `Requires GHAS`, or `No` | | ||
| | Last Default Branch Scan Date | Date of most recent scan on default branch | | ||
| | Scanned Languages | Languages scanned by CodeQL (semicolon-separated) | | ||
| | Unscanned CodeQL Languages | CodeQL-supported languages not being scanned | | ||
| | Open Alerts | (with `--fetch-alerts`) Total number of open code scanning alerts | | ||
| | Critical Alerts | (with `--fetch-alerts`) Number of critical severity alerts | | ||
| | Analysis Errors | Errors from most recent analysis | | ||
| | Analysis Warnings | Warnings from most recent analysis | | ||
| | Workflow Status | (with `--check-workflow-status`) `OK`, `Failing`, `No workflow`, or `Unknown` | | ||
|
|
||
| ## Sub-reports | ||
|
|
||
| When using `--output`, the script generates actionable sub-reports: | ||
|
|
||
| | File | Description | | ||
| |------|-------------| | ||
| | `*-disabled.csv` | Repos with CodeQL disabled or no scans | | ||
| | `*-stale.csv` | Repos modified after last scan (configurable with `--stale-days`, default: 90) | | ||
| | `*-missing-languages.csv` | Repos scanning but missing some CodeQL languages | | ||
| | `*-critical-alerts.csv` | Repos with critical severity code scanning alerts | | ||
| | `*-analysis-issues.csv` | Repos with analysis errors or warnings | | ||
|
|
||
| ## CodeQL Supported Languages | ||
|
|
||
| The script recognizes these CodeQL-supported languages: | ||
|
|
||
| - C/C++ (reported as `c-cpp`) | ||
| - C# (reported as `csharp`) | ||
| - Go | ||
| - Java/Kotlin (reported as `java-kotlin`) | ||
| - JavaScript/TypeScript (reported as `javascript-typescript`) | ||
| - Python | ||
| - Ruby | ||
| - Swift | ||
|
|
||
| ## Example Output | ||
|
|
||
| ```csv | ||
| Organization,Repository,Default Branch,Last Updated,Archived,Languages,CodeQL Enabled,Last Default Branch Scan Date,Scanned Languages,Unscanned CodeQL Languages,Open Alerts,Critical Alerts,Analysis Errors,Analysis Warnings | ||
| my-org,my-app,main,2025-12-01,No,JavaScript;TypeScript;Python,Yes,2025-12-15,javascript-typescript;python,None,5,1,None,None | ||
| my-org,legacy-service,master,2024-06-15,No,Java,Yes,2024-01-10,java-kotlin,None,0,0,None,None | ||
| my-org,new-project,main,2025-12-20,No,Go;Python,No Scans,Never,,go;python,N/A,N/A,, | ||
| ``` | ||
|
|
||
| <!-- Remove this section when the bash script is deleted | ||
| ## Comparison with Bash Version | ||
|
|
||
| This Node.js version offers several advantages over the bash script (`gh-cli/code-scanning-coverage-report.sh`): | ||
|
|
||
| - **Faster**: Parallel API calls (configurable concurrency) | ||
| - **More reliable**: Proper JSON handling, no regex parsing issues | ||
| - **Cross-platform**: Works identically on macOS, Linux, and Windows | ||
| - **Easier to maintain**: Clean data structures and error handling | ||
|
|
||
| The bash version is still useful if you: | ||
|
|
||
| - Don't have Node.js installed | ||
| - Prefer no dependencies beyond `gh` CLI | ||
| - Need to quickly inspect/modify the script | ||
|
|
||
| --> | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.