A comprehensive command-line utility to analyze API test coverage across multiple protocols: OpenAPI/Swagger (REST), gRPC Protocol Buffers, and GraphQL schemas. Generates unified HTML reports with protocol-specific insights. Check out the Example!**
swagger-coverage-cli is a comprehensive tool that helps you measure API test coverage across multiple protocols. It analyzes how much of your documented APIs are actually covered by your Postman tests. The tool supports:
- 📋 REST APIs: OpenAPI/Swagger specifications (v2/v3) in JSON or YAML format
- ⚡ gRPC APIs: Protocol Buffer (
.proto) files with service definitions - 🔀 GraphQL APIs: GraphQL schema (
.graphql,.gql) files with queries, mutations, and subscriptions - 📊 CSV APIs: Custom CSV format for flexible API documentation
- API Specifications: Single or multiple API files in supported formats
- Test Collections: Postman collections (JSON) with requests and test scripts
- Execution Reports: Newman run reports (JSON) with actual test execution results
The tool supports processing multiple API specifications in a single run, making it ideal for organizations managing microservices with diverse protocols. It calculates unified coverage percentages and produces detailed HTML reports with protocol-specific insights.
swagger-coverage-cli provides comprehensive support for modern API ecosystems with multiple protocols, enabling unified coverage analysis across your entire technology stack.
# Single protocol APIs
swagger-coverage-cli api.yaml collection.json # OpenAPI/REST
swagger-coverage-cli service.proto collection.json # gRPC
swagger-coverage-cli schema.graphql collection.json # GraphQL
swagger-coverage-cli api-docs.csv collection.json # CSV
# Mixed protocol APIs (Enterprise-ready)
swagger-coverage-cli "api.yaml,service.proto,schema.graphql" collection.json
# All existing options work across protocols
swagger-coverage-cli "api.yaml,service.proto" collection.json --verbose --strict-body- OpenAPI v2/v3: Full specification support
- Smart Path Matching: Handles parameter variations (
/users/{id}vs/users/{userId}) - Status Code Intelligence: Prioritizes 2xx → 4xx → 5xx responses
- Request Body Validation: JSON schema validation with strict mode
- Protocol Buffer Parsing: Automatic
.protofile analysis - Service Discovery: Extracts all services, methods, and message types
- Path Generation: Maps to HTTP/2 paths (
/package.service/method) - Content-Type Validation: Supports
application/grpcand variants
- Schema Analysis: Parses
.graphqland.gqlfiles - Operation Extraction: Identifies queries, mutations, and subscriptions
- Type System: Full support for arguments, unions, and interfaces
- Endpoint Unification: Maps all operations to
/graphqlendpoint
- Protocol Column: Color-coded identification (🟢 gRPC, 🔴 GraphQL, 🔵 REST)
- Mixed Statistics: Combined coverage metrics across all protocols
- Individual Breakdown: Per-API and per-protocol insights
- Smart Search: Protocol-aware filtering and search functionality
- 🌐 Multi-Protocol Support: Native support for REST (OpenAPI/Swagger), gRPC (Protocol Buffers), and GraphQL schemas
- 🔄 Mixed API Analysis: Process multiple API specifications with different protocols in a single run
- 🎯 Protocol-Aware Matching: Intelligent request matching tailored to each API protocol's characteristics
- 📊 Unified Reporting: Generate consolidated HTML reports with protocol-specific insights and color coding
- ⚡ Easy to Use: Simple CLI interface works across all supported protocols with consistent syntax
- 🔍 Multiple Input Types: Supports Postman collections and Newman run reports for maximum flexibility
- 🤖 Auto-Detection: Automatically detects API file types and Newman report formats
- 🏗️ Enterprise Ready: Perfect for microservices architectures using diverse API protocols
- 🎨 Smart Endpoint Mapping: Intelligent endpoint matching with status code prioritization and enhanced path matching
- 🔒 Strict Matching (Optional): Enforce strict checks for query parameters, request bodies, and more
- 🛡️ Flexible Validation: Skip spec validation with
--disable-spec-validationfor legacy APIs or specs with reference issues - 📈 Enhanced HTML Reports: Generates interactive
coverage-report.htmlwith protocol identification - 🧩 Extensible: Modular code structure allows customization of matching logic and protocol support
- 📋 CSV Support: Flexible API documentation format for teams preferring spreadsheet-based docs
- ✅ Unit Tested: Comprehensive Jest test suite covering all protocols and edge cases
flowchart LR
A[OpenAPI/Swagger Spec] --> B[Load & Parse Spec]
C[Postman Collection] --> D[Load & Parse Collection]
B --> E[Match Operations]
D --> E
E --> F[Calculate Coverage]
F --> G[Generate HTML & Console Report]
- Load & Parse Spec/CSV: The CLI reads your Swagger (YAML or JSON) or CSV file and extracts all documented operations (method, path, status codes, parameters, requestBody details, etc.).
- Load & Parse Collection: The CLI reads your Postman collection (JSON) to gather all requests, their URLs, methods, test scripts, and any explicitly tested status codes.
- *Match Operations: The tool compares each documented operation with each Postman request to see if it’s covered (based on path patterns, method, status codes tested, and optional strict checks on parameters and body).
- Calculate Coverage: It counts how many documented API operations are matched with Postman tests vs. how many total operations are defined in your spec or CSV.
- Generate Report: Prints a summary to the console and creates an HTML file (by default coverage-report.html) with matched and unmatched endpoints.
swagger-coverage-cli now supports processing multiple Swagger/OpenAPI specifications in a single run, making it perfect for organizations managing multiple APIs or microservices.
To analyze multiple APIs, provide comma-separated file paths:
swagger-coverage-cli "api1.yaml,api2.yaml,api3.json" collection.json- Unified Coverage Report: Get a single report showing coverage across all APIs
- API Identification: Each operation in the report is tagged with its source API
- Individual API Breakdown: The report header shows both the combined API list and individual API names
- Separate Operation Tracking: Operations with the same path/method from different APIs are tracked separately
- Backwards Compatibility: Single API mode works exactly as before
When processing multiple APIs, the console output will show:
=== Swagger Coverage Report ===
APIs analyzed: User API, Product API, Order API
Total operations in spec(s): 24
Matched operations in Postman: 18
Coverage: 75.00%
Unmatched Spec operations:
- [User API] [GET] /users/{id}/profile (statusCode=404)
- [Product API] [POST] /products (statusCode=400)
- [Order API] [DELETE] /orders/{id} (statusCode=204)
The HTML report will include:
- API column in the operations table to identify the source API
- Combined API information in the report header
- Individual API breakdown for detailed analysis
- Node.js version 12+ (16+ recommended).
- NPM (or Yarn) installed.
- Clone or download this repository.
- Install dependencies by running:
npm install -g swagger-coverage-cli
You will need:
- OpenAPI/Swagger specification file (e.g.,
openapi.yamlorswagger.json) OR CSV documentation file following the specified format. - Postman collection file (JSON format), which you can export from the Postman app, OR a Newman run report (JSON format) generated by running your collection with Newman.
Note:
- If using a Postman collection, make sure it includes actual test scripts that assert or check specific status codes (e.g.,
pm.response.to.have.status(200)).- Folder-level tests are fully supported: tests defined at the folder level in your Postman collection will be automatically applied to all requests within that folder, ensuring comprehensive coverage calculation.
- If using a Newman report, the tool will extract actual response codes and test results from the execution data.
Use the following command:
npm swagger-coverage-cli <swaggerFile> <postmanCollectionOrNewmanReport> [options]
For Multiple APIs:
npm swagger-coverage-cli "api1.yaml,api2.yaml,api3.json" collection.json [options]
Examples:
With Postman Collection:
npm swagger-coverage-cli openapi.yaml collection.json --verbose --strict-query --strict-body
With Newman Report:
npm swagger-coverage-cli openapi.yaml newman-report.json --newman --verbose
Multiple APIs with Newman Report:
npm swagger-coverage-cli "users-api.yaml,products-api.yaml" newman-report.json --newman --output multi-api-report.html
Options:
--verbose: Display additional logs (helpful for debugging).--newman: Treat input file as Newman run report instead of Postman collection.--strict-query: Enforce strict checks on query parameters (e.g., required params,enum,pattern, etc.).--strict-body: Verify thatapplication/jsonrequest bodies in the spec match raw JSON bodies in Postman requests.--disable-spec-validation: Disable OpenAPI/Swagger spec validation (useful for specs with validation or reference issues).--output <file>: Customize the name of the HTML report file (default iscoverage-report.html).
npm swagger-coverage-cli -- <swaggerFile> <postmanCollectionOrNewmanReport> [options]
After execution, you will see:
- Console Output:
=== Swagger Coverage Report ===
Total operations in spec: 12
Matched operations in Postman: 9
Coverage: 75.00%
Unmatched operations:
- [DELETE] /items/{id} (statusCode=204)
- [PUT] /items/{id} (statusCode=400)
...
- HTML Report:
- A file named
coverage-report.html(or the name you provided with--output) is generated. - Open it in your browser to see a table of matched/unmatched operations with color highlights.
- A file named
Use standard OpenAPI/Swagger files in YAML or JSON format:
# Single OpenAPI specification
swagger-coverage-cli api-spec.yaml collection.json
# Multiple REST APIs
swagger-coverage-cli "api-v1.yaml,api-v2.yaml,legacy.json" collection.json
# With strict validation
swagger-coverage-cli openapi.yaml collection.json --strict-query --strict-bodySupported OpenAPI features:
- Path parameters (
/users/{id},/users/{userId}) - Query parameters with schema validation
- Request body validation (JSON, form-data, etc.)
- Multiple response status codes per operation
- OpenAPI v2 and v3 specifications
Analyze Protocol Buffer service definitions:
# Single gRPC service
swagger-coverage-cli user-service.proto collection.json
# Multiple gRPC services
swagger-coverage-cli "user.proto,order.proto,payment.proto" collection.json
# Mixed with OpenAPI
swagger-coverage-cli "rest-api.yaml,grpc-service.proto" collection.jsongRPC-specific features:
- Service and method extraction from
.protofiles - HTTP/2 path mapping (
/package.service/method) - Content-type validation (
application/grpc,application/grpc+proto) - Nested package support (
company.api.v1.UserService)
Example Postman request for gRPC:
{
"method": "POST",
"url": "{{grpcUrl}}/user.v1.UserService/GetUser",
"header": [
{ "key": "Content-Type", "value": "application/grpc" }
],
"body": {
"mode": "raw",
"raw": "{\"user_id\": \"123\"}"
}
}Analyze GraphQL schema definitions:
# Single GraphQL API
swagger-coverage-cli schema.graphql collection.json
# Multiple GraphQL schemas
swagger-coverage-cli "user-schema.gql,product-schema.graphql" collection.json
# Full stack coverage
swagger-coverage-cli "api.yaml,service.proto,schema.graphql" collection.jsonGraphQL-specific features:
- Query, mutation, and subscription extraction
- Argument analysis with type information
- Union and interface type support
- Nested type relationship mapping
Example Postman request for GraphQL:
{
"method": "POST",
"url": "{{apiUrl}}/graphql",
"header": [
{ "key": "Content-Type", "value": "application/json" }
],
"body": {
"mode": "raw",
"raw": "{\"query\": \"query GetUser($id: ID!) { user(id: $id) { id name email } }\", \"variables\": {\"id\": \"123\"}}"
}
}Use CSV format for flexible API documentation:
# CSV-based API documentation
swagger-coverage-cli api-docs.csv collection.json
# Mixed with other formats
swagger-coverage-cli "api.yaml,docs.csv,service.proto" collection.jsonCSV format columns:
method: HTTP method (GET, POST, etc.)path: API endpoint pathstatusCode: Expected response status codedescription: Operation descriptiontags: Comma-separated tags for grouping
Microservices Architecture:
# Complete microservices stack
swagger-coverage-cli "gateway.yaml,user-service.proto,analytics.graphql,docs.csv" tests.json
# Per-team analysis
swagger-coverage-cli "team-a-api.yaml,team-b-service.proto" team-tests.jsonCI/CD Integration:
# Production coverage check
swagger-coverage-cli "$(find apis -name '*.yaml' -o -name '*.proto' -o -name '*.graphql' | tr '\n' ',')" collection.json --output coverage-$(date +%Y%m%d).htmlswagger-coverage-cli uses precise mathematical formulas to calculate API test coverage. Understanding these formulas helps you interpret coverage reports and set appropriate coverage targets.
The core coverage calculation is based on the ratio of matched operations to total operations in your API specification:
Coverage (%) = (Matched Operations / Total Operations) × 100
Where:
- Total Operations = Number of unique operation-status code combinations in your API specification
- Matched Operations = Number of operations that have corresponding tests in your Postman collection or Newman report
An operation is considered matched when ALL of the following criteria are satisfied:
- HTTP Method Match:
specOperation.method === postmanRequest.method - Path Pattern Match: Postman URL matches the Swagger path pattern (with parameter substitution)
- Status Code Verification: At least one expected status code from the spec is tested
- Query Parameters (if
--strict-queryenabled): Required parameters are present and valid - Request Body (if
--strict-bodyenabled): JSON body structure matches specification
Standard Coverage = Matched Operations / Total Spec Operations × 100
When using Postman collections, coverage is calculated based on:
- Static analysis of test scripts
- Expected status codes extracted from
pm.response.to.have.status(code)assertions - Request structure validation
When using Newman reports, coverage calculation includes:
- Actual execution data from real test runs
- Response codes from executed requests
- Assertion results (passed/failed tests)
Newman Coverage = (Executed Operations with Passing Tests / Total Spec Operations) × 100
For multiple API specifications, coverage is calculated as:
Combined Coverage = (Sum of Matched Operations across all APIs / Sum of Total Operations across all APIs) × 100
Each API operation is identified by: {API_Name}:{Method}:{Path}:{StatusCode}
Coverage can also be calculated per tag/group:
Tag Coverage = (Matched Operations in Tag / Total Operations in Tag) × 100
- API Specification: 20 operations (GET /users/200, POST /users/201, GET /users/404, etc.)
- Postman Tests: 15 operations covered
- Coverage: 15/20 × 100 = 75%
- API Specification: 18 operations
- Newman Report: 8 operations with actual execution data
- Postman Collection: 2 operations with static test scripts
- Newman Coverage: 8/18 × 100 = 44.44%
- Postman Coverage: 2/18 × 100 = 11.11%
- User API: 10 operations (7 matched) = 70% coverage
- Product API: 15 operations (12 matched) = 80% coverage
- Order API: 8 operations (5 matched) = 62.5% coverage
- Combined Coverage: (7+12+5)/(10+15+8) × 100 = 24/33 × 100 = 72.73%
Beyond basic percentage, consider these quality indicators:
- Status Code Coverage: Percentage of documented status codes that are tested
- Method Distribution: Coverage across different HTTP methods (GET, POST, PUT, DELETE)
- Critical Path Coverage: Coverage of business-critical API endpoints
- Error Case Coverage: Percentage of error scenarios (4xx, 5xx) that are tested
- Minimum Coverage: Aim for 80%+ coverage for production APIs
- Critical Endpoints: Ensure 100% coverage for authentication, payment, and data modification endpoints
- Error Testing: Include at least 50% of documented error cases
- Use Newman Reports: For more accurate coverage analysis in CI/CD pipelines
Sometimes API specifications may have validation errors or broken references, especially in legacy systems or during development. The --disable-spec-validation flag allows you to analyze coverage even when specs have issues.
Use --disable-spec-validation when:
- Working with legacy APIs that have incomplete specifications
- Specs contain broken
$refreferences that can't be resolved - External references aren't available in your CI/CD environment
- You need quick coverage analysis without fixing all spec issues first
- API specifications are still in development
# Standard usage (validation enabled - will fail on invalid specs)
swagger-coverage-cli api.yaml collection.json
# Disable validation for specs with issues
swagger-coverage-cli api.yaml collection.json --disable-spec-validation
# Works with all other flags
swagger-coverage-cli api.yaml collection.json --disable-spec-validation --verbose --strict-bodyWithout the flag (validation enabled):
$ swagger-coverage-cli broken-spec.yaml collection.json
Error: Token "NonExistentSchema" does not exist.With the flag (validation disabled):
$ swagger-coverage-cli broken-spec.yaml collection.json --disable-spec-validation
=== Swagger Coverage Report ===
Total operations in spec(s): 12
Matched operations in Postman/Newman: 9
Coverage: 75.00%
HTML report saved to: coverage-report.html- When validation is disabled, the tool parses the spec without validating references
- Coverage can still be calculated for the operations that are defined
- The tool won't catch structural issues in the spec
- Default behavior (with validation enabled) ensures spec quality
swagger-coverage-cli tries to match each operation from the spec with a request in Postman. An operation is considered covered if:
-
HTTP Method matches exactly (
GET,POST,PUT,DELETE, etc.). -
Path:
- The path pattern from Swagger (e.g.,
/users/{id}) is converted to a regex (like^/users/[^/]+$). - The Postman request URL (minus any base URL placeholders like
{{baseUrl}}) must match that regex.
- The path pattern from Swagger (e.g.,
-
Status Code:
- If the spec operation has a specific status code (e.g.,
200,404), the CLI checks the Postman test scripts to see if that status code is asserted (e.g.,pm.response.to.have.status(200)).
- If the spec operation has a specific status code (e.g.,
-
Query Parameters (only if
--strict-queryis enabled):- If the spec says a query parameter is required and has certain constraints (e.g.,
enum,pattern,type), the tool verifies that the Postman request includes that parameter and meets the constraints.
- If the spec says a query parameter is required and has certain constraints (e.g.,
-
Request Body (only if
--strict-bodyis enabled):- If the spec says
requestBodyincludesapplication/json, the CLI checks if the Postman request body is raw JSON and can be parsed without errors.
- If the spec says
If all criteria are satisfied, the operation is matched (covered). Otherwise, it’s reported as unmatched.
Smart endpoint mapping is an advanced feature that significantly improves coverage accuracy by using intelligent algorithms to match endpoints. It is enabled by default in all operations.
- Enhanced path matching for better parameter recognition
- Status Code Prioritization: Prioritizes successful (2xx) status codes over error codes
- Enhanced Path Matching: Better handling of parameter variations and naming conventions
- Confidence Scoring: Assigns quality scores to matches (0.0-1.0)
- Multi-API Support: Works seamlessly with microservices and complex architectures
# Smart mapping is enabled by default
swagger-coverage-cli api-spec.yaml collection.json --verbose
# Output shows smart mapping statistics:
# Smart mapping: 6 primary matches, 3 secondary matches
# Coverage: 50.00%Status Code Intelligence:
# API defines multiple status codes
GET /users:
responses:
'200': { description: Success }
'400': { description: Bad Request }
'500': { description: Server Error }
# Postman only tests success case
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
# Smart mapping result:
# ✅ Primary Match: GET /users (200) - Matched
# ❌ Secondary: GET /users (400, 500) - Unmatched but deprioritizedEnhanced Path Matching:
# API Spec: /users/{userId}/profile
# Postman: /users/123/profile
# Result: ✅ Intelligent parameter matching (confidence: 1.0)For comprehensive examples, use cases, and implementation details, see: 📖 Smart Mapping Examples & Use Cases
This document covers:
- 25+ detailed examples across 8 categories
- Real-world API scenarios (CRUD, microservices, versioning)
- Edge cases and error handling
- Performance testing and best practices
- CLI integration examples
Swagger/OpenAPI/.csv:
- JSON or YAML
- OpenAPI v2 (Swagger 2.0) or OpenAPI v3.x
- CSV: API documentation can be provided in CSV format following the specified structure.
Postman:
- Postman collection in JSON format (v2.1 or higher recommended).
- Typically exported via the Postman app: File → Export → Collection.
Newman Reports:
- Newman run report in JSON format.
- Generated by running:
newman run collection.json --reporters json --reporter-json-export newman-report.json
- Generated by running:
The tool supports two types of input for test data:
- What it is: The collection definition containing requests and test scripts
- Pros:
- Contains all test logic and assertions
- Can extract expected status codes from test scripts
- Supports folder-level tests: Tests defined at the folder level are automatically applied to all requests within that folder and its subfolders
- Cons:
- No actual execution data
- Relies on parsing test scripts to understand expected outcomes
- What it is: Actual execution results from running a Postman collection with Newman
- Pros:
- Contains real execution data with actual response codes
- Includes assertion results (passed/failed)
- More accurate representation of what was actually tested
- Includes response times and execution metadata
- Cons:
- Requires an additional step to generate the report
Recommendation: Use Newman reports when possible for more accurate coverage analysis, especially in CI/CD pipelines where collections are actually executed.
swagger-coverage-cli fully supports folder-level tests in Postman collections. Tests defined at the folder level are automatically applied to all requests within that folder and its subfolders, making it easy to apply common test assertions across multiple endpoints.
When you define tests at the folder level in your Postman collection:
- The tests are extracted from the folder's event scripts
- Status codes and assertions are identified from the test scripts
- These tests are automatically combined with any request-level tests
- All requests within the folder (and nested folders) inherit these tests
Consider this Postman collection structure:
{
"name": "Users API",
"item": [
{
"name": "Get User",
"request": { "method": "GET", "url": "/users/1" }
},
{
"name": "Create User",
"request": { "method": "POST", "url": "/users" }
}
],
"event": [
{
"listen": "test",
"script": {
"exec": [
"pm.test('Status code is 200 or 201', function () {",
" pm.expect(pm.response.code).to.be.oneOf([200, 201]);",
"});"
]
}
}
]
}In this example:
- The folder "Users API" has a test that checks for status codes 200 or 201
- Both "Get User" and "Create User" requests will inherit these status codes
- This means both endpoints will be counted as testing status codes 200 and 201
- Reduced duplication: Define common tests once at the folder level
- Better coverage: Ensures all requests within a folder test common scenarios
- Easier maintenance: Update folder-level tests to affect all child requests
- Nested support: Folder-level tests are inherited through multiple levels of nesting
Folder-level tests support the same patterns as request-level tests:
pm.response.to.have.status(200)pm.expect(pm.response.code).to.eql(201)pm.expect(pm.response.code).to.be.oneOf([200, 201, 204])- And other common Postman test assertions
The folder-level test feature handles various edge cases:
- Empty folders: Folders with no requests are handled gracefully
- Deep nesting: Supports unlimited levels of nested folders with test inheritance at each level
- Missing events: Requests or folders without event properties work correctly
- Mixed patterns: Multiple test patterns in the same folder or request are combined
- Null/undefined events: Folders with null or undefined event properties are handled safely
- Non-test events: Only "test" events are processed; other events (like "prerequest") are ignored
In addition to traditional OpenAPI/Swagger specifications, swagger-coverage-cli supports API documentation provided in a CSV format. This allows for a more flexible and easily editable documentation process, especially for teams that prefer spreadsheet-based documentation.
Your CSV file should adhere to the following structure to ensure compatibility with swagger-coverage-cli:
- Header Row: The first row must contain the following columns in exactly this order:
METHOD,URI,NAME,STATUS CODE,BODY,TAGS
- Data Rows: Each subsequent row represents an API endpoint response with data corresponding to the headers.
-
METHOD
- Type: String
- Description: The HTTP method used for the API endpoint (e.g.,
GET,POST,PUT,DELETE).
-
URI
- Type: String
- Description: The endpoint's URI path (e.g.,
/api/products).
-
NAME
- Type: String
- Description: A descriptive name for the API endpoint action (e.g.,
getProducts).
-
STATUS CODE
- Type: Integer
- Description: The HTTP status code returned by the API (e.g.,
200,400).
-
BODY
- Type: String (JSON format)
- Description: The response body returned by the API in JSON format. Ensure that JSON strings are properly escaped.
-
TAGS
- Type: String
- Description: Tags associated with the endpoint for categorization and filtering (e.g.,
products).
Below is an example of how your documentation.csv should be structured:
METHOD,URI,NAME,STATUS CODE,BODY,TAGS
GET,/api/products,getProducts,200,"{\"products\":\"updated\"}","products"
GET,/api/products,getProducts,400,"{\"error\":\"Invalid query parameters\"}","products"
Contributions are welcome! Please:
- Fork this repo
- Create a branch for your feature or fix (
git checkout -b feature/something) - Commit your changes
- Open a Pull Request
Feel free to add tests in the test/ folder to cover any new logic.
This project is licensed under the MIT License.
Happy testing! Feel free to open issues or submit pull requests for any improvements.

