Impact GateGuides
oasdiff vs API consumer impact analysis: what each report tells you
A grounded comparison of OpenAPI contract diffing and repository-level consumer impact analysis, including how to use both in a pull request review.
On this page
They answer different review questions#
API compatibility review usually has two separate questions: what changed in the published contract, and which callers rely on the changed behavior? A contract diff and consumer impact analysis can be used together because they operate on different evidence.
| Review question | Contract diff | Consumer impact analysis |
|---|---|---|
| What changed between base and head? | Compares the two OpenAPI documents and identifies contract differences. | Uses the changed operation or schema as the starting point for tracing caller evidence. |
| Who may be affected? | The diff alone does not identify the consumer repositories selected for review. | Inspects selected, indexed consumer repositories and links findings to source references. |
| Which field or request assumption matters? | Reports the changed contract location and before/after values when supported. | Checks supported call sites, field reads, request construction, and related code paths. |
| What remains uncertain? | Depends on the diff tool's output and the contract scope being compared. | Reports coverage notes, possible/cannot_tell results, unresolved host association, and missing repository or runtime scope. |
What oasdiff contributes in this workflow#
The current site documentation describes oasdiff as an open-source tool that lists breaking-change rules and runs in CI. In that role, it is useful for the contract-diff step: compare the provider document at the PR base and head, then review the reported contract changes.
- It gives the team a contract-focused view of paths, parameters, responses, schemas, and other OpenAPI changes in the documents being compared.
- It can run as part of CI, so a provider repository can surface a contract difference during pull request checks.
- Its output is the starting point for the downstream review. The team still needs a way to identify callers, inspect their source, and account for consumers outside the contract repository.
What consumer impact analysis contributes#
Impact Gate's consumer analysis starts with supported OpenAPI 3 contracts and selected GitHub repositories. It indexes provider and consumer snapshots, finds supported JavaScript, TypeScript, and Java HTTP client references, and publishes evidence tied to files, lines, and commits.
- It can connect a changed endpoint, request parameter, status, response field, type, enum, nullability, format, or constraint to a supported consumer reference.
- It distinguishes verified, possible/cannot_tell, not_affected, and unresolved coverage rather than treating an absent match as proof of no usage.
- It can combine static relationships with imported Prometheus request counts or Datadog indexed spans. A runtime dependency edge requires explicit caller identity and service mapping.
- It keeps the GitHub PR's exact base/head comparison separate from the indexed consumer snapshot, so the report can show both change commits and consumer evidence commits.
A combined pull request workflow#
Compare the provider contracts
Run the contract diff on the PR's base and head OpenAPI documents. Record the changed operations and the exact contract locations.
Confirm Impact Gate scope
Select the provider and relevant consumer repositories, wait for indexing, and inspect Coverage and indexed commit SHAs.
Trace downstream use
Use the changed operation to inspect consumer calls, request construction, response field access, and service identity in source.
Add runtime context
If connected, compare matched route activity and caller identity with static references. Treat missing, sampled, stale, or unmatched data as unknown.
Review and coordinate
Use the GitHub report to assign owners, choose a compatibility or migration approach, and record repositories or clients that remain outside the evidence.
Re-run at the final head
After pushing changes, confirm the report's analyzed PR head and review the remaining contract, consumer, and coverage findings before merge.
Which should you use?#
| Situation | Useful first step | Then ask |
|---|---|---|
| The provider team needs a quick contract check | Run a contract diff in CI. | Which selected consumers call the changed interface? |
| The provider has many separate client repositories | Index the provider and known consumers for consumer evidence. | Is the repository set complete, and what is the latest indexed commit? |
| A field or status changed but the route still exists | Read the contract diff location. | Which callers read, map, send, or branch on that value? |
| No static consumer was found | Check Coverage and repository scope, then consult runtime and owners. | Are generated, dynamic, external, scheduled, or unregistered clients missing? |
| The change is a removal or deprecation | Use the deprecation evidence checklist and runtime coverage. | Has the migration window completed, and is the evidence fresh enough for a removal PR? |
Limits of the comparison#
- A contract diff cannot see behavior-only implementation changes when the OpenAPI document stays the same. Consumer analysis starts from the supported contract and can likewise miss undocumented behavior changes.
- Consumer analysis is limited by selected repositories, supported language/client patterns, readable source, route normalization, and service/host identity. Dynamic, opaque, external, or non-HTTP usage can remain unknown.
- Prometheus and Datadog evidence is aggregate runtime context: Prometheus reports estimated counter increases, while Datadog reports indexed spans. Neither replaces the contract diff or source review.
- Neither workflow is a compatibility certification. Impact Gate is warn-only, and merge decisions remain with the engineering team.