Contract Testing OpenAPI with Schemathesis

This guide is part of Mock Servers & Contract Testing from OpenAPI within OpenAPI & AsyncAPI Schema Authoring. It covers the producer side: proving that a running API actually behaves the way its specification says. The prompt for this is usually a support ticket that turns out not to be a bug at all — the API works exactly as intended, and the documentation has quietly described something else for two releases.

Property-based contract testing loop The specification supplies both the input generator and the response oracle, so every generated request is checked against what the document promises. openapi.yaml generator AND oracle generated requests from parameter schemas the running API a disposable instance failures with a reproduction command responses checked against the same document that generated the input

Problem & Context

A specification that nothing executes is a description of intent. It can be perfectly valid OpenAPI, lint cleanly, render beautifully, and describe an API that no longer exists — because nothing in the pipeline ever compares it to the running system.

Hand-written contract tests help and do not scale. Someone writes assertions for the operations they were thinking about, and the tests then cover exactly the cases that author imagined. The failures that reach production are, by definition, the ones nobody imagined: the empty string in a path parameter, the integer at its documented boundary, the optional field that is null rather than absent.

Property-based testing inverts this. The specification already contains a machine-readable description of every valid input — that is what the parameter schemas are — so a generator can produce inputs nobody thought of, and the same document supplies the oracle for judging every response. The result is broad, shallow coverage of the entire documented surface, which is precisely the coverage hand-written tests are worst at.

Step-by-Step Solution

1. Point it at a bundled spec and a real base URL

python -m pip install 'schemathesis==3.39.5'
npx @redocly/cli@2 bundle openapi/api.yaml -o dist/api.bundled.yaml

schemathesis run dist/api.bundled.yaml \
  --base-url https://staging.api.acme.com \
  --header "Authorization: Bearer ${STAGING_TOKEN}"

2. Turn on every check

The default check set validates response bodies against their schemas. --checks all adds the ones that find drift:

schemathesis run dist/api.bundled.yaml \
  --base-url https://staging.api.acme.com \
  --checks all \
  --header "Authorization: Bearer ${STAGING_TOKEN}"

That adds status-code conformance (was this status documented at all?), content-type conformance, header conformance, and a server-error check that treats any 500 for a well-formed request as a failure.

3. Triage the first run by check type

The first run against a mature API produces a lot. Sort it by check name before fixing anything, because the two categories have different owners:

FAILED  GET /invoices/{id} [status_code_conformance]
    Received: 500
    Documented: 200, 404
    Reproduce with:
      curl -X GET 'https://staging.api.acme.com/invoices/%00'

FAILED  GET /invoices [response_schema_conformance]
    'next_cursor' is not of type 'string'
    Received: {"next_cursor": null, "data": []}
    Reproduce with:
      curl -X GET 'https://staging.api.acme.com/invoices?limit=1'

The first is an implementation bug: a null byte in a path parameter should be a 400, not a crash. The second is a documentation bug: the API legitimately returns null for next_cursor on the last page, and the schema should say type: [string, 'null']. Fix each in the place it actually belongs.

4. Cap the example budget in CI

Property-based testing is unbounded work. Cap it on pull requests and run the deep sweep on a schedule:

# On every pull request — fast enough that nobody routes around it
schemathesis run dist/api.bundled.yaml \
  --base-url "${STAGING_BASE_URL}" \
  --checks all \
  --hypothesis-max-examples 40 \
  --hypothesis-deadline 5000 \
  --header "Authorization: Bearer ${STAGING_TOKEN}"

5. Reproduce and regression-test each finding

Every failure prints the exact request that caused it. Copy it into a permanent test so the bug cannot return quietly:

curl -i -X GET 'https://staging.api.acme.com/invoices/%00'
Triage by check type Status-code and server-error failures are implementation bugs, while schema and content-type failures are usually documentation gaps. a failing check which owner? server_error status_code_conformance fix the implementation response_schema_conformance content_type_conformance usually fix the document either way, keep the repro

Complete Working Example

A workflow with a fast gate on pull requests and a deep nightly sweep, both reporting into the same place:

# .github/workflows/contract-test.yml
name: Contract test
on:
  pull_request:
    paths: ['openapi/**', 'src/**']
  schedule:
    - cron: '0 3 * * *'       # deep sweep, nightly

jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }

      - run: npm ci
      - run: python -m pip install 'schemathesis==3.39.5'

      - name: Bundle the contract
        run: npx @redocly/cli@2 bundle openapi/api.yaml -o dist/api.bundled.yaml

      # Fast on PRs, exhaustive nightly. Same checks either way.
      - name: Set the example budget
        id: budget
        run: |
          if [ "${{ github.event_name }}" = "schedule" ]; then
            echo "examples=400" >> "$GITHUB_OUTPUT"
          else
            echo "examples=40"  >> "$GITHUB_OUTPUT"
          fi

      - name: Run contract tests
        env:
          TOKEN: ${{ secrets.STAGING_TOKEN }}
        run: |
          schemathesis run dist/api.bundled.yaml \
            --base-url "${{ vars.STAGING_BASE_URL }}" \
            --checks all \
            --hypothesis-max-examples "${{ steps.budget.outputs.examples }}" \
            --hypothesis-deadline 5000 \
            --exclude-deprecated-operations \
            --exclude-method DELETE \
            --header "Authorization: Bearer ${TOKEN}" \
            --junit-xml report.xml \
            --show-trace

      - name: Publish the report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: contract-report-${{ github.run_id }}
          path: report.xml

Two exclusions in that command are deliberate rather than cosmetic. --exclude-method DELETE keeps generated traffic from destroying fixtures the rest of the suite depends on — those operations belong in a mock-backed test instead. --exclude-deprecated-operations stops the job reporting drift on endpoints you have already committed to removing, which would otherwise be noise you cannot act on.

Expected output on a healthy run:

================== Schemathesis test session starts ==================
Schema location: dist/api.bundled.yaml
Base URL: https://staging.api.acme.com
Specification version: Open API 3.1.0
Collected API operations: 24

GET /invoices .                                              [  4%]
GET /invoices/{id} .                                         [  8%]
...
============================ SUMMARY =================================
Performed checks:
    not_a_server_error                    960 / 960 passed  PASSED
    status_code_conformance               960 / 960 passed  PASSED
    content_type_conformance              960 / 960 passed  PASSED
    response_schema_conformance           960 / 960 passed  PASSED

======================= 24 passed in 41.28s ==========================

Gotchas & Edge Cases

Every operation returns 401 and the run is meaningless. No credential reached the tool, or the one supplied has expired. The symptom is a uniform wall of status_code_conformance failures, which looks alarming and says nothing. Verify with a single manual curl before investigating further, and use a dedicated test principal rather than a personal token that will expire without warning.

Generated writes fill the environment. Property-based testing will happily create thousands of records. Point it at an environment with a reset schedule, exclude destructive methods, and treat any operation you must exclude as a candidate for mock-backed coverage instead.

Unconstrained schemas generate pathological inputs. A bare type: string produces arbitrary Unicode, including inputs that time out a naive regex or blow a database column limit. Sometimes that is a genuine finding; often it means the schema should carry maxLength, pattern, or format. Tightening the schema improves the tests and the documentation together.

Stateful sequences are not covered. Schemathesis tests operations largely independently, so “create an invoice, then fetch it” is not exercised by default. Its stateful testing mode uses OpenAPI links to chain operations, and is worth enabling once the flat checks are green — but it needs links declared in the specification, which most documents lack.

A flaky staging environment reads as contract drift. A timeout or a 502 from an overloaded test instance is reported as a check failure, and if that happens often people stop trusting the job. Give the run a generous deadline, retry transport errors, and keep the deep sweep on a schedule where a slow environment costs nothing.

Agree who owns a failure before you make the job blocking. The first week produces a flood, and if nobody owns triage the team’s instinct will be to mark the job advisory — after which it is decoration and the next real drift ships unnoticed. Settle the default in advance: a server_error or status_code_conformance failure is an implementation bug, a response_schema_conformance failure is a documentation bug, and either one blocks the merge. Having a rule matters far more than which rule you pick, because the alternative is a per-failure argument that the gate always loses.

Clear the backlog before turning the gate on. Running this for the first time against a mature API surfaces years of accumulated drift, none of it caused by the pull request that happens to be open. Blocking that pull request on somebody else’s history is the fastest way to lose the team’s goodwill. Run it once against staging, file the findings as their own piece of work, clear them, and only then make the check required. From that point every failure genuinely belongs to the change that introduced it — which is the condition under which people fix them the same day instead of arguing about scope.

Two-phase adoption Run the checks in reporting mode to clear the historical backlog first, then make them blocking so every later failure belongs to the change that caused it. phase 1 — reporting only clear the historical backlog nobody is blocked meanwhile phase 2 — blocking every failure is new drift owned by the change that caused it backlog at zero

FAQ

Why does the first run report hundreds of failures?

Almost all of them are undocumented status codes rather than schema violations. The API returns responses the specification never mentioned — a 429, a validation 400, an auth 403 — which is real drift, but is fixed in the document rather than the code for most cases. Triage by check name before touching anything.

Will it create real data?

Yes, if you point it at an environment with write operations. Run it against a disposable environment with a reset schedule, and exclude genuinely destructive operations by method or tag, as the workflow above does.

How many examples should I generate per operation?

Around forty on pull requests, which finishes in a couple of minutes and catches drift, and several hundred on a nightly run where the time costs nothing. The obscure findings — boundary integers, odd Unicode, unusual parameter combinations — come from the nightly job.