Failing CI on Breaking OpenAPI Changes

This guide is part of Breaking Change Detection & Spec Diffing within SDK Generation & Changelog Automation. It turns the diff report into something with teeth: a required check that stops an accidental break reaching a release, without becoming the kind of obstacle teams disable. The distinction between those two outcomes is almost entirely about how the gate is designed rather than about the detection itself.

Merge gate routing by classification Patch and minor changes pass automatically, while a major change requires an approval label that leaves a permanent record. classification from the diff patch / minor check passes, comment posted major approved-breaking label? absent → check fails merge label added by an owner permanent record of who, when, and why

Problem & Context

Detection without enforcement changes nothing. A job that prints a breaking-change report into a log nobody opens is indistinguishable from no job at all, and the first accidental break ships exactly as it would have.

The obvious remedy — fail the build whenever a breaking change is detected — has a predictable half-life. Breaking changes are not always mistakes; a major version exists so that some of them can happen on purpose. When the gate blocks a deliberate, already-agreed break at an inconvenient moment, somebody makes the job advisory, or adds continue-on-error, or merges with the check bypassed. From then on the detector is decoration, and it will be decoration precisely when it matters.

A gate needs a legitimate path through it. An approval label provides one that costs seconds, requires a named person, and leaves a permanent record on the pull request — which turns out to be worth as much as the block itself, because the question people ask six months later is rarely “why did this break?” and almost always “did anyone decide this, or did it slip?”

Step-by-Step Solution

1. Classify the change in CI

Run the diff against the merge base and reduce it to one word:

      - name: Classify
        id: classify
        run: |
          set +e
          oasdiff breaking /tmp/base.bundled.yaml /tmp/head.bundled.yaml \
            --exclude-elements description,examples \
            --warn-ignore .oasdiff-ignore.yaml \
            --format text > /tmp/breaking.txt
          BREAKING=$?
          set -e
          oasdiff changelog /tmp/base.bundled.yaml /tmp/head.bundled.yaml \
            --exclude-elements description,examples \
            --format text > /tmp/changelog.txt

          if   [ "$BREAKING" -ne 0 ];      then echo 'level=major' >> "$GITHUB_OUTPUT"
          elif [ -s /tmp/changelog.txt ];  then echo 'level=minor' >> "$GITHUB_OUTPUT"
          else                                  echo 'level=patch' >> "$GITHUB_OUTPUT"
          fi

2. Comment the classification on the pull request

Post one updating comment so reviewers read the verdict before the diff:

      - uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const level = '${{ steps.classify.outputs.level }}';
            const marker = '<!-- spec-diff -->';
            const changes = fs.readFileSync('/tmp/changelog.txt', 'utf8').trim()
              || 'No contract changes.';
            const extra = level === 'major'
              ? `\n\n**⚠ Breaking.** Add the \`approved-breaking\` label to proceed.\n\n` +
                '```\n' + fs.readFileSync('/tmp/breaking.txt', 'utf8') + '\n```'
              : '';
            const body = `${marker}\n### Contract diff — \`${level}\`\n\n\`\`\`\n${changes}\n\`\`\`${extra}`;
            const {data: cs} = await github.rest.issues.listComments(
              {...context.repo, issue_number: context.issue.number});
            const prev = cs.find(c => c.body.includes(marker));
            if (prev) await github.rest.issues.updateComment(
              {...context.repo, comment_id: prev.id, body});
            else await github.rest.issues.createComment(
              {...context.repo, issue_number: context.issue.number, body});

Commenting on every level, not only on breaking ones, is deliberate: reviewers learn to glance at one word and spend their attention on whether that word is acceptable.

3. Gate on an approval label rather than failing outright

      - name: Require approval for a breaking change
        if: steps.classify.outputs.level == 'major'
        env:
          LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }}
        run: |
          if echo "$LABELS" | grep -q 'approved-breaking'; then
            echo "Breaking change approved via label."
          else
            echo "::error::Breaking contract change without the approved-breaking label."
            exit 1
          fi

4. Require a prior deprecation before a removal

A correctly classified removal is still a bad removal if consumers were never warned. Check the previously released contract:

# Every operation removed in this change must have been deprecated in the last release.
comm -23 \
  <(yq -r '.paths | keys[]' /tmp/released.bundled.yaml | sort) \
  <(yq -r '.paths | keys[]' /tmp/head.bundled.yaml     | sort) \
| while read -r removed; do
    was_deprecated=$(yq -r ".paths[\"${removed}\"][] | select(.deprecated == true) | .operationId" \
                     /tmp/released.bundled.yaml | head -1)
    if [ -z "${was_deprecated}" ]; then
      echo "::error::${removed} was removed without ever being deprecated"
      exit 1
    fi
    echo "OK   ${removed} was deprecated before removal"
  done

5. Make the check required and keep it fast

Mark the job a required status check in branch protection, and keep it under a minute. A gate that adds five minutes to every pull request generates pressure to remove it regardless of its value.

Hard failure versus approval gate over time A hard failure with no legitimate path is eventually bypassed or disabled, while an approval gate survives because deliberate changes have a fast route through it. hard failure, no override blocks a deliberate break urgent release, no path continue-on-error added detector is now decoration approval label blocks a deliberate break owner adds the label seconds, not an argument merge proceeds, record kept gate still trusted next week

Complete Working Example

# .github/workflows/contract-gate.yml
name: Contract gate
on:
  pull_request:
    types: [opened, synchronize, reopened, labeled, unlabeled]
    paths: ['openapi/**']

permissions:
  contents: read
  pull-requests: write

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0          # a shallow clone has no base commit to diff

      - uses: actions/setup-node@v4
        with: { node-version: '20', cache: 'npm' }
      - run: npm ci

      - name: Install oasdiff
        run: |
          curl -fsSL https://raw.githubusercontent.com/oasdiff/oasdiff/main/install.sh \
            | sh -s -- v1.11.7

      - name: Bundle both sides
        run: |
          git show "origin/${{ github.base_ref }}:openapi/api.yaml" > /tmp/base.yaml
          npx @redocly/cli@2 bundle /tmp/base.yaml   -o /tmp/base.bundled.yaml
          npx @redocly/cli@2 bundle openapi/api.yaml -o /tmp/head.bundled.yaml

      - name: Classify
        id: classify
        run: |
          set +e
          oasdiff breaking /tmp/base.bundled.yaml /tmp/head.bundled.yaml \
            --exclude-elements description,examples \
            --warn-ignore .oasdiff-ignore.yaml --format text > /tmp/breaking.txt
          BREAKING=$?
          set -e
          oasdiff changelog /tmp/base.bundled.yaml /tmp/head.bundled.yaml \
            --exclude-elements description,examples --format text > /tmp/changelog.txt
          if   [ "$BREAKING" -ne 0 ];     then echo 'level=major' >> "$GITHUB_OUTPUT"
          elif [ -s /tmp/changelog.txt ]; then echo 'level=minor' >> "$GITHUB_OUTPUT"
          else                                 echo 'level=patch' >> "$GITHUB_OUTPUT"
          fi

      - name: Comment the verdict
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const level = '${{ steps.classify.outputs.level }}';
            const marker = '<!-- spec-diff -->';
            const changes = fs.readFileSync('/tmp/changelog.txt','utf8').trim()
              || 'No contract changes.';
            const extra = level === 'major'
              ? '\n\n**⚠ Breaking.** Add the `approved-breaking` label to proceed.\n\n```\n'
                + fs.readFileSync('/tmp/breaking.txt','utf8') + '\n```'
              : '';
            const body = `${marker}\n### Contract diff — \`${level}\`\n\n\`\`\`\n${changes}\n\`\`\`${extra}`;
            const {data: cs} = await github.rest.issues.listComments(
              {...context.repo, issue_number: context.issue.number});
            const prev = cs.find(c => c.body.includes(marker));
            if (prev) await github.rest.issues.updateComment(
              {...context.repo, comment_id: prev.id, body});
            else await github.rest.issues.createComment(
              {...context.repo, issue_number: context.issue.number, body});

      - name: Require approval for a breaking change
        if: steps.classify.outputs.level == 'major'
        env:
          LABELS: ${{ toJson(github.event.pull_request.labels.*.name) }}
        run: |
          echo "$LABELS" | grep -q 'approved-breaking' \
            || { echo '::error::Breaking change without the approved-breaking label.'; exit 1; }

Including labeled and unlabeled in the trigger types is what makes the label usable. Without them, adding the label does not re-run the check, and the pull request stays blocked by a stale failing status until someone pushes an empty commit.

Gotchas & Edge Cases

The check never re-runs after the label is added. As above — add labeled and unlabeled to the types list. This is the single most common reason teams conclude the label approach “does not work”.

A shallow clone breaks the diff. fetch-depth: 0 is required; a depth-one checkout has no base commit and the bundle step fails with a confusing missing-file error.

Label permissions are broader than you think. Anyone who can label a pull request can approve a breaking change. If that set is wider than the set of people who should make that call, enforce it in the job by checking the actor against a team rather than relying on repository permissions alone.

The gate cannot see changes that already merged. Diffing against the merge base means breaking changes introduced earlier in the release cycle are invisible to this pull request. Add a separate release-time check that diffs against the last released tag, which is the question “is this release breaking?” rather than “is this change breaking?”.

Silence on non-breaking changes trains people to ignore the comment. Post the classification every time. A reviewer who only ever sees the comment when something is wrong will stop reading it between incidents; one who sees patch, minor, patch, major learns to notice the last one.

Pair the label with two conventions and the gate documents itself. First, require the pull request description to state the migration path whenever the label is present — a reviewer should not have to reconstruct what consumers must do, and the text you write there is the first draft of the release note. Second, keep the label on the merged pull request rather than removing it after approval, so the record survives. Between the label, the description, and the classification comment, a future reader has the complete story of a breaking change without needing anyone’s memory.

The record a merged breaking change leaves The classification comment, the approval label and the migration note in the description together document what broke, who approved it and what consumers must do. three artefacts, one complete record classification comment what changed, and that it was breaking written by the machine approval label who accepted it, and when kept after merge migration note what consumers must actually do becomes the release note

FAQ

Should the gate fail the build outright instead of using a label?

No. A gate with no legitimate path through it gets disabled the first time it blocks something urgent, and it will be disabled permanently rather than for that one occasion. An approval label keeps every intentional break visible and recorded while still stopping the accidental ones.

How do I stop the gate blocking on my own compatibility policy?

Encode the policy in the diff tool’s ignore file with a stated reason, rather than lowering severity or bypassing the job. The gate should reflect your published policy exactly, or it is measuring something other than what you promised consumers.

What if a breaking change must ship urgently?

The label path is the urgent path — it takes seconds and leaves a permanent record of who approved it. That is precisely why it is preferable to a hard failure that people have to route around under time pressure.