Broken Link Checking in Docs CI

This guide is part of Docs CI/CD & Preview Deployments within Developer Portal Frameworks & UI Setup. It sets up an automated gate that refuses to merge documentation containing a dead link. This becomes urgent the first time an operation is renamed: every deep link to it across guides, blog posts, and support replies breaks at once, and nothing in the build noticed.

Two layers of link checking The generator fails the build on unresolvable internal references, and the deployed check catches redirect, rewrite and external failures the build cannot see. layer 1 — the build onBrokenLinks: throw catches: a link to a page the generator never emitted layer 2 — the preview lychee on the live URL catches: rewrites, redirects, anchors, external 404s merge gate internal → block external → report uptime is not your bug

Problem & Context

Documentation links rot in three distinct ways, and a single check catches only one of them.

The first is an internal reference to a page that does not exist — a typo, or a link to a page that was renamed. The generator can catch this at build time because it knows every page it emitted.

The second is a link that resolves on disk but not over HTTP. /guides/auth is a directory on the filesystem and a request that must be rewritten to index.html by the host. Whether that rewrite happens depends on host configuration, so a link can be perfectly valid locally and 404 in production. No build-time check can see this, because the build has no host.

The third is an anchor that no longer exists. A link to /guides/auth/#refresh-tokens is a valid page reference regardless of whether that heading is still on the page, so it passes every structural check and lands the reader at the top of a long document wondering what they were meant to see.

A generator’s onBrokenLinks: 'throw' handles the first. Only a checker running against a deployed URL handles the other two, which is why this gate belongs on the preview rather than in the build.

Step-by-Step Solution

Free, instant, and catches the most common case before anything is deployed:

// docusaurus.config.js
module.exports = {
  onBrokenLinks: 'throw',
  onBrokenMarkdownLinks: 'throw',
  onBrokenAnchors: 'throw',   // v3.1+; catches the third failure mode at build time
};

2. Check the deployed preview, not the filesystem

Install the checker and point it at a URL rather than a path:

# lychee is a single fast binary; pin it in CI
curl -fsSL https://github.com/lycheeverse/lychee/releases/download/v0.15.1/lychee-v0.15.1-x86_64-unknown-linux-gnu.tar.gz \
  | tar -xz -C /usr/local/bin lychee
lychee --version

lychee --no-progress https://pr-482.docs-preview.example.com/

3. Drive the crawl from the sitemap

Starting at the root only reaches pages the navigation links to. Starting from the sitemap reaches everything you publish, including pages that are only reachable by deep link — which are precisely the ones nobody notices breaking:

lychee --no-progress --max-concurrency 8 \
  https://pr-482.docs-preview.example.com/sitemap.xml

4. Separate internal from external failures

Two invocations with different consequences. The internal one is the gate; the external one is information:

# Gate: internal links only. Any failure blocks the merge.
lychee --no-progress --exclude-all-private \
  --include "^https://pr-482\.docs-preview\.example\.com" \
  --exclude-mail \
  "https://pr-482.docs-preview.example.com/sitemap.xml"

# Report: everything else, never fails the build.
lychee --no-progress \
  --exclude "^https://pr-482\.docs-preview\.example\.com" \
  --format markdown --output external-links.md \
  "https://pr-482.docs-preview.example.com/sitemap.xml" || true

5. Cache results and respect rate limits

Most 429 responses come from hammering the same few hosts. Persist the cache between runs so unchanged URLs are checked once, and throttle:

lychee --cache --max-cache-age 14d --max-concurrency 6 \
  --retry-wait-time 4 --max-retries 3 \
  "https://pr-482.docs-preview.example.com/sitemap.xml"
Blocking versus reporting failures Internal 404s, failed rewrites and missing anchors block the merge, while third-party outages, rate limits and login walls are reported without blocking. blocks the merge internal 404 — a renamed page directory URL not rewritten anchor no longer on the page all of these are defects in this pull request reported, never blocks third-party host is down 429 rate limited 403 behind a login wall none of these are yours to fix today

Complete Working Example

# .github/workflows/link-check.yml
name: Link check
on:
  pull_request:
    paths: ['docs/**', 'openapi/**']
  schedule:
    - cron: '0 6 * * 1'      # weekly external sweep against production

permissions:
  contents: read
  pull-requests: write

jobs:
  links:
    runs-on: ubuntu-latest
    env:
      BASE: ${{ github.event_name == 'pull_request'
        && format('https://pr-{0}.docs-preview.example.com', github.event.number)
        || 'https://docs.example.com' }}
    steps:
      - uses: actions/checkout@v4

      # Persist the checker cache so unchanged URLs are not re-fetched every run.
      - uses: actions/cache@v4
        with:
          path: .lycheecache
          key: lychee-${{ github.run_id }}
          restore-keys: lychee-

      - name: Wait for the preview to be live
        run: npx wait-on "${BASE}/sitemap.xml" --timeout 120000

      # GATE — internal links only. A failure here is a defect in this change.
      - name: Check internal links
        uses: lycheeverse/lychee-action@v2
        with:
          args: >-
            --no-progress --cache --max-cache-age 14d
            --max-concurrency 8 --exclude-mail
            --include "^${{ env.BASE }}"
            "${{ env.BASE }}/sitemap.xml"
          fail: true

      # REPORT — external links. Never fails the build.
      - name: Check external links
        id: external
        continue-on-error: true
        uses: lycheeverse/lychee-action@v2
        with:
          args: >-
            --no-progress --cache --max-cache-age 14d
            --max-concurrency 4 --retry-wait-time 4 --max-retries 3
            --exclude "^${{ env.BASE }}"
            --exclude "^https://(twitter|x)\.com"
            "${{ env.BASE }}/sitemap.xml"
          output: external-links.md
          fail: false

      - name: Comment external findings
        if: steps.external.outcome == 'failure' && github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const body = '<!-- link-report -->\n### External links needing attention\n\n'
              + fs.readFileSync('external-links.md', 'utf8').slice(0, 60000);
            const {data: c} = await github.rest.issues.listComments(
              {...context.repo, issue_number: context.issue.number});
            const prev = c.find(x => x.body.includes('<!-- link-report -->'));
            if (prev) await github.rest.issues.updateComment(
              {...context.repo, comment_id: prev.id, body});
            else await github.rest.issues.createComment(
              {...context.repo, issue_number: context.issue.number, body});

The schedule trigger is what keeps external links honest without blocking anyone. A weekly sweep against production finds the third-party pages that disappeared since the last release, and files them as information rather than as an obstacle to a merge that has nothing to do with them.

Gotchas & Edge Cases

The checker races the deploy. Pointing it at a preview URL that is not live yet produces a run where every link fails, which reads as a catastrophic regression. The wait-on step is not optional; without it this job is flaky in a way that erodes trust in the gate faster than any false positive.

Anchors are checked only if you ask. Many checkers verify that a page responds and stop there, ignoring the fragment. Enable fragment checking explicitly, and be prepared for findings — anchor rot is invisible and therefore widespread. Renaming a heading changes its generated id, so a heading edit in one file quietly breaks links from every other file that pointed at it.

Login-walled and bot-hostile hosts return 403 forever. Some sites reject any request without a browser-like user agent, and no amount of retrying helps. Maintain a small, commented exclusion list rather than lowering the whole check’s strictness, and re-review it occasionally so a permanent exclusion does not hide a link that genuinely died.

Excluding a pattern is easy to over-scope. An exclusion written as a bare domain can match far more than intended once the site adds a subdomain. Anchor the regexes, and prefer excluding a specific host over excluding a substring that happens to appear in other URLs.

A checker that reports hundreds of findings will be ignored. The first run against a mature site almost always produces a long list, most of it accumulated over years and none of it caused by the pull request in front of you. Blocking that pull request on a backlog it did not create is how a gate loses its credibility in a single afternoon. Fix the backlog as its own piece of work — run the checker once against production, file the findings, clear them — and only then turn the gate on. From that point every finding genuinely belongs to the change that introduced it, which is the condition under which people fix them promptly instead of arguing about them.

Redirects deserve a decision, not a default. A link that returns 301 before reaching a 200 is not broken, and most checkers accept it. But a documentation site accumulating redirect hops is slowly getting slower and more fragile, and a chain of three is one removed hostname away from being genuinely broken. Configure the checker to report redirected internal links separately from failing ones, and treat the report as a cleanup queue: update the link to its final destination in the source, and keep the redirect for readers who bookmarked the old URL.

Reading the checker output is faster once you know that each status code points at a different owner. The mapping below covers almost everything a documentation crawl reports:

Response codes and their owners A 404 on an internal URL is yours to fix now, a 3xx chain is cleanup, a 403 or 429 is usually the remote host, and a timeout is often the checker configuration. the status code tells you who owns the fix 404 internal page gone yours — fix now blocks the merge 301 / 308 redirect chain yours — cleanup queue reported, not blocking 403 / 429 bot wall or throttle theirs — or your rate exclude or slow down timeout no response at all often the config check concurrency first

Treat the leftmost column as the only one that stops a merge. The other three belong in a queue that someone works through deliberately, which keeps the gate fast and its failures unambiguous.

FAQ

Why do links pass locally and fail on the deployed site?

A filesystem check resolves /guides/auth to guides/auth/index.html regardless of server behaviour, while the deployed host may not perform that rewrite. Checking the deployed URL is the only way to exercise the redirect and directory-index rules readers actually hit.

Should a broken external link fail the build?

No. A third-party site being down is not a defect in your pull request, and blocking merges on someone else’s uptime trains people to bypass the gate. Fail on internal links and report external ones, with a scheduled sweep against production to keep them from accumulating.

How do I stop the checker being rate limited?

Lower concurrency, persist the cache between runs so unchanged URLs are not re-fetched, and exclude hosts that aggressively throttle. Most 429 responses come from checking the same handful of domains hundreds of times in one run.