Scoping DocSearch Crawls to Versioned Docs

This guide is part of Algolia DocSearch Integration within Developer Portal Frameworks & UI Setup. It fixes a problem that appears the moment a portal publishes more than one API version: a reader searches for an operation and gets the same page three times, with a deprecated version ranked first. The documentation is correct, the search is working, and the result is still worse than useless.

Search results before and after scoping An unscoped index returns the same operation from every published version, while a scoped index returns only supported versions with the current one first. searching "create invoice" unscoped index /docs/0.9/api/create-invoice /docs/1.0/api/create-invoice /docs/api/create-invoice the usable result is third scoped index + facet /docs/api/create-invoice /docs/1.0/api/create-invoice 0.9 never entered the index current version first

Problem & Context

A crawler follows links, and a versioned portal links to every version it publishes. Left alone, the crawl indexes all of them, so each operation appears once per version. The index is not wrong — those pages genuinely exist — but relevance ranking has no idea that one of them is the answer and the others are history.

Worse, ranking often favours the old ones. Deprecated pages have accumulated inbound links and stable URLs for years, while the current page was published last month. So the practical result of adding search to a versioned portal is that developers reliably find the wrong version first, copy a request that no longer works, and conclude the documentation is inaccurate.

Two independent changes fix it. Excluding unsupported versions from the crawl removes noise at the source. Tagging the remainder with a version facet lets a reader on the v1 docs get v1 results, without maintaining separate indexes.

Step-by-Step Solution

1. Measure the damage first

Before changing anything, quantify it — this number is what justifies the work and proves the fix:

curl -s "https://${APP_ID}-dsn.algolia.net/1/indexes/api-portal/query" \
  -H "X-Algolia-API-Key: ${SEARCH_KEY}" \
  -H "X-Algolia-Application-Id: ${APP_ID}" \
  -d '{"query":"create invoice","hitsPerPage":10,"attributesToRetrieve":["url"]}' \
| jq -r '.hits[].url' | sed -E 's|.*/docs/([0-9.]+)?.*|\1|' | sort | uniq -c

      4 0.9
      3 1.0
      3           # blank = the current version

Four of the top ten results are from a version nobody supports.

2. Restrict the crawl with stop_urls

stop_urls takes regular expressions. Exclude the version prefixes you no longer support:

{
  "index_name": "api-portal",
  "start_urls": ["https://docs.example.com/"],
  "sitemap_urls": ["https://docs.example.com/sitemap.xml"],
  "stop_urls": [
    "https://docs\\.example\\.com/docs/0\\.[0-9]+/",
    "https://docs\\.example\\.com/docs/1\\.0/api/internal/"
  ],
  "selectors": {
    "lvl0": { "selector": ".sidebar .menu__link--active", "global": true,
              "default_value": "Documentation" },
    "lvl1": "article h1",
    "lvl2": "article h2",
    "lvl3": "article h3",
    "text": "article p, article li, article td"
  }
}

3. Tag every record with its version

Derive the version from the URL rather than from page markup — the URL is stable, the DOM is not:

{
  "custom_settings": {
    "attributesForFaceting": ["version", "lang"]
  },
  "js_render": false,
  "selectors_exclude": [".theme-doc-version-banner"],
  "conversion_function": "return (records) => records.map((r) => { const m = r.url.match(/\\/docs\\/(\\d+\\.\\d+)\\//); r.version = m ? m[1] : 'current'; return r; })"
}

Confirm the facet landed:

curl -s "https://${APP_ID}-dsn.algolia.net/1/indexes/api-portal/query" \
  -H "X-Algolia-API-Key: ${SEARCH_KEY}" -H "X-Algolia-Application-Id: ${APP_ID}" \
  -d '{"query":"","facets":["version"],"hitsPerPage":0}' | jq '.facets.version'

{
  "current": 214,
  "1.0": 189
}

The absence of 0.9 is the crawl exclusion working.

4. Filter at query time from the current URL

A reader on the v1 pages should get v1 results. Derive the filter from where they are, not from a preference they have to set:

import { DocSearch } from '@docsearch/react';

function currentDocsVersion() {
  const m = window.location.pathname.match(/^\/docs\/(\d+\.\d+)\//);
  return m ? m[1] : 'current';
}

export function Search() {
  return (
    <DocSearch
      appId="YOUR_APP_ID"
      apiKey="YOUR_SEARCH_ONLY_KEY"
      indexName="api-portal"
      // Results follow the version the reader is already reading.
      searchParameters={{ facetFilters: [`version:${currentDocsVersion()}`] }}
    />
  );
}

5. Serve noindex on unsupported versions

Your index is now scoped; public search engines still are not. Add the header at the CDN for the excluded prefixes:

/docs/0.*    X-Robots-Tag: noindex, follow

follow rather than nofollow is deliberate: the page should stop appearing in results while its outbound links to the current version keep passing value.

Three layers of version scoping The crawl exclusion removes unsupported versions from the index, the facet filter narrows results to the version being read, and the noindex header handles public search engines. three layers, three different audiences stop_urls unsupported versions never enter the index controls: what exists version facet filter supported versions, narrowed per reader controls: what is shown X-Robots-Tag public search engines stop ranking old pages controls: arrivals from outside

Complete Working Example

A crawler configuration and a verification script that fails CI if unsupported versions reappear:

{
  "index_name": "api-portal",
  "start_urls": [
    { "url": "https://docs.example.com/docs/", "tags": ["current"] },
    { "url": "https://docs.example.com/docs/1.0/", "tags": ["v1"] }
  ],
  "sitemap_urls": ["https://docs.example.com/sitemap.xml"],
  "stop_urls": [
    "https://docs\\.example\\.com/docs/0\\.[0-9]+/",
    "https://docs\\.example\\.com/docs/next/"
  ],
  "selectors": {
    "lvl0": { "selector": ".sidebar .menu__link--active", "global": true,
              "default_value": "Documentation" },
    "lvl1": "article h1",
    "lvl2": "article h2",
    "lvl3": "article h3",
    "text": "article p, article li, article td"
  },
  "selectors_exclude": [".theme-doc-version-banner", ".pagination-nav"],
  "custom_settings": {
    "attributesForFaceting": ["version", "lang"],
    "searchableAttributes": [
      "unordered(hierarchy.lvl0)", "unordered(hierarchy.lvl1)",
      "unordered(hierarchy.lvl2)", "unordered(hierarchy.lvl3)", "content"
    ],
    "customRanking": ["desc(weight.pageRank)", "desc(weight.level)"]
  }
}

#!/usr/bin/env bash
# verify-index-scope.sh — fail if unsupported versions are present in the index.
set -euo pipefail

: "${APP_ID:?}" "${SEARCH_KEY:?}" "${INDEX:=api-portal}"
SUPPORTED='current 1.0'

facets=$(curl -s "https://${APP_ID}-dsn.algolia.net/1/indexes/${INDEX}/query" \
  -H "X-Algolia-API-Key: ${SEARCH_KEY}" \
  -H "X-Algolia-Application-Id: ${APP_ID}" \
  -d '{"query":"","facets":["version"],"hitsPerPage":0}' | jq -r '.facets.version | keys[]')

fail=0
for v in ${facets}; do
  case " ${SUPPORTED} " in
    *" ${v} "*) echo "OK   ${v} is supported and indexed" ;;
    *) echo "FAIL ${v} is indexed but not supported — check stop_urls" >&2; fail=1 ;;
  esac
done

# A missing supported version is just as wrong as an extra unsupported one.
for v in ${SUPPORTED}; do
  echo "${facets}" | grep -qx "${v}" || { echo "FAIL ${v} is supported but NOT indexed" >&2; fail=1; }
done

exit "${fail}"

Expected output:

OK   current is supported and indexed
OK   1.0 is supported and indexed

Checking both directions matters. An over-broad stop_urls regex is easy to write — docs/1\. also matches docs/1.0/ — and it silently removes a version readers depend on. The second loop catches that, which the first cannot.

Gotchas & Edge Cases

stop_urls entries are regular expressions, not prefixes. An unescaped dot matches any character, so docs/0.9/ also matches docs/019/. Escape them, and test each pattern against a couple of real URLs before crawling.

The facet must be declared before it can be filtered. An attribute that is not in attributesForFaceting cannot be used in facetFilters; the query silently returns everything rather than erroring, so the filter appears to do nothing.

Cutting a new version does not update the crawler config. The config names versions explicitly, so it drifts every time you cut or prune one. Generate the stop_urls and supported-version list from versions.json in CI rather than maintaining both by hand — Versioned API docs in Docusaurus covers where that list lives.

Filtering hides results a reader may actually want. Someone on the v1 docs searching for a feature that only exists in the current version now gets nothing. Show a “no results in 1.0 — search all versions” affordance rather than a bare empty state, so the filter narrows by default without becoming a wall.

Removing pages from the index does not remove them from the index. Crawlers add and update records; they do not always delete ones whose URLs disappeared. After a large exclusion change, clear the index and crawl fresh rather than crawling on top of the old records.

Treat the zero-result list as the measure of whether the scoping is right. Once filtering is on, the interesting signal is not what people find but what they stop finding. Algolia records every query that returned nothing, and after a scoping change that list tells you two distinct things: queries where the reader was on an old version and the answer only exists in the current one, and queries where your stop_urls were too broad and removed something people genuinely need. The two look identical in the raw list and are distinguished by checking whether the page still exists on the site. Review it weekly for the first month after any change to the crawl scope, then monthly.

Version scoping and support policy have to move together. The crawler config, the facet filter, the noindex rules, and the published support window are four expressions of the same decision, and it is easy for them to drift apart — a version gets pruned from the portal but stays in the index, or leaves support but keeps ranking in Google. The durable fix is to derive all four from one source of truth. Keep the supported-version list in a single file, generate the crawler stop_urls and the CDN header rules from it in CI, and have the verification script above read the same file. That way retiring a version is one edit rather than four, and the four can never disagree.

One source of truth for version scoping A single supported-version list generates the crawler exclusions, the facet filter, the noindex header rules and the verification check. versions.json the supported list crawler stop_urls facet filter values X-Robots-Tag rules the verification check retiring a version is one edit

FAQ

Should I use one index per version or a version facet?

A facet. Separate indexes multiply record counts against your plan limits, complicate the widget configuration, and make cross-version search impossible. A facetable version attribute gives you filtering, an escape hatch to search everything, and one index to maintain.

Will excluding old versions break their deep links?

No. Excluding a path from the crawl only removes it from search results; the pages remain published and every existing link keeps working. It changes discovery, not availability.

Where does the version value come from?

Extract it from the URL path with a crawler transform, as in the configuration above. Deriving it from the URL rather than from page markup means it stays correct even when a theme upgrade renames the class your selectors depended on.