The problem
The question "is this site technically sound?" comes up at three specific moments. At pipeline and due-diligence stage, before there is any client access, when you need an independent read on a target's site. In kick-off week, when you want the technical baseline on record before the first campaign change. And on retainers, as a periodic health check that nothing has quietly broken. The numbers in the tile above are this site's own audit, run while writing this piece.
None of those moments needs a 40-tab spreadsheet. They need a one or two page summary, from a terminal, in about a minute: indexation health first, then the top three things worth fixing, with the evidence stated. Screaming Frog remains the right tool for the deep pass; this tool's job is telling you whether you need the deep pass at all.
So I built seo-audit: a single-file Python CLI, standard library only, Python 3.9 or later. Nothing to install, no licence key, no GUI. Point it at a domain and read the verdict.
What it does
Five stages, in order. Discover: given a bare domain it reads the Sitemap: lines from robots.txt, falls back to /sitemap.xml and /sitemap_index.xml, recurses sitemap indexes, and handles gzipped files. Sample: if the sitemap exceeds the page budget (default 100), it samples proportionally per child sitemap with the homepage and top-level pages always included, so a 5,000-URL site still gets an honest cross-section. Crawl: concurrent fetches with adaptive politeness (more on that below), manual redirect following so chains and loops are recorded, TTFB and total fetch time recorded separately. Check: roughly 30 checks spanning per-page basics, site-level patterns, canonical-target validation, per-URL robots rules, and an internal-link graph built from every anchor on every crawled page at zero extra request cost. Report:
VERDICT Indexation is broadly healthy. Nothing material was found; the top review item is 2 pages with titles outside the typical display length. COVERAGE Crawled 17 of 17 sitemap URLs in 8s. Indexable 17 · redirected 0 · broken 0 · noindexed 0 · rate-limited 0 · unreachable 0. TTFB median 0.19s · fetch median 0.20s, p90 0.25s. PRIORITY ACTIONS -- none: no blockers or material issues SECONDARY OPPORTUNITIES -- 2 review items, phrased as review items OBSERVATIONS -- context only, never inflated into problems
The verdict and priority actions are the client-facing takeaway. The coverage block is the honesty layer: what was sampled, what was rate-limited, what was skipped, so a finding from a sampled crawl is never quoted without its denominator. Issues cluster by source sitemap and first path segment, which is what turns "42 pages have X" into "42 of 48 crawled pages under /products/ have X: likely template-level, one fix".
The four finding classes
Most audit tooling fails by severity inflation: everything is critical, so nothing is. The report's design centre is a hard boundary around the word "blocker".
Indexation blockers mean a page cannot be crawled, indexed, or reached. Sitemap URLs returning 4xx or 5xx, noindex directives on sitemap-listed pages, robots rules blocking Googlebot, redirect loops. This class alone drives the exit code. Material issues are likely to affect discovery, consolidation, or presentation: missing titles, duplicate titles, canonicals pointing at broken or noindexed targets, internal links to dead pages. Opportunities are worth reviewing and are phrased as review items, never fault language: description and heading gaps, title-length display bands, thin visible text, slow-fetch outliers. Observations are context only: multiple H1s, missing structured data, oversized HTML.
The boundary is what makes the output quotable. When the tool exits 2 in a CI pipeline or a weekly cron, it means exactly one thing: something on this site cannot be indexed. Not "a title is 4 characters over a display band". That discipline came out of an early review round where missing titles sat in the blocker class; indexable pages are never blockers, however untidy, and the exit code was not trustworthy until the class was pure.
Scoring that shows its arithmetic
Priority ranking is a static table: each check carries an impact score and a confidence score, both 1 to 5, and the final score is impact × confidence × the affected fraction of crawled pages. Site-wide issues count every crawled page. A canonical or link target shared by 30 pages counts its 30 distinct sources, so one technical URL never dilutes a widespread problem to a rounding error. The default report renders this as plain language ("high impact, high confidence, affects 37 of 100 crawled pages"); a flag exposes the arithmetic.
Two rules keep the weaker signals honest. Timing is relative: a page is flagged slow only when its fetch time is both over 2.5s and over 3× the site's own median, with at least 5 successful fetches behind the median, because a uniform 1.5s TTFB across a whole sample is a platform trait, not a page problem. And orphan detection (sitemap URLs with zero internal inlinks) only reports at 80% or better sitemap coverage: a sampled crawl must never lead with an orphan claim it cannot support.
Politeness, learned the hard way
The first real run against a hosted storefront platform was educational. Eight default workers tripped the platform's rate limiter within seconds: 68% of the sample came back throttled, and the block outlived the crawl itself, with even robots.txt returning 429 minutes later. The first lesson: reactive backoff cannot un-trip a strict limiter. Once you have annoyed the platform, slowing down does not un-annoy it.
So politeness became layered and mostly proactive. Any robots.txt Crawl-delay is honoured as a delay floor. Any 429 or 503 doubles a delay shared across every worker, decaying only after 20 clean responses. Crawls over 20 pages slow-start. And known platforms are fingerprinted from response headers before the first real fetch, dropping straight to 2 workers with a 1s floor where the platform is known to be strict.
The fingerprint then failed silently in production, which was the second lesson. The platform's edge sends its identifying header in lowercase; the test fixture had faithfully copied the casing from the tool's own source code, so every test passed while the live check matched nothing.
The same instinct applies to robots.txt parsing. Python's stdlib robotparser predates the 2022 REP specification: on older versions it judges Allow: /public/ under a Disallow: / as blocked, and it misses wildcard rules entirely, which is precisely the pattern modern platforms use. A robots misread is a silently wrong indexation finding, the one class of error this tool cannot afford, so it ships its own parser: wildcards, longest-match precedence, per-agent groups. When a check's authority matters, own the parser.
Crank workers on an origin you don't control. A tripped rate limiter can outlive the crawl by minutes, and on some platforms it throttles robots.txt itself, poisoning every subsequent check in the run.
What it deliberately doesn't do
No JavaScript rendering. The crawler reads raw HTML, which is also what the first wave of indexing sees. A client-rendered site that serves empty HTML will report missing titles and thin content; that emptiness is itself a finding, but verify with a rendered crawl before reporting it as broken.
Sitemap-scoped discovery. Pages absent from the sitemap are never fetched. Internal links are analysed for the link graph but not followed, so crawl-depth and full link-equity analysis remain Screaming Frog's job. This tool tells you whether that deeper pass is needed.
No Core Web Vitals. TTFB is measured because it falls out of the fetch; LCP, CLS, and INP need a browser and belong to a different toolchain.
Protocol findings get re-verified before they are reported to anyone. A 307 redirect to https once surfaced as "no https redirect", false as stated, and a sandboxed session's egress proxy once answered a plain-http probe with its own 403. One curl of the response headers killed both wrong findings before they reached a report. The standing rule across the whole tool: weak signals are phrased as review items or omitted. Credibility over completeness.
Run it
# clone once; nothing to install git clone https://github.com/performify-github/seo-audit # bare domain: discovers the sitemap itself python3 seo-audit/seo_audit.py https://example.com # deeper sample, markdown report for filing python3 seo-audit/seo_audit.py https://example.com --max-pages 150 --markdown report.md # fragile production origin: pin a politeness floor python3 seo-audit/seo_audit.py https://example.com --workers 2 --delay 1
Exit code 0 means no indexation blockers; exit code 2 means at least one. That pair is the whole integration story: wire it into CI or a weekly cron and only open the report when the exit code tells you to. The repo carries the fixture test suite (46 tests) and the full check reference.
Where it fits the rest of the stack: a technically sound site is the precondition for everything the other Labs pieces build. If the crawl is clean, the interesting problems move up a layer, to consent, conversion signal, and how machines read the pages.
