Crawl budget and crawl traps
Faceted navigation, sort orders and session parameters can generate more URLs than you have products. I find where the crawl leaks and close it, without cutting off the facets that have search demand.

Technical SEO at scale
Technical SEO for websites with millions to tens of millions of URLs. I work from server logs, full crawls and raw Search Console exports, and I read the code that generates the pages.
What changes at scale
On a small website you can look at every page. At this size a desktop crawler gives up after a few million URLs, a spreadsheet long before that, and the Search Console interface shows you a thousand rows.
So the unit of work changes. I segment everything by template, count in SQL, and treat crawling, indexing and internal linking as systems with inputs and outputs. The question is where fifty million URLs came from, and which of them deserve to exist.
One bug in a template is a bug on four million URLs. One fix in a template is the cheapest SEO you will ever ship.
What I work on
I treat every one of them as a system: data you can query, a number you can move and a release that moves it.
Faceted navigation, sort orders and session parameters can generate more URLs than you have products. I find where the crawl leaks and close it, without cutting off the facets that have search demand.
Logs are the only record of what Googlebot did, as opposed to what a tool predicts it might do. Hits per template, status codes, and crawl frequency set against how often the content changes.
Deciding which URLs deserve an index entry, then making canonicals, noindex, sitemaps and status codes say the same thing. On large sites they usually contradict each other.
Fifty million nodes and a couple of billion edges. Click depth, orphans and link flow per template are graph queries. The fixes are changes to link modules, never to single pages.
What is in the raw HTML, what only appears after rendering, and what a crawler never gets to see. I compare both versions per template and move the critical parts to the server.
Language and country signals, canonicals and redirects across dozens of markets, generated from one source. I have ranked websites in all 244 countries Google recognises, including Christmas Island.
Domain moves, replatforming, URL restructures, brands merging. I do not turn away from risk. It can be reduced: redirect rules as code, staging crawls and a go or no-go list.
On a large website SEO usually breaks through someone else's release. Checks on templates, robots.txt and rendered output find a regression within hours, long before rankings show it.
GPTBot, ClaudeBot and the others are already in your logs. What they fetch, how often, what they get back, and whether your robots.txt says what you think it says.
Methods
No sampled crawl and no top-thousand export. The full dataset, segmented by template, in a query anyone on your team can rerun.
01
I start with thirty days of server logs, verified with reverse DNS so the fake Googlebots are out. Segmented by template, the pattern shows up fast: expired listings that still get crawled, filter URLs that eat a quarter of the budget, redirect chains on one page type.
02
The Search Console interface stops at a thousand rows. The BigQuery bulk export holds everything Google is willing to share. I join it with the sitemaps and the crawl, per template. A template where most URLs never get a single impression is an indexation problem, and usually a product decision as well.
How findings become a roadmap03
I fetch every template twice, raw and rendered, and diff the result: titles, canonicals, links, structured data and robots directives. The dangerous cases are the ones where the two versions disagree. At BrightonSEO in 2019 I presented what I learned about SEO from using the ten most used JavaScript frameworks. The frameworks have changed since. The failure modes have not.
04
Market targeting is a tag on ten pages and a data pipeline on ten million. I audit complete URL sets across all markets: alternates that do not link back, targets that redirect, canonicals that disagree, and language codes that do not exist. Then I move the generation to one place, so the next market launch does not start the argument again.
Who should own market targetingCrawl control
robots.txt says where crawlers should not go. Sitemaps say what you want indexed. On large websites the two often disagree with each other and with the internal links. I rewrite both against the logs and test every rule before it ships.
AI crawlers
Analytics filters bots out by design, so the log is the only place to see them. I report AI crawlers next to Googlebot, with the same rigour.
Hits per template, status codes, and whether robots.txt matches the policy that legal and marketing agreed on. Most of these crawlers fetch the HTML and do not run your JavaScript, which makes the raw response matter again.
No server logs on your platform? That gap is why I am building Botsanalyser. It is still on a waitlist.
My products for AI searchToolbox
I do not need a specific stack. I need access to the data, in whatever warehouse or bucket it lives.
The ground truth for every bot. If nothing is stored yet, getting logs delivered to a bucket is the first ticket I write.
The whole website, or a sample that is stratified by template when the whole is not practical. Stored as tables, so links become a graph you can query.
Every row instead of the first thousand. Joined with logs, crawl and sitemaps on the URL, which is where most of the answers are.
Up to ten years of Search Console history, migration comparison and change detection. Built for this work, never a requirement for it.
ProductsHigh-risk changes
A migration is the riskiest release most websites ever do. It is also the one moment the whole organisation is willing to fix old problems. I plan for both.
Size, logs, code, stack, and who does something with the findings afterwards.
There is no hard line. The methods on this page start to pay off around a million URLs, which is roughly where a desktop crawler and a spreadsheet stop being enough. The largest websites I work on have 50 million pages and more.
For most websites, no, and Google says so itself. For a marketplace with tens of millions of URLs and listings that expire every day, yes. When Googlebot spends a quarter of its requests on filter combinations, new listings are found later and expired ones stay in the index longer. You can measure both in the logs.
Then getting them is the first ticket. Most CDNs and load balancers can deliver request logs to a bucket, and that is enough to work with.
On hosted platforms where logs do not exist at all, bots have to be recorded another way. That is the reason I am building Botsanalyser. It has not launched yet, so for now the answer there is custom work.
Both. SQL and scripts are simply how the analysis gets done. For fixes I write tickets with acceptance criteria, and I prototype a solution when that is faster than describing it. I have close to thirty years of front-end and back-end development behind me. I do not need commit rights to your production code.
Whatever you have. A data warehouse or a bucket full of gzip files both work. I bring my own tooling where it saves time, such as GSC Wizard for Search Console history and Search Sentinel for change detection. Neither is required. They are on the products page.
That is the part most technical audits skip. Findings become tickets, tickets get a value, and the list gets defended in planning. How I do that inside a large organisation is on the enterprise SEO page. Who I am is on the about page.
Crawl budget, indexation, rendering, international targeting or a migration with a fixed date. Tell me the size of the website and what worries you.
Independent consultant · working with teams worldwide