Fixing Millions of Crawled URLs With Only Thousands of Valuable Pages
Search engine bots were crawling millions of URLs on a site that only had thousands of pages worth ranking. Filters, archives, parameters, internal search results, pagination variants and duplicates were soaking up crawl activity. Meek Media restructured how the site exposed URLs so crawlers concentrated on valuable pages and the index reflected what the business actually wanted found.
- Client
- Large website (name withheld)
- Services
- Technical SEO Services
- Privacy
- Client name withheld
How do you stop Google wasting crawl budget on useless URLs?
Stop generating and linking to low-value URLs, then tell crawlers clearly what to skip. Block infinite filter, parameter and internal search spaces in robots.txt, consolidate duplicates with canonicals, noindex or remove thin archives, keep pagination to one clean sequence, and list only valuable pages in sitemaps. Meek Media combined these to refocus crawling on pages that matter.
Why were bots crawling millions of URLs for a site with only thousands of useful pages?
The site's architecture was generating URLs far faster than it was producing content. Every filter combination, archive view, URL parameter, internal search query and pagination variant created a new address that crawlers could discover and request. Many of those addresses showed the same or nearly the same content as each other. The result was a crawlable space measured in millions, wrapped around a core of only thousands of pages with genuine search value.
For a large site, this matters. Google has finite time and resources for each site, and when most of that goes to near-duplicate and low-value URLs, the important pages can be crawled less often and updated changes can take longer to be picked up. Index quality suffers too: thin and duplicate pages dilute the signals of the pages you actually want to rank. It is worth being precise here. Google does not assign a crawl budget score, and crawl budget is mainly a concern for large or rapidly changing sites. This site met that description, so crawl efficiency and index quality became the focus of the engagement.
What we found
- Millions of URLs were being crawled against only thousands of valuable pages
- Filter combinations created a near-infinite URL space
- URL parameters produced duplicate versions of the same content
- Internal search result pages were exposed to crawlers
- Archives and pagination variants multiplied low-value URLs
- Duplicate content diluted the pages that deserved to rank
How did Meek Media refocus crawling on the pages that matter?
The work started with measurement, not blocking. We established which URL patterns were consuming crawl activity and which pages actually carried value. Each pattern then got the control that fits it, because robots.txt, canonicals, noindex and removal solve different problems, and using the wrong one can hide pages from Google while keeping them in the index.
-
1
Measure where crawl activity was going
We analysed server logs alongside a full site crawl and Search Console's Crawl Stats report to see which URL patterns search engine bots were requesting and how often. Grouping URLs by pattern, such as filters, parameters, search pages and archives, turned millions of addresses into a manageable set of problems. It also showed how often the valuable pages were being crawled compared with the waste around them.
-
2
Define the set of pages worth indexing
Before changing any directives, we agreed which pages deserved to be crawled and indexed: the thousands of URLs with real search demand, unique content or commercial value. Everything else was classified by pattern and assigned a treatment. This inventory became the reference for every later decision, and it made it easy to check that no valuable page was caught by a rule meant for waste.
-
3
Close off filter, parameter and search spaces
Filter combinations and internal search results create effectively infinite URL sets, so crawl paths into them were closed with robots.txt rules and by removing crawlable links to combinations with no search value. A small number of useful filtered pages were kept as clean, linkable URLs. Parameters that did not change content were consolidated so each page has one address.
-
4
Handle archives and pagination properly
Archive views that duplicated category content or offered nothing unique were noindexed or removed. Pagination was reduced to one clean sequence per listing, without variants created by combining page numbers with sort orders or filters. Each paginated page kept a self-referencing canonical, since Google no longer uses rel next and prev and canonicalizing every page to page one hides deeper content.
-
5
Consolidate duplicates with the right signal
Duplicate and near-duplicate pages were consolidated with canonical tags or redirects pointing to one preferred URL. We sequenced directives carefully: a page must stay crawlable for Google to see a noindex or canonical, so URLs were not blocked in robots.txt until they had dropped from the index. That avoids the common trap of blocked URLs lingering in results.
-
6
Rebuild sitemaps and internal links around value
XML sitemaps were rebuilt to list only canonical, indexable pages, giving Google a clean statement of what matters. Internal linking was adjusted so navigation and templates point to preferred URLs rather than parameter or filter variants. Crawl activity and index coverage were then tracked to confirm that bots were spending more of their time on the valuable pages.
What We Delivered
- Crawl and log analysis grouped by URL pattern
- Inventory of indexable, valuable pages
- Robots.txt rules for filter, parameter and search URL spaces
- Indexing rules for archives and pagination
- Canonical and redirect plan for duplicate content
- Rebuilt XML sitemaps listing only canonical pages
- Internal linking changes pointing to preferred URLs
What Other Teams Can Learn From This
Fix the generator, not just the symptoms
Blocking bad URLs after they exist is a holding action. The lasting fix is changing the templates, filters and links that create them. If the site keeps generating crawlable variants, new ones will appear faster than rules can catch them. Treat URL generation as an architecture decision that SEO signs off, not a side effect of features.
Robots.txt and noindex do different jobs
Robots.txt controls crawling, and noindex controls indexing. A URL blocked in robots.txt can still appear in results if other pages link to it, and Google cannot see a noindex on a page it is not allowed to crawl. Choose the directive based on the outcome you want, and sequence them when both are needed.
Crawl budget is a large-site problem
Most small sites do not need to worry about crawl budget, and there is no score to optimize. It becomes a real issue when a site has a very large number of URLs or changes rapidly. If your crawled URL count is far larger than your useful page count, that gap is the metric worth reducing.
Frequently Asked Questions
What is crawl budget and does my site need to worry about it?
Crawl budget describes how many URLs Google can and wants to crawl on a site in a given period. Google says it mainly matters for very large sites or sites whose content changes rapidly. If your site generates far more crawlable URLs than it has useful pages, crawl efficiency is worth addressing.
Should faceted navigation filters be blocked from Google?
Most filter combinations should not be crawlable, because they create huge numbers of near-duplicate URLs. Blocking them in robots.txt and avoiding crawlable links to them is usually right. A few filtered views with genuine search demand can be kept as clean, indexable pages with their own unique content and titles.
Is it better to use robots.txt or noindex for low-value pages?
Use noindex when you want a page removed from the index, and keep it crawlable until Google processes that directive. Use robots.txt when you want to stop crawling of large URL spaces that should never be requested. Blocking a page in robots.txt does not guarantee it leaves the index.
How does Google handle paginated pages today?
Google no longer uses rel next and rel prev as an indexing signal. Each paginated page should be reachable through normal links, have a self-referencing canonical and contain unique items. Avoid canonicalizing every page to the first one, and avoid creating extra pagination variants through sort or filter parameters.
Related Case Studies
Rebuilding a 10,000+ Page Website Without Losing SEO
A website with more than 10,000 pages needed a complete redesign, and thousands of those pages were indexed and ranking. Meek Media planned the rebuild so search value carried across: URL architecture, page templates, metadata migration, redirects, structured data, internal linking and a structured launch QA process that checked the new site before and after it went live.
Read the case study →
Recovering a Website After a Google Algorithm Traffic Collapse
After a major Google update, this site lost 60–80% of its organic traffic. Meek Media audited everything that shapes how Google judges a site: content quality, backlinks, site architecture, duplicate pages, search intent and technical SEO. We then rebuilt the organic strategy around the pages that deserved to rank, rather than chasing the update itself.
Read the case study →
Core Web Vitals Rescue for a High-Traffic Website
This website had strong traffic but failed Core Web Vitals. Heavy scripts, large images, layout shifts, slow server responses and third-party integrations were all dragging down the experience. Meek Media tackled the problem on two fronts, optimising the front-end architecture and the server infrastructure behind it, so pages loaded faster, responded quicker and stayed visually stable.
Read the case study →
Facing a similar problem?
Every engagement starts with a free audit. We find what is holding your site back, show you the fix, and scope the work to your goals before you commit.