A large online directory with more than 100 million pages had lost most of its organic traffic compared with its historical peak. The owner’s previous approach had focused on external links. SEMANTIKOS was asked to review the technical side of the site before any further investment.
What the audit found
At this scale, small template problems multiply across millions of URLs. The audit focused on how search engines were being admitted to the site, where crawl activity was going and what each page type told crawlers about itself.
1. Bot protection blocking legitimate crawlers
The site used aggressive bot protection to deter scraping. During large crawls, some legitimate search engine requests were being served a challenge page instead of the real content, and that challenge page carried a noindex directive. In effect, the site was intermittently telling search engines to drop its own pages.
2. Crawl activity spent on outbound redirects
Many internal links led to server-side redirects that sent visitors to partner sites. Crawlers followed them too, spending a large share of requests on URLs that were never meant to be indexed.
3. Opaque URL structure
Category and breadcrumb paths used internal ID codes rather than descriptive paths. That gave crawlers little context about how sections related to each other and made it harder to build a clear topical hierarchy.
4. Corrupted structured data
The breadcrumb markup contained double-encoded characters, so names and labels in the structured data did not match what users saw on the page. Beyond breadcrumbs, the page types had very little structured information describing what each record contained.
5. Heavy page weight
Large volumes of inline styling inflated every HTML response, increasing the cost of crawling and processing millions of pages.
The recommended roadmap
- Month 1: adjust bot protection so verified search engine and AI crawlers always receive the real page, and change how partner links are handled so crawlers are not sent through outbound redirects.
- Month 2: rebuild the structured data templates with clean, correctly encoded values and reduce inline page weight.
- Month 3: replace ID-based paths with descriptive category URLs and partition the XML sitemaps for a site of this size.
- Month 4: standardise canonical logic and restructure internal linking between hubs and records.
- Months 5 and 6: optimise templates for AI answers and retrieval, and monitor indexation and crawl behaviour through log analysis.
Why it matters
For very large sites, crawl access and template quality matter more than almost anything else. No amount of external promotion helps if search engines are intermittently blocked, or if much of their crawl activity goes to URLs that will never rank. Fixing the templates once improves millions of pages at the same time.
Client details have been withheld for confidentiality. This case describes the audit findings and the recommended plan.