Case Study #5 - 1,464 Crawls, 764 Filename Misses

Home Crew USA is fifteen days old. Week one was a launch stampede. Week two was quieter maintenance crawling with 292 misses. Week three is where the bots stop reading pages and start checking receipts.

This is case study five. Case study #1 built nonsenseo.com, case study #2 tracked its first week, case study #3 proved the pattern repeats on a second domain, and case study #4 covered the first maintenance week. This one watches what happens when the launch traffic drains away and only the crawlers keep the lights on.

The short version: total traffic fell 41 percent, AI crawler volume barely moved, and 764 of 912 misses were requests for files, not pages.

Six days of bursts

Between October 23 and 28 the site served 9,716 requests, down from 16,593 the week before. AI crawlers sent 1,464 of them, 15.1 percent, about 3.9 MB of responses. That share is up from 9.1 percent, not because the bots got busier but because everyone else left.

The AI line still moves in bursts: 422 on the 23rd, a weekend floor of 121 and 75, then 591 on the 26th, 200 on the 27th, and 55 to close. No steady stream, no daily drip. Six days, six different days of crawling.

The leaderboard flipped again

Every operator from week two came back. None of them behaved the same way.

Amazonbot is the only operator that grew: 457 requests, 394 answered. In week two it sent 440 and got 411 answered. Meta’s external agent collapsed from 456 requests and zero failures to 49 requests and 46 failures. Anthropic went from 137 requests with 6 failures to 128 requests with 126 failures, a near-total miss rate. Perplexity doubled its volume to 110 and still has not received a single successful answer across three weeks.

The served total is the number that matters: 552 of 1,464, or 37.7 percent, down from 80.6 percent in week two. The fleet is asking for more than the site has.

764 filename misses

Of the 1,464 AI responses, 912 were 4xx: 870 404s, 31 403s, and 11 405s. Still zero 5xx. The misses spread across 234 paths, and 764 of the 912 hits landed on something that looks like a file rather than a page.

The repeat offenders are a tour of other people’s stacks: /services/.env at 16 requests, /api/.env at 15, /files../.env at 13, /docker-compose.staging.yml at 13, /static../.env at 13, /terraform.tfstate.backup at 12, /.docker/config.json at 12, and /assets/.env at 12. Crawlers are not reading a static site. They are checking whether it is secretly a Vercel app with a leaked environment file.

Every one of those answers is a clean 404 at static-page cost. The site never flinches, and nothing on the origin wakes up.

What they actually read

AI hits touched 637 distinct paths, up from 602 last week. The sitemap now lists 461 URLs, one more than the week before, and robots.txt and the sitemap still answer on every crawl.

The gap between 637 requested paths and 461 sitemap entries is the error traffic: 234 distinct paths came back 4xx, and most of them are files the site never had.

Check it yourself

curl -s https://homecrewusa.com/sitemap-0.xml | grep -c "<loc>"
curl -s https://homecrewusa.com/services/.env -o /dev/null -w "%{http_code}\n"
curl -s https://homecrewusa.com/terraform.tfstate.backup -o /dev/null -w "%{http_code}\n"
curl -s https://homecrewusa.com/llms.txt | head -20

The sitemap returns 461 URLs. Both config probes answer 404, exactly as the numbers above say. Same artifacts, same receipts, one week later.

What we won’t tell you

How the pages get generated and refreshed. Which retrieval experiments run in the background. What the domain earns and what it costs. The engagement buys the outcome; the kitchen stays closed.

The misses are in here on purpose. A case study with only wins is an advertisement. 912 documented 4xx answers on paths that do not exist is evidence that someone actually looked.

FAQ

Did anything break in week three?

No. 534 responses were 200, one was 206, 17 were 301, 31 were 403, 870 were 404, and 11 were 405. Zero 5xx across all six days.

Who crawled the most?

Amazonbot, 457 requests with 394 answered. OAI-SearchBot followed at 183 and Googlebot at 153. Meta’s external agent fell from first place to eleventh.

Why did the served rate drop so far?

Week two served 80.6 percent of AI requests because most crawlers were still fetching pages from the launch index. Week three served 37.7 percent because the fleet shifted to config-file probing, and the site answers every probe with a 404.

What changed since Case Study #4?

Same domain, later phase. Total traffic fell 41 percent, AI requests held near flat at 1,464, error volume tripled to 912, and 764 of those errors were filename guesses rather than page misses.

If you want a site that keeps its receipts after launch week too, that is the service.