Why redirecting every missing page to the home page costs you in search

A visitor opens a link from two years ago and lands on the home page. No message, no explanation, just the front page again. The rule that does this is usually one line in a config file, added at some point to keep the error log quiet, and it sends every unmatched address to /. It works in the sense that nothing looks broken, and it costs you in search, because a search engine reads the status code rather than your intention.

What actually happens

A 404 is information and it has two readers. A crawler learns that the address it is holding is dead and can stop asking about it. Your own monitoring learns that a link somewhere is broken and can tell you which one. A blanket redirect to the home page erases both signals and replaces them with a chain that ends in 200 on a page that has nothing to do with the request.

Search engines have a name for what they see. A soft 404 is an address that answers as if it exists while the content says otherwise: a "page not found" message served with a 200, or a redirect to an unrelated page such as the home page. Google works this out from the content, files the address as not found anyway, and keeps checking it for a while in case it changes back.

What that costs, roughly in the order you notice it:

• Crawl budget goes to addresses that will never have content. On a ten page site this rounds to nothing. On a site with tens of thousands of URLs, where an old pattern keeps generating dead paths, it becomes most of what the crawler does on each visit.
• The home page collects duplicate signals. Every dead address resolves to it, so the crawler has to work out each time whether that address is another name for the front page.
• Your reports go quiet. A broken internal link now produces a 200 in the access log, so no alert fires, no report lists it, and the links rot for a year.
• A person who clicked through to an article gets the front page with no explanation. Most of them leave rather than search for the article again.

The redirect also hides the one thing you would want to fix. A 404 in the log carries a referrer, which tells you exactly which page still links to the dead address. A 301 to the home page tells you nothing.

How to see it

One request, with the status and the redirect target printed:

curl -o /dev/null -s -w '%{http_code} %{redirect_url}\n' https://example.com/this-page-never-existed
# 301 https://example.com/

A 301 or a 302 on an address that never existed is the bug. A healthy site prints 404 and an empty redirect target.

Run the same check over a list of addresses you expect to be dead: a random path, an old pattern from the logs, a path with a trailing slash, a missing asset.

for path in /no-such-page /old-section/gone/ /blog/2019/deleted-post /assets/missing.css; do
  printf '%-32s ' "$path"
  curl -o /dev/null -s -w '%{http_code}\n' "https://example.com$path"
done

The access log tells you how wide the problem is. Count what the redirect rule actually catches:

awk '$9 == 301 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

If that list is full of paths that were never real addresses, the rule is not tidying anything up. It is hiding the size of your dead link problem behind a 200.

The fix

On a static site the whole change is two directives. try_files ends with =404, so an unmatched path produces a not found, and error_page gives that status a body:

server {
  root /var/www/site;

  location / {
    try_files $uri $uri/ =404;
  }

  error_page 404 /404.html;

  location = /404.html {
    internal;
  }
}

internal stops the error page from being requested directly, which keeps it from being indexed as a page in its own right. The status stays 404 because nothing rewrites it, and that is the part most broken configs get wrong in one of two ways:

• error_page 404 =200 /404.html; serves the right body with the wrong status. The =200 is the entire bug, and it is usually copied from an answer about making a custom error page appear.
• error_page 404 /index.html; is how a single page application gets wired when nobody thinks about crawlers. Every dead address becomes the application shell with a 200.

The single page application case is real and it has no free solution. The server has to know which paths are pages, because the status code is sent long before any JavaScript runs. If the route list is known at build time, write a file per route and let everything else fall through to =404. If routes come from a database, generate the list during deploy, or let the application server answer and set the status itself. What you cannot do is decide on the client after the 200 has already gone out.

Redirects still have their place, and the test is whether an equivalent page exists:

1. The content moved and still exists: 301 to that exact address, one to one, not to the section index.
2. A section was merged into a page that genuinely replaces it: 301 to the replacement.
3. The content is gone for good and nothing replaces it: 404, or 410 if you want to say "gone" explicitly.
4. The address never existed, because it is a typo or a scanner probing your site: 404, always.

Before you switch a blanket redirect off, read the logs. Any dead path still receiving real traffic from a real referrer deserves a 301 to its equivalent. Everything else deserves the 404 it was going to get anyway.

How to check it worked

Three requests. A missing page answers 404, a real page answers 200, and a page that actually moved answers 301 with a target that is not the front page:

curl -sI https://example.com/no-such-page | head -1
# HTTP/2 404

curl -sI https://example.com/ | head -1
# HTTP/2 200

curl -o /dev/null -s -w '%{http_code} %{redirect_url}\n' https://example.com/old-address
# 301 https://example.com/new-address

Then open the 404 page in a browser with the network panel showing. The error page has to be a complete page: its stylesheet and its images return 200, and it offers a way back into the site. An error page whose own assets are missing is the same failure one level down.

What to watch out for

• A layer above your config can undo all of it. A CDN rule that serves the index on any 404, or a hosting panel that owns the vhost file, will override what you wrote, and the panel version comes back at the next certificate renewal. When the panel and the file disagree, the panel wins.
• Do not answer 404 for something that is only temporarily unavailable. A page behind a failing backend should answer 503, otherwise a crawler reads an outage as a deletion and starts removing pages that are coming back in an hour.
• try_files $uri $uri/ =404 with the wrong root can serve files you never meant to expose instead of a clean 404. It is the same line that leaks a dotfile when the root points at the wrong directory.
• Reversing a redirect later is its own trap, because browsers keep a permanent redirect long after you change your mind. If you are not certain a move is final, remember that a cached 301 is very hard to take back.
• Watch the 404 rate after the change. It will rise, and that is the point. The number that matters is the share of those 404s that arrive with one of your own pages as the referrer, and that one should be close to zero.

Status codes are an interface, and the audience is machines that will never read your explanation. A 404 costs one line in a log and buys you a crawler that stops asking, a monitoring tool that can name the broken link, and a visitor who knows what happened instead of guessing. The blanket redirect buys a quiet error log and pays for it with dead addresses that look alive for years. When you are choosing between the two, ask whether an equivalent page exists: if it does, redirect to that page, and if it does not, say so plainly.