How to Audit Stale URLs Before Google Keeps Rediscovering Them

The fastest way to fix stale URLs is to stop guessing where they came from.

Google Search Console can make an old URL feel like a current problem. It may show a 404, a redirect, or a “crawled - currently not indexed” warning for a URL that no longer exists in the current version of the site.

That does not always mean the live site is broken.

Sometimes Google is working from memory. Sometimes an old sitemap entry existed for a while. Sometimes a post moved. Sometimes an internal link, alias, tag, category, or redirect still points to the stale path.

This is the audit workflow I use before adding redirects or changing content. It builds on the earlier posts in this series:

• How I Fixed Google Search Console Indexing Issues With a Hugo and Netlify Site
• Netlify Redirects for Hugo Sites: A Practical Cleanup Guide

The goal is simple: prove whether a stale URL is still being generated, linked, redirected, or remembered.

Start With The URL, Not The Error Label

Search Console labels are useful, but the URL is the real evidence.

For each reported URL, I put it into one of these buckets:

1. Current canonical URL
2. Old URL with a real replacement
3. Old URL with no replacement
4. Accidental URL generated by front matter, aliases, tags, or categories
5. Internal link still pointing to an old path
6. External memory from Google or another site

That classification matters because each bucket has a different fix.

For example:

• a current canonical URL should return 200
• an old URL with a replacement should return a direct 301
• an old URL with no replacement can return 404
• an accidental taxonomy URL should be fixed in front matter
• an internal stale link should be corrected at the source

Do not add redirects before doing this audit. Redirects are useful, but they should not be used to hide source problems.

Create A Small Audit List

I start by reducing the problem to slugs.

Example stale URLs:

/post/cyber-security/dark-web/
/post/docker-without-sudo/
/post/api-authentication-and-authroization/
/tags/tag1/
/writing/

From that, I build a search pattern:

cyber-security/dark-web|docker-without-sudo|api-authentication-and-authroization|tag1|/writing/

This keeps the audit focused. I am not searching randomly through the project. I am searching for the exact traces that could regenerate or relink the stale URL.

Search The Source With rg

I use rg because it is fast and works well across a Hugo repository.

The first pass searches the source areas that can generate public URLs:

rg -n "cyber-security/dark-web|docker-without-sudo|api-authentication-and-authroization|tag1|/writing/" `
  "$SITE_ROOT\content" `
  "$SITE_ROOT\layouts" `
  "$SITE_ROOT\static" `
  "$SITE_ROOT\netlify.toml"

This tells me whether the stale URL is still present in:

• post body content
• front matter
• aliases
• static redirects
• layout links
• navigation or footer code
• Netlify redirect rules

If the only match is in netlify.toml or static/_redirects, that may be fine. It means the stale URL is being handled as a redirect source.

If the stale URL appears in article body content, a menu, a partial, or front matter, I fix that first.

Check Slugs, URLs, And Aliases

Hugo gives several ways to influence published URLs.

The fields I check first are:

slug:
url:
aliases:

These fields are useful, but they can also preserve old URLs longer than expected.

For a post, I want the canonical slug to be explicit when the filename and final URL might otherwise be confusing:

slug: api-authentication-and-authorization

Aliases are different. They intentionally generate redirect pages for old URLs.

Example page aliases:

aliases:
  - "/connect/"

That is useful when the old URL is a known public route and the replacement is obvious.

But aliases should still be intentional. If a stale URL appears in aliases, I ask:

Is this old URL still useful as a redirect source?
Does it point to the correct current page?
Would a Netlify redirect be clearer?

For cleanup work, I prefer one clear redirect strategy. On my site, most post-level URL moves now live in netlify.toml, while some simple page moves live in static/_redirects or aliases.

Search For Internal Links

Internal links are one of the easiest ways to keep stale URLs alive.

If a post still links to an old URL, Google can rediscover it even after the page itself is gone.

I search markdown and layout files together:

rg -n "\]\(/post/cyber-security/dark-web/|\]\(/post/docker-without-sudo/|href=\"/writing/|href='/writing/" `
  "$SITE_ROOT\content" `
  "$SITE_ROOT\layouts"

The exact pattern changes depending on the URL, but I am looking for:

• Markdown links: [text](/old-url/)
• HTML links: href="/old-url/"
• shortcode parameters containing old paths
• navigation links
• CTA links
• footer links

If I find an internal stale link, I update it to the canonical URL.

This is better than relying on a redirect. Internal links are signals. A site should signal its current structure, not keep pointing readers and crawlers through old paths.

Search Redirect Files Separately

After checking content and layout links, I inspect redirect rules directly.

rg -n "cyber-security/dark-web|docker-without-sudo|api-authentication-and-authroization|/writing/" `
  "$SITE_ROOT\netlify.toml" `
  "$SITE_ROOT\static\_redirects"

At this stage I want to see one of two things.

For old URLs with real replacements:

old URL -> direct 301 -> canonical URL

For old URLs with no real replacement:

no redirect

The mistake is redirecting every unknown old URL to the homepage. That may reduce visible 404 counts, but it creates a weaker site structure and makes future audits harder.

Build With A Clean Output Directory

Generated output can lie if old files are still sitting in public/.

So I rebuild with a clean destination:

hugo --source "$SITE_ROOT" `
  --config "$SITE_ROOT\hugo.toml" `
  --cleanDestinationDir `
  --minify

This removes stale generated files before Hugo writes the new site.

That matters because a leftover file in public/ can make it look like Hugo is still generating an old page when it is not.

Check Whether The Old Page Still Exists

After the clean build, I check for generated pages.

For example:

Test-Path "$SITE_ROOT\public\post\cyber-security\dark-web\index.html"
Test-Path "$SITE_ROOT\public\post\inside-the-dark-web\index.html"

The result I want is:

old path: false
canonical path: true

If the old path still exists, I do not add a redirect yet. I first find out why Hugo is generating it.

Possible causes:

• url: front matter
• slug: front matter
• aliases:
• nested content path plus permalink rules
• duplicate content file
• old generated files because the build was not cleaned

Check The Sitemap

The sitemap is one of the strongest signals in a static site audit.

I search the generated sitemap for stale paths:

rg -n "cyber-security/dark-web|docker-without-sudo|api-authentication-and-authroization|tag1" `
  "$SITE_ROOT\public\sitemap.xml"

For stale URLs, I want no matches.

Then I check for the canonical URLs:

rg -n "inside-the-dark-web|api-authentication-and-authorization|how-to-execute-docker-commands-using-non-root-users" `
  "$SITE_ROOT\public\sitemap.xml"

For current pages, I want matches.

This distinction is important:

• redirect-source URLs should not be in the sitemap
• canonical final URLs should be in the sitemap
• deleted pages should not be in the sitemap
• accidental tag pages should not be generated from placeholder tags

Check Accidental Tags And Categories

Stale URLs are not always post URLs.

Search Console can also report taxonomy pages such as:

/tags/tag1/

If that tag came from test front matter, the fix is not a redirect. The fix is to remove the bad tag from the content.

Search for placeholder taxonomy values:

rg -n "tag1|tag2|tag3|sample tag|test tag" "$SITE_ROOT\content"

Then replace them with real tags:

tags:
  - SSH
  - Security
  - DevOps

Taxonomy pages should exist because they help readers browse the site, not because test metadata slipped into a post.

Check Hugo Aliases Carefully

Hugo aliases create redirect pages in the generated output.

They are useful for content moves, but during a stale URL audit I treat them as active routing decisions.

Search all aliases:

rg -n "aliases:" "$SITE_ROOT\content"

Then inspect nearby lines:

rg -n -C 3 "aliases:" "$SITE_ROOT\content"

For every alias, I ask:

• Is this still needed?
• Does it point to the correct canonical page?
• Is the old URL also handled in netlify.toml or static/_redirects?
• Could this create duplicate or confusing redirect behavior?

I do not remove useful aliases blindly. But I do want to know they exist before deciding whether a URL is stale, redirected, or still intentionally supported.

Check For Old URLs In Static Assets

Sometimes old URLs live outside markdown.

Search the static folder:

rg -n "/post/|/writing/|/connect/|/tags/" "$SITE_ROOT\static"

This catches:

• static HTML
• JavaScript links
• JSON data
• _redirects
• manually created assets that include links

If a static asset links to an old URL, Google can still find it depending on how that asset is exposed.

Do Not Edit public/ As The Fix

If the stale URL appears in public/, that is evidence, not the source of truth.

The fix should happen in:

• content/
• layouts/
• static/
• data/
• netlify.toml
• Hugo config

The generated public/ directory should be disposable. Rebuild it after fixing the source.

My Stale URL Audit Checklist

This is the checklist I use before deciding what to do with a stale URL:

1. Copy the exact URL from Search Console.
2. Extract the meaningful slug or path fragment.
3. Search content, layouts, static files, and netlify.toml with rg.
4. Check slug, url, and aliases front matter.
5. Search markdown links and HTML href values.
6. Search redirect files separately.
7. Build Hugo with --cleanDestinationDir.
8. Test whether the old generated page exists under public/.
9. Test whether the canonical page exists under public/.
10. Search public/sitemap.xml for the stale URL.
11. Search public/sitemap.xml for the canonical URL.
12. Fix source links before adding redirects.
13. Add direct 301 redirects only when a real replacement exists.
14. Leave unrelated deleted pages as 404.
15. Validate old and final URLs after deployment.

What This Prevents

This audit prevents several common mistakes:

• recreating deleted pages just to satisfy Search Console
• redirecting unrelated URLs to the homepage
• adding redirects while the old URL is still internally linked
• keeping placeholder tag pages alive
• trusting stale files in public/
• requesting indexing for redirect-source URLs
• assuming every Search Console warning reflects the current site

The point is not to make Search Console quiet immediately. The point is to make the current site coherent.

Final Takeaway

A stale URL audit is a source-tracing exercise.

Before adding a redirect, I want to know whether the URL is still generated, still linked, still listed in the sitemap, still present as an alias, or simply remembered by Google.

Once I know that, the fix is usually straightforward:

• fix source metadata if Hugo is generating the wrong URL
• fix internal links if the site is rediscovering the old URL
• add a direct 301 if the old URL has a real replacement
• leave it as 404 if it is truly gone and unrelated

That discipline keeps the site clean and gives Google clearer signals over time.

• ← Previous Post