In a webmaster hangout, a publisher asked how fast Google removed pages from the index if they added a noindex nofollow to it. The publisher stated they had added noindex but the page remained in Google’s index. Google’s John Mueller responded with an answer that described how often some pages are indexed.

John Mueller revealed that URLs are crawled at different rates. That’s somewhat well understood. What was of more interest was that he said some URLs can be crawled as little as once every six months.

The publisher stated:

“We’re seeing stuff that’s from a long time ago, where we’ve changed the noindex nofollow but we’re still seeing it in the index. And this is several months after we’ve changed this.”

John Mueller answered:

“I think the hard part here is that we don’t crawl URLs with the same frequency all the time. So some URLs we will crawl daily. Some URLs maybe weekly. Other URLs every couple of months, maybe even every once half year or so.

So this is something that we try to find the right balance for, so that we don’t overload your server.

And if you made significant changes on your website across the board then probably a lot of those changes are picked up fairly quickly but there will be some leftover ones.

So in particular if you do things like site queries then there’s a chance that you’ll see those URLs that get crawled like once every half year. They’ll still be there after a couple of months.

And that’s kind of… the normal time for us to kind of reprocess/re-crawl things. So it’s not necessarily a sign that something is technically completely broken.

But it does mean that if you think that these URLs should really not be indexed at all, then maybe you can kind of back that up and say well here’s a sitemap file with the last modification date so that Google goes off and tries to double-check these a little bit faster than otherwise.”

Use The Site Map to Trigger Updated Crawling

John Mueller suggested updating the site map and letting Googlebot discover the last modified date and using that as a hint for it to go out and crawl the old web pages.

Google URL Inspection Tool

Something John Mueller didn’t mention is using Google’s URL Inspection tool. According to Google’s Webmaster Help page on re-indexing,  a submission can take up to a week or two.

The URL Inspection tool is useful if you have a few individual URLs that need re-crawling. If you have a large amount of web pages, Google recommends submitting a site map instead.