Crawling and Indexing
Are you making efforts to improve your SEO and not seeing results? You may have assumed this is because your efforts are unsuccessful. But this may not be the case. It could be down toa delay Google crawling and indexing your webpages. Here’s an explanation as to why, and advice on what you can do to overcome this.
When you make changes to your website’s pages, they need to be re-crawled by Google. This process involves them being vetted, passing through complex algorithms, so Google can decide how best to index them in the search results. If your pages are deemed high quality, they will be re-indexed to a new rank position. The process of SEO seeks to constantly improve this.
According to a recent reply to a publisher from Google’s John Mueller, URLs are crawled by Google at different rates.He said:
“I think the hard part here is that we don’t crawl URLs with the same frequency all the time. So, some URLs we will crawl daily. Some URLs maybe weekly. Other URLs every couple of months, maybe even every once half year or so”.
So, this can be as infrequent as every 6 months! This means that if you have spent a couple of labour-intensive months executing your SEO strategy, it may take up to half a year for the changes to show. Pretty frustrating huh? But at least it’s encouraging to the degree that you may still see the fruits of your labour, albeit it with a time lag.
Can I Do Anything to Speed up the Rate that Google Crawls and Indexes My Site?
In a word, yes you can. You can use the sitemap update function to push through these changes and serve an extra reminder to Google that things have changed. This is especially good if you have a large number of pages you want to be re-indexed.
The sitemap update function is simply a re-submission of your sitemap to Google and you can do this via Google Search Console if you have this set up, or via your CMS, depending on the platform you use.
Technically you shouldn’t need to re-submit a sitemap every time you make changes, but for the process of speeding things up, this is a worthy exercise and takes just a few minutes.
Another way to do this, is to use the URL Inspection Tool. This is really useful for individual URLs or small sets of URLs. If you have a time-sensitive campaign with a landing page on your site you want to be indexed quickly, this is a great option.The URL Inspection Tool also be found in the Google Search Console (formerly called Webmaster Tools).
If you haven’t yet got Google Search Console linked up to you site, we’d advise doing do. It’s a free tool and has a whole host of useful functions to keep abreast of any issues with your site’s pages and tools on how to fix them, to maximise your SEO.