Whitefish Wednesday – learning about XML Sitemaps and Robots files

SEO tips and advice on Sitemaps.xml and Robots.txt

[transcript of audio]

Hi, there and welcome to another one of our Whitefish Wednesday videos. My name is Chris Surridge, I’m the Marketing Director over here. And today we’re gonna have a quick discussion about sitemaps and robots.txt files. This is something, because it’s just come up in conversation just recently and I’m seeing more and more of it happen, yet it’s such a basic SEO thing to do.  I really just want to touch base on it. So, this is gonna be a short, quick, brief video all about sitemaps and robots.txt files.

What are Sitemaps on your website – why are they useful?

Right, so, let’s just have a quick discussion about what sitemaps are. Sitemaps are basically roadmaps for Google, like a said direction. So, when someone comes onto your website or rather when a spider crawler comes onto your website, they can go to the sitemap file and have a directory of where all your pages are and in what type of hierarchical order they’ll be in as well. Sitemaps – they get uploaded into your Google Search Console or Google Webmaster Tools, as some older folk might know them as. It’s very, very simple. You should be having an XML sitemap, which you’re then gonna wanna submit to Google.

Google will find your pages anyway, but by doing so, you’re offering them a much easier alternative by giving them a roadmap and directions on where to go. So, you know, it’s like… If someone’s asking for directions, you know, give them the directions. It’s really quite simple.

Using YOAST to generate your sitemap.xml

If you’re using something such as a WordPress design and… I’d recommend that you upload something called “Yoast” Y-O-A-S-T, Yoast SEO, which is probably quite familiar for many, many people. The Yoast plugin is exceptionally good. One of the key things that it will allow you to do is to create a sitemap, an XML sitemap ready to go. There are actually a number of different types of sitemaps available, be that you submit whether it be a post, whether it be imagery, whether it be video like we’re doing now and so on and so forth. Yoast will handle all that for you straight away if you’re using a WordPress plugin. If you’re not using WordPress then maybe have a chat with your web developer and see how they’re gonna go ahead and run a quick crawl of your website to create an XML sitemap. And also have a look and see the frequency of that XML sitemap and how often it’s going to be updated.

frequency of sitemap update


Upload your sitemap to Google Search Console

Once you have that, it will be something like www.yourdomain.com/sitemap.xml. So, you’re gonna take that sitemap.xml, you’re gonna upload it to Google Search Console and that’s it. Away it goes. Give it a quick test. There is a facility to go ahead and test your sitemap first to make sure it’s all working fine. Give it a quick test and away it goes.

submitting your xml sitemap to google search console

Different types of sitemaps for your website

Now, just briefly, whilst I’m just talking about sitemaps, there are two different types of sitemaps. The XML sitemap is more of a sitemap that you would give for the likes of Google or Yahoo, Bing, whoever it might be, who’s doing the crawling and searching for your site. Another type of sitemap is an HTML sitemap. Now, don’t get the two confused. The HTML sitemap is more for real human engagement. An HTML sitemap is very, very similar. It’s still a set of directions. It’s like an indices or contents of a reading book or a textbook. You’ve got to place that somewhere so if someone here needs to find a particular page, for easy navigation on your website, a real human being can do that.

The two are slightly different. An HTML sitemap is designed more for engagement factors. An XML sitemap is there more as directional use for Google and the likes of other search engines.

Moving onto Robots.txt files and how they work

Whilst we’re on the topic, the other thing that we want to have a look at here is your robots file. Okay. So, your robots file or robots text file, robots.txt, similar type of thing, www.yourdomain.com/robots.txt.  Basically, this is a way that you’re instructing the different types of robots or spiders or crawlers that come to your website to navigate them to areas that you’d want them to index, and also detract them from areas that you don’t want them to index. So, for example, let’s say you have a login area of your website so, again, WordPress might be the wp-admin area of your website. Let’s say you don’t want them to touch that area or rather, you wouldn’t want search engine crawlers to crawl the back-end of your website. So, you’re going to disallow that, whilst you’ll allow all your other pages, you’re going to want to disallow robots from indexing those pages. Why? Because they’re private.

Likewise as well, if you had some sort of membership club or if you’re a solicitor, conveyancer, or an accountant and you’ve got a client portal where your clients can login to access their personal files and the status of their conveyance, and whatever it might be, again, you might want to go ahead and disallow those types of pages. Again, I haven’t got any screenshots here for you at the moment. So, I’m terribly sorry. This was meant to be just a short, sharp, and brief video. But realistically, what you’d want to do is you want to disallow those kind of pages.

Robots.txt is purely a guide, not a hard and fast rule

The robots.txt file is not a hard and fast rule. Google and other search engines do not have to abide by them. It’s more the case of… it’s a set of guidelines or rather a request, a polite request, that you’re saying to Google, “Hi, please, don’t go ahead and search on these particular pages. Please, do go ahead and search these other pages instead.” Anyway, look, that’s just a quick, short, sharp video here. I hope you’ve enjoyed it. We’re gonna have quite a few more for what we call “Whitefish Wednesday’s” where we’re gonna offer a little bit more advice.

More on Whitefish Wednesday videos

Behind me, we’re actually gonna have the whiteboard soon and we’re actually gonna go and get down to more nitty-gritty kind of looks and feels of different things. For now, if you ever want to get hold of us, please, contact me. My name is Chris, Chris Surridge for full and I’m the director here at Whitefish Marketing. Have a look at whitefishmarketing.co.uk. We’re a specialist SEO and online marketing agency along with web development services. Give us a shout, and hopefully, speak to you soon. All the best.


About Chris Surridge

Chris Surridge is an experienced Digital Marketing Director with a wealth of knowledge on Search Marketing Strategies and Conversion Analysis. His value is in strategic planning for client accounts, and his consultative services.