Customise The "Robots.Txt" File, For Your Blog

The file "Robots.Txt", used to manage search engine access to your blog, is found in the blog root folder.

You can't edit "Robot.Txt" directly - but you can upload a new file. Long ago, you would use Webmaster Tools - which is now called Search Console.

Now, we can update "robots.txt" directly, using the Blogger dashboard "Custom robots.txt" wizard, in Settings - "Search preferences".

Here's the link to the "Robots.Txt" file, for this blog.

http://fixtemplates.blogspot.com/robots.txt

To view / copy the current contents, just click on the link. You can do the same with the file, for your blog.

If I want to change the file for this blog, I start with the current file. Excepting one small detail (my blog is published to "fixtemplates.blogspot.com", yours is published to your chosen URL), your file will be identical - if you have not tweaked it.

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://fixtemplates.blogspot.com/sitemap.xml

Let's look at a simple example - adding a sitemap.

Some people would like to index the static pages, in their blog. That's a simple enough task - just add the sitemap, for static pages, to your file.

"sitemap-pages.xml", like "sitemap.xml", is automatically generated for each blog.

User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://fixtemplates.blogspot.com/sitemap.xml
Sitemap: http://fixtemplates.blogspot.com/sitemap-pages.xml

Start from Settings - Search preferences.


Start with an empty ("[Disabled]") "Custom robots.txt".



Click on "Edit".



Click on "Yes".



Paste in your edited file content. Note that you will overlay the current contents with what you paste - so paste the entire file, as you wish to have.




Click on "Save changes".

Select "Yes" for "Enable custom robots.txt content?", if necessary.




Right now, though, I don't want to change my file. So, I'll leave my custom file in the wizard, and leave "No" selected.



And the custom file is
"[Disabled]", again.



Even if disabled, I can get it back again, easily enough. Just click on "Yes", and there it is, again. For right now, though, I'll leave it "[Disabled]".


Just be aware of the dangers of editing, without knowing what effect a given entry has, on search engine activity. Remember that you can get the default file back, with a couple mouse clicks.