Adding robots.txt to your Ghost blog

Ghost is a wonderful new blogging platform. Earlier I blogged about protecting a Ghost installation with Nginx and how to configure SMTP.

One thing that is missing however is a robots.txt file, instructing web robots such a the Googlebot and the Baidubot to index or not to index specific files or posts on your blog.

Typically you want everything to be publicly available, but you do not want robots hitting your administrative pages and filling up the error-logs.

Luckily, it is quite easy to configure a robots.txt file. Remember however that this is not a security feature, malicious web indexers might still ignore whatever you put into that file.

Installation

Go to your Ghost installation directory (replace installation_dir with the real location on your server. Move to your current theme installation directory (in the example the default casper theme).

    cd installation_dir
    cd content/themes/casper
    nano robots.txt

Put the following content in the robots.txt file:

    User-agent: *
    Disallow: /ghost/

This instructs web spiders to index everything, but not the administrative pages.

If you use a proxy such as Nginx and use rewriting rules, you might want to change the Disallow: /ghost/ to whatever your remapping rules dictate.

You can easily check if the installation was successful by surfing to http://YOUR.DOMAIN.NAME/robots.txt (change the YOUR.DOMAIN.NAME to the URL of your blog).

That's it! Easy, wasn't it? Have fun!

References

All you never wanted to know about a robots.txt file kan be found here.

Herman Stevens

Just some guy on the internet. Loves technology, diving, travelling, photography and Belgian Trappist beers.