What is Robots.txt file & How to create Robots.txt file for WordPress site ?

Share this

Posted by on

( What is Robots.txt file )

What is Robots.txt file ? What is does ? Robot.txt file is important for SEO ? How ? How to create a Robots.txt file for WordPress site ?  Don’t worry !!! Just Read the post. In this post, I would like to tell you all, the secrets of Robots.txt file and how to create Robots.txt file for your website.

What is Robots.txt file ?

Robots.txt file is a very important file which tells Search Engine Robots about those files & folders on your website, you don’t want bots to Crawl and Index. Special care must be taken while creating the robots.txt file because one wrong practice can remove your site from Search Results.

The Robots.txt file must be placed in your root directory .

For example : http://www.3nions.com/robots.txt

What is Robots.txt file

What’s the use of Robots.txt file ?

If you are running a website, it is obvious You don’t want Search Engines to crawl or index some part of your website. This can be easily done with Robots.txt file. When web crawlers visit your website they read robots.txt file from which, they get an idea which part of your website they can actually access.

It is recommended to create a robots.txt file. Though if you do not have a robots.txt file on your site, It will not block any pages or files and the search engines will crawl and index your website.

What will happen if you do not have robots.txt file ?
When you do not have robots.txt file the search engine bots will crawl and index all your web contents (files, pages – everything ).

Should you use no-index meta tag to remove individual pages from Search Results ?

To block a page you should use noindex meta tag <meta name=”robots” content=”noindex”>because it would be more effective than Robots.txt.

How to create Robots.txt File ?

What is Robots.txt file

Well, Creating a robots.txt file is as easy as writing a note in notepad or in any text editor because the file is just a text file and which can be created using notepad, wordpad or any other text editor.

You will have to upload your robots.txt file to your site’s root directory using FTP or by using Cpanel file manager. If you are a WordPress (Self hosted) user then you can make use of Plugins to create robots.txt file.

To create robots.txt file using Yoast SEO, go to Yoast SEO>Tools>File-editor.

What is Robots.txt file

If you do not have a robots.txt file then create using any text editor. Open a text editor create a new text file and give it a name robots.txt then save your file. Now switch to your site’s file manager and upload to  site’s root folder/directory.

You can always view your robots.txt file by visiting the link : http://www.yourwebsite.com/robots.txt

Know robots.txt syntax

There are three keywords that we use to create robots.txt file and once you know them, you will say that was really very easy. So let’s get started with the Robots.txt syntax.

Syntax for using the keywords are :

User-agent:*  Means this section is applied to all search engine bots and you need not to mention their names.

If you include this line into your robots.txt file it will simply block all the contents and search engine bots will not access your website.

This means you are allowing all your web contents I.e. robots will have access over all your web contents.

Blocking a Folder

Once you have created Robots.txt file, you should test your robots.txt with the Google’s robots.txt tester. Robots.txt tester tool informs you about your blocked and unblocked Urls.

     Use Robots.txt Tester

This is example of my site’s Robots.txt file. If you don’t want any trouble then you can use mine but do not forget to change my sitemaps Url with yours.

I hope this post helped you to learn about Robots.txt and how to create robots.txt file.

Liked this post ? If yes then share it on facebook and twitter with your friends….

Additional readings :

What is Robots.txt file

Last updated: June 18, 2016

Authored by :

Satyansh Tiwari is the co-founder of 3nions and currently an Electrical Engineering Student. He is a blogger, writer, graphic designer and SEO enthusiast. When he is free, he loves experiencing nature and traveling.

Leave a Reply