A Complete Guide to Understand WordPress robots.txt file 2024

The robots.txt file instruct search engine bots on how to crawl and index the site's pages

Robot.txt is an incredibly powerful tool when it comes to your site SEO. When you publish a post Search engine bots crawl your site for indexing. To ensure that your site or post ranks high in Search Engine Result Pages (SERPs), you need to make it easy for search engine ‘bots’ to explore its most important pages. And a well-structured robots.txt file will help direct those bots to the pages you want them to index. Here this post explains the Importance of a Robots.txt File for Your SEO, and how to create a robots txt file for your WordPress site.

What is robots txt WordPress?

The robots.txt file, also known as the robots exclusion protocol or standard, allows you to deny search engines access to certain files and folders. Or tell web robots (search engines) which pages, and posts on your site to crawl.

The Importance of a Robots.txt File for Your SEO?

As explained before Your Robots.txt file tells the search engines which pages to access and index on your website on which pages not to. For example, if you specify in your Robots.txt file that you don’t want the search engines to be able to access your feedback page, that page won’t be able to show up in the search results and web users won’t be able to find it. Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.

Where is robots txt located in WordPress?

This file is generally located in your site’s root folder. You can view this file by connecting to your site by using an FTP client. You can also do it by using the cPanel file manager.

Where is robots txt located in WordPress

Do I need a robots txt file WordPress?

By default, WordPress sets up robots.txt file but isn’t accessible at all, from any directory. But if you want to make changes to it, you’ll need to create your own file and upload it to your root folder as a replacement.

How do I create a robots txt file?

Creating robots txt file for your site is very easy and simple

  • Open Notepad, Microsoft Word or any text editor
  • Save the file as ‘robots,’ (making sure to choose .txt as the file type extension)
  • Next, add the following two lines of text to your file:

User-agent: *
Disallow:

‘User-agent’ is another word for robots or search engine spiders. The asterisk (*) denotes that this line applies to all of the spiders. Here, there is no file or folder listed in the Disallow line, implying that every directory on your site may be accessed. This is a basic robot text file.

Blocking the search engine spiders from your whole site is also one of the robots.txt options. To do this, add these two lines to the file:

User-agent: *
Disallow: /

If you’d like to block the spiders from certain areas of your site, your robots.txt might look something like this:

User-agent: *
Disallow: /database/
Disallow: /scripts/

The above three lines tell all robots that they are not allowed to access anything in the database and scripts directories or sub-directories.

Note: Keep in mind that only one file or folder can be used per the Disallow line. You may add as many Disallow lines as you need.

Be sure to add your search engine friendly XML sitemap file to the robots text file. This will ensure that the spiders can find your sitemap and easily index all of your site’s pages. Use this syntax:
Sitemap: http://www.mydomain.com/sitemap.xml

Once complete, save and upload your robots.txt file to the root directory of your site.

www.mydomain.com/robots.txt.

Use Yoast to create a robots.txt file

SEO plugins like Yoast have an option to create and edit your robots.txt file from within your WordPress dashboard.

  • Install Yoast SEO and navigate to the Tools page under the SEO menu.
  • Here you can see the robots.txt file of your website/blog.
  • If you don’t have a robots.txt file, create one by hitting Create robots.txt file button.

Use Yoast to create a robots.txt file

If you don’t see an option for file editor, it is because you have harden WordPress security. You can enable the file editing option by removing the following line from the wp-config.php file.

define(‘DISALLOW_FILE_EDIT’, true);

Your existing Robot.txt file will appear as:

User-agent: *

Disallow: /

If you are using  All in One SEO open the Feature Manager page in your dashboard. Inside, you’ll find an option called Robots.txt, with a conspicuous Activate button right below it.

Do you have any queries about how to edit robots.txt in WordPress? Let us know in the comments section below! Also read:

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More