Generate and Add a robots txt file to Blogger Websites

Create and Submit robot.txt file for blogger wordPress Shopify wix to Google Search Central

What is robot in SEO (Search Engine Optimisation ?

Simply, Robot is the communicator between the websites and the Server. Robot file is the Heart of the website. It contains a list of URLs of the website and Instructs / Guides the crawls (Google / Bing / MSN / Yahoo / Yandex ect,.) which is to be crawled and which should not be crawled. It is the Main GATE for the crawlers to Enter your Website. So, don't neglect to create this robot file.


Example, Robots are like the Postman where one can serve the Post cards to the right customer. Here, Robot is the Postman and the website is the right customer. Technically, The task of robots is to collect the Information from the websites and to provide the information to the relevant Searches(Users).
Robots can also called as Bots, Crawlers, Spiders.


A sitemap is a file (usually in XML) that lists all (or a selection of) URLs on a website, along with metadata like the last modified date, change frequency, and relative priority.


How long does it takes to fetch the robot file rules ?

Robot file is considered as the Heart of the website and also called as the Main GATE of the Website. If any Crawlers want to enter your website means, first these spiders / bots should Obey and follow the instructions in the file added by the website owners. So, while writing the robots.txt file instructions be carefull and never neglect to add the file. If you want your website URLs to be crawlled and Indexed fast means, Robot file can do it.


How to check whether the Robots file is Added Properly or Not

Robots file can be checked in 2 Methods.

  1. By Website
  2. Simply, In the browser, Enter your website and at the end add "/robots.txt". Thats it!! You can see all the Instructions of your website in the browser itself.

    NOTE: First, Generate and Add the robots.txt file from the below tool Before checking the robots file.


  3. By Search Console
  4. In Google Search Console, navigate towards Settings section robots.txt files. Thats it!! You can see all the Fetched URLs details with the file size here.


    How to check whether the robots file is Added Properly or Not in Google Search Console.






A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.




Types of Crawlers


There are different types of Crawlers for different browsers

How to create and submit robots.txt file for Blogger


Technically, First Know your Websites(URL / Locations) Properly and Completely.
For Example, Your website may consists of Private locations and You don't want Crawlers to crawl those URLs. In this case, You can request the bots to NOT CRAWL those pages by Instructing in the robots.txt file.

It may be yours : -
  • admin / logIns Page
  • Payment Page (If it is an eCommerce / Bussiness Website).
  • Plugins / Resources Page.
  • Some Importent Files / Invoices ect.,

For Example, Your website is - https://www.yourwebsite.blogspot.com/
Then, your website consists of -
https://www.yourwebsite.blogspot.com/login-page
https://www.yourwebsite.blogspot.com/payment-page
https://www.yourwebsite.blogspot.com/plugins
https://www.yourwebsite.blogspot.com/Invoices


If you don't want to crawl those pages / URLs. Then,
Instruct / write in the robots.txt file as -
Disallow:/https://www.yourwebsite.blogspot.com/login-page
Disallow:/https://www.yourwebsite.blogspot.com/payment-page
Disallow:/https://www.yourwebsite.blogspot.com/plugins
Disallow:/https://www.yourwebsite.blogspot.com/Invoices




How to create and submit robots.txt file for WordPress



Limitations of creating the robots.txt file


  • robots.txt file should be appended only to the root URL.
  • robots.txt file size should not exceed to 500 KiB (Kilobytes).



Free robot txt file generator

Create robots file Instantly for blogger

Design and Submit robots txt file to Google Search Central

Free Advanced robots.txt file generator for Blogger and WordPress

Write and Submit Advanced robots Instructions file for any Browser(Google, Bing, MSN, Yahoo, Baidu)