Generate and Add a robots txt file to Blogger Websites
What is robot in SEO (Search Engine Optimisation ?
Simply, Robot is the communicator between the websites and the Server. Robot file is the Heart of the website. It contains a list of URLs of the website and Instructs / Guides the crawls (Google / Bing / MSN / Yahoo / Yandex ect,.) which is to be crawled and which should not be crawled. It is the Main GATE for the crawlers to Enter your Website. So, don't neglect to create this robot file.
Example, Robots are like the Postman where one can serve the Post cards to
the right customer. Here, Robot is the Postman and the website is the
right customer. Technically, The task of robots is to collect the
Information from the websites and to provide the information to the
relevant Searches(Users).
Robots can also called as Bots, Crawlers,
Spiders.
A sitemap is a file (usually in XML) that lists all (or a selection of) URLs on a website, along with metadata like the last modified date, change frequency, and relative priority.
How long does it takes to fetch the robot file rules ?
Robot file is considered as the Heart of the website and also called as the Main GATE of the Website. If any Crawlers want to enter your website means, first these spiders / bots should Obey and follow the instructions in the file added by the website owners. So, while writing the robots.txt file instructions be carefull and never neglect to add the file. If you want your website URLs to be crawlled and Indexed fast means, Robot file can do it.
How to check whether the Robots file is Added Properly or Not
Robots file can be checked in 2 Methods.
- By Website
- By Search Console
Simply, In the browser, Enter your website and at the end add "/robots.txt". Thats it!! You can see all the Instructions of your website in the browser itself.
NOTE: First, Generate and Add the robots.txt file from the below tool Before checking the robots file.
In Google Search Console, navigate towards Settings section robots.txt files. Thats it!! You can see all the Fetched URLs details with the file size here.

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.
Types of Crawlers
There are different types of Crawlers for different browsers
How to create and submit robots.txt file for Blogger
Technically, First Know your Websites(URL / Locations) Properly and Completely.
For Example, Your website may consists of Private locations and You don't want Crawlers to crawl those URLs. In this case, You can request the bots to NOT CRAWL those pages by Instructing in the robots.txt file.
It may be yours : -
- admin / logIns Page
- Payment Page (If it is an eCommerce / Bussiness Website).
- Plugins / Resources Page.
- Some Importent Files / Invoices ect.,
For Example, Your website is - https://www.yourwebsite.blogspot.com/
Then, your website consists of -
https://www.yourwebsite.blogspot.com/login-page
https://www.yourwebsite.blogspot.com/payment-page
https://www.yourwebsite.blogspot.com/plugins
https://www.yourwebsite.blogspot.com/Invoices
If you don't want to crawl those pages / URLs. Then,
Instruct / write in the robots.txt file as -
Disallow:/https://www.yourwebsite.blogspot.com/login-page
Disallow:/https://www.yourwebsite.blogspot.com/payment-page
Disallow:/https://www.yourwebsite.blogspot.com/plugins
Disallow:/https://www.yourwebsite.blogspot.com/Invoices
How to create and submit robots.txt file for WordPress
Limitations of creating the robots.txt file
- robots.txt file should be appended only to the root URL.
- robots.txt file size should not exceed to 500 KiB (Kilobytes).
0 Comments
If you have any Doubts please don't hesitate to comment us.