Add Robots.txt File into Website with Full Detail Analysis

We all know that in order to get better rank in SEO the importance of robots.txt file can not be ignored. It will help a blogger or and a web developer to increase the SEO (Search Engine Optimization) rank as well as it makes a blog seo friendly. Today I will discuss about each and every parts of robots.txt with full and details analysis. Our discuss matter will be like Why should a blogger add robots.txt file in to blogger? If there is no robots.txt file in a blog or website then what will happen. What is the importance of crawling and indexing a site and how a robots.txt file will help googlebot to do it? A robots.txt file’s every parts meaning and full instruction of using it with screenshots. If you do not use this file in your web server then no problem your site’s links will be properly indexed by the search engines. Then the questions will come that why should we use robots.txt file in our web server, blogger or any other platform? No problem I must discuss about these matters step by step.

What is Robots.txt File?

A robots.txt file is a normal file containing some codes with the extension of .txt It is not a complicated file at all and there is no complicated codes in it. This file is used in a web server in order to give direction to the Search Robots in crawling and indexing. This is called The Robots Exclusion Protocol. In nutshell, Which file directs the search engine in indexing the site with including and excluding is called robots.txt file. In details, We all know that when we will make a website or blogsite in any kind of web server then search engines like google, being, yahoo, etc will find the website and crawl it as well as index all the necessary things and finally show in search results. Mentioning that search engine will index at its own. If you want to disallow the search engines to index some parts of your websites or the full websites then you have to use a file in your web server where the direction will be noted for the search engines according to your wish. And finally the search engines will follow your file’s direction for indexing your site. And this file is called robots.txt file. Perhaps I could make you understand.

Why Should You Use Robots.txt File in Your Servers?

If you do not use any robots.txt file in your web servers then no problem, search engines will find out your site and index your links. So in this case you may think that you need not add robots.txt file in your blog or websites. Firstly we have known that a robots.txt file will forbade the search engines or any particular search engine to index a website or any particular parts of a websites. In a word it imposes restriction of crawling and indexing any websites by the search engines. So you may again think that if you don’t want to impose any restriction and you want to index everything of your blogsite then you need not use any robots.txt file. But I will say that You must add the robots.txt file in your web servers. Because….

[yes_list]

It helps the search engine robots to properly index your site.
It makes your blog SEO friendly.
It helps to increase your blog traffic and make popular.
It helps to increase the page rank and alexa rank.
It helps to make you professional and following terms and conditions.
It helps to hide any personal issues from search engines.
It makes your blog important to the search engines.
It earns honor for your sites.
Both the search engines and the traffic be satisfy.
[/yes_list]

 A Robots.txt File and Blogger Blogspot

From the very beginning I am telling about the robots.txt file and saying again again that you have to use a robots.txt file in your web server. Now the questions come that in blogger blogspot I can not access the web server, I can only digitally make pages and posts so how can I add robots.txt filein the bloger web servers. No problem google has made an option to include this file in your blogger blog sites. Every blogger blog site has automatically added a robots.txt file by default. You can not remove this file rather you can edit this file in the blogger dashboard. Robots.txt file will of course increase your SEO rank if you able to put it in your blogger dashboard in proper way. The default blogger robots.txt file is like the following

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://psc-jsc-ssc-hsc.blogspot.com/feeds/posts/default?orderby=UPDATED

How to Add rotots.txt File in to Your Own C-Panel Website

Adding Robots.txt file in to cpanel hosted website is very easer than any other blog or cms site. We all know that we have to upload/add the robots.txt file in the web server and when you will make a cpanel hosted web site then it will me more easily to upload the robots.txt file in to your root directory. You can upload through the cpanel automatic system or Filezilla or any other software you want.

How to Add or Enable robots.txt File into Blogger

Log in to blogger with your blogger ID and Password (If not Click Here)
Select the blog in which you want to add robots.txt file
Now click on “Setting” from the left side menu
Now click on Search Preferences

Now scrolling to the Crawlers and Indexing Sections
Look For The Custom Robots.txt
Now you will find that it was disabled and you have to enable it
Click on Edit. The whole process will be like the image below
By default the radio button will remain selected “No”
Click On Yes and a Box Will Appear Like The Image Below
Now Copy Below Following Code and Paste it in the box:

User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://www.yourdomain.blogspot.com/feeds/posts/default?orderby=UPDATED

Remember to change the address “yourdomain.blogspot.com” to your own domain (blog address)
Now click on save button
The whole process will be like the image below
After saving it will be seen like the image below
So finally you have successfully added robots.txt file into your blogger blogspot. Now I shall discuss about every parts of the robots.txt file. Let’s continue

Analysis of Different Part of Robots.txt File

It is a very simple matter which the robots.txt file means. You have to just mention a search engine name and give it direction to index your site or not. Nothing else. Now see a result where we sill say google search engine Not to index my site. And the codes will be like below
User-agent: Mediapartners-Google
Disallow: /

Explanation:
Now see the codes where we will say the google search engine to index my site. And the codes will be like below
User-agent: Mediapartners-Google
Disallow:

Explanation:
Now we will see an example of codes where we will see all the search engines Not to index my site. In this case you don’t need to particularly mention all search engine names. Just follow the below codes
User-agent: *
Disallow: /

Explanation:
Now we will see an example of codes where we will see all the search engines to index my site. Again in this case you don’t need to particularly mention all search engine names. Just follow the below codes
User-agent: *
Disallow:

Now we will see an example of codes where we will say only some particular search engines to index my sites with all links. And the code will be like below
User-agent: Mediapartners-Google
Disallow:
User-agent: being
Disallow:
User-agent: BadBot
Disallow:

Now we will see an example of code where we will say some particular search engines Not to index some particular pages and categories of my site. And the codes will be like below
User-agent: Mediapartners-Google
Disallow: /privacy.html
Disallow: /personal-content/
User-agent: being
Disallow: /proxy.html
Disallow: /search
User-agent: BadBot
Disallow: /members/

Now we will see an example of code where we will say some all the search engines Not to index some particular pages and categories of my site. And the codes will be like below
User-agent: *
Disallow: /privacy.html
Disallow: /personal-content/
Disallow: /search

Now we will see an example of code where we will say some particular search engines to index some particular pages and categories of my site. And the codes will be like below
User-agent: Mediapartners-Google
Allow: /about-us.html
Allow: /post/
User-agent: being
Allow: /help.html
Allow: /education/
User-agent: BadBot
Allow: /members/

Now we will see an example of code where we will say all the search engines to index some particular pages and categories of my site. And the codes will be like below
User-agent: *
Allow: /privacy.html
Allow: /contact.html
Allow: /about-us.html
Allow: /post/

Now we will see an example of code where we will say only google to index my site and the other roots Not to index my site. The codes will be like bellow
User-agent: Mediapartners-GoogleDisallow: User-agent: *Disallow: /

Finally we will see an example of code where we will say google to index my whole sites with all links and we will the all search engines to index my whole sites with all links except the in page search results. We will also say the search engines to follow a sitemap. So the code will be like below.
User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /

Sitemap: http://www.yourdomain.blogspot.com/feeds/posts/default?orderby=UPDATED

After the Long Journey, Dear Followers
One thing bear in mind that robots.txt is not a page it is a file. It will not stay in your pages but the root servers. Search Engines will first enter your servers and find out the robots.txt file and index the pages according to the direction noted in the file. I have said all about the codes inside the file. So if you face any problem to understand any single part of this post so please convey me through the comments. I will try my best to solve your problems. So finally dear followers, this writing has come out from my little knowledge. If you have any broaden idea about this or any better suggestion, so please share with us. If you help one today, the others will help you tomorrow. Thanks to all and happy blogging.

Related Posts