What is Robots.txt file in terms of SEO

A robots.txt file determines which pages of your blog must be indexed and which should not be indexed. All  search engines use a web crawler robot to crawl and index your pages, with robots.txt file you can tell them which pages you want to index and which don't.

According to Wikipedia -
When a site owner wishes to give instructions to web robots they place a text file called robots.txt in the root of the web site hierarchy (e.g. https://www.example.com/robots.txt). This text file contains the instructions in a specific format. Robots that choose to follow the instructions try to fetch this file and read the instructions before fetching any other file from the web site. If this file doesn't exist, web robots assume that the web owner wishes to provide no specific instructions, and crawl the entire site.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) on how to crawl & index pages.

Points To Note;

  1. A robots.txt file must be located at the main domain with the name robots.txt, It cant be stored at http://example.com/afolder/robots.txt or it can't be stored like http://example.com/robots2.txt
  2. If there is no robots.txt file at specified place, search engines will assume that they have permission to crawl and index all your pages.
  3. A robots.txt file on a website will function as a request that specified robots ignore specified files or directories when crawling a site.
  4. A robots.txt file only covers one origin. For websites with multiple subdomains, each subdomain must have its own robots.txt file.

Example robots.txt File
The above lines tell every search engine to not index these three directories -


User-agent: *
Disallow: /images/
Disallow: /tmp/
Disallow: /template/


Now you know what a robots.txt file is.

Comments

Post a Comment

Popular posts from this blog

Flipkart Big Billion Day The Truth - Fake Offers and Out of Stock Products

Image
Flipkart is a reputed online shopping website for indian shoppers. few days back they released a TV advertisement and and put a banner on their website that the Big Billion Day sale is coming on 6th of october. They promised that they will offer a shopping experience like never before. But the truth is way worse than what they said. People are not able to purchase products and the server is crashing down and throwing a 500 internal server error. View Flipkart Big Billion Day Offers Flipkart Big Billion Day - Most products out of stock User reactions on facebook for big billion day sale Most of the users are saying that the flipkart big billion day deals are fake. and most of the products were out of stock before the offer even displayed on homepage. people were eagerly waiting for the deals on oct 6. but most of the people were not able to order a single product from flipkart because of the internal server error. Users not found big billion day's "Steal Deals...
Read more

Show Full Post on Front Page of Ghost, Not Excerpt

Image
I love writing about the Ghost platform. Its simple and just focused on blogging and that's why I love this platform. Some bloggers like to show full posts on homepage instead of post excerpts . By default Ghost(with default theme) displays excerpts on homepage. However if you would like to show full posts and not excerpts on the front page on your blog there is an easy way to change from excerpts to full posts on Ghost Homepage. Show full posts on Ghost Homepage : Tutorial Its very easy if you have access to your ghost blog by ftp/sftp or have Ghost hosted on your computer. you just have to be able to edit the index.hbs file. First open the index.hbs file. Find the above code: < section class="post-excerpt" > < p>{{ excerpt }}… » < /section > Replace it with: < section class="post-content" > {{ content }} < /section > You are done now, just save and restart Ghost. Warning: If you are displaying too much pos...
Read more

5 Beautiful Dark Blogger Templates

Image
Are you tired of looking at the default templates that Blogger provides? We decided to search for some of the different-looking beautiful dark Blogger templates and found these 5 cool dark blogger templates. Eclipse Dark theme that brings werewolves to mind. An enchanting template for the dark souls.Theme is designed beautifully with extra dark colors and glitter. Limax Another beautiful dark blogger template with striking colors. Butterfly Magic Another dark colors collection in this butterfly nagic blogger theme Xenon Xenon is a free dark blogger template adapted from WordPress with 2 columns, left sidebar and posts thumbnails. Cooperate Cooperate is a free blogger template adapted from WordPress with 2 columns, right sidebar, slideshow, social bookmarking icons, posts thumbnails, tabs, drop down menu and a background pattern. Excellent layout for blogs about any general topic. So we have given you 5 Cool Dark Blogger Templates, Now it;...
Read more