This Girl's Ass Shake will Make Your Jaws Drop ; Hottest Fitness Instructor - Eva Andressa ; U-S-A ... U-S-A ... U-S-oooooh ; Awas Uang Koin Palsu, Cocok-kan dulu koin-mu Dengan Gambar ini ; GIF - Dancer Embarassed Of Her Pussy
::| Keyword:       [Advance Search]
  Technology » Software & Internet » Webmaster  


Robots.txt and SEO

Friday, 05.06.2011, 08:58am (GMT-5)

Most webmasters know that SEO is very important, and sometimes the search engine robots, or spider, or crawlers get in the way and crawl some unwanted pages or folders in our websites. We can prevent this using robots.txt.

Robots.txt is seen by search engine spiders as the leaders to index which files or folders and which ones are not to be indexed. Robots.txt is very important file if we are working on e-commerce sites. If you e-commerce site has more than 1000 products meaning more than 1000 pages, robots.txt acts as the captain for these pages.

These robots.txt are being called by different names by different search engines, these are the list of the names:

Google Robot: googleBot
Yahoo Bot : slurp
MSN Bot : msnbot
Altavista : scooter

If you are using adsense or google analytics, then you are dealing with googlebot, you should include this in your robots.txt.

If you want to invite all crawlers to crawl and index all pages and root directories, no need to write more tags, simple write this one:

User-agent: *

* is wild card character and allow robots to crawl all pages on your site.

If you want to disallow some file(s) or folder not to be crawled or indexed, just write,

Disallow: /folder name/
Disallow: /file-name.htm

Mention Sitemap in robots.txt

You can mention xml, htm(l), php or simple txt sitemap with proper direction in robots.txt

Related Articles:
How To Boost Your Site Traffic For Free
Tips For Higher Page Rank
Google Page Rank
Meta Tags and SEO
Five Simple Steps To Improve Your Page Rank Through SEO
How To Prepare Your Article or Page For Search Engines
What is Search Engine Optimization (SEO)?
Cara Mudah Meningkatkan Pengunjung Blog

Re-Think $5.99 .COM!