How to use Robots.txt

How to Use Robots.txt This information will hopefully help you when  setting up your own robots.txt . I know that you can control spiders from search engines as well as more malicious sources. The only tools you need are notepad, and access to your  root folder via ftp.

What is Robots.txt? Robots.txt are special rows of text  that can be uploaded to server who’s primary function is to control Search Engine robots.

What are robots  you ask? Well most  Search Engine robots are commonly referred to as spiders. These spiders filter through your websites via links and make records of everything they find. This given is then used in different ways by the search engines to generated search results. Why would the Want to Control Spiders? Good question, since the robots are generally doing to good things  why would you want to hinder or stop them? Well the truth is  there are lots of reasons why you would want to control the spiders here are  a few examples of where robots.txt  can be handy.

Under Construction When your site is  under construction its not really much use to users or spiders. Prevent them indexing half or broken sites by using robots.txt  So you can created a  positive first impression when you’re ready to launch.

Bad Robots Unfortunately not all robots are friendly, robots.txt will help you stop the content and email scrapers as well as specific Search Engine spiders you don’t want checking your content or ones which are overloading your serveur. Search Engines Look for Them.

Most search engines I know check for to robots.txt.  Google even gives you information on what they have found in your robots.txt rows in webmaster tools.

Duplicated or Sensitive Content

There are plenty of legitimate reasons  for duplicate content both internally and externally. If you feel this is enough to endanger your site  in certain search engines you may want to stop them crawling the sensitive pages. There is also a possibility the content on sensitive loads of your pages is irrelevant or  you may wish to exclude these pages as well.

How to use Robots.txt Robots.txt is pretty easy to use, to created you just use notepad and follow the right syntax. Luckily this is also very similar and is comprised of only a  few elements. The rows are  made up of several records and each record features to user-agent section and to disallow section. You may choose to includes comments with the use of to ” #” at the start of  the lines. The ” user-agent” section defines which robots should follow the command.

To list multiple robots simply use multiple user-agent lines. You can also use the wildcard character ” *” to force ALL robots to obey the following command. An example of this line is below. User-agent: *

The ” Disallow” section is used for specifying the directory or rows that should not be accessed. It is fairly simply sytax and you just need to includes the directory (excluding you base URL) or rows similar to the following example.

Again you can have multiple lines to disallow to selection of files or directories.

Disallow: /folder1/ The ” allow” command can also like in useful, it lets you specify to specific rows to allow in to directory you may have disallowed. It still needs to be paired with to User-Agent command but can be added in with disallow commands. Here is an example of to full record using all of the above. User-Agent: * Disallow: /folder1/ Disallow: /folder2/ Disallow: /folder3/ Allow: /folder1/important.html Allow with all the robots getting to your site?

Just use this simple tails to open the gates and ensure you’re not blocking anything User-Agent: * Disallow: Block All Situated down for to while for construction? This will block everything from spidering. Don’t forget to remove it when you’re done! User-Agent: * Disallow: / Linking to your Sitemap

A handy little trick that loads robots allow is to link to your sitemap inside your robots rows. Simply use the following example and modify the URL to suite your own domain. Sitemap: http://www.example.com/sitemap.xml

Uploading Your Rows Once you’re ready you can upload the files to your server. It MUST be placed in the root of the domain. If you want to test that the right robots have access to the right pages the Google Webmaster Tools section has a good tool. Hope this  has helped you understand the loads of things you can do  with robots.txt and will help improve SEO and  secure your blog site.

Tagged as: Google Webmaster Tools, Robots exclusion standard, Search, Search engine, Search engine optimization, Searching, Uniform Resource Locator, Web page, Web search engine, Website