How to use Robots.txt How to Use Robots.txt This information will hopefully help you when setting up your own robots.txt . I know that you can control spiders from search engines as well as more malicious sources. The only tools you need are notepad, and access to your root folder via ftp. What is Robots.txt? [...]
Posts tagged as:
Robots exclusion standard
