How to use Robots.txt
How to Use Robots.txt This information will hopefully help you when setting up your own robots.txt . I know that you can control spiders from search engines as well as more malicious sources. The only tools you need are notepad, and access to your root folder via ftp.
What is Robots.txt? Robots.txt are [...]
Posts tagged as:
Web page
