Comments on: Robot txt mystery unraveled http://potpolitics.com/2009/07/02/robot-txt-mystery-unraveled/ Just another WordPress site Wed, 23 May 2012 18:47:58 +0000 hourly 1 http://wordpress.org/?v=3.3.2 By: Brian http://potpolitics.com/2009/07/02/robot-txt-mystery-unraveled/#comment-5147 Brian Wed, 08 Jul 2009 13:09:54 +0000 http://potpolitics.com/?p=2637#comment-5147 Kikolani, If the .com and .net have identical content, you should just use a 301 redirect on the .com to avoid the issue you talked about. (I assume you are using virtual hosts?) If you are using apache this is pretty easy to do with htaccess. Kikolani,

If the .com and .net have identical content, you should just use a 301 redirect on the .com to avoid the issue you talked about. (I assume you are using virtual hosts?) If you are using apache this is pretty easy to do with htaccess.

]]> By: Raleigh Fence Company http://potpolitics.com/2009/07/02/robot-txt-mystery-unraveled/#comment-5004 Raleigh Fence Company Fri, 03 Jul 2009 16:47:39 +0000 http://potpolitics.com/?p=2637#comment-5004 I have a tricky thing I'd like to use the robots for, but haven't figured out how yet. Basically, my client has both the .net and .com of his domain. The site files are all hosted under .com, but .net is the primary domain that shows up in search results. However, if you type in any page.com, it will pull up the same content as any page.net, which seems like a duplicate content issue. Can I block the robots from crawling the .com site, even if the files are hosted on the .com site? I know, pretty odd. ~ Kristi I have a tricky thing I’d like to use the robots for, but haven’t figured out how yet. Basically, my client has both the .net and .com of his domain. The site files are all hosted under .com, but .net is the primary domain that shows up in search results. However, if you type in any page.com, it will pull up the same content as any page.net, which seems like a duplicate content issue. Can I block the robots from crawling the .com site, even if the files are hosted on the .com site? I know, pretty odd.

~ Kristi

]]> By: orovo http://potpolitics.com/2009/07/02/robot-txt-mystery-unraveled/#comment-5002 orovo Fri, 03 Jul 2009 06:12:45 +0000 http://potpolitics.com/?p=2637#comment-5002 i am not feeling very shy for saying that i didn't know at all about the robot.txt but after reading your post i got to know some basics & seriously i am still in search of some information about the robot.txt & i am going for the forum discussion !Thanks for your perfect topic as robot .txt!it is very helpful to control bugs from the search engines! i am not feeling very shy for saying that i didn’t know at all about the robot.txt but after reading your post i got to know some basics & seriously i am still in search of some information about the robot.txt & i am going for the forum discussion !Thanks for your perfect topic as robot .txt!it is very helpful to control bugs from the search engines!

]]> By: Ste@DS Bundle http://potpolitics.com/2009/07/02/robot-txt-mystery-unraveled/#comment-4994 Ste@DS Bundle Thu, 02 Jul 2009 16:50:24 +0000 http://potpolitics.com/?p=2637#comment-4994 Some great tips here, robots.txt has always mystified me. Some great tips here, robots.txt has always mystified me.

]]> By: bbrian017 http://potpolitics.com/2009/07/02/robot-txt-mystery-unraveled/#comment-4988 bbrian017 Thu, 02 Jul 2009 08:57:28 +0000 http://potpolitics.com/?p=2637#comment-4988 Great read John! Ironic but I just added a robots.txt file to my social network about 8 weeks ago and since the pr update I got a page rank of 4. I get a lot of spam on the upcoming.php page and told the SE’s to not index of follow that link. I also added other pages that showed duplicate content. This is my file here, # All robots will spider the domain User-agent: * Disallow: /templates/ Disallow: /3rdparty/ Disallow: /libs/ Disallow: /modules/ Disallow: /plugins/ Disallow: /internal/ Disallow: /backup/ Disallow: /thickbox/ Disallow: /api/ Disallow: /evb/ Disallow: /avatars/ Disallow: /admin_index.php Disallow: /admin Disallow: /login.php Disallow: /js/ Disallow: /img/ Disallow: /upcoming.php p.s. I engaged this article! Great read John! Ironic but I just added a robots.txt file to my social network about 8 weeks ago and since the pr update I got a page rank of 4. I get a lot of spam on the upcoming.php page and told the SE’s to not index of follow that link. I also added other pages that showed duplicate content.

This is my file here,

# All robots will spider the domain
User-agent: *
Disallow: /templates/
Disallow: /3rdparty/
Disallow: /libs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /internal/
Disallow: /backup/
Disallow: /thickbox/
Disallow: /api/
Disallow: /evb/
Disallow: /avatars/
Disallow: /admin_index.php
Disallow: /admin
Disallow: /login.php
Disallow: /js/
Disallow: /img/
Disallow: /upcoming.php

p.s. I engaged this article!

]]>