Comments on: Robot txt mystery unraveled

By: Brian

Brian — Wed, 08 Jul 2009 17:09:54 +0000

Kikolani,

If the .com and .net have identical content, you should just use a 301 redirect on the .com to avoid the issue you talked about. (I assume you are using virtual hosts?) If you are using apache this is pretty easy to do with htaccess.

By: Kikolani

Kikolani — Fri, 03 Jul 2009 20:47:39 +0000

I have a tricky thing I’d like to use the robots for, but haven’t figured out how yet. Basically, my client has both the .net and .com of his domain. The site files are all hosted under .com, but .net is the primary domain that shows up in search results. However, if you type in any page.com, it will pull up the same content as any page.net, which seems like a duplicate content issue. Can I block the robots from crawling the .com site, even if the files are hosted on the .com site? I know, pretty odd.

~ Kristi
Kikolani´s last blog ..Fetching Friday – Resources Mashup, #FollowFriday, & Some Tennis Love

By: orovo

orovo — Fri, 03 Jul 2009 10:12:45 +0000

i am not feeling very shy for saying that i didn’t know at all about the robot.txt but after reading your post i got to know some basics & seriously i am still in search of some information about the robot.txt & i am going for the forum discussion !Thanks for your perfect topic as robot .txt!it is very helpful to control bugs from the search engines!

By: Tycoon Blogger @Make Money Blogging

Tycoon Blogger @Make Money Blogging — Fri, 03 Jul 2009 02:46:13 +0000

That is way over my head. I think I will outsource this to some one. Thanks for breaking it down though as I was not familiar with this.
Tycoon Blogger @Make Money Blogging´s last blog ..Twitter Voyeurism

By: Ste@DS Bundle

Ste@DS Bundle — Thu, 02 Jul 2009 20:50:24 +0000

Some great tips here, robots.txt has always mystified me.

By: bbrian017

bbrian017 — Thu, 02 Jul 2009 12:57:28 +0000

Great read John! Ironic but I just added a robots.txt file to my social network about 8 weeks ago and since the pr update I got a page rank of 4. I get a lot of spam on the upcoming.php page and told the SE’s to not index of follow that link. I also added other pages that showed duplicate content.

This is my file here,

# All robots will spider the domain
User-agent: *
Disallow: /templates/
Disallow: /3rdparty/
Disallow: /libs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /internal/
Disallow: /backup/
Disallow: /thickbox/
Disallow: /api/
Disallow: /evb/
Disallow: /avatars/
Disallow: /admin_index.php
Disallow: /admin
Disallow: /login.php
Disallow: /js/
Disallow: /img/
Disallow: /upcoming.php

p.s. I engaged this article!
bbrian017´s last blog ..7 Days 7 Colours Thailand | Thailand Art Photography