Author: pebender
Date: Thu Apr 2 08:12:09 2009
New Revision: 4541
Modified:
trunk/gar-minimyth/html/minimyth/robots.txt
Log:
- Updated the robots.txt file for the minimyth.org web site.
Modified: trunk/gar-minimyth/html/minimyth/robots.txt
==============================================================================
--- trunk/gar-minimyth/html/minimyth/robots.txt (original)
+++ trunk/gar-minimyth/html/minimyth/robots.txt Thu Apr 2 08:12:09 2009
@@ -1,2 +1,27 @@
User-agent: *
+Disallow: /error
Disallow: /error/
+
+
+# Don't crawl linpvr.org redirects that do not exist on minimyth.org.
+User-agent: *
+Disallow: *.shtml
+Disallow: /art3
+Disallow: /art3/
+Disallow: /bugs
+Disallow: /bugs/
+Disallow: /dnload
+Disallow: /dnload/
+Disallow: /isoboot
+Disallow: /rel-0*
+Disallow: /util
+Disallow: /util/
+
+# Don't crawl the funky links caused by the domains subdirectory.
+User-agent: *
+Disallow: /domains
+Disallow: /domains/
+Disallow: /bendercasa.net
+Disallow: /bendercasa.net/
+Disallow: /minimyth.org
+Disallow: /minimyth.org/
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"minimyth-commits" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/minimyth-commits?hl=en
-~----------~----~----~----~------~----~------~--~---
