Author: pebender
Date: Wed Apr 15 03:06:46 2009
New Revision: 4608
Modified:
trunk/gar-minimyth/html/minimyth/robots.txt
Log:
- Updated www.minimyth.org robots.txt file.
Modified: trunk/gar-minimyth/html/minimyth/robots.txt
==============================================================================
--- trunk/gar-minimyth/html/minimyth/robots.txt (original)
+++ trunk/gar-minimyth/html/minimyth/robots.txt Wed Apr 15 03:06:46 2009
@@ -1,27 +1,6 @@
User-agent: *
-Disallow: /error
Disallow: /error/
-
-# Don't crawl linpvr.org redirects that do not exist on minimyth.org.
-User-agent: *
-Disallow: *.shtml
-Disallow: /art3
-Disallow: /art3/
-Disallow: /bugs
-Disallow: /bugs/
-Disallow: /dnload
-Disallow: /dnload/
-Disallow: /isoboot
-Disallow: /rel-0*
-Disallow: /util
-Disallow: /util/
-
-# Don't crawl the funky links caused by the domains subdirectory.
+# Don't crawl the download directory as it changes to often.
User-agent: *
-Disallow: /domains
-Disallow: /domains/
-Disallow: /bendercasa.net
-Disallow: /bendercasa.net/
-Disallow: /minimyth.org
-Disallow: /minimyth.org/
+Disallow: /download/
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"minimyth-commits" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/minimyth-commits?hl=en
-~----------~----~----~----~------~----~------~--~---
