Disallow indexing of /latest and /search pages.
Patch by mark.
diffs (20 lines):
diff -r 3918d61981eb -r 90f4d2c698ae htdocs/robots.txt
--- a/htdocs/robots.txt Sat Oct 31 05:39:58 2009 +0000
+++ b/htdocs/robots.txt Sat Oct 31 20:29:21 2009 +0000
@@ -1,6 +1,8 @@ User-Agent: *
User-Agent: *
Disallow: /directory.bml
Disallow: /directory
+Disallow: /latest
+Disallow: /search
#
# Blocked journals aren't listed here because robots.txt files
@@ -9,6 +11,6 @@ Disallow: /directory
# Instead, blocked journals have HTML inserted in them which
# should prevent behaved spiders from indexing it.
#
-# Note that http://username.livejournal.com journals have an
+# Note that http://username.dreamwidth.org journals have an
# autogenerated robots.txt, since it can be small.
#
_______________________________________________
dw-changelog mailing list
[email protected]
http://lists.dwscoalition.org/mailman/listinfo/dw-changelog
