Re: [Dspace-tech] Issue about Google crawler

2010-10-05 Thread Kim Shepherd
Hi Panyarak, It might be an idea to add /displaystats to your JSPUI's robots.txt and to any Google Webmaster Tools robots.txt files or Page Removal Requests. For Google to de-index pages, it generally likes to see a 404 (not found) or a 410 (gone). Unfortunately, the servlet that handles

Re: [Dspace-tech] Issue about Google crawler

2010-10-05 Thread Kim Shepherd
I should point out that my robots.txt suggestions assume you don't want any stats pages crawled at all... if that's not true, it's probably best to apply the patch for DS-689 and wait for Google to de-index (and make the robots.txt entries more specific if there are only a few invalid handles

[Dspace-tech] Issue about Google crawler

2010-10-03 Thread Panyarak Ngamsritragul
Dear all, A couple of weeks ago I have posted questions about Google crawler and sitemaps. There was a response from Vinit, but I still could not reach the solution to what I am experiencing. I am running 1.6.2 and have registered the site (kb.psu.ac.th) to Google's webmaster tools. I