Tom De Mulder wrote:
> You should also ensure that the robots.txt is available at the root of the
> server... ie.
> https://dspace.stir.ac.uk<https://dspace.stir.ac.uk/dspace/browse-title?bottom=1893/214>/robots.txt
> and not just
> https://dspace.stir.ac.uk<https://dspace.stir.ac.uk/dspace/browse-title?bottom=1893/214>/dspace/robots.txt
Ah, thanks Tom.
I had just tried Graham's suggestion of adding /dspace to the Disallow values,
but was still getting spammed with Errors so was just wondering why it wasn't
working . . .
I note that my robots.txt is available at:
https://dspace.stir.ac.uk/dspace/robots.txt
- but not:
https://dspace.stir.ac.uk/robots.txt
Any pointers how to make it visible at that location?
Thanks,
Mike
Michael White
eLearning Developer
Centre for eLearning Development (CeLD)
3V3a, Cottrell
University of Stirling
Stirling SCOTLAND
FK9 4LA
Email: michael.wh...@stir.ac.uk<mailto:michael.wh...@stir.ac.uk>
Tel: +44 (0) 1786 466877
Fax: +44 (0) 1786 466880
http://www.is.stir.ac.uk/celd/
From: Graham Triggs [mailto:grahamtri...@gmail.com]
Sent: 11 February 2010 15:11
To: Tom De Mulder
Cc: Michael White; dspace-tech@lists.sourceforge.net
Subject: Re: [Dspace-tech] Bad robot! Googlebot and Internal Server Errors
On 11 February 2010 14:37, Tom De Mulder
<td...@cam.ac.uk<mailto:td...@cam.ac.uk>> wrote:
You should add "/dspace" to the start of those disallowed patterns,
because your DSpace URLs start with "/dspace" after the hostname.
You should also ensure that the robots.txt is available at the root of the
server... ie.
https://dspace.stir.ac.uk<https://dspace.stir.ac.uk/dspace/browse-title?bottom=1893/214>/robots.txt
and not just
https://dspace.stir.ac.uk<https://dspace.stir.ac.uk/dspace/browse-title?bottom=1893/214>/dspace/robots.txt
G
--
The Sunday Times Scottish University of the Year 2009/2010
The University of Stirling is a charity registered in Scotland,
number SC 011159.
------------------------------------------------------------------------------
SOLARIS 10 is the OS for Data Centers - provides features such as DTrace,
Predictive Self Healing and Award Winning ZFS. Get Solaris 10 NOW
http://p.sf.net/sfu/solaris-dev2dev
_______________________________________________
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
