Hi,
we've had similar issues with GSA. There are two things that helped
here:
Crawlers on that list in sitemap.xmap get special treatment with
regards to the 'last modified' date of files (if-modified-since); if
gsa isn't on the list then it will re-request the same file over and
over and over again.
I just made a pull request for this:
https://github.com/DSpace/DSpace/pull/1435
cheers,
Andrea
On 14/06/16 20:55, Louise Gruenberg
wrote:
The American Library Association has an archive
( https://alair.ala.org/) powered by DSpace Direct. We are
attempting to implement federated search across about 100 web
properties that shouldn't have so much content that they exceed
our license limit. The ALAIR has about 5,000 digital records but
the search appliance crawls about a quarter of a million on the
site. The /handler director is responsible for 183,000+. Does
anyone know what we would need to restrict to knock the crawl
back to a reasonable level?
--
You received this message because you are subscribed to the Google
Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.
--
Dr Andrea Schweer
Lead Software Developer, ITS Information Systems
The University of Waikato, Hamilton, New Zealand
+64-7-837 9120
--
You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/dspace-tech.
For more options, visit https://groups.google.com/d/optout.
|