[
https://jira.duraspace.org/browse/DS-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=26922#comment-26922
]
Richard Rodgers commented on DS-1387:
-------------------------------------
Hi Reinhard:
I'm not sure, but you might be making an assumption that all Google crawlers
can do is follow links. My understanding is that they also can do 'heuristic'
exploration of a site's URL space: take an URL, transform it to a random new
one, and see if it returns something. This was a big problem for DSpace when it
was afflicted by this Cocoon bug: https://jira.duraspace.org/browse/DS-768
(200s returned for invalid pages) - it tripped up these exploratory crawls
because they never reached the 404 'boundary'. Google Scholar (IIRC) alerted us
to this problem.
So its sort of border-line whether you call it bug or not: DSpace lets you put
a resource policy on any bitstream (including *.pdf.txt) that restricts access
to it; however few bother to do this for resources that aren't exposed in the
UI, which is *almost* always OK.
Richard
> Reports that Google Scholar is sometimes linking to DSpace extracted text
> (*.pdf.txt) files instead of original PDF
> -------------------------------------------------------------------------------------------------------------------
>
> Key: DS-1387
> URL: https://jira.duraspace.org/browse/DS-1387
> Project: DSpace
> Issue Type: Bug
> Components: XMLUI
> Reporter: Tim Donohue
>
> This ticket is a placeholder for several recent reports about PDF indexing
> oddities with Google Scholar and DSpace (seemingly XMLUI specific, though
> that is unconfirmed).
> In several cases, users have reported that Google Scholar is mistakenly
> linking to the internal extracted PDF text files (*.pdf.txt files). These
> internal ".pdf.txt" files are automatically generated by DSpace for its own
> indexing, and are not meant to be utilized by external search engines.
> Although the "*.pdf.txt" files are technically publicly accessible, they are
> currently not linked to from the main Item "splash page", so it's uncertain
> how they are being located by web spiders. (Some have speculated perhaps form
> the OAI interface, or from indexing of the XMLUI's "mets.xml" file)
> Here are a few threads describing this issues on dspace-tech mailing list:
> * http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg19303.html
> * http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg18831.html
> If anyone else has noticed this issue, we'd encourage you to provide examples
> in this JIRA ticket. It may help us to better track down whether this is a
> DSpace issue, a Google Scholar issue, or perhaps even a bit of both.
> When you add comments to this ticket, please provide the DSpace version you
> are using and whether you are using XMLUI or JSPUI and whether you have OAI
> enabled. If you have any examples you can link to in Google Scholar or any
> other oddities you've noticed, please note those as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
Dspace-devel mailing list
Dspace-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-devel
