[ https://jira.duraspace.org/browse/DS-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=26922#comment-26922 ]
Richard Rodgers commented on DS-1387: ------------------------------------- Hi Reinhard: I'm not sure, but you might be making an assumption that all Google crawlers can do is follow links. My understanding is that they also can do 'heuristic' exploration of a site's URL space: take an URL, transform it to a random new one, and see if it returns something. This was a big problem for DSpace when it was afflicted by this Cocoon bug: https://jira.duraspace.org/browse/DS-768 (200s returned for invalid pages) - it tripped up these exploratory crawls because they never reached the 404 'boundary'. Google Scholar (IIRC) alerted us to this problem. So its sort of border-line whether you call it bug or not: DSpace lets you put a resource policy on any bitstream (including *.pdf.txt) that restricts access to it; however few bother to do this for resources that aren't exposed in the UI, which is *almost* always OK. Richard > Reports that Google Scholar is sometimes linking to DSpace extracted text > (*.pdf.txt) files instead of original PDF > ------------------------------------------------------------------------------------------------------------------- > > Key: DS-1387 > URL: https://jira.duraspace.org/browse/DS-1387 > Project: DSpace > Issue Type: Bug > Components: XMLUI > Reporter: Tim Donohue > > This ticket is a placeholder for several recent reports about PDF indexing > oddities with Google Scholar and DSpace (seemingly XMLUI specific, though > that is unconfirmed). > In several cases, users have reported that Google Scholar is mistakenly > linking to the internal extracted PDF text files (*.pdf.txt files). These > internal ".pdf.txt" files are automatically generated by DSpace for its own > indexing, and are not meant to be utilized by external search engines. > Although the "*.pdf.txt" files are technically publicly accessible, they are > currently not linked to from the main Item "splash page", so it's uncertain > how they are being located by web spiders. (Some have speculated perhaps form > the OAI interface, or from indexing of the XMLUI's "mets.xml" file) > Here are a few threads describing this issues on dspace-tech mailing list: > * http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg19303.html > * http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg18831.html > If anyone else has noticed this issue, we'd encourage you to provide examples > in this JIRA ticket. It may help us to better track down whether this is a > DSpace issue, a Google Scholar issue, or perhaps even a bit of both. > When you add comments to this ticket, please provide the DSpace version you > are using and whether you are using XMLUI or JSPUI and whether you have OAI > enabled. If you have any examples you can link to in Google Scholar or any > other oddities you've noticed, please note those as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------------------------------ Monitor your physical, virtual and cloud infrastructure from a single web console. Get in-depth insight into apps, servers, databases, vmware, SAP, cloud infrastructure, etc. Download 30-day Free Trial. Pricing starts from $795 for 25 servers or applications! http://p.sf.net/sfu/zoho_dev2dev_nov _______________________________________________ Dspace-devel mailing list Dspace-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-devel