Thank you, Andrea! I really appreciate learning about all these nuances and I have answers to all your questions. I wonder if this is a case of malformed PDF/OCR/XML result but it seems clear that something unexpected was encountered. (On a side-note, I could launch a browser on the server and load these documents in the server I'm debugging (v5) but not on our production server (v3) so permissions must have changed to enable this by default on the server).
Yes this item is publicly viewable and searching for the title, author, contributer, abstract text etc. all find the item. I don't see any permissions information (untrained eye) but there is an array named "read" with string "g0" for the item in question. One additional thing I have discovered is that the [full text] entry *begins with [stream_size] information*. The problem seems to manifest itself irregardless of privacy settings for the item. Here is the hopefully tell-tale view of the item's "Full text" lead-up and output. Note that *the parts leading up to the fulltext get repeated* and absorbed as part of the full-text. However, instead of using the full text that is actually available, it is a duplication of the abstract (which has its own .pdf.txt generated via media-filter). It looks like some sort of indexing bug to me. -<arr name="bi_5_dis_partial"> (*just included for context -- not repeated inside the full text element*) <str>MacKay, Carolyn J. (Carolyn Joyce), 1954-</str> </arr> *(all the following get duplicated inside the [fulltext] nodes)* -<arr name="stream_size"> <str>1337</str> </arr> -<arr name="stream_content_type"> <str>text/plain</str> </arr> -<arr name="stream_name"> <str>HammonsJ_2012-2_ABSTRACT.pdf.txt</str> </arr> -<arr name="stream_source_info"> <str>HammonsJ_2012-2_ABSTRACT.pdf.txt</str> </arr> -<arr name="Content-Encoding"> <str>UTF-8</str> </arr> -<arr name="Content-Type"> <str>text/plain; charset=UTF-8</str> </arr> -<arr name="fulltext"> <str> stream_size 1337 stream_content_type text/plain stream_name HammonsJ_2012-2_ABSTRACT.pdf.txt stream_source_info HammonsJ_2012-2_ABSTRACT.pdf.txt Content-Encoding UTF-8 Content-Type text/plain; charset=UTF-8 ABSTRACT RESEARCH PAPER: Solidarity in an Online Community STUDENT: James William Hammons DEGREE: Master of Arts COLLEGE: Sciences and Humanities DATE: May 2012 PAGES: 64.....[*extracted for space reasons*].....is strongly supported. </str> </arr> -<arr name="fulltext_hl"> <str> stream_size 1337 stream_content_type text/plain stream_name HammonsJ_2012-2_ABSTRACT.pdf.txt stream_source_info HammonsJ_2012-2_ABSTRACT.pdf.txt Content-Encoding UTF-8 Content-Type text/plain; charset=UTF-8 ABSTRACT RESEARCH ...[*truncated*] Now that we can see the problem -- have you run into this before? What can we try to get the impacted items corrected? > > What I believe is happening is that *the two xml nodes [fulltext] and [fulltext_hl] are getting populated by the last bitstream's full text* (along with stream_size, stream_content_type, etc. for whatever reason). Whichever is placed last is getting put into the index. For items with only 1 bitstream the full text is searchable because the first *is the last* (e.g. no body + abstract pdfs -- but if the BODY was listed second I believe it would be searchable). BUT! Even the items with only 1 bitstream can still have the [stream_size], etc. dumped into the [fulltext] node. A search for "stream_size" returned 12k of our 25k items. And interestingly, 26 of the 120 <http://demo.dspace.org/xmlui/discover?scope=%2F&query=stream_size&submit=> items in the demo dspace. I wonder what the index looks like for http://demo.dspace.org/xmlui/handle/10673/5. Do you have access to view it? "Test word document" doesn't return that handle <http://demo.dspace.org/xmlui/handle/10673/5> even though the phrase is present in the document that is listed FIRST for the bitstreams for that handle. Stated another way: The handle for "Test PDF document" (#5) contains a word document with that exact phrase -- but handle#5 is not returned with an quoted search for it because the index is only capturing the full text of the second/last(?) bitstream. Did I uncover something new?! -- You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/dspace-tech. For more options, visit https://groups.google.com/d/optout.
