Mark,
That's correct, that the indexer only indexes files in the TEXT bundle.
But, that's why I had recommended to Susan to first run 'filter-media'
script. The 'filter-media' script will take text files in the CONTENT
bundle and essentially copy them over to the TEXT bundle for indexing.
So, you are correct that the *.txt files could be immediately put in the
TEXT bundle (which would also avoid them being exposed publicly). But,
the alternative would be to put the *.txt files in the CONTENT bundle
and run 'filter-media' to "filter" it into the TEXT bundle. (However, as
you noted, this latter option would require UI alteration to hide the
*.txt files, if they shouldn't be accessible).
- Tim
Diggory Mark wrote:
> Actually...
>
> Looking at the code of DSIndexer... I'm sure, written by among others...
> myself. We find that only Bitstreams within the "TEXT" bundle are
> actually indexed into Lucene:
>
>> for (int i = 0; i < myBundles.length; i++)
>> {
>> if ((myBundles[i].getName() != null)
>> && myBundles[i].getName().equals("TEXT"))
>> {
>
> I'm thinking this was a short-sightedness, but the unhappy consequence
> of which is that your text files will not get indexed if you place them
> into the "CONTENT" Bundle. There are two solutions
>
> A.) Put your text bitstreams into the TEXT bundle and not have to worry
> about them being exposed because the TEXT bundle will not be.
>
> B.) Put your text Bitstreams in the Content Bundle, alter the UI to hide
> them, and alter DSIndexer to index the CONTENT bundle.
>
> Mark
>
> On Jan 16, 2009, at 2:40 PM, Tim Donohue wrote:
>
>> Susan,
>>
>> Actually, the setting you'd want to change in your DSpace 1.4.2
>> dspace.cfg is this one:
>>
>> plugin.sequence.org.dspace.app.mediafilter.MediaFilter = ...
>>
>> You'd want to remove the entry for:
>> "org.dspace.app.mediafilter.PDFFilter"
>>
>> That'd ensure that the PDFFilter is no longer used by filter-media. The
>> setting that you referenced below just configures the PDF filter to
>> process files which are "Adobe PDF" format.
>>
>> [NOTE:] If you end up upgrading to DSpace 1.5.x, the above
>> "plugin.sequence.org.dspace.app.mediafilter.MediaFilter" setting no
>> longer exists. Instead, it was replaced by a more simplistic
>> "filter.plugins" setting. In that case, for DSpace 1.5.x, you'd just
>> remove "PDF Text Extractor" from the list of enabled "filter.plugins".
>> Again, this would ensure that 'filter-media' would no longer use the PDF
>> filter.
>>
>> Hopefully that all makes sense...Beyond that, as you mentioned, you'd
>> just need to hide those '*.txt' files from being displayed.
>>
>> - Tim
>>
>>
>>
>> Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] wrote:
>>> Hi Tim,
>>>
>>> So you're saying that our proposed solution would work as long as
>>> we remove (or comment out):
>>>
>>>
>>>
>>> *filter.org.dspace.app.mediafilter.PDFFilter.inputFormats = Adobe PDF*
>>>
>>>
>>>
>>> from dspace.cfg and make the change to not display the .txt files on the
>>> Item pages?
>>>
>>>
>>>
>>> Then we would still need to run filter-media which would only be to
>>> basically add our .txt files to the TEXT bundle for each Item?
>>>
>>>
>>>
>>> By the way, we have been using the 1.5 version of filter-media, with the
>>> addition of the two new configuration parameters in dspace.cfg, for
>>> awhile, even though we are running DSpace 1.4.2. I did this awhile back
>>> and yes, it has stopped the JAVA heap space errors from killing
>>> filter-media midstream.
>>>
>>>
>>>
>>> I do think this new plan is the better way to go for us. I believe the
>>> advantages would be:
>>>
>>> 1. No more filter-media running for soooo long – over 24 hours most of
>>> the time.
>>>
>>> 2. We would identify “problematic” .pdf files (ones that possibly
>>> wouldn’t filter) prior to importing them into DSpace, instead of
>>> after-the-fact. When these problems are caught at the scanning point,
>>> they could be dealt with there and then (rescanning/re-ocr’ing, etc).
>>>
>>> 3. Our Users wouldn’t have such a big job of identifying the
>>> “unfilterable” documents, locating them for rescanning, getting them
>>> back to us for re-import, etc etc.
>>>
>>> 4. Bottom line would be a more accurate full-text searchable
>>> repository.
>>>
>>>
>>>
>>> Thanks a bunch for the detailed feedback. We are processing a 1000
>>> document test with this new procedure and will let you know how it
>>> goes!!
>>>
>>> Sue
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Tim Donohue [mailto:[email protected]]
>>> Sent: Thursday, January 15, 2009 11:27 AM
>>> To: Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS]
>>> Cc: [email protected]; Kimbrough, Glenn W. (LARC-B7)[NCI
>>> INFORMATION SYSTEMS]; Warren, Douglas Lewis (LARC-B7)[NCI INFORMATION
>>> SYSTEMS]; Smail, James W. (LARC-B702)[NCI INFORMATION SYSTEMS]
>>> Subject: Re: [Dspace-tech] DSpace 1.4.2 / 1.5.x filter-media questions
>>>
>>>
>>>
>>> Sue,
>>>
>>>
>>>
>>> There were some improvements to 'filter-media' in DSpace 1.5.x.
>>>
>>> Primarily, there's the addition of two new PDF-specific settings in the
>>>
>>> dspace.cfg:
>>>
>>>
>>>
>>> pdffilter.largepdfs = true
>>>
>>> pdffilter.skiponmemoryexception = true
>>>
>>>
>>>
>>> The former ensures that all PDF text-extractions are written to
>>>
>>> temporary files during indexing. This helps avoid OutOfMemoryException
>>>
>>> & Heap space errors that were occasionally caused by larger PDFs being
>>>
>>> loaded into system memory all at once.
>>>
>>>
>>>
>>> The latter attempts to skip over any PDFs which still cause an
>>>
>>> OutOfMemoryException. So, if that exception still occurs on a PDF, then
>>>
>>> the PDF is skipped entirely and *not* indexed. This helps to avoid the
>>>
>>> entire 'filter-media' script "crashing" when an OutOfMemoryException
>>>
>>> occurs (which used to happen in 1.4.2).
>>>
>>>
>>>
>>> Despite these changes in 1.5.x, there is NO guarantee that *all* of your
>>>
>>> PDFs will index properly. As I've mentioned before, the 'filter-media'
>>>
>>> script uses third-party software (called PDFBox: http://www.pdfbox.org/)
>>>
>>> for indexing of PDF files. There are some known bugs in PDFBox that
>>>
>>> have yet to be fixed, so it does *not* always work for all PDFs. In
>>>
>>> some cases, PDFBox will also work inconsistently (and I don't know why
>>>
>>> that is). I've run into some inconsistency problems with larger-sized
>>>
>>> PDFs, which are originally scanned documents with embedded OCR.
>>>
>>> Occasionally PDFBox will index them fine, and other times it will cause
>>>
>>> an OutOfMemoryException (which, with DSpace 1.5 means that
>>>
>>> 'filter-media' will just skip that pdf).
>>>
>>>
>>>
>>> So, I guess the best way to sum this up is that DSpace currently cannot
>>>
>>> successfully index 100% of all PDFs, since PDFBox cannot do so. DSpace
>>>
>>> 1.5 has improvements in helping DSpace to safely handle PDFBox issues
>>>
>>> (like the OutOfMemoryExceptions), but it doesn't necessarily have
>>>
>>> drastic improvements in indexing capabilities.
>>>
>>>
>>>
>>> I answered your other questions inline below...
>>>
>>>
>>>
>>>
>>>
>>> Thornton, Susan M. (LARC-B702)[NCI INFORMATION SYSTEMS] wrote:
>>>
>>>
>>>
>>>> 1. Has the filter-media/index-all process changed
>>>
>>>> and/or improved significantly in DSpace 1.5? If so, we may just shelve
>>>
>>>> this issue until we’ve implemented 1.5.
>>>
>>>
>>>
>>> See above, obviously...
>>>
>>>
>>>
>>>> 2. In DSpace 1.4.2 (and 1.5), does it matter whether
>>>
>>>> your .txt files are plain or accessible .txt files? Can index-all
>>>
>>>> process either type?
>>>
>>>
>>>
>>> For text files, it doesn't really matter...in either case the
>>>
>>> 'filter-media' script just pulls out the plain text for indexing. I
>>>
>>> don't believe there'd be any significant difference between the "type"
>>>
>>> of .txt file.
>>>
>>>
>>>
>>> However, it's worth making this clear: for .txt files, you *still* need
>>>
>>> to run the 'filter-media' script for them to be indexed by 'index-all'.
>>>
>>> Essentially, 'index-all' only indexes plain text files in the "TEXT"
>>>
>>> bundle. The 'filter-media' script is what adds plain text to the "TEXT"
>>>
>>> bundle.
>>>
>>>
>>>
>>>>
>>>
>>>>
>>>
>>>> 3. If the process in 1.5 hasn’t changed and/or
>>>
>>>> improved significantly in 1.5, we are considering having our scanning
>>>
>>>> folks just create the .txt files along with the .pdf files at the time
>>>
>>>> the documents are scanned. Then when they send them to us, we would
>>>
>>>> just upload them in the import process along with the .pdf files for
>>>
>>>> each Item. The only thing we’d really have to change in our import
>>>
>>>> process is the addition of a second file name in the “contents” file
>>>> and
>>>
>>>> the addition of the .txt document in the Item’s import directory (right
>>>
>>>> along with the .pdf file). One other issue is we might have to make a
>>>
>>>> small modification to DSpace to **not** display the .txt file on the
>>>
>>>> Item page unless the User is in the Admin interface since we wouldn’t
>>>
>>>> want our Users clicking on/opening the .txt files. If we did this, we
>>>
>>>> could completely eliminate the filter-media job altogether. This would
>>>
>>>> ensure that we did not load any “unfilterable” documents into DSpace.
>>>
>>>> It would also eliminate the tedious process of identifying which
>>>
>>>> documents did not filter successfully, and the whole process of
>>>
>>>> rescanning and replacing them in DSpace.
>>>
>>>
>>>
>>> This sounds like a perfectly reasonable way of doing things, assuming
>>>
>>> you have the staff time to pre-generate those .txt files. You are
>>>
>>> correct that you'd no longer need to run 'filter-media' on those PDFs.
>>>
>>> But, you'd still need to run 'filter-media' to index those .txt files.
>>>
>>> You could do this by modifying the "Media Filter" settings in your
>>>
>>> dspace.cfg and *removing* the PDFFilter from the list (so 'filter-media'
>>>
>>> would no longer filter PDFs, but it would work on the other types of
>>>
>>> content).
>>>
>>>
>>>
>>> It would also require some custom coding to hide those .txt files from
>>>
>>> normal users, but that shouldn't be too horrible.
>>>
>>>
>>>
>>> If you did go this route, I'd make sure that you still OCR the PDFs that
>>>
>>> you put in, as it improves their accessibility overall.
>>>
>>>
>>>
>>> Hopefully that all makes sense...definitely let us know if you have
>>>
>>> further questions.
>>>
>>>
>>>
>>> - Tim
>>>
>>>
>>>
>>> --
>>>
>>> Tim Donohue
>>>
>>> Research Programmer, IDEALS
>>>
>>> http://www.ideals.uiuc.edu/
>>>
>>> University of Illinois
>>>
>>> [email protected] | (217) 333-4648
>>>
>>
>> --
>> Tim Donohue
>> Research Programmer, IDEALS
>> http://www.ideals.uiuc.edu/
>> University of Illinois
>> [email protected] | (217) 333-4648
>>
>> ------------------------------------------------------------------------------
>>
>>
>> This SF.net email is sponsored by:
>> SourcForge Community
>> SourceForge wants to tell your story.
>> http://p.sf.net/sfu/sf-spreadtheword
>> _______________________________________________
>> DSpace-tech mailing list
>> [email protected]
>> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
> ~~~~~~~~~~~~~
> Mark R. Diggory
> http://purl.org/net/mdiggory/homepage
>
>
>
>
--
Tim Donohue
Research Programmer, IDEALS
http://www.ideals.uiuc.edu/
University of Illinois
[email protected] | (217) 333-4648
------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
