A static variable is not necessary... a simple instance variable is just
fine.
On Fri Dec 05 2014 at 2:27:53 PM Ryan <freelanceflashga...@gmail.com> wrote:
> After running it with updated code, it seems like the problem has to do
> with something related to Tika since my output says that my input is the
> correct number of bytes (i.e. it's actually being sent in correctly). Going
> to test further to narrow down the problem.
>
> Pradeep, would you recommend using a static variable inside the
> ExtractTextFromPDFs function to store the PdfParser once it has been
> initialized once? I'm still learning how to best do things within the
> Pig/MapReduce/Hadoop framework
>
> Ryan
>
> On Fri, Dec 5, 2014 at 1:35 PM, Ryan <freelanceflashga...@gmail.com>
> wrote:
>
> > Thanks Pradeep! I'll give it a try and report back
> >
> > Ryan
> >
> > On Fri, Dec 5, 2014 at 12:30 PM, Pradeep Gollakota <pradeep...@gmail.com
> >
> > wrote:
> >
> >> I forgot to mention earlier that you should probably move the PdfParser
> >> initialization code out of the evaluate method. This will probably
> cause a
> >> significant overhead both in terms of gc and runtime performance. You'll
> >> want to initialize your parser once and evaluate all your docs against
> it.
> >>
> >> - Pradeep
> >>
> >> On Fri Dec 05 2014 at 9:18:16 AM Pradeep Gollakota <
> pradeep...@gmail.com>
> >> wrote:
> >>
> >> > Java string's are immutable. So "pdfText.concat()" returns a new
> string
> >> > and the original string is left unmolested. So at the end, all you're
> >> doing
> >> > is returning an empty string. Instead, you can do "pdfText =
> >> > pdfText.concat(...)". But the better way to write it is to use a
> >> > StringBuilder.
> >> >
> >> > StringBuilder pdfText = ...;
> >> > pdfText.append(...);
> >> > pdfText.append(...);
> >> > ...
> >> > return pdfText.toString();
> >> >
> >> > On Fri Dec 05 2014 at 9:12:37 AM Ryan <freelanceflashga...@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I'm working on an open source project attempting to convert raw
> content
> >> >> from a pdf (stored as a databytearray) into plain text using a Pig
> UDF
> >> and
> >> >> Apache Tika. I could use your help. For some reason, the UDF I'm
> using
> >> >> isn't working. The script succeeds but no output is written. *This is
> >> the
> >> >> Pig script I'm following:*
> >> >>
> >> >> register 'target/warcbase-0.1.0-SNAPSHOT-fatjar.jar';
> >> >> DEFINE ExtractTextFromPDFs
> >> >> org.warcbase.pig.piggybank.ExtractTextFromPDFs();
> >> >> DEFINE ArcLoader org.warcbase.pig.ArcLoader();
> >> >>
> >> >> raw = load '/data/arc/sample.arc' using ArcLoader as (url: chararray,
> >> >> date:
> >> >> chararray, mime: chararray, content: bytearray); --load the data
> >> >>
> >> >> a = FILTER raw BY (url matches '.*\\.pdf$'); --gets all PDF pages
> from
> >> >> the
> >> >> arc file
> >> >> b = LIMIT a 2; --limit to 2 pages to speed up testing time
> >> >> c = foreach b generate url, ExtractTextFromPDFs(content);
> >> >> store c into 'output/pdf_test';
> >> >>
> >> >>
> >> >> *This is the UDF I wrote:*
> >> >>
> >> >> public class ExtractTextFromPDFs extends EvalFunc<String> {
> >> >>
> >> >> @Override
> >> >> public String exec(Tuple input) throws IOException {
> >> >> String pdfText = "";
> >> >>
> >> >> if (input == null || input.size() == 0 || input.get(0) ==
> null) {
> >> >> return "N/A";
> >> >> }
> >> >>
> >> >> DataByteArray dba = (DataByteArray)input.get(0);
> >> >> pdfText.concat(String.valueOf(dba.size())); //my attempt at
> >> >> debugging. Nothing written
> >> >>
> >> >> InputStream is = new ByteArrayInputStream(dba.get());
> >> >>
> >> >> ContentHandler contenthandler = new BodyContentHandler();
> >> >> Metadata metadata = new Metadata();
> >> >> DefaultDetector detector = new DefaultDetector();
> >> >> AutoDetectParser pdfparser = new AutoDetectParser(detector);
> >> >>
> >> >> try {
> >> >> pdfparser.parse(is, contenthandler, metadata, new
> >> ParseContext());
> >> >> } catch (SAXException | TikaException e) {
> >> >> // TODO Auto-generated catch block
> >> >> e.printStackTrace();
> >> >> }
> >> >> pdfText.concat(" : "); //another attempt at debugging. Still
> >> nothing
> >> >> written
> >> >> pdfText.concat(contenthandler.toString());
> >> >>
> >> >> //close the input stream
> >> >> if(is != null){
> >> >> is.close();
> >> >> }
> >> >> return pdfText;
> >> >> }
> >> >>
> >> >> }
> >> >>
> >> >> Thank you for your assistance,
> >> >> Ryan
> >> >>
> >> >
> >>
> >
> >
>
