Calling close() on the MultipleOutputs objects in the cleanup() method of the reducer fixed the lzo file problem. Thanks!
~Ed On Thu, Oct 21, 2010 at 9:12 PM, ed <[email protected]> wrote: > Hi Todd, > > I don't have the code in front of me right but I was looking over the API > docs and it looks like I forgot to call close() on the MultipleOutput. I'll > post back if that fixes the problem. If not I'll put together a unit test. > Thanks! > > ~Ed > > > On Thu, Oct 21, 2010 at 6:31 PM, Todd Lipcon <[email protected]> wrote: > >> Hi Ed, >> >> Sounds like this might be a bug, either in MultipleOutputs or in LZO. >> >> Does it work properly with gzip compression? Which LZO implementation >> are you using? The one from google code or the more up to date one >> from github (either kevinweil's or mine)? >> >> Any chance you could write a unit test that shows the issue? >> >> Thanks >> -Todd >> >> On Thu, Oct 21, 2010 at 2:52 PM, ed <[email protected]> wrote: >> > Hello everyone, >> > >> > I am having problems using MultipleOutputs with LZO compression (could >> be a >> > bug or something wrong in my own code). >> > >> > In my driver I set >> > >> > MultipleOutputs.addNamedOutput(job, "test", TextOutputFormat.class, >> > NullWritable.class, Text.class); >> > >> > In my reducer I have: >> > >> > MultipleOutputs<NullWritable, Text> mOutput = new >> > MultipleOutputs<NullWritable, Text>(context); >> > >> > public String generateFileName(Key key){ >> > return "custom_file_name"; >> > } >> > >> > Then in the reduce() method I have: >> > >> > mOutput.write(mNullWritable, mValue, generateFileName(key)); >> > >> > This results in creating LZO files that do not decompress properly (lzop >> -d >> > throws the error "lzop: unexpected end of file: outputFile.lzo") >> > >> > If I switch back to the regular context.write(mNullWritable, mValue); >> > everything works fine. >> > >> > Am I forgetting a step needed when using MultipleOutputs or is this a >> > bug/non-feature of using LZO compression in Hadoop. >> > >> > Thank you! >> > >> > >> > ~Ed >> > >> >> >> >> -- >> Todd Lipcon >> Software Engineer, Cloudera >> > >
