Hadoop Serialization: Avro
Hey everyone, First time posting to the list. I'm currently writing a hadoop job that will run daily and whose output will be part of the part of the next day's input. Also, the output will potentially be read by other programs for later analysis. Since my program's output is used as part of the next day's input, it would be nice if it was stored in some binary format that is easy to read the next time around. But this format also needs to be readable by other outside programs, not necessarily written in Java. After searching for a while it seems that Avro is what I want to be using. In any case, I have been looking around for a while and I can't seem to find a single example of how to use Avro within a Hadoop job. It seems that in order to use Avro I need to change the io.serializations value, however I don't know which value should be specified. Furthermore, I found that there are classes Avro{Input,Output}Format but these use a series of other Avro classes which, as far as I understand, seem need the use of other Avro classes such as AvroWrapper, AvroKey, AvroValue, and as far as I am concerned Avro* (with * replaced with pretty much any Hadoop class name). It seems however that these are used so that the Avro format is used throughout the Hadoop process to pass objects around. I just want to use Avro to save my output and read it again as input next time around. So far I have been using SequenceFile{Input,Output}Format, and have implemented the Writable interface in the relevant classes, however this is not portable to other languages. Is there a way to use Avro without a substantial rewrite (using Avro* classes) of my Hadoop job? Thanks in advance, Best, -Leo -- Leo Urbina Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science Department of Mathematics lurb...@mit.edu
Re: Hadoop Serialization: Avro
Hi, Depending on the response you get here, you might also post the question separately on avro-user. On Sat, Nov 26, 2011 at 1:46 PM, Leonardo Urbina wrote: > Hey everyone, > > First time posting to the list. I'm currently writing a hadoop job that > will run daily and whose output will be part of the part of the next day's > input. Also, the output will potentially be read by other programs for > later analysis. > > Since my program's output is used as part of the next day's input, it would > be nice if it was stored in some binary format that is easy to read the > next time around. But this format also needs to be readable by other > outside programs, not necessarily written in Java. After searching for a > while it seems that Avro is what I want to be using. In any case, I have > been looking around for a while and I can't seem to find a single example > of how to use Avro within a Hadoop job. > > It seems that in order to use Avro I need to change the io.serializations > value, however I don't know which value should be specified. Furthermore, I > found that there are classes Avro{Input,Output}Format but these use a > series of other Avro classes which, as far as I understand, seem need the > use of other Avro classes such as AvroWrapper, AvroKey, AvroValue, and as > far as I am concerned Avro* (with * replaced with pretty much any Hadoop > class name). It seems however that these are used so that the Avro format > is used throughout the Hadoop process to pass objects around. > > I just want to use Avro to save my output and read it again as input next > time around. So far I have been using SequenceFile{Input,Output}Format, and > have implemented the Writable interface in the relevant classes, however > this is not portable to other languages. Is there a way to use Avro without > a substantial rewrite (using Avro* classes) of my Hadoop job? Thanks in > advance, > > Best, > -Leo > > -- > Leo Urbina > Massachusetts Institute of Technology > Department of Electrical Engineering and Computer Science > Department of Mathematics > lurb...@mit.edu >
Re: Hadoop Serialization: Avro
Thanks, I will send the question to that last as well, Best, -Leo Sent from my phone On Nov 26, 2011, at 7:32 PM, Brock Noland wrote: > Hi, > > Depending on the response you get here, you might also post the > question separately on avro-user. > > On Sat, Nov 26, 2011 at 1:46 PM, Leonardo Urbina wrote: >> Hey everyone, >> >> First time posting to the list. I'm currently writing a hadoop job that >> will run daily and whose output will be part of the part of the next day's >> input. Also, the output will potentially be read by other programs for >> later analysis. >> >> Since my program's output is used as part of the next day's input, it would >> be nice if it was stored in some binary format that is easy to read the >> next time around. But this format also needs to be readable by other >> outside programs, not necessarily written in Java. After searching for a >> while it seems that Avro is what I want to be using. In any case, I have >> been looking around for a while and I can't seem to find a single example >> of how to use Avro within a Hadoop job. >> >> It seems that in order to use Avro I need to change the io.serializations >> value, however I don't know which value should be specified. Furthermore, I >> found that there are classes Avro{Input,Output}Format but these use a >> series of other Avro classes which, as far as I understand, seem need the >> use of other Avro classes such as AvroWrapper, AvroKey, AvroValue, and as >> far as I am concerned Avro* (with * replaced with pretty much any Hadoop >> class name). It seems however that these are used so that the Avro format >> is used throughout the Hadoop process to pass objects around. >> >> I just want to use Avro to save my output and read it again as input next >> time around. So far I have been using SequenceFile{Input,Output}Format, and >> have implemented the Writable interface in the relevant classes, however >> this is not portable to other languages. Is there a way to use Avro without >> a substantial rewrite (using Avro* classes) of my Hadoop job? Thanks in >> advance, >> >> Best, >> -Leo >> >> -- >> Leo Urbina >> Massachusetts Institute of Technology >> Department of Electrical Engineering and Computer Science >> Department of Mathematics >> lurb...@mit.edu >>