Hi,
As of now in my jobs, I am using SequenceFileOutputFormat and I am emitting
custom java objects as MR output.
Now I am planning to emit it in avro format, I went through few blogs but
still have following doubts.
1) My current custom Writable objects has nested json format as toString(),
So
Avro. This will be
easiest, but you have to parse the data later when you use it.2) Use Avro
schema to match your json data, using matching structure from avro for your
data, like 'record, array, map' etc.
Yong
Date: Fri, 31 Jan 2014 00:13:59 +0530
Subject: shifting sequenceFileOutput f
from
> avro for your data, like 'record, array, map' etc.
>
> Yong
>
> ------
> Date: Fri, 31 Jan 2014 00:13:59 +0530
> Subject: shifting sequenceFileOutput format to Avro format
> From: akumarb2...@gmail.com
> To: user@hadoop.apache.o
s of data.
>>
>> If you have a Json string, you have 2 options to generate the Avro schema
>> for it:
>>
>> 1) Use "type: string" to store the whole Json string into Avro. This will
>> be easiest, but you have to parse the data later when you use
match your data. Avor's
>>> schema is very flexible and should be able to store all kinds of data.
>>>
>>> If you have a Json string, you have 2 options to generate the Avro
>>> schema for it:
>>>
>>> 1) Use "type: string&q
gt; In avro, you need to think about a schema to match your data. Avor's
>>>> schema is very flexible and should be able to store all kinds of data.
>>>>
>>>> If you have a Json string, you have 2 options to generate the Avro
>>>> schema for it:
>>>&
t;: {"type": "array",
"items":{"name":"Event","type":"map",
"values":"string"}}} ]}
yzhang$ java -jar ~/lib/avro-tools-1.7.6.jar compile schema test.avsc
o
