I am using the Map Reduce approach. I was looking into AVRO to create my
own custom Data types to pass from Mapper to Reducer.
With Avro I need to maintain the schema for all the types of Jason files I
am receiving and since there will be many different map reduce methods
running, so a different sc
Size is not a prob, frequently changing schema might be.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 7, 2013 at 6:25 PM, Panshul Whisper wrote:
> Hello,
>
> Thank you for the replies.
>
> I have not used pig yet. I am looking into it. I wanted to implement bo
Pig is like any other scripting language. Therefore maintainable.
Your script defines the schema of the data set and then does the processing you
tell it to do.
On Feb 7, 2013, at 6:55 AM, Panshul Whisper wrote:
> Hello,
>
> Thank you for the replies.
>
> I have not used pig yet. I am look
Hello,
Thank you for the replies.
I have not used pig yet. I am looking into it. I wanted to implement both
the approaches.
Are pig scripts maintainable? Because the Json structure that I will be
receiving will be changing quite often. Almost 3 times a month.
I will be processing 24 million Json
Good point sir. If Pig fits into Panshul's requirements then it's a much
better option.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 7, 2013 at 5:25 PM, Damien Hardy wrote:
> Hello,
> Why not using a PIG script for that ?
> make the json file available on HDF
Hello,
Why not using a PIG script for that ?
make the json file available on HDFS
Load with
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/builtin/JsonLoader.html
Store with
http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/backend/hadoop/hbase/HBaseStorage.html
http://pig.apache.org/do
Hello,
I am trying to write MapReduce jobs to read data from JSON files and load
it into HBase tables.
Please suggest me an efficient way to do it. I am trying to do it using
Spring Data Hbase Template to make it thread safe and enable table locking.
I use the Map methods to read and parse the JS
