Re: how to write custom log loader and store in JSON format

2015-07-06 Thread James Bond
I am not sure about Pig, but its easily achievable in MapReduce. We had a similar requirement, we had to convert logs from RFC syslog format (5424) into JSON. We have a MR job which does this for us. The reason why we chose MR was mainly for Error Handling - like missing fields in some records, rem

Re: how to write custom log loader and store in JSON format

2015-07-05 Thread Arvind S
i am not sure if you need a custom loader .. you could read this as a comma separated string into individual fields .. convert the last field into a map data type (using a UDF may be.. ) .. but if you still want to persue custom loader you can probably take hint from https://svn.apache.org/repos/a