[
https://issues.apache.org/jira/browse/MAPREDUCE-157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742723#action_12742723
]
Jeff Hammerbacher commented on MAPREDUCE-157:
---------------------------------------------
Hey,
As Philip notes, you guys clearly are working to get work done, so I don't want
to stand in the way. For future consideration, here are some reasons I thought
using Avro here might be a win for the project as a whole:
* There will certainly be more performance-critical uses of Avro to come inside
of Hadoop, so if we punt on using it here because of performance considerations
(2. and 2a.), I'd be concerned with regards to the feasibility of future uses
(e.g. data block transfer). I saw this application as a fairly safe place to
start getting comfortable with Avro in production use.
* Similarly, for concern 1, having an Avro schema would allow users to generate
intelligent parsers in many languages automagically, and having a useful Avro
schema to parse inside of Hadoop could potentially encourage the development of
new Avro client libraries.
* Parsing JSON may be easy in many languages but enforcing structure using
Avro's schemas will be valuable for communicating the format of the logs and
encouraging the format's stability.
* Also, it's likely that many of the tools in the Hadoop ecosystem that
manipulate structured data (Pig, Hive, etc.) will have utilities for Avro that
they may lack for JSON or other formats not central to the Hadoop ecosystem
like Avro.
Either way, I'm happy to see progress being made on a stable and sane format
for these logs!
Later,
Jeff
> Job History log file format is not friendly for external tools.
> ---------------------------------------------------------------
>
> Key: MAPREDUCE-157
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-157
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Owen O'Malley
> Assignee: Jothi Padmanabhan
>
> Currently, parsing the job history logs with external tools is very difficult
> because of the format. The most critical problem is that newlines aren't
> escaped in the strings. That makes using tools like grep, sed, and awk very
> tricky.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
