[
https://issues.apache.org/jira/browse/HADOOP-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12486407
]
Doug Cutting commented on HADOOP-1181:
--------------------------------------
Related to this, it would be great to have an InputFormat implementation that
includes all the task logs from a job. Folks should be able to do something
like:
job = new JobConf();
job.setInputFormatClass(TaskLogInputFormat.class);
TaskLogInputFormat.setJobId(jobId);
...
I mentioned this before, in HADOOP-342. Examining logs should be as
lightweight as possible, to facilitate debugging. It should not require a copy
to HDFS. A faster debug loop is like a faster search engine: it makes people
more productive. The sooner one can find that, e.g., most tasks failed with a
NullPointerException on line 723, the better.
Should I file a separate issue for this?
> userlogs reader
> ---------------
>
> Key: HADOOP-1181
> URL: https://issues.apache.org/jira/browse/HADOOP-1181
> Project: Hadoop
> Issue Type: Improvement
> Reporter: [EMAIL PROTECTED]
> Attachments: hadoop1181-v2.patch, hadoop1181.patch
>
>
> My jobs output lots of logging. I want to be able to quickly parse the logs
> across the cluster for anomalies. org.apache.hadoop.tool.Logalyzer looks
> promising at first but it does not know how to deal with the userlog format
> and it wants to first copy all logs local. Digging, there does not seem to
> currently be a reader for hadoop userlog format. TaskLog$Reader is not
> generally accessible and it too expects logs to be on the local filesystem
> (The latter is of little good if I want to run the analysis as a mapreduce
> job).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
