[
https://issues.apache.org/jira/browse/OOZIE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109992#comment-16109992
]
Robert Kanter commented on OOZIE-2983:
--------------------------------------
Another option here that might be simpler and faster is for Oozie to construct
the appropriate URL and redirect to the JHS and have the JHS serve the logs.
You can actually trick the JHS into serving non-MR job logs as long as you give
it the right URL. I was talking to [~wilfreds] and he was saying that the
Spark History Server does this. It won't be as nicely integrated, but it
should be much easier to implement and we won't have to worry about ACLs,
handling missing files, etc.]
[~andras.piros], [~gezapeti], what do you think?
> Stream the Launcher AM Logs
> ---------------------------
>
> Key: OOZIE-2983
> URL: https://issues.apache.org/jira/browse/OOZIE-2983
> Project: Oozie
> Issue Type: Sub-task
> Affects Versions: 5.0.0
> Reporter: Robert Kanter
> Assignee: Andras Piros
> Attachments: OOZIE-2983.001.patch
>
>
> Because OYA will get rid of the MR-based Launcher Job and have a custom AM
> instead, the Launcher Job info and logs will no longer show up in the JHS. It
> will show up in the ATSv2, however, the ATSv2 isn't quite ready yet and that
> doesn't help users on older versions of Hadoop. Launcher Jobs will stil show
> up in the RM, but the RM doesn't remember things forever (and can be quite
> short depending on cluster busyness and configs).
> We should have Oozie serve those logs itself (which would also be a nice
> thing to have anyway, as the launcher logs are typically the most useful).
> The code for reading those files is pretty easy to call. The yarn CLI, JHS,
> and this [Aggregated Log Files
> Exploder|https://github.com/rkanter/ALFExploder] tool I wrote all use it; the
> latter proves that you can do it outside of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
