[
https://issues.apache.org/jira/browse/CRUNCH-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047038#comment-14047038
]
Micah Whitacre commented on CRUNCH-272:
---------------------------------------
I completely agree with Robert that it'd be best if the Crunch action as part
of Oozie as it make integration and deployment a lot easier. The difficulty
the custom action will face is adding structure to how a Crunch Pipeline(s)
would get launched. What we will want to add in Crunch might be a common
launching API for the launching job to report the PipelineResult objects much
like the CrunchOozieLauncher. This way instead of the Oozie action taking a
generic main class it'd take a launcher class that returns the PipelineResults.
> Unable to correlate crunch jobs within Oozie
> --------------------------------------------
>
> Key: CRUNCH-272
> URL: https://issues.apache.org/jira/browse/CRUNCH-272
> Project: Crunch
> Issue Type: Improvement
> Reporter: Mike Zimmerman
> Assignee: Micah Whitacre
> Attachments: CRUNCH-272.patch, CRUNCH-272_prototype.patch
>
>
> I'm not really sure if this should be logged to Oozie or to Crunch, so please
> feel free to move as needed.
> I would like to request a way to decorate map/reduce jobs that are spawned by
> a Crunch pipeline so that I can programmatically determine their origin. The
> primary use case for this is integration with Oozie. Oozie launches a single
> map job to run a java action (in our case this java action runs a crunch
> job). Traceability from this original "launcher" job to the jobs created by
> the crunch job is impossible without trolling logs. This leaves a big black
> hole for the system operator to assess the performance/impact of these jobs.
> My initial thought was to provide a simple way to indicate a correlationId or
> similar on a map/reduce job and then make it accessible within Oozie to query
> for. Obviously, that request would have to come after the correlation
> feature was available within map/reduce.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
