Mike Zimmerman created CRUNCH-272:
-------------------------------------
Summary: Unable to correlate crunch jobs within Oozie
Key: CRUNCH-272
URL: https://issues.apache.org/jira/browse/CRUNCH-272
Project: Crunch
Issue Type: Improvement
Reporter: Mike Zimmerman
I'm not really sure if this should be logged to Oozie or to Crunch, so please
feel free to move as needed.
I would like to request a way to decorate map/reduce jobs that are spawned by a
Crunch pipeline so that I can programmatically determine their origin. The
primary use case for this is integration with Oozie. Oozie launches a single
map job to run a java action (in our case this java action runs a crunch job).
Traceability from this original "launcher" job to the jobs created by the
crunch job is impossible without trolling logs. This leaves a big black hole
for the system operator to assess the performance/impact of these jobs. My
initial thought was to provide a simple way to indicate a correlationId or
similar on a map/reduce job and then make it accessible within Oozie to query
for. Obviously, that request would have to come after the correlation feature
was available within map/reduce.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
