Default job.names - script name, load file pattern, store file pattern, sub
task type
-------------------------------------------------------------------------------------
Key: PIG-786
URL: https://issues.apache.org/jira/browse/PIG-786
Project: Pig
Issue Type: Improvement
Reporter: David Ciemiewicz
Priority: Trivial
I have very complex Pig scripts which are often concatenations and iterations
of a large number of map reduce tasks.
I've gotten into the habit of using the following construct in my code:
{code}set job.name '$DIR/$DATE/summary.bz';
A = load ...
...
store Z into '$DIR/$DATE/summary.bz' using PigStorage();{code}
But it would be really useful if Pig script parsing automagically set these
job.name values.
Ideally I'd like to have Pig just automagically construct job names for me so I
can trace execution of multihour jobs in the HOD progress pages. Something
like:
{code}process-dates.pig
A = LOAD /data/logs/daily/20090408
...
STORE Z into mysummary/20090408/summary.bz
map-group-combiner-sort{code}
Okay you say, I could construct this kind of job.name myself if this is what I
want.
Well:
1) I'd really like to have a default constructed by Pig so I don't have to
2) Pig has information about what is happening that I don't have such as:
* The name of the script passed to Pig
* The glob expansion of the file pathname in the LOAD statement
* The execution plan of pig that would tell me what the
map-group-combine-sort-reduce group looks like
* The name of intermediate STORE operations that are being performed
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.