Bill Graham created PIG-2906:
--------------------------------
Summary: Commands in Pig scripts shouldn't trigger executeBatch()
Key: PIG-2906
URL: https://issues.apache.org/jira/browse/PIG-2906
Project: Pig
Issue Type: Bug
Reporter: Bill Graham
There are a number of commands that can be used in Pig scripts that will
trigger an {{executeBatch()}} from {{GruntParser}} during parsing. This can
cause multiple runs of identical (non-shared) jobs. Consider this script:
{noformat}
A = ...
B = ...
C = ...
STORE C
rmf 'xyz'
D = JOIN C, A
STORE D
{noformat}
The DAG of jobs produced by A,B,C will run twice due to the {{executeBatch()}}
call tied to the {{rmf}} statement. Instead we should insert a marker into the
plan for these types of operations and execute them from the
{{JobControlCompiler}} at the appropriate time.
A quick look of the code shows the following commands currently trigger
{{executeBatch}}:
* cat
* move
* cp
* copyToLocal
* copyFromLocal
* rmf
* fs command
* sh command
* SQL command
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
