----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27429/ -----------------------------------------------------------
Review request for pig and Daniel Dai.
Bugs: PIG-4259
https://issues.apache.org/jira/browse/PIG-4259
Repository: pig
Description
-------
Patch addresses different issues encountered while trying to debug wrong
results for a production script.
Issues addressed:
- Optimized union followed directly by Limit also fixing possibility of
incorrect results when Limit could be totally removed by UnionOptimizer if
parallelism of union was also 1.
- Fixed wrong result in case of group by with secondary key followed by
Union (Union_14)
- Fixed CROSS for Union and multiquery.
- Fixed/Optimized POLimit to not process next input in bag redundantly if
limit is already reached.
- Fixed some issues in auto parallelism and modified overriding
parallelism of intermediate reducers (PIG-4162) only for required cases.
- Adjust the AM size based on total tasks. Pain to keep adjusting memory
size after task runs for a long time and then fails with OOM.
- Fixes NPE in logs while fetching counters when job fails
- Avoid printing counters everytime while printing dagStatus. Only print
tasks and diagnostics.
Diffs
-----
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POLimit.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezJob.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezJobCompiler.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezCompiler.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezOperPlan.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezOperator.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/CombinerOptimizer.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/MultiQueryOptimizerTez.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/ParallelismSetter.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/TezOperDependencyParallelismEstimator.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/UnionOptimizer.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/builtin/GFCross.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/Utils.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/test/e2e/pig/tests/nightly.conf
1635795
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/Util.java
1635795
http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/tez/TestTezAutoParallelism.java
1635795
Diff: https://reviews.apache.org/r/27429/diff/
Testing
-------
Added unit and e2e tests. In the process of running full suite of unit and e2e
tests.
Thanks,
Rohini Palaniswamy
