-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/17681/
-----------------------------------------------------------
(Updated Feb. 6, 2014, 11:48 p.m.)
Review request for pig, Cheolsoo Park and Daniel Dai.
Changes
-------
Updated patch. Had done some minor fixes. Also created tez staging dir in
FileLocalizer relativeRoot directory.
Bugs: PIG-3742
https://issues.apache.org/jira/browse/PIG-3742
Repository: pig
Description
-------
Changes made:
1) Converted the relevant MR settings to equivalent Tez settings and set them
on AM, Vertex and Edge.
2) Moved the util and helper classes (SecurityHelper and TezCompilerUtil) to a
util package. Does not show up cleanly in review board. Will be doing a svn mv
while committing.
3) Fixed a issue with 1-1 edge in orderby while running pigmix where
parallelism was not reflected in the second edge when the parallelism of first
vertex changed after input split calculation. Also made POIdentityOutTez work
with shuffle input as well when trying to test performance with 1-1 ege or
shuffle edge with round robin partitioner. Shuffle edge with round robin
partitioner or hash partitioner was very bad compared to MR. Even with 1-1
edge, performance is bad for L10.pig which orders by multiple columns. Still
need to work on order by performance. Hoping unsorted shuffle with TEZ-661
might make it better.
Diffs (updated)
-----
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POIdentityInOutTez.java
1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecurityHelper.java
1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java
1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java
1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java
1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java
1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java
1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java
1563492
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/MRToTezHelper.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/SecurityHelper.java
PRE-CREATION
http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/util/TezCompilerUtil.java
PRE-CREATION
Diff: https://reviews.apache.org/r/17681/diff/
Testing
-------
Unit and tez.conf e2e tests pass.
Thanks,
Rohini Palaniswamy
