----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/17191/ -----------------------------------------------------------
Review request for pig, Alex Bain, Cheolsoo Park, Daniel Dai, and Mark Wagner. Repository: pig Description ------- Added support for secondary key sort. Also fixed combiners and custom partitioners to work for multiple outputs. Fixed bugs in couple of places as I encountered them. Diffs ----- http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/MiniCluster.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/shims/test/hadoop23/org/apache/pig/test/TezMiniCluster.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/PigConfiguration.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/ColumnInfo.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/SecondaryKeyOptimizerMR.java PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/optimizer/SecondaryKeyOptimizer.java PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POLocalRearrange.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POReservoirSample.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/util/PlanHelper.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/CombinerOptimizer.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/POShuffleTezLoad.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SecondaryKeyOptimizerTez.java PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/SkewedPartitionerTez.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompiler.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezCompilerUtil.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezEdgeDescriptor.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezOperator.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/tez/TezPrinter.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/util/CombinerOptimizerUtil.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/backend/hadoop/executionengine/util/SecondaryKeyOptimizerUtil.java PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/NullableTuple.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/src/org/apache/pig/impl/io/PigNullableWritable.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/MiniGenericCluster.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestAccumulator.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestCombiner.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestCustomPartitioner.java PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestEvalPipeline2.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestPigServer.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestSecondarySort.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestSecondarySortMR.java PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestSkewedJoin.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/TestSplitStore.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/Util.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC14.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC15.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC16.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC4.gld 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC5.gld 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC8.gld 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/tez/TestSecondarySortTez.java PRE-CREATION http://svn.apache.org/repos/asf/pig/branches/tez/test/org/apache/pig/tez/TestTezCompiler.java 1559126 http://svn.apache.org/repos/asf/pig/branches/tez/test/tez-tests 1559126 Diff: https://reviews.apache.org/r/17191/diff/ Testing ------- Enabled more unit tests. Memory is a problem with few tests. Also seeing cases of Tez failing with file.out.index (map output) not found and hangs on waiting for input. Not sure if it is with me upgrading latest tez trunk. But definitely seems to be a Tez bug. Will create a TEZ jira once I dig a little more. Currently skewed join e2e tests - Join_7 and Join_8 fail with a NPE with this patch. Others pass. Will try and fix that shortly or in a different jira. Would like to get this patch in quickly as it has been long pending and rebasing is a pain. Thanks, Rohini Palaniswamy
