----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61265/ -----------------------------------------------------------
Review request for pig, Daniel Dai and Koji Noguchi. Bugs: PIG-5256 https://issues.apache.org/jira/browse/PIG-5256 Repository: pig Description ------- For each POForEach or POFilter operator in the plan, a corresponding class is created by inlining code for the input plans. Bytecode is generated directly through asm APIs and the generated class files are shipped to the tasks as a jar. On the frontend, the generated classes are dynamically added to the PigContext classloader. The explain command now decompiles the class files in the explain output directory. Refer to the golden files in the test for examples of generated code. TODO items to be addressed in a separate jira: 1) Support for MR and Spark. Currently only done for Tez 2) Support for Accumulator 3) Support for CROSS 4) Run the optimizer on combiner plans Diffs ----- http://svn.apache.org/repos/asf/pig/trunk/build.xml 1803384 http://svn.apache.org/repos/asf/pig/trunk/ivy.xml 1803384 http://svn.apache.org/repos/asf/pig/trunk/ivy/libraries.properties 1803384 http://svn.apache.org/repos/asf/pig/trunk/shade/pom.xml PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/PigConfiguration.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/PigServer.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/optimizer/AsmUtil.java PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/optimizer/CodeGenerator.java PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/optimizer/FilterCodeGenerator.java PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/optimizer/ForEachCodeGenerator.java PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Add.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Divide.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/EqualToExpr.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/GTOrEqualToExpr.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/GreaterThanExpr.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/LTOrEqualToExpr.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/LessThanExpr.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Mod.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Multiply.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/NotEqualToExpr.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POCast.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POProject.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/PORegexp.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/PORelationToExprProject.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Subtract.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/PODistinct.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POFilter.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POLimit.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSort.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSortedDistinct.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezLauncher.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezResourceManager.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/TezPlanContainer.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/plan/optimizer/RuntimeCodeGenOptimizer.java PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/builtin/InvokerGenerator.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/data/DataType.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/PigContext.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/io/FileLocalizer.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/plan/OperatorPlan.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/ClassDecompiler.java PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/JarManager.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/newplan/logical/expression/ExpToPhyTranslationVisitor.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/newplan/logical/expression/UserFuncExpression.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/parser/LogicalPlanBuilder.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/tools/grunt/GruntParser.java 1803384 http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestByteCodeGen.java PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilter/POFilter_scope_12.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilter/POForEach_scope_11.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilter/plan.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilterSplit/POFilter_scope_14.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilterSplit/POFilter_scope_9.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilterSplit/POForEach_scope_7.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testFilterSplit/plan.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POFilter_scope_20.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POFilter_scope_48.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_14.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_26.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_37.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_42.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_54.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_61.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_67.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_7.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_70.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/POForEach_scope_76.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachComplex/plan.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachSplit/POForEach_scope_20.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachSplit/POForEach_scope_32.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachSplit/POForEach_scope_9.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachSplit/plan.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachUDFEval/POForEach_scope_39.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachUDFEval/POForEach_scope_4.java.gld PRE-CREATION http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/data/GoldenFiles/bytecodegen/tez/testForeachUDFEval/plan.gld PRE-CREATION Diff: https://reviews.apache.org/r/61265/diff/1/ Testing ------- Running full suite of unit and e2e tests with pig.opt.bytecode=true. Will update once done Thanks, Rohini Palaniswamy