> On July 2, 2012, 10:50 p.m., Julien Le Dem wrote: > > Great work! > > some minor comments. > > This is getting really good!
Thanks Julien! > On July 2, 2012, 10:50 p.m., Julien Le Dem wrote: > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POFRJoin.java, > > line 447 > > <https://reviews.apache.org/r/4651/diff/10/?file=117525#file117525line447> > > > > why not just convert the tuple here, instead of extending ArrayList? > > It would seem a little more obvious. > > If you want a strategy pattern, it does not have to be in List. > > Jonathan Coveney wrote: > See above. Give thoughts w.r.t. that and I'll go with it. In this vein, I could create a side interface (kind of like TupleMaker) that would encapsulate the proper datatype, and not have the potential pitfalls of "oh the containsKey was or wasn't implement" or something like that? - Jonathan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4651/#review8812 ----------------------------------------------------------- On June 29, 2012, 9:55 p.m., Jonathan Coveney wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/4651/ > ----------------------------------------------------------- > > (Updated June 29, 2012, 9:55 p.m.) > > > Review request for pig and Julien Le Dem. > > > Description > ------- > > This work builds on Dmitriy's PrimitiveTuple work. The idea is that, knowing > the Schema on the frontend, we can code generate Tuples which can be used for > fun and profit. In rudimentary tests, the memory efficiency is 2-4x better, > and it's ~15% smaller serialized (heavily heavily depends on the data, > though). Need to do get/set tests, but assuming that it's on par (or even > faster) than Tuple, the memory gain is huge. > > Need to clean up the code and add tests. > > Right now, it generates a SchemaTuple for every inputSchema and outputSchema > given to UDF's. The next step is to make a SchemaBag, where I think the > serialization savings will be really huge. > > Needs tests and comments, but I want the code to settle a bit. > > > This addresses bug PIG-2632. > https://issues.apache.org/jira/browse/PIG-2632 > > > Diffs > ----- > > trunk/.gitignore 1355561 > trunk/conf/pig.properties 1355561 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java > 1355561 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapBase.java > 1355561 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigGenericMapReduce.java > 1355561 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigTupleDefaultRawComparator.java > 1355561 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java > 1355561 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java > 1355561 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POFRJoin.java > 1355561 > > trunk/src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POMergeJoin.java > 1355561 > trunk/src/org/apache/pig/builtin/mock/Storage.java 1355561 > trunk/src/org/apache/pig/data/AppendableSchemaTuple.java PRE-CREATION > trunk/src/org/apache/pig/data/BinInterSedes.java 1355561 > trunk/src/org/apache/pig/data/BinSedesTupleFactory.java 1355561 > trunk/src/org/apache/pig/data/DataByteArray.java 1355561 > trunk/src/org/apache/pig/data/FieldIsNullException.java PRE-CREATION > trunk/src/org/apache/pig/data/PBooleanTuple.java 1355561 > trunk/src/org/apache/pig/data/PDoubleTuple.java 1355561 > trunk/src/org/apache/pig/data/PFloatTuple.java 1355561 > trunk/src/org/apache/pig/data/PIntTuple.java 1355561 > trunk/src/org/apache/pig/data/PLongTuple.java 1355561 > trunk/src/org/apache/pig/data/PStringTuple.java 1355561 > trunk/src/org/apache/pig/data/PrimitiveFieldTuple.java 1355561 > trunk/src/org/apache/pig/data/PrimitiveTuple.java 1355561 > trunk/src/org/apache/pig/data/SchemaTuple.java PRE-CREATION > trunk/src/org/apache/pig/data/SchemaTupleBackend.java PRE-CREATION > trunk/src/org/apache/pig/data/SchemaTupleClassGenerator.java PRE-CREATION > trunk/src/org/apache/pig/data/SchemaTupleFactory.java PRE-CREATION > trunk/src/org/apache/pig/data/SchemaTupleFrontend.java PRE-CREATION > trunk/src/org/apache/pig/data/TupleFactory.java 1355561 > trunk/src/org/apache/pig/data/TupleMaker.java PRE-CREATION > trunk/src/org/apache/pig/data/TypeAwareTuple.java 1355561 > trunk/src/org/apache/pig/data/utils/BytesHelper.java PRE-CREATION > trunk/src/org/apache/pig/data/utils/MethodHelper.java PRE-CREATION > trunk/src/org/apache/pig/data/utils/SedesHelper.java PRE-CREATION > trunk/src/org/apache/pig/data/utils/StructuresHelper.java PRE-CREATION > trunk/src/org/apache/pig/impl/PigContext.java 1355561 > trunk/src/org/apache/pig/impl/io/InterRecordReader.java 1355561 > trunk/src/org/apache/pig/impl/io/NullableTuple.java 1355561 > > trunk/src/org/apache/pig/newplan/logical/expression/ExpToPhyTranslationVisitor.java > 1355561 > trunk/src/org/apache/pig/newplan/logical/expression/UserFuncExpression.java > 1355561 > > trunk/src/org/apache/pig/newplan/logical/relational/LogToPhyTranslationVisitor.java > 1355561 > > trunk/src/org/apache/pig/newplan/logical/relational/LogicalRelationalOperator.java > 1355561 > > trunk/src/org/apache/pig/newplan/logical/rules/GroupByConstParallelSetter.java > 1355561 > trunk/src/org/apache/pig/newplan/logical/rules/MergeForEach.java 1355561 > trunk/test/org/apache/pig/data/TestSchemaTuple.java PRE-CREATION > trunk/test/org/apache/pig/data/utils/TestMethodHelper.java PRE-CREATION > trunk/test/org/apache/pig/test/TestDataBag.java 1355561 > trunk/test/org/apache/pig/test/TestLogicalPlanBuilder.java 1355561 > trunk/test/org/apache/pig/test/TestPrimitiveFieldTuple.java 1355561 > trunk/test/org/apache/pig/test/TestPrimitiveTuple.java 1355561 > trunk/test/org/apache/pig/test/TestSchema.java 1355561 > > Diff: https://reviews.apache.org/r/4651/diff/ > > > Testing > ------- > > > Thanks, > > Jonathan Coveney > >
