> On Jan. 19, 2013, 7:29 p.m., Mathias Herberts wrote: > > src/org/apache/pig/data/DefaultTuple.java, line 359 > > <https://reviews.apache.org/r/9012/diff/1-2/?file=249804#file249804line359> > > > > Since BigDecimal have a scale which is an int, nothing prevents a > > BigDecimal to have a scale which won't fit on a short, and thus whose > > string representation might also be longer than a short. > > > > Is I understand this code correctly (thanks to your explanation), if > > type if CHARARRAY, field length is encoded on a short? This will be > > troublesome if scale > 0x7ffff.
This is a bit annoying/confusing, but there are two internal "CHARARRAYS," either a CHARARRAY or a BIGCHARRAY. So if it's bigger than a short, it will be an int (ie a BIGCHARARRAY) and it should deserialized correctly. Literally all DataType.BIGINTEGER in this case signals is "what comes after is a string," which we then serialized and deserialize in accordance with the rest of pig. - Jonathan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9012/#review15522 ----------------------------------------------------------- On Jan. 18, 2013, 10:11 p.m., Jonathan Coveney wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/9012/ > ----------------------------------------------------------- > > (Updated Jan. 18, 2013, 10:11 p.m.) > > > Review request for pig, Alan Gates and Mathias Herberts. > > > Description > ------- > > This patch adds big integer and big decimal support to Pig. It could use more > tests, something I'd appreciate feedback on (but I wanted to make sure the > core implementation is good) > > > This addresses bug PIG-2764. > https://issues.apache.org/jira/browse/PIG-2764 > > > Diffs > ----- > > .gitignore cc62d7d > src/org/apache/pig/LoadCaster.java 574769b > src/org/apache/pig/PigWarning.java 5de075f > src/org/apache/pig/StoreCaster.java 5fe48de > src/org/apache/pig/backend/hadoop/BigDecimalWritable.java PRE-CREATION > src/org/apache/pig/backend/hadoop/BigIntegerWritable.java PRE-CREATION > src/org/apache/pig/backend/hadoop/HDataType.java 84a56b8 > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java > 96fba6b > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigDecimalRawComparator.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/PigBigIntegerRawComparator.java > PRE-CREATION > > src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/partitioners/WeightedRangePartitioner.java > 9749339 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/PhysicalOperator.java > f40eb43 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Add.java > c84b767 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/ConstantExpression.java > db3840f > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Divide.java > 4656c28 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/EqualToExpr.java > 6683beb > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/ExpressionOperator.java > 2806336 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/GTOrEqualToExpr.java > d64a080 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/GreaterThanExpr.java > 704d0b8 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/LTOrEqualToExpr.java > 9dc929e > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/LessThanExpr.java > 0320698 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Mod.java > 6819185 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Multiply.java > 7b57bed > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/NotEqualToExpr.java > 79a4461 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POBinCond.java > 08544d5 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POCast.java > e8c2f2c > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POIsNull.java > f20b839 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/PONegative.java > c076ae7 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POProject.java > 8887133 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserComparisonFunc.java > 479eb83 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/POUserFunc.java > 3c7e741 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/expressionOperators/Subtract.java > 79d4c73 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POForEach.java > bf2ba08 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POLocalRearrange.java > ddb25f1 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPartialAgg.java > aa11409 > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPreCombinerLocalRearrange.java > 52401eb > > src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POSort.java > ad33e7b > src/org/apache/pig/backend/hadoop/hbase/HBaseBinaryConverter.java 60a5899 > src/org/apache/pig/backend/hadoop/hbase/HBaseStorage.java a6f4ea6 > src/org/apache/pig/builtin/ABS.java 8a7c631 > src/org/apache/pig/builtin/BigDecimalAbs.java PRE-CREATION > src/org/apache/pig/builtin/BigIntegerAbs.java PRE-CREATION > src/org/apache/pig/builtin/BinStorage.java 38b4492 > src/org/apache/pig/builtin/TextLoader.java d5bcf02 > src/org/apache/pig/builtin/Utf8StorageConverter.java da12ed6 > src/org/apache/pig/data/BinInterSedes.java e851d8b > src/org/apache/pig/data/DataReaderWriter.java 37a162a > src/org/apache/pig/data/DataType.java e4c7b98 > src/org/apache/pig/data/DefaultTuple.java e182b06 > src/org/apache/pig/data/SizeUtil.java 90e5d94 > src/org/apache/pig/data/TypeAwareTuple.java cf78d0a > src/org/apache/pig/impl/io/NullableBigDecimalWritable.java PRE-CREATION > src/org/apache/pig/impl/io/NullableBigIntegerWritable.java PRE-CREATION > src/org/apache/pig/impl/logicalLayer/schema/SchemaUtil.java c257ada > src/org/apache/pig/impl/util/CastUtils.java 309130a > src/org/apache/pig/impl/util/NumValCarrier.java af519fd > src/org/apache/pig/impl/util/StorageUtil.java 087651a > src/org/apache/pig/newplan/logical/rules/ConstExpEvaluator.java f968294 > src/org/apache/pig/newplan/logical/visitor/TypeCheckingExpVisitor.java > d7a1370 > src/org/apache/pig/parser/AliasMasker.g 8cb1b30 > src/org/apache/pig/parser/AstPrinter.g c6f7ff8 > src/org/apache/pig/parser/AstValidator.g 8646078 > src/org/apache/pig/parser/LogicalPlanBuilder.java 699f7a6 > src/org/apache/pig/parser/LogicalPlanGenerator.g 9b9c099 > src/org/apache/pig/parser/QueryLexer.g f201916 > src/org/apache/pig/parser/QueryParser.g 642884e > src/org/apache/pig/pen/AugmentBaseDataVisitor.java 57dcb14 > > test/e2e/pig/udfs/java/org/apache/pig/test/udf/storefunc/PigPerformanceLoader.java > 94c28be > test/org/apache/pig/test/TestAdd.java de9e90d > test/org/apache/pig/test/TestBestFitCast.java e3172d2 > test/org/apache/pig/test/TestPOCast.java 72787b3 > > Diff: https://reviews.apache.org/r/9012/diff/ > > > Testing > ------- > > > Thanks, > > Jonathan Coveney > >
