-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8104/
-----------------------------------------------------------
Review request for pig and Cheolsoo Park.
Description
-------
The current AvroStorage implementation has a lot of issues: it requires old
versions of Avro, it copies data much more than needed, and it's verbose and
complicated. (One pet peeve of mine is that old versions of Avro don't support
Snappy compression.)
I rewrote AvroStorage from scratch to fix these issues. In early tests, the new
implementation is significantly faster, and the code is a lot simpler.
Rewriting AvroStorage also enabled me to implement support for Trevni.
This is the latest version of the patch, complete with test cases and
TrevniStorage. (Test cases for TrevniStorage are still missing).
This addresses bug PIG-3015.
https://issues.apache.org/jira/browse/PIG-3015
Diffs
-----
build.xml 7d468a0
ivy.xml 70e8d50
ivy/libraries.properties 317564f
src/org/apache/pig/builtin/AvroStorage.java PRE-CREATION
src/org/apache/pig/builtin/TrevniStorage.java PRE-CREATION
src/org/apache/pig/impl/util/AvroBagWrapper.java PRE-CREATION
src/org/apache/pig/impl/util/AvroMapWrapper.java PRE-CREATION
src/org/apache/pig/impl/util/AvroRecordReader.java PRE-CREATION
src/org/apache/pig/impl/util/AvroRecordWriter.java PRE-CREATION
src/org/apache/pig/impl/util/AvroStorageDataConversionUtilities.java
PRE-CREATION
src/org/apache/pig/impl/util/AvroStorageSchemaConversionUtilities.java
PRE-CREATION
src/org/apache/pig/impl/util/AvroTupleWrapper.java PRE-CREATION
test/commit-tests 5081fbc
test/org/apache/pig/builtin/TestAvroStorage.java PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/directory_test.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_ai1_ao2.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_ao2.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_codec.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/identity_just_ao2.pig PRE-CREATION
test/org/apache/pig/builtin/avro/code/pig/recursive_tests.pig PRE-CREATION
test/org/apache/pig/builtin/avro/createTests.bash PRE-CREATION
test/org/apache/pig/builtin/avro/createests.py PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/deflate/records.avro PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/deflate/recordsAsOutputByPig.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/snappy/records.avro PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordWithRepeatedSubRecords.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/records.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsAsOutputByPig.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsOfArrays.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsOfArraysOfRecords.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsSubSchema.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsSubSchemaNullable.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsWithEnums.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsWithFixed.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsWithMaps.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsWithMapsOfRecords.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recordsWithNullableUnions.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/avro/uncompressed/recursiveRecord.avro
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordWithRepeatedSubRecords.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/records.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsAsOutputByPig.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsOfArrays.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsOfArraysOfRecords.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsSubSchema.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsSubSchemaNullable.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithEnums.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithFixed.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithMaps.json PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithMapsOfRecords.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recordsWithNullableUnions.json
PRE-CREATION
test/org/apache/pig/builtin/avro/data/json/recursiveRecord.json PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordWithRepeatedSubRecords.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/records.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsAsOutputByPig.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsOfArrays.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsOfArraysOfRecords.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsSubSchema.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsSubSchemaNullable.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithEnums.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithFixed.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithMaps.avsc PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithMapsOfRecords.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recordsWithNullableUnions.avsc
PRE-CREATION
test/org/apache/pig/builtin/avro/schema/recursiveRecord.avsc PRE-CREATION
test/unit-tests 0f18a0e
Diff: https://reviews.apache.org/r/8104/diff/
Testing
-------
Thanks,
Joseph Adler
