[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726135#comment-13726135 ] Hudson commented on HIVE-3264: -- ABORTED: Integrated in Hive-trunk-hadoop2 #319 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/319/]) HIVE-3264 : Add support for binary dataype to AvroSerde (Eli Reisman & Mark Wagner via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1508528) * /hive/trunk/data/files/csv.txt * /hive/trunk/ql/src/test/queries/clientpositive/avro_nullable_fields.q * /hive/trunk/ql/src/test/results/clientpositive/avro_nullable_fields.q.out * /hive/trunk/ql/src/test/results/clientpositive/avro_schema_literal.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan >Assignee: Eli Reisman > Labels: patch > Fix For: 0.12.0 > > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch, HIVE-3264.6.patch, HIVE-3264.7.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13725330#comment-13725330 ] Hudson commented on HIVE-3264: -- FAILURE: Integrated in Hive-trunk-h0.21 #2234 (See [https://builds.apache.org/job/Hive-trunk-h0.21/2234/]) HIVE-3264 : Add support for binary dataype to AvroSerde (Eli Reisman & Mark Wagner via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1508528) * /hive/trunk/data/files/csv.txt * /hive/trunk/ql/src/test/queries/clientpositive/avro_nullable_fields.q * /hive/trunk/ql/src/test/results/clientpositive/avro_nullable_fields.q.out * /hive/trunk/ql/src/test/results/clientpositive/avro_schema_literal.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan >Assignee: Eli Reisman > Labels: patch > Fix For: 0.12.0 > > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch, HIVE-3264.6.patch, HIVE-3264.7.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13724418#comment-13724418 ] Hudson commented on HIVE-3264: -- SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #109 (See [https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/109/]) HIVE-3264 : Add support for binary dataype to AvroSerde (Eli Reisman & Mark Wagner via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1508528) * /hive/trunk/data/files/csv.txt * /hive/trunk/ql/src/test/queries/clientpositive/avro_nullable_fields.q * /hive/trunk/ql/src/test/results/clientpositive/avro_nullable_fields.q.out * /hive/trunk/ql/src/test/results/clientpositive/avro_schema_literal.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan >Assignee: Eli Reisman > Labels: patch > Fix For: 0.12.0 > > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch, HIVE-3264.6.patch, HIVE-3264.7.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13724323#comment-13724323 ] Hudson commented on HIVE-3264: -- FAILURE: Integrated in Hive-trunk-hadoop2-ptest #37 (See [https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/37/]) HIVE-3264 : Add support for binary dataype to AvroSerde (Eli Reisman & Mark Wagner via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1508528) * /hive/trunk/data/files/csv.txt * /hive/trunk/ql/src/test/queries/clientpositive/avro_nullable_fields.q * /hive/trunk/ql/src/test/results/clientpositive/avro_nullable_fields.q.out * /hive/trunk/ql/src/test/results/clientpositive/avro_schema_literal.q.out * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerializer.java * /hive/trunk/serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroDeserializer.java * /hive/trunk/serde/src/test/org/apache/hadoop/hive/serde2/avro/TestAvroObjectInspectorGenerator.java > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan >Assignee: Eli Reisman > Labels: patch > Fix For: 0.12.0 > > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch, HIVE-3264.6.patch, HIVE-3264.7.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722704#comment-13722704 ] Ashutosh Chauhan commented on HIVE-3264: +1 > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan >Assignee: Eli Reisman > Labels: patch > Fix For: 0.12.0 > > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch, HIVE-3264.6.patch, HIVE-3264.7.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722674#comment-13722674 ] Jakob Homan commented on HIVE-3264: --- btw, Mark, once this goes through, please update the AvroSerde info (with the version number): https://cwiki.apache.org/confluence/display/Hive/AvroSerDe Ping the user list if you don't have write acesss. Thanks. > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan > Labels: patch > Fix For: 0.12.0 > > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch, HIVE-3264.6.patch, HIVE-3264.7.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13722636#comment-13722636 ] Jakob Homan commented on HIVE-3264: --- +1. Looks good. > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan > Labels: patch > Fix For: 0.12.0 > > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch, HIVE-3264.6.patch, HIVE-3264.7.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13707503#comment-13707503 ] Mark Wagner commented on HIVE-3264: --- Also posted to RB: https://reviews.apache.org/r/12531/ > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan > Labels: patch > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch, HIVE-3264.6.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13550963#comment-13550963 ] Eli Reisman commented on HIVE-3264: --- The patch isn't stale yet (whew!), I'm working on a test... > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan > Labels: patch > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499866#comment-13499866 ] Eli Reisman commented on HIVE-3264: --- This had fallen off my radar too, sorry. What needs to be done/added? When I was originally working on this, I was told the .q file approach was the test we needed. What sort of test should I add? > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan > Labels: patch > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461605#comment-13461605 ] Ashutosh Chauhan commented on HIVE-3264: I see that describe is already there in .q file and its printing the expected output as well. Adding more unit tests is always welcome. Looks like it will serialize binary to bytes. I see the code there. But there are no tests, (.q is just testing desirialization) so not sure. It will be good to add a test for it. > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan > Labels: patch > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461542#comment-13461542 ] Jakob Homan commented on HIVE-3264: --- Actually, can we add to the .q file a describe on the table to verify that Hive sees the new type correctly/ Also, there should be an equivalent unit test added to TestAvroDeserializer. Also, does this support serializing Hive binary to bytes? Sorry for this falling of my radar... > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan > Labels: patch > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461541#comment-13461541 ] Ashutosh Chauhan commented on HIVE-3264: +1 will commit if tests pass. > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 0.9.0 >Reporter: Jakob Homan > Labels: patch > Attachments: HIVE-3264-1.patch, HIVE-3264-2.patch, HIVE-3264-3.patch, > HIVE-3264-4.patch, HIVE-3264-5.patch > > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3264) Add support for binary dataype to AvroSerde
[ https://issues.apache.org/jira/browse/HIVE-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13426909#comment-13426909 ] Eli Reisman commented on HIVE-3264: --- Looking at the source, I'm seeing AvroSerDe code mapping BYTE to Constants.TINYINT_TYPE_NAME in Hive, but binary type (array/blob) already maps to Constants.BINARY_TYPE_NAME -- and inside the Hive constants, I see no "byte" type to replace TINYINT_TYPE_NAME, is there something I'm missing? I could not tell from the description whether the binary type is what you were discussing (looks like its already in place), or the single-byte type in specific. In that case, there is no replacement I can find for tiny int on the Hive side. I didn't see anything on the binary array being implemented as tiny ints any more in the code. I am looking in the org.apache.hive.serde2.* packages. > Add support for binary dataype to AvroSerde > --- > > Key: HIVE-3264 > URL: https://issues.apache.org/jira/browse/HIVE-3264 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Reporter: Jakob Homan > > When the AvroSerde was written, Hive didn't have a binary type, so Avro's > byte array type is converted an array of small ints. Now that HIVE-2380 is > in, this step isn't necessary and we can convert both Avro's bytes type and > probably fixed type to Hive's binary type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
