[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14162690#comment-14162690 ] Brock Noland commented on HIVE-5865: I think a release note is appropriate... Added! AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Assignee: Ben Roling Fix For: 0.15.0 Attachments: HIVE-5865-v2.patch, HIVE-5865.2.patch, HIVE-5865.2.patch, HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14161323#comment-14161323 ] Lefty Leverenz commented on HIVE-5865: -- Doc query: Should this be documented in the Avro SerDe wikidoc for release 0.15.0? * [Avro SerDe | https://cwiki.apache.org/confluence/display/Hive/AvroSerDe] Gentle nudge: How about a release note? AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Assignee: Ben Roling Fix For: 0.15.0 Attachments: HIVE-5865-v2.patch, HIVE-5865.2.patch, HIVE-5865.2.patch, HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[
https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157885#comment-14157885
]
Hive QA commented on HIVE-5865:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12672645/HIVE-5865.2.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 6542 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.pig.TestHCatLoader.testColumnarStorePushdown[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes[5]
org.apache.hive.hcatalog.pig.TestHCatStorer.testDynamicPartitioningMultiPartColsNoDataInDataNoSpec[5]
org.apache.hive.hcatalog.pig.TestHCatStorer.testEmptyStore[5]
org.apache.hive.hcatalog.pig.TestHCatStorer.testNoAlias[5]
org.apache.hive.hcatalog.pig.TestHCatStorer.testPartitionPublish[5]
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1099/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1099/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1099/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12672645
AvroDeserializer incorrectly assumes keys to Maps will always be of type
'org.apache.avro.util.Utf8'
Key: HIVE-5865
URL: https://issues.apache.org/jira/browse/HIVE-5865
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: Ben Roling
Attachments: HIVE-5865-v2.patch, HIVE-5865.2.patch,
HIVE-5865.2.patch, HIVE-5865.patch
AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys
will always be 'org.apache.avro.util.Utf8'. If the reader schema defines
avro.java.string=String, this assumption does not hold, resulting in a
ClassCastException.
I think a simple fix would be to define 'mapDatum' with type
MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has
the more general type of 'CharSequence' avoids the need to make an assumption
of either String or Utf8.
I discovered the issue when using Hive 0.11.0. Looking at the tags it is
also there is in 0.12.0 and trunk:
https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313
The reason I saw this issue was because I pointed my Hive table to a schema
file I populated based on pulling the schema from the SCHEMA$ attribute of an
Avro generated Java class and I used stringType=String in the configuration
of the avro-maven-plugin when generating my Java classes.
If I alter the schema my Hive table points to such that it doesn't have the
avro.java.string attribute on my map type objects then queries work fine
but if I leave those in there I get the ClassCastException anytime I try to
query the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158095#comment-14158095 ] Ben Roling commented on HIVE-5865: -- Hey [~brocknoland] - the HCatLoader tests failed again but as I stated previously, those tests are failing without any of the changes from this JIRA. Is there anything more you want me to do on this? Are those test failures something someone else is already looking at? I would have to assume they are being seen in other builds as well. AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Attachments: HIVE-5865-v2.patch, HIVE-5865.2.patch, HIVE-5865.2.patch, HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14158173#comment-14158173 ] Brock Noland commented on HIVE-5865: Those tests are failing due to: https://issues.apache.org/jira/browse/HIVE-8271?focusedCommentId=14157581page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14157581 I will commit this today. Thanks!! AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Attachments: HIVE-5865-v2.patch, HIVE-5865.2.patch, HIVE-5865.2.patch, HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[
https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156243#comment-14156243
]
Hive QA commented on HIVE-5865:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12672440/HIVE-5865.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 6503 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.hcatalog.pig.TestHCatLoader.testColumnarStorePushdown[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testConvertBooleanToInt[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testGetInputBytes[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testProjectionsBasic[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataBasic[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testReadPartitionedBasic[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadBasic[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadComplex[5]
org.apache.hive.hcatalog.pig.TestHCatLoader.testSchemaLoadPrimitiveTypes[5]
org.apache.hive.hcatalog.pig.TestHCatStorer.testDynamicPartitioningMultiPartColsNoDataInDataNoSpec[5]
org.apache.hive.hcatalog.pig.TestHCatStorer.testEmptyStore[5]
org.apache.hive.hcatalog.pig.TestHCatStorer.testNoAlias[5]
org.apache.hive.hcatalog.pig.TestHCatStorer.testPartitionPublish[5]
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1083/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1083/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1083/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12672440
AvroDeserializer incorrectly assumes keys to Maps will always be of type
'org.apache.avro.util.Utf8'
Key: HIVE-5865
URL: https://issues.apache.org/jira/browse/HIVE-5865
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0, 0.12.0, 0.13.0
Reporter: Ben Roling
Attachments: HIVE-5865.patch
AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys
will always be 'org.apache.avro.util.Utf8'. If the reader schema defines
avro.java.string=String, this assumption does not hold, resulting in a
ClassCastException.
I think a simple fix would be to define 'mapDatum' with type
MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has
the more general type of 'CharSequence' avoids the need to make an assumption
of either String or Utf8.
I discovered the issue when using Hive 0.11.0. Looking at the tags it is
also there is in 0.12.0 and trunk:
https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313
The reason I saw this issue was because I pointed my Hive table to a schema
file I populated based on pulling the schema from the SCHEMA$ attribute of an
Avro generated Java class and I used stringType=String in the configuration
of the avro-maven-plugin when generating my Java classes.
If I alter the schema my Hive table points to such that it doesn't have the
avro.java.string attribute on my map type objects then queries work fine
but if I leave those in there I get the ClassCastException anytime I try to
query the table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156692#comment-14156692 ] Ben Roling commented on HIVE-5865: -- Yeah, I'll take a look at the test failures and also see if I can create a new test specific to the issue. It was lazy of me not to have done that from the start -- I just didn't take the time to familiarize myself with the full build and testing setup of the project to do it as this is my first contribution to Hive. AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Attachments: HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156883#comment-14156883 ] Ben Roling commented on HIVE-5865: -- I don't know what exactly is the cause of the TestHCatalogLoader failures but I don't think it is related to my changes. That test seems to fail even with my changes backed out. AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Attachments: HIVE-5865-v2.patch, HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157198#comment-14157198 ] Brock Noland commented on HIVE-5865: +1 pending tests AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Attachments: HIVE-5865-v2.patch, HIVE-5865.2.patch, HIVE-5865.2.patch, HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14157207#comment-14157207 ] Ben Roling commented on HIVE-5865: -- Thanks [~brocknoland]! AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Attachments: HIVE-5865-v2.patch, HIVE-5865.2.patch, HIVE-5865.2.patch, HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-5865) AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8'
[ https://issues.apache.org/jira/browse/HIVE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155729#comment-14155729 ] Sean Busbey commented on HIVE-5865: --- Can you add a test case for a table backed by a schema that requires this patch to avoid failing? AvroDeserializer incorrectly assumes keys to Maps will always be of type 'org.apache.avro.util.Utf8' Key: HIVE-5865 URL: https://issues.apache.org/jira/browse/HIVE-5865 Project: Hive Issue Type: Bug Affects Versions: 0.11.0, 0.12.0, 0.13.0 Reporter: Ben Roling Attachments: HIVE-5865.patch AvroDeserializer. deserializeMap() incorrectly assumes the type of they keys will always be 'org.apache.avro.util.Utf8'. If the reader schema defines avro.java.string=String, this assumption does not hold, resulting in a ClassCastException. I think a simple fix would be to define 'mapDatum' with type MapCharSequence,Object instead of MapUtf8,Object. Assuming the key has the more general type of 'CharSequence' avoids the need to make an assumption of either String or Utf8. I discovered the issue when using Hive 0.11.0. Looking at the tags it is also there is in 0.12.0 and trunk: https://github.com/apache/hive/blob/99f5bfcdf64330d062a30c0c9d83be1fbee54c34/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroDeserializer.java#L313 The reason I saw this issue was because I pointed my Hive table to a schema file I populated based on pulling the schema from the SCHEMA$ attribute of an Avro generated Java class and I used stringType=String in the configuration of the avro-maven-plugin when generating my Java classes. If I alter the schema my Hive table points to such that it doesn't have the avro.java.string attribute on my map type objects then queries work fine but if I leave those in there I get the ClassCastException anytime I try to query the table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
