[jira] [Commented] (HIVE-10708) Add SchemaCompatibility check to AvroDeserializer
[ https://issues.apache.org/jira/browse/HIVE-10708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738132#comment-14738132 ] Swarnim Kulkarni commented on HIVE-10708: - Test failures seem unrelated to this patch. > Add SchemaCompatibility check to AvroDeserializer > - > > Key: HIVE-10708 > URL: https://issues.apache.org/jira/browse/HIVE-10708 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-10708.1.patch.txt > > > Avro provides a nice API[1] to check if the given reader schema can be used > to deserialize the data given its writer schema. I think it would be super > nice to integrate this into the AvroDeserializer so that we can fail fast and > gracefully if there is a bad schema compatibility > [1] > https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/SchemaCompatibility.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10708) Add SchemaCompatibility check to AvroDeserializer
[
https://issues.apache.org/jira/browse/HIVE-10708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738123#comment-14738123
]
Hive QA commented on HIVE-10708:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12754816/HIVE-10708.1.patch.txt
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 9424 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join35
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_metadata_only_queries
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats_noscan_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_timestamp_udf
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_percentile
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_distinct_2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_orderby_5
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5217/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5217/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5217/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12754816 - PreCommit-HIVE-TRUNK-Build
> Add SchemaCompatibility check to AvroDeserializer
> -
>
> Key: HIVE-10708
> URL: https://issues.apache.org/jira/browse/HIVE-10708
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Attachments: HIVE-10708.1.patch.txt
>
>
> Avro provides a nice API[1] to check if the given reader schema can be used
> to deserialize the data given its writer schema. I think it would be super
> nice to integrate this into the AvroDeserializer so that we can fail fast and
> gracefully if there is a bad schema compatibility
> [1]
> https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/SchemaCompatibility.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10708) Add SchemaCompatibility check to AvroDeserializer
[ https://issues.apache.org/jira/browse/HIVE-10708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736147#comment-14736147 ] Swarnim Kulkarni commented on HIVE-10708: - Decided to add a simple flag to turn this compatibility check on. Keeping this flag off by default for backwards compatibility. > Add SchemaCompatibility check to AvroDeserializer > - > > Key: HIVE-10708 > URL: https://issues.apache.org/jira/browse/HIVE-10708 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > > Avro provides a nice API[1] to check if the given reader schema can be used > to deserialize the data given its writer schema. I think it would be super > nice to integrate this into the AvroDeserializer so that we can fail fast and > gracefully if there is a bad schema compatibility > [1] > https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/SchemaCompatibility.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10708) Add SchemaCompatibility check to AvroDeserializer
[ https://issues.apache.org/jira/browse/HIVE-10708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736153#comment-14736153 ] Swarnim Kulkarni commented on HIVE-10708: - RB: https://reviews.apache.org/r/38203/ > Add SchemaCompatibility check to AvroDeserializer > - > > Key: HIVE-10708 > URL: https://issues.apache.org/jira/browse/HIVE-10708 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-10708.1.patch.txt > > > Avro provides a nice API[1] to check if the given reader schema can be used > to deserialize the data given its writer schema. I think it would be super > nice to integrate this into the AvroDeserializer so that we can fail fast and > gracefully if there is a bad schema compatibility > [1] > https://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/SchemaCompatibility.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
