[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 HIVE-24157 introduces a restriction to prohibit casting DATE/TIMESTAMP types to and from NUMERIC. It's enabled by default and can be turned off by set hive.strict.timestamp.conversion=false. This restriction breaks the data loading on avro_coldef and avro_extra_coldef tables, which results in empty data set and finally fails TestAvroSchemaResolution.test_avro_schema_resolution. This patch explicitly disables the restriction in loading these two avro tables. The Hive version currently used for development does not have HIVE-24157, but upstream Hive does have it. Adding hive.strict.timestamp.conversion does not cause problems for Hive versions that don't have HIVE-24157. Tests: - Run the data loading and test_avro_schema_resolution locally using a Hive that has HIVE-24157. - Run CORE tests - Run data loading with a Hive that doesn't have HIVE-24157. Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Reviewed-on: http://gerrit.cloudera.org:8080/21413 Reviewed-by: Impala Public Jenkins Tested-by: Joe McDonnell --- M testdata/avro_schema_resolution/create_table.sql 1 file changed, 4 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Patch Set 2: Verified+1 This is just a flaky test. This only impacts dataload and everything having to do with avro schema resolution has been fine. Going ahead with this. -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 09 May 2024 16:22:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Joe McDonnell has removed a vote on this change. Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10625/ -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 09 May 2024 15:53:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10625/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 09 May 2024 10:50:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 09 May 2024 10:50:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Patch Set 1: Code-Review+2 Thanks for digging into this! -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 09 May 2024 10:50:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 09 May 2024 02:37:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10621/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 May 2024 21:34:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21413 ) Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/16142/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 08 May 2024 21:24:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10451: Fix avro table loading failures caused by HIVE-24157
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21413 Change subject: IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 .. IMPALA-10451: Fix avro table loading failures caused by HIVE-24157 HIVE-24157 introduces a restriction to prohibit casting DATE/TIMESTAMP types to and from NUMERIC. It's enabled by default and can be turned off by set hive.strict.timestamp.conversion=false. This restriction breaks the data loading on avro_coldef and avro_extra_coldef tables, which results in empty data set and finally fails TestAvroSchemaResolution.test_avro_schema_resolution. This patch explicitly disables the restriction in loading these two avro tables. The Hive version currently used for development does not have HIVE-24157, but upstream Hive does have it. Adding hive.strict.timestamp.conversion does not cause problems for Hive versions that don't have HIVE-24157. Tests: - Run the data loading and test_avro_schema_resolution locally using a Hive that has HIVE-24157. - Run CORE tests - Run data loading with a Hive that doesn't have HIVE-24157. Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 --- M testdata/avro_schema_resolution/create_table.sql 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/21413/1 -- To view, visit http://gerrit.cloudera.org:8080/21413 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3e2a47d60d4079fece9c04091258215f3d6a7b52 Gerrit-Change-Number: 21413 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
