[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata If we insert data into an acid partitioned table from Hive, the generated events will be like open_txn -> alter_partition -> commit_txn. Previously we assumed the partition object with the alter_partition event has write id < current write id. However, that is not a valid assumption, the partition object is actually the write id allocated in this transaction. That means in commit_txn event, we will have a partition with write id equals to the write id of cached partition. So we need to modify the '<' condition to '<='. Tests: After IMPALA-10923, we now refresh file metadata while processing commit events. Therefore, we can add back the test disabled in IMPALA-9057. Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Reviewed-on: http://gerrit.cloudera.org:8080/18175 Tested-by: Impala Public Jenkins Reviewed-by: Sourabh Goyal Reviewed-by: Joe McDonnell --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M tests/metadata/test_event_processing.py 2 files changed, 1 insertion(+), 2 deletions(-) Approvals: Impala Public Jenkins: Verified Sourabh Goyal: Looks good to me, but someone else must approve Joe McDonnell: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 4 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 21:21:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39 PS3, Line 39: > Yes, the test will fail intermittently without fine-grained table refreshin Thanks for the clarification. -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 20:39:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39 PS3, Line 39: > @Yu-Wen: Please confirm the following: Yes, the test will fail intermittently without fine-grained table refreshing. The issue was that we previously refresh file metadata at alter partition event but while alter partition event was processed the transaction might not be committed yet. If it is committed, we could get new file metadata. Otherwise, we would still see stale file metadata. After my patch, we can now refresh file metadata at commit event. -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 18:06:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py File tests/metadata/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/18175/3/tests/metadata/test_event_processing.py@a39 PS3, Line 39: @Yu-Wen: Please confirm the following: 1. Are we enabling the test because the flakiness got fixed? 2. Does this test fail if we enable it without the fix in this patch? -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 17:48:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Thu, 03 Feb 2022 03:29:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7791/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Wed, 02 Feb 2022 20:52:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7776/ -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Sat, 29 Jan 2022 15:34:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7776/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Sat, 29 Jan 2022 09:03:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7772/ -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Sat, 29 Jan 2022 02:28:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10065/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 28 Jan 2022 22:07:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Yu-Wen Lai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata If we insert data into an acid partitioned table from Hive, the generated events will be like open_txn -> alter_partition -> commit_txn. Previously we assumed the partition object with the alter_partition event has write id < current write id. However, that is not a valid assumption, the partition object is actually the write id allocated in this transaction. That means in commit_txn event, we will have a partition with write id equals to the write id of cached partition. So we need to modify the '<' condition to '<='. Tests: After IMPALA-10923, we now refresh file metadata while processing commit events. Therefore, we can add back the test disabled in IMPALA-9057. Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M tests/metadata/test_event_processing.py 2 files changed, 1 insertion(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/18175/3 -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7772/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 3 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 28 Jan 2022 21:46:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Yu-Wen Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java File fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java: http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@255 PS1, Line 255: updateMinOpenWriteId(); > how is this related to this change? The minOpenWriteId is not used actually, so no harm as of now. I will remove this from the change and refactor this in another patch. -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 28 Jan 2022 21:42:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Sourabh Goyal has posted comments on this change. (
http://gerrit.cloudera.org:8080/18175 )
Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh
table file metadata
..
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2763
PS1, Line 2763: || hdfsPartition.getWriteId() <=
partition.getWriteId())) {
Please add a test for this since the bug was not caught in the existing test
suite.
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
File
fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java:
http://gerrit.cloudera.org:8080/#/c/18175/1/fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java@255
PS1, Line 255: updateMinOpenWriteId();
how is this related to this change?
On a side note, can we always derive minOpenWriteId from exceptions list and
abortedBits? If yes, we can skip maintaining minOpenWriteId
--
To view, visit http://gerrit.cloudera.org:8080/18175
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d
Gerrit-Change-Number: 18175
Gerrit-PatchSet: 1
Gerrit-Owner: Yu-Wen Lai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sourabh Goyal
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yu-Wen Lai
Gerrit-Comment-Date: Fri, 28 Jan 2022 12:05:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 28 Jan 2022 10:33:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/10057/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sourabh Goyal Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yu-Wen Lai Gerrit-Comment-Date: Fri, 28 Jan 2022 04:26:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18175 ) Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7768/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 28 Jan 2022 04:05:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata
Yu-Wen Lai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18175 Change subject: IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata .. IMPALA-11093: Fine grained table refreshing doesn't refresh table file metadata If we insert data into an acid partitioned table from Hive, the generated events will be like open_txn -> alter_partition -> commit_txn. Previously we assumed the partition object with the alter_partition event has write id < current write id. However, that is not a valid assumption, the partition object is actually the write id allocated in this transaction. That means in commit_txn event, we will have a partition with write id equals to the write id of cached partition. So we need to modify the '<' condition to '<='. Tests: Manually testing Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java 2 files changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/18175/1 -- To view, visit http://gerrit.cloudera.org:8080/18175 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idabeb522525c45f000ca0992348660fa5a5d4d2d Gerrit-Change-Number: 18175 Gerrit-PatchSet: 1 Gerrit-Owner: Yu-Wen Lai
