[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/14349/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 5 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 07 Nov 2023 03:43:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20614 to look at the new patch set (#5). Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. IMPALA-12513: Reset metadata when the CatalogD become active When switching active CatalogD, the loaded metadata in the candidate CatalogD may not be the latest. We should reset the metadata to ensure that there are no metadata inconsistency exceptions after failover. Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 52 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/5 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 5 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
ttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 4: > Patch Set 4: > > > > Patch Set 4: > > > > > > (1 comment) > > > > I understand. Indeed, in order to pursue performance, the standby > > catalog is likely to enable background metadata loading, despite > > the risk of metadata expiration. > > Could you add a flag to turn on resetting metadata when CatalogD becomes > active? Okay, I am willing to do it! -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 4 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Thu, 02 Nov 2023 15:43:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 4: > > Patch Set 4: > > > > (1 comment) > > I understand. Indeed, in order to pursue performance, the standby > catalog is likely to enable background metadata loading, despite > the risk of metadata expiration. Could you add a flag to turn on resetting metadata when CatalogD becomes active? -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 4 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Wed, 01 Nov 2023 21:14:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
ttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 4: > Patch Set 4: > > (1 comment) I understand. Indeed, in order to pursue performance, the standby catalog is likely to enable background metadata loading, despite the risk of metadata expiration. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 4 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Wed, 01 Nov 2023 03:49:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/20614/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20614/2//COMMIT_MSG@9 PS2, Line 9: the loaded metadata in the candidate : CatalogD may not be the latest. > There are serveral causes why the metadata loaded in the standby catalogd l For "The catalogd enables load metadata in the background", we can enable "load_catalog_in_background" for standby catalogd, though it increase the network traffic. For "Newly created user-defined functions will not be recognized after failover", this issue is tracked by IMPALA-10273 (Support function event). Since most of the users don't turn on load_catalog_in_background and don't create UDFs in ETL jobs, I prefer to add a flag reset_catalog_when_active. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 4 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Wed, 01 Nov 2023 03:32:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14243/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 4 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Wed, 25 Oct 2023 09:37:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
ttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 4: (1 comment) > Patch Set 2: > > (1 comment) http://gerrit.cloudera.org:8080/#/c/20614/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20614/2//COMMIT_MSG@9 PS2, Line 9: the loaded metadata in the candidate : CatalogD may not be the latest. > What are the causes for the loaded metadata in standby catalogd legging beh There are serveral causes why the metadata loaded in the standby catalogd legging behind, such as: 1?The catalogd enables load metadata in the background. 2?After multiple statestored restarts, the catalog's HA state changes as follows: active > stanbdy > active. 3?Newly created user-defined functions will not be recognized after failover. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 4 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Wed, 25 Oct 2023 09:07:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20614 to look at the new patch set (#4). Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. IMPALA-12513: Reset metadata when the CatalogD become active When switching active CatalogD, the loaded metadata in the candidate CatalogD may not be the latest. We should reset the metadata to ensure that there are no metadata inconsistency exceptions after failover. Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 46 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/4 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 4 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
ttz has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 3: > Patch Set 2: > > (1 comment) There are serveral causes why the metadata loaded in the standby catalogd legging behind, such as: 1、The catalogd enables load metadata in the background. 2、After multiple statestored restarts, the catalog's HA state changes as follows: active > stanbdy > active. 3、Newly created user-defined functions will not be recognized after failover. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 3 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Wed, 25 Oct 2023 09:04:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14242/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 3 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Wed, 25 Oct 2023 09:03:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/20614/3/tests/custom_cluster/test_catalogd_ha.py File tests/custom_cluster/test_catalogd_ha.py: http://gerrit.cloudera.org:8080/#/c/20614/3/tests/custom_cluster/test_catalogd_ha.py@426 PS3, Line 426: flake8: W292 no newline at end of file -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 3 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Wed, 25 Oct 2023 08:33:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20614 to look at the new patch set (#3). Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. IMPALA-12513: Reset metadata when the CatalogD become active When switching active CatalogD, the loaded metadata in the candidate CatalogD may not be the latest. We should reset the metadata to ensure that there are no metadata inconsistency exceptions after failover. Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc M tests/custom_cluster/test_catalogd_ha.py 2 files changed, 46 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/3 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 3 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/20614/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20614/2//COMMIT_MSG@9 PS2, Line 9: the loaded metadata in the candidate : CatalogD may not be the latest. What are the causes for the loaded metadata in standby catalogd legging behind? How about adding a flag to keep metadata ready on standby catalogd if the flag is enabled, otherwise reset the metadata when it becomes active. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 2 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 24 Oct 2023 19:33:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20614 to look at the new patch set (#2). Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. IMPALA-12513: Reset metadata when the CatalogD become active When switching active CatalogD, the loaded metadata in the candidate CatalogD may not be the latest. We should reset the metadata to ensure that there are no metadata inconsistency exceptions after failover. Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/2 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 2 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com>
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20614 ) Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14234/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 1 Gerrit-Owner: ttz <2433038...@qq.com> Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Reviewer: ttz <2433038...@qq.com> Gerrit-Comment-Date: Tue, 24 Oct 2023 13:06:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12513: Reset metadata when the CatalogD become active
ttz has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20614 Change subject: IMPALA-12513: Reset metadata when the CatalogD become active .. IMPALA-12513: Reset metadata when the CatalogD become active Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 --- M be/src/catalog/catalog-server.cc 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/20614/1 -- To view, visit http://gerrit.cloudera.org:8080/20614 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2b54f36f96e7901499105e51790d8e2300d7fea9 Gerrit-Change-Number: 20614 Gerrit-PatchSet: 1 Gerrit-Owner: ttz <2433038...@qq.com>