[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Quanlong Huang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. IMPALA-9118: Show catalog operation details in catalogd webUI This patch extends the /operations page in catalogd WebUI to show the in-flight and finished catalog operations. The following fields are shown for each operation: - Thread ID - Query ID - Client IP - Coordinator - Type - Target - User - Start Time - End Time (only shown for finished operations) - Duration - Status - Details Finished operation records are currently kept in memory and limited by the catalog_operation_log_size flag (defaults to 100). Tests: - Add e2e test in custom_cluster/test_web_pages.py - Manually verify the web pages when running a GVO job Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/service/client-request-state.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/custom_cluster/test_web_pages.py M www/catalog_operations.tmpl 18 files changed, 495 insertions(+), 102 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/20428/2 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20428 to look at the new patch set (#3). Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. IMPALA-9118: Show catalog operation details in catalogd webUI This patch extends the /operations page in catalogd WebUI to show the in-flight and finished catalog operations. The following fields are shown for each operation: - Thread ID - Query ID - Client IP - Coordinator - Type - Target - User - Start Time - End Time (only shown for finished operations) - Duration - Status - Details Finished operation records are currently kept in memory and limited by the catalog_operation_log_size flag (defaults to 100). To collect the above fields, this patch extends TCatalogServiceRequestHeader to contain the coordinator hostname. Also fix some catalog RPCs that didn't fill these fields. Tests: - Add e2e test in custom_cluster/test_web_pages.py - Manually verify the web pages when running a GVO job Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/service/client-request-state.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/custom_cluster/test_web_pages.py M www/catalog_operations.tmpl 18 files changed, 492 insertions(+), 101 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/20428/3 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13867/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 29 Aug 2023 08:09:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13868/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 29 Aug 2023 08:22:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20428 to look at the new patch set (#4). Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. IMPALA-9118: Show catalog operation details in catalogd webUI This patch extends the /operations page in catalogd WebUI to show the in-flight and finished catalog operations. The following fields are shown for each operation: - Thread ID - Query ID - Client IP - Coordinator - Type - Target - User - Start Time - End Time (only shown for finished operations) - Duration - Status - Details Finished operation records are currently kept in memory and limited by the catalog_operation_log_size flag (defaults to 100). To collect the above fields, this patch extends TCatalogServiceRequestHeader to contain the coordinator hostname. Also fix some catalog RPCs that didn't fill these fields. Tests: - Add e2e test in custom_cluster/test_web_pages.py - Manually verify the web pages when running a GVO job Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/service/client-request-state.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java D fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/custom_cluster/test_web_pages.py M www/catalog_operations.tmpl 20 files changed, 580 insertions(+), 194 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/20428/4 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/13983/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 12 Sep 2023 07:39:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 4: (2 comments) Hi Quanlong, this change looks great, left a nit and a comment. http://gerrit.cloudera.org:8080/#/c/20428/4/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/20428/4/be/src/catalog/catalog-server.cc@152 PS4, Line 152: operations nit: operation log records http://gerrit.cloudera.org:8080/#/c/20428/4/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java: http://gerrit.cloudera.org:8080/#/c/20428/4/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java@119 PS4, Line 119: if (ddlRequest.isSetHeader()) Users might find it confusing if the DDL Summary would show a different number of in-progress DDL requests that are displayed on the in-progress list. But I am not sure if this actually can be empty. -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 27 Sep 2023 13:54:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Hello Tamas Mate, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20428 to look at the new patch set (#5). Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. IMPALA-9118: Show catalog operation details in catalogd webUI This patch extends the /operations page in catalogd WebUI to show the in-flight and finished catalog operations. The following fields are shown for each operation: - Thread ID - Query ID - Client IP - Coordinator - Type - Target - User - Start Time - End Time (only shown for finished operations) - Duration - Status - Details Finished operation records are currently kept in memory and limited by the catalog_operation_log_size flag (defaults to 100). To collect the above fields, this patch extends TCatalogServiceRequestHeader to contain the coordinator hostname. Also fix some catalog RPCs that didn't fill these fields. Tests: - Add e2e test in custom_cluster/test_web_pages.py - Manually verify the web pages when running a GVO job Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java D fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/custom_cluster/test_web_pages.py M www/catalog_operations.tmpl 22 files changed, 592 insertions(+), 198 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/20428/5 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/20428/4/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/20428/4/be/src/catalog/catalog-server.cc@152 PS4, Line 152: operation > nit: operation log records Done http://gerrit.cloudera.org:8080/#/c/20428/4/be/src/exec/catalog-op-executor.cc File be/src/exec/catalog-op-executor.cc: http://gerrit.cloudera.org:8080/#/c/20428/4/be/src/exec/catalog-op-executor.cc@163 PS4, Line 163: Missing query id here. Fixed in PS5. http://gerrit.cloudera.org:8080/#/c/20428/4/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java: http://gerrit.cloudera.org:8080/#/c/20428/4/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java@119 PS4, Line 119: if (ddlRequest.ddl_type == TD > Users might find it confusing if the DDL Summary would show a different num I think missing the header here should be considered as a bug. But to be compatible with old version impalads during rolling upgrade, it'd be better to keep this. -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 12 Oct 2023 06:02:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14179/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Thu, 12 Oct 2023 06:20:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 5: (5 comments) http://gerrit.cloudera.org:8080/#/c/20428/5/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/20428/5/be/src/catalog/catalog-server.cc@953 PS5, Line 953: catalog_op.finish_time_ms > 0 ? : catalog_op.finish_time_ms : UnixMillis() end_time_ms will be set in line 956 and 961. It's duplicated here. http://gerrit.cloudera.org:8080/#/c/20428/5/common/thrift/JniCatalog.thrift File common/thrift/JniCatalog.thrift: http://gerrit.cloudera.org:8080/#/c/20428/5/common/thrift/JniCatalog.thrift@879 PS5, Line 879: required string details Is 'details' a required field? http://gerrit.cloudera.org:8080/#/c/20428/5/common/thrift/JniCatalog.thrift@886 PS5, Line 886: optional list in_flight_catalog_operations; nit: add comments for what's the order of lists of in-flight operations and finished operations. http://gerrit.cloudera.org:8080/#/c/20428/5/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java: http://gerrit.cloudera.org:8080/#/c/20428/5/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java@110 PS5, Line 110: catalogOperationLogSize If catalogOperationLogSize is negative, log size is unlimited http://gerrit.cloudera.org:8080/#/c/20428/5/tests/custom_cluster/test_web_pages.py File tests/custom_cluster/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/20428/5/tests/custom_cluster/test_web_pages.py@261 PS5, Line 261: --catalog_operation_log_size=2 Could you add test case with catalog_operation_log_size as -1? -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 16 Oct 2023 19:07:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/20428/5/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java File fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java: http://gerrit.cloudera.org:8080/#/c/20428/5/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java@110 PS5, Line 110: catalogOperationLogSize > If catalogOperationLogSize is negative, log size is unlimited Negative value is handled BackendConfig.java. Maybe add a DCHECK -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 16 Oct 2023 21:38:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Hello Tamas Mate, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20428 to look at the new patch set (#6). Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. IMPALA-9118: Show catalog operation details in catalogd webUI This patch extends the /operations page in catalogd WebUI to show the in-flight and finished catalog operations. The following fields are shown for each operation: - Thread ID - Query ID - Client IP - Coordinator - Type - Target - User - Start Time - End Time (only shown for finished operations) - Duration - Status - Details Finished operation records are currently kept in memory and limited by the catalog_operation_log_size flag (defaults to 100). To collect the above fields, this patch extends TCatalogServiceRequestHeader to contain the coordinator hostname. Also fix some catalog RPCs that didn't fill these fields. Tests: - Add e2e test in custom_cluster/test_web_pages.py - Manually verify the web pages when running a GVO job Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java D fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/custom_cluster/test_web_pages.py M www/catalog_operations.tmpl 23 files changed, 635 insertions(+), 199 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/20428/6 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/20428 )
Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI
..
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20428/5/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:
http://gerrit.cloudera.org:8080/#/c/20428/5/be/src/catalog/catalog-server.cc@953
PS5, Line 953:
: if (catalog_op.finish_time_ms > 0) {
> end_time_ms will be set in line 956 and 961. It's duplicated here.
Oops, removed them
http://gerrit.cloudera.org:8080/#/c/20428/5/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/20428/5/common/thrift/JniCatalog.thrift@879
PS5, Line 879: required string details
> Is 'details' a required field?
I think either way is ok. We currently show the query options in this 'details'
field so it's always non-empty.
http://gerrit.cloudera.org:8080/#/c/20428/5/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java
File
fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java:
http://gerrit.cloudera.org:8080/#/c/20428/5/fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java@110
PS5, Line 110:
> Negative value is handled BackendConfig.java. Maybe add a DCHECK
Done
http://gerrit.cloudera.org:8080/#/c/20428/5/tests/custom_cluster/test_web_pages.py
File tests/custom_cluster/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/20428/5/tests/custom_cluster/test_web_pages.py@261
PS5, Line 261: --catalog_operation_log_size=2
> Could you add test case with catalog_operation_log_size as -1?
Done
--
To view, visit http://gerrit.cloudera.org:8080/20428
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa
Gerrit-Change-Number: 20428
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tamas Mate
Gerrit-Reviewer: Wenzhe Zhou
Gerrit-Comment-Date: Mon, 23 Oct 2023 07:17:43 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14222/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 23 Oct 2023 07:48:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 23 Oct 2023 21:13:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Hello Tamas Mate, Wenzhe Zhou, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/20428 to look at the new patch set (#7). Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. IMPALA-9118: Show catalog operation details in catalogd webUI This patch extends the /operations page in catalogd WebUI to show the in-flight and finished catalog operations. The following fields are shown for each operation: - Thread ID - Query ID - Client IP - Coordinator - Type - Target - User - Start Time - End Time (only shown for finished operations) - Duration - Status - Details Finished operation records are currently kept in memory and limited by the catalog_operation_log_size flag (defaults to 100). To collect the above fields, this patch extends TCatalogServiceRequestHeader to contain the coordinator hostname. Also fix some catalog RPCs that didn't fill these fields. Tests: - Add e2e test in custom_cluster/test_web_pages.py - Manually verify the web pages when running a GVO job Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java D fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/custom_cluster/test_web_pages.py M www/catalog_operations.tmpl 23 files changed, 635 insertions(+), 199 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/20428/7 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/14231/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Tue, 24 Oct 2023 05:42:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 7: Code-Review+2 Thanks Quanlong, LGTM! -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 Oct 2023 09:05:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 7 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 Oct 2023 18:48:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 Oct 2023 18:49:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9843/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 Oct 2023 18:49:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. IMPALA-9118: Show catalog operation details in catalogd webUI This patch extends the /operations page in catalogd WebUI to show the in-flight and finished catalog operations. The following fields are shown for each operation: - Thread ID - Query ID - Client IP - Coordinator - Type - Target - User - Start Time - End Time (only shown for finished operations) - Duration - Status - Details Finished operation records are currently kept in memory and limited by the catalog_operation_log_size flag (defaults to 100). To collect the above fields, this patch extends TCatalogServiceRequestHeader to contain the coordinator hostname. Also fix some catalog RPCs that didn't fill these fields. Tests: - Add e2e test in custom_cluster/test_web_pages.py - Manually verify the web pages when running a GVO job Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Reviewed-on: http://gerrit.cloudera.org:8080/20428 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/exec/catalog-op-executor.cc M be/src/exec/catalog-op-executor.h M be/src/service/client-request-state.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogFinalizeDmlCounter.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogMonitor.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationCounter.java D fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationMetrics.java A fe/src/main/java/org/apache/impala/catalog/monitor/CatalogOperationTracker.java M fe/src/main/java/org/apache/impala/catalog/monitor/CatalogResetMetadataCounter.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/custom_cluster/test_web_pages.py M www/catalog_operations.tmpl 23 files changed, 635 insertions(+), 199 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 9 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9118: Show catalog operation details in catalogd webUI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20428 ) Change subject: IMPALA-9118: Show catalog operation details in catalogd webUI .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/20428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3cf3f0da2be2be79e546762a8083d4de338ff6aa Gerrit-Change-Number: 20428 Gerrit-PatchSet: 8 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 25 Oct 2023 23:14:39 + Gerrit-HasComments: No
