[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. IMPALA-4886: Expose table metrics in the catalog web UI. The following changes are included in this commit: * Adds a lightweight framework for registering metrics in the JVM. * Adds table-level metrics and enables these metrics to be exposed through the catalog web UI. * Adds a CatalogUsageMonitor class that monitors and reports the catalog usage in terms of the tables with the highest memory requirements and the tables with the highest number of metadata operations. The catalog usage information is exposed in the /catalog page of the catalog web UI. Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Reviewed-on: http://gerrit.cloudera.org:8080/8529 Reviewed-by: Dimitris Tsirogiannis Tested-by: Impala Public Jenkins --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M common/thrift/CatalogObjects.thrift M common/thrift/Frontend.thrift M common/thrift/JniCatalog.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/common/Metrics.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/main/java/org/apache/impala/util/TopNCache.java A fe/src/test/java/org/apache/impala/util/TestTopNCache.java M tests/webserver/test_web_pages.py M www/catalog.tmpl A www/table_metrics.tmpl 24 files changed, 1,206 insertions(+), 113 deletions(-) Approvals: Dimitris Tsirogiannis: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 7 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 6 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 19 Jan 2018 09:25:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1759/ -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 6 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 19 Jan 2018 05:44:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. Patch Set 6: Code-Review+2 Rebase and retry gvo due to failed flaky test (IMPALA-6092). -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 6 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 19 Jan 2018 05:44:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1755/ -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 5 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 19 Jan 2018 03:03:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1755/ -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 5 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 18 Jan 2018 23:28:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1754/ -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 5 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 18 Jan 2018 23:26:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. Patch Set 5: Code-Review+2 Rebase -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 5 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 18 Jan 2018 23:26:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Hello Bharath Vissapragada, Philip Zeyliger, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8529 to look at the new patch set (#5). Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. IMPALA-4886: Expose table metrics in the catalog web UI. The following changes are included in this commit: * Adds a lightweight framework for registering metrics in the JVM. * Adds table-level metrics and enables these metrics to be exposed through the catalog web UI. * Adds a CatalogUsageMonitor class that monitors and reports the catalog usage in terms of the tables with the highest memory requirements and the tables with the highest number of metadata operations. The catalog usage information is exposed in the /catalog page of the catalog web UI. Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M common/thrift/CatalogObjects.thrift M common/thrift/Frontend.thrift M common/thrift/JniCatalog.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java A fe/src/main/java/org/apache/impala/catalog/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java A fe/src/main/java/org/apache/impala/common/Metrics.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java A fe/src/main/java/org/apache/impala/util/TopNCache.java A fe/src/test/java/org/apache/impala/util/TestTopNCache.java M tests/webserver/test_web_pages.py M www/catalog.tmpl A www/table_metrics.tmpl 24 files changed, 1,206 insertions(+), 113 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/8529/5 -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 5 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-4886: Expose table metrics in the catalog web UI.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/8529 ) Change subject: IMPALA-4886: Expose table metrics in the catalog web UI. .. Patch Set 4: Code-Review+1 (5 comments) Please raise jira for the remaining to-do tasks (around metrics/observability) when you get a chance. They'd make nice ramp-up/newbie tasks. http://gerrit.cloudera.org:8080/#/c/8529/2/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/8529/2/be/src/catalog/catalog-server.cc@492 PS2, Line 492: // TODO: Enable json view of table metrics > The URL for table metrics already has a "?", so I am not sure how the ?json Yea could be wrong, I haven't tried it myself, sorry for the wrong pointer. http://gerrit.cloudera.org:8080/#/c/8529/3/be/src/catalog/catalog-server.cc File be/src/catalog/catalog-server.cc: http://gerrit.cloudera.org:8080/#/c/8529/3/be/src/catalog/catalog-server.cc@359 PS3, Line 359: GetCatalogUsage(document); > Why abandon loading the rest of the /catalog page if the catalog usage call My bad, I thought a single error breaks the whole page. http://gerrit.cloudera.org:8080/#/c/8529/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/8529/3/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1743 PS3, Line 1743: stats.numBlocks += tHdfsPartition.getNum_blocks(); : stats.numFiles += : tHdfsPartition.isSetFile_desc() ? tHdfsPartition.getFile_desc().size() : 0; : stats.totalFileBytes += tHdfsPartition.getTotal_file_size_bytes(); > includeFileDesc is always true when this is called in the catalogd. For the Agree. http://gerrit.cloudera.org:8080/#/c/8529/3/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/8529/3/fe/src/main/java/org/apache/impala/catalog/Table.java@151 PS3, Line 151: metrics_.addTimer(ALTER_DURATION_METRIC); > Hm, not sure if we need to add this check to every public function of table Fine by me. http://gerrit.cloudera.org:8080/#/c/8529/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/8529/3/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@362 PS3, Line 362: final Timer.Context context > That may not be a good idea since we will be accessing a table field in an Ah nice catch. -- To view, visit http://gerrit.cloudera.org:8080/8529 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37d407979e6d3b1a444b6b6265900b148facde9e Gerrit-Change-Number: 8529 Gerrit-PatchSet: 4 Gerrit-Owner: Dimitris Tsirogiannis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Sat, 16 Dec 2017 05:42:42 + Gerrit-HasComments: Yes
