Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20673 )
Change subject: IMPALA-12486: Add catalog metrics for metadata loading ...................................................................... Patch Set 3: (7 comments) http://gerrit.cloudera.org:8080/#/c/20673/2/be/src/catalog/catalog-server.h File be/src/catalog/catalog-server.h: http://gerrit.cloudera.org:8080/#/c/20673/2/be/src/catalog/catalog-server.h@172 PS2, Line 172: asynchronousl > Spelling: asynchronously Done http://gerrit.cloudera.org:8080/#/c/20673/2/common/thrift/JniCatalog.thrift File common/thrift/JniCatalog.thrift: http://gerrit.cloudera.org:8080/#/c/20673/2/common/thrift/JniCatalog.thrift@1041 PS2, Line 1041: 5: optional i32 catalog_num_file_metadata_loading_tasks > Why is this not i32 as well? I was afraid of large number of partitions being loading. But it seems i32 should be enough. http://gerrit.cloudera.org:8080/#/c/20673/2/common/thrift/JniCatalog.thrift@1054 PS2, Line 1054: 14: optional i32 catalog_num_hms_clients_in_use > Nit: "catalog_num_hms_clients_in_use" Done http://gerrit.cloudera.org:8080/#/c/20673/2/common/thrift/metrics.json File common/thrift/metrics.json: http://gerrit.cloudera.org:8080/#/c/20673/2/common/thrift/metrics.json@350 PS2, Line 350: catalog- > New metrics have "catalog." prefix, but label says "Catalog Server". Should We also have some existing metrics that have "catalog." prefix: "catalog.num-databases" "catalog.num-tables" "catalog.partial-fetch-rpc.queue-len" To be clear, we can use "catalog-server" prefix for catalogd-only metrics, and use "catalog" prefix for metrics that are both in catalogd and coordinator. Modified the metric names: catalog.num-file-metadata-loading-threads -> catalog-server.metadata.file.num-loading-threads catalog.num-file-metadata-loading-tasks -> catalog-server.metadata.file.num-loading-tasks catalog.num-tables-loading-file-metadata -> catalog-server.metadata.table.num-loading-file-metadata catalog.num-tables-loading-metadata -> catalog-server.metadata.table.num-loading-metadata catalog.num-tables-async-loading-metadata -> catalog-server.metadata.table.async-loading.num-in-progress catalog.num-tables-waiting-for-async-loading -> catalog-server.metadata.table.async-loading.queue-len catalog.num-hms-clients-idle -> catalog.hms-client-pool.num-idle catalog.num-hms-clients-in-use -> catalog.hms-client-pool.num-in-use Note1: Coordinator also has HMS clients (e.g. to open/close Hive transactions). Note2: "catalog.partial-fetch-rpc.queue-len" should have prefix "catalog-server" but it's an existing metric key so didn't change it. http://gerrit.cloudera.org:8080/#/c/20673/2/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/20673/2/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java@110 PS2, Line 110: Preconditions.checkArgument(recursive_, "ACID tables must be listed recursively"); > Add comment explaining when this is decremented (in load() I think, which m Done. Added variable comment. http://gerrit.cloudera.org:8080/#/c/20673/2/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java@236 PS2, Line 236: } > Maybe neater if this is in a finally block? Good point! I realized L170 could also throws exception. Added a finally block outside the whole method. http://gerrit.cloudera.org:8080/#/c/20673/2/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java File fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java: http://gerrit.cloudera.org:8080/#/c/20673/2/fe/src/main/java/org/apache/impala/catalog/MetaStoreClientPool.java@54 PS2, Line 54: numHmsClientsInUse_ > Nit: numHmsClientsInUse_ Done -- To view, visit http://gerrit.cloudera.org:8080/20673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icef7b123bdcb0f5b8572635eeaacd8294990f9ba Gerrit-Change-Number: 20673 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Comment-Date: Wed, 29 Nov 2023 08:14:34 +0000 Gerrit-HasComments: Yes
