Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20673
Change subject: IMPALA-12486: Add catalog metrics for metadata loading ...................................................................... IMPALA-12486: Add catalog metrics for metadata loading This patch adds the following catalog metrics which indicate the load on HDFS for loading file metadata: - catalog.num-file-metadata-loading-threads: The total size of all thread pools used in loading file metadata. - catalog.num-file-metadata-loading-tasks: The total number of unfinished file metadata loading tasks. Each task corresponds to a partition. - catalog.num-tables-loading-file-metadata: The total number of tables that are loading file metadata. Also adds some metrics for metadata loading on all tables. Note that metadata loading of an HDFS table consists of loading HMS metadata and HDFS file metadata, etc. - catalog.num-tables-loading-metadata: The total number of tables that are loading metadata. - catalog.num-tables-async-loading-metadata: The total number of tables that are loading metadata asynchorously. E.g. the initial metadata loading triggered by the first access on a table. - catalog.num-tables-waiting-for-async-loading: The total number of tables that are waiting for asynchorous loading. If this number raises, consider bumping --num_metadata_loading_threads. Three metrics about the catalog cache are also added: - catalog.num-databases - catalog.num-tables - catalog.num-functions Note that the first two are also shown in WebUI of coordinators and we plan to deprecate them. Tests - Launch catalogd locally with load_catalog_in_background=true and verified the metrics. - Add e2e tests in tests/webserver/test_web_pages.py Change-Id: Icef7b123bdcb0f5b8572635eeaacd8294990f9ba --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogObjectCache.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M tests/webserver/test_web_pages.py 19 files changed, 279 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/20673/1 -- To view, visit http://gerrit.cloudera.org:8080/20673 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icef7b123bdcb0f5b8572635eeaacd8294990f9ba Gerrit-Change-Number: 20673 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <huangquanl...@gmail.com>
