Hello Quanlong Huang, Jason Fehr, Zoltan Borok-Nagy, Pranav Lodha, Michael
Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23906
to look at the new patch set (#2).
Change subject: IMPALA-13122: Add detailed file metadata statistics to table
loading logs
......................................................................
IMPALA-13122: Add detailed file metadata statistics to table loading logs
This patch enhances table loading logs to include comprehensive file
metadata statistics, making it easier to identify small files issues
and diagnose slow storage performance.
The following statistics are now logged when loading file metadata:
- Number of files and blocks
- File sizes (min/avg/max)
- Total file size
- Modification times (min/max)
- Access times (min/max)
- Number of unique hosts and disks (HDFS/Ozone only)
Example log output:
Loaded file and block metadata for functional.alltypes partitions:
year=2009/month=1, year=2009/month=10, year=2009/month=11, and 21
others. Time taken: 11.749ms. Files: 24, Blocks: 24, Total size:
478.45KB, File sizes (min/avg/max): 18.12KB/19.93KB/20.36KB,
Modification times (min/max): 2025-12-17 01:40:53/2025-12-17 01:40:55,
Access times (min/max): 2026-01-25 22:09:47/2026-01-25 22:09:48,
Hosts: 3, Disks: 3
Testing:
- Added Junit tests to verify statistics collection accuracy
- Added new python end-to-end tests covering various cases
Change-Id: I6f4592f173c047e5064058402f83be6d1f5c9a79
---
M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java
M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
A tests/metadata/test_file_metadata_stats.py
6 files changed, 407 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/23906/2
--
To view, visit http://gerrit.cloudera.org:8080/23906
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6f4592f173c047e5064058402f83be6d1f5c9a79
Gerrit-Change-Number: 23906
Gerrit-PatchSet: 2
Gerrit-Owner: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Arnab Karmakar <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Pranav Lodha <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
