Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9370
Change subject: IMPALA-6530: Track time spent opening HDFS file handles ...................................................................... IMPALA-6530: Track time spent opening HDFS file handles When the HDFS NameNode is overloaded, opening file handles can be a significant source of query execution time. Currently, there is no statistic to track this time at the HDFS scan node level. This introduces a statistic "TotalRawHdfsOpenFileTime(*)" to track the time spent in HdfsOpenFile(). Here is an example of this statistic populated for the query "select * from functional_parquet.widetable_1000_cols", which is dominated by file handle opening time: - CachedFileHandlesHitCount: 0 (0) - CachedFileHandlesMissCount: 1.00K (1001) ... - ScannerThreadsTotalWallClockTime: 980.432ms - MaterializeTupleTime(*): 1.759ms - ScannerThreadsSysTime: 4.000ms - ScannerThreadsUserTime: 56.000ms - TotalRawHdfsOpenFileTime(*): 894.285ms - TotalRawHdfsReadTime(*): 25.188ms To make the TotalRawHdfsReadTime mutually exclusive from the TotalRawHdfsOpenFileTime, the timer tracking for the read timer moves from DiskIoMgr::ReadRange() to inside the ScanRange::Read() function. This allows it to exclude the portion of ScanRange::Read() that is getting a file handle from the file handle cache. Change-Id: Ia560af2d9b12f158e8811900a7b9d98f8e760858 --- M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/request-context.h M be/src/runtime/io/scan-range.cc 7 files changed, 18 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/9370/1 -- To view, visit http://gerrit.cloudera.org:8080/9370 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia560af2d9b12f158e8811900a7b9d98f8e760858 Gerrit-Change-Number: 9370 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>