Sahil Takiar created IMPALA-10117: ------------------------------------- Summary: Skip calls to FsPermissionCache for blob stores Key: IMPALA-10117 URL: https://issues.apache.org/jira/browse/IMPALA-10117 Project: IMPALA Issue Type: Improvement Reporter: Sahil Takiar
The {{FsPermissionCache}} is described as: {code:java} /** * Simple non-thread-safe cache for resolved file permissions. This allows * pre-caching permissions by listing the status of all files within a directory, * and then using that cache to avoid round trips to the FileSystem for later * queries of those paths. */ {code} I confirmed, and {{FsPermissionCache#precacheChildrenOf}} is actually called for data stored on S3. The issue is that {{FsPermissionCache#getPermissions}} is called inside {{HdfsTable#getAvailableAccessLevel}}, which is skipped for S3. So all the cached metadata is not used. The problem is that {{precacheChildrenOf}} calls {{getFileStatus}} for all files, which results in a bunch of unnecessary metadata operations to S3 + a bunch of cached metadata that is never used. {{precacheChildrenOf}} is actually only invoked in the specific scenario described below: {code} // Only preload permissions if the number of partitions to be added is // large (3x) relative to the number of existing partitions. This covers // two common cases: // // 1) initial load of a table (no existing partition metadata) // 2) ALTER TABLE RECOVER PARTITIONS after creating a table pointing to // an already-existing partition directory tree // // Without this heuristic, we would end up using a "listStatus" call to // potentially fetch a bunch of irrelevant information about existing // partitions when we only want to know about a small number of newly-added // partitions. {code} Regardless, skipping the call to {{precacheChildrenOf}} for blob stores should (1) improve table loading time for S3 backed tables, and (2) decrease catalogd memory requirements when loading a bunch of tables stored on S3. -- This message was sent by Atlassian Jira (v8.3.4#803005)