Quanlong Huang created IMPALA-12929: ---------------------------------------
Summary: Skip loading HDFS permissions in local-catalog mode Key: IMPALA-12929 URL: https://issues.apache.org/jira/browse/IMPALA-12929 Project: IMPALA Issue Type: Bug Components: Catalog Reporter: Quanlong Huang Assignee: Quanlong Huang As IMPALA-7539 is unresolved, HDFS file/dir permissions are not used at all in local catalog mode. However, in catalogd, we still load them (in single thread for a table!) which could dominant the table loading time when there are lots of partitions. Here is an example timeline for a REFRESH statement on an unloaded table: {noformat} Catalog Server Operation: 2s300ms - Got catalog version read lock: 26.407us (26.407us) - Start loading table: 314.663us (288.256us) - Got Metastore client: 629.599us (314.936us) - Fetched table from Metastore: 7.248ms (6.618ms) - Loaded table schema: 27.947ms (20.699ms) - Preloaded permissions cache for 1824 partitions: 1s514ms (1s486ms) - Got access level: 1s514ms (588.314us) - Created partition builders: 2s103ms (588.270ms) - Start loading file metadata: 2s103ms (49.760us) - Loaded file metadata for 1824 partitions: 2s282ms (179.839ms) - Async loaded table: 2s289ms (6.931ms) - Loaded table from scratch: 2s289ms (72.038us) - Got table read lock: 2s289ms (2.289us) - Finished resetMetadata request: 2s300ms (10.188ms){noformat} The majority of the time is spent in "Preloaded permissions cache for 1824 partitions". Currently, catalogd can skip loading HDFS permissions in local catalog mode until IMPALA-7539 is unresolved. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org