Michael Brown created IMPALA-7910: ------------------------------------- Summary: COMPUTE STATS does an unnecessary REFRESH after writing to the Metastore Key: IMPALA-7910 URL: https://issues.apache.org/jira/browse/IMPALA-7910 Project: IMPALA Issue Type: Bug Components: Catalog Affects Versions: Impala 2.12.0, Impala 2.11.0, Impala 2.9.0 Reporter: Michael Brown Assignee: Tianyi Wang
COMPUTE STATS and possibly other DDL operations unnecessarily do the equivalent of a REFRESH after writing to the Hive Metastore. This unnecessary operation can be very expensive, so should be avoided. The behavior can be confirmed from the catalogd logs: {code} compute stats functional_parquet.alltypes; +-------------------------------------------+ | summary | +-------------------------------------------+ | Updated 24 partition(s) and 11 column(s). | +-------------------------------------------+ Relevant catalogd.INFO snippet I0413 14:40:24.210749 27295 HdfsTable.java:1263] Incrementally loading table metadata for: functional_parquet.alltypes I0413 14:40:24.242122 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=1: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.244634 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=10: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.247174 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=11: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.249713 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=12: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.252288 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=2: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.254629 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=3: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.256991 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=4: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.259464 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=5: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.262197 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=6: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.264463 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=7: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.266736 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=8: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.269210 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2009/month=9: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.271800 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=1: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.274348 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=10: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.277053 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=11: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.282152 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=12: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.285684 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=2: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.288921 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=3: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.292757 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=4: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.303673 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=5: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.308387 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=6: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.311506 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=7: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.314600 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=8: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.317709 27295 HdfsTable.java:555] Refreshed file metadata for functional_parquet.alltypes Path: hdfs://localhost:20500/test-warehouse/alltypes_parquet/year=2010/month=9: Loaded files: 1 Hidden files: 0 Skipped files: 0 Unknown diskIDs: 0 I0413 14:40:24.317873 27295 HdfsTable.java:1273] Incrementally loaded table metadata for: functional_parquet.alltypes {code} The relevant code starts from CatalogOpExecutor: {code} boolean reloadMetadata = true; catalog_.getLock().writeLock().unlock(); if (tbl instanceof KuduTable && altersKuduTable(params.getAlter_type())) { alterKuduTable(params, response, (KuduTable) tbl, newCatalogVersion); return; } switch (params.getAlter_type()) { ... case UPDATE_STATS: Preconditions.checkState(params.isSetUpdate_stats_params()); Reference<Long> numUpdatedColumns = new Reference<>(0L); alterTableUpdateStats(tbl, params.getUpdate_stats_params(), numUpdatedPartitions, numUpdatedColumns); reloadTableSchema = true; addSummary(response, "Updated " + numUpdatedPartitions.getRef() + " partition(s) and " + numUpdatedColumns.getRef() + " column(s)."); break; .... } if (reloadMetadata) { <-- REFRESH here loadTableMetadata(tbl, newCatalogVersion, reloadFileMetadata, reloadTableSchema, null); addTableToCatalogUpdate(tbl, response.result); } {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)