Quanlong Huang created IMPALA-13009: ---------------------------------------
Summary: Possible leak of partition updates when the table has failed DDL and recovered by INVALIDATE METADATA Key: IMPALA-13009 URL: https://issues.apache.org/jira/browse/IMPALA-13009 Project: IMPALA Issue Type: Bug Components: Catalog Reporter: Quanlong Huang Assignee: Quanlong Huang Catalogd might not send partition deletions to the catalog topic in the following scenario: * Partitions of a table are dropped externally outside Impala. * Table dir is also removed on HDFS. * ALTER TABLE RECOVER PARTITIONS failed by FileNotFoundException on the table dir. * A subsequent INVALIDATE METADATA on the same table succeeds to invalidated the table. After the INVALIDATE finishes, catalogd might not send deletions of the dropped partitions to the catalog topic. Then the catalog topic only have the updates of those partitions, no deletions. This will be detected when a coordinator restarts: {noformat} E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog object: Received stale partition in a statestore update: THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, type:TColumnType(types:[TTypeNode(type:SCALAR, scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, stats:TTableStats(num_rows:-1), is_marked_cached:false, hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4, numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4, has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part, partition_name:p=106, hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1, collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT, blockSize:0)) Java exception follows: java.lang.IllegalStateException: Received stale partition in a statestore update: THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, type:TColumnType(types:[TTypeNode(type:SCALAR, scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, stats:TTableStats(num_rows:-1), is_marked_cached:false, hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4, numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4, has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part, partition_name:p=106, hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1, collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT, blockSize:0)) at com.google.common.base.Preconditions.checkState(Preconditions.java:512) at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523) at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334) at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262) at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120) at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565) at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196) {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org