[ https://issues.apache.org/jira/browse/IMPALA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17838407#comment-17838407 ]
Quanlong Huang commented on IMPALA-13009: ----------------------------------------- Found an easier way to reproduce the issue. Start impala with event processing disabled: {code:bash} bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=0 {code} Prepare a partitioned table with some partitions: {code:sql} create table if not exists my_part(i int) partitioned by (p int); insert into my_part partition(p) values (1,1),(2,2),(3,3);{code} Do these in one command so the following statements will be executed immediately: {code:sql} alter table my_part drop partition(p>0); invalidate metadata my_part; show partitions my_part;{code} Restart any impalad can see the stale partitions. > Possible leak of partition updates when the table has failed DDL and > recovered by INVALIDATE METADATA > ----------------------------------------------------------------------------------------------------- > > Key: IMPALA-13009 > URL: https://issues.apache.org/jira/browse/IMPALA-13009 > Project: IMPALA > Issue Type: Bug > Components: Catalog > Reporter: Quanlong Huang > Assignee: Quanlong Huang > Priority: Critical > > Catalogd might not send partition deletions to the catalog topic in the > following scenario: > * Partitions of a table are dropped externally outside Impala. > * Table dir is also removed on HDFS. > * ALTER TABLE RECOVER PARTITIONS failed by FileNotFoundException on the table > dir. > * A subsequent INVALIDATE METADATA on the same table succeeds to invalidate > the table. > After the INVALIDATE finishes, catalogd might not send deletions of the > dropped partitions to the catalog topic. Then the catalog topic only have the > updates of those partitions, no deletions. > This will be detected when a coordinator restarts: > {noformat} > E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog > object: Received stale partition in a statestore update: > THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, > type:TColumnType(types:[TTypeNode(type:SCALAR, > scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, > int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], > location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, > file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 > 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 > 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 > 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 > 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 > 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, > stats:TTableStats(num_rows:-1), is_marked_cached:false, > hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4, > numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4, > has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part, > partition_name:p=106, > hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1, > collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT, > blockSize:0)) > Java exception follows: > java.lang.IllegalStateException: Received stale partition in a statestore > update: > THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, > type:TColumnType(types:[TTypeNode(type:SCALAR, > scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, > int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], > location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, > file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 > 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 > 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 > 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 > 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 > 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, > stats:TTableStats(num_rows:-1), is_marked_cached:false, > hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4, > numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4, > has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part, > partition_name:p=106, > hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1, > collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT, > blockSize:0)) > at > com.google.common.base.Preconditions.checkState(Preconditions.java:512) > at > org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523) > at > org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334) > at > org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262) > at > org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120) > at > org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565) > at > org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196) > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org