[jira] [Commented] (IMPALA-13009) Potential leak of partition deletions in the catalog topic
[ https://issues.apache.org/jira/browse/IMPALA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844278#comment-17844278 ] ASF subversion and git services commented on IMPALA-13009: -- Commit 5d32919f46117213249c60574f77e3f9bb66ed90 in impala's branch refs/heads/branch-4.4.0 from stiga-huang [ https://gitbox.apache.org/repos/asf?p=impala.git;h=5d32919f4 ] IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions *Background* Since IMPALA-3127, catalogd sends incremental partition updates based on the last sent table snapshot ('maxSentPartitionId_' to be specific). Dropped partitions since the last catalog update are tracked in 'droppedPartitions_' of HdfsTable. When catalogd collects the next catalog update, they will be collected. HdfsTable then clears the set. See details in CatalogServiceCatalog#addHdfsPartitionsToCatalogDelta(). If an HdfsTable is invalidated, it's replaced with an IncompleteTable which doesn't track any partitions. The HdfsTable object is then added to the deleteLog so catalogd can send deletion updates for all its partitions. The same if the HdfsTable is dropped. However, the previously dropped partitions are not collected in this case, which results in a leak in the catalog topic if the partition name is not reused anymore. Note that in the catalog topic, the key of a partition update consists of the table name and the partition name. So if the partition is added back to the table, the topic key will be reused then resolves the leak. The leak will be observed when a coordinator restarts. In the initial catalog update sent from statestore, coordinator will find some partition updates that are not referenced by the HdfsTable (assuming the table is used again after the INVALIDATE). Then a Precondition check fails and the table is not added to the coordinator. *Overview of the patch* This patch fixes the leak by also collecting the dropped partitions when adding the HdfsTable to the deleteLog. A new field, dropped_partitions, is added in THdfsTable to collect them. It's only used when catalogd collects catalog updates. Removes the Precondition check in coordinator and just reports the stale partitions since IMPALA-12831 could also introduce them. Also adds a log line in CatalogOpExecutor.alterTableDropPartition() to show the dropped partition names for better diagnostics. Tests - Added e2e tests Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21 Reviewed-on: http://gerrit.cloudera.org:8080/21326 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins (cherry picked from commit ee21427d26620b40d38c706b4944d2831f84f6f5) > Potential leak of partition deletions in the catalog topic > -- > > Key: IMPALA-13009 > URL: https://issues.apache.org/jira/browse/IMPALA-13009 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 4.0.0, Impala 4.1.0, Impala 4.2.0, Impala 4.1.1, > Impala 4.1.2, Impala 4.3.0 >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Critical > Fix For: Impala 4.5.0 > > > Catalogd might not send partition deletions to the catalog topic in the > following scenario: > * Some partitions of a table are dropped. > * The HdfsTable object is removed sequentially before catalogd collects the > dropped partitions. > In such case, catalogd loses track of the dropped partitions so their updates > keep existing in the catalog topic, until the partition names are reused > again. > Note that the HdfsTable object can be removed by commands like DropTable or > INVALIDATE. > The leaked partitions will be detected when a coordinator restarts. An > IllegalStateException complaining stale partitions will be reported, causing > the table not being added to the catalog cache of coordinator. > {noformat} > E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog > object: Received stale partition in a statestore update: > THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, > type:TColumnType(types:[TTypeNode(type:SCALAR, > scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, > int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], > location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, > file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 > 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 > 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 > 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 > 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 > 78 74 00 00 0C 00 14 00 00 00 0C 00...)
[jira] [Commented] (IMPALA-13009) Potential leak of partition deletions in the catalog topic
[ https://issues.apache.org/jira/browse/IMPALA-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844099#comment-17844099 ] ASF subversion and git services commented on IMPALA-13009: -- Commit ee21427d26620b40d38c706b4944d2831f84f6f5 in impala's branch refs/heads/master from stiga-huang [ https://gitbox.apache.org/repos/asf?p=impala.git;h=ee21427d2 ] IMPALA-13009: Fix catalogd not sending deletion updates for some dropped partitions *Background* Since IMPALA-3127, catalogd sends incremental partition updates based on the last sent table snapshot ('maxSentPartitionId_' to be specific). Dropped partitions since the last catalog update are tracked in 'droppedPartitions_' of HdfsTable. When catalogd collects the next catalog update, they will be collected. HdfsTable then clears the set. See details in CatalogServiceCatalog#addHdfsPartitionsToCatalogDelta(). If an HdfsTable is invalidated, it's replaced with an IncompleteTable which doesn't track any partitions. The HdfsTable object is then added to the deleteLog so catalogd can send deletion updates for all its partitions. The same if the HdfsTable is dropped. However, the previously dropped partitions are not collected in this case, which results in a leak in the catalog topic if the partition name is not reused anymore. Note that in the catalog topic, the key of a partition update consists of the table name and the partition name. So if the partition is added back to the table, the topic key will be reused then resolves the leak. The leak will be observed when a coordinator restarts. In the initial catalog update sent from statestore, coordinator will find some partition updates that are not referenced by the HdfsTable (assuming the table is used again after the INVALIDATE). Then a Precondition check fails and the table is not added to the coordinator. *Overview of the patch* This patch fixes the leak by also collecting the dropped partitions when adding the HdfsTable to the deleteLog. A new field, dropped_partitions, is added in THdfsTable to collect them. It's only used when catalogd collects catalog updates. Removes the Precondition check in coordinator and just reports the stale partitions since IMPALA-12831 could also introduce them. Also adds a log line in CatalogOpExecutor.alterTableDropPartition() to show the dropped partition names for better diagnostics. Tests - Added e2e tests Change-Id: I12a68158dca18ee48c9564ea16b7484c9f5b5d21 Reviewed-on: http://gerrit.cloudera.org:8080/21326 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Potential leak of partition deletions in the catalog topic > -- > > Key: IMPALA-13009 > URL: https://issues.apache.org/jira/browse/IMPALA-13009 > Project: IMPALA > Issue Type: Bug > Components: Catalog >Affects Versions: Impala 4.0.0, Impala 4.1.0, Impala 4.2.0, Impala 4.1.1, > Impala 4.1.2, Impala 4.3.0 >Reporter: Quanlong Huang >Assignee: Quanlong Huang >Priority: Critical > > Catalogd might not send partition deletions to the catalog topic in the > following scenario: > * Some partitions of a table are dropped. > * The HdfsTable object is removed sequentially before catalogd collects the > dropped partitions. > In such case, catalogd loses track of the dropped partitions so their updates > keep existing in the catalog topic, until the partition names are reused > again. > Note that the HdfsTable object can be removed by commands like DropTable or > INVALIDATE. > The leaked partitions will be detected when a coordinator restarts. An > IllegalStateException complaining stale partitions will be reported, causing > the table not being added to the catalog cache of coordinator. > {noformat} > E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog > object: Received stale partition in a statestore update: > THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, > type:TColumnType(types:[TTypeNode(type:SCALAR, > scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, > int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], > location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, > file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 > 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 > 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 > 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 > 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 > 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, > stats:TTableStats(num_rows:-1), is_marked_cached:false, > hms_parameters:{transie