Hi Yaobin, LGTM, and I've already merged that into master branch, maybe you can check all operations and find more operations that still fail when any dn is done, and then try to fix them all.
Best regards, ----------------------- Yuan Tian On Thu, Jul 9, 2026 at 4:00 PM alpass <[email protected]> wrote: > Hi everyone, > > > Following the previous discussion, I have prepared an initial > implementation for the lease-based self-fencing framework（pr : > https://github.com/apache/iotdb/pull/18127）. The main changes are > summarized below. > > > 1. DataNode-side metadata lease and self-fencing > > > A new MetadataLeaseManager is introduced on the DataNode side. > > > It tracks the latest successful ConfigNode heartbeat using a monotonic > clock. If the DataNode does not receive a heartbeat within > metadata_lease_fence_ms, it enters a fenced state. > > > When fenced, the DataNode clears unsafe metadata caches and fails closed > for related metadata cache access, so a partitioned DataNode will not > continue serving stale metadata. > > > The protected cache paths currently include: > > > - Partition cache > - Table metadata cache > - Tree device schema cache > > > A recovery path is also added. Once the DataNode receives a ConfigNode > heartbeat again, it clears stale cache state and reloads metadata where > needed before returning to normal service. > > > 2. ConfigNode-side contact tracking and broadcast verdict > > > On the ConfigNode side, the implementation adds: > > > - DataNodeContactTracker > - ClusterCachePropagator > - MetadataBroadcastVerdict > > > DataNodeContactTracker records the ConfigNode-side timestamp of the last > successful heartbeat response from each DataNode. > > > ClusterCachePropagator uses this information when broadcasting metadata > cache updates. If a DataNode does not acknowledge the broadcast, the > ConfigNode can wait until the lease fence timeout plus a small margin. > After that point, the ConfigNode can safely infer that > the silent DataNode has self-fenced and allow the procedure to continue. > > > This avoids failing the whole metadata procedure simply because one > DataNode is temporarily down or partitioned. > > > 3. Table metadata procedure improvements > > > The implementation updates table metadata procedures to work with the > new lease-based broadcast semantics, including create, alter, drop, and > related table cache update paths. > > > For table deletion, a PreDeleteTsTable marker is introduced to make the > state transition safer. This helps avoid exposing an inconsistent table > state when a drop-table procedure is interrupted or partially completed. > > > A rollback plan for pre-delete table state is also added. > > > 4. Cache recovery and reload > > > A new DataNode lease recovery RPC path is added so that after a fenced > DataNode reconnects,Subject: Follow-up: Lease-based self-fencing framework > implementation update > > > Hi everyone, > > > Following the previous discussion, I have prepared an initial > implementation for the lease-based self-fencing framework. I would like to > summarize the main changes in this patch and clarify the current scope. > > > The major changes are: > > > 1. DataNode-side metadata lease management > > > A new MetadataLeaseManager is introduced on the DataNode side. > > > It tracks the last ConfigNode heartbeat using a monotonic clock. When > the heartbeat is not renewed within metadata_lease_fence_ms, the DataNode > marks itself as fenced, clears local metadata-related caches, and makes the > affected metadata cache access fail closed > instead of serving potentially stale metadata. > > > The default metadata caches covered by this mechanism include: > > > - partition cache > - table metadata cache > - tree device schema cache > > > After the lease is recovered, the DataNode can reload table metadata > from ConfigNode before returning to normal state. > > > 2. ConfigNode-side broadcast coordination > > > On the ConfigNode side, the patch introduces: > > > - DataNodeContactTracker > - ClusterCachePropagator > - MetadataBroadcastVerdict > > > The ConfigNode records the last successful heartbeat response time of > each DataNode. During metadata cache propagation, if a DataNode does not > acknowledge the update, the ConfigNode waits until the fencing timeout plus > a safety margin. After that, the ConfigNode can > safely infer that the silent DataNode has self-fenced and allow the > procedure to continue. > > > This avoids unnecessary rollback when a DataNode is already unreachable > or partitioned. > > > 3. Table metadata operation improvements > > > The implementation mainly targets table metadata HA scenarios, including > create/alter/drop table and related table cache updates. > > > For table deletion, a PreDeleteTsTable state is introduced. This helps > distinguish tables that are in the middle of deletion and prevents unsafe > state transitions when the procedure is interrupted or partially failed. > > > A rollback plan for pre-delete table state is also added to make the > procedure state transition safer. > > > 4. DataNode fail-closed behavior > > > Several DataNode metadata cache access paths now check the metadata > lease state before trusting local cache data. If the DataNode is fenced, > the operation fails explicitly with a metadata lease fenced error rather > than using stale cache. > > > This covers table cache, partition cache, tree device schema cache, and > authority cache related paths. > > > 5. Configuration and observability > > > A new configuration item metadata_lease_fence_ms is added, with a > default value of 20 seconds. > > > Metrics are also added for metadata lease status, so operators can > observe whether a DataNode is fenced and how long it has been since the > last ConfigNode heartbeat. > > > 6. Tests > > > The patch adds unit tests for: > > > - lease state transition and recovery > - ConfigNode broadcast verdict logic > - DataNode heartbeat contact tracking > - table cache fail-closed behavior > - partition cache fail-closed behavior > - tree schema cache fail-closed behavior > - authority cache invalidation under fenced state > > > There is also an integration test covering table DDL HA behavior in a > multi-DataNode cluster. > Any feedback on the scope, failure semantics, and whether more metadata > caches should be included in the lease-managed set would be very welcome. > > > > > > 主要改动如下： > > > 1. DataNode 侧元数据租约管理 > > > 本次新增了 MetadataLeaseManager，用于在 DataNode 侧管理元数据租约。 > > > 它会基于单调时钟记录最近一次收到 ConfigNode 心跳的时间。如果超过 metadata_lease_fence_ms > 仍未收到新的心跳，DataNode 会认为自己已经与集群隔离，并进入 fenced 状态。进入该状态后，DataNode 会清理 > 本地相关元数据缓存，并在访问这些缓存时 fail closed，避免继续使用可能已经过期的元数据。 > > > 当前默认纳入该机制保护的缓存包括： > > > - 分区缓存 > - 表元数据缓存 > - 树模型设备 Schema 缓存 > > > 当租约恢复后，DataNode 可以重新从 ConfigNode 拉取表元数据，并恢复到正常状态。 > > > 2. ConfigNode 侧广播协调 > > > 在 ConfigNode 侧，本次新增了： > > > - DataNodeContactTracker > - ClusterCachePropagator > - MetadataBroadcastVerdict > > > ConfigNode 会记录每个 DataNode 最近一次成功响应心跳的时间。在进行元数据缓存传播时，如果某个 DataNode 没有返回 > ack，ConfigNode 不会立即失败，而是等待到租约超时时间加上一个安全余量。超过该时间 > 后，ConfigNode 可以在逻辑上判断该静默 DataNode 已经完成自隔离，从而允许 procedure 继续执行。 > > > 这样可以避免某个 DataNode 已经宕机或发生网络分区时，整个元数据变更流程被不必要地回滚。 > > > 3. 表元数据操作改进 > > > 当前实现主要面向表元数据的 HA 场景，包括创建、修改、删除表以及相关表缓存更新流程。 > > > 对于删除表场景，本次引入了 PreDeleteTsTable 状态，用于表示表已经进入预删除阶段。这样可以在 procedure > 被中断或部分失败时，更清晰地区分正常表、预删除表和已删除表，避免不安全的状态转换。 > > > 同时也新增了预删除状态的 rollback plan，用于保证删除流程中的状态回滚更加安全。 > > > 4. DataNode fail-closed 行为 > > > 多个 DataNode 元数据缓存访问路径现在会在使用本地缓存前检查元数据租约状态。如果 DataNode 已经 fenced，则会明确返回 > metadata lease fenced 相关错误，而不是继续使用旧缓存。 > > > 目前覆盖的路径包括： > > > - 表缓存 > - 分区缓存 > - 树模型设备 Schema 缓存 > - 权限缓存相关路径 > > > 5. 配置与可观测性 > > > 本次新增配置项 metadata_lease_fence_ms，默认值为 20 秒。 > > > 同时增加了元数据租约相关 metrics，便于运维侧观察 DataNode 当前是否处于 fenced 状态，以及距离最近一次 > ConfigNode 心跳已经过去多长时间。 > > > 6. 测试 > > > 本次补充了多类测试，包括： > > > - 元数据租约状态切换与恢复 > - ConfigNode 广播裁决逻辑 > - DataNode 心跳响应时间追踪 > - 表缓存 fail-closed 行为 > - 分区缓存 fail-closed 行为 > - 树模型 Schema 缓存 fail-closed 行为 > - fenced 状态下权限缓存清理行为 > > > 同时也增加了一个多 DataNode 集群下的表 DDL HA 集成测试，用于验证异常场景下表元数据操作的可用性。 > > > Best wishes, > Yaobin Chen
