Hi all,
I would like to share a query retry bug found in tree-model queries and invite the community to review the fix. When it can happen The issue requires both of the following conditions: 1. A tree-model query contains a time predicate, for example: SELECT s1 FROM root.sg.d1 WHERE time >= 1000 AND time < 2000; 2. The first analysis succeeds, but query dispatch fails with a retryable error, causing QueryExecution to analyze the same parsed QueryStatement again. In the case we observed, the number of internal connections on the DataNode hosting the region leader had reached the limit of 1,000, so the first dispatch failed and triggered a retry. Other transient dispatch failures may potentially enter the same retry path. Problem and impact During the first analysis, the analyzer extracts the global time predicate and previously replaced the WHERE condition in the parser-owned AST with the remaining value predicate. When the same statement was analyzed again, the original time predicate was no longer available. Consequently, the retried query could be planned and executed without the intended time filter. Possible symptoms include: - returning data outside the requested time range and producing incorrect query results; - scanning significantly more data than expected; - increased query latency and additional CPU, memory, and I/O consumption. Because the problem only appears when a retryable dispatch failure occurs after the first analysis, it may look intermittent and is more likely to be observed during overload, leader changes, or transient connection failures. Queries that complete in the first attempt are not affected by this particular retry issue. While investigating this case, we also reviewed other analysis-time mutations in QueryStatement and related table-model device-query ASTs. Similar state reuse could affect predicates, ORDER BY keys, GROUP BY/HAVING expressions, LIMIT/OFFSET pushdown, template analysis, and repeated analysis of SHOW/COUNT DEVICES statements. Fix The fix mainly: - makes predicate extraction and simplification non-mutating; - introduces a per-analysis mutation journal with lazy copy-on-write, so a retry starts from the original parsed statement; - keeps analysis- or planning-derived state out of parser-owned ASTs; - uses per-analysis working statements for the affected table-model device queries. The implementation avoids an unconditional deep copy of the complete AST. Only fields or mutable components that are actually changed during analysis are recorded or copied. Regression unit tests have been added for repeated analysis, retry lifecycle, predicate preservation, ORDER BY planning, template queries, and table-model device queries. PR: https://github.com/apache/iotdb/pull/18179 Reviews and comments are very welcome, especially regarding the retry lifecycle, mutation coverage, and copy-on-write boundaries. Since this bug may lead to incorrect query results under transient dispatch failures, I would also like to propose including this fix in 2.0.11.1. 大家好， 这里同步一个树模型查询重试相关的 bug，并邀请大家 review 对应的修复。 可能触发问题的场景 该问题需要同时满足以下条件： 1. 树模型查询中包含时间过滤条件，例如： SELECT s1 FROM root.sg.d1 WHERE time >= 1000 AND time < 2000; 2. 第一次查询分析成功，但 dispatch 阶段发生了可重试的失败，导致 QueryExecution 使用同一个已经解析过的 QueryStatement 再次进行分析。 在我们实际遇到的 case 中，region leader 所在 DataNode 的 internal 连接数已经达到了 1000 的上限，导致第一次 dispatch 失败并触发重试。其他临时性的 dispatch 失败也可能进入相同的重试流程。 问题表现及影响 在第一次分析过程中，Analyzer 会提取全局时间过滤条件。此前的实现会直接修改 parser 持有的 AST，将 QueryStatement 中的 WHERE 条件替换为剩余的值过滤条件。 当重试流程再次分析同一个 QueryStatement 时，原始时间条件已经不存在。因此，重试后的查询可能在没有预期时间过滤条件的情况下完成规划和执行。 可能出现的表现包括： - 返回请求时间范围之外的数据，产生错误的查询结果； - 实际扫描的数据量显著增加； - 查询延迟升高，并消耗额外的 CPU、内存和 I/O 资源。 由于该问题只有在第一次分析完成后又发生可重试的 dispatch 失败时才会出现，因此表现可能是间歇性的，在系统过载、leader 切换或临时连接失败时更容易被观察到。能够在第一轮正常完成的查询不会受到这个特定重试问题的影响。 在排查该问题时，我们也检查了 QueryStatement 以及表模型设备查询相关 AST 中的其他 analysis-time 修改。类似的状态复用还可能影响谓词、ORDER BY 排序键、GROUP BY/HAVING 表达式、LIMIT/OFFSET 下推、模板查询分析，以及 SHOW/COUNT DEVICES 的重复分析。 修复方式 本次修复主要包括： - 将谓词提取和化简调整为非原地修改； - 引入单次 analysis attempt 的 mutation journal 和按需 copy-on-write，确保重试从原始解析结果开始； - 避免将分析或规划阶段产生的派生状态写入 parser 持有的 AST； - 对受影响的表模型设备查询使用单次分析专属的 working statement。 该实现不会无条件深拷贝完整 AST，只会记录或复制分析过程中实际发生修改的字段和可变组件。 同时补充了重复分析、重试生命周期、谓词保留、ORDER BY 规划、模板查询以及表模型设备查询等回归 UT。 PR：https://github.com/apache/iotdb/pull/18179 欢迎大家 review，尤其希望大家帮助检查重试生命周期、可能遗漏的 AST 修改点以及 copy-on-write 的边界。 考虑到该问题可能在临时 dispatch 失败时导致错误的查询结果，我也希望这个修复能够被加入 2.0.11.1。 Best regards, --------------- Yuan Tian
