Hi all, A tag index misalignment bug in `read_tsfile` was recently identified and fixed in PR #18191:
https://github.com/apache/iotdb/pull/18191 The bug occurs when projection pruning and tag predicate pushdown are combined. Suppose a TsFile contains tag columns `[tag1, tag2]` and a query like: ```sql SELECT time, value FROM read_tsfile(...) WHERE tag2 = 'x' ``` The optimizer does two things: 1. Since the output only requires `time` and `value`, it prunes both tag columns from the scan node's assignments. 2. It pushes `tag2 = 'x'` down to `read_tsfile` as a device filter, so devices can be filtered early. However, when constructing the pushdown filter, the code rebuilt a `TsTable` from the pruned assignments — which contained only `tag2`. The tag index was then recomputed against this pruned list, assigning `tag2` to index 0 instead of the correct index 1. Later, the device filter visitor reads `deviceID.segment(filter.getIndex() + 1)`, where `+1` skips the table name. Since `tag2` was mis-indexed as 0, it read `deviceID.segment(1)` — which is `tag1` — instead of `deviceID.segment(2)` — which is `tag2`. Concretely: - `WHERE tag2 = 'x'` was actually evaluated as `WHERE tag1 = 'x'` → returned empty - `WHERE tag2 = 'a'` was actually evaluated as `WHERE tag1 = 'a'` → returned wrong results **Fix**: Use `ExternalTsFileQueryResource.getTableColumnSchema()` instead of the pruned scan node `getAssignments()` when building the `TsTable` for device filtering. The query resource retains the complete, unpruned column schema from the initial planning phase, so tag indices are always correct regardless of projection pruning. This fix is a one-line change. A corresponding integration test has been added. Please review and let me know if you have any comments. --- 大家好， 近期发现并修复了 `read_tsfile` 中一个 "投影裁剪 + TAG 谓词下推" 组合场景下的 TAG 索引错位 bug，对应 PR #18191： https://github.com/apache/iotdb/pull/18191 该 bug 发生在投影裁剪和 TAG 谓词下推同时生效的场景。假设 TsFile 中有 TAG 列 `[tag1, tag2]`，执行以下查询： ```sql SELECT time, value FROM read_tsfile(...) WHERE tag2 = 'x' ``` 优化器做了两件事： 1. 因为输出只需要 `time` 和 `value`，对 TAG 列进行投影裁剪，从 scan node 的 assignments 中移除。 2. 将 `tag2 = 'x'` 下推到 `read_tsfile`，尝试提前筛选设备。 问题在于：构造下推过滤器时，代码从裁剪后的 assignments 重建了一个 `TsTable`（此时仅包含 `tag2`），随后基于这个不完整的 TAG 列表重新计算了索引，导致 `tag2` 被错误编号为 0（正确应为 1）。设备过滤器随后使用 `deviceID.segment(filter.getIndex() + 1)` 读取 TAG，其中 `+1` 是为了跳过 table_name。因为 `tag2` 被误编号为 0，实际读取的是 `deviceID.segment(1)` = `tag1`，而非 `deviceID.segment(2)` = `tag2`。 具体表现为： - `WHERE tag2 = 'x'` → 实际判断 `tag1 = 'x'` → 返回空 - `WHERE tag2 = 'a'` → 实际判断 `tag1 = 'a'` → 错误返回了数据 **修复**：在构建设备过滤器的 `TsTable` 时，使用 `ExternalTsFileQueryResource.getTableColumnSchema()`（该对象始终保留初始规划阶段的完整列 schema，TAG 列齐全且顺序正确），替代扫描节点中层被裁剪过的 `getAssignments()`。 该修复仅一行代码变更，并已添加相应的集成测试。 欢迎 review，如有任何问题欢迎指出。 Best regards, Wenwei Shu wenwei shu <[email protected]> 于2026年7月2日周四 09:49写道： > Hi all, I’d like to share a new feature introduced by PR #17951: > https://github.com/apache/iotdb/pull/17951 > This PR adds a new table-valued function, read_tsfile, which allows IoTDB > to query external TsFiles that are not managed by IoTDB. With this feature, > users can directly analyze standalone TsFile data through IoTDB’s table SQL > engine, without importing the files into an IoTDB data region first. > Example usage: > SELECT > time > , tag1, tag2, s1, s2 > FROM > read_tsfile( > PATHS > => > '/path/to/a.tsfile,/path/to/b.tsfile' > , > TABLE_NAME > => > 'table1' > ) > WHERE > tag1 > = > 'tag1_1' > ORDER BY > time > ; > It also supports querying directories: > SELECT > * > FROM > read_tsfile( > PATHS > => > '/path/to/external_tsfiles' > , > TABLE_NAME > => > 'table1' > ); > Main capabilities include: - Query one or more external TsFiles via table > SQL. - Recursively scan .tsfile files under directory inputs. - Infer or > specify the target table name. - Merge compatible schemas across multiple > TsFiles. - Support normal SQL operations such as projection, filtering, > aggregation, grouping, and joins. This feature is useful for scenarios such > as offline TsFile analysis, migration verification, troubleshooting, backup > inspection, and ad-hoc exploration of TsFile data outside an IoTDB cluster. > Feedback, suggestions, and follow-up improvements are welcome. 大家好， 我想分享一个在 > PR #17951 中引入的新功能： https://github.com/apache/iotdb/pull/17951 该 PR > 新增了一个表值函数 read_tsfile，使 IoTDB 可以查询不受 IoTDB 管理的外部 TsFile。通过这个功能，用户可以直接使用 > IoTDB 的表模型 SQL 引擎分析独立的 TsFile 数据，而无需先将这些文件导入到 IoTDB 的数据区域中。 使用示例： > SELECT > time > , tag1, tag2, s1, s2 > FROM > read_tsfile( > PATHS > => > '/path/to/a.tsfile,/path/to/b.tsfile' > , > TABLE_NAME > => > 'table1' > ) > WHERE > tag1 > = > 'tag1_1' > ORDER BY > time > ; > 它也支持查询目录： > SELECT > * > FROM > read_tsfile( > PATHS > => > '/path/to/external_tsfiles' > , > TABLE_NAME > => > 'table1' > ); > 主要能力包括： - 通过表模型 SQL 查询一个或多个外部 TsFile。 - 对目录输入递归扫描其中的 .tsfile 文件。 - > 支持推断或显式指定目标表名。 - 支持合并多个 TsFile 中兼容的 schema。 - 支持常规 SQL 操作，例如投影、过滤、聚合、分组和 > Join。 该功能适用于离线 TsFile 分析、迁移验证、问题排查、备份检查，以及对 IoTDB 集群之外的 TsFile 数据进行临时探索等场景。 > 欢迎大家提出反馈、建议以及后续改进意见。 Best regards, Wenwei Shu >
