ColinLeeo opened a new pull request, #20729: URL: https://github.com/apache/druid/pull/20729
## Summary Adds a community extension, `extensions-contrib/tsfile-extensions`, that lets Druid batch ingestion read [Apache TsFile](https://tsfile.apache.org/) files. TsFile is the columnar time-series format used by Apache IoTDB. This follows the discussion on dev@druid where the community extension approach under `extensions-contrib` was agreed, with the Apache TsFile community taking responsibility for maintenance and compatibility testing: https://lists.apache.org/thread/qmrkhobp7p07wkmcf4y4myq36k866zz0 ## What it does - New `InputFormat` type `tsfile`, implemented with `InputFormat` + `InputEntityReader` on top of `org.apache.tsfile:tsfile:2.5.0`, using the TsFile table-model reader (`TsFileReaderBuilder` -> `ITsFileReader` -> `ResultSet`). - Reads table-model TsFiles. Tree-model files (device paths without a table schema in the footer) are not supported by this first version. - Options: - `tables`: tables to read; all tables when omitted. - `columns`: columns to read; all columns when omitted. - `interval`: optional time range, pushed down to the TsFile reader, which skips chunks outside the range using the time statistics stored with each chunk. - TsFile's implicit `Time` column (epoch milliseconds) becomes the Druid timestamp, so ingestion specs use `"timestampSpec": {"column": "Time", "format": "millis"}`. - Local files are read in place; remote input sources are fetched to the task's temporary directory first. ## Compatibility | Item | Version | |------|---------| | TsFile Java library | `org.apache.tsfile:tsfile:2.5.0` | | TsFile format | files written by the TsFile 2.x table model | | Druid | 39.0.0 | ## Dependencies and licenses Only `org.apache.tsfile:tsfile` and `org.apache.tsfile:common` are new dependencies, both Apache-2.0. The transitive compression and parser libraries (`zstd-jni`, `snappy-java`, `lz4-java`, `xz`, `antlr4-runtime`) are already shipped by Druid. The extension declares `xz:1.10` and `lz4-java:1.11.4` explicitly because TsFile requires those versions and Druid currently manages older ones; the module would otherwise fail `RequireUpperBoundDeps`. ## Sample data and expected output - `extensions-contrib/tsfile-extensions/examples/data/sensor.tsfile`: three rows, one `sensor` table with tag columns `region`, `device` and value columns `temperature`, `humidity`, `status`. - `extensions-contrib/tsfile-extensions/examples/sensor-ingest.json`: ingestion spec for that file. - Expected input rows and the type mapping table are in `extensions-contrib/tsfile-extensions/README.md`. ## Test plan `TsFileInputFormatTest` (10 tests) covers: - reading all tag and field columns, including a null value; - the TsFile timestamp under the default and a custom `timestampSpec` column; - `tables` and `columns` selection; - `interval` pushdown, end-exclusive interval semantics, and an empty interval; - unknown table and unknown column errors; - Jackson round-trip with `"type": "tsfile"`; - reading the committed sample file against the documented expected rows. The module also passes checkstyle, PMD, forbiddenapis and RAT: ```bash mvn -P rat -pl extensions-contrib/tsfile-extensions verify ``` ## Known limitations - The input format is not splittable; parallelism comes from the input source (one file per task). - Druid does not pass the task's time window to input formats, so `interval` has to be set explicitly. - `VECTOR`, `OBJECT`, `ROW` and `UNKNOWN` TsFile columns are not supported and fail the task with a clear error. `BLOB` is exposed as a Base64 string and `DATE` as an epoch-day integer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
