Hello Druid community,
My name is Colin, and I am from the Apache TsFile community. We would like to discuss adding support for the Apache TsFile data format to Apache Druid. Apache TsFile is a columnar file format designed for time-series data and is used as the underlying storage file format by Apache IoTDB. It can also be integrated independently into other data-processing systems. TsFile provides implementations in Java, C, C++, and Python. It supports chunked storage, compression, encoding, time-range filtering, and column pruning for efficient storage and access to time-series data. Apache TsFile project: <https://github.com/apache/tsfile> We noticed that Druid already supports data formats and data sources such as Avro, ORC, Parquet, and Iceberg through its extension mechanism. We believe a similar mechanism could enable Druid to read and ingest TsFile data. Based on our current understanding of Druid's extension mechanism, one possible initial approach would be to implement a TsFile input format extension: 1. Implement Druid's `InputFormat` interface. 2. Implement an appropriate `InputEntityReader` for the binary TsFile format. 3. Use the Apache TsFile Java Reader to read the files. 4. Map a configured TsFile time column to Druid's timestamp field. 5. Convert the remaining TsFile fields into rows consumed by Druid's ingestion pipeline. 6. Initially support batch ingestion of TsFile files from local filesystems or object storage. The initial version could focus on the core TsFile batch-ingestion path and consider the following capabilities: - Configuring the TsFile column to use as the Druid timestamp; - Basic data type mapping; - Column selection and time-range filtering; - Basic schema discovery; - Single-file and multi-file batch ingestion. Before starting the implementation, we would appreciate feedback from the Druid community on the following questions: 1. Would the Druid community be interested in support for TsFile as an input format or data source? 2. For local files and object storage already supported by existing `InputSource` implementations, would it be appropriate to reuse those implementations and add a TsFile `InputFormat`? If TsFile requires format-specific file discovery or split planning, would a dedicated `InputSource` also be necessary? 3. If we proceed with the implementation, would this extension be suitable for the `extensions-contrib` directory in the `apache/druid` repository? 4. Do you have any other recommendations regarding the initial scope or data-model mapping? If the community considers this direction reasonable, we are willing to take responsibility for implementing, testing, and maintaining the extension. We would also like to discuss with the Druid community where the code should be hosted and maintained before we begin the implementation. As a related reference, the DuckDB TsFile extension we recently implemented has been accepted into DuckDB Community Extensions and can query TsFile data directly: - Extension repository: <https://github.com/ColinLeeo/tsfile-duckdb> - Merged DuckDB Community Extensions PR: <https://github.com/duckdb/community-extensions/pull/2435> We look forward to hearing your feedback and suggestions. Best regards, Colin Lee Apache TsFile
