ColinLeeo commented on code in PR #20729: URL: https://github.com/apache/druid/pull/20729#discussion_r4396069279
########## docs/development/extensions-contrib/tsfile.md: ########## @@ -0,0 +1,141 @@ +--- +id: tsfile +title: "Apache TsFile input format" +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `druid-tsfile-extensions` in the extensions load list. + +This extension enables Druid to ingest [Apache TsFile](https://tsfile.apache.org/) files. TsFile is the columnar +storage format for time series data used by [Apache IoTDB](https://iotdb.apache.org/). It is read through the +TsFile Java library, version 2.5.0. + +The extension supports batch ingestion from local files and from any input source that Druid can read as a file +(for example, S3). Local files are read in place; files from remote input sources are first fetched to the task's +temporary directory, so a non-local input source needs enough local disk for one file at a time. + +## TsFile input format + +| Field | Type | Description | Required | +|-------|------|-------------|----------| +| type | String | Must be `tsfile` | yes | +| tables | Array of String | Tables to read. When omitted or empty, all tables in the file are read. | no | +| columns | Array of String | Columns to read. When omitted or empty, all columns of each selected table are read. | no | +| interval | String | Time range to read, in Druid interval format (`start/end`). The range is end-exclusive. When omitted, each file is read in full. | no | + +### Time range pruning + +When `interval` is set, it is pushed down to the TsFile reader, which uses the time statistics stored with each +chunk to skip chunks that fall outside the range. Only matching rows are returned. + +Druid does not pass the task's time window to input formats, so the range has to be set explicitly here. Rows are +still filtered by the granularity spec as usual. + +### Timestamp + +TsFile stores its own timestamp, which is exposed to Druid under the column name `Time`. The ingestion spec must +therefore select `Time` as the timestamp column with a numeric format, for example: + +```json +"timestampSpec": { + "column": "Time", + "format": "millis" +} +``` + +If `timestampSpec.column` is set to another name, the TsFile timestamp is also placed under that name so that +existing ingestion specs keep working. In that case a TsFile column with the same name is shadowed. + +### Column types + +Tag, field, and attribute columns are all converted to input rows. The mapping of TsFile data types is: + +| TsFile type | Druid value | +|-------------|-------------| +| INT32 | Integer | +| INT64 | Long | +| FLOAT | Float | +| DOUBLE | Double | +| BOOLEAN | Boolean | +| TEXT, STRING | String | +| TIMESTAMP | Long (epoch milliseconds) | +| DATE | Integer (epoch day) | +| BLOB | Base64-encoded String | + +`VECTOR`, `OBJECT`, `ROW`, and `UNKNOWN` columns are not supported; requesting one of them fails the ingestion task. +Null values are omitted from the row. + +## Example: local batch ingestion Review Comment: Added a SQL-based ingestion example right after the JSON spec example: an MSQ `INSERT INTO ... SELECT ... FROM TABLE(EXTERN('<input source>', '<input format>', '<column signature>')) PARTITIONED BY DAY`, with a note that the implicit `Time` column is declared as `long` in the signature and converted with `MILLIS_TO_TIMESTAMP`. ########## licenses.yaml: ########## @@ -4051,7 +4051,7 @@ name: XZ license_category: binary module: java-core license_name: Public Domain -version: 1.9 +version: "1.10" Review Comment: The quotes are needed here. `check-licenses.py` reads `licenses.yaml` with `yaml.FullLoader` and normalizes the version through `str()`, so an unquoted `1.10` is parsed as the float `1.1` and no longer matches the resolved `xz` 1.10 artifact. That is exactly what packaging-check reported before I quoted it: `Missing licenses: [('java-core', 'org.tukaani', 'xz', '1.10', None)]`. The existing `"10.10"` entry has the same constraint; `"2.22"` and `"1.86"` are quoted even though those would round-trip unchanged. Happy to use a different representation if you would prefer one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
