Author: haonan Date: Thu Dec 25 01:48:06 2025 New Revision: 81521 Log: add 1.1.3 remove 1.1.1
Added: release/tsfile/1.1.3/ release/tsfile/1.1.3/README-zh.md release/tsfile/1.1.3/README.md release/tsfile/1.1.3/RELEASE_NOTES.md release/tsfile/1.1.3/apache-tsfile-1.1.3-source-release.zip (contents, props changed) release/tsfile/1.1.3/apache-tsfile-1.1.3-source-release.zip.asc release/tsfile/1.1.3/apache-tsfile-1.1.3-source-release.zip.sha512 Added: release/tsfile/1.1.3/README-zh.md ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ release/tsfile/1.1.3/README-zh.md Thu Dec 25 01:48:06 2025 (r81521) @@ -0,0 +1,130 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +[English](./README.md) | [中文](./README-zh.md) +# TsFile Document +<pre> +___________ ___________.__.__ +\__ ___/____\_ _____/|__| | ____ + | | / ___/| __) | | | _/ __ \ + | | \___ \ | \ | | |_\ ___/ + |____|/____ >\___ / |__|____/\___ > version 1.0.0 + \/ \/ \/ +</pre> +[](https://codecov.io/github/apache/tsfile) +[](http://search.maven.org/#search|gav|1|g:"org.apache.tsfile") + +## 简介 + +TsFile是一种为时间序列数据设计的列式存储文件格式，它支持高效压缩、高读写吞吐量，并且兼容多种框架，如Spark和Flink。TsFile很容易集成到物联网大数据处理框架中。 + +时序数据即时间序列数据，是指带时间标签（按照时间的顺序变化，即时间序列化）的数据，其来源多元、数据量庞大，可广泛应用于物联网、智能制造、金融分析等领域。在数据驱动的当下，时序数据的重要性不言而喻。 + +尽管时序数据如此普遍且重要，但长期以来，时序数据的管理都缺乏标准化的文件格式。TsFile 的出现为用户管理时序数据提供了统一的文件格式。 + +[点击查看更多](https://www.timecho.com/archives/tian-bu-shi-chang-kong-bai-apache-tsfile-ru-he-chong-xin-ding-yi-shi-xu-shu-ju-guan-li) + + +## TsFile 特性 + +TsFile 通过自研实现了时序数据高效率管理、高灵活传输，并支持多类软件深度集成。其特性包括： + +- 时序模型：专门为物联网设计的数据模型，每个时间序列与特定设备相关联，所有设备通过分层结构相互连接； + +- 跨语言独立使用：可以使用多种语言的 SDK 直接读写 TsFile，使得一些轻量级的数据读写场景成为可能。 + +- 高效写入和压缩：为时间序列量身定制的列式存储格式，将数据按设备进行组织，并保证每个序列的数据连续存储，最小化存储空间。相比 CSV，压缩比可提升 90% 以上。 + +- 高查询性能：通过设备、物理量和时间维度索引，TsFile 实现了基于特定时间范围的时序数据快速过滤和查询。相比通用文件格式，查询吞吐可提升 2-10 倍。 + +- 开放集成：TsFile 是时序数据库 IoTDB 的底层存储文件格式，可与 IoTDB 形成可插拔的存算分离架构。TsFile 支持与 Spark、Flink 等大数据软件建立无缝生态集成，从而确保跨不同数据处理环境的兼容性和互操作性，实现时序数据跨生态深度分析。 + +## TsFile 基本概念 + +TsFile 可管理多个设备的时序数据。每个设备可具有不同的物理量。 + +每个设备的每个物理量对应一条时间序列。 + +TsFile 数据模型（Schema）定义了所有设备物理量的集合，如下表所示（m1 ~ m5） + +| Time | deviceId | m1 | m2 | m3 | m4 | m5 | +|------|----------|----|----|----|----|----| +| 1 | device1 | 1 | 2 | 3 | | | +| 2 | device1 | 1 | 2 | 3 | | | +| 3 | device2 | 1 | | 3 | 4 | 5 | +| 4 | device2 | 1 | | 3 | 4 | 5 | +| 5 | device3 | 1 | 2 | 3 | 4 | 5 | + +其中 Time 和 deviceId 为内置字段，无需定义，可直接写入。 + +## TsFile 设计原理 + +### 文件结构 + +下为 Apache TsFile 的文件结构。 + +- Page：一段连续的时序数据，存储的基本单元，按时间升序排序，时间戳和值各有单独的列进行存储。 + +- Chunk：由同一序列的多个连续的 Page 组成，一个文件同一个序列可以存储多个 Chunk。 + +- ChunkGroup：由一个设备的一至多个 Chunk 组成，多个 Chunk 可共享一列时间存储（多值模型）。 + +- Index：TsFile 末尾的元数据文件包含序列内部时间维度的索引和序列间的索引信息。 + + + +### 编码和压缩 + +TsFile 通过采用二阶差分编码、游程编码（RLE）、位压缩和 Snappy 等先进的编码和压缩技术，优化时序数据的存储和访问，并支持对时间戳列和数据值列进行单独编码，以实现更好的数据处理效能。 + +其独特之处在于编码算法专为时序数据特性设计，聚焦在时间属性和数据之间的相关性。 + +TsFile、CSV 和 Parquet 三种文件格式的比较 + +| 维度 | TsFile | CSV | Parquet | +|---------|--------|-----|---------| +| 数据模型 | 物联网时序 | 无 | 嵌套 | +| 写入模式 | 批, 行 | 行 | 行 | +| 压缩 | 有 | 无 | 有 | +| 读取模式 | 查询, 扫描 | 扫描 | 查询 | +| 序列索引 | 有 | 无 | 无 | +| 时间索引 | 有 | 无 | 无 | + +基于对时序数据应用需求的深刻理解，TsFile 有助于实现时序数据高压缩比和实时访问速度，并为企业进一步构建高效、可扩展、灵活的数据分析平台提供底层文件技术支撑。 + +| 数据类型 | 推荐编码 | 推荐压缩算法 | +|---------|------------|--------| +| INT32 | TS_2DIFF | LZ4 | +| INT64 | TS_2DIFF | LZ4 | +| FLOAT | GORILLA | LZ4 | +| DOUBLE | GORILLA | LZ4 | +| BOOLEAN | RLE | LZ4 | +| TEXT | DICTIONARY | LZ4 | + +更多类型的编码和压缩方式参见[文档](https://iotdb.apache.org/zh/UserGuide/latest/Basic-Concept/Encoding-and-Compression.html) + +## 开发和使用 TsFile + +[Java](./java/tsfile/README-zh.md) + +[C++](./cpp/README-zh.md) + +[Python](./python/README-zh.md) Added: release/tsfile/1.1.3/README.md ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ release/tsfile/1.1.3/README.md Thu Dec 25 01:48:06 2025 (r81521) @@ -0,0 +1,129 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> + +[English](./README.md) | [中文](./README-zh.md) +# TsFile Document +<pre> +___________ ___________.__.__ +\__ ___/____\_ _____/|__| | ____ + | | / ___/| __) | | | _/ __ \ + | | \___ \ | \ | | |_\ ___/ + |____|/____ >\___ / |__|____/\___ > version 1.0.0 + \/ \/ \/ +</pre> +[](https://codecov.io/github/apache/tsfile) +[](http://search.maven.org/#search|gav|1|g:"org.apache.tsfile") + +## Introduction + +TsFile is a columnar storage file format designed for time series data, which supports efficient compression, high throughput of read and write, and compatibility with various frameworks, such as Spark and Flink. It is easy to integrate TsFile into IoT big data processing frameworks. + +Time series data is becoming increasingly important in a wide range of applications, including IoT, intelligent control, finance, log analysis, and monitoring systems. + +TsFile is the first existing standard file format for time series data. Despite the widespread presence and significance of temporal data, there has been a longstanding absence of standardized file formats for its management. The advent of TsFile introduces a unified file format to facilitate users in managing temporal data. + +[Click for More Information](https://www.timecho-global.com/archives/apache-tsfile-time-series-data-storage-redefined) + +## TsFile Features + +TsFile offers several distinctive features and benefits: + +- Mutil Language Independent Use: Multiple language SDK can be used to directly read and write TsFile, making it possible for some lightweight data reading and writing scenarios. + +- Efficient Writing and Compression: A column storage format tailored for time series, organizing data by device and ensuring continuous storage of data for each sequence, minimizing storage space. Compared to CSV, the compression ratio can be increased by more than 90%. + +- High Query Performance: By indexing devices, measurement, and time dimensions, TsFile implements fast filtering and querying of temporal data based on specific time ranges. Compared to general file formats, query throughput can be increased by 2-10 times. + +- Open Integration: TsFile is the underlying storage file format of the temporal database IoTDB, which can form a pluggable storage computing separation architecture with IoTDB. TsFile supports compatibility with Spark Flink and other big data software establish seamless ecosystem integration to ensure compatibility and interoperability across different data processing environments, and achieve deep analysis of temporal data across ecosystems. + +## TsFile Basic Concepts + +TsFile can manage the time series data of multiple devices. Each device can have different measurement. + +Each measurement of each device corresponds to a time series. + +The TsFile Scheme defines a set of measurement for all devices, as shown in the table below (m1~m5) + +| Time | deviceId | m1 | m2 | m3 | m4 | m5 | +|------|----------|----|----|----|----|----| +| 1 | device1 | 1 | 2 | 3 | | | +| 2 | device1 | 1 | 2 | 3 | | | +| 3 | device2 | 1 | | 3 | 4 | 5 | +| 4 | device2 | 1 | | 3 | 4 | 5 | +| 5 | device3 | 1 | 2 | 3 | 4 | 5 | + +Among them, Time and deviceId are built-in fields that do not need to be defined and can be written directly. + +## TsFile Design + +### File Structure + +TsFile adopts a columnar storage design, similar to other file formats, primarily to optimize time-series data's storage efficiency and query performance. This design aligns with the nature of time series data, which often involves large volumes of similar data types recorded over time. However, TsFile was developed particularly with a structure of page, chunk, chunk group, and index: + +- Page: The basic unit for storing time series data, sorted by time in ascending order with separate columns for timestamps and values. + +- Chunk: Comprising metadata headers and several pages, each chunk belongs to one time series, with variable sizes allowing for different compression and encoding methods. + +- Chunk Group: Multiple chunks within a chunk group belong to one or multiple series of a device written in the same period, facilitating efficient query processing. + +- Index: The file metadata at the end of TsFile contains a chunk-level index and file-level statistics for efficient data access. + + + +## Encoding and Compression + +TsFile employs advanced encoding and compression techniques to optimize storage and access for time series data. It uses methods like run-length encoding (RLE), bit-packing, and Snappy for efficient compression, allowing separate encoding of timestamp and value columns for better data processing. Its unique encoding algorithms are designed specifically for the characteristics of time series data in IoT scenarios, focusing on regular time intervals and the correlation among series. + +Its uniqueness lies in the encoding algorithm designed specifically for time series data characteristics, focusing on the correlation between time attributes and data. + +The table below compares 3 file formats in different dimensions. + +TsFile, CSV and Parquet in Comparison + +| Dimension | TsFile | CSV | Parquet | +|-----------------|--------------|-------|---------| +| Data Model | IoT | Plain | Nested | +| Write Mode | Tablet, Line | Line | Line | +| Compression | Yes | No | Yes | +| Read Mode | Query, Scan | Scan | Query | +| Index on Series | Yes | No | No | +| Index on Time | Yes | No | No | + +Its development facilitates efficient data encoding, compression, and access, reflecting a deep understanding of industry needs, pioneering a path toward efficient, scalable, and flexible data analytics platforms. + +| Data Type | Recommended Encoding | Recommended Compression | +|---------|------------|--------| +| INT32 | TS_2DIFF | LZ4 | +| INT64 | TS_2DIFF | LZ4 | +| FLOAT | GORILLA | LZ4 | +| DOUBLE | GORILLA | LZ4 | +| BOOLEAN | RLE | LZ4 | +| TEXT | DICTIONARY | LZ4 | + +more see [Docs](https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Encoding-and-Compression.html) + +## Build and Use TsFile + +[Java](./java/tsfile/README.md) + +[C++](./cpp/README.md) + +[Python](./python/README.md) Added: release/tsfile/1.1.3/RELEASE_NOTES.md ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ release/tsfile/1.1.3/RELEASE_NOTES.md Thu Dec 25 01:48:06 2025 (r81521) @@ -0,0 +1,103 @@ +<!-- + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +--> +# Apache TsFile 1.1.3 + +* perf: Optimize aligned object memory size calculation +* feat: add markRange / unmarkRange / merge for high-performance bit manipulation +* Modify the TsFileSequenceReaderTimeseriesMetadataIterator next function to return a LinkedHashMap +* fix GroupByMonthFilter.getTimeRanges +* Fix bug in PaginationController +* change config not found log to debug +* Init all series writer for AlignedChunkGroupWriter +* Added memory calculation for tablet +* Dont print exception log when thread is interrupted +* Bump lz4-java version to 1.10.1 + +# Apache TsFile 1.1.2 + +## Improvement/Bugfix + +* Fix the bug in parse date to int when year out of range by @HTHou in #500 +* Add TsFileLastReader for retrieving last points in a TsFile by @jt2594838 in #506 +* Added accountable function to measurementSchema by @Caideyipi in #509 +* Correct the retained size calculation for BinaryColumn and BinaryColumnBuilder by @JackieTien97 in #514 +* add switch to disable native lz4 (#480) by @jt2594838 in #515 +* Correct the memroy calculation of BinaryColumnBuilder by @JackieTien97 in #530 +* Fetch max tsblock line number each time from TSFileConfig by @JackieTien97 in #535 +* Support set default compression by data type & Bump org.apache.commons:commons-lang3 from 3.15.0 to 3.18.0 by @jt2594838 in #547 +* Avoid calculating shallow size of map by @shuwenwei in #566 +* Add methods for RamUsageEstimator by @shuwenwei in #570 + +# Apache TsFile 1.1.1 + +## Improvement/Bugfix +* Fixed the issue that the time of the first data item written to TSFile by measurement cannot be a negative number (#297) +* Add LongConsumer ioSizeRecorder in TsFileSequenceReader for IoTDB scan (#301) +* Add readItimeseriesMetadata method (#312) +* Tablet.serialize() may throw an exception due to null values in the Date column (#330) +* Add FlushChunkMetadataListener (#328) +* Add final for readData methods (#347) +* Bump logback to 1.3.15 (#362) +* Fix example compile issue (#400) +* Fixed the empty string ser/de bug & null string[] array calculation bug (#449) + +# Apache TsFile 1.1.0 + +## New Feature +- Support new data types: STRING, BLOB, TIMESTAMP, DATE by @Cpaulyz in #76 +- Add an equivalent .getLongs() method to .getTimes() in TimeColumn. by @Sh-Zh-7 in #61 +- Return all columns in TsBlock class by @Sh-Zh-7 in #80 + +## Improvement/Bugfix + +- Fix value filter allSatisfy bug by @liuminghui233 in #41 +- Fix error log caused by ClosedByInterruptException by @shuwenwei in #47 +- Fix the mistaken argument in LZ4 Uncompressor by @jt2594838 in #57 +- Remove duplicate lookups in dictionary encoder by @MrQuansy in #54 +- Optimize SeriesScanUtil by memorizing the order time and satisfied information for each Seq and Unseq Resource by @JackieTien97 in #58 +- Fix TsBlockBuilder bug in AlignedPageReader and PageReader. by @JackieTien97 in #77 +- Fix ZstdUncompressor by @lancelly in #132 +- fix RLBE Encoding for float and double by @gzh23 in #143 +- Fix uncompress page data by @shuwenwei in #161 +- Fix encoder and decoder construction of RLBE by @jt2594838 in #162 +- Fix aligned TimeValuePair npe by @shuwenwei in #173 +- Fix StringStatistics data type by @shuwenwei in #177 +- Fix bug in the conversion of int types to timestamp. by @FearfulTomcat27 in #224 +- Fix error when write aligned tablet with null date by @HTHou in #251 + +# Apache TsFile 1.0.0 + +## New Features + +- Support registering devices +- Support registering measurements +- Support adding additional measurements +- Support writing timeseries data without pre-defined schema +- Support writing timeseries data with pre-defined schema +- Support writing with tsRecord +- Support writing with Tablet +- Support writing data into a closed TsFile +- Support query timeseries data without any filter +- Support query timeseries data with time filter +- Support query timeseries data with value filter +- Support BOOLEAN, INT32, INT64, FLOAT, DOUBLE, TEXT data types +- Support PLAIN, DICTIONARY, RLE, TS_2DIFF, GORILLA, ZIGZAG, CHIMP, SPRINTZ, RLBE encoding algorithm +- Support UNCOMPRESSED, SNAPPY, GZIP, LZ4, ZSTD, LZMA2 compression algorithm Added: release/tsfile/1.1.3/apache-tsfile-1.1.3-source-release.zip ============================================================================== Binary file. No diff available. Added: release/tsfile/1.1.3/apache-tsfile-1.1.3-source-release.zip.asc ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ release/tsfile/1.1.3/apache-tsfile-1.1.3-source-release.zip.asc Thu Dec 25 01:48:06 2025 (r81521) @@ -0,0 +1,16 @@ +-----BEGIN PGP SIGNATURE----- + +iQIzBAABCgAdFiEEzhaDbaKuItvzc/VQLM9+e8qjyj0FAmlGgIoACgkQLM9+e8qj +yj2PkA//YtkexaLQyz40jL0iIlkERRwA+C/BjfjQ06H2UHQxXWRt/TkEi93zHcuk +igOb5UchXZdLvHN5WbETFL+ov8zKnwm2nxBIzCcAPtzt8wdtFRJMCbHBGYB77bc8 +SKoGR9LJKrmBDHtxyFoEOJGiOD20G/kNnC1cHAGM2DUej+678TDOL1ODGm4hMeqE +7ZWMjHZ3wE+7KSav1lxOZXX36E+9ZTm3zzuBzvEPJSVId4Nwk9nRbgcZaErLPEPs +aNAWrCjeKsLPsQBQ7S46BSyqsxmNBvxGY5AuA3VdiuOpaRWokhZSOmUfI95j7An6 +5yJHCFO8ijM/1BY0T2lcZg9yeoxPXb2aj+hY71BHtu/46pJdB8FbeSnFWM+oETKQ +0BLPFMBLwGQ/42bXhBBl38R1aHphMBMZ9W2zJfNJhsqJyVaOH6s3r2viuo8Pxtak +QEUp5Xf7d88nRnnUTLucf93N5Gq1vi/tZr4OHYHf9mqoyX6AcjvuP21RlHXEV1q/ +kr+gOOIsrBHmFCxt8+UYcRVXs9dG/BN2HoHZItOX8w4qdwhrcH6FY0W6pWuA649H +q3DRJNY9fQ7XGRdWji1Hfuf0resU2UvQoFlnuWKtXdZho1kS7mwSaaP2QNIbe7Eg +I3hHBYYRyyvnXXh6AJRTd5o3sWl5giG8eHWNxr7ueZx6JEYvo6g= +=eobz +-----END PGP SIGNATURE----- Added: release/tsfile/1.1.3/apache-tsfile-1.1.3-source-release.zip.sha512 ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ release/tsfile/1.1.3/apache-tsfile-1.1.3-source-release.zip.sha512 Thu Dec 25 01:48:06 2025 (r81521) @@ -0,0 +1 @@ +641032c71bafe2dbd415ca6b84f8654733ce756c187548cbb453a5587b9f4420e26d8a46c9b03bad15799d73dd1716e232417727045fde7537839936f198bd6c \ No newline at end of file
