Hi, After TsFile is being accepted by Apache, I will create an TsFile-Rust repo.
Thanks, ————————————————— Jialin Qiao Apache IoTDB PMC Giorgio Zoppi <giorgio.zo...@gmail.com> 于2023年10月25日周三 15:46写道： > > Hello, > can you add in apache a initial ts-native for the rust project? Sorry for > being pedantic. > Best Regards, > Giorgio > > Il giorno mer 25 ott 2023 alle ore 05:04 Jialin Qiao <qiaojia...@apache.org> > ha scritto: > > > Hi, > > > > The TsFile proposal is as follows, feel free to give advice :-) > > > > Abstract > > > > TsFile is a columnar storage file format designed for time series > > data, which supports efficient compression, high throughput of read > > and write, and compatibility with various frameworks, such as Spark > > and Flink. It is easy to integrate TsFile into IoT big data processing > > frameworks. > > > > Proposal > > > > TsFile is used for managing time series data. Although it is firstly > > used inside IoTDB, many users and companies use TsFile directly as a > > time series data management solution independently. Besides, there is > > a growing requirement for Multi-Language Support of TsFile > > implementations, such as C++, Go and Rust. > > > > Apache IoTDB community hereby submit this proposal of TsFile as an > > independent Apache project. This proposal outlines the key features > > and benefits of TsFile, along with the integration plan and the need > > for multi-language support. > > > > Background > > > > Time series data is becoming increasingly important in a wide range of > > applications, including IoT, intelligent control, finance, log > > analysis, and monitoring systems. > > TsFile has been developed by Apache IoTDB community in Java and is in > > IoTDB repository. Users could store time series data using TsFile, > > then read and analyze it in IoTDB, Spark and Flink. IoTDB could also > > generate TsFiles and syncronize TsFiles between two IoTDB instances. > > Furthermore, the demand for TsFile implementations in multiple > > programming languages has been growing, as it allows developers to > > leverage TsFile's capabilities in their preferred language. > > > > TsFile offers several distinctive features and benefits: > > Efficient Storage and Compression: TsFile employs advanced compression > > techniques to minimize storage requirements, resulting in reduced disk > > space consumption and improved system efficiency. > > Flexible Schema and Metadata Management: TsFile allows for directly > > write data without pre defining the schema, which is flexible for data > > aquisition. > > High Query Performance with time range: TsFile has indexed devices, > > sensors and time dimensions to accelerate query performance, enabling > > fast filtering and retrieval of time series data. > > Seamless Integration: TsFile is designed to seamlessly integrate with > > existing big data frameworks, such as Spark, Flink and Hadoop. > > > > > > Rationale > > > > Before using TsFile, there doesn't exist a file format for time > > series. The industry companies usually write time series data in > > various user-defined file format without unification, or use general > > columnar file format such as Parquet and ORC, which makes data > > collection and processing complicated without a standard. After > > TsFile, organizations could write data in TsFile inside end devices or > > gateway, then transfer TsFile to the cloud for unified management in > > IoTDB and other systems. In this way, we lower the network > > transmission and the computing resource consumption in the cloud. > > > > Initial Goals > > > > The initial goals include: > > > > Make TsFile an independent project that has its own SDK and > > documentation that is easier to use. > > Multi-Language Support of TsFile implementations apart from Java, such > > as C++, Go and Rust. > > Integrate more encoding and compression method in TsFile. > > More tools for TsFile: visualization tool, parsing tool, repair tool. > > > > > > Current Status > > > > Meritocracy > > > > We plan to invite the IoTDB committer to be the initial committer of > > TsFile. And we would like to follow ASF meritocratic principles and > > invite additional developers to participate. We will establish the > > documentation and encourage and monitor community participation so > > that privileges can be extended to those that contribute. > > > > Community > > > > The TsFile community is grown from the Apache IoTDB Community. The > > IoTDB community is introducing TsFile at many technical conferences. > > Next, we will build the mailing list for more convenience, broader > > communication and archived discussions. We are open to recruiting more > > developers from diverse backgrounds. > > > > Core Developers > > > > TsFile initial PMCs are from IoTDB community: Christofer Dutz, > > Xiangdong Huang, Jialin Qiao, Steve Yurong Su, Jinrui Zhang, Yuan > > Tian, Xinyu Tan, Haonan Hou, Gaofei Cao, Tian Jiang, Chao > > Wang(wangchao316), Chao Wang(mychaow), Houliang Qi, Kun Liu. > > These people has extensive experience in building database and data > > management system. > > > > Alignment > > > > The ASF is the natural choice to host the TsFile project as its goal > > of encouraging community-driven open-source projects fits with our > > vision for TsFile. Additionally, many other projects with which we are > > familiar with and expect TsFile to integrate with, such as Apache > > Spark, Apache Flink and Apache IoTDB are hosted by the ASF and we will > > benefit and provide benefits in close proximity to them. > > > > Known Risks > > > > Project Name > > > > TsFile project is used in IoTDB and other scenarios for over 7 years, > > its name is unique. > > > > Orphaned Products > > > > The core developers plan to work full time on the project. There is > > very little risk of TsFile being abandoned as it is part of Apache > > IoTDB's internal infrastructure. Tsinghua and NEL-BDS Lab relies on > > TsFile as a platform for a large number of long-term research > > projects. Companies such as Timcho, Huawei, BONC, Yonyou will also > > participated in this project. > > > > Inexperience with Open Source > > > > All of the core developers have experience with open source > > development. We have 1 Apache Board member(Christofer Dutz), 2 Apache > > Members(Xiangdong Huang, Jialin Qiao) and 11 Apache IoTDB PMCs in > > TsFile initial PMCs. We have the experience of the Apache way, such as > > community over code, license management, version release, CEV > > processing, attract committer/pmcs and meetups. > > > > Length of Incubation > > > > There are enough initial PMCs of TsFile know the ASF process well, so > > we apply to go straight to TLP. > > > > Homogeneous Developers > > > > The current core developers are from diverse groups: Apache IoTDB > > Community, Timecho, BONC, Huawei, eBay, Yonyou and Tsinghua > > University. > > > > Reliance on Salaried Developers > > > > Currently, the developers are paid to do work at Timecho, BONC, Huawei > > and Tsinghua University. Also we have students and > > researchers/professors community in universities, and their researches > > focus on big data management and analytics. It is unlikely that they > > will change their research focus away from big data management. We > > will work to ensure that the ability for the project to continuously > > be stewarded and to proceed forward independent of salaried developers > > is continued. > > > > Relationships with Other Apache Products > > > > TsFile is used by Apache IoTDB project as the default data file > > format. TsFile-Spark-connector and TsFile-Flink-connector have been > > developed to support analysing time series data by using Apache Spark > > and Flink. > > Overall, TsFile is designed as an open architecture, and it can be > > integrated with many other systems in the future. > > > > An Excessive Fascination with the Apache Brand > > > > We respect the reputation of the Apache brand and have no doubt that > > it will attract contributors and users. Most of the initial developers > > come from Apache IoTDB Community, so we have no intent to use the > > Apache brand for profit. Our goal is making TsFile integrating more > > with Apache projects and letting Apache community more professional in > > IoT data management. > > > > Documentation > > > > The Documentations: > > https://iotdb.apache.org/UserGuide/V1.2.x/API/Programming-TsFile-API.html > > Examples: https://github.com/apache/iotdb/tree/master/example/tsfile > > > > Initial Source > > > > TsFile core: https://github.com/apache/iotdb/tree/master/iotdb-core/tsfile > > Spark TsFile Connector： > > https://github.com/apache/iotdb/blob/master/iotdb-connector/spark-tsfile > > Flink TsFile Connector: > > > > https://github.com/apache/iotdb/blob/master/iotdb-connector/flink-tsfile-connector > > > > > > Source and Intellectual Property Submission Plan > > > > External Dependencies > > > > zstd-jni: BSD 2-Clause License > > logback-classic: Eclipse Public License - v 1.0 > > snappy-java: Apache License 2.0 > > commons-io: Apache License 2.0 > > commons-lang3: Apache License 2.0 > > lz4-java: Apache License 2.0 > > gson: Apache License 2.0 > > slice: Apache License 2.0 > > xz: Public Domain > > Mock: Apache License 2.0 > > > > > > Cryptography > > > > All codes are public under Apache license 2.0. > > > > Required Resources > > > > > > Mailing Lists > > > > priv...@tsfile.apache.org > > d...@tsfile.apache.org > > comm...@tsfile.apache.org > > > > Subversion Directory > > > > Git is the preferred source control system: git://git.apache.org/tsfile > > > > Issue Tracking > > > > JIRA TsFile (TsFile) > > > > Initial Committers > > > > Yuan Tian (jackietien at apache dot org) > > Chao Wang (wangchao316 at apache dot org) > > Christofer Dutz (cdutz at apache dot org) > > Jinrui Zhang (xingtanzjr at apache dot org) > > Steve Yurong Su (rong at apache dot org) > > Xinyu Tan (tanxinyu at apache dot org) > > Haohan Hou (haonan at apache dot org) > > Gaofei Cao (gaogaofei at apache dot org) > > Jialin Qiao (qiaojialin at apache dot org) > > Kun Liu (liukun at apache dot org) > > Houliang Qi (neuyilan at apache dot org) > > Xiangdong Huang (hxd at apache dot org) > > Chao Wang (chaow at apache dot org) > > Jianmin Wang (jimwang at apache dot org) > > Tian Jiang (jiangtian at apache dot org） > > Xinyi Zhao (zhaoxinyi at apache dot org) > > Shuo Zhang (shuozhagn at apache dot org) > > > > Sponsors > > Champion:TBD > > Nominated Mentors:TBD > > > > > > Thanks, > > ————————————————— > > Jialin Qiao > > Apache IoTDB PMC > > > > > -- > Life is a chess game - Anonymous.
