So, it seems that this document is based on an incubation proposal … sorry … I
just reviewed it on a content-level.
As we’re going straight TLP this should look quite a bit different. So I’ll try
to whip up a version:
WHEREAS, the Board of Directors deems it to be in the best interests of
the Foundation and consistent with the Foundation's purpose to
establish a Project Management Committee charged with the creation and
maintenance of open-source software, for distribution at no charge to
the public, related to a columnar storage file format designed for time
series data, which supports efficient compression, high throughput of
read and write, and compatibility with various frameworks, such as Spark
and Flink. It is easy to integrate TsFile into IoT big data processing
frameworks.
NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
(PMC), to be known as the "Apache TsFile Project", be and hereby is
established pursuant to Bylaws of the Foundation; and be it further
RESOLVED, that the Apache TsFile be and hereby is responsible for
the creation and maintenance of software related to a columnar storage
file format designed for time series data, which supports efficient
compression, high throughput of read and write, and compatibility with
various frameworks, such as Spark and Flink. It is easy to integrate
TsFile
into IoT big data processing frameworks.
RESOLVED, that the office of "Vice President, Apache TsFile" be and
hereby is created, the person holding such office to serve at the
direction of the Board of Directors as the chair of the Apache
TsFile Project, and to have primary responsibility for management of
the projects within the scope of responsibility of the Apache TsFile
Project; and be it further
RESOLVED, that the persons listed immediately below be and hereby are
appointed to serve as the initial members of the Apache TsFile
Project:
- Yuan Tian (jackietien at apache dot org)
- Chao Wang (wangchao316 at apache dot org)
- Christofer Dutz (cdutz at apache dot org)
- Jinrui Zhang (xingtanzjr at apache dot org)
- Steve Yurong Su (rong at apache dot org)
- Xinyu Tan (tanxinyu at apache dot org)
- Haohan Hou (haonan at apache dot org)
- Gaofei Cao (gaogaofei at apache dot org)
- Jialin Qiao (qiaojialin at apache dot org)
- Kun Liu (liukun at apache dot org)
- Houliang Qi (neuyilan at apache dot org)
- Xiangdong Huang (hxd at apache dot org)
- Chao Wang (chaow at apache dot org)
- Jianmin Wang (jimwang at apache dot org)
- Tian Jiang (jiangtian at apache dot org）
- Xinyi Zhao (zhaoxinyi at apache dot org)
- Shuo Zhang (shuozhagn at apache dot org)
NOW, THEREFORE, BE IT FURTHER RESOLVED, that {name of the chair} be
appointed to the office of Vice President, Apache TsFile, to serve in
accordance with and subject to the direction of the Board of Directors
and the Bylaws of the Foundation until death, resignation, retirement,
removal or disqualification, or until a successor is appointed; and be
it further
RESOLVED, that the Apache TsFile Project be and hereby is tasked with
the migration and rationalization of the Apache IoTDB TsFile subproject;
and be it further
RESOLVED, that all responsibilities pertaining to the Apache IoTDB
TsFile subproject encumbered upon the Apache IoTDB PMC are hereafter
discharged.
Von: Christofer Dutz <christofer.d...@c-ware.de>
Datum: Mittwoch, 25. Oktober 2023 um 12:24
An: dev@iotdb.apache.org <dev@iotdb.apache.org>
Betreff: AW: [DISCUSS] Apache TsFile Proposal
So, I gave the text a little run and tweaked a few parts (I do hope I based
that on the latest version discussed here):
# Abstract
TsFile is a columnar storage file format designed for time series data, which
supports efficient compression, high throughput of read and write, and
compatibility with various frameworks, such as Spark and Flink. It is easy to
integrate TsFile into IoT big data processing frameworks.
# Proposal
TsFile is used for managing time series data. Although it is firstly used
inside IoTDB, many users and companies use TsFile directly as a time series
data management solution independently. Besides, there is a growing requirement
for Multi-Language Support of TsFile implementations, such as C++, Go and Rust.
Apache IoTDB community hereby submit this proposal of TsFile as an independent
Apache project. This proposal outlines the key features and benefits of TsFile,
along with the integration plan and the need for multi-language support.
# Background
Time series data is becoming increasingly important in a wide range of
applications, including IoT, intelligent control, finance, log analysis, and
monitoring systems.
TsFile has been developed by the Apache IoTDB community in Java and is
currently located inside the Apache IoTDB code repository. Users could store
time series data using TsFile, then read and analyze it in IoTDB, Spark and
Flink. IoTDB could also generate TsFiles and synchronize TsFiles between two
IoTDB instances.
Furthermore, the demand for TsFile implementations in multiple programming
languages has been growing, as it allows developers to leverage TsFile's
capabilities in their preferred language.
# TsFile offers several distinctive features and benefits:
Efficient Storage and Compression: TsFile employs advanced compression
techniques to minimize storage requirements, resulting in reduced disk space
consumption and improved system efficiency.
Flexible Schema and Metadata Management: TsFile allows for directly write data
without predefining the schema, which is flexible for data acquisition.
High Query Performance with time range: TsFile has indexed devices, sensors,
and time dimensions to accelerate query performance, enabling fast filtering
and retrieval of time series data.
Seamless Integration: TsFile is designed to seamlessly integrate with existing
big data frameworks, such as Spark, Flink and Hadoop.
# Rationale
TsFile is the first existing standard file format for time series data. In the
industry companies usually write time series data without unification or use
general columnar file format such as Parquet and ORC, which makes data
collection and processing complicated without a standard. With TsFile,
organizations could write data in TsFile inside end devices or gateway, then
transfer TsFile to the cloud for unified management in IoTDB and other systems.
In this way, we lower the network transmission and the computing resource
consumption in the cloud.
# Initial Goals
The initial goals include:
Make TsFile an independent project that has its own SDK and documentation that
is easier to use.
Multi-Language Support of TsFile implementations apart from Java, such as C++,
Go and Rust.
Integrate more encoding and compression methods in TsFile.
More tools for TsFile: visualization tool, parsing tool, repair tool.
# Current Status
## Meritocracy
We plan to invite a subset of the Apache IoTDB committers as initial committers
of Apache TsFile. And we would like to follow ASF meritocratic principles and
invite additional developers to participate. We will establish the
documentation and encourage and monitor community participation so that
privileges can be extended to those that contribute.
## Community
The TsFile community is grown from the Apache IoTDB Community. The IoTDB
community is introducing TsFile at many technical conferences. Next, we will
build the mailing list for more convenience, broader communication and archived
discussions. We are open to recruiting more developers from diverse backgrounds.
## Core Developers
TsFile initial PMCs are from IoTDB community: Christofer Dutz, Xiangdong Huang,
Jialin Qiao, Steve Yurong Su, Jinrui Zhang, Yuan Tian, Xinyu Tan, Haonan Hou,
Gaofei Cao, Tian Jiang, Chao Wang(wangchao316), Chao Wang(mychaow), Houliang
Qi, Kun Liu.
These people have extensive experience in building database and data management
system.
## Alignment
The ASF is the natural choice to host the TsFile project as its goal of
encouraging community-driven open-source projects fits with our vision for
TsFile. Additionally, many other projects with which we are familiar with and
expect TsFile to integrate with, such as Apache Spark, Apache Flink and Apache
IoTDB are hosted by the ASF and we will benefit and provide benefits in close
proximity to them.
# Known Risks
## Project Name
TsFile project is used in IoTDB and other scenarios for over 7 years, its name
is unique.
## Orphaned Products
The core developers plan to work full time on the project. There is very little
risk of TsFile being abandoned as it is part of Apache IoTDB's internal
infrastructure. Tsinghua and NEL-BDS Lab relies on TsFile as a platform for
many long-term research projects. Companies such as Timcho, Huawei, BONC,
Yonyou will also participate in this project.
## Inexperience with Open Source
All of the core developers have experience with open-source development. We
have 1 Apache Board member(Christofer Dutz), 2 Apache Members(Xiangdong Huang,
Jialin Qiao) and 11 Apache IoTDB PMCs in TsFile initial PMCs. We have the
experience of the Apache way, such as community over code, license management,
version release, CEV processing, attract committer/pmcs and meetups.
## Length of Incubation
There are enough initial PMCs of TsFile know the ASF process well, so we apply
to go straight to TLP.
## Homogeneous Developers
The current core developers are from diverse groups: Apache IoTDB Community,
Timecho, BONC, Huawei, eBay, Yonyou and Tsinghua University.
## Reliance on Salaried Developers
Currently, the developers are paid to do work at Timecho, BONC, Huawei and
Tsinghua University. Also, we have students and researchers/professors
community in universities, and their researches focus on big data management
and analytics. It is unlikely that they will change their research focus away
from big data management. We will work to ensure that the ability for the
project to continuously be stewarded and to proceed forward independent of
salaried developers is continued.
## Relationships with Other Apache Products
TsFile is used by Apache IoTDB project as the default data file format.
TsFile-Spark-connector and TsFile-Flink-connector have been developed to
support analyzing time series data by using Apache Spark and Flink.
Overall, TsFile is designed as an open architecture, and it can be integrated
with many other systems in the future.
## An Excessive Fascination with the Apache Brand
We respect the reputation of the Apache brand and have no doubt that it will
attract contributors and users. Most of the initial developers come from Apache
IoTDB Community, so we have no intent to use the Apache brand for profit. Our
goal is making TsFile integrating more with Apache projects and letting Apache
community more professional in IoT data management.
# Documentation
The Documentations:
https://iotdb.apache.org/UserGuide/V1.2.x/API/Programming-TsFile-API.html
Examples: https://github.com/apache/iotdb/tree/master/example/tsfile
# Initial Source
TsFile core: https://github.com/apache/iotdb/tree/master/iotdb-core/tsfile
Spark TsFile
Connector：https://github.com/apache/iotdb/blob/master/iotdb-connector/spark-tsfile
Flink TsFile Connector:
https://github.com/apache/iotdb/blob/master/iotdb-connector/flink-tsfile-connector
# Source and Intellectual Property Submission Plan
## External Dependencies
- zstd-jni: BSD 2-Clause License
- logback-classic: Eclipse Public License - v 1.0
- snappy-java: Apache License 2.0
- commons-io: Apache License 2.0
- commons-lang3: Apache License 2.0
- lz4-java: Apache License 2.0
- gson: Apache License 2.0
- Jol-core:
- slice: Apache License 2.0
- xz: Public Domain
- Mock: Apache License 2.0
## Cryptography
All codes are public under Apache license 2.0.
# Required Resources
## Mailing Lists
priv...@tsfile.apache.org<mailto:priv...@tsfile.apache.org>
d...@tsfile.apache.org<mailto:d...@tsfile.apache.org>
comm...@tsfile.apache.org<mailto:comm...@tsfile.apache.org>
## Git Directory
Git is the preferred source control system: git://git.apache.org/tsfile
## Issue Tracking
We will use the github issue for TsFile.
# Initial Committers and PMC
All initial committers will also be members of the PMC
- Yuan Tian (jackietien at apache dot org)
- Chao Wang (wangchao316 at apache dot org)
- Christofer Dutz (cdutz at apache dot org)
- Jinrui Zhang (xingtanzjr at apache dot org)
- Steve Yurong Su (rong at apache dot org)
- Xinyu Tan (tanxinyu at apache dot org)
- Haohan Hou (haonan at apache dot org)
- Gaofei Cao (gaogaofei at apache dot org)
- Jialin Qiao (qiaojialin at apache dot org)
- Kun Liu (liukun at apache dot org)
- Houliang Qi (neuyilan at apache dot org)
- Xiangdong Huang (hxd at apache dot org)
- Chao Wang (chaow at apache dot org)
- Jianmin Wang (jimwang at apache dot org)
- Tian Jiang (jiangtian at apache dot org）
- Xinyi Zhao (zhaoxinyi at apache dot org)
- Shuo Zhang (shuozhagn at apache dot org)
# Chair
We propose to nominate {name once this has been decided} as Chair of the Apache
TsFile PMC.
# Sponsors
Apache IoTDB
# Champion
Christofer Dutz (cdutz at apache dot org)
Von: Xiangdong Huang <saint...@gmail.com>
Datum: Mittwoch, 25. Oktober 2023 um 10:01
An: dev@iotdb.apache.org <dev@iotdb.apache.org>
Betreff: Re: [DISCUSS] Apache TsFile Proposal
> can you add in apache a initial ts-native for the rust project?
I think supporting Rust has been introduced:
- The initial goals include: Multi-Language Support of TsFile
implementations apart from Java, such as C++, Go and Rust.
-----------------------------------
Xiangdong Huang
Giorgio Zoppi <giorgio.zo...@gmail.com> 于2023年10月25日周三 15:46写道：
> Hello,
> can you add in apache a initial ts-native for the rust project? Sorry for
> being pedantic.
> Best Regards,
> Giorgio
>
> Il giorno mer 25 ott 2023 alle ore 05:04 Jialin Qiao <
> qiaojia...@apache.org>
> ha scritto:
>
> > Hi,
> >
> > The TsFile proposal is as follows, feel free to give advice :-)
> >
> > Abstract
> >
> > TsFile is a columnar storage file format designed for time series
> > data, which supports efficient compression, high throughput of read
> > and write, and compatibility with various frameworks, such as Spark
> > and Flink. It is easy to integrate TsFile into IoT big data processing
> > frameworks.
> >
> > Proposal
> >
> > TsFile is used for managing time series data. Although it is firstly
> > used inside IoTDB, many users and companies use TsFile directly as a
> > time series data management solution independently. Besides, there is
> > a growing requirement for Multi-Language Support of TsFile
> > implementations, such as C++, Go and Rust.
> >
> > Apache IoTDB community hereby submit this proposal of TsFile as an
> > independent Apache project. This proposal outlines the key features
> > and benefits of TsFile, along with the integration plan and the need
> > for multi-language support.
> >
> > Background
> >
> > Time series data is becoming increasingly important in a wide range of
> > applications, including IoT, intelligent control, finance, log
> > analysis, and monitoring systems.
> > TsFile has been developed by Apache IoTDB community in Java and is in
> > IoTDB repository. Users could store time series data using TsFile,
> > then read and analyze it in IoTDB, Spark and Flink. IoTDB could also
> > generate TsFiles and syncronize TsFiles between two IoTDB instances.
> > Furthermore, the demand for TsFile implementations in multiple
> > programming languages has been growing, as it allows developers to
> > leverage TsFile's capabilities in their preferred language.
> >
> > TsFile offers several distinctive features and benefits:
> > Efficient Storage and Compression: TsFile employs advanced compression
> > techniques to minimize storage requirements, resulting in reduced disk
> > space consumption and improved system efficiency.
> > Flexible Schema and Metadata Management: TsFile allows for directly
> > write data without pre defining the schema, which is flexible for data
> > aquisition.
> > High Query Performance with time range: TsFile has indexed devices,
> > sensors and time dimensions to accelerate query performance, enabling
> > fast filtering and retrieval of time series data.
> > Seamless Integration: TsFile is designed to seamlessly integrate with
> > existing big data frameworks, such as Spark, Flink and Hadoop.
> >
> >
> > Rationale
> >
> > Before using TsFile, there doesn't exist a file format for time
> > series. The industry companies usually write time series data in
> > various user-defined file format without unification, or use general
> > columnar file format such as Parquet and ORC, which makes data
> > collection and processing complicated without a standard. After
> > TsFile, organizations could write data in TsFile inside end devices or
> > gateway, then transfer TsFile to the cloud for unified management in
> > IoTDB and other systems. In this way, we lower the network
> > transmission and the computing resource consumption in the cloud.
> >
> > Initial Goals
> >
> > The initial goals include:
> >
> > Make TsFile an independent project that has its own SDK and
> > documentation that is easier to use.
> > Multi-Language Support of TsFile implementations apart from Java, such
> > as C++, Go and Rust.
> > Integrate more encoding and compression method in TsFile.
> > More tools for TsFile: visualization tool, parsing tool, repair tool.
> >
> >
> > Current Status
> >
> > Meritocracy
> >
> > We plan to invite the IoTDB committer to be the initial committer of
> > TsFile. And we would like to follow ASF meritocratic principles and
> > invite additional developers to participate. We will establish the
> > documentation and encourage and monitor community participation so
> > that privileges can be extended to those that contribute.
> >
> > Community
> >
> > The TsFile community is grown from the Apache IoTDB Community. The
> > IoTDB community is introducing TsFile at many technical conferences.
> > Next, we will build the mailing list for more convenience, broader
> > communication and archived discussions. We are open to recruiting more
> > developers from diverse backgrounds.
> >
> > Core Developers
> >
> > TsFile initial PMCs are from IoTDB community: Christofer Dutz,
> > Xiangdong Huang, Jialin Qiao, Steve Yurong Su, Jinrui Zhang, Yuan
> > Tian, Xinyu Tan, Haonan Hou, Gaofei Cao, Tian Jiang, Chao
> > Wang(wangchao316), Chao Wang(mychaow), Houliang Qi, Kun Liu.
> > These people has extensive experience in building database and data
> > management system.
> >
> > Alignment
> >
> > The ASF is the natural choice to host the TsFile project as its goal
> > of encouraging community-driven open-source projects fits with our
> > vision for TsFile. Additionally, many other projects with which we are
> > familiar with and expect TsFile to integrate with, such as Apache
> > Spark, Apache Flink and Apache IoTDB are hosted by the ASF and we will
> > benefit and provide benefits in close proximity to them.
> >
> > Known Risks
> >
> > Project Name
> >
> > TsFile project is used in IoTDB and other scenarios for over 7 years,
> > its name is unique.
> >
> > Orphaned Products
> >
> > The core developers plan to work full time on the project. There is
> > very little risk of TsFile being abandoned as it is part of Apache
> > IoTDB's internal infrastructure. Tsinghua and NEL-BDS Lab relies on
> > TsFile as a platform for a large number of long-term research
> > projects. Companies such as Timcho, Huawei, BONC, Yonyou will also
> > participated in this project.
> >
> > Inexperience with Open Source
> >
> > All of the core developers have experience with open source
> > development. We have 1 Apache Board member(Christofer Dutz), 2 Apache
> > Members(Xiangdong Huang, Jialin Qiao) and 11 Apache IoTDB PMCs in
> > TsFile initial PMCs. We have the experience of the Apache way, such as
> > community over code, license management, version release, CEV
> > processing, attract committer/pmcs and meetups.
> >
> > Length of Incubation
> >
> > There are enough initial PMCs of TsFile know the ASF process well, so
> > we apply to go straight to TLP.
> >
> > Homogeneous Developers
> >
> > The current core developers are from diverse groups: Apache IoTDB
> > Community, Timecho, BONC, Huawei, eBay, Yonyou and Tsinghua
> > University.
> >
> > Reliance on Salaried Developers
> >
> > Currently, the developers are paid to do work at Timecho, BONC, Huawei
> > and Tsinghua University. Also we have students and
> > researchers/professors community in universities, and their researches
> > focus on big data management and analytics. It is unlikely that they
> > will change their research focus away from big data management. We
> > will work to ensure that the ability for the project to continuously
> > be stewarded and to proceed forward independent of salaried developers
> > is continued.
> >
> > Relationships with Other Apache Products
> >
> > TsFile is used by Apache IoTDB project as the default data file
> > format. TsFile-Spark-connector and TsFile-Flink-connector have been
> > developed to support analysing time series data by using Apache Spark
> > and Flink.
> > Overall, TsFile is designed as an open architecture, and it can be
> > integrated with many other systems in the future.
> >
> > An Excessive Fascination with the Apache Brand
> >
> > We respect the reputation of the Apache brand and have no doubt that
> > it will attract contributors and users. Most of the initial developers
> > come from Apache IoTDB Community, so we have no intent to use the
> > Apache brand for profit. Our goal is making TsFile integrating more
> > with Apache projects and letting Apache community more professional in
> > IoT data management.
> >
> > Documentation
> >
> > The Documentations:
> >
> https://iotdb.apache.org/UserGuide/V1.2.x/API/Programming-TsFile-API.html
> > Examples: https://github.com/apache/iotdb/tree/master/example/tsfile
> >
> > Initial Source
> >
> > TsFile core:
> https://github.com/apache/iotdb/tree/master/iotdb-core/tsfile
> > Spark TsFile Connector：
> > https://github.com/apache/iotdb/blob/master/iotdb-connector/spark-tsfile
> > Flink TsFile Connector:
> >
> >
> https://github.com/apache/iotdb/blob/master/iotdb-connector/flink-tsfile-connector
> >
> >
> > Source and Intellectual Property Submission Plan
> >
> > External Dependencies
> >
> > zstd-jni: BSD 2-Clause License
> > logback-classic: Eclipse Public License - v 1.0
> > snappy-java: Apache License 2.0
> > commons-io: Apache License 2.0
> > commons-lang3: Apache License 2.0
> > lz4-java: Apache License 2.0
> > gson: Apache License 2.0
> > slice: Apache License 2.0
> > xz: Public Domain
> > Mock: Apache License 2.0
> >
> >
> > Cryptography
> >
> > All codes are public under Apache license 2.0.
> >
> > Required Resources
> >
> >
> > Mailing Lists
> >
> > priv...@tsfile.apache.org
> > d...@tsfile.apache.org
> > comm...@tsfile.apache.org
> >
> > Subversion Directory
> >
> > Git is the preferred source control system: git://git.apache.org/tsfile
> >
> > Issue Tracking
> >
> > JIRA TsFile (TsFile)
> >
> > Initial Committers
> >
> > Yuan Tian (jackietien at apache dot org)
> > Chao Wang (wangchao316 at apache dot org)
> > Christofer Dutz (cdutz at apache dot org)
> > Jinrui Zhang (xingtanzjr at apache dot org)
> > Steve Yurong Su (rong at apache dot org)
> > Xinyu Tan (tanxinyu at apache dot org)
> > Haohan Hou (haonan at apache dot org)
> > Gaofei Cao (gaogaofei at apache dot org)
> > Jialin Qiao (qiaojialin at apache dot org)
> > Kun Liu (liukun at apache dot org)
> > Houliang Qi (neuyilan at apache dot org)
> > Xiangdong Huang (hxd at apache dot org)
> > Chao Wang (chaow at apache dot org)
> > Jianmin Wang (jimwang at apache dot org)
> > Tian Jiang (jiangtian at apache dot org）
> > Xinyi Zhao (zhaoxinyi at apache dot org)
> > Shuo Zhang (shuozhagn at apache dot org)
> >
> > Sponsors
> > Champion:TBD
> > Nominated Mentors:TBD
> >
> >
> > Thanks,
> > -----------------
> > Jialin Qiao
> > Apache IoTDB PMC
> >
>
>
> --
> Life is a chess game - Anonymous.
>
