Hi, It looks interesting. Quick questions about the current state of the project/roadmap: how the users implement the ETL pipeline ? Is it using a kind of SDK/DSL or do you plan an UI ? Is it very coupled to Flink or the runtime is "pluggable" ? Do you plan to have some kind of optional storage, similar to Apache Druid, ... ?
Regards JB On Tue, Feb 22, 2022 at 7:41 AM Apache <apa...@dtstack.com> wrote: > > > Dear Apache Incubator Community, > We propose to contribute Chunjun as an Apache Incubator project. > We are still looking for possible Champion and Mentors if anyone would like > to volunteer. Thanks a lot. > Best Regards, > Real-time computing engine team of DTStack. > > #Chunjun Proposal > > ##Abstract > Chunjun is a distributed ETL tool and distributed data integration tool. > Currently Chunjun is based on Apache Flink. It was initially known as FlinkX > and renamed Chunjun on February 22, 2022. > - Chunjun codebase: https://github.com/DTStack/chunjun > > ##Proposal > We propose to contribute the Chunjun codebase to the Apache Software > Foundation with the intent of forming a productive, meritocratic and open > community around Chunjun’s continued development, according to the 'Apache > Way'. The Chunjun's source code is already under the Apache License Version > 2.0. > > ##Background > We developed Chunjun in DTStack company in 2017, when we needed a low-code > development and high-performance data integration tool. It has been an > open-source project on GitHub since April 2018. Chunjun is running in DTStack > production environment all the time. Chunjun has also been widely used by > companies in China, including DTStack (https://www.dtstack.com/), > Qihu360(https://www.360.cn/), Iflytek (https://www.iflytek.com/), XPeng > Motors (https://en.xiaopeng.com/), WeBank (https://www.webank.com/), > Asiainfo(https://asiainfo.com/), Guazi(https://www.guazi.com/), Hello Inc > (https://www.hello-inc.com/), etc. Nowadays, Chunjun has a strong community > in China. > > ##Rationale > High-performance of Chunjun is based on Apache Flink, and Chunjun can > integrate data from different data source. Users only need to configure a > JSON file to complete the data reading, transformer, and writing. Users can > implement new reader/writer plugins to meet their requirements. Chunjun have > implemented plugins that can capture data change for MySQL to restore data > for Apache Doris. > Chunjun has the following feature: > real-time and offline integrate data from different data sources. > change data capture(CDC) to merge restore data. > resume from broken-point. > capture and collect dirty data. > limit data transferring rate. > thoughput metrics. > capture and restore schema evolution. (TODO) > > ##Current Status###Meritocracy > Since Chunjun was open-sourced, many enterprises have adopted Chunjun to > build up their data integration system. In return, we have received many > issue reports or enhancements from them simultaneously. The codebase is now > mainly managed by the development team inside DTStack who's responsible for > building internal data integration system too.###Community > Chunjun has been building a community around contributors and users to this > framework for the last five years. We organized one meetup in 2020. > Currently, we communicate in Github issues and in chinese DingTalk group. > There are about 3000 people in this group. And we believe that we can get a > lot of help from the Apache Flink community too. We will organize a meetup > again in 2022.###Core Developers > (In alphabetical order) > Chao Xu (https://github.com/zoudaokoulife) > Gongjiang Tang, (https://github.com/kyo-tom) > Huai Yang, (https://github.com/yanghuaiGit) > Jiangbo Li, (https://github.com/lijiangbo) > Luning Wong, (https://github.com/deadwind4) > Luo Li, (https://github.com/kanata163) > Sishu Yang, (https://github.com/yangsishu) > Tianzhu Wen, (https://github.com/WTZ468071157) > Weiliang Hao, (https://github.com/xiuzhu9527) > Wenqiang Liu, (https://github.com/meng1222) > Xing Liu, (https://github.com/simenliuxing) > Yang Lan, (https://github.com/HiLany) > Yanquan Lv, (https://github.com/lvyanquan) > Yifan Hu, (https://github.com/demotto) > Zaiyue Yu, (https://github.com/tonybobam) > Zhangwan Zhao, (https://github.com/jiemotongxue) > Zhiqiang Li, (https://github.com/ChestnutQiang) > They are almost working in real-time computing engine team of DTStack. Only > Yifan Hu working for CaoCao Tech. Most of them are Apache Flink contributor. > > ##Known Risks###Project Name > The name of the project is Chunjun. Chunjun comes from mandarin chinese > Pinyin "Chun Jun", and it is one of the top ten famous swords in > China.###Orphaned products > More than 20 contributors and thousands of forks and star further show that > Chunjun is actively supported, and we seek to further prosper the community > with the aid of Apache. As a consequence, Chunjun is unlikely to be reduced > to an orphaned project.###Inexperience with Open Source > Many of the Chunjun committers have experience working on open source > projects. They are also active contributors to other Apache projects. > ###Homogenous Developers > The most of core developers are from DTStack, and Chunjun received some bug > fixes and enhancements from other developers not working at DTStack. > ###Reliance on Salaried Developers > Currently, most of core developers are paid to work on Chunjun project by > DTStack. We look forward to attracting more people outside DTStack to join > this project.###Relationships with Other Apache Products > We have integrated with Apache Flink, Apache Hadoop, Apache Common and Apache > HttpComponents, Log4J and Maven. > Usage of Apache projects related to Chunjun plugin > Apache Hive > Apache Solr > Apache Doris > Apache HBase > Apache Kudu > Apache Kafka > Apache Pulsar (TODO)###An Excessive Fascination with the Apache Brand > We acknowledge the value and reputation that the Apache brand would bring to > Chunjun. However, our primary interest is in the excellent community provided > by Apache Software Foundation, in which all the projects could gain stability > for long-term development. > > ##Documentation > A complete set of documents is provided on GitHub, including English and > Simplified Chinese versions. > English: https://github.com/DTStack/chunjun/blob/master/README.md > Chinese: https://github.com/DTStack/chunjun/blob/master/README_CH.md > > ##Initial Code > https://github.com/DTStack/chunjun > > ##Initial Source and Intellectual Property Submission Plan > The codebase is already licensed under the Apache License 2.0 and the > copyright is assigned to DTStack. If the project enters incubator, DTStack > will transfer the source code & trademark ownership to ASF via a Software > Grant Agreement. Our initial committers will submit iCLA(s), SGA, and > CCLA(s). ##External DependenciesApache-2.0 licenses > Apache Avro > Apache Commons > Apache Curator > Apache Flink > Apache Hadoop > Apache HttpComponents > Apache Log4j > Gson > Guava > Jackson > Powermock > PrometheusEclipse Distribution License > JUnitEPL licenses > LogbackMIT licenses > Mockito > SLF4J > > ##Required Resources ###Git Repositories > https://github.com/apache/incubator-chunjun###Issue Tracking > The community would like to continue using GitHub Issues.###Mailing List > priv...@chunjun.incubator.apache.org > d...@chunjun.incubator.apache.org > comm...@chunjun.incubator.apache.org###Continuous Integration tool > GitHub Action > > ##Initial Committers > (In alphabetical order) > Chao Xu (https://github.com/zoudaokoulife, xuchao at dtstack dot com) > Luning Wong (https://github.com/deadwind4, gfeng48 at gmail dot com) > Sishu Yang (https://github.com/yangsishu, sishu at dtstack dot com) > Yang Huai (https://github.com/yanghuaiGit, dujie at dtstack dot com) > Zhiqiang Li (https://github.com/ChestnutQiang, wujuan at dtstack dot com) > > > ##Affiliations > The initial committers are employees of DTStack. The nominated mentors and > champion are employees of TODO. > > ##Sponsors > > ###Champion > TODO ###Nominated Mentors > TODO --------------------------------------------------------------------- To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
