hi, if ChunJun also plan to support Spark or other runtimes, it will be highly similar to Apache SeaTunnel, have you considered contributing to Apache SeaTunnel?
Best Regards --------------- Apache DolphinScheduler PMC Chair Lidong Dai lidong...@apache.org Linkedin: https://www.linkedin.com/in/dailidong Twitter: @WorkflowEasy <https://twitter.com/WorkflowEasy> --------------- On Fri, Feb 25, 2022 at 2:29 PM LuNing Wang <wang4lun...@gmail.com> wrote: > Hi, > > Thanks a lot Jean-Baptiste Onofré. > > > how the users implement the ETL pipeline ? > > Users implement the ETL pipeline by configuring a json file or SQL. > the following is one of examples: > > https://github.com/DTStack/chunjun/blob/master/flinkx-examples/json/doris/mysql_doris.json > > > > Is it using a kind of SDK/DSL or do you plan an UI ? > > It is using a json file or SQL file. We had a plan to implement an UI > a year ago, but current core developers have no time to implement an > UI and are not good at Web frontend skills. > > > > Is it very coupled to Flink or the runtime is "pluggable" ? Do you plan > to have some kind of optional storage, similar to Apache Druid. > > Nowadays, the ChunJun is coupled to Flink. We plan to support Spark or > other runtimes, but no person in the ChunJun community is good at > Apache Spark. We need to bring in more contributors. > > Best, > LuNing Wang 王鲁宁 > > Daniel Widdis <wid...@gmail.com> 于2022年2月25日周五 11:11写道： > > > > Hi, LuNing. > > > > I am not an IPC member, just an interested open source enthusiast > looking for a project to contribute to. > > > > I was excited about contributing to another project that recently joined > the incubator but as someone who only speaks English, I have had a > challenge when a large number of issues are written in Chinese without > enough translation for me to help. > > > > Reading your proposal I thought this may be another opportunity for me > to help, but I just visited your github site and found that most of the > current open issues are not understandable to me, and while I am > enthusiastic and want to help, I do not see how I can do so. > > > > I am concerned about your ability to gain members of your project who > do not speak your language. > > > > On 2/23/22, 9:15 PM, "LuNing Wang" <wang4lun...@gmail.com> wrote: > > > > Hi, > > > > I am LuNing Wang who sent the Chunjun proposal using ' > apa...@dtstack.com' > > and I'm one of the maintainer of Chunjun project. > > As Apache is a community of peers, I will use this email to reply to > all > > questions and issues in this thread, after I read The Apache > Incubator > > Cookbook. > > > > May I use this email to continue to communicate with the Apache > community > > in this thread? > > > > Best, > > LuNing Wang 王鲁宁 > > > > > > Apache <apa...@dtstack.com> 于2022年2月24日周四 09:57写道： > > > > > > > > > > > ------------------------------------------------------------------ > > > 发件人：Calvin Kirs <k...@apache.org> > > > 发送时间：2022年2月23日(星期三) 00:07 > > > 收件人：general <general@incubator.apache.org> > > > 主 题：Re: Chunjun Proposal > > > > > > Hi, > > > > > > I second with Tison and I'm glad to see your proposal, > > > you must have put a lot of effort into drafting this proposal, > > > but it needs to be clear what you expect and what you know about > Apache[1], > > > and are we are on the same page? > > > > > > > > > You can see the following information: > > > The Apache Incubator Cookbook[2] > > > The Apache Way[3] > > > > > > I'd be happy to help you if you need it. > > > > > > [1]https://www.apache.org/ > > > [2]https://incubator.apache.org/cookbook/ > > > [3]https://www.apache.org/theapacheway/index.html > > > > > > tison <wander4...@gmail.com> 于2022年2月22日周二 23:17写道： > > > > > > > Hi, > > > > > > > > I have two questions here: > > > > > > > > 1. How should I name you in this thread? Apache is a community > of peers. > > > I > > > > can't image I'm talking to the whole Chunjun community or > "Apache" expect > > > > its your name. > > > > 2. What's your expectations on going into the incubator? Among > the whole > > > > proposal it's almost about what the current state of Chunjun and > the only > > > > statement about your expectations is: > > > > > > > > > we seek to further prosper the community with the aid of Apache > > > > > > > > Could you elaborate a bit the motivation here? What help are you > seeking? > > > > > > > > Also I second to Sheng's comment that it's confused about your > > > expressions > > > > of contributors and initial committers. If your community > continuously > > > > promote contributors , why the initial committer list is quite a > bit > > > > limited? > > > > > > > > Best, > > > > tison. > > > > > > > > > > > > Sheng Wu <wu.sheng.841...@gmail.com> 于2022年2月22日周二 20:54写道： > > > > > > > > > I think this description is incorrect. > > > > > > > > > > > Our initial committers will submit iCLA(s), SGA, and CCLA(s). > > > > > > > > > > Committers are individuals, who should only submit ICLA, their > > > > > employers are recommended to submit CCLA, the owner of the > project > > > > > should sign the SGA. > > > > > ___ > > > > > > > > > > Also, I noticed a conflict in your description > > > > > On one side, you mentioned `The initial committers are > employees of > > > > > DTStack.` with only 5 initial committers, and on the other > hand, you > > > > > gave a very long vendor list and core contributors list. > > > > > So, which is an accurate description? If you have those > contributors, > > > > > why were all of them invited as PPMC members? Do you have any > public > > > > > discussion about this decision? > > > > > Such as GitHub ID(demotto) is the #8 in the contributor list, > and also > > > > > listed in the core contributor list, but can't find it in the > initial > > > > > committer list. > > > > > > > > > > > > > > > Sheng Wu 吴晟 > > > > > Twitter, wusheng1108 > > > > > > > > > > Lidong Dai <lidong...@apache.org> 于2022年2月22日周二 20:41写道： > > > > > > > > > > > > I am curious that Chunjun(was Flinkx) is built on the Flink > CDC, so > > > > what > > > > > is > > > > > > its innovation? > > > > > > > > > > > > BTW, you shouldn't use the mail(apa...@dtstack.com) as your > user > > > > > account, > > > > > > apache is a registered trademark of ASF > > > > > > > > > > > > > > > > > > > > > > > > Best Regards > > > > > > > > > > > > > > > > > > > > > > > > --------------- > > > > > > Apache DolphinScheduler PMC Chair > > > > > > Lidong Dai > > > > > > lidong...@apache.org > > > > > > Linkedin: https://www.linkedin.com/in/dailidong > > > > > > Twitter: @WorkflowEasy <https://twitter.com/WorkflowEasy> > > > > > > > > > > > > --------------- > > > > > > > > > > > > > > > > > > On Tue, Feb 22, 2022 at 3:10 PM Jean-Baptiste Onofré < > > > j...@nanthrax.net> > > > > > > wrote: > > > > > > > > > > > > > By the way, if you need a mentor, you can ping me. > > > > > > > > > > > > > > Regards > > > > > > > JB > > > > > > > > > > > > > > On Tue, Feb 22, 2022 at 8:08 AM Jean-Baptiste Onofré < > > > > j...@nanthrax.net> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > It looks interesting. Quick questions about the current > state of > > > > the > > > > > > > > project/roadmap: how the users implement the ETL > pipeline ? Is it > > > > > > > > using a kind of SDK/DSL or do you plan an UI ? > > > > > > > > Is it very coupled to Flink or the runtime is > "pluggable" ? Do > > > you > > > > > > > > plan to have some kind of optional storage, similar to > Apache > > > > Druid, > > > > > > > > ... ? > > > > > > > > > > > > > > > > Regards > > > > > > > > JB > > > > > > > > > > > > > > > > On Tue, Feb 22, 2022 at 7:41 AM Apache < > apa...@dtstack.com> > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Dear Apache Incubator Community, > > > > > > > > > We propose to contribute Chunjun as an Apache Incubator > > > project. > > > > > > > > > We are still looking for possible Champion and Mentors > if > > > anyone > > > > > would > > > > > > > like to volunteer. Thanks a lot. > > > > > > > > > Best Regards, > > > > > > > > > Real-time computing engine team of DTStack. > > > > > > > > > > > > > > > > > > #Chunjun Proposal > > > > > > > > > > > > > > > > > > ##Abstract > > > > > > > > > Chunjun is a distributed ETL tool and distributed data > > > > integration > > > > > > > tool. Currently Chunjun is based on Apache Flink. It was > initially > > > > > known as > > > > > > > FlinkX and renamed Chunjun on February 22, 2022. > > > > > > > > > - Chunjun codebase: https://github.com/DTStack/chunjun > > > > > > > > > > > > > > > > > > ##Proposal > > > > > > > > > We propose to contribute the Chunjun codebase to the > Apache > > > > > Software > > > > > > > Foundation with the intent of forming a productive, > meritocratic > > > and > > > > > open > > > > > > > community around Chunjun’s continued development, > according to the > > > > > 'Apache > > > > > > > Way'. The Chunjun's source code is already under the > Apache License > > > > > Version > > > > > > > 2.0. > > > > > > > > > > > > > > > > > > ##Background > > > > > > > > > We developed Chunjun in DTStack company in 2017, when > we > > > needed a > > > > > > > low-code development and high-performance data integration > tool. It > > > > has > > > > > > > been an open-source project on GitHub since April 2018. > Chunjun is > > > > > running > > > > > > > in DTStack production environment all the time. Chunjun > has also > > > been > > > > > > > widely used by companies in China, including DTStack ( > > > > > > > https://www.dtstack.com/), Qihu360(https://www.360.cn/), > Iflytek ( > > > > > > > https://www.iflytek.com/), XPeng Motors ( > https://en.xiaopeng.com/ > > > ), > > > > > > > WeBank (https://www.webank.com/), Asiainfo( > https://asiainfo.com/), > > > > > Guazi( > > > > > > > https://www.guazi.com/), Hello Inc ( > https://www.hello-inc.com/), > > > > etc. > > > > > > > Nowadays, Chunjun has a strong community in China. > > > > > > > > > > > > > > > > > > ##Rationale > > > > > > > > > High-performance of Chunjun is based on Apache Flink, > and > > > Chunjun > > > > > can > > > > > > > integrate data from different data source. Users only need > to > > > > > configure a > > > > > > > JSON file to complete the data reading, transformer, and > writing. > > > > > Users can > > > > > > > implement new reader/writer plugins to meet their > requirements. > > > > Chunjun > > > > > > > have implemented plugins that can capture data change for > MySQL to > > > > > restore > > > > > > > data for Apache Doris. > > > > > > > > > Chunjun has the following feature: > > > > > > > > > real-time and offline integrate data from different > data > > > sources. > > > > > > > > > change data capture(CDC) to merge restore data. > > > > > > > > > resume from broken-point. > > > > > > > > > capture and collect dirty data. > > > > > > > > > limit data transferring rate. > > > > > > > > > thoughput metrics. > > > > > > > > > capture and restore schema evolution. (TODO) > > > > > > > > > > > > > > > > > > ##Current Status###Meritocracy > > > > > > > > > Since Chunjun was open-sourced, many enterprises have > adopted > > > > > Chunjun > > > > > > > to build up their data integration system. In return, we > have > > > > received > > > > > many > > > > > > > issue reports or enhancements from them simultaneously. The > > > codebase > > > > > is now > > > > > > > mainly managed by the development team inside DTStack who's > > > > > responsible for > > > > > > > building internal data integration system too.###Community > > > > > > > > > Chunjun has been building a community around > contributors and > > > > > users to > > > > > > > this framework for the last five years. We organized one > meetup in > > > > > 2020. > > > > > > > Currently, we communicate in Github issues and in chinese > DingTalk > > > > > group. > > > > > > > There are about 3000 people in this group. And we believe > that we > > > can > > > > > get a > > > > > > > lot of help from the Apache Flink community too. We will > organize a > > > > > meetup > > > > > > > again in 2022.###Core Developers > > > > > > > > > (In alphabetical order) > > > > > > > > > Chao Xu (https://github.com/zoudaokoulife) > > > > > > > > > Gongjiang Tang, (https://github.com/kyo-tom) > > > > > > > > > Huai Yang, (https://github.com/yanghuaiGit) > > > > > > > > > Jiangbo Li, (https://github.com/lijiangbo) > > > > > > > > > Luning Wong, (https://github.com/deadwind4) > > > > > > > > > Luo Li, (https://github.com/kanata163) > > > > > > > > > Sishu Yang, (https://github.com/yangsishu) > > > > > > > > > Tianzhu Wen, (https://github.com/WTZ468071157) > > > > > > > > > Weiliang Hao, (https://github.com/xiuzhu9527) > > > > > > > > > Wenqiang Liu, (https://github.com/meng1222) > > > > > > > > > Xing Liu, (https://github.com/simenliuxing) > > > > > > > > > Yang Lan, (https://github.com/HiLany) > > > > > > > > > Yanquan Lv, (https://github.com/lvyanquan) > > > > > > > > > Yifan Hu, (https://github.com/demotto) > > > > > > > > > Zaiyue Yu, (https://github.com/tonybobam) > > > > > > > > > Zhangwan Zhao, (https://github.com/jiemotongxue) > > > > > > > > > Zhiqiang Li, (https://github.com/ChestnutQiang) > > > > > > > > > They are almost working in real-time computing engine > team of > > > > > DTStack. > > > > > > > Only Yifan Hu working for CaoCao Tech. Most of them are > Apache > > > Flink > > > > > > > contributor. > > > > > > > > > > > > > > > > > > ##Known Risks###Project Name > > > > > > > > > The name of the project is Chunjun. Chunjun comes from > mandarin > > > > > > > chinese Pinyin "Chun Jun", and it is one of the top ten > famous > > > swords > > > > > in > > > > > > > China.###Orphaned products > > > > > > > > > More than 20 contributors and thousands of forks and > star > > > further > > > > > show > > > > > > > that Chunjun is actively supported, and we seek to further > prosper > > > > the > > > > > > > community with the aid of Apache. As a consequence, > Chunjun is > > > > > unlikely to > > > > > > > be reduced to an orphaned project.###Inexperience with > Open Source > > > > > > > > > Many of the Chunjun committers have experience working > on open > > > > > source > > > > > > > projects. They are also active contributors to other Apache > > > projects. > > > > > > > > > ###Homogenous Developers > > > > > > > > > The most of core developers are from DTStack, and > Chunjun > > > > received > > > > > > > some bug fixes and enhancements from other developers not > working > > > at > > > > > > > DTStack. ###Reliance on Salaried Developers > > > > > > > > > Currently, most of core developers are paid to work on > Chunjun > > > > > project > > > > > > > by DTStack. We look forward to attracting more people > outside > > > DTStack > > > > > to > > > > > > > join this project.###Relationships with Other Apache > Products > > > > > > > > > We have integrated with Apache Flink, Apache Hadoop, > Apache > > > > Common > > > > > and > > > > > > > Apache HttpComponents, Log4J and Maven. > > > > > > > > > Usage of Apache projects related to Chunjun plugin > > > > > > > > > Apache Hive > > > > > > > > > Apache Solr > > > > > > > > > Apache Doris > > > > > > > > > Apache HBase > > > > > > > > > Apache Kudu > > > > > > > > > Apache Kafka > > > > > > > > > Apache Pulsar (TODO)###An Excessive Fascination with > the Apache > > > > > Brand > > > > > > > > > We acknowledge the value and reputation that the > Apache brand > > > > would > > > > > > > bring to Chunjun. However, our primary interest is in the > excellent > > > > > > > community provided by Apache Software Foundation, in which > all the > > > > > projects > > > > > > > could gain stability for long-term development. > > > > > > > > > > > > > > > > > > ##Documentation > > > > > > > > > A complete set of documents is provided on GitHub, > including > > > > > English > > > > > > > and Simplified Chinese versions. > > > > > > > > > English: > > > > https://github.com/DTStack/chunjun/blob/master/README.md > > > > > > > > > Chinese: > > > > > https://github.com/DTStack/chunjun/blob/master/README_CH.md > > > > > > > > > > > > > > > > > > ##Initial Code > > > > > > > > > https://github.com/DTStack/chunjun > > > > > > > > > > > > > > > > > > ##Initial Source and Intellectual Property Submission > Plan > > > > > > > > > The codebase is already licensed under the Apache > License 2.0 > > > and > > > > > the > > > > > > > copyright is assigned to DTStack. If the project enters > incubator, > > > > > DTStack > > > > > > > will transfer the source code & trademark ownership to ASF > via a > > > > > Software > > > > > > > Grant Agreement. Our initial committers will submit > iCLA(s), SGA, > > > and > > > > > > > CCLA(s). ##External DependenciesApache-2.0 licenses > > > > > > > > > Apache Avro > > > > > > > > > Apache Commons > > > > > > > > > Apache Curator > > > > > > > > > Apache Flink > > > > > > > > > Apache Hadoop > > > > > > > > > Apache HttpComponents > > > > > > > > > Apache Log4j > > > > > > > > > Gson > > > > > > > > > Guava > > > > > > > > > Jackson > > > > > > > > > Powermock > > > > > > > > > PrometheusEclipse Distribution License > > > > > > > > > JUnitEPL licenses > > > > > > > > > LogbackMIT licenses > > > > > > > > > Mockito > > > > > > > > > SLF4J > > > > > > > > > > > > > > > > > > ##Required Resources ###Git Repositories > > > > > > > > > https://github.com/apache/incubator-chunjun###Issue > Tracking > > > > > > > > > The community would like to continue using GitHub > > > > Issues.###Mailing > > > > > > > List > > > > > > > > > priv...@chunjun.incubator.apache.org > > > > > > > > > d...@chunjun.incubator.apache.org > > > > > > > > > comm...@chunjun.incubator.apache.org###Continuous > <http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > > > <http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > > > > <http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > > > > > <http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > > > > > > > < > http://comm...@chunjun.incubator.apache.org#%23%23Continuous> > > > > > > > Integration tool > > > > > > > > > GitHub Action > > > > > > > > > > > > > > > > > > ##Initial Committers > > > > > > > > > (In alphabetical order) > > > > > > > > > Chao Xu (https://github.com/zoudaokoulife, xuchao at > dtstack > > > dot > > > > > com) > > > > > > > > > Luning Wong (https://github.com/deadwind4, gfeng48 at > gmail > > > dot > > > > > com) > > > > > > > > > Sishu Yang (https://github.com/yangsishu, sishu at > dtstack dot > > > > > com) > > > > > > > > > Yang Huai (https://github.com/yanghuaiGit, dujie at > dtstack > > > dot > > > > > com) > > > > > > > > > Zhiqiang Li (https://github.com/ChestnutQiang, wujuan > at > > > dtstack > > > > > dot > > > > > > > com) > > > > > > > > > > > > > > > > > > > > > > > > > > > ##Affiliations > > > > > > > > > The initial committers are employees of DTStack. The > nominated > > > > > mentors > > > > > > > and champion are employees of TODO. > > > > > > > > > > > > > > > > > > ##Sponsors > > > > > > > > > > > > > > > > > > ###Champion > > > > > > > > > TODO ###Nominated Mentors > > > > > > > > > TODO > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: > general-unsubscr...@incubator.apache.org > > > > > > > For additional commands, e-mail: > general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: > general-unsubscr...@incubator.apache.org > > > > > For additional commands, e-mail: > general-h...@incubator.apache.org > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best wishes！ > > > CalvinKirs > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
