What is the conclusion of this discussion? Are we gonna merge the old source/sink or do we believe that FLIP-27 will be completed in time so that we can merge the new source right away?
Cheers, Till On Tue, Sep 24, 2019 at 3:43 AM 郭士榕 <gsra...@163.com> wrote: > > > unscribe > > > > > > > At 2019-09-24 08:53:25, "Becket Qin" <becket....@gmail.com> wrote: > >Thanks, Stephan. > > > >Sounds good to me. We can still try our best to get new Pulsar connector > in > >Flink 1.10. In case we do not have time to do that, we will prominently > >link the Pulsar connector from the Flink connector docs. > > > >Thanks, > > > >Jiangjie (Becket) Qin > > > >On Mon, Sep 23, 2019 at 4:11 PM Stephan Ewen <se...@apache.org> wrote: > > > >> Okay, I see your point, Becket. > >> > >> Then let us prominently link the Pulsar connector from the Flink > connector > >> docs then, so that users can find it easily. > >> > >> As soon as FLIP 27 is done, we reach out the Pulsar folks to contribute > a > >> new connector. > >> > >> On Mon, Sep 23, 2019 at 3:11 AM Becket Qin <becket....@gmail.com> > wrote: > >> > >> > Hi Stephan, > >> > > >> > I have no doubt about the value of adding Pulsar connector to Flink > repo. > >> > My concern is about how exactly we are going to do it. > >> > > >> > As mentioned before, I believe that we can handle connectors more > >> > > pragmatically and less strict than the core of Flink, if it helps > >> > unlocking > >> > > users. > >> > > >> > I can see the benefit of being less restrict for the initial connector > >> code > >> > adoption. However, I don't think we should be less restrict on the > >> > maintenance commitment once the code is in Flink repo. It only makes > >> sense > >> > to check in something and ask users to use if we plan to maintain it. > >> > > >> > If I understand correctly, the current plan so far is following: > >> > 1. release 1.10 > >> > - Check in Pulsar connector on old interface and label it as beta > >> > version. > >> > - encourage users to try it and report bugs. > >> > 2. release 1.11 > >> > - Check in Pulsar connector on new interface (a.k.a new Pulsar > >> > connector) and label it as beta version > >> > - Deprecate the old Pulsar connector > >> > - Fix bugs reported on old Pulsar connector from release 1.10 > >> > - Ask users to migrate from old Pulsar connector to new Pulsar > >> connector > >> > 3. release 1.12 > >> > - Announce end of support for old Pulsar connector and remove the > code > >> > - Fix bugs reported on new Pulsar connector. > >> > > >> > If this is the plan, it seems neither Flink nor the users trying the > old > >> > Pulsar connector will benefit from this experimental old Pulsar > >> connector, > >> > because whatever feedbacks we got or bugs we fix on the old Pulsar > >> > connector are immediately thrown away in one or two releases. > >> > > >> > If we check in the old Pulsar connector right now, the only option I > see > >> is > >> > to maintain it for a while (e.g. a year or more). IMO, the immediate > >> > deprecation and code removal hurts the users much more than asking > them > >> to > >> > wait for another release. I personally think that we can avoid this > >> > maintenance burden by going directly to the new Pulsar connector, > >> > especially given that users can still use the connector even if they > are > >> > not in Flink repo. That said, I am OK with maintaining both old and > new > >> > Pulsar connector if we believe that having the Pulsar connector > available > >> > right now in Flink repo is more important. > >> > > >> > Thanks, > >> > > >> > Jiangjie (Becket) Qin > >> > > >> > On Sun, Sep 22, 2019 at 9:10 PM Stephan Ewen <se...@apache.org> > wrote: > >> > > >> > > My assumption is as Sijie's, that once the connector is either part > of > >> > > Flink, or part of the streamnative repo. No double maintenance. > >> > > > >> > > I feel this discussion is very much caught in problems that are all > >> > > solvable if we want to solve them. > >> > > Maybe we can think what our goal for users and the communities is? > >> > > > >> > > - Do we want to help build a relationship between the Pulsar and > >> Flink > >> > > open source communities? > >> > > - Will users find a connector in the streamnative repository? > >> > > - Will users trust a connector that is not part of Flink as much? > >> > > > >> > > And then decide what is best according to the overall goals there. > >> > > As mentioned before, I believe that we can handle connectors more > >> > > pragmatically and less strict than the core of Flink, if it helps > >> > unlocking > >> > > users. > >> > > > >> > > Best, > >> > > Stephan > >> > > > >> > > > >> > > > >> > > On Fri, Sep 20, 2019 at 2:10 PM Sijie Guo <guosi...@gmail.com> > wrote: > >> > > > >> > > > Thanks Becket. > >> > > > > >> > > > I think it is better for the Flink community to judge the > benefits of > >> > > doing > >> > > > this. I was trying to provide some views from outsiders. > >> > > > > >> > > > Thanks, > >> > > > Sijie > >> > > > > >> > > > On Fri, Sep 20, 2019 at 10:25 AM Becket Qin <becket....@gmail.com > > > >> > > wrote: > >> > > > > >> > > > > Hi Sijie, > >> > > > > > >> > > > > Yes, we will have to support existing old connectors and new > >> > connectors > >> > > > in > >> > > > > parallel for a while. We have to take that maintenance overhead > >> > because > >> > > > > existing connectors have been used by the users for a long > time. I > >> > > guess > >> > > > It > >> > > > > may take at least a year for us to fully remove the old > connectors. > >> > > > > > >> > > > > Process wise, we can do the same for Pulsar connector. But I am > not > >> > > sure > >> > > > if > >> > > > > we want to have the same burden on Pulsar connector, and I would > >> like > >> > > to > >> > > > > understand the benefit of doing that. > >> > > > > > >> > > > > For users, the benefit of having the old Pulsar connector > checked > >> in > >> > > > seems > >> > > > > limited because 1) that code base will be immediately > deprecated in > >> > the > >> > > > > next release in 3-4 months; 2) users can always use it even if > it > >> is > >> > > not > >> > > > in > >> > > > > the Flink code base. Admittedly it is not as convenient as > having > >> it > >> > in > >> > > > > Flink code base, but doesn't seem super either. And after 3-4 > >> months, > >> > > > users > >> > > > > can just use the new connector in Flink repo. > >> > > > > > >> > > > > For Flink developers, the old connector code base is not > something > >> > that > >> > > > we > >> > > > > want to evolve later. Instead, these code will be deprecated and > >> > > > > removed. So why do we want to get a beta version out to attract > >> > people > >> > > to > >> > > > > use something we don't want to maintain? > >> > > > > > >> > > > > Thanks, > >> > > > > > >> > > > > Jiangjie (Becket) Qin > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Fri, Sep 20, 2019 at 10:12 AM Sijie Guo <guosi...@gmail.com> > >> > wrote: > >> > > > > > >> > > > > > Thanks everyone here. Sorry for jumping into the discussion > here. > >> > > > > > > >> > > > > > I am not very familiar about the deprecation process in Flink. > >> If I > >> > > > > > misunderstood the process, please fix me. > >> > > > > > > >> > > > > > As far as I understand, FLIP-27 is introducing a new unified > API > >> > for > >> > > > > > connectors. After it introduces the new API > >> > > > > > and before moving all the existing connectors from old API to > new > >> > > API, > >> > > > > both > >> > > > > > old ApI and new API will co-exist > >> > > > > > for a while until Flink moves all existing connectors to new > API. > >> > So > >> > > > the > >> > > > > > Pulsar connector (using old API) can > >> > > > > > follow the deprecation process with other connector using old > API > >> > and > >> > > > the > >> > > > > > deprecation of old API, no? > >> > > > > > > >> > > > > > If that's the case, I think contributing the current connector > >> back > >> > > to > >> > > > > > Flink rather than maintaining it outside Flink > >> > > > > > would provide a bit more benefits. We can deprecate the > existing > >> > > > > > streamnative/pulsar-flink repo and point the users > >> > > > > > to use the connector in Flink repo. So all the review > processes > >> > will > >> > > > > happen > >> > > > > > within Flink for both old connector and > >> > > > > > new connector. It also reduces the confusions for the users as > >> the > >> > > > > > documentation and code base happen in one place. > >> > > > > > > >> > > > > > Thoughts? > >> > > > > > > >> > > > > > - Sijie > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > On Fri, Sep 20, 2019 at 12:53 AM Becket Qin < > >> becket....@gmail.com> > >> > > > > wrote: > >> > > > > > > >> > > > > > > Thanks for the explanation, Stephan. I have a few questions > / > >> > > > thoughts. > >> > > > > > > > >> > > > > > > So that means we will remove the old connector without a > major > >> > > > version > >> > > > > > > bump, is that correct? > >> > > > > > > > >> > > > > > > I am not 100% sure if mixing 1.10 connectors with 1.11 > >> connectors > >> > > > will > >> > > > > > > always work because we saw some dependency class collisions > in > >> > the > >> > > > > past. > >> > > > > > To > >> > > > > > > make it safe we may have to maintain the old code for one > more > >> > > > release. > >> > > > > > > > >> > > > > > > To be honest I am still wondering if we have to put the old > >> > > connector > >> > > > > in > >> > > > > > > Flink repo. if we check in the old connector to Flink. We > will > >> > end > >> > > up > >> > > > > in > >> > > > > > > the following situation: > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo. > >> > > > > > > 2. Old connector in Flink Repo, which may be different from > the > >> > one > >> > > > in > >> > > > > > > Pulsar repo. (Added in 1.10, deprecated in 1.11, removed in > >> 1.12) > >> > > > > > > 3. New connector in Flink Repo. > >> > > > > > > > >> > > > > > > We need to think about how to make the users in each case > >> happy. > >> > > > > > > - For users of (1), I assume Sijie and Yijie will have to > >> > maintain > >> > > > the > >> > > > > > code > >> > > > > > > a bit longer for its own compatibility even after we have > (2). > >> In > >> > > > that > >> > > > > > > case, bugs found in old connector may or may not need to be > >> fixed > >> > > in > >> > > > > both > >> > > > > > > Flink and the streamnative/pulsar-flink repo. > >> > > > > > > - For users of (2), will we provide bug fixes? If we do, it > >> will > >> > > be a > >> > > > > > > little awkward because those bug fixes will be immediately > >> > > deprecated > >> > > > > in > >> > > > > > > 1.11, and removed in 1.12. So we are essentially asking > users > >> to > >> > > > > migrate > >> > > > > > > away from the bug fix. After Flink 1.12, users may still > have > >> to > >> > > > switch > >> > > > > > to > >> > > > > > > use (3) due to the potential dependency class conflicts > >> mentioned > >> > > > > above. > >> > > > > > > - Users of (3) have a much easier life and don't need to > worry > >> > too > >> > > > > much. > >> > > > > > > > >> > > > > > > The above story seems a little complicated to tell. I think > it > >> > will > >> > > > be > >> > > > > > much > >> > > > > > > easier to not have (2) at all. > >> > > > > > > 1. Old connector in streamnative/pulsar-flink repo. > >> > > > > > > 3. New connector in Flink Repo. > >> > > > > > > > >> > > > > > > - Old connector will only be maintained in > >> > > streamnative/pulsar-flink > >> > > > > repo > >> > > > > > > until it is fully deprecated. Users can always use the > existing > >> > > > Pulsar > >> > > > > > > connector in that repo. > >> > > > > > > - New connector will be in Flink repo and maintained like > the > >> > other > >> > > > > > > connectors. > >> > > > > > > > >> > > > > > > This seems much simpler for users understand and they will > not > >> be > >> > > > > blocked > >> > > > > > > from using the old connector. If the concern is about the > >> quality > >> > > of > >> > > > > the > >> > > > > > > connector in streamnative/pulsar-flink repo, is it enough > for > >> us > >> > > just > >> > > > > to > >> > > > > > > review the code in streamnative/pulsar-flink connector to > make > >> > sure > >> > > > it > >> > > > > > > looks good from Flink's perspective? > >> > > > > > > > >> > > > > > > What do you think? > >> > > > > > > > >> > > > > > > Thanks, > >> > > > > > > > >> > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > >> > > > > > > > >> > > > > > > On Thu, Sep 19, 2019 at 6:58 PM Stephan Ewen < > se...@apache.org > >> > > >> > > > wrote: > >> > > > > > > > >> > > > > > > > My take would be the following: > >> > > > > > > > > >> > > > > > > > - If we merge the connector now and replace it with a > >> FLIP-27 > >> > > > > version > >> > > > > > > > before the 1.10 release, then we need no deprecation > process > >> > > > > > > > - If we don't manage to replace it with a FLIP-27 > version > >> > > before > >> > > > > the > >> > > > > > > 1.10 > >> > > > > > > > release, than it is good that we have the other version, > so > >> no > >> > > > users > >> > > > > > get > >> > > > > > > > blocked. > >> > > > > > > > > >> > > > > > > > In the latter case we can see how we want to do it. > Immediate > >> > > > removal > >> > > > > > of > >> > > > > > > > the old version or deprecation label and keeping it for > one > >> > more > >> > > > > > release. > >> > > > > > > > Given that you should be able to use a Flink 1.10 > connector > >> > with > >> > > > > Flink > >> > > > > > > 1.11 > >> > > > > > > > as well (stable public APIs) there is also a workaround if > >> you > >> > > need > >> > > > > an > >> > > > > > > old > >> > > > > > > > connector in a newer version. So immediate removal might > even > >> > be > >> > > > > > > feasible. > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Thu, Sep 19, 2019 at 11:09 AM Becket Qin < > >> > > becket....@gmail.com> > >> > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Hi Stephan, > >> > > > > > > > > > >> > > > > > > > > Thanks for the clarification. I completely agree with > you > >> and > >> > > > > Thomas > >> > > > > > on > >> > > > > > > > the > >> > > > > > > > > process of adding connectors to Flink repo. However, I > am > >> > > > wondering > >> > > > > > > what > >> > > > > > > > is > >> > > > > > > > > the deprecation process? Given the main concern here was > >> that > >> > > we > >> > > > > may > >> > > > > > > have > >> > > > > > > > > to maintain two Pulsar connector code bases until the > old > >> one > >> > > is > >> > > > > > > removed > >> > > > > > > > > from the repo, it would be good to know how long we > have to > >> > do > >> > > > > that. > >> > > > > > > > > > >> > > > > > > > > Thanks, > >> > > > > > > > > > >> > > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > > > >> > > > > > > > > On Thu, Sep 19, 2019 at 3:54 PM Stephan Ewen < > >> > se...@apache.org > >> > > > > >> > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Some quick thoughts on the connector contribution > >> process. > >> > I > >> > > > > > > basically > >> > > > > > > > > > reiterate here what Thomas mentioned in another thread > >> > about > >> > > > the > >> > > > > > > > Kinesis > >> > > > > > > > > > connector. > >> > > > > > > > > > > >> > > > > > > > > > For connectors, we should favor a low-overhead > >> contribution > >> > > > > > process, > >> > > > > > > > and > >> > > > > > > > > > accept user code and changes more readily than in the > >> core > >> > > > > system. > >> > > > > > > > > > That is because connectors have both a big variety of > >> > > scenarios > >> > > > > > they > >> > > > > > > > get > >> > > > > > > > > > used in (only through use and many small > contributions do > >> > > they > >> > > > > > become > >> > > > > > > > > > really useful over time) and at the same time, and > >> > committers > >> > > > do > >> > > > > > not > >> > > > > > > > use > >> > > > > > > > > > the connector themselves and usually cannot foresee > too > >> > well > >> > > > what > >> > > > > > is > >> > > > > > > > > > needed. > >> > > > > > > > > > > >> > > > > > > > > > Further more, a missing connector (or connector > feature) > >> is > >> > > > > often a > >> > > > > > > > > bigger > >> > > > > > > > > > show stopper for users than a missing API or system > >> > feature. > >> > > > > > > > > > > >> > > > > > > > > > Along these lines of thougt, the conclusion would be > to > >> > take > >> > > > the > >> > > > > > > Pulsar > >> > > > > > > > > > connector now, focus the review on > >> legal/dependencies/rough > >> > > > code > >> > > > > > > style > >> > > > > > > > > and > >> > > > > > > > > > conventions, label it as "beta" (in the sense of "new > >> code" > >> > > > that > >> > > > > is > >> > > > > > > > "not > >> > > > > > > > > > yet tested through longer use") and go ahead. And then > >> > evolve > >> > > > it > >> > > > > > > > quickly > >> > > > > > > > > > without putting formal blockers in the way, meaning > also > >> > > > adding a > >> > > > > > new > >> > > > > > > > > FLIP > >> > > > > > > > > > 27 version when it is there. > >> > > > > > > > > > > >> > > > > > > > > > Best, > >> > > > > > > > > > Stephan > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > On Thu, Sep 19, 2019 at 3:47 AM Becket Qin < > >> > > > becket....@gmail.com > >> > > > > > > >> > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Hi Yijie, > >> > > > > > > > > > > > >> > > > > > > > > > > Could you please follow the FLIP process to start a > new > >> > > FLIP > >> > > > > > > > > [DISCUSSION] > >> > > > > > > > > > > thread in the mailing list? > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals#FlinkImprovementProposals-Process > >> > > > > > > > > > > > >> > > > > > > > > > > I see two FLIP-69 discussion in the mailing list > now. > >> So > >> > > > there > >> > > > > > is a > >> > > > > > > > > FLIP > >> > > > > > > > > > > number collision. Can you change the FLIP number to > 72? > >> > > > > > > > > > > > >> > > > > > > > > > > Thanks, > >> > > > > > > > > > > > >> > > > > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > > > > > >> > > > > > > > > > > On Thu, Sep 19, 2019 at 12:23 AM Rong Rong < > >> > > > > walter...@gmail.com> > >> > > > > > > > > wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > Hi Yijie, > >> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks for sharing the pulsar FLIP. > >> > > > > > > > > > > > Would you mind enabling comments/suggestions on > the > >> > > google > >> > > > > doc > >> > > > > > > > link? > >> > > > > > > > > > This > >> > > > > > > > > > > > way the contributors from the community can > comment > >> on > >> > > the > >> > > > > doc. > >> > > > > > > > > > > > > >> > > > > > > > > > > > Best, > >> > > > > > > > > > > > Rong > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Mon, Sep 16, 2019 at 5:43 AM Yijie Shen < > >> > > > > > > > > henry.yijies...@gmail.com> > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > > Hello everyone, > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > I've drafted a FLIP that describes the current > >> design > >> > > of > >> > > > > the > >> > > > > > > > Pulsar > >> > > > > > > > > > > > > connector: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit# > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Please take a look and let me know what you > think. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > Yijie > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Sat, Sep 14, 2019 at 12:08 AM Rong Rong < > >> > > > > > > walter...@gmail.com> > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Hi All, > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Sorry for joining the discussion late and > thanks > >> > > Yijie > >> > > > & > >> > > > > > > Sijie > >> > > > > > > > > for > >> > > > > > > > > > > > > driving > >> > > > > > > > > > > > > > the discussion. > >> > > > > > > > > > > > > > I also think the Pulsar connector would be a > very > >> > > > > valuable > >> > > > > > > > > addition > >> > > > > > > > > > > to > >> > > > > > > > > > > > > > Flink. I can also help out a bit on the review > >> side > >> > > :-) > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Regarding the timeline, I also share concerns > >> with > >> > > > Becket > >> > > > > > on > >> > > > > > > > the > >> > > > > > > > > > > > > > relationship between the new Pulsar connector > and > >> > > > > FLIP-27. > >> > > > > > > > > > > > > > There's also another discussion just started > by > >> > > Stephan > >> > > > > on > >> > > > > > > > > dropping > >> > > > > > > > > > > > Kafka > >> > > > > > > > > > > > > > 9/10 support on next Flink release [1]. > Although > >> > the > >> > > > > > > situation > >> > > > > > > > > is > >> > > > > > > > > > > > > somewhat > >> > > > > > > > > > > > > > different, and Kafka 9/10 connector has been > in > >> > Flink > >> > > > for > >> > > > > > > > almost > >> > > > > > > > > > 3-4 > >> > > > > > > > > > > > > years, > >> > > > > > > > > > > > > > based on the discussion I am not sure if a > major > >> > > > version > >> > > > > > > > release > >> > > > > > > > > > is a > >> > > > > > > > > > > > > > requirement for removing old connector > supports. > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > I think there shouldn't be a blocker if we > agree > >> > the > >> > > > old > >> > > > > > > > > connector > >> > > > > > > > > > > will > >> > > > > > > > > > > > > be > >> > > > > > > > > > > > > > removed once FLIP-27 based Pulsar connector is > >> > there. > >> > > > As > >> > > > > > > > Stephan > >> > > > > > > > > > > > stated, > >> > > > > > > > > > > > > it > >> > > > > > > > > > > > > > is easier to contribute the source sooner and > >> > adjust > >> > > it > >> > > > > > > later. > >> > > > > > > > > > > > > > We should also ensure we clearly communicate > the > >> > > > message: > >> > > > > > for > >> > > > > > > > > > > example, > >> > > > > > > > > > > > > > putting an experimental flag on the pre-FLIP27 > >> > > > connector > >> > > > > > page > >> > > > > > > > of > >> > > > > > > > > > the > >> > > > > > > > > > > > > > website, documentations, etc. Any other > thoughts? > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > -- > >> > > > > > > > > > > > > > Rong > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > [1] > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop-older-versions-of-Kafka-Connectors-0-9-0-10-for-Flink-1-10-td29916.html > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Fri, Sep 13, 2019 at 8:15 AM Becket Qin < > >> > > > > > > > becket....@gmail.com > >> > > > > > > > > > > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Technically speaking, removing the old > >> connector > >> > > code > >> > > > > is > >> > > > > > a > >> > > > > > > > > > > backwards > >> > > > > > > > > > > > > > > incompatible change which requires a major > >> > version > >> > > > > bump, > >> > > > > > > i.e. > >> > > > > > > > > > Flink > >> > > > > > > > > > > > > 2.x. > >> > > > > > > > > > > > > > > Given that we don't have a clear plan on > when > >> to > >> > > have > >> > > > > the > >> > > > > > > > next > >> > > > > > > > > > > major > >> > > > > > > > > > > > > > > version release, it seems unclear how long > the > >> > old > >> > > > > > > connector > >> > > > > > > > > code > >> > > > > > > > > > > > will > >> > > > > > > > > > > > > be > >> > > > > > > > > > > > > > > there if we check it in right now. Or will > we > >> > > remove > >> > > > > the > >> > > > > > > old > >> > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > without a major version bump? In any case, > it > >> > > sounds > >> > > > > not > >> > > > > > > > quite > >> > > > > > > > > > user > >> > > > > > > > > > > > > > > friendly to the those who might use the old > >> > Pulsar > >> > > > > > > > connector. I > >> > > > > > > > > > am > >> > > > > > > > > > > > not > >> > > > > > > > > > > > > sure > >> > > > > > > > > > > > > > > if it is worth these potential problems in > >> order > >> > to > >> > > > > have > >> > > > > > > the > >> > > > > > > > > > Pulsar > >> > > > > > > > > > > > > source > >> > > > > > > > > > > > > > > connector checked in one or two months > earlier. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:52 PM Stephan > Ewen < > >> > > > > > > > se...@apache.org > >> > > > > > > > > > > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Agreed, if we check in the old code, we > >> should > >> > > make > >> > > > > it > >> > > > > > > > clear > >> > > > > > > > > > that > >> > > > > > > > > > > > it > >> > > > > > > > > > > > > will > >> > > > > > > > > > > > > > > > be removed as soon as the FLIP-27 based > >> version > >> > > of > >> > > > > the > >> > > > > > > > > > connector > >> > > > > > > > > > > is > >> > > > > > > > > > > > > > > there. > >> > > > > > > > > > > > > > > > We should not commit to maintaining the > old > >> > > > version, > >> > > > > > that > >> > > > > > > > > would > >> > > > > > > > > > > be > >> > > > > > > > > > > > > indeed > >> > > > > > > > > > > > > > > > too much overhead. > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Thu, Sep 12, 2019 at 3:30 AM Becket > Qin < > >> > > > > > > > > > becket....@gmail.com > >> > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Hi Stephan, > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks for the volunteering to help. > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Yes, the overhead would just be review > >> > > capacity. > >> > > > In > >> > > > > > > > fact, I > >> > > > > > > > > > am > >> > > > > > > > > > > > not > >> > > > > > > > > > > > > > > > worrying > >> > > > > > > > > > > > > > > > > too much about the review capacity. > That is > >> > > just > >> > > > a > >> > > > > > one > >> > > > > > > > time > >> > > > > > > > > > > cost. > >> > > > > > > > > > > > > My > >> > > > > > > > > > > > > > > > > concern is mainly about the long term > >> burden. > >> > > > > Assume > >> > > > > > we > >> > > > > > > > > have > >> > > > > > > > > > > new > >> > > > > > > > > > > > > source > >> > > > > > > > > > > > > > > > > interface ready in 1.10 with newly added > >> > Pulsar > >> > > > > > > > connectors > >> > > > > > > > > in > >> > > > > > > > > > > old > >> > > > > > > > > > > > > > > > > interface. Later on if we migrate > Pulsar to > >> > new > >> > > > > > source > >> > > > > > > > > > > interface, > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > old > >> > > > > > > > > > > > > > > > > Pulsar connector might be deprecated > almost > >> > > > > > immediately > >> > > > > > > > > after > >> > > > > > > > > > > > > checked > >> > > > > > > > > > > > > > > in, > >> > > > > > > > > > > > > > > > > but we may still have to maintain two > code > >> > > bases. > >> > > > > For > >> > > > > > > the > >> > > > > > > > > > > > existing > >> > > > > > > > > > > > > > > > > connectors, we have to do that anyways. > But > >> > it > >> > > > > would > >> > > > > > be > >> > > > > > > > > good > >> > > > > > > > > > to > >> > > > > > > > > > > > > avoid > >> > > > > > > > > > > > > > > > > introducing a new connector with the > same > >> > > > problem. > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 6:51 PM Stephan > >> Ewen > >> > < > >> > > > > > > > > > se...@apache.org > >> > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Hi all! > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Nice to see this lively discussion > about > >> > the > >> > > > > Pulsar > >> > > > > > > > > > > connector. > >> > > > > > > > > > > > > > > > > > Some thoughts on the open questions: > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > ## Contribute to Flink or maintain as > a > >> > > > community > >> > > > > > > > package > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Looks like the discussion is more > going > >> > > towards > >> > > > > > > > > > > contribution. I > >> > > > > > > > > > > > > think > >> > > > > > > > > > > > > > > > > that > >> > > > > > > > > > > > > > > > > > is good, especially if we think that > we > >> > want > >> > > to > >> > > > > > > build a > >> > > > > > > > > > > > similarly > >> > > > > > > > > > > > > > > deep > >> > > > > > > > > > > > > > > > > > integration with Pulsar as we have for > >> > > example > >> > > > > with > >> > > > > > > > > Kafka. > >> > > > > > > > > > > The > >> > > > > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > already looks like a more thorough > >> > connector > >> > > > than > >> > > > > > > many > >> > > > > > > > > > others > >> > > > > > > > > > > > we > >> > > > > > > > > > > > > have > >> > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > the repository. > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > With either a repo split, or the new > >> build > >> > > > > system, > >> > > > > > I > >> > > > > > > > hope > >> > > > > > > > > > > that > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > build > >> > > > > > > > > > > > > > > > > > overhead is not a problem. > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > ## Committer Support > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Becket offered some help already, I > can > >> > also > >> > > > > help a > >> > > > > > > > bit. > >> > > > > > > > > I > >> > > > > > > > > > > hope > >> > > > > > > > > > > > > that > >> > > > > > > > > > > > > > > > > > between us, we can cover this. > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > ## Contribute now, or wait for FLIP-27 > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > As Becket said, FLIP-27 is actually > >> making > >> > > some > >> > > > > > > PoC-ing > >> > > > > > > > > > > > > progress, but > >> > > > > > > > > > > > > > > > > will > >> > > > > > > > > > > > > > > > > > take 2 more months, I would estimate, > >> > before > >> > > it > >> > > > > is > >> > > > > > > > fully > >> > > > > > > > > > > > > available. > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > If we want to be on the safe side with > >> the > >> > > > > > > > contribution, > >> > > > > > > > > we > >> > > > > > > > > > > > > should > >> > > > > > > > > > > > > > > > > > contribute the source sooner and > adjust > >> it > >> > > > later. > >> > > > > > > That > >> > > > > > > > > > would > >> > > > > > > > > > > > also > >> > > > > > > > > > > > > > > help > >> > > > > > > > > > > > > > > > us > >> > > > > > > > > > > > > > > > > > in case things get crazy towards the > 1.10 > >> > > > feature > >> > > > > > > > freeze > >> > > > > > > > > > and > >> > > > > > > > > > > it > >> > > > > > > > > > > > > would > >> > > > > > > > > > > > > > > > be > >> > > > > > > > > > > > > > > > > > hard to find time to review the new > >> > changes. > >> > > > > > > > > > > > > > > > > > What would be the overhead of > >> contributing > >> > > now? > >> > > > > > Given > >> > > > > > > > > that > >> > > > > > > > > > > the > >> > > > > > > > > > > > > code > >> > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > already there, it looks like it would > be > >> > only > >> > > > > > review > >> > > > > > > > > > > capacity, > >> > > > > > > > > > > > > right? > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Best, > >> > > > > > > > > > > > > > > > > > Stephan > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 11:04 AM Yijie > >> > Shen < > >> > > > > > > > > > > > > > > henry.yijies...@gmail.com > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Hi everyone! > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for your attention and the > >> > promotion > >> > > > of > >> > > > > > this > >> > > > > > > > > work. > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > We will prepare a FLIP as soon as > >> > possible > >> > > > for > >> > > > > > more > >> > > > > > > > > > > specific > >> > > > > > > > > > > > > > > > > discussions. > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > For FLIP-27, it seems that we have > not > >> > > > reached > >> > > > > a > >> > > > > > > > > > consensus. > >> > > > > > > > > > > > > > > > Therefore, > >> > > > > > > > > > > > > > > > > > > I will explain all the > functionalities > >> of > >> > > the > >> > > > > > > > existing > >> > > > > > > > > > > > > connector in > >> > > > > > > > > > > > > > > > > > > the FLIP (including Source, Sink, > and > >> > > > Catalog) > >> > > > > to > >> > > > > > > > > > continue > >> > > > > > > > > > > > our > >> > > > > > > > > > > > > > > > > > > discussions in FLIP. > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for your kind help. > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Best, > >> > > > > > > > > > > > > > > > > > > Yijie > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Tue, Sep 10, 2019 at 9:57 AM > Becket > >> > Qin > >> > > < > >> > > > > > > > > > > > > becket....@gmail.com> > >> > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Hi Sijie, > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > If we agree that the goal is to > have > >> > > Pulsar > >> > > > > > > > connector > >> > > > > > > > > > in > >> > > > > > > > > > > > > 1.10, > >> > > > > > > > > > > > > > > how > >> > > > > > > > > > > > > > > > > > about > >> > > > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > > do the following: > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > 0. Start a FLIP to add Pulsar > >> connector > >> > > to > >> > > > > > Flink > >> > > > > > > > main > >> > > > > > > > > > > repo > >> > > > > > > > > > > > > as it > >> > > > > > > > > > > > > > > > is a > >> > > > > > > > > > > > > > > > > > new > >> > > > > > > > > > > > > > > > > > > > public interface to Flink main > repo. > >> > > > > > > > > > > > > > > > > > > > 1. Start to review the Pulsar sink > >> > right > >> > > > away > >> > > > > > as > >> > > > > > > > > there > >> > > > > > > > > > is > >> > > > > > > > > > > > no > >> > > > > > > > > > > > > > > change > >> > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > sink interface so far. > >> > > > > > > > > > > > > > > > > > > > 2. Wait a little bit on FLIP-27. > >> Flink > >> > > 1.10 > >> > > > > is > >> > > > > > > > going > >> > > > > > > > > to > >> > > > > > > > > > > be > >> > > > > > > > > > > > > code > >> > > > > > > > > > > > > > > > > freeze > >> > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > late Nov and let's say we give a > >> month > >> > to > >> > > > the > >> > > > > > > > > > development > >> > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > review > >> > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > Pulsar connector, we need to have > >> > FLIP-27 > >> > > > by > >> > > > > > late > >> > > > > > > > > Oct. > >> > > > > > > > > > > > There > >> > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > > still > >> > > > > > > > > > > > > > > > > > 7 > >> > > > > > > > > > > > > > > > > > > > weeks. Personally I think it is > >> doable. > >> > > If > >> > > > > > > FLIP-27 > >> > > > > > > > is > >> > > > > > > > > > not > >> > > > > > > > > > > > > ready > >> > > > > > > > > > > > > > > by > >> > > > > > > > > > > > > > > > > late > >> > > > > > > > > > > > > > > > > > > > Oct, we can review and check in > >> Pulsar > >> > > > > > connector > >> > > > > > > > with > >> > > > > > > > > > the > >> > > > > > > > > > > > > > > existing > >> > > > > > > > > > > > > > > > > > source > >> > > > > > > > > > > > > > > > > > > > interface. This means we will have > >> > Pulsar > >> > > > > > > connector > >> > > > > > > > > in > >> > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > 1.10, > >> > > > > > > > > > > > > > > > > > either > >> > > > > > > > > > > > > > > > > > > > with or without FLIP-27. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Because we are going to have > Pulsar > >> > sink > >> > > > and > >> > > > > > > source > >> > > > > > > > > > > checked > >> > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > separately, > >> > > > > > > > > > > > > > > > > > > > it might make sense to have two > >> FLIPs, > >> > > one > >> > > > > for > >> > > > > > > > Pulsar > >> > > > > > > > > > > sink > >> > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > another > >> > > > > > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > > > > Pulsar source. And we can start > the > >> > work > >> > > on > >> > > > > > > Pulsar > >> > > > > > > > > sink > >> > > > > > > > > > > > right > >> > > > > > > > > > > > > > > away. > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Mon, Sep 9, 2019 at 4:13 PM > Sijie > >> > Guo > >> > > < > >> > > > > > > > > > > > guosi...@gmail.com > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Thank you Bowen and Becket. > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > What's the take from Flink > >> community? > >> > > > Shall > >> > > > > > we > >> > > > > > > > wait > >> > > > > > > > > > for > >> > > > > > > > > > > > > FLIP-27 > >> > > > > > > > > > > > > > > > or > >> > > > > > > > > > > > > > > > > > > shall we > >> > > > > > > > > > > > > > > > > > > > > proceed to next steps? And what > the > >> > > next > >> > > > > > steps > >> > > > > > > > are? > >> > > > > > > > > > :-) > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > > > > > > > Sijie > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 2:43 PM > >> Bowen > >> > > Li < > >> > > > > > > > > > > > > bowenl...@gmail.com> > >> > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Hi, > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I think having a Pulsar > connector > >> > in > >> > > > > Flink > >> > > > > > > can > >> > > > > > > > > be a > >> > > > > > > > > > > > good > >> > > > > > > > > > > > > > > mutual > >> > > > > > > > > > > > > > > > > > > benefit > >> > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > both communities. > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Another perspective is that > >> Pulsar > >> > > > > > connector > >> > > > > > > is > >> > > > > > > > > the > >> > > > > > > > > > > 1st > >> > > > > > > > > > > > > > > > streaming > >> > > > > > > > > > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > > > > > that integrates with Flink's > >> > metadata > >> > > > > > > > management > >> > > > > > > > > > > system > >> > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > Catalog > >> > > > > > > > > > > > > > > > > > > APIs. > >> > > > > > > > > > > > > > > > > > > > > > It'll be cool to see how the > >> > > > integration > >> > > > > > > turns > >> > > > > > > > > out > >> > > > > > > > > > > and > >> > > > > > > > > > > > > > > whether > >> > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > need to > >> > > > > > > > > > > > > > > > > > > > > > improve Flink Catalog stack, > >> which > >> > > are > >> > > > > > > > currently > >> > > > > > > > > in > >> > > > > > > > > > > > > Beta, to > >> > > > > > > > > > > > > > > > > cater > >> > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > streaming source/sink. Thus > I'm > >> in > >> > > > favor > >> > > > > of > >> > > > > > > > > merging > >> > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > > into > >> > > > > > > > > > > > > > > > > > > > > > Flink 1.10. > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > I'd suggest to submit smaller > >> sized > >> > > > PRs, > >> > > > > > e.g. > >> > > > > > > > > maybe > >> > > > > > > > > > > one > >> > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > basic > >> > > > > > > > > > > > > > > > > > > > > > source/sink functionalities > and > >> > > another > >> > > > > for > >> > > > > > > > > schema > >> > > > > > > > > > > and > >> > > > > > > > > > > > > > > catalog > >> > > > > > > > > > > > > > > > > > > > > integration, > >> > > > > > > > > > > > > > > > > > > > > > just to make them easier to > >> review. > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > It doesn't seem to hurt to > wait > >> for > >> > > > > > FLIP-27. > >> > > > > > > > But > >> > > > > > > > > I > >> > > > > > > > > > > > don't > >> > > > > > > > > > > > > > > think > >> > > > > > > > > > > > > > > > > > > FLIP-27 > >> > > > > > > > > > > > > > > > > > > > > > should be a blocker in cases > >> where > >> > it > >> > > > > > cannot > >> > > > > > > > make > >> > > > > > > > > > its > >> > > > > > > > > > > > way > >> > > > > > > > > > > > > > > into > >> > > > > > > > > > > > > > > > > 1.10 > >> > > > > > > > > > > > > > > > > > > or > >> > > > > > > > > > > > > > > > > > > > > > doesn't leave reasonable > amount > >> of > >> > > time > >> > > > > for > >> > > > > > > > > > > committers > >> > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > review > >> > > > > > > > > > > > > > > > > or > >> > > > > > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > > > > > > Pulsar connector to fully > adapt > >> to > >> > > new > >> > > > > > > > > interfaces. > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Bowen > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:21 AM > >> > Becket > >> > > > > Qin < > >> > > > > > > > > > > > > > > > becket....@gmail.com> > >> > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Hi Till, > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > You are right. It all > depends > >> on > >> > > when > >> > > > > the > >> > > > > > > new > >> > > > > > > > > > > source > >> > > > > > > > > > > > > > > > interface > >> > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > going > >> > > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > be ready. Personally I > think it > >> > > would > >> > > > > be > >> > > > > > > > there > >> > > > > > > > > in > >> > > > > > > > > > > > > about a > >> > > > > > > > > > > > > > > > month > >> > > > > > > > > > > > > > > > > > or > >> > > > > > > > > > > > > > > > > > > so. > >> > > > > > > > > > > > > > > > > > > > > > But > >> > > > > > > > > > > > > > > > > > > > > > > I could be too optimistic. > It > >> > would > >> > > > > also > >> > > > > > be > >> > > > > > > > > good > >> > > > > > > > > > to > >> > > > > > > > > > > > > hear > >> > > > > > > > > > > > > > > what > >> > > > > > > > > > > > > > > > > do > >> > > > > > > > > > > > > > > > > > > > > Aljoscha > >> > > > > > > > > > > > > > > > > > > > > > > and Stephan think as they > are > >> > also > >> > > > > > involved > >> > > > > > > > in > >> > > > > > > > > > > > FLIP-27. > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > In general I think we should > >> have > >> > > > > Pulsar > >> > > > > > > > > > connector > >> > > > > > > > > > > in > >> > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > 1.10, > >> > > > > > > > > > > > > > > > > > > > > > > preferably with the new > source > >> > > > > interface. > >> > > > > > > We > >> > > > > > > > > can > >> > > > > > > > > > > also > >> > > > > > > > > > > > > check > >> > > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > right > >> > > > > > > > > > > > > > > > > > > > > > now > >> > > > > > > > > > > > > > > > > > > > > > > with old source interface, > but > >> I > >> > > > > suspect > >> > > > > > > few > >> > > > > > > > > > users > >> > > > > > > > > > > > > will use > >> > > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > > before > >> > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > next official release. > >> Therefore, > >> > > it > >> > > > > > seems > >> > > > > > > > > > > reasonable > >> > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > wait a > >> > > > > > > > > > > > > > > > > > > little > >> > > > > > > > > > > > > > > > > > > > > > bit > >> > > > > > > > > > > > > > > > > > > > > > > to see whether we can jump > to > >> the > >> > > new > >> > > > > > > source > >> > > > > > > > > > > > > interface. As > >> > > > > > > > > > > > > > > > long > >> > > > > > > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > > > > make > >> > > > > > > > > > > > > > > > > > > > > > > sure Flink 1.10 has it, > >> waiting a > >> > > > > little > >> > > > > > > bit > >> > > > > > > > > > > doesn't > >> > > > > > > > > > > > > seem > >> > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > hurt > >> > > > > > > > > > > > > > > > > > > much. > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at 3:59 > PM > >> > Till > >> > > > > > > Rohrmann > >> > > > > > > > < > >> > > > > > > > > > > > > > > > > > trohrm...@apache.org > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'm wondering what the > >> problem > >> > > > would > >> > > > > be > >> > > > > > > if > >> > > > > > > > we > >> > > > > > > > > > > > > committed > >> > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > connector before the new > >> source > >> > > > > > interface > >> > > > > > > > is > >> > > > > > > > > > > ready. > >> > > > > > > > > > > > > If I > >> > > > > > > > > > > > > > > > > > > understood > >> > > > > > > > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > > > > > > > correctly, then we need to > >> > > support > >> > > > > the > >> > > > > > > old > >> > > > > > > > > > source > >> > > > > > > > > > > > > > > interface > >> > > > > > > > > > > > > > > > > > > anyway > >> > > > > > > > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > existing connectors. By > >> > checking > >> > > it > >> > > > > in > >> > > > > > > > early > >> > > > > > > > > I > >> > > > > > > > > > > > could > >> > > > > > > > > > > > > see > >> > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > benefit > >> > > > > > > > > > > > > > > > > > > > > > that > >> > > > > > > > > > > > > > > > > > > > > > > > our users could start > using > >> the > >> > > > > > connector > >> > > > > > > > > > > earlier. > >> > > > > > > > > > > > > > > > Moreover, > >> > > > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > > would > >> > > > > > > > > > > > > > > > > > > > > > > > prevent that the Pulsar > >> > > integration > >> > > > > is > >> > > > > > > > being > >> > > > > > > > > > > > delayed > >> > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > case > >> > > > > > > > > > > > > > > > > > > that the > >> > > > > > > > > > > > > > > > > > > > > > > > source interface should be > >> > > delayed. > >> > > > > The > >> > > > > > > > only > >> > > > > > > > > > > > > downside I > >> > > > > > > > > > > > > > > see > >> > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > extra > >> > > > > > > > > > > > > > > > > > > > > > > > review effort and > potential > >> > fixes > >> > > > > which > >> > > > > > > > might > >> > > > > > > > > > be > >> > > > > > > > > > > > > > > irrelevant > >> > > > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > new > >> > > > > > > > > > > > > > > > > > > > > > > > source interface > >> > implementation. > >> > > I > >> > > > > > guess > >> > > > > > > it > >> > > > > > > > > > > mainly > >> > > > > > > > > > > > > > > depends > >> > > > > > > > > > > > > > > > on > >> > > > > > > > > > > > > > > > > > how > >> > > > > > > > > > > > > > > > > > > > > > certain > >> > > > > > > > > > > > > > > > > > > > > > > > we are when the new source > >> > > > interface > >> > > > > > will > >> > > > > > > > be > >> > > > > > > > > > > ready. > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Cheers, > >> > > > > > > > > > > > > > > > > > > > > > > > Till > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at > 8:56 > >> AM > >> > > > Becket > >> > > > > > > Qin < > >> > > > > > > > > > > > > > > > > > becket....@gmail.com> > >> > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Hi Sijie and Yijie, > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks for sharing your > >> > > thoughts. > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Just want to have some > >> update > >> > > on > >> > > > > > > FLIP-27. > >> > > > > > > > > > > > Although > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > FLIP > >> > > > > > > > > > > > > > > > > > > wiki > >> > > > > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > discussion thread has > been > >> > > quiet > >> > > > > for > >> > > > > > > some > >> > > > > > > > > > > time, a > >> > > > > > > > > > > > > few > >> > > > > > > > > > > > > > > > > > > committer / > >> > > > > > > > > > > > > > > > > > > > > > > > > contributors in Flink > >> > community > >> > > > > were > >> > > > > > > > > actually > >> > > > > > > > > > > > > > > prototyping > >> > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > entire > >> > > > > > > > > > > > > > > > > > > > > > > > thing. > >> > > > > > > > > > > > > > > > > > > > > > > > > We have made some good > >> > progress > >> > > > > there > >> > > > > > > but > >> > > > > > > > > > want > >> > > > > > > > > > > to > >> > > > > > > > > > > > > > > update > >> > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > FLIP > >> > > > > > > > > > > > > > > > > > > > > > wiki > >> > > > > > > > > > > > > > > > > > > > > > > > > after the entire thing > is > >> > > > verified > >> > > > > to > >> > > > > > > > work > >> > > > > > > > > in > >> > > > > > > > > > > > case > >> > > > > > > > > > > > > > > there > >> > > > > > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > > > > some > >> > > > > > > > > > > > > > > > > > > > > > last > >> > > > > > > > > > > > > > > > > > > > > > > > > minute surprise in the > >> > > > > > implementation. > >> > > > > > > I > >> > > > > > > > > > don't > >> > > > > > > > > > > > > have an > >> > > > > > > > > > > > > > > > > exact > >> > > > > > > > > > > > > > > > > > > ETA > >> > > > > > > > > > > > > > > > > > > > > yet, > >> > > > > > > > > > > > > > > > > > > > > > > > but I > >> > > > > > > > > > > > > > > > > > > > > > > > > guess it is going to be > >> > within > >> > > a > >> > > > > > month > >> > > > > > > or > >> > > > > > > > > so. > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > I am happy to review the > >> > > current > >> > > > > > Flink > >> > > > > > > > > Pulsar > >> > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > see if > >> > > > > > > > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > > > > > > > would > >> > > > > > > > > > > > > > > > > > > > > > > > > fit in FLIP-27. It > would be > >> > > good > >> > > > to > >> > > > > > > avoid > >> > > > > > > > > the > >> > > > > > > > > > > > case > >> > > > > > > > > > > > > that > >> > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > checked > >> > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector with > some > >> > > review > >> > > > > > > efforts > >> > > > > > > > > and > >> > > > > > > > > > > > > shortly > >> > > > > > > > > > > > > > > > after > >> > > > > > > > > > > > > > > > > > > that > >> > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > new > >> > > > > > > > > > > > > > > > > > > > > > > > > Source interface is > ready. > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at > 8:39 > >> > AM > >> > > > > Yijie > >> > > > > > > > Shen < > >> > > > > > > > > > > > > > > > > > > > > henry.yijies...@gmail.com > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for all the > >> feedback > >> > > and > >> > > > > > > > > > suggestions! > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > As Sijie said, the > goal > >> of > >> > > the > >> > > > > > > > connector > >> > > > > > > > > > has > >> > > > > > > > > > > > > always > >> > > > > > > > > > > > > > > > been > >> > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > provide > >> > > > > > > > > > > > > > > > > > > > > > > > > > users with the latest > >> > > features > >> > > > of > >> > > > > > > both > >> > > > > > > > > > > systems > >> > > > > > > > > > > > as > >> > > > > > > > > > > > > > > soon > >> > > > > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > > > > > > > possible. > >> > > > > > > > > > > > > > > > > > > > > > > We > >> > > > > > > > > > > > > > > > > > > > > > > > > > propose to contribute > the > >> > > > > connector > >> > > > > > > to > >> > > > > > > > > > Flink > >> > > > > > > > > > > > and > >> > > > > > > > > > > > > hope > >> > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > get > >> > > > > > > > > > > > > > > > > > > more > >> > > > > > > > > > > > > > > > > > > > > > > > > > suggestions and > feedback > >> > from > >> > > > > Flink > >> > > > > > > > > experts > >> > > > > > > > > > > to > >> > > > > > > > > > > > > ensure > >> > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > high > >> > > > > > > > > > > > > > > > > > > > > > > quality > >> > > > > > > > > > > > > > > > > > > > > > > > > > of the connector. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > For FLIP-27, we > noticed > >> its > >> > > > > > existence > >> > > > > > > > at > >> > > > > > > > > > the > >> > > > > > > > > > > > > > > beginning > >> > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > reworking > >> > > > > > > > > > > > > > > > > > > > > > > > > > the connector > >> > implementation > >> > > > > based > >> > > > > > on > >> > > > > > > > > Flink > >> > > > > > > > > > > > 1.9; > >> > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > also > >> > > > > > > > > > > > > > > > > > > wanted > >> > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > > > > build a connector that > >> > > supports > >> > > > > > both > >> > > > > > > > > batch > >> > > > > > > > > > > and > >> > > > > > > > > > > > > stream > >> > > > > > > > > > > > > > > > > > > computing > >> > > > > > > > > > > > > > > > > > > > > > based > >> > > > > > > > > > > > > > > > > > > > > > > > > > on it. > >> > > > > > > > > > > > > > > > > > > > > > > > > > However, it has been > >> > inactive > >> > > > for > >> > > > > > > some > >> > > > > > > > > > time, > >> > > > > > > > > > > so > >> > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > decided > >> > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > provide > >> > > > > > > > > > > > > > > > > > > > > > > > > > a connector with most > of > >> > the > >> > > > new > >> > > > > > > > > features, > >> > > > > > > > > > > such > >> > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > new > >> > > > > > > > > > > > > > > > > > > type > >> > > > > > > > > > > > > > > > > > > > > > > system > >> > > > > > > > > > > > > > > > > > > > > > > > > > and the new catalog > API > >> > > first. > >> > > > We > >> > > > > > > will > >> > > > > > > > > pay > >> > > > > > > > > > > > > attention > >> > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > progress > >> > > > > > > > > > > > > > > > > > > > > > > > > > of FLIP-27 continually > >> and > >> > > > > > > incorporate > >> > > > > > > > it > >> > > > > > > > > > > with > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > > > > > > > > soon > >> > > > > > > > > > > > > > > > > > > > > > > > > > as possible. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Regarding the test > status > >> > of > >> > > > the > >> > > > > > > > > connector, > >> > > > > > > > > > > we > >> > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > > > following > >> > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > other > >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors' test in > Flink > >> > > > > > repository > >> > > > > > > > and > >> > > > > > > > > > > aimed > >> > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > provide > >> > > > > > > > > > > > > > > > > > > > > > throughout > >> > > > > > > > > > > > > > > > > > > > > > > > > > tests as we could. We > are > >> > > also > >> > > > > > happy > >> > > > > > > to > >> > > > > > > > > > hear > >> > > > > > > > > > > > > > > > suggestions > >> > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > > supervision from the > >> Flink > >> > > > > > community > >> > > > > > > to > >> > > > > > > > > > > improve > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > stability and > >> > > > > > > > > > > > > > > > > > > > > > > > > > performance of the > >> > connector > >> > > > > > > > > continuously. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Best, > >> > > > > > > > > > > > > > > > > > > > > > > > > > Yijie > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Sep 5, 2019 at > >> 5:59 > >> > > AM > >> > > > > > Sijie > >> > > > > > > > Guo > >> > > > > > > > > < > >> > > > > > > > > > > > > > > > > > guosi...@gmail.com > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks everyone for > the > >> > > > > comments > >> > > > > > > and > >> > > > > > > > > > > > feedback. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > It seems to me that > the > >> > > main > >> > > > > > > question > >> > > > > > > > > > here > >> > > > > > > > > > > is > >> > > > > > > > > > > > > > > about - > >> > > > > > > > > > > > > > > > > > "how > >> > > > > > > > > > > > > > > > > > > can > >> > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community maintain > the > >> > > > > > connector?". > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Here are two > thoughts > >> > from > >> > > > > > myself. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > 1) I think how and > >> where > >> > to > >> > > > > host > >> > > > > > > this > >> > > > > > > > > > > > > integration > >> > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > kind > >> > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > less > >> > > > > > > > > > > > > > > > > > > > > > > > > > important > >> > > > > > > > > > > > > > > > > > > > > > > > > > > here. I believe > there > >> can > >> > > be > >> > > > > many > >> > > > > > > > ways > >> > > > > > > > > to > >> > > > > > > > > > > > > achieve > >> > > > > > > > > > > > > > > it. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > As part of the > >> > > contribution, > >> > > > > what > >> > > > > > > we > >> > > > > > > > > are > >> > > > > > > > > > > > > looking > >> > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > here > >> > > > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > how > >> > > > > > > > > > > > > > > > > > > > > > > > these > >> > > > > > > > > > > > > > > > > > > > > > > > > > two > >> > > > > > > > > > > > > > > > > > > > > > > > > > > communities can > build > >> the > >> > > > > > > > collaboration > >> > > > > > > > > > > > > > > relationship > >> > > > > > > > > > > > > > > > on > >> > > > > > > > > > > > > > > > > > > > > > developing > >> > > > > > > > > > > > > > > > > > > > > > > > > > > the integration > between > >> > > > Pulsar > >> > > > > > and > >> > > > > > > > > Flink. > >> > > > > > > > > > > > Even > >> > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > can > >> > > > > > > > > > > > > > > > > try > >> > > > > > > > > > > > > > > > > > > our > >> > > > > > > > > > > > > > > > > > > > > > best > >> > > > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > > > > catch > >> > > > > > > > > > > > > > > > > > > > > > > > > > > up all the updates > in > >> > Flink > >> > > > > > > > community. > >> > > > > > > > > We > >> > > > > > > > > > > are > >> > > > > > > > > > > > > still > >> > > > > > > > > > > > > > > > > > > > > > > > > > > facing the fact > that we > >> > > have > >> > > > > less > >> > > > > > > > > > > experiences > >> > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > than > >> > > > > > > > > > > > > > > > > > > > > folks > >> > > > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community. In order > to > >> > make > >> > > > > sure > >> > > > > > we > >> > > > > > > > > > > maintain > >> > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > deliver > >> > > > > > > > > > > > > > > > > > > > > > > > > > > a high-quality > >> > pulsar-flink > >> > > > > > > > integration > >> > > > > > > > > > to > >> > > > > > > > > > > > the > >> > > > > > > > > > > > > > > users > >> > > > > > > > > > > > > > > > > who > >> > > > > > > > > > > > > > > > > > > use > >> > > > > > > > > > > > > > > > > > > > > both > >> > > > > > > > > > > > > > > > > > > > > > > > > > > technologies, we > need > >> > some > >> > > > help > >> > > > > > > from > >> > > > > > > > > the > >> > > > > > > > > > > > > experts > >> > > > > > > > > > > > > > > from > >> > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > community. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > 2) We have been > >> following > >> > > > > FLIP-27 > >> > > > > > > > for a > >> > > > > > > > > > > > while. > >> > > > > > > > > > > > > > > > > Originally > >> > > > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > > > were > >> > > > > > > > > > > > > > > > > > > > > > > > > > thinking > >> > > > > > > > > > > > > > > > > > > > > > > > > > > of contributing the > >> > > > connectors > >> > > > > > back > >> > > > > > > > > after > >> > > > > > > > > > > > > > > integrating > >> > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > new API introduced > in > >> > > > FLIP-27. > >> > > > > > But > >> > > > > > > we > >> > > > > > > > > > > decided > >> > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > initiate > >> > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > conversation > >> > > > > > > > > > > > > > > > > > > > > > > > > > > as early as > possible. > >> > > Because > >> > > > > we > >> > > > > > > > > believe > >> > > > > > > > > > > > there > >> > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > more > >> > > > > > > > > > > > > > > > > > > > > benefits > >> > > > > > > > > > > > > > > > > > > > > > > > doing > >> > > > > > > > > > > > > > > > > > > > > > > > > > > it now rather than > >> later. > >> > > As > >> > > > > part > >> > > > > > > of > >> > > > > > > > > > > > > contribution, > >> > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > can > >> > > > > > > > > > > > > > > > > > > help > >> > > > > > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > community understand > >> more > >> > > > about > >> > > > > > > > Pulsar > >> > > > > > > > > > and > >> > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > potential > >> > > > > > > > > > > > > > > > > > > > > > > integration > >> > > > > > > > > > > > > > > > > > > > > > > > > > points. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Also we can also > help > >> > Flink > >> > > > > > > community > >> > > > > > > > > > > verify > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > new > >> > > > > > > > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > > > > > API > >> > > > > > > > > > > > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > > > > > > > > > > > > well > >> > > > > > > > > > > > > > > > > > > > > > > > > > > as other new API > (e.g. > >> > > > catalog > >> > > > > > > API). > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Sijie > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, 2019 > at > >> > 5:24 > >> > > > AM > >> > > > > > > Becket > >> > > > > > > > > > Qin < > >> > > > > > > > > > > > > > > > > > > > > becket....@gmail.com> > >> > > > > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Yijie, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the > >> interest > >> > > in > >> > > > > > > > > contributing > >> > > > > > > > > > > the > >> > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > connector. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > In general, I > think > >> > > having > >> > > > > > Pulsar > >> > > > > > > > > > > connector > >> > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > strong > >> > > > > > > > > > > > > > > > > > > > > support > >> > > > > > > > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > > > > a > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > valuable addition > to > >> > > Flink. > >> > > > > So > >> > > > > > I > >> > > > > > > am > >> > > > > > > > > > happy > >> > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > shepherd > >> > > > > > > > > > > > > > > > > > > this > >> > > > > > > > > > > > > > > > > > > > > > > effort. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Meanwhile, I would > >> also > >> > > > like > >> > > > > to > >> > > > > > > > > provide > >> > > > > > > > > > > > some > >> > > > > > > > > > > > > > > > context > >> > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > recent > >> > > > > > > > > > > > > > > > > > > > > > > > > > efforts on > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the Flink > connectors > >> > > > > ecosystem. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > The current way > Flink > >> > > > > maintains > >> > > > > > > its > >> > > > > > > > > > > > > connector has > >> > > > > > > > > > > > > > > > hit > >> > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > scalability > >> > > > > > > > > > > > > > > > > > > > > > > > > > bar. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > With more and more > >> > > > connectors > >> > > > > > > > coming > >> > > > > > > > > > into > >> > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > repo, > >> > > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > > > > > > > > > facing a > >> > > > > > > > > > > > > > > > > > > > > > > > > > few > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > problems such as > long > >> > > build > >> > > > > and > >> > > > > > > > > testing > >> > > > > > > > > > > > > time. To > >> > > > > > > > > > > > > > > > > > address > >> > > > > > > > > > > > > > > > > > > this > >> > > > > > > > > > > > > > > > > > > > > > > > > problem, > >> > > > > > > > > > > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > have attempted to > do > >> > the > >> > > > > > > following: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Split out the > >> > > connectors > >> > > > > > into > >> > > > > > > a > >> > > > > > > > > > > separate > >> > > > > > > > > > > > > > > > > repository. > >> > > > > > > > > > > > > > > > > > > This > >> > > > > > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > > > > > > temporarily > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > on hold due to > >> > potential > >> > > > > > solution > >> > > > > > > > to > >> > > > > > > > > > > > shorten > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > build > >> > > > > > > > > > > > > > > > > > > time. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Encourage the > >> > > connectors > >> > > > > to > >> > > > > > > stay > >> > > > > > > > > as > >> > > > > > > > > > > > > ecosystem > >> > > > > > > > > > > > > > > > > > project > >> > > > > > > > > > > > > > > > > > > > > while > >> > > > > > > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > tries > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to provide good > >> support > >> > > for > >> > > > > > > > > > functionality > >> > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > compatibility > >> > > > > > > > > > > > > > > > > > > > > > > tests. > >> > > > > > > > > > > > > > > > > > > > > > > > > > Robert > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > has driven to > create > >> a > >> > > > Flink > >> > > > > > > > > Ecosystem > >> > > > > > > > > > > > > project > >> > > > > > > > > > > > > > > > > website > >> > > > > > > > > > > > > > > > > > > and it > >> > > > > > > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > > > > > going > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > through some final > >> > > approval > >> > > > > > > > process. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Given the above > >> > efforts, > >> > > it > >> > > > > > would > >> > > > > > > > be > >> > > > > > > > > > > great > >> > > > > > > > > > > > to > >> > > > > > > > > > > > > > > first > >> > > > > > > > > > > > > > > > > see > >> > > > > > > > > > > > > > > > > > > if we > >> > > > > > > > > > > > > > > > > > > > > > can > >> > > > > > > > > > > > > > > > > > > > > > > > > have > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar connector > as > >> an > >> > > > > > ecosystem > >> > > > > > > > > > project > >> > > > > > > > > > > > with > >> > > > > > > > > > > > > > > great > >> > > > > > > > > > > > > > > > > > > support. > >> > > > > > > > > > > > > > > > > > > > > It > >> > > > > > > > > > > > > > > > > > > > > > > > would > >> > > > > > > > > > > > > > > > > > > > > > > > > > be > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > good to hear how > the > >> > > Flink > >> > > > > > Pulsar > >> > > > > > > > > > > connector > >> > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > tested > >> > > > > > > > > > > > > > > > > > > > > currently > >> > > > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > > > see > >> > > > > > > > > > > > > > > > > > > > > > > > > > if > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > we can learn > >> something > >> > to > >> > > > > > > maintain > >> > > > > > > > it > >> > > > > > > > > > as > >> > > > > > > > > > > an > >> > > > > > > > > > > > > > > > ecosystem > >> > > > > > > > > > > > > > > > > > > project > >> > > > > > > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > > > > good > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > quality and test > >> > > coverage. > >> > > > If > >> > > > > > the > >> > > > > > > > > > quality > >> > > > > > > > > > > > as > >> > > > > > > > > > > > > an > >> > > > > > > > > > > > > > > > > > ecosystem > >> > > > > > > > > > > > > > > > > > > > > > project > >> > > > > > > > > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > > > > > > hard > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > to guarantee, we > may > >> as > >> > > > well > >> > > > > > > adopt > >> > > > > > > > it > >> > > > > > > > > > > into > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > main > >> > > > > > > > > > > > > > > > > > repo. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > BTW, another > ongoing > >> > > effort > >> > > > > is > >> > > > > > > > > FLIP-27 > >> > > > > > > > > > > > where > >> > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > > > > making > >> > > > > > > > > > > > > > > > > > > > > > > changes > >> > > > > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink source > >> connector > >> > > > > > > architecture > >> > > > > > > > > and > >> > > > > > > > > > > > > > > interface. > >> > > > > > > > > > > > > > > > > This > >> > > > > > > > > > > > > > > > > > > > > change > >> > > > > > > > > > > > > > > > > > > > > > > will > >> > > > > > > > > > > > > > > > > > > > > > > > > > likely > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > land in 1.10. > >> Therefore > >> > > > > timing > >> > > > > > > > wise, > >> > > > > > > > > if > >> > > > > > > > > > > we > >> > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > going > >> > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > have > >> > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > connector in main > >> > repo, I > >> > > > am > >> > > > > > > > > wondering > >> > > > > > > > > > if > >> > > > > > > > > > > > we > >> > > > > > > > > > > > > > > should > >> > > > > > > > > > > > > > > > > > hold > >> > > > > > > > > > > > > > > > > > > a > >> > > > > > > > > > > > > > > > > > > > > > little > >> > > > > > > > > > > > > > > > > > > > > > > > bit > >> > > > > > > > > > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > let the Pulsar > >> > connector > >> > > > > adapt > >> > > > > > to > >> > > > > > > > the > >> > > > > > > > > > new > >> > > > > > > > > > > > > > > interface > >> > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > avoid > >> > > > > > > > > > > > > > > > > > > > > > > > shortly > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > deprecated work? > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) > Qin > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, > 2019 > >> at > >> > > 4:32 > >> > > > > PM > >> > > > > > > > > Chesnay > >> > > > > > > > > > > > > Schepler < > >> > > > > > > > > > > > > > > > > > > > > > > > ches...@apache.org> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm quite > worried > >> > that > >> > > we > >> > > > > may > >> > > > > > > end > >> > > > > > > > > up > >> > > > > > > > > > > > > repeating > >> > > > > > > > > > > > > > > > > > history. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > There were > already > >> 2 > >> > > > > attempts > >> > > > > > > at > >> > > > > > > > > > > > > contributing a > >> > > > > > > > > > > > > > > > > > pulsar > >> > > > > > > > > > > > > > > > > > > > > > > connector, > >> > > > > > > > > > > > > > > > > > > > > > > > > > both > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of which failed > >> > because > >> > > > no > >> > > > > > > > > committer > >> > > > > > > > > > > was > >> > > > > > > > > > > > > > > getting > >> > > > > > > > > > > > > > > > > > > involved, > >> > > > > > > > > > > > > > > > > > > > > > > > despite > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contributor > >> opening a > >> > > > > > dedicated > >> > > > > > > > > > > > discussion > >> > > > > > > > > > > > > > > thread > >> > > > > > > > > > > > > > > > > > > about the > >> > > > > > > > > > > > > > > > > > > > > > > > > > contribution > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > beforehand and > >> > getting > >> > > > > > several > >> > > > > > > > +1's > >> > > > > > > > > > > from > >> > > > > > > > > > > > > > > > > committers. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should really > >> make > >> > > > sure > >> > > > > > that > >> > > > > > > > if > >> > > > > > > > > we > >> > > > > > > > > > > > > > > > > welcome/approve > >> > > > > > > > > > > > > > > > > > > such > >> > > > > > > > > > > > > > > > > > > > > a > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > contribution it > >> will > >> > > > > actually > >> > > > > > > get > >> > > > > > > > > the > >> > > > > > > > > > > > > attention > >> > > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > > > > deserves. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > As such, I'm > >> inclined > >> > > to > >> > > > > > > > recommend > >> > > > > > > > > > > > > maintaining > >> > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > > > > > > > > outside > >> > > > > > > > > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. We could > >> link > >> > to > >> > > > it > >> > > > > > from > >> > > > > > > > the > >> > > > > > > > > > > > > > > documentation > >> > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > give > >> > > > > > > > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > > > > > > more > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > exposure. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > With the > upcoming > >> > page > >> > > > for > >> > > > > > > > sharing > >> > > > > > > > > > > > > artifacts > >> > > > > > > > > > > > > > > > among > >> > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > community > >> > > > > > > > > > > > > > > > > > > > > > > > > > (what's > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the state of > that > >> > > > anyway?), > >> > > > > > > this > >> > > > > > > > > may > >> > > > > > > > > > > be a > >> > > > > > > > > > > > > > > better > >> > > > > > > > > > > > > > > > > > > option. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 04/09/2019 > >> 10:16, > >> > > Till > >> > > > > > > > Rohrmann > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks a lot > for > >> > > > starting > >> > > > > > > this > >> > > > > > > > > > > > discussion > >> > > > > > > > > > > > > > > > Yijie. > >> > > > > > > > > > > > > > > > > I > >> > > > > > > > > > > > > > > > > > > think > >> > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector > would > >> be > >> > a > >> > > > very > >> > > > > > > > > valuable > >> > > > > > > > > > > > > addition > >> > > > > > > > > > > > > > > > since > >> > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > becomes > >> > > > > > > > > > > > > > > > > > > > > > > > > > more > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > more popular > and > >> it > >> > > > would > >> > > > > > > > further > >> > > > > > > > > > > > expand > >> > > > > > > > > > > > > > > > Flink's > >> > > > > > > > > > > > > > > > > > > > > > > > > interoperability. > >> > > > > > > > > > > > > > > > > > > > > > > > > > Also > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from a project > >> > > > > perspective > >> > > > > > it > >> > > > > > > > > makes > >> > > > > > > > > > > > > sense for > >> > > > > > > > > > > > > > > > me > >> > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > place > >> > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the downstream > >> > > project. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > My main > >> > > > concern/question > >> > > > > is > >> > > > > > > how > >> > > > > > > > > can > >> > > > > > > > > > > the > >> > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > community > >> > > > > > > > > > > > > > > > > > > > > > > > maintain > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > connector? We > >> have > >> > > seen > >> > > > > in > >> > > > > > > the > >> > > > > > > > > past > >> > > > > > > > > > > > that > >> > > > > > > > > > > > > > > > > connectors > >> > > > > > > > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > > > > > > > some > >> > > > > > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > most > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actively > >> developed > >> > > > > > components > >> > > > > > > > > > because > >> > > > > > > > > > > > > they > >> > > > > > > > > > > > > > > need > >> > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > be > >> > > > > > > > > > > > > > > > > > > > > kept > >> > > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > > > sync > >> > > > > > > > > > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > external > system > >> and > >> > > > with > >> > > > > > > Flink. > >> > > > > > > > > > Given > >> > > > > > > > > > > > > that > >> > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > community > >> > > > > > > > > > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > willing > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to help with > >> > > > maintaining, > >> > > > > > > > > improving > >> > > > > > > > > > > and > >> > > > > > > > > > > > > > > > evolving > >> > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > connector, > >> > > > > > > > > > > > > > > > > > > > > > > > > I'm > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > optimistic > that > >> we > >> > > can > >> > > > > > > achieve > >> > > > > > > > > > this. > >> > > > > > > > > > > > > Hence, > >> > > > > > > > > > > > > > > +1 > >> > > > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > > > > > > > contributing > >> > > > > > > > > > > > > > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > > > > > > > > > > > > back > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Till > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 4, > >> 2019 > >> > > at > >> > > > > 2:03 > >> > > > > > > AM > >> > > > > > > > > > Sijie > >> > > > > > > > > > > > Guo > >> > > > > > > > > > > > > < > >> > > > > > > > > > > > > > > > > > > > > > guosi...@gmail.com > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hi Yun, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Since I was > the > >> > main > >> > > > > > driver > >> > > > > > > > > behind > >> > > > > > > > > > > > > > > FLINK-9641 > >> > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > FLINK-9168, > >> > > > > > > > > > > > > > > > > > > > > > > > > let > >> > > > > > > > > > > > > > > > > > > > > > > > > > me > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > try to > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> add more > context > >> > on > >> > > > > this. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> FLINK-9641 > and > >> > > > > FLINK-9168 > >> > > > > > > was > >> > > > > > > > > > > created > >> > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > bringing > >> > > > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > > > > > > > > > > > > source > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> sink for > Flink. > >> > The > >> > > > > > > > integration > >> > > > > > > > > > was > >> > > > > > > > > > > > done > >> > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > 1.6.0. > >> > > > > > > > > > > > > > > > > > > > > > > We > >> > > > > > > > > > > > > > > > > > > > > > > > > > sent out > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pull > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> requests > about a > >> > > year > >> > > > > ago > >> > > > > > > and > >> > > > > > > > we > >> > > > > > > > > > > ended > >> > > > > > > > > > > > > up > >> > > > > > > > > > > > > > > > > > > maintaining > >> > > > > > > > > > > > > > > > > > > > > > those > >> > > > > > > > > > > > > > > > > > > > > > > > > > connectors > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar for > >> Pulsar > >> > > > users > >> > > > > to > >> > > > > > > use > >> > > > > > > > > > Flink > >> > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > process > >> > > > > > > > > > > > > > > > > > > event > >> > > > > > > > > > > > > > > > > > > > > > > streams > >> > > > > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> (See > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > https://github.com/apache/pulsar/tree/master/pulsar-flink > >> > > > > > > > > > > > > > > > > > > > > > > > > ). > >> > > > > > > > > > > > > > > > > > > > > > > > > > The > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> 1.6 > integration > >> is > >> > > > > pretty > >> > > > > > > > simple > >> > > > > > > > > > and > >> > > > > > > > > > > > > there > >> > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > no > >> > > > > > > > > > > > > > > > > > > schema > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > considerations. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> In the past > >> year, > >> > we > >> > > > > have > >> > > > > > > > made a > >> > > > > > > > > > lot > >> > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > changes > >> > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > > brought > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Pulsar > schema as > >> > the > >> > > > > > > > first-class > >> > > > > > > > > > > > > citizen in > >> > > > > > > > > > > > > > > > > > Pulsar. > >> > > > > > > > > > > > > > > > > > > We > >> > > > > > > > > > > > > > > > > > > > > > also > >> > > > > > > > > > > > > > > > > > > > > > > > > > integrated > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> other > computing > >> > > > engines > >> > > > > > for > >> > > > > > > > > > > processing > >> > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > event > >> > > > > > > > > > > > > > > > > > > > > > streams > >> > > > > > > > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> schema. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> It led us to > >> > rethink > >> > > > how > >> > > > > > to > >> > > > > > > > > > > integrate > >> > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > best > >> > > > > > > > > > > > > > > > > > > > > > > > > way. > >> > > > > > > > > > > > > > > > > > > > > > > > > > Then > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> reimplement > the > >> > > > > > pulsar-flink > >> > > > > > > > > > > > connectors > >> > > > > > > > > > > > > from > >> > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > ground > >> > > > > > > > > > > > > > > > > > > > > up > >> > > > > > > > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > > > > > schema > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> bring table > API > >> > and > >> > > > > > catalog > >> > > > > > > > API > >> > > > > > > > > as > >> > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > first-class > >> > > > > > > > > > > > > > > > > > > > > citizen > >> > > > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. > >> With > >> > > that > >> > > > > > being > >> > > > > > > > > said, > >> > > > > > > > > > > in > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > new > >> > > > > > > > > > > > > > > > > > > > > pulsar-flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > implementation, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> you can > register > >> > > > pulsar > >> > > > > > as a > >> > > > > > > > > flink > >> > > > > > > > > > > > > catalog > >> > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > query / > >> > > > > > > > > > > > > > > > > > > > > > > process > >> > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > event > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> streams using > >> > Flink > >> > > > SQL. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> This is an > >> example > >> > > > about > >> > > > > > how > >> > > > > > > > to > >> > > > > > > > > > use > >> > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > > a > >> > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > catalog: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://github.com/streamnative/pulsar-flink/blob/3eeddec5625fc7dddc3f8a3ec69f72e1614ca9c9/README.md#use-pulsar-catalog > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Yijie has > also > >> > > > written a > >> > > > > > > blog > >> > > > > > > > > post > >> > > > > > > > > > > > > > > explaining > >> > > > > > > > > > > > > > > > > why > >> > > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > > > > > > > re-implement > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector > with > >> > Flink > >> > > > 1.9 > >> > > > > > and > >> > > > > > > > > what > >> > > > > > > > > > > are > >> > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > changes > >> > > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > > > made > >> > > > > > > > > > > > > > > > > > > > > > in > >> > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > new > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> connector: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://medium.com/streamnative/use-apache-pulsar-as-streaming-table-with-8-lines-of-code-39033a93947f > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> We believe > >> Pulsar > >> > is > >> > > > not > >> > > > > > > just > >> > > > > > > > a > >> > > > > > > > > > > simple > >> > > > > > > > > > > > > data > >> > > > > > > > > > > > > > > > sink > >> > > > > > > > > > > > > > > > > > or > >> > > > > > > > > > > > > > > > > > > > > source > >> > > > > > > > > > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > It > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> actually can > be > >> a > >> > > > fully > >> > > > > > > > > integrated > >> > > > > > > > > > > > > streaming > >> > > > > > > > > > > > > > > > > data > >> > > > > > > > > > > > > > > > > > > > > storage > >> > > > > > > > > > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink in > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > many > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> areas (sink, > >> > source, > >> > > > > > > > > > schema/catalog > >> > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > state). > >> > > > > > > > > > > > > > > > > > The > >> > > > > > > > > > > > > > > > > > > > > > > > combination > >> > > > > > > > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> and Pulsar > can > >> > > create > >> > > > a > >> > > > > > > great > >> > > > > > > > > > > > streaming > >> > > > > > > > > > > > > > > > > warehouse > >> > > > > > > > > > > > > > > > > > > > > > > architecture > >> > > > > > > > > > > > > > > > > > > > > > > > > for > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > streaming-first, > >> > > > unified > >> > > > > > > data > >> > > > > > > > > > > > > processing. > >> > > > > > > > > > > > > > > > Since > >> > > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > > > > > > > > talking > >> > > > > > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> contribute > >> Pulsar > >> > > > > > > integration > >> > > > > > > > to > >> > > > > > > > > > > Flink > >> > > > > > > > > > > > > here, > >> > > > > > > > > > > > > > > > we > >> > > > > > > > > > > > > > > > > > are > >> > > > > > > > > > > > > > > > > > > also > >> > > > > > > > > > > > > > > > > > > > > > > > > > dedicated to > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> maintain, > >> improve > >> > > and > >> > > > > > evolve > >> > > > > > > > the > >> > > > > > > > > > > > > integration > >> > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > > help > >> > > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > users > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> who use both > >> Flink > >> > > and > >> > > > > > > Pulsar. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Hope this > give > >> > you a > >> > > > bit > >> > > > > > > more > >> > > > > > > > > > > > background > >> > > > > > > > > > > > > > > about > >> > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> integration. > Let > >> > me > >> > > > know > >> > > > > > > what > >> > > > > > > > > are > >> > > > > > > > > > > your > >> > > > > > > > > > > > > > > > thoughts. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Thanks, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Sijie > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On Tue, Sep > 3, > >> > 2019 > >> > > at > >> > > > > > 11:54 > >> > > > > > > > AM > >> > > > > > > > > > Yun > >> > > > > > > > > > > > > Tang < > >> > > > > > > > > > > > > > > > > > > > > > myas...@live.com> > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Hi Yijie > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I can see > that > >> > > Pulsar > >> > > > > > > becomes > >> > > > > > > > > > more > >> > > > > > > > > > > > and > >> > > > > > > > > > > > > more > >> > > > > > > > > > > > > > > > > > popular > >> > > > > > > > > > > > > > > > > > > > > > > recently > >> > > > > > > > > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > > very > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> glad > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> to see more > >> > people > >> > > > > > willing > >> > > > > > > to > >> > > > > > > > > > > > > contribute to > >> > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > ecosystem. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Before any > >> > further > >> > > > > > > > discussion, > >> > > > > > > > > > > would > >> > > > > > > > > > > > > you > >> > > > > > > > > > > > > > > > please > >> > > > > > > > > > > > > > > > > > > give > >> > > > > > > > > > > > > > > > > > > > > some > >> > > > > > > > > > > > > > > > > > > > > > > > > > explanation > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> the > >> relationship > >> > > > > between > >> > > > > > > this > >> > > > > > > > > > > thread > >> > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > current > >> > > > > > > > > > > > > > > > > > > > > existing > >> > > > > > > > > > > > > > > > > > > > > > > > JIRAs > >> > > > > > > > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> source [1] > and > >> > sink > >> > > > [2] > >> > > > > > > > > > connector? > >> > > > > > > > > > > > > Will the > >> > > > > > > > > > > > > > > > > > > > > contribution > >> > > > > > > > > > > > > > > > > > > > > > > > > contains > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > part > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> those PRs or > >> > > totally > >> > > > > > > > different > >> > > > > > > > > > > > > > > > implementation? > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > >> > > > > > > > > > > > > > > > > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-9641 > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > >> > > > > > > > > > > > > > > > > > > > >> > > > > https://issues.apache.org/jira/browse/FLINK-9168 > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yun Tang > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > ________________________________ > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> From: Yijie > >> Shen > >> > < > >> > > > > > > > > > > > > > > henry.yijies...@gmail.com> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Sent: > Tuesday, > >> > > > > September > >> > > > > > 3, > >> > > > > > > > > 2019 > >> > > > > > > > > > > > 13:57 > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> To: > >> > > > > dev@flink.apache.org > >> > > > > > < > >> > > > > > > > > > > > > > > > dev@flink.apache.org > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Subject: > >> > [DISCUSS] > >> > > > > > > Contribute > >> > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > > > > back > >> > > > > > > > > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Dear Flink > >> > > Community! > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> I would > like to > >> > > open > >> > > > > the > >> > > > > > > > > > discussion > >> > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > contributing > >> > > > > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > > Flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> connector > [0] > >> > back > >> > > to > >> > > > > > > Flink. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## A brief > >> > > > introduction > >> > > > > > to > >> > > > > > > > > Apache > >> > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Apache > >> Pulsar[1] > >> > > is a > >> > > > > > > > > > multi-tenant, > >> > > > > > > > > > > > > > > > > > > high-performance > >> > > > > > > > > > > > > > > > > > > > > > > > > distributed > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> pub-sub > >> messaging > >> > > > > system. > >> > > > > > > > > Pulsar > >> > > > > > > > > > > > > includes > >> > > > > > > > > > > > > > > > > > multiple > >> > > > > > > > > > > > > > > > > > > > > > features > >> > > > > > > > > > > > > > > > > > > > > > > > > such > >> > > > > > > > > > > > > > > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> native > support > >> > for > >> > > > > > multiple > >> > > > > > > > > > > clusters > >> > > > > > > > > > > > > in a > >> > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > instance, > >> > > > > > > > > > > > > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> seamless > >> > > > > geo-replication > >> > > > > > of > >> > > > > > > > > > > messages > >> > > > > > > > > > > > > across > >> > > > > > > > > > > > > > > > > > > clusters, > >> > > > > > > > > > > > > > > > > > > > > > very > >> > > > > > > > > > > > > > > > > > > > > > > > low > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > publish > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and > end-to-end > >> > > > latency, > >> > > > > > > > > seamless > >> > > > > > > > > > > > > > > scalability > >> > > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > over a > >> > > > > > > > > > > > > > > > > > > > > > > > million > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > topics, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> and > guaranteed > >> > > > message > >> > > > > > > > delivery > >> > > > > > > > > > > with > >> > > > > > > > > > > > > > > > persistent > >> > > > > > > > > > > > > > > > > > > message > >> > > > > > > > > > > > > > > > > > > > > > > > storage > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> provided by > >> > Apache > >> > > > > > > > BookKeeper. > >> > > > > > > > > > > > > Nowadays, > >> > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > has > >> > > > > > > > > > > > > > > > > > > > > been > >> > > > > > > > > > > > > > > > > > > > > > > > > adopted > >> > > > > > > > > > > > > > > > > > > > > > > > > > by > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> more and > more > >> > > > > > companies[2]. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## The > status > >> of > >> > > > Pulsar > >> > > > > > > Flink > >> > > > > > > > > > > > connector > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> The Pulsar > >> Flink > >> > > > > > connector > >> > > > > > > we > >> > > > > > > > > are > >> > > > > > > > > > > > > planning > >> > > > > > > > > > > > > > > to > >> > > > > > > > > > > > > > > > > > > > > contribute > >> > > > > > > > > > > > > > > > > > > > > > is > >> > > > > > > > > > > > > > > > > > > > > > > > > built > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > upon > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Flink 1.9.0 > and > >> > > > Pulsar > >> > > > > > > 2.4.0. > >> > > > > > > > > The > >> > > > > > > > > > > > main > >> > > > > > > > > > > > > > > > features > >> > > > > > > > > > > > > > > > > > > are: > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Pulsar as > a > >> > > > streaming > >> > > > > > > > source > >> > > > > > > > > > with > >> > > > > > > > > > > > > > > > > exactly-once > >> > > > > > > > > > > > > > > > > > > > > > guarantee. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Sink > >> streaming > >> > > > > results > >> > > > > > to > >> > > > > > > > > > Pulsar > >> > > > > > > > > > > > with > >> > > > > > > > > > > > > > > > > > > at-least-once > >> > > > > > > > > > > > > > > > > > > > > > > > > semantics. > >> > > > > > > > > > > > > > > > > > > > > > > > > > (We > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> would update > >> this > >> > > to > >> > > > > > > > > exactly-once > >> > > > > > > > > > > as > >> > > > > > > > > > > > > well > >> > > > > > > > > > > > > > > > when > >> > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > gets > >> > > > > > > > > > > > > > > > > > > > > > > > all > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> transaction > >> > > features > >> > > > > > ready > >> > > > > > > in > >> > > > > > > > > its > >> > > > > > > > > > > > 2.5.0 > >> > > > > > > > > > > > > > > > > version) > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Build upon > >> > Flink > >> > > > new > >> > > > > > > Table > >> > > > > > > > > API > >> > > > > > > > > > > Type > >> > > > > > > > > > > > > > > system > >> > > > > > > > > > > > > > > > > > > > > > (FLIP-37[3]), > >> > > > > > > > > > > > > > > > > > > > > > > > and > >> > > > > > > > > > > > > > > > > > > > > > > > > > can > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > automatically > >> > > > > > (de)serialize > >> > > > > > > > > > > messages > >> > > > > > > > > > > > > with > >> > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > help > >> > > > > > > > > > > > > > > > > > > of > >> > > > > > > > > > > > > > > > > > > > > > > Pulsar > >> > > > > > > > > > > > > > > > > > > > > > > > > > schema. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> - Integrate > >> with > >> > > > Flink > >> > > > > > new > >> > > > > > > > > > Catalog > >> > > > > > > > > > > > API > >> > > > > > > > > > > > > > > > > > > (FLIP-30[4]), > >> > > > > > > > > > > > > > > > > > > > > > which > >> > > > > > > > > > > > > > > > > > > > > > > > > > enables > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> use of > Pulsar > >> > > topics > >> > > > as > >> > > > > > > > tables > >> > > > > > > > > in > >> > > > > > > > > > > > > Table API > >> > > > > > > > > > > > > > > > as > >> > > > > > > > > > > > > > > > > > > well as > >> > > > > > > > > > > > > > > > > > > > > > SQL > >> > > > > > > > > > > > > > > > > > > > > > > > > > client. > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> ## Reference > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [0] > >> > > > > > > > > > > > > > > > > > >> https://github.com/streamnative/pulsar-flink > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [1] > >> > > > > > > > https://pulsar.apache.org/ > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [2] > >> > > > > > > > > > > > > > > https://pulsar.apache.org/en/powered-by/ > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [3] > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> [4] > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Best, > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> Yijie Shen > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >>> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> >
