Hi Till, 1) Who from the Flink community will mentor this effort and could take > responsibility for it? >
I'd be happy to mentor the transition. It remains to be seen who is doing mainly the maintenance in the long run. If all fails, I can also take that over but I was hoping that the respective connector authors would also become Flink contributors/committers in the long run. 2) How can Pulsar be tested together with Flink (is there a Pulsar mini >> cluster for IT tests)? >> > > We use testcontainer for running a Pulsar standalone cluster to do the > integration tests. If it is not suitable for Flink project, we are happy to > adopt any technologies the Flink community prefers. > We currently try to establish testcontainers for some external systems, so this sounds perfect to me. 3) Which Pulsar versions will/can be supported? >> > > Pulsar has a pretty good backward compatibility between versions. In our > experience, people have been running the connector with Pulsar clusters > from 2.5.x to 2.7.x in production. > I must admit that I didn't think too much about versions yet. Pulsar has a fast development pace, so I'm assuming that we also need to cut support for older versions rather quickly. We should avoid supporting several major versions like we do with other connectors. Maybe we can phase out old versions into flink-packages.org? If the Pulsar community is willing to help with some of these tasks, then > this should be much easier but we need buy in from someone in the Flink > community to help with this effort. > I hope that Sijie and I addressed your concerns. Best, Arvid On Wed, Jan 6, 2021 at 11:28 AM Sijie Guo <s...@streamnative.io> wrote: > Hi Till, > > Thank you for your email! Please find my comments inline. > > On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <trohrm...@apache.org> > wrote: > >> Hi Jianyun, >> >> Thanks a lot for reviving this discussion. I think it would be great to >> have a well working Pulsar connector for Flink. Before diving into the >> detailed plan on how to do it technically, I think we should discuss where >> exactly the connector should live. From the Flink community's perspective >> every additional connector adds a considerable maintenance burden and, >> thus, needs to be carefully considered. >> >> 1) Who from the Flink community will mentor this effort and could take >> responsibility for it? >> > > We had a conversation with Stephen and Arvid about this. I think Arvid > Heise was willing to mentor this effort. > > >> 2) How can Pulsar be tested together with Flink (is there a Pulsar mini >> cluster for IT tests)? >> > > We use testcontainer for running a Pulsar standalone cluster to do the > integration tests. If it is not suitable for Flink project, we are happy to > adopt any technologies the Flink community prefers. > > >> 2.1) Do we need additional e2e tests? >> > > The Pulsar-Flink connector repository contains pretty good test coverage. > If it is not sufficient, we are happy to add more to follow Flink's best > practices. > > >> 3) Which Pulsar versions will/can be supported? >> > > Pulsar has a pretty good backward compatibility between versions. In our > experience, people have been running the connector with Pulsar clusters > from 2.5.x to 2.7.x in production. > > >> >> If the Pulsar community is willing to help with some of these tasks, then >> this should be much easier but we need buy in from someone in the Flink >> community to help with this effort. >> > > I am one of the Pulsar PMC members. Some of our Pulsar PMC members and > committers are committed to helping maintain this Pulsar Flink connector. > Hope this helps eliminate some concerns here. > >> >> The Flink community might be a bit unresponsive in the next couple of >> days because of Christmas and New Year. Beginning of January the discussion >> should see a couple of additional comments. >> >> Cheers, >> Till >> >> On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zhaojianyu...@outlook.com> wrote: >> >>> We are now done with the Flink Pulsar connector. It is ready to be >>> merged. >>> >>> Now our plan is >>> >>> 1. merge Source and Sink based on Function, because it has been >>> verified by users and is reliable. >>> 2. Next is to merge the DynamicTable feature. table implementation >>> depends on Step 1. >>> 3. merge Pulsar Catalog >>> 4. merge the PulsarSource based on FLIP-27: Refactor Source Interface< >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface> >>> of PulsarSource >>> 5. merge PulsarSink based on FLIP-143: Unified Sink API< >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API >>> > >>> 6. merge upsert-pulsar >>> >>> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector< >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector>, >>> but I don't have access to it. >>> >>> StreamNative/Pulsar Flink Connector: >>> https://github.com/streamnative/pulsar-flink >>> >>> >>> Looking forward to your feedback~ >>> >>> Jianyun Zhao >>> >>> -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
