Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi everyone, I've updated the Catalog PR and make all settings small case. And tests are added as well. Hi Bowen, could you please take a look. https://github.com/apache/flink/pull/10455 For the sink part of the connector, I've made a separate PR https://github.com/apache/flink/pull/10875. Could

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Yijie, There's just one more concern on the yaml configs. Otherwise, I think we should be good to go. Can you update your PR and ensure all tests pass? I can help review and merge in the next couple weeks. Thanks, Bowen On Mon, Dec 23, 2019 at 7:03 PM Yijie Shen wrote: > Hi Bowen, > >

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Bowen, I've done updated the design doc, PTAL. Btw the PR for catalog is https://github.com/apache/flink/pull/10455, could you please take a look? Best, Yijie On Mon, Dec 9, 2019 at 8:44 AM Bowen Li wrote: > Hi Yijie, > > I took a look at the design doc. LGTM overall, left a few questions.

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Yijie, I took a look at the design doc. LGTM overall, left a few questions. On Tue, Dec 3, 2019 at 10:39 PM Becket Qin wrote: > Yes, you are absolutely right. Cannot believe I posted in the wrong > thread... > > On Wed, Dec 4, 2019 at 1:46 PM Jark Wu wrote: > >> Thanks Becket the the

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Yes, you are absolutely right. Cannot believe I posted in the wrong thread... On Wed, Dec 4, 2019 at 1:46 PM Jark Wu wrote: > Thanks Becket the the updating, > > But shouldn't this message be posted in FLIP-27 discussion thread[1]? > > > Best, > Jark > > [1]: > >

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Thanks Becket the the updating, But shouldn't this message be posted in FLIP-27 discussion thread[1]? Best, Jark [1]: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-27-Refactor-Source-Interface-td24952.html On Wed, 4 Dec 2019 at 12:12, Becket Qin wrote: > Hi

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi all, Sorry for the long belated update. I have updated FLIP-27 wiki page with the latest proposals. Some noticeable changes include: 1. A new generic communication mechanism between SplitEnumerator and SourceReader. 2. Some detail API method signature changes. We left a few things out of this

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi everyone, I've put the catalog part design in separate doc with more details for easier communication. https://docs.google.com/document/d/1LMnABtXn-wQedsmWv8hopvx-B-jbdr8-jHbIiDhdsoE/edit?usp=sharing I would love to hear your thoughts on this. Best, Yijie On Mon, Oct 21, 2019 at 11:15 AM

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi everyone, Glad to receive your valuable feedbacks. I'd first separate the Pulsar catalog as another doc and show more design and implementation details there. For the current FLIP-72, I would separate it into the sink part for current work and keep the source part as future works until we

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Yijie, I also agree with Jark on separating the Catalog part into another FLIP. With FLIP-27[1] also in the air, it is also probably great to split and unblock the sink implementation contribution. I would suggest either putting in a detail implementation plan section in the doc, or (maybe

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Yijie, Thanks for the design document. I agree with Bowen that the catalog part needs more details. And I would suggest to separate Pulsar Catalog as another FLIP. IMO, it has little to do with source/sink. Having a separate FLIP can unblock the contribution for sink (or source) and keep the

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Bowen, Thanks for your comments. I'll add catalog details as you suggested. One more question: since we decide to not implement source part of the connector at the moment. What can users do with a Pulsar catalog? Create a table backed by Pulsar and check existing pulsar tables to see their

Re: [DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi Yijie, Per the discussion, maybe you can move pulsar source to 'future work' section in the FLIP for now? Besides, the FLIP seems to be quite rough at the moment, and I'd recommend to add more details . A few questions mainly regarding the proposed pulsar catalog. - Can you provide some

[DISCUSS] FLIP-72: Introduce Pulsar Connector

Hi everyone, Per discussion in the previous thread , I have created FLIP-72 to kick off a more detailed discussion on the Flink Pulsar connector: