Hi Yijie, Thanks for the design document. I agree with Bowen that the catalog part needs more details. And I would suggest to separate Pulsar Catalog as another FLIP. IMO, it has little to do with source/sink. Having a separate FLIP can unblock the contribution for sink (or source) and keep the discussion more focus. I also left some comments in the documentation.
Thanks, Jark On Thu, 17 Oct 2019 at 11:24, Yijie Shen <henry.yijies...@gmail.com> wrote: > Hi Bowen, > > Thanks for your comments. I'll add catalog details as you suggested. > > One more question: since we decide to not implement source part of the > connector at the moment. > What can users do with a Pulsar catalog? > Create a table backed by Pulsar and check existing pulsar tables to see > their schemas? Drop tables maybe? > > Best, > Yijie > > On Thu, Oct 17, 2019 at 1:04 AM Bowen Li <bowenl...@gmail.com> wrote: > > > Hi Yijie, > > > > Per the discussion, maybe you can move pulsar source to 'future work' > > section in the FLIP for now? > > > > Besides, the FLIP seems to be quite rough at the moment, and I'd > recommend > > to add more details . > > > > A few questions mainly regarding the proposed pulsar catalog. > > > > - Can you provide some background of pulsar schema registry and how it > > works? > > - The proposed design of pulsar catalog is very vague now, can you > > share some details of how a pulsar catalog would work internally? E.g. > > - which APIs does it support exactly? E.g. I see from your > > prototype that table creation is supported but not alteration. > > - is it going to connect to a pulsar schema registry via a http > > client or a pulsar client, etc > > - will it be able to handle multiple versions of pulsar, or just > > one? How is compatibility handles between different Flink-Pulsar > versions? > > - will it support only reading from pulsar schema registry , or > > both read/write? Will it work end-to-end in Flink SQL for users to > create > > and manipulate a pulsar table such as "CREATE TABLE t WITH > > PROPERTIES(type=pulsar)" and "DROP TABLE t"? > > - Is a pulsar topic always gonna be a non-partitioned table? How is > > a partitioned topic mapped to a Flink table? > > - How to map Flink's catalog/database namespace to pulsar's > > multi-tenant namespaces? I'm not very familiar with how multi tenancy > works > > in pulsar, and some background context/use cases may help here too. > E.g. > > - can a pulsar client/consumer/producer be multiple-tenant at the > > same time? > > - how does authentication work in pulsar's multi-tenancy and the > > catalog? asking since I didn't see the proposed pulsar catalog has > > username/password configs > > - the FLIP seems propose mapping a pulsar cluster and > > 'tenant/namespace' respectively to Flink's 'catalog' and > 'database'. I > > wonder whether it totally makes sense, or should we actually map > "tenant" > > to "catalog", and "namespace" to "database"? > > > > Cheers, > > Bowen > > > > On Fri, Sep 20, 2019 at 1:16 AM Yijie Shen <henry.yijies...@gmail.com> > > wrote: > > > >> Hi everyone, > >> > >> Per discussion in the previous thread > >> < > >> > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Contribute-Pulsar-Flink-connector-back-to-Flink-tc32538.html > >> >, > >> I have created FLIP-72 to kick off a more detailed discussion on the > Flink > >> Pulsar connector: > >> > >> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector > >> > >> In short, the connector has the following features: > >> > >> - > >> > >> Pulsar as a streaming source with exactly-once guarantee. > >> - > >> > >> Sink streaming results to Pulsar with at-least-once semantics. > >> - > >> > >> Build upon Flink new Table API Type system (FLIP-37 > >> < > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-37%3A+Rework+of+the+Table+API+Type+System > >> > > >> ), and can automatically (de)serialize messages with the help of > Pulsar > >> schema. > >> - > >> > >> Integrate with Flink new Catalog API (FLIP-30 > >> < > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs > >> >), > >> which enables the use of Pulsar topics as tables in Table API as well > >> as > >> SQL client. > >> > >> > >> > >> > https://docs.google.com/document/d/1rES79eKhkJxrRfQp1b3u8LB2aPaq-6JaDHDPJIA8kMY/edit#heading=h.28v5v23yeq1u > >> > >> > >> Would love to here your thoughts on this. > >> > >> Best, > >> Yijie > >> > > >