Is this what we want to do in the long run, i.e. implement copies of connectors for different SDKs? I thought the plan was to enable using connectors written in different languages, i.e. use the Java Kafka I/O from python. This way we wouldn't duplicate bugs for three different language (Java, Python, and Go for now).
Best, Aljoscha > On 29. Apr 2018, at 20:46, Eugene Kirpichov <kirpic...@google.com> wrote: > > Thanks Cham, this is great! I left just a couple of comments on the doc. > > On Fri, Apr 27, 2018 at 10:06 PM Chamikara Jayalath <chamik...@google.com > <mailto:chamik...@google.com>> wrote: > Hi All, > > I'm looking into adding a Kafka connector to Beam Python SDK. I think this > will benefits many Python SDK users and will serve as a good example for > recently added Splittable DoFn API (Fn API support which will allow all > runners to use Python Splittable DoFn is in active development). I created a > document [1] that makes the case for adding this connector and compares the > performance of available Python Kafka client libraries. Also I created a POC > [2] that illustrates the API and how Python SDF API can be used to implement > a Kafka source. I extremely appreciate any feedback related to this. > > [1] > https://docs.google.com/document/d/1ogRS-e-HYYTHsXi_l2zDUUOnvfzEbub3BFkPrYIOawU/edit?usp=sharing > > <https://docs.google.com/document/d/1ogRS-e-HYYTHsXi_l2zDUUOnvfzEbub3BFkPrYIOawU/edit?usp=sharing> > [2] > https://github.com/chamikaramj/beam/commit/982767b69198579b22522de6794242142d12c5f9 > > <https://github.com/chamikaramj/beam/commit/982767b69198579b22522de6794242142d12c5f9> > > Thanks, > Cham
