Hi All, I'm looking into adding a Kafka connector to Beam Python SDK. I think this will benefits many Python SDK users and will serve as a good example for recently added Splittable DoFn API (Fn API support which will allow all runners to use Python Splittable DoFn is in active development). I created a document [1] that makes the case for adding this connector and compares the performance of available Python Kafka client libraries. Also I created a POC [2] that illustrates the API and how Python SDF API can be used to implement a Kafka source. I extremely appreciate any feedback related to this.
[1] https://docs.google.com/document/d/1ogRS-e-HYYTHsXi_l2zDUUOnvfzEbub3BFkPrYIOawU/edit?usp=sharing [2] https://github.com/chamikaramj/beam/commit/982767b69198579b22522de6794242142d12c5f9 Thanks, Cham
