Thanks for the response JB, It would make sense to me to have this up at the Beam level as opposed to down in the Flink runner level, too. The reason that I wrote specifically about the FlinkKafkaProducer08 is because there is symmetry with the FlinkKafkaConsumer08, which is what’s being used in the example.
Bill > On Mar 18, 2016, at 12:34 PM, Jean-Baptiste Onofré <[email protected]> wrote: > > Hi Bill, > > I would prefer to have some Beam IO for that more than specific connector on > the runner. > > It's clearly an area where we have to extend our coverage (and we are working > on it). > Kafka IO is already on track. I have some others on plan. > > Anyway, we love contribution ;) > > Regards > JB > > On 03/18/2016 05:23 PM, William McCarthy wrote: >> Hi, >> >> I’m trying to write a proof-of-concept which takes messages from Kafka, >> transforms them using Beam on Flink, then pushes the results onto a >> different Kafka topic. >> >> I’ve used the KafkaWindowedWordCountExample as a starting point, and that’s >> doing the first part of what I want to do, but it outputs to text files as >> opposed to Kafka. FlinkKafkaProducer08 looks promising, but I can’t figure >> out how to plug it into the pipeline. I was thinking that it would be >> wrapped with an UnboundedFlinkSink, or some such, but that doesn’t seem to >> exist. >> >> Any advice or thoughts on what I’m trying to do? >> >> I’m running the latest incubator-beam (as of last night from Github), Flink >> 1.0.0 in cluster mode and Kafka 0.9.0.1, all on Google Compute Engine >> (Debian Jessie). >> >> Thanks, >> >> Bill McCarthy >> > > -- > Jean-Baptiste Onofré > [email protected] > http://blog.nanthrax.net > Talend - http://www.talend.com
