Hello Bill, This is a known limitation of the Flink Runner. There is a JIRA issue for that https://issues.apache.org/jira/browse/BEAM-127 <https://issues.apache.org/jira/browse/BEAM-127>
A wrapper for Flink sinks will come soon and as Beam evolves, a more Beam-y solution will come as well. Kostas > On Mar 18, 2016, at 5:38 PM, William McCarthy <[email protected]> > wrote: > > Thanks for the response JB, > > It would make sense to me to have this up at the Beam level as opposed to > down in the Flink runner level, too. The reason that I wrote specifically > about the FlinkKafkaProducer08 is because there is symmetry with the > FlinkKafkaConsumer08, which is what’s being used in the example. > > Bill > >> On Mar 18, 2016, at 12:34 PM, Jean-Baptiste Onofré <[email protected]> wrote: >> >> Hi Bill, >> >> I would prefer to have some Beam IO for that more than specific connector on >> the runner. >> >> It's clearly an area where we have to extend our coverage (and we are >> working on it). >> Kafka IO is already on track. I have some others on plan. >> >> Anyway, we love contribution ;) >> >> Regards >> JB >> >> On 03/18/2016 05:23 PM, William McCarthy wrote: >>> Hi, >>> >>> I’m trying to write a proof-of-concept which takes messages from Kafka, >>> transforms them using Beam on Flink, then pushes the results onto a >>> different Kafka topic. >>> >>> I’ve used the KafkaWindowedWordCountExample as a starting point, and that’s >>> doing the first part of what I want to do, but it outputs to text files as >>> opposed to Kafka. FlinkKafkaProducer08 looks promising, but I can’t figure >>> out how to plug it into the pipeline. I was thinking that it would be >>> wrapped with an UnboundedFlinkSink, or some such, but that doesn’t seem to >>> exist. >>> >>> Any advice or thoughts on what I’m trying to do? >>> >>> I’m running the latest incubator-beam (as of last night from Github), Flink >>> 1.0.0 in cluster mode and Kafka 0.9.0.1, all on Google Compute Engine >>> (Debian Jessie). >>> >>> Thanks, >>> >>> Bill McCarthy >>> >> >> -- >> Jean-Baptiste Onofré >> [email protected] >> http://blog.nanthrax.net >> Talend - http://www.talend.com >
