Hi Bill,
I would prefer to have some Beam IO for that more than specific
connector on the runner.
It's clearly an area where we have to extend our coverage (and we are
working on it).
Kafka IO is already on track. I have some others on plan.
Anyway, we love contribution ;)
Regards
JB
On 03/18/2016 05:23 PM, William McCarthy wrote:
Hi,
I’m trying to write a proof-of-concept which takes messages from Kafka,
transforms them using Beam on Flink, then pushes the results onto a different
Kafka topic.
I’ve used the KafkaWindowedWordCountExample as a starting point, and that’s
doing the first part of what I want to do, but it outputs to text files as
opposed to Kafka. FlinkKafkaProducer08 looks promising, but I can’t figure out
how to plug it into the pipeline. I was thinking that it would be wrapped with
an UnboundedFlinkSink, or some such, but that doesn’t seem to exist.
Any advice or thoughts on what I’m trying to do?
I’m running the latest incubator-beam (as of last night from Github), Flink
1.0.0 in cluster mode and Kafka 0.9.0.1, all on Google Compute Engine (Debian
Jessie).
Thanks,
Bill McCarthy
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
