Great, thanks! From: Fabian Hueske [mailto:[email protected]] Sent: Thursday, June 6, 2019 3:07 PM To: Smirnov Sergey Vladimirovich <[email protected]> Cc: [email protected] Subject: Re: Change sink topology
Hi Sergey, I would not consider this to be a topology change (the sink operator would still be a Kafka producer). It seems that dynamic topic selection is possible with a KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]). Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <[email protected]<mailto:[email protected]>>: Hi flink, Im wonder, is it possible to dynamically (while job running) change sink topology* - by adding new sink on the fly? Say, we have input stream and by analyzing message property we decided to put this message into some kafka topic, i.e. choosen_topic = function(message.property). Simplifying: sink_topic = ‘logger_group_’+message.groupId When job was launched we don`t know list of all possible groupId. These sink topics (and groupId) are created (and even removed) dynamically upon executing some external rules during the entire job period. Best, Sergey
