[ https://issues.apache.org/jira/browse/BEAM-6751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maximilian Michels closed BEAM-6751. ------------------------------------ Resolution: Fixed > KafkaIO's EOS mode does not work with FlinkRunner > ------------------------------------------------- > > Key: BEAM-6751 > URL: https://issues.apache.org/jira/browse/BEAM-6751 > Project: Beam > Issue Type: Bug > Components: io-java-kafka, runner-flink > Reporter: Maximilian Michels > Assignee: Maximilian Michels > Priority: Major > Fix For: 2.12.0 > > Time Spent: 10h 20m > Remaining Estimate: 0h > > KafkaIO has a validation check which whitelists certain runners capable of > provide exactly-once semantics: > {noformat} > if ("org.apache.beam.runners.direct.DirectRunner".equals(runner) > || runner.startsWith("org.apache.beam.runners.dataflow.") > || runner.startsWith("org.apache.beam.runners.spark.") { > ... > {noformat} > The Flink supports exactly-once checkpointing but the Flink Runner can't > utilize it in the way KafkaIO intends it. > I think we should remove the check in favor of checking for translation of > {{@RequiresStableInput}}. Changes to KafkaIO might have to be made to support > EOS efficiently with the Flink Runner. -- This message was sent by Atlassian JIRA (v7.6.3#76005)