Hi Robert,
I have few more questions to clarify.
1) Why do you say printing the values to the standard out would display
duplicates even if exactly once works? What is the reason for this? Could
you brief me on this?
2) I observed duplicates (by writing to a file) starting from the
Hi Robert,
Thanks a lot for the reply.
To further explain how I verify the presence of duplicates, I write the
output stream received at the FlinkKafkaConsumer (after being sent from the
KafkaProducer) to a csv file.
Then the content of the file is scanned to see whether we received the exact
Hi Gordan,
Thanks alot for the reply.
The events are produced using a KafkaProducer, submitted to a topic and
thereby consumed by the Flink application using a FlinkKafkaConsumer. I
verified that during a failure recovery scenario(of the Flink application)
the KafkaProducer was not interrupted,
Hi all,
I'm working with Flink 1.2.0, Kafka 0.10.0.1 and Hadoop 2.7.3.
I have a Flink Highly Available cluster that reads data from a Kafka
producer and processes them within the cluster. I randomly kill a Task
Manager to introduce failure. Restart strategy is configured and the cluster
does
Hi Gordan,
Appreciate your prompt reply. Thanks alot for pointing that out that Kafka
Producer has at least once guarantee of message delivery. That seems to be
the reason why I encountered duplicated data on a flink failure recovery
scenario.
--
View this message in context:
Hi,
Thanks a lot for the reply. I configured a restart strategy as suggested and
now the TM failure scenario is working as expected. Once a TM is killed
another active TM automatically recovers the job.
--
View this message in context:
Hi,
I'm working with Apache Flink 1.1.2 and testing on High Availability mode.
In the case of Task Manager failures they say a standby TM will recover the
work of the failed TM. In my case, I have 4 TM's running in parallel and
when a TM is killed the state goes to Cancelling and then to Failed
