Hi Gordon, I was able to reproduce the data loss on standalone flink cluster
also. I have stripped down version of our code with here:
Environment:
Flink standalone 1.3.0
Kafka 0.9
*What the code is doing:*
-consume messages from kafka topic ('event.filter.topic' property in
application.properties)
-group them by key
-analyze the events in a window and filter some messages.
-send remaining messages to kafka topc ('sep.http.topic' property in
application.properties)
To build:
./gradlew clean assemble
The jar needs path to 'application.properties' file to run
Important properties in application.properties:
window.session.interval.sec
kafka.brokers
event.filter.topic --> source topic
sep.http.topic --> destination topic
To test:
-Use 'EventGenerator' class to publish messages to source kafka topic
The data published won't be filtered by the logic. If you publish 10
messages to the source topic,
those 10 messages should be sent to the destination topic.
-Once we see that flink has received all the messages, bring down all kafka
brokers
-Let Flink jobs fail for 2-3 times.
-Restart kafka brokers.
Note: Data loss isn't observed frequently. 1/4 times or so.
Thanks for all your help.
eventFilter.zip
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n14522/eventFilter.zip>
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Fink-KafkaProducer-Data-Loss-tp11413p14522.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
