Jeff Hu created FLINK-23132: ------------------------------- Summary: flink upgrade issue(1.11.3->1.13.0) Key: FLINK-23132 URL: https://issues.apache.org/jira/browse/FLINK-23132 Project: Flink Issue Type: Bug Reporter: Jeff Hu
In order to improve the performance of data process, we store events to a map and do not process them untill event count reaches 100. in the meantime, start a timer in open method, so data is processed every 60 seconds this works when flink version is *1.11.3*, after upgrading flink version to *1.13.0* I found sometimes events were consumed from Kafka continuously, but were not processed in RichFlatMapFunction, it means data was missing. after restarting service, it works well, but several hours later the same thing happened again. any known issue for this flink version? any suggestions are appreciated. {{public class MyJob \{ public static void main(String[] args) throws Exception { ... DataStream<String> rawEventSource = env.addSource(flinkKafkaConsumer); ... }}} {{public class MyMapFunction extends RichFlatMapFunction<String, String> implements Serializable \{ @Override public void open(Configuration parameters) { ... long periodTimeout = 60; pool.scheduleAtFixedRate(() -> { // processing data }, periodTimeout, periodTimeout, TimeUnit.SECONDS); } @Override public void flatMap(String message, Collector<String> out) \{ // store event to map // count event, // when count = 100, start data processing } }}} -- This message was sent by Atlassian Jira (v8.3.4#803005)