curcur edited a comment on pull request #11725: URL: https://github.com/apache/flink/pull/11725#issuecomment-621721146
Updates to commit "Kafka Shuffle and Test Case" 1. add log4j2 file for better testing and debugging. 2. I found the original tests are occasionally failing when testing for a large number of data. The reason is the default property "auto.offset.reset" is set to "latest". That means if the consumer starts later than the producer starting to writing data, then I am going to lose some data occasionally. So, I switched to use KafkaTestEnvironment.getStandardProperties which set the auto.offset.reset = earliest In a bit more detail, Kafka Fetcher reset the consuming offset when first starting (when there are no committed offsets). This initial consuming offset is determined by "auto.offset.reset". ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
