[GitHub] [spark] jerrypeng commented on pull request #38898: [SPARK-41375][SS] Avoid empty latest KafkaSourceOffset

2022-12-07 Thread GitBox
jerrypeng commented on PR #38898: URL: https://github.com/apache/spark/pull/38898#issuecomment-1341350829 > To avoid the data duplication in the extreme cases where spark fetch empty latest Kafka source offset. @wecharyu how does an empty latest Kafka source offset cause data duplica

[GitHub] [spark] jerrypeng commented on pull request #38898: [SPARK-41375][SS] Avoid empty latest KafkaSourceOffset

2022-12-06 Thread GitBox
jerrypeng commented on PR #38898: URL: https://github.com/apache/spark/pull/38898#issuecomment-1340530153 @wecharyu can you run one batch and then delete all the partitions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and