ReemaAlzaid commented on PR #13183: URL: https://github.com/apache/gluten/pull/13183#issuecomment-6225960681
Hi @zhouyuan , I found a case where the reader times out even when `failOnDataLoss=false`. If Spark plans to read 100 messages but Kafka’s log gets truncated to 10, the reader reads those 10 and then keeps waiting for the rest. I’ve prepared a fix so it finishes with the available data when data loss is allowed, and reports data loss otherwise. It also checks that an open transaction isn’t being mistaken for truncation [72e9a8d](https://github.com/apache/gluten/pull/13183/commits/72e9a8ddae8f5ff006b1cad0f0c25a91c833d46d) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
