Github user koeninger commented on the issue: https://github.com/apache/spark/pull/15355 @zsxwing good eye, thanks. It's not that auto.offset.reset.earliest doesn't work, it's that there's a potential race condition that poll gets called twice slowly enough for consumer position to be modified before topicpartitions are paused. https://github.com/apache/spark/pull/15387 should address that. It's something that whoever works on the duplicated equivalent code in the structured streaming module is going to have to address, also.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org