Hi Spark users, especially Structured Streaming users who are using Kafka as data source,
I'm pleased to introduce Kafka offset committer, which enables commit offsets which batch has been processed. The tool is basically an implementation of streaming query listener, which listens for events and commit offsets for each batch. Please refer README.md in the repository to see more details. Here's Github repository for this tool: https://github.com/HeartSaVioR/spark-sql-kafka-offset-committer Currently it hasn't be published to Maven central, so you might need to build the source and add jar via "--jars" option until artifact is published. I'd be happy to hear new ideas of improvements, and much appreciated for contributions! Enjoy! Thanks, Jungtaek Lim (HeartSaVioR)
