GitHub user koeninger opened a pull request: https://github.com/apache/spark/pull/20572
[SPARK-17147][STREAMING][KAFKA] Allow non-consecutive offsets ## What changes were proposed in this pull request? Add a configuration spark.streaming.kafka.allowNonConsecutiveOffsets to allow streaming jobs to proceed on compacted topics (or other situations involving gaps between offsets in the log). ## How was this patch tested? Added new unit test @justinrmiller has been testing this branch in production for a few weeks You can merge this pull request into a Git repository by running: $ git pull https://github.com/koeninger/spark-1 SPARK-17147 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/20572.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #20572 ---- commit 3082de7e43e8c381dc2227005d1e0fc5bd2c3d29 Author: cody koeninger <cody@...> Date: 2016-10-08T21:21:48Z [SPARK-17147][STREAMING][KAFKA] failing test for compacted topics commit e8ea89ea10527c6723df4af2685004ea67d872cd Author: cody koeninger <cody@...> Date: 2016-10-09T04:59:39Z [SPARK-17147][STREAMING][KAFKA] test passing for compacted topics commit 182943e36f596d0cb5841a9c63471bea1dd9047b Author: cody koeninger <cody@...> Date: 2018-02-11T04:09:38Z spark.streaming.kafka.allowNonConsecutiveOffsets commit 89f4bc5f4de78cdcc22b5c9b26a27ee9263048c8 Author: cody koeninger <cody@...> Date: 2018-02-11T04:13:49Z [SPARK-17147][STREAMING][KAFKA] remove stray param doc commit 12e65bedddbcd2407598e69fa3c6fcbcdfc67e5d Author: cody koeninger <cody@...> Date: 2018-02-11T04:28:22Z [SPARK-17147][STREAMING][KAFKA] prepare for merge of master commit 2ed51f1f73ee75ffd08355265a72e68e83ef592d Author: cody koeninger <cody@...> Date: 2018-02-11T05:19:31Z Merge branch 'master' into SPARK-17147 ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org