Github user koeninger commented on the issue:
https://github.com/apache/spark/pull/21917
Still playing devil's advocate here, I don't think stopping at 3 in your
example actually tells you anything about the cause of the gaps in the sequence
at 4. I'm not sure you can know that the gap is because of a transaction
marker, without a modified kafka consumer library.
If the actual problem is that when allowNonConsecutiveOffsets is set we
need to allow gaps even at the end of an offset range... why not just fix that
directly?
Master is updated to kafka 2.0 at this point, so we should be able to write
a test for your original jira example of a partition consisting of 1 message
followed by 1 transaction commit.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
