On 17/10/2018 16.43, Ryanne Dolan wrote:
I see, thanks.
On the other hand, the commitRecord() callback provides the functionality
you require in this case. In commitRecord() your SourceTask can track the
offsets of records that have been ack'd by the producer client, and then in
commit() you can be sure that those offsets have been flushed.
That is the trick I am currently using - more or less.
But unfortunately it does not work 100% either. It is possible that
commitRecord() is called with a record R, and then commit() is called
after that, without the offsets of R having been written/flushed. The
call to commitRecord() means that the "actual data" of R has been
send/acknowledged, but unfortunately this does not guarantee that the
offsets of R have been written/flushed at the next commit() call
