Handling Bounded Sources with KafkaSource

2021-03-12 Thread Rion Williams
Hi all, I've been using the KafkaSource API as opposed to the classic consumer and things have been going well. I configured my source such that it could be used in either a streaming or bounded mode, with the bounded approach specifically aimed at improving testing (unit/integration). I've notic

Re: Handling Bounded Sources with KafkaSource

2021-03-13 Thread Rion Williams
Following up on this issue, I realized my initial problem was that my test case only contained a single message to send through the pipeline. This resulted in the earliest offset also being the latest and things didn’t exactly work as expected. Once I added several other messages and sent them t

Re: Handling Bounded Sources with KafkaSource

2021-03-14 Thread Maciej Obuchowski
Hey Rion, We solved this issue by using usual, unbounded streams, and using awaitility library to express conditions that would end the test - for example, having particular data in a table. IMO this type of testing has the advantage that you won't have divergent behavior from production as you h