Hi Ryan,
So long as you're not committing/storing offsets until that batch has been
successfully persisted you're fine (or if you're okay with some data loss
in the event of an app failing or being scaled down).
We use that approach in a couple of our apps that are persisting data into
datastores
You don't think it's weird if I just batch in memory manually do you? I
wrote a small snippet:
// Replicates Base Consumer
while (true) {
List records = consumer.poll();
// Batch records in-memory until criteria is satisfied
consumer.recordBuffer.addAll(records);
if (isNull(co
I think it is a regression, since it was introduced in KAFKA-9437: KIP-559,
which is in 2.5.0, so it means for the same scenario described in Boyang's
PR, 2.5 would fail while 2.4 would not.
Guozhang
On Fri, Mar 20, 2020 at 6:59 PM Ismael Juma wrote:
> Hi Boyang,
>
> Is this a regression?
>
> I
Lukasz,
Thanks for the reply. I set max.poll.interval.ms to 1 hour and my consumer
is polling with no issues every 500 ms and still I'm seeing the problem.
I'm not worried about consumer dying and rebalancing instead after the
rebalance it forgets it's previous state.
Thanks
On Fri, Mar 20, 2020
