Re: Exactly once kafka connect query

2023-03-08 Thread Chris Egerton
Hi Nitty, I'm a little confused about what you mean by this part: > transaction is not getting completed because it is not commiting the transaction offest. The only conditions required for a transaction to be completed when a connector is defining its own transaction boundaries are: 1. The tas

Re: producer purgatory

2023-03-08 Thread David Ballano Fernandez
Thanks Andrew. I'll give linger.ms a try. I was testing worse case scenarios so linger.ms was set to 0. also the producer was doing ack=all. which definitely adds all the producer requests to the purgatory waiting to be acknowledged. thanks. On Sat, Mar 4, 2023 at 2:57 PM Andrew Grant wrote:

Re: Exactly once kafka connect query

2023-03-08 Thread NITTY BENNY
Hi Chris, Sorry for the typo in my previous email. Regarding the point 2,* the task returns a batch of records from SourceTask::poll (and, if using* *the per-record API provided by the TransactionContext class, includes atleast one record that should trigger a transaction commit/abort in thatbat

Re: Exactly once kafka connect query

2023-03-08 Thread NITTY BENNY
Hi Chris, I am not sure if you are able to see the images I shared with you . Copying the code snippet below, if (expectedRecordCount >= 0) { int missingCount = expectedRecordCount - (int) this.recordOffset() - 1; if (missingCount > 0) { if (transactionConte

max.poll.interval.ms rebalance override

2023-03-08 Thread Mcs Vemuri
Hello, Is there a way to override a rebalance caused by max.poll.interval.ms? We have a case where there are multiple consumers in the same group- but some of them might start sooner than the rest and all consumers can be lazy pollers  The issue is that we need to set the poll interval to a large

Re: max.poll.interval.ms rebalance override

2023-03-08 Thread Mcs Vemuri
I overlooked the pause resume mechanism- it should work for us I think  On Wednesday, March 8, 2023, 4:52 PM, Mcs Vemuri wrote: Hello, Is there a way to override a rebalance caused by max.poll.interval.ms? We have a case where there are multiple consumers in the same group- but some of them m