Re: Exactly once kafka connect query

2023-03-13 Thread NITTY BENNY
Hi Chris, I really don't understand why a graceful shutdown will happen during a commit operation? Am I understanding something wrong here?. I see this happens when I have a batch of 2 valid records and in the second batch the record is invalid. In that case I want to commit the valid records. So

Re: Exactly once kafka connect query

2023-03-13 Thread NITTY BENNY
Hi Chris, The difference is in the Task Classes, no difference for value/key convertors. I don’t see log messages for graceful shutdown. I am not clear on what you mean by shutting down the task. I called the commit operation for the successful records. Should I perform any other steps if I have

Re: Exactly once kafka connect query

2023-03-13 Thread Chris Egerton
Hi Nitty, Thanks again for all the details here, especially the log messages. > The below mentioned issue is happening for Json connector only. Is there any difference with asn1,binary,csv and json connector? Can you clarify if the difference here is in the Connector/Task classens, or if it's in

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

2023-03-13 Thread Satish Duggana
Congratulations David! On Mon, 13 Mar 2023 at 16:08, Rajini Sivaram wrote: > Congratulations, David! > > Regards, > > Rajini > > On Mon, Mar 13, 2023 at 9:06 AM Bruno Cadonna wrote: > > > Congrats, David! > > > > Bruno > > > > On 10.03.23 01:36, Matthias J. Sax wrote: > > > Congrats! > > > > >

Re: Exactly once kafka connect query

2023-03-13 Thread NITTY BENNY
Hi Chris, As you said, the below message is coming when I call an abort if there is an invalid record, then for the next transaction I can see the below message and then the connector will be stopped. 2023-03-13 14:28:26,043 INFO [json-sftp-source-connector|task-0] Aborting transaction for batch a

Re: Exactly once kafka connect query

2023-03-13 Thread NITTY BENNY
Hi Chris, The below mentioned issue is happening for Json connector only. Is there any difference with asn1,binary,csv and json connector? Thanks, Nitty On Mon, Mar 13, 2023 at 9:16 AM NITTY BENNY wrote: > Hi Chris, > > Sorry Chris, I am not able to reproduce the above issue. > > I want to sha

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

2023-03-13 Thread Rajini Sivaram
Congratulations, David! Regards, Rajini On Mon, Mar 13, 2023 at 9:06 AM Bruno Cadonna wrote: > Congrats, David! > > Bruno > > On 10.03.23 01:36, Matthias J. Sax wrote: > > Congrats! > > > > On 3/9/23 2:59 PM, José Armando García Sancio wrote: > >> Congrats David! > >> > >> On Thu, Mar 9, 2023

Re: Exactly once kafka connect query

2023-03-13 Thread NITTY BENNY
Hi Chris, Sorry Chris, I am not able to reproduce the above issue. I want to share with you one more use case I found. The use case is in the first batch it returns 2 valid records and then in the next batch it is an invalid record.Below is the transaction_state topic, when I call a commit while

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

2023-03-13 Thread Bruno Cadonna
Congrats, David! Bruno On 10.03.23 01:36, Matthias J. Sax wrote: Congrats! On 3/9/23 2:59 PM, José Armando García Sancio wrote: Congrats David! On Thu, Mar 9, 2023 at 2:00 PM Kowshik Prakasam wrote: Congrats David! On Thu, Mar 9, 2023 at 12:09 PM Lucas Brutschy wrote: Congratulations