Hi Gunnar and Sergei, I think it's good to have a retriable exception handling during task#start.
> are retriable exceptions during start disallowed by design, and the task must not throw retriable exceptions during start, or it's just currently not supported by the Connect framework and I just need to implement proper error handling in the connector? > Would it require a KIP? Sorry, I'm not sure if it's by design or not supported and needed to be implemented. But I guess if you want to implement the error handling in connector, you might leverage existing retry configuration, or you'll create a new one. Either way, I think it needs a small KIP as you mentioned, task#start is not covered in KIP-298. On the other hand, I think having a KIP first is good, to make sure you're on the right track, before you get your hand dirty. Besides, KIP discussion would have more attention, I think. :) Thank you. Luke On Fri, Nov 26, 2021 at 4:09 PM Gunnar Morling <gunnar.morl...@googlemail.com.invalid> wrote: > Hi all, > > We encountered a similar situation in Debezium again, where an exception > during Task::start() would be desirable to be retried. > > Would anything speak against implementing retriable support for > Task::start() in Kafka Connect? Would it require a KIP? > > Thanks, > > --Gunnar > > > Am Mo., 9. Aug. 2021 um 10:47 Uhr schrieb Gunnar Morling < > gunnar.morl...@googlemail.com>: > > > Hi, > > > > To ask slightly differently: would there be interest in a pull request > for > > implementing retries, in case RetriableException is thrown from the > > Task::start() method? > > > > Thanks, > > > > --Gunnar > > > > > > Am Do., 5. Aug. 2021 um 22:27 Uhr schrieb Sergei Morozov <moro...@tut.by > >: > > > >> Hi, > >> > >> I'm trying to address an issue in Debezium (DBZ-3823 > >> <https://issues.redhat.com/browse/DBZ-3823>) where a source connector > >> task > >> cannot recover from a retriable exception. > >> > >> The root cause is that the task interacts with the source database > during > >> SourceTask#start but Kafka Connect doesn't handle retriable exceptions > >> thrown at this stage as retriable. KIP-298 > >> < > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-298%3A+Error+Handling+in+Connect > >> > > >> that > >> originally introduced handling of retriable exception doesn't describe > >> handling task start exceptions, so it's unclear to me whether those > aren't > >> allowed by design or it was just out of the scope of the KIP. > >> > >> My current working solution > >> <https://github.com/debezium/debezium/pull/2572> relies > >> on the internal Debezium implementation of the task restart which > >> introduces certain risks (the details are in the PR description). > >> > >> The question is: are retriable exceptions during start disallowed by > >> design, and the task must not throw retriable exceptions during start, > or > >> it's just currently not supported by the Connect framework and I just > need > >> to implement proper error handling in the connector? > >> > >> Thanks! > >> > >> -- > >> Sergei Morozov > >> > > >
