Hello Greg (sorry about the duplicate e-mail, forgot to cc users mailing list)
Thanks a lot for your detailed reply. I'm using JDBC Source connectors from kafka-connect-jdbc <https://github.com/confluentinc/kafka-connect-jdbc>. Indeed the `poll()` implementation is blocked, so it only processes a `stop()` when it returns from the current `poll()`execution. There was a change in the past to fix a similar problem <https://github.com/confluentinc/kafka-connect-jdbc/pull/677>, but not involving `stop()` from the same thread. I've just raised one <https://github.com/confluentinc/kafka-connect-jdbc/issues/1360>. Unfortunately setting a lower poll interval is not an option for me, as this is being used in a heavy data load operation, being executed only once per day. Will see if I'm able to come up with a change, although not sure yet. First time using kafka, kafka connect and kafka connect jdbc =D Kind regards Robson On Thu, 17 Aug 2023 at 00:37, Greg Harris <greg.har...@aiven.io.invalid> wrote: > Hi Robson, > > Thank you for the detailed bug report. > > I believe the behavior that you're describing is caused by this flaw: > https://issues.apache.org/jira/browse/KAFKA-15090 which is still under > discussion. Since the above flaw was introduced in 3.0, source > connectors need to return from poll() before the graceful shutdown > timeout to avoid the error. > You may be able to mitigate the error if the connector allows you to > reduce its poll timeout/interval to something less than the graceful > timeout, but that will depend on the specific connector > implementation, so check the documentation for your connector. I know > some implementations have received patches to compensate for this > behavior in the framework, so also consider upgrading or checking > release notes for your connectors. > > As for the effects of this error: whenever a non-graceful stop occurs, > the runtime will immediately close the producer so that the task will > not be able to write any further records. However, it will still leave > resources for that task (threads, in-memory records, database > connections, etc) allocated, until the task does finally return from > poll(). While this is not desirable behavior, it seems to be treated > as just a nuisance error by most operators. > > I hope this gives some context for the error message you're seeing. > > Thanks, > Greg >
