Hello Greg Thanks a *lot* for your help on this.
Indeed the empty poll is not the issue for us. As mentioned, our setup is a poll every 24 hours. So the `stop()` being stuck due to the `poll()` is hitting us hard. I did a trace today on my dev environment, I can indeed see this waiting log entry every 100 ms <https://github.com/confluentinc/kafka-connect-jdbc/blob/master/src/main/java/io/confluent/connect/jdbc/source/JdbcSourceTask.java#L427>. Then I call a DEL on this connector, and the stop is not processed until the next loop in the `poll()`. Your initial diagnosis is 100% correct. You mentioned other connectors already changed to support a stop signal from the same thread. Would u have any concrete connector impl to point me? Kind regards Robson On Mon, 21 Aug 2023 at 19:23, Greg Harris <greg.har...@aiven.io.invalid> wrote: > Hey Robson, > > Thanks for opening an issue on the JDBC repo, I think this is > certainly relevant feedback for the connector developers. I commented > on the issue with a potential regression that I saw, you can try > downgrading your connector to see if the behavior improves. > I also know that kafka-connect-jdbc received a patch to improve this > behavior when no data is being emitted: > https://github.com/confluentinc/kafka-connect-jdbc/pull/947 but I'm > not sure if that is relevant to your situation. > > Thanks! > Greg > > On Mon, Aug 21, 2023 at 6:53 AM Robson Hermes <robsonher...@gmail.com> > wrote: > > > > No, it stops them also. > > The problem is precisely what Greg described, now the stop signal comes > > from the same thread. So any source task which is running in a blocking > way > > will not process the stop signal until the current poll finishes. > > So would need to patch source jdbc connector. > > > > On Mon, 21 Aug 2023 at 15:48, sunil chaudhari < > sunilmchaudhar...@gmail.com> > > wrote: > > > > > I think when you delete connector it removes the task and workers > continues > > > to run. > > > When you stop it actually stops the worker. > > > Both different things. > > > Point to be noted is Worker has connector. > > > So connector should be removed before stopping the worker. > > > > > > Though I am not expert in this. > > > > > > On Mon, 21 Aug 2023 at 7:10 PM, Robson Hermes <robsonher...@gmail.com> > > > wrote: > > > > > > > Hello Sunil > > > > > > > > I'm not calling a stop, I'm straight deleting the connectors with the > > > > DELETE. Stopping the connector is done internally during deletion. > > > > > > > > Regards > > > > Robson > > > > > > > > On Mon, 21 Aug 2023 at 15:36, sunil chaudhari < > > > sunilmchaudhar...@gmail.com > > > > > > > > > wrote: > > > > > > > > > You have to remove connectors first using delete api > > > > > and then stop the connector > > > > > > > > > > On Thu, 17 Aug 2023 at 2:51 AM, Robson Hermes < > robsonher...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hello > > > > > > > > > > > > I'm using kafka connect 7.4.0 to read data from Postgres views > and > > > > write > > > > > to > > > > > > another Postgres tables. So using JDBC source and sink > connectors. > > > > > > All works good, but whenever I stop the source connectors via the > > > rest > > > > > api: > > > > > > > > > > > > DEL http://kafka-connect:8083/connectors/connector_name_here > > > > > > > > > > > > The connector stops fine, but not the task: > > > > > > > > > > > > > > > > > > Graceful stop of connector (connector-name-here) succeeded. > > > > > > > > > > > > Graceful stop of task (task-name-here) failed. > > > > > > > > > > > > > > > > > > It only happens with the *source* connector tasks. The sink > connector > > > > > > and tasks shutdown gracefully and fine. > > > > > > > > > > > > The timeout for task shutdown has been increased, but didn't > help: > > > > > > > > > > > > task.shutdown.graceful.timeout.ms=60000 > > > > > > > > > > > > > > > > > > > > > > > > The connectors are running once per day (during the night) to > load a > > > > > > lot of data, and the error happens when I try to delete the > > > connectors > > > > > > in the middle of the day. That is, they are not actually > > > > > > executing/loading any data, it has finished already. > > > > > > > > > > > > offset.flush.interval.ms=10000 in development and integration > > > > > > environments. > > > > > > > > > > > > offset.flush.interval.ms=60000 in production and uat. > > > > > > > > > > > > > > > > > > The rest of the config is pretty much the default. > > > > > > > > > > > > What could be the issue? > > > > > > > > > > > > The errors of the graceful stop of the tasks are triggering our > alert > > > > > > system, so trying to get rid of those. > > > > > > > > > > > > > > > > > > Thanks a lot > > > > > > > > > > > > Robson > > > > > > > > > > > > > > > > > > >
