Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect
Hi all, It seems like there are no objections to this KIP, so I've kicked off a vote thread: https://lists.apache.org/thread/dgq332o5j25rwddbvfydf7ttrclldw17 Cheers, Chris On Fri, Nov 24, 2023 at 10:39 PM Chris Egerton wrote: > Hi Yash, > > Thanks for taking a look! Yeah, it looks like we'll be forced to hold onto > the property until 5.0 if we can't introduce it at least one minor release > before 4.0. I don't think this is the worst thing. Although it'd be nice to > have the property completely removed when designing features like KIP-987, > if necessary, we can also declare any new features incompatible with > connectors that have opted out of enforcement of the tasks.max property > (and likely enforce this restraint programmatically via preflight > validation, failing connectors/tasks, log messages, etc.). > > Cheers, > > Chris > > On Wed, Nov 22, 2023 at 10:04 PM Yash Mayya wrote: > > > Hi Chris, > > > > Thanks for the well written and comprehensive KIP! Given that we're > already > > past the KIP freeze deadline for 3.7.0 ( > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0) > and > > there may not be a 3.8.0 release before the 4.0.0 release, would we then > be > > forced to punt the removal of "tasks.max.enforce" to a future 5.0.0 > > release? I don't have any other comments, and the proposed changes make > > sense to me. > > > > Thanks, > > Yash > > > > On Mon, Nov 20, 2023 at 10:50 PM Chris Egerton > > wrote: > > > > > Hi Hector, > > > > > > Thanks for taking a look! I think the key difference between the > proposed > > > behavior and the rejected alternative is that the set of tasks that > will > > be > > > running with the former is still a complete set of tasks, whereas the > set > > > of tasks in the latter is a subset of tasks. Also noteworthy but > slightly > > > less important: the problem will be more visible to users with the > former > > > (the connector will still be marked FAILED) than with the latter. > > > > > > Cheers, > > > > > > Chris > > > > > > On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > hgerald...@bloomberg.net> wrote: > > > > > > > Thanks for the KIP Chris, adding this check makes total sense. > > > > > > > > I do have one question. The second paragraph in the Public Interfaces > > > > section states: > > > > > > > > "If the connector generated excessive tasks after being reconfigured, > > > then > > > > any existing tasks for the connector will be allowed to continue > > running, > > > > unless that existing set of tasks also exceeds the tasks.max > property." > > > > > > > > Would not failing the connector land us in the second scenario of > > > > 'Rejected Alternatives'? > > > > > > > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: > > > > dev@kafka.apache.org > > > > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka > > Connect > > > > > > > > Hi all, > > > > > > > > I'd like to open up KIP-1004 for discussion: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ > > > > property+in+Kafka+Connect > > > > > > > > As a brief summary: this KIP proposes that the Kafka Connect runtime > > > start > > > > failing connectors that generate a greater number of tasks than the > > > > tasks.max property, with an optional emergency override that can be > > used > > > to > > > > continue running these (probably-buggy) connectors if absolutely > > > necessary. > > > > > > > > I'll be taking time off most of the next three weeks, so response > > latency > > > > may be a bit higher than usual, but I wanted to kick off the > discussion > > > in > > > > case we can land this in time for the upcoming 3.7.0 release. > > > > > > > > Cheers, > > > > > > > > Chris > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect
Hi Yash, Thanks for taking a look! Yeah, it looks like we'll be forced to hold onto the property until 5.0 if we can't introduce it at least one minor release before 4.0. I don't think this is the worst thing. Although it'd be nice to have the property completely removed when designing features like KIP-987, if necessary, we can also declare any new features incompatible with connectors that have opted out of enforcement of the tasks.max property (and likely enforce this restraint programmatically via preflight validation, failing connectors/tasks, log messages, etc.). Cheers, Chris On Wed, Nov 22, 2023 at 10:04 PM Yash Mayya wrote: > Hi Chris, > > Thanks for the well written and comprehensive KIP! Given that we're already > past the KIP freeze deadline for 3.7.0 ( > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0) and > there may not be a 3.8.0 release before the 4.0.0 release, would we then be > forced to punt the removal of "tasks.max.enforce" to a future 5.0.0 > release? I don't have any other comments, and the proposed changes make > sense to me. > > Thanks, > Yash > > On Mon, Nov 20, 2023 at 10:50 PM Chris Egerton > wrote: > > > Hi Hector, > > > > Thanks for taking a look! I think the key difference between the proposed > > behavior and the rejected alternative is that the set of tasks that will > be > > running with the former is still a complete set of tasks, whereas the set > > of tasks in the latter is a subset of tasks. Also noteworthy but slightly > > less important: the problem will be more visible to users with the former > > (the connector will still be marked FAILED) than with the latter. > > > > Cheers, > > > > Chris > > > > On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > hgerald...@bloomberg.net> wrote: > > > > > Thanks for the KIP Chris, adding this check makes total sense. > > > > > > I do have one question. The second paragraph in the Public Interfaces > > > section states: > > > > > > "If the connector generated excessive tasks after being reconfigured, > > then > > > any existing tasks for the connector will be allowed to continue > running, > > > unless that existing set of tasks also exceeds the tasks.max property." > > > > > > Would not failing the connector land us in the second scenario of > > > 'Rejected Alternatives'? > > > > > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: > > > dev@kafka.apache.org > > > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka > Connect > > > > > > Hi all, > > > > > > I'd like to open up KIP-1004 for discussion: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ > > > property+in+Kafka+Connect > > > > > > As a brief summary: this KIP proposes that the Kafka Connect runtime > > start > > > failing connectors that generate a greater number of tasks than the > > > tasks.max property, with an optional emergency override that can be > used > > to > > > continue running these (probably-buggy) connectors if absolutely > > necessary. > > > > > > I'll be taking time off most of the next three weeks, so response > latency > > > may be a bit higher than usual, but I wanted to kick off the discussion > > in > > > case we can land this in time for the upcoming 3.7.0 release. > > > > > > Cheers, > > > > > > Chris > > > > > > > > > > > >
Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect
Hi Chris, Thanks for the well written and comprehensive KIP! Given that we're already past the KIP freeze deadline for 3.7.0 ( https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0) and there may not be a 3.8.0 release before the 4.0.0 release, would we then be forced to punt the removal of "tasks.max.enforce" to a future 5.0.0 release? I don't have any other comments, and the proposed changes make sense to me. Thanks, Yash On Mon, Nov 20, 2023 at 10:50 PM Chris Egerton wrote: > Hi Hector, > > Thanks for taking a look! I think the key difference between the proposed > behavior and the rejected alternative is that the set of tasks that will be > running with the former is still a complete set of tasks, whereas the set > of tasks in the latter is a subset of tasks. Also noteworthy but slightly > less important: the problem will be more visible to users with the former > (the connector will still be marked FAILED) than with the latter. > > Cheers, > > Chris > > On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) < > hgerald...@bloomberg.net> wrote: > > > Thanks for the KIP Chris, adding this check makes total sense. > > > > I do have one question. The second paragraph in the Public Interfaces > > section states: > > > > "If the connector generated excessive tasks after being reconfigured, > then > > any existing tasks for the connector will be allowed to continue running, > > unless that existing set of tasks also exceeds the tasks.max property." > > > > Would not failing the connector land us in the second scenario of > > 'Rejected Alternatives'? > > > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: > > dev@kafka.apache.org > > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect > > > > Hi all, > > > > I'd like to open up KIP-1004 for discussion: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ > > property+in+Kafka+Connect > > > > As a brief summary: this KIP proposes that the Kafka Connect runtime > start > > failing connectors that generate a greater number of tasks than the > > tasks.max property, with an optional emergency override that can be used > to > > continue running these (probably-buggy) connectors if absolutely > necessary. > > > > I'll be taking time off most of the next three weeks, so response latency > > may be a bit higher than usual, but I wanted to kick off the discussion > in > > case we can land this in time for the upcoming 3.7.0 release. > > > > Cheers, > > > > Chris > > > > > > >
Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect
Hi Hector, Thanks for taking a look! I think the key difference between the proposed behavior and the rejected alternative is that the set of tasks that will be running with the former is still a complete set of tasks, whereas the set of tasks in the latter is a subset of tasks. Also noteworthy but slightly less important: the problem will be more visible to users with the former (the connector will still be marked FAILED) than with the latter. Cheers, Chris On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) < hgerald...@bloomberg.net> wrote: > Thanks for the KIP Chris, adding this check makes total sense. > > I do have one question. The second paragraph in the Public Interfaces > section states: > > "If the connector generated excessive tasks after being reconfigured, then > any existing tasks for the connector will be allowed to continue running, > unless that existing set of tasks also exceeds the tasks.max property." > > Would not failing the connector land us in the second scenario of > 'Rejected Alternatives'? > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: > dev@kafka.apache.org > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect > > Hi all, > > I'd like to open up KIP-1004 for discussion: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ > property+in+Kafka+Connect > > As a brief summary: this KIP proposes that the Kafka Connect runtime start > failing connectors that generate a greater number of tasks than the > tasks.max property, with an optional emergency override that can be used to > continue running these (probably-buggy) connectors if absolutely necessary. > > I'll be taking time off most of the next three weeks, so response latency > may be a bit higher than usual, but I wanted to kick off the discussion in > case we can land this in time for the upcoming 3.7.0 release. > > Cheers, > > Chris > > >
Re:[DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect
Thanks for the KIP Chris, adding this check makes total sense. I do have one question. The second paragraph in the Public Interfaces section states: "If the connector generated excessive tasks after being reconfigured, then any existing tasks for the connector will be allowed to continue running, unless that existing set of tasks also exceeds the tasks.max property." Would not failing the connector land us in the second scenario of 'Rejected Alternatives'? From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: dev@kafka.apache.org Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect Hi all, I'd like to open up KIP-1004 for discussion: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ property+in+Kafka+Connect As a brief summary: this KIP proposes that the Kafka Connect runtime start failing connectors that generate a greater number of tasks than the tasks.max property, with an optional emergency override that can be used to continue running these (probably-buggy) connectors if absolutely necessary. I'll be taking time off most of the next three weeks, so response latency may be a bit higher than usual, but I wanted to kick off the discussion in case we can land this in time for the upcoming 3.7.0 release. Cheers, Chris
