Hi Yash, Thanks for taking a look! Yeah, it looks like we'll be forced to hold onto the property until 5.0 if we can't introduce it at least one minor release before 4.0. I don't think this is the worst thing. Although it'd be nice to have the property completely removed when designing features like KIP-987, if necessary, we can also declare any new features incompatible with connectors that have opted out of enforcement of the tasks.max property (and likely enforce this restraint programmatically via preflight validation, failing connectors/tasks, log messages, etc.).
Cheers, Chris On Wed, Nov 22, 2023 at 10:04 PM Yash Mayya <yash.ma...@gmail.com> wrote: > Hi Chris, > > Thanks for the well written and comprehensive KIP! Given that we're already > past the KIP freeze deadline for 3.7.0 ( > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0) and > there may not be a 3.8.0 release before the 4.0.0 release, would we then be > forced to punt the removal of "tasks.max.enforce" to a future 5.0.0 > release? I don't have any other comments, and the proposed changes make > sense to me. > > Thanks, > Yash > > On Mon, Nov 20, 2023 at 10:50 PM Chris Egerton <fearthecel...@gmail.com> > wrote: > > > Hi Hector, > > > > Thanks for taking a look! I think the key difference between the proposed > > behavior and the rejected alternative is that the set of tasks that will > be > > running with the former is still a complete set of tasks, whereas the set > > of tasks in the latter is a subset of tasks. Also noteworthy but slightly > > less important: the problem will be more visible to users with the former > > (the connector will still be marked FAILED) than with the latter. > > > > Cheers, > > > > Chris > > > > On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > hgerald...@bloomberg.net> wrote: > > > > > Thanks for the KIP Chris, adding this check makes total sense. > > > > > > I do have one question. The second paragraph in the Public Interfaces > > > section states: > > > > > > "If the connector generated excessive tasks after being reconfigured, > > then > > > any existing tasks for the connector will be allowed to continue > running, > > > unless that existing set of tasks also exceeds the tasks.max property." > > > > > > Would not failing the connector land us in the second scenario of > > > 'Rejected Alternatives'? > > > > > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: > > > dev@kafka.apache.org > > > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka > Connect > > > > > > Hi all, > > > > > > I'd like to open up KIP-1004 for discussion: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ > > > property+in+Kafka+Connect > > > > > > As a brief summary: this KIP proposes that the Kafka Connect runtime > > start > > > failing connectors that generate a greater number of tasks than the > > > tasks.max property, with an optional emergency override that can be > used > > to > > > continue running these (probably-buggy) connectors if absolutely > > necessary. > > > > > > I'll be taking time off most of the next three weeks, so response > latency > > > may be a bit higher than usual, but I wanted to kick off the discussion > > in > > > case we can land this in time for the upcoming 3.7.0 release. > > > > > > Cheers, > > > > > > Chris > > > > > > > > > > > >
