Thanks Przemek and Niko. Quick update: #75945 (xchwan) already implements dagProcessor.sets close to what I proposed, so I'll leave the dag processor side to that PR. I've reviewed it with the cases we hit in production.
Next I'll open triggerer.sets, following the same conventions as #75945 (enableDefault, <component>-set label, PDB exclusion, ServiceAccount handling). On Wed, Nov 11, 2026 08:24 PM, Przemysław Mirowski via dev < [email protected]> wrote: > Hi Mathieu, > > Thanks for proposal and feel free to create PRs (according to the plan > which you've mentioned) if you didn't already (I'm a little behind in > the reviewing PRs, but sooner or later I will catch up and review all > waiting helm chart-related). > > Regarding your questions: > > > For the dag processor, is a separate `sets` list (referencing bundles > by name) acceptable, or would you prefer per-bundle overrides on the > dagBundleConfigList entries (as discussed on #61039)? The triggerer is > scoped by team, not bundle, so `sets` seems the natural fit there either > way. > > As of now I would say "we will see". Implementation wise and in terms of > behaviour, the way how the feature works should be the same between > components. As it is possible to do for the workers and triggerer, it > raises a question for the Dag Processor as it can have different schema, > which would make more sense. One of the biggest concern regarding the > Worker Sets during the implementation was the complexity of it and this > is the reason for having (if possible and reasonable) one implementation > behind the `sets` feature. Nevertheless, when we will reach out the PRs, > maybe I will have ready the thing which I plan to do which is how the > schema for Helm Chart 2 should look like as the reference point for the > work (with also one issue currently affecting the chart itself which is > the schema validation). Maybe before the PR I will have that ready. > Let's differ that one (unless someone will have some opinion on that > too) till the implementation phase. > > > Should this target 2.0.0? > > Yes. I think we might consider the back-porting of it if reasonable > amount of users will want to and the works on Helm Chart 2 will take too > long (there is no specific target date, but I plan to work on it a bit > more in the upcoming couple months). > > > The default instance's selectors also match set pods (same as worker > sets today). Worth adding a `<component>-set: default` label to > all three components while the 2.0 window allows breaking changes? > > Yes. I think I saw PR with it somewhere already, but I haven't finished > review of it yet. > > Best, > Przemek > > On 20/10/2026 20:12, Nikolas Oliveira wrote: > > Hey Mathieu, > > > > I appreciate you contributing this back! I don't work very closely with > > Helm, so I'm not the best candidate to review this proposal, but from a > > multi team perspective, I'd really love to have this, especially for > > triggerers, where having one per team is critical. I think the proposal > > looks reasonable to me. If you haven't already, I'd start putting up some > > PRs if you're not getting much traction here. > > > > Cheers, > > Niko > > > > On Mon, Oct 12, 2026 at 4:37 AM Mathieu Monet<[email protected]> > > wrote: > > > >> Hi all, > >> > >> I'd like to propose adding `sets` to the triggerer and dag processor in > >> the Helm chart, following the existing `workers.celery.sets` pattern, > and > >> get direction before opening PRs. > >> > >> Why > >> --- > >> With `core.multi_team` enabled, each team needs a triggerer started with > >> `--team-name`, otherwise deferred tasks of that team's Dags never resume > >> (multi-team.rst, "Triggerer" section). The chart can only render one > >> triggerer today, so a multi-team deployment on the official chart is > >> not fully functional without patching templates. Running one dag > >> processor per team or bundle (`--bundle-name`) is the matching need on > >> the parsing side, for isolation and sizing. > >> > >> We run this in production (Airflow 3.3.2, several teams in one instance) > >> through a wrapper chart that copies the upstream triggerer and dag > >> processor templates in a loop. I'd rather contribute to it than keep > that > >> copy in sync. > >> > >> Proposal > >> -------- > >> triggerer: > >> enableDefault: true # render the unsuffixed triggerer > >> teamName: ~ # -> --team-name > >> queues: ~ # -> --queues > >> extraArgs: [] # appended to the default args > >> sets: [] # each item overrides any triggerer.* key > >> dagProcessor: > >> enableDefault: true > >> bundleNames: [] # -> repeated --bundle-name > >> extraArgs: [] > >> sets: [] > >> > >> - A set `<n>` renders `<fullname>-<component>-<n>` with its own > >> ServiceAccount, PDB and ScaledObject, and a `<component>-set: <n>` > >> label. Values merge exactly like worker sets (shared helper). > >> - Sets reference bundles and teams by name only. Render-time validation > >> checks them against dagBundleConfigList, so there is no second list > >> to keep in sync (the concern raised on #61039). Users set small scope > >> fields instead of a full CLI in `args` (the concern raised on > #61963). > >> - With every new field empty, the output is identical to today's. > >> > >> Planned as small PRs: shared set helpers (generalising > >> workersMergeValues), triggerer.sets, dagProcessor.sets, then optional > >> multi-team conveniences (deriving bundles from team_name, team-aware > >> KEDA query). > >> > >> Related: #26494, #61037, #61039, #61963, #64037. > >> > >> Questions > >> --------- > >> 1. For the dag processor, is a separate `sets` list (referencing > >> bundles by name) acceptable, or would you prefer per-bundle > overrides > >> on the dagBundleConfigList entries (as discussed on #61039)? The > >> triggerer is scoped by team, not bundle, so `sets` seems the natural > >> fit there either way. > >> 2. Should this target 2.0.0? > >> 3. The default instance's selectors also match set pods (same as worker > >> sets today). Worth adding a `<component>-set: default` label to all > >> three components while the 2.0 window allows breaking changes? > >> > >> Thanks, > >> Mathieu Monet > >> > >> -- > >> Drafted-by: Claude Code (AI assistant) and > >> reviewed by me before sending. > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail:[email protected] > >> For additional commands, e-mail:[email protected] > >> > >>
