Hi Mathieu,
Thanks for proposal and feel free to create PRs (according to the plan
which you've mentioned) if you didn't already (I'm a little behind in
the reviewing PRs, but sooner or later I will catch up and review all
waiting helm chart-related).
Regarding your questions:
> For the dag processor, is a separate `sets` list (referencing bundles
by name) acceptable, or would you prefer per-bundle overrides on the
dagBundleConfigList entries (as discussed on #61039)? The triggerer is
scoped by team, not bundle, so `sets` seems the natural fit there either
way.
As of now I would say "we will see". Implementation wise and in terms of
behaviour, the way how the feature works should be the same between
components. As it is possible to do for the workers and triggerer, it
raises a question for the Dag Processor as it can have different schema,
which would make more sense. One of the biggest concern regarding the
Worker Sets during the implementation was the complexity of it and this
is the reason for having (if possible and reasonable) one implementation
behind the `sets` feature. Nevertheless, when we will reach out the PRs,
maybe I will have ready the thing which I plan to do which is how the
schema for Helm Chart 2 should look like as the reference point for the
work (with also one issue currently affecting the chart itself which is
the schema validation). Maybe before the PR I will have that ready.
Let's differ that one (unless someone will have some opinion on that
too) till the implementation phase.
> Should this target 2.0.0?
Yes. I think we might consider the back-porting of it if reasonable
amount of users will want to and the works on Helm Chart 2 will take too
long (there is no specific target date, but I plan to work on it a bit
more in the upcoming couple months).
> The default instance's selectors also match set pods (same as worker
sets today). Worth adding a `<component>-set: default` label to
all three components while the 2.0 window allows breaking changes?
Yes. I think I saw PR with it somewhere already, but I haven't finished
review of it yet.
Best,
Przemek
On 20/10/2026 20:12, Nikolas Oliveira wrote:
Hey Mathieu,
I appreciate you contributing this back! I don't work very closely with
Helm, so I'm not the best candidate to review this proposal, but from a
multi team perspective, I'd really love to have this, especially for
triggerers, where having one per team is critical. I think the proposal
looks reasonable to me. If you haven't already, I'd start putting up some
PRs if you're not getting much traction here.
Cheers,
Niko
On Mon, Oct 12, 2026 at 4:37 AM Mathieu Monet<[email protected]>
wrote:
Hi all,
I'd like to propose adding `sets` to the triggerer and dag processor in
the Helm chart, following the existing `workers.celery.sets` pattern, and
get direction before opening PRs.
Why
---
With `core.multi_team` enabled, each team needs a triggerer started with
`--team-name`, otherwise deferred tasks of that team's Dags never resume
(multi-team.rst, "Triggerer" section). The chart can only render one
triggerer today, so a multi-team deployment on the official chart is
not fully functional without patching templates. Running one dag
processor per team or bundle (`--bundle-name`) is the matching need on
the parsing side, for isolation and sizing.
We run this in production (Airflow 3.3.2, several teams in one instance)
through a wrapper chart that copies the upstream triggerer and dag
processor templates in a loop. I'd rather contribute to it than keep that
copy in sync.
Proposal
--------
triggerer:
enableDefault: true # render the unsuffixed triggerer
teamName: ~ # -> --team-name
queues: ~ # -> --queues
extraArgs: [] # appended to the default args
sets: [] # each item overrides any triggerer.* key
dagProcessor:
enableDefault: true
bundleNames: [] # -> repeated --bundle-name
extraArgs: []
sets: []
- A set `<n>` renders `<fullname>-<component>-<n>` with its own
ServiceAccount, PDB and ScaledObject, and a `<component>-set: <n>`
label. Values merge exactly like worker sets (shared helper).
- Sets reference bundles and teams by name only. Render-time validation
checks them against dagBundleConfigList, so there is no second list
to keep in sync (the concern raised on #61039). Users set small scope
fields instead of a full CLI in `args` (the concern raised on #61963).
- With every new field empty, the output is identical to today's.
Planned as small PRs: shared set helpers (generalising
workersMergeValues), triggerer.sets, dagProcessor.sets, then optional
multi-team conveniences (deriving bundles from team_name, team-aware
KEDA query).
Related: #26494, #61037, #61039, #61963, #64037.
Questions
---------
1. For the dag processor, is a separate `sets` list (referencing
bundles by name) acceptable, or would you prefer per-bundle overrides
on the dagBundleConfigList entries (as discussed on #61039)? The
triggerer is scoped by team, not bundle, so `sets` seems the natural
fit there either way.
2. Should this target 2.0.0?
3. The default instance's selectors also match set pods (same as worker
sets today). Worth adding a `<component>-set: default` label to all
three components while the 2.0 window allows breaking changes?
Thanks,
Mathieu Monet
--
Drafted-by: Claude Code (AI assistant) and
reviewed by me before sending.
---------------------------------------------------------------------
To unsubscribe, e-mail:[email protected]
For additional commands, e-mail:[email protected]
