Thanks Przemek and Niko.

Quick update: #75945 (xchwan) already implements dagProcessor.sets close
to what I proposed, so I'll leave the dag processor side to that PR. I've
reviewed it with the cases we hit in production.

Next I'll open triggerer.sets, following the same conventions as #75945
(enableDefault, <component>-set label, PDB exclusion, ServiceAccount
handling).


On Wed, Nov 11, 2026 08:24 PM, Przemysław Mirowski via dev <
[email protected]> wrote:

> Hi Mathieu,
>
> Thanks for proposal and feel free to create PRs (according to the plan
> which you've mentioned) if you didn't already (I'm a little behind in
> the reviewing PRs, but sooner or later I will catch up and review all
> waiting helm chart-related).
>
> Regarding your questions:
>
>  > For the dag processor, is a separate `sets` list (referencing bundles
> by name) acceptable, or would you prefer per-bundle overrides on the
> dagBundleConfigList entries (as discussed on #61039)? The triggerer is
> scoped by team, not bundle, so `sets` seems the natural fit there either
> way.
>
> As of now I would say "we will see". Implementation wise and in terms of
> behaviour, the way how the feature works should be the same between
> components. As it is possible to do for the workers and triggerer, it
> raises a question for the Dag Processor as it can have different schema,
> which would make more sense. One of the biggest concern regarding the
> Worker Sets during the implementation was the complexity of it and this
> is the reason for having (if possible and reasonable) one implementation
> behind the `sets` feature. Nevertheless, when we will reach out the PRs,
> maybe I will have ready the thing which I plan to do which is how the
> schema for Helm Chart 2 should look like as the reference point for the
> work (with also one issue currently affecting the chart itself which is
> the schema validation). Maybe before the PR I will have that ready.
> Let's differ that one (unless someone will have some opinion on that
> too) till the implementation phase.
>
>  > Should this target 2.0.0?
>
> Yes. I think we might consider the back-porting of it if reasonable
> amount of users will want to and the works on Helm Chart 2 will take too
> long (there is no specific target date, but I plan to work on it a bit
> more in the upcoming couple months).
>
>  > The default instance's selectors also match set pods (same as worker
> sets today). Worth adding a `<component>-set: default` label to
> all three components while the 2.0 window allows breaking changes?
>
> Yes. I think I saw PR with it somewhere already, but I haven't finished
> review of it yet.
>
> Best,
> Przemek
>
> On 20/10/2026 20:12, Nikolas Oliveira wrote:
> > Hey Mathieu,
> >
> > I appreciate you contributing this back! I don't work very closely with
> > Helm, so I'm not the best candidate to review this proposal, but from a
> > multi team perspective, I'd really love to have this, especially for
> > triggerers, where having one per team is critical. I think the proposal
> > looks reasonable to me. If you haven't already, I'd start putting up some
> > PRs if you're not getting much traction here.
> >
> > Cheers,
> > Niko
> >
> > On Mon, Oct 12, 2026 at 4:37 AM Mathieu Monet<[email protected]>
> > wrote:
> >
> >> Hi all,
> >>
> >> I'd like to propose adding `sets` to the triggerer and dag processor in
> >> the Helm chart, following the existing `workers.celery.sets` pattern,
> and
> >> get direction before opening PRs.
> >>
> >> Why
> >> ---
> >> With `core.multi_team` enabled, each team needs a triggerer started with
> >> `--team-name`, otherwise deferred tasks of that team's Dags never resume
> >> (multi-team.rst, "Triggerer" section). The chart can only render one
> >> triggerer today, so a multi-team deployment on the official chart is
> >> not fully functional without patching templates. Running one dag
> >> processor per team or bundle (`--bundle-name`) is the matching need on
> >> the parsing side, for isolation and sizing.
> >>
> >> We run this in production (Airflow 3.3.2, several teams in one instance)
> >> through a wrapper chart that copies the upstream triggerer and dag
> >> processor templates in a loop. I'd rather contribute to it than keep
> that
> >> copy in sync.
> >>
> >> Proposal
> >> --------
> >> triggerer:
> >>    enableDefault: true    # render the unsuffixed triggerer
> >>    teamName: ~            # -> --team-name
> >>    queues: ~              # -> --queues
> >>    extraArgs: []          # appended to the default args
> >>    sets: []               # each item overrides any triggerer.* key
> >> dagProcessor:
> >>    enableDefault: true
> >>    bundleNames: []        # -> repeated --bundle-name
> >>    extraArgs: []
> >>    sets: []
> >>
> >> - A set `<n>` renders `<fullname>-<component>-<n>` with its own
> >>    ServiceAccount, PDB and ScaledObject, and a `<component>-set: <n>`
> >>    label. Values merge exactly like worker sets (shared helper).
> >> - Sets reference bundles and teams by name only. Render-time validation
> >>    checks them against dagBundleConfigList, so there is no second list
> >>    to keep in sync (the concern raised on #61039). Users set small scope
> >>    fields instead of a full CLI in `args` (the concern raised on
> #61963).
> >> - With every new field empty, the output is identical to today's.
> >>
> >> Planned as small PRs: shared set helpers (generalising
> >> workersMergeValues), triggerer.sets, dagProcessor.sets, then optional
> >> multi-team conveniences (deriving bundles from team_name, team-aware
> >> KEDA query).
> >>
> >> Related: #26494, #61037, #61039, #61963, #64037.
> >>
> >> Questions
> >> ---------
> >> 1. For the dag processor, is a separate `sets` list (referencing
> >>     bundles by name) acceptable, or would you prefer per-bundle
> overrides
> >>     on the dagBundleConfigList entries (as discussed on #61039)? The
> >>     triggerer is scoped by team, not bundle, so `sets` seems the natural
> >>     fit there either way.
> >> 2. Should this target 2.0.0?
> >> 3. The default instance's selectors also match set pods (same as worker
> >>     sets today). Worth adding a `<component>-set: default` label to all
> >>     three components while the 2.0 window allows breaking changes?
> >>
> >> Thanks,
> >> Mathieu Monet
> >>
> >> --
> >> Drafted-by: Claude Code (AI assistant) and
> >> reviewed by me before sending.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail:[email protected]
> >> For additional commands, e-mail:[email protected]
> >>
> >>

Reply via email to