Re: [DISCUSS] Backfill concurrency re AIP-78

Ok want to focus the discussion on one question. Proposal: DAG.max_active_runs will apply to non-backfill dag runs and will *not* apply to backfill dag runs. Backfill.max_active_runs is a required param for backfills and is evaluated entirely separately from DAG.max_active_runs. In other words, DA

Re: [DISCUSS] Backfill concurrency re AIP-78

Thanks Jarek. Let me respond to your comments. Then I'll follow up with another message trying to focus the discussion on my question. > I think it should be controllable at the moment when you start backfill. Current state is, when you create a backfill you set max_active_runs for it. And t

Re: [DISCUSS] Backfill concurrency re AIP-78

I think it should be controllable at the moment when you start backfill. IMHO default behaviour should be that there should be very little concurrency reserved for backfills so that you could run backfills without impacting regular runs - say "max 10 backfill task instances" and "max 3 backfill ru

[DISCUSS] Backfill concurrency re AIP-78

Just adding the [DISCUSS] prefix, which I forgot to add. On Thu, Oct 3, 2024 at 4:23 PM Daniel Standish < daniel.stand...@astronomer.io> wrote: > Ok so, I'm thinking through what makes sense re concurrency control in > backfill. > > It was referred to >