GitHub user dstandish closed a discussion: Should you be able to override pools in backfill?
Note: I originally posted this on slack but I'm reposting here for posterity So i was going to implement the "pool override" behavior on backfill. But it strikes me that, the option to **provide one "pool override" that overrides pool for all tasks in the backfill**, that just doesn't make a lot of sense. Obviously different tasks may use different pools. so all-or-nothing override doesn't make a lot of sense. And note, in **the old backfill, it just straight up ignored pools unless you provided an override** --- by default just no pools were used. this was surprising to me. But ok, i don't really like that option. so what's the alternative? we could **allow user to provide a dict of tasks or something** where you optionally override the pool for any specific task. and then if no override is provided, then use the normal pool. this would work and make more sense. but it seems like a rather kludgy interface. WDYT? Separately though, **does anyone feel that backfill should still completely ignore pools by default?** Anyone think that behavior should be preserved? Responses from slack: > > raphaelauv > Yesterday at 1:22 PM > Backfill was ignoring pools , that's a bug > Backfill is a trick on time , not on the code of the dag , airflow should > respect the code of the dag no matter what > > > Daniel Standish > [Yesterday at 1:24 > PM](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729455867348749?thread_ts=1729436754.752759&cid=C06K9Q5G2UA) > And your opinion concerning a possible interface for overriding pools? > [1:24](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729455889539649?thread_ts=1729436754.752759&cid=C06K9Q5G2UA) > Maybe it’s not really necessary, or not as important, now that we will have > separate concurrency controls for backfill? > > > Jarek Potiuk > Yesterday at 2:54 PM > Yeah. Backfill ignoring pools was a bug. > I think we should not override pools at all with backfill - see the discusion > on backfill vs. deferrable. There is a reason a task needs a pool, and > backfill ignoring that is a bad idea. (edited) > :heavy_plus_sign: > 2 > > [2:56](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729461380822389?thread_ts=1729436754.752759&cid=C06K9Q5G2UA) > There are probably a couple of “override” scenarios, but - paraphrasing your > request > [@dstandish](https://apache-airflow.slack.com/team/UG7RJQEV9) > from the backfill/deferrable discussion - we should only allow > overridability if someone comes up with - maybe a made up but realistic - > case where it makes sense (edited) > > > raphaelauv > Yesterday at 5:05 PM > a possible interface for overriding pools > maybe some people would expect this feature ( and I don't see what are the > purpose of pools in that case in this scenario ) , but they can always > copy paste the dag > sufix the name > edit the pool usage in the tasks > trigger the backill of the dag > > > Daniel Standish > [6 minutes > ago](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539875337429?thread_ts=1729436754.752759&cid=C06K9Q5G2UA) > and I don't see what are the purpose of pools in that case in this scenario > [12:44](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539888781359?thread_ts=1729436754.752759&cid=C06K9Q5G2UA) > pools are a bit of an enigma > [12:45](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539918844969?thread_ts=1729436754.752759&cid=C06K9Q5G2UA) > alright, well, if people say no override, then no override there shall be > [12:45](https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1729539926740689?thread_ts=1729436754.752759&cid=C06K9Q5G2UA) > i will make a lazy consensus vote > > > raphaelauv > 4 minutes ago > pools are a bit of an enigma > I only use them as cross-dag semaphore > > GitHub link: https://github.com/apache/airflow/discussions/43235 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
