To be clear, you’re hoping that setting the slots to 1 will cause the tasks across district dags to run in order based on the assumption that they’ll queue up and then execute off the pool?
I don’t think it will quite work that way - there’s no guarantee the scheduler will execute your tasks across dags in any particular sequence, and if 1 is “faster” than the other for sure they don’t “line up”. Thus, no way to ensure they’ll queue in the right order. I successfully use pools across many dags to limit access to an expensive resource and it works really well, but my design doesn’t require they execute in any particular order, each idempotent. I’m curious as to your design/constraints - could you elaborate? Brian Sent from a device with less than stellar autocorrect > On Apr 6, 2018, at 3:46 PM, Manish Trivedi <trivman...@gmail.com> wrote: > > Hi Airflow devs, > > I have a use case to limit the # of calls to a certain database. I am using > the pool along with priority weight to schedule the tasks to the slot pool. > I have around 5 operators that I need to execute in serial order across > different dags. > > Slot pool is created with "1" slot to ensure sequential exection. I am not > able to achieve the desired function with current setup.
