Hey Gerard, For our use case we have 1 pool per dag and each dag has between 50 and 100 tasks. We have the pool size set between 3 and 5 in most cases and have seen up to 50 running tasks within the pool
Best, David > On Nov 1, 2016, at 11:17 AM, Gerard Toonstra <gtoons...@gmail.com> wrote: > > David, > > When you say "massive" oversubscribing, are you running a lot of dags in > parallel that > use your configured pools? Access to pools is not atomic at the moment. > > Can you also quantify "massive" ? Not that it matters, but to get a better > idea. > > Rgds, > > Gerard > > > On Tue, Nov 1, 2016 at 4:09 PM, siddharth anand <san...@apache.org> wrote: > >> Yes.. we have seen the over-subscription of pools. We do need a fix for it >> --- I don't believe there is one. We need someone to own and fix it.. happy >> to review a PR. >> >> We use pools at Agari for all of our needs. We are okay with mild >> oversubscription, so we do see numbers slightly higher, but our pipelines >> work fine with that level. In your example, the oversubscription is much >> higher. >> -s >> >> On Tue, Nov 1, 2016 at 6:38 AM, David Kegley <k...@b23.io> wrote: >> >>> I've been seeing some weird behavior when using Airflow's execution pool >>> feature. Pools have been massively over-filled leading to failed tasks. >> I >>> created a bug for this issue, but in the mean time, has anyone else >>> experienced this behavior? >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-584 >>> >>> Best, >>> David >>> >>
