I think I lean towards the built-in/manager approach as it is less concurrency code we have to manage/maintain in Airflow, though I'm not hugely happy about another process :(
-ash > On 29 Apr 2019, at 07:33, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > Hello Everyone, > > I think we need some more pairs of eyes to take a look at potential fixes > we have for the pesky LocalExecutorTest that we are all experiencing with > our Travis builds. Once we solve it I think we should be much closer to > have stable builds - including some other flaky test fixes merged recently. > > It turned out that the problem relates to quite deep internals of how data > is passed between processes using multiprocessing queues. It's really deep > in the core processing of Airflow so I think it would be great if also > other experienced Airflowers review and comment it and help to select the > best solution as we could have missed something. > > I was looking at it together with Ash and Bas and I (a bit too fast) merged > a preliminary version of the fix last week. We reverted it later as it > turned out to have some side effects, so we know we have to be careful with > this one. > > After more detailed analysis and discussions with Omar, we have now two > potential candidates to fix it. Both are green and from local testing - > both are solving the problem in a different way. > > - https://github.com/apache/airflow/pull/5199 > - https://github.com/apache/airflow/pull/5200 > > I tried to describe the problem, solution candidates with Pros and Cons in > the JIRA ticket : > https://issues.apache.org/jira/browse/AIRFLOW-4401 > > I'd love if we can get reviews in the PRs and input to discussion on which > solution to choose. > > J. > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > E: jarek.pot...@polidea.com
