Following discussion Dask's gitter, I created an issue in Dask's github : https://github.com/dask/dask/issues/5803
Let's see if we can get someone from Dask community interested. On Fri, Jan 17, 2020 at 10:00 PM Jarek Potiuk <[email protected]> wrote: > Good idea :) doing that, > > On Fri, Jan 17, 2020 at 9:58 PM Daniel Imberman <[email protected]> > wrote: > >> Maybe we can reach out to a company that does Dask as a service? >> >> via Newton Mail [ >> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 >> ] >> On Fri, Jan 17, 2020 at 9:31 AM, Jarek Potiuk <[email protected]> >> wrote: >> Yeah. I think if we do not find anyone willing to champion it (no matter >> committer or contributor), I would be for dropping it. >> >> J. >> >> On Fri, Jan 17, 2020 at 6:07 PM Daniel Imberman < >> [email protected]> >> wrote: >> >> > I think we need to ask “who is going to champion this executor.” I see >> > that it is being used (a bit), but am concerned if no one with >> knowledge of >> > this executor is willing to maintain it. >> > >> > I’ve personally never used Dask and the DaskExecutor isn’t super high on >> > my priority list compared to things like autoscaling, DAG serialization, >> > etc. >> > >> > via Newton Mail [ >> > >> https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 >> > ] >> > On Fri, Jan 17, 2020 at 6:07 AM, Jarek Potiuk <[email protected] >> > >> > wrote: >> > Do we have anyone here who uses Dask Executor and would like to test >> it/fix >> > the tests. They are marked now as xfailed (expected to fail) and it >> would >> > be great to fix them. >> > >> > J. >> > >> > >> > On Tue, Jan 14, 2020 at 12:18 AM Darren Weber < >> [email protected] >> > > >> > wrote: >> > >> > > +1 for keeping it and fixing tests >> > > >> > > PS, I also noticed the skipped tests while looking at an option to use >> > the >> > > async client feature; if/when I get time to get back on that and >> figure >> > out >> > > how the test setup needs to work, I might also discover how to enable >> > tests >> > > for the non-async executor. No promises, just noting that I'm aware >> of it >> > > too. >> > > >> > > On Mon, Jan 13, 2020 at 8:06 AM Jarek Potiuk < >> [email protected]> >> > > wrote: >> > > >> > > > For now I marked the skipped tests we had (including Dask) as >> > > > pytest.mark.xfail (means - expected to fail). They will be executed >> and >> > > > summarized as XFail tests and we will have to deal with them at some >> > > point. >> > > > >> > > > I think we will have to decide if we want to keep it or not, and >> either >> > > > remove both tests and executor or fix the tests. >> > > > >> > > > J. >> > > > >> > > > On Mon, Jan 13, 2020 at 4:40 PM Shaw, Damian P. < >> > > > [email protected]> wrote: >> > > > >> > > > > FYI I used Dash instead of Local Executor when first starting >> > Airflow, >> > > it >> > > > > was a great way to make sure the Executor and Scheduler weren’t >> tied >> > to >> > > > > each other with no difficulty in set-up. But once I actually >> started >> > > > > deploying to multiple boxes I needed queue names pretty quickly. >> So >> > not >> > > > > going to say it's needed but for me it was a helpful stepping >> stone. >> > > > > >> > > > > >> > > > > -----Original Message----- >> > > > > From: Ash Berlin-Taylor <[email protected]> >> > > > > Sent: Sunday, January 12, 2020 17:38 >> > > > > To: [email protected] >> > > > > Cc: [email protected] >> > > > > Subject: Re: Remove Dask Executor in Airflow 2.0 ? >> > > > > >> > > > > It hasn't been discussed before, but unlike the Mesos one this one >> > was >> > > > > seen a (tiny) bit of activity in 1.10 so at least one person is >> using >> > > it >> > > > > https://github.com/apache/airflow/pull/5273 >> > > > > >> > > > > On Jan 12 2020, at 9:05 pm, Jarek Potiuk < >> [email protected]> >> > > > wrote: >> > > > > > I am finishing the PR on separating integrations and improving >> our >> > CI >> > > > > > footprint (https://github.com/apache/airflow/pull/7091) but >> during >> > > > > > this change I have found that we have - apparently - >> dysfunctional >> > > > > > DaskExecutor in Airflow 2.0. >> > > > > > >> > > > > > There is a "test_dask_executor.py" for which all tests are >> skipped. >> > > > > > And they fail when I try to run the tests. I tried to look for >> any >> > > > > > reference in devlist archives but I couldn't find anything about >> > it. >> > > > > > >> > > > > > Can someone shed some light on this? Should we remove Dask >> executor >> > > > > > completely from Airflow 2.0 ? Or should we fix the >> tests/executor ? >> > > > > > Has it been discussed ? >> > > > > > >> > > > > > J. >> > > > > > >> > > > > > -- >> > > > > > Jarek Potiuk >> > > > > > Polidea <https://www.polidea.com/> | Principal Software >> Engineer >> > > > > > >> > > > > > M: +48 660 796 129 <+48660796129> >> > > > > > [image: Polidea] <https://www.polidea.com/> >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> > >> =============================================================================== >> > > > > >> > > > > Please access the attached hyperlink for an important electronic >> > > > > communications disclaimer: >> > > > > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html >> > > > > >> > > > >> > > >> > >> =============================================================================== >> > > > > >> > > > > >> > > > >> > > > >> > > > -- >> > > > >> > > > Jarek Potiuk >> > > > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > > > >> > > > M: +48 660 796 129 <+48660796129> >> > > > [image: Polidea] <https://www.polidea.com/> >> > > > >> > > >> > > >> > > -- >> > > Darren L. Weber, Ph.D. >> > > http://psdlw.users.sourceforge.net/ >> > > http://psdlw.users.sourceforge.net/wordpress/ >> > > >> > >> > >> > -- >> > >> > Jarek Potiuk >> > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > >> > M: +48 660 796 129 <+48660796129> >> > [image: Polidea] <https://www.polidea.com/> >> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
