Are we going to remove the MesosExecutor? I think it is a safe bet since in the Survey of Ash (thanks again for that!), it also shows that nobody is really using it.
https://jira.apache.org/jira/browse/AIRFLOW-3969 Cheers, Fokko Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <a...@apache.org>: > I’m glad yarn wasn’t the only option - it would have meant I’d have never > been in a position to use Airflow! (Many of our workflows don’t touch > EMR/Hadoop, and running Celery is much more of a known element to a python > developer than configuring Yarn) > > > On 11 Feb 2019, at 17:14, Maxime Beauchemin <maximebeauche...@gmail.com> > wrote: > > > > From memory, I think MesosExecutor depends on pickling to get DAG > > definitions to workers, which we should also deprecate. > > > > About CeleryExecutor, we never had the intention to make it the > recommended > > option for production early on. The intent back in 2014 was to write a > > YarnExecutor quickly (that was before the rise of k8s), but realized that > > container support wasn't ready in Yarn at the time, and had hard > > restrictions on what we could install on Hadoop/Yarn data nodes, so we > > moved forward with Celery. > > > > I think it's practical to have LocalExecutor around though for unit test, > > local use, and for POCs. > > > > Max > > > > On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < > > o.girar...@lateral-thoughts.com> wrote: > > > >> Hi, we tried to use it for a new project 1year ago, but it was too > broken > >> to be usable, and we didn't have the time to make it work. > >> Of course we are still hoping to use it one day, but that doesn't mean > it > >> should not move to contrib, it was already clear from the documentation > >> that it was not a first class citizen. > >> > >> Regards, > >> > >> Olivier. > >> > >> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <a...@apache.org> a > écrit > >> : > >> > >>> That PR pre-dates 1.9.0 being released. > >>> > >>>> On 27 Jan 2019, at 06:57, airflowuser <airflowu...@protonmail.com > >> .INVALID> > >>> wrote: > >>>> > >>>> The Mesos executor is being used (at least by some people) and there > is > >>> an open PR to extend it: > >>>> https://github.com/apache/airflow/pull/3739 > >>>> > >>>> > >>>> Sent with ProtonMail Secure Email. > >>>> > >>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > >>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <kaxiln...@gmail.com> > >>> wrote: > >>>> > >>>>> What about Dask Executor as well. > >>>>> > >>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < > >> daniel.imber...@gmail.com > >>>>> wrote: > >>>>> > >>>>>> @sid basically my response lol > >>>>>> I haven't seen any emails from any users and if it's been broken for > >>> two > >>>>>> versions then there doesn't seem to be enough interest to maintain > >> it. > >>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39...@gmail.com wrote: > >>>>>>> > >>>>>>> LOL :_0 > >>>>>>> -s > >>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbr...@gmail.com > >>>>>>> wrote: > >>>>>>> > >>>>>>>> I agree. Maybe we should make a choice on what we consider 1st > >> class > >>>>>>>> executors and move others to contrib (I'm looking at you celery) > >>>>>>>> Sent from my iPhone > >>>>>>>> > >>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor a...@apache.org > wrote: > >>>>>>>>> Is anyone using the Mesos Executor? I think we should deprecate > >> and > >>>>>>>>> remove it. > >>>>>>>>> The reason I say that is I have a feeling it's been broken since > >>>>>>>>> 1.9.0 > >>>>>>> > >>>>>>>> and no-one has noticed: > >>>>>>>> > >>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin): > >>>>>>>>> def start(self): > >>>>>>>>> # ... > >>>>>>>>> self.log.info( > >>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : > >>>>>>>>> %s, > >>>>>>> > >>>>>>>> checkpoint : %s', > >>>>>>>> > >>>>>>>>> master, framework.name, > >>>>>>>>> str(task_cpu), str(task_memory), > >> str(framework.checkpoint) > >>>>>>>>> ) > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But > >> given > >>>>>>>>> this > >>>>>>>>> bug has been in since > >>>>>> > >>>>>> > >>> > >> > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29 > >>>>>> > >>>>>>>> which was in the 1.9.0 release I propose we simply remove this > >>> executor > >>>>>>>> from master. > >>>>>>>> > >>>>>>>>> Thoughts? > >>>>>>>>> Ash > >>>> > >>>> > >>> > >>> > >> > >
