I’m glad yarn wasn’t the only option - it would have meant I’d have never been in a position to use Airflow! (Many of our workflows don’t touch EMR/Hadoop, and running Celery is much more of a known element to a python developer than configuring Yarn)
> On 11 Feb 2019, at 17:14, Maxime Beauchemin <maximebeauche...@gmail.com> > wrote: > > From memory, I think MesosExecutor depends on pickling to get DAG > definitions to workers, which we should also deprecate. > > About CeleryExecutor, we never had the intention to make it the recommended > option for production early on. The intent back in 2014 was to write a > YarnExecutor quickly (that was before the rise of k8s), but realized that > container support wasn't ready in Yarn at the time, and had hard > restrictions on what we could install on Hadoop/Yarn data nodes, so we > moved forward with Celery. > > I think it's practical to have LocalExecutor around though for unit test, > local use, and for POCs. > > Max > > On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < > o.girar...@lateral-thoughts.com> wrote: > >> Hi, we tried to use it for a new project 1year ago, but it was too broken >> to be usable, and we didn't have the time to make it work. >> Of course we are still hoping to use it one day, but that doesn't mean it >> should not move to contrib, it was already clear from the documentation >> that it was not a first class citizen. >> >> Regards, >> >> Olivier. >> >> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <a...@apache.org> a écrit >> : >> >>> That PR pre-dates 1.9.0 being released. >>> >>>> On 27 Jan 2019, at 06:57, airflowuser <airflowu...@protonmail.com >> .INVALID> >>> wrote: >>>> >>>> The Mesos executor is being used (at least by some people) and there is >>> an open PR to extend it: >>>> https://github.com/apache/airflow/pull/3739 >>>> >>>> >>>> Sent with ProtonMail Secure Email. >>>> >>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ >>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik <kaxiln...@gmail.com> >>> wrote: >>>> >>>>> What about Dask Executor as well. >>>>> >>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < >> daniel.imber...@gmail.com >>>>> wrote: >>>>> >>>>>> @sid basically my response lol >>>>>> I haven't seen any emails from any users and if it's been broken for >>> two >>>>>> versions then there doesn't seem to be enough interest to maintain >> it. >>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39...@gmail.com wrote: >>>>>>> >>>>>>> LOL :_0 >>>>>>> -s >>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbr...@gmail.com >>>>>>> wrote: >>>>>>> >>>>>>>> I agree. Maybe we should make a choice on what we consider 1st >> class >>>>>>>> executors and move others to contrib (I'm looking at you celery) >>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor a...@apache.org wrote: >>>>>>>>> Is anyone using the Mesos Executor? I think we should deprecate >> and >>>>>>>>> remove it. >>>>>>>>> The reason I say that is I have a feeling it's been broken since >>>>>>>>> 1.9.0 >>>>>>> >>>>>>>> and no-one has noticed: >>>>>>>> >>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin): >>>>>>>>> def start(self): >>>>>>>>> # ... >>>>>>>>> self.log.info( >>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : >>>>>>>>> %s, >>>>>>> >>>>>>>> checkpoint : %s', >>>>>>>> >>>>>>>>> master, framework.name, >>>>>>>>> str(task_cpu), str(task_memory), >> str(framework.checkpoint) >>>>>>>>> ) >>>>>>>>> >>>>>>>>> >>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But >> given >>>>>>>>> this >>>>>>>>> bug has been in since >>>>>> >>>>>> >>> >> https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29 >>>>>> >>>>>>>> which was in the 1.9.0 release I propose we simply remove this >>> executor >>>>>>>> from master. >>>>>>>> >>>>>>>>> Thoughts? >>>>>>>>> Ash >>>> >>>> >>> >>> >>
