+1 as it relies on pickles On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <a...@apache.org> wrote:
> I say lets remove it in 2.0. > > > On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl> > wrote: > > > > Are we going to remove the MesosExecutor? I think it is a safe bet since > in > > the Survey of Ash (thanks again for that!), it also shows that nobody is > > really using it. > > > > https://jira.apache.org/jira/browse/AIRFLOW-3969 > > > > Cheers, Fokko > > > > Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <a...@apache.org>: > > > >> I’m glad yarn wasn’t the only option - it would have meant I’d have > never > >> been in a position to use Airflow! (Many of our workflows don’t touch > >> EMR/Hadoop, and running Celery is much more of a known element to a > python > >> developer than configuring Yarn) > >> > >>> On 11 Feb 2019, at 17:14, Maxime Beauchemin < > maximebeauche...@gmail.com> > >> wrote: > >>> > >>> From memory, I think MesosExecutor depends on pickling to get DAG > >>> definitions to workers, which we should also deprecate. > >>> > >>> About CeleryExecutor, we never had the intention to make it the > >> recommended > >>> option for production early on. The intent back in 2014 was to write a > >>> YarnExecutor quickly (that was before the rise of k8s), but realized > that > >>> container support wasn't ready in Yarn at the time, and had hard > >>> restrictions on what we could install on Hadoop/Yarn data nodes, so we > >>> moved forward with Celery. > >>> > >>> I think it's practical to have LocalExecutor around though for unit > test, > >>> local use, and for POCs. > >>> > >>> Max > >>> > >>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < > >>> o.girar...@lateral-thoughts.com> wrote: > >>> > >>>> Hi, we tried to use it for a new project 1year ago, but it was too > >> broken > >>>> to be usable, and we didn't have the time to make it work. > >>>> Of course we are still hoping to use it one day, but that doesn't mean > >> it > >>>> should not move to contrib, it was already clear from the > documentation > >>>> that it was not a first class citizen. > >>>> > >>>> Regards, > >>>> > >>>> Olivier. > >>>> > >>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <a...@apache.org> a > >> écrit > >>>> : > >>>> > >>>>> That PR pre-dates 1.9.0 being released. > >>>>> > >>>>>> On 27 Jan 2019, at 06:57, airflowuser <airflowu...@protonmail.com > >>>> .INVALID> > >>>>> wrote: > >>>>>> > >>>>>> The Mesos executor is being used (at least by some people) and there > >> is > >>>>> an open PR to extend it: > >>>>>> https://github.com/apache/airflow/pull/3739 > >>>>>> > >>>>>> > >>>>>> Sent with ProtonMail Secure Email. > >>>>>> > >>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > >>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik < > kaxiln...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>>> What about Dask Executor as well. > >>>>>>> > >>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < > >>>> daniel.imber...@gmail.com > >>>>>>> wrote: > >>>>>>> > >>>>>>>> @sid basically my response lol > >>>>>>>> I haven't seen any emails from any users and if it's been broken > for > >>>>> two > >>>>>>>> versions then there doesn't seem to be enough interest to maintain > >>>> it. > >>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39...@gmail.com wrote: > >>>>>>>>> > >>>>>>>>> LOL :_0 > >>>>>>>>> -s > >>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin bdbr...@gmail.com > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> I agree. Maybe we should make a choice on what we consider 1st > >>>> class > >>>>>>>>>> executors and move others to contrib (I'm looking at you celery) > >>>>>>>>>> Sent from my iPhone > >>>>>>>>>> > >>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor a...@apache.org > >> wrote: > >>>>>>>>>>> Is anyone using the Mesos Executor? I think we should deprecate > >>>> and > >>>>>>>>>>> remove it. > >>>>>>>>>>> The reason I say that is I have a feeling it's been broken > since > >>>>>>>>>>> 1.9.0 > >>>>>>>>> > >>>>>>>>>> and no-one has noticed: > >>>>>>>>>> > >>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin): > >>>>>>>>>>> def start(self): > >>>>>>>>>>> # ... > >>>>>>>>>>> self.log.info( > >>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : > >>>>>>>>>>> %s, > >>>>>>>>> > >>>>>>>>>> checkpoint : %s', > >>>>>>>>>> > >>>>>>>>>>> master, framework.name, > >>>>>>>>>>> str(task_cpu), str(task_memory), > >>>> str(framework.checkpoint) > >>>>>>>>>>> ) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But > >>>> given > >>>>>>>>>>> this > >>>>>>>>>>> bug has been in since > >>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29 > >>>>>>>> > >>>>>>>>>> which was in the 1.9.0 release I propose we simply remove this > >>>>> executor > >>>>>>>>>> from master. > >>>>>>>>>> > >>>>>>>>>>> Thoughts? > >>>>>>>>>>> Ash > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >> > >> > >
