Since we are now doing version 2.0 spring cleaning can we revisit this? On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin < maximebeauche...@gmail.com> wrote:
> +1 as it relies on pickles > > On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <a...@apache.org> wrote: > > > I say lets remove it in 2.0. > > > > > On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl> > > wrote: > > > > > > Are we going to remove the MesosExecutor? I think it is a safe bet > since > > in > > > the Survey of Ash (thanks again for that!), it also shows that nobody > is > > > really using it. > > > > > > https://jira.apache.org/jira/browse/AIRFLOW-3969 > > > > > > Cheers, Fokko > > > > > > Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor <a...@apache.org > >: > > > > > >> I’m glad yarn wasn’t the only option - it would have meant I’d have > > never > > >> been in a position to use Airflow! (Many of our workflows don’t touch > > >> EMR/Hadoop, and running Celery is much more of a known element to a > > python > > >> developer than configuring Yarn) > > >> > > >>> On 11 Feb 2019, at 17:14, Maxime Beauchemin < > > maximebeauche...@gmail.com> > > >> wrote: > > >>> > > >>> From memory, I think MesosExecutor depends on pickling to get DAG > > >>> definitions to workers, which we should also deprecate. > > >>> > > >>> About CeleryExecutor, we never had the intention to make it the > > >> recommended > > >>> option for production early on. The intent back in 2014 was to write > a > > >>> YarnExecutor quickly (that was before the rise of k8s), but realized > > that > > >>> container support wasn't ready in Yarn at the time, and had hard > > >>> restrictions on what we could install on Hadoop/Yarn data nodes, so > we > > >>> moved forward with Celery. > > >>> > > >>> I think it's practical to have LocalExecutor around though for unit > > test, > > >>> local use, and for POCs. > > >>> > > >>> Max > > >>> > > >>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < > > >>> o.girar...@lateral-thoughts.com> wrote: > > >>> > > >>>> Hi, we tried to use it for a new project 1year ago, but it was too > > >> broken > > >>>> to be usable, and we didn't have the time to make it work. > > >>>> Of course we are still hoping to use it one day, but that doesn't > mean > > >> it > > >>>> should not move to contrib, it was already clear from the > > documentation > > >>>> that it was not a first class citizen. > > >>>> > > >>>> Regards, > > >>>> > > >>>> Olivier. > > >>>> > > >>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <a...@apache.org> a > > >> écrit > > >>>> : > > >>>> > > >>>>> That PR pre-dates 1.9.0 being released. > > >>>>> > > >>>>>> On 27 Jan 2019, at 06:57, airflowuser <airflowu...@protonmail.com > > >>>> .INVALID> > > >>>>> wrote: > > >>>>>> > > >>>>>> The Mesos executor is being used (at least by some people) and > there > > >> is > > >>>>> an open PR to extend it: > > >>>>>> https://github.com/apache/airflow/pull/3739 > > >>>>>> > > >>>>>> > > >>>>>> Sent with ProtonMail Secure Email. > > >>>>>> > > >>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > >>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik < > > kaxiln...@gmail.com> > > >>>>> wrote: > > >>>>>> > > >>>>>>> What about Dask Executor as well. > > >>>>>>> > > >>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < > > >>>> daniel.imber...@gmail.com > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>> @sid basically my response lol > > >>>>>>>> I haven't seen any emails from any users and if it's been broken > > for > > >>>>> two > > >>>>>>>> versions then there doesn't seem to be enough interest to > maintain > > >>>> it. > > >>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39...@gmail.com > wrote: > > >>>>>>>>> > > >>>>>>>>> LOL :_0 > > >>>>>>>>> -s > > >>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin > bdbr...@gmail.com > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> I agree. Maybe we should make a choice on what we consider 1st > > >>>> class > > >>>>>>>>>> executors and move others to contrib (I'm looking at you > celery) > > >>>>>>>>>> Sent from my iPhone > > >>>>>>>>>> > > >>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor a...@apache.org > > >> wrote: > > >>>>>>>>>>> Is anyone using the Mesos Executor? I think we should > deprecate > > >>>> and > > >>>>>>>>>>> remove it. > > >>>>>>>>>>> The reason I say that is I have a feeling it's been broken > > since > > >>>>>>>>>>> 1.9.0 > > >>>>>>>>> > > >>>>>>>>>> and no-one has noticed: > > >>>>>>>>>> > > >>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin): > > >>>>>>>>>>> def start(self): > > >>>>>>>>>>> # ... > > >>>>>>>>>>> self.log.info( > > >>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : > > >>>>>>>>>>> %s, > > >>>>>>>>> > > >>>>>>>>>> checkpoint : %s', > > >>>>>>>>>> > > >>>>>>>>>>> master, framework.name, > > >>>>>>>>>>> str(task_cpu), str(task_memory), > > >>>> str(framework.checkpoint) > > >>>>>>>>>>> ) > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. But > > >>>> given > > >>>>>>>>>>> this > > >>>>>>>>>>> bug has been in since > > >>>>>>>> > > >>>>>>>> > > >>>>> > > >>>> > > >> > > > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29 > > >>>>>>>> > > >>>>>>>>>> which was in the 1.9.0 release I propose we simply remove this > > >>>>> executor > > >>>>>>>>>> from master. > > >>>>>>>>>> > > >>>>>>>>>>> Thoughts? > > >>>>>>>>>>> Ash > > >>>>>> > > >>>>>> > > >>>>> > > >>>>> > > >>>> > > >> > > >> > > > > >
