+1 on removing mesos executor. On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman <daniel.imber...@gmail.com> wrote:
> Since we are now doing version 2.0 spring cleaning can we revisit this? > > On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > > > +1 as it relies on pickles > > > > On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > I say lets remove it in 2.0. > > > > > > > On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl> > > > wrote: > > > > > > > > Are we going to remove the MesosExecutor? I think it is a safe bet > > since > > > in > > > > the Survey of Ash (thanks again for that!), it also shows that nobody > > is > > > > really using it. > > > > > > > > https://jira.apache.org/jira/browse/AIRFLOW-3969 > > > > > > > > Cheers, Fokko > > > > > > > > Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor < > a...@apache.org > > >: > > > > > > > >> I’m glad yarn wasn’t the only option - it would have meant I’d have > > > never > > > >> been in a position to use Airflow! (Many of our workflows don’t > touch > > > >> EMR/Hadoop, and running Celery is much more of a known element to a > > > python > > > >> developer than configuring Yarn) > > > >> > > > >>> On 11 Feb 2019, at 17:14, Maxime Beauchemin < > > > maximebeauche...@gmail.com> > > > >> wrote: > > > >>> > > > >>> From memory, I think MesosExecutor depends on pickling to get DAG > > > >>> definitions to workers, which we should also deprecate. > > > >>> > > > >>> About CeleryExecutor, we never had the intention to make it the > > > >> recommended > > > >>> option for production early on. The intent back in 2014 was to > write > > a > > > >>> YarnExecutor quickly (that was before the rise of k8s), but > realized > > > that > > > >>> container support wasn't ready in Yarn at the time, and had hard > > > >>> restrictions on what we could install on Hadoop/Yarn data nodes, so > > we > > > >>> moved forward with Celery. > > > >>> > > > >>> I think it's practical to have LocalExecutor around though for unit > > > test, > > > >>> local use, and for POCs. > > > >>> > > > >>> Max > > > >>> > > > >>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < > > > >>> o.girar...@lateral-thoughts.com> wrote: > > > >>> > > > >>>> Hi, we tried to use it for a new project 1year ago, but it was too > > > >> broken > > > >>>> to be usable, and we didn't have the time to make it work. > > > >>>> Of course we are still hoping to use it one day, but that doesn't > > mean > > > >> it > > > >>>> should not move to contrib, it was already clear from the > > > documentation > > > >>>> that it was not a first class citizen. > > > >>>> > > > >>>> Regards, > > > >>>> > > > >>>> Olivier. > > > >>>> > > > >>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <a...@apache.org> > a > > > >> écrit > > > >>>> : > > > >>>> > > > >>>>> That PR pre-dates 1.9.0 being released. > > > >>>>> > > > >>>>>> On 27 Jan 2019, at 06:57, airflowuser < > airflowu...@protonmail.com > > > >>>> .INVALID> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>> The Mesos executor is being used (at least by some people) and > > there > > > >> is > > > >>>>> an open PR to extend it: > > > >>>>>> https://github.com/apache/airflow/pull/3739 > > > >>>>>> > > > >>>>>> > > > >>>>>> Sent with ProtonMail Secure Email. > > > >>>>>> > > > >>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > > >>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik < > > > kaxiln...@gmail.com> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>>> What about Dask Executor as well. > > > >>>>>>> > > > >>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < > > > >>>> daniel.imber...@gmail.com > > > >>>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> @sid basically my response lol > > > >>>>>>>> I haven't seen any emails from any users and if it's been > broken > > > for > > > >>>>> two > > > >>>>>>>> versions then there doesn't seem to be enough interest to > > maintain > > > >>>> it. > > > >>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39...@gmail.com > > wrote: > > > >>>>>>>>> > > > >>>>>>>>> LOL :_0 > > > >>>>>>>>> -s > > > >>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin > > bdbr...@gmail.com > > > >>>>>>>>> wrote: > > > >>>>>>>>> > > > >>>>>>>>>> I agree. Maybe we should make a choice on what we consider > 1st > > > >>>> class > > > >>>>>>>>>> executors and move others to contrib (I'm looking at you > > celery) > > > >>>>>>>>>> Sent from my iPhone > > > >>>>>>>>>> > > > >>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor a...@apache.org > > > >> wrote: > > > >>>>>>>>>>> Is anyone using the Mesos Executor? I think we should > > deprecate > > > >>>> and > > > >>>>>>>>>>> remove it. > > > >>>>>>>>>>> The reason I say that is I have a feeling it's been broken > > > since > > > >>>>>>>>>>> 1.9.0 > > > >>>>>>>>> > > > >>>>>>>>>> and no-one has noticed: > > > >>>>>>>>>> > > > >>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin): > > > >>>>>>>>>>> def start(self): > > > >>>>>>>>>>> # ... > > > >>>>>>>>>>> self.log.info( > > > >>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : > > > >>>>>>>>>>> %s, > > > >>>>>>>>> > > > >>>>>>>>>> checkpoint : %s', > > > >>>>>>>>>> > > > >>>>>>>>>>> master, framework.name, > > > >>>>>>>>>>> str(task_cpu), str(task_memory), > > > >>>> str(framework.checkpoint) > > > >>>>>>>>>>> ) > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. > But > > > >>>> given > > > >>>>>>>>>>> this > > > >>>>>>>>>>> bug has been in since > > > >>>>>>>> > > > >>>>>>>> > > > >>>>> > > > >>>> > > > >> > > > > > > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29 > > > >>>>>>>> > > > >>>>>>>>>> which was in the 1.9.0 release I propose we simply remove > this > > > >>>>> executor > > > >>>>>>>>>> from master. > > > >>>>>>>>>> > > > >>>>>>>>>>> Thoughts? > > > >>>>>>>>>>> Ash > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>> > > > >> > > > >> > > > > > > > > >
