https://jira.apache.org/jira/browse/AIRFLOW-4313
:-) Op za 13 apr. 2019 om 12:28 schreef Felix Uellendall < felix.uellend...@gmx.de>: > +1 (non-binding) > > Am 12/04/2019 um 23:42 schrieb Tao Feng: > > +1 on removing mesos executor. > > > > On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman < > daniel.imber...@gmail.com> > > wrote: > > > >> Since we are now doing version 2.0 spring cleaning can we revisit this? > >> > >> On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin < > >> maximebeauche...@gmail.com> wrote: > >> > >>> +1 as it relies on pickles > >>> > >>> On Wed, Feb 27, 2019 at 4:51 AM Ash Berlin-Taylor <a...@apache.org> > >> wrote: > >>>> I say lets remove it in 2.0. > >>>> > >>>>> On 27 Feb 2019, at 11:50, Driesprong, Fokko <fo...@driesprong.frl> > >>>> wrote: > >>>>> Are we going to remove the MesosExecutor? I think it is a safe bet > >>> since > >>>> in > >>>>> the Survey of Ash (thanks again for that!), it also shows that nobody > >>> is > >>>>> really using it. > >>>>> > >>>>> https://jira.apache.org/jira/browse/AIRFLOW-3969 > >>>>> > >>>>> Cheers, Fokko > >>>>> > >>>>> Op di 12 feb. 2019 om 10:28 schreef Ash Berlin-Taylor < > >> a...@apache.org > >>>> : > >>>>>> I’m glad yarn wasn’t the only option - it would have meant I’d have > >>>> never > >>>>>> been in a position to use Airflow! (Many of our workflows don’t > >> touch > >>>>>> EMR/Hadoop, and running Celery is much more of a known element to a > >>>> python > >>>>>> developer than configuring Yarn) > >>>>>> > >>>>>>> On 11 Feb 2019, at 17:14, Maxime Beauchemin < > >>>> maximebeauche...@gmail.com> > >>>>>> wrote: > >>>>>>> From memory, I think MesosExecutor depends on pickling to get DAG > >>>>>>> definitions to workers, which we should also deprecate. > >>>>>>> > >>>>>>> About CeleryExecutor, we never had the intention to make it the > >>>>>> recommended > >>>>>>> option for production early on. The intent back in 2014 was to > >> write > >>> a > >>>>>>> YarnExecutor quickly (that was before the rise of k8s), but > >> realized > >>>> that > >>>>>>> container support wasn't ready in Yarn at the time, and had hard > >>>>>>> restrictions on what we could install on Hadoop/Yarn data nodes, so > >>> we > >>>>>>> moved forward with Celery. > >>>>>>> > >>>>>>> I think it's practical to have LocalExecutor around though for unit > >>>> test, > >>>>>>> local use, and for POCs. > >>>>>>> > >>>>>>> Max > >>>>>>> > >>>>>>> On Sat, Feb 9, 2019 at 5:59 AM Olivier Girardot < > >>>>>>> o.girar...@lateral-thoughts.com> wrote: > >>>>>>> > >>>>>>>> Hi, we tried to use it for a new project 1year ago, but it was too > >>>>>> broken > >>>>>>>> to be usable, and we didn't have the time to make it work. > >>>>>>>> Of course we are still hoping to use it one day, but that doesn't > >>> mean > >>>>>> it > >>>>>>>> should not move to contrib, it was already clear from the > >>>> documentation > >>>>>>>> that it was not a first class citizen. > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> > >>>>>>>> Olivier. > >>>>>>>> > >>>>>>>> Le dim. 27 janv. 2019 à 12:12, Ash Berlin-Taylor <a...@apache.org> > >> a > >>>>>> écrit > >>>>>>>> : > >>>>>>>> > >>>>>>>>> That PR pre-dates 1.9.0 being released. > >>>>>>>>> > >>>>>>>>>> On 27 Jan 2019, at 06:57, airflowuser < > >> airflowu...@protonmail.com > >>>>>>>> .INVALID> > >>>>>>>>> wrote: > >>>>>>>>>> The Mesos executor is being used (at least by some people) and > >>> there > >>>>>> is > >>>>>>>>> an open PR to extend it: > >>>>>>>>>> https://github.com/apache/airflow/pull/3739 > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Sent with ProtonMail Secure Email. > >>>>>>>>>> > >>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > >>>>>>>>>> On Friday, January 25, 2019 7:30 PM, Kaxil Naik < > >>>> kaxiln...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>>>> What about Dask Executor as well. > >>>>>>>>>>> > >>>>>>>>>>> On Fri, Jan 25, 2019, 16:35 Daniel Imberman < > >>>>>>>> daniel.imber...@gmail.com > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> @sid basically my response lol > >>>>>>>>>>>> I haven't seen any emails from any users and if it's been > >> broken > >>>> for > >>>>>>>>> two > >>>>>>>>>>>> versions then there doesn't seem to be enough interest to > >>> maintain > >>>>>>>> it. > >>>>>>>>>>>>> On Fri, Jan 25, 2019, 8:27 AM Sid Anand r39...@gmail.com > >>> wrote: > >>>>>>>>>>>>> LOL :_0 > >>>>>>>>>>>>> -s > >>>>>>>>>>>>> On Fri, Jan 25, 2019 at 8:25 AM Bolke de Bruin > >>> bdbr...@gmail.com > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> I agree. Maybe we should make a choice on what we consider > >> 1st > >>>>>>>> class > >>>>>>>>>>>>>> executors and move others to contrib (I'm looking at you > >>> celery) > >>>>>>>>>>>>>> Sent from my iPhone > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 25 Jan 2019, at 12:46, Ash Berlin-Taylor a...@apache.org > >>>>>> wrote: > >>>>>>>>>>>>>>> Is anyone using the Mesos Executor? I think we should > >>> deprecate > >>>>>>>> and > >>>>>>>>>>>>>>> remove it. > >>>>>>>>>>>>>>> The reason I say that is I have a feeling it's been broken > >>>> since > >>>>>>>>>>>>>>> 1.9.0 > >>>>>>>>>>>>>> and no-one has noticed: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> class MesosExecutor(BaseExecutor, LoginMixin): > >>>>>>>>>>>>>>> def start(self): > >>>>>>>>>>>>>>> # ... > >>>>>>>>>>>>>>> self.log.info( > >>>>>>>>>>>>>>> 'MesosFramework master : %s, name : %s, cpu : %s, mem : > >>>>>>>>>>>>>>> %s, > >>>>>>>>>>>>>> checkpoint : %s', > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> master, framework.name, > >>>>>>>>>>>>>>> str(task_cpu), str(task_memory), > >>>>>>>> str(framework.checkpoint) > >>>>>>>>>>>>>>> ) > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> That mixin should have been LoggingMixin, not LoginMixin. > >> But > >>>>>>>> given > >>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>> bug has been in since > >>>>>>>>>>>> > >> > https://github.com/apache/airflow/commit/a7a518902dcf1e7fd4bf477cf57cee691f181b29 > >>>>>>>>>>>>>> which was in the 1.9.0 release I propose we simply remove > >> this > >>>>>>>>> executor > >>>>>>>>>>>>>> from master. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thoughts? > >>>>>>>>>>>>>>> Ash > >>>>>>>>>> > >>>>>>>>> > >>>>>> > >>>> >
