Do we need to have a vote on it? I'm mostly interested in answering the question about vote in general terms rather than this specific case)
What do we need votes on, and when is "yeah no one complained, let's do it" enough? For example if someone had created a PR for this and had appropriate instructions in UPDATING I would feel okay merging it. -a On 18 February 2020 09:37:13 GMT, Jarek Potiuk <jarek.pot...@polidea.com> wrote: >All right. I turn it into vote then :) > > >On Tue, Feb 18, 2020 at 7:45 AM Driesprong, Fokko ><fo...@driesprong.frl> >wrote: > >> I don't have any objection, however, this isn't a [VOTE] right? ;) >> >> Op di 18 feb. 2020 om 00:08 schreef Jarek Potiuk ><jarek.pot...@polidea.com >> >: >> >> > I see that it's quite welcome change, so I think if no-one else >objects >> > within three days, I consider that a lazy consensus (not that lazy >in >> fact) >> > :) https://www.apache.org/foundation/voting.html >> > >> > On Mon, Feb 17, 2020 at 6:14 PM Maxime Beauchemin < >> > maximebeauche...@gmail.com> wrote: >> > >> > > +1 >> > > >> > > On Mon, Feb 17, 2020 at 7:32 AM Daniel Imberman < >> > daniel.imber...@gmail.com >> > > > >> > > wrote: >> > > >> > > > +1 on my end! >> > > > >> > > > via Newton Mail >> > > > [ >> > > > >> > > >> > >> >https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 >> > > > ] >> > > > On Mon, Feb 17, 2020 at 12:30 AM, Driesprong, Fokko >> > <fo...@driesprong.frl >> > > > >> > > > >> > > > wrote: >> > > > I like this as well. It will hopefully also reduce the memory >> footprint >> > > of >> > > > Airflow. >> > > > >> > > > The only thing I can think of is that it will reduce the test >> coverage, >> > > but >> > > > that's a vanity metric anyway :-) >> > > > >> > > > Cheers, Fokko >> > > > >> > > > Op za 15 feb. 2020 om 13:37 schreef Ash Berlin-Taylor < >> a...@apache.org >> > >: >> > > > >> > > > > I'm massively in favour of this. And as a side effect it >would >> solve >> > > an >> > > > > issue a reports almost two years ago >> > > > > https://issues.apache.org/jira/browse/AIRFLOW-1931 ( >> > > > > >> > > > >> > > > >> > > >> > >> >https://issues.apache.org/jira/browse/AIRFLOW-1931?jql=project%20%3D%20AIRFLOW%20AND%20text%20~%20%22logging%20import%22 >> > > > > ) >> > > > > >> > > > > The one outstanding question is how/where we move >> > settings.initialize >> > > > and >> > > > > integrate_plugins to. I'm specifically thinking of usecases >> outside >> > of >> > > > > someone running an airflow subcommand, such as in tests, >where you >> > > want >> > > > > airflow to be initialized. >> > > > > Perhaps: >> > > > > import airflow; airflow.initialize() >> > > > > Or I wonder if we need that at all? Things sould maybe >integrate >> > > plugins >> > > > > when they need to (by making a property/method somewhere >that is >> > > > memoized) >> > > > > and likewise in settings? Callers not having to do this >would be >> > > nicer, >> > > > > certainly. >> > > > > -a >> > > > > On Feb 15 2020, at 12:31 pm, Jarek Potiuk < >> jarek.pot...@polidea.com >> > > >> > > > > wrote: >> > > > > > TL;DR; I would like to ask the community for opinion about >> > reducing >> > > > (or >> > > > > > even removing) the number of automated imports we have in >> > > > > > `airflow/__init__.py` for Airflow 2.0. >> > > > > > >> > > > > > This issue is plaguing us for quite a while already and I >think >> we >> > > > have a >> > > > > > perfect opportunity to solve it in AIrflow 2.0. Currently >our >> > > > > > `airflow/__init__.py` file contains the code I copied >below. >> While >> > > > > looking >> > > > > > fairly innocent it causes a lot of problems - because >importing >> > > > anything >> > > > > > from any airflow package automatically imports probably >90% of >> the >> > > > > airflow >> > > > > > internal code - all models, configurations, utils, Task >> Instance, >> > > > > > BaseOperator and plenty others (also we initialise all >plugins >> > where >> > > > they >> > > > > > are mostly not needed). What it really is - we have >implicit >> > > > dependencies >> > > > > > in our code that are causing various side effects: >> > > > > > >> > > > > > - pylint detects cyclic dependencies that are super-hard >and >> > > sometimes >> > > > > > impossible to remove >> > > > > > - mypy and pylint are very slow - mypy parallel more is >slowed >> > down >> > > by >> > > > > > having to parse whole airflow in multiple instances, and >pylint >> > > > cannot >> > > > be >> > > > > > run in parallel at all as it starts behaving randomly >w/regards >> > > cyclic >> > > > > > dependency detections >> > > > > > - we cannot really apply pylint and type annotations to >most of >> > the >> > > > core >> > > > > > classes as it will add even more cyclic dependencies >> > > > > > - last but not least - our CLI is really, really slow >because of >> > > that >> > > > - >> > > > > > right now any CLI command even `airflow version` has to >pull in >> > and >> > > > > > initialise all the classes. Solving that slowness is >impossible >> > > > without >> > > > > > removing the __init__.py code >> > > > > > >> > > > > > The effect of this change is that most of DAGs and plugins >> written >> > > so >> > > > far >> > > > > > for 1.10.* will not be compatible with Airflow 2.0 - in >all of >> the >> > > > DAGs >> > > > > > import paths will have to be changed. >> > > > > > >> > > > > > However as I see it - it's not a problem whatsoever. >People will >> > > have >> > > > to >> > > > > > perform migration from 1.10.* -> 2.0 and we know it's not >going >> to >> > > be >> > > > > > seamless. We are going to write some tools for the >migration and >> > > > changing >> > > > > > such import paths is super easy fix that we can automate >> > > super-easily. >> > > > > > >> > > > > > I'd love to hear community opinion on that. >> > > > > > J. >> > > > > > >> > > > > > *Current `airflow/__init__.py`:* >> > > > > > from typing import Callable, Optional >> > > > > > from airflow import utils >> > > > > > from airflow import settings >> > > > > > from airflow import version >> > > > > > from airflow.utils.log.logging_mixin import LoggingMixin >> > > > > > from airflow.configuration import conf >> > > > > > from airflow.exceptions import AirflowException >> > > > > > from airflow.models.dag import DAG >> > > > > > >> > > > > > __version__ = version.version >> > > > > > settings.initialize() >> > > > > > from airflow.plugins_manager import integrate_plugins >> > > > > > login: Optional[Callable] = None >> > > > > > integrate_plugins() >> > > > > > -- >> > > > > > Jarek Potiuk >> > > > > > Polidea <https://www.polidea.com/> | Principal Software >> Engineer >> > > > > > >> > > > > > M: +48 660 796 129 <+48660796129> >> > > > > > [image: Polidea] <https://www.polidea.com/> >> > > > > > >> > > > > >> > > > > >> > > >> > >> > >> > -- >> > >> > Jarek Potiuk >> > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > >> > M: +48 660 796 129 <+48660796129> >> > [image: Polidea] <https://www.polidea.com/> >> > >> > > >-- > >Jarek Potiuk >Polidea <https://www.polidea.com/> | Principal Software Engineer > >M: +48 660 796 129 <+48660796129> >[image: Polidea] <https://www.polidea.com/>
