So.. In the meantime our new contributor Matt implemented the move and we are merging it https://github.com/apache/airflow/pull/7456 as we have already several approvals. I hope it will help us to move faster a bit :)
We might have another discussions on the models __init__.py stuff removal and plugin mechanism refeactoring but those are other topics and I think we can discuss it in the PR that will come next. No need to discuss it all here J. On Wed, Feb 19, 2020 at 10:53 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > +1 - happy with that > > On Wed, Feb 19, 2020 at 8:09 AM Driesprong, Fokko <fo...@driesprong.frl> > wrote: > > > I'm totally fine with that. > > > > Cheers, Fokko > > > > Op di 18 feb. 2020 om 13:46 schreef Jarek Potiuk < > jarek.pot...@polidea.com > > >: > > > > > I believe this is one of the cases where we can just go with the > > consensus > > > indeed :). > > > > > > J. > > > > > > On Tue, Feb 18, 2020 at 12:51 PM Ash Berlin-Taylor <a...@apache.org> > > wrote: > > > > > > > Do we need to have a vote on it? I'm mostly interested in answering > the > > > > question about vote in general terms rather than this specific case) > > > > > > > > What do we need votes on, and when is "yeah no one complained, let's > do > > > > it" enough? > > > > > > > > For example if someone had created a PR for this and had appropriate > > > > instructions in UPDATING I would feel okay merging it. > > > > > > > > -a > > > > > > > > On 18 February 2020 09:37:13 GMT, Jarek Potiuk < > > jarek.pot...@polidea.com > > > > > > > > wrote: > > > > >All right. I turn it into vote then :) > > > > > > > > > > > > > > >On Tue, Feb 18, 2020 at 7:45 AM Driesprong, Fokko > > > > ><fo...@driesprong.frl> > > > > >wrote: > > > > > > > > > >> I don't have any objection, however, this isn't a [VOTE] right? ;) > > > > >> > > > > >> Op di 18 feb. 2020 om 00:08 schreef Jarek Potiuk > > > > ><jarek.pot...@polidea.com > > > > >> >: > > > > >> > > > > >> > I see that it's quite welcome change, so I think if no-one else > > > > >objects > > > > >> > within three days, I consider that a lazy consensus (not that > lazy > > > > >in > > > > >> fact) > > > > >> > :) https://www.apache.org/foundation/voting.html > > > > >> > > > > > >> > On Mon, Feb 17, 2020 at 6:14 PM Maxime Beauchemin < > > > > >> > maximebeauche...@gmail.com> wrote: > > > > >> > > > > > >> > > +1 > > > > >> > > > > > > >> > > On Mon, Feb 17, 2020 at 7:32 AM Daniel Imberman < > > > > >> > daniel.imber...@gmail.com > > > > >> > > > > > > > >> > > wrote: > > > > >> > > > > > > >> > > > +1 on my end! > > > > >> > > > > > > > >> > > > via Newton Mail > > > > >> > > > [ > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 > > > > >> > > > ] > > > > >> > > > On Mon, Feb 17, 2020 at 12:30 AM, Driesprong, Fokko > > > > >> > <fo...@driesprong.frl > > > > >> > > > > > > > >> > > > > > > > >> > > > wrote: > > > > >> > > > I like this as well. It will hopefully also reduce the > memory > > > > >> footprint > > > > >> > > of > > > > >> > > > Airflow. > > > > >> > > > > > > > >> > > > The only thing I can think of is that it will reduce the > test > > > > >> coverage, > > > > >> > > but > > > > >> > > > that's a vanity metric anyway :-) > > > > >> > > > > > > > >> > > > Cheers, Fokko > > > > >> > > > > > > > >> > > > Op za 15 feb. 2020 om 13:37 schreef Ash Berlin-Taylor < > > > > >> a...@apache.org > > > > >> > >: > > > > >> > > > > > > > >> > > > > I'm massively in favour of this. And as a side effect it > > > > >would > > > > >> solve > > > > >> > > an > > > > >> > > > > issue a reports almost two years ago > > > > >> > > > > https://issues.apache.org/jira/browse/AIRFLOW-1931 ( > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/AIRFLOW-1931?jql=project%20%3D%20AIRFLOW%20AND%20text%20~%20%22logging%20import%22 > > > > >> > > > > ) > > > > >> > > > > > > > > >> > > > > The one outstanding question is how/where we move > > > > >> > settings.initialize > > > > >> > > > and > > > > >> > > > > integrate_plugins to. I'm specifically thinking of > usecases > > > > >> outside > > > > >> > of > > > > >> > > > > someone running an airflow subcommand, such as in tests, > > > > >where you > > > > >> > > want > > > > >> > > > > airflow to be initialized. > > > > >> > > > > Perhaps: > > > > >> > > > > import airflow; airflow.initialize() > > > > >> > > > > Or I wonder if we need that at all? Things sould maybe > > > > >integrate > > > > >> > > plugins > > > > >> > > > > when they need to (by making a property/method somewhere > > > > >that is > > > > >> > > > memoized) > > > > >> > > > > and likewise in settings? Callers not having to do this > > > > >would be > > > > >> > > nicer, > > > > >> > > > > certainly. > > > > >> > > > > -a > > > > >> > > > > On Feb 15 2020, at 12:31 pm, Jarek Potiuk < > > > > >> jarek.pot...@polidea.com > > > > >> > > > > > > >> > > > > wrote: > > > > >> > > > > > TL;DR; I would like to ask the community for opinion > > about > > > > >> > reducing > > > > >> > > > (or > > > > >> > > > > > even removing) the number of automated imports we have > in > > > > >> > > > > > `airflow/__init__.py` for Airflow 2.0. > > > > >> > > > > > > > > > >> > > > > > This issue is plaguing us for quite a while already > and I > > > > >think > > > > >> we > > > > >> > > > have a > > > > >> > > > > > perfect opportunity to solve it in AIrflow 2.0. > Currently > > > > >our > > > > >> > > > > > `airflow/__init__.py` file contains the code I copied > > > > >below. > > > > >> While > > > > >> > > > > looking > > > > >> > > > > > fairly innocent it causes a lot of problems - because > > > > >importing > > > > >> > > > anything > > > > >> > > > > > from any airflow package automatically imports probably > > > > >90% of > > > > >> the > > > > >> > > > > airflow > > > > >> > > > > > internal code - all models, configurations, utils, Task > > > > >> Instance, > > > > >> > > > > > BaseOperator and plenty others (also we initialise all > > > > >plugins > > > > >> > where > > > > >> > > > they > > > > >> > > > > > are mostly not needed). What it really is - we have > > > > >implicit > > > > >> > > > dependencies > > > > >> > > > > > in our code that are causing various side effects: > > > > >> > > > > > > > > > >> > > > > > - pylint detects cyclic dependencies that are > super-hard > > > > >and > > > > >> > > sometimes > > > > >> > > > > > impossible to remove > > > > >> > > > > > - mypy and pylint are very slow - mypy parallel more is > > > > >slowed > > > > >> > down > > > > >> > > by > > > > >> > > > > > having to parse whole airflow in multiple instances, > and > > > > >pylint > > > > >> > > > cannot > > > > >> > > > be > > > > >> > > > > > run in parallel at all as it starts behaving randomly > > > > >w/regards > > > > >> > > cyclic > > > > >> > > > > > dependency detections > > > > >> > > > > > - we cannot really apply pylint and type annotations to > > > > >most of > > > > >> > the > > > > >> > > > core > > > > >> > > > > > classes as it will add even more cyclic dependencies > > > > >> > > > > > - last but not least - our CLI is really, really slow > > > > >because of > > > > >> > > that > > > > >> > > > - > > > > >> > > > > > right now any CLI command even `airflow version` has to > > > > >pull in > > > > >> > and > > > > >> > > > > > initialise all the classes. Solving that slowness is > > > > >impossible > > > > >> > > > without > > > > >> > > > > > removing the __init__.py code > > > > >> > > > > > > > > > >> > > > > > The effect of this change is that most of DAGs and > > plugins > > > > >> written > > > > >> > > so > > > > >> > > > far > > > > >> > > > > > for 1.10.* will not be compatible with Airflow 2.0 - in > > > > >all of > > > > >> the > > > > >> > > > DAGs > > > > >> > > > > > import paths will have to be changed. > > > > >> > > > > > > > > > >> > > > > > However as I see it - it's not a problem whatsoever. > > > > >People will > > > > >> > > have > > > > >> > > > to > > > > >> > > > > > perform migration from 1.10.* -> 2.0 and we know it's > not > > > > >going > > > > >> to > > > > >> > > be > > > > >> > > > > > seamless. We are going to write some tools for the > > > > >migration and > > > > >> > > > changing > > > > >> > > > > > such import paths is super easy fix that we can > automate > > > > >> > > super-easily. > > > > >> > > > > > > > > > >> > > > > > I'd love to hear community opinion on that. > > > > >> > > > > > J. > > > > >> > > > > > > > > > >> > > > > > *Current `airflow/__init__.py`:* > > > > >> > > > > > from typing import Callable, Optional > > > > >> > > > > > from airflow import utils > > > > >> > > > > > from airflow import settings > > > > >> > > > > > from airflow import version > > > > >> > > > > > from airflow.utils.log.logging_mixin import > LoggingMixin > > > > >> > > > > > from airflow.configuration import conf > > > > >> > > > > > from airflow.exceptions import AirflowException > > > > >> > > > > > from airflow.models.dag import DAG > > > > >> > > > > > > > > > >> > > > > > __version__ = version.version > > > > >> > > > > > settings.initialize() > > > > >> > > > > > from airflow.plugins_manager import integrate_plugins > > > > >> > > > > > login: Optional[Callable] = None > > > > >> > > > > > integrate_plugins() > > > > >> > > > > > -- > > > > >> > > > > > Jarek Potiuk > > > > >> > > > > > Polidea <https://www.polidea.com/> | Principal > Software > > > > >> Engineer > > > > >> > > > > > > > > > >> > > > > > M: +48 660 796 129 <+48660796129> > > > > >> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > -- > > > > >> > > > > > >> > Jarek Potiuk > > > > >> > Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > > >> > > > > > >> > M: +48 660 796 129 <+48660796129> > > > > >> > [image: Polidea] <https://www.polidea.com/> > > > > >> > > > > > >> > > > > > > > > > > > > > > >-- > > > > > > > > > >Jarek Potiuk > > > > >Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > >M: +48 660 796 129 <+48660796129> > > > > >[image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48660796129> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
