Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

2020-02-23 Thread Kaxil Naik
Yay !! Nice suggestion Kamil, good work Ash ᐧ On Sun, Feb 23, 2020 at 10:51 PM Ash Berlin-Taylor wrote: > Thanks Kamil! You rock > > https://github.com/apache/airflow/pull/7517 > On Feb 23 2020, at 10:36 pm, Ash Berlin-Taylor wrote: > > Yeah both Pycharm and VSCode are perfectly happy with that

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

2020-02-23 Thread Ash Berlin-Taylor
Thanks Kamil! You rock https://github.com/apache/airflow/pull/7517 On Feb 23 2020, at 10:36 pm, Ash Berlin-Taylor wrote: > Yeah both Pycharm and VSCode are perfectly happy with that (and jump to > impl/defintion works right which is better than the type stub version too) > > PR incoming. I don't

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

2020-02-23 Thread Ash Berlin-Taylor
Yeah both Pycharm and VSCode are perfectly happy with that (and jump to impl/defintion works right which is better than the type stub version too) PR incoming. I don't think we need to worry about the method Celery is going to as (I think) Py3.6 supports PEP-562 natively. -ash On Feb 23 2020, at

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

2020-02-23 Thread Ash Berlin-Taylor
Ahha, I was wondering if there was something like typing.TYPE_CHECKING but for static analyzers/IDEs. I like that approach. Testing it out now. On Feb 23 2020, at 10:28 pm, Kamil Breguła wrote: > Hello, > > I admit that I do not follow the subject closely, but I think that > here is the answer.

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

2020-02-23 Thread Kamil Breguła
Hello, I admit that I do not follow the subject closely, but I think that here is the answer. https://github.com/celery/celery/blob/master/celery/__init__.py#L63-L78 We can also create a module with lazy-loaded attributes: https://github.com/celery/celery/blob/master/celery/__init__.py#L158-L184

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

2020-02-23 Thread Ash Berlin-Taylor
I've managed to get type stubs working in VSCode (and intellij), so that may be an option. TBH I think I'm leaning somewhat towards 5 -- after all that would then go some way to clarify the distinction between DAG and DagModel (i.e. DAG doesn't belong in models, it's not a DB model) On Feb 23 2

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

2020-02-23 Thread Kaxil Naik
1. Just have a deprecation warning (which it's not even clear if PEP-562 style will show a nice (!) warning in IDEs or just at run time 2. I learn to live with from airflow.models.dag import DAG 3. After we change airflow.models to not import all the sub-modules automatically, go bac

Re: [DISCUSS] Reduce (remove?) automated imports in Airflow 2.0

2020-02-23 Thread Ash Berlin-Taylor
Gah not only that, vscode also ignores __all__ settings which is really bad form. Pycharm at least does respect __all__ for what to show in the module, and also respects type stub files (.pyi) when completing constructor args etc. In short IDEs are terrible at dynamic languages, but it is what i

Re: [DISCUSS] Airflow's extensibility options ?

2020-02-23 Thread Ash Berlin-Taylor
Yes I've done some fairly horrible things to Airflow in a plugin (horrible only in how I did it - using functools.wrap) to: - create custom db tables in upgrade db func - start an extra background thread when the scheduler started. But I did it using wraps/"monkey-patching" precisely because I c

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-02-23 Thread Jarek Potiuk
I like the idea a lot. Good direction. I know we have a few people who are better in functional thinking than me so I think I would love those people to work it out. Happy to listen to the discussions :) One thing that struck me however. I am not sure if that falls in the same camp, but I have a

Re: [DISCUSS] Airflow's extensibility options ?

2020-02-23 Thread Jarek Potiuk
> The main extension point that is missed by us is DagBag. Its current > implementation is multi-layered, and it looks like it's starting to be the > main point of interest around the community. > > DagBag affects user deployments in the following ways: > * being able to read dags from remote F