Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

2020-08-11 Thread QP Hou
I think it's best to divide the discussion into two separate topics. First one is to replace the existing json_client with the new to be created official Airflow Python Client backed by the new RESTful API. This IMHO is a must have considering we are to deprecate experimental API going forward.

Re: [AIP-34] Rewrite SubDagOperator

2020-08-11 Thread James Coder
I agree this looks great, one question, how does the tree view look? James Coder > On Aug 11, 2020, at 6:48 PM, Gerard Casas Saez > wrote: > > First of all, this is awesome!! > > Secondly, checking your UI code, seems you are loading all operators at > once. Wondering if we can load them as

Re: [Meeting Notes] Airflow 2.0 Dev Call #1 - Planning

2020-08-11 Thread Vikram Koka
Thank you Kaxil, this looks great. I just updated the list of attendees to include a couple of people who I had noticed in the meeting. On Tue, Aug 11, 2020 at 5:17 PM Kaxil Naik wrote: > Hi all, > > I have created a document to summarize the discussion from the first Dev > call for Airflow

[Meeting Notes] Airflow 2.0 Dev Call #1 - Planning

2020-08-11 Thread Kaxil Naik
Hi all, I have created a document to summarize the discussion from the first Dev call for Airflow 2.0 *Doc Link*: https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#1:10Aug2020-Planning To all those who attended, can you please double-check and add if I have missed

Re: [AIP-34] Rewrite SubDagOperator

2020-08-11 Thread Gerard Casas Saez
First of all, this is awesome!! Secondly, checking your UI code, seems you are loading all operators at once. Wondering if we can load them as needed (aka load whenever we click the TaskGroup). Some of our DAGs are so large that take forever to load on the Graph view, so worried about this still

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

2020-08-11 Thread Ash Berlin-Taylor
-1 from me without a firm plan how we will replace it. I see keeping it and extending to use the new API would ensure that everything the CLI can do locally (i.e. when airflow webserver isn't up yet, with the ) also works over the API with the exception of db utilities. -ash On 11 August 2020

Re: [2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

2020-08-11 Thread QP Hou
+1 for replacing the existing remote mode client with the open api based client. IMO, we don't really have other options here because the experimental API will be deprecated in the future. For OpenAPI based Airflow REST clients, the current plan is to maintain all the code gen automation within

Re: [PROPOSAL][AIP-36 DAG Versioning]

2020-08-11 Thread Ash Berlin-Taylor
Anything to doing with the process of building wheels should be a "power user" only feature, and should not be required for many users - many many users of airflow are not primarily Python developers, but data scientists, and needing them to understand anything about the python build toolchain

[2.0 spring cleaning] Remove remote mode in Airflow CLI and in-core API Client

2020-08-11 Thread Kamil Breguła
Hello, I think we should remove remote mode in CLI and in-core API Client (airflow.api.client package). Here is docs about remote mode: https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance Since these features were introduced, it has never been

Re: [AIP-34] Rewrite SubDagOperator

2020-08-11 Thread Xinbin Huang
Hi Yu, Thank you so much for taking on this. I was fairly distracted previously and I didn't have the time to update the proposal. In fact, after discussing with Ash, Kaxil and Daniel, the direction of this AIP has been changed to favor the concept of TaskGroup instead of rewriting SubDagOperator

Re: [AIP-34] Rewrite SubDagOperator

2020-08-11 Thread Yu Qian
Hi, all, I've added the basic UI changes to my proposed implementation of TaskGroup as UI grouping concept: https://github.com/apache/airflow/pull/10153 I think Chris had a pretty good specification of TaskGroup so i'm quoting it here. The only thing I don't fully agree with is the restriction