Removing the experimental is a fundamental breaking change to users' workflows, and so we should remove it before 2.0.
-ash On 13 August 2020 10:14:02 BST, Jarek Potiuk <jarek.pot...@polidea.com> wrote: >And I think we should make the whole experimenta API deprecated in >1.10.12 >possibly ? > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk ><jarek.pot...@polidea.com> >wrote: > >> I think no matter what, Maybe we should simply make it deprecated in >the >> upcoming (today?) release of 1.10.12 ? Then we can decide if in - >> potential - 1.10.13 we remove it or leave it as it is. >> >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła ><kamil.breg...@polidea.com> >> wrote: >> >>> I started this thread mainly to discuss what we want to do with this >>> remote >>> mode prior to the Airflow 2.0 release. This is mainly due to the >fact that >>> he is using an experimental API which will be deprecated. >>> >>> In my opinion, we have several solutions. >>> a) Delete this mode as unused and not supported. >>> b) Rewrite in-core API client to support stable API >>> c) Prepare OpenAPI based client and rewrite CLI to use it >>> d) Leave as-is >>> >>> There were various expectations on the mailing list about this mode, >but I >>> haven't seen anyone actively contribute to it. This brings me >further >>> questions. Does anyone use this mode in its current form? If no one >is >>> using it, I think we can take more radical steps to start with a >blank >>> page. This will make it easier to start work and be able to iterate >over >>> it >>> faster in the future. This looks like a simple task, but if we want >to be >>> sure that no breaking-changes are made, we should pay off some >technical >>> debt and increase testing coverage before we can think about making >more >>> changes. It may not be necessary if we choose a different path of >>> development. >>> >>> I think now is a good time to try to make some decisions if someone >is >>> actually interested in developing these features. I do not think we >need a >>> precise vision of the development, if currently this feature is not >used >>> by >>> anyone and no one is really interested in its development. >>> >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <q...@scribd.com> wrote: >>> >>> > I think it's best to divide the discussion into two separate >topics. >>> > >>> > First one is to replace the existing json_client with the new to >be >>> created >>> > official Airflow Python Client backed by the new RESTful API. This >IMHO >>> is >>> > a must have considering we are to deprecate experimental API going >>> forward. >>> > >>> > The second topic is to create a better CLI experience leveraging >the new >>> > APIs. This is much more controversial. I remember us having a >>> > similar discussion in the dev list a year ago, which didn't get >much >>> > traction. It's not possible to fit all existing CLI >functionalities into >>> > REST APIs. DB utils use-case that Ash mentioned is a great >example. So I >>> > think one potential solution is to split the CLI into two. Keep >the >>> > existing CLI as the admin/management CLI can communicate directly >with >>> the >>> > DB and taps into airflow core code base. On the other hand, we can >>> create a >>> > separate user facing CLI that's light weight, fast and remote >only. It >>> > doesn't even need to be written in python to make it easier to >>> distribute >>> > as a single binary. >>> > >>> > Thanks, >>> > QP Hou >>> > >>> > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor ><a...@apache.org> >>> wrote: >>> > >>> > > -1 from me without a firm plan how we will replace it. >>> > > >>> > > I see keeping it and extending to use the new API would ensure >that >>> > > everything the CLI can do locally (i.e. when airflow webserver >isn't >>> up >>> > > yet, with the ) also works over the API with the exception of db >>> > utilities. >>> > > >>> > > -ash >>> > > >>> > > On 11 August 2020 20:05:56 BST, QP Hou <q...@scribd.com> wrote: >>> > > >+1 for replacing the existing remote mode client with the open >api >>> > > >based >>> > > >client. IMO, we don't really have other options here because >the >>> > > >experimental API will be deprecated in the future. >>> > > > >>> > > >For OpenAPI based Airflow REST clients, the current plan is to >>> maintain >>> > > >all >>> > > >the code gen automation within the main source tree [1], then >use it >>> to >>> > > >populate each individual language specific client repo like the >go >>> > > >client >>> > > >mentioned earlier. So far, we have the go client completed and >>> > > >validated to >>> > > >make sure this development flow will meet our needs. The next >step I >>> > > >think >>> > > >the community should focus on is getting API auth implemented >[2] >>> > > >before we >>> > > >move on to generate the python client. How we do API auth could >have >>> a >>> > > >big >>> > > >impact on client code gen automation, so it is worth waiting >for. >>> > > > >>> > > >Once we have authentication implemented in both Airflow core >and >>> > > >clients, >>> > > >we should be all good to start doing version releases for our >API >>> > > >clients. >>> > > > >>> > > >That said, adopting open api based clients in the CLI alone >won't >>> > > >address >>> > > >the issue of CLI depending on full airflow installation. Some >of the >>> > > >cli >>> > > >commands like `dags test` depend on a full airflow installation >by >>> > > >design. >>> > > >We will have to either develop a separate CLI intended for >remote >>> only >>> > > >use >>> > > >or add a flag in the existing cli so it can run in a pure >remote mode >>> > > >where >>> > > >it would disable loading of code that requires airflow >installation >>> > > >entirely. >>> > > > >>> > > >[1]: https://github.com/apache/airflow/tree/master/clients, >>> > > >[2]: https://github.com/apache/airflow/issues/8112 >>> > > > >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła >>> > > ><kamil.breg...@polidea.com> >>> > > >wrote: >>> > > > >>> > > >> Hello, >>> > > >> >>> > > >> I think we should remove remote mode in CLI and in-core API >Client >>> > > >> (airflow.api.client package). >>> > > >> Here is docs about remote mode: >>> > > >> >>> > > >> >>> > > > >>> > > >>> > >>> >https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance >>> > > >> >>> > > >> Since these features were introduced, it has never been >actively >>> > > >developed >>> > > >> and I don't think it's widely used. At the same time, Apache >>> Airflow >>> > > >is >>> > > >> evolving, and this code stands out more and more from the >rest. >>> > > >> >>> > > >> My main reservations about these features: >>> > > >> - Remote mode/in-core API Client is rarely used. I asked a >few >>> people >>> > > >and >>> > > >> none of them used it in production. Does anyone use it? >>> > > >> - A very small number of commands are available (7 pools >command >>> and >>> > > >2 dags >>> > > >> command only) >>> > > >> - Remote mode/API Client depends on experimental REST API. >>> > > >> - Remote mode/API Client is a handwritten code that is >difficult to >>> > > >> maintain. >>> > > >> - No documentation for API client >>> > > >> - Remote mode/API Client has low test coverage. >>> > > >> - Remote mode does not provide a good level of security, >because it >>> > > >depends >>> > > >> on experimental API. There is the only authentication, but >the >>> > > >> authenticated user can perform any operation. >>> > > >> - Requires full Airflow to be installed along with a large >number >>> of >>> > > >> unnecessary dependencies. Some of them are difficult to >install in >>> > > >some >>> > > >> environments, e.g. setproctitle on Windows >>> > > >> - Using this client API changes the logger configuration >because it >>> > > >> requires importing the airflow package. >>> > > >> >>> > > >> I think this remote mode in CLI is something valuable, but I >think >>> we >>> > > >can >>> > > >> do it in a different way in the future, e.g. generate a >CLI/API >>> > > >Client >>> > > >> based on the OpenAPI specification. >>> > > >> >>> > > >> Generated API clients can be installed independently of >airflow and >>> > > >will be >>> > > >> easier to maintain. We already have one API client for golang >>> > > >implemented >>> > > >> in this way, so new languages will only be developing this >idea. >>> > > >> - https://github.com/apache/airflow-client-go >>> > > >> >>> > > >> I will be happy to discuss the vision of the development of >these >>> two >>> > > >> things. How do we want to develop these two things? >>> > > >> >>> > > >> Best regards, >>> > > >> Kamil Bregula >>> > > >> >>> > > >>> > >>> >> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > >-- > >Jarek Potiuk >Polidea <https://www.polidea.com/> | Principal Software Engineer > >M: +48 660 796 129 <+48660796129> >[image: Polidea] <https://www.polidea.com/>
