Yep I agree removal (especially now in 1.10) is not a good idea.That's fine not to remove it in 2.0 but disable by default.
But I still think we should mark the whole API deprecated NOW (in 1.10.12). It would be great to have at least one release in 1.10. where we explicitly mark it as deprecated. We already know that we are going to use the new API in 2.0 and it's pretty much "ready" to be released - so I'd say - let's mark it as deprecated even if we do not provide a direct replacement immediately, just to let know people to whe will try to use it now, that they should think twice before doing it. I don't think "@deprecated" means "we provide a direct replacement already". It's more " do not use it because it will disappear". Whether it is replaced by something else or not is an entirely different story. J. On Thu, Aug 13, 2020 at 9:18 PM Ry Walker <r...@rywalker.com> wrote: > Yes I agree the new API would be made default, and experimental would be > disabled by default. > > Before the new API goes live, we should allow minor improvements to the > Experimental API (for example https://github.com/apache/airflow/pull/10308 > , > which addresses to some degree the security hole in the experimental API > for users of Airflow RBAC) which could change your last sentence a little, > Kamil. > > > On Thu, Aug 13, 2020 at 1:54 PM Kamil Breguła <kamil.breg...@polidea.com> > wrote: > > > I agree. We do not have to completely delete the experimental API in > > Airflow 2.0, but I think it is worth turning off by default so that the > > user has to make a conscious decision that they want to use the API, > which > > provides a limited level of security- no permission control, all > > authorized users have full power. > > > > On Thu, Aug 13, 2020 at 7:42 PM QP Hou <q...@scribd.com> wrote: > > > > > Yeah, i am also curious to know more about the reason why we want to > nuke > > > the experimental api code soon instead of just marking it as > deprecated. > > > > > > As for getting more insights into remote mode cli usage, would it make > > > sense to make it part of the airflow user survey? > > > > > > Thanks, > > > QP Hou > > > > > > > > > On Thu, Aug 13, 2020 at 7:18 AM Ry Walker <r...@rywalker.com> wrote: > > > > > > > I would think we would deprecate the old API once we say the new API > is > > > > “ready to go” - and leave it in place a while as users transition to > > new > > > > API. Why is there an urgency to remove it from codebase? > > > > > > > > On Thu, Aug 13, 2020 at 5:46 AM Ash Berlin-Taylor <a...@apache.org> > > > wrote: > > > > > > > > > Removing the experimental is a fundamental breaking change to > users' > > > > > workflows, and so we should remove it before 2.0. > > > > > > > > > > -ash > > > > > > > > > > On 13 August 2020 10:14:02 BST, Jarek Potiuk < > > jarek.pot...@polidea.com > > > > > > > > > wrote: > > > > > >And I think we should make the whole experimenta API deprecated in > > > > > >1.10.12 > > > > > >possibly ? > > > > > > > > > > > >On Thu, Aug 13, 2020 at 11:12 AM Jarek Potiuk > > > > > ><jarek.pot...@polidea.com> > > > > > >wrote: > > > > > > > > > > > >> I think no matter what, Maybe we should simply make it > deprecated > > in > > > > > >the > > > > > >> upcoming (today?) release of 1.10.12 ? Then we can decide if in > - > > > > > >> potential - 1.10.13 we remove it or leave it as it is. > > > > > >> > > > > > >> On Wed, Aug 12, 2020 at 6:37 PM Kamil Breguła > > > > > ><kamil.breg...@polidea.com> > > > > > >> wrote: > > > > > >> > > > > > >>> I started this thread mainly to discuss what we want to do with > > > this > > > > > >>> remote > > > > > >>> mode prior to the Airflow 2.0 release. This is mainly due to > the > > > > > >fact that > > > > > >>> he is using an experimental API which will be deprecated. > > > > > >>> > > > > > >>> In my opinion, we have several solutions. > > > > > >>> a) Delete this mode as unused and not supported. > > > > > >>> b) Rewrite in-core API client to support stable API > > > > > >>> c) Prepare OpenAPI based client and rewrite CLI to use it > > > > > >>> d) Leave as-is > > > > > >>> > > > > > >>> There were various expectations on the mailing list about this > > > mode, > > > > > >but I > > > > > >>> haven't seen anyone actively contribute to it. This brings me > > > > > >further > > > > > >>> questions. Does anyone use this mode in its current form? If no > > one > > > > > >is > > > > > >>> using it, I think we can take more radical steps to start with > a > > > > > >blank > > > > > >>> page. This will make it easier to start work and be able to > > iterate > > > > > >over > > > > > >>> it > > > > > >>> faster in the future. This looks like a simple task, but if we > > want > > > > > >to be > > > > > >>> sure that no breaking-changes are made, we should pay off some > > > > > >technical > > > > > >>> debt and increase testing coverage before we can think about > > making > > > > > >more > > > > > >>> changes. It may not be necessary if we choose a different path > of > > > > > >>> development. > > > > > >>> > > > > > >>> I think now is a good time to try to make some decisions if > > someone > > > > > >is > > > > > >>> actually interested in developing these features. I do not > think > > we > > > > > >need a > > > > > >>> precise vision of the development, if currently this feature is > > not > > > > > >used > > > > > >>> by > > > > > >>> anyone and no one is really interested in its development. > > > > > >>> > > > > > >>> On Wed, Aug 12, 2020 at 7:16 AM QP Hou <q...@scribd.com> wrote: > > > > > >>> > > > > > >>> > I think it's best to divide the discussion into two separate > > > > > >topics. > > > > > >>> > > > > > > >>> > First one is to replace the existing json_client with the new > > to > > > > > >be > > > > > >>> created > > > > > >>> > official Airflow Python Client backed by the new RESTful API. > > > This > > > > > >IMHO > > > > > >>> is > > > > > >>> > a must have considering we are to deprecate experimental API > > > going > > > > > >>> forward. > > > > > >>> > > > > > > >>> > The second topic is to create a better CLI experience > > leveraging > > > > > >the new > > > > > >>> > APIs. This is much more controversial. I remember us having a > > > > > >>> > similar discussion in the dev list a year ago, which didn't > get > > > > > >much > > > > > >>> > traction. It's not possible to fit all existing CLI > > > > > >functionalities into > > > > > >>> > REST APIs. DB utils use-case that Ash mentioned is a great > > > > > >example. So I > > > > > >>> > think one potential solution is to split the CLI into two. > Keep > > > > > >the > > > > > >>> > existing CLI as the admin/management CLI can communicate > > directly > > > > > >with > > > > > >>> the > > > > > >>> > DB and taps into airflow core code base. On the other hand, > we > > > can > > > > > >>> create a > > > > > >>> > separate user facing CLI that's light weight, fast and remote > > > > > >only. It > > > > > >>> > doesn't even need to be written in python to make it easier > to > > > > > >>> distribute > > > > > >>> > as a single binary. > > > > > >>> > > > > > > >>> > Thanks, > > > > > >>> > QP Hou > > > > > >>> > > > > > > >>> > > > > > > >>> > On Tue, Aug 11, 2020 at 12:44 PM Ash Berlin-Taylor > > > > > ><a...@apache.org> > > > > > >>> wrote: > > > > > >>> > > > > > > >>> > > -1 from me without a firm plan how we will replace it. > > > > > >>> > > > > > > > >>> > > I see keeping it and extending to use the new API would > > ensure > > > > > >that > > > > > >>> > > everything the CLI can do locally (i.e. when airflow > > webserver > > > > > >isn't > > > > > >>> up > > > > > >>> > > yet, with the ) also works over the API with the exception > of > > > db > > > > > >>> > utilities. > > > > > >>> > > > > > > > >>> > > -ash > > > > > >>> > > > > > > > >>> > > On 11 August 2020 20:05:56 BST, QP Hou <q...@scribd.com> > > wrote: > > > > > >>> > > >+1 for replacing the existing remote mode client with the > > open > > > > > >api > > > > > >>> > > >based > > > > > >>> > > >client. IMO, we don't really have other options here > because > > > > > >the > > > > > >>> > > >experimental API will be deprecated in the future. > > > > > >>> > > > > > > > > >>> > > >For OpenAPI based Airflow REST clients, the current plan > is > > to > > > > > >>> maintain > > > > > >>> > > >all > > > > > >>> > > >the code gen automation within the main source tree [1], > > then > > > > > >use it > > > > > >>> to > > > > > >>> > > >populate each individual language specific client repo > like > > > the > > > > > >go > > > > > >>> > > >client > > > > > >>> > > >mentioned earlier. So far, we have the go client completed > > and > > > > > >>> > > >validated to > > > > > >>> > > >make sure this development flow will meet our needs. The > > next > > > > > >step I > > > > > >>> > > >think > > > > > >>> > > >the community should focus on is getting API auth > > implemented > > > > > >[2] > > > > > >>> > > >before we > > > > > >>> > > >move on to generate the python client. How we do API auth > > > could > > > > > >have > > > > > >>> a > > > > > >>> > > >big > > > > > >>> > > >impact on client code gen automation, so it is worth > waiting > > > > > >for. > > > > > >>> > > > > > > > > >>> > > >Once we have authentication implemented in both Airflow > core > > > > > >and > > > > > >>> > > >clients, > > > > > >>> > > >we should be all good to start doing version releases for > > our > > > > > >API > > > > > >>> > > >clients. > > > > > >>> > > > > > > > > >>> > > >That said, adopting open api based clients in the CLI > alone > > > > > >won't > > > > > >>> > > >address > > > > > >>> > > >the issue of CLI depending on full airflow installation. > > Some > > > > > >of the > > > > > >>> > > >cli > > > > > >>> > > >commands like `dags test` depend on a full airflow > > > installation > > > > > >by > > > > > >>> > > >design. > > > > > >>> > > >We will have to either develop a separate CLI intended for > > > > > >remote > > > > > >>> only > > > > > >>> > > >use > > > > > >>> > > >or add a flag in the existing cli so it can run in a pure > > > > > >remote mode > > > > > >>> > > >where > > > > > >>> > > >it would disable loading of code that requires airflow > > > > > >installation > > > > > >>> > > >entirely. > > > > > >>> > > > > > > > > >>> > > >[1]: > https://github.com/apache/airflow/tree/master/clients, > > > > > >>> > > >[2]: https://github.com/apache/airflow/issues/8112 > > > > > >>> > > > > > > > > >>> > > >On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła > > > > > >>> > > ><kamil.breg...@polidea.com> > > > > > >>> > > >wrote: > > > > > >>> > > > > > > > > >>> > > >> Hello, > > > > > >>> > > >> > > > > > >>> > > >> I think we should remove remote mode in CLI and in-core > > API > > > > > >Client > > > > > >>> > > >> (airflow.api.client package). > > > > > >>> > > >> Here is docs about remote mode: > > > > > >>> > > >> > > > > > >>> > > >> > > > > > >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance > > > > > >>> > > >> > > > > > >>> > > >> Since these features were introduced, it has never been > > > > > >actively > > > > > >>> > > >developed > > > > > >>> > > >> and I don't think it's widely used. At the same time, > > Apache > > > > > >>> Airflow > > > > > >>> > > >is > > > > > >>> > > >> evolving, and this code stands out more and more from > the > > > > > >rest. > > > > > >>> > > >> > > > > > >>> > > >> My main reservations about these features: > > > > > >>> > > >> - Remote mode/in-core API Client is rarely used. I > asked a > > > > > >few > > > > > >>> people > > > > > >>> > > >and > > > > > >>> > > >> none of them used it in production. Does anyone use it? > > > > > >>> > > >> - A very small number of commands are available (7 pools > > > > > >command > > > > > >>> and > > > > > >>> > > >2 dags > > > > > >>> > > >> command only) > > > > > >>> > > >> - Remote mode/API Client depends on experimental REST > API. > > > > > >>> > > >> - Remote mode/API Client is a handwritten code that is > > > > > >difficult to > > > > > >>> > > >> maintain. > > > > > >>> > > >> - No documentation for API client > > > > > >>> > > >> - Remote mode/API Client has low test coverage. > > > > > >>> > > >> - Remote mode does not provide a good level of security, > > > > > >because it > > > > > >>> > > >depends > > > > > >>> > > >> on experimental API. There is the only authentication, > but > > > > > >the > > > > > >>> > > >> authenticated user can perform any operation. > > > > > >>> > > >> - Requires full Airflow to be installed along with a > large > > > > > >number > > > > > >>> of > > > > > >>> > > >> unnecessary dependencies. Some of them are difficult to > > > > > >install in > > > > > >>> > > >some > > > > > >>> > > >> environments, e.g. setproctitle on Windows > > > > > >>> > > >> - Using this client API changes the logger configuration > > > > > >because it > > > > > >>> > > >> requires importing the airflow package. > > > > > >>> > > >> > > > > > >>> > > >> I think this remote mode in CLI is something valuable, > > but I > > > > > >think > > > > > >>> we > > > > > >>> > > >can > > > > > >>> > > >> do it in a different way in the future, e.g. generate a > > > > > >CLI/API > > > > > >>> > > >Client > > > > > >>> > > >> based on the OpenAPI specification. > > > > > >>> > > >> > > > > > >>> > > >> Generated API clients can be installed independently of > > > > > >airflow and > > > > > >>> > > >will be > > > > > >>> > > >> easier to maintain. We already have one API client for > > > golang > > > > > >>> > > >implemented > > > > > >>> > > >> in this way, so new languages will only be developing > this > > > > > >idea. > > > > > >>> > > >> - https://github.com/apache/airflow-client-go > > > > > >>> > > >> > > > > > >>> > > >> I will be happy to discuss the vision of the development > > of > > > > > >these > > > > > >>> two > > > > > >>> > > >> things. How do we want to develop these two things? > > > > > >>> > > >> > > > > > >>> > > >> Best regards, > > > > > >>> > > >> Kamil Bregula > > > > > >>> > > >> > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> > > > > > >> Jarek Potiuk > > > > > >> Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > > > >> > > > > > >> M: +48 660 796 129 <+48660796129> > > > > > >> [image: Polidea] <https://www.polidea.com/> > > > > > >> > > > > > >> > > > > > > > > > > > >-- > > > > > > > > > > > >Jarek Potiuk > > > > > >Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > > >M: +48 660 796 129 <+48660796129> > > > > > >[image: Polidea] <https://www.polidea.com/> > > > > > > > > > -- > > > > Sent from Gmail Mobile > > > > > > > > > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
