+1 for replacing the existing remote mode client with the open api based client. IMO, we don't really have other options here because the experimental API will be deprecated in the future.
For OpenAPI based Airflow REST clients, the current plan is to maintain all the code gen automation within the main source tree [1], then use it to populate each individual language specific client repo like the go client mentioned earlier. So far, we have the go client completed and validated to make sure this development flow will meet our needs. The next step I think the community should focus on is getting API auth implemented [2] before we move on to generate the python client. How we do API auth could have a big impact on client code gen automation, so it is worth waiting for. Once we have authentication implemented in both Airflow core and clients, we should be all good to start doing version releases for our API clients. That said, adopting open api based clients in the CLI alone won't address the issue of CLI depending on full airflow installation. Some of the cli commands like `dags test` depend on a full airflow installation by design. We will have to either develop a separate CLI intended for remote only use or add a flag in the existing cli so it can run in a pure remote mode where it would disable loading of code that requires airflow installation entirely. [1]: https://github.com/apache/airflow/tree/master/clients, [2]: https://github.com/apache/airflow/issues/8112 On Tue, Aug 11, 2020 at 11:05 AM Kamil Breguła <kamil.breg...@polidea.com> wrote: > Hello, > > I think we should remove remote mode in CLI and in-core API Client > (airflow.api.client package). > Here is docs about remote mode: > > https://airflow.readthedocs.io/en/latest/usage-cli.html#set-up-connection-to-a-remote-airflow-instance > > Since these features were introduced, it has never been actively developed > and I don't think it's widely used. At the same time, Apache Airflow is > evolving, and this code stands out more and more from the rest. > > My main reservations about these features: > - Remote mode/in-core API Client is rarely used. I asked a few people and > none of them used it in production. Does anyone use it? > - A very small number of commands are available (7 pools command and 2 dags > command only) > - Remote mode/API Client depends on experimental REST API. > - Remote mode/API Client is a handwritten code that is difficult to > maintain. > - No documentation for API client > - Remote mode/API Client has low test coverage. > - Remote mode does not provide a good level of security, because it depends > on experimental API. There is the only authentication, but the > authenticated user can perform any operation. > - Requires full Airflow to be installed along with a large number of > unnecessary dependencies. Some of them are difficult to install in some > environments, e.g. setproctitle on Windows > - Using this client API changes the logger configuration because it > requires importing the airflow package. > > I think this remote mode in CLI is something valuable, but I think we can > do it in a different way in the future, e.g. generate a CLI/API Client > based on the OpenAPI specification. > > Generated API clients can be installed independently of airflow and will be > easier to maintain. We already have one API client for golang implemented > in this way, so new languages will only be developing this idea. > - https://github.com/apache/airflow-client-go > > I will be happy to discuss the vision of the development of these two > things. How do we want to develop these two things? > > Best regards, > Kamil Bregula >
