I would absolutely prefer to see them in the UI, and there are some annoying details in the way that some hooks use env vars in non-standard ways (I'm looking at you S3Hook) or have extra parameters that are hard (impossible?) to include via the environment.
It might be nice if the UI pulled connection displays from the same place that the configuration is pulling them (config / env / etc.) There is also the slight wrinkle that we want people to be able to query our metadata database but we don't necessarily want all of those people to be able to read connection information or other secrets. I hope the fernet key is good enough to make this a non-issue but, as I said, some of the connections behave in funny ways and I certainly haven't had the chance to dig deep enough to be entirely certain. It's sub-ideal and I'll be the first to admit my distaste for having multiple ways to configure the same thing but, for the time being, it's how we use Airflow. --George Leslie-Waksman On Fri, Sep 30, 2016 at 11:47 PM siddharth anand <san...@apache.org> wrote: > I see. Thanks for the explanation. > > Then, wouldn't you like Airflow's connections views to read, edit, and > delete connection information, whether that connection information is > stored in the db or in the OS environment? Currently, OS env-provided > connections are not exposed by the web app! Sure, they can be accessed by > the hooks, but the connection list/edit views don't expose them. Do you > find this a short-coming? > > -s > > On Fri, Sep 30, 2016 at 10:07 PM, George Leslie-Waksman < > geo...@cloverhealth.com> wrote: > > > We (Clover Health) currently deploy airflow to a 12-factor style > > environment (Aptible) and exclusively use environment variables to define > > connections. Removal of this feature would require a significant refactor > > of how we use Airflow and would likely force us away from the mainline > code > > branch. > > > > I would be happy to discuss things further, if you need more details. > > Certainly we could work around, and certainly there are things we don't > > like about the current system, but we only have so many places that we > can > > devote resources at any given point in time. > > > > --George Leslie-Waksman (Clover Health) > > > > > > On Fri, Sep 30, 2016 at 9:20 PM siddharth anand <san...@apache.org> > wrote: > > > > I'm trying to understand the impetus for a feature - the ability to > define > > connections using OS environment vars. > > > > https://airflow.incubator.apache.org/concepts.html#connections > > > > It seems folks use this so that they can set up Airflow connections using > > automation (e.g. Ansible, chef, puppet. etc...). > > > > I've never used this feature and would prefer to have CLI commands set up > > connections (e.g. https://github.com/apache/incubator-airflow/pull/1802 > ). > > > > It does not seem to me that these connections are revealed in the > > connection list view page or imported into the db table. It seems like > the > > OS environment serves as a parallel repository for connection > information, > > side-by-side the airflow metadata db. I don't like this this "parallel" > > repository as it is one more source of metadata to keep track of. > > > > If anyone is using the connections being set via env_var for any purpose > > aside from what I've listed here, please make it known. Else, I would > like > > to add this feature to the list of 2.0 deprecated features. > > > > In the meantime, I will spend some time to review and push the CLI > > connection PR forward. We currently already support setting Variables and > > Pools via the CLI. It makes sense to manage Connections in the same way. > > The CLI can be called via automation toolsets like those listed above. > > > > -s > > >
