Airflow fetches connections by name, but doesn't enforce unique names. My
team got bit by this, since it's very unexpected behavior for most types of
data entry. The reason for this behavior is explained in the docs:

"Many connections with the same conn_id can be defined and when that is the
case, and when the hooks uses the get_connection method from BaseHook,
Airflow will choose one connection randomly, allowing for some basic load
balancing and fault tolerance when used in conjunction with retries."

I think this is very non-intuitive UX. If we even want to support this
feature within Airflow - and I don't think that is a given - it would make
much more sense to require a unique (conn_id, conn_type) but allow storing
multiple related records. This wouldn't be a huge data modeling change, but
would require changing the web UI to appear as a form with subforms.

Reply via email to