Re: [2.0 spring cleaning] Require unique conn_id

2019-04-14 Thread Kevin Yang
Yup unfortunately we Airbnb are relaying on the "feature" for some load balanching and also something like sensing partitions from 2 clusters at the same time( yup it is ugly). And at the same time we got bitten by having duplicate connections while one has outdated info. I think it does make sens

Re: [2.0 spring cleaning] Require unique conn_id

2019-04-13 Thread airflowuser
It can get more confusing because airflow allow to create two connection with same conn_id but different conn_type https://issues.apache.org/jira/browse/AIRFLOW-2784 Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Sunday, April 14, 2019 12:22 AM, Maxime Beauchemin wrote

Re: [2.0 spring cleaning] Require unique conn_id

2019-04-13 Thread Maxime Beauchemin
People may rely on this feature for [poor man's] load balancing though, I forgot what the exact use case was but used this at Airbnb at some point. Maybe the solution is to make the UI/UX/log output much more clear around this. Making the CLI log more clear should be really easy to do, web server

[2.0 spring cleaning] Require unique conn_id

2019-04-12 Thread James Meickle
Airflow fetches connections by name, but doesn't enforce unique names. My team got bit by this, since it's very unexpected behavior for most types of data entry. The reason for this behavior is explained in the docs: "Many connections with the same conn_id can be defined and when that is the case,