I'm not sure this is the right solution. I haven't explored all the code but it seems to me that the actual database connections are managed in the hooks. Different databases will want to handle this differently, and modifying SqlSensor seems too heavy handed to me.
If you look at the get_records method of the generic DbApiHook ( https://pythonhosted.org/airflow/_modules/dbapi_hook.html#DbApiHook) then you can see it does closer the connections it is making. Is there specific hook for Teradata, and have you looked at how that is handling connections? Chris On Jan 16, 2018 9:32 AM, "Bolke de Bruin" <bdbr...@gmail.com> wrote: Of course. PR welcome. It would be nice to know how to test for connection leakage. Sent from my iPhone > On 16 Jan 2018, at 16:01, Alexis Rolland <alexis.roll...@ubisoft.com> wrote: > > Hello everyone, > > I’m reaching out to discuss / suggest a small improvement in the class SqlSensor: > https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html > > We are currently using SqlSensors on top of Teradata in several DAGs. When the DAGs execute we receive the following error message from Teradata engine: Error 8024 All virtual circuits are currently in use. > This error message would typically appear when we reach the maximum number of simultaneous connections to the database. > > I am suspecting the SqlSensor task to create a lot of connections - basically every time it (re)tries, and these connections would end up in idle state. > Does closing the connection at the end of the SqlSensor poke method sounds feasible? > > I’d like to take this opportunity as well to thank you for the awesome work you’ve been doing with Airflow. > Keep it up! > > Best, > > >
