Ok indeed I can see the connection is getting closed in get_records! Thanks
Alexis Rolland +86 138 1602 1449 > Le 17 janv. 2018 à 03:18, Bolke de Bruin <bdbr...@gmail.com> a écrit : > > Good point Chris. I overlooked that. > > B. > >> On 16 Jan 2018, at 20:09, Chris Palmer <crisp...@gmail.com> wrote: >> >> I'm not sure this is the right solution. I haven't explored all the code >> but it seems to me that the actual database connections are managed in the >> hooks. Different databases will want to handle this differently, and >> modifying SqlSensor seems too heavy handed to me. >> >> >> If you look at the get_records method of the generic DbApiHook ( >> https://pythonhosted.org/airflow/_modules/dbapi_hook.html#DbApiHook) then >> you can see it does closer the connections it is making. >> >> Is there specific hook for Teradata, and have you looked at how that is >> handling connections? >> >> Chris >> >> >> On Jan 16, 2018 9:32 AM, "Bolke de Bruin" <bdbr...@gmail.com> wrote: >> >> Of course. PR welcome. It would be nice to know how to test for connection >> leakage. >> >> Sent from my iPhone >> >>> On 16 Jan 2018, at 16:01, Alexis Rolland <alexis.roll...@ubisoft.com> >> wrote: >>> >>> Hello everyone, >>> >>> I’m reaching out to discuss / suggest a small improvement in the class >> SqlSensor: >>> https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html >>> >>> We are currently using SqlSensors on top of Teradata in several DAGs. >> When the DAGs execute we receive the following error message from Teradata >> engine: Error 8024 All virtual circuits are currently in use. >>> This error message would typically appear when we reach the maximum >> number of simultaneous connections to the database. >>> >>> I am suspecting the SqlSensor task to create a lot of connections - >> basically every time it (re)tries, and these connections would end up in >> idle state. >>> Does closing the connection at the end of the SqlSensor poke method >> sounds feasible? >>> >>> I’d like to take this opportunity as well to thank you for the awesome >> work you’ve been doing with Airflow. >>> Keep it up! >>> >>> Best, >>> >>> >>> >
