Re: Close SqlSensor Connection

[Alexis Rolland]

Ok indeed I can see the connection is getting closed in get_records!
Thanks

Alexis Rolland
+86 138 1602 1449

> Le 17 janv. 2018 à 03:18, Bolke de Bruin  a écrit :
> 
> Good point Chris. I overlooked that.
> 
> B.
> 
>> On 16 Jan 2018, at 20:09, Chris Palmer  wrote:
>> 
>> I'm not sure this is the right solution. I haven't explored all the code
>> but it seems to me that the actual database connections are managed in the
>> hooks. Different databases will want to handle this differently, and
>> modifying SqlSensor seems too heavy handed to me.
>> 
>> 
>> If you look at the get_records method of the generic DbApiHook (
>> https://pythonhosted.org/airflow/_modules/dbapi_hook.html#DbApiHook) then
>> you can see it does closer the connections it is making.
>> 
>> Is there specific hook for Teradata, and have you looked at how that is
>> handling connections?
>> 
>> Chris
>> 
>> 
>> On Jan 16, 2018 9:32 AM, "Bolke de Bruin"  wrote:
>> 
>> Of course. PR welcome. It would be nice to know how to test for connection
>> leakage.
>> 
>> Sent from my iPhone
>> 
>>> On 16 Jan 2018, at 16:01, Alexis Rolland 
>> wrote:
>>> 
>>> Hello everyone,
>>> 
>>> I’m reaching out to discuss / suggest a small improvement in the class
>> SqlSensor:
>>> https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html
>>> 
>>> We are currently using SqlSensors on top of Teradata in several DAGs.
>> When the DAGs execute we receive the following error message from Teradata
>> engine: Error 8024 All virtual circuits are currently in use.
>>> This error message would typically appear when we reach the maximum
>> number of simultaneous connections to the database.
>>> 
>>> I am suspecting the SqlSensor task to create a lot of connections -
>> basically every time it (re)tries, and these connections would end up in
>> idle state.
>>> Does closing the connection at the end of the SqlSensor poke method
>> sounds feasible?
>>> 
>>> I’d like to take this opportunity as well to thank you for the awesome
>> work you’ve been doing with Airflow.
>>> Keep it up!
>>> 
>>> Best,
>>> 
>>> 
>>> 
> 

Re: Close SqlSensor Connection

[Bolke de Bruin]

Good point Chris. I overlooked that.

B.

> On 16 Jan 2018, at 20:09, Chris Palmer  wrote:
> 
> I'm not sure this is the right solution. I haven't explored all the code
> but it seems to me that the actual database connections are managed in the
> hooks. Different databases will want to handle this differently, and
> modifying SqlSensor seems too heavy handed to me.
> 
> 
> If you look at the get_records method of the generic DbApiHook (
> https://pythonhosted.org/airflow/_modules/dbapi_hook.html#DbApiHook) then
> you can see it does closer the connections it is making.
> 
> Is there specific hook for Teradata, and have you looked at how that is
> handling connections?
> 
> Chris
> 
> 
> On Jan 16, 2018 9:32 AM, "Bolke de Bruin"  wrote:
> 
> Of course. PR welcome. It would be nice to know how to test for connection
> leakage.
> 
> Sent from my iPhone
> 
>> On 16 Jan 2018, at 16:01, Alexis Rolland 
> wrote:
>> 
>> Hello everyone,
>> 
>> I’m reaching out to discuss / suggest a small improvement in the class
> SqlSensor:
>> https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html
>> 
>> We are currently using SqlSensors on top of Teradata in several DAGs.
> When the DAGs execute we receive the following error message from Teradata
> engine: Error 8024 All virtual circuits are currently in use.
>> This error message would typically appear when we reach the maximum
> number of simultaneous connections to the database.
>> 
>> I am suspecting the SqlSensor task to create a lot of connections -
> basically every time it (re)tries, and these connections would end up in
> idle state.
>> Does closing the connection at the end of the SqlSensor poke method
> sounds feasible?
>> 
>> I’d like to take this opportunity as well to thank you for the awesome
> work you’ve been doing with Airflow.
>> Keep it up!
>> 
>> Best,
>> 
>> 
>> 

Re: Close SqlSensor Connection

[Chris Palmer]

I'm not sure this is the right solution. I haven't explored all the code
but it seems to me that the actual database connections are managed in the
hooks. Different databases will want to handle this differently, and
modifying SqlSensor seems too heavy handed to me.


If you look at the get_records method of the generic DbApiHook (
https://pythonhosted.org/airflow/_modules/dbapi_hook.html#DbApiHook) then
you can see it does closer the connections it is making.

Is there specific hook for Teradata, and have you looked at how that is
handling connections?

Chris


On Jan 16, 2018 9:32 AM, "Bolke de Bruin"  wrote:

Of course. PR welcome. It would be nice to know how to test for connection
leakage.

Sent from my iPhone

> On 16 Jan 2018, at 16:01, Alexis Rolland 
wrote:
>
> Hello everyone,
>
> I’m reaching out to discuss / suggest a small improvement in the class
SqlSensor:
> https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html
>
> We are currently using SqlSensors on top of Teradata in several DAGs.
When the DAGs execute we receive the following error message from Teradata
engine: Error 8024 All virtual circuits are currently in use.
> This error message would typically appear when we reach the maximum
number of simultaneous connections to the database.
>
> I am suspecting the SqlSensor task to create a lot of connections -
basically every time it (re)tries, and these connections would end up in
idle state.
> Does closing the connection at the end of the SqlSensor poke method
sounds feasible?
>
> I’d like to take this opportunity as well to thank you for the awesome
work you’ve been doing with Airflow.
> Keep it up!
>
> Best,
>
>
>

Re: Close SqlSensor Connection

[Bolke de Bruin]

Of course. PR welcome. It would be nice to know how to test for connection 
leakage. 

Sent from my iPhone

> On 16 Jan 2018, at 16:01, Alexis Rolland  wrote:
> 
> Hello everyone,
>  
> I’m reaching out to discuss / suggest a small improvement in the class 
> SqlSensor:
> https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html
>  
> We are currently using SqlSensors on top of Teradata in several DAGs. When 
> the DAGs execute we receive the following error message from Teradata engine: 
> Error 8024 All virtual circuits are currently in use.
> This error message would typically appear when we reach the maximum number of 
> simultaneous connections to the database.
>  
> I am suspecting the SqlSensor task to create a lot of connections - basically 
> every time it (re)tries, and these connections would end up in idle state.
> Does closing the connection at the end of the SqlSensor poke method sounds 
> feasible?
>  
> I’d like to take this opportunity as well to thank you for the awesome work 
> you’ve been doing with Airflow.
> Keep it up!
>  
> Best,
>  
> 
>  

Close SqlSensor Connection

[Alexis Rolland]

Hello everyone,

I'm reaching out to discuss / suggest a small improvement in the class 
SqlSensor:
https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html

We are currently using SqlSensors on top of Teradata in several DAGs. When the 
DAGs execute we receive the following error message from Teradata engine: Error 
8024 All virtual circuits are currently in use.
This error message would typically appear when we reach the maximum number of 
simultaneous connections to the database.

I am suspecting the SqlSensor task to create a lot of connections - basically 
every time it (re)tries, and these connections would end up in idle state.
Does closing the connection at the end of the SqlSensor poke method sounds 
feasible?

I'd like to take this opportunity as well to thank you for the awesome work 
you've been doing with Airflow.
Keep it up!

Best,

[far-cry-signature]

Close SqlSensor Connection

[Alexis Rolland]

[Edit: resending e-mail now that my account is whitelisted]

Hello everyone,

I'm reaching out to discuss / suggest a small improvement in the class 
SqlSensor:
https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html

We are currently using SqlSensors on top of Teradata in several DAGs. When the 
DAGs execute we receive the following error message from Teradata engine: Error 
8024 All virtual circuits are currently in use.
This error message would typically appear when we reach the maximum number of 
simultaneous connections to the database.

I am suspecting the SqlSensor task to create a lot of connections - basically 
every time it (re)tries, and these connections would end up in idle state.
Does closing the connection at the end of the SqlSensor poke method sounds 
feasible?

I'd like to take this opportunity as well to thank you for the awesome work 
you've been doing with Airflow.
Keep it up!

Best,

[far-cry-signature]

Close SqlSensor Connection

[Alexis Rolland]

Hello everyone,

I'm reaching out to discuss / suggest a small improvement in the class 
SqlSensor:
https://pythonhosted.org/airflow/_modules/airflow/operators/sensors.html

We are currently using SqlSensors on top of Teradata in several DAGs. When the 
DAGs execute we receive the following error message from Teradata engine: Error 
8024 All virtual circuits are currently in use.
This error message would typically appear when we reach the maximum number of 
simultaneous connections to the database.

I am suspecting the SqlSensor task to create a lot of connections - basically 
every time it (re)tries, and these connections would end up in idle state.
Does closing the connection at the end of the SqlSensor poke method sounds 
feasible?

I'd like to take this opportunity as well to thank you for the awesome work 
you've been doing with Airflow.
Keep it up!

Best,

[far-cry-signature]