Thanks, this is really useful to know! I often write my own
Operators/Sensors/Hooks and was just looking at doing the same with the
SFTPSensor and Operator.
I've never formalized it but my current pattern is the follow:
Hooks,
Set self._conn to None on __init__, and have a property "self.conn" that checks
if "self._conn" is None,
*if None create a new connection set it to self._conn and return it
* if not None run a check to see if the connection is still alive, if is alive
return self._conn, otherwise create a new connection
Sensor/Operators,
On __init__ set self.conn_id to the conn_id string, and set
"self._{conn_type}_hook" to None and have a property "self.{conn_type}_hook"
In property check if "self._{conn_type}_hook" is None and if so create a new
Hook, if not None then return "self._{conn_type}_hook"
I would be really appreciative on any best practices here others could share.
-----Original Message-----
From: James Meickle [mailto:jmeic...@quantopian.com.INVALID]
Sent: Friday, August 16, 2019 11:27 AM
To: dev@airflow.apache.org
Subject: Outage report
We had an outage last night that was rather complex and difficult to debug.
Rather than just writing up the bug, I included what we did for various
debug steps. Hope some folks who are also cluster maintainers may find it
interesting!
https://issues.apache.org/jira/browse/AIRFLOW-5238
===============================================================================
Please access the attached hyperlink for an important electronic communications
disclaimer:
http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
===============================================================================
