Nice work! Always love reading these sort of “bug reports from hell” and the
work required to find the cause.
Also strongly agree we should standardize hooks in some way.
Cheers,
Bas
> On 16 Aug 2019, at 17:52, Shaw, Damian P. <damian.sha...@credit-suisse.com>
> wrote:
>
> Thanks, this is really useful to know! I often write my own
> Operators/Sensors/Hooks and was just looking at doing the same with the
> SFTPSensor and Operator.
>
> I've never formalized it but my current pattern is the follow:
>
> Hooks,
> Set self._conn to None on __init__, and have a property "self.conn" that
> checks if "self._conn" is None,
> *if None create a new connection set it to self._conn and return it
> * if not None run a check to see if the connection is still alive, if is
> alive return self._conn, otherwise create a new connection
>
> Sensor/Operators,
> On __init__ set self.conn_id to the conn_id string, and set
> "self._{conn_type}_hook" to None and have a property "self.{conn_type}_hook"
> In property check if "self._{conn_type}_hook" is None and if so create a new
> Hook, if not None then return "self._{conn_type}_hook"
>
> I would be really appreciative on any best practices here others could
> share.
>
>
> -----Original Message-----
> From: James Meickle [mailto:jmeic...@quantopian.com.INVALID]
> Sent: Friday, August 16, 2019 11:27 AM
> To: dev@airflow.apache.org
> Subject: Outage report
>
> We had an outage last night that was rather complex and difficult to debug.
> Rather than just writing up the bug, I included what we did for various
> debug steps. Hope some folks who are also cluster maintainers may find it
> interesting!
>
> https://issues.apache.org/jira/browse/AIRFLOW-5238
>
>
>
> ===============================================================================
>
> Please access the attached hyperlink for an important electronic
> communications disclaimer:
> http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> ===============================================================================
>
