Hi
This problem also exists in GCP operators. I have noticed this problem long
time ago and I will want to solve it
https://issues.apache.org/jira/browse/AIRFLOW-4771
This problem limits the use of AIrflow in the multitenant
environment, because the scheduler connects to the connection table.
Greets
On Sat, Aug 17, 2019, 11:17 AM Bas Harenslak <basharens...@godatadriven.com>
wrote:
> Nice work! Always love reading these sort of “bug reports from hell” and
> the work required to find the cause.
>
> Also strongly agree we should standardize hooks in some way.
>
> Cheers,
> Bas
>
> > On 16 Aug 2019, at 17:52, Shaw, Damian P. <
> damian.sha...@credit-suisse.com> wrote:
> >
> > Thanks, this is really useful to know! I often write my own
> Operators/Sensors/Hooks and was just looking at doing the same with the
> SFTPSensor and Operator.
> >
> > I've never formalized it but my current pattern is the follow:
> >
> > Hooks,
> > Set self._conn to None on __init__, and have a property "self.conn" that
> checks if "self._conn" is None,
> > *if None create a new connection set it to self._conn and return it
> > * if not None run a check to see if the connection is still alive, if is
> alive return self._conn, otherwise create a new connection
> >
> > Sensor/Operators,
> > On __init__ set self.conn_id to the conn_id string, and set
> "self._{conn_type}_hook" to None and have a property "self.{conn_type}_hook"
> > In property check if "self._{conn_type}_hook" is None and if so create a
> new Hook, if not None then return "self._{conn_type}_hook"
> >
> > I would be really appreciative on any best practices here others could
> share.
> >
> >
> > -----Original Message-----
> > From: James Meickle [mailto:jmeic...@quantopian.com.INVALID]
> > Sent: Friday, August 16, 2019 11:27 AM
> > To: dev@airflow.apache.org
> > Subject: Outage report
> >
> > We had an outage last night that was rather complex and difficult to
> debug.
> > Rather than just writing up the bug, I included what we did for various
> > debug steps. Hope some folks who are also cluster maintainers may find it
> > interesting!
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-5238
> >
> >
> >
> >
> ===============================================================================
>
> > Please access the attached hyperlink for an important electronic
> communications disclaimer:
> > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html
> >
> ===============================================================================
>
>
>
