> > > > > > > > of
> > > > > > > > > the class to the airflow configuration and do *something*
> to
> > > > their
> > > > > > > sensor
> > > > > > > > > class, including override the "is_smart_sensor" method
> (why a
&
sors inherited from BaseSmartSensor would be
> > > "Smart"
> > > > > > when
> > > > > > > > smart sensors are enabled in the configuration and not smart
> > when
> > > > > smart
> > > > > >
.
> > > > > > >
> > > > > > > Enabling/Disabling the smart sensor is a system level config
> > which
> > > is
> > > > > > > transparent to the individual users. An example of smart sensor
> > > > enabled
> > >
gt; > > > > > shards = 5
> > > > > >
> > > > > > sensor_enabled = NamedHivePartitionSensor,
> MetastorePartitionSensor
> > > > > >
> > > > > >
> > > > > > The "use_sma
; class names (e.g. HivePartitionSensor) in their DAGs and they don’t
> > > have
> > > > > the control to use smart sensors or not, unless they exclude their
> > > tasks
> > > > > explicits.
> > > > >
> > > > &g
nsor class names that will use the smart sensor. The users use
> the
> > > > same
> > > > > class names (e.g. HivePartitionSensor) in their DAGs and they don’t
> > > have
> > > > > the control to use smart sensors or not, unless they exclude their
&g
gt; smart
> > > > sensor.
> > > >
> > > >
> > > > “Is_smart_sensor_compatible” is a class level configuration (instead
> of
> > > > instance-level) so that the system knows if a particular sensor
> > operator
> > >
> > >
> > >Add a class attribute "poke_context_fields" to the operator.
> > >"poke_context_fields" include all key names used for initializing a
> > > sensor
> > >object.
> > >2.
> > >
> > >In airflo
;“[smart_sensor]” with the field “sensors_enabled” as follows.
> >
> >
> > Yingbo
> >
> > On Fri, Jun 19, 2020 at 7:27 AM Shaw, Damian P. <
> > damian.sha...@credit-suisse.com> wrote:
> >
> > > Also +1 (non-binding) on the AIP but questions on t
for this. It looks like they need to manually add the name
> of
> > the class to the airflow configuration and do *something* to their sensor
> > class, including override the "is_smart_sensor" method (why a method and
> > not an attribute?)
> >
> > Havin
from BaseSmartSensor would be "Smart" when
> smart sensors are enabled in the configuration and not smart when smart
> sensors are not enaled.
>
> Damian
>
> -----Original Message-
> From: Vikram Koka
> Sent: Friday, June 19, 2020 00:57
> To: dev@airflow.a
aseSmartSensor would be "Smart" when smart
sensors are enabled in the configuration and not smart when smart sensors are
not enaled.
Damian
-Original Message-
From: Vikram Koka
Sent: Friday, June 19, 2020 00:57
To: dev@airflow.apache.org
Subject: Re: [VOTE] AIP-17: Consolidate
+1 (non-binding) for this AIP.
I really like the concept and the efficiency improvements. The general
SmartSensor concept and the ability to add additional sensor classes is
elegant.
>From an implementation perspective, my one area of concern is the
"sharding" concept and the configuration / mana
Hello everyone!
This email calls for a vote to add the airflow smart sensor at
https://github.com/apache/airflow/pull/5499
AIP:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor
Change summary:
- Add a new mode cal
14 matches
Mail list logo
