; > know or care about Airflow. This approach changes that. And
> > this
> > > is
> > > > > not
> > > > > > > > a decision to be made lightly.
> > > > > > > >
> > > > > > > > 3.
signals get delivered to just one scheduler, or all
> > schedulers?
> > > > > > >
> > > > > > > 5. Does this now mean to get started out with Airflow you need
> to
> > > run
> > > > > > > three components: webserver,
t; > > > -ash
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Jun 17 2020, at 10:21 am, 蒋晓峰
> > wrote:
> > > > > >
> > > > > > > Hi Gerard,
> > > > >
t
> > > with
> > > > > > the conditions. If the received signal is met with the
> > corresponding
> > > > > > condition and the execution of task instances also meet the
> > > > > > conditions, the
> > > > > >
he
> > > > idea
> > > > > > is to run multiple evaluations of the online trained model (on a
> > > > > > permanently running DAGRun). So what happens when you have
> > triggered the
> > > > > > downstream operators using a si
d Casas Saez
> > > > > Twitter | Cortex | @casassaez
> > > > > On Jun 16, 2020, 7:00 AM -0600, 蒋晓峰 ,
> wrote:
> > > > > > Hello everyone,
> > > > > >
> > > > > > Sending a message to everyone and collecting feedb
t; > job that
> > > > > keeps polling records from Kafka and uses a model to do prediction.
> > And
> > > > the
> > > > > availability of a new model requires that streaming prediction job
> > either
> > > > > to restart,
based scheduling to let the operators
> send
> >> > signals to the scheduler to trigger a scheduling action, such as
> starting
> >> > jobs, stopping jobs and restarting jobs. With the change of
> >> > scheduling mechanism, a streaming job can send signals
; with such workflows consisting of both batch and streaming jobs.
>> I
>> > > > think
>> > > > > > having a system that can help smoothly work with a mixture of
>> > > streaming
>> > > > > and
>> > > > > > ba
; signals to the scheduler to trigger a scheduling action, such as
> > > > starting
> > > > > > jobs, stopping jobs and restarting jobs. With the change of
> > > > > > scheduling mechanism, a streaming job can send signals to the
> > > scheduler
> &
uch as periodic execution and manual trigger at per operator
>> > granularity.
>> >
>> > Changes proposed:
>> >
>> > - Signals are used to define conditions that must be met to run an
>> > operator. State change of the upstream tasks is one typ
we propose to introduce SignalOperator which
> > is mentioned in the public interface section.
> > - A notification service is necessary to help receive and propagate the
> > signals from the operators and other sources to the scheduler. Upon
> > receiving a signal, the scheduler c
dependency state immediately without periodically querying the
> > metadata
> > > > > database. This also allows potential support for richer scheduling
> > > > > semantics such as periodic execution and manual trigger at per
> > operator
> > > >
trigger at per
> operator
> > > > granularity.
> > > >
> > > > Changes proposed:
> > > >
> > > > - Signals are used to define conditions that must be met to run an
> > > > operator. State change of the upstream tasks is one type
t; > There may be other types of signals. The scheduler may take different
> > > actions when receiving different signals. To let the operators take
> > signals
> > > as their starting condition, we propose to introduce SignalOperator
> which
> > > is mentioned in the pu
is one type of the
> signals.
> > > There may be other types of signals. The scheduler may take different
> > > actions when receiving different signals. To let the operators take
> > signals
> > > as their starting condition, we propose to introduce Sign
tarting condition, we propose to introduce SignalOperator which
> > is mentioned in the public interface section.
> > - A notification service is necessary to help receive and propagate the
> > signals from the operators and other sources to the scheduler. Upon
> > receiving
he
> predefined signal-based conditions on the operators. Therefore we propose
> to introduce a Signal Notification Service component to Airflow.
>
> Please see related documents and PRs for details:
>
> AIP:
> https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Base
Service component to Airflow.
Please see related documents and PRs for details:
AIP:
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-35+Add+Signal+Based+Scheduling+To+Airflow
Issue: https://github.com/apache/airflow/issues/9321
Please let me know if there are any aspects that you agree
19 matches
Mail list logo
