Re: [AIP-34] Rewrite SubDagOperator

2020-06-17 Thread
be ignored. Or if we decide to use TaskGroup to replace SubDag, there will > be no subdag schedule_interval. > > Bin > > On Wed, Jun 17, 2020 at 6:21 PM 蒋晓峰 wrote: > > > Hi Bin, > > Thanks for your good proposal. I was confused whether the schedule > > interv

Re: [AIP-34] Rewrite SubDagOperator

2020-06-17 Thread
Hi Bin, Thanks for your good proposal. I was confused whether the schedule interval of SubDAG is different from that of the parent DAG? I have discussed with Jiajie Zhong about the schedule interval of SubDAG. If the SubDagOperator has a different schedule interval, what will happen for the sch

Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-17 Thread
p until its listener got a start signal from the > upstream task. > > Just my two cents with hope to get more ideas on this topic. > > > Cheers, > Kevin Y > > On Tue, Jun 16, 2020 at 11:06 PM 蒋晓峰 wrote: > > > Hi barath, > > > >Could you pleas

Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-17 Thread
nal based triggers, as the relationship between upstream tasks > to downstream tasks is no longer 1:1 but 1:many. > > Gerard Casas Saez > Twitter | Cortex | @casassaez > On Jun 16, 2020, 7:00 AM -0600, 蒋晓峰 , wrote: > > Hello everyone, > > > > Sending a message to every

Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-16 Thread
> > it’s also related to the ML field). > > > > > > My main concern would be that you may need to change a few more things > to > > > support signal based triggers, as the relationship between upstream > tasks > > > to downstream tasks is no longer 1:1 but 1:man

Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-16 Thread
ull/253 (interesting read as > > it’s also related to the ML field). > > > > My main concern would be that you may need to change a few more things to > > support signal based triggers, as the relationship between upstream tasks > > to downstream tasks is no longer 1:1 b

Re: [UPDATE] AIP-31 .output update

2020-06-16 Thread
+1(not binding) On Wed, Jun 17, 2020 at 3:03 AM Gerard Casas Saez wrote: > Hi everyone, > > Sending an email here to consolidate an update to the AIP-31 that has > happened while we have been implementing this. > > When AIP-31 was proposed, the proposal mentioned that all operators would > have

[AIP-35] Add Signal Based Scheduling To Airflow

2020-06-16 Thread
Hello everyone, Sending a message to everyone and collecting feedback on the AIP-35 on adding signal-based scheduling. This was previously briefly mentioned in the discussion of development slack channel. The key motivation of this proposal is to support a mixture of batch and stream jobs in the s

Apply permission to create AIP

2020-06-15 Thread
Hello community, I don't have permission to create AIP for new ideas to Airflow. Anyone could help to grant permission to me? My WIKI ID: Nicholas Jiang WIKI Username: nicholasjiang Nice, Nicholas