Re: [VOTE] AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor

2020-06-18 Thread Vikram Koka
+1 (non-binding) for this AIP. I really like the concept and the efficiency improvements. The general SmartSensor concept and the ability to add additional sensor classes is elegant. >From an implementation perspective, my one area of concern is the "sharding" concept and the configuration / mana

[VOTE] AIP-17: Consolidate and de-duplicate sensor tasks in airflow Smart Sensor

2020-06-18 Thread Yingbo Wang
Hello everyone! This email calls for a vote to add the airflow smart sensor at https://github.com/apache/airflow/pull/5499 AIP: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor Change summary: - Add a new mode cal

Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-18 Thread Dan Davydov
Metacomment: You might want to consider moving this discussion to a google doc or something since it seems like a lot of divergent threads are being created due to the scope of this change, which can be a bit hard to keep track of in email. If folks are concerned about history we can dump the googl

Re: [UPDATE] AIP-31 .output update

2020-06-18 Thread Gerard Casas Saez
Closing this vote after 48h as announced. +5 (bindings) +2 (non-binding) No negative votes or objections. We will proceed with the current updated AIP. Thank you everyone! Gerard Casas Saez Twitter | Cortex | @casassaez On Jun 18, 2020, 11:02 AM -0600, Jarek Potiuk , wrote: > +1 (binding). I lik

Re: [AIP-35] Add Signal Based Scheduling To Airflow

2020-06-18 Thread Gerard Casas Saez
Re-read this a couple times.  Some thoughts: • I agree on calling it event based scheduling, may be less misleading. • I believe we can mostly all agree that this is something interesting to add to Airflow and a use case that would be good to support. Nailing the details and path to implement is

Re: [UPDATE] AIP-31 .output update

2020-06-18 Thread Jarek Potiuk
+1 (binding). I like where the functional DAG is going. :) On Thu, Jun 18, 2020 at 6:41 PM Xinbin Huang wrote: > +1 (non-binding) > > On Thu, Jun 18, 2020 at 9:36 AM Kamil Breguła > wrote: > > > +1 (binding) > > > > On Thu, Jun 18, 2020, 18:33 Dan Davydov > > wrote: > > > > > +1 (binding) > >

Re: [UPDATE] AIP-31 .output update

2020-06-18 Thread Xinbin Huang
+1 (non-binding) On Thu, Jun 18, 2020 at 9:36 AM Kamil Breguła wrote: > +1 (binding) > > On Thu, Jun 18, 2020, 18:33 Dan Davydov > wrote: > > > +1 (binding) > > > > On Thu, Jun 18, 2020 at 12:32 PM Daniel Imberman < > > daniel.imber...@gmail.com> > > wrote: > > > > > +1 (binding) > > > > > > vi

Re: [UPDATE] AIP-31 .output update

2020-06-18 Thread Kamil Breguła
+1 (binding) On Thu, Jun 18, 2020, 18:33 Dan Davydov wrote: > +1 (binding) > > On Thu, Jun 18, 2020 at 12:32 PM Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > +1 (binding) > > > > via Newton Mail [ > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_f

Re: [AIP-34] Rewrite SubDagOperator

2020-06-18 Thread Xinbin Huang
Hi Daniel I agree that the TaskGroup should have the same API as a DAG object related to task dependencies, but it will not have anything related to actual execution or scheduling. I will update the AIP according to this over the weekend. > We could even make a “DAGTemplate” object s.t. when you

Re: [UPDATE] AIP-31 .output update

2020-06-18 Thread Dan Davydov
+1 (binding) On Thu, Jun 18, 2020 at 12:32 PM Daniel Imberman wrote: > +1 (binding) > > via Newton Mail [ > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2 > ] > On Wed, Jun 17, 2020 at 12:13 AM, Tomasz Urbaszek > wrote: > +1 (binding) > > On Wed, Jun 17, 20

Re: [UPDATE] AIP-31 .output update

2020-06-18 Thread Daniel Imberman
+1 (binding) via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.14.6&source=email_footer_2] On Wed, Jun 17, 2020 at 12:13 AM, Tomasz Urbaszek wrote: +1 (binding) On Wed, Jun 17, 2020 at 3:39 AM 蒋晓峰 wrote: > +1(not binding) > > On Wed, Jun 17, 2020 at 3:03 AM Gerard Cas

Re: [AIP-34] Rewrite SubDagOperator

2020-06-18 Thread Daniel Imberman
Hi Bin, Why not give the TaskGroup the same API as a DAG object (e.g. the bitwise operator fro task dependencies). We could even make a “DAGTemplate” object s.t. when you import the object you can import it with parameters to determine the shape of the DAG. On Wed, Jun 17, 2020 at 8:54 PM, Xi