Re: [Discuss] Airflow sensor optimization

Thank Yingbo for starting this and everyone for joining the discussion, great point about sharding. This would be really useful for large scale clusters. I image at the first stage we can reuse the existing logic and make the smart sensor a special kind of operator( maybe even make scheduler treat

Re: Welcoming Xiaodong Deng as a new commiter!

Thanks Everyone! XD On Thu, Mar 7, 2019 at 21:25 Ash Berlin-Taylor wrote: > Hello Airflowers, > > Please join the Apache Airflow PMC in welcoming our newest committer: > Xiaodong Deng (a.k.a. XD-DENG ). > > XD has been an active member of the community helping other

Re: [Discuss] Airflow sensor optimization

On 2019/03/06 14:31:57, Yingbo Wang wrote: > hi,> > > I would like to open an AIP for Airflow sensor optimization.> > > > *Motivation*:> > > Low efficiency in Airflow Sensor Implementation> > > Sensors are a special kind of operator that will keep running until a> > certain criterion is met. Examp

Re: Welcoming Xiaodong Deng as a new commiter!

Congrats Xiaodong! On Thu, Mar 7, 2019 at 5:25 AM Ash Berlin-Taylor wrote: > Hello Airflowers, > > Please join the Apache Airflow PMC in welcoming our newest committer: > Xiaodong Deng (a.k.a. XD-DENG ). > > XD has been an active member of the community helping other

Re: Welcoming Xiaodong Deng as a new commiter!

Congratulations Xiaodong! Aizhamal Nurmamat kyzy 于2019年3月7日 周四上午10:58写道： > Congratulations Xiaodong! > On Thu, Mar 7, 2019 at 07:10 jiajie zhong > wrote: > > > Welcoming XD > > > > -zhongjiajie > > > > > On Mar 7, 2019, at 22:38, Sid Anand wrote: > > > > > > Welcome aboard Xiaodong (XD)! > > >

Re: Welcoming Xiaodong Deng as a new commiter!

Congratulations Xiaodong! On Thu, Mar 7, 2019 at 07:10 jiajie zhong wrote: > Welcoming XD > > -zhongjiajie > > > On Mar 7, 2019, at 22:38, Sid Anand wrote: > > > > Welcome aboard Xiaodong (XD)! > > -s > > > > On Thu, Mar 7, 2019 at 2:20 PM Jarek Potiuk > > wrote: > > > >> +1 > >> > >> On Thu, M

Re: [Discuss] Airflow sensor optimization

There are two dimension to evaluate how much resource all sensors take in Airflow: the number of sensors and the duration of each sensor task take. Batch/smart sensor idea is proposed for the first one and the rescheduling is for the second one. For airflow cluster running large number of sensor ta

Re: [External] Re: Snowflake hook with key pair auth

Ok, great suggestion! I'll do that. Thanks On Wed, Mar 6, 2019 at 10:57 PM jiajie zhong wrote: > Due to this file just use it unittest, I think the best way is to generate > it in test file, maybe you could see > > https://github.com/apache/airflow/blob/68c66a1f8ae2398bbfc8b9556fbf72b7becb/t

Re: Welcoming Xiaodong Deng as a new commiter!

Welcoming XD -zhongjiajie > On Mar 7, 2019, at 22:38, Sid Anand wrote: > > Welcome aboard Xiaodong (XD)! > -s > > On Thu, Mar 7, 2019 at 2:20 PM Jarek Potiuk > wrote: > >> +1 >> >> On Thu, Mar 7, 2019 at 2:58 PM Driesprong, Fokko >> wrote: >> >>> Welcome XD! >>> >>> Op do 7 mrt. 2019 om

Re: Welcoming Xiaodong Deng as a new commiter!

Welcome Xiaodong! On Thu, Mar 7, 2019 at 5:25 AM Ash Berlin-Taylor wrote: > Hello Airflowers, > > Please join the Apache Airflow PMC in welcoming our newest committer: > Xiaodong Deng (a.k.a. XD-DENG ). > > XD has been an active member of the community helping other

Re: Welcoming Xiaodong Deng as a new commiter!

Welcome aboard Xiaodong (XD)! -s On Thu, Mar 7, 2019 at 2:20 PM Jarek Potiuk wrote: > +1 > > On Thu, Mar 7, 2019 at 2:58 PM Driesprong, Fokko > wrote: > > > Welcome XD! > > > > Op do 7 mrt. 2019 om 14:32 schreef Felix Uellendall < > > felix.uellend...@gmx.de > > > > > > > > Congratulations XD!

Re: Welcoming Xiaodong Deng as a new commiter!

+1 On Thu, Mar 7, 2019 at 2:58 PM Driesprong, Fokko wrote: > Welcome XD! > > Op do 7 mrt. 2019 om 14:32 schreef Felix Uellendall < > felix.uellend...@gmx.de > > > > > Congratulations XD! I'm happy for you :) > > > > Am 07/03/2019 um 14:24 schrieb Ash Berlin-Taylor: > > > Hello Airflowers, > > >

Re: Welcoming Xiaodong Deng as a new commiter!

Welcome XD! Op do 7 mrt. 2019 om 14:32 schreef Felix Uellendall > Congratulations XD! I'm happy for you :) > > Am 07/03/2019 um 14:24 schrieb Ash Berlin-Taylor: > > Hello Airflowers, > > > > Please join the Apache Airflow PMC in welcoming our newest committer: > Xiaodong Deng (a.k.a. XD-DENG

Re: Welcoming Xiaodong Deng as a new commiter!

Congratulations XD! I'm happy for you :) Am 07/03/2019 um 14:24 schrieb Ash Berlin-Taylor: Hello Airflowers, Please join the Apache Airflow PMC in welcoming our newest committer: Xiaodong Deng (a.k.a. XD-DENG ). XD has been an active member of the community helpin

Re: Welcoming Xiaodong Deng as a new commiter!

Congrats!! Always happy to see the committers growing Daniel On Thu, 7 Mar 2019 at 13:25, Ash Berlin-Taylor wrote: > Hello Airflowers, > > Please join the Apache Airflow PMC in welcoming our newest committer: > Xiaodong Deng (a.k.a. XD-DENG ). > > XD has been an act

Welcoming Xiaodong Deng as a new commiter!

Hello Airflowers, Please join the Apache Airflow PMC in welcoming our newest committer: Xiaodong Deng (a.k.a. XD-DENG ). XD has been an active member of the community helping other people on Slack, reviewing other's PRs and submitting PRs too and I'm sure XD will be

Re: [Discuss] Airflow sensor optimization

Rescheduling is of massive use for a DAG where we are waiting for a weekly S3 file delivery from a third party supplier with _massive_ variance in the delivery time. It'll appear at some point between Thursday AM and Sunday evening. Not having an executor slot tied up with the S3KeySensor is gre

Re: custom auth

The problem is you are creating a new User record, which maps to an INSERT statment every request/log in. Instead you should query first to find the record. Something like this: user = session.query(models.User).filter( models.User.username == username).first() if not

Re: Snowflake hook with key pair auth

Agree with Jiajie. You can consider writing the sample private key into a temp file at the beginning of your test, and clean it once it is done. XD On Thu, Mar 7, 2019 at 14:57 jiajie zhong wrote: > Due to this file just use it unittest, I think the best way is to generate > it in test file, ma