Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

2020-02-25 Thread Sumit Maheshwari
Awesome work guys!! Kudos to all of you  On Wed, Feb 26, 2020 at 6:59 AM Jiajie Zhong wrote: > Good work! Thanks Kamil > > Best Wish > — Jiajie

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

2020-02-25 Thread Jiajie Zhong
Good work! Thanks Kamil Best Wish — Jiajie

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-02-25 Thread Gerard Casas Saez
Short update on status after receiving some feedback on Slack and Confluence comments: • Updated description of PythonFunctionalOperator • Added a few clarifying comments on how each component change plays along each other Reached out to Bolke to see if I can get his thoughts on how Lineage

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

2020-02-25 Thread Maxime Beauchemin
Nice! On Tue, Feb 25, 2020 at 12:11 AM Robin Edwards wrote: > This is brilliant work, thank you! Looking forward to watching my RDS > metrics when this gets deployed :-) > > On Tue, 25 Feb 2020, 07:08 Driesprong, Fokko, > wrote: > > > Sweet work Kamil and others! I'll try to go through them

Re: Airflow and Machine Learning

2020-02-25 Thread Jarek Potiuk
I certainly will take a look. I figured that something like that might be available already :). It seems it might be very useful for our development environment indeed (especially that our kubernetes tests are hard locally). It might also be interesting to see if this can work in the cases you

Re: Airflow and Machine Learning

2020-02-25 Thread James Meickle
Jarek, have you looked at garden.io? We've been experimenting with that for our local, Kubernetes-native development. It's hard to describe, but imagine: Docker Compose, but cross-git-repo, with builds occurring in Kubernetes and deployments managed by Helm charts. With this workflow, rather than

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

2020-02-25 Thread Robin Edwards
This is brilliant work, thank you! Looking forward to watching my RDS metrics when this gets deployed :-) On Tue, 25 Feb 2020, 07:08 Driesprong, Fokko, wrote: > Sweet work Kamil and others! I'll try to go through them today! > > Cheers, Fokko > > Op ma 24 feb. 2020 om 22:37 schreef Tao Feng : >