Re: Give it up for Fokko!

+100 On Fri, Apr 13, 2018 at 4:08 PM, Alex Tronchin-James 949-412-7220 < alex.n.ja...@gmail.com> wrote: > Bravo!!! Bien fait! > > On Fri, Apr 13, 2018 at 3:54 PM Joy Gao wrote: > > >  > > > > On Fri, Apr 13, 2018 at 11:47 AM, Naik Kaxil wrote: > > > > >

Re: Give it up for Fokko!

Bravo!!! Bien fait! On Fri, Apr 13, 2018 at 3:54 PM Joy Gao wrote: >  > > On Fri, Apr 13, 2018 at 11:47 AM, Naik Kaxil wrote: > > > Couldn't agree more. Thanks Fokko > > > > On 13/04/2018, 17:56, "Maxime Beauchemin" > > wrote:

Re: Give it up for Fokko!

 On Fri, Apr 13, 2018 at 11:47 AM, Naik Kaxil wrote: > Couldn't agree more. Thanks Fokko > > On 13/04/2018, 17:56, "Maxime Beauchemin" > wrote: > > Hey all, > > I wanted to point out the amazing work that Fokko is doing, >

Re: Give it up for Fokko!

+1 ! On Fri, Apr 13, 2018 at 10:27 AM, Tao Feng wrote: > Thanks Fokko. Really appreciate your great feedback and your hard work on > helping airflow project thrive:) > > On Fri, Apr 13, 2018 at 9:55 AM, Maxime Beauchemin < > maximebeauche...@gmail.com> wrote: > > > Hey all,

Re: Give it up for Fokko!

Thanks Fokko. Really appreciate your great feedback and your hard work on helping airflow project thrive:) On Fri, Apr 13, 2018 at 9:55 AM, Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > Hey all, > > I wanted to point out the amazing work that Fokko is doing, > reviewing/merging PRs

Re: Benchmarking of Airflow Scheduler with Celery Executor

If you're concerned about scheduler scalability I'd go with a bigger box. The scheduler uses multiprocessing so more CPU power means more throughput. Also you may want to provision a beefy MySQL box to make sure that doesn't become the bottleneck. 10k tasks heartbeating to the DB every 30 seconds

Re: Give it up for Fokko!

Couldn't agree more. Appreciate you Fokko! Thanks for all of your hard work Cheers! Ace > On Apr 13, 2018, at 9:55 AM, Maxime Beauchemin > wrote: > > Hey all, > > I wanted to point out the amazing work that Fokko is doing, > reviewing/merging PRs and doing

Give it up for Fokko!

Hey all, I wanted to point out the amazing work that Fokko is doing, reviewing/merging PRs and doing fantastic committer & maintainer work. It takes a variety of contributions to make projects like Airflow thrive, but without this kind of involvement it wouldn't be possible to keep shipping

Re: Benchmarking of Airflow Scheduler with Celery Executor

Thanks Ry, Just wondering if there is any approximate number on concurrent tasks a scheduler can run on say 16 GB RAM and 8 core machine. If its already been done that would be useful. We did some benchmarking with local executor and observed that each TaskInstance was taking ~100MB of memory so

Re: Cancel a Running dag

Thanks Bolke, Will it become part of Airflow 1.10 release. Is there any tentative timeline for same. -Raman Gupta On 2018/04/12 19:19:07, Bolke de Bruin wrote: > This is now fixed in master. Clearing tasks will now properly terminate a > running task. If you pause the dag

Re: Cancel a Running dag

I have a similar use case, to support that I have implemented on_kill methods for all the operators. To cancel current run: Get all the running send a kill signal, in on_kill make sure current run for the task terminates gracefully, after the kill signal set state to failed(state should be set

Re: Cancel a Running dag

Like Bolke said, it has been fixed in master. One of the perquisites is support by the operator. For example, the Spark operator has implemented how to kill the Spark job on YARN, Local and Kubernetes. If you are running something else, you might want to check if this is implemented. Implemented