Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

Hello, In my opinion, it will be easier because the database will be less loaded, but more importantly. I added wait_for_update in one method, which means that another scheduler will not be able to damage the state of the database. Most of these changes even streamline HA, because more things are

Re: Airflow growth graphic?

That's just stars, but I'm working on a blog post that will share a jupyter notebook that gets metrics out of Github and ship a generic Superset dashboard on top of it. Some metrics it'll expose: * PRs - opened / merged / closed * Issues - opened / closed * comments * reactions * durations (time

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

Hey, I wanted to echo the awesomeness once more, but also bring up the question as to whether any of this work may make it harder to distribute / HA the scheduler down the line (?) I almost started analyzing the code and thought it'd just be easier to ask the authors. Max On Wed, Feb 26, 2020

Re: [DISCUSS] AIP-31: Airflow functional DAG definition (was API)

Really like the proposal, updated AIP doc for now. Updates: • Changed name (as pointed by Ash) • Updated example with new decorator interface • Updated description of decorator to match the new proposed interface • Added link to Airflow Slack thread into AIP for clarity Gerard Casas Saez

Re: Airflow growth graphic?

Thanks Max!  On Wed, Feb 26, 2020 at 12:35 PM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > > https://star-history.t9t.io/#apache/airflow/oozie/luigi/pinball/dbt/dagster/prefect > [image: Screen Shot 2020-02-26 at 12.34.01 PM.png] > > On Wed, Feb 26,

Re: Proposal: SIG-Kubernetes

+1 On Wed, Feb 26, 2020 at 8:40 PM Daniel Imberman wrote: > I’m glad to see that there is interest in this! > > For a scheduled meeting time would something similar to the > SIG-autoscaling work? > > Perhaps the second Wednesday of the month at 8AM or 9AM PST? > > @tomek I think that is an

Re: Proposal: SIG-Kubernetes

I’m glad to see that there is interest in this! For a scheduled meeting time would something similar to the SIG-autoscaling work? Perhaps the second Wednesday of the month at 8AM or 9AM PST? @tomek I think that is an interesting point that we can discuss further in the slack group. Now that

Re: Airflow growth graphic?

https://star-history.t9t.io/#apache/airflow/oozie/luigi/pinball/dbt/dagster/prefect [image: Screen Shot 2020-02-26 at 12.34.01 PM.png] On Wed, Feb 26, 2020 at 12:06 PM Alex Tronchin-James 949-412-7220 < alex.n.ja...@gmail.com> wrote: > Hello and happy Wednesday, Airflow community mind! > > I saw

Re: register

Thanks, please raise a PR to fix it :) On Wed, Feb 26, 2020 at 8:02 PM Gu, Baoshan (NIH/NLM/NCBI) [E] wrote: > Hi, > > 1. I am a developer and use airflow in our process. I noticed an issue > when using airflow==1.10.9 with mssql database, and I have reported >

Airflow growth graphic?

Hello and happy Wednesday, Airflow community mind! I saw a graphic recently (maybe even on this list) showing the number of committers or commits over time for airflow, oozie, and a couple of other open source scheduling projects, but couldn't find it anywhere this morning. If I can find it I was

register

Hi, 1. I am a developer and use airflow in our process. I noticed an issue when using airflow==1.10.9 with mssql database, and I have reported AIRFLOW-6931: I plan to submit a bug fix for it. Thank you, Sincerely, Baoshan Gu

Re: Proposal: SIG-Kubernetes

+1 would love to be a part of this On Wed, Feb 26, 2020, 2:34 PM Jarek Potiuk wrote: > +1 > > On Wed, Feb 26, 2020 at 8:32 PM Daniel Stevenson wrote: > > > +1 great idea > > > > > > > On Feb 26, 2020, at 20:30, Kevin Yang wrote: > > > > > > +1 would love to join. > > > > > >> On Wed, Feb 26,

Re: Big performance optimization of Scheduler - 10x faster , 2000+ fewer queries count

Really awesome indeed! I hadn’t enough time to look into all of it yet but will definitely do. Thanks Polidea and Databand for your efforts you put into this! Kamil especially! -Felix Sent from ProtonMail Mobile On Wed, Feb 26, 2020 at 08:54, Sumit Maheshwari wrote: > Awesome work guys!!

Re: Proposal: SIG-Kubernetes

+1 On Wed, Feb 26, 2020 at 8:32 PM Daniel Stevenson wrote: > +1 great idea > > > > On Feb 26, 2020, at 20:30, Kevin Yang wrote: > > > > +1 would love to join. > > > >> On Wed, Feb 26, 2020 at 10:58 AM Xinbin Huang > wrote: > >> > >> +1 would love to join. > >> > >> On Wed, Feb 26, 2020 at

Re: Proposal: SIG-Kubernetes

+1 great idea > On Feb 26, 2020, at 20:30, Kevin Yang wrote: > > +1 would love to join. > >> On Wed, Feb 26, 2020 at 10:58 AM Xinbin Huang wrote: >> >> +1 would love to join. >> >> On Wed, Feb 26, 2020 at 10:24 AM Dan Davydov >> >> wrote: >> >>> +1 would love to listen in on these >>>

Re: Proposal: SIG-Kubernetes

+1 would love to join. On Wed, Feb 26, 2020 at 10:58 AM Xinbin Huang wrote: > +1 would love to join. > > On Wed, Feb 26, 2020 at 10:24 AM Dan Davydov > > wrote: > > > +1 would love to listen in on these > > > > On Wed, Feb 26, 2020 at 11:52 AM Tomasz Urbaszek < > > tomasz.urbas...@polidea.com>

Re: Proposal: SIG-Kubernetes

+1 would love to join. On Wed, Feb 26, 2020 at 10:24 AM Dan Davydov wrote: > +1 would love to listen in on these > > On Wed, Feb 26, 2020 at 11:52 AM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> wrote: > > > +1 for the idea. Should Airflow k8s operator be included in those > > discussions?

Re: Proposal: SIG-Kubernetes

+1 would love to listen in on these On Wed, Feb 26, 2020 at 11:52 AM Tomasz Urbaszek < tomasz.urbas...@polidea.com> wrote: > +1 for the idea. Should Airflow k8s operator be included in those > discussions? > > Also, I'm not sure if we have any more need to have autoscaling-only SIG? > Should we

Re: [DISCUSS] AIP-31: Airflow functional DAG definition (was API)

Chatting with Gerard on slack a bit and we think a new name for this is "Functional DAG definition" -- yes this is still an API, but it's too easily confused with an HTTP/REST API. The page now lives at

RE: task failed without running

I get the same error in airflow-worker.err, the real underlying cause only revealed itself in airflow-worker.out Damian -Original Message- From: heng gu Sent: Wednesday, February 26, 2020 12:35 To: dev@airflow.apache.org Subject: Re: task failed without running thanks for your

Re: task failed without running

thanks for your help! I have similar DB timeout problem with other tasks, but not this one. The error message I got for this one from airflow-worker.err is    File "python3.6/site-packages/airflow/executors/celery_executor.py", line 67, in execute_command    close_fds=True, env=env)  File

RE: task failed without running

You should look in airflow-worker.out for an exception that's happening at about the same time. Damian -Original Message- From: heng gu Sent: Wednesday, February 26, 2020 11:55 To: dev@airflow.apache.org Subject: Re: task failed without running I got an advice from slack to check

Re: task failed without running

I got an advice from slack to check airflow-worker.err, and here is what I found   File "python3.6/site-packages/airflow/executors/celery_executor.py", line 67, in execute_command    close_fds=True, env=env)  File "pypacks/lib/python3.6/subprocess.py", line 311, in check_call    raise

Re: Proposal: SIG-Kubernetes

+1 for the idea. Should Airflow k8s operator be included in those discussions? Also, I'm not sure if we have any more need to have autoscaling-only SIG? Should we rename the existing one? T. On Wed, Feb 26, 2020 at 4:43 PM Daniel Imberman wrote: > Hello Airflowers! > Recently we've had some

[PROPOSAL][AIP-32] Airflow REST API

Hello, I just created "AIP-32 - Airflow REST API" proposal and would love community feedback and comments. https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-32%3A+Airflow+REST+API I would love to know what is your expectation from the API. We currently have one experimental API, but

Proposal: SIG-Kubernetes

Hello Airflowers! Recently we've had some pretty great success with the SIG-Autoscaling and SIG-DAG-Serialization groups. In this same vein, I'd like to propose a SIG-Kubernetes group where we can discuss all efforts to improve the airflow Kubernetes story. We would meet once a month and

RE: task failed without running

FYI, I've had this issue for a few different reasons, the first was that my worker failed to read the file (this for me was a permission issue). Now I'm having it a second time and it appears Airflow timeouts connecting to the DB while setting up the task, for me I can see this in the stdout

Airflow Summit 2020 new Website and CFP still open

On behalf of Airflow Summit organisers - I just wanted you to know that we have a new shiny website for the Airflow Summit! https://airflowsummit.org/ Expect first speakers to be announced soon - in the meantime here are some social links to share:

Re: task failed without running

Hi Heng, I have same problem. Failed jobs do not dump log files. Do you have this problem too? Have you been using Redis as message queue? Also can you send configurations related parallelism and concurrent in airflow.cfg file? Best regards, Mehmet. heng gu , 24 Şub 2020 Pzt, 17:51 tarihinde