Re: Performance.

2022-02-01 Thread Jarek Potiuk
No apologies needed :) . It's not always easy to figure out all the communication "standards" in the community :). J. On Tue, Feb 1, 2022 at 10:53 PM Giorgio Zoppi wrote: > Hello, > my apologies for the inappropriate questions. > I'll file an issue when it's needed. > Best Regards, > Giorgio.

Re: Performance.

2022-02-01 Thread Giorgio Zoppi
Hello, my apologies for the inappropriate questions. I'll file an issue when it's needed. Best Regards, Giorgio. Il giorno mar 1 feb 2022 alle ore 22:38 Jarek Potiuk ha scritto: > I think this is not the best place to ask those kinds of questions. Please > open a slack question or Github

Re: Performance.

2022-02-01 Thread Jarek Potiuk
I think this is not the best place to ask those kinds of questions. Please open a slack question or Github Discussions if you want to discuss "using" of airflow. All links here: https://airflow.apache.org/community/ Also - if I may suggest you - you asked a very imprecise question and you are

Performance.

2022-02-01 Thread Giorgio Zoppi
Hi devs, I have tried multiple LocalExecutors in 16GB vm and sometimes I have experienced workers consuming a lot of memory, Is this normal? Is there any incoming activity to improve worker performance for python operators? Is there any java/go operator available? Best Regards, Giorgio

Re: [sig-multitenancy]: Meeting for multi-tenancy state and AIP-45/AIP-46 introductory discussions

2022-02-01 Thread Jarek Potiuk
Sure - anyone can join - just sign up in doodle with the available slots and I will announce the time for the meeting time on Friday. J. On Tue, Feb 1, 2022 at 6:47 PM Giorgio Zoppi wrote: > Hello, > can i join the meeting? I am reviewing airflow in depth. I am interested > in helping. > Best

Re: [sig-multitenancy]: Meeting for multi-tenancy state and AIP-45/AIP-46 introductory discussions

2022-02-01 Thread Giorgio Zoppi
Hello, can i join the meeting? I am reviewing airflow in depth. I am interested in helping. Best Regards, Giorgio.

Invitation: [sig-multitenancy]: Meeting for multi-tenancy state and A... @ Wed Feb 2 - Fri Feb 4, 2022 (dev@airflow.apache.org)

2022-02-01 Thread alanx . yan
BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN METHOD:REQUEST BEGIN:VEVENT DTSTART;VALUE=DATE:20220202 DTEND;VALUE=DATE:20220205 DTSTAMP:20220201T174720Z ORGANIZER;CN=alanx@gmail.com:mailto:alanx@gmail.com

[sig-multitenancy]: Meeting for multi-tenancy state and AIP-45/AIP-46 introductory discussions

2022-02-01 Thread Jarek Potiuk
Hello Everyone, I think it's about the time for the next sig-multitenancy meeting : I created a doodle poll for next week - please mark your availability till Friday the 4th. https://doodle.com/poll/axvu2gz7zhv8ieye?utm_source=poll_medium=link I think what the rough agenda will be: * AIP-43

Re: [Discuss] AIP-44 Airflow Database API

2022-02-01 Thread Jarek Potiuk
Since we have AIP-43 already approved, I think I would love to have more questions and discussions about AIP-44 - The "Airflow Internal API" https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-44+Airflow+Internal+API (as the new name is). For those who would like to get more context -

Re: [DISCUSS] deprecate `days_ago` dates utility

2022-02-01 Thread Jarek Potiuk
Good point Bas. While I am all for deprecation, I agree this one is very much "hard-wired" into many users of Airflow, for many likely this is almost a muscle-memory or copying from existing DAGs. It's not enough to mandate it but we also should be empathetic and provide a viable replacement.

AWS sponsorship of Airflow CI continued

2022-02-01 Thread Jarek Potiuk
Following the discussions from a week ago. As of last Friday, we also have USD 10.000 worth credits from the AWS for our CI. Thanks John and AWS :) J. On Fri, Jan 21, 2022 at 7:24 PM Jarek Potiuk wrote: > Just a small update. Let's hold-off with the sponsorship change. Who knows > what will

Re: [DISCUSSION] AIP-47 New design of Airflow System Tests

2022-02-01 Thread Mateusz Nojek
Hello everyone, Thanks for the questions, Niko! I will try to address them now... 1. The proposed solution is to have these provider-specific projects for running DAGs being set up for system tests execution and use GitHub CI just as a trigger for them. This way we will unload the

Re: [DISCUSS] deprecate `days_ago` dates utility

2022-02-01 Thread Bas Harenslak
Regardless the behaviour of days_ago(), a lot of people use it so we’ll definitely need to document it well with some good examples of alternatives. That said, I think the usage of days_ago() is actually a side-effect of users that don’t really need their DAGs to start at X days ago, but want