Apache Airflow Performance tests framework has been accepted.
5 "+1" binding votes received:
- Scheffler Jens
- Jarek Potiuk
- Vincent Beck
- Amogh Desai
- Xiaodong DENG
6 "+1" non-binding votes received:
- Rajeshwar Bishundeo
- Aritra Basu
- Wei Lee
- Rahul Vats
- Mehta, Shu
+1 (non-binding)
From: Bartosz Jankiewicz
Sent: Tuesday, March 12, 2024 11:40:13 PM
To: dev@airflow.apache.org
Subject: [EXTERNAL] [COURRIEL EXTERNE] [VOTE] AIP-59 Performance testing
framework
CAUTION: This email originated from outside of the organization. Do
t;
> Thanks & Regards,
> Amogh Desai
>
> On Thu, Mar 14, 2024 at 12:28 AM Rahul Vats <mailto:rah.sharm...@gmail.com>> wrote:
>
>> +1 (non- binding)
>>
>> On Tue, 12 Mar, 2024, 23:40 Bartosz Jankiewicz,
>> mailto:bjankiew...@google.com.inva>lid&
gh Desai
>
> On Thu, Mar 14, 2024 at 12:28 AM Rahul Vats wrote:
>
>> +1 (non- binding)
>>
>> On Tue, 12 Mar, 2024, 23:40 Bartosz Jankiewicz,
>> wrote:
>>
>>> Hi folks,
>>>
>>> The AIP for performance testing has been in review fo
+1 binding
Glad to see this happen. Go for it!
Thanks & Regards,
Amogh Desai
On Thu, Mar 14, 2024 at 12:28 AM Rahul Vats wrote:
> +1 (non- binding)
>
> On Tue, 12 Mar, 2024, 23:40 Bartosz Jankiewicz,
> wrote:
>
> > Hi folks,
> >
> > The AIP for perfo
+1 (non- binding)
On Tue, 12 Mar, 2024, 23:40 Bartosz Jankiewicz,
wrote:
> Hi folks,
>
> The AIP for performance testing has been in review for quite some time and
> I've included your feedback in the document.
>
> I'd like to call a vote, and if you agree I&
lt;mailto:jens.scheff...@de.bosch.com>
> > >
> > > Sitz: Stuttgart, Registergericht: Amtsgericht Stuttgart, HRB 14000;
> > > Aufsichtsratsvorsitzender: Prof. Dr. Stefan Asenkerschbaumer;
> > > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr
mer;
> > Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> Forschner,
> > Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
> >
> > -Original Message-
> > From: Jarek Potiuk mailto:ja...@potiuk.com>>
> &g
> Geschäftsführung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> Forschner,
> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
>
> -Original Message-
> From: Jarek Potiuk mailto:ja...@potiuk.com>>
> Sent: Mittwoch, 13. März 2024 10:05
hrung: Dr. Stefan Hartung, Dr. Christian Fischer, Dr. Markus
> Forschner,
> Stefan Grosch, Dr. Markus Heyn, Dr. Frank Meyer, Dr. Tanja Rückert
>
> -Original Message-
> From: Jarek Potiuk
> Sent: Mittwoch, 13. März 2024 10:05
> To: dev@airflow.apache.org
> Subject: R
-
From: Jarek Potiuk
Sent: Mittwoch, 13. März 2024 10:05
To: dev@airflow.apache.org
Subject: Re: [VOTE] AIP-59 Performance testing framework
+1 (binding)
On Wed, Mar 13, 2024 at 7:40 AM Bartosz Jankiewicz
wrote:
>
> Hi folks,
>
> The AIP for performance testing has been in revi
+1 (binding)
On Wed, Mar 13, 2024 at 7:40 AM Bartosz Jankiewicz
wrote:
>
> Hi folks,
>
> The AIP for performance testing has been in review for quite some time and
> I've included your feedback in the document.
>
> I'd like to call a vote, and if you agree
Hi folks,
The AIP for performance testing has been in review for quite some time and
I've included your feedback in the document.
I'd like to call a vote, and if you agree I'd start a development of the
framework.
The AIP can be found below:
https://cwiki.apache.org/confluence/
thread. I propose you open separate threads for each of the questions you
>> have (in Github Discussions) this is a much more efficient way of asking
>> for help. But Again DevList is likely not the best place to ask those
>> questions.
>>
>> J
>>
>>
>> On
;> Hi devs,
>> I have tried multiple LocalExecutors in 16GB vm and sometimes
>> I have experienced workers consuming a lot of memory, Is this normal?
>> Is there any incoming activity to improve worker performance for python
>> operators?
>> Is there any java/go operator available?
>> Best Regards,
>> Giorgio
>>
>>
>>
>>
--
Life is a chess game - Anonymous.
in 16GB vm and sometimes
> I have experienced workers consuming a lot of memory, Is this normal?
> Is there any incoming activity to improve worker performance for python
> operators?
> Is there any java/go operator available?
> Best Regards,
> Giorgio
>
>
>
>
Hi devs,
I have tried multiple LocalExecutors in 16GB vm and sometimes
I have experienced workers consuming a lot of memory, Is this normal?
Is there any incoming activity to improve worker performance for python
operators?
Is there any java/go operator available?
Best Regards,
Giorgio
This vote has passed with 5 +1 votes and no -1 votes.
Work will begin on this soon :)
-ash
On Mar 19 2020, at 8:53 pm, Maxime Beauchemin
wrote:
> +1 (binding)
>
> Solid work!
>
> On Tue, Mar 17, 2020 at 10:43 PM Jarek Potiuk
> wrote:
>
>> +1 (binding)
>>
>> On Tue, Mar 17, 2020 at 11:16 P
+1 (binding)
Solid work!
On Tue, Mar 17, 2020 at 10:43 PM Jarek Potiuk
wrote:
> +1 (binding)
>
> On Tue, Mar 17, 2020 at 11:16 PM Kaxil Naik wrote:
> >
> > +1 (binding)
> >
> > On Tue, Mar 17, 2020 at 10:06 PM Deng Xiaodong
> wrote:
> >
> > > +1 (binding).
> > >
> > > Thanks for proceeding th
+1 (binding)
On Tue, Mar 17, 2020 at 11:16 PM Kaxil Naik wrote:
>
> +1 (binding)
>
> On Tue, Mar 17, 2020 at 10:06 PM Deng Xiaodong wrote:
>
> > +1 (binding).
> >
> > Thanks for proceeding this AIP, Ash.
> >
> >
> > XD
> >
> > On Tue, Mar 17, 2020 at 22:40 Ash Berlin-Taylor wrote:
> >
> > > Hi
+1 (binding)
On Tue, Mar 17, 2020 at 10:06 PM Deng Xiaodong wrote:
> +1 (binding).
>
> Thanks for proceeding this AIP, Ash.
>
>
> XD
>
> On Tue, Mar 17, 2020 at 22:40 Ash Berlin-Taylor wrote:
>
> > Hi all,
> >
> > This email calls for a vote on the design proposed in AIP-15, found here
> >
> >
+1 (binding).
Thanks for proceeding this AIP, Ash.
XD
On Tue, Mar 17, 2020 at 22:40 Ash Berlin-Taylor wrote:
> Hi all,
>
> This email calls for a vote on the design proposed in AIP-15, found here
>
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651
>
> A few notes
Hi all,
This email calls for a vote on the design proposed in AIP-15, found here
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651
A few notes
- The proposed architecture is to use an active/active architecture
where each scheduler is fully capable
- Nothing in thi
ss stable and predictable than PostgresSQL and this can >
> > >cause
> > > various stability problems. I know that Astronomer uses > >PostgreSQL,
> > but
> > > Airflow supports RDMS in a production environment and > >both must work
> > > p
gt; various stability problems. I know that Astronomer uses > >PostgreSQL,
> but
> > Airflow supports RDMS in a production environment and > >both must work
> > properly in this AIP. > > > >Best regards, > >Kamil > > > >On Tue, Mar 3,
> >
> >both must work
> properly in this AIP. > > > >Best regards, > >Kamil > > > >On Tue, Mar 3,
> 2020 at 12:50 PM Kaxil Naik wrote: > >> > >> Good work on the Proposal Ash
> & Vikram. > >> > >> > >>
2020 at 12:50 PM Kaxil
> Naik wrote: > >> > >> Good work on the Proposal Ash & Vikram. > >> > >> > >>
> > >> On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka > > > >> wrote: > >> > >> >
> Team, &g
regards,
> >Kamil
> >
> >On Tue, Mar 3, 2020 at 12:50 PM Kaxil Naik wrote:
> >>
> >> Good work on the Proposal Ash & Vikram.
> >>
> >>
> >>
> >> On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka
> >
> >> wrote:
> >>
e, Mar 3, 2020 at 12:50 PM Kaxil Naik wrote:
>>
>> Good work on the Proposal Ash & Vikram.
>>
>>
>>
>> On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka
>
>> wrote:
>>
>> > Team,
>> >
>> >
>> >
>> &
> Team,
> >
> >
> >
> > We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better
> > Scheduling Performance' on Confluence and would very much appreciate
> > feedback and suggestions from the community.
> >
> >
> &
Good work on the Proposal Ash & Vikram.
On Fri, Feb 28, 2020 at 10:39 PM Vikram Koka
wrote:
> Team,
>
>
>
> We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better
> Scheduling Performance' on Confluence and would very much appreciate
> feed
Team,
We just updated 'AIP-15 Support Multiple-Schedulers for HA & Better
Scheduling Performance' on Confluence and would very much appreciate
feedback and suggestions from the community.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103092651
The original
Yes. Here is PR.
https://github.com/apache/airflow/pull/6697
It made this work easier.
On Thu, Feb 27, 2020 at 12:18 PM Ash Berlin-Taylor wrote:
> We're just finializing the AIP document around scheduler HA and hope to
> have it published by the end of this week.
>
> DagFileProcessor is a new cl
We're just finializing the AIP document around scheduler HA and hope to have it
published by the end of this week.
DagFileProcessor is a new class, but the code in it is not new -- it's the
same, gradually evolving scheduler and parsing code, but refactored/moved to
live elsewhere, right?
-ash
Hello,
In my opinion, it will be easier because the database will be less loaded,
but more importantly. I added wait_for_update in one method, which means
that another scheduler will not be able to damage the state of the
database. Most of these changes even streamline HA, because more things are
Hey, I wanted to echo the awesomeness once more, but also bring up the
question as to whether any of this work may make it harder to distribute /
HA the scheduler down the line (?)
I almost started analyzing the code and thought it'd just be easier to ask
the authors.
Max
On Wed, Feb 26, 2020 at
Really awesome indeed! I hadn’t enough time to look into all of it yet but will
definitely do.
Thanks Polidea and Databand for your efforts you put into this! Kamil
especially!
-Felix
Sent from ProtonMail Mobile
On Wed, Feb 26, 2020 at 08:54, Sumit Maheshwari wrote:
> Awesome work guys!! Ku
Awesome work guys!! Kudos to all of you 👏
On Wed, Feb 26, 2020 at 6:59 AM Jiajie Zhong
wrote:
> Good work! Thanks Kamil
>
> Best Wish
> — Jiajie
Good work! Thanks Kamil
Best Wish
— Jiajie
hose optimizations into our
> > > > production environment.
> > > >
> > > > Thanks,
> > > > QP Hou
> > > >
> > > > On Mon, Feb 24, 2020 at 8:35 AM Kamil Breguła <
> > kamil.breg...@polidea.com
> > > >
> &g
AM Kamil Breguła <
> kamil.breg...@polidea.com
> > >
> > > wrote:
> > > >
> > > > Hello,
> > > >
> > > > Polidea [1] together with Databand [2] has taken steps to optimize
> > > > scheduler performance.
> &
> On Mon, Feb 24, 2020 at 8:35 AM Kamil Breguła >
> > wrote:
> > >
> > > Hello,
> > >
> > > Polidea [1] together with Databand [2] has taken steps to optimize
> > > scheduler performance.
> > > I made many changes last weekend:
>
eploy those optimizations into our
> production environment.
>
> Thanks,
> QP Hou
>
> On Mon, Feb 24, 2020 at 8:35 AM Kamil Breguła
> wrote:
> >
> > Hello,
> >
> > Polidea [1] together with Databand [2] has taken steps to optimize
> > scheduler performan
] has taken steps to optimize
> scheduler performance.
> I made many changes last weekend:
> 1. [AIRFLOW-6856] Bulk fetch paused_dag_ids
> https://github.com/apache/airflow/pull/7476
> 2. [AIRFLOW-6857] Bulk sync DAGs
> https://github.com/apache/airflow/pull/7477
> 3. [AIRFLOW-686
> wrote:
>
> > Hello,
> >
> > Polidea [1] together with Databand [2] has taken steps to optimize
> > scheduler performance.
> > I made many changes last weekend:
> > 1. [AIRFLOW-6856] Bulk fetch paused_dag_ids
> > https://github.com/apache/airflow/pull/74
Those are all great improvements Kamil! It would be great to have them
reviewed, tested and merged for 2.0 !
J.
On Mon, Feb 24, 2020 at 5:35 PM Kamil Breguła
wrote:
> Hello,
>
> Polidea [1] together with Databand [2] has taken steps to optimize
> scheduler performance.
> I ma
imize
> scheduler performance.
> I made many changes last weekend:
> 1. [AIRFLOW-6856] Bulk fetch paused_dag_ids
> https://github.com/apache/airflow/pull/7476
> 2. [AIRFLOW-6857] Bulk sync DAGs
> https://github.com/apache/airflow/pull/7477
> 3. [AIRFLOW-6862] Do not check the fresh
Hello,
Polidea [1] together with Databand [2] has taken steps to optimize
scheduler performance.
I made many changes last weekend:
1. [AIRFLOW-6856] Bulk fetch paused_dag_ids
https://github.com/apache/airflow/pull/7476
2. [AIRFLOW-6857] Bulk sync DAGs
https://github.com/apache/airflow/pull/7477
>
>
> As I mentioned in that article I think it's critical both for performance
> testing of changes, and "regression" testing of releases that we have a
> repeatable performance test framework. I somewhat simplified the script I
> used in the article, so I need
3:42 PM Ash Berlin-Taylor wrote:
> As part of my performance testing work I created a simple driver script,
> one of the steps it did was to "reset" the DB (at least for a given dag) so
> that the code was operating on the same rows in the same state each time --
> I ran
As part of my performance testing work I created a simple driver script, one of
the steps it did was to "reset" the DB (at least for a given dag) so that the
code was operating on the same rows in the same state each time -- I ran all my
tests about 10 times to get mean and variance.
This is obviously a subject dear to my heart :)
As I mentioned in that article I think it's critical both for performance
testing of changes, and "regression" testing of releases that we have a
repeatable performance test framework. I somewhat simplified the script I used
in th
*TL;DR;* We are gearing up @ Polidea to work on Apache Airflow performance
and I wanted to start a discussion that might lead to creating a new AIP
and implementing it :). Here is a high-level summary of the discussions we
had so far, so this might be a starting point to get some details worked
53 matches
Mail list logo
