Nataneljpwd commented on PR #53492: URL: https://github.com/apache/airflow/pull/53492#issuecomment-3168304983
> > as of now we want to get performance data and we also want to get it ready to be as an experimental feature in airflow 3.1 preferrably > > I would just say, we don't need to necessarily rush this out. Just wondering, have you considered patching your airflow and running it in your shop to see what the performance is? This is a thing that folks do sometimes. Can help make the case. Not saying we can't just add it but just asking the question. @dstandish that is part of the plan and a great idea indeed, first I want to test it locally and see that it gives decent performance, then we will try running it on the shop with a cluster policy that sets all operators to empty operator, and testing at a higher scale (maybe enable backfill) to really test it under very high load with multiple instances of the DB on the same server and multiple airflow deployments just like @ashb proposed, though first I will do it locally with a smaller scale of 100k or more schedulable tasks on a decently powerful linux machine, and collect metrics. I will publish the test on a different branch in case someone wants to test it as well. Testing it locally first to tune (or understand how to tune) Max tis. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
