> > Attached is the graph of queue size for Apache projects where you can > see how dramatic improvements it was (other projects of Apache do not > have self-hosted runners as we do so they are far bigger users).
Do you have a link to this attachment? Thanks! Best, Solomon On Fri, Nov 11, 2022 at 1:17 AM Jarek Potiuk <[email protected]> wrote: > Hello Everyone, > > I thought it's a good time to share some news after yesterday's ASF > "Builds meeting". The latest news for our CI is that ~ 2 weeks ago > GitHub bumped the parallelism of jobs for Public runners for Apache > Software Foundation from 300 to 900. > > Attached is the graph of queue size for Apache projects where you can > see how dramatic improvements it was (other projects of Apache do not > have self-hosted runners as we do so they are far bigger users). > Thanks to Tobiasz Kedzierski - one of the contributors to Airflow and > my friend who developed and maintains the graphs so that ASF can see > some stats. > > This means that builds from forks of our contributors should > experience far, far less queuing. Some new things are coming as well - > we (ASF with our support) are talking to GitHub to get self-hosted > runners on Azure, as well as enabling bigger instances / ARM instances > for running our CI jobs. > > This will likely allow us to optimize some of the build times as well. > I already have some ideas how we can make our builds leaner, faster > and with less number of failures. The goal is always to get the CI > feedback as fast as possible with as little as possible > false-negatives (and without burning too much money that our sponsors > give us). It's a moving target but we have some good possibilities now > on how to make those better. > > Stay tuned. > > Also I think this is the right moment to thank both Astronomer and > Amazon for the AWS credits (Amazon) and money (Astronomer). Again this > year we got USD 15.000 from Amazon (last year we got USD 10.000 and > Astronomer pays whatever we exceed above that and invested quite some > time (mostly of Ash) to get our "VM" infrastructure up and running. > > There are also likely more news coming in the light of AIP-47 - where > we are working with Amazon, Google, Databricks, Snowflake (and others) > to make our "System" test infrastructure even stronger and be able to > automatically test the stability and regressions of integrations of > Airflow. > > J. >
