Hello Everyone, I thought it's a good time to share some news after yesterday's ASF "Builds meeting". The latest news for our CI is that ~ 2 weeks ago GitHub bumped the parallelism of jobs for Public runners for Apache Software Foundation from 300 to 900.
Attached is the graph of queue size for Apache projects where you can see how dramatic improvements it was (other projects of Apache do not have self-hosted runners as we do so they are far bigger users). Thanks to Tobiasz Kedzierski - one of the contributors to Airflow and my friend who developed and maintains the graphs so that ASF can see some stats. This means that builds from forks of our contributors should experience far, far less queuing. Some new things are coming as well - we (ASF with our support) are talking to GitHub to get self-hosted runners on Azure, as well as enabling bigger instances / ARM instances for running our CI jobs. This will likely allow us to optimize some of the build times as well. I already have some ideas how we can make our builds leaner, faster and with less number of failures. The goal is always to get the CI feedback as fast as possible with as little as possible false-negatives (and without burning too much money that our sponsors give us). It's a moving target but we have some good possibilities now on how to make those better. Stay tuned. Also I think this is the right moment to thank both Astronomer and Amazon for the AWS credits (Amazon) and money (Astronomer). Again this year we got USD 15.000 from Amazon (last year we got USD 10.000 and Astronomer pays whatever we exceed above that and invested quite some time (mostly of Ash) to get our "VM" infrastructure up and running. There are also likely more news coming in the light of AIP-47 - where we are working with Amazon, Google, Databricks, Snowflake (and others) to make our "System" test infrastructure even stronger and be able to automatically test the stability and regressions of integrations of Airflow. J.
