Hello everyone, Probably some of you noticed (especially committers but also regular contributors) that our CI got quite a bit speedier over the last couple of months and also quite a bit more stable.
But we need a bit more cloud credits to keep it this way :(. I am reaching out to the community with the kind usk, because maybe there are some sources of credits that we can tap into. It would be great if the community of Airflow could help with that :D. The speed-up has been a joined effort of a number of people - mostly Ash who built the whole auto-scaling infrastructure for it and customized the Github Runner to safely run it on our self-hosted infrastructure, and myself working on optimizing and parallelizing the CI builds (which also got much nicer and more usable, colorful output). However that was at the cost of ... credits from AWS. Unfortunately the burn rate turned quite a bit more than we (my estimations actually) initially anticipated and we are running out of those credits pretty quickly. At the current rate (which I still hope to bring down soon) we will run out the current credits by mid May (or earlier, depending on how many PRs we'll get merged :) ). We are working on getting more credits (keep fingers crossed) from both AWS and GCP teams (there we will need to replicate the infrastructure we have in AWS now). I also have a very fresh idea from the Apache Spark team https://github.com/apache/spark/pull/32092 that resulted from our discussions at [email protected]. We are going to explore which might help to drive the cost down significantly (more on that soon). So eventually the regular cost of the infrastructure might be much cheaper, but this might take a bit of time to get deployed. If you have some ideas, sources of credits (AWS but GCP as well) please reach out to me personally. We can make a very good use of those. J. -- +48 660 796 129
