Hi, During the weekend I created a new GCP project for Apache Airflow CI/CD purposes and I provided access to this project to Jarek et al.
Right now this project is linked to a billing account that has $8000 of GCP credits. It should be good enough for us to start replicating the CI/CD/testing infrastructure on GCP. Regards, Rafal. Cloud Composer <https://cloud.google.com/composer> On Sat, Apr 10, 2021 at 2:51 PM Jarek Potiuk <[email protected]> wrote: > Hello everyone, > > Probably some of you noticed (especially committers but also regular > contributors) that our CI got quite a bit speedier over the last couple of > months and also quite a bit more stable. > > But we need a bit more cloud credits to keep it this way :(. I am > reaching out to the community with the kind usk, because maybe there are > some sources of credits that we can tap into. It would be great if the > community of Airflow could help with that :D. > > The speed-up has been a joined effort of a number of people - mostly Ash > who built the whole auto-scaling infrastructure for it and customized the > Github Runner to safely run it on our self-hosted infrastructure, and > myself working on optimizing and parallelizing the CI builds (which also > got much nicer and more usable, colorful output). > > However that was at the cost of ... credits from AWS. Unfortunately the > burn rate turned quite a bit more than we (my estimations actually) > initially anticipated and we are running out of those credits pretty > quickly. At the current rate (which I still hope to bring down soon) we > will run out the current credits by mid May (or earlier, depending on how > many PRs we'll get merged :) ). > > We are working on getting more credits (keep fingers crossed) from both > AWS and GCP teams (there we will need to replicate the infrastructure we > have in AWS now). > > I also have a very fresh idea from the Apache Spark team > https://github.com/apache/spark/pull/32092 that resulted from our > discussions at [email protected]. We are going to explore which might > help to drive the cost down significantly (more on that soon). So > eventually the regular cost of the infrastructure might be much cheaper, > but this might take a bit of time to get deployed. > > If you have some ideas, sources of credits (AWS but GCP as well) please > reach out to me personally. We can make a very good use of those. > > J. > > > -- > +48 660 796 129 <+48%20660%20796%20129> >
