Fantastic news Rafał! Thank you and the Composer team for that ! On Tue, Apr 13, 2021 at 9:27 PM Rafal Biegacz <[email protected]> wrote:
> Hi, > > During the weekend I created a new GCP project for Apache Airflow CI/CD > purposes and I provided access to this project to Jarek et al. > > Right now this project is linked to a billing account that has $8000 of > GCP credits. It should be good enough for us to start replicating the > CI/CD/testing infrastructure on GCP. > > Regards, Rafal. > Cloud Composer <https://cloud.google.com/composer> > > > > > On Sat, Apr 10, 2021 at 2:51 PM Jarek Potiuk <[email protected]> wrote: > >> Hello everyone, >> >> Probably some of you noticed (especially committers but also regular >> contributors) that our CI got quite a bit speedier over the last couple of >> months and also quite a bit more stable. >> >> But we need a bit more cloud credits to keep it this way :(. I am >> reaching out to the community with the kind usk, because maybe there are >> some sources of credits that we can tap into. It would be great if the >> community of Airflow could help with that :D. >> >> The speed-up has been a joined effort of a number of people - mostly Ash >> who built the whole auto-scaling infrastructure for it and customized the >> Github Runner to safely run it on our self-hosted infrastructure, and >> myself working on optimizing and parallelizing the CI builds (which also >> got much nicer and more usable, colorful output). >> >> However that was at the cost of ... credits from AWS. Unfortunately the >> burn rate turned quite a bit more than we (my estimations actually) >> initially anticipated and we are running out of those credits pretty >> quickly. At the current rate (which I still hope to bring down soon) we >> will run out the current credits by mid May (or earlier, depending on how >> many PRs we'll get merged :) ). >> >> We are working on getting more credits (keep fingers crossed) from both >> AWS and GCP teams (there we will need to replicate the infrastructure we >> have in AWS now). >> >> I also have a very fresh idea from the Apache Spark team >> https://github.com/apache/spark/pull/32092 that resulted from our >> discussions at [email protected]. We are going to explore which might >> help to drive the cost down significantly (more on that soon). So >> eventually the regular cost of the infrastructure might be much cheaper, >> but this might take a bit of time to get deployed. >> >> If you have some ideas, sources of credits (AWS but GCP as well) please >> reach out to me personally. We can make a very good use of those. >> >> J. >> >> >> -- >> +48 660 796 129 <+48%20660%20796%20129> >> > -- +48 660 796 129
