By default builds are run on Github runners. A public repo has 2000 minutes monthly, that's definitely not enough for us. But having having GCP resources donated to us we should be able to overcome this problem.
T. On Mon, Dec 9, 2019 at 11:02 PM Pablo Estrada <pabl...@google.com.invalid> wrote: > That looks pretty cool! It helps that they're github native. Where does the > compute power come from? Is it provided by github? > > On Mon, Dec 9, 2019 at 1:57 PM Tomasz Urbaszek < > tomasz.urbas...@polidea.com> > wrote: > > > Hi all, > > > > It sometime since we last discussed using other CI than Travis. One of > the > > main reasons behind considering Gitlab CI was its ability to work on > > self-hosted runner. However, over time of few long weeks Github Actions > > matured enough to allow using self-hosted runners! > > > > Github Actions are still growing but using them have few big advantages: > > - they are Github natives > > - forking repo and enabling actions will run CI on your fork > automatically > > - variety of actions (PR checks, greetings, etc) > > > > I put together a PoC of CI in our internal repo: > > https://github.com/PolideaInternal/airflow/pull/542 > > My impression is quite good. I like information about steps successes at > > the PR level (no need to go to CI to check which step failed). The build > > log view is a little bit clumsy but it works. > > > > Does any of you have any experience with Github Actions? Any thoughts > > about using it? > > > > Best, > > Tomek > > > > On 2019/08/09 13:55:11, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > > FYI: Interesting article about the history behind GitLabCI (featuring > > > Kamil, my friend). > > > > > > https://about.gitlab.com/2019/08/08/built-in-ci-cd-version-control-secret/?fbclid=IwAR2tEfqLaDXTCd1mD6XUZMX7hGYBfZcohPtI2BP3-oK_Yk_EHIXF4zLDixk > > > > > > On Mon, Aug 5, 2019 at 7:14 PM Jarek Potiuk <jarek.pot...@polidea.com> > > > wrote: > > > > > > > Some update on my GitLab experiences so far: > > > > > > > > TL;DR; I think the POC has shown that we can fairly easily replicate > > the > > > > CI in GitLab + Kubernetes. I think i can say - it generally works, I > > can > > > > plug it in for master/v1-10-test builds in the main Airflow project > > for a > > > > few weeks to see how it is doing (while I am no holidays) and once we > > see > > > > it running and get the support for PRs from GitLab we can switch to > it. > > > > > > > > What do you think ? Should i call a vote or just try to set it up ? > > > > > > > > Some details > > > > > > > > - I manged to get full working builds in GitLabCI + kubernetes - > > > > without the kubernetes-specific tests yet, but this should be > > rather easy > > > > with kind (looking at it next): > > > > - Working example here - you can take a look and compare the > UI/how > > it > > > > is to navigate, comparing to Travis etc: > > > > https://gitlab.com/Jarek.Potiuk/airflow/pipelines/74625817 > > > > - Per-job it is a bit slower than Travis so far (still around 35 > > > > minutes in total), but I plan to optimise it further. I can play > > with > > > > memory/cpu settings of individual workers (Got some reasonable > > values now), > > > > I can use local SSD disk as Docker storage/logs/etc > > > > - I got an approval for 72vCPU quota (up for initial 24) - that > > should > > > > let us build 3 builds in parallel independently from each other. > > > > - I managed to get Preemptible nodes working (we have built in > retry > > > > mechanism in GitLab to work in case of system failures like that > > > > - Current spending with > 120 builds is 40 USD. We should be way > > below > > > > 500 USD/month according to my back-of-the-envelope calculations. > > Likely > > > > well below > > > > - The current setup does not use GCR as cache and Kaniko as I > > > > originally planned. GCR would require custom authentication (and > > > > easy-to-steal secrets) and Kaniko does not yet well handle > > multi-staging > > > > builds (cache does not work > > > > https://github.com/GoogleContainerTools/kaniko/issues/682). I > > updated > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-23+Migrate+out+of+Travis+CI > > to > > > > reflect that. > > > > - We only use GCR as mirroring of DockerHub - so that we can have > > > > reliable downloads not depending on DockerHub's stability (it has > > problems > > > > sometimes) > > > > - All in-all, it's GCP-independent. It could be run in any > > Kubernetes > > > > cluster (some optimisations like local volumes mounting for docker > > engine > > > > might have GCP-specific assumptions, but should be generally > > replicable). > > > > - You can take a look at the current source code in > > > > https://github.com/potiuk/airflow/commits/test-gitlab-ci > > > > - There will be some updates (I will get rid of custom builder > > Docker, > > > > simplify it a bit and implement kubernetes tests) - it's mostly > some > > > > cleanups + removal of Travis-Specific variables + gitlab.ci yaml > > with > > > > job definitions. > > > > > > > > J. > > > > > > > > > > > > On Wed, Jul 31, 2019 at 10:57 AM Jarek Potiuk < > > jarek.pot...@polidea.com> > > > > wrote: > > > > > > > >> So GitLab already works on automatically running builds from for PRs > > :). > > > >> > > > >> Kamil got involved and will be out advocate on it: > > > >> https://gitlab.com/gitlab-org/gitlab-ce/issues/65139 > > > >> J. > > > >> > > > >> Principal Software Engineer > > > >> Phone: +48660796129 <+48%20660%20796%20129> > > > >> > > > >> pt., 26 lip 2019, 18:12 użytkownik Jarek Potiuk < > > jarek.pot...@polidea.com> > > > >> napisał: > > > >> > > > >>> Update: I added appropriate comment in the GitLab CI issue about > PRs > > and > > > >>> we are getting attention of Jason Lenny - director of Product > > Management @ > > > >>> GitLab. Let's hope they prioritise it quickly enough. > > > >>> > > > >>> Speaking of potential complexity/Maintenance - in order to > alleviate > > any > > > >>> maintenance worries, I think about setting up the whole system on > > GitLab > > > >>> CI + GKE and running it in parallel to Travis for quite some time > > (even > > > >>> months) so that we can switch it at any time. Then we will be able > > to tune > > > >>> it according to real use cases and compare the experience of both > > systems. > > > >>> > > > >>> Also I am going for holidays in two weeks and I will make sure that > > > >>> there will be someone with GitLab + Kubernetes experience (from my > > company) > > > >>> who can take over and make sure there will be no problems. However > I > > am > > > >>> quite confident :D nothing is going to happen while I am away. I > > would also > > > >>> invite whoever from committers who would like to join the project > and > > > >>> gitlab instance (once I setup POC) to learn and see how easy it is > > and how > > > >>> maintenance free it is going to be. > > > >>> > > > >>> J. > > > >>> > > > >>> On Fri, Jul 26, 2019 at 2:56 PM Kamil Breguła < > > kamil.breg...@polidea.com> > > > >>> wrote: > > > >>> > > > >>>> GKE and its own CI will allow us to solve other problems - > building > > > >>>> and publishing documentation from the master branch. Currently, > > > >>>> building is done using the RTD service. Unfortunately, our project > > is > > > >>>> too large and often the documentation is not built properly. > > > >>>> https://readthedocs.org/projects/airflow/builds/ > > > >>>> We should think about another way to build documentation. In the > > ideal > > > >>>> world, building documentation should use the same environment as > > > >>>> checking documentation on CI. Adding this step to Travis can > further > > > >>>> reduce our development opportunities. > > > >>>> Discussion on Slack about it: > > > >>>> > > https://apache-airflow.slack.com/archives/CJ1LVREHX/p1561756652021900 > > > >>>> > > > >>>> It is worth thinking about the fact that our project will soon > have > > a > > > >>>> website and our documentation will also be available in many > > > >>>> languages. Currently, talks are taking place with the design > studio > > > >>>> and developers who can make these websites ;-) > > > >>>> > > > >>>> > > > https://lists.apache.org/thread.html/982c7baa06742ad722f2baa0db53ad99aea6c26b14b7d6d4aa522677@%3Cdev.airflow.apache.org%3E > > > >>>> We should provide an environment that will allow you to build a > > > >>>> website and documentation. At best, these tasks should be > combined. > > I > > > >>>> hope that we will be able to create a website that will be a real > > > >>>> support for the community on current events, so it will be updated > > > >>>> frequently. > > > >>>> > > > >>>> It seems to me that the project will grow. If we now have problems > > > >>>> with Travis, then the significance of these problems in the future > > can > > > >>>> only grow. Now we have a chance to provide a stable infrastructure > > for > > > >>>> the project for a long time. > > > >>>> > > > >>>> I would like to share another situation which was not pleasant for > > me. > > > >>>> Recently I wanted to send >10 PR, but because of Travis, I had to > > wait > > > >>>> for the weekend to send changes. If I would send my changes in a > > week, > > > >>>> I would block the queue for a few hours. Although I did it over > the > > > >>>> weekend, I got the message that the queue is blocked on Travis by > my > > > >>>> jobs. > > > >>>> > > > >>>> On Tue, Jul 23, 2019 at 6:12 PM Jarek Potiuk < > > jarek.pot...@polidea.com> > > > >>>> wrote: > > > >>>> > > > > >>>> > Hello Everyone, > > > >>>> > > > > >>>> > I prepared a short docs where I described general architecture > of > > the > > > >>>> > solution I imagine we can deploy fairly quickly - having GitLab > CI > > > >>>> support > > > >>>> > and Google provided funding for GCP resources. > > > >>>> > > > > >>>> > I am going to start working on Proof-Of-Concept soon but before > I > > > >>>> start > > > >>>> > doing it, I would like to get some comments and opinions on the > > > >>>> proposed > > > >>>> > approach. I discussed the basic approach with my friend Kamil > who > > > >>>> works at > > > >>>> > GitLab and he is a CI maintainer and this is what we think will > be > > > >>>> > achievable in fairly short time. > > > >>>> > > > > >>>> > > > > >>>> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-23+Migrate+out+of+Travis+CI > > > >>>> > > > > >>>> > I am happy to discuss details and make changes to the proposal - > > we > > > >>>> can > > > >>>> > discuss it here or as comments in the document. > > > >>>> > > > > >>>> > Let's see what people think about it and if we get to some > > consensus > > > >>>> we > > > >>>> > might want to cast a vote (or maybe go via lasy consensus as > this > > is > > > >>>> > something we should have rather quickly) > > > >>>> > > > > >>>> > Looking forward to your comments! > > > >>>> > > > > >>>> > J. > > > >>>> > > > > >>>> > -- > > > >>>> > > > > >>>> > Jarek Potiuk > > > >>>> > Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > >>>> > > > > >>>> > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > > <+48%20660%20796%20129>> > > > >>>> > [image: Polidea] <https://www.polidea.com/> > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> > > > >>> Jarek Potiuk > > > >>> Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >>> > > > >>> M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > > <+48%20660%20796%20129>> > > > >>> [image: Polidea] <https://www.polidea.com/> > > > >>> > > > >>> > > > > > > > > -- > > > > > > > > Jarek Potiuk > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > > <+48%20660%20796%20129>> > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > -- > > > > > > Jarek Potiuk > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > M: +48 660 796 129 <+48%20660%20796%20129> <+48660796129 > > <+48%20660%20796%20129>> > > > [image: Polidea] <https://www.polidea.com/> > > > > > > -- Tomasz Urbaszek Polidea <https://www.polidea.com/> | Junior Software Engineer M: +48 505 628 493 <+48505628493> E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> Unique Tech Check out our projects! <https://www.polidea.com/our-work>
