Thank you for sharing this Ismaël. This 180 jobs limit across all Apache projects sounds like a problem for Beam, because we are running quite a bit of GH actions already. Following the Airflow suggestions, we can add VMs to apache-beam-testing projects to add Beam specifici private runners to address the issue. GHs suggestion against using private VMs in public projects [1] is related to the risk of unauthorized PRs running unexpected workloads in these VMs. As far as I remember, we did not have this problem with our jenkins machines and anyone being able to run code with their PRs. And Airflow has the suggestion of use preemptible machines. We can do the same and these machines are always recycled after 24 hours limiting the risks.
/cc @Tyson Hamilton <[email protected]> @David Lu <[email protected]> @Alan Myrvold <[email protected]> [1] https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories On Mon, Feb 8, 2021 at 3:30 AM JB Onofré <[email protected]> wrote: > Hi Ismaël. > > Thanks for sharing. I started to evaluate GitHub actions on some other > Apache projects and the doc is interesting. > > Regards > JB > > > Le 8 févr. 2021 à 12:22, Ismaël Mejía <[email protected]> a écrit : > > > > Just for reference and related to this thread. It seems we may end up > > also having this queue issue (even if we don't fully move to Github > > actions). > > "For Apache projects, starting December 2020 we are experiencing a > > high strain of GitHub Actions jobs. All Apache projects are sharing > > 180 jobs and as more projects are using GitHub Actions the job queue > > becomes a serious bottleneck." > > > > An interesting document shared recently on builds@ goes deeper on how > > the Airflow project is dealing with this: > > > https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit# > > > >> On Mon, Jan 18, 2021 at 1:28 PM Elliotte Rusty Harold > >> <[email protected]> wrote: > >> > >>> On Mon, Jan 18, 2021 at 10:49 AM Ismaël Mejía <[email protected]> > wrote: > >>> > >>> Thanks for sharing this Pablo, This looks super interesting. We should > >>> see if it could make sense to migrate our Jenkins infra to GitHub > >>> Actions given that it is free and quickly becoming the new 'standard', > >>> Good points it is 'free' because we will bring our machines and Google > >>> pays :) bad points we will become 100% github dependant. > >>> > >> > >> Github actions have a really big advantage over Jenkins: they run on > >> forks, not just branches. This is very useful to non-commmiter > >> contributors. > >> > >> On the minus side it's not clear if one can see the logs from the > >> integration tests, which is blocking some work in the > >> maven-site-plugin: > >> > >> > https://github.com/apache/maven-site-plugin/pull/34#issuecomment-762207488 > >> > >> -- > >> Elliotte Rusty Harold > >> [email protected] > >
