Hi all, In case you may find this interesting / valuable: Airflow has configured their own machines for Github actions.
Here's the PR https://github.com/apache/airflow/pull/13730 And here's the thread: https://lists.apache.org/thread.html/r2e398f86479e4cbfca13c22e4499fb0becdbba20dd9d6d47e1ed30bd%40%3Cdev.airflow.apache.org%3E On Mon, Feb 8, 2021 at 2:56 PM Ahmet Altay <[email protected]> wrote: > Thank you for sharing this Ismaël. > > This 180 jobs limit across all Apache projects sounds like a problem for > Beam, because we are running quite a bit of GH actions already. Following > the Airflow suggestions, we can add VMs to apache-beam-testing projects to > add Beam specifici private runners to address the issue. GHs suggestion > against using private VMs in public projects [1] is related to the risk of > unauthorized PRs running unexpected workloads in these VMs. As far as I > remember, we did not have this problem with our jenkins machines and anyone > being able to run code with their PRs. And Airflow has the suggestion of > use preemptible machines. We can do the same and these machines are always > recycled after 24 hours limiting the risks. > > /cc @Tyson Hamilton <[email protected]> @David Lu <[email protected]> @Alan > Myrvold <[email protected]> > > [1] > https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories > > On Mon, Feb 8, 2021 at 3:30 AM JB Onofré <[email protected]> wrote: > >> Hi Ismaël. >> >> Thanks for sharing. I started to evaluate GitHub actions on some other >> Apache projects and the doc is interesting. >> >> Regards >> JB >> >> > Le 8 févr. 2021 à 12:22, Ismaël Mejía <[email protected]> a écrit : >> > >> > Just for reference and related to this thread. It seems we may end up >> > also having this queue issue (even if we don't fully move to Github >> > actions). >> > "For Apache projects, starting December 2020 we are experiencing a >> > high strain of GitHub Actions jobs. All Apache projects are sharing >> > 180 jobs and as more projects are using GitHub Actions the job queue >> > becomes a serious bottleneck." >> > >> > An interesting document shared recently on builds@ goes deeper on how >> > the Airflow project is dealing with this: >> > >> https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit# >> > >> >> On Mon, Jan 18, 2021 at 1:28 PM Elliotte Rusty Harold >> >> <[email protected]> wrote: >> >> >> >>> On Mon, Jan 18, 2021 at 10:49 AM Ismaël Mejía <[email protected]> >> wrote: >> >>> >> >>> Thanks for sharing this Pablo, This looks super interesting. We should >> >>> see if it could make sense to migrate our Jenkins infra to GitHub >> >>> Actions given that it is free and quickly becoming the new 'standard', >> >>> Good points it is 'free' because we will bring our machines and Google >> >>> pays :) bad points we will become 100% github dependant. >> >>> >> >> >> >> Github actions have a really big advantage over Jenkins: they run on >> >> forks, not just branches. This is very useful to non-commmiter >> >> contributors. >> >> >> >> On the minus side it's not clear if one can see the logs from the >> >> integration tests, which is blocking some work in the >> >> maven-site-plugin: >> >> >> >> >> https://github.com/apache/maven-site-plugin/pull/34#issuecomment-762207488 >> >> >> >> -- >> >> Elliotte Rusty Harold >> >> [email protected] >> >>
