Comment from outside - I looked briefly at the implementation and docs and the GHA controller looks very clear and straightforward to implement. Fantastic job Jacob and big shoutout to Voltron Data for implementing and open-sourcing it.
I am going to try it out in Apache Airflow very soon. We were waiting for something that GitHub Actions are cooking up https://github.com/orgs/github/projects/4247 but it just moved from Q4 2022 to Q1 2022 so .... you never know :). One small comment for the security of hosting your self-hosted runners that you might want to take into account. While this is great there are ephemeral runners (they provide all the necessary security boundaries, escaping from a container in K8S is not an easy feat), there is still one case where allowing any PRs to run code in your self-hosted runners is potentially problematic - i.e. possibility of using the processing power of your machines by anyone to do any kind of jobs (and do it with your donated credits or money). For example cryptomining. This is not an academic problem - this has already happened in the past https://github.blog/2021-04-22-github-actions-update-helping-maintainers-combat-bad-actors/ and that's why GitHub Actions introduced mandatory "Approval" for first-time-users Pull requests - because the bad actors were actuallly abusing Github's public runners to mine crypto. The approval workflow actually protects against the "mass abuse" - i.e. creating new accounts and using them to exploit this on multiple repos, but it does not protect you against the case that some collaborators will use your self-hosted runners to do any kind of computing. There are likely ways to mitigate it like limiting the maximum time container can run, and of course attempts to do so might be caught during reviews (and the offending user can be called out) - but I think if you want powerful CI and have a lot of contributors, this might slip under the radar easily unless you have some monitoring in place. The fact that it is not mass-exploitable by new users, makes it less likely to occur (because the regular users might lose their reputation if they attempt to do it), but it is still a possibility. It's up to you if you would like to protect against it in some ways (in Airflow we will likely continue using https://github.com/ashb/runner and limit the self-hosted workflows to "main" workflows and to maintainer's PRs) and it is not a blocker, but I wanted you to be aware of this potential abuse scenario. J. On Fri, Dec 16, 2022 at 7:27 PM Jacob Wujciak <ja...@voltrondata.com.invalid> wrote: > No news with regards to arrow specific S390x machines but apparently IBM > has donated a number of S390x VMs to the ASF which we should be able to use > but I have not had the time yet to investigate this option. > > > Matt Topol <zotthewiz...@gmail.com> schrieb am Fr., 16. Dez. 2022, 17:01: > > > These are awesome! Has there been any luck in reaching out to IBM to see > if > > they could donate one or more s390x VMs to use as runners for testing the > > s390x builds? That is probably my only concern with Travis going away at > > EOY, since we don't have a way currently to test those builds on GH > > Actions. > > > > --Matt > > > > On Fri, Dec 16, 2022 at 8:46 AM Jacob Wujciak > > <ja...@voltrondata.com.invalid> > > wrote: > > > > > I would like to propose the addition of a self-hosted runner system to > > the > > > arrow repository to add speciality runners (arm64 and CUDA). This will > > > allow us to compensate for the arm64 jobs that previously ran on > Travis, > > > which will be turned off EOY[1]. > > > > > > The migration to GitHub Issues will require a significant extension of > > our > > > existing “comment bot”-workflows (e.g. assigning and labeling issues > for > > > non-committers, see [3]), with such a system we could add reserved > > runners > > > that only pick up these “comment bot”-jobs to guarantee a smooth > > developer > > > experience, regardless of the state of the ASF CI resources. > > > > > > As the allocation of GitHub-hosted runners for the Apache software > > > foundation was recently increased, the queue times are currently low, > but > > > this will inevitably change and such a system would enable us to react > > > quickly to such changes by adding new Windows and Linux nodes without > any > > > need for INFRA intervention. > > > > > > We at Voltron Data have been working on a Kubernetes based system to > > deploy > > > auto-scaling ephemeral GitHub runners that can be seamlessly added to > the > > > arrow repository via a Github App. As the runners are ephemeral (each > job > > > is run in an isolated environment that is destroyed once the job is > done) > > > the usual security issues with self-hosted runners do not apply [2]. > > > > > > Voltron Data has open sourced the necessary Infrastructure as Code [4], > > > this makes it possible for other interested parties to donate CI > capacity > > > to arrow or other ASF projects by cloning the IaC, setting up and > > > maintaining their own Instance of the system. Voltron Data will set up > > and > > > maintain one instance of the system. > > > > > > The dockerfiles for the runners will be added to the main arrow repo to > > > facilitate easy changes and updates to the runner configuration for the > > > community. > > > > > > Best, > > > Jacob > > > > > > [1]: > https://cwiki.apache.org/confluence/display/INFRA/Travis+Migrations > > > > > > [2]: > > > > > > > > > https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security > > > > > > [3]: https://github.com/apache/arrow/actions/workflows/comment_bot.yml > > > > > > [4]: https://github.com/voltrondata-labs/gha-controller-infra > > > > > >
