Hi, all I tried to verify the possibility of *Linux arm64 scheduled job *using self-hosted action, below is some progress and I would like to hear suggestion from you in the next step (continue or stop).
Related JIRA: SPARK-35607 <https://issues.apache.org/jira/browse/SPARK-35607> *## About self-hosted Github Action:* Currently, self-hosted action supported x64(Linux, macOS, Windows), ARM64(Linux only), ARM32(Linux only) <https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#architectures> . There is guidance on self-hosted runners from Apache Infra <https://cwiki.apache.org/confluence/display/INFRA/GitHub+-+self-hosted+runners>. The gap to enable self-hosted runner on Apache repo is resource security considerations, specifically, it's to prevent the self-hosted runner from being accessed by unallow users' PR. As info and suggestion from ASF, the apache/airflow team maintained a custom runner <https://github.com/ashb/runner/tree/releases/pr-security-options>, and it's also used by apache/airflow in their CI. So, we could just use this directly. TLDR, what we needed is setup resource with custom runner, then enable these resources in self-hosted action. *## Test on self-hosted Github Action with custom runner:* Here is some tries on my local repo: 1. Spark Maven/SBT test: PR: https://github.com/apache/spark/pull/35088 TEST: https://github.com/Yikun/spark/pull/51 2. PySpark test: PR: https://github.com/apache/spark/pull/35049 TEST: https://github.com/Yikun/spark/pull/53 3. Pull request test on unallow user: TEST: https://github.com/Yikun/spark/pull/60 The self-hosted runner will prevent the PR access the runner due to "Running job on worker spark-github-runner-0001 disallowed by security policy". *## Pros of self-hosted github aciton:* - Satisfy the simple demands of Linux arm64 sheduled jobs. - Reuse the main workflow of github action. - All changes are visible on github is easy to review. - Easy to migrate when official GA arm64 support ready. *## What's the next step:* * If we can also consider self-hosted action as optional, I will submit a JIRA on Apache Infra to request the token to continue, like: https://issues.apache.org/jira/browse/INFRA-21305 * If we certainly think that self-hosted action is not a wise choice, I will try to find other way. There are also some initial discusson, just FYI: https://github.com/dongjoon-hyun/ApacheSparkGitHubActionImage/pull/6 Regards, Yikun
