Neal Richardson created ARROW-7203:
--------------------------------------
Summary: [CI] Trigger GitHub Action cron workflows on demand
Key: ARROW-7203
URL: https://issues.apache.org/jira/browse/ARROW-7203
Project: Apache Arrow
Issue Type: Wish
Components: Continuous Integration, Developer Tools
Reporter: Neal Richardson
The new GitHub Actions workflows in place of Travis-CI are great. So much
faster feedback than before. One question we've had since migrating is how
should Crossbow and the GHA jobs interact. GHA makes it easy enough to schedule
nightly jobs, so we don't really need the Crossbow trickery to get the nightly
builds to run. However, we don't yet have a solution for the other way we use
Crossbow: triggering builds via PR comments (using ursabot).
As it turns out, last week I was at the GitHub Universe conference and spoke to
some of the lead devs about our experience and needs. I asked especially about
this because the day before, I had debugged and fixed a cron workflow
(https://issues.apache.org/jira/browse/ARROW-7164) and struggled with how to
test that I had fixed it. (Interestingly, I happened to talk to the dev who was
responsible for the action code change that caused our job to start failing.)
Triggering cron workflows on demand was not a feature they (or at least he) had
considered. We brainstormed a few ways we might be able to do it. None of them
were simple or clean. Here's what we discussed. See also the
[docs|https://help.github.com/en/actions/automating-your-workflow-with-github-actions/events-that-trigger-workflows]
for events that trigger workflows.
Ideally, we could use the {{issue_comment}} event to trigger a workflow, just
as we do now via ursabot and crossbow. But there are some challenges:
* We can trigger based on an issue_comment being created, but then we'd have to
write some code to parse the payload of the event (in the {{github.event}}
object) and then do things only if the right build name is found. This would
meant that for every cron workflow we have, this logic would run every time
anyone makes any comment on any issue or PR. That might have some undesirable
side effects.
* The issue_comment event takes the workflow script from the master branch. So
if you're testing changes to the workflow yaml itself, you're out of luck.
* You could work around this by also conditionally running the workflow also if
the workflow file itself is changed.
* Because the event triggers from master, you need to modify the checkout step
to checkout a commit/ref. Unfortunately, the commit SHA isn't present in the
[event
payload|https://developer.github.com/v3/activity/events/types/#issuecommentevent].
You could probably parse it out of one of the URLs in the payload, though.
Assuming you do, then you probably need to make this behavior conditional on
whether you're running from an issue_comment (in which case you parse the event
and go with master) or from cron or some other means (in which case you don't
want to specify a ref).
* Think of all of the complexity here, and realize that unless there's some way
to package this up into an action or template or something, we have to
replicate this for every workflow we want to be able to trigger like this.
An alternative strategy would be to use the existing ursabot integration and
trigger GitHub workflows from it using a [repository
dispatch|https://developer.github.com/v3/repos/#create-a-repository-dispatch-event]
event. The repository dispatch event would have an event_type that (somehow?)
we would map to the workflow, and then in the client_payload we could include
any additional build params. This would have to include the PR number or commit
SHA because, just as with issue_comments, the workflow will run from master so
we'll need to explicitly checkout something else. The other limitation is that
repository dispatch requires an API token with repo write access; fortunately,
ursabot already has had to deal with this.
A further ursabot-centric approach would extend crossbow to be able to create
GHA workflows, and triggering on demand a workflow via crossbow (through
ursabot comment bot or otherwise) would essentially copy the workflow to its
repository, amending the workflow to checkout the repo and commit and to run on
push.
In sum, it's not straightforward to do this, and as it stands now, there's a
bit of code to write somewhere for this.
cc [~kou] [~kszucs]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
