On Thu, Mar 5, 2020 at 5:01 AM Krisztián Szűcs <[email protected]> wrote: > > On Thu, Mar 5, 2020 at 10:59 AM Antoine Pitrou <[email protected]> wrote: > > > > > > It sounds like this would be a good reason to use BuildKite, which AFAIU > > can automatically provision and operate cloud resources for us? > That's true for buildbot as well [1] including openstack and mesos support > which > would be nice for physical machines.
I think we should try to reduce the number of services we have to configure and sysadmin, it has a non-zero cost in time and money. > With the ursa machines down we ended up with two missing services: > - nightly trigger and report: I've already ported the cron jobs triggering the > builds and reporting to github actions [2] > - the comment bot: it is implemented in ursabot including the github event > listener, so buildkite wouldn't provide a solution here because we need > to maintain a web service for it. > > The solution for the comment bot could be to factor out the implementation > from ursabot and run it from a github actions build triggered using the > issue_comment event [3] (buildkite doesn't seem to support it currently [4]). > > We can also host the buildbot buildmaster in the cloud, at least until we > decomission all of its services. > > [1] > https://docs.buildbot.net/latest/manual/configuration/workers.html#supported-latent-workers > [2] https://github.com/ursa-labs/crossbow/tree/master/.github/workflows > [3] > https://help.github.com/en/actions/reference/events-that-trigger-workflows#issue-comment-event-issue_comment > [4] https://github.com/buildkite/feedback/issues/288 > > > > > > Le 04/03/2020 à 16:21, Wes McKinney a écrit : > > > hi folks, > > > > > > The tornado the night before last in Nashville, Tennessee temporarily > > > disabled the physical hardware that I have been running there where > > > we've been running "Ursabot" builds and where we've been experimenting > > > with other self-hosted CI solutions like GitHub Actions Self-Hosted > > > Runners and Buildkite. > > > > > > While dedicated physical hardware can be useful to reduce cloud > > > computing costs, I think this natural disaster should help inform our > > > approach to this problem: > > > > > > * In the event that physical hosted infrastructure becomes > > > unavailable, we eventually should have the capability to spin up > > > machines in the cloud with the desired properties (GCE provides both > > > Linux and Windows VMs, for example) > > > * Adding new machines to our CI process ideally should not require a > > > human in-the-loop (GHA presently requires a human -- in particular > > > someone from ASF Infra -- in the loop to add workers, so this IMHO > > > should be taken into consideration) > > > > > > Any other thoughts about this topic would be welcome. > > > > > > Thanks > > > Wes > > > > > > On Thu, Feb 20, 2020 at 9:27 AM Krisztián Szűcs > > > <[email protected]> wrote: > > >> > > >> On Thu, Feb 20, 2020 at 3:53 PM Wes McKinney <[email protected]> wrote: > > >>> > > >>> On Thu, Feb 20, 2020 at 8:40 AM Krisztián Szűcs > > >>> <[email protected]> wrote: > > >>>> > > >>>> On Thu, Feb 20, 2020 at 12:14 PM Wes McKinney <[email protected]> > > >>>> wrote: > > >>>>> > > >>>>> hi Ganesh, > > >>>>> > > >>>>> Thanks for writing. > > >>>>> > > >>>>> I've been working on setting up Buildkite (BK) as a way for third > > >>>>> parties for attach machines to run builds on, with a free organization > > >>>>> at > > >>>>> > > >>>>> https://buildkite.com/apache-arrow > > >>>>> > > >>>>> Configuring a new machine to accept builds is very easy [1] and takes > > >>>>> less than 60 seconds on Linux or macOS (though maybe a bit more work > > >>>>> on Windows). Currently I've attached 6 machines: > > >>>>> > > >>>>> * 2 CUDA-capable Linux x86 > > >>>>> * 3 armhf machines (not super high-powered), 1 CUDA-capable > > >>>>> * 1 macOS > > >>>>> > > >>>>> We're still waiting on ASF Infra to twiddle some bits so that builds > > >>>>> triggered in BK can report commit statuses on GitHub [2] > > >>>>> > > >>>>> It's possible we can use self-hosted GitHub Actions (GHA) for this > > >>>>> also but the workflow for new machines to be contributed needs to be > > >>>>> proven out. > > >>>> I've already tried it out, and setting up self-hosted github runners > > >>>> is just as > > >>>> easy as with buildkite, drawbacks: > > >>> > > >>> I don't mean to be argumentative, but I don't see how this can be true > > >>> if we don't have access to the "Settings" tab on GitHub > > >> On a fork where I have access for that tab. > > >>> > > >>> https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners > > >>> > > >>>> - I'm unsure how would the tagging selection work in practice [1] > > >>>> - We won't have access to the runners dashboard in lack of admin rights > > >>>> for the apache/arrow repository - so we need to test out the > > >>>> workflow. > > >>>> > > >>>> I've created an INFRA ticket to get some information and to track it: > > >>>> https://issues.apache.org/jira/browse/INFRA-19875 > > >>>> > > >>>> [1] > > >>>> https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-a-self-hosted-runner > > >>>>> > > >>>>> Thanks, > > >>>>> Wes > > >>>>> > > >>>>> [1]: > > >>>>> https://github.com/ursa-labs/dev-tools/blob/master/buildkite/debian_agent_bootstrap.sh > > >>>>> [2]: https://issues.apache.org/jira/browse/INFRA-19217 > > >>>>> > > >>>>> On Wed, Feb 19, 2020 at 3:38 PM Ganesh Raju <[email protected]> > > >>>>> wrote: > > >>>>>> > > >>>>>> Hi, > > >>>>>> I am following up on the discussion from here > > >>>>>> <https://github.com/apache/arrow/pull/6253>, with interest to have > > >>>>>> dedicated arm hardware for CI setup. We can surely help with that if > > >>>>>> we get > > >>>>>> a go-ahead from the project. > > >>>>>> > > >>>>>> Thanks, > > >>>>>> Ganesh > > >>>>>> > > >>>>>> -- > > >>>>>> IRC: ganeshraju@#linaro on irc.freenode.ne > > >>>>>> <http://irc.freenode.net/>t
