Hi Wes, Just a quick update here.
I spawned a small ubuntu VM on Azure with a mounted volume to try out the dockerization. I used rootless mode <https://docs.docker.com/engine/security/rootless/> since we would like to run PRs but we have no control over what people PR to the repo and the pipeline is read directly from the PR(*). This works as intended. It is roughly bootstrapped as: 1. partition the mounted volume 2. symlink the external mount to docker's share/docker 2. add non-root user 3. add authorized_key to non-root user 4. ssh into non-root user (why <https://unix.stackexchange.com/a/593756/36325>) 5. install rootless docker 6. enable docker via systemctl --user 7. start build-agent using a token I have the exact commands I used on each step, if needed. There are still some improvements, but IMO it is a starting point. The build is on-going here <https://buildkite.com/test-811/example/builds/85#>. (*) We can revisit this later. This currently fails to run Rust's coverage, which requires `docker run --privileged`. Best, Jorge On Fri, Jan 8, 2021 at 12:38 AM Wes McKinney <wesmck...@gmail.com> wrote: > Good point re re-use of our other Docker images. Probably the best > path would be to attach a few bare-metal machines or spin up some > persistent cloud instances (if we figure out how they're going to be > paid for). > > On Thu, Jan 7, 2021 at 2:15 PM Jorge Cardoso Leitão > <jorgecarlei...@gmail.com> wrote: > > > > Hi Wes, > > > > Thanks for the suggestion and for the offer; much appreciated. I was able > > to run it on my own macbook on the dockerized agent > > <https://buildkite.com/docs/agent/v3/docker#running-via-docker>, and it > > just ran out of the box. > > > > The main limitation of this is that it is more difficult to run our own > > docker images. We can probably make it work via docker on docker, > > but it is likely more evolved (I haven't tried that yet). We could also > > build an image on top of "buildkite/agent:3" for this, but well... > > > > Best, > > Jorge > > > > > > On Thu, Jan 7, 2021 at 8:34 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > > > Jorge -- if you want to test a Linux agent, you could run the > > > buildkite-agent in a Docker container. We (Ursa) could possibly look > > > into adding Dockerized agents on some of our physical machines, > > > particularly if we set up a well-documented procedure for setting this > > > up on a new machine. > > > > > > On Wed, Jan 6, 2021 at 11:29 PM Jorge Cardoso Leitão > > > <jorgecarlei...@gmail.com> wrote: > > > > > > > > Hi Jacob, > > > > > > > > Neal already requested those (and other) actions to be whitelisted > here > > > > https://issues.apache.org/jira/browse/INFRA-21239, but there was no > > > > response yet. > > > > > > > > Best, > > > > Jorge > > > > > > > > > > > > On Thu, Jan 7, 2021 at 6:20 AM Jacob Quinn <quinn.jac...@gmail.com> > > > wrote: > > > > > > > > > From this page, it looks like there have been certain github > > > organizations > > > > > that have been "whitelisted" to allow their github actions to run. > Is > > > there > > > > > a process to do this whitelisting? If the `julia-actions` github > org > > > was > > > > > allowed to run, that would enable everything needed for Julia CI to > > > run. > > > > > > > > > > -Jacob > > > > > > > > > > On Wed, Jan 6, 2021 at 10:00 PM Sutou Kouhei <k...@clear-code.com> > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > > I wasn't following the build queue's state lately, but I think > we > > > > > > > should consolidate the build configurations. > > > > > > > Possible candidates are the PR* workflows > > > > > > > > > > > > https://github.com/apache/arrow/pull/9120 > > > > > > > > > > > > > > > > > > Thanks, > > > > > > -- > > > > > > kou > > > > > > > > > > > > In < > > > cahm19a7v+mclvzmu1jtuztojsbtgdazcm5nswrefjw2w0+c...@mail.gmail.com> > > > > > > "Github Actions feedback time" on Tue, 5 Jan 2021 13:33:38 > +0100, > > > > > > Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I'm concerned about the overall feedback time we have on pull > > > requests. > > > > > > > I have a simple PR to make the comment bot working again, but > no > > > > > > > builds are running even after 30 minutes. > > > > > > > I can see 2-4 running builds, which will make our work much > harder > > > > > > > right before the release. > > > > > > > > > > > > > > I wasn't following the build queue's state lately, but I think > we > > > > > > > should consolidate the build configurations. > > > > > > > Possible candidates are the PR* workflows and good to have > tests > > > which > > > > > > > we could trigger on master instead. > > > > > > > > > > > > > > Opinions? > > > > > > > > > > > > > > Regards, Krisztian > > > > > > > > > > > > > > >
