Re: New CI system: Ursabot
It was resolved with https://github.com/ursa-labs/ursabot/pull/118 See the builders here https://ci.ursalabs.org/#/builders?tags=%2Bcuda On Tue, Jun 25, 2019 at 9:54 PM Wes McKinney wrote: > Krisz -- can the CUDA issue be tracked somewhere so we remember to do > it (and set ARROW_CUDA=on)? > > On Mon, Jun 24, 2019 at 4:42 AM Krisztián Szűcs > wrote: > > > > We already have a CUDA builder in ursabot [1], just need to enable > > --runtime=nvidia for the docker worker. > > > > [1]: > > > https://github.com/ursa-labs/ursabot/blob/master/ursabot/builders.py#L445 > > > > On Fri, Jun 21, 2019 at 9:58 PM Keith Kraus wrote: > > > > > There's nvidia-docker (https://github.com/NVIDIA/nvidia-docker) which > > > handles passing through the GPU devices and necessary driver modules > into a > > > docker container. CUDA doesn't get mapped in as it's userspace so > you'll > > > need to either use an image with CUDA baked in (i.e. > > > https://hub.docker.com/r/nvidia/cuda) or install CUDA yourself into > your > > > container. > > > > > > -Keith > > > > > > On 6/21/19, 2:20 PM, "Antoine Pitrou" wrote: > > > > > > > > > Is it possible to test CUDA under a Docker container? > > > > > > I feel like I'm the only person who routinely tests CUDA on my home > > > machine :-) And of course I only do that on Linux... > > > > > > Regards > > > > > > Antoine. > > > > > > > > > On Fri, 21 Jun 2019 12:23:10 -0500 > > > Wes McKinney wrote: > > > > hi folks, > > > > > > > > I would suggest the following development approach to help with > > > > increasing our CI capacity: > > > > > > > > 1. For all Linux builds, refactor Travis CI jobs to be > Docker-based > > > > and not depend on Travis-CI-specific state or environment > variables > > > > 2. Add such jobs to Ursabot. If there is satisfaction with the > > > service > > > > provided by these builds, then the Travis CI entry can be toggled > > > off, > > > > but we should preserve the Travis CI configuration so they can be > > > > turned back on > > > > > > > > I'm not sure what to do about Windows and macOS jobs. > > > > > > > > Obvious initial candidate for this process is the lint job, to > give > > > > faster linter failures on PRs which currently can take a while > > > > > > > > Thoughts? > > > > > > > > - Wes > > > > > > > > On Mon, Jun 17, 2019 at 3:48 PM Krisztián Szűcs > > > > wrote: > > > > > > > > > > That's right, OWNER, MEMBER and CONTRIBUTOR roles are allowed: > > > > > CONTRIBUTOR > > > > > > > > > > Author has previously committed to the repository. > > > > > MEMBER > > > > > > > > > > Author is a member of the organization that owns the > repository. > > > > > OWNER > > > > > > > > > > Author is the owner of the repository. > > > > > See > https://developer.github.com/v4/enum/commentauthorassociation/ > > > > > > > > > > On Mon, Jun 17, 2019 at 3:16 PM Wes McKinney < > wesmck...@gmail.com> > > > wrote: > > > > > > > > > > > On Mon, Jun 17, 2019 at 7:25 AM Krisztián Szűcs > > > > > > wrote: > > > > > > > > > > > > > > On Sun, Jun 16, 2019 at 6:17 AM Micah Kornfield < > > > emkornfi...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Krisztian, > > > > > > > > This is really cool, thank you for doing this. Two > > > questions: > > > > > > > > 1. How reliable is the build setup? Is it reliable > enough > > > at this > > > > > > point to > > > > > > > > be considered a merge blocker if a build fails? > > > > > > > > > > > > > > > IMO yes. > > > > > > > > > > > > > > > 2. What is the permission model for triggering runs? > Is it > > > open to > > > > > > > > anybody on github? Only Ursalab members? Committers? > > > > > > > > > > > > > > > Most of the builders are automatically triggered on each > > > commits. > > > > > > > Specific control buttons are available for ursalabs member > at > > > the moment, > > > > > > > but I can grant access to other organizations (e.g. apache) > > > and > > > > > > individual > > > > > > > members. > > > > > > > > > > > > > > > > > > > You're talking about the Buildbot UI here? Suffice to say if > any > > > CI > > > > > > system is going to be depended on for decision-making, then > any > > > > > > _contributor_ needs to be able to trigger runs. It seems that > > > > > > presently any contributor can trigger builds from GitHub > > > comments, is > > > > > > that right? > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Micah > > > > > > > > > > > > > > > > On Fri, Jun 14, 2019 at 2:30 PM Antoine Pitrou < > > > anto...@python.org> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : > > > > > > >
Re: New CI system: Ursabot
Hi Eric! On Thu, Jul 11, 2019 at 3:34 PM Eric Erhardt wrote: > My apologies if this is already covered in the docs, but I couldn't find > it. > > How do I re-run a single leg in the Ursabot tests? The 'AMD64 Debian 9 > Rust 1.35' failed on my PR, and I wanted to try re-running just that leg, > but the only option I found was to re-run all Ursabot legs. > Currently you can't restart a single builder, just the whole buildset. There is a ticket about supporting @ursabot build command and another ticker to provide control access for apache members. Once the latter one is set up, I can also grant access for contributors outside of apache. BTW I think you can safely ignore the rust failure because it uses the nightly toolchain. > > Eric > > -Original Message- > From: Krisztián Szűcs > Sent: Friday, June 14, 2019 9:48 AM > To: dev@arrow.apache.org > Subject: New CI system: Ursabot > > Hello All, > > We're developing a buildbot application to utilize Ursa Labs’ > physical machines called Ursabot. Buildbot [1] is used by major open > source projects, like CPython and WebKit [2]. > > The source code is hosted at [3], the web interface is accessible at [4]. > The repository contains a short guide about the goals, implementation and > the interfaces we can drive ursabot. The most notable way to trigger > ursabot builds is via sending github comments mentioning @ursabot machine > account, for more see [5]. > > Currently we have builders for the C++ implementation and the Python > bindings on AMD64 and ARM64 architectures. > It is quite easy to attach workers to the buildmaster [7], so We can scale > our build cluster to test and run on-demand builds (like benchmarks, > packaging tasks) on more platforms. > > Yesterday we've enabled the github status push reporter to improve the > visibility of ursabot, although we were testing the builders in the last > couple of weeks. I hope no one has a hard objection against this new CI. > Arrow has already started to outgrow Travis-CI and Appveyor's capacity and > we're trying to make the build system quicker and more robust. > > Please don't hesitate to ask any questions! > > Thanks, Krisztian > > [1]: > https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbuildbot.net%2Fdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204963990536sdata=CX7t5kh2wLH%2BHYZq%2BwMG3cGIeg1ZHx%2BDHnGqlyRw81g%3Dreserved=0 > [2]: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbuildbot%2Fbuildbot%2Fwiki%2FSuccessStoriesdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204963990536sdata=vxbos9e%2BrJi7ZBIoqjUNbyj2Xmlfpj9JxsFbDc1CXrI%3Dreserved=0 > [3]: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fursa-labs%2Fursabotdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204963990536sdata=77aMN03BotaAVZM4LhI1ER4lkEqVrYb%2B848yvELq%2BEk%3Dreserved=0 > [4]: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.ursalabs.orgdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204963990536sdata=JKLOOems6daX9OQGfZwsjxuvdYXxuM9Pj3r7BR869fg%3Dreserved=0 > [5]: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fursa-labs%2Fursabot%23driving-ursabotdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204964000528sdata=x5oOrTOeedkfmvP9K9R4FYZZnR3jD1A7Q%2F5Qu8EC7M8%3Dreserved=0 > [7]: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fursa-labs%2Fursabot%2Fblob%2Fmaster%2Fdefault.yaml%23L115data=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204964000528sdata=FoqVfr4RPDmhEXxXWOK%2BUchzwm5mTv8tsN4nSrjKggQ%3Dreserved=0 >
RE: New CI system: Ursabot
My apologies if this is already covered in the docs, but I couldn't find it. How do I re-run a single leg in the Ursabot tests? The 'AMD64 Debian 9 Rust 1.35' failed on my PR, and I wanted to try re-running just that leg, but the only option I found was to re-run all Ursabot legs. Eric -Original Message- From: Krisztián Szűcs Sent: Friday, June 14, 2019 9:48 AM To: dev@arrow.apache.org Subject: New CI system: Ursabot Hello All, We're developing a buildbot application to utilize Ursa Labs’ physical machines called Ursabot. Buildbot [1] is used by major open source projects, like CPython and WebKit [2]. The source code is hosted at [3], the web interface is accessible at [4]. The repository contains a short guide about the goals, implementation and the interfaces we can drive ursabot. The most notable way to trigger ursabot builds is via sending github comments mentioning @ursabot machine account, for more see [5]. Currently we have builders for the C++ implementation and the Python bindings on AMD64 and ARM64 architectures. It is quite easy to attach workers to the buildmaster [7], so We can scale our build cluster to test and run on-demand builds (like benchmarks, packaging tasks) on more platforms. Yesterday we've enabled the github status push reporter to improve the visibility of ursabot, although we were testing the builders in the last couple of weeks. I hope no one has a hard objection against this new CI. Arrow has already started to outgrow Travis-CI and Appveyor's capacity and we're trying to make the build system quicker and more robust. Please don't hesitate to ask any questions! Thanks, Krisztian [1]: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbuildbot.net%2Fdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204963990536sdata=CX7t5kh2wLH%2BHYZq%2BwMG3cGIeg1ZHx%2BDHnGqlyRw81g%3Dreserved=0 [2]: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbuildbot%2Fbuildbot%2Fwiki%2FSuccessStoriesdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204963990536sdata=vxbos9e%2BrJi7ZBIoqjUNbyj2Xmlfpj9JxsFbDc1CXrI%3Dreserved=0 [3]: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fursa-labs%2Fursabotdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204963990536sdata=77aMN03BotaAVZM4LhI1ER4lkEqVrYb%2B848yvELq%2BEk%3Dreserved=0 [4]: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.ursalabs.orgdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204963990536sdata=JKLOOems6daX9OQGfZwsjxuvdYXxuM9Pj3r7BR869fg%3Dreserved=0 [5]: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fursa-labs%2Fursabot%23driving-ursabotdata=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204964000528sdata=x5oOrTOeedkfmvP9K9R4FYZZnR3jD1A7Q%2F5Qu8EC7M8%3Dreserved=0 [7]: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fursa-labs%2Fursabot%2Fblob%2Fmaster%2Fdefault.yaml%23L115data=02%7C01%7CEric.Erhardt%40microsoft.com%7C7df1445a86f747c3db9608d6f0d75462%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961204964000528sdata=FoqVfr4RPDmhEXxXWOK%2BUchzwm5mTv8tsN4nSrjKggQ%3Dreserved=0
Re: New CI system: Ursabot
Krisz -- can the CUDA issue be tracked somewhere so we remember to do it (and set ARROW_CUDA=on)? On Mon, Jun 24, 2019 at 4:42 AM Krisztián Szűcs wrote: > > We already have a CUDA builder in ursabot [1], just need to enable > --runtime=nvidia for the docker worker. > > [1]: > https://github.com/ursa-labs/ursabot/blob/master/ursabot/builders.py#L445 > > On Fri, Jun 21, 2019 at 9:58 PM Keith Kraus wrote: > > > There's nvidia-docker (https://github.com/NVIDIA/nvidia-docker) which > > handles passing through the GPU devices and necessary driver modules into a > > docker container. CUDA doesn't get mapped in as it's userspace so you'll > > need to either use an image with CUDA baked in (i.e. > > https://hub.docker.com/r/nvidia/cuda) or install CUDA yourself into your > > container. > > > > -Keith > > > > On 6/21/19, 2:20 PM, "Antoine Pitrou" wrote: > > > > > > Is it possible to test CUDA under a Docker container? > > > > I feel like I'm the only person who routinely tests CUDA on my home > > machine :-) And of course I only do that on Linux... > > > > Regards > > > > Antoine. > > > > > > On Fri, 21 Jun 2019 12:23:10 -0500 > > Wes McKinney wrote: > > > hi folks, > > > > > > I would suggest the following development approach to help with > > > increasing our CI capacity: > > > > > > 1. For all Linux builds, refactor Travis CI jobs to be Docker-based > > > and not depend on Travis-CI-specific state or environment variables > > > 2. Add such jobs to Ursabot. If there is satisfaction with the > > service > > > provided by these builds, then the Travis CI entry can be toggled > > off, > > > but we should preserve the Travis CI configuration so they can be > > > turned back on > > > > > > I'm not sure what to do about Windows and macOS jobs. > > > > > > Obvious initial candidate for this process is the lint job, to give > > > faster linter failures on PRs which currently can take a while > > > > > > Thoughts? > > > > > > - Wes > > > > > > On Mon, Jun 17, 2019 at 3:48 PM Krisztián Szűcs > > > wrote: > > > > > > > > That's right, OWNER, MEMBER and CONTRIBUTOR roles are allowed: > > > > CONTRIBUTOR > > > > > > > > Author has previously committed to the repository. > > > > MEMBER > > > > > > > > Author is a member of the organization that owns the repository. > > > > OWNER > > > > > > > > Author is the owner of the repository. > > > > See https://developer.github.com/v4/enum/commentauthorassociation/ > > > > > > > > On Mon, Jun 17, 2019 at 3:16 PM Wes McKinney > > wrote: > > > > > > > > > On Mon, Jun 17, 2019 at 7:25 AM Krisztián Szűcs > > > > > wrote: > > > > > > > > > > > > On Sun, Jun 16, 2019 at 6:17 AM Micah Kornfield < > > emkornfi...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi Krisztian, > > > > > > > This is really cool, thank you for doing this. Two > > questions: > > > > > > > 1. How reliable is the build setup? Is it reliable enough > > at this > > > > > point to > > > > > > > be considered a merge blocker if a build fails? > > > > > > > > > > > > > IMO yes. > > > > > > > > > > > > > 2. What is the permission model for triggering runs? Is it > > open to > > > > > > > anybody on github? Only Ursalab members? Committers? > > > > > > > > > > > > > Most of the builders are automatically triggered on each > > commits. > > > > > > Specific control buttons are available for ursalabs member at > > the moment, > > > > > > but I can grant access to other organizations (e.g. apache) > > and > > > > > individual > > > > > > members. > > > > > > > > > > > > > > > > You're talking about the Buildbot UI here? Suffice to say if any > > CI > > > > > system is going to be depended on for decision-making, then any > > > > > _contributor_ needs to be able to trigger runs. It seems that > > > > > presently any contributor can trigger builds from GitHub > > comments, is > > > > > that right? > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > Micah > > > > > > > > > > > > > > On Fri, Jun 14, 2019 at 2:30 PM Antoine Pitrou < > > anto...@python.org> > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : > > > > > > > > >> > > > > > > > > >> * Do machines have to be co-located on the same > > physical network > > > > > as > > > > > > > > >> the master, or can they reside in other locations? > > > > > > > > >> > > > > > > > > > It is preferable to have a master in the same network > > where the > > > > > workers > > > > > > > > are, > > > > > > > > > because the build steps are rpc calls made by the > > master. > > > > > > > > > > > > > > > > I'm unaware that this is
Re: New CI system: Ursabot
We already have a CUDA builder in ursabot [1], just need to enable --runtime=nvidia for the docker worker. [1]: https://github.com/ursa-labs/ursabot/blob/master/ursabot/builders.py#L445 On Fri, Jun 21, 2019 at 9:58 PM Keith Kraus wrote: > There's nvidia-docker (https://github.com/NVIDIA/nvidia-docker) which > handles passing through the GPU devices and necessary driver modules into a > docker container. CUDA doesn't get mapped in as it's userspace so you'll > need to either use an image with CUDA baked in (i.e. > https://hub.docker.com/r/nvidia/cuda) or install CUDA yourself into your > container. > > -Keith > > On 6/21/19, 2:20 PM, "Antoine Pitrou" wrote: > > > Is it possible to test CUDA under a Docker container? > > I feel like I'm the only person who routinely tests CUDA on my home > machine :-) And of course I only do that on Linux... > > Regards > > Antoine. > > > On Fri, 21 Jun 2019 12:23:10 -0500 > Wes McKinney wrote: > > hi folks, > > > > I would suggest the following development approach to help with > > increasing our CI capacity: > > > > 1. For all Linux builds, refactor Travis CI jobs to be Docker-based > > and not depend on Travis-CI-specific state or environment variables > > 2. Add such jobs to Ursabot. If there is satisfaction with the > service > > provided by these builds, then the Travis CI entry can be toggled > off, > > but we should preserve the Travis CI configuration so they can be > > turned back on > > > > I'm not sure what to do about Windows and macOS jobs. > > > > Obvious initial candidate for this process is the lint job, to give > > faster linter failures on PRs which currently can take a while > > > > Thoughts? > > > > - Wes > > > > On Mon, Jun 17, 2019 at 3:48 PM Krisztián Szűcs > > wrote: > > > > > > That's right, OWNER, MEMBER and CONTRIBUTOR roles are allowed: > > > CONTRIBUTOR > > > > > > Author has previously committed to the repository. > > > MEMBER > > > > > > Author is a member of the organization that owns the repository. > > > OWNER > > > > > > Author is the owner of the repository. > > > See https://developer.github.com/v4/enum/commentauthorassociation/ > > > > > > On Mon, Jun 17, 2019 at 3:16 PM Wes McKinney > wrote: > > > > > > > On Mon, Jun 17, 2019 at 7:25 AM Krisztián Szűcs > > > > wrote: > > > > > > > > > > On Sun, Jun 16, 2019 at 6:17 AM Micah Kornfield < > emkornfi...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Krisztian, > > > > > > This is really cool, thank you for doing this. Two > questions: > > > > > > 1. How reliable is the build setup? Is it reliable enough > at this > > > > point to > > > > > > be considered a merge blocker if a build fails? > > > > > > > > > > > IMO yes. > > > > > > > > > > > 2. What is the permission model for triggering runs? Is it > open to > > > > > > anybody on github? Only Ursalab members? Committers? > > > > > > > > > > > Most of the builders are automatically triggered on each > commits. > > > > > Specific control buttons are available for ursalabs member at > the moment, > > > > > but I can grant access to other organizations (e.g. apache) > and > > > > individual > > > > > members. > > > > > > > > > > > > > You're talking about the Buildbot UI here? Suffice to say if any > CI > > > > system is going to be depended on for decision-making, then any > > > > _contributor_ needs to be able to trigger runs. It seems that > > > > presently any contributor can trigger builds from GitHub > comments, is > > > > that right? > > > > > > > > > > > > > > > > Thanks, > > > > > > Micah > > > > > > > > > > > > On Fri, Jun 14, 2019 at 2:30 PM Antoine Pitrou < > anto...@python.org> > > > > wrote: > > > > > > > > > > > > > > > > > > > > Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : > > > > > > > >> > > > > > > > >> * Do machines have to be co-located on the same > physical network > > > > as > > > > > > > >> the master, or can they reside in other locations? > > > > > > > >> > > > > > > > > It is preferable to have a master in the same network > where the > > > > workers > > > > > > > are, > > > > > > > > because the build steps are rpc calls made by the > master. > > > > > > > > > > > > > > I'm unaware that this is a problem. > > > > > > > CPython has build workers all over the world (contributed > by > > > > volunteers) > > > > > > > connected to a single build master. > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > > > > > > > > > > --- > This email
Re: New CI system: Ursabot
Yes please. Locally reproducible failed builds/tests should be a top priority. I would also propose that we cache docker images on a nightly basis and export them via quay or other channel. I spent the last days looking at docker build with wall of apt and conda lines. François On Fri, Jun 21, 2019 at 1:23 PM Wes McKinney wrote: > > hi folks, > > I would suggest the following development approach to help with > increasing our CI capacity: > > 1. For all Linux builds, refactor Travis CI jobs to be Docker-based > and not depend on Travis-CI-specific state or environment variables > 2. Add such jobs to Ursabot. If there is satisfaction with the service > provided by these builds, then the Travis CI entry can be toggled off, > but we should preserve the Travis CI configuration so they can be > turned back on > > I'm not sure what to do about Windows and macOS jobs. > > Obvious initial candidate for this process is the lint job, to give > faster linter failures on PRs which currently can take a while > > Thoughts? > > - Wes > > On Mon, Jun 17, 2019 at 3:48 PM Krisztián Szűcs > wrote: > > > > That's right, OWNER, MEMBER and CONTRIBUTOR roles are allowed: > > CONTRIBUTOR > > > > Author has previously committed to the repository. > > MEMBER > > > > Author is a member of the organization that owns the repository. > > OWNER > > > > Author is the owner of the repository. > > See https://developer.github.com/v4/enum/commentauthorassociation/ > > > > On Mon, Jun 17, 2019 at 3:16 PM Wes McKinney wrote: > > > > > On Mon, Jun 17, 2019 at 7:25 AM Krisztián Szűcs > > > wrote: > > > > > > > > On Sun, Jun 16, 2019 at 6:17 AM Micah Kornfield > > > > wrote: > > > > > > > > > Hi Krisztian, > > > > > This is really cool, thank you for doing this. Two questions: > > > > > 1. How reliable is the build setup? Is it reliable enough at this > > > point to > > > > > be considered a merge blocker if a build fails? > > > > > > > > > IMO yes. > > > > > > > > > 2. What is the permission model for triggering runs? Is it open to > > > > > anybody on github? Only Ursalab members? Committers? > > > > > > > > > Most of the builders are automatically triggered on each commits. > > > > Specific control buttons are available for ursalabs member at the > > > > moment, > > > > but I can grant access to other organizations (e.g. apache) and > > > individual > > > > members. > > > > > > > > > > You're talking about the Buildbot UI here? Suffice to say if any CI > > > system is going to be depended on for decision-making, then any > > > _contributor_ needs to be able to trigger runs. It seems that > > > presently any contributor can trigger builds from GitHub comments, is > > > that right? > > > > > > > > > > > > > Thanks, > > > > > Micah > > > > > > > > > > On Fri, Jun 14, 2019 at 2:30 PM Antoine Pitrou > > > wrote: > > > > > > > > > > > > > > > > > Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : > > > > > > >> > > > > > > >> * Do machines have to be co-located on the same physical network > > > as > > > > > > >> the master, or can they reside in other locations? > > > > > > >> > > > > > > > It is preferable to have a master in the same network where the > > > workers > > > > > > are, > > > > > > > because the build steps are rpc calls made by the master. > > > > > > > > > > > > I'm unaware that this is a problem. > > > > > > CPython has build workers all over the world (contributed by > > > volunteers) > > > > > > connected to a single build master. > > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > >
Re: New CI system: Ursabot
hi folks, I would suggest the following development approach to help with increasing our CI capacity: 1. For all Linux builds, refactor Travis CI jobs to be Docker-based and not depend on Travis-CI-specific state or environment variables 2. Add such jobs to Ursabot. If there is satisfaction with the service provided by these builds, then the Travis CI entry can be toggled off, but we should preserve the Travis CI configuration so they can be turned back on I'm not sure what to do about Windows and macOS jobs. Obvious initial candidate for this process is the lint job, to give faster linter failures on PRs which currently can take a while Thoughts? - Wes On Mon, Jun 17, 2019 at 3:48 PM Krisztián Szűcs wrote: > > That's right, OWNER, MEMBER and CONTRIBUTOR roles are allowed: > CONTRIBUTOR > > Author has previously committed to the repository. > MEMBER > > Author is a member of the organization that owns the repository. > OWNER > > Author is the owner of the repository. > See https://developer.github.com/v4/enum/commentauthorassociation/ > > On Mon, Jun 17, 2019 at 3:16 PM Wes McKinney wrote: > > > On Mon, Jun 17, 2019 at 7:25 AM Krisztián Szűcs > > wrote: > > > > > > On Sun, Jun 16, 2019 at 6:17 AM Micah Kornfield > > > wrote: > > > > > > > Hi Krisztian, > > > > This is really cool, thank you for doing this. Two questions: > > > > 1. How reliable is the build setup? Is it reliable enough at this > > point to > > > > be considered a merge blocker if a build fails? > > > > > > > IMO yes. > > > > > > > 2. What is the permission model for triggering runs? Is it open to > > > > anybody on github? Only Ursalab members? Committers? > > > > > > > Most of the builders are automatically triggered on each commits. > > > Specific control buttons are available for ursalabs member at the moment, > > > but I can grant access to other organizations (e.g. apache) and > > individual > > > members. > > > > > > > You're talking about the Buildbot UI here? Suffice to say if any CI > > system is going to be depended on for decision-making, then any > > _contributor_ needs to be able to trigger runs. It seems that > > presently any contributor can trigger builds from GitHub comments, is > > that right? > > > > > > > > > > Thanks, > > > > Micah > > > > > > > > On Fri, Jun 14, 2019 at 2:30 PM Antoine Pitrou > > wrote: > > > > > > > > > > > > > > Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : > > > > > >> > > > > > >> * Do machines have to be co-located on the same physical network > > as > > > > > >> the master, or can they reside in other locations? > > > > > >> > > > > > > It is preferable to have a master in the same network where the > > workers > > > > > are, > > > > > > because the build steps are rpc calls made by the master. > > > > > > > > > > I'm unaware that this is a problem. > > > > > CPython has build workers all over the world (contributed by > > volunteers) > > > > > connected to a single build master. > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > > > > > > >
Re: New CI system: Ursabot
That's right, OWNER, MEMBER and CONTRIBUTOR roles are allowed: CONTRIBUTOR Author has previously committed to the repository. MEMBER Author is a member of the organization that owns the repository. OWNER Author is the owner of the repository. See https://developer.github.com/v4/enum/commentauthorassociation/ On Mon, Jun 17, 2019 at 3:16 PM Wes McKinney wrote: > On Mon, Jun 17, 2019 at 7:25 AM Krisztián Szűcs > wrote: > > > > On Sun, Jun 16, 2019 at 6:17 AM Micah Kornfield > > wrote: > > > > > Hi Krisztian, > > > This is really cool, thank you for doing this. Two questions: > > > 1. How reliable is the build setup? Is it reliable enough at this > point to > > > be considered a merge blocker if a build fails? > > > > > IMO yes. > > > > > 2. What is the permission model for triggering runs? Is it open to > > > anybody on github? Only Ursalab members? Committers? > > > > > Most of the builders are automatically triggered on each commits. > > Specific control buttons are available for ursalabs member at the moment, > > but I can grant access to other organizations (e.g. apache) and > individual > > members. > > > > You're talking about the Buildbot UI here? Suffice to say if any CI > system is going to be depended on for decision-making, then any > _contributor_ needs to be able to trigger runs. It seems that > presently any contributor can trigger builds from GitHub comments, is > that right? > > > > > > > Thanks, > > > Micah > > > > > > On Fri, Jun 14, 2019 at 2:30 PM Antoine Pitrou > wrote: > > > > > > > > > > > Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : > > > > >> > > > > >> * Do machines have to be co-located on the same physical network > as > > > > >> the master, or can they reside in other locations? > > > > >> > > > > > It is preferable to have a master in the same network where the > workers > > > > are, > > > > > because the build steps are rpc calls made by the master. > > > > > > > > I'm unaware that this is a problem. > > > > CPython has build workers all over the world (contributed by > volunteers) > > > > connected to a single build master. > > > > > > > > Regards > > > > > > > > Antoine. > > > > > > > >
Re: New CI system: Ursabot
On Mon, Jun 17, 2019 at 7:25 AM Krisztián Szűcs wrote: > > On Sun, Jun 16, 2019 at 6:17 AM Micah Kornfield > wrote: > > > Hi Krisztian, > > This is really cool, thank you for doing this. Two questions: > > 1. How reliable is the build setup? Is it reliable enough at this point to > > be considered a merge blocker if a build fails? > > > IMO yes. > > > 2. What is the permission model for triggering runs? Is it open to > > anybody on github? Only Ursalab members? Committers? > > > Most of the builders are automatically triggered on each commits. > Specific control buttons are available for ursalabs member at the moment, > but I can grant access to other organizations (e.g. apache) and individual > members. > You're talking about the Buildbot UI here? Suffice to say if any CI system is going to be depended on for decision-making, then any _contributor_ needs to be able to trigger runs. It seems that presently any contributor can trigger builds from GitHub comments, is that right? > > > > Thanks, > > Micah > > > > On Fri, Jun 14, 2019 at 2:30 PM Antoine Pitrou wrote: > > > > > > > > Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : > > > >> > > > >> * Do machines have to be co-located on the same physical network as > > > >> the master, or can they reside in other locations? > > > >> > > > > It is preferable to have a master in the same network where the workers > > > are, > > > > because the build steps are rpc calls made by the master. > > > > > > I'm unaware that this is a problem. > > > CPython has build workers all over the world (contributed by volunteers) > > > connected to a single build master. > > > > > > Regards > > > > > > Antoine. > > > > >
Re: New CI system: Ursabot
On Fri, Jun 14, 2019, at 11:23 PM, Krisztián Szűcs wrote: > On Fri, Jun 14, 2019 at 9:04 PM Wes McKinney wrote: > > > hi Krisz, > > > > Thanks for working on this! It already helped me fix a Python 2.7-only > > bug yesterday https://github.com/apache/arrow/pull/4553 > > > > I have a bunch of questions: > > > > * What is the license of the ursabot codebase? Seems like it could be > > GPL if Buildbot itself is [2] and you have reused any Buildbot code. > > This is not mentioned in the README so I think you need to call this > > out and advise people to be careful that any code contributed to this > > codebase is stuck in GPL-land. Is it possible to separate the work > > you've done from any GPL-ness? I don't think you can be too paranoid > > about this kind of thing, and the longer you wait to draw a clear line > > around any contaminated code the harder it will be to disentangle. > > > Created an issue for it https://github.com/ursa-labs/ursabot/issues/105 > > > > > * How brittle is the Buildbot master? It currently resides in my home, > > but what if a natural disaster (like [1] from 2010) occurs in > > Nashville causing an extended power outage (or a temporary outage > > requiring human intervention while I'm out of town)? Is the Buildbot > > master state backed up? Can it be easily migrated to a new host > > (Dockerized, even)? Either way we need a contingency plan. > > > It doesn't have a backup yet, although it only matters for historical > reasons. > If I prune its database and restart the buildmaster nothing changes except > the losing the previous builds' result - so everything will work the same > way > as before erasing the database. > It can be pretty easily migrated to another host, besides setting up the > networking with the workers and a database server (or sqlite) nothing more > esoteric is required. > > > > > * The availability of Buildbot suggests we should try to decouple our > > CI procedures from anything Travis CI specific and use Docker instead, > > at least for the Linux builds. This has the side benefit of enabling > > contributors to reproduce CI build locally. Can you create some JIRAs > > about this? > > > Yes, IMO docker is preferable, even for Windows containers [1]. Will do, but > I'm curious for other's opinion too. @Uwe? Yes, also for Windows builds docker is really nice for getting a clean reproducible state. Only for OSX builds, you will need to boot VMs these days when you want clean separation. Uwe
Re: New CI system: Ursabot
Hi Krisztian, This is really cool, thank you for doing this. Two questions: 1. How reliable is the build setup? Is it reliable enough at this point to be considered a merge blocker if a build fails? 2. What is the permission model for triggering runs? Is it open to anybody on github? Only Ursalab members? Committers? Thanks, Micah On Fri, Jun 14, 2019 at 2:30 PM Antoine Pitrou wrote: > > Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : > >> > >> * Do machines have to be co-located on the same physical network as > >> the master, or can they reside in other locations? > >> > > It is preferable to have a master in the same network where the workers > are, > > because the build steps are rpc calls made by the master. > > I'm unaware that this is a problem. > CPython has build workers all over the world (contributed by volunteers) > connected to a single build master. > > Regards > > Antoine. >
Re: New CI system: Ursabot
Le 14/06/2019 à 23:22, Krisztián Szűcs a écrit : >> >> * Do machines have to be co-located on the same physical network as >> the master, or can they reside in other locations? >> > It is preferable to have a master in the same network where the workers are, > because the build steps are rpc calls made by the master. I'm unaware that this is a problem. CPython has build workers all over the world (contributed by volunteers) connected to a single build master. Regards Antoine.
Re: New CI system: Ursabot
On Fri, Jun 14, 2019 at 9:04 PM Wes McKinney wrote: > hi Krisz, > > Thanks for working on this! It already helped me fix a Python 2.7-only > bug yesterday https://github.com/apache/arrow/pull/4553 > > I have a bunch of questions: > > * What is the license of the ursabot codebase? Seems like it could be > GPL if Buildbot itself is [2] and you have reused any Buildbot code. > This is not mentioned in the README so I think you need to call this > out and advise people to be careful that any code contributed to this > codebase is stuck in GPL-land. Is it possible to separate the work > you've done from any GPL-ness? I don't think you can be too paranoid > about this kind of thing, and the longer you wait to draw a clear line > around any contaminated code the harder it will be to disentangle. > Created an issue for it https://github.com/ursa-labs/ursabot/issues/105 > > * How brittle is the Buildbot master? It currently resides in my home, > but what if a natural disaster (like [1] from 2010) occurs in > Nashville causing an extended power outage (or a temporary outage > requiring human intervention while I'm out of town)? Is the Buildbot > master state backed up? Can it be easily migrated to a new host > (Dockerized, even)? Either way we need a contingency plan. > It doesn't have a backup yet, although it only matters for historical reasons. If I prune its database and restart the buildmaster nothing changes except the losing the previous builds' result - so everything will work the same way as before erasing the database. It can be pretty easily migrated to another host, besides setting up the networking with the workers and a database server (or sqlite) nothing more esoteric is required. > > * The availability of Buildbot suggests we should try to decouple our > CI procedures from anything Travis CI specific and use Docker instead, > at least for the Linux builds. This has the side benefit of enabling > contributors to reproduce CI build locally. Can you create some JIRAs > about this? > Yes, IMO docker is preferable, even for Windows containers [1]. Will do, but I'm curious for other's opinion too. @Uwe? > > * Currently there are only Linux-based x86 and Aarch64 builders. Given > the periodic bandwidth issues with Windows on Appveyor, getting more > builds off of Appveyor would help us all. If I wanted to set up a new > machine (including a Windows-based one), how do I do that? > We don't have windows builders yet, so it'll require some (not much) development. Theoretically we can either use windows workers running within windows docker containers or directly on the windows host. After having a windows worker connected to the buildmaster We need to setup builders which are able to run on windows (this is mostly about converting the build steps to windows specific ones [2], although it might work out of the box - haven't tested it yet so cannot say much more about it). > > * Do machines have to be co-located on the same physical network as > the master, or can they reside in other locations? > It is preferable to have a master in the same network where the workers are, because the build steps are rpc calls made by the master. In order to scale our cluster we can have a multi-master setup [3], so spining up workers at different locations is possible. For example we could use machines from public cloud providers to offload the current travis/appveyor builds (with better availability then your/Wes' network). [1]: https://www.docker.com/products/windows-containers [2]: https://github.com/ursa-labs/ursabot/blob/master/ursabot/builders.py#L420 [3]: http://docs.buildbot.net/2.3.1/full.html#multi-master-mode > Thanks, > Wes > > [1]: https://en.wikipedia.org/wiki/2010_Tennessee_floods > [2]: https://github.com/buildbot/buildbot > > On Fri, Jun 14, 2019 at 9:48 AM Krisztián Szűcs > wrote: > > > > Hello All, > > > > We're developing a buildbot application to utilize Ursa Labs’ > > physical machines called Ursabot. Buildbot [1] is used by > > major open source projects, like CPython and WebKit [2]. > > > > The source code is hosted at [3], the web interface is > > accessible at [4]. The repository contains a short guide > > about the goals, implementation and the interfaces we can > > drive ursabot. The most notable way to trigger ursabot builds > > is via sending github comments mentioning @ursabot machine > > account, for more see [5]. > > > > Currently we have builders for the C++ implementation and > > the Python bindings on AMD64 and ARM64 architectures. > > It is quite easy to attach workers to the buildmaster [7], so > > We can scale our build cluster to test and run on-demand builds > > (like benchmarks, packaging tasks) on more platforms. > > > > Yesterday we've enabled the github status push reporter > > to improve the visibility of ursabot, although we were testing > > the builders in the last couple of weeks. I hope no one has > > a hard objection against this new CI. Arrow has already started > >
Re: New CI system: Ursabot
hi Krisz, Thanks for working on this! It already helped me fix a Python 2.7-only bug yesterday https://github.com/apache/arrow/pull/4553 I have a bunch of questions: * What is the license of the ursabot codebase? Seems like it could be GPL if Buildbot itself is [2] and you have reused any Buildbot code. This is not mentioned in the README so I think you need to call this out and advise people to be careful that any code contributed to this codebase is stuck in GPL-land. Is it possible to separate the work you've done from any GPL-ness? I don't think you can be too paranoid about this kind of thing, and the longer you wait to draw a clear line around any contaminated code the harder it will be to disentangle. * How brittle is the Buildbot master? It currently resides in my home, but what if a natural disaster (like [1] from 2010) occurs in Nashville causing an extended power outage (or a temporary outage requiring human intervention while I'm out of town)? Is the Buildbot master state backed up? Can it be easily migrated to a new host (Dockerized, even)? Either way we need a contingency plan. * The availability of Buildbot suggests we should try to decouple our CI procedures from anything Travis CI specific and use Docker instead, at least for the Linux builds. This has the side benefit of enabling contributors to reproduce CI build locally. Can you create some JIRAs about this? * Currently there are only Linux-based x86 and Aarch64 builders. Given the periodic bandwidth issues with Windows on Appveyor, getting more builds off of Appveyor would help us all. If I wanted to set up a new machine (including a Windows-based one), how do I do that? * Do machines have to be co-located on the same physical network as the master, or can they reside in other locations? Thanks, Wes [1]: https://en.wikipedia.org/wiki/2010_Tennessee_floods [2]: https://github.com/buildbot/buildbot On Fri, Jun 14, 2019 at 9:48 AM Krisztián Szűcs wrote: > > Hello All, > > We're developing a buildbot application to utilize Ursa Labs’ > physical machines called Ursabot. Buildbot [1] is used by > major open source projects, like CPython and WebKit [2]. > > The source code is hosted at [3], the web interface is > accessible at [4]. The repository contains a short guide > about the goals, implementation and the interfaces we can > drive ursabot. The most notable way to trigger ursabot builds > is via sending github comments mentioning @ursabot machine > account, for more see [5]. > > Currently we have builders for the C++ implementation and > the Python bindings on AMD64 and ARM64 architectures. > It is quite easy to attach workers to the buildmaster [7], so > We can scale our build cluster to test and run on-demand builds > (like benchmarks, packaging tasks) on more platforms. > > Yesterday we've enabled the github status push reporter > to improve the visibility of ursabot, although we were testing > the builders in the last couple of weeks. I hope no one has > a hard objection against this new CI. Arrow has already started > to outgrow Travis-CI and Appveyor's capacity and we're trying > to make the build system quicker and more robust. > > Please don't hesitate to ask any questions! > > Thanks, Krisztian > > [1]: http://buildbot.net/ > [2]: https://github.com/buildbot/buildbot/wiki/SuccessStories > [3]: https://github.com/ursa-labs/ursabot > [4]: https://ci.ursalabs.org > [5]: https://github.com/ursa-labs/ursabot#driving-ursabot > [7]: https://github.com/ursa-labs/ursabot/blob/master/default.yaml#L115
New CI system: Ursabot
Hello All, We're developing a buildbot application to utilize Ursa Labs’ physical machines called Ursabot. Buildbot [1] is used by major open source projects, like CPython and WebKit [2]. The source code is hosted at [3], the web interface is accessible at [4]. The repository contains a short guide about the goals, implementation and the interfaces we can drive ursabot. The most notable way to trigger ursabot builds is via sending github comments mentioning @ursabot machine account, for more see [5]. Currently we have builders for the C++ implementation and the Python bindings on AMD64 and ARM64 architectures. It is quite easy to attach workers to the buildmaster [7], so We can scale our build cluster to test and run on-demand builds (like benchmarks, packaging tasks) on more platforms. Yesterday we've enabled the github status push reporter to improve the visibility of ursabot, although we were testing the builders in the last couple of weeks. I hope no one has a hard objection against this new CI. Arrow has already started to outgrow Travis-CI and Appveyor's capacity and we're trying to make the build system quicker and more robust. Please don't hesitate to ask any questions! Thanks, Krisztian [1]: http://buildbot.net/ [2]: https://github.com/buildbot/buildbot/wiki/SuccessStories [3]: https://github.com/ursa-labs/ursabot [4]: https://ci.ursalabs.org [5]: https://github.com/ursa-labs/ursabot#driving-ursabot [7]: https://github.com/ursa-labs/ursabot/blob/master/default.yaml#L115
