> > Hi Krisz -- we can't have primary CI configurations in a third party > repository. > I have a solution in mind, most of the arrow related parts are clearly separated, just need to organize to codebase accordingly. (there are a couple of shared services like the github hook handling which requires special treatment). I can start to work on it from next week.
> > We need to figure out how to make 100% of builds runnable from the > apache/arrow repository. > > On Wed, Jul 17, 2019, 12:29 PM Krisztián Szűcs <szucs.kriszt...@gmail.com> > wrote: > > > Besides the builders listed at https://ci.ursalabs.org/#/builders > > I've also migrated JS and R is in the queue > > https://github.com/ursa-labs/ursabot/pull/135 > > > > On Wed, Jul 17, 2019 at 6:23 PM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > Indeed, Docker runs on macOS and Windows > > > > > > In our Travis CI build matrix we have > > > > > > * 11 Linux builds > > > * 3 macOS builds > > > > > > If we migrated just the Linux builds it might make the Travis wait > > > times more acceptable in the short term > > > > > > On Wed, Jul 17, 2019 at 11:16 AM Uwe L. Korn <uw...@xhochy.com> wrote: > > > > > > > > Docker works well for all people on all OSes. > > > > > > > > Interesting will be Windows, OSX or aarch64 builds which require a > > > special system. > > > > > > > > Uwe > > > > > > > > On Wed, Jul 17, 2019, at 6:11 PM, Antoine Pitrou wrote: > > > > > > > > > > I'm not sure how Docker will work for people not on Linux though? > > > > > (and/or for macOS builds) > > > > > > > > > > Regards > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > On Wed, 17 Jul 2019 10:54:13 -0500 > > > > > Wes McKinney <wesmck...@gmail.com> wrote: > > > > > > > > > > > Presumably a migration away from Travis also means that we have > to > > > > > > develop tools to allow contributors to test their patches outside > > of > > > > > > the GitHub pull request. If something is Docker-based, then it > can > > be > > > > > > run locally, of course. > > > > > > > > > > > > We definitely can't persist under the current circumstances where > > > > > > builds take hours to even begin. Here's an example of a PR that > was > > > > > > approved 3 hours ago but whose Travis builds only started about > 10 > > > > > > minutes ago (and will have to run for at least another 30-60 > > minutes) > > > > > > > > > > > > https://github.com/apache/arrow/pull/4894 > > > > > > > > > > > > I think we need to get to an SLA where we're getting feedback on > > PRs > > > > > > in 90 minutes or less. > > > > > > > > > > > > On Wed, Jul 17, 2019 at 10:47 AM Neal Richardson > > > > > > <neal.p.richard...@gmail.com> wrote: > > > > > > > > > > > > > > Won't moving CI away from Travis to our own infrastructure mean > > > that we > > > > > > > won't get any CI on our personal forks? > > > > > > > > > > > > > > On Wed, Jul 17, 2019 at 8:23 AM Wes McKinney < > > wesmck...@gmail.com> > > > wrote: > > > > > > > > > > > > > > > On Wed, Jul 17, 2019 at 10:22 AM Wes McKinney < > > > wesmck...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > hi folks -- I noticed this last night on > > > > > > > > > https://github.com/apache/arrow/pull/4841 and it surprised > > > me. Others > > > > > > > > > may not be aware. > > > > > > > > > > > > > > > > > > We have been using builds on Appveyor and Travis CI to > decide > > > whether > > > > > > > > > to merge PRs. The trouble is these builds are not > equivalent > > > to the > > > > > > > > > builds that Travis runs inside the PR (using the > apache/arrow > > > build > > > > > > > > > queue). The differences are: > > > > > > > > > > > > > > > > > > > > > > > > > *missing crucial detail: "builds on personal forks" > > > > > > > > > > > > > > > > > * They do not take into account changes in master (IOW to > > test > > > if the > > > > > > > > > build works after `git merge`) > > > > > > > > > * They only test the latest commit versus the previous one > in > > > the branch > > > > > > > > > > > > > > > > > > This latter item is insidious, because of the > > > `detect-changes.py` > > > > > > > > > script. Suppose that you have a large PR that touches many > > > components, > > > > > > > > > and you push a commit that only affects one of them. Then > the > > > > > > > > > detect-changes.py script will cause Travis to only run > builds > > > for the > > > > > > > > > affected component in the most recent commit. > > > > > > > > > > > > > > > > > > Here's an example of such a spurious build > > > > > > > > > > > > > > > > > > https://travis-ci.org/wesm/arrow/builds/559745190 > > > > > > > > > > > > > > > > > > There are a few ways we can mitigate this last issue: > > > > > > > > > > > > > > > > > > * If you need a faster build, you can squash your commits > and > > > rebase > > > > > > > > > on master before pushing to make sure that Travis "sees" > > > everything. > > > > > > > > > Note this still carries risk of conflicting changes in > master > > > causing > > > > > > > > > a broken build post-merge > > > > > > > > > > > > > > > > > > * We can change the Travis configuration to try to detect > > > whether or > > > > > > > > > not we are testing a PR -- the detect-changes.py logic is > > > really only > > > > > > > > > intended to speed up builds in apache/arrow > > > > > > > > > > > > > > > > > > Overall, I think we need to accelerate our exodus from > Travis > > > CI since > > > > > > > > > it's hurting the project's productivity to be waiting so > long > > > on > > > > > > > > > builds. We've moved a couple of jobs to be Docker-based but > > we > > > have > > > > > > > > > quite a lot more work to do to decouple ourselves. > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > Wes > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jul 17, 2019 at 7:32 PM Wes McKinney <wesmck...@gmail.com> wrote:
