On Sun, Jun 30, 2019 at 12:03 AM Sutou Kouhei <k...@clear-code.com> wrote:
> Hi, > > How about creating a mirror repository on > https://gitlab.com/ only to run CI jobs? > > This is an idea that is described in > https://issues.apache.org/jira/browse/ARROW-5673 . > I do agree that We should investigate the features provided by GitLab. Buildbot might not be so familiar to others, so I'm trying to provide some details to see how it compares to GitLab CI. > > GitLab CI can attach external workers. So we can increase CI > capacity by adding our new workers. GitLab also provides > Docker registry. It means that we can cache built Docker > images for our CI. It will reduce CI time. > Currently all of the workers are running within docker daemons, so all of the images are cached once they are pulled to a docker daemon. The `worker_preparation` takes only 3 seconds: https://ci.ursalabs.org/#/builders/66/builds/2157 > > The feature to create a mirror repository for CI isn't > included in the Free tier on https://gitlab.com/ . But > https://gitlab.com/ provides the Gold tier features to open > source project: > https://about.gitlab.com/solutions/github/#open-source-projects > So we can use this feature. > > > Here are advantages I think to use GitLab CI: > > * We can increase CI capacity by adding our new workers. > * GitLab Runner (CI job runner) can work on GNU/Linux, macOS > and Windows: https://docs.gitlab.com/runner/#requirements > It means that we can increase CI capacity of all of them. > The same is true for buildbot, the workers are basically python twisted applications, so we can host them on any platform. > > * We can reduce CI time by caching built Docker images. > * It will reduce package build job time especially. > As I mentioned above the same is true for buildbot, but the package build scripts needs to be ported to either crossbow or ursabot or the new gitlab CI. Mentioning crossbow, I've recently added support for azure pipelines and circleci. For the docker tests (which are running the docker-compose commands) we could add a gitlab specific template yml to test gitlab's docker capabilities, see the current templates at https://github.com/apache/arrow/tree/master/dev/tasks/docker-tests > > * We can run CUDA related tests in CI by adding CUDA > enabled workers. > Here is a PR which passes `--runtime=nvidia` option to the docker run command, thus making CUDA enabled tests possible on buildbot's docker workers: https://github.com/ursa-labs/ursabot/pull/118 > > * We can manage CI jobs in https://github.com/apache/arrow > repository. > * GitLab CI uses .gitlab-ci.yml like .travis.yml for > Travis CI. > We can also maintain our buildbot configuration in apache/arrow similarly, but with a more flexible python based DSL. > > > If we create a mirror repository for CI on > https://gitlab.com/ , https://gitlab.com/ursa-labs/arrow > will be a good URL. > > > Thanks, > -- > kou > > In <CAJPUwMBpxzCLZEM2=wwb+necdwz0z0kjd5egss3uqoiirk-...@mail.gmail.com> > "Re: [DISCUSS] Ongoing Travis CI service degradation" on Sat, 29 Jun > 2019 14:54:19 -0500, > Wes McKinney <wesmck...@gmail.com> wrote: > > > hi Rok, > > > > I would guess that GitHub Actions will have the same resource and > > hardware limitations that Travis CI and Appveyor currently have, as > > well as organization-level resource contention with the rest of the > > ASF. > > > > We need to have dedicated, powerful hardware (more cores, more RAM), > > with more capabilities (architectures other than x86, and with GPUs), > > that can run jobs longer than 50 minutes, with the ability to scale up > > as the project grows in # of contributions per month. In the past > > month Arrow had 4300 hours of builds on Travis CI. What will happen > > when we need 10,000 or more hours per month to verify all of our > > patches? At the current rapid rate of project growth it is only a > > matter of time. > > > > I made a graph of commits to master by month: > > > > https://imgur.com/a/02TtGXx > > > > With nearly ~300 commits in the month of June alone, it begs the > > question how to support 500 commits per month, or 1000. > > > > - Wes > > > > > > > > On Sat, Jun 29, 2019 at 5:19 AM Rok Mihevc <rok.mih...@gmail.com> wrote: > >> > >> GitHub Actions are currently in limited public beta and appear to be > >> similar to GitLab CI: https://github.com/features/actions > >> More here: https://help.github.com/en/articles/about-github-actions > >> > >> Rok > >> > >> On Fri, Jun 28, 2019 at 7:06 PM Wes McKinney <wesmck...@gmail.com> > wrote: > >> > >> > Based on the discussion in > >> > https://issues.apache.org/jira/browse/INFRA-18533 it does not appear > >> > to be ASF Infra's inclination to allow projects to donate money to the > >> > Foundation to get more build resources on Travis CI. Our likely only > >> > solution is going to be to reduce our dependence on Travis CI. In the > >> > short term, I would say that the sooner we can migrate all of our > >> > Linux builds to docker-compose form to aid in this transition, the > >> > better > >> > > >> > We are hiring in our organization (Ursa Labs) for a dedicated role to > >> > support CI and development lifecycle automation (packaging, > >> > benchmarking, releases, etc.) in the Apache Arrow project, so I hope > >> > that we can provide even more help to resolve these issues in the > >> > future than we already are > >> > > >> > On Wed, Jun 26, 2019 at 11:35 AM Antoine Pitrou <anto...@python.org> > >> > wrote: > >> > > > >> > > > >> > > Also note that the situation with AppVeyor isn't much better. > >> > > > >> > > Any "free as in beer" CI service is probably too capacity-limited > for > >> > > our needs now, unless it allows private workers (which apparently > Gitlab > >> > > CI does). > >> > > > >> > > Regards > >> > > > >> > > Antoine. > >> > > > >> > > > >> > > Le 26/06/2019 à 18:32, Wes McKinney a écrit : > >> > > > It seems that there is intermittent Apache-wide degradation of > Travis > >> > > > CI services -- I was looking at https://travis-ci.org/apache > today and > >> > > > there appeared to be a stretch of 3-4 hours where no queued > builds on > >> > > > github.com/apache were running at all. I initially thought that > the > >> > > > issue was contention with other Apache projects but even with > >> > > > round-robin allocation and a concurrency limit (e.g. no Apache > project > >> > > > having more than 5-6 concurrent builds) that wouldn't explain why > NO > >> > > > builds are running. > >> > > > > >> > > > This is obviously disturbing given how reliant we are on Travis > CI to > >> > > > validate patches to be merged. > >> > > > > >> > > > I've opened a support ticket with Travis CI to see if they can > provide > >> > > > some insight into what's going on. There is also an INFRA ticket > where > >> > > > other projects have reported some similar experiences > >> > > > > >> > > > https://issues.apache.org/jira/browse/INFRA-18533 > >> > > > > >> > > > As a meta-comment, at some point Apache Arrow is going to need to > move > >> > > > off of public CI services for patch validation so that we can have > >> > > > unilateral control over scaling our build / test resources as the > >> > > > community grows larger. As the most active merger of patches (I > have > >> > > > merged over 50% of pull requests over the project's history) this > >> > > > affects me greatly as I am often monitoring builds on many open > PRs so > >> > > > that I can merge them as soon as possible. We are often resorting > to > >> > > > builds on contributor's forks (assuming they have enabled Travis > CI / > >> > > > Appveyor) > >> > > > > >> > > > As some context around Travis CI in particular, in January Travis > CI > >> > > > was acquired by Idera, a private equity (I think?) developer tools > >> > > > conglomerate. It's likely that we're seeing some "maximize profit, > >> > > > minimize costs" behavior in play, so the recent experience could > >> > > > become the new normal. > >> > > > > >> > > > - Wes > >> > > > > >> > >
