Re: [sympy] Benchmarking CI

2019-05-21 Thread Ondřej Čertík
> On Tue, May 21, 2019 at 12:41 AM Isuru Fernando wrote: > > > Yes, but not by default, one has to setup a bot to do that. Here is how > > > somebody already did exactly that: > > > > There's a new feature in gitlab that takes care of this. See > > https://docs.gitlab.com/ee/ci/ci_cd_for_extern

Re: [sympy] Benchmarking CI

2019-05-21 Thread Aaron Meurer
On Mon, May 20, 2019 at 9:49 PM Ondřej Čertík wrote: > > > > On Mon, May 20, 2019, at 5:06 PM, Aaron Meurer wrote: > > On Mon, May 20, 2019 at 4:41 PM Ondřej Čertík wrote: > > > > > > > > > > > > On Mon, May 20, 2019, at 4:15 PM, Aaron Meurer wrote: > > > > On Mon, May 20, 2019 at 3:51 PM Ondřej

Re: [sympy] Benchmarking CI

2019-05-20 Thread Isuru Fernando
Forget that. That was because I also pushed to a branch. For PRs, we'll have to set up a bot to push to a gitlab branch. Isuru On Tue, May 21, 2019 at 12:41 AM Isuru Fernando wrote: > > Yes, but not by default, one has to setup a bot to do that. Here is how > somebody already did exactly that:

Re: [sympy] Benchmarking CI

2019-05-20 Thread Isuru Fernando
> Yes, but not by default, one has to setup a bot to do that. Here is how somebody already did exactly that: There's a new feature in gitlab that takes care of this. See https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html I just created https://gitlab.com/isuruf/sympy/

Re: [sympy] Benchmarking CI

2019-05-20 Thread Ondřej Čertík
On Mon, May 20, 2019, at 5:06 PM, Aaron Meurer wrote: > On Mon, May 20, 2019 at 4:41 PM Ondřej Čertík wrote: > > > > > > > > On Mon, May 20, 2019, at 4:15 PM, Aaron Meurer wrote: > > > On Mon, May 20, 2019 at 3:51 PM Ondřej Čertík wrote: > > > > > > > > > > > > > > > > On Mon, May 20, 2019, at

Re: [sympy] Benchmarking CI

2019-05-20 Thread Aaron Meurer
On Mon, May 20, 2019 at 4:41 PM Ondřej Čertík wrote: > > > > On Mon, May 20, 2019, at 4:15 PM, Aaron Meurer wrote: > > On Mon, May 20, 2019 at 3:51 PM Ondřej Čertík wrote: > > > > > > > > > > > > On Mon, May 20, 2019, at 3:44 PM, Isuru Fernando wrote: > > > > > > > > > > > > On Mon, May 20, 2019

Re: [sympy] Benchmarking CI

2019-05-20 Thread Ondřej Čertík
On Mon, May 20, 2019, at 4:15 PM, Aaron Meurer wrote: > On Mon, May 20, 2019 at 3:51 PM Ondřej Čertík wrote: > > > > > > > > On Mon, May 20, 2019, at 3:44 PM, Isuru Fernando wrote: > > > > > > > > > On Mon, May 20, 2019 at 4:30 PM Aaron Meurer wrote: > > > > On Mon, May 20, 2019 at 2:45 PM Isu

Re: [sympy] Benchmarking CI

2019-05-20 Thread Aaron Meurer
On Mon, May 20, 2019 at 3:51 PM Ondřej Čertík wrote: > > > > On Mon, May 20, 2019, at 3:44 PM, Isuru Fernando wrote: > > > > > > On Mon, May 20, 2019 at 4:30 PM Aaron Meurer wrote: > > > On Mon, May 20, 2019 at 2:45 PM Isuru Fernando wrote: > > > > > > > > > > > > On Mon, May 20, 2019 at 3:34

Re: [sympy] Benchmarking CI

2019-05-20 Thread Ondřej Čertík
On Mon, May 20, 2019, at 3:44 PM, Isuru Fernando wrote: > > > On Mon, May 20, 2019 at 4:30 PM Aaron Meurer wrote: > > On Mon, May 20, 2019 at 2:45 PM Isuru Fernando wrote: > > > > > > > > > On Mon, May 20, 2019 at 3:34 PM Aaron Meurer wrote: > > >> > > >> Thanks. My biggest question ha

Re: [sympy] Benchmarking CI

2019-05-20 Thread Isuru Fernando
On Mon, May 20, 2019 at 4:30 PM Aaron Meurer wrote: > On Mon, May 20, 2019 at 2:45 PM Isuru Fernando wrote: > > > > > > On Mon, May 20, 2019 at 3:34 PM Aaron Meurer wrote: > >> > >> Thanks. My biggest question has been how we can do it in the cloud. > >> Most CI services run multiple concurrent

Re: [sympy] Benchmarking CI

2019-05-20 Thread Aaron Meurer
On Mon, May 20, 2019 at 2:45 PM Isuru Fernando wrote: > > > On Mon, May 20, 2019 at 3:34 PM Aaron Meurer wrote: >> >> Thanks. My biggest question has been how we can do it in the cloud. >> Most CI services run multiple concurrent jobs on the same machine, >> making the performance inconsistent. D

Re: [sympy] Benchmarking CI

2019-05-20 Thread Isuru Fernando
On Mon, May 20, 2019 at 3:34 PM Aaron Meurer wrote: > Thanks. My biggest question has been how we can do it in the cloud. > Most CI services run multiple concurrent jobs on the same machine, > making the performance inconsistent. Does drone.io let you have a > dedicated machine? > Drone.io has a

Re: [sympy] Benchmarking CI

2019-05-20 Thread Aaron Meurer
Thanks. My biggest question has been how we can do it in the cloud. Most CI services run multiple concurrent jobs on the same machine, making the performance inconsistent. Does drone.io let you have a dedicated machine? If we can't do it in the cloud, another alternative would be to buy a cheap de

Re: [sympy] Benchmarking CI

2019-05-20 Thread Isuru Fernando
Hi, I setup asv in Gitlab CI for https://gitlab.tiker.net/inducer/sumpy. It runs the PR head and master (cached) and errors if there is a test with 10% regression. Script is at https://gitlab.tiker.net/inducer/ci-support/blob/master/build-and-benchmark-py-project.sh. It runs on a dedicated Gitlab

[sympy] Benchmarking CI

2019-05-20 Thread Aaron Meurer
Hello everyone. I would like to start looking into setting up some kind of benchmarking CI for SymPy. Ideally the CI would work just like Travis or codecov. It would run the benchmark suite on every pull request, and report if there are any major performance regressions. If anyone has any though