On Mon, May 20, 2019 at 4:30 PM Aaron Meurer <asmeu...@gmail.com> wrote:
> On Mon, May 20, 2019 at 2:45 PM Isuru Fernando <isu...@gmail.com> wrote: > > > > > > On Mon, May 20, 2019 at 3:34 PM Aaron Meurer <asmeu...@gmail.com> wrote: > >> > >> Thanks. My biggest question has been how we can do it in the cloud. > >> Most CI services run multiple concurrent jobs on the same machine, > >> making the performance inconsistent. Does drone.io let you have a > >> dedicated machine? > > > > > > Drone.io has a shared cloud offering which we don't want. As you > mentioned we can buy a cheap dedicated machine and install drone on it for > free. > > > > Drone might support dedicated machines on the cloud, but I'm not sure. > Travis-CI supports dedicated machines, but we looked at this for > conda-forge and they were quite expensive. > > I see, so Drone is a CI software, similar to gitlab CI or Azure. Yes > We > will need to decide which of those is the most appropriate to use. > I've heard good things about GitLab CI. Can it not be used with GitHub > repos? > I though it couldn't, but looks like they do, https://about.gitlab.com/handbook/marketing/product-marketing/enablement/github-ci-cd-faq/#open-source-projects-opportunity I like Gitlab CI and I know Ondrej has experience setting up Gitlab runners on dedicated machines. > Regarding the hardware, the question is if there are cloud providers > that provide dedicated machines, and how much they cost. If anyone has > any suggestions for this let me know. If we go with Gitlab CI with github integration, we can use something like https://aws.amazon.com/ec2/pricing/dedicated-instances/ Isuru > Alternately, what's a reasonably > cheap box we can put together to host ourselves, and what are the > tradeoffs there? > > Aaron Meurer > > > > >> > >> If we can't do it in the cloud, another alternative would be to buy a > >> cheap dedicated machine that we can provision ourselves. We have funds > >> in our NumFOCUS account to do either of these, so the question is > >> which will be easier for us to set up and maintain. What do the sumpy > >> benchmarks run on? > > > > > > It's a dedicated machine somewhere in the university running a gitlab > runner service. Not a cloud machine. > > > > Isuru > > > >> > >> Aaron Meurer > >> > >> On Mon, May 20, 2019 at 1:59 PM Isuru Fernando <isu...@gmail.com> > wrote: > >> > > >> > Hi, > >> > > >> > I setup asv in Gitlab CI for https://gitlab.tiker.net/inducer/sumpy. > It runs the PR head and master (cached) and errors if there is a test with > 10% regression. > >> > Script is at > https://gitlab.tiker.net/inducer/ci-support/blob/master/build-and-benchmark-py-project.sh. > It runs on a dedicated Gitlab runner. > >> > > >> > For sympy, you'll have to have a dedicated machine for benchmarking > and some CI service that integrates with github installed. (Maybe Drone CI > which is free for 15000 builds per year. > https://drone.io/enterprise/starter/#features). You'll have to limit to 1 > concurrent job to avoid multiple PRs affecting each other. > >> > > >> > Isuru > >> > > >> > On Mon, May 20, 2019 at 1:20 PM Aaron Meurer <asmeu...@gmail.com> > wrote: > >> >> > >> >> Hello everyone. > >> >> > >> >> I would like to start looking into setting up some kind of > >> >> benchmarking CI for SymPy. > >> >> > >> >> Ideally the CI would work just like Travis or codecov. It would run > >> >> the benchmark suite on every pull request, and report if there are > any > >> >> major performance regressions. > >> >> > >> >> If anyone has any thoughts on how to go about doing this, let me > know. > >> >> If anyone knows of any projects that already do this, let me know > that > >> >> as well. > >> >> > >> >> Finally, if anyone is interested in helping out with this, please > let me know. > >> >> > >> >> Aaron Meurer > >> >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > Groups "sympy" group. > >> >> To unsubscribe from this group and stop receiving emails from it, > send an email to sympy+unsubscr...@googlegroups.com. > >> >> To post to this group, send email to sympy@googlegroups.com. > >> >> Visit this group at https://groups.google.com/group/sympy. > >> >> To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CAKgW%3D6Lks6y8CKbiBRauwD1DQ5%3DcnMzQNbXuhhD9MefbN%3DQQrQ%40mail.gmail.com > . > >> >> For more options, visit https://groups.google.com/d/optout. > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > Groups "sympy" group. > >> > To unsubscribe from this group and stop receiving emails from it, > send an email to sympy+unsubscr...@googlegroups.com. > >> > To post to this group, send email to sympy@googlegroups.com. > >> > Visit this group at https://groups.google.com/group/sympy. > >> > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CA%2B01voNuyrPRemoJ-3Q0HgucebO4%3DB1g0JScFqs0WwMj3-5odg%40mail.gmail.com > . > >> > For more options, visit https://groups.google.com/d/optout. > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups "sympy" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an email to sympy+unsubscr...@googlegroups.com. > >> To post to this group, send email to sympy@googlegroups.com. > >> Visit this group at https://groups.google.com/group/sympy. > >> To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CAKgW%3D6L%3Ddx%3DXpxEH%2BBG%2B9nW%3DaLWceAsyRSv2xD6Zm-iQCHmDLw%40mail.gmail.com > . > >> For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups "sympy" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to sympy+unsubscr...@googlegroups.com. > > To post to this group, send email to sympy@googlegroups.com. > > Visit this group at https://groups.google.com/group/sympy. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CA%2B01voNzw3ZzZnan-%3DPMeYtGHD7zJ6O9UDrCdPsBMgMCYCi34A%40mail.gmail.com > . > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sympy+unsubscr...@googlegroups.com. > To post to this group, send email to sympy@googlegroups.com. > Visit this group at https://groups.google.com/group/sympy. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BQ6eHPqERxn6btbncGFYuf5pg_pHKnRvF6%2B5KQTEbVrw%40mail.gmail.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at https://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CA%2B01voMGJHvv0N2tFY9Fa6wSvSXobtHpQXSjV6G308Tq0_CRJg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
