Re: [sympy] Re: GSOC: Benchmark Project Questions

[Sam Lubelsky]

Is there any specific problems with the current pull request benchmarking 
system that this project should address?
On Sunday, March 31, 2024 at 1:41:58 PM UTC-5 moore...@gmail.com wrote:

> HI Sam,
>
> I think that idea could be a bit outdated. I'm not sure if the text was 
> updated for this year. If it was, then someone else can speak up about it.
>
> I think that improving our sympy_benchmarks repository with more and 
> better benchmarks and making the benchmarking system that we have setup 
> with each pull request to sympy more useful is a better focus. I'm not sure 
> we can run the benchmarks on a dedicated machine unless we spend some sympy 
> funds to do that.
>
> We basically want to know if a pull request slows down sympy and make sure 
> the pull request authors are warned about this in a clear way before 
> merging. In the past it was helpful to see the historical speed of various 
> SymPy benchmarks (here is an example I used to maintain: 
> https://www.moorepants.info/misc/sympy-asv/) but that does require a 
> dedicated machine so that benchmarks are comparable over time.
>
> Another thing I thought would be useful in the past, is to run benchmarks 
> as part of the release process (or just before) so we can see if the 
> upcoming release is slower than the prior release.
>
> Jason
> moorepants.info
> +01 530-601-9791 <(530)%20601-9791>
>
>
> On Sun, Mar 31, 2024 at 8:13 PM Sam Lubelsky  wrote:
>
>> Sorry if it is kinda intimidating that I put so many questions.  I really 
>> just need the answer to the first one to make my proposal.  I know I am a 
>> little late to GSOC, but I've really enjoyed getting to know the Sympy 
>> community a little bit in this past week and I am committed to putting 
>> together a good project proposal.
>> Thanks, 
>> Sam.
>> On Friday, March 29, 2024 at 4:37:55 PM UTC-5 Sam Lubelsky wrote:
>>
>>> I put an introduction a few emails down, but to recap my name is Sam, 
>>> I'm a college freshman, and I'm very interested in working on improving 
>>> Sympy's benchmarking services over this summer through GSOC.  
>>>
>>> While going through the project description I had a few questions:
>>>
>>> 1) *"It also needs an automated system to run them"*
>>> What exactly is meant by this.  Right now, github actions seems to be 
>>> already automatically running benchmarking after each pr.  Why is this not 
>>> an automated system?  Is the meaning of automated system something that 
>>> runs weekly/monthly on the whole repo, generates a benchmark report and 
>>> sends it* somewhere?*
>>>
>>> 2) *How to go about hosting benchmarks on a remote, dedicated machine?*  
>>> What's the general idea of how to go about this in open source project.  Is 
>>> there money available to pay some cloud provider to host it?  Free hosting 
>>> options?(doesn't seem reliable enough for benchmarking).
>>>
>>> 3) *SymEngine vs SymPy.  *I'm not familiar with SymEngine.  
>>> Approximately how similar are SymPy and SymEngine?  Is making the project 
>>> also work with SymEngine more of a quick fix(≈1-2 weeks) or should I expect 
>>> it to take longer?
>>>
>>> 4) *Current Benchmark Suite*
>>> *"We currently have a benchmarking suite and run the benchmarks on 
>>> GitHub Actions, but this is limited and is often buggy"*
>>>  
>>> What are the limitation(s) to github actions that this project should 
>>> address?
>>> If we don't use github actions, is there another way to make it run 
>>> after every PR like we have now?
>>>
>>> 5) *Where are the tests run now?*
>>> On the project description it says " the results are run and hosted Ad 
>>> Hoc", which I assumes means whatever computer is running all the other PR 
>>> tests.  Just want to make sure this is correct.
>>>
>>>
>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "sympy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to sympy+un...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/sympy/9c277927-7ac7-4c91-8c10-9ec63263f307n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sympy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sympy/577753cf-ff9d-4707-a3d3-8695bbf10c77n%40googlegroups.com.

Re: [sympy] Re: GSOC: Benchmark Project Questions

[Jason Moore]

HI Sam,

I think that idea could be a bit outdated. I'm not sure if the text was
updated for this year. If it was, then someone else can speak up about it.

I think that improving our sympy_benchmarks repository with more and better
benchmarks and making the benchmarking system that we have setup with each
pull request to sympy more useful is a better focus. I'm not sure we can
run the benchmarks on a dedicated machine unless we spend some sympy funds
to do that.

We basically want to know if a pull request slows down sympy and make sure
the pull request authors are warned about this in a clear way before
merging. In the past it was helpful to see the historical speed of various
SymPy benchmarks (here is an example I used to maintain:
https://www.moorepants.info/misc/sympy-asv/) but that does require a
dedicated machine so that benchmarks are comparable over time.

Another thing I thought would be useful in the past, is to run benchmarks
as part of the release process (or just before) so we can see if the
upcoming release is slower than the prior release.

Jason
moorepants.info
+01 530-601-9791


On Sun, Mar 31, 2024 at 8:13 PM Sam Lubelsky  wrote:

> Sorry if it is kinda intimidating that I put so many questions.  I really
> just need the answer to the first one to make my proposal.  I know I am a
> little late to GSOC, but I've really enjoyed getting to know the Sympy
> community a little bit in this past week and I am committed to putting
> together a good project proposal.
> Thanks,
> Sam.
> On Friday, March 29, 2024 at 4:37:55 PM UTC-5 Sam Lubelsky wrote:
>
>> I put an introduction a few emails down, but to recap my name is Sam, I'm
>> a college freshman, and I'm very interested in working on improving Sympy's
>> benchmarking services over this summer through GSOC.
>>
>> While going through the project description I had a few questions:
>>
>> 1) *"It also needs an automated system to run them"*
>> What exactly is meant by this.  Right now, github actions seems to be
>> already automatically running benchmarking after each pr.  Why is this not
>> an automated system?  Is the meaning of automated system something that
>> runs weekly/monthly on the whole repo, generates a benchmark report and
>> sends it* somewhere?*
>>
>> 2) *How to go about hosting benchmarks on a remote, dedicated machine?*
>> What's the general idea of how to go about this in open source project.  Is
>> there money available to pay some cloud provider to host it?  Free hosting
>> options?(doesn't seem reliable enough for benchmarking).
>>
>> 3) *SymEngine vs SymPy.  *I'm not familiar with SymEngine.
>> Approximately how similar are SymPy and SymEngine?  Is making the project
>> also work with SymEngine more of a quick fix(≈1-2 weeks) or should I expect
>> it to take longer?
>>
>> 4) *Current Benchmark Suite*
>> *"We currently have a benchmarking suite and run the benchmarks on GitHub
>> Actions, but this is limited and is often buggy"*
>>
>> What are the limitation(s) to github actions that this project should
>> address?
>> If we don't use github actions, is there another way to make it run after
>> every PR like we have now?
>>
>> 5) *Where are the tests run now?*
>> On the project description it says " the results are run and hosted Ad
>> Hoc", which I assumes means whatever computer is running all the other PR
>> tests.  Just want to make sure this is correct.
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sympy/9c277927-7ac7-4c91-8c10-9ec63263f307n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sympy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sympy/CAP7f1AgMBvGuzZC2c_jahOtpcF%2B8viaKLW%2BayxTQXoHSxAcw3A%40mail.gmail.com.

[sympy] Re: GSOC: Benchmark Project Questions

[Sam Lubelsky]

Sorry if it is kinda intimidating that I put so many questions.  I really 
just need the answer to the first one to make my proposal.  I know I am a 
little late to GSOC, but I've really enjoyed getting to know the Sympy 
community a little bit in this past week and I am committed to putting 
together a good project proposal.
Thanks, 
Sam.
On Friday, March 29, 2024 at 4:37:55 PM UTC-5 Sam Lubelsky wrote:

> I put an introduction a few emails down, but to recap my name is Sam, I'm 
> a college freshman, and I'm very interested in working on improving Sympy's 
> benchmarking services over this summer through GSOC.  
>
> While going through the project description I had a few questions:
>
> 1) *"It also needs an automated system to run them"*
> What exactly is meant by this.  Right now, github actions seems to be 
> already automatically running benchmarking after each pr.  Why is this not 
> an automated system?  Is the meaning of automated system something that 
> runs weekly/monthly on the whole repo, generates a benchmark report and 
> sends it* somewhere?*
>
> 2) *How to go about hosting benchmarks on a remote, dedicated machine?*  
> What's the general idea of how to go about this in open source project.  Is 
> there money available to pay some cloud provider to host it?  Free hosting 
> options?(doesn't seem reliable enough for benchmarking).
>
> 3) *SymEngine vs SymPy.  *I'm not familiar with SymEngine.  Approximately 
> how similar are SymPy and SymEngine?  Is making the project also work with 
> SymEngine more of a quick fix(≈1-2 weeks) or should I expect it to take 
> longer?
>
> 4) *Current Benchmark Suite*
> *"We currently have a benchmarking suite and run the benchmarks on GitHub 
> Actions, but this is limited and is often buggy"*
>  
> What are the limitation(s) to github actions that this project should 
> address?
> If we don't use github actions, is there another way to make it run after 
> every PR like we have now?
>
> 5) *Where are the tests run now?*
> On the project description it says " the results are run and hosted Ad 
> Hoc", which I assumes means whatever computer is running all the other PR 
> tests.  Just want to make sure this is correct.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sympy+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/sympy/9c277927-7ac7-4c91-8c10-9ec63263f307n%40googlegroups.com.