Re: [pypy-dev] [Speed] adding a pypy runner for speed.python.org

[Miquel Torres]

Hi Matti,

I'll gladly help setting up the speed site

El El mié, 31 ene 2018 a las 13:47, Victor Stinner 
escribió:

> Hi,
>
> I tried but failed to find someone in PyPy to adjust performance
> benchmarks for PyPy. Currently, the JIT is not properly warmed up, and
> the results can be dishonnest or not reliable.
>
> My latest attempt to support is PyPy is:
> http://vstinner.readthedocs.io/pypy_warmups.html
>
> IMHO we need to develop a statistical methodology in perf to compute
> when values become stable. That's hard to define and it was proven
> that "performance stability" doesn't exist (see " Virtual Machine
> Warmup Blows Hot and Cold" paper).
>
> Fijal from PyPy would like to use hardcoded configuration for the
> number of warmup values. I like the idea, but nobody implemented it
> yet.
>
> Victor
>
> 2018-01-30 21:37 GMT+01:00 Matti Picus :
> > I am cross-posting to both speed and pypy-dev to ask what needs to be
> done
> > to get pypy2 and pypy3 benchmark runners onto speed.python.org
> > I am willing to be the contact person from the PyPy side, who do we need
> to
> > talk to on the speed maintainers' side?
> > Instead of spamming the lists again, we could discuss this off-line, and
> > report back with a plan, required resources, and a timeline.
> >
> > Thanks,
> > Matti Picus
> > ___
> > Speed mailing list
> > sp...@python.org
> > https://mail.python.org/mailman/listinfo/speed
> ___
> Speed mailing list
> sp...@python.org
> https://mail.python.org/mailman/listinfo/speed
>
___
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] 404 and DEBUG=True on speed.pypy.org

[Miquel Torres]

Thanks Davide. It's fixed now.

Cheers
Miquel


2011/11/4 Davide Setti davide.se...@gmail.com:
 Hi,

 If i click on the permalink link on

 http://speed.pypy.org/timeline/#/?exe=3,6,1,5base=2+472ben=gridenv=1revs=200equid=off

 i get:

 http://speed.pypy.org/timeline/?exe=3%2C6%2C1%2C5base=2%2B472ben=gridenv=1revs=200equid=off

 DEBUG=True is not good...

 Regards.
 --

 Davide Setti
 code: http://github.com/vad
 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] speed and 1.6

[Miquel Torres]

Done, I tagged revision 46161:eb30a0ef328e (1st of August) as PyPy
1.6. Can be seen now on the start page.

Cheers,
Miquel


2011/9/8 Antonio Cuni anto.c...@gmail.com:
 On 08/09/11 12:15, Miquel Torres wrote:

 which sadly doesn't have data on speed.pypy.org (not all data saved on
 the removed environment was kept, sorry). The August 1st revision is
 probably not acceptable to portrait as being 1.6 ...

 looking at the graphs, I don't see any big difference between august 1st and
 august 3rd, so I think we could just use that and be happy.

 ciao,
 anto

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] [Speed] Moving the project forward

[Miquel Torres]

You can also do that in Github, which I prefer.

However, since CPython and PyPy use mercurial, the general preference
for Bitbucket is understandable.


2011/9/1 Brett Cannon br...@python.org:
 On Thu, Sep 1, 2011 at 01:10, Nick Coghlan ncogh...@gmail.com wrote:
 On Thu, Sep 1, 2011 at 5:50 PM, Antonio Cuni anto.c...@gmail.com wrote:
 On 31/08/11 22:11, Brett Cannon wrote:

 The PyPy folk could answer this as they have their repo on bitbucket
 already. Else I guess we can just create a standalone account that
 represents the official speed.python.org account.

 for pypy we do exactly that. There is a bitbucket user named pypy whose
 credentials are shared among all the core devs.

 The security auditing part of my brain has its fingers in its ears and
 is singing La La La rather loudly :)

 What about Google Code? Projects there can have multiple owners and
 they support hg, have a tracker, and a wiki.



 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

 ___
 Speed mailing list
 sp...@python.org
 http://mail.python.org/mailman/listinfo/speed

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Moving the project forward

[Miquel Torres]

Hi all,

though I took up on the task of installing a Codespeed instance
myself, I didn't have time until now. This weekend I will definitely
have  a *lot* of time to work on this, so count on that task being
done by then.

The bitbucket issue tracker is a start (though a organization account
would be better) and the splash page is great. So let's get started
organizing things.

Regarding the deployment strategy, it turns out I use Chef at work, so
I am in full agreement with Noah here (yey!). Actually, I am the
author of LittleChef (which we can use as a tool to execute Chef on
the node).

So, Configuration Management. I would propose that Noah starts the
repo with the Chef cookbooks (preferably a complete LittleChef
kitchen, but that is not a must :), and gets the main recipes (apache,
django) going, while I create a cookbook for Codespeed. What do you
think?

The benchmark runner question is still open. We need to clarify that.
Use the pypy runner? Tennessee's work?

Regarding repositories and issues, we could maybe have a speed
organization account (not sure on Bitbucket, you can do that in
Github), where we have a wiki, issues, and runner + config management
repo + other stuff.

Cheers,
Miquel

2011/8/31 Jesse Noller jnol...@gmail.com:
 I've put up a splash page for the project this AM:

 http://speed.python.org/

 jesse
 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] [Speed] Moving the project forward

[Miquel Torres]

Oh, cool, so there will be an Opscode hosted account for the PSF,
right? Then the Chef repo should be for the PSF. Maybe in a current
account somewhere? What do you propose?

Miquel


2011/8/31 Noah Kantrowitz n...@coderanger.net:
 Opscode has already agreed to donate a Hosted account as long we keep it 
 under ~20 clients :-) I can hand out the info for it to anyone that wants. As 
 for setting up the Chef repo, just remember we are trying to not manage this 
 system in isolation and that it will be part of a bigger PSF infrastructure 
 management effort.

 --Noah

 On Aug 31, 2011, at 11:34 AM, Miquel Torres wrote:

 Hi all,

 though I took up on the task of installing a Codespeed instance
 myself, I didn't have time until now. This weekend I will definitely
 have  a *lot* of time to work on this, so count on that task being
 done by then.

 The bitbucket issue tracker is a start (though a organization account
 would be better) and the splash page is great. So let's get started
 organizing things.

 Regarding the deployment strategy, it turns out I use Chef at work, so
 I am in full agreement with Noah here (yey!). Actually, I am the
 author of LittleChef (which we can use as a tool to execute Chef on
 the node).

 So, Configuration Management. I would propose that Noah starts the
 repo with the Chef cookbooks (preferably a complete LittleChef
 kitchen, but that is not a must :), and gets the main recipes (apache,
 django) going, while I create a cookbook for Codespeed. What do you
 think?

 The benchmark runner question is still open. We need to clarify that.
 Use the pypy runner? Tennessee's work?

 Regarding repositories and issues, we could maybe have a speed
 organization account (not sure on Bitbucket, you can do that in
 Github), where we have a wiki, issues, and runner + config management
 repo + other stuff.

 Cheers,
 Miquel

 2011/8/31 Jesse Noller jnol...@gmail.com:
 I've put up a splash page for the project this AM:

 http://speed.python.org/

 jesse
 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

 ___
 Speed mailing list
 sp...@python.org
 http://mail.python.org/mailman/listinfo/speed


___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] python 3

[Miquel Torres]

so the situation is even better: even if the PyPy Python 2.x
implementation was not updated any more after 1.6 (which won't be the
case), future versions of PyPy could have *both* a 2.x and a 3.x
interpreter (separately packaged), and *both* would leverage the newer
JIT versions. Is that correct Maciej?

In that case, the question for PyPy as a project would be when it
makes sense to invest the time to implement a 3.x interpreter if no
one steps up and just does it ;-)

Miquel


2011/8/18 Maciej Fijalkowski fij...@gmail.com:
 On Thu, Aug 18, 2011 at 2:13 AM, Eli Stevens (Gmail)
 wickedg...@gmail.com wrote:
 On Wed, Aug 17, 2011 at 4:01 PM, Yury Selivanov yselivanov...@gmail.com 
 wrote:
 +1 to the question.  Why can't it be that way?

 If by that way you mean leave python 2.x behind post 1.6 I'd like
 to note that IMO pypy has been under-acknowledged by the wider python
 community for a very long time.  That's finally starting to change
 (pypy production releases, cpython devs devoting resources to make
 alternate implementations not second-class citizens, etc.), but by
 abandoning the segment of the language with the largest userbase, the
 project would go back to niche status again.  Yeah, doing so might
 position pypy well to become the default python 3 implementation, but
 I find it hard to imagine that tacking on another N years until pypy
 is a significant percentage of python deployments is going to be good
 for the project.

 My $0.02,
 Eli

 Just to answer some questions:

 There is no way we're leaving python 2 support for the forseeable
 future. It's all constructed precisely for the reason so JIT
 improvements will benefit every interpreter written in RPython, not
 just a specific one. In either of scenarios presented above, new
 releases will include both py 3.x and 2.7 as release targets.

 Cheers,
 fijal
 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy at last infinitely fast

[Miquel Torres]

Hi Antonio,

I'm afraid you are right, the solution I proposed makes no sense.
Sorry I gave you a wrong answer. A revision is unique for a project
(well, now to a branch of a project), and thus they are not separated
by environment. Codespeed was not really designed with revisions in
mind that sometimes have results, and sometimes not. To solve that,
revisions would need to depend on an executable as well, or introduce
a check that so that the revision list is tailored to a particular
exe, but it would be ugly.

There is a way though to solve it right now. Separate executables
into two different projects. pypy32, pypy64, instead of different
environements. The revision list shown does change on-the-fly
depending on the project the selected exe belongs to.

Cheers,
Miquel


2011/7/27 Antonio Cuni anto.c...@gmail.com:
 Hi Miquel,

 On 25/07/11 21:09, Miquel Torres wrote:

 Btw., in any case you can start saving results in separated
 environments. That will at least make changes work right away, and it
 does make sense to have tannit 32 bits and tannit 64 bits.

 are you sure that having two separate environments will fix the changes
 page? I tried to add tannit-64 (no results yet, a build is running right
 now), but in the dropdown menu of the Changes page I can see all the
 revisions that are also in tannit. I'd have expected those two to be
 completely separate.

 ciao,
 Anto

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy at last infinitely fast

[Miquel Torres]

Alternatively, the timeline view could allow to display several
environments at the same time...


2011/7/21 Antonio Cuni anto.c...@gmail.com:
 On 21/07/11 09:13, Maciej Fijalkowski wrote:
 I think having them at the same graph is more important than having
 changes showing correct things. I might give it a go if nobody wants
 to

 Not sure. Having them in the same graph is important only to quickly spot
 cases in which one backend is much slower than the other, which seems not to
 be the case.

 On the other hand, to spot regressions it's enough to have them in two
 separate graphs.

 ciao,
 Anto

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy at last infinitely fast

[Miquel Torres]

sorry, I have currently two busy days and couldn't look yet at it.
Will do as soon as I can


2011/7/19 Antonio Cuni anto.c...@gmail.com:
 On 18/07/11 22:36, Miquel Torres wrote:
 Ok, it is fixed now.

 AND branch support is in. Results saved with branch other than
 default will be available in the comparison view. Got to talk to
 fijal yet to see how we should benchmark branches...

 wow, all of this is very cool, thank you!

 However, there was a problem with the uploading of the results tonight :-/
 http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-32/builds/791/steps/shell_7/logs/stdio

 Do you know what it could be?

 ciao,
 Anto

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy at last infinitely fast

[Miquel Torres]

 I fixed it. Having a working changes tab would be cool, but not
 immediately needed
Thanks Maciej. It was just a case of adding the new branch parameter
to the result dictionary, as you did in
pypy/benchmarks/src/saveresults

I wanted to save you the time to look it up and change it but well, I couldn't.

Regarding the broken changes table, it is obviously a related problem,
together with not making the user having to browse through empty data.
All of those issues are caused by only benchmarking some of the
executables with one revision, and other with different revisions.
Each can be more or less solved with a couple of lines of code, but it
would introduce quite a bit of overhead, so we need to consider
(test/benchmark) the possible solution carefully.

Could you imagine implementing Antonio's suggestion?:
Citing:
A quick workaround would be to force buildbot to update to a more specific
revision. E.g., the highest revision of today at 00:00, or something like
this. This should ensure to have the same revision for all our 
benchmarks/tests.

Sounds good to me.

Miquel


2011/7/20 Maciej Fijalkowski fij...@gmail.com:
 On Wed, Jul 20, 2011 at 3:56 PM, Miquel Torres tob...@googlemail.com wrote:
 sorry, I have currently two busy days and couldn't look yet at it.
 Will do as soon as I can

 I fixed it. Having a working changes tab would be cool, but not
 immediately needed



 2011/7/19 Antonio Cuni anto.c...@gmail.com:
 On 18/07/11 22:36, Miquel Torres wrote:
 Ok, it is fixed now.

 AND branch support is in. Results saved with branch other than
 default will be available in the comparison view. Got to talk to
 fijal yet to see how we should benchmark branches...

 wow, all of this is very cool, thank you!

 However, there was a problem with the uploading of the results tonight :-/
 http://buildbot.pypy.org/builders/jit-benchmark-linux-x86-32/builds/791/steps/shell_7/logs/stdio

 Do you know what it could be?

 ciao,
 Anto

 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev


___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] just saw: speed.pypy.org reports 4x faster

[Miquel Torres]

@Phyo: it's because revision 45700 shows a large improvement for telco
and spitfire_stringio (40%+), which makes for an average improvement
of over 6%. See detailed changes table:
http://speed.pypy.org/changes/?tre=10rev=45700%3Ac3294f3c5888exe=1env=1

Due to Maciej's commit (merge numpy-str-repr) or a benchmarking anomaly?

Miquel


2011/7/18 Phyo Arkar phyo.arkarl...@gmail.com:
 So This is really happening in 1.6 or Speed.pypy bug?

 On 7/19/11, Maciej Fijalkowski fij...@gmail.com wrote:
 On Mon, Jul 18, 2011 at 11:30 PM, Massa, Harald Armin c...@ghum.de wrote:
 I recommend to wrap the code and release it with the subtitle the 4 times
 faster release

 I like codenames more. Like PyPy 1.6 Kickass Panda

 Best wishes,
 Harald
 --
 GHUM GmbH
 Harald Armin Massa
 Spielberger Straße 49
 70435 Stuttgart
 0173/9409607

 Amtsgericht Stuttgart, HRB 734971
 -
 persuadere.
 et programmare

 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev


 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] PyPy at last infinitely fast

[Miquel Torres]

Ok, it is fixed now.

AND branch support is in. Results saved with branch other than
default will be available in the comparison view. Got to talk to
fijal yet to see how we should benchmark branches...

Ciao!
Miquel


2011/7/18 Antonio Cuni anto.c...@gmail.com:
 On 18/07/11 21:37, Miquel Torres wrote:
 Hi,

 speed.pypy.org currently shows a very encouraging performance picture
 for PyPy: it is infinite times faster than CPython. No, it is note
 yet April 1st.

 hooray! We finally finished pypy :-)

 Codespeed creates the front page plots using the latest tested
 revision... which currently has no data for pypy-c-jit (32 bits).
 There is already a ticket for that issue, as well as to fix the
 Changes view letting you choose a revision with no data. The
 assumption was that a revision would be tested for all executables of
 a given project, which is no longer the case for PyPy.

 Still, even though there will be a fix, it may be a good idea to test
 all exes for the same revision to easy comparisons.

 this is not completely easy, because buildbot just pulls and updates to the
 latest revision. If someone pushes in between the two runs, the revision is
 different.

 A quick workaround would be to force buildbot to update to a more specific
 revision. E.g., the highest revision of today at 00:00, or something like
 this. This should ensure to have the same revision for all our 
 benchmarks/tests.

 ciao,
 Anto

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Benchmarks

[Miquel Torres]

Branches are already implemented. speed.pypy.org just needs to be
upgraded to Codespeed 0.8.x and its data migrated.

That has not yet been done because we wanted to move to ep.io, which
has not yet happend, and we are working on speed.python.org and
somehow things have stalled.

Migrating current speed.pypy.org (I mean keeping it on the current
server but upgrading) would not take me very long. But it is on the
way out...



2011/7/12 Antonio Cuni anto.c...@gmail.com:
 On 12/07/11 09:01, Maciej Fijalkowski wrote:
 I'll follow up on the branches, but the issue is a bit irrelevant - we
 still have performance regressions by trunk checkins as well.

 it's not irrelevant. I won't solve the current issues, but it will help having
 more in the future.

 ciao,
 Anto
 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] Benchmarks

[Miquel Torres]

They *are* being merged. The question here is to have branches *now*,
as we can't know how long it will take for speed.python.org to be
online.

All things considered, I think the way to go would be to keep current
speed.pypy.org hosting if at all possible until PyPy moves to
speed.python.org. If that can be done, we can upgrade the current
setup in no time.



2011/7/12 holger krekel hol...@merlinux.eu:
 On Tue, Jul 12, 2011 at 09:19 +0200, Maciej Fijalkowski wrote:
 On Tue, Jul 12, 2011 at 9:16 AM, Miquel Torres tob...@googlemail.com wrote:
  Branches are already implemented. speed.pypy.org just needs to be
  upgraded to Codespeed 0.8.x and its data migrated.

 I can help you with that, but that means having local changes from
 speed.pypy.org commited somewhere else.

 
  That has not yet been done because we wanted to move to ep.io, which
  has not yet happend, and we are working on speed.python.org and
  somehow things have stalled.

 speed.python.org is completely irrelevant here.

 Probably i am missing something but couldn't speed.pypy.org and
 speed.python.org mostly merge?  (maybe except for the front page and
 default comparisons/settings but those could be easily hosted wherever)

 best,
 holger

  Migrating current speed.pypy.org (I mean keeping it on the current
  server but upgrading) would not take me very long. But it is on the
  way out...

 We either should do the ep.io move (data has been imported up until
 very recently) or upgrade now.



 
 
 
  2011/7/12 Antonio Cuni anto.c...@gmail.com:
  On 12/07/11 09:01, Maciej Fijalkowski wrote:
  I'll follow up on the branches, but the issue is a bit irrelevant - we
  still have performance regressions by trunk checkins as well.
 
  it's not irrelevant. I won't solve the current issues, but it will help 
  having
  more in the future.
 
  ciao,
  Anto
  ___
  pypy-dev mailing list
  pypy-dev@python.org
  http://mail.python.org/mailman/listinfo/pypy-dev
 
 
 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev


___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Re: [pypy-dev] codespeed setup

[Miquel Torres]

I have answered on the Codespeed discussion group.

Maciej is right that you should actually run bechmarks on a separate,
clean box, which then posts results to the Codespeed instance.

Miquel


2011/6/16 Maciej Fijalkowski fij...@gmail.com:
 On Thu, Jun 16, 2011 at 4:32 AM, Chuck Remes cremes.devl...@mac.com wrote:
 I'm working on setting up codespeed for several Ruby projects. The codespeed 
 project points at this list as the place to ask questions and get help.

 There is a codespeed-dev list somewhere as well, it is ok to ask here though.


 From the README on the project page (https://github.com/tobami/codespeed) 
 it says that codespeed needs to clone any repositories locally that it is 
 tracking.

 Note: Git and Mercurial need to locally clone the repository. That means 
 that your codespeed/speedcenter/repos directory will need to be owned by 
 the server. In the case of a typical Apache installation, you'll need to 
 type sudo chown www-data:www-data codespeed/speedcenter/repos

 This implies that serving up all web requests to get access to the stats 
 happens on the *same* box that is also running the benchmarks. I would think 
 this would skew results over time especially if there is heavy web traffic. 
 I would like to avoid having the resource consumption from web hosting 
 impact the benchmarking output.

 No, it does not imply that in any way. Requiring local copy doesn't
 mean you have to have benchmarks run there.


 Once I'm ready to take this into production, is there a django configuration 
 change I can make so that the box offloads all web hosting to another host? 
 Ideally, as commits are detected in the repositories and the benchmarks are 
 rerun and the results posted, codespeed would be able to regen a bunch of 
 static files and ship them off to the web server.

 Am I on the right track with this or am I missing something obvious?

 Pages are a bit too dynamic for that. You can request particular
 ranges of data etc. In theory it's maybe possible, but I don't think
 with the current codebase.


 cr

 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

 ___
 pypy-dev mailing list
 pypy-dev@python.org
 http://mail.python.org/mailman/listinfo/pypy-dev

___
pypy-dev mailing list
pypy-dev@python.org
http://mail.python.org/mailman/listinfo/pypy-dev