[pypy-dev] Improving the pypy website (and benchmarks)
Hi, this is my first post on the pypy-dev mailing list. I've commented on the pypy blog, and was encouraged by fijal to continue here. My daily job involves some web coding (HTML, javascript, Django...), python coding (frameworks, scipts), usability analysis, and opensource project management among other tasks, so I have some ideas on how to improve pypy's website, as well as the project's visibility. I find the pypy project extremely interesting (and important!), and I think that once a version of the JIT gets into a release the project will get a lot more attention. There are two things I want to discuss: One is improving pypy's main website. While the current site has served its purpose, it is mostly a pypy developers site. Better structure and navigation would be desirable when pypy becomes popular among mainstream python developers. So there are two options: to keep the current http://codespeak.net/pypy/dist/pypy/doc/index.html site for developers and develop a new www.pypy.org site, or improve the current one. The tasks to perform would be: - Agree on a new website or keeping and improving the current one - Choose a CMS (or hand-code or whatever) to craft the website - Define a navigation menu with key areas (about, download, news, roadmap, benchmarks, developement...) - Visual design - Code ;.) I can help with some (or all) of these tasks. Another matter are benchmarks. Because it is the project's most visible "feature" or "result", it would be great to publish a set of benchmarks so that python users can keep track of performance across different versions (cpython 2.6 vs pypy1.1, Jython, etc...). That way they can keep track of performance improvements as well as decide when it becomes attractive for them to make the switch from cpython. It would be the best advertisement for the project. The best case would be if you internally perform performance test to prevent performance regression on new releases, and that same data could be also be automatically published on the web, in the dev pages during development, and .in the "public" pages for final releases. So the tasks here would be: - Define a set of standard benchmarks that will serve as performance tests for every new release (including alphas and betas) - Create a script that gathers all the data for developers to analyse and spot performance regressions and bugs AND outputs the data in such a way that it can be automatically published on the website (so no extra maintenance workload) - Code the web page that beautifully shows the data in a suitable format (tables, graphs) I have recently done some work on dynamic javascript (or python) plotting, so I can take care of the last part relatively easily. I could also help with the second task. So I leave it there for you to discuss. What do you think of it all? Cheers, Miquel ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] benchmarking input
Anders Hammarquist wrote: > Hi all, Hi Iko, hi all [cut] > So benchmarks > will obviously have to specify what interpreter(s) they should be run > by somehow. I think this is a vital requirement. I can imagine various scenarios where you want to specify which interpreters to benchmark, e.g.: 1) benchmarking pypy-cli vs IronPython or pypy-jvm vs Jython 2) benchmarking pypy-cs at different svn revisions 3) benchmarking pypy-c-trunk vs pypy-c-some-branch (maybe with the possibility of specifying pypy-c-trunk-at-the-revision-where-the-branch-was-created, to avoid noise) 4) benchmarking pypy-cs with different build options 5) bencharmking with profiling enabled (I'd say that profiling should be off by default) > The bigger question is how to get those interpreters. Should running > the benchmarks also trigger building one (or more) pypy interpreters > according to specs in the benchmarking framework? (but then if you > only want it to run one benchmark, you may have to wait for all the > interpreters to build) Perhaps each benchmark should build its own > interpreter (though this seems slow, given that most benchmarks > can probably run on an identically built interpreter). > > Or maybe the installed infrastructure should only care about history, > and if you want to run a single benchmark, you do that on your own. Conceptually, I would say that you need to rebuild the required pypys every time you run the benchmarks. Concretely, we can think of putting them into a cache, so that if you need a pypy-c that for some reason has already been built, you just reuse it. Moreover, it could be nice if you could select the pypy to benchmark from a list of already built pypys, if you want to same time. Also, we may need to think how to deal with excessive loading: if everyone of us tries to run his own set of benchmark, the benchmarking machine could become too overloaded to be useful in any sense. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] benchmarking input
Hi all, I need some input for the benchmarking infrastructure. I'm nearly at the point where I need to have some place to run it before continuing (i.e. I need to try and use it, not just speculate). Anyway, what I was thinking about, and need input on, is how to get at the interpreters to run the benchmark. When we were talking just benchmarks, and not profiling, my thought was to just use whatever python the machine has, and fetch the pypy from the last buildbot run, but for profiling that will not work (and anyway, running the profiling on the standard python is quite pointless). So benchmarks will obviously have to specify what interpreter(s) they should be run by somehow. The bigger question is how to get those interpreters. Should running the benchmarks also trigger building one (or more) pypy interpreters according to specs in the benchmarking framework? (but then if you only want it to run one benchmark, you may have to wait for all the interpreters to build) Perhaps each benchmark should build its own interpreter (though this seems slow, given that most benchmarks can probably run on an identically built interpreter). Or maybe the installed infrastructure should only care about history, and if you want to run a single benchmark, you do that on your own. Thoughts please! /Anders ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
