Re: [pypy-dev] Fwd: [IronPython] SciPy
Oh wow, that's really cool. On Tue, Dec 21, 2010 at 6:15 PM, Gary Robinson wrote: >> I thought this email could be relevant for those interested in SciPy / Numpy >> on pypy. With enthought implementing a smaller core and using compatibility >> layers for alternative platforms, it would seem to be a good basis for a >> pypy port. > > All I can do is give a BIG +1 to anything that can get numpy/scipy up more > quickly. > > PyPy is starting to give us the speed we need for statistical/scientific work > on python (the speed it still lacks compared to C or Java is made up for, for > my purposes, by the ease of writing python compared to those languages). The > recent 64-bit functionality lets us process a lot of data. The fast-forward > branch is giving us the multiprocessing we need. (I recognize that there are > other solutions, but for simple things we need to write quickly, the > multiprocessing module is really sweet.) > > The main thing missing now is numpy/scipy. The addition of that will make > PyPy a huge win in the scientific community, IMO. > > Anyway, I mention it in case the opinion of one person who is using Python in > production for statistical processing is of interest. > > -- > > Gary Robinson > CTO > Emergent Discovery, LLC > personal email: gary...@me.com > work email: grobin...@emergentdiscovery.com > Company: http://www.emergentdiscovery.com > Blog: http://www.garyrobinson.net > > > > > ___ > pypy-dev@codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] funding/popularity?
I sort of figured it out, although I don't have a ready solution. This affects opera 11, stable ff, ff 4.0b7; amazingly it does not affect ie8. Basically memory consumption of the plot appears to be proportional to plot area. Normal plot that you get by default at http://speed.pypy.org/comparison/ costs about 100M of browser memory consumption: opera 130M, stable ff 70M, beta ff 90M at window size 1680x1050; opera 80M, stable ff 55M, beta ff 70M at window size 1024x600; Switching to "horizontal" produces a tall plot of same width and costs about 300~700M of browser memory: opera 720M, stable ff 370M, beta ff 370M at window wize 1680x1050; opera 350M, stable ff 370M, beta ff 370M at window size 1024x600; Suprisingly window size only matters while javascript produces the plot, and not when window is resized, even though plot it resized with the window correctly. This alone is pretty heavy, but doesn't grind the browser. What really grinds is that every time you change a tickbox on the left, a plot is redrawn and another 200M of browser memory is wasted. This is not double buffering, as next change adds yet another 200M or so and so on, it appears that either js doesn't free something, or browser caches or saves the previous page state. As memory consumption grows, at some point browser hits the wall, causes heavy swapping for some time, and I think garbage collection, because practical (but not virtual) memory usage first drops to something like 20~50M and then returns to "normal" 300M. opera ~30 seconds, stable ff about a minute, beta ff several minutes (total system mem 1G, cpu Atom @1.6GHz) Perhaps OS also plays a role in the grind, as it is clearly swapping and swaps out too much? or triggers gc too late and gc has to pull the pages back from disk to perform collection? ie8 doesn't use that much memory, as a matter of fact memory consumption starts little (40M) and changes very little (only +10M) if you go to horizonatal view; the price is very slow rendering, more than 10 seconds per column change. I'll post this on firefox bugzilla too, let's see if someone has a solution. Meanwhile perhaps pypy speed center could start with a smaller plot area (or fewer columns as that makes horizontal plot smaller) to accomodate varying hardware and system mem usage that users might have? The simplest would be a warning next to "horizontal" checkbox. On 21 December 2010 01:06, Miquel Torres wrote: > Hi Dima, > >> another temp problem with speed pypy is that it's terrubly slow in ff >> beta. it also occasionally grinds in stable ff and opera, but I guess >> this can be forgiven for the sake of simplicity / developer effort. > > Well, speed.pypy is actually fast in all modern browsers. The problem > you are referring to is probably caused by a bug in the javascript > plotting library (jqPplot) that is triggered in the comparison view > when there are some results with 0 values. It only appears for some > plot types, but it is very annoying because it grinds the browser to a > halt like you say. Is that what you meant? > > We are looking into it, and will fix that library if necessary. > > Cheers, > Miquel > > > 2010/12/21 Dima Tisnek : >> On 20 December 2010 19:21, William ML Leslie >> wrote: >>> On 21 December 2010 11:59, Dima Tisnek wrote: More visibility for performance achievements would do good too. >>> >>> Where are pypy's performance achievements *not* visible, but should be? >> >> for example http://shootout.alioth.debian.org/ >> doesn't say which pypy version is used, what options, doesn't have >> performance figures for multithreaded/multicore >> >> also benchmarks are kinda small, most of them are not docuemented, and >> I couldn't find any info if the same python code was used for cpython >> and pypy (both shootout and speed pypy) or interpreter-specific >> verions were used, that is each tweaked for best performance given the >> known tradeoffs for each implementation.further the most standard >> benchmarks, pystone, etc. completely ignore the fact that real world >> programs are large and only a few small paths are execured often. >> >> another temp problem with speed pypy is that it's terrubly slow in ff >> beta. it also occasionally grinds in stable ff and opera, but I guess >> this can be forgiven for the sake of simplicity / developer effort. >> >> if you google for 'python performance' you don't get a single link to >> pypy on the first page, as a matter of fact, codespeak is poorly >> indexed, it took me quite some time to get some of my questions >> answered with a search. also if you look up 'pypy gc' you get a page >> on codespeak, but to interpret what the data actually means is so far >> beyond me. >> >> a good overview is found in the mainling list >> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >> slowspitfire and spambayes bits are outdated by now. >> >> the definitive good thing about pypy is that it's much easier to find >> out about its inne
Re: [pypy-dev] funding/popularity?
yes, same problem, but the grind is not so aweful, surviveable even, tried with opera and normal ff On 21 December 2010 12:30, Miquel Torres wrote: > It is weird that it happens as you de-select benchmarks. > > Does it happen with non-beta browsers? > > > 2010/12/21 Dima Tisnek : >> Yes it does grind ff 4b7 to an almost halt, little cpu usage, >> reasonable memory size and constant disk activity for several minutes, >> very weird... >> So far, the difference between browsers is so large that I doubt it's >> dependent on data. >> Also it seems to tirgger as I remove columns progressively, thus new >> zero values should not appear, right? >> >> I'll invesitage some more... >> >> On 21 December 2010 01:06, Miquel Torres wrote: >>> Hi Dima, >>> another temp problem with speed pypy is that it's terrubly slow in ff beta. it also occasionally grinds in stable ff and opera, but I guess this can be forgiven for the sake of simplicity / developer effort. >>> >>> Well, speed.pypy is actually fast in all modern browsers. The problem >>> you are referring to is probably caused by a bug in the javascript >>> plotting library (jqPplot) that is triggered in the comparison view >>> when there are some results with 0 values. It only appears for some >>> plot types, but it is very annoying because it grinds the browser to a >>> halt like you say. Is that what you meant? >>> >>> We are looking into it, and will fix that library if necessary. >>> >>> Cheers, >>> Miquel >>> >>> >>> 2010/12/21 Dima Tisnek : On 20 December 2010 19:21, William ML Leslie wrote: > On 21 December 2010 11:59, Dima Tisnek wrote: >> More visibility for performance achievements would do >> good too. > > Where are pypy's performance achievements *not* visible, but should be? for example http://shootout.alioth.debian.org/ doesn't say which pypy version is used, what options, doesn't have performance figures for multithreaded/multicore also benchmarks are kinda small, most of them are not docuemented, and I couldn't find any info if the same python code was used for cpython and pypy (both shootout and speed pypy) or interpreter-specific verions were used, that is each tweaked for best performance given the known tradeoffs for each implementation.further the most standard benchmarks, pystone, etc. completely ignore the fact that real world programs are large and only a few small paths are execured often. another temp problem with speed pypy is that it's terrubly slow in ff beta. it also occasionally grinds in stable ff and opera, but I guess this can be forgiven for the sake of simplicity / developer effort. if you google for 'python performance' you don't get a single link to pypy on the first page, as a matter of fact, codespeak is poorly indexed, it took me quite some time to get some of my questions answered with a search. also if you look up 'pypy gc' you get a page on codespeak, but to interpret what the data actually means is so far beyond me. a good overview is found in the mainling list http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again slowspitfire and spambayes bits are outdated by now. the definitive good thing about pypy is that it's much easier to find out about its inner workings than that of cpython! hopefully a bit more of end-user stuff get known. let's call it pypy public outreach (feature request) > >> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >> hard would it be to use same techniques and most of backends for some >> unrelated language that doesn't have jit yet, e.g. php? > > You know that pypy already has implementations of other languages, > right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't > integrated with the translated pypy-c, but they show that it is not > too difficult to write a runtime for any dynamic language you choose. Oh I didn't know there were so many, and I mistakenly though that js was a target, not implmented langauge. In any case I read somewhere that js support was retired... > >> And how hard >> would it be to marry two dynamic languages, so that modules from one >> could be used in the other? Or that modules written in rpython could >> be used in several langs? > > It's in the "interesting problems" bucket, and the effort required > depends on the level of integration between languages you want. There > are several projects already attempting to do decent integration > between several languages, besides the approach used on the JVM, there > are also GNU Guile, Racket, and Parrot, among others. It might be > worth waiting to see how these different projects pan out, before > spending a bunch of effort just to be an also-ran in
Re: [pypy-dev] funding/popularity?
It is weird that it happens as you de-select benchmarks. Does it happen with non-beta browsers? 2010/12/21 Dima Tisnek : > Yes it does grind ff 4b7 to an almost halt, little cpu usage, > reasonable memory size and constant disk activity for several minutes, > very weird... > So far, the difference between browsers is so large that I doubt it's > dependent on data. > Also it seems to tirgger as I remove columns progressively, thus new > zero values should not appear, right? > > I'll invesitage some more... > > On 21 December 2010 01:06, Miquel Torres wrote: >> Hi Dima, >> >>> another temp problem with speed pypy is that it's terrubly slow in ff >>> beta. it also occasionally grinds in stable ff and opera, but I guess >>> this can be forgiven for the sake of simplicity / developer effort. >> >> Well, speed.pypy is actually fast in all modern browsers. The problem >> you are referring to is probably caused by a bug in the javascript >> plotting library (jqPplot) that is triggered in the comparison view >> when there are some results with 0 values. It only appears for some >> plot types, but it is very annoying because it grinds the browser to a >> halt like you say. Is that what you meant? >> >> We are looking into it, and will fix that library if necessary. >> >> Cheers, >> Miquel >> >> >> 2010/12/21 Dima Tisnek : >>> On 20 December 2010 19:21, William ML Leslie >>> wrote: On 21 December 2010 11:59, Dima Tisnek wrote: > More visibility for performance achievements would do > good too. Where are pypy's performance achievements *not* visible, but should be? >>> >>> for example http://shootout.alioth.debian.org/ >>> doesn't say which pypy version is used, what options, doesn't have >>> performance figures for multithreaded/multicore >>> >>> also benchmarks are kinda small, most of them are not docuemented, and >>> I couldn't find any info if the same python code was used for cpython >>> and pypy (both shootout and speed pypy) or interpreter-specific >>> verions were used, that is each tweaked for best performance given the >>> known tradeoffs for each implementation.further the most standard >>> benchmarks, pystone, etc. completely ignore the fact that real world >>> programs are large and only a few small paths are execured often. >>> >>> another temp problem with speed pypy is that it's terrubly slow in ff >>> beta. it also occasionally grinds in stable ff and opera, but I guess >>> this can be forgiven for the sake of simplicity / developer effort. >>> >>> if you google for 'python performance' you don't get a single link to >>> pypy on the first page, as a matter of fact, codespeak is poorly >>> indexed, it took me quite some time to get some of my questions >>> answered with a search. also if you look up 'pypy gc' you get a page >>> on codespeak, but to interpret what the data actually means is so far >>> beyond me. >>> >>> a good overview is found in the mainling list >>> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >>> slowspitfire and spambayes bits are outdated by now. >>> >>> the definitive good thing about pypy is that it's much easier to find >>> out about its inner workings than that of cpython! >>> >>> hopefully a bit more of end-user stuff get known. >>> let's call it pypy public outreach (feature request) >>> > Sidetracking... one day when pypy jit/gc/etc are all worked out, how > hard would it be to use same techniques and most of backends for some > unrelated language that doesn't have jit yet, e.g. php? You know that pypy already has implementations of other languages, right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't integrated with the translated pypy-c, but they show that it is not too difficult to write a runtime for any dynamic language you choose. >>> >>> Oh I didn't know there were so many, and I mistakenly though that js >>> was a target, not implmented langauge. In any case I read somewhere >>> that js support was retired... >>> > And how hard > would it be to marry two dynamic languages, so that modules from one > could be used in the other? Or that modules written in rpython could > be used in several langs? It's in the "interesting problems" bucket, and the effort required depends on the level of integration between languages you want. There are several projects already attempting to do decent integration between several languages, besides the approach used on the JVM, there are also GNU Guile, Racket, and Parrot, among others. It might be worth waiting to see how these different projects pan out, before spending a bunch of effort just to be an also-ran in the multi-language runtime market. However, implementing more languages in rpython has the side-effect of propagating the l * o * p problem: it introduces more and more implementations that then have to be maintained, so good c
Re: [pypy-dev] [PATCH] Improving readability of generated .c code
On Tue, 2010-12-14 at 15:14 +0100, Carl Friedrich Bolz wrote: > Hi David, > > On 12/14/2010 01:57 AM, David Malcolm wrote: Thanks for looking at this; sorry for the belated reply. I'm attaching the latest work-in-progress version of the patch, addressing some of the areas discussed. (snip) > > Hopefully this makes the generated .c much more readable. > > I like the general approach, though I have the vague fear that it will > make the generated C code much bigger. Could you check that this is not > the case? The .c code seems to get roughly 15% bigger (which I don't think is an issue), but the translation may be getting significantly longer (40% more time; it's a real pain to benchmark this, though). I'm also not getting as many comments as I'd like for inlining (see below), and I expect the code to grow when I fix that. Method: Control: - clean pypy hg checkout of 40149:cd083843b67a - cd pypy/translator/goal - python translate.py (i.e. without any flags) - wait for the build to complete - locate the generated code in the "testing_1" subdirectory: - invoke David A. Wheeler's "SLOCCount" tool on the "testing_1" dir (http://www.dwheeler.com/sloccount/) - run "du -h" on the "testing_1" dir Experiment: - as above, but with the attached changes to the source tree Actually, the first time I built the clean version, I was hacking on the debug tree, and lost my copy of the relevant /tmp/usession-N directory, so I set PYPY_USESSION_DIR=/tmp/clean-build and redid it. (I was also checking mail, IRC, and other stuff on the machine, but I don't think this significantly affects the timings). Size of generated source:: Output of "wc *.c":: (trimmed to final summary, giving newline, word, and byte counts) Unpatched:: 5166080 15803859 186007458 total With the patch:: 5544105 18458780 213205652 total Disk usage:: This is: du -h -c *.c omitting the individual counts Unpatched:: 178M total With the patch:: 204M total "sloccount":: Unpatched:: SLOCDirectory SLOC-by-Language (Sorted) 4728922 testing_1 ansic=4728922 With the patch:: SLOCDirectory SLOC-by-Language (Sorted) 4730181 testing_1 ansic=4730181 I haven't measured memory usage during conversion. The above looks fine to me (I'll happily take a 15% increase in on-disk size of files that are used at runtime only for debugging, in exchange for easier debugging). However, conversion time seems to increase roughly 40%. Unpatched: [Timer] Timings: [Timer] annotate --- 1180.5 s [Timer] rtype_lltype --- 730.7 s [Timer] backendopt_lltype --- 468.5 s [Timer] stackcheckinsertion_lltype --- 68.9 s [Timer] database_c --- 546.4 s [Timer] source_c --- 1323.5 s [Timer] compile_c --- 347.4 s [Timer] === [Timer] Total: --- 4665.9 s Patched: [Timer] Timings: [Timer] annotate --- 1428.2 s [Timer] rtype_lltype --- 913.6 s [Timer] backendopt_lltype --- 831.9 s [Timer] stackcheckinsertion_lltype --- 106.6 s [Timer] database_c --- 991.1 s [Timer] source_c --- 2385.8 s [Timer] compile_c --- 400.2 s [Timer] === [Timer] Total: --- 7057.3 s > > (I hope to > > work on the names of locals also; ideally they ought to embed the python > > indentifier) > > This should already work in theory, see e.g. "l_cls_0" above, which > embeds the Python variable name "cls". I guess that the information is > simply lost in some of the transformation steps. If you feel like > hunting down where, you can probably get better naming in some cases. I think I was wrong about this: the variables that are getting uninformative names are the temporaries; .c locals that "are" RPython locals are being named informatively. > > > This is on a Fedora 13 x86_64 box, with cpython 2.6.5 > > > > Caveats: > >- this is my first patch to PyPy (please be gentle!): I hope I've > > correctly understood the various levels, but I suspect I may still be > > missing things at the conceptual level > > I think you are doing great. You might have to grep around a bit for > more places that create SpaceOperations, or did you look at all of them? There are numerous places where ops are created, (in llops.genops iirc), and somewhere in that process, many of the resulting operations are only getting a partial "path" of CodeLoc instances (see below). I plan to look into that next. > >- haven't completed the full unit tests yet. > > This is the biggest problem of the patch, of course. I think that you > should at least try to write tests for each of the places where you
Re: [pypy-dev] funding/popularity?
Yes it does grind ff 4b7 to an almost halt, little cpu usage, reasonable memory size and constant disk activity for several minutes, very weird... So far, the difference between browsers is so large that I doubt it's dependent on data. Also it seems to tirgger as I remove columns progressively, thus new zero values should not appear, right? I'll invesitage some more... On 21 December 2010 01:06, Miquel Torres wrote: > Hi Dima, > >> another temp problem with speed pypy is that it's terrubly slow in ff >> beta. it also occasionally grinds in stable ff and opera, but I guess >> this can be forgiven for the sake of simplicity / developer effort. > > Well, speed.pypy is actually fast in all modern browsers. The problem > you are referring to is probably caused by a bug in the javascript > plotting library (jqPplot) that is triggered in the comparison view > when there are some results with 0 values. It only appears for some > plot types, but it is very annoying because it grinds the browser to a > halt like you say. Is that what you meant? > > We are looking into it, and will fix that library if necessary. > > Cheers, > Miquel > > > 2010/12/21 Dima Tisnek : >> On 20 December 2010 19:21, William ML Leslie >> wrote: >>> On 21 December 2010 11:59, Dima Tisnek wrote: More visibility for performance achievements would do good too. >>> >>> Where are pypy's performance achievements *not* visible, but should be? >> >> for example http://shootout.alioth.debian.org/ >> doesn't say which pypy version is used, what options, doesn't have >> performance figures for multithreaded/multicore >> >> also benchmarks are kinda small, most of them are not docuemented, and >> I couldn't find any info if the same python code was used for cpython >> and pypy (both shootout and speed pypy) or interpreter-specific >> verions were used, that is each tweaked for best performance given the >> known tradeoffs for each implementation.further the most standard >> benchmarks, pystone, etc. completely ignore the fact that real world >> programs are large and only a few small paths are execured often. >> >> another temp problem with speed pypy is that it's terrubly slow in ff >> beta. it also occasionally grinds in stable ff and opera, but I guess >> this can be forgiven for the sake of simplicity / developer effort. >> >> if you google for 'python performance' you don't get a single link to >> pypy on the first page, as a matter of fact, codespeak is poorly >> indexed, it took me quite some time to get some of my questions >> answered with a search. also if you look up 'pypy gc' you get a page >> on codespeak, but to interpret what the data actually means is so far >> beyond me. >> >> a good overview is found in the mainling list >> http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again >> slowspitfire and spambayes bits are outdated by now. >> >> the definitive good thing about pypy is that it's much easier to find >> out about its inner workings than that of cpython! >> >> hopefully a bit more of end-user stuff get known. >> let's call it pypy public outreach (feature request) >> >>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how hard would it be to use same techniques and most of backends for some unrelated language that doesn't have jit yet, e.g. php? >>> >>> You know that pypy already has implementations of other languages, >>> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >>> integrated with the translated pypy-c, but they show that it is not >>> too difficult to write a runtime for any dynamic language you choose. >> >> Oh I didn't know there were so many, and I mistakenly though that js >> was a target, not implmented langauge. In any case I read somewhere >> that js support was retired... >> >>> And how hard would it be to marry two dynamic languages, so that modules from one could be used in the other? Or that modules written in rpython could be used in several langs? >>> >>> It's in the "interesting problems" bucket, and the effort required >>> depends on the level of integration between languages you want. There >>> are several projects already attempting to do decent integration >>> between several languages, besides the approach used on the JVM, there >>> are also GNU Guile, Racket, and Parrot, among others. It might be >>> worth waiting to see how these different projects pan out, before >>> spending a bunch of effort just to be an also-ran in the >>> multi-language runtime market. >>> >>> However, implementing more languages in rpython has the side-effect of >>> propagating the l * o * p problem: it introduces more and more >>> implementations that then have to be maintained, so good >>> cross-language integration probably belongs /outside/ pypy itself, so >>> existing runtimes can hook into it. >> >> Makes perfect sense, after all any given other language hardly has the >> same data model as python. >> >>> >>> But it woul
[pypy-dev] PyPy 1.4.1
Hi all, Here is PyPy 1.4.1 :-) Note that the Windows and the Mac OS X 64 builds are still missing; we are waiting for contributions here (amaury, mvt? :-). A bientôt, Armin. === PyPy 1.4.1 === We're pleased to announce the 1.4.1 release of PyPy. This release consolidates all the bug fixes that occurred since the previous release. To everyone that took the trouble to report them, we want to say thank you. http://pypy.org/download.html What is PyPy PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython. Note that it still only emulates Python 2.5 by default; the ``fast-forward`` branch with Python 2.7 support is slowly getting ready but will only be integrated in the next release. In two words, the advantage of trying out PyPy instead of CPython (the default implementation of Python) is, for now, the performance. Not all programs are faster in PyPy, but we are confident that any CPU-intensive task will be much faster, at least if it runs for long enough (the JIT has a slow warm-up phase, which can take several seconds or even one minute on the largest programs). Note again that we do support compiling and using C extension modules from CPython (``pypy setup.py install``). However, this is still an alpha feature, and the most complex modules typically fail for various reasons; others work (e.g. ``PIL``) but take a serious performance hit. Also, for Mac OS X see below. Please note also that PyPy's performance was optimized almost exclusively on Linux. It seems from some reports that on Windows as well as Mac OS X (probably for different reasons) the performance might be lower. We did not investigate much so far. More highlights === * We migrated to Mercurial (thanks to Ronny Pfannschmidt and Antonio Cuni) for the effort) and moved to bitbucket. The new command to check out a copy of PyPy is:: hg clone http://bitbucket.org/pypy/pypy * In long-running processes, the assembler generated by old JIT-compilations is now freed. There should be no more leak, however long the process runs. * Improve a lot the performance of the ``binascii`` module, and of ``hashlib.md5`` and ``hashlib.sha``. * Made sys.setrecursionlimit() a no-op. Instead, we rely purely on the built-in stack overflow detection mechanism, which also gives you a RuntimeError -- just not at some exact recursion level. * Fix argument processing (now e.g. ``pypy -OScpass`` works like it does on CPython --- if you have a clue what it does there ``:-)`` ) * cpyext on Mac OS X: it still does not seem to work. I get systematically a segfault in dlopen(). Contributions welcome. * Fix two corner cases in the GC (one in minimark, one in asmgcc+JIT). This notably prevented "pypy translate.py -Ojit" from working on Windows, leading to crashes. * Fixed a corner case in the JIT's optimizer, leading to "Fatal RPython error: AssertionError". * Added some missing built-in functions into the 'os' module. * Fix ctypes (it was not propagating keepalive information from c_void_p). Cheers, Armin Rigo, for the rest of the team ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] Fwd: [IronPython] SciPy
> I thought this email could be relevant for those interested in SciPy / Numpy > on pypy. With enthought implementing a smaller core and using compatibility > layers for alternative platforms, it would seem to be a good basis for a > pypy port. All I can do is give a BIG +1 to anything that can get numpy/scipy up more quickly. PyPy is starting to give us the speed we need for statistical/scientific work on python (the speed it still lacks compared to C or Java is made up for, for my purposes, by the ease of writing python compared to those languages). The recent 64-bit functionality lets us process a lot of data. The fast-forward branch is giving us the multiprocessing we need. (I recognize that there are other solutions, but for simple things we need to write quickly, the multiprocessing module is really sweet.) The main thing missing now is numpy/scipy. The addition of that will make PyPy a huge win in the scientific community, IMO. Anyway, I mention it in case the opinion of one person who is using Python in production for statistical processing is of interest. -- Gary Robinson CTO Emergent Discovery, LLC personal email: gary...@me.com work email: grobin...@emergentdiscovery.com Company: http://www.emergentdiscovery.com Blog:http://www.garyrobinson.net ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] Fwd: [IronPython] SciPy
I thought this email could be relevant for those interested in SciPy / Numpy on pypy. With enthought implementing a smaller core and using compatibility layers for alternative platforms, it would seem to be a good basis for a pypy port. -- Forwarded message -- From: Jason McCampbell Date: 20 December 2010 15:13 Subject: Re: [IronPython] SciPy To: Discussion of IronPython Hi Mark, As Dino mentioned we (Enthought) are working on refactoring Numpy into a pure "C" core with CPython and IronPython interface layers. This is largely complete and available at github (https://github.com/numpy/numpy-refactor), though the core layer is largely undocumented thus far. This is the multi-dimensional array. SciPy is in progress and we are updating it to work with the refactored numpy core and to add an IronPython interface. I assume you are looking for IronPython interfaces to SciPy as opposed to a C interface, correct? Regards, Jason On Thu, Dec 16, 2010 at 1:56 PM, Dino Viehland wrote: > Enthought has been working on getting numpy/scipy ported over to work w/ > IronPython. I believe numpy is working but I’m not sure of how far along > SciPy is. There’s a separate mailing list for this at: > > > > https://mail.enthought.com/mailman/listinfo/scipy4dotnet > > > > It’s very low traffic – it’s usually just working through issues Enthought > has run into and either workarounds or suggested changes to IronPython. I’d > suggest sending a mail there – they might have something you can try. > > > > > > *From:* users-boun...@lists.ironpython.com [mailto: > users-boun...@lists.ironpython.com] *On Behalf Of *Mark Senko > *Sent:* Thursday, December 16, 2010 11:49 AM > *To:* us...@lists.ironpython.com > *Subject:* [IronPython] SciPy > > > > I’ve been searching for the current state of support for “C” based > libraries, specifically SciPy (I’m just looking for a decent numerical > analysis package). The responses I’ve seen on various websites are somewhat > dated. > > What is the latest status, or is there no effort towards accommodating the > C API? Is IronClad still the best option? Any info, suggestions and warnings > would be appreciated before I start to invest a lot of time into installing > and learning these packages. > > > > *Mark Senko* > > Complete Genomics, Inc. > > 2071 Stierlin Court > > Mountain View, CA 94043 > > > > > > > > > > > > The contents of this e-mail and any attachments are confidential and only for > > use by the intended recipient. Any unauthorized use, distribution or copying > > of this message is strictly prohibited. If you are not the intended recipient > > please inform the sender immediately by reply e-mail and delete this message > > from your system. Thank you for your co-operation. > > > ___ > Users mailing list > us...@lists.ironpython.com > http://lists.ironpython.com/listinfo.cgi/users-ironpython.com > > ___ Users mailing list us...@lists.ironpython.com http://lists.ironpython.com/listinfo.cgi/users-ironpython.com -- http://www.voidspace.org.uk ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
Hi René, On Tue, Dec 21, 2010 at 1:24 PM, René Dudfield wrote: > sys.getsizeof(obj) Ah, thank you, didn't know about it. It's a 2.6 feature only though. A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
sys.getsizeof(obj) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
Hi, On Tue, Dec 21, 2010 at 9:58 AM, Arnd Rechenburg wrote: > In my code I need something like > > long.__itemsize__ > > in Python. I suppose the question is "why"? This is supposed to be the size in bytes occupied by one "element" of the type, and one element of "long" happens to be 15 bits, so that's why long.__itemsize__ is 2. It has no other meaning. If you are interested in an estimate of how many bytes some actual object takes, we have some functions in the 'gc' module, but even that is incomplete -- there are cases were it's hard to come up with a definitive answer, like objects based on "mapdicts". A bientôt, Armin. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
On Tue, Dec 21, 2010 at 1:31 PM, Antonio Cuni wrote: > On 21/12/10 12:05, Maciej Fijalkowski wrote: > >>> __itemsize__ - in bytes, corresponds to item size field in the types >>> definition structure. >>> >>> It's a field for types. >>> See: >>> http://docs.python.org/c-api/typeobj.html#tp_itemsize >>> >> >> Well... Those are docs for C API. It doesn't say it's exposed at >> applevel nor since which version. (Also, to be precise, C API is known >> to be implementation specific) > > Moreover, I don't think we could give it a sane semantics on PyPy, given that > the same applevel type can be potentially implemented by many different low > level structures with different sizes. > Not even "potentially", it actually is in some places. ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
On 21/12/10 12:05, Maciej Fijalkowski wrote: >> __itemsize__ - in bytes, corresponds to item size field in the types >> definition structure. >> >> It's a field for types. >> See: >>http://docs.python.org/c-api/typeobj.html#tp_itemsize >> > > Well... Those are docs for C API. It doesn't say it's exposed at > applevel nor since which version. (Also, to be precise, C API is known > to be implementation specific) Moreover, I don't think we could give it a sane semantics on PyPy, given that the same applevel type can be potentially implemented by many different low level structures with different sizes. ciao, Anto ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
On Tue, Dec 21, 2010 at 1:02 PM, René Dudfield wrote: > Hi, > > __itemsize__ - in bytes, corresponds to item size field in the types > definition structure. > > It's a field for types. > See: > http://docs.python.org/c-api/typeobj.html#tp_itemsize > Well... Those are docs for C API. It doesn't say it's exposed at applevel nor since which version. (Also, to be precise, C API is known to be implementation specific) ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
Hi, __itemsize__ - in bytes, corresponds to item size field in the types definition structure. It's a field for types. See: http://docs.python.org/c-api/typeobj.html#tp_itemsize On Tue, Dec 21, 2010 at 9:36 AM, Maciej Fijalkowski wrote: > > On Tue, Dec 21, 2010 at 10:58 AM, Arnd Rechenburg > wrote: > > Hi, > > > > > > > > In my code I need something like > > > > long.__itemsize__ > > > > in Python. > > > > > > > > Any suggestions to do it with pypy? > > > > Hey. > > This attribute is undocumented and untested as far as I can tell. What > does it do? > > Cheers, > fijal > > > > > > > > Regards, > > > > Arnd > > > > ___ > > pypy-dev@codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > ___ > pypy-dev@codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] long.__itemsize__
On Tue, Dec 21, 2010 at 10:58 AM, Arnd Rechenburg wrote: > Hi, > > > > In my code I need something like > > long.__itemsize__ > > in Python. > > > > Any suggestions to do it with pypy? > Hey. This attribute is undocumented and untested as far as I can tell. What does it do? Cheers, fijal > > > Regards, > > Arnd > > ___ > pypy-dev@codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
[pypy-dev] long.__itemsize__
Hi, In my code I need something like long.__itemsize__ in Python. Any suggestions to do it with pypy? Regards, Arnd ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] ImportError: No module named _sqlite3
Hi, I saw my mistake already by myself. Regards, Arnd From: Arnd Rechenburg Sent: Montag, 20. Dezember 2010 19:16 To: 'pypy-dev@codespeak.net' Subject: ImportError: No module named _sqlite3 Hi, I tried to use sqlite3 with pypy but I got the following error: import sqlite Traceback (most recent call last): File "", line 1, in ImportError: No module named sqlite import sqlite3 Traceback (most recent call last): File "", line 1, in File "sqlite3/__init__.py", line 24, in from dbapi2 import * File "sqlite3/dbapi2.py", line 27, in from _sqlite3 import * ImportError: No module named _sqlite3 Someone knows where to put the _sqlite3.so library? Any help would be appreciated. Thx in advance, Arnd Rechenburg ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Re: [pypy-dev] funding/popularity?
Hi Dima, > another temp problem with speed pypy is that it's terrubly slow in ff > beta. it also occasionally grinds in stable ff and opera, but I guess > this can be forgiven for the sake of simplicity / developer effort. Well, speed.pypy is actually fast in all modern browsers. The problem you are referring to is probably caused by a bug in the javascript plotting library (jqPplot) that is triggered in the comparison view when there are some results with 0 values. It only appears for some plot types, but it is very annoying because it grinds the browser to a halt like you say. Is that what you meant? We are looking into it, and will fix that library if necessary. Cheers, Miquel 2010/12/21 Dima Tisnek : > On 20 December 2010 19:21, William ML Leslie > wrote: >> On 21 December 2010 11:59, Dima Tisnek wrote: >>> More visibility for performance achievements would do >>> good too. >> >> Where are pypy's performance achievements *not* visible, but should be? > > for example http://shootout.alioth.debian.org/ > doesn't say which pypy version is used, what options, doesn't have > performance figures for multithreaded/multicore > > also benchmarks are kinda small, most of them are not docuemented, and > I couldn't find any info if the same python code was used for cpython > and pypy (both shootout and speed pypy) or interpreter-specific > verions were used, that is each tweaked for best performance given the > known tradeoffs for each implementation.further the most standard > benchmarks, pystone, etc. completely ignore the fact that real world > programs are large and only a few small paths are execured often. > > another temp problem with speed pypy is that it's terrubly slow in ff > beta. it also occasionally grinds in stable ff and opera, but I guess > this can be forgiven for the sake of simplicity / developer effort. > > if you google for 'python performance' you don't get a single link to > pypy on the first page, as a matter of fact, codespeak is poorly > indexed, it took me quite some time to get some of my questions > answered with a search. also if you look up 'pypy gc' you get a page > on codespeak, but to interpret what the data actually means is so far > beyond me. > > a good overview is found in the mainling list > http://codespeak.net/pipermail/pypy-dev/2010q1/005757.html then again > slowspitfire and spambayes bits are outdated by now. > > the definitive good thing about pypy is that it's much easier to find > out about its inner workings than that of cpython! > > hopefully a bit more of end-user stuff get known. > let's call it pypy public outreach (feature request) > >> >>> Sidetracking... one day when pypy jit/gc/etc are all worked out, how >>> hard would it be to use same techniques and most of backends for some >>> unrelated language that doesn't have jit yet, e.g. php? >> >> You know that pypy already has implementations of other languages, >> right - Scheme, Javascript, Prolog, IO and smallTalk? They aren't >> integrated with the translated pypy-c, but they show that it is not >> too difficult to write a runtime for any dynamic language you choose. > > Oh I didn't know there were so many, and I mistakenly though that js > was a target, not implmented langauge. In any case I read somewhere > that js support was retired... > >> >>> And how hard >>> would it be to marry two dynamic languages, so that modules from one >>> could be used in the other? Or that modules written in rpython could >>> be used in several langs? >> >> It's in the "interesting problems" bucket, and the effort required >> depends on the level of integration between languages you want. There >> are several projects already attempting to do decent integration >> between several languages, besides the approach used on the JVM, there >> are also GNU Guile, Racket, and Parrot, among others. It might be >> worth waiting to see how these different projects pan out, before >> spending a bunch of effort just to be an also-ran in the >> multi-language runtime market. >> >> However, implementing more languages in rpython has the side-effect of >> propagating the l * o * p problem: it introduces more and more >> implementations that then have to be maintained, so good >> cross-language integration probably belongs /outside/ pypy itself, so >> existing runtimes can hook into it. > > Makes perfect sense, after all any given other language hardly has the > same data model as python. > >> >> But it would be an interesting experiment (to marry the various >> interpreters pypy ships with), if you wanted to try it. >> >> My two cents. >> >> -- >> William Leslie >> > ___ > pypy-dev@codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev ___ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
