Re: [Python-Dev] A suggestion: Do proto-PEPs in Google Docs
On Thu, Feb 19, 2009 at 7:17 PM, Stephen J. Turnbull wrote: > On Python-Ideas, Guido van Rossum writes: > > > On Thu, Feb 19, 2009 at 2:12 AM, Greg Ewing wrote: > > > > Fifth draft of the PEP. Re-worded a few things slightly > > > to hopefully make the proposal a bit clearer up front. > > > > Wow, how I long for the days when we routinely put things like this > > under revision control so its easy to compare versions. > > FWIW, Google Docs is almost there. Working with Brett et al on early > drafts of PEP 0374 was easy and pleasant, and Google Docs gives > control of access to the document to the editor, not the Subversion > admin. The ability to make comments that are not visible to > non-editors was nice. Now that it's in Subversion it's much less > convenient for me (a non-committer). I actually have to *decide* to > work on it, rather than simply raising a browser window, hitting > "refresh" and fixing a typo or two (then back to "day job" work). > > The main problem with Google Docs is that is records a revision > automatically every so often (good) but doesn't prune the automatic > commits (possibly hard to do efficiently) OR mark user saves specially > (easy to do). This lack of marking "important" revisions makes the > diff functionality kind of tedious. > > I don't know how automatic the conversion to reST was, but the PEP in > Subversion is a quite accurate conversion of the Google Doc version. > > Overall, I recommend use of Google Docs for "Python-Ideas" level of > PEP drafts. Rietveld would also be a good option: it offers more at-will revision control (rather than "whenever Google Docs decides"), allows you to attach comments to the revisions, and will give you nice diffs between PEP iterations. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] IO implementation: in C and Python?
On Thu, Feb 19, 2009 at 9:07 PM, Guido van Rossum wrote: > On Thu, Feb 19, 2009 at 8:38 PM, Brett Cannon wrote: >> On Thu, Feb 19, 2009 at 19:41, Benjamin Peterson >> wrote: >>> As we prepare to merge the io-c branch, the question has come up [1] >>> about the original Python implementation. Should it just be deleted in >>> favor C version? The wish to maintain the two implementations together >>> has been raised on the basis that Python is easier to experiment on >>> and read (for other vm implementors). >> >> Probably not a surprise, but +1 from me for keeping the pure Python version >> around for the benefit of other VMs as well as a reference implementation. > > You have been practice channeling me again, haven't you? I like the > idea of having two (closely matching) implementations very much. Agreed. In particular, this helps any projects that are focused on improving the performance of pure-Python code: they can work on minimizing the delta between the Python and C versions. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Pickler/Unpickler API clarification
I'm working on some performance patches for cPickle, and one of the bigger wins so far has been replacing the Pickler's memo dict with a custom hashtable (and hence removing memo's getters and setters). In looking over this, Jeffrey Yasskin commented that this would break anyone who was accessing the memo attribute. I've found a few examples of code using the memo attribute ([1], [2], [3]), and there are probably more out there, but the memo attribute doesn't look like part of the API to me. It's only documented in http://docs.python.org/library/pickle.html as "you used to need this before Python 2.3, but don't anymore". However: I don't believe you should ever need this attribute. The usages of memo I've seen break down into two camps: clearing the memo, and wanting to explicitly populate the memo with predefined values. Clearing the memo is recommended as part of reusing Pickler objects, but I can't fathom when you would want to reuse a Pickler *without* clearing the memo. Reusing the Pickler without clearing the memo will produce pickles that are, as best I can see, invalid -- at least, pickletools.dis() rejects this, which is the closest thing we have to a validator. Explicitly setting memo values has the same problem: an easy, very brittle way to produce invalid data. So my questions are these: 1) Should Pickler/Unpickler objects automatically clear their memos when dumping/loading? 2) Is memo an intentionally exposed, supported part of the Pickler/Unpickler API, despite the lack of documentation and tests? Thanks, Collin [1] - http://google.com/codesearch/p?hl=en#Qx8E-7HUBTk/trunk/google/appengine/api/memcache/__init__.py&q=lang:py%20%5C.memo [2] - http://google.com/codesearch/p?hl=en#M-DDI-lCOgE/lib/python2.4/site-packages/cvs2svn_lib/primed_pickle.py&q=lang:py%20%5C.memo [3] - http://google.com/codesearch/p?hl=en#l_w_cA4dKMY/AtlasAnalysis/2.0.3-LST-1/PhysicsAnalysis/PyAnalysis/PyAnalysisUtils/python/root_pickle.py&q=lang:py%20pick.*%5C.memo%5Cb ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pickler/Unpickler API clarification
On Fri, Mar 6, 2009 at 10:01 AM, Michael Haggerty wrote: > Antoine Pitrou wrote: >> Le vendredi 06 mars 2009 à 13:44 +0100, Michael Haggerty a écrit : >>> Antoine Pitrou wrote: Michael Haggerty alum.mit.edu> writes: > It is easy to optimize the pickling of instances by giving them > __getstate__() and __setstate__() methods. But the pickler still > records the type of each object (essentially, the name of its class) in > each record. The space for these strings constituted a large fraction > of the database size. If these strings are not interned, then perhaps they should be. There is a similar optimization proposal (w/ patch) for attribute names: http://bugs.python.org/issue5084 >>> If I understand correctly, this would not help: >>> >>> - on writing, the strings are identical anyway, because they are read >>> out of the class's __name__ and __module__ fields. Therefore the >>> Pickler's usual memoizing behavior will prevent the strings from being >>> written more than once. >> >> Then why did you say that "the space for these strings constituted a >> large fraction of the database size", if they are already shared? Are >> your objects so tiny that even the space taken by the pointer to the >> type name grows the size of the database significantly? > > Sorry for the confusion. I thought you were suggesting the change to > help the more typical use case, when a single Pickler is used for a lot > of data. That use case will not be helped by interning the class > __name__ and __module__ strings, for the reasons given in my previous email. > > In my case, the strings are shared via the Pickler memoizing mechanism > because I pre-populate the memo (using the API that the OP proposes to > remove), so your suggestion won't help my current code, either. It was > before I implemented the pre-populated memoizer that "the space for > these strings constituted a large fraction of the database size". But > your suggestion wouldn't help that case, either. > > Here are the main use cases: > > 1. Saving and loading one large record. A class's __name__ string is > the same string object every time it is retrieved, so it only needs to > be stored once and the Pickler memo mechanism works. Similarly for the > class's __module__ string. > > 2. Saving and loading lots of records sequentially. Provided a single > Pickler is used for all records and its memo is never cleared, this > works just as well as case 1. > > 3. Saving and loading lots of records in random order, as for example in > the shelve module. It is not possible to reuse a Pickler with retained > memo, because the Unpickler might not encounter objects in the right > order. There are two subcases: > > a. Use a clean Pickler/Unpickler object for each record. In this > case the __name__ and __module__ of a class will appear once in each > record in which the class appears. (This is the case regardless of > whether they are interned.) On reading, the __name__ and __module__ are > only used to look up the class, so interning them won't help. It is > thus impossible to avoid wasting a lot of space in the database. > > b. Use a Pickler/Unpickler with a preset memo for each record (my > unorthodox technique). In this case the class __name__ and __module__ > will be memoized in the shared memo, so in other records only their ID > needs to be stored (in fact, only the ID of the class object itself). > This allows the database to be smaller, but does not have any effect on > the RAM usage of the loaded objects. > > If the OP's proposal is accepted, 3b will become impossible. The > technique seems not to be well known, so maybe it doesn't need to be > supported. It would mean some extra work for me on the cvs2svn project > though :-( Talking it over with Guido, support for the memo attribute will have to stay. I shall add it back to my patches. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.1 performance
On Sun, Mar 8, 2009 at 7:30 AM, Christian Heimes wrote: > Antoine Pitrou wrote: >> Hi, >> >> Victor Stinner haypocalc.com> writes: >>> Summary (minimum total) on 32 bits CPU: >>> * Python 2.6.1: 8762 ms >>> * Python 3.0.1: 8977 ms >>> * Python 3.1a1: 9228 ms (slower than 3.0) >> >> Have you compiled with or without "--with-computed-gotos"? > > Why is the feature still disabled by default? > > Christian > > PS: Holy moly! Computed gotos totally put my Python on fire! The feature > increases the minimum run-time by approx. 25% and the average run-time > by approx. 40% on my Ubuntu 8.10 box (AMD64, Intel(R) Core(TM)2 CPU > T7600 @ 2.33GHz). Note that of the benchmarks tested, PyBench benefits the most from threaded eval loop designs. Other systems benefit less; for example, Django template benchmarks were only sped up by 7-8% when I was testing it. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] speeding up PyObject_GetItem
2009/3/24 Daniel Stutzbach : > On Tue, Mar 24, 2009 at 10:13 AM, Mark Dickinson wrote: >> >> 2009/3/24 Daniel Stutzbach : >> > [...] >> > 100 nanoseconds, py3k trunk: >> > ceval -> PyObject_GetItem (object.c) -> list_subscript (listobject.c) -> >> > PyNumber_AsSsize_t (object.c) -> PyLong_AsSsize_t (longobject.c) >> > [more timings snipped] >> >> Does removing the PyLong_Check call in PyLong_AsSsize_t >> make any noticeable difference to these timings? > > Making no other changes from the trunk, removing the PyLong_Check and NULL > check from PyLong_AsSsize_t shaves off 4 nanoseconds (or around 4% since the > trunk is around 100 nanoseconds). > > Here's what I'm testing with, by the way: > > ./python.exe Lib/timeit.py -r 10 -s 'x = list(range(10))' 'x[5]' What difference does it make on real applications? Are you running any macro-benchmarks against this? Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GPython?
On Thu, Mar 26, 2009 at 8:05 PM, Terry Reedy wrote: > An ars technica articla just linked to in a python-list post > > http://arstechnica.com/open-source/news/2009/03/google-launches-project-to-boost-python-performance-by-5x.ars > > calls the following project "Google launched" > http://code.google.com/p/unladen-swallow/wiki/ProjectPlan > > (Though the project page does not really claim that.) Hi, I'm the tech lead for Unladen Swallow. Jeffrey Yasskin and Thomas Wouters are also working on this project. Unladen Swallow is Google-sponsored, but not Google-owned. This is an open-source branch that we're working on, focused on performance, and we want to move all of our work upstream as quickly as possible. In fact, right now I'm adding a last few tests before putting our cPickle patches up on the tracker for further review. > I am sure some people here might find this interesting. > > I'd love to have a faster CPython, but this note: > "Will probably kill Python Windows support (for now)." > would kill merger back into mainline (for now) without one opposing being > 'conservative'. To clarify, when I wrote 'conservative', I wasn't being disparaging. A resistance to change can certainly be a good thing, and something that I think is very healthy in these situations. We certainly have to prove ourselves, especially given some of the fairly radical things we're thinking of [1]. We believe we can justify these changes, but I *do* want to be forced to justify them publicly; I don't think python-dev would be doing its job if some of these things were merely accepted without discussion. In particular, Windows support is one of those things we'll need to address on our end. LLVM's Windows support may be spotty, or there may be other Windows issues we inadvertently introduce. None of the three of us have Windows machines, nor do we particularly want to acquire them :), and Windows support isn't going to be a big priority. If we find that some of our patches have Windows issues, we will certainly fix those before proposing their inclusion in CPython. > If one adds type annotations so that values can be unboxed, would not > Cython, etc, do even better for speedup? Possibly, but we want to see how far we can push the current language before we even start thinking of tinkering with the language spec. Assigning meaning to function annotations is something that PEP 3107 explicitly avoids, and I'm not sure Unladen Swallow (or anyone else) would want to take the plunge into coming up with broadly-acceptable type systems for Python. That would be a bikeshed discussion of such magnitude, you'd have to invent new colors to paint the thing. Collin Winter [1] - http://code.google.com/p/unladen-swallow/wiki/ProjectPlan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GPython?
On Thu, Mar 26, 2009 at 11:26 PM, Alexandre Vassalotti wrote: > On Thu, Mar 26, 2009 at 11:40 PM, Collin Winter wrote: >> In fact, right now I'm adding a last few tests before putting our cPickle >> patches up on the tracker for further review. >> > > Put me in the nosy list when you do; and when I get some free time, I > will give your patches a complete review. I've already taken a quick > look at cPickle changes you did in Unladen and I think some (i.e., the > custom memo table) are definitely worthy to be merged in the > mainlines. Will do, thanks for volunteering! jyasskin has already reviewed them internally, but it'll be good to put them through another set of eyes. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] GPython?
On Fri, Mar 27, 2009 at 5:50 AM, Paul Moore wrote: > 2009/3/27 Collin Winter : >> In particular, Windows support is one of those things we'll need to >> address on our end. LLVM's Windows support may be spotty, or there may >> be other Windows issues we inadvertently introduce. None of the three >> of us have Windows machines, nor do we particularly want to acquire >> them :), and Windows support isn't going to be a big priority. If we >> find that some of our patches have Windows issues, we will certainly >> fix those before proposing their inclusion in CPython. > > On the assumption (sorry, I've done little more than read the press > releases so far) that you're starting from the CPython base and > incrementally patching things, you currently have strong Windows > support. It would be a shame if that got gradually chipped away > through neglect, until it became a big job to reinstate it. That's correct, we're starting with CPython 2.6.1. > If the Unladen Swallow team doesn't include any Windows developers, > you're a bit stuck, I guess, but could you not at least have a Windows > buildbot which keeps tabs on the current status? Then you might > encourage interested Windows bystanders to check in occasionally and > maybe offer fixes. We're definitely going to set up buildslaves for Windows and other platforms (currently we're only running Linux buildslaves). We're trying to solicit 20% time help from Google Windows developers, but that experience is relatively rare compared to the vast sea of Linux-focused engineers (though that's true of the open-source community in general). Also, it may be that some of the components we're reusing don't support Windows, or perhaps worse, offer degraded performance on Windows. We believe we can fix these problems as they come up -- we certainly don't want Windows issues to prevent patches from going into mainline -- but it's still a risk that Windows issues may slow down our development or prevent us from doing something fancy down the road, and I wanted to be up front about that risk. I've updated our ProjectPlan in hopes of clarifying this. That section of the docs was copy/pasted off a slide, and was a bit too terse :) Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Partial 2to3?
2009/3/27 :
> Following up on yesterday's conversation about 2to3 and 3to2, I wonder if
> it's easily possible to run 2to3 with a specific small subset of its fixers?
> For example, people not wanting to make the 2->3 leap yet might still be
> interersted in the exception handling changes ("except Foo as exc")?
Sure, that's easily possible: run 2to3 -f
some_fixer,other_fixer,this_fixer,that_fixer. You can get a full list
of fixers using the --list-fixes option.
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3to2 Project
On Mon, Mar 30, 2009 at 7:44 AM, Jesse Noller wrote: > During the Language summit this past Thursday, pretty much everyone > agreed that a python 3 to python 2 tool would be a very large > improvement in helping developers be able to write "pure" python 3 > code. The idea being a large project such as Django could completely > cut over to Python3, but then run the 3to2 tool on the code based to > continue to support version 2.x. > > I raised my hand to help move this along, I've spoke to Benjamin > Peterson, and he's amendable to mentoring a GSoC student for this > project and he's already received at least one proposal for this. > > Additionally, there's been a number of developers here at PyCon who > are more than ready to help contribute. > > So, if people are interested in helping, coordinating work/etc - feel > free to sync up with Benjamin - he's started a wiki page here: > > http://wiki.python.org/moin/3to2 If anyone is interested in working on this during the PyCon sprints or otherwise, here are some easy, concrete starter projects that would really help move this along: - The core refactoring engine needs to be broken out from 2to3. In particular, the tests/ and fixes/ need to get pulled up a directory, out of lib2to3/. - Once that's done, lib2to3 should then be renamed to something like librefactor or something else that indicates its more general nature. This will allow both 2to3 and 3to2 to more easily share the core components. - If you're more performance-minded, 2to3 and 3to2 would benefit heavily from some work on the pattern matching system. The current pattern matcher is a fairly simple AST interpreter; compiling the patterns down to pure Python code would be a win, I believe. This is all pretty heavily tested, so you wouldn't run much risk of breaking it. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Wed, Apr 1, 2009 at 4:29 PM, John Ehresman wrote: > I've written a proof of concept patch to add a hook to PyDict_SetItem at > http://bugs.python.org/issue5654 My motivation is to enable watchpoints in > a python debugger that are called when an attribute or global changes. I > know that this won't cover function locals and objects with slots (as Martin > pointed out). > > We talked about this at the sprints and a few issues came up: > > * Is this worth it for debugger watchpoint support? This is a feature that > probably wouldn't be used regularly but is extremely useful in some > situations. > > * Would it be better to create a namespace dict subclass of dict, use it for > modules, classes, & instances, and only allow watches of the subclass > instances? > > * To what extent should non-debugger code use the hook? At one end of the > spectrum, the hook could be made readily available for non-debug use and at > the other end, it could be documented as being debug only, disabled in > python -O, & not exposed in the stdlib to python code. Have you measured the impact on performance? Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Fri, Apr 3, 2009 at 2:27 AM, Antoine Pitrou wrote: > Thomas Wouters python.org> writes: >> >> >> Pystone is pretty much a useless benchmark. If it measures anything, it's the > speed of the bytecode dispatcher (and it doesn't measure it particularly > well.) > PyBench isn't any better, in my experience. > > I don't think pybench is useless. It gives a lot of performance data about > crucial internal operations of the interpreter. It is of course very little > real-world, but conversely makes you know immediately where a performance > regression has happened. (by contrast, if you witness a regression in a > high-level benchmark, you still have a lot of investigation to do to find out > where exactly something bad happened) > > Perhaps someone should start maintaining a suite of benchmarks, high-level and > low-level; we currently have them all scattered around (pybench, pystone, > stringbench, richard, iobench, and the various Unladen Swallow benchmarks; not > to mention other third-party stuff that can be found in e.g. the Computer > Language Shootout). Already in the works :) As part of the common standard library and test suite that we agreed on at the PyCon language summit last week, we're going to include a common benchmark suite that all Python implementations can share. This is still some months off, though, so there'll be plenty of time to bikeshed^Wrationally discuss which benchmarks should go in there. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Fri, Apr 3, 2009 at 9:43 AM, Antoine Pitrou wrote: > Thomas Wouters python.org> writes: >> >> Really? Have you tried it? I get at least 5% noise between runs without any > changes. I have gotten results that include *negative* run times. > > That's an implementation problem, not an issue with the tests themselves. > Perhaps a better timing mechanism could be inspired from the timeit module. > Perhaps the default numbers of iterations should be higher (many subtests run > in less than 100ms on a modern CPU, which might be too low for accurate > measurement). Perhaps the so-called "calibration" should just be disabled. > etc. > >> The tests in PyBench are not micro-benchmarks (they do way too much for > that), > > Then I wonder what you call a micro-benchmark. Should it involve direct calls > to > low-level C API functions? I agree that a suite of microbenchmarks is supremely useful: I would very much like to be able to isolate, say, raise statement performance. PyBench suffers from implementation defects that in its current incarnation make it unsuitable for this, though: - It does not effectively isolate component performance as it claims. When I was working on a change to BINARY_MODULO to make string formatting faster, PyBench would report that floating point math got slower, or that generator yields got slower. There is a lot of random noise in the results. - We have observed overall performance swings of 10-15% between runs on the same machine, using the same Python binary. Using the same binary on the same unloaded machine should give as close an answer to 0% as possible. - I wish PyBench actually did more isolation. Call.py:ComplexPythonFunctionCalls is on my mind right now; I wish it didn't put keyword arguments and **kwargs in the same microbenchmark. - In experimenting with gcc 4.4's FDO support, I produced a training load that resulted in a 15-30% performance improvement (depending on benchmark) across all benchmarks. Using this trained binary, PyBench slowed down by 10%. - I would like to see PyBench incorporate better statistics for indicating the significance of the observed performance difference. I don't believe that these are insurmountable problems, though. A great contribution to Python performance work would be an improved version of PyBench that corrects these problems and offers more precise measurements. Is that something you might be interested in contributing to? As performance moves more into the wider consciousness, having good tools will become increasingly important. Thanks, Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Fri, Apr 3, 2009 at 10:28 AM, Michael Foord wrote: > Collin Winter wrote: >> As part of the common standard library and test suite that we agreed >> on at the PyCon language summit last week, we're going to include a >> common benchmark suite that all Python implementations can share. This >> is still some months off, though, so there'll be plenty of time to >> bikeshed^Wrationally discuss which benchmarks should go in there. >> > > Where is the right place for us to discuss this common benchmark and test > suite? > > As the benchmark is developed I would like to ensure it can run on > IronPython. > > The test suite changes will need some discussion as well - Jython and > IronPython (and probably PyPy) have almost identical changes to tests that > currently rely on deterministic finalisation (reference counting) so it > makes sense to test changes on both platforms and commit a single solution. I believe Brett Cannon is the best person to talk to about this kind of thing. I don't know that any common mailing list has been set up, though there may be and Brett just hasn't told anyone yet :) Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyDict_SetItem hook
On Fri, Apr 3, 2009 at 10:50 AM, Antoine Pitrou wrote: > Collin Winter gmail.com> writes: >> >> - I wish PyBench actually did more isolation. >> Call.py:ComplexPythonFunctionCalls is on my mind right now; I wish it >> didn't put keyword arguments and **kwargs in the same microbenchmark. > > Well, there is a balance to be found between having more subtests and keeping > a > reasonable total running time :-) > (I have to plead guilty for ComplexPythonFunctionCalls, btw) Sure, there's definitely a balance to maintain. With perf.py, we're going down the road of having different tiers of benchmarks: the default set is the one we pay the most attention to, with other benchmarks available for benchmarking certain specific subsystems or workloads (like pickling list-heavy input data). Something similar could be done for PyBench, giving the user the option of increasing the level of detail (and run-time) as appropriate. >> - I would like to see PyBench incorporate better statistics for >> indicating the significance of the observed performance difference. > > I see you already have this kind of measurement in your perf.py script, would > it > be easy to port it? Yes, it should be straightforward to incorporate these statistics into PyBench. In the same directory as perf.py, you'll find test_perf.py which includes tests for the stats functions we're using. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] core python tests
On Sat, Apr 4, 2009 at 7:33 AM, Michael Foord wrote: > Antoine Pitrou wrote: >> >> Nick Coghlan gmail.com> writes: >> >>> >>> C. Titus Brown wrote: >>> I vote for a separate mailing list -- 'python-tests'? -- but I don't know exactly how splintered to make the conversation. It probably belongs at python.org but if you want me to host it, I can. >>> >>> If too many things get moved off to SIGs there won't be anything left >>> for python-dev to talk about ;) >>> >> >> There is already an stdlib-sig, which has been almost unused. >> >> > > stdlib-sig isn't *quite* right (the testing and benchmarking are as much > about core python as the stdlib) - although we could view the benchmarks and > tests themselves as part of the standard library... > > Either way we should get it underway. Collin and Jeffrey - happy to use > stdlib-sig? Works for me. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rethinking intern() and its data structure
Hi John, On Thu, Apr 9, 2009 at 8:02 AM, John Arbash Meinel wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > I've been doing some memory profiling of my application, and I've found > some interesting results with how intern() works. I was pretty surprised > to see that the "interned" dict was actually consuming a significant > amount of total memory. > To give the specific values, after doing: > bzr branch A B > of a small project, the total memory consumption is ~21MB [snip] > Anyway, I the internals of intern() could be done a bit better. Here are > some concrete things: [snip] Memory usage is definitely something we're interested in improving. Since you've already looked at this in some detail, could you try implementing one or two of your ideas and see if it makes a difference in memory consumption? Changing from a dict to a set looks promising, and should be a fairly self-contained way of starting on this. If it works, please post the patch on http://bugs.python.org with your results and assign it to me for review. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rethinking intern() and its data structure
On Thu, Apr 9, 2009 at 9:34 AM, John Arbash Meinel wrote: > ... > >>> Anyway, I the internals of intern() could be done a bit better. Here are >>> some concrete things: >>> >> >> [snip] >> >> Memory usage is definitely something we're interested in improving. >> Since you've already looked at this in some detail, could you try >> implementing one or two of your ideas and see if it makes a difference >> in memory consumption? Changing from a dict to a set looks promising, >> and should be a fairly self-contained way of starting on this. If it >> works, please post the patch on http://bugs.python.org with your >> results and assign it to me for review. >> >> Thanks, >> Collin Winter >> > (I did end up subscribing, just with a different email address :) > > What is the best branch to start working from? "trunk"? That's a good place to start, yes. If the idea works well, we'll want to port it to the py3k branch, too, but that can wait. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Rethinking intern() and its data structure
On Thu, Apr 9, 2009 at 6:24 PM, John Arbash Meinel wrote: > Greg Ewing wrote: >> John Arbash Meinel wrote: >>> And the way intern is currently >>> written, there is a third cost when the item doesn't exist yet, which is >>> another lookup to insert the object. >> >> That's even rarer still, since it only happens the first >> time you load a piece of code that uses a given variable >> name anywhere in any module. >> > > Somewhat true, though I know it happens 25k times during startup of > bzr... And I would be a *lot* happier if startup time was 100ms instead > of 400ms. Quite so. We have a number of internal tools, and they find that frequently just starting up Python takes several times the duration of the actual work unit itself. I'd be very interested to review any patches you come up with to improve start-up time; so far on this thread, there's been a lot of theory and not much practice. I'd approach this iteratively: first replace the dict with a set, then if that bears fruit, consider a customized data structure; if that bears fruit, etc. Good luck, and be sure to let us know what you find, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wordcode-based Python
Hi Cesare, On Mon, May 11, 2009 at 11:00 AM, Cesare Di Mauro wrote: > At the last PyCon3 at Italy I've presented a new Python implementation, > which you'll find at http://code.google.com/p/wpython/ Good to see some more attention on Python performance! There's quite a bit going on in your changes; do you have an optimization-by-optimization breakdown, to give an idea about how much performance each optimization gives? Looking over the slides, I see that you still need to implement functionality to make test_trace pass, for example; do you have a notion of how much performance it will cost to implement the rest of Python's semantics in these areas? Also, I checked out wpython at head to run Unladen Swallow's benchmarks against it, but it refuses to compile with either gcc 4.0.1 or 4.3.1 on Linux (fails in Python/ast.c). I can send you the build failures off-list, if you're interested. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wordcode-based Python
On Tue, May 12, 2009 at 4:45 AM, Cesare Di Mauro wrote: > Another note. Fredrik Johansson let me note just few minutes ago that I've > compiled my sources without PGO optimizations enabled. > > That's because I used Visual Studio Express Edition. > > So another gain in performances can be obtained. :) FWIW, Unladen Swallow experimented with gcc 4.4's FDO and got an additional 10-30% (depending on the benchmark). The training load is important, though: some training sets offered better performance than others. I'd be interested in how MSVC's PGO compares to gcc's FDO in terms of overall effectiveness. The results for gcc FDO with our 2009Q1 release are at the bottom of http://code.google.com/p/unladen-swallow/wiki/Releases. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Shorter release schedule?
On Tue, May 12, 2009 at 3:06 PM, Antoine Pitrou wrote: > Hello, > > Just food for thought here, but seeing how 3.1 is going to be a real > featureful > schedule despite being released shortly after 3.0, wouldn't it make sense to > tighten future release planning a little? I was thinking something like doing > a > major release every 12 months (rather than 18 to 24 months as has been > heuristically the case lately). This could also imply switching to some kind > of > loosely time-based release system. I'd be in favor of a shorter, 12-month release cycle. I think the limiting resource would be the time and energy of the release managers and the package builders for Windows, etc. Provided it's not a tax on the release staff, I think shorter release cycles would be a benefit to the community. My own experience with time-based releases at work is that it greatly helps focus energy and attention, knowing that you can't simply delay the release if you slack off on your features/bugs. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft pep: backwards compatibility
On Thu, Jun 18, 2009 at 7:17 PM, Benjamin Peterson wrote: [snip] > Backwards Compatibility Rules > = > > This policy applys to all public APIs. These include the C-API, the standard > library, and the core language including syntax and operation as defined by > the > reference manual. > > This is the basic policy for backwards compatibility: > > * The behavior of an API *must* not change between any two consecutive > releases. Is this intended to include performance changes? Clearly no-one will complain if things simply get faster, but I'm thinking about cases where, say, a function runs in half the time but uses double the memory (or vice versa). Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Anyone against having a loop option for regrtest?
On Mon, Jun 29, 2009 at 6:59 PM, Jesse Noller wrote: > Something that's been helping me squirrel out "wacky" and "fun" bugs > in multiprocessing is running the tests in a loop - sometimes hundreds > of times. Right now, I hack this up with a bash script, but I'm > sitting here wondering if adding a "loop for x iterations" option to > regrtest.py would be useful to others as well. > > Any thoughts? Does anyone hate this idea with the power of a thousand suns? +1 for having this in regrtest. I've wished for this in the past, and ended up going the bash route, same as you. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fast Implementation for ZIP decryption
On Sun, Aug 30, 2009 at 7:34 AM, Shashank Singh wrote: > just to give you an idea of the speed up: > > a 3.3 mb zip file extracted using the current all-python implementation on > my machine (win xp 1.67Ghz 1.5GB) > takes approximately 38 seconds. > > the same file when extracted using c implementation takes 0.4 seconds. Are there any applications/frameworks which have zip files on their critical path, where this kind of (admittedly impressive) speedup would be beneficial? What was the motivation for writing the C version? Collin Winter > On Sun, Aug 30, 2009 at 6:35 PM, wrote: >> >> On 12:59 pm, st...@pearwood.info wrote: >>> >>> On Sun, 30 Aug 2009 06:55:33 pm Martin v. Löwis wrote: >>>> >>>> > Does it sound worthy enough to create a patch for and integrate >>>> > into python itself? >>>> >>>> Probably not, given that people think that the algorithm itself is >>>> fairly useless. >>> >>> I would think that for most people, the threat model isn't "the CIA is >>> reading my files" but "my little brother or nosey co-worker is reading >>> my files", and for that, zip encryption with a good password is >>> probably perfectly adequate. E.g. OpenOffice uses it for >>> password-protected documents. >>> >>> Given that Python already supports ZIP decryption (as it should), are >>> there any reasons to prefer the current pure-Python implementation over >>> a faster version? >> >> Given that the use case is "protect my biology homework from my little >> brother", how fast does the implementation really need to be? Is speeding >> it up from 0.1 seconds to 0.001 seconds worth the potential new problems >> that come with more C code (more code to maintain, less portability to other >> runtimes, potential for interpreter crashes or even arbitrary code execution >> vulnerabilities from specially crafted files)? >> >> Jean-Paul >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/shashank.sunny.singh%40gmail.com >> > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/collinw%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] default of returning None hurts performance?
On Mon, Aug 31, 2009 at 3:07 PM, Gregory P. Smith wrote: > > > On Mon, Aug 31, 2009 at 2:20 PM, Antoine Pitrou wrote: >> >> Gregory P. Smith krypto.org> writes: >> > >> > food for thought as noticed by a coworker who has been profiling some >> > hot code >> to optimize a library...If a function does not have a return statement we >> return >> None. Ironically this makes the foo2 function below faster than the bar2 >> function at least as measured using bytecode size >> >> I would be surprised if this "bytecode size" difference made a significant >> difference in runtimes, given that function call cost should dwarf the >> cumulated >> cost of POP_TOP and LOAD_CONST (two of the simplest opcodes you could >> find). >> > > the attached sample code repeatably shows that it makes a difference though > its really not much of one (2-3%). > > I was just wondering if a bytecode for a superinstruction of the common > sequence: > > 6 POP_TOP > 7 LOAD_CONST 0 (None) > 10 RETURN_VALUE > > might be worth it. I doubt it. You'd save a bit of stack manipulation, but since this will only appear at the end of a function, I'd be skeptical that this would make any macrobenchmarks (statistically) significantly faster. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
On Sun, Oct 25, 2009 at 1:22 PM, Antoine Pitrou wrote: > Having other people test it would be fine. Even better if you have an > actual multi-threaded py3k application. But ccbench results for other > OSes would be nice too :-) My results for an 2.4 GHz Intel Core 2 Duo MacBook Pro (OS X 10.5.8): Control (py3k @ r75723) --- Throughput --- Pi calculation (Python) threads=1: 633 iterations/s. threads=2: 468 ( 74 %) threads=3: 443 ( 70 %) threads=4: 442 ( 69 %) regular expression (C) threads=1: 281 iterations/s. threads=2: 282 ( 100 %) threads=3: 282 ( 100 %) threads=4: 282 ( 100 %) bz2 compression (C) threads=1: 379 iterations/s. threads=2: 735 ( 193 %) threads=3: 733 ( 193 %) threads=4: 724 ( 190 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 1 ms. (std dev: 1 ms.) CPU threads=2: 1 ms. (std dev: 2 ms.) CPU threads=3: 3 ms. (std dev: 6 ms.) CPU threads=4: 2 ms. (std dev: 3 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 975 ms. (std dev: 577 ms.) CPU threads=2: 1035 ms. (std dev: 571 ms.) CPU threads=3: 1098 ms. (std dev: 556 ms.) CPU threads=4: 1195 ms. (std dev: 557 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 2 ms.) CPU threads=2: 4 ms. (std dev: 5 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 1 ms. (std dev: 4 ms.) Experiment (newgil branch @ r75723) --- Throughput --- Pi calculation (Python) threads=1: 651 iterations/s. threads=2: 643 ( 98 %) threads=3: 637 ( 97 %) threads=4: 625 ( 95 %) regular expression (C) threads=1: 298 iterations/s. threads=2: 296 ( 99 %) threads=3: 288 ( 96 %) threads=4: 287 ( 96 %) bz2 compression (C) threads=1: 378 iterations/s. threads=2: 720 ( 190 %) threads=3: 724 ( 191 %) threads=4: 718 ( 189 %) --- Latency --- Background CPU task: Pi calculation (Python) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 1 ms.) CPU threads=2: 0 ms. (std dev: 1 ms.) CPU threads=3: 0 ms. (std dev: 0 ms.) CPU threads=4: 1 ms. (std dev: 5 ms.) Background CPU task: regular expression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 1 ms. (std dev: 0 ms.) CPU threads=2: 2 ms. (std dev: 1 ms.) CPU threads=3: 2 ms. (std dev: 2 ms.) CPU threads=4: 2 ms. (std dev: 1 ms.) Background CPU task: bz2 compression (C) CPU threads=0: 0 ms. (std dev: 0 ms.) CPU threads=1: 0 ms. (std dev: 0 ms.) CPU threads=2: 2 ms. (std dev: 3 ms.) CPU threads=3: 0 ms. (std dev: 1 ms.) CPU threads=4: 0 ms. (std dev: 0 ms.) I also ran this through Unladen Swallow's threading microbenchmark, which is a straight copy of what David Beazley was experimenting with (simply iterating over 100 ints in pure Python) [1]. "iterative_count" is doing the loops one after the other, "threaded_count" is doing the loops in parallel using threads. The results below are benchmarking py3k as the control, newgil as the experiment. When it says "x% faster", that is a measure of newgil's performance over py3k's. With two threads: iterative_count: Min: 0.336573 -> 0.387782: 13.21% slower # I've run this configuration multiple times and gotten the same slowdown. Avg: 0.338473 -> 0.418559: 19.13% slower Significant (t=-38.434785, a=0.95) threaded_count: Min: 0.529859 -> 0.397134: 33.42% faster Avg: 0.581786 -> 0.429933: 35.32% faster Significant (t=70.100445, a=0.95) With four threads: iterative_count: Min: 0.766617 -> 0.734354: 4.39% faster Avg: 0.771954 -> 0.751374: 2.74% faster Significant (t=22.164103, a=0.95) Stddev: 0.00262 -> 0.00891: 70.53% larger threaded_count: Min: 1.175750 -> 0.829181: 41.80% faster Avg: 1.224157 -> 0.867506: 41.11% faster Significant (t=161.715477, a=0.95) Stddev: 0.01900 -> 0.01120: 69.65% smaller With eight threads: iterative_count: Min: 1.527794 -> 1.447421: 5.55% faster Avg: 1.536911 -> 1.479940: 3.85% faster Significant (t=35.559595, a=0.95) Stddev: 0.00394 -> 0.01553: 74.61% larger threaded_count: Min: 2.424553 -> 1.677180: 44.56% faster Avg: 2.484922 -> 1.723093: 44.21% faster Significant (t=184.766131, a=0.95) Stddev: 0.02874 -> 0.02956: 2.78% larger I'd be interested in multithreaded benchmarks with less-homogenous workloads. Collin Winter [1] - http://code.google.com/p/unladen-swallow/source/browse/tests/performance/bm_threading.py ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Reworking the GIL
On Mon, Oct 26, 2009 at 2:43 PM, Antoine Pitrou wrote:
> Collin Winter gmail.com> writes:
> [the Dave Beazley benchmark]
>> The results below are benchmarking py3k as the control, newgil as the
>> experiment. When it says "x% faster", that is a measure of newgil's
>> performance over py3k's.
>>
>> With two threads:
>>
>> iterative_count:
>> Min: 0.336573 -> 0.387782: 13.21% slower # I've run this
>> configuration multiple times and gotten the same slowdown.
>> Avg: 0.338473 -> 0.418559: 19.13% slower
>
> Those numbers are not very in line with the other "iterative_count" results.
> Since iterative_count just runs the loop N times in a row, results should be
> proportional to the number N ("number of threads").
>
> Besides, there's no reason for single-threaded performance to be degraded
> since
> the fast path of the eval loop actually got a bit streamlined (there is no
> volatile ticker to decrement).
I agree those numbers are out of line with the others and make no
sense. I've run it with two threads several times and the results are
consistent on this machine. I'm digging into it a bit more.
Collin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A wordcode-based Python
On Wed, Nov 4, 2009 at 4:20 AM, Mart Sõmermaa wrote: > On Tue, May 12, 2009 at 8:54 AM, Cesare Di Mauro > wrote: >>> Also, I checked out wpython at head to run Unladen Swallow's >>> benchmarks against it, but it refuses to compile with either gcc 4.0.1 >>> or 4.3.1 on Linux (fails in Python/ast.c). I can send you the build >>> failures off-list, if you're interested. >>> >>> Thanks, >>> Collin Winter >> >> I'm very interested, thanks. That's because I worked only on Windows >> machines, so I definitely need to test and fix it to let it run on any other >> platform. >> >> Cesare > > Re-animating an old discussion -- Cesare, any news on the wpython front? > > I did a checkout from http://wpython.googlecode.com/svn/trunk and > was able to ./configure and make successfully on my 64-bit Linux box > as well as to run the Unladen benchmarks. > > Given svn co http://svn.python.org/projects/python/tags/r261 in py261 > and svn co http://wpython.googlecode.com/svn/trunk in wpy, > > $ python unladen-tests/perf.py -rm --benchmarks=-2to3,all py261/python > wpy/python Do note that the --track_memory option to perf.py imposes some overhead that interferes with the performance figures. I'd recommend running the benchmarks again without --track_memory. That extra overhead is almost certainly what's causing some of the variability in the results. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2to3, 3to2: official status
Hi Ben, On Wed, Nov 4, 2009 at 6:49 PM, Ben Finney wrote: > "Martin v. Löwis" writes: > >> Ben Finney wrote: >> > "Martin v. Löwis" writes: >> > >> >> Well, 3to2 would then be an option for you: use Python 3 as the >> >> source language. >> > >> > I was under the impression that 2to3 was officially supported as >> > part of Python, but 3to2 was a third-party tool. […] Is it an >> > official part of Python? >> >> No, the status is exactly as you describe it. > > Okay. It's probably best for anyone with their Python developer hat on > (which, in this forum, is all the time for any Python developer) to make > the status of 3to2 clear when recommending it to people concerned about > future plans. Are you implying that we shouldn't recommend 3to2 to people wanting to develop in Py3k and back-translate to 2.x? Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3003 - Python Language Moratorium
On Thu, Nov 5, 2009 at 10:35 AM, Dino Viehland wrote: > Stefan wrote: >> It /does/ make some static assumptions in that it considers builtins >> true >> builtins. However, it does not prevent you from replacing them in your >> code, as long as you do it inside the module. Certainly a restriction >> compared to Python, where you can import a module into a changed dict >> environment that redefines 'object', but not a major restriction IMO, >> and certainly not one that impacts much code. > > To me this is a deal breaker which prevents Cython from being a Python > implementation. From a talk given by Colin Winter at the LLVM dev meeting > (http://llvm.org/devmtg/2009-10/) it seems like Unladen Swallow wanted to > do something like this as well and Guido said no. In this case the breaking > change is so subtle that I'd personally hate to run into something like > this porting code to Cython and having to figure out why it's not working. To clarify, I was joking when I told that story (or at least I was joking with Guido when I asked him if we could break that). It clearly *would* be easier if we could just ignore this point of Python compatibility, but that's not an option, so we've had to optimize around it. It's not that hard to do, but it's still extra work. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
e the startup time penalty. We're interested in getting guidance from the CPython community as to what kind of a startup slow down would be sufficient in exchange for greater runtime performance. > I guess what I am mainly saying is that there are several possible ways to > speed up Python 3 execution (including others not mentioned here) and it is > not at all clear to me that this particular one is in any sense 'best of > breed'. If it disables other approaches, I think it should be optional for > the standard PSF distribution. We considered the three approaches you mentioned (Psyco, changing the language, using function annotations), but found them unworkable or inapplicable to Google's needs. Adding a just-in-time compiler to Python 2.6 while designing our changes for ease of portability to Python 3 made more sense for our environment, and we believe, is more applicable to the environments of other Python consumers and better-suited to the future roadmap of CPython. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyCon Keynote
On Thu, Jan 21, 2010 at 8:37 AM, Kortatu wrote: > Hi! > > For me, could be very interesting something about Unladen Swallow, and your > opinion about JIT compilers. FWIW, there will be a separate talk about Unladen Swallow at PyCon. I for one would like to hear Guido talk about something else :) Collin > 2010/1/21 Michael Foord >> >> On 21/01/2010 15:03, Thomas Wouters wrote: >> >> On Wed, Jan 13, 2010 at 19:51, Guido van Rossum wrote: >>> >>> Please mail me topics you'd like to hear me talk about in my keynote >>> at PyCon this year. >> >> How about something completely different... ? >> >> Your history of Python stuff has been really interesting. >> >> >> I'd like to hear you lay to rest that nonsense about you retiring :> >> >> Well, ditto. :-) >> >> All the best, >> >> Michael >> >> >> -- >> Thomas Wouters >> >> Hi! I'm a .signature virus! copy me into your .signature file to help me >> spread! >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> >> >> -- >> http://www.ironpythoninaction.com/ >> http://www.voidspace.org.uk/blog >> >> READ CAREFULLY. By accepting and reading this email you agree, on behalf >> of your employer, to release me from all obligations and waivers arising >> from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, >> shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, >> non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have >> entered into with your employer, its partners, licensors, agents and >> assigns, in perpetuity, without prejudice to my ongoing rights and >> privileges. You further represent that you have the authority to release me >> from any BOGUS AGREEMENTS on behalf of your employer. >> >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/gloryboy84%40gmail.com >> > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/collinw%40gmail.com > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hi Dirkjan, On Wed, Jan 20, 2010 at 10:55 PM, Dirkjan Ochtman wrote: > On Thu, Jan 21, 2010 at 02:56, Collin Winter wrote: >> Agreed. We are actively working to improve the startup time penalty. >> We're interested in getting guidance from the CPython community as to >> what kind of a startup slow down would be sufficient in exchange for >> greater runtime performance. > > For some apps (like Mercurial, which I happen to sometimes hack on), > increased startup time really sucks. We already have our demandimport > code (I believe bzr has something similar) to try and delay imports, > to prevent us spending time on imports we don't need. Maybe it would > be possible to do something like that in u-s? It could possibly also > keep track of the thorny issues, like imports where there's an except > ImportError that can do fallbacks. I added startup benchmarks for Mercurial and Bazaar yesterday (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we can use them as more macro-ish benchmarks, rather than merely starting the CPython binary over and over again. If you have ideas for better Mercurial/Bazaar startup scenarios, I'd love to hear them. The new hg_startup and bzr_startup benchmarks should give us some more data points for measuring improvements in startup time. One idea we had for improving startup time for apps like Mercurial was to allow the creation of hermetic Python "binaries", with all necessary modules preloaded. This would be something like Smalltalk images. We haven't yet really fleshed out this idea, though. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Dirkjan, On Thu, Jan 21, 2010 at 11:16 AM, Dirkjan Ochtman wrote: > On Thu, Jan 21, 2010 at 18:32, Collin Winter wrote: >> I added startup benchmarks for Mercurial and Bazaar yesterday >> (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we >> can use them as more macro-ish benchmarks, rather than merely starting >> the CPython binary over and over again. If you have ideas for better >> Mercurial/Bazaar startup scenarios, I'd love to hear them. The new >> hg_startup and bzr_startup benchmarks should give us some more data >> points for measuring improvements in startup time. > > Sounds good! I seem to remember from a while ago that you included the > Mercurial test suite in your performance tests, but maybe those were > the correctness tests rather than the performance tests (or maybe I'm > just mistaken). I didn't see any mention of that in the proto-PEP, in > any case. We used to run the Mercurial correctness tests at every revision, but they were incredibly slow and a bit flaky under CPython 2.6. Bazaar's tests were faster, but were flakier, so we ended up disabling them, too. We only run these tests occasionally. >> One idea we had for improving startup time for apps like Mercurial was >> to allow the creation of hermetic Python "binaries", with all >> necessary modules preloaded. This would be something like Smalltalk >> images. We haven't yet really fleshed out this idea, though. > > Yeah, that might be interesting. I think V8 can do something similar, right? Correct; V8 loads a pre-compiled image of its builtins to reduce startup time. > What I personally would consider interesting for the PEP is a (not too > big) section evaluating where other Python-performance efforts are at. > E.g. does it make sense to propose a u-s merge now when, by the time > 3.3 (or whatever) is released, there'll be a very good PyPy that > sports memory usage competitive for embedded development (already does > right now, I think) and a good tracing JIT? Or when we can compile > Python using Cython, or Shedskin -- probably not as likely; but I > think it might be worth assessing the landscape a bit before this huge > change is implemented. I can definitely work on that. http://codespeak.net:8099/plotsummary.html should give you a quick starting point for PyPy's performance. My reading of those graphs is that it does very well on heavily-numerical workloads, but is much slower than CPython on more diverse workloads. When I initially benchmarked PyPy vs CPython last year, PyPy was 3-5x slower on non-numerical workloads, and 60x slower on one benchmark (./perf.py -b pickle,unpickle, IIRC). My quick take on Cython and Shedskin is that they are useful-but-limited workarounds for CPython's historically-poor performance. Shedskin, for example, does not support the entire Python language or standard library (http://shedskin.googlecode.com/files/shedskin-tutorial-0.3.html). Cython is a super-set of Python, and files annotated for maximum Cython performance are no longer valid Python code, and will not run on any other Python implementation. The advantage of using an integrated JIT compiler is that we can support Python-as-specified, without workarounds or changes in workflow. The compiler can observe which parts of user code are static (or static-ish) and take advantage of that, without the manual annotations needed by Cython. Cython is good for writing extension modules without worrying about the details of reference counting, etc, but I don't see it as an either-or alternative for a JIT compiler. > P.S. Is there any chance of LLVM doing something like tracing JITs? > Those seem somewhat more promising to me (even though I understand > they're quite hard in the face of Python features like stack frames). Yes, you could implement a tracing JIT with LLVM. We chose a function-at-a-time JIT because it would a) be an easy-to-implement baseline to measure future improvement, and b) create much of the infrastructure for a future tracing JIT. Implementing a tracing JIT that crosses the C/Python boundary would be interesting. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Greg, On Wed, Jan 20, 2010 at 10:54 PM, Gregory P. Smith wrote: > +1 > My biggest concern is memory usage but it sounds like addressing that is > already in your mind. I don't so much mind an additional up front constant > and per-line-of-code hit for instrumentation but leaks are unacceptable. > Any instrumentation data or jit caches should be managed (and tunable at > run time when possible and it makes sense). Reducing memory usage is a high priority. One thing being worked on right now is to avoid collecting runtime data for functions that will never be considered hot. That's one "leak" in the current implementation. > I think having a run time flag (or environment variable for those who like > that) to disable the use of JIT at python3 execution time would be a good > idea. Yep, we already have a -j flag that supports "don't ever use the JIT" (-j never), "use the JIT when you think you should" (-j whenhot), and "always the use the JIT" (-j always) options. I'll mention this in the PEP (we'll clearly need to make this an -X option before merger). Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Barry, On Thu, Jan 21, 2010 at 3:34 AM, Barry Warsaw wrote: > On Jan 20, 2010, at 11:05 PM, Jack Diederich wrote: > >>Does disabling the LLVM change binary compatibility between modules >>targeted at the same version? At tonight's Boston PIG we had some >>binary package maintainers but most people (including myself) only >>cared about source compatibility. I assume linux distros care about >>binary compatibility _a lot_. > > A few questions come to mind: > > 1. What are the implications for PEP 384 (Stable ABI) if U-S is added? PEP 384 looks to be incomplete at this writing, but reading the section "Structures", it says """ Only the following structures and structure fields are accessible to applications: - PyObject (ob_refcnt, ob_type) - PyVarObject (ob_base, ob_size) - Py_buffer (buf, obj, len, itemsize, readonly, ndim, shape, strides, suboffsets, smalltable, internal) - PyMethodDef (ml_name, ml_meth, ml_flags, ml_doc) - PyMemberDef (name, type, offset, flags, doc) - PyGetSetDef (name, get, set, doc, closure) """ Of these, the only one we have changed is PyMethodDef, and then to add two fields to the end of the structure. We have changed other types (dicts and code come to mind), but I believe we have only appended fields and not deleted or reordered existing fields. I don't believe that introducing the Unladen Swallow JIT will make maintaining a stable ABI per PEP 384 more difficult. We've been careful about not exporting any C++ symbols via PyAPI_FUNC(), so I don't believe that will be an issue either, but Jeffrey can comment more deeply on this issue. If PEP 384 is accepted, I'd like it to include a testing strategy so that we can be sure that we haven't accidentally broken ABI compatibility. That testing should ideally be automated. > 2. What effect does requiring C++ have on the embedded applications across the > set of platforms that Python is currently compatible on? In a previous > life I had to integrate a C++ library with Python as an embedded language > and had lots of problems on some OSes (IIRC Solaris and Windows) getting > all the necessary components to link properly. To be clear, you're talking about embedding Python in a C/C++ application/library? We have successfully integrated Unladen Swallow into a large C++ application that uses Python as an embedded scripting language. There were no special issues or restrictions that I had to overcome to do this. If you have any applications/libraries in particular that you'd like me to test, I'd be happy to do that. > 3. Will the U-S bits come with a roadmap to the code? It seems like this is > dropping a big black box of code on the Python developers, and I would want > to reduce the learning curve as much as possible. Yes; there is http://code.google.com/p/unladen-swallow/source/browse/trunk/Python/llvm_notes.txt, which goes into developer-level detail about various optimizations and subsystems. We have other documentation in the Unladen Swallow wiki that is being merged into llvm_notes.txt. Simply dropping this code onto python-dev without a guide to it would be unacceptable. llvm_notes.txt also details available instrumentation, useful to CPython developers who are investigating performance changes. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter wrote: [snip] > Incremental builds, however, are significantly slower. The table below shows > incremental rebuild times after touching ``Objects/listobject.c``. > > +-+---+---+--+ > | Incr make | CPython 2.6.4 | CPython 3.1.1 | Unladen Swallow r988 | > +=+===+===+==+ > | Run 1 | 0m1.854s | 0m1.456s | 0m24.464s | > +-+---+---+--+ > | Run 2 | 0m1.437s | 0m1.442s | 0m24.416s | > +-+---+---+--+ > | Run 3 | 0m1.440s | 0m1.425s | 0m24.352s | > +-+---+---+--+ http://code.google.com/p/unladen-swallow/source/detail?r=1015 has significantly improved this situation. The new table of incremental build times: +-+---+---+---+ | Incr make | CPython 2.6.4 | CPython 3.1.1 | Unladen Swallow r1024 | +=+===+===+===+ | Run 1 | 0m1.854s | 0m1.456s | 0m6.680s | +-+---+---+---+ | Run 2 | 0m1.437s | 0m1.442s | 0m5.310s | +-+---+---+---+ | Run 3 | 0m1.440s | 0m1.425s | 0m7.639s | +-+---+---+---+ The remaining increase is from statically linking LLVM into libpython. PEP updated: http://codereview.appspot.com/186247/diff2/1:4/5 Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hi Paul, On Thu, Jan 21, 2010 at 12:56 PM, Paul Moore wrote: > I'm concerned about the memory and startup time penalties. It's nice > that you're working on them - I'd like to see them remain a priority. > Ultimately a *lot* of people use Python for short-running transient > commands (not just adhoc scripts, think "hg log") and startup time and > memory penalties can really hurt there. I think final merger from the proposed py3k-jit branch into py3k should block on reducing the startup and memory usage penalties. Improving startup time is high on my list of priorities, and I think there's a fair amount of low-hanging fruit there. I've just updated http://code.google.com/p/unladen-swallow/issues/detail?id=64 with some ideas that have recently come up to improve startup time, as well results from the recently-added hg_startup and bzr_startup benchmarks. I'll also update the PEP with these benchmark results, since they're important to a lot of people. > Windows compatibility is a big deal to me. And IMHO, it's a great > strength of Python at the moment that it has solid Windows support. I > would be strongly *against* this PEP if it was going to be Unix or > Linux only. As it is, I have concerns that Windows could suffer from > the common "none of the developers use Windows, but we do our best" > problem. I'm hoping that having U-S integrated into the core will mean > that there will be more Windows developers able to contribute and > alleviate that problem. One of our contributors, James Abbatiello (cc'd), has done a bang-up job of making Unladen Swallow work on Windows. My understanding from his last update is that Unladen Swallow works well on Windows, but he can comment further as to the precise state of Windows support and any remaining challenges faced on that platform, if any. > One question - once Unladen Swallow is integrated, will Google's > support (in terms of dedicated developer time) remain? If not, I'd > rather see more of the potential gains realised before integration, as > otherwise it could be a long time before it happens. Ideally, I'd like > to see a commitment from Google - otherwise the cynic in me is > inclined to say "no" until the suggested speed benefits have > materialised and only then accept U-S for integration. Less cynically, > it's clear that there's quite a way to go before the key advertised > benefits of U-S are achieved, and I don't want the project to lose > steam before it gets there. While this decision is not mine, I don't believe our director would be open to an open-ended commitment of full-time Google engineering resources, though we still have another few quarters of engineering time allocated to the Google team (myself, Jeffrey Yasskin). At this point, the clear majority of Unladen Swallow's performance patches are coming from the non-Google developers on the project (Jeffrey and I are mostly working on infrastructure), and I believe that pattern will continue once the py3k-jit branch is established. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Thu, Jan 21, 2010 at 2:24 PM, Collin Winter wrote: > I'll also update the PEP with these benchmark results, since they're > important to a lot of people. Done; see http://codereview.appspot.com/186247/diff2/4:8/9 for the wording change and new startup data. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Antoine, On Thu, Jan 21, 2010 at 4:25 AM, Antoine Pitrou wrote: >> The increased memory usage comes from a) LLVM code generation, analysis >> and optimization libraries; b) native code; c) memory usage issues or >> leaks in LLVM; d) data structures needed to optimize and generate >> machine code; e) as-yet uncategorized other sources. > > Does the increase in memory occupation disappear when the JIT is disabled > from the command-line? It does not disappear, but it is significantly reduced. Running our django benchmark against three different configurations gives me these max memory usage numbers: CPython 2.6.4: 8508 kb Unladen Swallow default: 26768 kb Unladen Swallow -j never: 15144 kb "-j never" is Unladen Swallow's flag to disable JIT compilation. As it stands right now, -j never gives a 1.76x reduction in memory usage, but is still 1.77x larger than CPython. It occurs to me that we're still doing a lot of LLVM-side initialization and setup that we don't need to do under -j never. We're also collecting runtime feedback in the eval loop, which is yet more memory usage. Optimizing this mode has not yet been a priority for us, but it seems to be the emerging consensus of python-dev that we need to give -j never some more love. There's a lot of low-hanging fruit there. I've added this information to http://code.google.com/p/unladen-swallow/issues/detail?id=123, which is our issue tracking -j never improvements. > Do you think LLVM might suffer from a lot of memory leaks? I don't know that it suffers from a lot of memory leaks, though we have certainly observed and fixed quadratic memory usage in some of the optimization passes. We've fixed all the memory leaks that Google's internal heapchecker has found. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Thu, Jan 21, 2010 at 10:14 AM, Reid Kleckner wrote: > On Thu, Jan 21, 2010 at 9:35 AM, Floris Bruynooghe > wrote: >> I just compiled with the --without-llvm option and see that the >> binary, while only an acceptable 4.1M, still links with libstdc++. Is >> it possible to completely get rid of the C++ dependency if this option >> is used? Introducing a C++ dependency on all platforms for no >> additional benefit (with --without-llvm) seems like a bad tradeoff to >> me. > > There isn't (and shouldn't be) any real source-level dependency on > libstdc++ when LLVM is turned off. However, the eval loop is now > compiled as C++, and that may be adding some hidden dependency > (exception handling code?). The final binary is linked with $(CXX), > which adds an implicit -lstdc++, I think. Someone just has to go and > track this down. We've opened http://code.google.com/p/unladen-swallow/issues/detail?id=124 to track this issue. It should be straight-forward to fix. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Glyph, On Thu, Jan 21, 2010 at 9:11 AM, Glyph Lefkowitz wrote: > It would be hard for me to put an exact number on what I would find > acceptable, but I was really hoping that we could get a *reduced* memory > footprint in the long term. > > My real concern here is not absolute memory usage, but usage for each > additional Python process on a system; even if Python supported fast, > GIL-free multithreading, I'd still prefer the additional isolation of > multiprocess concurrency. As it currently stands, starting cores+1 Python > processes can start to really hurt, especially in many-core-low-RAM > environments like the Playstation 3. > > So, if memory usage went up by 20%, but per-interpreter overhead were > decreased by more than that, I'd personally be happy. There's been a recent thread on our mailing list about a patch that dramatically reduces the memory footprint of multiprocess concurrency by separating reference counts from objects. We're looking at possibly incorporating this work into Unladen Swallow, though I think it should really go into upstream CPython first (since it's largely orthogonal to the JIT work). You can see the thread here: http://groups.google.com/group/unladen-swallow/browse_thread/thread/21d7248e8279b328/2343816abd1bd669 Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Thu, Jan 21, 2010 at 12:20 PM, Collin Winter wrote: > Hey Greg, > > On Wed, Jan 20, 2010 at 10:54 PM, Gregory P. Smith wrote: >> I think having a run time flag (or environment variable for those who like >> that) to disable the use of JIT at python3 execution time would be a good >> idea. > > Yep, we already have a -j flag that supports "don't ever use the JIT" > (-j never), "use the JIT when you think you should" (-j whenhot), and > "always the use the JIT" (-j always) options. I'll mention this in the > PEP (we'll clearly need to make this an -X option before merger). FYI, I just committed http://code.google.com/p/unladen-swallow/source/detail?r=1027, which dramatically improves the performance of Unladen Swallow when running with `-j never`, making disabling the JIT at runtime more viable. We're continuing to make progress minimizing the impact of the JIT when running under `-j never`. Progress can be tracked at http://code.google.com/p/unladen-swallow/issues/detail?id=123. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Thu, Jan 21, 2010 at 11:37 PM, Glyph Lefkowitz wrote: > > On Jan 21, 2010, at 6:48 PM, Collin Winter wrote: > > Hey Glyph, > > There's been a recent thread on our mailing list about a patch that > dramatically reduces the memory footprint of multiprocess concurrency > by separating reference counts from objects. We're looking at possibly > incorporating this work into Unladen Swallow, though I think it should > really go into upstream CPython first (since it's largely orthogonal > to the JIT work). You can see the thread here: > http://groups.google.com/group/unladen-swallow/browse_thread/thread/21d7248e8279b328/2343816abd1bd669 > > AWESOME. > Thanks for the pointer. I read through both of the threads but I didn't see > any numbers on savings-per-multi-process. Do you have any? The data I've seen comes from http://groups.google.com/group/comp.lang.python/msg/c18b671f2c4fef9e: """ This test code[1] consumes roughly 2G of RAM on an x86_64 with python 2.6.1, with the patch, it *should* use 2.3G of RAM (as specified by its output), so you can see the footprint overhead... but better page sharing makes it consume about 6 times less - roughly 400M... which is the size of the dataset. Ie: near-optimal data sharing. """ Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Tony, On Fri, Jan 22, 2010 at 10:11 AM, Tony Nelson wrote: > On 10-01-22 02:53:21, Collin Winter wrote: >> On Thu, Jan 21, 2010 at 11:37 PM, Glyph Lefkowitz >> wrote: >> > >> > On Jan 21, 2010, at 6:48 PM, Collin Winter wrote: > ... >> > There's been a recent thread on our mailing list about a patch that >> > dramatically reduces the memory footprint of multiprocess >> > concurrency by separating reference counts from objects. ... > > Currently, CPython gets a performance advantage from having reference > counts hot in the cache when the referenced object is used. There is > still the write pressure from updating the counts. With separate > reference counts, an extra cache line must be loaded from memory (it is > unlikely to be in the cache unless the program is trivial). I see from > the referenced posting that this is a 10% speed hit (the poster > attributes the hit to extra instructions). > > Perhaps the speed and memory hits could be minimized by only doing this > for some objects? Only objects that are fully shared (such as read- > only data) benefit from this change. I don't know but shared objects > may already be treated separately. One option that we discussed was to create a ./configure flag to toggle between inline refcounts and separate refcounts. Advanced users that care about the memory usage of multiprocess concurrency could compile their own CPython binary to enable this space optimization at the cost of some performance. On the other hand, once we get enough performance out of the JIT that python-dev is willing to take a 10% hit, then I'd say we should just turn the space optimization on by default. In the meantime, though, a configure flag would be a useful intermediate point for a number of people. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Jake, On Thu, Jan 21, 2010 at 10:48 AM, Jake McGuire wrote: > On Thu, Jan 21, 2010 at 10:19 AM, Reid Kleckner wrote: >> On Thu, Jan 21, 2010 at 12:27 PM, Jake McGuire wrote: >>> On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter >>> wrote: >>>> Profiling >>>> - >>>> >>>> Unladen Swallow integrates with oProfile 0.9.4 and newer [#oprofile]_ to >>>> support >>>> assembly-level profiling on Linux systems. This means that oProfile will >>>> correctly symbolize JIT-compiled functions in its reports. >>> >>> Do the current python profiling tools (profile/cProfile/pstats) still >>> work with Unladen Swallow? >> >> Sort of. They disable the use of JITed code, so they don't quite work >> the way you would want them to. Checking tstate->c_tracefunc every >> line generated too much code. They still give you a rough idea of >> where your application hotspots are, though, which I think is >> acceptable. > > Hmm. So cProfile doesn't break, but it causes code to run under a > completely different execution model so the numbers it produces are > not connected to reality? > > We've found the call graph and associated execution time information > from cProfile to be extremely useful for understanding performance > issues and tracking down regressions. Giving that up would be a huge > blow. FWIW, cProfile's call graph information is still perfectly accurate, but you're right: turning on cProfile does trigger execution under a different codepath. That's regrettable, but instrumentation-based profiling is always going to introduce skew into your numbers. That's why we opted to improve oProfile, since we believe sampling-based profiling to be a better model. Profiling was problematic to support in machine code because in Python, you can turn profiling on from user code at arbitrary points. To correctly support that, we would need to add lots of hooks to the generated code to check whether profiling is enabled, and if so, call out to the profiler. Those "is profiling enabled now?" checks are (almost) always going to be false, which means we spend cycles for no real benefit. Can YouTube use oProfile for profiling, or is instrumented profiling critical? oProfile does have its downsides for profiling user code: you see all the C-language support functions, not just the pure-Python functions. That extra data might be useful, but it's probably more information than most people want. YouTube might want it, though. Assuming YouTube can't use oProfile as-is, there are some options: - Write a script around oProfile's reporting tool to strip out all C functions from the report. Enhance oProfile to fix any deficiencies compared to cProfile's reporting. - Develop a sampling profiler for Python that only samples pure-Python functions, ignoring C code (but including JIT-compiled Python code). - Add the necessary profiling hooks to JITted code to better support cProfile, but add a command-line flag (something explicit like -O3) that removes the hooks and activates the current behaviour (or something even more restrictive, possibly). - Initially compile Python code without the hooks, but have a trip-wire set to detect the installation of profiling hooks. When profiling hooks are installed, purge all machine code from the system and recompile all hot functions to include the profiling hooks. Thoughts? Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hi Floris, On Sun, Jan 24, 2010 at 3:40 AM, Floris Bruynooghe wrote: > On Sat, Jan 23, 2010 at 10:09:14PM +0100, Cesare Di Mauro wrote: >> Introducing C++ is a big step, also. Aside the problems it can bring on some >> platforms, it means that C++ can now be used by CPython developers. It >> doesn't make sense to force people use C for everything but the JIT part. In >> the end, CPython could become a mix of C and C++ code, so a bit more >> difficult to understand and manage. > > Introducing C++ is a big step, but I disagree that it means C++ should > be allowed in the other CPython code. C++ can be problematic on more > obscure platforms (certainly when static initialisers are used) and > being able to build a python without C++ (no JIT/LLVM) would be a huge > benefit, effectively having the option to build an old-style CPython > at compile time. (This is why I ased about --without-llvm being able > not to link with libstdc++). I'm working on a patch to completely remove all traces of C++ with configured with --without-llvm. It's a straightforward change, and should present no difficulties. For reference, what are these "obscure platforms" where static initializers cause problems? Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hi Cesare, On Sat, Jan 23, 2010 at 1:09 PM, Cesare Di Mauro wrote: > Hi Collin > > IMO it'll be better to make Unladen Swallow project a module, to be > installed and used if needed, so demanding to users the choice of having it > or not. The same way psyco does, indeed. > Nowadays it requires too much memory, longer loading time, and fat binaries > for not-so-great performances. I know that some issues have being worked on, > but I don't think that they'll show something comparable to the current > CPython status. You're proposing that, even once the issues of memory usage and startup time are addressed, Unladen Swallow should still be an extension module? I don't see why. You're assuming that these issues cannot be fixed, which I disagree with. I think maintaining something like a JIT compiler out-of-line, as Psyco is, causes long-term maintainability problems. Such extension modules are forever playing catchup with the CPython code, depending on implementation details that the CPython developers are right to regard as open to change. It also limits what kind of optimizations you can implement or forces those optimizations to be implemented with workarounds that might be suboptimal or fragile. I'd recommend reading the Psyco codebase, if you haven't yet. As others have requested, we are working hard to minimize the impact of the JIT so that it can be turned off entirely at runtime. We have an active issue tracking our progress at http://code.google.com/p/unladen-swallow/issues/detail?id=123. > Introducing C++ is a big step, also. Aside the problems it can bring on some > platforms, it means that C++ can now be used by CPython developers. Which platforms, specifically? What is it about C++ on those platforms that is problematic? Can you please provide details? > It > doesn't make sense to force people use C for everything but the JIT part. In > the end, CPython could become a mix of C and C++ code, so a bit more > difficult to understand and manage. Whether CPython should allow wider usage of C++ or whether developer should be "force[d]" to use C is not our decision, and is not part of this PEP. With the exception of Python/eval.c, we deliberately have not converted any CPython code to C++ so that if you're not working on the JIT, python-dev's workflow remains the same. Even within eval.cc, the only C++ parts are related to the JIT, and so disappear completely with configured with --without-llvm (or if you're not working on the JIT). In any case, developers can easily tell which language to use based on file extension. The compiler errors that would result from compiling C++ with a C compiler would be a good indication as well. > What I see is that LLVM is a too big project for the goal of having "just" a > JIT-ed Python VM. It can be surely easier to use and integrate into CPython, > but requires too much resources Which resources do you feel that LLVM would tax, machine resources or developer resources? Are you referring to the portions of LLVM used by Unladen Swallow, or the entire wider LLVM project, including the pieces Unladen Swallow doesn't use at runtime? > (on the contrary, Psyco demands little > resources, give very good performances, but seems to be like a mess to > manage and extend). This is not my experience. For the workloads I have experience with, Psyco doubles memory usage while only providing a 15-30% speed improvement. Psyco's benefits are not uniform. Unladen Swallow has been designed to be much more maintainable and easier to extend and modify than Psyco: the compiler and its attendant optimizations are well-tested (see Lib/test/test_llvm.py, for one) and well-documented (see Python/llvm_notes.txt for one). I think that the project is bearing out the success of our design: Google's full-time engineers are a small minority on the project at this point, and almost all performance-improving patches are coming from non-Google developers. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Floris, On Mon, Jan 25, 2010 at 1:25 PM, Floris Bruynooghe wrote: > On Mon, Jan 25, 2010 at 10:14:35AM -0800, Collin Winter wrote: >> I'm working on a patch to completely remove all traces of C++ with >> configured with --without-llvm. It's a straightforward change, and >> should present no difficulties. > > Great to hear that, thanks for caring. This has now been resolved. As of http://code.google.com/p/unladen-swallow/source/detail?r=1036, ./configure --without-llvm has no dependency on libstdc++: Before: $ otool -L ./python.exe ./python.exe: /usr/lib/libSystem.B.dylib /usr/lib/libstdc++.6.dylib /usr/lib/libgcc_s.1.dylib After: $ otool -L ./python.exe ./python.exe: /usr/lib/libSystem.B.dylib /usr/lib/libgcc_s.1.dylib I've explicitly noted this in the PEP (see http://codereview.appspot.com/186247/diff2/2001:4001/5001). Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hi Cesare, On Tue, Jan 26, 2010 at 12:29 AM, Cesare Di Mauro wrote: > Hi Collin, > > One more question: is it easy to support more opcodes, or a different opcode > structure, in Unladen Swallow project? I assume you're asking about integrating WPython. Yes, adding new opcodes to Unladen Swallow is still pretty easy. The PEP includes a section on this, http://www.python.org/dev/peps/pep-3146/#experimenting-with-changes-to-python-or-cpython-bytecode, though it doesn't cover something more complex like converting from bytecode to wordcode, as a purely hypothetical example ;) Let me know if that section is unclear or needs more data. Converting from bytecode to wordcode should be relatively straightforward, assuming that the arrangement of opcode arguments is the main change. I believe the only real place you would need to update is the JIT compiler's bytecode iterator (see http://code.google.com/p/unladen-swallow/source/browse/trunk/Util/PyBytecodeIterator.cc). Depending on the nature of the changes, the runtime feedback system might need to be updated, too, but it wouldn't be too difficult, and the changes should be localized. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Martin, On Thu, Jan 21, 2010 at 2:25 PM, "Martin v. Löwis" wrote: > Reid Kleckner wrote: >> On Thu, Jan 21, 2010 at 4:34 PM, "Martin v. Löwis" >> wrote: >>>> How large is the LLVM shared library? One surprising data point is that the >>>> binary is much larger than some of the memory footprint measurements given >>>> in >>>> the PEP. >>> Could it be that you need to strip the binary, or otherwise remove >>> unneeded debug information? >> >> Python is always built with debug information (-g), at least it was in >> 2.6.1 which unladen is based off of, and we've made sure to build LLVM >> the same way. We had to muck with the LLVM build system to get it to >> include debugging information. On my system, stripping the python >> binary takes it from 82 MB to 9.7 MB. So yes, it contains extra debug >> info, which explains the footprint measurements. The question is >> whether we want LLVM built with debug info or not. > > Ok, so if 70MB are debug information, I think a lot of the concerns are > removed: > - debug information doesn't consume any main memory, as it doesn't get > mapped when the process is started. > - debug information also doesn't take up space in the system > distributions, as they distribute stripped binaries. > > As 10MB is still 10 times as large as a current Python binary,people > will probably search for ways to reduce that further, or at least split > it up into pieces. 70MB of the increase was indeed debug information. Since the Linux distros that I checked ship stripped Python binaries, I've stripped the Unladen Swallow binaries as well, and while the size increase is still significant, it's not as large as it once was. Stripped CPython 2.6.4: 1.3 MB Stripped CPython 3.1.1: 1.4 MB Stripped Unladen r1041: 12 MB A 9x increase is better than a 20x increase, but it's not great, either. There is still room to trim the set of LLVM libraries used by Unladen Swallow, and we're continuing to investigate reducing on-disk binary size (http://code.google.com/p/unladen-swallow/issues/detail?id=118 tracks this). I've updated the PEP to reflect this configuration, since it's what most users will pick up via their system package managers. The exact change to the PEP wording is http://codereview.appspot.com/186247/diff2/6001:6003/5002. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hi William, On Wed, Jan 27, 2010 at 7:26 AM, William Dode wrote: > Hi (as a simple user), > > I'd like to know why you didn't followed the same way as V8 Javascript, > or the opposite, why for V8 they didn't choose llvm ? > > I imagine that startup time and memory was also critical for V8. Startup time and memory usage are arguably *more* critical for a Javascript implementation, since if you only spend a few milliseconds executing Javascript code, but your engine takes 10-20ms to startup, then you've lost. Also, a minimized memory profile is important if you plan to embed your JS engine on a mobile platform, for example, or you need to run in a heavily-multiprocessed browser on low-memory consumer desktops and netbooks. Among other reasons we chose LLVM, we didn't want to write code generators for each platform we were targeting. LLVM has done this for us. V8, on the other hand, has to implement a new code generator for each new platform they want to target. This is non-trivial work: it takes a long time, has a lot of finicky details, and it greatly increases the maintenance burden on the team. We felt that requiring python-dev to understand code generation on multiple platforms was a distraction from what python-dev is trying to do -- develop Python. V8 still doesn't have x86-64 code generation working on Windows (http://code.google.com/p/v8/issues/detail?id=330), so I wouldn't underestimate the time required for that kind of project. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
onical representation of the LLVM > you're using is in the embedded copy in Utils/llvm, rather than e.g. a > recipe like "grab upstream release foo and apply this short list of > patches" (a distributed SCM might help here, or a tool like jhbuild [3]) We generally use LLVM trunk verbatim, but will sometimes cherrypick upstream patches that we want (since doing the full vendor merge can take a while). Right now, we're using an unmodified snapshot of LLVM. I've added language to the PEP to clarify some of these points: - No in-tree copies of LLVM/Clang: http://codereview.appspot.com/186247/diff2/5004:5006/5007 - Shared linking of LLVM: http://codereview.appspot.com/186247/diff2/5006:6007/5008 I've filed http://code.google.com/p/unladen-swallow/issues/detail?id=130 so that we have our own issue to track this, in addition to the upstream LLVM bug (http://llvm.org/PR3201). Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hi William, On Wed, Jan 27, 2010 at 11:02 AM, William Dode wrote: > The startup time and memory comsumption are a limitation of llvm that > their developers plan to resolve or is it only specific to the current > python integration ? I mean the work to correct this is more on U-S or > on llvm ? Part of it is LLVM, part of it is Unladen Swallow. LLVM is very flexible, and there's a price for that. We have also found and fixed several cases of quadratic memory usage in LLVM optimization passes, and there may be more of those lurking around. On the Unladen Swallow side, there are doubtless things we can do to improve our usage of LLVM; http://code.google.com/p/unladen-swallow/issues/detail?id=68 has most of our work on this, and there are still more ideas to implement. Part of the issue is that Unladen Swallow is using LLVM's JIT infrastructure in ways that it really hasn't been used before, and so there's a fair amount of low-hanging fruit left in LLVM that no-one has needed to pick yet. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlan wrote: > Antoine Pitrou wrote: >> Or you could submit patches piecewise on http://bugs.python.org >> I think the first step would be to switch to 16-bit bytecodes. It would be >> uncontroversial (the increase in code size probably has no negative effect) >> and >> would provide the foundation for all of your optimizations. > > I wouldn't consider changing from bytecode to wordcode uncontroversial - > the potential to have an effect on cache hit ratios means it needs to be > benchmarked (the U-S performance tests should be helpful there). > > It's the same basic problem where any changes to the ceval loop can have > surprising performance effects due to the way they affect the compiled > switch statements ability to fit into the cache and other low level > processor weirdness. Agreed. We originally switched Unladen Swallow to wordcode in our 2009Q1 release, and saw a performance improvement from this across the board. We switched back to bytecode for the JIT compiler to make upstream merger easier. The Unladen Swallow benchmark suite should provided a thorough assessment of the impact of the wordcode -> bytecode switch. This would be complementary to a JIT compiler, rather than a replacement for it. I would note that the switch will introduce incompatibilities with libraries like Twisted. IIRC, Twisted has a traceback prettifier that removes its trampoline functions from the traceback, parsing CPython's bytecode in the process. If running under CPython, it assumes that the bytecode is as it expects. We broke this in Unladen's wordcode switch. I think parsing bytecode is a bad idea, but any switch to wordcode should be advertised widely. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Terry, On Fri, Jan 29, 2010 at 2:47 PM, Terry Reedy wrote: > On 1/29/2010 4:19 PM, Collin Winter wrote: >> >> On Fri, Jan 29, 2010 at 7:22 AM, Nick Coghlan wrote: > >> Agreed. We originally switched Unladen Swallow to wordcode in our >> 2009Q1 release, and saw a performance improvement from this across the >> board. We switched back to bytecode for the JIT compiler to make >> upstream merger easier. The Unladen Swallow benchmark suite should >> provided a thorough assessment of the impact of the wordcode -> >> bytecode switch. This would be complementary to a JIT compiler, rather >> than a replacement for it. >> >> I would note that the switch will introduce incompatibilities with >> libraries like Twisted. IIRC, Twisted has a traceback prettifier that >> removes its trampoline functions from the traceback, parsing CPython's >> bytecode in the process. If running under CPython, it assumes that the >> bytecode is as it expects. We broke this in Unladen's wordcode switch. >> I think parsing bytecode is a bad idea, but any switch to wordcode >> should be advertised widely. > > Several years, there was serious consideration of switching to a > registerbased vm, which would have been even more of a change. Since I > learned 1.4, Guido has consistently insisted that the CPython vm is not part > of the language definition and, as far as I know, he has rejected any > byte-code hackery in the stdlib. While he is not one to, say, randomly > permute the codes just to frustrate such hacks, I believe he has always > considered vm details private and subject to change and any usage thereof > 'at one's own risk'. No, I agree entirely: bytecode is an implementation detail that could be changed at any time. But like reference counting, it's an implementation detail that people have -- for better or worse -- come to rely on. My only point was that a switch to wordcode should be announced prominently in the release notes and not assumed to be without impact on user code. That people are directly munging CPython bytecode means that CPython should provide a better, more abstract way to do the same thing that's more resistant to these kinds of changes. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey MA, On Mon, Feb 1, 2010 at 9:58 AM, M.-A. Lemburg wrote: > BTW: Some years ago we discussed the idea of pluggable VMs for > Python. Wouldn't U-S be a good motivation to revisit this idea ? > > We could then have a VM based on byte code using a stack > machines, one based on word code using a register machine > and perhaps one that uses the Stackless approach. What is the usecase for having pluggable VMs? Is the idea that, at runtime, the user would select which virtual machine they want to run their code under? How would the user make that determination intelligently? I think this idea underestimates a) how deeply the current CPython VM is intertwined with the rest of the implementation, and b) the nature of the changes required by these separate VMs. For example, Unladen Swallow adds fields to the C-level structs for dicts, code objects and frame objects; how would those changes be pluggable? Stackless requires so many modifications that it is effectively a fork; how would those changes be pluggable? Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Mon, Feb 1, 2010 at 11:17 AM, M.-A. Lemburg wrote: > Collin Winter wrote: >> I think this idea underestimates a) how deeply the current CPython VM >> is intertwined with the rest of the implementation, and b) the nature >> of the changes required by these separate VMs. For example, Unladen >> Swallow adds fields to the C-level structs for dicts, code objects and >> frame objects; how would those changes be pluggable? Stackless >> requires so many modifications that it is effectively a fork; how >> would those changes be pluggable? > > They wouldn't be pluggable. Such changes would have to be made > in a more general way in order to serve more than just one VM. I believe these VMs would have little overlap. I cannot imagine that Unladen Swallow's needs have much in common with Stackless's, or with those of a hypothetical register machine to replace the current stack machine. Let's consider that last example in more detail: a register machine would require completely different bytecode. This would require replacing the bytecode compiler, the peephole optimizer, and the bytecode eval loop. The frame object would need to be changed to hold the registers and a new blockstack design; the code object would have to potentially hold a new bytecode layout. I suppose making all this pluggable would be possible, but I don't see the point. This kind of experimentation is ideal for a branch: go off, test your idea, report your findings, merge back. Let the branch be long-lived, if need be. The Mercurial migration will make all this easier. > Getting the right would certainly require a major effort, but it > would also reduce the need to have several branches of C-based > Python implementations. If such a restrictive plugin-based scheme had been available when we began Unladen Swallow, I do not doubt that we would have ignored it entirely. I do not like the idea of artificially tying the hands of people trying to make CPython faster. I do not see any part of Unladen Swallow that would have been made easier by such a scheme. If anything, it would have made our project more difficult. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] API for VM branches (was: PEP 3146)
[Moving to python-ideas; python-dev to bcc] On Tue, Feb 2, 2010 at 2:02 AM, M.-A. Lemburg wrote: > Collin Winter wrote: [snip] >> If such a restrictive plugin-based scheme had been available when we >> began Unladen Swallow, I do not doubt that we would have ignored it >> entirely. I do not like the idea of artificially tying the hands of >> people trying to make CPython faster. I do not see any part of Unladen >> Swallow that would have been made easier by such a scheme. If >> anything, it would have made our project more difficult. > > I don't think that it has to be restrictive - much to the contrary, > it would provide a consistent API to those CPython internals and > also clarify the separation between the various parts. Something > which currently does not exist in CPython. We do not need an API to CPython's internals: we are not interfacing with them, we are replacing and augmenting them. > Note that it may be easier for you (and others) to just take > CPython and patch it as necessary. However, this doesn't relieve > you from the needed maintenance - which, I presume, is one of the > reasons why you are suggesting to merge U-S back into CPython ;-) That is incorrect. In the year we have been working on Unladen Swallow, we have only updated our vendor branch of CPython 2.6 once, going from 2.6.1 to 2.6.4. We have occasionally cherrypicked patches from the 2.6 maintenance branch to fix specific problems. The maintenance required by upstream CPython changes has been effectively zero. We are seeking to merge with CPython for three reasons: 1) verify that python-dev is interested in this project, and that we are not wasting our time; 2) expose the codebase to a wider, more heterogenous testing environment; 3) accelerate development by having more hands on the code. Upstream maintenance requirements have had zero impact on our planning. In any case, I'll be interested in reading your PEP that outlines how the plugin interface should work, which systems will be pluggable, and exactly how Unladen Swallow, WPython and Stackless would benefit. Let's move further discussion of this to python-ideas until there's something more concrete here. The py3k-jit branch will live long enough that we could update it to work with a plugin system, assuming it is demonstrated to be beneficial. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Dirkjan, [Circling back to this part of the thread] On Thu, Jan 21, 2010 at 1:37 PM, Dirkjan Ochtman wrote: > On Thu, Jan 21, 2010 at 21:14, Collin Winter wrote: [snip] >> My quick take on Cython and Shedskin is that they are >> useful-but-limited workarounds for CPython's historically-poor >> performance. Shedskin, for example, does not support the entire Python >> language or standard library >> (http://shedskin.googlecode.com/files/shedskin-tutorial-0.3.html). > > Perfect, now put something like this in the PEP, please. ;) Done. The diff is at http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed Cython, Shedskin and a bunch of other alternatives to pure CPython. Some of that information is based on conversations I've had with the respective developers, and I'd appreciate corrections if I'm out of date. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey MA, On Fri, Jan 29, 2010 at 11:14 AM, M.-A. Lemburg wrote: > Collin Winter wrote: >> I added startup benchmarks for Mercurial and Bazaar yesterday >> (http://code.google.com/p/unladen-swallow/source/detail?r=1019) so we >> can use them as more macro-ish benchmarks, rather than merely starting >> the CPython binary over and over again. If you have ideas for better >> Mercurial/Bazaar startup scenarios, I'd love to hear them. The new >> hg_startup and bzr_startup benchmarks should give us some more data >> points for measuring improvements in startup time. >> >> One idea we had for improving startup time for apps like Mercurial was >> to allow the creation of hermetic Python "binaries", with all >> necessary modules preloaded. This would be something like Smalltalk >> images. We haven't yet really fleshed out this idea, though. > > In Python you can do the same with the freeze.py utility. See > > http://www.egenix.com/www2002/python/mxCGIPython.html > > for an old project where we basically put the Python > interpreter and stdlib into a single executable. > > We've recently revisited that project and created something > we call "pyrun". It fits Python 2.5 into a single executable > and a set of shared modules (which for various reasons cannot > be linked statically)... 12MB in total. > > If you load lots of modules from the stdlib this does provide > a significant improvement over standard Python. Good to know there are options. One feature we had in mind for a system of this sort would be the ability to take advantage of the limited/known set of modules in the image to optimize the application further, similar to link-time optimizations in gcc/LLVM (http://www.airs.com/blog/archives/100). > Back to the PEP's proposal: > > Looking at the data you currently have, the negative results > currently don't really look good in the light of the small > performance improvements. The JIT compiler we are offering is more than just its current performance benefit. An interpreter loop will simply never be as fast as machine code. An interpreter loop, no matter how well-optimized, will hit a performance ceiling and before that ceiling will run into diminishing returns. Machine code is a more versatile optimization target, and as such, allows many optimizations that would be impossible or prohibitively difficult in an interpreter. Unladen Swallow offers a platform to extract increasing performance for years to come. The current generation of modern, JIT-based JavaScript engines are instructive in this regard: V8 (which I'm most familiar with) delivers consistently improving performance release-over-release (see the graphs at the top of http://googleblog.blogspot.com/2009/09/google-chrome-after-year-sporting-new.html). I'd like to see CPython be able to achieve the same thing, like the new implementations of JavaScript and Ruby are able to do. We are aware that Unladen Swallow is not finished; that's why we're not asking to go into py3k directly. Unladen Swallow's memory usage will continue to decrease, and its performance will only go up. The current state is not its permanent state; I'd hate to see the perfect become the enemy of the good. > Wouldn't it be possible to have the compiler approach work > in three phases in order to reduce the memory footprint and > startup time hit, ie. > > 1. run an instrumented Python interpreter to collect all > the needed compiler information; write this information into > a .pys file (Python stats) > > 2. create compiled versions of the code for various often > used code paths and type combinations by reading the > .pys file and generating an .so file as regular > Python extension module > > 3. run an uninstrumented Python interpreter and let it > use the .so files instead of the .py ones > > In production, you'd then only use step 3 and avoid the > overhead of steps 1 and 2. That is certainly a possibility if we are unable to reduce memory usage to a satisfactory level. I've added a "Contingency Plans" section to the PEP, including this option: http://codereview.appspot.com/186247/diff2/8004:7005/8006. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hi Craig, On Tue, Feb 2, 2010 at 4:42 PM, Craig Citro wrote: >> Done. The diff is at >> http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed >> Cython, Shedskin and a bunch of other alternatives to pure CPython. >> Some of that information is based on conversations I've had with the >> respective developers, and I'd appreciate corrections if I'm out of >> date. >> > > Well, it's a minor nit, but it might be more fair to say something > like "Cython provides the biggest improvements once type annotations > are added to the code." After all, Cython is more than happy to take > arbitrary Python code as input -- it's just much more effective when > it knows something about types. The code to make Cython handle > closures has just been merged ... hopefully support for the full > Python language isn't so far off. (Let me know if you want me to > actually make a comment on Rietveld ...) Indeed, you're quite right. I've corrected the description here: http://codereview.appspot.com/186247/diff2/7005:9001/10001 > Now what's more interesting is whether or not U-S and Cython could > play off one another -- take a Python program, run it with some > "generic input data" under Unladen and record info about which > functions are hot, and what types they tend to take, then let > Cython/gcc -O3 have a go at these, and lather, rinse, repeat ... JIT > compilation and static compilation obviously serve different purposes, > but I'm curious if there aren't other interesting ways to take > advantage of both. Definitely! Someone approached me about possibly reusing the profile data for a feedback-enhanced code coverage tool, which has interesting potential, too. I've added a note about this under the "Future Work" section: http://codereview.appspot.com/186247/diff2/9001:10002/9003 Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
To follow up on some of the open issues: On Wed, Jan 20, 2010 at 2:27 PM, Collin Winter wrote: [snip] > Open Issues > === > > - *Code review policy for the ``py3k-jit`` branch.* How does the CPython > community want us to procede with respect to checkins on the ``py3k-jit`` > branch? Pre-commit reviews? Post-commit reviews? > > Unladen Swallow has enforced pre-commit reviews in our trunk, but we realize > this may lead to long review/checkin cycles in a purely-volunteer > organization. We would like a non-Google-affiliated member of the CPython > development team to review our work for correctness and compatibility, but we > realize this may not be possible for every commit. The feedback we've gotten so far is that at most, only larger, more critical commits should be sent for review, while most commits can just go into the branch. Is that broadly agreeable to python-dev? > - *How to link LLVM.* Should we change LLVM to better support shared linking, > and then use shared linking to link the parts of it we need into CPython? The consensus has been that we should link shared against LLVM. Jeffrey Yasskin is now working on this in upstream LLVM. We are tracking this at http://code.google.com/p/unladen-swallow/issues/detail?id=130 and http://llvm.org/PR3201. > - *Prioritization of remaining issues.* We would like input from the CPython > development team on how to prioritize the remaining issues in the Unladen > Swallow codebase. Some issues like memory usage are obviously critical before > merger with ``py3k``, but others may fall into a "nice to have" category that > could be kept for resolution into a future CPython 3.x release. The big-ticket items here are what we expected: reducing memory usage and startup time. We also need to improve profiling options, both for oProfile and cProfile. > - *Create a C++ style guide.* Should PEP 7 be extended to include C++, or > should a separate C++ style PEP be created? Unladen Swallow maintains its own > style guide [#us-styleguide]_, which may serve as a starting point; the > Unladen Swallow style guide is based on both LLVM's [#llvm-styleguide]_ and > Google's [#google-styleguide]_ C++ style guides. Any thoughts on a CPython C++ style guide? My personal preference would be to extend PEP 7 to cover C++ by taking elements from http://code.google.com/p/unladen-swallow/wiki/StyleGuide and the LLVM and Google style guides (which is how we've been developing Unladen Swallow). If that's broadly agreeable, Jeffrey and I will work on a patch to PEP 7. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
Hey Maciej, On Thu, Feb 11, 2010 at 6:39 AM, Maciej Fijalkowski wrote: > Snippet from: > > http://codereview.appspot.com/186247/diff2/5014:8003/7002 > > *PyPy*: PyPy [#pypy]_ has good performance on numerical code, but is > slower than Unladen Swallow on non-numerical workloads. PyPy only > supports 32-bit x86 code generation. It has poor support for CPython > extension modules, making migration for large applications > prohibitively expensive. > > That part at the very least has some sort of personal opinion > "prohibitively", Of course; difficulty is always in the eye of the person doing the work. Simply put, PyPy is not a drop-in replacement for CPython: there is no embedding API, much less the same one exported by CPython; important libraries, such as MySQLdb and pycrypto, do not build against PyPy; PyPy is 32-bit x86 only. All of these problems can be overcome with enough time/effort/money, but I think you'd agree that, if all I'm trying to do is speed up my application, adding a new x86-64 backend or implementing support for CPython extension modules is certainly north of "prohibitively expensive". I stand by that wording. I'm willing to enumerate all of PyPy's deficiencies in this regard in the PEP, rather than the current vaguer wording, if you'd prefer. > while the other part is not completely true "slower > than US on non-numerical workloads". Fancy providing a proof for that? > I'm well aware that there are benchmarks on which PyPy is slower than > CPython or US, however, I would like a bit more weighted opinion in > the PEP. Based on the benchmarks you're running at http://codespeak.net:8099/plotsummary.html, PyPy is slower than CPython on many non-numerical workloads, which Unladen Swallow is faster than CPython at. Looking at the benchmarks there at which PyPy is faster than CPython, they are primarily numerical; this was the basis for the wording in the PEP. My own recent benchmarking of PyPy and Unladen Swallow (both trunk; PyPy wouldn't run some benchmarks): | Benchmark| PyPy | Unladen | Change | +==+===+=+=+ | ai | 0.61 | 0.51| 1.1921x faster | | django | 0.68 | 0.8 | 1.1898x slower | | float| 0.03 | 0.07| 2.7108x slower | | html5lib | 20.04 | 16.42 | 1.2201x faster | | pickle | 17.7 | 1.09| 16.2465x faster | | rietveld | 1.09 | 0.59| 1.8597x faster | | slowpickle | 0.43 | 0.56| 1.2956x slower | | slowspitfire | 2.5 | 0.63| 3.9853x faster | | slowunpickle | 0.26 | 0.27| 1.0585x slower | | unpickle | 28.45 | 0.78| 36.6427x faster | I'm happy to change the wording to "slower than US on some workloads". Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
On Sat, Feb 13, 2010 at 12:12 AM, Maciej Fijalkowski wrote: > I like this wording far more. It's at the very least far more precise. > Those examples are fair enough (except the fact that PyPy is not 32bit > x86 only, the JIT is). [snip] > "slower than US on some workloads" is true, while not really telling > much to a potential reader. For any X and Y implementing the same > language "X is faster than Y on some workloads" is usually true. > > To be precise you would need to include the above table in the PEP, > which is probably a bit too much, given that PEP is not about PyPy at > all. I'm fine with any wording that is at least correct. I've updated the language: http://codereview.appspot.com/186247/diff2/9005:11001/11002. Thanks for the clarifications. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Mercurial repository for Python benchmarks
Hey Dirkjan, Would it be possible for us to get a Mercurial repository on python.org for the Unladen Swallow benchmarks? Maciej and I would like to move the benchmark suite out of Unladen Swallow and into python.org, where all implementations can share it and contribute to it. PyPy has been adding some benchmarks to their copy of the Unladen benchmarks, and we'd like to have as well, and Mercurial seems to be an ideal solution to this. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial repository for Python benchmarks
On Sun, Feb 21, 2010 at 3:31 PM, Dirkjan Ochtman wrote: > Hi Collin (and others), > > On Sun, Feb 21, 2010 at 15:28, Collin Winter wrote: >> Would it be possible for us to get a Mercurial repository on >> python.org for the Unladen Swallow benchmarks? Maciej and I would like >> to move the benchmark suite out of Unladen Swallow and into >> python.org, where all implementations can share it and contribute to >> it. PyPy has been adding some benchmarks to their copy of the Unladen >> benchmarks, and we'd like to have as well, and Mercurial seems to be >> an ideal solution to this. > > Just a repository on hg.python.org? > > Sounds good to me. Are you staying for the sprints? We'll just do it. > (Might need to figure out some hooks we want to put up with it.) Yep, that's all we want. I'll be around for the sprints through Tuesday, sitting at the Unladen Swallow sprint. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial repository for Python benchmarks
Hey Daniel, On Sun, Feb 21, 2010 at 4:51 PM, Daniel Stutzbach wrote: > On Sun, Feb 21, 2010 at 2:28 PM, Collin Winter > wrote: >> >> Would it be possible for us to get a Mercurial repository on >> python.org for the Unladen Swallow benchmarks? Maciej and I would like >> to move the benchmark suite out of Unladen Swallow and into >> python.org, where all implementations can share it and contribute to >> it. PyPy has been adding some benchmarks to their copy of the Unladen >> benchmarks, and we'd like to have as well, and Mercurial seems to be >> an ideal solution to this. > > If and when you have a benchmark repository set up, could you announce it > via a reply to this thread? I'd like to check it out. Will do. In the meantime, you can read http://code.google.com/p/unladen-swallow/wiki/Benchmarks to find out how to check out the current draft of the benchmarks, as well as which benchmarks are currently included. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial repository for Python benchmarks
On Sun, Feb 21, 2010 at 9:43 PM, Collin Winter wrote: > Hey Daniel, > > On Sun, Feb 21, 2010 at 4:51 PM, Daniel Stutzbach > wrote: >> On Sun, Feb 21, 2010 at 2:28 PM, Collin Winter >> wrote: >>> >>> Would it be possible for us to get a Mercurial repository on >>> python.org for the Unladen Swallow benchmarks? Maciej and I would like >>> to move the benchmark suite out of Unladen Swallow and into >>> python.org, where all implementations can share it and contribute to >>> it. PyPy has been adding some benchmarks to their copy of the Unladen >>> benchmarks, and we'd like to have as well, and Mercurial seems to be >>> an ideal solution to this. >> >> If and when you have a benchmark repository set up, could you announce it >> via a reply to this thread? I'd like to check it out. > > Will do. The benchmarks repository is now available at http://hg.python.org/benchmarks/. It contains all the benchmarks that the Unladen Swallow svn repository contains, including the beginnings of a README.txt that describes the available benchmarks and a quick-start guide for running perf.py (the main interface to the benchmarks). This will eventually contain all the information from http://code.google.com/p/unladen-swallow/wiki/Benchmarks, as well as guidelines on how to write good benchmarks. If you have svn commit access, you should be able to run `hg clone ssh://h...@hg.python.org/repos/benchmarks`. I'm not sure how to get read-only access; Dirkjan can comment on that. Still todo: - Replace the static snapshots of 2to3, Mercurial and other hg-based projects with clones of the respective repositories. - Fix the 2to3 and nbody benchmarks to work with Python 2.5 for Jython and PyPy. - Import some of the benchmarks PyPy has been using. Any access problems with the hg repo should be directed to Dirkjan. Thanks so much for getting the repo set up so fast! Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385 progress report
On Sat, Feb 13, 2010 at 2:23 PM, Benjamin Peterson wrote: > 2010/2/13 "Martin v. Löwis" : >> I still think that the best approach for projects to use 2to3 is to run >> 2to3 at install time from a single-source release. For that, projects >> will have to adjust to whatever bugs certain 2to3 releases have, rather >> than requiring users to download a newer version of 2to3 that fixes >> them. For this use case, a tightly-integrated lib2to3 (with that name >> and sole purpose) is the best thing. > > Alright. That is reasonable. > > The other thing is that we will loose some vcs history and some > history granularity by switching development to the trunk version, > since just the svnmerged revisions will be converted. So the consensus is that 2to3 should be pulled out of the main Python tree? Should the 2to3 hg repository be deleted, then? Thanks, Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mercurial repository for Python benchmarks
On Mon, Feb 22, 2010 at 3:17 PM, Collin Winter wrote: > The benchmarks repository is now available at > http://hg.python.org/benchmarks/. It contains all the benchmarks that > the Unladen Swallow svn repository contains, including the beginnings > of a README.txt that describes the available benchmarks and a > quick-start guide for running perf.py (the main interface to the > benchmarks). This will eventually contain all the information from > http://code.google.com/p/unladen-swallow/wiki/Benchmarks, as well as > guidelines on how to write good benchmarks. We now have a "Benchmarks" component in the bug tracker. Suggestions for new benchmarks, feature requests for perf.py, and bugs in existing benchmarks should be reported under that component. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385 progress report
On Mon, Feb 22, 2010 at 4:27 PM, "Martin v. Löwis" wrote: >>> The other thing is that we will loose some vcs history and some >>> history granularity by switching development to the trunk version, >>> since just the svnmerged revisions will be converted. >> >> So the consensus is that 2to3 should be pulled out of the main Python >> tree? > > Not sure what you mean by "pull out"; I had expect that the right verb > should be "pull into": 2to3 should be pulled into the main Python tree. Sorry, I meant "pulled out" as in: I want an updated version for the benchmark suite, where should I get that? >> Should the 2to3 hg repository be deleted, then? > > Which one? To my knowledge, there is no official 2to3 repository yet. > When the switchover happens, 2to3 should not be converted to its own hg > repository, yes. This one: http://hg.python.org/2to3 Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385 progress report
On Mon, Feb 22, 2010 at 5:03 PM, Nick Coghlan wrote: > Dirkjan Ochtman wrote: >> On Mon, Feb 22, 2010 at 16:09, Collin Winter wrote: >>> So the consensus is that 2to3 should be pulled out of the main Python >>> tree? Should the 2to3 hg repository be deleted, then? >> >> Wouldn't the former be reason to officialize the hg repository, >> instead of deleting it? > > I think the difference between "pull out" and "pull from" is causing > confusion here (and no, I'm not sure which of those Collin actually > meant either). Sorry, I meant "pull from". I want an updated snapshot of 2to3 for the benchmark suite, and I'm looking for the best place to grab it from. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Packaging JIT-less versions of Python
Hey packaging guys, We recently committed a change to Unladen Swallow [1] that moves all the JIT infrastructure into a Python extension module. The theory [2] behind this patch was that this would make it easier for downstream packagers to ship a JIT-less Python package, with the JIT compiler available via an optional add-on package. Some questions for you, so we're not shooting blind here: - Have you guys thought about how a JIT-enabled Python 3 installation would be packaged by your respective distros? - Would you prefer the default python3.x package to have a JIT, or would you omit the JIT by default? - How would you prefer to build the JIT-less package (current options: via a ./configure flag; or by deleting _llvmjit.so from the JIT-enabled package)? - Would the two packages be able to exist side-by-side, or would they be mutually exclusive? My strong preference would be to have the JIT included by default so that it receives as much testing as possible. Thanks, Collin Winter [1] - http://code.google.com/p/unladen-swallow/source/detail?r=1110 [2] - http://code.google.com/p/unladen-swallow/issues/detail?id=136 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging JIT-less versions of Python
Hey David, On Mon, Mar 1, 2010 at 7:29 PM, David Malcolm wrote: > On Mon, 2010-03-01 at 15:35 -0800, Collin Winter wrote: [snip] >> - How would you prefer to build the JIT-less package (current options: >> via a ./configure flag; or by deleting _llvmjit.so from the >> JIT-enabled package)? >> - Would the two packages be able to exist side-by-side, or would they >> be mutually exclusive? > > I have a particular interest in ABI compatibility: if turning JIT on and > off is going to change the ABI of extension modules, that would be a > major pain, as I hope that we will have dozens of C extension modules > available via RPM for our Python 3 stack by the time of the "great > unladen merger". Do you have a good way of testing ABI compatibility, or is it just "build modules, see if they work"? Some general way of testing ABI compatibility would be really useful for PEP 384, too. > So I'm keen for the ability to toggle the JIT code in the face of bugs > and have it not affect ABI. "-Xjit" will do this at runtime (once > that's renamed), but I think it would be useful to be able to toggle the > JIT on/off default during the build, so that I can fix a broken > architecture for non-technical users, but have individual testers opt > back in with -Xjit whilst tracking down a major bug. That's something we can definitely do: you'd just change the default value for the -Xjit flag from "whenhot" to "never". Those individual testers would pass -Xjit=whenhot to opt back in. We could make that a ./configure flag if it would be useful to you and the other distros. > In either case, I don't want to have to recompile 30 extension modules > to try with/without JIT; that would introduce too much change during > bug-hunts, and be no fun at all. That would suck indeed; I want to avoid that. I think that kind of thing falls under PEP 384, which we will have to obey once it is accepted. > (In the blue-sky nirvana future, I'd love to be able to ship > ahead-of-time compiled versions of the stdlib, pre-optimized based on > realworld workloads. Back in my reality, though, I have bugs to fix > before I can work on _that_ patch :( ) Reid Kleckner may be looking at that for his Master's project. It's definitely doable. >> My strong preference would be to have the JIT included by default so >> that it receives as much testing as possible. > > Sounds reasonable. Hope the above made sense and is useful. Thanks for your perspective, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Caching function pointers in type objects
Hey Daniel, On Wed, Mar 3, 2010 at 1:24 PM, Daniel Stutzbach wrote: > On Tue, Mar 2, 2010 at 9:06 PM, Reid Kleckner wrote: >> >> I don't think this will help you solve your problem, but one thing >> we've done in unladen swallow is to hack PyType_Modified to invalidate >> our own descriptor caches. We may eventually want to extend that into >> a callback interface, but it probably will never be considered an API >> that outside code should depend on. > > Thanks Reid and Benjamin for the information. > > I think I see a way to dramatically speed up PyObject_RichCompareBool when > comparing immutable, built-in, non-container objects (int, float, str, > etc.). It would speed up list.sort when the key is one of those types, as > well as most operations on the ubiquitous dictionary with str keys. That definitely sounds worth pursuing. > Is that a worthwhile avenue to pursue, or is it likely to be redundant with > Unladen Swallow's optimizations? I don't believe it will be redundant with the optimizations in Unladen Swallow. > If I can find time to pursue it, would it be best for me to implement it as > a patch to Unladen Swallow, CPython trunk, or CPython py3k? I would recommend patching py3k, with a backport to trunk. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Caching function pointers in type objects
On Wed, Mar 3, 2010 at 2:41 PM, Daniel Stutzbach wrote: > On Wed, Mar 3, 2010 at 4:34 PM, Collin Winter > wrote: >> >> I would recommend patching py3k, with a backport to trunk. > > After thinking it over, I'm inclined to patch trunk, so I can run the > Unladen Swallow macro-benchmarks, then forward-port to py3k. > > I'm correct in understanding that it will be a while before the Unladen > Swallow benchmarks can support Python 3, right? That's correct; porting the full benchmark suite to Python 3 will require projects like Django to support Python 3. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] interesting article on regex performance
On Fri, Mar 12, 2010 at 8:12 AM, Nick Coghlan wrote: [snip] > To bring this on-topic for python-dev by considering how it could apply > to Python's default re engine, I think the key issue is that any updates > to the default engine would need to remain backwards compatible with all > of the tricks that re2 doesn't support. > > There are major practical problems associated with making such a leap > directly (Google's re2 engine is in C++ rather than C and we'd have to > keep the existing implementation around regardless to handle the > features that re2 doesn't support). I don't see why C++ would be a deal-breaker in this case, since it would be restricted to an extension module. > I would say it is better to let re2 bake for a while and see if anyone > is motivated to come up with Python bindings for it and release them on > PyPI. FWIW, re2 is heavily, heavily used in production at Google. Stabilizing any proposed Python bindings would be a good idea, but I'm not sure how much more "baking" re2's core functionality needs. > Once that happens (and assuming the bindings earn a good > reputation), the first step towards integration would be to include a > "See Also" in the re module documentation to point people towards the > more limited (but more robust) regex engine implementation. > > The next step would probably be a hybrid third party library that > exploits the NFA approach when it can, but resorts to backtracking when > it has to in order to handle full regex functionality. (Although > developers would need to be able to disable the backtracking support in > order to restore re2's guarantees of linear time execution) We considered such a hybrid approach for Unladen Swallow, but rejected it on the advice of the V8 team [1]: you end up maintaining two totally separate, incompatible regex engines; the hybrid system comes with interesting, possibly unintuitive performance/correctness issues when bailing from one implementation to another; performance is unstable as small changes are made to the regexes; and it's not obvious that enough Python regexes would benefit from re2's performance/restrictions tradeoff to make such a hybrid system worthwhile in the long term. (There is no representative corpus of real-world Python regexes weighted for dynamic execution frequency to use in assessing such tradeoffs empirically like there is for JavaScript.) re2 is very useful when you want to run user-provided regexes and want to protect your backends against pathological/malicious regex input, but I'm not sure how applicable it is to Python. I think there are more promising strategies to improve regex performance, such as reusing the new JIT infrastructure to JIT-compile regular expressions to machine code (along the lines of V8's irregexp). Some work has already been done in this direction, and I'd be thrilled to mentor any GSoC students interested in working on such a project this summer. Lastly, anyone interested in working on Python regex performance should take a look at the regex benchmarks in the standard benchmark suite [2]. Thanks, Collin Winter [1] - http://blog.chromium.org/2009/02/irregexp-google-chromes-new-regexp.html#c4843826268005492354 [2] - http://hg.python.org/benchmarks/file/5b8fe389710b/performance ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] interesting article on regex performance
On Fri, Mar 12, 2010 at 11:29 AM, wrote: > > >> There are major practical problems associated with making such a leap > >> directly (Google's re2 engine is in C++ rather than C and we'd have > >> to keep the existing implementation around regardless to handle the > >> features that re2 doesn't support). > > Collin> I don't see why C++ would be a deal-breaker in this case, since > Collin> it would be restricted to an extension module. > > Traditionally Python has run on some (minority) platforms where C++ was > unavailable. While the re module is a dynamically linked extension module > and thus could be considered "optional", I doubt anybody thinks of it as > optional nowadays. It's used in the regression test suite anyway. It would > be tough to run unit tests on such minority platforms without it. You'd > have to maintain both the current sre implementation and the new re2 > implementation for a long while into the future. re2 is not a full replacement for Python's current regex semantics: it would only serve as an accelerator for a subset of the current regex language. Given that, it makes perfect sense that it would be optional on such minority platforms (much like the incoming JIT). Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution
On Fri, Apr 23, 2010 at 11:49 AM, Alexandre Vassalotti wrote: > On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti > wrote: >> Collin Winter wrote a simple optimization pass for cPickle in Unladen >> Swallow [1]. The code reads through the stream and remove all the >> unnecessary PUTs in-place. >> > > I just noticed the code removes *all* PUT opcodes, regardless if they > are needed or not. So, this code can only be used if there's no GET in > the stream (which is unlikely for a large stream). I believe Collin > made this trade-off for performance reasons. However, it wouldn't be > hard to make the current code to work like pickletools.optimize(). The optimization pass is only run if you don't use any GETs. The optimization is also disabled if you're writing to a file-like object. These tradeoffs were appropriate for the workload I was optimizing against. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution
On Fri, Apr 23, 2010 at 11:53 AM, Collin Winter wrote: > On Fri, Apr 23, 2010 at 11:49 AM, Alexandre Vassalotti > wrote: >> On Fri, Apr 23, 2010 at 2:38 PM, Alexandre Vassalotti >> wrote: >>> Collin Winter wrote a simple optimization pass for cPickle in Unladen >>> Swallow [1]. The code reads through the stream and remove all the >>> unnecessary PUTs in-place. >>> >> >> I just noticed the code removes *all* PUT opcodes, regardless if they >> are needed or not. So, this code can only be used if there's no GET in >> the stream (which is unlikely for a large stream). I believe Collin >> made this trade-off for performance reasons. However, it wouldn't be >> hard to make the current code to work like pickletools.optimize(). > > The optimization pass is only run if you don't use any GETs. The > optimization is also disabled if you're writing to a file-like object. > These tradeoffs were appropriate for the workload I was optimizing > against. I should add that, adding the necessary bookkeeping to remove only unused PUTs (instead of the current all-or-nothing scheme) should not be hard. I'd watch out for a further performance/memory hit; the pickling benchmarks in the benchmark suite should help assess this. The current optimization penalizes pickling to speed up unpickling, which made sense when optimizing pickles that would go into memcache and be read out 13-15x more often than they were written. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unpickling memory usage problem, and a proposed solution
On Fri, Apr 23, 2010 at 1:53 PM, Alexandre Vassalotti wrote: > On Fri, Apr 23, 2010 at 3:57 PM, Dan Gindikin wrote: >> This wouldn't help our use case, your code needs the entire pickle >> stream to be in memory, which in our case would be about 475mb, this >> is on top of the 300mb+ data structures that generated the pickle >> stream. >> > > In that case, the best we could do is a two-pass algorithm to remove > the unused PUTs. That won't be efficient, but it will satisfy the > memory constraint. Another solution is to not generate the PUTs at all > by setting the 'fast' attribute on Pickler. But that won't work if you > have a recursive structure, or have code that requires that the > identity of objects to be preserved. I don't think it's possible in general to remove any PUTs if the pickle is being written to a file-like object. It is possible to reuse a single Pickler to pickle multiple objects: this causes the Pickler's memo dict to be shared between the objects being pickled. If you pickle foo, bar, and baz, foo may not have any GETs, but bar and baz may have GETs that reference data added to the memo by foo's PUT operations. Because you can't know what will be written to the file-like object later, you can't remove any of the PUT instructions in this scenario. This kind of thing is done in real-world code like cvs2svn (which I broke when I was optimizing cPickle; don't break cvs2svn, it's not fun to fix :). I added some basic tests for this support in cPython's Lib/test/pickletester.py. There might be room for app-specific optimizations that do this, but I'm not sure it would work for a general-usage cPickle that needs to stay compatible with the current system. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New regex module for 3.2?
On Fri, Jul 9, 2010 at 10:28 AM, MRAB wrote: > anatoly techtonik wrote: >> >> On Thu, Jul 8, 2010 at 10:52 PM, MRAB wrote: >>> >>> Hi all, >>> >>> I re-implemented the re module, adding new features and speed >>> improvements. It's available at: >>> >>> http://pypi.python.org/pypi/regex >>> >>> under the name "regex" so that it can be tried alongside "re". >>> >>> I'd be interested in any comments or feedback. How does it compare with >>> "re" in terms of speed on real-world data? The benchmarks suggest it >>> should be faster, or at worst comparable. >> >> And where are the benchmarks? >> In particular it would be interesting to see it compared both to re >> from stdlib and re2 from http://code.google.com/p/re2/ >> > The benchmarks bm_regex_effbot.py and bm_regex_v8.py both perform > multiple runs of the tests multiple times, giving just the total times > for each set. Here are the averages: > > Python26 > BENCHMARK re regex ratio > bm_regex_effbot 0.135secs 0.083secs 1.63 > bm_regex_v8 0.153secs 0.085secs 1.80 > > > Python31 > BENCHMARK re regex ratio > bm_regex_effbot 0.138secs 0.083secs 1.66 > bm_regex_v8 0.170secs 0.091secs 1.87 Out of curiosity, what are the results for the bm_regex_compile benchmark? Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] commit privs
On Sun, Jul 11, 2010 at 9:28 AM, Antoine Pitrou wrote: > On Sun, 11 Jul 2010 13:23:13 + > Reid Kleckner wrote: >> >> I'm also expecting to be doing more work merging unladen-swallow into >> the py3k-jit branch, so I was wondering if I could get commit >> privileges for that. > > It sounds good to me. Also, thanks for your threading patches! +1 from me. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New regex module for 3.2?
On Mon, Jul 12, 2010 at 8:18 AM, Michael Foord wrote: > On 12/07/2010 15:07, Nick Coghlan wrote: >> >> On Mon, Jul 12, 2010 at 9:42 AM, Steven D'Aprano >> wrote: >> >>> >>> On Sun, 11 Jul 2010 09:37:22 pm Eric Smith wrote: >>> > > re2 comparison is interesting from the point of if it should be > included in stdlib. > Is "it" re2 or regex? I don't see having 2 regular expression engines in the stdlib. >>> >>> There's precedence though... the old regex engine and the new re engine >>> were side-by-side for many years before regex was deprecated and >>> finally removed in 2.5. Hypothetically, re2 could similarly be added to >>> the standard library while re is deprecated. >>> >> >> re2 deliberately omits some features for efficiency reasons, hence is >> not even on the table as a possible replacement for the standard >> library version. If someone is in a position where re2 can solve their >> problems with the re module, they should also be in a position where >> they can track it down for themselves. >> >> > > If it has *partial* compatibility, and big enough performance improvements > for common cases, it could perhaps be used where the regex doesn't use > unsupported features. This would have some extra cost in the compile phase, > but would mean Python could ship with two regex engines but only one > interface exposed to the programmer... FWIW, this has all been discussed before: http://aspn.activestate.com/ASPN/Mail/Message/python-dev/3829265. In particular, I still believe that, "it's not obvious that enough Python regexes would benefit from re2's performance/restrictions tradeoff to make such a hybrid system worthwhile in the long term. (There is no representative corpus of real-world Python regexes weighted for dynamic execution frequency to use in assessing such tradeoffs empirically like there is for JavaScript.)" Collin >> MRAB's module offers a superset of re's features rather than a subset >> though, so once it has had more of a chance to bake on PyPI it may be >> worth another look. >> >> Cheers, >> Nick. >> >> > > > -- > http://www.ironpythoninaction.com/ > http://www.voidspace.org.uk/blog > > READ CAREFULLY. By accepting and reading this email you agree, on behalf of > your employer, to release me from all obligations and waivers arising from > any and all NON-NEGOTIATED agreements, licenses, terms-of-service, > shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, > non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have > entered into with your employer, its partners, licensors, agents and > assigns, in perpetuity, without prejudice to my ongoing rights and > privileges. You further represent that you have the authority to release me > from any BOGUS AGREEMENTS on behalf of your employer. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/collinwinter%40google.com > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unladen swallow status
On Wed, Jul 21, 2010 at 2:43 PM, Maciej Fijalkowski wrote: > On Wed, Jul 21, 2010 at 6:50 PM, Reid Kleckner > wrote: >> On Wed, Jul 21, 2010 at 8:11 AM, Tim Golden wrote: >>> Brett suggested that >>> the Unladen Swallow merge to trunk was waiting for some work to complete >>> on the JIT compiler and Georg, as release manager for 3.2, confirmed that >>> Unladen Swallow would not be merged before 3.3. >> >> Yeah, this has slipped. I have patches that need review, and Jeff and >> Collin have been distracted with other work. Hopefully when one of >> them gets around to that, I can proceed with the merge without >> blocking on them. >> >> Reid > > The merge py3k-jit to trunk? I believe he's talking about the merger of the Unladen tree into the py3k-jit branch. Collin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Which version of distutils to ship with Python 2.5?
Is it intentional that Python 2.5 is (currently) shipping with distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3) shipped with distutils 2.4.1? Judging from my own tests, distutils 2.4.1 fixed several bugs that some of my test suites depend on (the fixes, not the bugs ; ). Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Which version of distutils to ship with Python 2.5?
On 7/27/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Collin Winter wrote: > > Is it intentional that Python 2.5 is (currently) shipping with > > distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3) > > shipped with distutils 2.4.1? Judging from my own tests, distutils > > 2.4.1 fixed several bugs that some of my test suites depend on (the > > fixes, not the bugs ; ). > > Are these bugs not fixed in the distutils that shipped with Python 2.5b2? I now believe this to be a new regression that I had confused with an earlier bug report. I've filed a new report, http://python.org/sf/1529871. I'd appreciate it if anyone could shed some light on this. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Questions on unittest behaviour
[Sorry for accidentally cross-posting this to python-list]
While working on a test suite for unittest these past few weeks, I've
run across some behaviours that, while not obviously wrong, don't
strike me as quite right, either. Submitted for your consideration:
1) TestCase.tearDown() is only run if TestCase.setUp() succeeded. It
seems to me that tearDown() should always be run, regardless of any
failures in setUp() or the test method itself.
The case I'm considering is something like this, ie, a multi-part setUp():
> def setUp(self)
> lock_file(testfile) # open_socket(), connect_to_database(), etc
>
> something_that_raises_an_exception()
>
> def tearDown(self):
> if file_is_locked(testfile):
> unlock_file(testfile)
In this pseudo-code example, the file won't be unlocked if some later
operation in setUp() raises an exception. I propose that
TestCase.run() be changed to always run tearDown(), even if setUp()
raise an exception.
I'm undecided if this is a new feature (so it should go in for 2.6) or
a bug fix; I'm leaning toward the latter.
2) The TestLoader.testMethodPrefix attribute currently allows anything
to be assigned to it, including invalid objects and the empty string.
While the former will cause errors to be raised when one of
TestLoader's loadTestsFrom*() methods is called, the empty string will
raise no exception; rather, the loadTestsFrom*() methods will
interpret every possible attribute as being a test method, e.g.,
meaning you get things like assertEqual(), failUnlessEqual(), etc,
when TestLoader.loadTestsFromTestCase() is run.
I propose protecting testMethodPrefix with a property that validates
the assigned value, restricting input to non-empty instances of str. I
see this as a bug fix that should go in before 2.5-final.
3) TestLoader.loadTestsFromTestCase() accepts objects that are not
test cases and will happily look for appropriately-named methods on
any object you give it. This flexibility should be documented, or
proper input validation should be done (a bug fix for 2.5).
4) TestLoader.loadTestsFromName() (and by extension,
loadTestsFromNames(), too) raises an AttributeError if the name is the
empty string because -- as it correctly asserts -- the object does not
contain an attribute named ''. I recommend that this be tested for and
ValueError be raised (bug fix for 2.5).
This of course leads into the question of how much input validation
should be done on these names. Should loadTestsFrom{Name,Names}() make
sure the names are valid attribute names, or is this overkill?
5) When TestLoader.loadTestsFrom{Name,Names}() are given a name that
resolves to a classmethod on a TestCase subclass, the method is not
invoked. From the docs:
> The specifier name is a ``dotted name'' that may resolve either to a module,
> a test
> case class, a TestSuite instance, a test method within a test case class, or a
> callable object which returns a TestCase or TestSuite instance.
It is not documented which of these tests takes priority: is the
classmethod "a test method within a test case class" or is it a
callable? The same issue applies to staticmethods as well.
Once I get answers to these questions, I can finish off the last few
bits of the test suite and have it ready for 2.5-final.
Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Questions on unittest behaviour
On 8/19/06, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Alternatively, someone who prefers your style (with a tearDown() method that > can handle a partially executed call to the setUp() method), can just write > it as: > > def setUp(self) > try: > lock_file(testfile) # open_socket(), connect_to_database(), etc > something_that_may_raise_an_exception() > except: > self.tearDown() > raise > > def tearDown(self): > if file_is_locked(testfile): > unlock_file(testfile) > > Consider the parallel to PEP 343's __enter__ and __exit__ methods - __exit__ > is allowed to assume that it will only be called if __enter__ succeeded, > because that is part of the semantics of the with statement. I can accept that. Any thoughts on the other four items? Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] A test suite for unittest
I've just uploaded a trio of unittest-related patches: #1550272 (http://python.org/sf/1550272) is a test suite for the mission-critical parts of unittest. #1550273 (http://python.org/sf/1550273) fixes 6 issues uncovered while writing the test suite. Several other items that I raised earlier (http://mail.python.org/pipermail/python-dev/2006-August/068378.html) were judged to be either non-issues or behaviours that, while suboptimal, people have come to rely on. #1550263 (http://python.org/sf/1550263) follows up on an earlier patch I submitted for unittest's docs. This new patch corrects and clarifies numerous sections of the module's documentation. I'd appreciate it if these changes could make it into 2.5-final or at least 2.5.1. What follows is a list of the issues fixed in patch #1550273: 1) TestLoader.loadTestsFromName() failed to return a suite when resolving a name to a callable that returns a TestCase instance. 2) Fix a bug in both TestSuite.addTest() and TestSuite.addTests() concerning a lack of input checking on the input test case(s)/suite(s). 3) Fix a bug in both TestLoader.loadTestsFromName() and TestLoader.loadTestsFromNames() that had ValueError being raised instead of TypeError. The problem occured when the given name resolved to a callable and the callable returned something of the wrong type. 4) When a name resolves to a method on a TestCase subclass, TestLoader.loadTestsFromName() did not return a suite as promised. 5) TestLoader.loadTestsFromName() would raise a ValueError (rather than a TypeError) if a name resolved to an invalid object. This has been fixed so that a TypeError is raised. 6) TestResult.shouldStop was being initialised to 0 in TestResult.__init__. Since this attribute is always used in a boolean context, it's better to use the False spelling. Thanks, Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater
On 1/10/07, Thomas Wouters <[EMAIL PROTECTED]> wrote: > On 1/10/07, Raymond Hettinger <[EMAIL PROTECTED]> wrote: > > It is my strong preference that we not go down this path. > > Instead, the 2.6 vs 3.0 difference analysis should go in an > > external lint utility. > > > > The Py2.x series may live-on for some time and should do so > > as if Py3.x did not exist. Burdening the 2.x code with loads > > of warnings will only clutter the source code and make maintenance > > more difficult. There may also be some performance impact. > > > > We should resolve that Py2.6 remain as clean as possible > > and that Py3.0 be kept in its own world. Forging a new > > blade does not have to entail dulling the trusty old blade. > > The idea is that we only generate the warnings optionally, only for things > that can be written in a manner compatible with prevalent Python versions, > and in the most efficient manner we can manage, *except* for the few things > that are already considered (by many) criminal to use: input(), backtics, > mixed tabs and spaces. In other words, for any code written even remotely > sane in the last five years, no extra warnings will be generated. I'd rather see this effort invested in a tool like Guido's 2to3, rather than in sprinkling warnings all through the 2.x codebase (even if they are wrapped in #ifdef's). I don't imagine people developing day-to-day in a 2.x environment are going to be so terribly interested in "is this 3.x compliant?" that downloading a separate tool would be an undue burden. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
I like the general idea, but the syntax looks like dirt on my monitor. The period is too easy to lose visually and without it, there's nothing to distinguish this from a function call. Also, like Anthony Baxter said, someone coming across this for the first time will think it's a syntax error, allusions to MATLAB and assembly indirection syntax not withstanding. Ignoring the syntax, I'm -1 on the 2-argument form, especially since it can only be used in an expression context; getattr() can be kept around for this. I'm +0 on the idea, -1 on the means. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
On 2/12/07, Brett Cannon <[EMAIL PROTECTED]> wrote: > On 2/12/07, Raymond Hettinger <[EMAIL PROTECTED]> wrote: > > [Jack Jansen] > > > I like the functionality, but I don't like the syntax, to me it looks > > > too much like a method call. > > > > > > To me self.[method_name] = self.metadata.[method_name] looks better: > > > what we're doing here is more like dictionary lookup than calling > > > functions. > > > > I also like the functionality. > > > > Rather than munge existing syntaxes, an altogether new one would be more > > clear: > > > >self->name = self.metadata->name > > > > I like the arrow syntax because is the lookup process can be more involved > > than a simple dictionary lookup (perhaps traveling up to base classes). > > IOW, getattr(a,n) is not always the same as a.__dict__[n]. > > The a.__getattribute__(n) process can be more complex than that > > and a bracketed dictionary-like syntax would misleadingly mask the lookup > > process. > > > > I actually kind of like that. The connection to pointer indirection > meshes well with the idea of indirectly figuring out what attribute to > access at runtime. There's a connection, but I'd say it's the wrong one. In C, "x->y" dereferences x, while in Python, "x->y" would dereference y. That's just begging for trouble. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Recent experience with the _ast module
I've been working this past week at converting the stdlib's compiler
package to use 2.5's new _ast module instead of the package's own AST.
Overall, _ast was a joy to work with, save for the following nitpicks:
1) There are times when the _fields attribute on some AST nodes is
None; if this was done to indicate that a given node has no
AST-related attributes, it would be much easier if _fields was [] in
this case. As it is, I had to special-case "node._fields is None" in
the visitor so that I don't accidentally iterate over it.
2) {BinOp,AugAssign,BoolOp,etc}.op is an instance of, eg, Add, Sub,
Mult, Mod, meaning you have to dispatch based on tests like
"isinstance(node.op, x)" or "type(node.op) is x". I would much, much
prefer to spell this "node.op is x", ie, use "node.op = Add" rather
than the current "node.op = Add()" when constructing the nodes.
3) I'd like there to be an Else() node for If.orelse, While.orelse,
etc. My motivation is that I need the lineno and col_offset values of
the "else" statement for a code-coverage utility; as it is, I have to
find the last lineno of the if/while/etc suite and the first lineno of
the "else" suite and then search between those two for the "else"
statement.
Thoughts?
Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New syntax for 'dynamic' attribute access
On 2/13/07, Nick Coghlan <[EMAIL PROTECTED]> wrote: [snip] > I've tried this out on Brett's urllib & urllib2 examples below. (calling > the new builtin attrview() to emphasise the fact that it retains a > reference to the original instance). I don't consider it any uglier than > the proposed syntax changes, and it provides a few other benefits: > >- the two-argument form is naturally available as the .get() method > on the resulting dict-like object (e.g. "attrview(obj).get(some_attr, > None)") > >- hasattr() is naturally replaced by containment testing (e.g. > "some_attr in attrview(obj)") > >- keywords/builtins are easier to look up in the documentation than > symbolic syntax > > With this approach, performance would be attained by arranging to create > the view objects once, and then performing multiple dynamic attribute > accesses using those view objects. This changes my vote: +1 on including attrview(), -1 on the syntax proposal. Collin Winter ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Recent experience with the _ast module
On 2/12/07, Brett Cannon <[EMAIL PROTECTED]> wrote:
> They all sound reasonable. And it's nice to have a wanted feature be
> found from actual use. =)
I've just uploaded patch #1659410 to SF, implementing the changes I outlined:
1) I changed some of the code emitted by asdl_c.py so _fields is
always a tuple. Where _fields used to be None, it's now a 0-element
tuple.
2) It turned out that {BinOp, BoolOp,AugAssign,etc}.op were already
singleton instances of their respective classes. I've changed
asdl_c.py to no longer emit the *_singleton names and to use the
corresponding *_type types in their place, which will enable me to
write "node.op is _ast.Add", etc.
3) Adding an Else node was a little more involved, but it proved
do-able. TryFinally.orelse, While.orelse, For.orelse and If.orelse can
now be either None or an instance of Else. Else instances have a
'body' attribute, which is a sequence of statements.
Thanks,
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Recent experience with the _ast module
On 2/14/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote:
> Collin Winter schrieb:
> > 2) It turned out that {BinOp, BoolOp,AugAssign,etc}.op were already
> > singleton instances of their respective classes. I've changed
> > asdl_c.py to no longer emit the *_singleton names and to use the
> > corresponding *_type types in their place, which will enable me to
> > write "node.op is _ast.Add", etc.
>
> I don't really like this - it's inconsistent. I'd rather prefer
> if the singletons where exposed under a name, e.g.
> _ast.Add.singleton, or _ast.add (if that doesn't cause conflicts).
What's inconsistent about it? That classes are being used for the
_ast.{Add,Sub,Mult,etc} names?
I don't see the need for both _ast.Add and _ast.Add.singleton or
_ast.add or however else it might be spelled. I'd be perfectly happy
doing something like "_ast.Add = object()" (when initializing the _ast
module), so long as I can write "node.op is _ast.Add", "node.op ==
_ast.Add", or something equally brief and to-the-point.
Collin Winter
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
