Re: [Python-Dev] 2.4 news reaches interesting places
Hi Andrew, On Thu, Dec 09, 2004 at 03:32:10PM +1300, Andrew McGregor wrote: > The other thing we can do is finish the portable backend for psyco and > make it a standard module. Then Python won't be slow, it will be > compiled, and py2exe will be able to make a single-file executable. You probably mean that Psyco can dynamically compile Python bytecodes even if they have been hidden into an .exe file by py2exe. At first reading, it appears that you are propagating the misconception "compiler == speed == .exe" i.e. Psyco could "compile your program into an .exe file". As far as I see, this is definitely not possible. Of course, for PR purposes the difference is small -- an .exe file that runs at compiled speed, or at least machine-level VM speed -- but advertising should work around and not build on top of existing confusion IMHO. A bientot, Armin ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
Hi, On Wed, Dec 08, 2004 at 11:43:49PM -0500, Raymond Hettinger wrote: > Acceptance for Py2.4 partially hinges on how quickly third party apps > have their binaries updated. > > I wonder if there is anything we can do to help. For people like myself, Linux programmers not developing on Windows every day, there is precious little information available about how to compile our extension modules for the new Windows distribution. I was actually very disappointed to have to google my way around until I found a page that explained to me that to use Microsoft's free compilers you need to manually patch the distutils included with 2.4. (I know this is being worked on right now but I'd have expected it to be present in the 2.4 release.) (The only page I could find still refers to a pre-final 2.4 so their distutils patch doesn't apply without hand-editing, though that should be fixed by now.) Moreover, the standard documentation's "Extending and Embedding" has apparently not been updated at all since 2.3. That's quite unexpected too... In other words, if you want 3rd parties to compile Windows binaries for 2.4, tell them how. (I even hoped that just doing "python setup.py build" would fail but refer me to a web page that explains me which free compilers I should get and install from Microsoft.) A bientot, Armin. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4 news reaches interesting places
>> The other thing we can do is finish the portable backend for psyco >> and make it a standard module. Then Python won't be slow, it will be >> compiled, and py2exe will be able to make a single-file executable. Armin> You probably mean that Psyco can dynamically compile Python Armin> bytecodes even if they have been hidden into an .exe file by Armin> py2exe. I didn't read it that way. My impression was that py2exe be modified to include and enable psyco if it's available when building an .exe. You would, in theory, get a single file distribution as well as dynamic compilation. Skip ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4 news reaches interesting places
Skip Montanaro <[EMAIL PROTECTED]> writes: > >> The other thing we can do is finish the portable backend for psyco > >> and make it a standard module. Then Python won't be slow, it will be > >> compiled, and py2exe will be able to make a single-file executable. > > Armin> You probably mean that Psyco can dynamically compile Python > Armin> bytecodes even if they have been hidden into an .exe file by > Armin> py2exe. > > I didn't read it that way. My impression was that py2exe be modified to > include and enable psyco if it's available when building an .exe. You > would, in theory, get a single file distribution as well as dynamic > compilation. I haven't tried it, but using psyco in a script and building an exe from it with py2exe should work right out of the box - but maybe this isn't what you had in mind? And single-file exes with py2exe are on my plan for a medium time range, although currently I'm not really sure which way to go. Thomas ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4 news reaches interesting places
Jeremy Hylton <[EMAIL PROTECTED]> writes: > I agree, although it's not clear to me how much faster it will be in > the future. Making a *fast* Python based on our own virtual execution > environment (as opposed to piggybacking a JVM or CLR) is a big > project. It's not clear than anyone has enough resources to make > credible effort I think the EU does! Cheers, mwh -- NUTRIMAT: That drink was individually tailored to meet your personal requirements for nutrition and pleasure. ARTHUR: Ah. So I'm a masochist on a diet am I? -- The Hitch-Hikers Guide to the Galaxy, Episode 9 ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyOS_InputHook enhancement proposal
Keith Dart <[EMAIL PROTECTED]> writes: > I did modify the readline module that hooks this and can call back > to a Python function. There are also methods for installing and > removing the Python function. I did this for a different reason. I > need Python signal handlers to run, and they don't run when the > execution path is currently in some C code (such as readline). The > hook forces the interpreter to run, and check for signals as a side > effect. Using this I can be sitting in raw_input(), or interactive > mode, and signal handlers are still run. Have you seen what happened to the signal handling code in readline in Python 2.4? I don't think your modifications will be needed anymore, but you should check. Cheers, mwh -- I don't remember any dirty green trousers. -- Ian Jackson, ucam.chat ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
On Fri, 10 Dec 2004 10:06:59 +, Armin Rigo <[EMAIL PROTECTED]> wrote: > For people like myself, Linux programmers not developing on Windows every day, > there is precious little information available about how to compile our > extension modules for the new Windows distribution. I was actually very > disappointed to have to google my way around until I found a page that > explained to me that to use Microsoft's free compilers you need to manually > patch the distutils included with 2.4. (I know this is being worked on right > now but I'd have expected it to be present in the 2.4 release.) (The only > page I could find still refers to a pre-final 2.4 so their distutils patch > doesn't apply without hand-editing, though that should be fixed by now.) [...] > In other words, if you want 3rd parties to compile Windows binaries for 2.4, > tell them how. I think that the details about the Microsoft free compilers is a bit misleading. Sure, it's great that it exists, but at the present time, it is not the best free option (too many downloads required, too many rough edges). For most C extensions, the best free option is mingw. This is fully supported by distutils, and has been for a while. It is documented in the manuals, but basically all you need is to do the following: 1. Obtain a reasonably recent mingw (3.3.3 definitely works, you need msvcr71 support). 2. Install it, and make sure it is in your PATH. 3. Build libpython24.a as documented in the Python manuals (recent versions of mingw can read MS import libraries, so this may no longer be needed, but I haven't tested this yet). 4. python setup.py build --compiler=mingw32. The only difficulties are: 1. Obtaining support libraries like jpeg, sdl, etc. It *may* be a nuisance getting mingw-compatible build, but as I say above, MS-format libraries may now be fine, and there are good sources of mingw-compatible libraries like gnuwin32.sourceforge.net. 2. Microsoft-specific code. (As I'm replying to Armin, maybe the fact that inline assembler formats are different is relevant :-) :-)) To be honest, I'd only ever use the MS free compilers if I had a critical need to build an extension which had some highly MS-specific code or build processes. And even then, I'd probably give up and wait for someone with MSVC7 to produce a binary... If anyone has any particular modules (of their own, or third party) they have problems with, I'd be happy to have a go at building, and report back. Maybe a page on the Python Wiki about building modules using mingw would be worth adding. Hmm, might do that tonight... Hope this helps, Paul. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
At 10:06 AM 12/10/04 +, Armin Rigo wrote: Hi, On Wed, Dec 08, 2004 at 11:43:49PM -0500, Raymond Hettinger wrote: > Acceptance for Py2.4 partially hinges on how quickly third party apps > have their binaries updated. > > I wonder if there is anything we can do to help. For people like myself, Linux programmers not developing on Windows every day, there is precious little information available about how to compile our extension modules for the new Windows distribution. I personally find MinGW (by way of Cygwin) to be the easiest way to do it. I posted here previously with the procedure and even a script to prepare the necessary libpython24.a file. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4 news reaches interesting places
Hi Skip, On Fri, Dec 10, 2004 at 04:49:30AM -0600, Skip Montanaro wrote: > > >> The other thing we can do is finish the portable backend for psyco > >> and make it a standard module. Then Python won't be slow, it will be > >> compiled, and py2exe will be able to make a single-file executable. > > Armin> You probably mean that Psyco can dynamically compile Python > Armin> bytecodes even if they have been hidden into an .exe file by > Armin> py2exe. > > I didn't read it that way. My impression was that py2exe be modified to > include and enable psyco if it's available when building an .exe. You > would, in theory, get a single file distribution as well as dynamic > compilation. Yes, I agree with this. What I meant is that when I first read the original paragraph (the 1st one quoted above), I thought it meant that in the future py2exe and Psyco could be combined in such a way that we'd essentially have a compiler from Python producing a "classical" compiled binary. A lot of people could read it that way. The question is if we should advertise a Psyco+py2exe combination using a similar wording, such that it superficially sounds like we are doing a "classical" compilation from .py to .exe, whereas it actually means that we are hiding Psyco with the bytecodes in a .exe. After all, from a user's (or journalist's) point of view the result is similar, performancewise. Whether Psyco is reliable enough for this is yet another issue... I'd classify Psyco as "mostly reliable" only... A bientot, Armin. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
At 05:19 PM 12/10/04 +, Armin Rigo wrote: Another note: can you report on whether building libpython24.a can be skipped for mingw? I'm thinking about the specific situation where we want on-site compilation of extension modules with a minimal number of things to install first. E.g. if we need to compile libpython24.a it means we need to fetch the Python sources themselves first. Actually, no, you don't need the sources. See: http://mail.python.org/pipermail/python-dev/2004-January/041676.html for a script that builds libpython24.a from the python24.lib distributed with Python for Windows. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
On Dec 10, 2004, at 1:05 PM, Phillip J. Eby wrote: At 05:19 PM 12/10/04 +, Armin Rigo wrote: Another note: can you report on whether building libpython24.a can be skipped for mingw? I'm thinking about the specific situation where we want on-site compilation of extension modules with a minimal number of things to install first. E.g. if we need to compile libpython24.a it means we need to fetch the Python sources themselves first. Actually, no, you don't need the sources. See: http://mail.python.org/pipermail/python-dev/2004-January/041676.html for a script that builds libpython24.a from the python24.lib distributed with Python for Windows. Shouldn't this be distributed with binary distributions of Python, to save people the trouble? Or, if it can be skipped, the procedure for doing a mingw build with the .lib should be documented and that'd be the end of it. -bob ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
At 01:12 PM 12/10/04 -0500, Bob Ippolito wrote: On Dec 10, 2004, at 1:05 PM, Phillip J. Eby wrote: At 05:19 PM 12/10/04 +, Armin Rigo wrote: Another note: can you report on whether building libpython24.a can be skipped for mingw? I'm thinking about the specific situation where we want on-site compilation of extension modules with a minimal number of things to install first. E.g. if we need to compile libpython24.a it means we need to fetch the Python sources themselves first. Actually, no, you don't need the sources. See: http://mail.python.org/pipermail/python-dev/2004-January/041676.html for a script that builds libpython24.a from the python24.lib distributed with Python for Windows. Shouldn't this be distributed with binary distributions of Python, to save people the trouble? The Python developers who produce the Windows binaries don't use mingw/cygwin, so this would put a maintenance burden on them. Or, if it can be skipped, the procedure for doing a mingw build with the .lib should be documented and that'd be the end of it. It's actually documented now, in the "Installing Python Modules" manual. See: http://docs.python.org/inst/tweak-flags.html#SECTION000622000 It just would need to have the libpython.a instructions removed in that case. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: 2.4 news reaches interesting places
On Fri, 10 Dec 2004 14:01:55 -0500, A.M. Kuchling <[EMAIL PROTECTED]> wrote: > On Thu, Dec 09, 2004 at 05:11:04PM +0300, Oleg Broytmann wrote: > > some popular areas. Let's add another topic, "Making things fast". Let's > > even make it the first topic, though I personnaly dont see a need for > > this. > > The topic guides are migrating into the Wiki, and there's already a Wiki page > about this: http://www.python.org/moin/PythonSpeed The Wiki entry seems to reinforce the impression that bugged Guido to begin with. It provides a bunch of "but ..." explanations about why Python's speed isn't that important. Python is slow, but "speed of development is far more important." If you want to avoid people saying "Python is slow, but ..." you need to give them a different message. "Writing code quickly is crucial in today's marketplace." (whatever, that's not much of a message.) David's dynamic languages whitepaper has more of message, for example. Jeremy ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
On Fri, 10 Dec 2004 17:19:21 +, Armin Rigo <[EMAIL PROTECTED]> wrote: > Another note: can you report on whether building libpython24.a can be skipped > for mingw? A first attempt seems to almost work. There is a problem with structures, however - I get an error about unresolved references to _imp___Py_NoneStruct. My guess is that there's some issue with resolving references to data (as opposed to functions) from DLLs. I'll investigate further and then report back. But this is specifically if you *don't* build libpytho24.a. If you build that, there's no problem at all. Paul. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
On Fri, 10 Dec 2004 20:40:10 +, Paul Moore <[EMAIL PROTECTED]> wrote: > On Fri, 10 Dec 2004 17:19:21 +, Armin Rigo <[EMAIL PROTECTED]> wrote: > > Another note: can you report on whether building libpython24.a can be > > skipped > > for mingw? > > A first attempt seems to almost work. There is a problem with > structures, however - I get an error about unresolved references to > _imp___Py_NoneStruct. My guess is that there's some issue with > resolving references to data (as opposed to functions) from DLLs. I'll > investigate further and then report back. OK. After some more extensive investigation: 1. You cannot use python24.lib (ie, skip building libpython24.a). If you do, data values exported from python24.dll (specifically, Py_NoneStruct, which is quite important :-)) do not get exported correctly. I don't know exactly why - I guess it's something to do with how the MS LIB format works. 2. You can, however, link direct to python24.dll, which works fine. Unfortunately, there are 2 problems with this - first, distutils doesn't support it as things stand, and second, it isn't possible to get the full path of the Python DLL without using Mark Hammond's pywin32, or custom C code :-( (You need to get from sys.dllhandle to the filename, and the API to do this isn't exposed). 3. In some cases, an extension built with mingw still contains references to msvcrt (I think Martin has noted this). This seems *not* to be an issue. What is happening, is that any symbols referenced from user code are satisfied from msvcr71, and so match the Python DLL. However, if the mingw startup code uses symbols which have not been satisfied by the time all references from user code are resolved, then these references are satisfied from msvcrt. But I can't see any way that these references are going to do any harm, as they are startup code, which by its nature will have been written to work cleanly in this situation (linking like this is explicitly supported by mingw). Martin, as the CRT compatibility expert, does this sound reasonable? The mingw people certainly seem to think that the situation is OK... You could probably incorporate Philip Eby's script, posted here a while ago, which he mentioned earlier in this thread, into a setup.py so that if libpython24.a didn't already exist, the setup script would build it. This would give you a reasonably clean way of making sure it was present. Philip's script doesn't need anything but mingw installed, AIUI. Longer term, I'll see if I can put together a distutils patch to make it reference pythonXX.dll directly, so removing the need for libpythonXX.a in the future. Obviously, this is only going to be relevant for Python 2.5 and later, though. Sorry for the long mail, but I thought it would be useful to get the details in the archives... Paul. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
Phillip J. Eby wrote: The Python developers who produce the Windows binaries don't use mingw/cygwin, so this would put a maintenance burden on them. If this were completely automatic (e.g. part of msi.py), I'd happily add all utilities needed to integrated this into the build process. For this to happen: 1. Somebody else needs to do the actual work. I.e. write a patch to msi.py that invokes all the necessary magic, and test that patch. 2. The patch should avoid manual configuration of the "use cygwin" case. I.e. it should try to find the cygwin on the path, or in their standard location, or use any other mechanism available to locate cygwin. It should produce an error message if cygwin tools cannot be found, and then proceed without out creating libpython24.a Regards, Martin ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Supporting Third Party Modules (was The other Py2.4 issue)
>Raymond: > >Acceptance for Py2.4 partially hinges on how quickly third party apps >have their binaries updated. > >I wonder if there is anything we can do to help. One good way of helping out is to provide an dynamic loading function that third party modules could access the basic python functions such as PyArgParseTuple, PyString_AsString etc regardless of which python the user is running. This would be similar to the COM approach. You can load all the function pointers into a struct and then call them. Third party modules would link against this DLL independent of which python is being used. Chui ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] removing aclocal.m4
Can we remove aclocal.m4? The last log message states: fix for bug #811160 - autoconf vs. hp/ux system header files. also applied to release23-maint. Note that aclocal.m4 can go away when autoconf 2.58 is out. And configure says: # Generated by GNU Autoconf 2.59 for python 2.5. Neal ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.4 news reaches interesting places
Thomas> I haven't tried it, but using psyco in a script and building an Thomas> exe from it with py2exe should work right out of the box - but Thomas> maybe this isn't what you had in mind? I was thinking of implicitly mixing in psyco, even if the script didn't use it. Maybe I have too much faith in psyco. ;-) Skip ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
Hi, On Fri, Dec 10, 2004 at 12:06:01PM +, Paul Moore wrote: > For most C extensions, the best free option is mingw. Sorry, I was not aware that mingw supports the new VC7.1-type of runtime that is needed for the extension module to load with the official Python 2.4 distribution. Note that compiling with Microsoft's free compiler (after the huge downloads and the manual patching) worked just fine, so I think it is worth a note somewhere. Another note: can you report on whether building libpython24.a can be skipped for mingw? I'm thinking about the specific situation where we want on-site compilation of extension modules with a minimal number of things to install first. E.g. if we need to compile libpython24.a it means we need to fetch the Python sources themselves first. (A use case is for prorgams that dynamically produce and compile extension modules, as "weave" or the PyPy test suite do.) A bientot, Armin. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: 2.4 news reaches interesting places
On Thu, Dec 09, 2004 at 05:11:04PM +0300, Oleg Broytmann wrote: > some popular areas. Let's add another topic, "Making things fast". Let's > even make it the first topic, though I personnaly dont see a need for > this. The topic guides are migrating into the Wiki, and there's already a Wiki page about this: http://www.python.org/moin/PythonSpeed --amk ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: 2.4 news reaches interesting places
On Fri, Dec 10, 2004 at 02:01:55PM -0500, A.M. Kuchling wrote: > On Thu, Dec 09, 2004 at 05:11:04PM +0300, Oleg Broytmann wrote: > > some popular areas. Let's add another topic, "Making things fast". Let's > > even make it the first topic, though I personnaly dont see a need for > > this. > > The topic guides are migrating into the Wiki, and there's already a Wiki page > about this: http://www.python.org/moin/PythonSpeed Thank you. Actually, I've thought about wiki when I was writing that email, but I was afarid that people who (in my very humble opinion) are overreacting - will not tolerate putting such an important topic into a wiki. My personnal opinion is that a good topic about the speed of Python in the wiki is enough. Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
RE: [Python-Dev] Re: 2.4 news reaches interesting places
> The Wiki entry seems to reinforce the impression that bugged Guido to > begin with. It provides a bunch of "but ..." explanations about why > Python's speed isn't that important. Python is slow, but "speed of > development is far more important." I felt the same way when reading it. Also, it seemed to embody the political outlook that optimization is sinful. The document could be much more positive, fact based, and informative. Also, the wording seems somewhat outdated. A draft for a new entry is included below. Review and feedback are requested. Raymond Techniques for Improving Performance and Scalability Here are some coding guidelines for applications that demand peek performance (in terms of memory utilization, speed, or scalability). Use the best algorithms and fastest tools - . Membership testing with sets and dictionaries is much faster, O(1), than searching sequences, O(n). When testing "a in b", b should be a set or dictionary instead of a list or tuple. . String concatenation is best done with ''.join(seq) which is an O(n) process. In contrast, using the '+' or '+=' operators can result in an O(n**2) process because new strings may be built for each intermediate step. Py2.4 mitigates this issue somewhat; however, ''.join(seq) remains the best practice. . Many tools come in both list form and iterator form (map and itertools.imap, list comprehension and generator expressions, dict.items and dict.iteritems). In general, the iterator forms are more memory friendly and more scalable. They are preferred whenever a real list is not required. . Many core building blocks are coded in optimized C. Applications that take advantage of them can make substantial performance gains. The building blocks include all of the builtin datatypes (lists, tuples, sets, and dictionaries) and extension modules like array, itertools, and collections.deque. . Likewise, the builtin functions run faster than hand-built equivalents. For example, map(operator.add, v1, v2) is faster than map(lambda x,y: x+y, v1, v2). . List comprehensions run a bit faster than equivalent for-loops. . Lists perform well as fixed length arrays and as stacks. For queue applications using pop(0) or insert(0,v)), collections.deque() offers superior O(1) performance because it avoids rebuilding a full list for each insertion or deletion, an O(n) process. . Custom sort ordering is best performed with Py2.4's key= option or with the traditional decorate-sort-undecorate technique. Both approaches call the key function just once per element. In contrast, sort's cmp= option is called many times per element during a sort. For example, sort(key=str.lower) is faster than sort(cmp=lambda a,b: cmp(a.lower(), b.lower())). Take advantage of interpreter optimizations --- . In functions, local variables are accessed more quickly than global variables, builtins, and attribute lookups. So, it is sometimes worth localizing variable access in inner-loops. For example, the code for random.shuffle() localizes access with the line, random=self.random. That saves the shuffling loop from having to repeatedly lookup self.random. Outside of loops, the gain is minimal and rarely worth it. . The previous recommendation is a generalization of the rule to factor constant expressions out of loops. Also, constant folding should be done manually. Inside loops, write "x=3" instead of "x=1+2". . Function call overhead is large compared to other instructions. Accordingly, it is sometimes worth in-lining code in the middle of a few time-critical loops. . Starting with Py2.3, the interpreter optimizes "while 1" to just a single jump. In contrast "while True" takes several more steps. While the latter is preferred for clarity, time critical code should use the first form. . Multiple assignment is slower than individual assignment. For example "x,y=a,b" is slower than "x=a; y=b". However, multiple assignment is faster for variable swaps. For example, "x,y=y,x" is faster than "t=x; x=y; y=t". . Chained comparisons are faster than using the "and" operator. Write "x < y < z" instead of "x < y and y < z". . A few fast approaches should be considered hacks and reserved for only the most demanding applications. For example, "x and True or False" is faster than "bool(x)". Take advantage of diagnostic tools -- . The hotshot and profile modules help identify performance bottlenecks. . The timeit module offers immediate performance comparisons between alternative approaches to writing individual lines of code. Performance can dictate overall strategy . SAX is typically faster and more memory efficient than DOM approaches to XM
Re: [Python-Dev] The other Py2.4 issue
Armin Rigo wrote: In other words, if you want 3rd parties to compile Windows binaries for 2.4, tell them how. The straight-forward answer is: Get VC7.1 (aka VS.NET 2003), and invoke python setup.py bdist_wininst That's not too hard to do, I think. Regards, Martin ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] decorators article for DDJ?
Hi folks. Is anyone interested in doing an article on Python decorators for "Doctor Dobb's Journal"? If so, please drop me a line... Thanks, Greg ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Re: 2.4 news reaches interesting places
Raymond Hettinger wrote: guidelines for applications that demand peek performance (in terms of memory Peak performance, perhaps? :) Anyway, it looks pretty good to me, but I have a few additional ideas. Add a section of Caveats (we know they exist - might as well be upfront about it): Caveats . This document is based primarily on the CPython interpreter from www.python.org. However, most of the recommendations should apply to any Python interpreter that supports the mentioned feature. . For small data sets, some of the suggestions will perform more slowly than the approaches they are given as alternatives too. This is due to the fact that those suggestions introduce a little more fixed overhead in order to avoid overhead in processing each data item. The crossover point (where the more scaleable approach begins to show a net performance gain) is generally application specific. Use the diagnostic tools mentioned later to make an informed decision for your application. Use the best algorithms and fastest tools - > . Membership testing with sets and dictionaries is much faster, > O(1), than searching sequences, O(n). When testing "a in b", b should > be a set or dictionary instead of a list or tuple. Should the bisect module be mentioned? If you have a sorted list already, using bisect may be faster than constructing an intermediate set. Also, you can use bisect if an item's *position* in the list is important. list.index(x) uses a linear search (since it can't assume a sorted list), whereas bisect expects the list to be pre-sorted. intermediate step. Py2.4 mitigates this issue somewhat; however, ''.join(seq) remains the best practice. I'd say 'The CPython 2.4 interpreter', rather than just Py2.4. . List comprehensions run a bit faster than equivalent for-loops. I'd move this to the next section (the algorithms are the same, but the interpreter can speed up one more than the other) . Custom sort ordering is best performed with Py2.4's key= option or with the traditional decorate-sort-undecorate technique. Both approaches call the key function just once per element. In contrast, sort's cmp= option is called many times per element during a sort. For example, sort(key=str.lower) is faster than sort(cmp=lambda a,b: cmp(a.lower(), b.lower())). If sorting is going to be performed repeatedly (e.g. maintaining a list in sorted order), it may be feasible to store the list of keys, rather than regenerating them every time. This also plays nicely with using the bisect module to update the list (as the list of keys is available to determine the correct index for insertion). I can't find a 'sorted list' recipe in the Cookbook, so I might have to create one. Take advantage of interpreter optimizations --- [snip] . Prefer iteration (using for loops, list comprehensions, or generator expressions) over iterables to explicit invocation of .next() Take advantage of diagnostic tools -- . The hotshot and profile modules help identify performance bottlenecks. Currently profile is a fair bit more useful than hotshot, mainly due to the fact that it isolates time spent in C functions from the Python code that calls those functions. Perhaps another section for External Libraries? If you're doing serious number crunching in Python, using NumPy is practically a requirement. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://boredomandlaziness.skystorm.net ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The other Py2.4 issue
E.g. if we need to compile libpython24.a it means we need to fetch the Python sources themselves first. Actually, no, you don't need the sources. See: http://mail.python.org/pipermail/python-dev/2004-January/041676.html for a script that builds libpython24.a from the python24.lib distributed with Python for Windows. Previously I built libpython24.a; it can be downloaded from my website. See http://bonsai.ims.u-tokyo.ac.jp/~mdehoon/software/python/cygwin.html --Michiel. -- Michiel de Hoon, Assistant Professor University of Tokyo, Institute of Medical Science Human Genome Center 4-6-1 Shirokane-dai, Minato-ku Tokyo 108-8639 Japan http://bonsai.ims.u-tokyo.ac.jp/~mdehoon ___ Python-Dev mailing list [EMAIL PROTECTED] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
