Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/13/07, Greg Ewing [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: what if ... we use atomic test-and-set to handle reference counting (with a lock for those CPU architectures where we haven't written the necessary assembler fragment), then implement a lock for each mutable type and another for global state (thread state, interpreter state, etc)? Could be worth a try. A first step might be to just implement the atomic refcounting, and run that single-threaded to see if it has terribly bad effects on performance. I've done this experiment. It was about 12% on my box. Later, once I had everything else setup so I could run two threads simultaneously, I found much worse costs. All those literals become shared objects that create contention. I'm now working on an approach that writes out refcounts in batches to reduce contention. The initial cost is much higher, but it scales better too. I've currently got it to just under 50% cost, meaning two threads is a slight net gain. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Hrvoje Nikšić wrote: On Thu, 2007-09-13 at 13:15 +0200, Martin v. Löwis wrote: To put it another way, would it actually matter if the reference counts for such objects became hopelessly wrong due to non-atomic adjustments? If they drop to zero (which may happen due to non-atomic adjustments), Python will try to release the static memory, which will crash the malloc implementation. More precisely, Python will call the deallocator appropriate for the object type. If that deallocator does nothing, the object continues to live. Such objects could also start out with a refcount of sys.maxint or so to ensure that calls to the no-op deallocator are unlikely. The thought of adding references is amusing. What happens when a refcount becomes negative by overflow? I know, I should read the source ... regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Sorry, the dog ate my .sigline ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/14/07, Adam Olsen [EMAIL PROTECTED] wrote: Could be worth a try. A first step might be to just implement the atomic refcounting, and run that single-threaded to see if it has terribly bad effects on performance. I've done this experiment. It was about 12% on my box. Later, once I had everything else setup so I could run two threads simultaneously, I found much worse costs. All those literals become shared objects that create contention. It's hard to argue with cold hard facts when all we have is raw speculation. What do you think of a model where there is a global thread count that keeps track of how many threads reference an object? Then there are thread-specific reference counters for each object. When a thread's refcount goes to 0, it decrefs the object's thread count. If you did this right, hopefully there would only be cache updates when you update the thread count, which will only be when a thread first references an object and when it last references an object. I mentioned this idea earlier and it's growing on me. Since you've actually messed around with the code, do you think this would alleviate some of the contention issues? Justin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Thu, 2007-09-13 at 18:38 -0500, [EMAIL PROTECTED] wrote: Hrvoje More precisely, Python will call the deallocator appropriate for Hrvoje the object type. If that deallocator does nothing, the object Hrvoje continues to live. Such objects could also start out with a Hrvoje refcount of sys.maxint or so to ensure that calls to the no-op Hrvoje deallocator are unlikely. Maybe sys.maxint/2? You'd hate for the first incref to invoke the deallocator even if it was a no-op. ob_refcnt is signed. :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Decimal news
Hi people! After some months, Decimal is now in the trunk again. It's fully updated to the latest Cowlishaw specification, and complying with the latest test cases (from a few days ago, which even take in consideration some feedback from ours). I want to thank so much to Mark Dickinson, who made *a lot* of this work, not only the math part (he's a mathematician himself), but also a lot of cleaning and speeding up. Now we will put our hands in the documentation, for it to be 100% OK way before 2.6 arrives. Py3 will come after that. Regards, -- .Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily Windows Installers
Martin v. Löwis schrieb: Together with David Bolen, I set up a series of buildbot slaves that create an MSI installer from the 2.5, 2.6, and 3.0 branches every day. The result files are available from http://www.python.org/dev/daily-msi/ The buildbot pages themselves are at http://www.python.org/dev/buildbot/msi/ There are still some glitches with that installation (in particular, the Microsoft help compiler seems to crash occasionally). I hope this isn't due to the files that Sphinx creates. I had a nasty crash with HTML Help Workshop when I generated an invalid index file -- but this was reproducible of course. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily Windows Installers
On 14/09/2007, Martin v. Löwis [EMAIL PROTECTED] wrote: Together with David Bolen, I set up a series of buildbot slaves that create an MSI installer from the 2.5, 2.6, and 3.0 branches every day. That's good news. Thanks for doing this. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily Windows Installers
I hope this isn't due to the files that Sphinx creates. I had a nasty crash with HTML Help Workshop when I generated an invalid index file -- but this was reproducible of course. It's not clear what precisely the problem is, but yes, it must have to do with the input :-) If you fixed that problem fairly recently (within the last 48 hours), this may have been the one we were seeing. Unfortunately, this is again one of the Windows problems which make buildbot on Windows so difficult: it brings up an error window, and then hangs. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/14/07, Justin Tulloss [EMAIL PROTECTED] wrote: On 9/14/07, Adam Olsen [EMAIL PROTECTED] wrote: Could be worth a try. A first step might be to just implement the atomic refcounting, and run that single-threaded to see if it has terribly bad effects on performance. I've done this experiment. It was about 12% on my box. Later, once I had everything else setup so I could run two threads simultaneously, I found much worse costs. All those literals become shared objects that create contention. It's hard to argue with cold hard facts when all we have is raw speculation. What do you think of a model where there is a global thread count that keeps track of how many threads reference an object? Then there are thread-specific reference counters for each object. When a thread's refcount goes to 0, it decrefs the object's thread count. If you did this right, hopefully there would only be cache updates when you update the thread count, which will only be when a thread first references an object and when it last references an object. I mentioned this idea earlier and it's growing on me. Since you've actually messed around with the code, do you think this would alleviate some of the contention issues? There would be some poor worst-case behaviour. In the case of literals you'd start referencing them when you call a function, then stop when the function returns. Same for any shared datastructure. I think caching/buffering refcounts in general holds promise though. My current approach uses a crude hash table as a cache and only flushes when there's a collision or when the tracing GC starts up. So far I've only got about 50% of the normal performance, but that's with 90% or more scalability, and I'm hoping to keep improving it. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
At 1:51 AM -0500 9/14/07, Justin Tulloss wrote: On 9/14/07, Adam Olsen mailto:[EMAIL PROTECTED][EMAIL PROTECTED] wrote: Could be worth a try. A first step might be to just implement the atomic refcounting, and run that single-threaded to see if it has terribly bad effects on performance. I've done this experiment. It was about 12% on my box. Later, once I had everything else setup so I could run two threads simultaneously, I found much worse costs. All those literals become shared objects that create contention. It's hard to argue with cold hard facts when all we have is raw speculation. What do you think of a model where there is a global thread count that keeps track of how many threads reference an object? Then there are thread-specific reference counters for each object. When a thread's refcount goes to 0, it decrefs the object's thread count. If you did this right, hopefully there would only be cache updates when you update the thread count, which will only be when a thread first references an object and when it last references an object. It's likely that cache line contention is the issue, so don't glom all the different threads' refcount for an object into one vector. Keep each thread's refcounts in a per-thread vector of objects, so only that thread will cache that vector, or make refcounts so large that each will be in its own cache line (usu. 64 bytes, not too horrible for testing purposes). I don't know all what would be required for separate vectors of refcounts, but each object could contain its index into the vectors, which would all be the same size (Go Virtual Memory!). I mentioned this idea earlier and it's growing on me. Since you've actually messed around with the code, do you think this would alleviate some of the contention issues? Justin Your idea can be combined with the maxint/2 initial refcount for non-disposable objects, which should about eliminate thread-count updates for them. -- TonyN.:' mailto:[EMAIL PROTECTED] ' http://www.georgeanelson.com/___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Summary of Tracker Issues
ACTIVITY SUMMARY (09/07/07 - 09/14/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1274 open (+24) / 11372 closed (+11) / 12646 total (+35) Average duration of open issues: 672 days. Median duration of open issues: 640 days. Open Issues Breakdown open 1270 (+24) pending 4 ( +0) Issues Created Or Reopened (35) ___ OpenSSL detection broken for Python 3.0a109/07/07 CLOSED http://bugs.python.org/issue1129created pythonmeister Idle - Save (buffer) - closes IDLE and does not save file (Windo 09/08/07 http://bugs.python.org/issue1130created infixum Reference Manual: for statement links to break statement 09/08/07 http://bugs.python.org/issue1131created Martoon compile error in poplib.py 09/08/07 CLOSED http://bugs.python.org/issue1132created andre python3.0-config raises SyntaxError 09/09/07 CLOSED http://bugs.python.org/issue1133created complex Parsing a simple script eats all of your memory 09/09/07 http://bugs.python.org/issue1134created complex xview/yview of Tix.Grid is broken09/09/07 http://bugs.python.org/issue1135created ocean-city Bdb documentation09/09/07 CLOSED http://bugs.python.org/issue1136created arklad pyexpat patch for changing buffer_size 09/09/07 http://bugs.python.org/issue1137created AchimGaedke Fixer needed for __future__ imports 09/09/07 http://bugs.python.org/issue1138created collinwinter PyFile_Encoding should be PyFile_SetEncoding 09/10/07 CLOSED http://bugs.python.org/issue1139created gagenellina re.sub returns str when processing empty unicode string 09/10/07 http://bugs.python.org/issue1140created beda reading large files 09/10/07 http://bugs.python.org/issue1141created [EMAIL PROTECTED] code sample showing errors reading large files with py 2.5/3.0 09/10/07 http://bugs.python.org/issue1142created [EMAIL PROTECTED] Update to latest ElementTree in Python 2.6 09/11/07 http://bugs.python.org/issue1143created effbot parsermodule validation out of sync with Grammar 09/11/07 http://bugs.python.org/issue1144created dbinger Allow str.join to join non-string types (as per PEP 3100)09/11/07 http://bugs.python.org/issue1145created thomas.lee TextWrap vs words 1-character shorter than the width 09/11/07 http://bugs.python.org/issue1146created sam string exceptions inconsistently deprecated/disabled 09/11/07 CLOSED http://bugs.python.org/issue1147created exarkun TypeError on join - httplib mixing str and bytes 09/11/07 CLOSED http://bugs.python.org/issue1148created eopadoan fdopen does not work as expected 09/11/07 http://bugs.python.org/issue1149created [EMAIL PROTECTED] Rename PyBUF_WRITEABLE to PyBUF_WRITABLE 09/11/07 http://bugs.python.org/issue1150created gvanrossum TypeError: expected string, bytes found instead of KeyboardInt 09/11/07 http://bugs.python.org/issue1151created eopadoan Bug in documentation for SimpleXMLRPCServer 09/12/07 CLOSED http://bugs.python.org/issue1152created FrankMillman help(pickle) fails: unorderable types: type() type() 09/12/07 CLOSED http://bugs.python.org/issue1153created Qrczak Carbon.CF memory leak09/12/07 CLOSED http://bugs.python.org/issue1154created hhas Carbon.CF memory management problem 09/12/07 http://bugs.python.org/issue1155created hhas Suggested change to _exit function description in os module docu 09/12/07 http://bugs.python.org/issue1156created jtonsing test_urllib2net fails on test_ftp09/12/07 http://bugs.python.org/issue1157created gvanrossum %f format for datetime objects 09/13/07 http://bugs.python.org/issue1158
Re: [Python-Dev] base64 -- should b64encode introduce line breaks?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sep 13, 2007, at 2:43 PM, Bill Janssen wrote: I see that base64.b64encode and base64.standard_b64encode no longer introduce line breaks into the output strings, as base64.encodestring does. Shouldn't there be an option on one of them to do this? See: http://mail.python.org/pipermail/python-bugs-list/2001-October/ 007856.html section 2.1 of http://www.faqs.org/rfcs/rfc3548.html Perhaps adding MIME_b64encode() and PEM_b64encode() routines? Or just an optional parameter to standard_b64encode, called max_line_length, defaulting to 0, meaning no max? It turns out to be inconvenient in other contexts to do the line splitting at this lower level, so I would prefer to leave the current methods as is (that means, no change in semantics or arguments). I wouldn't necessarily be opposed to new functions that did the line splitting, but ideally, you could design an API that provided that behavior for any of the existing alternatives in the base64 module, without duplicating them all. It's not clear to me how you'd do that though (or if it's worth it). - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iQCUAwUBRurURHEjvBPtnXfVAQLSZAP4pdn3lUPvVfqSl+RT4GBzYmL1uUTMrmJx +lc+7SEaOj0sphfQbTmN9kKlwS2cJQ7UdZQzXM6t5+zlM+b4GRl6pA0CEk/M3PUI VWs3JkxgMRQA0CoeF5AflLru7ZxEL7pYej88y9KPAZCQ7H6e0+b8TCr/6Qj0YiYw c2eLfZoSAA== =klKj -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Your idea can be combined with the maxint/2 initial refcount for non-disposable objects, which should about eliminate thread-count updates for them. -- I don't really like the maxint/2 idea because it requires us to differentiate between globals and everything else. Plus, it's a hack. I'd like a more elegant solution if possible. Justin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] base64 -- should b64encode introduce line breaks?
I see that base64.b64encode and base64.standard_b64encode no longer introduce line breaks into the output strings, as base64.encodestring does. Shouldn't there be an option on one of them to do this? See: http://mail.python.org/pipermail/python-bugs-list/2001-October/ 007856.html section 2.1 of http://www.faqs.org/rfcs/rfc3548.html Perhaps adding MIME_b64encode() and PEM_b64encode() routines? Or just an optional parameter to standard_b64encode, called max_line_length, defaulting to 0, meaning no max? It turns out to be inconvenient in other contexts to do the line splitting at this lower level, so I would prefer to leave the current methods as is (that means, no change in semantics or arguments). I wouldn't necessarily be opposed to new functions that did the line splitting, but ideally, you could design an API that provided that behavior for any of the existing alternatives in the base64 module, without duplicating them all. It's not clear to me how you'd do that though (or if it's worth it). I think that's probably right. I just added the PEM line-wrapping to the code in the ssl module. Though I hate to keep adding line-wrapping code here and there... Perhaps just adding a utility function, wrap_lines(), or some such to the module would suffice. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily Windows Installers
Georg Brandl [EMAIL PROTECTED] writes: I hope this isn't due to the files that Sphinx creates. I had a nasty crash with HTML Help Workshop when I generated an invalid index file -- but this was reproducible of course. The really annoying thing is that this only occurs (so far) in the 3.0 tree when run beneath the buildbot, although it seems consistent there. Using the same tree right after a crash, and running the same build command interactively always seems to work fine. I thought it might be a stdout/console thing but redirecting the compiler's output to a file still crashes. I think, but can't prove it has parsed all the input files, since the last bit of output even in verbose mode is still buffered in its process when it crashes. I did determine that genindex.html is being created with malformed HTML ( and in operators aren't being quoted as lt; and gt;), but manually fixing that didn't resolve the crash. And even in the 2.6 branch (which builds fine) genindex.html has erroneous uses of protocol that isn't quoted either. For the moment I'm probably going to work to ensure we don't get the pop-up box (which blocks the rest of the processing) so at least an MSI can get created even if the chm is bad. -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Fri, 14 Sep 2007 14:13:47 -0500, Justin Tulloss [EMAIL PROTECTED] wrote: Your idea can be combined with the maxint/2 initial refcount for non-disposable objects, which should about eliminate thread-count updates for them. -- I don't really like the maxint/2 idea because it requires us to differentiate between globals and everything else. Plus, it's a hack. I'd like a more elegant solution if possible. It's not really a solution either. If your program runs for a couple minutes and then exits, maybe it won't trigger some catastrophic behavior from this hack, but if you have a long running process then you're almost certain to be screwed over by this (it wouldn't even have to be *very* long running - a month or two could do it on a 32bit platform). Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] base64 -- should b64encode introduce line breaks?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sep 14, 2007, at 3:20 PM, Bill Janssen wrote: I think that's probably right. I just added the PEM line-wrapping to the code in the ssl module. Though I hate to keep adding line-wrapping code here and there... Perhaps just adding a utility function, wrap_lines(), or some such to the module would suffice. Does anything in textwrap already do the trick? If not, that might be the best place to refactor similar code to. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRurm8HEjvBPtnXfVAQIGFwP9G/hYbZ/cR7n9X4NtKATFqm/Mp+q8SH3b jFUEuvf/y0/0Ri6aKpC9QJzLNg+ZlgthmaYNRT488SXPplbB4mtysFbJg+A9x3d3 fi4rkqXnrvJt6Msqbti7wt6sGYZRisDveztuKM5Sh8t+die+55e3bZg7ght6Vyuk +N6V9lg2/3A= =iknB -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] SSL and asyncore update (and SSL_shutdown)
I've now got an HTTPS server (more importantly, one built on asyncore and SocketServer and BaseHTTPServer), running in the test suite. Also, I think that, for the moment, I'm going to take ssl.ssl_shutdown() out of the library. The state machine implemented at the GoogleSprint really only does the client side of SSL_shutdown(); it will take a bit more work for the server side. I'll check this in this weekend, and update the 2.3-compatible package. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Jean-Paul Calderone wrote: On Fri, 14 Sep 2007 14:13:47 -0500, Justin Tulloss [EMAIL PROTECTED] wrote: Your idea can be combined with the maxint/2 initial refcount for non-disposable objects, which should about eliminate thread-count updates for them. -- I don't really like the maxint/2 idea because it requires us to differentiate between globals and everything else. Plus, it's a hack. I'd like a more elegant solution if possible. It's not really a solution either. If your program runs for a couple minutes and then exits, maybe it won't trigger some catastrophic behavior from this hack, but if you have a long running process then you're almost certain to be screwed over by this (it wouldn't even have to be *very* long running - a month or two could do it on a 32bit platform). Could each class define the value to be added to or subtracted from the refcount? We'd only need a bit to store the value (since it would always be zero or one), but the execution time might increase quite a lot if there's no nifty way to conditionally add or subtract one. regards Steve -- Steve Holden+1 571 484 6266 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Sorry, the dog ate my .sigline ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily Windows Installers
David Bolen schrieb: Georg Brandl [EMAIL PROTECTED] writes: I hope this isn't due to the files that Sphinx creates. I had a nasty crash with HTML Help Workshop when I generated an invalid index file -- but this was reproducible of course. The really annoying thing is that this only occurs (so far) in the 3.0 tree when run beneath the buildbot, although it seems consistent there. Using the same tree right after a crash, and running the same build command interactively always seems to work fine. I thought it might be a stdout/console thing but redirecting the compiler's output to a file still crashes. I think, but can't prove it has parsed all the input files, since the last bit of output even in verbose mode is still buffered in its process when it crashes. Can't help you there, just notice that this is the same point where I saw my crash. I did determine that genindex.html is being created with malformed HTML ( and in operators aren't being quoted as lt; and gt;), but manually fixing that didn't resolve the crash. And even in the 2.6 branch (which builds fine) genindex.html has erroneous uses of protocol that isn't quoted either. Okay, I should really fix this. Added a todo-list item. For the moment I'm probably going to work to ensure we don't get the pop-up box (which blocks the rest of the processing) so at least an MSI can get created even if the chm is bad. Thanks for handling this! Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] import file extensions
I think you're looking for a PEP 302 style meta hook. On 9/14/07, tomer filiba [EMAIL PROTECTED] wrote: a quick question: i'm working on a pythonic build system, where the build scripts are plain python files. but i want to differentiate them from normal python files (.py) by a different suffix (say .pyy), but then i can't import them. so i'm wondering, is there a quick way to just add another extension to import mechanism? or do i have to write a fully fledged import hook? -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] import file extensions
so i'm wondering, is there a quick way to just add another extension to import mechanism? or do i have to write a fully fledged import hook? [this question is off-topic for python-dev] If recompiling Python is an option, the quick way is to edit the interpreter, and add that extension. If that is not an option, but it is an option to put all .pyy files in a single directory, the quick way is to add an entry to sys.path_hooks. If that is also not an option, the quick way is to add an entry to sys.meta_path. The best way would be to not use import, but provide a separate function (e.g. calling it require). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [python] Re: import file extensions
Guido van Rossum wrote: I think you're looking for a PEP 302 style meta hook. Or even execfile in a context... Michael Foord http://www.manning.com/foord On 9/14/07, tomer filiba [EMAIL PROTECTED] wrote: a quick question: i'm working on a pythonic build system, where the build scripts are plain python files. but i want to differentiate them from normal python files (.py) by a different suffix (say .pyy), but then i can't import them. so i'm wondering, is there a quick way to just add another extension to import mechanism? or do i have to write a fully fledged import hook? -tomer ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] import file extensions
On 9/14/07, Martin v. Löwis [EMAIL PROTECTED] wrote: The best way would be to not use import, but provide a separate function (e.g. calling it require). yepp, that's probably the cleanest and quickest solution. i needed to see all the alternatives to realize this though. sorry. -- An NCO and a Gentleman ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Daily Windows Installers
Georg Brandl [EMAIL PROTECTED] writes: David Bolen schrieb: Georg Brandl [EMAIL PROTECTED] writes: (...) For the moment I'm probably going to work to ensure we don't get the pop-up box (which blocks the rest of the processing) so at least an MSI can get created even if the chm is bad. Thanks for handling this! I hit it with a sledge-hammer and modified my build slave to disable error boxes for anything it runs, so we'll get the 3.0 MSI now but with a bad chm until it gets figured out. -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On Fri, 14 Sep 2007 17:43:39 -0400, James Y Knight [EMAIL PROTECTED] wrote: On Sep 14, 2007, at 3:30 PM, Jean-Paul Calderone wrote: On Fri, 14 Sep 2007 14:13:47 -0500, Justin Tulloss [EMAIL PROTECTED] wrote: Your idea can be combined with the maxint/2 initial refcount for non-disposable objects, which should about eliminate thread-count updates for them. -- I don't really like the maxint/2 idea because it requires us to differentiate between globals and everything else. Plus, it's a hack. I'd like a more elegant solution if possible. It's not really a solution either. If your program runs for a couple minutes and then exits, maybe it won't trigger some catastrophic behavior from this hack, but if you have a long running process then you're almost certain to be screwed over by this (it wouldn't even have to be *very* long running - a month or two could do it on a 32bit platform). Not true: the refcount becoming 0 only calls a dealloc function.. For objects which are not deletable, the dealloc function should simply set the refcount back to maxint/2. Done. So, eg, replace the Py_FatalError in none_dealloc with an assignment to ob_refcnt? Good point, sounds like it could work (I'm pretty sure you know more about deallocation in CPython than I :). Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] base64 -- should b64encode introduce line breaks?
Does anything in textwrap already do the trick? If not, that might be the best place to refactor similar code to. Yes, textwrap.fill. Thanks for pointing it out. Bill ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/14/07, Jean-Paul Calderone [EMAIL PROTECTED] wrote: On Fri, 14 Sep 2007 17:43:39 -0400, James Y Knight [EMAIL PROTECTED] wrote: On Sep 14, 2007, at 3:30 PM, Jean-Paul Calderone wrote: On Fri, 14 Sep 2007 14:13:47 -0500, Justin Tulloss [EMAIL PROTECTED] wrote: Your idea can be combined with the maxint/2 initial refcount for non-disposable objects, which should about eliminate thread-count updates for them. -- I don't really like the maxint/2 idea because it requires us to differentiate between globals and everything else. Plus, it's a hack. I'd like a more elegant solution if possible. It's not really a solution either. If your program runs for a couple minutes and then exits, maybe it won't trigger some catastrophic behavior from this hack, but if you have a long running process then you're almost certain to be screwed over by this (it wouldn't even have to be *very* long running - a month or two could do it on a 32bit platform). Not true: the refcount becoming 0 only calls a dealloc function.. For objects which are not deletable, the dealloc function should simply set the refcount back to maxint/2. Done. So, eg, replace the Py_FatalError in none_dealloc with an assignment to ob_refcnt? Good point, sounds like it could work (I'm pretty sure you know more about deallocation in CPython than I :). As I've said, this is all moot. The cache coherence protocols on x86 means this will be nearly as slow as proper atomic refcounting, and will not scale if multiple threads regularly touch the object. My experience is that they will touch it regularly. -- Adam Olsen, aka Rhamphoryncus ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Justin Tulloss wrote: What do you think of a model where there is a global thread count that keeps track of how many threads reference an object? I've thought about that sort of thing before. The problem is how you keep track of how many threads reference an object, without introducing far more overhead than you're trying to eliminate. Then there are thread-specific reference counters for each object. What happens when a new thread comes into existence? Do you go through all existing objects and add another element to their refcount arrays? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] base64 -- should b64encode introduce line breaks?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sep 14, 2007, at 6:01 PM, Bill Janssen wrote: Does anything in textwrap already do the trick? If not, that might be the best place to refactor similar code to. Yes, textwrap.fill. Thanks for pointing it out. /me tries to remember that for Py3k's email package. ;) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRusNRnEjvBPtnXfVAQIpNgP/SosDYX/GDMolxcv3U2WrGzMjQa+gd8ai J/Oaw2vdSf8H84eU9ziKaWHQtK0obS9XrnUTLUDyfAKObNffZVvldG1KUV9vAKhr 3JuNJ3xiIk7RKXdkKd5mA7SXXqRd80NVN26Za0H8bkl16mhdpZM7OqJmhaIkCkXr AJtjJP5esWQ= =v/C8 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Adam Olsen wrote: I'm now working on an approach that writes out refcounts in batches to reduce contention. The initial cost is much higher, but it scales better too. I've currently got it to just under 50% cost, meaning two threads is a slight net gain. http://www.research.ibm.com/people/d/dfb/publications.html Look at the various papers on 'Recycler'. The way it works is that for each thread, there is an addref buffer and a decref buffer. The buffers are arrays of pointers. Each time a reference is addref'd, its appended to the addref buffer, likewise for decref. When a buffer gets full, it is added to a queue and then a new buffer is allocated. There is a background thread that actually applies the refcounts from the buffers and frees the objects. Since this background thread is the only thread that ever touches the actual refcount field of the object, there's no need for locking. -- Talin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
