Re: [Python-Dev] deja-vu .. python locking
Greg Ewing wrote: > Martin Devera wrote: > >> As I've written in "Big reader lock" paragraph of the original >> proposal, these >> objects could be handled by not blocking in read path > > But as was just pointed out, because of refcounting, > there's really no such thing as read-only access to > an object. What *looks* like read-only access at the > Python level involves refcount updates just from the > act of touching the object. > Yes I was thinking about atomic inc/dec (locked inc/dec in x86) as used in G.Stein's patch. I have to admit that I haven't measured its performance, I was hoping for decent one. But from http://www.linuxjournal.com/article/6993 it seems that atomic inc is rather expensive too (75ns on 1.8GHz P4) :-( Greg, what change do you have in mind regarding that "3 instruction addition" to refcounting ? thanks, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] deja-vu .. python locking
> Ah, I think I understand now. First the minor critique: I believe
> the locking algorithm isn't thread-safe:
>
> while (ob->owner_thread != self_thread()) {
>unlock_mutex(thread_mutex[self_thread()])
> // wait for owning thread to go to quiscent state
>lock_mutex(thread_mutex[ob->owner_thread])
>ob->owner_thread = self_thread()
>unlock_mutex(thread_mutex[ob->owner_thread])
>lock_mutex(thread_mutex[self_thread()])
> }
>
> If two threads are competing for the same object held by a third
> thread, they may simultaneously enter the while loop, and then
> simultaneously try to lock the owner_thread. Now, one will win,
> and own the object. Later, the other will gain the lock, and
> unconditionally overwrite ownership. This will cause two threads
> to own the objects, which is an error.
oops .. well it seems as very stupid error on my side. Yes you are
absolutely right, I'll have to rethink it. I hope it is possible
to do it in correct way...
> The more fundamental critique is: Why? It seems you do this
> to improve efficiency, (implicitly) claiming that it is
> more efficient to keep holding the lock, instead of releasing
> and re-acquiring it each time.
>
> I claim that this doesn't really matter: any reasonable
> mutex implementation will be "fast" if there is no lock
> contention. On locking, it will not invoke any system
> call if the lock is currently not held (but just
> atomically test-and-set some field of the lock); on
> unlocking, it will not invoke any system call if
> the wait list is empty. As you also need to test, there
> shouldn't be much of a performance difference.
I measured it. Lock op in futex based linux locking is of the same
speed as windows critical section and it is about 30 cycles on my
P4 1.8GHz in uncontented case.
As explained in already mentioned http://www.linuxjournal.com/article/6993
it seems due to pipeline flush during cmpxchg insn.
And there will be cacheline transfer penalty which is much larger. So
that mutex locking will take time comparable with protected code itself
(assuming fast code like dict/list read).
Single compare will take ten times less.
Am I missing something ?
thanks, Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
On Tue, Sep 19, 2006 at 03:46:59PM +1200, Greg Ewing wrote: > There really shouldn't be > any such thing as sys.path -- the view that any > given module has of the package namespace should > depend only on where it is I do not understand this. Can you show an example? Imagine I have two servers, Linux and FreeBSD, and on Linux python is in /usr/bin, home is /home/phd, on BSD these are /usr/local/bin and /usr/home/phd. I have some modules in site-packages and some modules in $HOME/lib/python. How can I move programs from one server to the other without rewriting them (how can I not to put full paths to modules)? I use PYTHONPATH manipulation - its enough to write a shell script that starts daemons once and use it for many years. How can I do this without sys.path?! Oleg. -- Oleg Broytmannhttp://phd.pp.ru/[EMAIL PROTECTED] Programmers don't die, they just GOSUB without RETURN. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Josiah Carlson wrote: > As it stands, in order to "work around" this particular feature, one > would need to write a 'loader' to handle importing and/or main() calling > in subpackage1/module1.py . Yup. At the moment, you can rely on PEP 328, or an PEP 338, but not both at the same time. This was previously discussed back in June/July with Anthony convincing me that the solution to the current poor interaction shouldn't be rushed [1]. It is, however, pretty trivial to write a runpy.run_module based launcher that will execute your module and use something other than "__name__ == '__main__'" to indicate that the module is the main module. By letting run_module set __name__ normally, relative imports will "just work". For example: #mypkg/launch.py # Runs a script, using the global _launched to indicate whether or not # the module is the main module if "_launched" not in globals(): _launched = False if (__name__ == "__main__") or _launched: import runpy # Run the module specified as the next command line argument if len(sys.argv) < 2: print >> sys.stderr, "No module specified for execution" else: del sys.argv[0] # Make the requested module sys.argv[0] run_module(sys.argv[0], init_globals=dict(_launched=True), alter_sys=True) Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2006-July/067077.html -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Greg Ewing wrote: > Armin Rigo wrote: > > >>My (limited) understanding of the motivation for relative imports is >>that they are only here as a transitional feature. Fully-absolute >>imports are the official future. > > > Guido does seem to have a dislike for relative imports, > but I don't really understand why. The usefulness of > being able to make a package self-contained and movable > to another place in the package hierarchy without hacking > it seems self-evident to me. > > What's happening in Py3k? Will relative imports still > exist? > > >>there >>is no clean way from a test module 'foo.bar.test.test_hello' to import >>'foo.bar.hello': the top-level directory must first be inserted into >>sys.path magically. > > > I've felt for a long time that problems like this > wouldn't arise so much if there were a closer > connection between the package hierarchy and the > file system structure. There really shouldn't be > any such thing as sys.path -- the view that any > given module has of the package namespace should > depend only on where it is, not on the history of > how it came to be invoked. > This does, of course, assume that you're importing modules from the filestore, which assumption is no longer valid in the presence of PEP 302 importers. The current initialization code actually looks for os.py as a means of establishing path elements. This should really be better integrated with the PEP 302 mechanism: ideally Python should work on systems that don't rely on filestore for import (even though for the foreseeable future all systems will continue to do this). regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] RELEASED Python 2.5 (FINAL)
It's been nearly 20 months since the last major release of Python (2.4), and 5 months since the first alpha release of this cycle, so I'm absolutely thrilled to be able to say: On behalf of the Python development team and the Python community, I'm happy to announce the FINAL release of Python 2.5. This is a *production* release of Python 2.5. Yes, that's right, it's finally here. Python 2.5 is probably the most significant new release of Python since 2.2, way back in the dark ages of 2001. There's been a wide variety of changes and additions, both user-visible and underneath the hood. In addition, we've switched to SVN for development and now use Buildbot to do continuous testing of the Python codebase. Much more information (as well as source distributions and Windows and Universal Mac OSX installers) are available from the 2.5 website: http://www.python.org/2.5/ The new features in Python 2.5 are described in Andrew Kuchling's What's New In Python 2.5. It's available from the 2.5 web page. Amongst the new features of Python 2.5 are conditional expressions, the with statement, the merge of try/except and try/finally into try/except/finally, enhancements to generators to produce coroutine functionality, and a brand new AST-based compiler implementation underneath the hood. There's a variety of smaller new features as well. New to the standard library are hashlib, ElementTree, sqlite3, wsgiref, uuid and ctypes. As well, a new higher-performance profiling module (cProfile) was added. Extra-special thanks on behalf of the entire Python community should go out to Neal Norwitz, who's done absolutely sterling work in shepherding Python 2.5 through to it's final release. Enjoy this new release, (and Woo-HOO! It's done!) Anthony Anthony Baxter [EMAIL PROTECTED] Python Release Manager (on behalf of the entire python-dev team) pgpA8ImA53iae.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] release25-maint branch - please keep frozen for a day or two more.
Could people please treat the release25-maint branch as frozen for a day or two, just in case we have to cut an ohmygodnononokillme release? Thanks, Anthony -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] release25-maint branch - please keep frozen for a day or two more.
Anthony Baxter wrote: > Could people please treat the release25-maint branch as frozen for a day or > two, just in case we have to cut an ohmygodnononokillme release? Thanks, Otherwise to be known as 2.5.005? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] deja-vu .. python locking
Martin Devera schrieb: > I measured it. Lock op in futex based linux locking is of the same > speed as windows critical section and it is about 30 cycles on my > P4 1.8GHz in uncontented case. > As explained in already mentioned http://www.linuxjournal.com/article/6993 > it seems due to pipeline flush during cmpxchg insn. > And there will be cacheline transfer penalty which is much larger. So > that mutex locking will take time comparable with protected code itself > (assuming fast code like dict/list read). > Single compare will take ten times less. > Am I missing something ? I'll have to wait for your revised algorithm, but likely, you will need some kind of memory barrier also, or else it can't work in the multi-processor case. In any case, if to judge whether 30 cycles is few or little, measurements of the alternative approach are necessary. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Download URL typo
Hiho, in case noone didn't notice yet: the "Windows MSI Installer" link at http://www.python.org/download/releases/2.5/ points to Python 2.4! Regards, Michael ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Download URL typo
Michael Walter schrieb: > in case noone didn't notice yet: the "Windows MSI Installer" link at > http://www.python.org/download/releases/2.5/ points to Python 2.4! Why is this a problem? The link is actually correct: The MSI documentation is the same. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Download URL typo
Martin v. Löwis schrieb: > Michael Walter schrieb: >> in case noone didn't notice yet: the "Windows MSI Installer" link at >> http://www.python.org/download/releases/2.5/ points to Python 2.4! > > Why is this a problem? The link is actually correct: The MSI > documentation is the same. I reconsidered. Even though the documentation was nearly correct (except that one limitation went away long ago), it's probably better to have the documentation state "2.5" throughout. So I copied it, changed the version numbers, and changed the links to refer to the copy. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] deja-vu .. python locking
Martin Devera wrote: > Greg, what change do you have in mind regarding that "3 instruction > addition" to refcounting ? I don't have any change in mind. If even an atomic inc is too expensive, it seems there's no hope for us. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Steve Holden wrote: > This does, of course, assume that you're importing modules from the > filestore, which assumption is no longer valid in the presence of PEP > 302 importers. Well, you need to allow for a sufficiently abstract notion of "filesystem". I haven't really thought it through in detail. It just seems as though it would be a lot less confusing if you could figure out from static information which module will get imported by a given import statement, instead of having it depend on the history of run-time modifications to sys.path. One such kind of static information is the layout of the filesystem. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] New relative import issue
Greg Ewing wrote: > Steve Holden wrote: > > >>This does, of course, assume that you're importing modules from the >>filestore, which assumption is no longer valid in the presence of PEP >>302 importers. > > > Well, you need to allow for a sufficiently abstract > notion of "filesystem". > For some value of "sufficiently" ... > I haven't really thought it through in detail. It > just seems as though it would be a lot less confusing > if you could figure out from static information which > module will get imported by a given import statement, > instead of having it depend on the history of run-time > modifications to sys.path. One such kind of static > information is the layout of the filesystem. > Less confusing, but sadly also less realistic. I suspect what's really needed is *more* importer behavior rather than less but, like you, I haven't yet thought it through in detail. All I *can* tell you is once you start importing modules for a database the whole import mechanism starts to look a bit under-specified an over-complicated. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Download URL typo
Martin v. Löwis wrote: > Martin v. Löwis schrieb: > >>Michael Walter schrieb: >> >>>in case noone didn't notice yet: the "Windows MSI Installer" link at >>>http://www.python.org/download/releases/2.5/ points to Python 2.4! >> >>Why is this a problem? The link is actually correct: The MSI >>documentation is the same. > > > I reconsidered. Even though the documentation was nearly correct > (except that one limitation went away long ago), it's probably better > to have the documentation state "2.5" throughout. So I copied it, > changed the version numbers, and changed the links to refer to the > copy. > As I write the situation is an ugly mess, since the most visible link is just plain wrong. The page http://www.python.org/download/releases/2.5/ has a block at the top right whose last link is "Windows MSI installer". That links to http://www.python.org/download/releases/2.5/msi/ which *also* has a block at the top right whose last link is "Windows MSI installer". Unfortunately that takes you to http://www.python.org/download/releases/2.5/msi/msi by which time you have completely lost contact with any style sheet, and despite the potential infinite regress have still not located the actual installer. The correct link is in-line: http://www.python.org/download/releases/2.5/python-2.5.msi I think the next time we redesign the web production system we should take the release managers' needs into consideration. They should have a simple form to fill in, with defaults already provided. As indeed should many other people ... regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] deja-vu .. python locking
Greg Ewing wrote: > Martin Devera wrote: > >> Greg, what change do you have in mind regarding that "3 instruction >> addition" to refcounting ? > > I don't have any change in mind. If even an atomic inc > is too expensive, it seems there's no hope for us. Just from curiosity, would be a big problem removing refcounting and live with garbage collection only ? I'm not sure if some parts of py code depends on exact refcnt behaviour (I guess it should not). Probably not for mainstream, but maybe as compile time option as part of freethreading solution only for those who need it. Even if you can do fast atomic inc/dec, it forces cacheline with refcounter to ping-pong between caches of referencing cpus (for read only class dicts for example) so that you can probably never get good SMP scalability. Consider main memory latency 100ns, then on 8 way 2GHz SMP system where paralel computation within the same py class is going on all cpus. When you manage to do a lot of class references in a loop, say 6400 instructions apart (quite realistic) then at least one CPU each time will block on that inc/dec, so that you lost one cpu in overhead... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] deja-vu .. python locking
Martin Devera schrieb: > Just from curiosity, would be a big problem removing refcounting and live > with garbage collection only ? I'm not sure if some parts of py code > depends on exact refcnt behaviour (I guess it should not). Now, this gives a true deja-vu. Python applications often rely on reference counting (in particular, that releasing a file object will immediately close the file), despite the language reference saying that this is not a Python feature, just one of the implementation. In addition, implementing a tracing garbage collection would either be tedious or have additional consequences on semantics: with a conservative GC, some objects may never get collected, with a precise GC, you have to declare GC roots on the C level. Things get more complicated if the GC is also compacting. See the current thread on the py3k list. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Download URL typo
Steve Holden schrieb: > That links to > >http://www.python.org/download/releases/2.5/msi/ > > which *also* has a block at the top right whose last link is "Windows > MSI installer". Unfortunately that takes you to > >http://www.python.org/download/releases/2.5/msi/msi I noticed, but my pyramid fu is not good enough to fix it. Should I submit a pyramid/web site bug report? Or can you fix it? Notice that the Highlights page behaves the same way, whereas the License and Bugs pages works correctly. I can't really spot a difference in the sources: the subnav.yml files are identical in all these. Actually, looking more closely, it appears that the "working" pages have a line subnav: !fragment subnav.yml in content.yml; this seems to make a difference. What does that line mean? Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
