Re: [Python-Dev] Draft PEP: Simplified Package Layout and Partitioning
Antoine Pitrou wrote: The additional confusion lies in the fact that a module can be shadowed by something which is not a module (a mere global variable). I find it rather baffling. I think we're stuck with that as long as we use the same syntax for importing a submodule and importing a non-module name from a module, i.e. 'from x import y'. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Draft PEP: Simplified Package Layout and Partitioning
P.J. Eby wrote: from x import y means import x; y = x.y. It actually means slightly more that that if y is a submodule, in which case it means import x.y; y = x.y. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Import lock considered mysterious
I recently encountered a very mysterious bug in which
a background thread would inexplicably hang while attempting
to make a connection using httplib.
Debugging as far as I could at the Python level led to
the surprising conclusion that it was hanging while
using the ascii codec to encode the host name.
I couldn't imagine why, until I resorted to breaking into
it with gdb and found that it was trying to import
codecs.ascii, and blocked on acquiring the import lock.
The reason for *that* was that my main module was a stub
that imported the real main module, which did all its
work directly from the module code. So the whole program
was effectively running inside an import statement and
holding onto the import lock.
Once I realised what was happening, it was easy to fix,
but I'm a bit disturbed about how difficult it was to
track down the cause.
This whole episode seems to have resulted from a collision
between two rather obscure things: that import statements
involve locking things, and that some fairly innocuous
looking calls, such as s.encode('ascii'), can trigger an
import.
I'm wondering whether anything can be done to make problems
like this less likely to occur, or at least to make the
cause more readily apparent. One shouldn't really have to
use gdb to track down bugs in Python code.
Is it really necessary to hold the import lock for so long?
Presumably the import lock is there to protect access to
things like sys.modules. Is that all? Could it be released
after the module code is loaded and sys.modules has been
updated, but before executing the module body?
--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python launcher command line usage (Was: 3.2.1 encoding surprise)
On 22/07/2011 02:30, Vlad Riscutia wrote: If versioned filenames are added in addition to python.exe, it still might look confusing for most users: Why do I have python and python3.2 executables? What's the difference? I'd rather go with -v argument either way, for people that /know/ they want to call Python 3.2 instead of Python 3.1... It doesn't seem to be too confusing for Linux / Mac OS X users where you have both. What's more it's very useful. I still like being able to specify version from the launcher, it's probably what I will use it most for (on the rare occasions I still use Windows). Michael Thank you, Vlad On Thu, Jul 21, 2011 at 6:07 PM, Éric Araujo mer...@netwok.org mailto:mer...@netwok.org wrote: Hi, Le 22/07/2011 03:03, Vlad Riscutia a écrit : I'm kind of -1 on changing Python executable name. It would make sense for different major versions, where there are known incompatibilities, so python2-python3 would make sense but python31 python32 not that much... If my team is using Python and it gets pre-installed with other dev-tools, do I need to let everyone know they must call python*31*? And if we upgrade, make sure everyone knows they should now call python*32*? What if we have scripts that call python? Make sure we update all of them whenever minor version is changed? If I understand correctly, adding versioned filenames like python3.3.exe would happen in addition of python.exe, not in replacement. Regards ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import lock considered mysterious
On 22Jul2011 21:29, Greg Ewing greg.ew...@canterbury.ac.nz wrote:
[...]
| This whole episode seems to have resulted from a collision
| between two rather obscure things: that import statements
| involve locking things,
Necessary to avoid performing the module definitons twice when a module
is imported twice, surely?
| and that some fairly innocuous
| looking calls, such as s.encode('ascii'), can trigger an
| import.
There are plenty of occasions where an import might be deferred until a
function call - it is a common workaround for otherwise circular
imports. Personally I also sometimes defer an import if I'm importing
something large that the module would only depend upon on in unusual
(or anyway, often avoidable) circumstances; for example, a facility not
in the stdlib, which a caller might avoid requiring by choosing a
different aspect of the importing module such as choosing a CSV storage
backend over and SQLalchemy backend.
| Is it really necessary to hold the import lock for so long?
| Presumably the import lock is there to protect access to
| things like sys.modules. Is that all? Could it be released
| after the module code is loaded and sys.modules has been
| updated, but before executing the module body?
Sometimes names are made by executing the module body.
Since one expects to access the module's names after the import, how is
this avoidable?
Cheers,
--
Cameron Simpson c...@zip.com.au DoD#743
http://www.cskk.ezoshosting.com/cs/
What's the point of having Nebraska if we can't put it to its highest and
best use? - Patrick Bedard arguing for a 100 mph speed limit
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import lock considered mysterious
On 22 July 2011 10:29, Greg Ewing greg.ew...@canterbury.ac.nz wrote: The reason for *that* was that my main module was a stub that imported the real main module, which did all its work directly from the module code. So the whole program was effectively running inside an import statement and holding onto the import lock. I realise you probably know this, but I think this is covered by the standard advice to avoid having substantial code running at module level (i.e., on import). It may be useful to clarify or further emphasize that advice in the Python documentation. Paul. PS This is not to say that I oppose trying to reduce the chance of this type of thing happening... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import lock considered mysterious
On Fri, 22 Jul 2011 21:29:23 +1200
Greg Ewing greg.ew...@canterbury.ac.nz wrote:
This whole episode seems to have resulted from a collision
between two rather obscure things: that import statements
involve locking things, and that some fairly innocuous
looking calls, such as s.encode('ascii'), can trigger an
import.
Indeed.
I'm wondering whether anything can be done to make problems
like this less likely to occur, or at least to make the
cause more readily apparent. One shouldn't really have to
use gdb to track down bugs in Python code.
See http://bugs.python.org/issue9260
There's a patch there but it needs additional sophistication to remove
deadlocks when doing concurrent circular imports.
Regards
Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python launcher command line usage (Was: 3.2.1 encoding surprise)
On Thu, Jul 21, 2011 at 20:30, Vlad Riscutia riscutiav...@gmail.com wrote: If versioned filenames are added in addition to python.exe, it still might look confusing for most users: Why do I have python and python3.2 executables? What's the difference? I'd rather go with -v argument either way, for people that *know* they want to call Python 3.2 instead of Python 3.1... Thank you, Vlad Honestly, would it really be that confusing? Seeing python32.exe inside C:\Python32 shouldn't be a huge surprise, and ActiveState has been doing something like this for years (forever?). Versioned executables in addition to the standard python.exe is something I've wanted for a while, but that's outside of this PEP. This way you could have C:\Python27 and C:\Python32 on your path and explicitly open up the right one. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import lock considered mysterious
On Fri, Jul 22, 2011 at 7:29 PM, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Is it really necessary to hold the import lock for so long? Presumably the import lock is there to protect access to things like sys.modules. Is that all? Could it be released after the module code is loaded and sys.modules has been updated, but before executing the module body? No, because we don't know if we're keeping it until the module body is executed successfully and we don't want other threads to see the partially initialised module. There's a reason the docs say never to spawn and then wait for threads as a side effect of import (http://docs.python.org/library/threading#importing-in-threaded-code) If you want to delegate main module execution to another file, then runpy.run_module is available. Antoine's fine grained import locking patch (or something along those lines) would improve matters, but currently causes its own deadlock problems. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import lock considered mysterious
At 02:48 PM 7/22/2011 +0200, Antoine Pitrou wrote: See http://bugs.python.org/issue9260 There's a patch there but it needs additional sophistication to remove deadlocks when doing concurrent circular imports. I don't think that approach can work, as PEP 302 loaders can currently assume the global import lock is being held when they run... and in general, there are too many global data structures in sys that need to be protected by code that uses such things. A simpler solution to Greg's problem would be to have a timeout on attempts to acquire the import lock, and have it fail with a RuntimeError describing the problem. (*Not* an ImportError, mind you, which might get ignored and trigger a different code path.) The timeout would need to be on the order of seconds to prevent false positives, and there'd need to be a way to change or remove the timeout in the event somebody really needs to. But it would eliminate the mysteriousness. A unique and Google-able error message would let someone find a clear explanation of what's going on, as well. A second thing that *could* be helpful would be to issue a warning when a new thread is started (or waited on) while the import lock is held. This is already known to be a bad thing to do. The tricky part is issuing the warning for the right caller level, but I suppose you could walk back up the call stack until you found some module-level code, and then fingered that line of code as the culprit. Yes, that might do it: the code for starting or waiting on a thread, could check to see if the import lock is held by the current thread, and if so, walk up the stack to find a module frame (one where f_globals is f_locals and __name__ in f_locals and sys.modules[__name__].__dict__ is f_locals), and if one is found, issue a warning about not starting or waiting on threads in module-level code. Between that and the timeout, the mysteriousness could be completely done away with, without throwing a monkey wrench into the current import mechanisms. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Import lock considered mysterious
See http://bugs.python.org/issue9260 There's a patch there but it needs additional sophistication to remove deadlocks when doing concurrent circular imports. I don't think that approach can work, as PEP 302 loaders can currently assume the global import lock is being held when they run... and in general, there are too many global data structures in sys that need to be protected by code that uses such things. Some of the comments in the issue are about that. The global import lock could *still* protect those structures when needed. It just doesn't have to be held when executing a module's body. Between that and the timeout, the mysteriousness could be completely done away with, without throwing a monkey wrench into the current import mechanisms. Isn't the current import mechanism already a monkey wrench? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python launcher command line usage (Was: 3.2.1 encoding surprise)
OK then. I don't have a *strong* opinion against it, just thought that most people have one version of Python, maybe 2 versions as in 2.x and 3.x, so I would understand python2.exe, python3.exe but yeah, it's not that big of a deal either way. Thank you, Vlad On Fri, Jul 22, 2011 at 6:41 AM, Brian Curtin brian.cur...@gmail.comwrote: On Thu, Jul 21, 2011 at 20:30, Vlad Riscutia riscutiav...@gmail.comwrote: If versioned filenames are added in addition to python.exe, it still might look confusing for most users: Why do I have python and python3.2 executables? What's the difference? I'd rather go with -v argument either way, for people that *know* they want to call Python 3.2 instead of Python 3.1... Thank you, Vlad Honestly, would it really be that confusing? Seeing python32.exe inside C:\Python32 shouldn't be a huge surprise, and ActiveState has been doing something like this for years (forever?). Versioned executables in addition to the standard python.exe is something I've wanted for a while, but that's outside of this PEP. This way you could have C:\Python27 and C:\Python32 on your path and explicitly open up the right one. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 3.2.1 encoding surprise
On 7/21/2011 5:44 PM, Mark Hammond wrote: On 22/07/2011 9:02 AM, Glenn Linderman wrote: Bad logic is get_configured_value! get_configured_value only looks in the global configuration file if there is a local configuration file that doesn't have the setting. It should look in the global configuration file if there is no local configuration file _OR_ the is a local configuration file without the setting. I'll await a new launcher. I just pushed a fix and hopefully Vinay will push a new MSI soon. Thanks, Mark What (free?) toolset is needed for building the launcher? I don't even have a C compiler installed on this computer yet. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
