Re: Portable code: __import__ demands different string types between 2 and 3
On Sun, Dec 14, 2014 at 11:29 PM, Ben Finney ben+pyt...@benfinney.id.au wrote:
I'm slowly learning about making Python code that will run under both
Python 2 (version 2.6 or above) and Python 3 (version 3.2 or above).
This entails, I believe, the admonition to ensure text literals are
Unicode by default::
from __future__ import unicode_literals
Ordinarily, for 2.x/3.3+ code I would suggest not doing this --
instead, b'...' for bytes, u'...' for unicode, and '...' for native
string type (bytes on 2.x, unicode on 3.x). This is the main benefit
of the u'...' syntax addition.
For 3.2, you'll have to do b'...' for bytes, '...' for unicode, and
str('...') for platform-specific strings (bytes on 2.x, unicode on
3.x). It is in good taste to make an alias of str so it's less
confusing, e.g. native_str = str.
So for example, it's __import__(str(foo)), and getattr(foo, str(bar), baz).
-- Devin
--
https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: __import__ demands different string types between 2 and 3
Ben Finney wrote: Howdy all, What should I do, in a world where all text literals are Unicode by default, to make ‘__import__’ work in both Python 2 and 3? One approach I frequently use is a conditional import. Off the top of my head, I'd do something like this: try: import builtins # Python 3.x. except ImportError: # We're probably running Python 2.x. import __builtin__ as builtins # Untested, just to give you an idea of what I mean. try: _ = __import__(sys, fromlist=[version]) except TypeError: # Shadow the built-in with a patched version. def __import__(*args, **kwargs): if fromlist in kwargs: kwargs[fromlist] = [str(name) for name in kwargs[fromlist]] return builtins.__import__(*args, **kwargs) If you're really brave, you can even monkey-patch builtins with your own version. Obviously you still need to keep the old version somewhere. A closure would be perfect for that: # Again, untested. def patch_import(original_import=__import__): def __import__(*args, **kwargs): if fromlist in kwargs: kwargs[fromlist] = [str(name) for name in kwargs[fromlist]] return original_import(*args, **kwargs) builtins.__import__ = __import__ Monkey-patching builtins.__import__ is one of the few not-completely-frowned-upon uses of monkey-patching. Perhaps a better approach might be to eschew the use of __import__ and see whether the functions in imputil (deprecated) or importlib do what you need. https://docs.python.org/2/library/imputil.html https://docs.python.org/3/library/importlib.html Aside: __import__ is not recommended for user code. Direct use of __import__() is also discouraged in favor of importlib.import_module(). https://docs.python.org/3/library/functions.html#__import__ -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: __import__ demands different string types between 2 and 3
On 12/14/2014 11:29 PM, Ben Finney wrote:
This entails, I believe, the admonition to ensure text literals are
Unicode by default::
from __future__ import unicode_literals
and to specify bytes literals explicitly with ‘b'wibble'’ if needed.
For some scripts this works great (I have some), and for some it does not (I
have some of those, too).
The ‘__import__’ built-in function, though, is tripping up.
One work-around I have used is:
if isinstance(some_var, bytes):
some_var = some_var.decode('ascii')
# at this point some_var is unicode in both Pythons
--
~Ethan~
signature.asc
Description: OpenPGP digital signature
--
https://mail.python.org/mailman/listinfo/python-list
Portable code: ‘from __future__ import unicode_literals’ a good idea? (was: Portable code: __import__ demands different string types between 2 and 3)
Devin Jeanpierre jeanpierr...@gmail.com writes: On Sun, Dec 14, 2014 at 11:29 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: from __future__ import unicode_literals Ordinarily, for 2.x/3.3+ code I would suggest not doing this -- instead, b'...' for bytes, u'...' for unicode, and '...' for native string type (bytes on 2.x, unicode on 3.x). This is the main benefit of the u'...' syntax addition. That latter point would seemingly also apply to ‘from __future__ import unicode_literals’, so is moot in this context. As for the advice to avoid such a declaration, you're arguing against the official guide for porting Python 2 code to 2-and-3 compatible code: For text you should either use the from __future__ import unicode_literals statement or add a u prefix to the text literal. URL:https://docs.python.org/3.4/howto/pyporting.html#text-versus-binary-data So, the declarative import is specifically recommended. You'll need to present a case for why I shouldn't follow that recommendation. -- \ “I disapprove of what you say, but I will defend to the death | `\ your right to say it.” —Evelyn Beatrice Hall, _The Friends of | _o__) Voltaire_, 1906 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: ‘from __future__ import unicode_literals’ a good idea? (was: Portable code: __import__ demands different string types between 2 and 3)
On Mon, Dec 15, 2014 at 4:42 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: Devin Jeanpierre jeanpierr...@gmail.com writes: On Sun, Dec 14, 2014 at 11:29 PM, Ben Finney ben+pyt...@benfinney.id.au wrote: from __future__ import unicode_literals Ordinarily, for 2.x/3.3+ code I would suggest not doing this -- instead, b'...' for bytes, u'...' for unicode, and '...' for native string type (bytes on 2.x, unicode on 3.x). This is the main benefit of the u'...' syntax addition. That latter point would seemingly also apply to ‘from __future__ import unicode_literals’, so is moot in this context. As for the advice to avoid such a declaration, you're arguing against the official guide for porting Python 2 code to 2-and-3 compatible code: For text you should either use the from __future__ import unicode_literals statement or add a u prefix to the text literal. URL:https://docs.python.org/3.4/howto/pyporting.html#text-versus-binary-data So, the declarative import is specifically recommended. You'll need to present a case for why I shouldn't follow that recommendation. All the doc is saying there is to use unicode for text, in a way that works in both 2.x and 3.x. I am saying that the unicode_literals import is not necessarily the best way to do so. Python has three categories of strings: things that should be bytes in 2.x and 3.x, things that should be unicode in 2.x and 3.x, and things that should be bytes on 2.x and unicode on 3.x. The last category applies mostly to identifiers -- the strings you pass to functions like getattr or __import__, and the strings that are valid keys in **kwargs, and so on. You need a way to represent the last kind of string, whether it's unadorned ... literals, or function calls like identifier_string(...). If you can solve that, you are good. The official porting guide offers no advice. -- Devin -- https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: __import__ demands different string types between 2 and 3
Ethan Furman et...@stoneleaf.us writes:
On 12/14/2014 11:29 PM, Ben Finney wrote:
The ‘__import__’ built-in function, though, is tripping up.
One work-around I have used is:
if isinstance(some_var, bytes):
some_var = some_var.decode('ascii')
# at this point some_var is unicode in both Pythons
That's not the issue at all. I know how to declare a literal such that
it is Unicode in Python 2 and Python 3 (that's what the ‘from __future__
import unicode_literals’ does).
Rather, the problem is ‘__import__’ having incompatible expectations:
the ‘fromlist’ parameter's items must be bytes in Python 2, and must be
Unicode in Python 3.
The same parameter value can't satisfy both those requirements in a
single 2-and-3 compatible code base, without either using the
bytes-and-text ambiguity of ‘str’, or some kludge over-riding a simple
‘__import__’ call. Both of those are ugly and make for buggy code.
I'm increasingly of the opinion this is a subtle bug in ‘__import__’
that should be fixed instead of worked around. But it will likely be
rejected because the documentation advises against using ‘__import__’.
Bah.
--
\ “Try adding “as long as you don't breach the terms of service – |
`\ according to our sole judgement” to the end of any cloud |
_o__) computing pitch.” —Simon Phipps, 2010-12-11 |
Ben Finney
--
https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: __import__ demands different string types between 2 and 3
On 12/15/2014 05:36 PM, Ben Finney wrote: That's not the issue at all. I know how to declare a literal such that it is Unicode in Python 2 and Python 3 (that's what the ‘from __future__ import unicode_literals’ does). Rather, the problem is ‘__import__’ having incompatible expectations: the ‘fromlist’ parameter's items must be bytes in Python 2, and must be Unicode in Python 3. Ah. Well, then you do not want the `unicode_literals` import or it won't be bytes in Python 2 (as I'm sure you know). The same parameter value can't satisfy both those requirements in a single 2-and-3 compatible code base, without either using the bytes-and-text ambiguity of ‘str’, or some kludge over-riding a simple ‘__import__’ call. Both of those are ugly and make for buggy code. If this is for a large(ish) application, make one module with stuff that needs the ambiguity to work correctly. Comment the heck out of it. ;) I'm increasingly of the opinion this is a subtle bug in ‘__import__’ that should be fixed instead of worked around. Of course. But you'll still need to work around it for previous versions, unless you can say you only support 2.7.10+ (maybe 2.7.9+ if it gets fixed quick enough). But it will likely be rejected because the documentation advises against using ‘__import__’. Functions that should accept str but barf on unicode have a tendency to get fixed. -- ~Ethan~ signature.asc Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: __import__ demands different string types between 2 and 3
Ben Finney wrote: I'm increasingly of the opinion this is a subtle bug in ‘__import__’ that should be fixed instead of worked around. But it will likely be rejected because the documentation advises against using ‘__import__’. Bah. I think its worth raising an issue on the bug tracker and see what the core devs think. If you don't have and won't open an account on the tracker, if you can come up with a minimal example that demonstrates the issue, I'll open an issue for you. (But I'd rather you did it yourself :-) -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: __import__ demands different string types between 2 and 3
Ethan Furman et...@stoneleaf.us writes: On 12/15/2014 05:36 PM, Ben Finney wrote: I'm increasingly of the opinion this is a subtle bug in ‘__import__’ that should be fixed instead of worked around. And other people agree: URL:https://bugs.python.org/issue21720. Of course. But you'll still need to work around it for previous versions, unless you can say you only support 2.7.10+ (maybe 2.7.9+ if it gets fixed quick enough). If I can eventually drop the kludge by dropping support for Python 2.7 at some future point (instead of waiting until I can drop Python 3), that would still be an improvement. Functions that should accept str but barf on unicode have a tendency to get fixed. I hope you're right. The bug report currently focusses on improving the error message only. -- \ “They who can give up essential liberty to obtain a little | `\temporary safety, deserve neither liberty nor safety.” | _o__) —Benjamin Franklin, 1775-02-17 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: __import__ demands different string types between 2 and 3
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: If you don't have and won't open an account on the tracker, if you can come up with a minimal example that demonstrates the issue, I'll open an issue for you. (But I'd rather you did it yourself :-) Thank you for the offer. Fortunately, the Python bug tracker does not require me to maintain credentials for that site (I can use any OpenID), so I've got an account there. -- \ “Our task must be to free ourselves from our prison by widening | `\our circle of compassion to embrace all humanity and the whole | _o__) of nature in its beauty.” —Albert Einstein | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Portable code: __import__ demands different string types between 2 and 3
Howdy all, What should I do, in a world where all text literals are Unicode by default, to make ‘__import__’ work in both Python 2 and 3? I'm slowly learning about making Python code that will run under both Python 2 (version 2.6 or above) and Python 3 (version 3.2 or above). This entails, I believe, the admonition to ensure text literals are Unicode by default:: from __future__ import unicode_literals and to specify bytes literals explicitly with ‘b'wibble'’ if needed. The ‘__import__’ built-in function, though, is tripping up. A contrived, trivial project layout:: $ mkdir fooproject/ $ cd fooproject/ $ mkdir foo/ $ printf foo/__init__.py $ mkdir foo/bar/ $ printf foo/bar/__init__.py Here's a simple ‘fooproject/setup.py’:: from __future__ import unicode_literals main_module_name = 'foo' main_module = __import__(main_module_name, fromlist=['bar']) assert main_module.bar That fails under Python 2, but runs fine under Python 3:: $ python2 ./setup.py Traceback (most recent call last): File ./setup.py, line 4, in module main_module = __import__(main_module_name, fromlist=['bar']) TypeError: Item in ``from list'' not a string $ python3 ./setup.py We've deliberately made unadorned strings Unicode by default. By “not a string”, I presume Python 2 means “not a ‘bytes’ object”. Okay, so we'll explicitly set that to a ‘bytes’ literal:: from __future__ import unicode_literals main_module_name = 'foo' main_module = __import__(main_module_name, fromlist=[b'bar']) assert main_module.bar Now Python 2 is satisfied, but Python 3 complains:: $ python2 ./setup.py $ python3 ./setup.py Traceback (most recent call last): File ./setup.py, line 4, in module main_module = __import__(main_module_name, fromlist=[b'bar']) File frozen importlib._bootstrap, line 2281, in _handle_fromlist TypeError: hasattr(): attribute name must be string How can I get that ‘__import__’ call, complete with its ‘fromlist’ parameter, working correctly under both Python 2 and Python 3, keeping the ‘unicode_literals’ setting? If some kind of kludge is needed to make it work between versions, is this a bug which should be fixed so “use Unicode for text” remains applicable advice? -- \ “I do not believe in immortality of the individual, and I | `\consider ethics to be an exclusively human concern with no | _o__) superhuman authority behind it.” —Albert Einstein, letter, 1953 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Portable code: __import__ demands different string types between 2 and 3
On Mon, Dec 15, 2014 at 1:29 AM, Ben Finney ben+pyt...@benfinney.id.au wrote:
How can I get that ‘__import__’ call, complete with its ‘fromlist’
parameter, working correctly under both Python 2 and Python 3, keeping
the ‘unicode_literals’ setting?
How about str('bar')?
If some kind of kludge is needed to make it work between versions, is
this a bug which should be fixed so “use Unicode for text” remains
applicable advice?
Fixed in which version? The problem is that Python 2 mixes the
notions of strings and bytestrings, and the solution is Python 3.
I believe six also has a give me a native string function to help with this.
--
Zach
--
https://mail.python.org/mailman/listinfo/python-list
