[issue33327] Add a method to move messages to IMAPlib
Hans-Peter Jansen added the comment:
If I'm not mistaken, this is applied to the openSUSE TW version of Python.
For some reason, this seems to not play well with .uid('move',...)
on a cyrus imap server (v2.4.19). Is that to be expected?
```
2020-07-03 18:04:05 INFO: [imap_reorg] move b'10399' from 2017-01-01
06:30:35+02:00 to INBOX.2017
Traceback (most recent call last):
File "./imap_reorg.py", line 431, in
sys.exit(main())
File "./imap_reorg.py", line 425, in main
return process()
File "./imap_reorg.py", line 133, in trace_and_call
result = func(*args, **kwargs)
File "./imap_reorg.py", line 358, in process
ret |= reorg.run_expr(expr)
File "./imap_reorg.py", line 345, in run_expr
return method(*args)
File "./imap_reorg.py", line 328, in yearly
ret = self.imap.uid('move', uid, dest)
File "/usr/lib64/python3.8/imaplib.py", line 881, in uid
typ, dat = self._simple_command(name, command, *args)
File "/usr/lib64/python3.8/imaplib.py", line 1205, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/usr/lib64/python3.8/imaplib.py", line 1030, in _command_complete
raise self.error('%s command error: %s %s' % (name, typ, data))
imaplib.error: UID command error: BAD [b'Unrecognized UID subcommand']
```
--
nosy: +frispete
___
Python tracker
<https://bugs.python.org/issue33327>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Re: PyCA cryptography 2.3 released
Hi Paul, you have a version mismatch in subject and text. Cheers, Pete On Mittwoch, 18. Juli 2018 05:19:27 Paul Kehrer wrote: > PyCA cryptography 2.2.2 has been released to PyPI. cryptography includes > both high level recipes and low level interfaces to common cryptographic > algorithms such as symmetric ciphers, message digests, and key derivation > functions. We support Python 2.7, Python 3.4+, and PyPy. > > Changelog (https://cryptography.io/en/latest/changelog/#v2-3): > > * SECURITY ISSUE: finalize_with_tag() allowed tag truncation by default > which can allow tag forgery in some cases. The method now enforces the > min_tag_length provided to the GCM constructor. > * Added support for Python 3.7. > * Added extract_timestamp() to get the authenticated timestamp of a Fernet > token. > * Support for Python 2.7.x without hmac.compare_digest has been deprecated. > We will require Python 2.7.7 or higher (or 2.7.6 on Ubuntu) in the next > cryptography release. > * Fixed multiple issues preventing cryptography from compiling against > LibreSSL 2.7.x. > * Added get_revoked_certificate_by_serial_number for quick serial number > searches in CRLs. > * The RelativeDistinguishedName class now preserves the order of > attributes. Duplicate attributes now raise an error instead of silently > discarding duplicates. > * aes_key_unwrap() and aes_key_unwrap_with_padding() now raise > InvalidUnwrap if the wrapped key is an invalid length, instead of > ValueError. > > -Paul Kehrer (reaperhulk) -- https://mail.python.org/mailman/listinfo/python-list
Re: keyrings.cryptfile released on github
On Dienstag, 14. März 2017 00:33:34 Hans-Peter Jansen wrote: > > I plan to add authenticated service and username support via associated data > as well (that protects against tampering with these values). Done. > Cheers, > Pete -- https://mail.python.org/mailman/listinfo/python-list
Re: keyrings.cryptfile released on github
On Freitag, 10. März 2017 13:31:41 Paul Rubin wrote: > Hans-Peter Jansen <h...@urpla.net> writes: > > [1] http://web.cs.ucdavis.edu/~rogaway/ocb/license.htm > > Oh that's interesting, he's expanded the free licenses. Still though, > while OCB is very clever and it was important as the first satisfactory > AEAD mode, I don't think it's that important these days. GCM is > standardized, does similar things, and while it's potentially slower, > some CPUs even have hardware support for it now. If you library doesn't > support GCM (I haven't checked yet) then it probably should. It does support GCM now, it's even the new default(!). I've added support for all AEAD modes, that PyCryptodome supports, and supplied a small encryption mode conversion tool as well. I plan to add authenticated service and username support via associated data as well (that protects against tampering with these values). It might be a good idea to record and secure the number of entries as well. Cheers, Pete -- https://mail.python.org/mailman/listinfo/python-list
Re: keyrings.cryptfile released on github
On Donnerstag, 9. März 2017 23:09:09 ng0 wrote: > Hans-Peter Jansen transcribed 3.8K bytes: > > Hi, > > > > since the PyCrypto ML is dead, I'm looking for advise/feedback from some > > cryptography aware people. > > > > I've released a keyring companion package today: > > https://github.com/frispete/keyrings.cryptfile > > > > Its primary purpose is a decent encrypted file backend for python > > keyrings. > > As such, it uses manually parameterized argon2 hashes as KDF, and AES in > > OCB mode as stream cipher (well, it just encrypts the password for a > > given service/user name). Granted, the advantages of OCB are not /that/ > > crucial here :wink:, but apart from technical factors, the exclusion of > > military uses > I was looking for some proprietary EULA or something else locked down > license and instead just saw the Expat license. Don't get you here. Yes, the package contains just a MIT license file for now. > I assume that you are > aware that no anti-mil clause exists in co-existence with free software > licenses > and your sentence was just written in an > odd way? First, bear with me, as I'm not a native speaker. Sometimes, my expressions tend to confuse people. I'm sorry about that. Second, I'm not a lawyer. As long as I use OCB in OSS and does not sue somebody else about it, License 1 is granted, if I read the OCB license options correctly [1]. Anyway, I will add the OCB license to the package and add a note in the initial password retrieval dialog about it.. In this projects state, I would like to discuss the technical issues first. As said, I sympathize with the idea of the license, since I personally believe, that military conflicts doesn't solve any problems, but many todays problems originate from them. [1] http://web.cs.ucdavis.edu/~rogaway/ocb/license.htm > > by its license is rather *attractive* from my POV(!). But I'm open for > > discussions of course. Cheers, Pete -- https://mail.python.org/mailman/listinfo/python-list
keyrings.cryptfile released on github
Hi,
since the PyCrypto ML is dead, I'm looking for advise/feedback from some
cryptography aware people.
I've released a keyring companion package today:
https://github.com/frispete/keyrings.cryptfile
Its primary purpose is a decent encrypted file backend for python keyrings.
As such, it uses manually parameterized argon2 hashes as KDF, and AES in OCB
mode as stream cipher (well, it just encrypts the password for a given
service/user name). Granted, the advantages of OCB are not /that/ crucial
here :wink:, but apart from technical factors, the exclusion of military uses
by its license is rather *attractive* from my POV(!). But I'm open for
discussions of course.
Still interested? Here we go:
To get you started, I expect you to have a python3 environment and git
available. You might want to provide the packages argon2-cffi, keyring,
pycryptodome and their dependencies (most notably SecretStorage and
cryptography, or use a local venv, but that will depend on a compiler and some
development packages.
Example session, create an encrypted keyring:
$ git clone https://github.com/frispete/keyrings.cryptfile
$ cd keyrings.cryptfile
$ pyvenv env
$ . env/bin/activate
(env) $ pip install -e .
[...] # should succeed, some development packages might be missing otherwise
(env) $ python3
Python 3.4.5 (default, Jul 03 2016, 12:57:15) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from keyrings.cryptfile.cryptfile import CryptFileKeyring
>>> kr = CryptFileKeyring()
>>> kr.set_password("service", "user", "secret")
Please set a password for your new keyring:
Please confirm the password:
>>> ^d
Second session, retrieve the stored secret from the keyring:
(env) $ python3
Python 3.4.5 (default, Jul 03 2016, 12:57:15) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from keyrings.cryptfile.cryptfile import CryptFileKeyring
>>> kr = CryptFileKeyring()
>>> kr.get_password("service", "user")
Please enter password for encrypted keyring:
'secret'
>>> ^d
Note, that the KDF might delay the {set,get}_password() operations for a
few seconds (~1 sec. on a capable system).
The resulting file is located here (by default) and might look similar to:
(env) $ cat ~/.local/share/python_keyring/cryptfile_pass.cfg
[keyring_2Dsetting]
password_20reference =
eyJub25jZSI6ICJQdVdWVUIwUHNYbEFqYUUxZ2l2RlxuIiwgIm1hYyI6ICIvVTFIVDBWTnRheTFl
TjA5TVlHb0dRPT1cbiIsICJzYWx0IjogIklMdDNBU1hMUENrbWZ2NzFudmtBSUE9PVxuIiwgImRh
dGEiOiAidW1EQkNvQ2dRUTk5WEVaNkZ4NWt3NXRkSUZDOHFIUE5ZOHhWXG4ifQ==
scheme = PyCryptodome [Argon2] AES OCB
version = 1.0
[service]
user =
eyJub25jZSI6ICI5SUU3UGp2eDU2SXNQdHlLUGRtaFxuIiwgIm1hYyI6ICJKcFR1NXMxaDd0UGlW
OW9XL3d5cFdBPT1cbiIsICJzYWx0IjogIlpBeEhJdXlqYnRuTkgzb3BMNTFvdkE9PVxuIiwgImRh
dGEiOiAiT2I3Z1JJbXR5aVJLXG4ifQ==
The values can be decoded like this:
(env) $ python3
>>> import base64
>>> base64.decodebytes(b"""
... eyJub25jZSI6ICI5SUU3UGp2eDU2SXNQdHlLUGRtaFxuIiwgIm1hYyI6ICJKcFR1NXMxaDd0UGlW
... OW9XL3d5cFdBPT1cbiIsICJzYWx0IjogIlpBeEhJdXlqYnRuTkgzb3BMNTFvdkE9PVxuIiwgImRh
... dGEiOiAiT2I3Z1JJbXR5aVJLXG4ifQ==""")
b'{"nonce": "9IE7Pjvx56IsPtyKPdmh\\n", "mac": "JpTu5s1h7tPiV9oW/wypWA==\\n",
"salt": "ZAxHIuyjbtnNH3opL51ovA==\\n", "data": "Ob7gRImtyiRK\\n"}'
The items should be self explanatory. In theory, it should be considerable
hard to get back to the plain values of data without knowing the password.
Any cryptography experts attending?
What do you think? The class hierarchy is inherited from keyrings.alt, and
not exactly easy to follow, but the interesting parts are all in cryptfile,
which is quite brief.
I would be glad to hear something from you about my handling of cryptography.
Is it ready for the public in that form or should I better locked away? :wink:
TIA,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: ANN: psutil 5.1.0 with hardware sensors released
On Mittwoch, 1. Februar 2017 21:54:06 Giampaolo Rodola' wrote: > Hello all, > I'm glad to announce the release of psutil 5.1.1: ^ Guess, you meant to say 5.1.0 here, or probably your time machine broke ;) Cheers, Pete -- https://mail.python.org/mailman/listinfo/python-list
Re: Referencing section name by interpolation in ConfigParser
On Mittwoch, 25. Januar 2017 10:01:56 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > I would like to use a interpolated section name, e.g.:
> >
> > [Section]
> > secref: %{section}s/whatever
> >
> > should result in:
> >>>> config['Section']['secref']
> >
> > 'Section/whatever'
> >
> > Any idea anybody, how to archive this with minimum fuzz?
>
> If you can live with the existing "basic interpolation", i. e. %(...)s, not
> %{...}s:
Yes, of course.. Sorry for the typo.
> $ cat config.ini
> [Foo]
> secref: %(section)s/whatever
> [Bar]
> secref: %(section)s/whatever
> $ cat demo.py
> import configparser
>
>
> class Interpolation(configparser.BasicInterpolation):
> def before_get(self, parser, section, option, value, defaults):
> defaults = defaults.copy()
> defaults["section"] = section
> return super().before_get(parser, section, option, value, defaults)
>
>
> p = configparser.ConfigParser(interpolation=Interpolation())
> p.read("config.ini")
> for section in "Foo", "Bar":
> print(section, "-->", p[section]["secref"])
> $ python3 demo.py
> Foo --> Foo/whatever
> Bar --> Bar/whatever
> $
Brilliant as usual, thank you, Peter.
This is exactly, what I was after.
Have a nice weekend,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Referencing section name by interpolation in ConfigParser
Hi,
I would like to use a interpolated section name, e.g.:
[Section]
secref: %{section}s/whatever
should result in:
>>> config['Section']['secref']
'Section/whatever'
Any idea anybody, how to archive this with minimum fuzz?
Thanks,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: Using sudo with pip3?
On Samstag, 7. Januar 2017 19:07:55 Clint Moyer wrote: > I would lightly advise against, assuming both Pip and your package > manager are trying to accomplish nearly the same thing. Stick with > updating through the repo. > > If you find that the version your OS provides is out-of-date compared > to what's on PyPi or Github, then you might want to remove from your > OS and re-install through Pip, for those discrete cases. That's the > platform agnostic route. Or take the ninja way, I do: Build all packages, you want (updated) for your distribution: https://build.opensuse.org/project/monitor/home:frispete:python https://build.opensuse.org/project/monitor/home:frispete:python3 The red labels are reminders... SCR, Pete -- https://mail.python.org/mailman/listinfo/python-list
Re: Choosing a Python IDE. what is your Pythonish recommendation? I
On Montag, 2. Januar 2017 03:38:53 Antonio Caminero Garcia wrote: > Hello, I am having a hard time deciding what IDE or IDE-like code editor > should I use. This can be overwhelming. > > So far, I have used Vim, Sublime, Atom, Eclipse with PyDev, Pycharm, > IntelliJ with Python plugin. Well, since nobody mentioned it, yet: eric is doing quite nice here. With on the fly error checking, jedi and qscintilla calltips and autocompletion, git integration (using a plugin), graphical debugger, it's grown to a capable IDE over the years. Given, it's a fully open source, PyQt based project, it also shows the powers of Python3 and PyQt. Pete -- https://mail.python.org/mailman/listinfo/python-list
Re: Choosing a Python IDE. what is your Pythonish recommendation? I do not know what to choose.
On Montag, 2. Januar 2017 03:38:53 Antonio Caminero Garcia wrote: > Hello, I am having a hard time deciding what IDE or IDE-like code editor > should I use. This can be overwhelming. > > So far, I have used Vim, Sublime, Atom, Eclipse with PyDev, Pycharm, > IntelliJ with Python plugin. Well, since nobody mentioned it, yet: eric is doing quite nice here. With on the fly error checking, jedi and qscintilla calltips and autocompletion, git integration (using a plugin), graphical debugger, it's grown to a capable IDE over the years. Given, it's a fully open source, PyQt based project, it also shows the powers of Python3 and PyQt. Pete -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
Dear Eryk,
thanks for chiming in.
On Donnerstag, 29. Dezember 2016 21:27:56 eryk sun wrote:
> On Thu, Dec 29, 2016 at 12:18 PM, Hans-Peter Jansen <h...@urpla.net> wrote:
> >> >>> import weakref, ctypes
> >> >>> T = ctypes.c_ubyte * 3
> >> >>> t = T()
> >> >>> bytes(t) == b"\0" * 3
> >>
> >> True
> >>
> >> >>> bytes(weakref.proxy(t)) == b"\0" * 3
> >>
> >> Traceback (most recent call last):
> >> File "", line 1, in
> >>
> >> AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__'
> >>
> >> That looks like a leaky abstraction. While I found a workaround
> >>
> >> >>> bytes(weakref.proxy(t)[:]) == b"\0" * 3
> >>
> >> True
> >
> > I found a couple of other rough corners already, when working with the
> > ctypes module. Obviously, this module is lacking some love.
>
> That's not the fault of ctypes. There's no requirement for objects
> that implement the buffer protocol to also implement __bytes__. You'd
> have the same problem if you tried to proxy a memoryview.
>
> However, using a proxy seems particularly wHaorthless for ctypes. Type
> checking is integral to the design of ctypes, and a weakproxy won't
Did you follow the discussion?
I'm trying to make context manager work with ctypes.from_buffer on mmapped
files:
import ctypes
import mmap
import weakref
NOPROB=False
#NOPROB=True
from contextlib import contextmanager
class T(ctypes.Structure):
_fields_ = [("foo", ctypes.c_uint32)]
@contextmanager
def map_struct(m, n, struct, offset = 0):
m.resize(n * mmap.PAGESIZE)
inst = struct.from_buffer(m, offset)
yield inst
SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
with map_struct(m, 1, T) as a:
a.foo = 1
if NOPROB:
del a
with map_struct(m, 2, T) as b:
b.foo = 2
if NOPROB:
del b
offset = ctypes.sizeof(T)
rest = m.size() - offset
overhang = ctypes.c_ubyte * rest
with map_struct(m, 2, overhang, offset) as c:
assert(bytes(c) == bytes(rest))
if NOPROB:
del c
Without these dreaded del statements, this code doesn't work:
$ python3 mmap_test2.py
Traceback (most recent call last):
File "mmap_test2.py", line 30, in
with map_struct(m, 2, T) as b:
File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
return next(self.gen)
File "mmap_test2.py", line 16, in map_struct
m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.
It will work, if you define NOPROB=True.
The weakref approach was an attempt to make this work.
Do you have an idea how to create the context manager in a way, that obsoletes
these ugly dels? Something like ctypes.from_buffer_release() that is able to
actively release the mapping is needed here, AFAICS.
This code works with Python2 due to the mmap module not checking for any
existing mappings which may lead to segfaults, if the mmap is resized.
Thanks,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Donnerstag, 29. Dezember 2016 09:33:59 Peter Otten wrote: > Hans-Peter Jansen wrote: > > On Mittwoch, 28. Dezember 2016 16:53:53 Hans-Peter Jansen wrote: > > The minimal example is > > >>> import weakref, ctypes > >>> T = ctypes.c_ubyte * 3 > >>> t = T() > >>> bytes(t) == b"\0" * 3 > > True > > >>> bytes(weakref.proxy(t)) == b"\0" * 3 > > Traceback (most recent call last): > File "", line 1, in > AttributeError: 'c_ubyte_Array_3' object has no attribute '__bytes__' > > That looks like a leaky abstraction. While I found a workaround > > >>> bytes(weakref.proxy(t)[:]) == b"\0" * 3 > > True I found a couple of other rough corners already, when working with the ctypes module. Obviously, this module is lacking some love. > to me your whole approach is beginning to look very questionable. You know, > > "If the implementation is hard to explain, it's a bad idea." > > What do you gain from using the mmap/ctypes combo instead of regular file > operations and the struct module? Your sample code seems to touch every > single byte of the file once so that there are little to no gains from > caching. And then your offset is basically a file position managed manually > instead of implicitly with read, write, and seek. Of course, the real code is a bit more complex... The code presented here is for demonstration purposes only. I'm not allowed to reveal the projects' code, but I can state, that using this combination allows for crawling through huge files (5-25GB) in unbelievable performance (without any further optimization), and updating parts of it. By delegating the whole I/O management to the kernel, one can observe, that python runs at full speed managing the data just by reference and assignment operations, all (mostly) in place. The resource usage is impressively low at the same time. Since the code is meant to be executed with many instances in parallel on a single machine, this is an important design criteria. While I would love to get rid of these dreaded and unpythonic del statements, I can accept them for now, until a better approach is found. Will dig through the ctypes module again, when I find time. Thanks again for taking your valuable time, Peter. Much appreciated. I wish you a Happy New Year! Cheers, Pete -- https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Mittwoch, 28. Dezember 2016 16:53:53 Hans-Peter Jansen wrote:
> On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> > > Hans-Peter Jansen wrote:
> > > > Dear Peter,
> > > >
> > > > thanks for taking valuable time to look into my issue.
> > >
> > > You're welcome!
> > >
> > > @contextmanager
> > > def map_struct(m, n):
> > > m.resize(n * mmap.PAGESIZE)
> > > keep_me = T.from_buffer(m)
> > > yield weakref.proxy(keep_me)
> >
> > Hooray, that did the trick. Great solution, thank you very much!
>
> Sorry for bothering you again, Peter, but after applying it to the real
> project, that fails consistently similar to:
$ python3 mmap_test_weakref.py
Traceback (most recent call last):
File "mmap_test_weakref.py", line 32, in
assert(bytes(c) == bytes(rest))
AttributeError: 'c_ubyte_Array_8188' object has no attribute '__bytes__'
$ cat mmap_test_weakref.py
import ctypes
import mmap
import weakref
from contextlib import contextmanager
class T(ctypes.Structure):
_fields_ = [("foo", ctypes.c_uint32)]
@contextmanager
def map_struct(m, n, struct, offset = 0):
m.resize(n * mmap.PAGESIZE)
inst = struct.from_buffer(m, offset)
yield weakref.proxy(inst)
SIZE = mmap.PAGESIZE
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
with map_struct(m, 1, T) as a:
a.foo = 1
with map_struct(m, 2, T) as b:
b.foo = 2
offset = ctypes.sizeof(T)
rest = m.size() - offset
overhang = ctypes.c_ubyte * rest
with map_struct(m, 2, overhang, offset) as c:
assert(bytes(c) == bytes(rest))
With weakref and mmap.resize() disabled, this acts as expected.
BTW: mmapped files need the first page initialized, the rest is done in the
kernel (filled with zeros on resize).
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Mittwoch, 28. Dezember 2016 21:58:38 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> >> Hans-Peter Jansen wrote:
> > It leaves the question on why is Python2 acting as one would expect
> > related to context managers, and Python3 needs this weakref juggling.
> > Maybe something, that's worth to be placed in python-dev. What do you
> > think?
>
> Well, given
>
> $ cat mmap_resize.py
> import ctypes
> import mmap
>
> class T(ctypes.Structure):
> _fields_ = [("foo", ctypes.c_uint32)]
>
> SIZE = 2 * mmap.PAGESIZE
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
>
> m = mmap.mmap(f.fileno(), SIZE)
>
> a = T.from_buffer(m, mmap.PAGESIZE)
> m.resize(mmap.PAGESIZE)
> a.foo = 42
> $ python3 mmap_resize.py
> Traceback (most recent call last):
> File "mmap_resize.py", line 15, in
> m.resize(mmap.PAGESIZE)
> BufferError: mmap can't resize with extant buffers exported.
> $ python2.7 mmap_resize.py
> Segmentation fault
>
> do you really prefer the Python 2 behaviour?
Hmm, so Python2 behavior was "working by chance", or better "working without
any safety net" which isn't the Real McCoy either in that area.
Now, if only the weakref version would behave with the ctypes objects...
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Mittwoch, 28. Dezember 2016 15:17:22 Hans-Peter Jansen wrote:
> On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> > Hans-Peter Jansen wrote:
> > > Dear Peter,
> > >
> > > thanks for taking valuable time to look into my issue.
> >
> > You're welcome!
> >
> > > It might be related to my distinct silliness, but the problem persists
> > > with your code as well.
> >
> > Unfortunately I posted the broken toy example rather than the fixed one.
> > Here's the latter. Basically you have to keep a reference in the context
> > manager (whether you implement it as a class or a generator doesn't
> > matter)
> > without giving another reference away to client code:
> >
> > $ cat mmap_after.py
> > import ctypes
> > import mmap
> > import weakref
> >
> > from contextlib import contextmanager
> >
> > class T(ctypes.Structure):
> > _fields = [("foo", ctypes.c_uint32)]
> >
> > @contextmanager
> >
> > def map_struct(m, n):
> > m.resize(n * mmap.PAGESIZE)
> > keep_me = T.from_buffer(m)
> > yield weakref.proxy(keep_me)
>
> Hooray, that did the trick. Great solution, thank you very much!
Sorry for bothering you again, Peter, but after applying it to the real
project, that fails consistently similar to:
# first run, check explicitly disabled
$> python3 ctypes_mmap_ctx.py
DEBUG: starting offset: 0x10
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: add_data: header
DEBUG: add_data: 5168
DEBUG: final offset: 0xa1d0
$> python3 ctypes_mmap_ctx.py
Traceback (most recent call last):
File "ctypes_mmap_ctx.py", line 163, in
mf = MapFile(mapfile)
File "ctypes_mmap_ctx.py", line 109, in __init__
if bytes(blk) != bytes(rest):
AttributeError: 'c_ubyte_Array_3632' object has no attribute '__bytes__'
The new issue appears in some consistency checking code. The mapfile is
checked for zeroed out overhang on that line. The weakref proxy isn't behaving
well at that point. Any idea, what could be going wrong with it?
Updated the gist to demonstrate the issue after switching to weakref.proxy().
(has grown even more code in that process, sorry).
It looks like a minor issue, but the code quality is really degraded from
these ugly del statements.
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
On Mittwoch, 28. Dezember 2016 13:48:48 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> > Dear Peter,
> >
> > thanks for taking valuable time to look into my issue.
>
> You're welcome!
>
> > It might be related to my distinct silliness, but the problem persists
> > with your code as well.
>
> Unfortunately I posted the broken toy example rather than the fixed one.
> Here's the latter. Basically you have to keep a reference in the context
> manager (whether you implement it as a class or a generator doesn't matter)
> without giving another reference away to client code:
>
> $ cat mmap_after.py
> import ctypes
> import mmap
> import weakref
>
> from contextlib import contextmanager
>
> class T(ctypes.Structure):
> _fields = [("foo", ctypes.c_uint32)]
>
>
> @contextmanager
> def map_struct(m, n):
> m.resize(n * mmap.PAGESIZE)
> keep_me = T.from_buffer(m)
> yield weakref.proxy(keep_me)
Hooray, that did the trick. Great solution, thank you very much!
If you don't mind, I will mention you and your solution at the various places,
I placed this issue over the last weeks.
You made my day, Peter!
It leaves the question on why is Python2 acting as one would expect related to
context managers, and Python3 needs this weakref juggling. Maybe something,
that's worth to be placed in python-dev. What do you think?
Cheers,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: ctypes, memory mapped files and context manager
Dear Peter,
thanks for taking valuable time to look into my issue.
It might be related to my distinct silliness, but the problem persists with
your code as well. Further comments inlined.
On Dienstag, 27. Dezember 2016 21:39:51 Peter Otten wrote:
> Hans-Peter Jansen wrote:
> >
> > def __enter__(self):
> > # resize the mmap (and backing file), if structure exceeds mmap
> > # size mmap size must be aligned to mmap.PAGESIZE
> > cssize = ctypes.sizeof(self._cstruct)
> >
> > if self._offset + cssize > self._mm.size():
> > newsize = align(self._offset + cssize, mmap.PAGESIZE)
> > self._mm.resize(newsize)
> >
> > self._csinst = self._cstruct.from_buffer(self._mm, self._offset)
> > return self._csinst
>
> Here you give away a reference to the ctypes.BigEndianStructure. That means
> you no longer control the lifetime of self._csinst which in turn holds a
> reference to the underlying mmap or whatever it's called.
Here's the code, based on contextmanager:
@contextmanager
def cstructmap(cstruct, mm, offset = 0):
# resize the mmap (and backing file), if structure exceeds mmap size
# mmap size must be aligned to mmap.PAGESIZE
cssize = ctypes.sizeof(cstruct)
if offset + cssize > mm.size():
newsize = align(offset + cssize, mmap.PAGESIZE)
mm.resize(newsize)
yield cstruct.from_buffer(mm, offset)
While much more concise, I don't understand, how it should make a difference
relative to the "with" variable lifetime, when used.
> There might be a way to release the mmap reference while the wrapper
> structure is still alive, but the cleaner way is probably to not give it
> away in the first place, and create a proxy instead with
>
> return weakref.proxy(self._csinst)
This fails, as it doesn't keep the reference long enough.
> > def __exit__(self, exc_type, exc_value, exc_traceback):
> > # free all references into mmap
> > del self._csinst
>
> The line above is redundant. It removes the attribute from the instance
> __dict__ and implicitly decreases its refcount. It does not actually
> physically delete the referenced object. If you remove the del statement the
> line below will still decrease the refcount.
>
> Make sure you understand this to avoid littering your code with cargo cult
> del-s ;)
Yes, I was aware of this. It was a testing relic, that survived somehow.
Sorry. Yes, I usually try to avoid cargo cultry in my code. ;)
> > The issue: when creating a mapping via context manager, we assign a local
> > variable (with ..), that keep existing in the local context, even when the
> > manager context was left. This keeps a reference on the ctypes mapped area
> > alive, even if we try everything to destroy it in __exit__. We have to del
> > the with var manually.
> >
> > Now, I want to get rid of the ugly any error prone del statements.
> >
> > What is needed, is a ctypes operation, that removes the mapping actively,
> > and that could be added to the __exit__ part of the context manager.
Revised code (including your test code):
https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239
> > The script creates a memory mapped file in the current directory named
> > "mapfile". When started without arguments, it copies itself into this
> > file, until 10 * mmap.PAGESIZE growth is reached (or it errored out
> > before..).
> >
> > IF you change NOPROB to True, it will actively destruct the context
> > manager vars, and should work as advertized.
> >
> > Any ideas are much appreciated.
>
> You might put some more effort into composing example scripts. Something
> like the script below would have saved me some time...
I'm very sorry about this.
> import ctypes
> import mmap
>
> from contextlib import contextmanager
>
> class T(ctypes.Structure):
> _fields = [("foo", ctypes.c_uint32)]
>
>
> @contextmanager
> def map_struct(m, n):
> m.resize(n * mmap.PAGESIZE)
> yield T.from_buffer(m)
>
> SIZE = mmap.PAGESIZE * 2
> f = open("tmp.dat", "w+b")
> f.write(b"\0" * SIZE)
> f.seek(0)
> m = mmap.mmap(f.fileno(), mmap.PAGESIZE)
>
> with map_struct(m, 1) as a:
> a.foo = 1
> with map_struct(m, 2) as b:
> b.foo = 2
Unfortunately, your code behaves exactly like mine:
$> python3 mmap_test.py
Traceback (most recent call last):
File "mmap_test.py", line 23, in
with map_struct(m, 2) as b:
File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
return next(self.gen)
File "mmap_test.py", line 12, in map_struct
m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.
BTW, Python2 makes a difference in this respect, but my project is Python3
based. Have you tested this with Python3? It would be interesting to explore
the reasons of this difference, which is, ähem, pretty surprising.
Thanks,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
ctypes, memory mapped files and context manager
Hi, I'm using $subjects combination successfully in a project for creating/iterating over huge binary files (> 5GB) with impressive performance, while resource usage keeps pretty low, all with plain Python3 code. Nice! Environment: (Python 3.4.5, Linux 4.8.14, openSUSE/x86_64, NFS4 and XFS filesystems) The idea is: map a ctypes structure onto the file at a certain offset, act on the structure, and release the mapping. The latter is necessary for keeping the mmap file properly resizable and closable (due to the nature of mmaps and Python's posix implementation thereof). Hence, a context manager serves us well (in theory). Here's some code excerpt: class cstructmap: def __init__(self, cstruct, mm, offset = 0): self._cstruct = cstruct self._mm = mm self._offset = offset self._csinst = None def __enter__(self): # resize the mmap (and backing file), if structure exceeds mmap size # mmap size must be aligned to mmap.PAGESIZE cssize = ctypes.sizeof(self._cstruct) if self._offset + cssize > self._mm.size(): newsize = align(self._offset + cssize, mmap.PAGESIZE) self._mm.resize(newsize) self._csinst = self._cstruct.from_buffer(self._mm, self._offset) return self._csinst def __exit__(self, exc_type, exc_value, exc_traceback): # free all references into mmap del self._csinst self._csinst = None def work(): with cstructmap(ItemHeader, self._mm, self._offset) as ih: ih.identifier = ItemHeader.Identifier ih.length = ItemHeaderSize + datasize blktype = ctypes.c_char * datasize with cstructmap(blktype, self._mm, self._offset) as blk: blk.raw = data In practice, this results in: Traceback (most recent call last): File "ctypes_mmap_ctx.py", line 146, in mf.add_data(data) File "ctypes_mmap_ctx.py", line 113, in add_data with cstructmap(blktype, self._mm, self._offset) as blk: File "ctypes_mmap_ctx.py", line 42, in __enter__ self._mm.resize(newsize) BufferError: mmap can't resize with extant buffers exported. The issue: when creating a mapping via context manager, we assign a local variable (with ..), that keep existing in the local context, even when the manager context was left. This keeps a reference on the ctypes mapped area alive, even if we try everything to destroy it in __exit__. We have to del the with var manually. Now, I want to get rid of the ugly any error prone del statements. What is needed, is a ctypes operation, that removes the mapping actively, and that could be added to the __exit__ part of the context manager. Full working code example: https://gist.github.com/frispete/97c27e24a0aae1bcaf1375e2e463d239 The script creates a memory mapped file in the current directory named "mapfile". When started without arguments, it copies itself into this file, until 10 * mmap.PAGESIZE growth is reached (or it errored out before..). IF you change NOPROB to True, it will actively destruct the context manager vars, and should work as advertized. Any ideas are much appreciated. Thanks in advance, Pete -- https://mail.python.org/mailman/listinfo/python-list
[issue9253] argparse: optional subparsers
Changes by Hans-Peter Jansen <h...@urpla.net>: -- nosy: +frispete ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue9253> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28970] ctypes.from_buffer counterpart to actively remove the mapping
New submission from Hans-Peter Jansen: In an attempt of using ctypes.from_buffer() to map a structure to a memory mapped file, it is important to destroy the mapping after use, because the mmap won't be resizable or freeable correctly until then. The best approach, I was able to came up with is using a context manager, but calling the dtor of the mapping in __exit__ is not enough, which results to code like this: with StructMap(ctypes_struct, mm, offest) as smap: smap.xxx = 'blabla' del smap # necessary, but ugly Without the del, the "with" variable still exist in the local context, hence the mapping still exist, until it is explicitly destroyed. I hope, that ctypes is able to (or can be made to) actively remove the mapping in the __exit__ part of the context manager. I've put some code on stackoverflow to play with this: http://stackoverflow.com/questions/41077696/python-ctypes-from-buffer-mapping-with-context-manager-into-memory-mapped-file The problem seems to exist with the linux mmap implementation only. -- components: ctypes messages: 283188 nosy: frispete priority: normal severity: normal status: open title: ctypes.from_buffer counterpart to actively remove the mapping type: behavior versions: Python 3.7 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28970> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
distutils_ui 0.1.1 released
For those of you, who like PyQt{4,5} as much as I do, as well as for those who
don't like it that much, because of the poor integration with setuptools
et.al., here's another piece of software to bridge the gap:
A distutils build extension for PyQt{4,5} applications
that makes handling the PyQt tool chain easier than ever:
https://pypi.python.org/pypi/distutils_ui
Ahem, well, it wasn't that easy before. Most of us were using dreaded
Makefiles or other such crutches to generate translation files, .py modules of
forms, and resource modules. Scratch the crutches, here's what you're looking
for.
Feedback welcome.
Enjoy,
Pete
--
https://mail.python.org/mailman/listinfo/python-announce-list
Support the Python Software Foundation:
http://www.python.org/psf/donations/
distutils_ui 0.1.1 released
For those of you, who like PyQt{4,5} as much as I do, as well as for those who
don't like it that much, because of the poor integration with setuptools
et.al., here's another piece of software to bridge the gap:
A distutils build extension for PyQt{4,5} applications
that makes handling the PyQt tool chain easier than ever:
https://pypi.python.org/pypi/distutils_ui
Ahem, well, it wasn't that easy before. Most of us were using dreaded
Makefiles or other such crutches to generate translation files, .py modules of
forms, and resource modules. Scratch the crutches, here's what you're looking
for.
Feedback welcome.
Enjoy,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
[issue27568] "HTTPoxy", use of HTTP_PROXY flag supplied by attacker in CGI scripts
Hans-Peter Jansen added the comment: > (In msg271688, I pondered if I need to backport a behavior change from > issue26804 which will allow lower cased proxies, but then, I decided against > it as it will introduce unnecessary changes to this security fix releases). Hmm, Senthil, while I understand, that you want to avoid unnecessary changes, doesn't this result in non deterministic behaviour of proxy handling without my patch? + header. If you need to use an HTTP proxy in a CGI environment, either use + ``ProxyHandler`` explicitly, or make sure the variable name is in + lowercase (or at least the ``_proxy`` suffix). Without 26804, this fix works by chance only for 3.3 and 3.4, since it depends on os.environ dictionary order, which is non deterministic by definition. 26804 resolves this by making sure, a lower case _proxy var has a higher priority over the other variants. -- nosy: +frispete ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27568> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27513] email.utils.getaddresses raises exception from erroneous message get_all input
Hans-Peter Jansen added the comment: message.get cannot decode the header correctly, and returns a Header instance instead, which makes email.utils.getaddresses stumble upon... A much better behavior for getaddresses in this case would be returning the perfectly valid address, and ignoring the invalid dtext part. -- Added file: http://bugs.python.org/file43715/decode_from_header.py ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27513> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27513] email.utils.getaddresses raises exception from erroneous message get_all input
New submission from Hans-Peter Jansen:
An unfortunate combination of get_all and getaddresses results in a Traceback:
Traceback (most recent call last):
File "misc/decode_from_header.py", line 17, in
print('From: %s' % email.utils.getaddresses(val))
File "/usr/lib64/python3.4/email/utils.py", line 112, in getaddresses
all = COMMASPACE.join(fieldvalues)
TypeError: sequence item 0: expected str instance, Header found
Here's the relevant part of it:
Content-type: text/html;charset=iso-8859-1
From: Itaú Uniclass. <comunicado.com...@atendimento.gotdns.ch>
Obviously, the From header is iso-8859-1 encoded as well, and violates RFC 2822
as such. But making it crash in the usual combination of
val = msg.get('from')
email.utils.getaddresses([val])
isn't the real McCoy either..
--
components: email
files: iso-8859-1-encoded-from-header.mail
messages: 270399
nosy: barry, frispete, r.david.murray
priority: normal
severity: normal
status: open
title: email.utils.getaddresses raises exception from erroneous message get_all
input
type: crash
versions: Python 3.4
Added file: http://bugs.python.org/file43714/iso-8859-1-encoded-from-header.mail
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue27513>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27256] email header indentation destroyed
Hans-Peter Jansen added the comment: Sorry guys for not providing this earlier. It turned out, that the sub optimal behaviour is related to a unfortunate policy choice: email.policy.SMTP. -- Added file: http://bugs.python.org/file43417/email_flatten.py ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27256> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27257] get_addresses results in traceback with an addrspec with an empty local part.
Hans-Peter Jansen added the comment: Sorry guys for not providing this earlier. It turned out, that the sub optimal behaviour is related to a unfortunate policy choice: email.policy.SMTP. -- Added file: http://bugs.python.org/file43416/email_flatten.py ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27258] Exception in BytesGenerator.flatten
Hans-Peter Jansen added the comment: Sorry guys for not providing this earlier. It turned out, that the sub optimal behaviour is related to a unfortunate policy choice: email.policy.SMTP. -- Added file: http://bugs.python.org/file43415/email_flatten.py ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27258> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: In a couple of systems, I have to stick with 3.4. Is there a chance to have this patch in 3.4 as well, if a new release 3.4 is made? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27257] get_addresses results in traceback with a valid? header
Hans-Peter Jansen added the comment: Dear Stephen, thanks for your care. I'm glad, that you're able to reproduce it. This header is added from the email provider (the biggest here in Germany), so yes, it deserves an entry in the defects list, but must not traceback, of course. It is not expected to provide a sensible way of interoperability otherwise. The unlisted-recipients part is a bit more useful in this respect. -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27257> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27258] Exception in BytesGenerator.flatten
New submission from Hans-Peter Jansen: Attached mail, parsed with email.message_from_binary_file results in: Traceback (most recent call last): File "./mail_filter.py", line 616, in ret = main.run() File "./mail_filter.py", line 605, in run self.process(fp) File "./mail_filter.py", line 589, in process self.save_message(msg, self._fname + '.out') File "./mail_filter.py", line 103, in save_message ofd.write(msg.as_bytes()) File "/usr/lib64/python3.4/email/message.py", line 179, in as_bytes g.flatten(self, unixfrom=unixfrom) File "/usr/lib64/python3.4/email/generator.py", line 115, in flatten self._write(msg) File "/usr/lib64/python3.4/email/generator.py", line 195, in _write self._write_headers(msg) File "/usr/lib64/python3.4/email/generator.py", line 422, in _write_headers self._fp.write(self.policy.fold_binary(h, v)) File "/usr/lib64/python3.4/email/policy.py", line 190, in fold_binary folded = self._fold(name, value, refold_binary=self.cte_type=='7bit') File "/usr/lib64/python3.4/email/policy.py", line 204, in _fold return self.header_factory(name, ''.join(lines)).fold(policy=self) File "/usr/lib64/python3.4/email/headerregistry.py", line 255, in fold return header.fold(policy=policy) File "/usr/lib64/python3.4/email/_header_value_parser.py", line 300, in fold self._fold(folded) File "/usr/lib64/python3.4/email/_header_value_parser.py", line 1228, in _fold rest._fold(folded) File "/usr/lib64/python3.4/email/_header_value_parser.py", line 338, in _fold if folded.append_if_fits(part, tstr): File "/usr/lib64/python3.4/email/_header_value_parser.py", line 149, in append_if_fits token._fold(self) File "/usr/lib64/python3.4/email/_header_value_parser.py", line 324, in _fold for part in self.parts: File "/usr/lib64/python3.4/email/_header_value_parser.py", line 254, in parts if token.startswith_fws(): File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws return self[0].startswith_fws() File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws return self[0].startswith_fws() File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws return self[0].startswith_fws() File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws return self[0].startswith_fws() File "/usr/lib64/python3.4/email/_header_value_parser.py", line 267, in startswith_fws return self[0].startswith_fws() IndexError: list index out of range when flattened with BytesGenerator. -- components: email files: flatten-exception.mail messages: 267736 nosy: barry, frispete, r.david.murray priority: normal severity: normal status: open title: Exception in BytesGenerator.flatten versions: Python 3.4, Python 3.5, Python 3.6 Added file: http://bugs.python.org/file43288/flatten-exception.mail ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27258> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27257] get_addresses results in traceback with a valid? header
New submission from Hans-Peter Jansen:
In the course of replacing an old Python 2.7 email filter tool with a rewritten
Python3 version, I stumbled across a ugly case, where such an header:
To: unlisted-recipients: ;,
""@pop.kundenserver.de (no To-header on input)
results in a Traceback (most recent call last):
File "./mail_filter.py", line 606, in
ret = main.run()
File "./mail_filter.py", line 595, in run
self.process(fp)
File "./mail_filter.py", line 520, in process
config.recipients = self.get_addresses('to', msg)
File "./mail_filter.py", line 103, in get_addresses
vals = msg.get_all(field, [])
File "/usr/lib64/python3.4/email/message.py", line 511, in get_all
values.append(self.policy.header_fetch_parse(k, v))
File "/usr/lib64/python3.4/email/policy.py", line 145, in header_fetch_parse
return self.header_factory(name, ''.join(value.splitlines()))
File "/usr/lib64/python3.4/email/headerregistry.py", line 584, in __call__
return self[name](name, value)
File "/usr/lib64/python3.4/email/headerregistry.py", line 195, in __new__
cls.parse(value, kwds)
File "/usr/lib64/python3.4/email/headerregistry.py", line 342, in parse
for mb in addr.all_mailboxes]))
File "/usr/lib64/python3.4/email/headerregistry.py", line 342, in
for mb in addr.all_mailboxes]))
File "/usr/lib64/python3.4/email/_header_value_parser.py", line 837, in
local_part
return self[0].local_part
File "/usr/lib64/python3.4/email/_header_value_parser.py", line 889, in
local_part
return self[0].local_part
File "/usr/lib64/python3.4/email/_header_value_parser.py", line 984, in
local_part
tok[0].token_type == 'cfws'):
IndexError: list index out of range
I'm not completely sure, if the Top header, as added from my email provider, is
perfectly valid, but none of the other parts of my mail infrastructure neither
complained, nor behave strange with such headers.
This happens with 3.4.4, but also with the email module from current hg for
testing.
--
components: email
files: lkml-exception.mail
messages: 267733
nosy: barry, frispete, r.david.murray
priority: normal
severity: normal
status: open
title: get_addresses results in traceback with a valid? header
versions: Python 3.4, Python 3.5, Python 3.6
Added file: http://bugs.python.org/file43287/lkml-exception.mail
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue27257>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27256] header indentation destroyed
Changes by Hans-Peter Jansen <h...@urpla.net>: Added file: http://bugs.python.org/file43286/mf.9__mi0bf.out ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27256> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27256] header indentation destroyed
New submission from Hans-Peter Jansen: In the course of replacing an old Python 2.7 email filter tool with a rewritten Python3 version, I stumbled across a ugly case, where such an header: X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTDJQUjAyTUI1MTQ7MjM6bEtRRlNaUHQvVTk5WCttdktlOUVrUGQvVFBH?= =?utf-8?B?cDFJemVUeXFzOGNzYnZOYWlwMDZpR0YzbXZyY09WaTBKM2pkeUl4S1VDMkxw?= =?utf-8?B?eVRkNWthRW9waUhJTzczTWd5WDZOQ3hMNU1haGFvQTVzVTdRZmxJUnZlblpW?= ... is regenerated as: X-Microsoft-Exchange-Diagnostics: 1;BL2PR02MB514;23:lKQFSZPt/U99X+mvKe9EkPd/TPG p1IzeTyqs8csbvNaip06iGF3mvrcOVi0J3jdyIxKUC2Lp yTd5kaEopiHIO73MgyX6NCxL5MahaoA5sU7QflIRvenZV which is plain wrong of course. I'm using email.message_from_binary_file for parsing and BytesGenerator.flatten for regeneration. Since those are LKML public mails, I'm attaching both versions. I'm using 3.4.4, but also the email module from current hg for testing. -- components: email files: utf8-header-failure.mail messages: 267732 nosy: barry, frispete, r.david.murray priority: normal severity: normal status: open title: header indentation destroyed versions: Python 3.4, Python 3.5, Python 3.6 Added file: http://bugs.python.org/file43285/utf8-header-failure.mail ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue27256> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10808] ssl unwrap fails with Error 0
Hans-Peter Jansen added the comment:
Poor old bug.
Just being bitten from it today, while trying to package pyftpdlib on the
openSUSE build service, which creates a clean reproducible build environment
for all packages, and testing fails.
Part of the game: openssl 1.0.1k, Python 2.7.8
https://build.opensuse.org/package/show/home:frispete:python/python-pyftpdlib
It happens reproducible for i586 only, but not for x86_64, with all the same
versions, and not with a local (much faster) build host.
So it is smells like a timing problem.
[ 97s] ERROR: test_nlst (test_functional_ssl.TestFtpListingCmdsTLSMixin)
[ 97s] --
[ 97s] Traceback (most recent call last):
[ 97s] File
"/home/abuild/rpmbuild/BUILD/pyftpdlib-1.5.1/pyftpdlib/test/test_functional_ssl.py",
line 139, in test_nlst
[ 97s] super(TestFtpListingCmdsTLSMixin, self).test_nlst()
[ 97s] File
"/home/abuild/.local/lib/python2.7/site-packages/pyftpdlib-1.5.1-py2.7.egg/pyftpdlib/test/test_functional.py",
line 1187, in test_nlst
[ 97s] self._test_listing_cmds('nlst')
[ 97s] File
"/home/abuild/.local/lib/python2.7/site-packages/pyftpdlib-1.5.1-py2.7.egg/pyftpdlib/test/test_functional.py",
line 1180, in _test_listing_cmds
[ 97s] self.client.retrlines('%s %s' % (cmd, tempdir), x.append)
[ 97s] File "/usr/lib/python2.7/ftplib.py", line 735, in retrlines
[ 97s] conn.unwrap()
[ 97s] File "/usr/lib/python2.7/ssl.py", line 289, in unwrap
[ 97s] s = self._sslobj.shutdown()
[ 97s] error: [Errno 0] Error
--
nosy: +frispete
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue10808>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: v7: - reorder test code in order to improve edibility -- Added file: http://bugs.python.org/file42586/python-urllib-prefer-lowercase-proxies-v7.diff ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: > In Python 2, it looks like the proxy_bypass_etc() functions are defined > in urllib and imported into urllib2, so it makes sense to include the > tests in test_urllib rather than test_urllib2. The tests are in test_urllib. test_urllib2 is testing proxy behaviour on a higher level, so I think, they're in the correct module, aren't they? -- ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: * blatant error fixed * one test case added -- Added file: http://bugs.python.org/file42582/python-urllib-prefer-lowercase-proxies-v6.diff ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26831] ConfigParser parsing failures with default_section and ExtendedInterpolation options
New submission from Hans-Peter Jansen:
ConfigParser fails in interesting ways, when using default_section and
ExtendedInterpolation options. Running the attached script results in:
ConfigParser() with expected result:
global: [('loglevel', 'WARNING'), ('logfile', '-')]
section1: [('key_a', 'value'), ('key_b', 'morevalue')]
section2: [('key_c', 'othervalue'), ('key_d', 'differentvalue')]
ConfigParser(default_section='global') mangles section separation:
section1: [('loglevel', 'WARNING'), ('logfile', '-'), ('key_a', 'value'),
('key_b', 'morevalue')]
section2: [('loglevel', 'WARNING'), ('logfile', '-'), ('key_c', 'othervalue'),
('key_d', 'differentvalue')]
ConfigParser(interpolation=ExtendedInterpolation) fails with strange error:
Traceback (most recent call last):
File "configparser-test.py", line 36, in
print_sections(cp)
File "configparser-test.py", line 21, in print_sections
cp.read_string(__doc__)
File "/usr/lib64/python3.4/configparser.py", line 696, in read_string
self.read_file(sfile, source)
File "/usr/lib64/python3.4/configparser.py", line 691, in read_file
self._read(f, source)
File "/usr/lib64/python3.4/configparser.py", line 1089, in _read
self._join_multiline_values()
File "/usr/lib64/python3.4/configparser.py", line 1101, in
_join_multiline_values
name, val)
TypeError: before_read() missing 1 required positional argument: 'value'
while it is expected to behave identical.
--
components: Library (Lib)
files: configparser-test.py
messages: 264031
nosy: frispete
priority: normal
severity: normal
status: open
title: ConfigParser parsing failures with default_section and
ExtendedInterpolation options
versions: Python 3.4
Added file: http://bugs.python.org/file42573/configparser-test.py
___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue26831>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: v5: don't require the proxies argument in proxy_bypass_environment() -- Added file: http://bugs.python.org/file42565/python-urllib-prefer-lowercase-proxies-v5.diff ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: Here's the finalized version of this patch, including unit tests. -- Added file: http://bugs.python.org/file42552/python-urllib-prefer-lowercase-proxies-v4.diff ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: Here we go: v3 fixes following issues: * prefer lowercase proxy environment settings, if multiple (disagreeing) settings are specified with differing case schemes (e.g. HTTP_PROXY vs. http_proxy) * an empty proxy variable value resets the related setting correctly * apply this logic to no_proxy, too * document this behaviour * fix misleading docstrings related to proxy_bypass -- Added file: http://bugs.python.org/file42548/python-urllib-prefer-lowercase-proxies-v3.diff ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: Hi Martin, hi Senthil, please find a new patch attached, that incorporates your suggestions. * added a comment to get_proxies doc in urllib.rst * documented and fixed the mixed case scheme * added a note to proxy_bypass_environment, that behaves slightly different in this respect Yes, mixed case situations are not handled in proxy_bypass_environment, just lowercase and uppercase, while lowercase is preferred correctly. I think, that the mixed case situation is pathologic enough and deserves to be ignored here. BTW, while looking at the code, I noticed, that most docstrings of the callers of proxy_bypass_environment are wrong: they say, that the proxies dict is returned, but they return the value of proxy_bypass_environment(), not get_proxies(). A follow up patch could do this in order to clean up this mess: since there's always a call to get_proxies preceding the call to proxy_bypass_environment, we could add a second argument to the latter, passing in the proxy dict, that is thrown away at the moment. Since that carries a 'no' key already, if it exists, using it here would fix this ambiguity. While at it, fix up the affected docstrings. What do you think about the attached patch and the last paragraph? -- Added file: http://bugs.python.org/file42544/python-urllib-prefer-lowercase-proxies.diff ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Hans-Peter Jansen added the comment: Hi Martin, hi Senthil, thanks for the valuable comments. Will incorporate your suggestions later today. Yes, Martin, it's a bug, and should be fixed for 2.7 and 3.5 as well, but I was unsure, if I get some feedback at all... Hence, this is a very nice experience for me. I'm out for jogging now, Pete -- versions: +Python 2.7, Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
Changes by Hans-Peter Jansen <h...@urpla.net>: -- versions: +Python 3.6 -Python 3.5 ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26804] Prioritize lowercase proxy variables in urllib.request
New submission from Hans-Peter Jansen: During programming a function, that replaces a wget call, I noticed, that something is wrong with urllibs proxy handling. I usually use the scheme "http_proxy= wget -N -nd URL" when I need to bypass the proxy. Hence I was pretty confused, that this doesn't work with python(3). Creating an empty ProxyHandler isn't the real Mc Coy either. Diving into the issue, I found getproxies_environment, but couldn't make much sense out of its behavior, up until I noticed, that my OS (openSUSE ) creates both variants of environment variables behind the scenes: uppercase and lowercase. Consequence: python3 needs the scheme "http_proxy= HTTP_PROXY= python3 ..." Since I, like everyone else, prefer gentle tones over the loud, and want to spare this surprise for others in the future, I propose the attached patch. Process environment variables in two passes, first uppercase, then lowercase, allowing an empty lowercase value to overrule any uppercase value. Please consider applying this. -- components: Extension Modules files: prioritize_lowercase_proxy_vars_in_urllib_request.diff keywords: patch messages: 263720 nosy: frispete priority: normal severity: normal status: open title: Prioritize lowercase proxy variables in urllib.request type: behavior versions: Python 3.5 Added file: http://bugs.python.org/file42516/prioritize_lowercase_proxy_vars_in_urllib_request.diff ___ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue26804> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
Python 2.7.5: Strange and differing behavior depending on sys.setdefaultencoding being set
Hi,
I'm experiencing strange behavior with attached code, that differs depending
on sys.setdefaultencoding being set or not. If it is set, the code works as
expected, if not - what should be the usual case - the code fails with some
non-sensible traceback.
I tried to boil it down to a comprehensible state, but some LOC are still
involved. I use similar code to represent database records, where the repr's
appear in user visible logs for example.
It is greatly appreciated, if some kind soul could shed some light on this
ungrateful behavior.
Thanks in advance,
Pete#!/usr/bin/env python
# -*- coding: utf-8 -*-
u
# this code will usually crash with python 2.7.5, except some poor soul has thrown
# a file called sitecustomize.py into systems site-packages containing:
import sys
sys.setdefaultencoding('utf-8')
# question is, why does it crash without defining a default encoding?
# these are basically two classes with a __repr__ method, where one class
# refers to the other. On a repr, a two level recursion should happen.
# from the log traces, one can see, that in the good case, this is dealt
# well, in the normal case, the recursion isn't handled well: when it
# should continue, it restarts the loop for some reason.
# with default encoding set (comments starting the #):
DEBUG: ['a', 'b', 'c', 'd', 'e', 'f'] # iterate over these elements of D2Rec
DEBUG: a
DEBUG: b
DEBUG: c
DEBUG: d
DEBUG: ['bla', 'ho', 'hu', 'pi', 'tup'] # iterate over these elements of D1Rec
DEBUG: bla
DEBUG: ho
DEBUG: hu
DEBUG: pi
DEBUG: tup # D1Rec exhausted
DEBUG: e# continue with rest of D2Rec
DEBUG: f# finish
DEBUG: D2Rec( # dump structure
a: 1
b: 2
c: 3
d: [D1Rec(
bla: []
ho: u'äöü'
hu: 'hu'
pi: 3.14
tup: ()
)]
e: u'fünf'
f: 'sechs'
)
# all is good, without (the normal, but failing way):
DEBUG: ['a', 'b', 'c', 'd', 'e', 'f'] # iterate over these elements of D2Rec
DEBUG: a
DEBUG: b
DEBUG: c
DEBUG: d
DEBUG: ['bla', 'ho', 'hu', 'pi', 'tup'] # iterate over these elements of D1Rec
DEBUG: bla
DEBUG: ho
DEBUG: hu
DEBUG: pi
DEBUG: tup # D1Rec exhausted
DEBUG: ['a', 'b', 'c', 'd', 'e', 'f'] # it should continue with rest of D1Rec
DEBUG: a# but restarts the process for unknown reasons
DEBUG: b# effectively processing these items twice
DEBUG: c# only to commit suicide thereafter
DEBUG: d
DEBUG: ['bla', 'ho', 'hu', 'pi', 'tup']
DEBUG: bla
DEBUG: ho
DEBUG: hu
DEBUG: pi
DEBUG: tup
Traceback (most recent call last):
File /usr/lib64/python2.7/logging/__init__.py, line 851, in emit
msg = self.format(record)
File /usr/lib64/python2.7/logging/__init__.py, line 724, in format
return fmt.format(record)
File /usr/lib64/python2.7/logging/__init__.py, line 467, in format
s = self._fmt % record.__dict__
File reprtest.py, line 80, in __repr__
return u'%s(\n%s\n)' % (self.__class__.__name__, frec(self.__dict__))
File reprtest.py, line 66, in frec
ret.append(u'%*s: %s' % (maxklen, key, reprstr(rec[key])))
File reprtest.py, line 53, in reprstr
s = repr(s)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 22-24: ordinal not in range(128)
Logged from file reprtest.py, line 108
# Consequently, the traceback doesn't make any sense..
import sys
import logging
logconfig = {
'level': logging.DEBUG,
'format': '%(levelname)s: %(message)s',
'encoding': 'utf8',
}
logging.basicConfig(**logconfig)
log = logging.getLogger(__name__)
def isascii(s):
tests a string, if it can be represented as pure ascii
return all(ord(c) 128 for c in s)
def reprstr(s):
helper to format values in a python 2 compatible way,
using unicode only, where necessary, and quote strings
if isinstance(s, basestring):
if isascii(s):
s = repr(str(s))
else:
assert isinstance(s, unicode), only unicode for non ascii strings allowed: %r % s
s = u'%s' % s.replace(', \\')
else:
s = repr(s)
return s
def frec(rec):
'''format a dict in a easy to read sorted record presentation
'''
ret = []
keys = [key for key in rec]
maxklen = len(keys) and max([len(key) for key in keys]) or 0
log.debug(sorted(keys))
for key in sorted(keys):
log.debug(key)
ret.append(u'%*s: %s' % (maxklen, key, reprstr(rec[key])))
return u'\n'.join(ret)
def recordfactory(classname, **kwargs):
record factory, returning a class name classname,
and keyword args assigned as class members
class Record(object):
represent a Record, carrying its attributes as class members
def __init__(self, **kwargs):
self.__dict__.update(kwargs)
Re: Python 2.7.5: Strange and differing behavior depending on sys.setdefaultencoding being set
Hi Chris,
On Mittwoch, 4. Dezember 2013 10:20:31 Chris Angelico wrote:
On Wed, Dec 4, 2013 at 9:32 AM, Hans-Peter Jansen h...@urpla.net wrote:
I'm experiencing strange behavior with attached code, that differs
depending on sys.setdefaultencoding being set or not. If it is set, the
code works as expected, if not - what should be the usual case - the code
fails with some non-sensible traceback.
Interesting. You're mixing str and unicode objects a lot here. The
cleanest solution, IMO, would be to either switch to Python 3 or add
this to the top of your code:
from __future__ import unicode_literals
Either way, you'll have all your quoted strings be Unicode, rather
than byte, strings. Then take away the requirement that Unicode
strings contain non-ASCII characters, and let everything go through
that code branch.
Looking at this line in reprstr():
s = u'%s' % s.replace(', \\')
Two potential problems with that. Firstly, the representation is
flawed: a backslash in the input string won't be changed, so it's not
a true repr; but if this is just for debugging output, that's not a
big deal. Secondly, this code might produce either a str or a unicode,
depending on the type of s. That may cause messes later; since you
seem to be mostly working with the unicode type after that, it'd
probably be simpler/safer to make that always return one:
The code serves three purposes: make simple strings more readable, document
the others as being unicode, and display those correctly ;)
s = uu'%s' % s.replace(', \\')
But the actual problem, I think, is that repr() guarantees to return a
str, and you're trying to return a unicode. Here's an illustration:
# -*- coding: utf-8 -*-
class Foo(object):
def __repr__(self):
return u'äöü'
foo = Foo()
print(foo.__repr__())
print(repr(foo))
The first one succeeds, because building up that string isn't at all a
problem. The second one then tries to turn the return value of
__repr__ into a string using the default encoding - which defaults to
'ascii', hence the problem you're seeing.
Solution 1: Switch to Python 3, in which this will work fine (because
repr() in Py3 returns a Unicode string, since _everything_ is
Unicode).
Solution 2: Explicitly encode in frec, or at the end of Record.__repr__():
def __repr__(self):
s = u'%s(\n%s\n)' % (self.__class__.__name__,
frec(self.__dict__)) return s.encode(utf-8)
(that could be a one-liner, but it's already pushing 80-chars, so if
you have a length limit, breaking it helps)
Solution 3: Don't use __repr__ here, but simply have your frec
function intelligently handle Record types. Effectively, you have your
own method of generating a debug description of a Record, which could
then return a unicode instead of a str.
Thanks for all your considerations, they are very helpful indeed. Even more
helpful, that I understand the issue in question now. I will take some rest
and then decide, what to do about this with your precious help.
I personally recommend switching to Python 3 :) But presumably that's
not an option, or you'd already have considered it.
You nailed it ;)
Given the amount of special unicode handling code, that is necessary to keep
Python 2 happy, makes proceeding with it no real fun on a longer term..
And the biggest proponent for hacking in Python IS the fun part of it. Then
productivity, elegance, ..., you name it.
Have-a-good-day-ly y'rs,
Pete
--
https://mail.python.org/mailman/listinfo/python-list
Re: debugging segfaults in pythen PyQt (QWebview)
On Thursday 03 March 2011, 10:40:20 Gelonida wrote: Hi, I have a QWebview application, which segfaults rather often, but not all the time. I assume it is some kind of race condition when loading a certain web page with quite some built in AJax. [...] The application crashes under Windows and under Linux. [...] Is there any way to obtain the related backtrace of the python script? [...] Before you're able to track your issue back into python, you will have to prepare a debuggable environment. In Linux, there's usually a way to install the debug symbols of the packages in question. In your case, you might want to install debug symbols (and probably debug sources) of (package names differ from distro to distro, of course): - libqt4 - libwebkit - python - python-sip - python-qt4 Next, you run your script: gdb python -ex set args script.py -ex run Now, get it to crash and type bt at the gdb prompt. Paste all info before and after bt into a mail, and send it to p...@riverbankcomputing.com ML (at least). That's the proper place for such issues, and chances are high, that you will get further help over there. It's a good idea to subscribe that (low traffic) list, even if you hack PyQt only occasionly. It has an astonishing high S/R ratio, btw (compared to the 50 other lists, that I'm subscribed at). Don't forget to specify your environment as detailed as possible (arch, package versions, etc..). A perfect report includes a minimum runnable example script. Good luck, Pete -- http://mail.python.org/mailman/listinfo/python-list
floating point woes
Hi, while I usually cope with the woes of floating point issues, this is one, that I didn't expect: round(2.385, 2) 2.3799 Doesn't the docs say, it's rounded up for this case? quote Values are rounded to the closest multiple of 10 to the power minus n; if two multiples are equally close, rounding is done away from 0 /quote Well, that one is clearly rounding down. What's up, eh, down here? Pete Python 2.6 (r26:66714, Feb 8 2011, 08:50:11) [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2 -- http://mail.python.org/mailman/listinfo/python-list
Re: floating point woes
On Wednesday 16 February 2011, 01:06:08 Benjamin Kaplan wrote: On Tue, Feb 15, 2011 at 6:49 PM, Hans-Peter Jansen h...@urpla.net wrote: Hi, while I usually cope with the woes of floating point issues, this is one, that I didn't expect: round(2.385, 2) 2.3799 Doesn't the docs say, it's rounded up for this case? quote Values are rounded to the closest multiple of 10 to the power minus n; if two multiples are equally close, rounding is done away from 0 /quote Well, that one is clearly rounding down. What's up, eh, down here? Pete The number you are rounding is not 2.385. It is not possible to represent that number in binary, just like you cannot represent the value 1/3 in decimal. So instead, you're using the nearest approximation that an IEEE 754 Double-Precision Floating Point number can get you, which happens to be about 2.3848. And that rounds down to 2.38. Which also cannot be precisely represented in binary, so you get 2.3799 instead. Thanks for the explanation, Benjamin. Not that I like it, but anyway. If I hadn't quitted smoking a long time ago, I would go and ask, what these engineers smoked during the course of inventing this sh*t. Even more probably, they took way too much of a special form of lysergic acid. OTOH, cdecimals, as in Stefan Krah's package are long overdue to get into the core. Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: floating point woes
On Wednesday 16 February 2011, 01:24:59 Chris Rebert wrote:
On Tue, Feb 15, 2011 at 4:09 PM, Chris Rebert c...@rebertia.com
wrote:
On Tue, Feb 15, 2011 at 3:49 PM, Hans-Peter Jansen h...@urpla.net
wrote:
Hi,
while I usually cope with the woes of floating point issues, this
is
one, that I didn't expect:
round(2.385, 2)
2.3799
Doesn't the docs say, it's rounded up for this case?
quote
Values are rounded to the closest multiple of 10 to the power
minus n; if two multiples are equally close, rounding is done away
from 0 /quote
Well, that one is clearly rounding down.
What's up, eh, down here?
http://docs.python.org/library/functions.html#round :
Note: The behavior of round() for floats can be surprising: for
example, round(2.675, 2) gives 2.67 instead of the expected 2.68.
This is not a bug: it’s a result of the fact that most decimal
fractions can’t be represented exactly as a float. See Floating
Point Arithmetic: Issues and Limitations[1] for more information.
[1]: http://docs.python.org/tutorial/floatingpoint.html
And indeed:
from decimal import Decimal
Decimal(2.385)
Decimal('2.3847868371792719699442386627197265625')
Which, rounded to 2 decimal places, gives us 2.38, which is in turn
approximated as:
If that only wouldn't be so arkward to use:
from cdecimal import Decimal, ROUND_HALF_UP
d = Decimal(2.385)
d
Decimal('2.385')
d.quantize(Decimal('1.00'))
Decimal('2.38')
hrmpf.
d.quantize(Decimal('1.00'), ROUND_HALF_UP)
Decimal('2.39')
Oh, well. This is a bit too Cobolesque. (Yes, sure, I know, I can define
any context, I like.)
[*whacks forehead hard*]
Nevermind.
Too true.
- Chris
Pete
--
http://mail.python.org/mailman/listinfo/python-list
Re: GUI Tools for Python 3.1
On Friday 24 December 2010, 03:58:15 Randy Given wrote: Lots of stuff for 2.6 and 2.7 -- what GUI tools are there for 3.1? PyQt4 of course. http://www.riverbankcomputing.com Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: Read / Write OpenOffice SpreadSheet ?
On Friday 17 December 2010, 02:07:07 Torsten Mohr wrote: Hi, i search for a possibility to access OpenOffoce SpreadSheets from Python with a reasonably new version of Python. Can anybody point me to a package that can do this? http://ooopy.sourceforge.net/ Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: while True or while 1
On Tuesday 14 December 2010, 21:38:47 Arnaud Delobelle wrote: Christian Heimes li...@cheimes.de writes: [...] Tres Seavers once told me a joke like this: True = not not Who's at the door? # say it out loud! This was back in the old days of Zope 2.5 and Python 2.1, which didn't have True and False. I almost used: True = to be or not to be # that is the question That's wrong: to be or not to be 'to be' You need to wrap it with bool() at least (even without interpreting Pythons answer to the duality contradiction of consciousness for now ;-)) but didn't dare! Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: default argument in method
On Monday 13 December 2010, 18:14:27 Godson Gera wrote: On Sun, Dec 12, 2010 at 5:05 PM, ernest nfdi...@gmail.com wrote: Hi, I'd like to have a reference to an instance attribute as default argument in a method. It doesn't work because self is not defined at the time the method signature is evaluated. For example: class C(object): def __init__(self): self.foo = 5 def m(self, val=self.foo): return val Raises NameError because 'self' is not defined. You can defined foo outside the __init__ and can access it inside methods using self.foo class C(object): foo=5 def __init__(self): print self.foo def m(self,val=foo): return val class attributes can be accessed with out self before them in method signatures. However, becareful if you change the value of foo inside any method, the method signature will still hold on to old value. Since this is a major pitfall, it might be worth mentioning, that mutable default arguments are generally a bad idea, as the default arguments are evaluated just once, hence e.g. using an empty list might contain the items, that were appended in earlier calls of this method.. Code, that _relies_ on such behavior should be yanked instantaneous and the producer of such code should be punished with coding APL¹ on a dubeolsik hangul keyboard² for a year at least.. Pete ¹) Not, that APL is a bad language per se, but even one liners tend to be rather cryptic for usual brains. Let's say it with Dijkstra: APL is a mistake, carried out through perfection... but given the second constraint, it's going to be a, hmm, challenge. http://en.wikipedia.org/wiki/APL_(programming_language) ²) http://en.wikipedia.org/wiki/Keyboard_layout#Dubeolsik -- http://mail.python.org/mailman/listinfo/python-list
Re: default argument in method
On Thursday 16 December 2010, 00:56:31 Steven D'Aprano wrote: On Wed, 15 Dec 2010 21:10:05 +0100, Hans-Peter Jansen wrote: Since this is a major pitfall, it might be worth mentioning, that mutable default arguments are generally a bad idea, as the default arguments are evaluated just once, hence e.g. using an empty list might contain the items, that were appended in earlier calls of this method.. It's only a pitfall for users who expect that default arguments are re- created every time you call the function; it's only a bad idea for code which relies on the default arguments being re-created each time. If you hold misunderstandings about the behaviour of a language, you'll have trouble understanding what code does. Default arguments are no different from any other feature. Code, that _relies_ on such behavior should be yanked instantaneous and the producer of such code should be punished with coding APL¹ on a dubeolsik hangul keyboard² for a year at least.. Python code that relies on default arguments to *not* be re-created on each function call is no worse than (say) Ruby code that relies on default arguments *to* be re-created each time. I don't mean to be elitist (ah, who am I fooling, of course I do), but when coders of the skill and experience of the Effbot and Guido use mutable defaults, who are you to say they shouldn't? http://effbot.org/zone/default-values.htm http://www.python.org/doc/essays/graphs/ Hmm, thanks for the pointers, Steven. I stand corrected, as I won't argue with taste.. I like the part about the disastrous results specially. If such code would be used in any collaborations, I would expect an explicit comment at least. Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: while True or while 1
On Tuesday 14 December 2010, 10:19:04 Gregory Ewing wrote: Steven D'Aprano wrote: while True: ... print Looping ... True = 0 Just remember that if you use that inside a function, you'll have to initialise True to True before... er, wait a moment, that won't work... ah, I know: def f(true = True): True = true while True: ... True = False Thankfully, with Python 3 this code falls flat on its face. If I would have to _consume_ code like that more often, it would require me to also use a vomit resistant keyboard cover.. Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: remote control firefox with python
On Sunday 28 November 2010, 16:22:33 News123 wrote:
Hi,
I wondered whether there is a simpe way to
'remote' control fire fox with python.
With remote controlling I mean:
- enter a url in the title bar and click on it
- create a new tab
- enter another url click on it
- save the html document of this page
- Probably the most difficult one: emulate a click or 'right click'
on a certain button or link of the current page.
- other interesting things would be to be able to enter the master
password from a script
- to enable disable proxy settings while running.
The reason why I want to stay within Firefox and not use any other
'mechanize' frame work is, that the pages I want to automate might
contain a lot of javascript for the construction of the actual page.
If webkit based rendering in an option (since its javascript engine is
respected by web developers nowadays..), you might want to check out
PyQt, based on current versions of Qt. It provides very easy access to
a full featured web browser engine without sacrificing low level
details. All your requirements are provided easily (if you're able to
grok the Qt documentation, e.g. ignore all C++ clutter, you're set).
I've transcoded all available QtWebKit examples to python lately,
available here:
http://www.riverbankcomputing.com/pipermail/pyqt/2010-November/028614.html
The attachment is a tar.bz2 archive, btw.
Clicking is archived by:
webelement.evaluateJavaScript(
var event = document.createEvent('MouseEvents');
event.initEvent('click', true, true);
this.dispatchEvent(event);
)
Cheers,
Pete
--
http://mail.python.org/mailman/listinfo/python-list
Re: PyQt Installation Problem on Windows
On Wednesday 24 November 2010, 23:03:14 Saul Spatz wrote: Hi, I've been trying to install PyQt on Windows XP Pro so that I can try out eric ide. I used the binary windows installer for PyQt. I can run eric as administrator, but not with my ordinary user account. By running eric.bat with the --debug flag, I found that he crux of the problem is that if I type import PyQt4 in the python shell, it works for both users, but if I type import PyQt4.QtCore it works for the administrator, but the non-privileged account gets the message ImportError: DLL load failed: The specified module could not be found. My WinXP installations are always a bit lacking, since I do work 97% of my time in Linux (and 2% on MacOSX), but Python 2.6.6 and PyQt 4.7.7 do work fine here as a user win XP SP3/i586. Things like what you describe usually point to broken installations (remaining manual build artefacts, unfinished installs, incomplete deinstallations, ...). Just uninstall Python and PyQt, remove everything below your Python base path, and start over (a registry check/cleanup might be worth, too). In the file the file pyqtconfig.py from Python26\Lib\site-packages \PyQt4 I have the line 'pyqt_config_args': '--confirm-license -b C:\\Python26\\Lib\\site- packages\\PyQt4\\bin', I checked with a friend who uses eric, and his file does not have the --confirm-license parameter. As far as we can tell, we followed the same installation procedures. This is unrelated, since it is only a detail in the build process, but points to differing versions. By the way, I have tried this with python 3.1 on the same machine with similar results. I've blown a whole day playing with this, so I'd really appreciate any help you can give me. Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: http error 301 for urlopen
On Tuesday 09 November 2010, 03:10:24 Lawrence D'Oliveiro wrote: In message 4cd7987e$0$1674$742ec...@news.sonic.net, John Nagle wrote: It's the New York Times' paywall. They're trying to set a cookie, and will redirect the URL until you store and return the cookie. And if they find out you’re acessing them from a script, they’ll probably try to find a way to block that as well. ..which could be alleviated by carefully crafting the requests ;-) Luckily, unpleasant related ground work was already done by others, e.g.: http://bugs.python.org/issue2275 Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: ANN: PyQt v4.8.1 Released
Am Monday 08 November 2010 02:26:51 schrieb Robert Kern: On 2010-11-07 18:53 , Lawrence D'Oliveiro wrote: In messagemailman.720.1289149298.2218.python-l...@python.org, Robert Kern wrote: Everyone here knew exactly what he meant. But if you don’t banana the right tomato, everybody could be grapefruit, right? You know what I mean. And as I reiterated in the part that you snipped, he is not using commercial as a synonym for proprietary. It is the license that he sells commercially. ..with a pretty affordable price, I may add. I don't understand all the fuzz about some obvious formulation. It it a concern only for those who want to _sell_ their _products_ based on PyQt and consequently don't want to reveal their source code. Pete (Not being afflicted in any way with Riverbankcomputing, other than being a satisfied commercial licencee). -- http://mail.python.org/mailman/listinfo/python-list
Re: Python changes
On Thursday 28 October 2010, 21:23:03 Craig McRoberts wrote: Oh, I like to browse brick-and-mortar enough. But it's been forever since I've bought something there. If you can get your hands on a copy of Mark Summerfield's Programming in Python3, check it out. He really raised the accustomed quality levels of technical writings for me. Pete On Oct 28, 2010, at 15:16, Teenan t33...@gmail.com wrote: On Thu, 2010-10-28 at 15:03 -0400, Craig McRoberts wrote: Thanks for the prompt replies. Sounds like it's time to hit a bookstore. Craig McRoberts You could do a lot worse than getting 'Dive into Python' (There's even a nice new version just for python 3.0) hmmm bookstore.. those are the things they had before Amazon right? ;) -- http://mail.python.org/mailman/listinfo/python-list
ANN: automated daily snapshot builds for PyQt and friend on openSUSE build service
[Sorry for cross posting] Hi PyQtnistas, I proudly announce the availability of automated builds of the most current PyQt and related packages including snapshots on openSUSEs build service for openSUSE 11.1, 11.2 and 11.3, here: https://build.opensuse.org/project/monitor?project=home%3Afrispete%3APyQt https://build.opensuse.org/project/monitor?project=home%3Afrispete%3APyQt-next New sip4, PyQt3 and PyQt4 snapshots and release get build against a range of gcc and Qt versions automatically, e.g. without human intervention (if all goes well, famous last words..). dip and PyQtMobility will probably follow soon. If you add both home:frispete:PyQt home:frispete:PyQt-next to your list of repos, than you get the current snapshot builds of qscintilla, sip4, PyQt3 and PyQt4, with dependent packages, like PyQwt5, PyKDE3 and PyKDE4. Omitting or deactivating the latter, you can switch back to the current released versions with: zypper dup -r home_frispete_PyQt BTW, home:frispete:PyQt contains the builds of the current versions of a lot of our favorite stuff: e.g. eric4, PyQwt5. eric is lacking the newest release, but I didn't manage to automate the sourceforge download process, yet. How to choose your target? Depending on which other repos you're using, choose your target accordingly, e.g. if you have the KDE:Distro:Stable (KDE 4.4) repo included, use KDE_Distro_Stable_openSUSE_11.x, KDE_Distro_Factory_openSUSE_11.x for KDE:Distro:Factory (KDE 4.5), or none of them, then use plain openSUSE_11.x. Note, that you implicitely choose your systems Qt4 version with this decision. I hope, this fullfills the most common needs. PyKDE4 is only provided for KDE_Distro_Stable_x ATM, since I didn't got around splitting this package into a 4.4 and 4.5 version. All in all, these repos provide the the cheapest way of keeping current with the PyQt project, that I know of. Comments welcome. Enjoy, Pete -- http://mail.python.org/mailman/listinfo/python-list
C++ vs. Python Was: Re: help!!!
On Wednesday 06 October 2010, 06:28:51 Dennis Lee Bieber wrote: On Tue, 05 Oct 2010 23:54:00 -0400, fkr...@aboutrafi.net23.net declaimed the following in gmane.comp.python.general: plz can u convert this cpp file into python i need that badly as soon as possible... I am new to python. I just wanna learn it Step one... DON'T TRY TO PORT C++ to Python... the object models are quite different. I do this all the time without any adverse effects (other than being glad to only rarely having the need of doing it the other way around ;-)). And the models aren't _that_ different, the syntax is. Check yourself: http://doc.qt.nokia.com/qtmobility-1.0/lightmaps.html vs. http://www.riverbankcomputing.com/pipermail/pyqt/2010-October/028040.html Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: PyQt imageViewer does not working properly...
On Tuesday 05 October 2010, 00:29:04 Polimeno wrote: Hello guys, I have been looking throughout the web for some PyQt Image Viewer : http://nullege.com/codes/show/src%40pyformex-0.8.2%40pyformex%40gui%40ima geViewer.py/78/PyQt4.QtGui.QImage# Unfortunately, everytime I input any kind of image type (.jpeg, .tga, .png, whatver) It doesn´t show me the image inside the widget itself looks like it ignores the path I did pick... Even if I use a simple snippet like one below, I can´t get my display image... from PyQt4.QtGui import * from PyQt4.QtCore import * import sys class ImageViewer(QWidget): def __init__(self, imgFile): QWidget.__init__(self) self.image = QImage(imgFile) print self, file_Path, '\n' self.update() def paintEvent(self, event): self.painter = QPainter(self) self.painter.drawImage(0, 0, self.image) if __name__ == __main__: Qapp = QApplication(sys.argv) Iviewer = ImageViewer(imgFile='C:\\Users\\Administrador\\Desktop\ \Img_001.jpg') Iviewer.show() Qapp.exec_() What am I missing ? Hmm, first of all, a QLabel would be better suited for the task in question, since that one would display some image just fine without any painter, but anyway, here's a working example: import sys from PyQt4 import QtCore, QtGui class Widget(QtGui.QWidget): def __init__(self, imgFile, parent = None): super(Widget, self).__init__(parent) self.image = QtGui.QImage(imgFile) self.resize(self.image.size()) def paintEvent(self, event): p = QtGui.QPainter(self) p.drawImage(event.rect(), self.image) try: imgFile = sys.argv[1] except IndexError: print sys.stderr, %s: image % sys.argv[0] sys.exit(1) app = QtGui.QApplication(sys.argv) iv = Widget(imgFile) iv.show() sys.exit(app.exec_()) Depending on the window frame size, the image might be distorted with small images, because the painter tries to fit the image into the window. On the plus side, this code is able to cope with all supported image formats, hence even svg files. Hth, Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: segfault with small pyqt script
On Monday 16 August 2010, 09:22:27 Gelonida wrote: Hi Hans-Peter, It seems, that my other posts did not get through. On 08/15/2010 11:17 PM, Hans-Peter Jansen wrote: For a starter, tell us the versions of python-sip, and python-qt4 or however they're called in Ubuntu. For the record, python-sip-4.10.5-1.1 python-qt4-4.7.4-1.1 doesn't show this behavior. The problem seems to be known for 4.7.2. For simple code I managed to work around the issue. For the real more complicated I didn't. So it seems I'll have to avoid 4.7.2. [...] For more complex multi widget examples it doesn't seem enough to just destroy the main widget. probably I had to recursively assign all widgets / dialogues sub widgets to None. So I'll just try to stay away from this pyqt release and stick with older or newer ones. If you test sip 4.10.5 by chance, it would be nice to leave a brief note about the outcome. As long as you replace (qscintilla, if you use it), sip and PyQt, the probability of a bad influence on your system is pretty small. Replacing PyKDE aka kdebindings4 might not be that easy, though.. I might even get around also building Ubuntu packages in openSUSE build service one day... Various openSUSE builds are available here: http://download.opensuse.org/repositories/home:/frispete:/pyqt/ Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: segfault with small pyqt script
On Thursday 12 August 2010, 01:07:25 Gelonida wrote: Hi Guys, I'm desperate. I'm having a real application, which fails rather often when finishing it. I'm not sure, whether any serious problem could be hidden behind it The script is a pyqt script, which segfaults most of the time on my ubuntu 10.4 linux 64 bit and I'm having trouble to understand why. Trying to create the smallest possible test case I ended up with following script, which I named dodo.py and which i made executable with chmod +x ./dodo.py #!/usr/bin/env python from PyQt4.QtGui import QDialog,QGridLayout,QLabel,QComboBox,QPushButton,QApplication a = one b = unused c = also unused d= ans also unused e = another var f = something class MyForm(QDialog): def __init__(self,parent=None,config=None,ini_info=None): super(MyForm,self).__init__(parent=parent) grid = QGridLayout() quit_btn = QPushButton(Quit) quit_btn.clicked.connect(self.quit) grid.addWidget(quit_btn,0,0) name = a_name vals_box = QComboBox() vals_box.addItem(one) vals_box.addItem(two) grid.addWidget(vals_box,0,1) self.setLayout(grid) def quit(self): self.close() if __name__ == __main__: app = QApplication([]) myform = MyForm() myform.show() retcode = app.exec_() print last In order to perform the test several times I typed on the command line: a= ; while [ $a = ] ; do ./dodo.py ; read -t 1 a ; done As soon as the window shows up I click twice (slowly ) on 'one' and then on quit. For a starter, tell us the versions of python-sip, and python-qt4 or however they're called in Ubuntu. For the record, python-sip-4.10.5-1.1 python-qt4-4.7.4-1.1 doesn't show this behavior. Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: About one class/function per module
On Tuesday 03 November 2009, 12:52:20 Diez B. Roggisch wrote: Peng Yu wrote: On Mon, Nov 2, 2009 at 9:39 PM, alex23 wuwe...@gmail.com wrote: Peng Yu pengyu...@gmail.com wrote: I don't think that this is a problem that can not be overcome. A simple solution might be to associate a unique identifier to each file, so that even the filename has been changed, the new version and the old version can still be identified as actually the same file. Or, again, you could work _with_ the tools you're using in the way they're meant to be used, rather than re-inventing the whole process of version control yourself. I'm not trying to reinvent a new version control. But due to this drawback, I avoid use a version control system. Rather, I compressed my source code in a tar file whenever necessary. But if a version control system has this capability, I'd love to use it. And I don't think that no version control system support this is because of any technical difficult but rather because of practical issue (maybe it takes a lot efforts to add this feature to an existing version control system?). There are those people who realize if *everything* they encounter needs adjustment to fit their needs, their needs need adjustment. Others fight an uphill battle bicker about things not being as they want them. Don't get me wrong - innovation often comes from scratching ones personal itch. But you seem to be suffering from a rather bad case of neurodermatitis. Diez, sorry for chiming in that lately, but while the whole thread is spilled over for no good reason, your QOTW remembered me on a quote of R.A.W., that sounds like a perfect fit: Whatever the Thinker thinks, the Prover will prove. And if the Thinker thinks passionately enough, the Prover will prove the thought so conclusively that you will never talk a person out of such a belief, even if it is something as remarkable as the notion that there is a gaseous vertebrate of astronomical heft (GOD) who will spend all eternity torturing people who do not believe in his religion. From Prometheus Rising by Robert Anton Wilson Pete http://en.wikiquote.org/wiki/Robert_Anton_Wilson -- http://mail.python.org/mailman/listinfo/python-list
[issue2275] urllib2 header capitalization
Hans-Peter Jansen [EMAIL PROTECTED] added the comment: Facundo, first of all: *really* nice work, thanks a lot. While I don't fully understand the issues raised lately here, especially what broke (user code). I can see, that it completely solves my original problem, which is great. While reading the patch, I noticed, that the modifications to Doc/library/urllib2.rst contain two typos (retrive instead of retrieve). The only remaining issue left in this area is using some form of ordered dict for headers in order to control - how it comes - the order of headers ;-), but that's left for another issue to raise some day. ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue2275 ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2275] urllib2 header capitalization
Hans-Peter Jansen [EMAIL PROTECTED] added the comment: But should not this patch be handled in a way wherein. key.capitalize() is just replaced by key.upper()? Hmm, are you sure? hello.upper() 'HELLO' but the issue is with values containing dashes: 'accept-charset'.capitalize() 'Accept-charset' whereas the rest of the world would expect: 'Accept-Charset' ^ This is especially useful, if you try to emulate the behavior of a typical browser as close as possible. __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2275 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2275] urllib2 header capitalization
Hans-Peter Jansen [EMAIL PROTECTED] added the comment: Hi Senthil, that looks promising, and the title() trick is nice, as it fixes my issue.. Thanks, Pete __ Tracker [EMAIL PROTECTED] http://bugs.python.org/issue2275 __ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue2275] urllib2 header capitalization
New submission from Hans-Peter Jansen [EMAIL PROTECTED]:
The urllib2 behavior related to headers is - hmm - improvable.
It simply capitalize() the key, which leads to funny results like:
Accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
while this is seemingly conforming to the specs, it's simply different
to every other implementation of such things..
And we can do better. How about:
--- /usr/lib/python/urllib2.py 2008-01-10 19:03:55.0 +0100
+++ urllib2.py 2008-03-11 21:25:33.523890670 +0100
@@ -261,13 +261,16 @@ class Request:
def is_unverifiable(self):
return self.unverifiable
+def _cap_header_key(self, key):
+return '-'.join((ck.capitalize() for ck in key.split('-')))
+
def add_header(self, key, val):
# useful for something like authentication
-self.headers[key.capitalize()] = val
+self.headers[self._cap_header_key(key)] = val
def add_unredirected_header(self, key, val):
# will not be added to a redirected request
-self.unredirected_hdrs[key.capitalize()] = val
+self.unredirected_hdrs[self._cap_header_key(key)] = val
def has_header(self, header_name):
return (header_name in self.headers or
I'm not happy with the _cap_header_key name, but you get the idea.
The patch is optimized to operate with maximum locality. It's also
attached.
I would be very grateful, if something similar could be applied.
Opinions?
--
components: Library (Lib)
files: urllib2-cap-headers.diff
keywords: patch
messages: 63466
nosy: frispete
severity: minor
status: open
title: urllib2 header capitalization
type: behavior
versions: Python 2.5
Added file: http://bugs.python.org/file9658/urllib2-cap-headers.diff
__
Tracker [EMAIL PROTECTED]
http://bugs.python.org/issue2275
__
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
trouble with generators
Hi Pythonistas, I'm stuck in a maze of new style classes and generators. While I love the concepts, I obviously didn't grok them throughout. I'm trying to generate a bunch of similar classes, where some are contained in list attributes of others, e.g.: class A: def __init__(self): self.id = 'A1' self.b = [instances of B] class B: def __init__(self): self.id = 'B1' Here's the test code, I have: #!/usr/bin/env python # -*- coding: utf8 -*- class A(object): A def __init__(self): self.id = None self.b = [] class B(object): B def __init__(self): self.id = None class Gen(object): def records(self, cls): for i in range(3): setattr(cls, id, %s%s % (cls.__doc__, i)) yield cls def display(self, rec): for i in rec.__dict__.keys(): if not i.startswith(_): print %s: %s: %s % (rec.__doc__, i, rec.__dict__[i]) class GenA(Gen): def __init__(self): self.genB = GenB() def records(self): for a in Gen.records(self, A()): for b in self.genB.records(): #self.genB.display(b) a.b.append(b) #self.display(a) yield a class GenB(Gen): def records(self): return Gen.records(self, B()) # testing.. aRecs = [] bRecs = [] for i, r in enumerate(GenB().records()): bRecs.append(r) print i, r.id, r for i, r in enumerate(GenA().records()): aRecs.append(r) print i, r.id, r for b in r.b: print b.id, b Here's the commented output: # even if I keep a reference to each rec, the object is reused: 0 B0 __main__.B object at 0xb7bd0f8c 1 B1 __main__.B object at 0xb7bd0f8c 2 B2 __main__.B object at 0xb7bd0f8c # same here, with additional quadratic behavior, I do not understand 0 A0 __main__.A object at 0xb7bd206c B2 __main__.B object at 0xb7bd210c B2 __main__.B object at 0xb7bd210c B2 __main__.B object at 0xb7bd210c 1 A1 __main__.A object at 0xb7bd206c B2 __main__.B object at 0xb7bd210c B2 __main__.B object at 0xb7bd210c B2 __main__.B object at 0xb7bd210c B2 __main__.B object at 0xb7bd20ec B2 __main__.B object at 0xb7bd20ec B2 __main__.B object at 0xb7bd20ec 2 A2 __main__.A object at 0xb7bd206c B2 __main__.B object at 0xb7bd210c B2 __main__.B object at 0xb7bd210c B2 __main__.B object at 0xb7bd210c B2 __main__.B object at 0xb7bd20ec B2 __main__.B object at 0xb7bd20ec B2 __main__.B object at 0xb7bd20ec B2 __main__.B object at 0xb7bd0f8c B2 __main__.B object at 0xb7bd0f8c B2 __main__.B object at 0xb7bd0f8c I expected to get 3 different class objects from both sections, with each A containing 3 different Bs in the latter section, but obviously got something else. Could some kind soul help me to distangle my mind twist here? Am I healable? TIA, Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: trouble with generators
Hi Diez, first, thanks for your comprehensive answer. Diez B. Roggisch wrote: Hans-Peter Jansen schrieb: I'm trying to generate a bunch of similar classes, where some are contained in list attributes of others, e.g.: All your code below shows that you don't create classes, but _instances_ of classes. So - is that what you mean to do? Yes, exactly. Sorry for the confusion.. In a nutshell, you do this: b = B() res = [] for i in xrange(3): b.id = i res.append(b) Always the same b. What you seem to want is this class B(object): ...pass ... res = [] for i in xrange(3): ... class BSubClass(B): ... pass ... BSubClass.id = i ... res.append(BSubClass) ... print [c.id for c in res] [0, 1, 2] I'm still not sure what you want - do you want instances created, or classes? For the former, you need constructor calls on your classes, and pass the class instead of an instance. Like this: class B(object): pass def g(cls): for i in xrange(3): o = cls() o.id = i yield o list(g(B)) Yes, that did the trick. Silly me. Rookie error. Here's what I was after: #!/usr/bin/env python # -*- coding: utf8 -*- class A(object): A def __init__(self): self.id = None self.b = [] class B(object): B def __init__(self): self.id = None class Gen(object): Gen def records(self, cls): for n in range(3): i = cls() i.id = %s%s % (i.__doc__, n) yield i class GenA(Gen): def __init__(self): self.genB = GenB() def records(self): for a in Gen.records(self, A): for b in self.genB.records(): a.b.append(b) yield a class GenB(Gen): def records(self): return Gen.records(self, B) aRecs = [] bRecs = [] for i, r in enumerate(GenB().records()): bRecs.append(r) print i, r.id, r for i, r in enumerate(GenA().records()): aRecs.append(r) print i, r.id, r for b in r.b: print b.id, b created pretty nice different _instances_ of what I wanted: 0 B0 __main__.B object at 0xb7cacfec 1 B1 __main__.B object at 0xb7cae04c 2 B2 __main__.B object at 0xb7cae08c 0 A0 __main__.A object at 0xb7cae12c B0 __main__.B object at 0xb7cae1ac B1 __main__.B object at 0xb7cae1ec B2 __main__.B object at 0xb7cae22c 1 A1 __main__.A object at 0xb7cae16c B0 __main__.B object at 0xb7cae2ac B1 __main__.B object at 0xb7cae2ec B2 __main__.B object at 0xb7cae32c 2 A2 __main__.A object at 0xb7cae26c B0 __main__.B object at 0xb7cae3ac B1 __main__.B object at 0xb7cae3ec B2 __main__.B object at 0xb7cae42c Didn't found the forest because all the trees. Thanks again. Greetings to Berlin, Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: trouble with generators
Marc 'BlackJack' Rintsch wrote: In [EMAIL PROTECTED], Hans-Peter Jansen wrote: class Gen(object): def records(self, cls): for i in range(3): setattr(cls, id, %s%s % (cls.__doc__, i)) yield cls […] class GenA(Gen): def __init__(self): self.genB = GenB() def records(self): for a in Gen.records(self, A()): Here you create an instance of `A` and pass that *instance* and not the *class*. If you would pass the class here, you must create objects in `Gen.records()`. Yes, that was my fault, as you both found. Ciao, Marc 'BlackJack' Rintsch Thanks, Marc. Cheers, Pete -- http://mail.python.org/mailman/listinfo/python-list
convert ascii escapes into binary form
Hi Pythonistas, I need to convert ascii escapes into binary form, e.g.: \f - ^L [EMAIL PROTECTED] - [EMAIL PROTECTED]@ (rvalues in terminal representation) Any idea, how to do this most elegantly in python? Do I really need to do a search n'replace orgy, combined with regex for this task? TIA, Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: convert ascii escapes into binary form
Robert Kern wrote:
Hans-Peter Jansen wrote:
Hi Pythonistas,
I need to convert ascii escapes into binary form, e.g.:
\f - ^L
[EMAIL PROTECTED] - [EMAIL PROTECTED]@
(rvalues in terminal representation)
Any idea, how to do this most elegantly in python?
Do I really need to do a search n'replace orgy, combined with
regex for this task?
In [11]: s = '\\f'
In [12]: s.decode('string_escape')
Out[12]: '\x0c'
That did the trick, thanks a lot, Peter. Unfortunately, on the
target system, there's still python 2.0 running :-( Looks like
I've to bite the apple..
Pete
--
http://mail.python.org/mailman/listinfo/python-list
Re: convert ascii escapes into binary form
Hi Robert, Hans-Peter Jansen wrote: Robert Kern wrote: That did the trick, thanks a lot, Peter. Unfortunately, on the s/Peter/Robert/g Sorry, Robert. That's the price to pay for doing multiple replies at the same time. Mea culpa.. target system, there's still python 2.0 running :-( Looks like I've to bite the apple.. Pete -- http://mail.python.org/mailman/listinfo/python-list
Re: wxPython vs. pyQt
[EMAIL PROTECTED] wrote: I've narrowed down my toolkit selection for my project to wxPython and pyQt, and now i'd like to hear any opinions, war stories, peeves, etc, about them, particularly from anyone who's used _both_toolkits_. I'm only mildly interested in the IDEs and UI designers for each, as i want to do as much as i can in just Xemacs and xterm. Feel free to rant, rave, pontificate, whatever. I used both toolkits in projects under linux (mostly for inhouse database applications and the like). First experiences with tkinter and it's look within linux brought me to the decision to try others. Although not necessary that time, being able to use it under windows was appealing. Because of the Qt license, I decided to start with wx, but several issues with it turned my head around soon thereafter. Here is what I still remember: - inconsistent api (for historical and conceptual reasons) - solutions to some problems weren't always obvious - somewhat opaque documentation - painfully slow start of wx applications - gui layering makes it hard to locate a problem domain inside the tk - widgets behave different on different architectures/base libs - patched swig needed for full build/own extensions - tools (gui builder) weren't appealing - printing a single fax page image resulted in a ~50MB PS job (because it wasn't possible to prevent the conversation to a 24 bit image) - spurious gtk error messages While in PyQt world, I found these advantages: + conceptually vastly superior + powerful api/widgets/features + fast as hell due to the efficient binding of a quite efficient lib + cool tools, that are unicode/translation aware + very efficient programming environment/unbeatable productivity While this sounds like the average sales talk, I will try to backup these claims a bit: Phil Thompson does a great job with sip: since V.4, it's able to wrap c++ libs directly without any additional trampoline modules, most other wrappers (includding sip 3) produces. It creates a very thin layer, which results in a tight and efficient binding. Besides all the necessary essential work there, it also realizes lazy binding in order to speed up loading big libs like qt once more. Qt itself is a good example of a mostly well thought out toolkit including a quite consistent api, usable documentation and powerful tools (designer, assistant, linguist), which PyQt fully deploys into the wonderful world of our favorite programming language. Accompanied by eric, itself a great example on how far you can go with PyQt and limited human resources (unfortunately), it is big fun to get some real work done. The power of eric is largely based on qscintilla, Phil's Qt port of the scintilla editor component besides PyQt. For the scientists out there, there's also a cool extension lib available: PyQwt. On the down side let me note: - due to the tight coupling of Qt, PyQt inherits one of Qt's few down sides, which boils down to: garbage in - (probably) crash, but well, we use a real operating system for a reason, don't we ;-) - to get most from the documentation, one gets to a c++ - python converter over time (trains the eye to skip the right portions).. Conclusion: PyQt is the most underrated python gui toolkit out there, but beware: once you get infected, your brain will tend to refuse to work with more long winded toolkits. It allowed me to do things in hours, which literally took weeks with tkinter, while still running quicker, looking and feeling better, providing more features and a noticable higher user satisfaction: IOW, much more fun. Well-I'm-biased-ly-y'rs, Pete -- http://mail.python.org/mailman/listinfo/python-list
