[Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have 10MB pickled structure generated in Python 2.7. I only use basic types (no clases) like sets, dictionaries, lists, strings, etc. The pickle stores a lot of strings. Some of them should be bytes, while other should be unicode. My idea is to import ALL the strings as bytes in Python 3.2 and navigate the data structure to convert the appropiate values to unicode, in a one-time operation (I version the structure, so I can know if this conversion is already done, simply storing a new version value). But I get this error: Python 3.2 (r32:88445, Feb 21 2011, 13:34:07) [GCC 4.4.3] on linux2 Type help, copyright, credits or license for more information. f=open(file.pickle, mode=rb).read() len(f) 9847316 b=pickle.loads(f,encoding=latin1) Traceback (most recent call last): File stdin, line 1, in module ValueError: operation forbidden on released memoryview object I use the encoding latin1 for transparent byte/unicode conversion (do not touch the values!). This seems to be a bug in Python 3.2. Any suggestion?. PS: The bytestream is protocol 2. PPS: If there is consensus that this is a real bug, I would create an issue in the tracker and try to get a minimal testcase. - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTWOomplgi5GaxT1NAQLTYAP/U+eqhQ5nIJyBAqSgYwPmkH4DOlMj4JnH Jgt6okvOV0hRIXlZ7kbWI2l9OuQyUM4gAeTNDSjFaKs9Hswy26Ro6xhtjidivXDS TKw6ocRx92/eHvgsOdEZjrE0D8l0dOqodZddbXELp2DjpYs9aozzAsjTHqNZDE1L fujeTOhtUKw= =/bcO -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
On 22/02/2011 12:14, Jesus Cea wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have 10MB pickled structure generated in Python 2.7. I only use basic types (no clases) like sets, dictionaries, lists, strings, etc. The pickle stores a lot of strings. Some of them should be bytes, while other should be unicode. My idea is to import ALL the strings as bytes in Python 3.2 and navigate the data structure to convert the appropiate values to unicode, in a one-time operation (I version the structure, so I can know if this conversion is already done, simply storing a new version value). But I get this error: Python 3.2 (r32:88445, Feb 21 2011, 13:34:07) [GCC 4.4.3] on linux2 Type help, copyright, credits or license for more information. f=open(file.pickle, mode=rb).read() len(f) 9847316 b=pickle.loads(f,encoding=latin1) Traceback (most recent call last): File stdin, line 1, inmodule ValueError: operation forbidden on released memoryview object That seems like an odd error, but the decision was made that Python 2 byte-strings would be unpickled on Python 3 as Unicode strings. See the discussion here: http://bugs.python.org/issue6784 This is basically because many people do the wrong thing and use Python 2 byte strings for restoring text. What it means though is that people who do the right thing and store binary data in Python 2 byte strings can't use Python 2 pickles from Python 3. It also means that only ascii data can be unpickled. A custom pickler / unpickler is suggested as the solution in this issue. All the best, Michael Foord I use the encoding latin1 for transparent byte/unicode conversion (do not touch the values!). This seems to be a bug in Python 3.2. Any suggestion?. PS: The bytestream is protocol 2. PPS: If there is consensus that this is a real bug, I would create an issue in the tracker and try to get a minimal testcase. - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTWOomplgi5GaxT1NAQLTYAP/U+eqhQ5nIJyBAqSgYwPmkH4DOlMj4JnH Jgt6okvOV0hRIXlZ7kbWI2l9OuQyUM4gAeTNDSjFaKs9Hswy26Ro6xhtjidivXDS TKw6ocRx92/eHvgsOdEZjrE0D8l0dOqodZddbXELp2DjpYs9aozzAsjTHqNZDE1L fujeTOhtUKw= =/bcO -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
On Tue, 22 Feb 2011 13:14:18 +0100 Jesus Cea j...@jcea.es wrote: This seems to be a bug in Python 3.2. Any suggestion?. Report an issue and investigate :) Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 22/02/11 13:20, Michael Foord wrote: Traceback (most recent call last): File stdin, line 1, inmodule ValueError: operation forbidden on released memoryview object That seems like an odd error, but the decision was made that Python 2 byte-strings would be unpickled on Python 3 as Unicode strings. This problem seems NOT related to unicode. In fact, when saying 'encoding=latin1', my binary strings should be converted to unicode without any kind of issue (my plan was, then, to scan the datastructure and convert them to native bytes). The fact is that I get a strange error: ValueError: operation forbidden on released memoryview object. Seems like a bug to me. Google shows no hits. I want to discard any obvious overlooked point. PS: Just checked... Python 3.1.3 imports the pickle just fine. So busy migrating my projects to 3.2 (it was my compromise two years ago :), I don't have time to debug this :). - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTWO7l5lgi5GaxT1NAQJVLQP/fZMxsybbHfkbwDEJ/DVaBSj8VZ2dkO38 oXsH9ojspbxRTv9BCNakKt8SyDMtzJIB6kaZ10qScxftDAGs22xlkpOJyGxBYgNZ Ut5U425YuUTCyFQyYfREWNs2AqUQOWymnXgIlThDS93n1Y+W2S1ovcT9WJaHyebe ZVDabLUZYlw= =IBN8 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
PS: Just checked... Python 3.1.3 imports the pickle just fine. So busy migrating my projects to 3.2 (it was my compromise two years ago :), I don't have time to debug this :). I hope you do have a time to open an issue, though :-) Eli ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 22/02/11 15:32, Eli Bendersky wrote: PS: Just checked... Python 3.1.3 imports the pickle just fine. So busy migrating my projects to 3.2 (it was my compromise two years ago :), I don't have time to debug this :). I hope you do have a time to open an issue, though :-) Eli Bugs bug me a lot :). I spend a couple of hours trying to reduce my pickle to something I could post (the original pickle has tons of propietary information): http://bugs.python.org/issue11286 I got a reproductible pickle testcase in only 41 bytes. Seems to be a SERIOUS regression in 3.2. I can't progress further. Pickle internals are out of my expertise. - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTWPRz5lgi5GaxT1NAQKwdgQAjV5zKB3LhuVMU+JDbPeZjo/oFu1Yz++Z 1xFPuXTtaeMGMYuQH16j5rghqp90Q4u0M/VGaXI99uxcyTR9fpGGVEBE2L0qnVTg 1sbRyCaaVrPDVju3tTonw5QEe7eXnsec9INuK7KCIXUqEZK7klbqoWFFflU5g/Ui hcxe8Zt6lQE= =nWu0 -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Actual Mercurial Roadmap for February (Was: svn outage on Friday)
On Fri, Feb 18, 2011 at 4:00 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Fri, Feb 18, 2011 at 14:41, anatoly techtonik techto...@gmail.com wrote: Do you have a public list of stuff to be done (i.e. Roadmap)? BTW, what is the size of Mercurial clone for Python repository? There is a TODO file in the pymigr repo (though I think that is currently inaccessible). Can you provide a link? I don't know where to search. Should we open a src.python.org domain? I don't have a recent optimized clone to check the size of, yet. What is the size of non-optimized clone then? I know that a clone of such relatively small project as MoinMoin is about 250Mb. ISTM that Python repository may take more than 1GB, and that's not acceptable IMHO. BTW, what do you mean by optimization - I hope not stripping the history? [1] http://moinmo.in/MoinMoin2.0/InfrastructurePlans -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Actual Mercurial Roadmap for February (Was: svn outage on Friday)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 2/22/2011 9:41 AM, anatoly techtonik wrote: On Fri, Feb 18, 2011 at 4:00 PM, Dirkjan Ochtman dirk...@ochtman.nl wrote: On Fri, Feb 18, 2011 at 14:41, anatoly techtonik techto...@gmail.com wrote: Do you have a public list of stuff to be done (i.e. Roadmap)? BTW, what is the size of Mercurial clone for Python repository? There is a TODO file in the pymigr repo (though I think that is currently inaccessible). Can you provide a link? I don't know where to search. Should we open a src.python.org domain? I don't have a recent optimized clone to check the size of, yet. What is the size of non-optimized clone then? I know that a clone of such relatively small project as MoinMoin is about 250Mb. ISTM that Python repository may take more than 1GB, and that's not acceptable IMHO. BTW, what do you mean by optimization - I hope not stripping the history? Mercurial repositories are sensitive to the order that data is inserted into them. So re-ordering the topological insert can dramatically improve compression. The quick overview is that in a given file's history, each diff is computed to the previous text in that file. So if you have a history like: foo | \ foo baz bar foo | / baz foo bar This can be stored as either: foo +bar -bar +baz +bar This matters more if you have a long divergent history for a while: A |\ B C | | D E | | F G : : X Y |/ Z In this case, you could end up with contents that look like: A +B +D +F +X -BDFX+C +E +G +Y +ABDFXZ Or you could have the history 'interleaved': A +B -B+C -C+BD -BD+CE -BDF+CEG -... There are tools that take a history file, and rewrite it to be more compact. I don't know much more than that myself. But especially with something like an svn conversion which probably works on each revision at a time, where people are committing to different branches concurrently, I would imagine the conversion could easily end up in the pessimistic case. John =:- -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (Cygwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk1j6qoACgkQJdeBCYSNAAPzPgCdEOJsHf4Xf4lZH+jHX42FQb8J sQoAn3JuCmDcsyv0JZpXtbVJoGewA+7t =M8DI -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] %-formatting depracation
Greetings! According to these release notes in Python 3.0, %-formatting will be going away. http://docs.python.org/release/3.0.1/whatsnew/3.0.html#pep-3101-a-new-approach-to-string-formatting However, I was unable to find any further evidence of actual deprecation in 3.1 or 3.2... does anybody know the status of this? ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Mon, Feb 21, 2011 at 15:34, David Claridge da...@daave.com wrote: Hi, I was wondering if there is some reason why C API functions like PyObject_CallMethod[1] and PySys_GetObject[2] take char* arguments rather than const char*s? If there is some reason these methods will modify their string arguments, it would be nice if it was documented, because at the moment it's unclear whether it is safe to cast a string literal to char* in these cases. For instance it seems reasonable that I should be able to call PySys_GetObject(path) without having to deal with a 'deprecated conversion from string constant to ‘char*’' error. Probably because (a) the person who first wrote them used char* instead of const char*, and (b) it gives us API flexibility by not promising to not alter the char array at some point in the future. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] %-formatting depracation
On Tue, Feb 22, 2011 at 10:43, Ethan Furman et...@stoneleaf.us wrote: Greetings! According to these release notes in Python 3.0, %-formatting will be going away. http://docs.python.org/release/3.0.1/whatsnew/3.0.html#pep-3101-a-new-approach-to-string-formatting However, I was unable to find any further evidence of actual deprecation in 3.1 or 3.2... does anybody know the status of this? There isn't one. =) The very long term view is for %-formatting to go away, but that's as far as the thinking has gone. There are currently no plans to introduce any deprecation warning, and I highly doubt we will even remove the feature in Python 3, giving you probably at least another decade of use at our current major version release schedule. =) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Tue, Feb 22, 2011 at 10:55 AM, Brett Cannon br...@python.org wrote: On Mon, Feb 21, 2011 at 15:34, David Claridge da...@daave.com wrote: Hi, I was wondering if there is some reason why C API functions like PyObject_CallMethod[1] and PySys_GetObject[2] take char* arguments rather than const char*s? If there is some reason these methods will modify their string arguments, it would be nice if it was documented, because at the moment it's unclear whether it is safe to cast a string literal to char* in these cases. For instance it seems reasonable that I should be able to call PySys_GetObject(path) without having to deal with a 'deprecated conversion from string constant to ‘char*’' error. Probably because (a) the person who first wrote them used char* instead of const char*, and (b) it gives us API flexibility by not promising to not alter the char array at some point in the future. I'm sorry, but (b) does not make sense. These APIs typically get passed string literals and modifying the argument would be a bad idea. FWIW, in Python 3.2, PySys_GetObject() is actually using const char *. I don't see a problem with changing PyObject_CallMethod() and its variations as well. Though I do not get that warning -- which compiler and version issues it? Is it a C or a C++ compiler? (If it is a C++ compiler I'm tempted to reject the complaint -- CPython is not written in C++ even though at various points we try to keep the headers usable with C++ as well as with C.) -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] %-formatting depracation
On 02/22/2011 01:43 PM, Ethan Furman wrote: Greetings! According to these release notes in Python 3.0, %-formatting will be going away. http://docs.python.org/release/3.0.1/whatsnew/3.0.html#pep-3101-a-new-approach-to-string-formatting However, I was unable to find any further evidence of actual deprecation in 3.1 or 3.2... does anybody know the status of this? The last thread on this I have a reference to was in September 2009. I think was basically the outcome: http://mail.python.org/pipermail/python-dev/2009-September/092399.html So there will be no deprecation, just a gradual shift to PEP 3101 formatting. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On 02/22/2011 01:55 PM, Brett Cannon wrote: On Mon, Feb 21, 2011 at 15:34, David Claridge da...@daave.com mailto:da...@daave.com wrote: Hi, I was wondering if there is some reason why C API functions like PyObject_CallMethod[1] and PySys_GetObject[2] take char* arguments rather than const char*s? If there is some reason these methods will modify their string arguments, it would be nice if it was documented, because at the moment it's unclear whether it is safe to cast a string literal to char* in these cases. For instance it seems reasonable that I should be able to call PySys_GetObject(path) without having to deal with a 'deprecated conversion from string constant to ‘char*’' error. Probably because (a) the person who first wrote them used char* instead of const char*, and (b) it gives us API flexibility by not promising to not alter the char array at some point in the future. Also changing it now would be a giant hassle, leading to so-called const poisoning where many, many APIs need to be changed before everything would again work. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Tue, 22 Feb 2011 11:06:41 -0800 Guido van Rossum gu...@python.org wrote: Probably because (a) the person who first wrote them used char* instead of const char*, and (b) it gives us API flexibility by not promising to not alter the char array at some point in the future. I'm sorry, but (b) does not make sense. These APIs typically get passed string literals and modifying the argument would be a bad idea. +1. If we started mutating strings passed to such APIs, it would certainly break a lot of third-party code. FWIW, in Python 3.2, PySys_GetObject() is actually using const char *. I don't see a problem with changing PyObject_CallMethod() and its variations as well. Though I do not get that warning -- which compiler and version issues it? Is it a C or a C++ compiler? Well, which warning are you talking about? I don't think changing a char * to a const char * specification would be harmful, since it makes the API more tolerant: with the latter, you can pass it both const and non-const strings. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Tue, Feb 22, 2011 at 2:09 PM, Eric Smith e...@trueblade.com wrote: Also changing it now would be a giant hassle, leading to so-called const poisoning where many, many APIs need to be changed before everything would again work. The poisoning will not break any users of the API, though, since they can pass const and non-const pointers. Internally Python would have to go through and add const keywords as appropriate when passing strings around. IMO it's worth it to not cause this warning for users. Reid ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
Reid Kleckner, 22.02.2011 21:21: On Tue, Feb 22, 2011 at 2:09 PM, Eric Smith wrote: Also changing it now would be a giant hassle, leading to so-called const poisoning where many, many APIs need to be changed before everything would again work. The poisoning will not break any users of the API, though, since they can pass const and non-const pointers. Internally Python would have to go through and add const keywords as appropriate when passing strings around. IMO it's worth it to not cause this warning for users. The problem is that Python's C-API functions are used both internally and externally, so changes like this can easily impact other public API functions because the function being changed uses them. I think that's what const poisoning refers to here. Stefan ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Tue, 22 Feb 2011 21:48:51 +0100 Stefan Behnel stefan...@behnel.de wrote: Reid Kleckner, 22.02.2011 21:21: On Tue, Feb 22, 2011 at 2:09 PM, Eric Smith wrote: Also changing it now would be a giant hassle, leading to so-called const poisoning where many, many APIs need to be changed before everything would again work. The poisoning will not break any users of the API, though, since they can pass const and non-const pointers. Internally Python would have to go through and add const keywords as appropriate when passing strings around. IMO it's worth it to not cause this warning for users. The problem is that Python's C-API functions are used both internally and externally, so changes like this can easily impact other public API functions because the function being changed uses them. How so? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On 2011-02-22, at 21:55 , Antoine Pitrou wrote: On Tue, 22 Feb 2011 21:48:51 +0100 Stefan Behnel stefan...@behnel.de wrote: Reid Kleckner, 22.02.2011 21:21: On Tue, Feb 22, 2011 at 2:09 PM, Eric Smith wrote: Also changing it now would be a giant hassle, leading to so-called const poisoning where many, many APIs need to be changed before everything would again work. The poisoning will not break any users of the API, though, since they can pass const and non-const pointers. Internally Python would have to go through and add const keywords as appropriate when passing strings around. IMO it's worth it to not cause this warning for users. The problem is that Python's C-API functions are used both internally and externally, so changes like this can easily impact other public API functions because the function being changed uses them. How so? If the parameters are passed from the newly const'ed function to an other public-API function, that one will have to be const'ed as well (or the const will have to be cast away which generally isn't considered good style and may lead to UBs), which may cascade into yet an other public-API function, the end result being that numerous functions would have to be const'ed: Adding const qualification may propagate through a program; as you add const qualifiers, still more become necessary. This phenomenon is sometimes called const-poisoning. Const-poisoning can frequently lead to violations of recommendation EXP05-C. Do not cast away a const qualification. While const qualification is a good idea, the costs may outweigh the value in the remediation of existing code. https://www.securecoding.cert.org/confluence/display/seccode/STR05-C.+Use+pointers+to+const+when+referring+to+string+literals ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Tue, Feb 22, 2011 at 1:17 PM, Xavier Morel python-...@masklinn.net wrote: On 2011-02-22, at 21:55 , Antoine Pitrou wrote: On Tue, 22 Feb 2011 21:48:51 +0100 Stefan Behnel stefan...@behnel.de wrote: Reid Kleckner, 22.02.2011 21:21: On Tue, Feb 22, 2011 at 2:09 PM, Eric Smith wrote: Also changing it now would be a giant hassle, leading to so-called const poisoning where many, many APIs need to be changed before everything would again work. The poisoning will not break any users of the API, though, since they can pass const and non-const pointers. Internally Python would have to go through and add const keywords as appropriate when passing strings around. IMO it's worth it to not cause this warning for users. The problem is that Python's C-API functions are used both internally and externally, so changes like this can easily impact other public API functions because the function being changed uses them. How so? If the parameters are passed from the newly const'ed function to an other public-API function, that one will have to be const'ed as well (or the const will have to be cast away which generally isn't considered good style and may lead to UBs), which may cascade into yet an other public-API function, the end result being that numerous functions would have to be const'ed: I tried this in my codebase. I found no need for further const propagation in this particular case. Adding const qualification may propagate through a program; as you add const qualifiers, still more become necessary. This phenomenon is sometimes called const-poisoning. Const-poisoning can frequently lead to violations of recommendation EXP05-C. Do not cast away a const qualification. While const qualification is a good idea, the costs may outweigh the value in the remediation of existing code. https://www.securecoding.cert.org/confluence/display/seccode/STR05-C.+Use+pointers+to+const+when+referring+to+string+literals ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] %-formatting depracation
On Wed, Feb 23, 2011 at 4:51 AM, Brett Cannon br...@python.org wrote: The very long term view is for %-formatting to go away, but that's as far as the thinking has gone. There are currently no plans to introduce any deprecation warning, and I highly doubt we will even remove the feature in Python 3, giving you probably at least another decade of use at our current major version release schedule. =) This. Without a systematic way to detect and convert %-style to {}-style formatting, enforcing the switch in the 3.x series just isn't practical. Heck, we still haven't figured out how to convert a lot of higher level APIs to the new scheme in a backwards compatible way (that's why modules like logging still default to %-style, with {}-style only available via options and wrapper objects). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
Also changing it now would be a giant hassle, leading to so-called const poisoning where many, many APIs need to be changed before everything would again work. We have been adding const to many places over the years. I think the specific case was just missed (i.e. nobody cared about adding const there). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] %-formatting depracation
On Wed, 23 Feb 2011 07:52:23 +1000 Nick Coghlan ncogh...@gmail.com wrote: On Wed, Feb 23, 2011 at 4:51 AM, Brett Cannon br...@python.org wrote: The very long term view is for %-formatting to go away, but that's as far as the thinking has gone. There are currently no plans to introduce any deprecation warning, and I highly doubt we will even remove the feature in Python 3, giving you probably at least another decade of use at our current major version release schedule. =) This. Without a systematic way to detect and convert %-style to {}-style formatting, enforcing the switch in the 3.x series just isn't practical. Heck, we still haven't figured out how to convert a lot of higher level APIs to the new scheme in a backwards compatible way (that's why modules like logging still default to %-style, with {}-style only available via options and wrapper objects). I think there are many people still finding %-style more practical for simple uses, so this might be a case of practicality beats purity over there should be one obvious way to do it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
Though I do not get that warning -- which compiler and version issues it? Is it a C or a C++ compiler? Well, which warning are you talking about? I think Guido assumed that the OP was getting actual complaints from some actual compiler - else he wouldn't have asked the question. However, he didn't actually say he got compile issues. If you compile #include Python.h int main() { const char* s = stdin; PyObject_CallMethod(0, s, s); } with a g++, you get a.cc: In function ‘int main()’: a.cc:6: error: invalid conversion from ‘const char*’ to ‘char*’ a.cc:6: error: initializing argument 2 of ‘PyObject* PyObject_CallMethod(PyObject*, char*, char*, ...)’ a.cc:6: error: invalid conversion from ‘const char*’ to ‘char*’ a.cc:6: error: initializing argument 3 of ‘PyObject* PyObject_CallMethod(PyObject*, char*, char*, ...)’ If you compile #include Python.h int main() { PyObject_CallMethod(0, stdin, stdin); } you get a.cc: In function ‘int main()’: a.cc:5: warning: deprecated conversion from string constant to ‘char*’ a.cc:5: warning: deprecated conversion from string constant to ‘char*’ Since most people likely use string literals, and since g++ only started warning about the deprecated conversion only recently, most people probably haven't run into the issue. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] %-formatting depracation
The very long term view is for %-formatting to go away Add to that that this view isn't universally shared among contributors. Many of us would rather see % formatting stay indefinitely. I regularly use it for new code. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] %-formatting depracation
On Wed, Feb 23, 2011 at 8:03 AM, Antoine Pitrou solip...@pitrou.net wrote: I think there are many people still finding %-style more practical for simple uses, A lot of the sting went out of that objection when field autonumbering was added to new-style formatting ('%s' % (obj,) vs '{}'.format(obj) as the minimal case that correctly handles tuples). The addition of format_map() reduces it even further. so this might be a case of practicality beats purity over there should be one obvious way to do it. I agree with that part though, just for historical reasons rather than seeing much in the way in the way of inherent advantages with %-style formatting. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fwd: deep question re dict as formatting input
One of the students on an introductory Python 3 class asks a very good question about string formatting. This could be because the course materials are misleading, so I would like to understand. It would appear from tests that {0[X]}.format(...) first tries to convert the string X to in integer. If it succeeds then __getitem__() is called with the integer as an argument, otherwise it is called with the string itself as an argument. Is this correct? The documentation at http://docs.python.org/library/string.html#formatspec is silent on whether strings were ever intended to be used as subscripts. Does this seem sensible? Was it considered during design? Should I alter the materials so that only integer subscripts are used? regards Steve Begin forwarded message: From: kirby urner kirby.ur...@gmail.com Date: February 22, 2011 2:31:08 PM PST To: Steve Holden st...@holdenweb.com Subject: deep question re dict as formatting input d {'Steve': 'Holden', 'Tim': 'Peters', 'Guido': 'van Rossum', '1': 'string', 1: 'integer'} {0[Guido]} is cool.format(d) 'van Rossum is cool' {0[1]} is cool.format(d) 'integer is cool' {0['1']} is cool.format(d) Traceback (most recent call last): File pyshell#19, line 1, in module {0['1']} is cool.format(d) KeyError: '1' Student question: Good morning! Question on .format(), interactive session follows: -- d = {Steve: Holden, ... Guido: van Rossum, ... Tim: Peters, ... 1: string, ... 1: integer} -- d {'Steve': 'Holden', 'Tim': 'Peters', '1': 'string', 1: 'integer', 'Guido': 'van Rossum'} -- d[1] 'integer' -- d['1'] 'string' -- {dct[1]}.format(dct=d) 'integer' -- {dct[Guido]}.format(dct=d) 'van Rossum' -- {dct['1']}.format(dct=d) Traceback (most recent call last): File console, line 1, in module KeyError: '1' Question: If {dct[Guido]} treats Guido as str, why doesn't {dct[1]} treate 1 as str? Feels like an automatic conversion from str to int. Furthermore, how does one access the key '1' in a format statement? ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Mon, Feb 21, 2011 at 6:34 PM, David Claridge da...@daave.com wrote: .. I was wondering if there is some reason why C API functions like PyObject_CallMethod[1] and PySys_GetObject[2] take char* arguments rather than const char*s? The later is addressed by issue 1699259 http://bugs.python.org/issue1699259. It looks like const was added in 3.2, but deemed not important enough to backport to 2.7. The issue is still open, so interested parties can argue for backport there. The former (PyObject_CallMethod) is similarly subject of an open issue in a commit review stage: http://bugs.python.org/issue9369. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
If you compile #include Python.h int main() { PyObject_CallMethod(0, stdin, stdin); } you get a.cc: In function ‘int main()’: a.cc:5: warning: deprecated conversion from string constant to ‘char*’ a.cc:5: warning: deprecated conversion from string constant to ‘char*’ Since most people likely use string literals, and since g++ only started warning about the deprecated conversion only recently, most people probably haven't run into the issue. Regards, Martin This is precisely the warning I've been getting, I've been embedding Python 2.6 in an application built using g++ 4.4. The later is addressed by issue 1699259 http://bugs.python.org/issue1699259. It looks like const was added in 3.2, but deemed not important enough to backport to 2.7. The issue is still open, so interested parties can argue for backport there. The former (PyObject_CallMethod) is similarly subject of an open issue in a commit review stage: http://bugs.python.org/issue9369. Thanks, I'll add my 2c worth on those issues. Even if it is eventually decided not to backport those patches to 2.7, it would be nice if the documentation could be updated to indicate that strings passed to those functions won't be modified, so that API users like myself can feel a little safer when passing literals in, without having to trawl through the interpreter source to see if the arguments are modified. Thanks, -- David Claridge http://daave.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: deep question re dict as formatting input
On Tue, Feb 22, 2011 at 6:01 PM, Steve Holden st...@holdenweb.com wrote: ... It would appear from tests that {0[X]}.format(...) first tries to convert the string X to in integer. If it succeeds then __getitem__() is called with the integer as an argument, otherwise it is called with the string itself as an argument. Is this correct? This is addressed in the PEP 3101: The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. http://www.python.org/dev/peps/pep-3101/ If current implementation does something more involved, I would say it is a bug. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Tue, 22 Feb 2011 18:08:01 -0500 Alexander Belopolsky alexander.belopol...@gmail.com wrote: On Mon, Feb 21, 2011 at 6:34 PM, David Claridge da...@daave.com wrote: .. I was wondering if there is some reason why C API functions like PyObject_CallMethod[1] and PySys_GetObject[2] take char* arguments rather than const char*s? The later is addressed by issue 1699259 http://bugs.python.org/issue1699259. It looks like const was added in 3.2, but deemed not important enough to backport to 2.7. The issue is still open, so interested parties can argue for backport there. I don't think it's a good idea to backport visible API changes. (someone successfully compiling on 2.7.N could then have users complaining that compilation fails on 2.7.N-1). Moreover, it doesn't really fix a bug. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: deep question re dict as formatting input
Quoting PEP 3101: An example of the 'getitem' syntax: My name is {0[name]}.format(dict(name='Fred')) It should be noted that the use of 'getitem' within a format string is much more limited than its conventional usage. In the above example, the string 'name' really is the literal string 'name', not a variable named 'name'. The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. On 2/22/2011 6:01 PM, Steve Holden wrote: One of the students on an introductory Python 3 class asks a very good question about string formatting. This could be because the course materials are misleading, so I would like to understand. It would appear from tests that {0[X]}.format(...) first tries to convert the string X to in integer. If it succeeds then __getitem__() is called with the integer as an argument, otherwise it is called with the string itself as an argument. Is this correct? The documentation at http://docs.python.org/library/string.html#formatspec is silent on whether strings were ever intended to be used as subscripts. Does this seem sensible? Was it considered during design? Should I alter the materials so that only integer subscripts are used? regards Steve Begin forwarded message: *From: *kirby urner kirby.ur...@gmail.com mailto:kirby.ur...@gmail.com *Date: *February 22, 2011 2:31:08 PM PST *To: *Steve Holden st...@holdenweb.com mailto:st...@holdenweb.com *Subject: **deep question re dict as formatting input* d {'Steve': 'Holden', 'Tim': 'Peters', 'Guido': 'van Rossum', '1': 'string', 1: 'integer'} {0[Guido]} is cool.format(d) 'van Rossum is cool' {0[1]} is cool.format(d) 'integer is cool' {0['1']} is cool.format(d) Traceback (most recent call last): File pyshell#19, line 1, in module {0['1']} is cool.format(d) KeyError: '1' Student question: Good morning! Question on .format(), interactive session follows: -- d = {Steve: Holden, ... Guido: van Rossum, ... Tim: Peters, ... 1: string, ... 1: integer} -- d {'Steve': 'Holden', 'Tim': 'Peters', '1': 'string', 1: 'integer', 'Guido': 'van Rossum'} -- d[1] 'integer' -- d['1'] 'string' -- {dct[1]}.format(dct=d) 'integer' -- {dct[Guido]}.format(dct=d) 'van Rossum' -- {dct['1']}.format(dct=d) Traceback (most recent call last): File console, line 1, in module KeyError: '1' Question: If {dct[Guido]} treats Guido as str, why doesn't {dct[1]} treate 1 as str? Feels like an automatic conversion from str to int. Furthermore, how does one access the key '1' in a format statement? ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: deep question re dict as formatting input
On Feb 22, 2011, at 3:08 PM, Eric Smith wrote: Quoting PEP 3101: An example of the 'getitem' syntax: My name is {0[name]}.format(dict(name='Fred')) It should be noted that the use of 'getitem' within a format string is much more limited than its conventional usage. In the above example, the string 'name' really is the literal string 'name', not a variable named 'name'. The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. That's not strictly true: d = {Steve:Holden, Guido:van Rossum, 21.2:float} d[21.1] Traceback (most recent call last): File stdin, line 1, in module KeyError: 21.1 d[21.2] 'float' {0[21.2]}.format(d) Traceback (most recent call last): File stdin, line 1, in module KeyError: '21.2' But I take your point, and should have thought to read the PEP. Thanks! Kirby: Please apologize to Ethan. I can't remember being aware of the PEP 3101 specification quoted by Eric above. We will probably need to modify the course materials to take this wrinkle into account (at least by demonstrating that we are aware of it). regards Steve On 2/22/2011 6:01 PM, Steve Holden wrote: One of the students on an introductory Python 3 class asks a very good question about string formatting. This could be because the course materials are misleading, so I would like to understand. It would appear from tests that {0[X]}.format(...) first tries to convert the string X to in integer. If it succeeds then __getitem__() is called with the integer as an argument, otherwise it is called with the string itself as an argument. Is this correct? The documentation at http://docs.python.org/library/string.html#formatspec is silent on whether strings were ever intended to be used as subscripts. Does this seem sensible? Was it considered during design? Should I alter the materials so that only integer subscripts are used? regards Steve Begin forwarded message: *From: *kirby urner kirby.ur...@gmail.com mailto:kirby.ur...@gmail.com *Date: *February 22, 2011 2:31:08 PM PST *To: *Steve Holden st...@holdenweb.com mailto:st...@holdenweb.com *Subject: **deep question re dict as formatting input* d {'Steve': 'Holden', 'Tim': 'Peters', 'Guido': 'van Rossum', '1': 'string', 1: 'integer'} {0[Guido]} is cool.format(d) 'van Rossum is cool' {0[1]} is cool.format(d) 'integer is cool' {0['1']} is cool.format(d) Traceback (most recent call last): File pyshell#19, line 1, in module {0['1']} is cool.format(d) KeyError: '1' Student question: Good morning! Question on .format(), interactive session follows: -- d = {Steve: Holden, ... Guido: van Rossum, ... Tim: Peters, ... 1: string, ... 1: integer} -- d {'Steve': 'Holden', 'Tim': 'Peters', '1': 'string', 1: 'integer', 'Guido': 'van Rossum'} -- d[1] 'integer' -- d['1'] 'string' -- {dct[1]}.format(dct=d) 'integer' -- {dct[Guido]}.format(dct=d) 'van Rossum' -- {dct['1']}.format(dct=d) Traceback (most recent call last): File console, line 1, in module KeyError: '1' Question: If {dct[Guido]} treats Guido as str, why doesn't {dct[1]} treate 1 as str? Feels like an automatic conversion from str to int. Furthermore, how does one access the key '1' in a format statement? ~Ethan~ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Tue, Feb 22, 2011 at 6:18 PM, Antoine Pitrou solip...@pitrou.net wrote: .. I don't think it's a good idea to backport visible API changes. (someone successfully compiling on 2.7.N could then have users complaining that compilation fails on 2.7.N-1). Moreover, it doesn't really fix a bug. That was the argument that I voiced http://bugs.python.org/issue1699259#msg119032, but I cannot really think of the code that would be broken by changing the type of an API function *argument* from char * to const char *. Given that 2.7 is an odd-ball with extended maintenance period, I am not sure it doesn't really fix a bug argument is dispositive. I am still -0 on backport, however. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: deep question re dict as formatting input
On Tue, Feb 22, 2011 at 6:01 PM, Steve Holden st...@holdenweb.com wrote: ... It would appear from tests that {0[X]}.format(...) first tries to convert the string X to in integer. If it succeeds then __getitem__() is called with the integer as an argument, otherwise it is called with the string itself as an argument. Is this correct? This is addressed in the PEP 3101: The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. http://www.python.org/dev/peps/pep-3101/ To the other question : Furthermore, how does one access the key '1' in a format statement? ~Ethan~ I think, parsing rule already helps to understand the problem with the key like '1'. The PEP also explicitly states that: Because keys are not quote-delimited, it is not possible to specify arbitrary dictionary keys (e.g., the strings 10 or :-]) from within a format string. -- Senthil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] %-formatting depracation
On Tue, 2011-02-22 at 23:03 +0100, Antoine Pitrou wrote: On Wed, 23 Feb 2011 07:52:23 +1000 Nick Coghlan ncogh...@gmail.com wrote: On Wed, Feb 23, 2011 at 4:51 AM, Brett Cannon br...@python.org wrote: The very long term view is for %-formatting to go away, but that's as far as the thinking has gone. There are currently no plans to introduce any deprecation warning, and I highly doubt we will even remove the feature in Python 3, giving you probably at least another decade of use at our current major version release schedule. =) This. Without a systematic way to detect and convert %-style to {}-style formatting, enforcing the switch in the 3.x series just isn't practical. Heck, we still haven't figured out how to convert a lot of higher level APIs to the new scheme in a backwards compatible way (that's why modules like logging still default to %-style, with {}-style only available via options and wrapper objects). I think there are many people still finding %-style more practical for simple uses, so this might be a case of practicality beats purity over there should be one obvious way to do it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/anikom15%40gmail.com I tried to use the format method, but I found the syntax was getting on my nerves and have since gone back to %-formatting. So I don't think it's going away anytime soon. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
Le mardi 22 février 2011 à 18:30 -0500, Alexander Belopolsky a écrit : On Tue, Feb 22, 2011 at 6:18 PM, Antoine Pitrou solip...@pitrou.net wrote: .. I don't think it's a good idea to backport visible API changes. (someone successfully compiling on 2.7.N could then have users complaining that compilation fails on 2.7.N-1). Moreover, it doesn't really fix a bug. That was the argument that I voiced http://bugs.python.org/issue1699259#msg119032, but I cannot really think of the code that would be broken by changing the type of an API function *argument* from char * to const char *. To quote the message above: someone successfully compiling on 2.7.N could then have users complaining that compilation fails on 2.7.N-1. (note: *N-1*) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
Even if it is eventually decided not to backport those patches to 2.7, it would be nice if the documentation could be updated to indicate that strings passed to those functions won't be modified, so that API users like myself can feel a little safer when passing literals in, without having to trawl through the interpreter source to see if the arguments are modified. Can you provide the proper patches? In writing them, you can assume that Python never modifies parameters, except when it is already documented that it does. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
On Tue, Feb 22, 2011 at 6:36 PM, Antoine Pitrou solip...@pitrou.net wrote: .. To quote the message above: someone successfully compiling on 2.7.N could then have users complaining that compilation fails on 2.7.N-1. (note: *N-1*) I missed that. Yes, this is a valid concern. I change my vote from -0 to -1. :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Const-correctness in C-API Object Protocol
I don't think it's a good idea to backport visible API changes. (someone successfully compiling on 2.7.N could then have users complaining that compilation fails on 2.7.N-1). +1. If it was a bug fix (which it isn't), having this kind of breakage would be fine, of course (i.e. code relying on the bug fix will reasonably break when used with buggy versions). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: deep question re dict as formatting input
On 2/22/2011 6:32 PM, Senthil Kumaran wrote: On Tue, Feb 22, 2011 at 6:01 PM, Steve Holdenst...@holdenweb.com wrote: ... It would appear from tests that {0[X]}.format(...) first tries to convert the string X to in integer. If it succeeds then __getitem__() is called with the integer as an argument, otherwise it is called with the string itself as an argument. Is this correct? This is addressed in the PEP 3101: The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. http://www.python.org/dev/peps/pep-3101/ To the other question : Furthermore, how does one access the key '1' in a format statement? ~Ethan~ I think, parsing rule already helps to understand the problem with the key like '1'. The PEP also explicitly states that: Because keys are not quote-delimited, it is not possible to specify arbitrary dictionary keys (e.g., the strings 10 or :-]) from within a format string. Is this all specific in the lib docs? If not, it should be. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: deep question re dict as formatting input
On 2/22/2011 6:28 PM, Steve Holden wrote: On Feb 22, 2011, at 3:08 PM, Eric Smith wrote: Quoting PEP 3101: An example of the 'getitem' syntax: My name is {0[name]}.format(dict(name='Fred')) It should be noted that the use of 'getitem' within a format string is much more limited than its conventional usage. In the above example, the string 'name' really is the literal string 'name', not a variable named 'name'. The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. That's not strictly true: d = {Steve:Holden, Guido:van Rossum, 21.2:float} d[21.1] Traceback (most recent call last): File stdin, line 1, inmodule KeyError: 21.1 d[21.2] 'float' {0[21.2]}.format(d) Traceback (most recent call last): File stdin, line 1, inmodule KeyError: '21.2' You are correct, I didn't exactly implement the PEP on this point, probably as a shortcut. I think there's an issue somewhere that discusses this, but I can't find it. The CPython implementation is really using If every character is a digit, then it is treated as an integer, otherwise it is used as a string. See find_name_split in Objects/stringlib/string_format.h, in particular the call to get_integer() and the interpretation of the result. Eric. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r88505 - python/branches/py3k/Lib/ftplib.py
You should maybe backport this fix to Python 3.2. Le mardi 22 février 2011 à 20:24 +0100, giampaolo.rodola a écrit : Author: giampaolo.rodola Date: Tue Feb 22 20:24:33 2011 New Revision: 88505 Log: In FTP.close() method, make sure to also close the socket object, not only the file. Modified: python/branches/py3k/Lib/ftplib.py Modified: python/branches/py3k/Lib/ftplib.py == --- python/branches/py3k/Lib/ftplib.py(original) +++ python/branches/py3k/Lib/ftplib.pyTue Feb 22 20:24:33 2011 @@ -589,11 +589,11 @@ def close(self): '''Close the connection without assuming anything about it.''' -if self.file: +if self.file is not None: self.file.close() +if self.sock is not None: self.sock.close() -self.file = self.sock = None - +self.file = self.sock = None ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn outage on Friday
On Tue, Feb 15, 2011 at 6:30 PM, Martin v. Löwis mar...@v.loewis.de wrote: I'm going to perform a Debian upgrade of svn.python.org on Friday, between 9:00 UTC and 11:00 UTC. I'll be disabling write access during that time. The outage shouldn't be longer than an hour. This may have caused some issues with the SVN viewer - I'm currently getting code with no syntax highlighting and all leading whitespace stripped when attempting to look at file contents. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn outage on Friday
On Feb 22, 2011, at 5:43 PM, Nick Coghlan wrote: On Tue, Feb 15, 2011 at 6:30 PM, Martin v. Löwis mar...@v.loewis.de wrote: I'm going to perform a Debian upgrade of svn.python.org on Friday, between 9:00 UTC and 11:00 UTC. I'll be disabling write access during that time. The outage shouldn't be longer than an hour. This may have caused some issues with the SVN viewer - I'm currently getting code with no syntax highlighting and all leading whitespace stripped when attempting to look at file contents. I saw the same thing yesterday when viewing with the Chrome browser but it looked fine on Firefox. Later, after restarting the browsers the problem disappeared. I wonder if there was some sort of caching issue. Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r88503 - in python/branches/py3k: Lib/lib2to3/__main__.py Tools/scripts/2to3
2011/2/22 brett.cannon python-check...@python.org: Author: brett.cannon Date: Tue Feb 22 20:12:43 2011 New Revision: 88503 Log: Add lib2to3.__main__ to make it easier for debugging purposes to run 2to3. Please revert this and do it in the sandbox. Added: python/branches/py3k/Lib/lib2to3/__main__.py Modified: python/branches/py3k/Tools/scripts/2to3 Added: python/branches/py3k/Lib/lib2to3/__main__.py == --- (empty file) +++ python/branches/py3k/Lib/lib2to3/__main__.py Tue Feb 22 20:12:43 2011 @@ -0,0 +1,4 @@ +import sys +from .main import main + +sys.exit(main(lib2to3.fixes)) Modified: python/branches/py3k/Tools/scripts/2to3 == --- python/branches/py3k/Tools/scripts/2to3 (original) +++ python/branches/py3k/Tools/scripts/2to3 Tue Feb 22 20:12:43 2011 @@ -1,5 +1,4 @@ #!/usr/bin/env python -import sys -from lib2to3.main import main +import runpy -sys.exit(main(lib2to3.fixes)) +runpy.run_module('lib2to3', run_name='__main__', alter_sys=True) ___ Python-checkins mailing list python-check...@python.org http://mail.python.org/mailman/listinfo/python-checkins -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
Jesus Cea writes: PPS: If there is consensus that this is a real bug, I would create an issue in the tracker and try to get a minimal testcase. All bugs are issues, but not all issues are bugs. Please don't wait for consensus or even a second opinion to file the issue. It's reasonable for a new Python user to ask whether something is a bug or not, but if somebody with your experience and contribution level to Python doesn't understand something, at the very least we have to suspect a doc bug. So please file the issue to ensure that something will be done to address the issue, and even if it turns out to be some misunderstanding that only you are ever likely to makewink, at least the issue will remain as documentation if somebody else has the same misunderstanding. OTOH, the testcase might require a lot of effort on your part. Of course it's reasonable for you to check whether it's a simple misunderstanding before exerting that effort. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
On Tue, Feb 22, 2011 at 9:31 PM, Stephen J. Turnbull step...@xemacs.org wrote: Jesus Cea writes: PPS: If there is consensus that this is a real bug, I would create an issue in the tracker and try to get a minimal testcase. All bugs are issues, but not all issues are bugs. Please don't wait for consensus or even a second opinion to file the issue. AFAICT, it was not mentioned in this thread, but the issue has been created on the tracker: http://bugs.python.org/issue11286 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 23/02/11 03:31, Stephen J. Turnbull wrote: Please don't wait for consensus or even a second opinion to file the issue. It's reasonable for a new Python user to ask whether something is a bug or not, but if somebody with your experience and contribution level to Python doesn't understand something, at the very least we have to suspect a doc bug. Every time I read a message from Antoine Pitrou, Brett Cannon, Nick Coghlan, Martin Löwis, Eric Smith, Steve Holden, Benjamin Peterson, Victor Stinner, Greog Brandl, Raymond Hettinger, Guido and so many others python-devs (not an exhaustive list, if you are not there, you probably should, sorry :), I feel I am faking my knowledge of Python :-). I am a pretender :). BTW, this project is my first real python 3 code (I promised to myself to move after 3.2 release, a year ago), for a real/big project, and I was thinking that maybe I was overlooking something obvious for any seasoned real python programmer. I overcame my fear of being seen as a fool last millenia, so I am not afraid of asking. Sometimes I even ask too much. OTOH, the testcase might require a lot of effort on your part. Of course it's reasonable for you to check whether it's a simple misunderstanding before exerting that effort. In fact, I invested *hours* trying to reduce my multimegabyte problematic pickle to 41 bytes, but at this time I was already convinced that I had hit an ugly and serious bug. Issue filed. It already has a patch. That was fast!. Now I can sit back waiting for 3.2.1 before touching my project again :). Mixed feelings about the waiting. I hope it is short. Life sucks sometimes :). Thanks $DEITY there are quite a few better python-devs than me :). - -- Jesus Cea Avion _/_/ _/_/_/_/_/_/ j...@jcea.es - http://www.jcea.es/ _/_/_/_/ _/_/_/_/ _/_/ jabber / xmpp:j...@jabber.org _/_/_/_/ _/_/_/_/_/ . _/_/ _/_/_/_/ _/_/ _/_/ Things are not so easy _/_/ _/_/_/_/ _/_/_/_/ _/_/ My name is Dump, Core Dump _/_/_/_/_/_/ _/_/ _/_/ El amor es poner tu felicidad en la felicidad de otro - Leibniz -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTWSOY5lgi5GaxT1NAQIqQAP/Wf/e2iN3FRb7istoGCpqCgjDv7UyCOWF RzOYMJWh0xhNL5ydZZ2YwtcQNEWrQS538zrr8piOqvV3ielOBCgSWArqChWaQTHU ZC3gdaw8N5VMr0AXGBMXwcflkLaQ7BrBtiQBizFL9KLYGDI9JG8+O1YjpjamUeQv iXEyGdqWUp4= =XwYt -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r88505 - python/branches/py3k/Lib/ftplib.py
I'll do. 2011/2/23 Victor Stinner victor.stin...@haypocalc.com: You should maybe backport this fix to Python 3.2. Le mardi 22 février 2011 à 20:24 +0100, giampaolo.rodola a écrit : Author: giampaolo.rodola Date: Tue Feb 22 20:24:33 2011 New Revision: 88505 Log: In FTP.close() method, make sure to also close the socket object, not only the file. Modified: python/branches/py3k/Lib/ftplib.py Modified: python/branches/py3k/Lib/ftplib.py == --- python/branches/py3k/Lib/ftplib.py (original) +++ python/branches/py3k/Lib/ftplib.py Tue Feb 22 20:24:33 2011 @@ -589,11 +589,11 @@ def close(self): '''Close the connection without assuming anything about it.''' - if self.file: + if self.file is not None: self.file.close() + if self.sock is not None: self.sock.close() - self.file = self.sock = None - + self.file = self.sock = None ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange error importing a Pickle from 2.7 to 3.2
Jesus Cea writes: Every time I read a message from [long, incompletewink list] and so many others python-devs (not an exhaustive list, if you are not there, you probably should, sorry :), I feel I am faking my knowledge of Python :-). I am a pretender :). Sure. I suspect even some of those *on* the list feel that way sometimes. That's what's so great about the list, the people who contribute! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] svn outage on Friday
Am 23.02.2011 02:43, schrieb Nick Coghlan: On Tue, Feb 15, 2011 at 6:30 PM, Martin v. Löwis mar...@v.loewis.de wrote: I'm going to perform a Debian upgrade of svn.python.org on Friday, between 9:00 UTC and 11:00 UTC. I'll be disabling write access during that time. The outage shouldn't be longer than an hour. This may have caused some issues with the SVN viewer - I'm currently getting code with no syntax highlighting and all leading whitespace stripped when attempting to look at file contents. The ViewVC configuration has significantly changed, so I originally tried to adjust the old configuration. I went now the other path of configuring starting with the new configuration template. I may have broken things in the process; if so, please let me know. Post the specific URL that gives troubles. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r88501 - python/branches/py3k/Lib/smtplib.py
You're sure this will not cause tedious conflicts with backports? Georg On 22.02.2011 16:56, giampaolo.rodola wrote: Author: giampaolo.rodola Date: Tue Feb 22 16:56:20 2011 New Revision: 88501 Log: smtlib.py PEP8 normalization via pep8.py script. Modified: python/branches/py3k/Lib/smtplib.py Modified: python/branches/py3k/Lib/smtplib.py == --- python/branches/py3k/Lib/smtplib.py (original) +++ python/branches/py3k/Lib/smtplib.py Tue Feb 22 16:56:20 2011 @@ -52,15 +52,15 @@ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] r88516 - in python/branches/py3k/Python: dynload_aix.c dynload_dl.c dynload_hpux.c dynload_next.c dynload_os2.c dynload_shlib.c dynload_win.c importdl.c
This commit introduced tabs, at least in dynload_dl.c. Georg On 23.02.2011 00:16, victor.stinner wrote: Author: victor.stinner Date: Wed Feb 23 00:16:19 2011 New Revision: 88516 Log: Issue #3080: Remove unused argument of _PyImport_GetDynLoadFunc() The first argument, fqname, was not used. Modified: python/branches/py3k/Python/dynload_aix.c python/branches/py3k/Python/dynload_dl.c python/branches/py3k/Python/dynload_hpux.c python/branches/py3k/Python/dynload_next.c python/branches/py3k/Python/dynload_os2.c python/branches/py3k/Python/dynload_shlib.c python/branches/py3k/Python/dynload_win.c python/branches/py3k/Python/importdl.c Modified: python/branches/py3k/Python/dynload_aix.c == --- python/branches/py3k/Python/dynload_aix.c (original) +++ python/branches/py3k/Python/dynload_aix.c Wed Feb 23 00:16:19 2011 @@ -154,7 +154,7 @@ } -dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname, +dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, const char *pathname, FILE *fp) { dl_funcptr p; Modified: python/branches/py3k/Python/dynload_dl.c == --- python/branches/py3k/Python/dynload_dl.c (original) +++ python/branches/py3k/Python/dynload_dl.c Wed Feb 23 00:16:19 2011 @@ -16,7 +16,7 @@ }; -dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname, +dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, const char *pathname, FILE *fp) { char funcname[258]; Modified: python/branches/py3k/Python/dynload_hpux.c == --- python/branches/py3k/Python/dynload_hpux.c(original) +++ python/branches/py3k/Python/dynload_hpux.cWed Feb 23 00:16:19 2011 @@ -19,7 +19,7 @@ {0, 0} }; -dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname, +dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, const char *pathname, FILE *fp) { dl_funcptr p; Modified: python/branches/py3k/Python/dynload_next.c == --- python/branches/py3k/Python/dynload_next.c(original) +++ python/branches/py3k/Python/dynload_next.cWed Feb 23 00:16:19 2011 @@ -31,8 +31,8 @@ #define LINKOPTIONS NSLINKMODULE_OPTION_BINDNOW| \ NSLINKMODULE_OPTION_RETURN_ON_ERROR|NSLINKMODULE_OPTION_PRIVATE #endif -dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname, -const char *pathname, FILE *fp) +dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, +const char *pathname, FILE *fp) { dl_funcptr p = NULL; char funcname[258]; Modified: python/branches/py3k/Python/dynload_os2.c == --- python/branches/py3k/Python/dynload_os2.c (original) +++ python/branches/py3k/Python/dynload_os2.c Wed Feb 23 00:16:19 2011 @@ -15,7 +15,7 @@ {0, 0} }; -dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname, +dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, const char *pathname, FILE *fp) { dl_funcptr p; Modified: python/branches/py3k/Python/dynload_shlib.c == --- python/branches/py3k/Python/dynload_shlib.c (original) +++ python/branches/py3k/Python/dynload_shlib.c Wed Feb 23 00:16:19 2011 @@ -75,7 +75,7 @@ static int nhandles = 0; -dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname, +dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, const char *pathname, FILE *fp) { dl_funcptr p; Modified: python/branches/py3k/Python/dynload_win.c == --- python/branches/py3k/Python/dynload_win.c (original) +++ python/branches/py3k/Python/dynload_win.c Wed Feb 23 00:16:19 2011 @@ -171,7 +171,7 @@ return NULL; } -dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname, +dl_funcptr _PyImport_GetDynLoadFunc(const char *shortname, const char *pathname, FILE *fp) { dl_funcptr p; Modified: python/branches/py3k/Python/importdl.c == --- python/branches/py3k/Python/importdl.c(original) +++ python/branches/py3k/Python/importdl.cWed