[Python-Dev] __str__ bug?
Is this a bug? If not, how do I override __str__ on a unicode derived class? class S(str): def __str__(self): return '__str__ overridden' class U(unicode): def __str__(self): return '__str__ overridden' def __unicode__(self): return u'__unicode__ overridden' s = S() u = U() print 's:', s print "str(s):", str(s) print 's substitued is "%s"\n' % s print 'u:', u print "str(u):", str(u) print 'u substitued is "%s"' % u - s: __str__ overridden str(s): __str__ overridden s substitued is "__str__ overridden" u: str(u): __str__ overridden u substitued is "" Results are identical for 2.4.2 and 2.5c2 (running under windows). Mike ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] RELEASED Python 2.3.6, release candidate 1
On behalf of the Python development team and the Python community, I'm announcing the release of Python 2.3.6 (release candidate 1). Python 2.3.6 is a security bug-fix release. While Python 2.5 is the latest version of Python, we're making this release for people who are still running Python 2.3. Unlike the recently released 2.4.4, this release only contains a small handful of security-related bugfixes. See the website for more. * Python 2.3.6 contains a fix for PSF-2006-001, a buffer overrun * in repr() of unicode strings in wide unicode (UCS-4) builds. * See http://www.python.org/news/security/PSF-2006-001/ for more. This is a **source only** release. The Windows and Mac binaries of 2.3.5 were built with UCS-2 unicode, and are therefore not vulnerable to the problem outlined in PSF-2006-001. The PCRE fix is for a long-deprecated module (you should use the 're' module instead) and the email fix can be obtained by downloading the standalone version of the email package. Most vendors who ship Python should have already released a patched version of 2.3.5 with the above fixes, this release is for people who need or want to build their own release, but don't want to mess around with patch or svn. Assuming no major problems crop up, a final release of Python 2.3.6 will follow in about a week's time. Python 2.3.6 will complete python.org's response to PSF-2006-001. If you're still on Python 2.2 for some reason and need to work with UCS-4 unicode strings, please obtain the patch from the PSF-2006-001 security advisory page. Python 2.4.4 and Python 2.5 have both already been released and contain the fix for this security problem. For more information on Python 2.3.6, including download links for source archives, release notes, and known issues, please see: http://www.python.org/2.3.6 Highlights of this new release include: - A fix for PSF-2006-001, a bug in repr() for unicode strings on UCS-4 (wide unicode) builds. - Two other, less critical, security fixes. Enjoy this release, Anthony Anthony Baxter [EMAIL PROTECTED] Python Release Manager (on behalf of the entire python-dev team) pgp4EtX0NCP6g.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
Larry Hastings schrieb: > Am I correct in understanding that changing the Python minor revision > number (2.5 -> 2.6) requires external modules to recompile? (It > certainly does on Windows.) There is an ongoing debate on that. The original intent was that you normally *shouldn't* have to recompile modules just because the Python version changes. Instead, you should do so when PYTHON_API_VERSION changes. Of course, such a change would also cause a change to PYTHON_API_VERSION. Then, even if PYTHON_API_VERSION changes, you aren't *required* to recompile your extension modules. Instead, you get a warning that the API version is different and *might* require recompilation: it does require recompilation if the extension module relies on some of the changed API. With this change, people not recompiling their extension modules would likely see Python crash rather quickly after seeing the warning about incompatible APIs. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
[EMAIL PROTECTED] schrieb: > >> Anyway, it was my intent to post the patch and see what happened. > >> Being a first-timer at this, and not having even read the core > >> development mailing lists for very long, I had no idea what to > >> expect. Though I genuinely didn't expect it to be this brusque. > > Martin> I could have told you :-) The "problem" really is that you are > Martin> suggesting a major, significant change to the implementation of > Martin> Python, and one that doesn't fix an obvious bug. > > Come on Martin. Give Larry a break. I'm seriously not complaining, I'm explaining. > Lots of changes have been accepted to > to the Python core which weren't obvious "bug fixes". Surely many new features have been implemented over time, but in many cases, they weren't really "big changes", in the sense that you could ignore them if you don't like them. This wouldn't be so in this case: as the string type is very fundamental, people feel a higher interest in its implementation. > In fact, I seem to > recall a sprint held recently in Reykjavik where the whole point was just to > make Python faster. That's true. I also recall there were serious complaints about the outcome of this sprint, and the changes to the struct module in particular. Still, the struct module is of lesser importance than the string type, so the concerns were smaller. > I believe that was exactly Larry's point in posting the > patch. The "one obvious way to do" concatenation and slicing for one of the > most heavily used types in python appears to be faster. That seems like a > win to me. Have you reviewed the patch and can vouch for its correctness, even in boundary cases? Have you tested it in a real application and found a real performance improvement? I have done neither, so I can't speak on the advantages of the patch. I didn't actually object to the inclusion of the patch, either. I was merely stating what I think the problems with "that kind of" patch are. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
On Mon, 23 Oct 2006 09:07:51 -0700, Josiah Carlson <[EMAIL PROTECTED]> wrote: > >"Paul Moore" <[EMAIL PROTECTED]> wrote: >> I had picked up on this comment, and I have to say that I had been a >> little surprised by the resistance to the change based on the "code >> would break" argument, when you had made such a thorough attempt to >> address this. Perhaps others had missed this point, though. > >I'm also concerned about future usability. Me too (perhaps in a different way though). >Word in the Py3k list is >that Python 2.6 will be just about the last Python in the 2.x series, >and by directing his implementation at only Python 2.x strings, he's >just about guaranteeing obsolescence. People will be using 2.x for a long time to come. And in the long run, isn't all software obsolete? :) >By building with unicode and/or >objects with a buffer interface in mind, Larry could build with both 2.x >and 3.x in mind, and his code wouldn't be obsolete the moment it was >released. (I'm not sure what the antecedent of "it" is in the above, I'm going to assume it's Python 3.x.) Supporting unicode strings and objects providing the buffer interface seems like a good idea in general, even disregarding Py3k. Starting with str is reasonable though, since there's still plenty of code that will benefit from this change, if it is indeed a beneficial change. Larry, I'm going to try to do some benchmarks against Twisted using this patch, but given my current time constraints, you may be able to beat me to this :) If you're interested, Twisted [EMAIL PROTECTED] plus this trial plugin: http://twistedmatrix.com/trac/browser/sandbox/exarkun/merit/trunk will let you do some gross measurements using the Twisted test suite. I can give some more specific pointers if this sounds like something you'd want to mess with. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
"Paul Moore" <[EMAIL PROTECTED]> wrote: > I had picked up on this comment, and I have to say that I had been a > little surprised by the resistance to the change based on the "code > would break" argument, when you had made such a thorough attempt to > address this. Perhaps others had missed this point, though. I'm also concerned about future usability. Word in the Py3k list is that Python 2.6 will be just about the last Python in the 2.x series, and by directing his implementation at only Python 2.x strings, he's just about guaranteeing obsolescence. By building with unicode and/or objects with a buffer interface in mind, Larry could build with both 2.x and 3.x in mind, and his code wouldn't be obsolete the moment it was released. - Josiah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
Larry Hastings wrote: > Am I correct in understanding that changing the Python minor revision > number (2.5 -> 2.6) requires external modules to recompile? not, in general, on Unix. it's recommended, but things usually work quite well anyway. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
Larry> The only function that *might* return a non-terminated char * is Larry> PyString_AsUnterminatedString(). This function is static to Larry> stringobject.c--and I would be shocked if it were ever otherwise. If it's static to stringobject.c it doesn't need a PyString_ prefix. In fact, I'd argue that it shouldn't have one so that people reading the code won't miss the "static" and think it is part of the published API. Larry> Am I correct in understanding that changing the Python minor Larry> revision number (2.5 -> 2.6) requires external modules to Larry> recompile? Yes, in general, though you can often get away without it if you don't mind Python screaming at you about version mismatches. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
On 10/23/06, Larry Hastings <[EMAIL PROTECTED]> wrote:
>
> Steve Holden wrote:
>
> But it seems to me that the only major issue is the inability to provide
> zero-byte terminators with this new representation.
>
> I guess I wasn't clear in my description of the patch; sorry about that.
>
> Like "lazy concatenation objects", "lazy slices" render when you call
> PyString_AsString() on them. Before rendering, the lazy slice's ob_sval
> will be NULL. Afterwards it will point to a proper zero-terminated string,
> at which point the object behaves exactly like any other PyStringObject.
I had picked up on this comment, and I have to say that I had been a
little surprised by the resistance to the change based on the "code
would break" argument, when you had made such a thorough attempt to
address this. Perhaps others had missed this point, though.
> I genuinely don't know how many external Python extension modules are
> well-behaved in this regard. But in case it helps: I just checked PIL,
> NumPy, PyWin32, and SWIG, and all of them were well-behaved.
There's code out there which was written to the Python 1.4 API, and
has not been updated since (I know, I wrote some of it!) I wouldn't
call it "well-behaved" (it writes directly into the string's character
buffer) but I don't believe it would fail (it only uses
PyString_AsString to get the buffer address).
/* Allocate an Python string object, with uninitialised contents. We
* must do it this way, so that we can modify the string in place
* later. See the Python source, Objects/stringobject.c for details.
*/
result = PyString_FromStringAndSize(NULL, len);
if (result == NULL)
return NULL;
p = PyString_AsString(result);
while (*str)
{
if (*str == '\n')
*p = '\0';
else
*p = *str;
++p;
++str;
}
> Am I correct in understanding that changing the Python minor revision
> number (2.5 -> 2.6) requires external modules to recompile? (It certainly
> does on Windows.) If so, I could mitigate the problem by renaming ob_sval.
> That way, code making explicit reference to it would fail to compile, which
> I feel is better than silently recompiling unsafe code.
I think you've covered pretty much all the possible backward
compatibility bases. A sufficiently evil extension could blow up, I
guess, but that's always going to be true.
OTOH, I don't have a comment on the desirability of the patch per se,
as (a) I've never been hit by the speed issue, and (b) I'm thoroughly
indoctrinated, so I always use ''.join() :-)
Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
On Mon, 23 Oct 2006 07:58:25 -0700, Larry Hastings <[EMAIL PROTECTED]> wrote: > [snip] >If external Python extension modules are as well-behaved as the shipping >Python source tree, there simply wouldn't be a problem. Python source is >delightfully consistent about using the macro PyString_AS_STRING() to get at >the creamy char *center of a PyStringObject *. When code religiously uses >that macro (or calls PyString_AsString() directly), all it needs is a >recompile with the current stringobject.h and it will Just Work. > >I genuinely don't know how many external Python extension modules are well- >behaved in this regard. But in case it helps: I just checked PIL, NumPy, >PyWin32, and SWIG, and all of them were well-behaved. FWIW, http://www.google.com/codesearch?q=+ob_sval Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
Steve Holden wrote: But it seems to me that the only major issue is the inability to provide zero-byte terminators with this new representation. I guess I wasn't clear in my description of the patch; sorry about that. Like "lazy concatenation objects", "lazy slices" render when you call PyString_AsString() on them. Before rendering, the lazy slice's ob_sval will be NULL. Afterwards it will point to a proper zero-terminated string, at which point the object behaves exactly like any other PyStringObject. The only function that *might* return a non-terminated char * is PyString_AsUnterminatedString(). This function is static to stringobject.c--and I would be shocked if it were ever otherwise. If there were any reliable way to make sure these objects never got passed to extension modules then I'd say "go for it". If external Python extension modules are as well-behaved as the shipping Python source tree, there simply wouldn't be a problem. Python source is delightfully consistent about using the macro PyString_AS_STRING() to get at the creamy char *center of a PyStringObject *. When code religiously uses that macro (or calls PyString_AsString() directly), all it needs is a recompile with the current stringobject.h and it will Just Work. I genuinely don't know how many external Python extension modules are well-behaved in this regard. But in case it helps: I just checked PIL, NumPy, PyWin32, and SWIG, and all of them were well-behaved. Apart from stringobject.c, there was exactly one spot in the Python source tree which made assumptions about the structure of PyStringObjects (Mac/Modules/macos.c). It's in the block starting with the comment "This is a hack:". Note that this is unfixed in my patch, so just now all code using that self-avowed "hack" will break. Am I correct in understanding that changing the Python minor revision number (2.5 -> 2.6) requires external modules to recompile? (It certainly does on Windows.) If so, I could mitigate the problem by renaming ob_sval. That way, code making explicit reference to it would fail to compile, which I feel is better than silently recompiling unsafe code. Cheers, larry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
[EMAIL PROTECTED] wrote: > >> Anyway, it was my intent to post the patch and see what happened. > >> Being a first-timer at this, and not having even read the core > >> development mailing lists for very long, I had no idea what to > >> expect. Though I genuinely didn't expect it to be this brusque. > > Martin> I could have told you :-) The "problem" really is that you are > Martin> suggesting a major, significant change to the implementation of > Martin> Python, and one that doesn't fix an obvious bug. > The "obvious bug" that it fixes is slowness <0.75 wink>. > Come on Martin. Give Larry a break. Lots of changes have been accepted to > to the Python core which weren't obvious "bug fixes". In fact, I seem to > recall a sprint held recently in Reykjavik where the whole point was just to > make Python faster. I believe that was exactly Larry's point in posting the > patch. The "one obvious way to do" concatenation and slicing for one of the > most heavily used types in python appears to be faster. That seems like a > win to me. > I did point out to Larry when he went to c.l.py with the original patch that he would face resistance, so this hasn't blind-sided him. But it seems to me that the only major issue is the inability to provide zero-byte terminators with this new representation. Because Larry's proposal for handling this involves the introduction of a new API that can't already be in use in extensions it's obviously the extension writers who would be given most problems by this patch. I can understand resistance on that score, and I could understand resistance if there were other clear disadvantages to its implementation, but in their absence it seems like the extension modules are the killers. If there were any reliable way to make sure these objects never got passed to extension modules then I'd say "go for it". Without that it does seem like a potentially widespread change to the C API that could affect much code outside the interpreter. This is a great shame. I think Larry showed inventiveness and tenacity to get this far, and deserves credit for his achievements no matter whether or not they get into the core. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
>> Anyway, it was my intent to post the patch and see what happened. >> Being a first-timer at this, and not having even read the core >> development mailing lists for very long, I had no idea what to >> expect. Though I genuinely didn't expect it to be this brusque. Martin> I could have told you :-) The "problem" really is that you are Martin> suggesting a major, significant change to the implementation of Martin> Python, and one that doesn't fix an obvious bug. Come on Martin. Give Larry a break. Lots of changes have been accepted to to the Python core which weren't obvious "bug fixes". In fact, I seem to recall a sprint held recently in Reykjavik where the whole point was just to make Python faster. I believe that was exactly Larry's point in posting the patch. The "one obvious way to do" concatenation and slicing for one of the most heavily used types in python appears to be faster. That seems like a win to me. Skip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The "lazy strings" patch
Josiah Carlson wrote: > Want my advice? Aim for Py3k text as your primary target, but as a > wrapper, not as the core type (I put the odds at somewhere around 0 for > such a core type change). If you are good, and want to make guys like > me happy, you could even make it support the buffer interface for > non-text (bytes, array, mmap, etc.), unifying (via wrapper) the behavior > of bytes and text. This is still my preferred approach, too - for local optimisation of an algorithm, a string view type strikes me as an excellent idea. For the core data type, though, keeping the behaviour comparatively simple and predictable counterbalances the desire for more speed. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
