[Python-Dev] Divorcing str and unicode (no more implicit conversions).
Hi. Like a lot of people (or so I hear in the blogosphere...), I've been experiencing some friction in my code with unicode conversion problems. Even when being super extra careful with the types of str's or unicode objects that my variables can contain, there is always some case or oversight where something unexpected happens which results in a conversion which triggers a decode error. str.join() of a list of strs, where one unicode object appears unexpectedly, and voila! exception galore. Sometimes the problem shows up late because your test code doesn't always contain accented characters. I'm sure many of you experienced that or some variant at some point. I came to realize recently that this problem shares strong similarity with the problem of implicit type conversions in C++, or at least it feels the same: Stuff just happens implicitly, and it's hard to track down where and when it happens by just looking at the code. Part of the problem is that the unicode object acts a lot like a str, which is convenient, but... What if we could completely disable the implicit conversions between unicode and str? In other words, if you would ALWAYS be forced to call either .encode() or .decode() to convert between one and the other... wouldn't that help a lot deal with that issue? How hard would that be to implement? Would it break a lot of code? Would some people want that? (I know I would, at least for some of my code.) It seems to me that this would make the code more explicit and force the programmer to become more aware of those conversions. Any opinions welcome. cheers, ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On 10/2/05, Christopher Armstrong <[EMAIL PROTECTED]> wrote:
> On 10/3/05, Martin Blais <[EMAIL PROTECTED]> wrote:
> > On 10/1/05, Antoine <[EMAIL PROTECTED]> wrote:
> > >
> > > > like this with their "deferred objects", no? I figure they would
> > > > need to do something like this too. I will have to check.)
> > >
> > > A Deferred object is just the abstraction of a callback - or, rather, two
> > > callbacks: one for success and one for failure. Twisted is architected
> > > around an event loop, which calls your code back when a registered event
> > > happens (for example when an operation is finished, or when some data
> > > arrives on the wire). Compared to generators, it is a different way of
> > > expressing cooperative multi-threading.
> >
> > So, the question is, in Twisted, if I want to defer on an operation
> > that is going to block, say I'm making a call to run a database query
> > that I'm expecting will take much time, and want to yield ("defer")
> > for other events to be processed while the query is executed, how do I
> > do that? As far as I remember the Twisted docs I read a long time ago
> > did not provide a solution for that.
>
> Deferreds don't make blocking code non-blocking; they're just a way to
> make it nicer to write non-blocking code. There are utilities in
> Twisted for wrapping a blocking function call in a thread and having
> the result returned in a Deferred, though (see deferToThread). There
> is also a lightweight and complete wrapper for DB-API2 database
> modules in twisted.enterprise.adbapi, which does the threading
> interaction for you.
>
> So, since this then exposes a non-blocking API, you can do stuff like
>
> d = pool.runQuery('SELECT User_ID FROM Users')
> d.addCallback(gotDBData)
> d2 = ldapfoo.getUser('bob')
> d2.addCallback(gotLDAPData)
>
> And both the database call and the ldap request will be worked on
> concurrently.
Very nice!
However, if you're using a thread to do just that, it's just using a
part of what threads were designed for: it's really just using the
low-level kernel knowledge about resource access and when they become
ready to wait on the resource, since you're not going to run much
actual code in the thread itself (apart from setting up to do the
blocking call and returning its value).
Now, if we had something in the language that allows us to do
something like that--make the most important potentially blocking
calls asynchronously-- we could implement a more complete scheduler
that could really leverage generators to create a more interesting
concurrency solution with less overhead. For example, imagine that
some class of generators are used as tasks, like we were discussing
before. When you would call the special yield_read() call (a
variation on e.g. os.read() call), there is an implicit yield that
allows other generators which are ready to run until the data is
available, without the overhead of
1. context switching to the helper threads and back;
2. synchronization for communcation with the helper threads (I assume
threads would not be created dynamically, for efficiency. I imagine
there is a pool of helpers waiting to do the async call jobs, and
communication with them to dispatch the call jobs does not come for
free (i.e. locking)).
We really don't need threads at all to do that (at least for the
common blocking calls), just some low-level support for building a
scheduler. Using threads to do that has a cost, it is more or less a
kludge, in that context (but we have nothing better for now).
cheers,
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On 10/3/05, Martin Blais <[EMAIL PROTECTED]> wrote:
> On 10/1/05, Antoine <[EMAIL PROTECTED]> wrote:
> >
> > > like this with their "deferred objects", no? I figure they would
> > > need to do something like this too. I will have to check.)
> >
> > A Deferred object is just the abstraction of a callback - or, rather, two
> > callbacks: one for success and one for failure. Twisted is architected
> > around an event loop, which calls your code back when a registered event
> > happens (for example when an operation is finished, or when some data
> > arrives on the wire). Compared to generators, it is a different way of
> > expressing cooperative multi-threading.
>
> So, the question is, in Twisted, if I want to defer on an operation
> that is going to block, say I'm making a call to run a database query
> that I'm expecting will take much time, and want to yield ("defer")
> for other events to be processed while the query is executed, how do I
> do that? As far as I remember the Twisted docs I read a long time ago
> did not provide a solution for that.
Deferreds don't make blocking code non-blocking; they're just a way to
make it nicer to write non-blocking code. There are utilities in
Twisted for wrapping a blocking function call in a thread and having
the result returned in a Deferred, though (see deferToThread). There
is also a lightweight and complete wrapper for DB-API2 database
modules in twisted.enterprise.adbapi, which does the threading
interaction for you.
So, since this then exposes a non-blocking API, you can do stuff like
d = pool.runQuery('SELECT User_ID FROM Users')
d.addCallback(gotDBData)
d2 = ldapfoo.getUser('bob')
d2.addCallback(gotLDAPData)
And both the database call and the ldap request will be worked on concurrently.
--
Twisted | Christopher Armstrong: International Man of Twistery
Radix|-- http://radix.twistedmatrix.com
| Release Manager, Twisted Project
\\\V/// |-- http://twistedmatrix.com
|o O||
wvw-+
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Pythonic concurrency - cooperative MT
On 10/1/05, Antoine <[EMAIL PROTECTED]> wrote:
>
> > like this with their "deferred objects", no? I figure they would
> > need to do something like this too. I will have to check.)
>
> A Deferred object is just the abstraction of a callback - or, rather, two
> callbacks: one for success and one for failure. Twisted is architected
> around an event loop, which calls your code back when a registered event
> happens (for example when an operation is finished, or when some data
> arrives on the wire). Compared to generators, it is a different way of
> expressing cooperative multi-threading.
So, the question is, in Twisted, if I want to defer on an operation
that is going to block, say I'm making a call to run a database query
that I'm expecting will take much time, and want to yield ("defer")
for other events to be processed while the query is executed, how do I
do that? As far as I remember the Twisted docs I read a long time ago
did not provide a solution for that.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why does __getitem__ slot of builtin call sequence methods first?
Travis Oliphant <[EMAIL PROTECTED]> writes: > Thanks for the tip. I think I figured out the problem, and it was my > misunderstanding of how types inherit in C that was the source of my > problem. > > Basically, Python is doing what you would expect, the mp_item is used > for __getitem__ if both mp_item and sq_item are present. However, the > addition of these descriptors (and therefore the resolution of any > comptetion for __getitem__ calls) is done *before* the inheritance of > any slots takes place. Oof. That'd do it. > The new ndarray object inherits from a "big" array object that doesn't > define the sequence and buffer protocols (which have the size limiting > int dependencing in their interfaces). The ndarray object has standard > tp_as_sequence and tp_as_buffer slots filled. I guess the reason this hasn't come up before is that non-trivial C inheritance is still pretty rare. > The easy fix was to initialize the tp_as_mapping slot before calling > PyType_Ready.Hopefully, somebody else searching in the future for an > answer to their problem will find this discussion useful. Well, it sounds like a bug that should be easy to fix. I can't think of a reason to do slot wrapper generation before slot inheritance, though I wouldn't like to bet more than a beer on not having missed something... Cheers, mwh -- There are two kinds of large software systems: those that evolved from small systems and those that don't work. -- Seen on slashdot.org, then quoted by amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tests and unicode
Reinhold Birkenfeld <[EMAIL PROTECTED]> writes: > Hi, > > I looked whether I could make the test suite pass again > when compiled with --disable-unicode. > > One problem is that no Unicode escapes can be used since compiling > the file raises ValueErrors for them. Such strings would have to > be produced using unichr(). Yeah, I've bumped into this. > Is this the right way? Or is disabling Unicode not supported any more? I don't know. More particularly, I don't know if anyone actually uses a unicode-disabled build. If noone does, we might as well rip the code out. Cheers, mwh -- Sufficiently advanced political correctness is indistinguishable from irony. -- Erik Naggum ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] C API doc fix
Jim Jewett wrote: > > > Py_UNICODE > Python uses this type to store Unicode ordinals. It is > typically a typedef alias, but the underlying type -- and > the size of that type -- varies across different systems. > I think I objected to such a formulation, requesting that the precise procedure is documented for chosing the alias. I then went on saying what the precise procedure is, and then an argument about that procedure arose. I still believe that the precise procedure should be documented (in addition to saying that its outcome may vary across installations). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tests and unicode
Reinhold Birkenfeld wrote: > One problem is that no Unicode escapes can be used since compiling > the file raises ValueErrors for them. Such strings would have to > be produced using unichr(). You mean, in Unicode literals? There are various approaches, depending on context: - you could encode the literals as UTF-8, and decode it when the module/test case is imported. See test_support.TESTFN_UNICODE for an example. - you could use unichr - you could use eval, see test_re for an example > Is this the right way? Or is disabling Unicode not supported any more? There are certainly tests that cannot be executed when Unicode is not available. It would be good if such tests get skipped instead of being failing, and it would be good if all tests that do not require Unicode support run even when Unicode support is missing. Whether "it is supported" is a tricky question: your message indicates that, right now, it is *not* supported (or else you wouldn't have noticed a problem). Whether we think it should be supported depends on who "we" is, as with all these minor features: some think it is a waste of time, some think it should be supported if reasonably possible, and some think this a conditio sine qua non. It certainly isn't a release-critical feature. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Help needed with MSI permissions
I have various reports that the Python 2.4 installer does not work if you are trying to install in a non-standard location as a non-privileged user, e.g. #1298962, #1234328, #1232947, #1199808. Despite many attempts, I haven't been able to reproduce any such problem, and the submitters weren't really able to experiment much, either. So, if anybody is able to reproduce any of these reports, and give me instructions on how to reproduce it myself, that would be very much appreciated. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
