Re: [Python-Dev] The endless GIL debate: why not remove thread support instead?
2008/12/17 Greg Ewing : > Nick Coghlan wrote: > >> Actually, I believe 3.0 already took a big step towards allowing this by >> changing the way modules are initialised. > > It's a step, but I wouldn't call it a big one. There are > many other problems to be solved before fully independent > interpreters are possible. Do you know if these remaining problems are listed anywhere? AIUI, certain software (for example mod_python) has been using multiple interpreters for a long while now - admittedly not without issues, but certainly enough to imply that multiple interpreters are at least "possible" - although not perfect. Experience with such software would probably be a great guide to where the issues exist. Maybe a page on the Python Wiki, or a FAQ entry, would be useful here. If only to make things explicit, and clear up some of the FUD around multiple interpreters. Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.0 urllib fails with chunked HTTP responses
On Wed, Dec 17, 2008 at 1:05 PM, Guido van Rossum wrote: > The inheritance from io.RawIOBase seems fine. There is a small problem with the interaction between HTTPResponse and RawIOBase, but I think the problem is more on the http side. You may recall that the HTTP code has a habit of closing the connection for you. In a variety of cases, once you've read the last bytes of the response, the HTTPResponse object calls its own close() method. This interacts poorly with RawIOBase, because it raises a ValueError for any operation on a closed io object. This prevents iterators from working correctly. The iterator implementation expects the final call to readline() to return an empty string and converts that to a StopIteration. Instead, it's seeing a ValueError that propagates out. It's always been odd to me that the connection closed itself. It's going to be tricky to fix the current bug (chunked responses) and keep the self-closing behavior, but I worry that change the self-closing behavior too dramatically isn't appropriate for a bug fix. Will look some more at this tomorrow. Jeremy > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > On Mon, Dec 15, 2008 at 11:19 AM, Jeremy Hylton wrote: >> I have a patch that appears to fix this bug >> http://bugs.python.org/file12361/urllib-chunked.diff >> but I'm not sure about its interaction with the io module and >> RawIOBase. Is there a new IO expert who could take a look at it for >> me? >> >> Jeremy >> >> On Sun, Dec 14, 2008 at 11:06 PM, Jeremy Hylton wrote: >>> This bug is pretty serious, because urllib will insert garbage into >>> the application-visible data for a chunked response. It simply >>> ignores the fact that it's reading a chunked response and includes the >>> chunked header data is payload data. The original bug was reported in >>> September, but no one noticed it. It was reported again recently. >>> >>> http://bugs.python.org/issue3761 >>> http://bugs.python.org/issue4631 >>> >>> I suspect we'd want to get a 3.0.1 out as soon as this is fixed, but >>> that's not my call. >>> >>> Jeremy >>> >> ___ >> Python-Dev mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.0 urllib fails with chunked HTTP responses
It sounds like the self-closing is an implementation detail, meant to make sure the socket is closed as early as possible (which I suppose is a good thing if there's a server waiting for the final ACK on the other side). Perhaps it should not use close() but something slightly lower level that affects the socket directly? --Guido van Rossum (home page: http://www.python.org/~guido/) On Thu, Dec 18, 2008 at 5:22 AM, Jeremy Hylton wrote: > On Wed, Dec 17, 2008 at 1:05 PM, Guido van Rossum wrote: >> The inheritance from io.RawIOBase seems fine. > > There is a small problem with the interaction between HTTPResponse and > RawIOBase, but I think the problem is more on the http side. You may > recall that the HTTP code has a habit of closing the connection for > you. In a variety of cases, once you've read the last bytes of the > response, the HTTPResponse object calls its own close() method. This > interacts poorly with RawIOBase, because it raises a ValueError for > any operation on a closed io object. This prevents iterators from > working correctly. The iterator implementation expects the final call > to readline() to return an empty string and converts that to a > StopIteration. Instead, it's seeing a ValueError that propagates out. > > It's always been odd to me that the connection closed itself. It's > going to be tricky to fix the current bug (chunked responses) and keep > the self-closing behavior, but I worry that change the self-closing > behavior too dramatically isn't appropriate for a bug fix. Will look > some more at this tomorrow. > > Jeremy > >> --Guido van Rossum (home page: http://www.python.org/~guido/) >> >> >> >> On Mon, Dec 15, 2008 at 11:19 AM, Jeremy Hylton wrote: >>> I have a patch that appears to fix this bug >>> http://bugs.python.org/file12361/urllib-chunked.diff >>> but I'm not sure about its interaction with the io module and >>> RawIOBase. Is there a new IO expert who could take a look at it for >>> me? >>> >>> Jeremy >>> >>> On Sun, Dec 14, 2008 at 11:06 PM, Jeremy Hylton wrote: This bug is pretty serious, because urllib will insert garbage into the application-visible data for a chunked response. It simply ignores the fact that it's reading a chunked response and includes the chunked header data is payload data. The original bug was reported in September, but no one noticed it. It was reported again recently. http://bugs.python.org/issue3761 http://bugs.python.org/issue4631 I suspect we'd want to get a 3.0.1 out as soon as this is fixed, but that's not my call. Jeremy >>> ___ >>> Python-Dev mailing list >>> [email protected] >>> http://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> http://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >> > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.0 urllib fails with chunked HTTP responses
Jeremy Hylton wrote: > but I worry that change the self-closing > behavior too dramatically isn't appropriate for a bug fix. Will look > some more at this tomorrow. Reading through the code, it looks like you've already fixed bug 1348. Thanks! Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.0 urllib fails with chunked HTTP responses
On Thu, Dec 18, 2008 at 12:27 PM, Guido van Rossum wrote: > It sounds like the self-closing is an implementation detail, meant to > make sure the socket is closed as early as possible (which I suppose > is a good thing if there's a server waiting for the final ACK on the > other side). Perhaps it should not use close() but something slightly > lower level that affects the socket directly? That's what I'm thinking, too. I had 10 minutes last night after the kids went to bed, and my first attempt didn't work :-). Jeremy > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > On Thu, Dec 18, 2008 at 5:22 AM, Jeremy Hylton wrote: >> On Wed, Dec 17, 2008 at 1:05 PM, Guido van Rossum wrote: >>> The inheritance from io.RawIOBase seems fine. >> >> There is a small problem with the interaction between HTTPResponse and >> RawIOBase, but I think the problem is more on the http side. You may >> recall that the HTTP code has a habit of closing the connection for >> you. In a variety of cases, once you've read the last bytes of the >> response, the HTTPResponse object calls its own close() method. This >> interacts poorly with RawIOBase, because it raises a ValueError for >> any operation on a closed io object. This prevents iterators from >> working correctly. The iterator implementation expects the final call >> to readline() to return an empty string and converts that to a >> StopIteration. Instead, it's seeing a ValueError that propagates out. >> >> It's always been odd to me that the connection closed itself. It's >> going to be tricky to fix the current bug (chunked responses) and keep >> the self-closing behavior, but I worry that change the self-closing >> behavior too dramatically isn't appropriate for a bug fix. Will look >> some more at this tomorrow. >> >> Jeremy >> >>> --Guido van Rossum (home page: http://www.python.org/~guido/) >>> >>> >>> >>> On Mon, Dec 15, 2008 at 11:19 AM, Jeremy Hylton wrote: I have a patch that appears to fix this bug http://bugs.python.org/file12361/urllib-chunked.diff but I'm not sure about its interaction with the io module and RawIOBase. Is there a new IO expert who could take a look at it for me? Jeremy On Sun, Dec 14, 2008 at 11:06 PM, Jeremy Hylton wrote: > This bug is pretty serious, because urllib will insert garbage into > the application-visible data for a chunked response. It simply > ignores the fact that it's reading a chunked response and includes the > chunked header data is payload data. The original bug was reported in > September, but no one noticed it. It was reported again recently. > > http://bugs.python.org/issue3761 > http://bugs.python.org/issue4631 > > I suspect we'd want to get a 3.0.1 out as soon as this is fixed, but > that's not my call. > > Jeremy > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org >>> >> > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The endless GIL debate: why not remove thread support instead?
Paul Moore wrote: Do you know if these remaining problems are listed anywhere? There was a big discussion about this in comp.lang.python not long ago. Basically all the built-in types and constants are shared between interpreters, which means you still need a GIL to stop different interpreters stepping on each other's toes. AIUI, certain software (for example mod_python) has been using multiple interpreters for a long while now Multiple interpeters are possible, they're just not completely independent. Whether this is a problem depends on the reason you want multiple interpreters. In the Apache case, it's probably more about providing virtual Python environments than free-threading between interpreters. -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The endless GIL debate: why not remove thread support instead?
Greg Ewing wrote: > Paul Moore wrote: >> Do you know if these remaining problems are listed anywhere? > > There was a big discussion about this in comp.lang.python > not long ago. Basically all the built-in types and constants > are shared between interpreters, which means you still need > a GIL to stop different interpreters stepping on each other's > toes. That kind of thing is under the core's control though - the 2.x module initialisation problem means that you can't write a multiple interpreter friendly extension module even if you want to. The new per-interpreter state mechanism could also be used internally by the core to duplicate some of that global state for each new interpreter. I see the introduction of the interpreter specific state mechanism as a big step because it provides an underlying mechanism that makes the problem solvable *in principle* through a combination of per-interpreter state and finer grained shared locking, making it just a practical implementation problem to see if that can be done without adversely impacting single interpreter performance. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The endless GIL debate: why not remove thread support instead?
2008/12/18 Greg Ewing : > Paul Moore wrote: >> >> Do you know if these remaining problems are listed anywhere? > > There was a big discussion about this in comp.lang.python > not long ago. Basically all the built-in types and constants > are shared between interpreters, which means you still need > a GIL to stop different interpreters stepping on each other's > toes. > >> AIUI, >> certain software (for example mod_python) has been using multiple >> interpreters for a long while now > > Multiple interpeters are possible, they're just not completely > independent. Whether this is a problem depends on the reason > you want multiple interpreters. In the Apache case, it's > probably more about providing virtual Python environments than > free-threading between interpreters. OK, but how close is it to providing isolation for threads running under the control of the GIL? I'm thinking of something along the lines of an in-process version of fork(), which spawns a new interpreter and runs the 2 interpreters as threads, still using the GIL to enforce serialisation, but otherwise independent. I believe that Perl uses this model for its "interpreter threads" implementation. Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The endless GIL debate: why not remove thread support instead?
Paul Moore schrieb: > OK, but how close is it to providing isolation for threads running > under the control of the GIL? I'm thinking of something along the > lines of an in-process version of fork(), which spawns a new > interpreter and runs the 2 interpreters as threads, still using the > GIL to enforce serialisation, but otherwise independent. I believe > that Perl uses this model for its "interpreter threads" > implementation. How is your idea different from subinterpreters? Today you can have multiple subinterpreters inside a single process. Each subinterpreter has its own state and can see only its own objects. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The endless GIL debate: why not remove thread support instead?
> OK, but how close is it to providing isolation for threads running > under the control of the GIL? They won't be indedepent. If an extension module has a global variable, that will be shared across interpreters. If that variable supports modifiable state, such modifications will "leak" across interpreters. For example, there will be only a single object class. With that in mind, take a look at object.__subclasses__(); it would provide access to all classes, including those in the other interpreters. Likewise, gc.get_objects() will give you the complete list of all objects. So the isolation is not strong enough to run untrusted code isolated from other code. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
