Re: [Python-ideas] async objects

Your are right about the importance of Python C API, it often goes under my radar. For the past 20 years I have only used it a couple of times (to integrate Python into some existing C-code) therefore it is not as much in focus as it should be and definiatly are by others. I get your innovators

Re: [Python-ideas] async objects

The problem is that if your goal is to make a practical proposal, it's not enough to look at Python-the-language. You're absolutely right, AFAICT there's nothing stopping someone from making a nice implementation of Python-the-language that has erlang-style cheap shared-nothing threads with some ef

Re: [Python-ideas] async objects

Regarding the Python C-runtime and async, I just had a good talk with Kresten Krab at Trifork. He implemented “Erjang” the Java implementation of the Erlang VM (www.erjang.org ). Doing this he had access to the Erlang (C) VM. It turn’s out that the Erlang VM and the Pyth

Re: [Python-ideas] async objects

> On 7 Oct 2016, at 16:18, Nick Coghlan wrote: > > However, if you're running in a context that embeds CPython inside a > larger application (e.g. mod_wsgi inside Apache), then gevent's > assumptions about how the C thread states are managed may be wrong, > and hence you may be in for some "inte

Re: [Python-ideas] async objects

On 7 October 2016 at 16:42, Nathaniel Smith wrote: > For folks who prefer the > gevent API, is it really easier to port libraries to asyncio than to > port them to gevent? It's definitely *not* easier, as gevent lets you suspend execution inside arbitrary CPython magic method calls. That's why yo

Re: [Python-ideas] async objects

On Thu, Oct 6, 2016 at 4:12 PM, Greg Ewing wrote: > Nathaniel Smith wrote: >> >> The core distinguishing feature between >> async/await and gevent is the visibility of suspension points, so it >> might as well be the case that async/await is designed for exactly >> those people who want visible su

Re: [Python-ideas] async objects

Yury Selivanov wrote: To start, no matter how exactly you want to approach this, it would require us to do a *complete rewrite* of CPython internals. This is so complex that we wouldn't be able to even estimate how long it would take us. You could ask the author of Stackless -- he did exactl

Re: [Python-ideas] async objects

Nick Coghlan wrote: When a language usage pattern is supported for that long, but folks still don't grok how it might benefit them, you have a UX problem, and one of the ways to address it is to take the existing pattern and give it dedicated syntax, which is exactly what PEP 492 did. However,

Re: [Python-ideas] async objects

Nick Coghlan wrote: The pay-off that CPython gets from this is that we get to delegate 99.9% of the work for supporting different CPU architectures to C compiler developers, and we get a lot of capabilities "for free" when it comes to stack management. One of the main benefits is that it's very

Re: [Python-ideas] async objects

Nathaniel Smith wrote: The core distinguishing feature between async/await and gevent is the visibility of suspension points, so it might as well be the case that async/await is designed for exactly those people who want visible suspension points. They're not quite independent axes, though. Gev

Re: [Python-ideas] async objects

On 2016-10-06 03:27, Nick Coghlan wrote: It's not a question that's up for debate - as a point of factual history, Python's runtime model is anchored in the C runtime model, and this pervades the entire language design. Simply wishing that Python's core runtime design was other than it is doesn't

Re: [Python-ideas] async objects

Interestingly, this just showed up on Python List: On 10/06/2016 05:09 AM, Frank Millman wrote: I have used itertools.groupby before, and I love it. I used it to process a csv file and 'break' on change of a particular field. It worked very well. Now I want to use it to process a database ta

Re: [Python-ideas] async objects

On 2016-10-06 1:15 AM, Stephen J. Turnbull wrote: These may be hard to explain, and I know you, Yury, and Guido are very busy. But it's frustrating for all to see this go around in a circle: "it's like it is because it has to be that way, so that's the way it is". To add to what Nick said. I

Re: [Python-ideas] async objects

2016-10-06 13:50 GMT+02:00 Nick Coghlan : > The shadow thread idea will hopefully prove successful in addressing > the last major rough spot in the UX, which is the ability to easily > integrate asynchronous components into an otherwise synchronous > application. > That's my opinion as well. If I

Re: [Python-ideas] async objects

On 6 October 2016 at 17:45, Greg Ewing wrote: > Saying that 'await' is good for you because it > makes the suspension points visible seems to me > a rationalisation after the fact. It was something > that emerged from the implementation, not a > prior design requirement. I'd say it emerged from m

Re: [Python-ideas] async objects

On 6 October 2016 at 15:15, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > Python's core runtime model is the C runtime model: threads (with a > > local stack and access to a global process heap) and processes (which > > contain a heap and one or more threads). Anything else we do (whe

Re: [Python-ideas] async objects

On 6 October 2016 at 05:20, Sven R. Kunze wrote: > On 05.10.2016 18:06, Nick Coghlan wrote: >> >> [runtime matters] > > > I think I understand your point. > > I also hope that others and me could provide you with our perspective. We > see Python not as a C-like runtime but as an abstract modelling

Re: [Python-ideas] async objects

On Thu, Oct 6, 2016 at 12:45 AM, Greg Ewing wrote: > Nathaniel Smith wrote: >> >> It wasn't that we created these keywords to solve some >> implementation problem and then inflicted them on users. > > > I disagree -- looking at the history of how we > ended up with async/await, it looks to me like

Re: [Python-ideas] async objects

Nathaniel Smith wrote: It wasn't that we created these keywords to solve some implementation problem and then inflicted them on users. I disagree -- looking at the history of how we ended up with async/await, it looks to me like this is exactly what *did* happen. First we had generators. Then

Re: [Python-ideas] async objects

On Wed, Oct 5, 2016 at 1:28 PM, Rene Nejsum wrote: > When I first read about the async idea, I initially expected that it would be > some stackless like additions to Python. My wish for Python was an addition > to the language the allowed an easy an elegant concurrent model on the > language le

Re: [Python-ideas] async objects

> On 06 Oct 2016, at 07:15, Stephen J. Turnbull > wrote: > > Nick Coghlan writes: > >> Python's core runtime model is the C runtime model: threads (with a >> local stack and access to a global process heap) and processes (which >> contain a heap and one or more threads). Anything else we do (w

Re: [Python-ideas] async objects

Nick Coghlan writes: > Python's core runtime model is the C runtime model: threads (with a > local stack and access to a global process heap) and processes (which > contain a heap and one or more threads). Anything else we do (whether > it's generators, coroutines, or some other form of paused

Re: [Python-ideas] async objects

Paul Moore wrote: I don't know *that* much about Erlang, but Python's model is that of a single shared address space with (potentially multiple) threads of code running, having access to that address space. I don't know much about Erlang either, but from what I gather, it's a functional languag

Re: [Python-ideas] async objects

On 5 October 2016 at 21:28, Rene Nejsum wrote: > But, are the runtimes for Python and Erlang that fundamentally different? Is > it Python’s tight integration with C that is the big difference? I don't know *that* much about Erlang, but Python's model is that of a single shared address space with

Re: [Python-ideas] async objects

> On 05 Oct 2016, at 21:20, Sven R. Kunze wrote: > > On 05.10.2016 18:06, Nick Coghlan wrote: >> [runtime matters] > > I think I understand your point. > > I also hope that others and me could provide you with our perspective. We see > Python not as a C-like runtime but as an abstract modelli

Re: [Python-ideas] async objects

On 10/05/2016 12:20 PM, Sven R. Kunze wrote: On 05.10.2016 18:06, Nick Coghlan wrote: Guido's idea of a shadow thread to let synchronous threads run coroutines without needing to actually run a foreground event loop should provide a manageable way of getting the two runtime models (traditional

Re: [Python-ideas] async objects

On 05.10.2016 18:06, Nick Coghlan wrote: [runtime matters] I think I understand your point. I also hope that others and me could provide you with our perspective. We see Python not as a C-like runtime but as an abstract modelling language. I know that it's different from the point of view of

Re: [Python-ideas] async objects

On 05.10.2016 20:23, Michel Desmoulin wrote: On the other hand, await / async is a fantastic interface to unify all concurrent paradigms and asyncio already provide a bridge with threads and subprocess. So it kinda make sense. Almost if it would not require duplicate pieces of code. But maybe

Re: [Python-ideas] async objects

On 05.10.2016 08:49, Rene Nejsum wrote: As a result of past discussions, I wrote the module "xfork" which basically does this "golang goroutine" stuff. It's just a thin wrapper around "futures" but it allows to avoid that what René and Anthony objects about. I had a look at xfork, and really

Re: [Python-ideas] async objects

On the other hand, await / async is a fantastic interface to unify all concurrent paradigms and asyncio already provide a bridge with threads and subprocess. So it kinda make sense. Le 04/10/2016 à 18:40, Sven R. Kunze a écrit : On 04.10.2016 13:30, Nick Coghlan wrote: What it *doesn't* do, a

Re: [Python-ideas] async objects

On 5 October 2016 at 16:49, Rene Nejsum wrote: >> On 04 Oct 2016, at 18:40, Sven R. Kunze wrote: >> I don't think that's actually what I wanted here. One simple keyword should >> have sufficed just like golang did. So, the developer gets a way to decide >> whether or not he needs it blocking or

Re: [Python-ideas] async objects

Rene Nejsum writes: > On 04 Oct 2016, at 18:40, Sven R. Kunze wrote: > > I don't think that's actually what I wanted here. One simple > > keyword should have sufficed just like golang did. So, the > > developer gets a way to decide whether or not he needs it > > blocking or nonblocking **whe

Re: [Python-ideas] async objects

> On 04 Oct 2016, at 18:15, Guido van Rossum wrote: > > On Tue, Oct 4, 2016 at 4:30 AM, Nick Coghlan wrote: >>class SomeClass(object): >>def some_sync_method(self): >>return 42 >>async def some_async_method(self): >>await asyncio.sleep(3) >>

Re: [Python-ideas] async objects

> On 04 Oct 2016, at 18:40, Sven R. Kunze wrote: > > On 04.10.2016 09:50, Stephen J. Turnbull wrote: >> As I understand the main point, Sven and Rene don't believe that [the >> kind of] async code [they want to write] should need any keywords; >> just start the event loop and invoke functions, a

Re: [Python-ideas] async objects

On 5 October 2016 at 02:15, Guido van Rossum wrote: > Honestly it feels like many things can go wrong with this API model, > esp. you haven't answered what should happen when a method of > SomeClass (either a synchronous one or an async one) calls > run_in_foreground() on something -- or, more lik

Re: [Python-ideas] async objects

On Wed, Oct 5, 2016 at 3:40 AM, Sven R. Kunze wrote: > I don't think that's actually what I wanted here. One simple keyword should > have sufficed just like golang did. So, the developer gets a way to decide > whether or not he needs it blocking or nonblocking **when using a > function**. He doesn

Re: [Python-ideas] async objects

On 04.10.2016 09:50, Stephen J. Turnbull wrote: As I understand the main point, Sven and Rene don't believe that [the kind of] async code [they want to write] should need any keywords; just start the event loop and invoke functions, and that somehow automatically DTRTs. [reading my name second t

Re: [Python-ideas] async objects

On 04.10.2016 13:30, Nick Coghlan wrote: What it *doesn't* do, and what you need greenlet for, is making that common interface look like you're using plain old synchronous C threads. If folks really want to do that, that's fine - they just need to add gevent/greenlet as a dependency, just as the

Re: [Python-ideas] async objects

On Tue, Oct 4, 2016 at 4:30 AM, Nick Coghlan wrote: > class SomeClass(object): > def some_sync_method(self): > return 42 > async def some_async_method(self): > await asyncio.sleep(3) > return 42 > > o = auto_schedule(SomeClass()) # Indicatin

Re: [Python-ideas] async objects

On Mon, Oct 3, 2016 at 10:37 PM, Rene Nejsum wrote: > Ideally I think a language (would love it to be Python) should > permit many (millions) of what we know as coroutines and then have as many > threads as the CPU have cores to execute this coroutines, but I do not thing > you as a programmer sho

Re: [Python-ideas] async objects

On 4 October 2016 at 17:50, Stephen J. Turnbull wrote: > Nick Coghlan writes: > > > What's not well-defined are the interfaces for calling into > > asynchronous code from synchronous code. > > I don't understand the relevance to the content of the thread. Given the schedule_coroutine/run_in_for

Re: [Python-ideas] async objects

Nick Coghlan writes: > What's not well-defined are the interfaces for calling into > asynchronous code from synchronous code. I don't understand the relevance to the content of the thread. As I understand the main point, Sven and Rene don't believe that [the kind of] async code [they want to w

Re: [Python-ideas] async objects

> On 04 Oct 2016, at 07:26, Chris Angelico wrote: > > On Tue, Oct 4, 2016 at 4:25 PM, Rene Nejsum wrote: >>> On 04 Oct 2016, at 02:09, Stephen J. Turnbull >>> wrote: >>> >>> Rene Nejsum writes: I believe that you should be able to code concurrent code, without being to explicit abo

Re: [Python-ideas] async objects

> On 03 Oct 2016, at 23:32, Greg Ewing wrote: > > Yann Kaiser wrote: >> The way I see it, the great thing about async/await as opposed to threading >> is that it is explicit about when execution will "take a break" from your >> function or resume into it. > > Another thing is that async/await

Re: [Python-ideas] async objects

> On 04 Oct 2016, at 02:48, C Anthony Risinger wrote: > > On Oct 3, 2016 7:09 PM, "Stephen J. Turnbull" > > wrote: > > > > Rene Nejsum writes: > > > > > I believe that you should be able to code concurrent code, without > > > being to explicit abou

Re: [Python-ideas] async objects

On Tue, Oct 4, 2016 at 4:25 PM, Rene Nejsum wrote: >> On 04 Oct 2016, at 02:09, Stephen J. Turnbull >> wrote: >> >> Rene Nejsum writes: >>> I believe that you should be able to code concurrent code, without >>> being to explicit about it, but let the runtime handle low-level >>> timing, as long

Re: [Python-ideas] async objects

> On 04 Oct 2016, at 02:09, Stephen J. Turnbull > wrote: > > Rene Nejsum writes: >> I believe that you should be able to code concurrent code, without >> being to explicit about it, but let the runtime handle low-level >> timing, as long as you know your code will execute in the intended >> ord

Re: [Python-ideas] async objects

On Tue, Oct 4, 2016 at 8:32 AM, Greg Ewing wrote: > Yann Kaiser wrote: >> >> The way I see it, the great thing about async/await as opposed to >> threading is that it is explicit about when execution will "take a break" >> from your function or resume into it. > > > Another thing is that async/awa

Re: [Python-ideas] async objects

On 4 October 2016 at 10:48, C Anthony Risinger wrote: > In Go I can spawn a new control state (goroutine) at any time against any > function. This is clear in the code. In Erlang I can spawn a new control > state (Erlang process) at any time and it's also clear. Erlang is a little > different beca

Re: [Python-ideas] async objects

Yann Kaiser wrote: The way I see it, the great thing about async/await as opposed to threading is that it is explicit about when execution will "take a break" from your function or resume into it. Another thing is that async/await tasks are very lightweight compared to OS threads, so you can a

Re: [Python-ideas] async objects

On Oct 3, 2016 7:09 PM, "Stephen J. Turnbull" < turnbull.stephen...@u.tsukuba.ac.jp> wrote: > > Rene Nejsum writes: > > > I believe that you should be able to code concurrent code, without > > being to explicit about it, but let the runtime handle low-level > > timing, as long as you know your c

Re: [Python-ideas] async objects

Rene Nejsum writes: > I believe that you should be able to code concurrent code, without > being to explicit about it, but let the runtime handle low-level > timing, as long as you know your code will execute in the intended > order. Isn't "concurrent code whose order of execution you know" a

Re: [Python-ideas] async objects

Hi Yann/ > On 03 Oct 2016, at 17:46, Yann Kaiser wrote: > > The way I see it, the great thing about async/await as opposed to threading > is that it is explicit about when execution will "take a break" from your > function or resume into it. This is made clear and readable through the use > o

Re: [Python-ideas] async objects

On 2016-10-03 16:46, Yann Kaiser wrote: The way I see it, the great thing about async/await as opposed to threading is that it is explicit about when execution will "take a break" from your function or resume into it. This is made clear and readable through the use of `await` keywords. Your prop

Re: [Python-ideas] async objects

The way I see it, the great thing about async/await as opposed to threading is that it is explicit about when execution will "take a break" from your function or resume into it. This is made clear and readable through the use of `await` keywords. Your proposal unfortunately goes directly against t

Re: [Python-ideas] async objects

On 3 October 2016 at 15:52, Giampaolo Rodola' wrote: > Independently from what the proposed solution is, I think you raised a very > valid concern: the DRY principle. > Right now the stdlib has tons of client network libraries which do not > support the new async model. > As such, library vendors

Re: [Python-ideas] async objects

Independently from what the proposed solution is, I think you raised a very valid concern: the DRY principle. Right now the stdlib has tons of client network libraries which do not support the new async model. As such, library vendors will have to rewrite them by using the new syntax and provide an

[Python-ideas] async objects

Having followed Yury Selivanov yselivanov.ml at gmail.com proposal to add async/await to Python (PEP 492 Coroutines with async and await syntax and (PEP 525 Asynchronous Generators) and and especially the discussion about PEP 530: Asynchronous Comprehensions I would like t