[Python-ideas] Re: Extract variable name from itself

[Rene Nejsum]

A little late, but the requirement to "Extract variable name from itself" is 
widely used in Django. Each field in a Django model, knows it's own variable 
name

user_name = models.CharField(...)

The instance of CharField knows that it's variable name is "user_name", so that 
it can name the row in the database the same.

On startup Django runs through the Model class searching for instances of Field 
and gets the name that way
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/SJUKWXOJY6CLPAVTH4NZHDNFXND6USDM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: channel or message queue in asyncio

[Rene Nejsum]

Sound like a good idea. 

Would be supernice if the channel could receive (ch.get()) multiple types of 
events like: network messages (socket), UI input (mouse and keyboard events), 
file events (select?), timeouts, kernel events (shutdown) and signals besides 
internal messages (ch.put) 

br
/Rene
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/35LMAHKRJTBJOGLBVBA4Q34RZXBDK255/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] f-string "debug" conversion

[Rene Nejsum]

+1 for this, I would use it all the time for debugging and tracing programs

breakpoints and IDE’s can be nice, but my code is filled with lines like:

logger.debug(f“transaction_id={transaction_id}, state={state}, 
amount={amount}, etc={etc}”)

So yeah, well +10 actually :-)

/rene
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Civility on this mailing list

[Rene Nejsum]

Dear Brett/

I have been reading the python-idea archive from time to time over the past 
years and I joined the list about a month ago to promote my “crazy” async 
object idea. I did fear the response to a newcomer with an unlikely idea, but I 
must say the *everyone* has been extremely nice, writing often long answer to 
discussions and trying to understand where I’m coming from with this idea. And 
it definitely made me try to think a little extra before sending responses …

I did also raise an eye-brow when reading some of the comments in the thread 
you mentioned, they seam a little out of touch with my experience on other 
threads here.

Hope some time off will do you good, my best advice to you and others is 
something that have helped me, in similar situations is the old saying  “Other 
peoples opinion of you, are none of your business” :-) It took me some years to 
get it, but now it helps me every time i get worked up about something another 
person says to me or about me.

best
/Rene



> On 17 Oct 2016, at 20:29, Brett Cannon  wrote:
> 
> 
> Based on some emails I read in the " unpacking generalisations for list 
> comprehension", I feel like I need to address this entire list about its 
> general behaviour.
> 
> If you don't follow me on Twitter you may not be aware that I am taking the 
> entire month of October off from volunteering any personal time on Python for 
> my personal well-being (this reply is being done on work time for instance). 
> This stems from my wife pointing out that I had been rather stressed in July 
> and August outside of work in relation to my Python volunteering (having your 
> weekends ruined is never fun). That stress stemmed primarily from two rather 
> bad interactions I had to contend with on the issue track in July and August 
> ... and this mailing list.
> 
> When I have talked to people about this mailing list it's often referred to 
> by others as the "wild west" of Python development discussions (if you're not 
> familiar with US culture, that turn of phrase basically means "anything 
> goes"). To me that is not a compliment. When I created this list with Titus 
> the goal was to provide a safe place where people could bring up ideas for 
> Python where people could quickly provide basic feedback so people could know 
> whether there was any chance that python-dev would consider the proposal. 
> This was meant to be a win for proposers by not feeling like they were 
> wasting python-dev's time and a win for python-dev by keeping that list 
> focused on the development of Python and not fielding every idea that people 
> want to propose.
> 
> And while this list has definitely helped with the cognitive load on 
> python-dev, it has not always provided a safe place for people to express 
> ideas. I have seen people completely dismiss people's expertise and opinion. 
> There has been name calling and yelling at people (which is always 
> unnecessary). There have been threads that have completely derailed itself 
> and gone entirely off-topic. IOW I would not hold this mailing list up as an 
> example of the general discourse that I experience elsewhere within the 
> community.
> 
> Now I realize that we are all human beings coming from different cultural 
> backgrounds and lives. We all have bad days and may not take the time to stop 
> and think about what we are typing before sending it, leading to emails that 
> are worded in a way that can be hurtful to others. It's also easy to forget 
> that various cultures views things differently and so that can lead to people 
> "reading between the lines" a lot and picking up things that were never 
> intended. There are 1,031 people on this mailing list from around the world 
> and it's easy to forget that e.g. Canadian humour may not translate well to 
> Ukrainian culture (or something). What this means is it's okay to nicely say 
> that something bothered you, but also try to give people the benefit of the 
> doubt as you don't know what their day had been like before they wrote that 
> email (I personally don't like the "just mute the thread" approach to dealing 
> with bad actors when the muting is silent as that doesn't help new people who 
> join this mailing list and the first email they see is someone being rude 
> that everyone else didn't see because they muted the thread days ago).
> 
> As for the off-topic threads, please remember there are 1,031 people on this 
> mailing list (this doesn't count people reading through gmane or Google 
> Groups). Being extremely generous and assuming every person on this list only 
> spends 10 seconds deciding if they care about your email, that's still nearly 
> 3 hours of cumulative time spent on your email. So please be cognisant when 
> you reply, and if you want to have an off-topic conversation, please take it 
> off-list.
> 
> And finally, as one of the list administrators I am in a position of power 
> when it comes to the rules of this list and the CoC. While 

Re: [Python-ideas] async objects

[Rene Nejsum]

Your are right about the importance of Python C API, it often goes under my 
radar. For the past 20 years I have only used it a couple of times (to 
integrate Python into some existing C-code) therefore it is not as much in 
focus as it should be and definiatly are by others.

I get your innovators dilemma all to well, just look at Python 3 and the time 
it took us to shift from 2.

But, watching Larry Hastings talk on his awesome gilectomy project, it was my 
understanding that he at least saw it as a possibility to do a backward 
compatible extension of the C-API for his GIL removal project.

As I understand he proposes that the Python runtime should check whether a 
given C-lib has been upgraded to support non-GIL, if not run it as an old 
version. I am not sure how much it will take in this case, but i thought “hey, 
if Larry Hastings is removing the GIL and proposing an extension to the C-api, 
at least it can be done” :-)

/Rene

> On 17 Oct 2016, at 20:39, Nathaniel Smith  wrote:
> 
> The problem is that if your goal is to make a practical proposal, it's
> not enough to look at Python-the-language. You're absolutely right,
> AFAICT there's nothing stopping someone from making a nice
> implementation of Python-the-language that has erlang-style cheap
> shared-nothing threads with some efficient message-passing mechanism.
> 
> But! It turns out that unless your new implementation supports the
> CPython C API, then it's almost certainly not viable as a mainstream
> CPython alternative, because there's this huge huge pile of libraries
> that have been written against that C API. You're not competing
> against CPython, you're competing against CPython+thousands of
> libraries that you don't have and that your users expect. And
> unfortunately, it turns out that the C API locks in a bunch of the
> implementation assumptions (refcounting, the GIL, use of the C stack,
> poor support for isolation between different interpreter states, ...)
> that you were trying to get away from.
> 
> I mean, in many ways it's a good problem to have, that our current
> ecosystem is just so attractive that it's hard to compete with!
> (Though a pessimist could point out that this difficulty with
> competing with yourself is exactly what tends to eventually undermine
> incumbents -- cf. the innovator's dilemma.) And it's "just" a matter
> of implementation, not Python-the-language itself. But the bottom line
> is: this is *the* core problem that you have to grapple with if you
> want to make any radical improvements in the Python runtime and have
> people actually use them.
> 
> -n
> 
> On Mon, Oct 17, 2016 at 9:36 AM, Rene Nejsum  wrote:
>> Regarding the Python C-runtime and async, I just had a good talk with
>> Kresten Krab at Trifork. He implemented “Erjang” the Java implementation of
>> the Erlang VM (www.erjang.org). Doing this he had access to the Erlang (C)
>> VM.
>> 
>> It turn’s out that the Erlang VM and the Python VM has a lot of similarities
>> and the differences are more in the language, than in the VM
>> 
>> Differences between the Erlang VM and Python related to async are:
>> 
>> 1) Most variables in Erlang are immutable
>> Making it easier to have coroutines
>> 
>> 2) coroutines are built into the Erlang using the “spawn” keyword
>> Leaving the specific implementation to the VM, but never implemented with OS
>> threads.
>> 
>> 3) All coroutines have their own heap and stack (initially 200 bytes), but
>> can grow as needed
>> 
>> 4) coroutines are managed in “ready-queue”, from which the VM thread
>> executes the next ready job
>> Each job gets 2000 “instructions” (or until IO block) and the next coroutine
>> is executed
>> 
>> Because of this, when multicore CPU’s entered the game, it was quite easy to
>> change the Erlang VM to add a thread per core to pull from the ready-queue.
>> This makes an Erlang program run twice as fast (almost) every time the
>> number of cores are doubled!
>> 
>> Given this, I am still convinced that:
>> 
>> obj = async SomeObject()
>> 
>> should be feasible, even though there will be some “golang” like issues
>> about shared data, but there could be several ways to handle this.
>> 
>> br
>> /Rene
>> 
>> 
>> On 05 Oct 2016, at 18:06, Nick Coghlan  wrote:
>> 
>> On 5 October 2016 at 16:49, Rene Nejsum  wrote:
>> 
>> On 04 Oct 2016, at 18:40, Sven R. Kunze  wrote:
>> I don't think that's actually what I wanted here. One simple keyword should
>> have sufficed just like golang did. So, the developer gets a way to dec

Re: [Python-ideas] async objects

[Rene Nejsum]

Regarding the Python C-runtime and async, I just had a good talk with Kresten 
Krab at Trifork. He implemented “Erjang” the Java implementation of the Erlang 
VM (www.erjang.org <http://www.erjang.org/>). Doing this he had access to the 
Erlang (C) VM.

It turn’s out that the Erlang VM and the Python VM has a lot of similarities 
and the differences are more in the language, than in the VM

Differences between the Erlang VM and Python related to async are:

1) Most variables in Erlang are immutable 
Making it easier to have coroutines

2) coroutines are built into the Erlang using the “spawn” keyword
Leaving the specific implementation to the VM, but never implemented with OS 
threads.

3) All coroutines have their own heap and stack (initially 200 bytes), but can 
grow as needed

4) coroutines are managed in “ready-queue”, from which the VM thread executes 
the next ready job
Each job gets 2000 “instructions” (or until IO block) and the next coroutine is 
executed

Because of this, when multicore CPU’s entered the game, it was quite easy to 
change the Erlang VM to add a thread per core to pull from the ready-queue. 
This makes an Erlang program run twice as fast (almost) every time the number 
of cores are doubled!

Given this, I am still convinced that:

obj = async SomeObject()

should be feasible, even though there will be some “golang” like issues about 
shared data, but there could be several ways to handle this.

br
/Rene


> On 05 Oct 2016, at 18:06, Nick Coghlan  wrote:
> 
> On 5 October 2016 at 16:49, Rene Nejsum  wrote:
>>> On 04 Oct 2016, at 18:40, Sven R. Kunze  wrote:
>>> I don't think that's actually what I wanted here. One simple keyword should 
>>> have sufficed just like golang did. So, the developer gets a way to decide 
>>> whether or not he needs it blocking or nonblocking **when using a 
>>> function**. He doesn't need to decide it **when writing the function**.
>> 
>> I agree, that’s why i proposed to put the async keyword in when creating the 
>> object, saying in this instance I want asynchronous communication with the 
>> object.
> 
> OK, I think there may be a piece of foundational knowledge regarding
> runtime design that's contributing to the confusion here.
> 
> Python's core runtime model is the C runtime model: threads (with a
> local stack and access to a global process heap) and processes (which
> contain a heap and one or more threads). Anything else we do (whether
> it's generators, coroutines, or some other form of paused execution
> like callback management) gets layered on top of that runtime model.
> When folks ask questions like "Why can't Python be more like Go?",
> "Why can't Python be more like Erlang?", or "Why can't Python be more
> like Rust?" and get a negative response, it's usually because there's
> an inherent conflict between the C runtime model and whatever piece of
> the Go/Erlang/Rust runtime model we want to steal.
> 
> So the "async" keyword in "async def", "async for" and "async with" is
> essentially a marker saying "This is not a C-like runtime concept
> anymore!" (The closest C-ish equivalent I'm aware of would be Apple's
> Grand Central Dispatch in Objective-C and that shows many of the
> async/await characteristics also seen in Python and C#:
> https://www.raywenderlich.com/60749/grand-central-dispatch-in-depth-part-1
> )
> 
> Go (as with Erlang before it) avoided these problems by not providing
> C-equivalent functions in the first place. Accordingly, *every* normal
> function defined in Go can also be used as a goroutine, rather than
> needing to be a distinct type - their special case is defining
> functions that interoperate with external C libraries. Python (along
> with other languages built on the C runtime model like C# and
> Objective-C) doesn't have that luxury - we need to distinguish
> coroutines from regular functions, since we can't just handle them
> according to the underlying C runtime model any more.
> 
> Guido's idea of a shadow thread to let synchronous threads run
> coroutines without needing to actually run a foreground event loop
> should provide a manageable way of getting the two runtime models
> (traditional C and asynchronous coroutines) to play nicely together in
> a single application, and has the virtue of being something folks can
> readily experiment with for themselves before we commit to anything
> specific in the standard library (since all the building blocks of
> thread local storage, event loop management, and inter-thread message
> passing primitives are already available).
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 06 Oct 2016, at 07:15, Stephen J. Turnbull 
>  wrote:
> 
> Nick Coghlan writes:
> 
>> Python's core runtime model is the C runtime model: threads (with a
>> local stack and access to a global process heap) and processes (which
>> contain a heap and one or more threads). Anything else we do (whether
>> it's generators, coroutines, or some other form of paused execution
>> like callback management) gets layered on top of that runtime model.
>> When folks ask questions like "Why can't Python be more like Go?",
>> "Why can't Python be more like Erlang?", or "Why can't Python be more
>> like Rust?" and get a negative response, it's usually because there's
>> an inherent conflict between the C runtime model and whatever piece of
>> the Go/Erlang/Rust runtime model we want to steal.
> 
> How can there be a conflict between Python implementing the C runtime
> model *itself* which says "you can do anything anywhere anytime", and
> some part of Python implementing the more restricted models that allow
> safe concurrency?  If you can do anything, well, you can voluntarily
> submit to compiler discipline to a restricted set.  No?  So it must be
> that the existing constructions (functions, for, with) that need an
> "async" marker have an implementation that is itself unsafe.  This
> need is not being explained very well.  What is also not being
> explained is what would be lost by simply using the "safe"
> implementations generated by the async versions everywhere.

Agree, well put. The Erlang runtime (VM) is also written in C, so anything 
should be possible. 

I do not advocate that Python should be a “new” Erlang or Go, just saying that 
since we are introducing some level of concurrency in Python that we look at 
some of the elegant ways others have achieved this and try to implement 
something like that in Python.

> These may be hard to explain, and I know you, Yury, and Guido are very
> busy.  But it's frustrating for all to see this go around in a circle:
> "it's like it is because it has to be that way, so that's the way it is”.

I understand that there is a lot of backwards compatibility, especially in 
regards to the Python/C interface, but I think that it is possible to find an 
elegant solution to this.

> There's also the question of "is async/await really a language
> feature, or is it patching up a deficiency in the CPython
> implementation that other implementations don't necessarily have?" 
> (which has been brought up before, in less contentious terms).
> 
>> So the "async" keyword in "async def", "async for" and "async with" is
>> essentially a marker saying "This is not a C-like runtime concept
>> anymore!"
> 
> That's understood, of course.  The question that isn't being answered
> well is "why can't that non-C-like runtime concept be like Go or
> Erlang or Rust?"  Or, less obtusely, "what exactly is the 'async'
> runtime concept, and why is it preferred to the concepts implemented
> by Go or Erlang or Rust or gevent or greenlets or Stackless?”

This would be very interesting to understand.

> I guess the answer to "why not Stackless?" is buried in the archives
> for Python-Dev somewhere, but I need to get back to $DAYJOB, maybe
> I'll look it up later.

I will try to look for that, I have some time on my hands, not sure I have have 
the %BRAINSKILL, but never the less…

br
/Rene

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 05 Oct 2016, at 21:20, Sven R. Kunze  wrote:
> 
> On 05.10.2016 18:06, Nick Coghlan wrote:
>> [runtime matters]
> 
> I think I understand your point.
> 
> I also hope that others and me could provide you with our perspective. We see 
> Python not as a C-like runtime but as an abstract modelling language. I know 
> that it's different from the point of view of CPython internals, however from 
> the outside Python suggests to be much more than a simple wrapper around C. 
> Just two different perspectives.

Excellent point. For me CPython, Jython, IronPython, PyPy are the same (99.9%) 
and the important part is the Python the language. For a long time I tested 
PYWORKS again all implementations and were happy that it ran on all. Clearly, 
for others CPython (incl. runtime and C-bindings) is the fact and the others 
are far from the same, especially because the missing C-integration.

But, are the runtimes for Python and Erlang that fundamentally different? Is it 
Python’s tight integration with C that is the big difference? 

When I first read about the async idea, I initially expected that it would be 
some stackless like additions to Python. My wish for Python was an addition to 
the language the allowed an easy an elegant concurrent model on the language 
level. Ideally a Python program with 1000 async objects parsing a 10TB XML 
in-memory file, should run twice as fast on a 8-core CPU, compared to  a 4-core 
ditto. 


> Unfortunately, your runtime explanations still don't address the DRY issue. 
> :-/
> 
>> Guido's idea of a shadow thread to let synchronous threads run
>> coroutines without needing to actually run a foreground event loop
>> should provide a manageable way of getting the two runtime models
>> (traditional C and asynchronous coroutines) to play nicely together in
>> a single application, and has the virtue of being something folks can
>> readily experiment with for themselves before we commit to anything
>> specific in the standard library (since all the building blocks of
>> thread local storage, event loop management, and inter-thread message
>> passing primitives are already available).
> 
> I needed to think about this further when Guido mentioned it. But I like it 
> now.
> 
> If you check https://github.com/srkunze/fork/tree/asyncio , I already started 
> working on integrating asyncio into xfork at long time ago. But I still 
> couldn't wrap my mind around it and it stalled. But IIRC, I would have 
> implemented a shadow thread solution as well. So, if his idea goes into the 
> stdlib first, I welcome it even more as it would do the heavy lifting for me. 
> xfork would then be just a common interface to threads, processes and 
> coroutines.

xfork (as pyworks) implements a proxy object, which “almost” behaves like the 
real object, but it is still a proxy. If fork (or spawn, chan, async, 
whatever.) was a part of the language it would be more clean.

br
/Rene


> 
> Cheers,
> Sven
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 04 Oct 2016, at 18:15, Guido van Rossum  wrote:
> 
> On Tue, Oct 4, 2016 at 4:30 AM, Nick Coghlan  wrote:
>>class SomeClass(object):
>>def some_sync_method(self):
>>return 42
>>async def some_async_method(self):
>>await asyncio.sleep(3)
>>return 42
>> 
>> o = auto_schedule(SomeClass())  # Indicating that the user wants an
>> async version of the object
>> r1 = o.some_sync_method() # Automatically run in a background thread
>> r2 = o.some_async_method() # Automatically scheduled as a coroutine
>> print(run_in_foreground(r1))
>> print(run_in_foreground(r2))
> 
> So maybe r1 and r2 are just concurrent.futures.Futures, and
> run_in_foreground(r) wraps r.result(). And auto_schedule() is a proxy
> that turns all method calls into async calls with a (concurrent)
> Future to wait for the result. There's an event loop somewhere that
> sits idle except when you call run_in_foreground() on somethong; it's
> only used for the async methods, since the sync methods run in a
> background thread (pool, I hope). Or perhaps r2 is an asyncio.Future
> and run_in_foreground(r2) wraps loop.run_until_complete(r2). I suppose
> the event loop should also be activated when waiting for r1, so maybe
> r1 should be an asyncio Future that wraps a concurrent Future (using
> asyncio.wrap_future(), which can do just that thing).
> 
> Honestly it feels like many things can go wrong with this API model,
> esp. you haven't answered what should happen when a method of
> SomeClass (either a synchronous one or an async one) calls
> run_in_foreground() on something -- or, more likely, calls some
> harmless-looking function that calls another harmless-looking function
> that calls run_in_foreground(). At that point you have pre-emptive
> scheduling back in play (or your coroutines may be blocked
> unnecessarily) and I think you have nothing except a more complicated
> API to work with threads.

I am a little out on deep water here, but I think that if an object instance 
was guaranteed - by Python runtime - to run in one coroutine/thread and only 
the message passing of method call and return values was allowed to pass 
between coroutine/thread context, then at least all local instance variable 
reference would be fine?


> I think I am ready to offer a counterproposal where the event loop
> runs in one thread and synchronous code runs in another thread and we
> give the synchronous code a way to synchronously await a coroutine or
> an asyncio.Future. This can be based on
> asyncio.run_coroutine_threadsafe(), which takes a coroutine or an
> asyncio.Future and returns a concurrent Future. (It also takes a loop,
> and it assumes that loop runs in a different thread. I think it should
> assert that.)
> 
> The main feature of my counterproposal as I see it is that async code
> should not call back into synchronous code, IOW once you are writing
> coroutines, you have to use the coroutine API for everything you do.
> And if something doesn't have a coroutine API, you run it in a
> background thread using loop.run_in_executor().
> 
> So either you buy into the async way of living and it's coroutines all
> the way down from there, no looking back -- or you stay on the safe
> side of the fence, and you interact with coroutines only using a very
> limited "remote manipulator" API. The two don't mix any better than
> that.

Maybe not, but I am hoping for something better :-)

> 
> -- 
> --Guido van Rossum (python.org/~guido)
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 04 Oct 2016, at 18:40, Sven R. Kunze  wrote:
> 
> On 04.10.2016 09:50, Stephen J. Turnbull wrote:
>> As I understand the main point, Sven and Rene don't believe that [the
>> kind of] async code [they want to write] should need any keywords;
>> just start the event loop and invoke functions, and that somehow
>> automatically DTRTs.
> [reading my name second time]
> 
> 
> I don't think that's actually what I wanted here. One simple keyword should 
> have sufficed just like golang did. So, the developer gets a way to decide 
> whether or not he needs it blocking or nonblocking **when using a function**. 
> He doesn't need to decide it **when writing the function**.

I agree, that’s why i proposed to put the async keyword in when creating the 
object, saying in this instance I want asynchronous communication with the 
object.

> You might wonder why this is relevant. DRY principle has been mentioned but 
> there's more to it. Only the caller **can decide** whether it needs to wait 
> or not. Why? Because, the caller works WITH the result of the called function 
> (whatever results means to you). The caller is (what Nick probably would 
> call) the orchestrator, as it has the knowledge about the relation and 
> interaction between domain-specific function calls.

+1

> As a result of past discussions, I wrote the module "xfork" which basically 
> does this "golang goroutine" stuff. It's just a thin wrapper around "futures" 
> but it allows to avoid that what René and Anthony objects about.

I had a look at xfork, and really like it. It is implemented much like the 
lower level of PYWORKS and PYWORKS could build on xfork instead. 

I think that the “model” of doing async should be defined in the Python 
language/runtime (like in Go, Erlang, ABCL) . I the ideal case it should be up 
to the runtime implementation (CPython, PyPy, Jython, IronPython etc.) how the 
asynchronous behaviour is implemented (greenlets, threads, roll-it-own, etc)

br
/Rene

> 
> Cheers,
> Sven
> 
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 04 Oct 2016, at 07:26, Chris Angelico  wrote:
> 
> On Tue, Oct 4, 2016 at 4:25 PM, Rene Nejsum  wrote:
>>> On 04 Oct 2016, at 02:09, Stephen J. Turnbull 
>>>  wrote:
>>> 
>>> Rene Nejsum writes:
>>>> I believe that you should be able to code concurrent code, without
>>>> being to explicit about it, but let the runtime handle low-level
>>>> timing, as long as you know your code will execute in the intended
>>>> order.
>>> 
>>> Isn't "concurrent code whose order of execution you know" an oxymoron?
>> 
>> You are right, I should have been more specific. What I ment was that I 
>> don’t need code filled with async/await, I don’t care where it blocks, as 
>> long as it (the specific code block iI am looking at) runs in the order i 
>> wrote it :-)
>> 
> 
> Then you want threads. Easy!

Well, yes and no. I other languages (Java/C#) where I have implemented 
concurrent objects ala PYWORKS it works pretty well, as long as you have less 
than maybe 10.000 threads

But, in Python (CPython2 on multicore CPU) threads does not work! The GIL makes 
it impossible to have for example 100 threads sending messages between each 
other (See the Ring example in PYWORKS), that’s one reason why it would be 
interesting to have some kind of concurrency support built into the Python 
runtime.

Today I see all kinds of tricks and workarounds to get around the GIL. Raging 
from starting several Python interpreters to difficult to read code using yield 
(now async/await), but when you have seen much more elegant support (Go, 
Erlang, maybe even ABCL) you kind of wish this could be added to you own 
favourite language.

br
/Rene

> 
> ChrisA
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 03 Oct 2016, at 23:32, Greg Ewing  wrote:
> 
> Yann Kaiser wrote:
>> The way I see it, the great thing about async/await as opposed to threading 
>> is that it is explicit about when execution will "take a break" from your 
>> function or resume into it.
> 
> Another thing is that async/await tasks are very lightweight
> compared to OS threads, so you can afford to have a large
> number of them active at once.
> 
> Rene's approach seems to be based on ordinary threads, so
> it would not have this property.

My implementation is, but it should not (have to) be, it only reflects my 
limited ability and time :-)

The programmer should not need to be aware of where concurrency is achieved 
though coroutines or threads, ideally there should be one OS thread per core in 
the CPU running many (millions) of coroutines…

br
/Rene

> 
> -- 
> Greg
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 04 Oct 2016, at 02:48, C Anthony Risinger  wrote:
> 
> On Oct 3, 2016 7:09 PM, "Stephen J. Turnbull" 
>  <mailto:turnbull.stephen...@u.tsukuba.ac.jp>> wrote:
> >
> > Rene Nejsum writes:
> >
> >  > I believe that you should be able to code concurrent code, without
> >  > being to explicit about it, but let the runtime handle low-level
> >  > timing, as long as you know your code will execute in the intended
> >  > order.
> >
> > Isn't "concurrent code whose order of execution you know" an oxymoron?
> 
> They are referring to the synchronous nature of any independent control 
> state. Whether it's a thread, a coroutine, a continuation, or whatever else 
> doesn't really matter much. When a thing runs concurrently along side other 
> things, it's still synchronous with respect to itself regardless of how many 
> context switches occur before completion. Such things only need mechanisms to 
> synchronize in order to cooperate.
> 
I agree 100%. Ideally I think a language (would love it to be Python) should 
permit many (millions) of what we know as coroutines and then have as many 
threads as the CPU have cores to execute this coroutines, but I do not thing 
you as a programmer should be especially aware of this as you code. (Just like 
GC handles your alloc/free, the runtime should handle your “concurrency”)
> People want to know how they are suppose to write unified, 
> non-insane-and-ugly code in this a/sync python 2/3 world we now find 
> ourselves in. I've been eagerly watching this thread for the answer, thus far 
> to no avail.
> 
Agree
> Sans-io suggests we write bite-sized synchronous code that can be driven by 
> a/sync consumers. While this is all well and good, how does one write said 
> consuming library for both I/O styles without duplication?
> 
> The answer seems to be "write everything you ever wanted as async and throw 
> some sync wrappers around it". Which means all the actual code I write will 
> be peppered with async and await keywords.
> 
Have a look at the examples in David Beazley’s curio, he is one of the most 
knowable Python people I have met, but that code is almost impossible to read 
and understand.
> In Go I can spawn a new control state (goroutine) at any time against any 
> function. This is clear in the code. In Erlang I can spawn a new control 
> state (Erlang process) at any time and it's also clear. Erlang is a little 
> different because it will preempt me, but the point is I am simply choosing a 
> target function to run in a new context. Gevent and even threading module is 
> another example of this pattern.
> 
Having thought some more about it, I think that putting async i front of the 
object, could be kind of a channel i Go and other languages?

> In all reality you don't typically need many suspension points other than 
> around I/O, and occasionally heavy CPU, so I think folks are struggling to 
> understand (I admit, myself included) why the runtime doesn't want to be more 
> help and instead punts back to the developer.
> 
Well put, we are definitely on the same page here, thank you.

br
/Rene


> --
> 
> C Anthony
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

> On 04 Oct 2016, at 02:09, Stephen J. Turnbull 
>  wrote:
> 
> Rene Nejsum writes:
>> I believe that you should be able to code concurrent code, without
>> being to explicit about it, but let the runtime handle low-level
>> timing, as long as you know your code will execute in the intended
>> order.
> 
> Isn't "concurrent code whose order of execution you know" an oxymoron?

You are right, I should have been more specific. What I ment was that I don’t 
need code filled with async/await, I don’t care where it blocks, as long as it 
(the specific code block iI am looking at) runs in the order i wrote it :-)

br
/Rene


> 
> 

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] async objects

[Rene Nejsum]

Hi Yann/

> On 03 Oct 2016, at 17:46, Yann Kaiser  wrote:
> 
> The way I see it, the great thing about async/await as opposed to threading 
> is that it is explicit about when execution will "take a break" from your 
> function or resume into it. This is made clear and readable through the use 
> of `await` keywords.

The way I read this argument, a parallel could be “the great thing about 
alloc/free is that it is explicit about when allocation will happen”, but I 
believe that the more control you can leave to the runtime the better.

> Your proposal unfortunately goes directly against this idea of explicitness. 
> You won't know what function will need to be fed into an event loop or not. 
> You won't know where your code is going to lose or gain control.

I believe that you should be able to code concurrent code, without being to 
explicit about it, but let the runtime handle low-level timing, as long as you 
know your code will execute in the intended order.

br
/Rene


> 
> On Sun, Oct 2, 2016, 14:26 Rene Nejsum  <mailto:r...@stranden.com>> wrote:
> Having followed Yury Selivanov yselivanov.ml <http://yselivanov.ml/> at 
> gmail.com <http://gmail.com/> proposal to add async/await to Python  (PEP 492 
> Coroutines with async and await syntax and (PEP 525  Asynchronous Generators) 
> and and especially the discussion about PEP 530: Asynchronous Comprehensions  
> I would like to add some concerns about the direction Python is taking on 
> this.
> 
> As Sven R. Kunze srkunze at mail.de <http://mail.de/> mentions the is a risk 
> of having to double a lot of methods/functions to have an Async 
> implementation. Just look at the mess in .NET  when Microsoft introduced 
> async/await in their library, a huge number of functions had to be 
> implemented with a Async version of each member. Definitely not the DRY 
> principle.
> 
> While I think parallelism and concurrency are very important features in a 
> language, I feel the direction Python is taking right now is getting to 
> complicated, being difficult to understand and implement correct.
> 
> I thought it might be worth to look at using async at a higher level. Instead 
> of making methods, generators and lists async, why not make the object itself 
> async? Meaning that the method call (message to object) is async
> 
> Example:
> 
> class SomeClass(object):
>def some_method(self):
>return 42
> 
> o = async SomeClass()  # Indicating that the user want’s an async version of 
> the object
> r = o.some_method()# Will implicit be a async/await “wrapped” method no 
> matter impl.
> # Here other code could execute, until the result (r) is referenced
> print r
> 
> I think above code is easier to implement, use and understand, while it 
> handles some of the use cases handled by defining a lot of methods as 
> async/await.
> 
> I have made a small implementation called PYWORKS 
> (https://github.com/pylots/pyworks <https://github.com/pylots/pyworks>), 
> somewhat based on the idea above. PYWORKS has been used in several real world 
> implementation and seams to be fairly easy for developers to understand and 
> use.
> 
> br
> /Rene
> 
> PS. This is my first post to python-ideas, please be gentle :-)
> ___
> Python-ideas mailing list
> Python-ideas@python.org <mailto:Python-ideas@python.org>
> https://mail.python.org/mailman/listinfo/python-ideas 
> <https://mail.python.org/mailman/listinfo/python-ideas>
> Code of Conduct: http://python.org/psf/codeofconduct/ 
> <http://python.org/psf/codeofconduct/>
> -- 
> Yann Kaiser
> kaiser.y...@gmail.com <mailto:kaiser.y...@gmail.com>
> yann.kai...@efrei.net <mailto:yann.kai...@efrei.net>
> +33 6 51 64 01 89
> https://github.com/epsy <https://github.com/epsy>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] async objects

[Rene Nejsum]

Having followed Yury Selivanov yselivanov.ml at gmail.com  
proposal to add async/await to Python  (PEP 492 Coroutines with async and await 
syntax and (PEP 525  Asynchronous Generators) and and especially the discussion 
about PEP 530: Asynchronous Comprehensions  I would like to add some concerns 
about the direction Python is taking on this.

As Sven R. Kunze srkunze at mail.de  mentions the is a risk of 
having to double a lot of methods/functions to have an Async implementation. 
Just look at the mess in .NET  when Microsoft introduced async/await in their 
library, a huge number of functions had to be implemented with a Async version 
of each member. Definitely not the DRY principle.

While I think parallelism and concurrency are very important features in a 
language, I feel the direction Python is taking right now is getting to 
complicated, being difficult to understand and implement correct.

I thought it might be worth to look at using async at a higher level. Instead 
of making methods, generators and lists async, why not make the object itself 
async? Meaning that the method call (message to object) is async

Example:

class SomeClass(object):
   def some_method(self):
   return 42

o = async SomeClass()  # Indicating that the user want’s an async version of 
the object
r = o.some_method()# Will implicit be a async/await “wrapped” method no 
matter impl.
# Here other code could execute, until the result (r) is referenced
print r

I think above code is easier to implement, use and understand, while it handles 
some of the use cases handled by defining a lot of methods as async/await.

I have made a small implementation called PYWORKS 
(https://github.com/pylots/pyworks ), 
somewhat based on the idea above. PYWORKS has been used in several real world 
implementation and seams to be fairly easy for developers to understand and use.

br
/Rene

PS. This is my first post to python-ideas, please be gentle :-)___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/