[Python-ideas] Re: asyncio: return from multiple coroutines
I believe asyncio.wait() with "return_when=FIRST_COMPLETED" would
perform the functionality you're looking for with the
"asyncio.on_first_return()". For details on the functionality of
asyncio.wait(), see
https://docs.python.org/3/library/asyncio-task.html#asyncio.wait.
> I understand that I can create two coroutines that call the same function,
> but it would be much cleaner (because of implementation issues) if I can
> simply create a coroutine that yields the result of whichever connection
> arrives first.
You can use an asynchronous generator that will continuously yield the
result of the first recv() that finishes (I'm assuming you mean
"yields" literally and want multiple results from a generator, but I
might be misinterpreting that part).
Here's a brief example, using the recv() coroutine function from the
pastebin linked:
```
import asyncio
import random
async def recv(message: str, max_sleep: int):
sleep_time = max_sleep * random.random()
await asyncio.sleep(sleep_time)
return f'{message} awaited for {sleep_time:.2f}s'
async def _start():
while True:
msgs = [
asyncio.create_task(recv("Messager 1", max_sleep=1)),
asyncio.create_task(recv("Messager 2", max_sleep=1))
]
done, _ = await asyncio.wait(msgs,
return_when=asyncio.FIRST_COMPLETED)
result = done.pop()
yield await result
async def main():
async for result in _start():
print(result)
asyncio.run(main())
```
Note that in the above example, in "msgs", you can technically pass
the coroutine objects directly to asyncio.wait(), as they will be
implicitly converted to tasks. However, we decided to deprecate that
functionality in Python 3.8 since it can be rather confusing. So
creating and passing the tasks is a better practice.
> Again, it's quite likely I am not seeing something obvious, but I didn't know
> where else to ask.
If you're not mostly certain or relatively inexperienced with the
specific area that the question pertains to, I'd recommend asking on
python-list first (or another Python user community). python-ideas is
primarily intended for new feature proposals/suggestions. Although if
you've tried other resources and haven't found an answer, it's
perfectly fine to ask a question as part of the suggestion post.
On Mon, Jun 22, 2020 at 6:24 PM Pablo Alcain wrote:
>
> Hey everyone. I have been looking into asyncio lately, and even though I have
> had my fair share of work, I still have some of it very shaky, so first of
> all forgive me if what I am saying here is already implemented and I totally
> missed it (so far, it looks *really* likely).
>
> Basically this is the situation: I have an application that listens on two
> websockets through the async library https://websockets.readthedocs.io/ and I
> have to perform the same function on the result, no matter where the message
> came from. I understand that I can create two coroutines that call the same
> function, but it would be much cleaner (because of implementation issues) if
> I can simply create a coroutine that yields the result of whichever
> connection arrives first.
>
> I have implemented a rather cumbersome solution with async Queues (as I would
> do in threading), in which each connection puts its message in a queue and an
> adapter class awaits the first element of the queue on "receive". Here I
> attach a pastebin with the minimal working example:
> https://pastebin.com/BzaxRbtF
>
> However, it looks like a more async-friendly solution should exist, something
> like
>
> ```
> async def _start():
> msg1 = recv("Messager 1", sleep_time=1)
> msg2 = recv("Messager 2", sleep_time=2)
> while True:
> result = await asyncio.on_first_return(msg1, msg2)
> print(result)
> ```
>
> (I understand that this implementation would not work because the event loop
> doesn't know that it is "the same task repeated", but it's just to tell you
> the general idea)
>
> Again, it's quite likely I am not seeing something obvious, but I didn't know
> where else to ask.
>
> Thank you very much,
> Pablo
>
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/XBR5QPXRBCCJELDVEWMKRBPTNG4SJM64/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/TFLM7AZWUWZKZIVJBSTQOIDFW7V3VSCV/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] asyncio: return from multiple coroutines
Hey everyone. I have been looking into asyncio lately, and even though I
have had my fair share of work, I still have some of it very shaky, so
first of all forgive me if what I am saying here is already implemented and
I totally missed it (so far, it looks *really* likely).
Basically this is the situation: I have an application that listens on two
websockets through the async library https://websockets.readthedocs.io/ and
I have to perform the same function on the result, no matter where the
message came from. I understand that I can create two coroutines that call
the same function, but it would be much cleaner (because of implementation
issues) if I can simply create a coroutine that yields the result of
whichever connection arrives first.
I have implemented a rather cumbersome solution with async Queues (as I
would do in threading), in which each connection puts its message in a
queue and an adapter class awaits the first element of the queue on
"receive". Here I attach a pastebin with the minimal working example:
https://pastebin.com/BzaxRbtF
However, it looks like a more async-friendly solution should exist,
something like
```
async def _start():
msg1 = recv("Messager 1", sleep_time=1)
msg2 = recv("Messager 2", sleep_time=2)
while True:
result = await asyncio.on_first_return(msg1, msg2)
print(result)
```
(I understand that this implementation would not work because the event
loop doesn't know that it is "the same task repeated", but it's just to
tell you the general idea)
Again, it's quite likely I am not seeing something obvious, but I didn't
know where else to ask.
Thank you very much,
Pablo
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/XBR5QPXRBCCJELDVEWMKRBPTNG4SJM64/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Please consider adding numbers.Boolean
On Mon, Jun 22, 2020 at 10:19 AM Serhiy Storchaka wrote: > The abstract class for booleans is useless, because all objects can be > used in boolean context, and all boolean operations (and, or, not) work > with arbitrary objects. So Boolean would be identical to objects. > I'm highly ambivalent about Python's model of "Truthiness" -- it's really convenient often, but also gives me a lack of control. So I think it would be helpful, in static type checking cases in particular to say something HAS to be a Bool, and only a Bool. And in that context, I don't think I'd want allow integers. Also, in the reverse, I would wan't folks to be able to use Bools when an integer or other number is expected. So in short: no, Bool should not be considered a Numeric "type" -CHB > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/RUQBDQJGDQ5MQBIWEVNULIMMB3EHOJVP/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DBMZTOHIYCM5AJOTZMSVANQM4VKCTBVH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Please consider adding numbers.Boolean
22.06.20 03:19, Neil Girdhar пише: I'm just curious if there was a reason why Boolean was omitted from the numeric tower in the numbers library? It seems that builtins.bool and numpy.bool_ would both be elements of Boolean, and Boolean itself would be Integral? The abstract class for booleans is useless, because all objects can be used in boolean context, and all boolean operations (and, or, not) work with arbitrary objects. So Boolean would be identical to objects. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RUQBDQJGDQ5MQBIWEVNULIMMB3EHOJVP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Proposal to introduce pattern matching syntax
22.06.20 18:17, nate lust пише: Matching begins by calling a __match__ (class)method on the type, with the match target as a parameter. The match method must return an object that can be evaluated as a bool. If the return value is True, the code block in this match branch is executed, and execution is passed to whatever comes after the match syntax. If __match__ returns False, execution is passed to the next match branch for testing. If a match branch contains a group of parameters, they are used in the matching process as well. If __match__ returns True, then the match target will be tested for the presence of a __unpack__ method. If there is no such method, the match target is tried as a sequence. If both of these fail, execution moves on. If there is a __unpack__ method, it is called and is expected to return a sequence. The length of the sequence (either the result of __unpack__, or the match target itself) is compared to the number of supplied arguments. If they match the sequence is unpacked into variables defined by the arguments and the match is considered a success and the body is executed. If the length of the sequence does not match the number of arguments, the match branch fails and execution continues. This is useful for differentiating tuples of different lengths, or objects that unpack differently depending on state. I afraid this is very poor kind of pattern matching, and most users will need something more powerful. For tuples: * If its length is 2, and its first item is GET, and its second item's type is int, then do A. * If its first item is GET, and its second item's type is str, then save the second item and do B (the rest of items are ignored). * If its first item is PUT, then ignore other items and do C. For dicts: * If its "type" key is INT, and it has key "value", then save the value for "value", and save all other key-values, and do A. * If its "type" key is STRING, and it has keys "data" and "encoding", then save the value for "data" and "encoding" and do B. * If it is an empty dict, then do C. * If it does not have the "type" key, then do D. For arbitrary objects: * If it is an instance of Dog, and its "name" attribute has type str and starts with "F", and its "color" attribute is contained in required_colors, and its "legs" attribute > 3, and calling its "has_tail()" method returns false (not necessary literal False, but arbitrary false value), then do A. Oh, and it should work for nested patterns. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DA2NAYGUUMGTCYVVBGZYUDCXFLACAT4F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Proposal to introduce pattern matching syntax
One should never underestimate the power of Guido's time machine. > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/4JLKRRBP37IFHTGYHV53Z5MLHVDYCVSM/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Proposal to introduce pattern matching syntax
On Mon, Jun 22, 2020 at 8:20 AM nate lust wrote:
> I have been working on an idea that would introduce pattern matching
> syntax to python. I now have this syntax implemented in cpython, and feel
> this is the right time to gather further input. The repository and branch
> can be found at https://github.com/natelust/cpython/tree/match_syntax.
> The new syntax would clean up readability, ease things like visitor pattern
> style programming, localize matching behavior to a class, and support
> better signaling amongst other things. This is the tl;dr, I will get into a
> longer discussion below, but first I want to introduce the syntax and how
> it works with the following simple example
>
Great! This is a time of synchronicity -- with a few core devs and some
others I have been quietly working on a similar proposal. We've got an
implementation and a draft PEP -- we were planning to publish the PEP last
week but got distracted and there were a bunch of last-minute edits we
wanted to apply (and we thought there was no hurry).
We should join forces somehow.
> result = some_function_call()
>
> try match result: # try match some_function_call(): is also supported
>
> as Dog:
>
> print("is a dog")
>
> as Cat(lives):
>
> print(f"is a cat with {lives} lives")
>
> as tuple(result1, result2):
>
> print(f"got two results {result1} and {result2}")
>
> else:
>
> print("unknown result")
>
Our syntax is similar but uses plain "match" instead of "try match". We
considered "as", but in the end settled on "case". We ended up rejecting a
separate "else" clause, using "case _" (where "_" matches everything)
instead. There are other small syntactic differences but the general feel
is very similar!
> The statement begins with a new compound keyword "try match" . This is
> treated as one logical block, the word match is not being turned into a
> keyword. There are no backwards compatibility issues as previously no
> symbols were allowed between try and :. The try match compound keyword was
> chosen to make it clearer to users that this is distinct from a try block,
> provide a hint on what the block is doing, and follow the Python tradition
> of being sensible when spoken out loud to an English speaker. This keyword
> is followed by an expression that is to be matched, called the match target.
>
Note that the new PEG parser makes it possible to recognize plain "match
:" without fear of backward incompatibility.
> What follows is one or more match blocks. A match block is started with
> the keyword ‘as’ followed by a type, and optionally parameters.
>
> Matching begins by calling a __match__ (class)method on the type, with the
> match target as a parameter. The match method must return an object that
> can be evaluated as a bool. If the return value is True, the code block in
> this match branch is executed, and execution is passed to whatever comes
> after the match syntax. If __match__ returns False, execution is passed to
> the next match branch for testing.
>
We have a __match__ class method too, though its semantics are somewhat
different.
> If a match branch contains a group of parameters, they are used in the
> matching process as well. If __match__ returns True, then the match target
> will be tested for the presence of a __unpack__ method. If there is no such
> method, the match target is tried as a sequence. If both of these fail,
> execution moves on. If there is a __unpack__ method, it is called and is
> expected to return a sequence. The length of the sequence (either the
> result of __unpack__, or the match target itself) is compared to the number
> of supplied arguments. If they match the sequence is unpacked into
> variables defined by the arguments and the match is considered a success
> and the body is executed. If the length of the sequence does not match the
> number of arguments, the match branch fails and execution continues. This
> is useful for differentiating tuples of different lengths, or objects that
> unpack differently depending on state.
>
Our __match__ handles the __unpack__ functionality.
> If all the match blocks are tested and fail, the try match statement will
> check for the presence of an else clause. If this is present the body is
> executed. This serves as a default execution block.
>
> What is the __match__ method and how does it determine if a match target
> is a match? This change introduces __match__ as a new default method on
> ‘object’. The default implementation first checks the match target with the
> ‘is’ operator against the object containing the __match__ method. If that
> is false, then it checks the match target using isinstnace. Objects are
> free to implement whatever __match__ method they want provided it matches
> the interface.
>
> This proposal also introduces __unpack__ as a new interface, but does not
> define any default implementation. This method should return a sequence.
> There is no specific form
[Python-ideas] Re: Proposal to introduce pattern matching syntax
On Mon, 22 Jun 2020 at 13:08, Michael Christensen
wrote:
> Concerning parameters in the as section: I think using `with` would make
> it easier to understand.
>
> try match result:
> as Dog:
> print("Dog")
> as Cat with lives:
> print(f"Cat with {lives} lives")
> as tuple with (arg1, arg2):
> print(f"Tuple with args {arg1}, {arg2}")
> else:
> print("Nothing matched")
>
>
I for one could not even parse the example in the first e-mail. This
version
using "with" makes sense for me.
To start, it should be a starter that between `as` and `:` one should able
to write
an arbitrary Python expression, and not trying to invent yet another new
language
inside Python for this syntax (as what has taken place with annotations)
> Sent via the Samsung Galaxy S® 6, an AT 4G LTE smartphone
> Get Outlook for Android
>
> --
> *From:* nate lust
> *Sent:* Monday, June 22, 2020 9:17:01 AM
> *To:* python-ideas
> *Subject:* [Python-ideas] Proposal to introduce pattern matching syntax
>
>
> Hello,
>
> I have been working on an idea that would introduce pattern matching
> syntax to python. I now have this syntax implemented in cpython, and feel
> this is the right time to gather further input. The repository and branch
> can be found at https://github.com/natelust/cpython/tree/match_syntax.
> The new syntax would clean up readability, ease things like visitor pattern
> style programming, localize matching behavior to a class, and support
> better signaling amongst other things. This is the tl;dr, I will get into a
> longer discussion below, but first I want to introduce the syntax and how
> it works with the following simple example.
>
> result = some_function_call()
>
> try match result: # try match some_function_call(): is also supported
>
> as Dog:
>
> print("is a dog")
>
> as Cat(lives):
>
> print(f"is a cat with {lives} lives")
>
> as tuple(result1, result2):
>
> print(f"got two results {result1} and {result2}")
>
> else:
>
> print("unknown result")
>
>
The statement begins with a new compound keyword "try match" . This is
> treated as one logical block, the word match is not being turned into a
> keyword. There are no backwards compatibility issues as previously no
> symbols were allowed between try and :. The try match compound keyword was
> chosen to make it clearer to users that this is distinct from a try block,
> provide a hint on what the block is doing, and follow the Python tradition
> of being sensible when spoken out loud to an English speaker. This keyword
> is followed by an expression that is to be matched, called the match target.
>
> What follows is one or more match blocks. A match block is started with
> the keyword ‘as’ followed by a type, and optionally parameters.
>
> Matching begins by calling a __match__ (class)method on the type, with the
> match target as a parameter. The match method must return an object that
> can be evaluated as a bool. If the return value is True, the code block in
> this match branch is executed, and execution is passed to whatever comes
> after the match syntax. If __match__ returns False, execution is passed to
> the next match branch for testing.
>
> If a match branch contains a group of parameters, they are used in the
> matching process as well. If __match__ returns True, then the match target
> will be tested for the presence of a __unpack__ method. If there is no such
> method, the match target is tried as a sequence. If both of these fail,
> execution moves on. If there is a __unpack__ method, it is called and is
> expected to return a sequence. The length of the sequence (either the
> result of __unpack__, or the match target itself) is compared to the number
> of supplied arguments. If they match the sequence is unpacked into
> variables defined by the arguments and the match is considered a success
> and the body is executed. If the length of the sequence does not match the
> number of arguments, the match branch fails and execution continues. This
> is useful for differentiating tuples of different lengths, or objects that
> unpack differently depending on state.
>
> If all the match blocks are tested and fail, the try match statement will
> check for the presence of an else clause. If this is present the body is
> executed. This serves as a default execution block.
>
> What is the __match__ method and how does it determine if a match target
> is a match? This change introduces __match__ as a new default method on
> ‘object’. The default implementation first checks the match target with the
> ‘is’ operator against the object containing the __match__ method. If that
> is false, then it checks the match target using isinstnace. Objects are
> free to implement whatever __match__ method they want provided it matches
> the interface.
>
> This proposal also introduces __unpack__ as a new interface, but does not
[Python-ideas] Re: Proposal to introduce pattern matching syntax
Concerning parameters in the as section: I think using `with` would make it
easier to understand.
try match result:
as Dog:
print("Dog")
as Cat with lives:
print(f"Cat with {lives} lives")
as tuple with (arg1, arg2):
print(f"Tuple with args {arg1}, {arg2}")
else:
print("Nothing matched")
Sent via the Samsung Galaxy S® 6, an AT 4G LTE smartphone
Get Outlook for Android
From: nate lust
Sent: Monday, June 22, 2020 9:17:01 AM
To: python-ideas
Subject: [Python-ideas] Proposal to introduce pattern matching syntax
Hello,
I have been working on an idea that would introduce pattern matching syntax to
python. I now have this syntax implemented in cpython, and feel this is the
right time to gather further input. The repository and branch can be found at
https://github.com/natelust/cpython/tree/match_syntax. The new syntax would
clean up readability, ease things like visitor pattern style programming,
localize matching behavior to a class, and support better signaling amongst
other things. This is the tl;dr, I will get into a longer discussion below, but
first I want to introduce the syntax and how it works with the following simple
example.
result = some_function_call()
try match result: # try match some_function_call(): is also supported
as Dog:
print("is a dog")
as Cat(lives):
print(f"is a cat with {lives} lives")
as tuple(result1, result2):
print(f"got two results {result1} and {result2}")
else:
print("unknown result")
The statement begins with a new compound keyword "try match" . This is treated
as one logical block, the word match is not being turned into a keyword. There
are no backwards compatibility issues as previously no symbols were allowed
between try and :. The try match compound keyword was chosen to make it
clearer to users that this is distinct from a try block, provide a hint on what
the block is doing, and follow the Python tradition of being sensible when
spoken out loud to an English speaker. This keyword is followed by an
expression that is to be matched, called the match target.
What follows is one or more match blocks. A match block is started with the
keyword ‘as’ followed by a type, and optionally parameters.
Matching begins by calling a __match__ (class)method on the type, with the
match target as a parameter. The match method must return an object that can be
evaluated as a bool. If the return value is True, the code block in this match
branch is executed, and execution is passed to whatever comes after the match
syntax. If __match__ returns False, execution is passed to the next match
branch for testing.
If a match branch contains a group of parameters, they are used in the
matching process as well. If __match__ returns True, then the match target will
be tested for the presence of a __unpack__ method. If there is no such method,
the match target is tried as a sequence. If both of these fail, execution moves
on. If there is a __unpack__ method, it is called and is expected to return a
sequence. The length of the sequence (either the result of __unpack__, or the
match target itself) is compared to the number of supplied arguments. If they
match the sequence is unpacked into variables defined by the arguments and the
match is considered a success and the body is executed. If the length of the
sequence does not match the number of arguments, the match branch fails and
execution continues. This is useful for differentiating tuples of different
lengths, or objects that unpack differently depending on state.
If all the match blocks are tested and fail, the try match statement will check
for the presence of an else clause. If this is present the body is executed.
This serves as a default execution block.
What is the __match__ method and how does it determine if a match target is a
match? This change introduces __match__ as a new default method on ‘object’.
The default implementation first checks the match target with the ‘is’ operator
against the object containing the __match__ method. If that is false, then it
checks the match target using isinstnace. Objects are free to implement
whatever __match__ method they want provided it matches the interface.
This proposal also introduces __unpack__ as a new interface, but does not
define any default implementation. This method should return a sequence. There
is no specific form outside this definition, a class author is free to
implement whatever representation they would like to use in a match statement.
One use case for this method could be a class that stores the parameters passed
to __init__ (or some other parameters) so someone could construct a new object
such as Animal(*animal_instance.__unpack__()) [possibly with a builtin for
calling unpack]. Another use, in keeping with the match syntax, is something
like
[Python-ideas] Re: Please consider adding numbers.Boolean
[Neil Girdhar] > It seems that builtins.bool and numpy.bool_ would both be elements of Boolean I'd be wary of trying to abstract over `bool` and `numpy.bool_`. There are some fundamental differences: - `bool` subclasses `int`; `numpy.bool_` is not a subclass of any integer type - moreover, `numpy.bool_` isn't considered integer-like. It _does_ currently support `__index__`, but that support is deprecated, so at some point in the future a `numpy.bool_` may not be usable in many of the contexts that a Python bool is. (cf https://bugs.python.org/issue37980) - some bitwise operations behave differently: `~False` is not equal to `~np.False_`, and similarly for `True` What's the motivation for this? Is this intended primarily for use in type annotations, or did you have some other use in mind? -- Mark ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5H24NXYMFM5NIJ5YMGIH7W5FCLPOCL63/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Please consider adding numbers.Boolean
On 6/21/2020 8:19 PM, Neil Girdhar wrote: I'm just curious if there was a reason why Boolean was omitted from the numeric tower in the numbers library? It seems that builtins.bool and numpy.bool_ would both be elements of Boolean, and Boolean itself would be Integral? Is this being proposed for completeness, or is there some actual problem that would be solved by adding this? Eric ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WX37CSQHDUJHNIACTZMRPKSXEZ73GKLW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Please consider adding numbers.Boolean
I disagree that doing arithmetic on boolean variables doesn't make sense. Indicator variables which take a value of either zero are one are extremely common. Dirac or Kronecker delta functions are also examples of functions where it arithmetically makes sense to talk about values in the boolean domain. Of course that does mean that you define True as 1 and False as 0, but that is a very standard convention and it is the case in Python. I don't have a strong opinion on the proposal itself, but saying "arithmetic on booleans doesn't doesn't make mathematical sense" is False. On Mon, Jun 22, 2020 at 5:08 AM Greg Ewing wrote: > On 22/06/20 12:19 pm, Neil Girdhar wrote: > > I'm just curious if there was a reason why Boolean was omitted from the > > numeric tower in the numbers library? > > Python's bool type is a subclass of int for historical reasons, > not because it's conceptually a numeric type. Doing arithmetic on > it doesn't really make mathematical sense. > > -- > Greg > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/CMDSMDVCT5ABSLILV3U772QDLI2IUCAZ/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- -Dr. Jon Crall (him) ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WZTFFH4BHZAS4YSKVZOSS6JTA3VYFIDZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Proposal to introduce pattern matching syntax
Hello,
I have been working on an idea that would introduce pattern matching syntax
to python. I now have this syntax implemented in cpython, and feel this is
the right time to gather further input. The repository and branch can be
found at https://github.com/natelust/cpython/tree/match_syntax. The new
syntax would clean up readability, ease things like visitor pattern style
programming, localize matching behavior to a class, and support better
signaling amongst other things. This is the tl;dr, I will get into a longer
discussion below, but first I want to introduce the syntax and how it works
with the following simple example.
result = some_function_call()
try match result: # try match some_function_call(): is also supported
as Dog:
print("is a dog")
as Cat(lives):
print(f"is a cat with {lives} lives")
as tuple(result1, result2):
print(f"got two results {result1} and {result2}")
else:
print("unknown result")
The statement begins with a new compound keyword "try match" . This is
treated as one logical block, the word match is not being turned into a
keyword. There are no backwards compatibility issues as previously no
symbols were allowed between try and :. The try match compound keyword was
chosen to make it clearer to users that this is distinct from a try block,
provide a hint on what the block is doing, and follow the Python tradition
of being sensible when spoken out loud to an English speaker. This keyword
is followed by an expression that is to be matched, called the match target.
What follows is one or more match blocks. A match block is started with the
keyword ‘as’ followed by a type, and optionally parameters.
Matching begins by calling a __match__ (class)method on the type, with the
match target as a parameter. The match method must return an object that
can be evaluated as a bool. If the return value is True, the code block in
this match branch is executed, and execution is passed to whatever comes
after the match syntax. If __match__ returns False, execution is passed to
the next match branch for testing.
If a match branch contains a group of parameters, they are used in the
matching process as well. If __match__ returns True, then the match target
will be tested for the presence of a __unpack__ method. If there is no such
method, the match target is tried as a sequence. If both of these fail,
execution moves on. If there is a __unpack__ method, it is called and is
expected to return a sequence. The length of the sequence (either the
result of __unpack__, or the match target itself) is compared to the number
of supplied arguments. If they match the sequence is unpacked into
variables defined by the arguments and the match is considered a success
and the body is executed. If the length of the sequence does not match the
number of arguments, the match branch fails and execution continues. This
is useful for differentiating tuples of different lengths, or objects that
unpack differently depending on state.
If all the match blocks are tested and fail, the try match statement will
check for the presence of an else clause. If this is present the body is
executed. This serves as a default execution block.
What is the __match__ method and how does it determine if a match target is
a match? This change introduces __match__ as a new default method on
‘object’. The default implementation first checks the match target with the
‘is’ operator against the object containing the __match__ method. If that
is false, then it checks the match target using isinstnace. Objects are
free to implement whatever __match__ method they want provided it matches
the interface.
This proposal also introduces __unpack__ as a new interface, but does not
define any default implementation. This method should return a sequence.
There is no specific form outside this definition, a class author is free
to implement whatever representation they would like to use in a match
statement. One use case for this method could be a class that stores the
parameters passed to __init__ (or some other parameters) so someone could
construct a new object such as Animal(*animal_instance.__unpack__())
[possibly with a builtin for calling unpack]. Another use, in keeping with
the match syntax, is something like Stateful Enums.
The behavior covered by try match can be emulated with some combination of
compound and or nested if statements alongside type checking and parameter
unpacking, so why introduce the new syntax? The benefits I see are:
* Much easier to read and follow compared complicated if branching logic
* the matching logic is now defined alongside the class in the __match__
method. This is in contrast to if statements where the logic is duplicated
in each place. If it factored out into a function, the function may be
unknown or unused in a cross package setting. Refactoring distributed logic
to live next to an object is similar to the introduction of the format
method
[Python-ideas] Re: Permanent code objects (less memory, quicker load, less Unix Copy On Write)
Hi SUMMARY: We're starting to discuss implementation. I'm going to focus on what can be done, with only a few changes to the interpreter. First consider this: >>> from sys import getrefcount as grc >>> def fn(obj): return grc(obj) >>> grc(fn.__code__), grc(fn.__code__.co_code) (2, 2) >>> fn(fn.__code__), fn(fn.__code__.co_code) (5, 4) >>> grc(fn.__code__), grc(fn.__code__.co_code) (2, 2) # This is the bytecode. >>> fn.__code__.co_code b't\x00|\x00\x83\x01S\x00' What's happening here? While the interpreter executes the pure Python function fn, it changes the refcount of both fn.__code__ and fn.__code__.co_code. This is one of the problems we have to solve, to make progress. These refcounts are stored in the objects themselves. So unless the interpreter is changed, these Python objects can't be stored in read-only memory. We may have to change code and co_code objects also. Let's focus on the bytecode, as it's the busiest and often largest part of the code object. The (ordinary) code object has a field which is (a pointer to) the co_code attribute, which is a Python bytes object. This is the bytecode, as a Python object. Let's instead give the C implementation of fn.__code__ TWO fields. The first is a pointer, as usual, to the co_code attribute of the code object. The second is a pointer to the raw data of the co_code object. When the interpreter executes the code object, the second field tells the interpreter where to start executing. (This might be why the refcount of fn.__code__.co_code is incremented during the execution of fn.) The interpreter doesn't even have to look at the first field. If we want the raw bytecode of a code object to lie in read-only memory, it is enough to set the second pointer to that location. In both cases, the interpreter reads the memory location of the raw bytecode and executes accordingly. This leaves the problem of the first field. At present, it can only be a bytes object. When the raw bytecode is in read-only memory, we need a second sort of object. It's purpose is to 'do the right thing'. Let's call this sort of object perma_bytes. It's like a bytes object, except the data is stored elsewhere, in read-only permanent storage. Aside: If the co_code attribute of a code object is ordinary bytes - not perma_bytes -- then the two pointer addresses differ by a constant, namely the size of the header of a Python bytes object. Any Python language operation on perma_bytes is done by performing the same Python operation on bytes, but on the raw data that is pointed to. (That raw data had better still be there, otherwise chaos or worse will result.) So what have we gained, and what have we lost. LOST: 1. fn.code object bigger by the size of a pointer. 2. Added perma_bytes object. GAINED: 1. Can store co_code data in read-only permanent storage. 2. Bytes on fn.__code__.co_code objects are slower. 3. perma_bytes might be useful elsewhere. It may be possible to improve the outcome, by making more changes to the interpreter. I don't see a way of getting a useful outcome, by making fewer. Here's another way of looking at things. If all the refcounts were stored in a single array, and the data stored elsewhere, the changing refcount wouldn't be a problem. Using perma_bytes allows the refount and the data to be stored at different locations, thereby avoiding the refcount problem! I hope this is clear enough, and that it helps. And that it is correct. I'll let T. S. Eliot have the last word: https://faculty.washington.edu/smcohen/453/NamingCats.html The Naming of Cats is a difficult matter, It isn’t just one of your holiday games; You may think at first I’m as mad as a hatter When I tell you, a cat must have THREE DIFFERENT NAMES. We're giving the raw data TWO DIFFERENT POINTERS. with best wishes Jonathan ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/3AN53F4XEM4BHD4QKX3VOUHXDBM6HEIP/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Permanent code objects (less memory, quicker load, less Unix Copy On Write)
I like where this is going. It would be nice if certain constants could also be loaded from RO memory. On Mon, Jun 22, 2020 at 00:16 Inada Naoki wrote: > On Mon, Jun 22, 2020 at 12:00 AM Guido van Rossum > wrote: > > > > > > I believe this was what Greg Stein's idea here was about. (As well as > Jonathan Fine's in this thread?) But the current use of code objects makes > this hard. Perhaps the code objects could have a memoryview object to hold > the bytecode instead of a bytes object. > > > > memoryview is heavy object. Using memoryview instead of bytes object > will increase memory usage. > I think lightweight bytes-like object is better. My rough idea is: > > * New code and pyc format > * pyc has "rodata" segment > * It can be copied into single memory block, or can be mmapped. > * co_code should be aligned at least 2 bytes. > * code.co_code can point to memory block in "rodata". > * docstring, signature, and lnotab can be lazy load from "rodata". > * signature is serialized in JSON like format. > * Allow multiple modules in single file > * Reduce fileno when using mmap > * Merge more constants > * New Python object: PyROData. > * It is like read-only bytearray. > * But body may be mmap-ped, instead of malloc-ed > * code objects owns reference of PyROData. > * When PyROData is deallocated, it munmap or free "rodata" segment. > > Regards, > -- > Inada Naoki > -- --Guido (mobile) ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/7QFQUUAKA24PAZNSPYQKLMKGK6DVZA5U/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Permanent code objects (less memory, quicker load, less Unix Copy On Write)
If you want to proceed in this direction, it would be better to do some more research into current CPU architectures and then build a VM optimized byte code storage object, which is well aligned, fits into today's caches and improves locality. freeze.py could then write out this format as well, so that the object can directly point to the structure and have the OS deal with memory mapping and sharing byte code across processes. I've had some good performance increases when I used the above approach in the mxTextTools tagging engine, which is a low-level VM for tagging and searching in text. If compilers are made aware of the structure the VM will use, they may also be able to apply additional optimizations for faster byte code access, prefetching, etc. On 22.06.2020 09:15, Inada Naoki wrote: > On Mon, Jun 22, 2020 at 12:00 AM Guido van Rossum wrote: >> >> >> I believe this was what Greg Stein's idea here was about. (As well as >> Jonathan Fine's in this thread?) But the current use of code objects makes >> this hard. Perhaps the code objects could have a memoryview object to hold >> the bytecode instead of a bytes object. >> > > memoryview is heavy object. Using memoryview instead of bytes object > will increase memory usage. > I think lightweight bytes-like object is better. My rough idea is: > > * New code and pyc format > * pyc has "rodata" segment > * It can be copied into single memory block, or can be mmapped. > * co_code should be aligned at least 2 bytes. > * code.co_code can point to memory block in "rodata". > * docstring, signature, and lnotab can be lazy load from "rodata". > * signature is serialized in JSON like format. > * Allow multiple modules in single file > * Reduce fileno when using mmap > * Merge more constants > * New Python object: PyROData. > * It is like read-only bytearray. > * But body may be mmap-ped, instead of malloc-ed > * code objects owns reference of PyROData. > * When PyROData is deallocated, it munmap or free "rodata" segment. > > Regards, > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Jun 22 2020) >>> Python Projects, Coaching and Support ...https://www.egenix.com/ >>> Python Product Development ...https://consulting.egenix.com/ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LIS4FGWJ4RH4VYP2HICPPAWS2VVZZMZ7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Permanent code objects (less memory, quicker load, less Unix Copy On Write)
On Mon, Jun 22, 2020 at 8:27 PM Barry Scott wrote: > > * New code and pyc format > * pyc has "rodata" segment >* It can be copied into single memory block, or can be mmapped. > * co_code should be aligned at least 2 bytes. > > > Would higher alignment help? malloc is using 8 or 16 byte alignment isn't it? > Would that be better for packing the byte code into cache lines? > It may. But I am not sure. I said "at least 2 byte" because we use "word code". We read the "word code" by `uint16_t *`, not `unsignet char *`. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PMPWPP2YF7JSTT6XOVCXNF6G7X6WSSEU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Permanent code objects (less memory, quicker load, less Unix Copy On Write)
Let's try again... > On 22 Jun 2020, at 08:15, Inada Naoki wrote: > > On Mon, Jun 22, 2020 at 12:00 AM Guido van Rossum wrote: >> >> >> I believe this was what Greg Stein's idea here was about. (As well as >> Jonathan Fine's in this thread?) But the current use of code objects makes >> this hard. Perhaps the code objects could have a memoryview object to hold >> the bytecode instead of a bytes object. >> > > memoryview is heavy object. Using memoryview instead of bytes object > will increase memory usage. > I think lightweight bytes-like object is better. My rough idea is: > > * New code and pyc format > * pyc has "rodata" segment >* It can be copied into single memory block, or can be mmapped. > * co_code should be aligned at least 2 bytes. Would higher alignment help? malloc is using 8 or 16 byte alignment isn't it? Would that be better for packing the byte code into cache lines? > * code.co_code can point to memory block in "rodata". > * docstring, signature, and lnotab can be lazy load from "rodata". >* signature is serialized in JSON like format. > * Allow multiple modules in single file >* Reduce fileno when using mmap >* Merge more constants > * New Python object: PyROData. >* It is like read-only bytearray. > * But body may be mmap-ped, instead of malloc-ed >* code objects owns reference of PyROData. > * When PyROData is deallocated, it munmap or free "rodata" segment. > > Regards, > -- > Inada Naoki Barry > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/VKBXY7KDI2OGESB7IPAMAIIHKR4TC7TQ/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WVQYYFALSV4E22SFR66PNUN4X7TB4ZJA/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Permanent code objects (less memory, quicker load, less Unix Copy On Write)
> On 22 Jun 2020, at 08:15, Inada Naoki wrote: > > On Mon, Jun 22, 2020 at 12:00 AM Guido van Rossum wrote: >> >> >> I believe this was what Greg Stein's idea here was about. (As well as >> Jonathan Fine's in this thread?) But the current use of code objects makes >> this hard. Perhaps the code objects could have a memoryview object to hold >> the bytecode instead of a bytes object. >> > > memoryview is heavy object. Using memoryview instead of bytes object > will increase memory usage. > I think lightweight bytes-like object is better. My rough idea is: > > * New code and pyc format > * pyc has "rodata" segment >* It can be copied into single memory block, or can be mmapped. > * co_code should be aligned at least 2 bytes. Would higher alignment help? malloc is use 8 or 16 byte alignment isn't it? Would that better load the code into cache lines? > * code.co_code can point to memory block in "rodata". > * docstring, signature, and lnotab can be lazy load from "rodata". >* signature is serialized in JSON like format. > * Allow multiple modules in single file >* Reduce fileno when using mmap >* Merge more constants > * New Python object: PyROData. >* It is like read-only bytearray. > * But body may be mmap-ped, instead of malloc-ed >* code objects owns reference of PyROData. > * When PyROData is deallocated, it munmap or free "rodata" segment. > > Regards, > -- > Inada Naoki > Barry > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/VKBXY7KDI2OGESB7IPAMAIIHKR4TC7TQ/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5MAUB7BC4R2YGCTPKVXDKSSPERTRX7JY/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Please consider adding numbers.Boolean
On 22/06/20 12:19 pm, Neil Girdhar wrote: I'm just curious if there was a reason why Boolean was omitted from the numeric tower in the numbers library? Python's bool type is a subclass of int for historical reasons, not because it's conceptually a numeric type. Doing arithmetic on it doesn't really make mathematical sense. -- Greg ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/CMDSMDVCT5ABSLILV3U772QDLI2IUCAZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Permanent code objects (less memory, quicker load, less Unix Copy On Write)
On Mon, Jun 22, 2020 at 5:19 PM Inada Naoki wrote: > I think lightweight bytes-like object is better. My rough idea is: > > * New code and pyc format > * pyc has "rodata" segment > * It can be copied into single memory block, or can be mmapped. > * co_code should be aligned at least 2 bytes. > * code.co_code can point to memory block in "rodata". > * docstring, signature, and lnotab can be lazy load from "rodata". > * signature is serialized in JSON like format. > * Allow multiple modules in single file > * Reduce fileno when using mmap > * Merge more constants > * New Python object: PyROData. > * It is like read-only bytearray. > * But body may be mmap-ped, instead of malloc-ed > * code objects owns reference of PyROData. > * When PyROData is deallocated, it munmap or free "rodata" segment. > With exec, it's possible to create a function that doesn't have any corresponding pyc or module. Would functions and code objects need to cope with both this style and the current model? ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/52FLX52YF63MG4K76PIHA3PDDPMW53SK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Permanent code objects (less memory, quicker load, less Unix Copy On Write)
On Mon, Jun 22, 2020 at 12:00 AM Guido van Rossum wrote: > > > I believe this was what Greg Stein's idea here was about. (As well as > Jonathan Fine's in this thread?) But the current use of code objects makes > this hard. Perhaps the code objects could have a memoryview object to hold > the bytecode instead of a bytes object. > memoryview is heavy object. Using memoryview instead of bytes object will increase memory usage. I think lightweight bytes-like object is better. My rough idea is: * New code and pyc format * pyc has "rodata" segment * It can be copied into single memory block, or can be mmapped. * co_code should be aligned at least 2 bytes. * code.co_code can point to memory block in "rodata". * docstring, signature, and lnotab can be lazy load from "rodata". * signature is serialized in JSON like format. * Allow multiple modules in single file * Reduce fileno when using mmap * Merge more constants * New Python object: PyROData. * It is like read-only bytearray. * But body may be mmap-ped, instead of malloc-ed * code objects owns reference of PyROData. * When PyROData is deallocated, it munmap or free "rodata" segment. Regards, -- Inada Naoki ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VKBXY7KDI2OGESB7IPAMAIIHKR4TC7TQ/ Code of Conduct: http://python.org/psf/codeofconduct/
