[Python-Dev] Re: Python multithreading without the GIL

[Barry Warsaw]

Thank you Sam, this additional detail really helps me understand your proposal.

-Barry

> On Oct 11, 2021, at 12:06, Sam Gross  wrote:
> 
> I’m unclear what is actually retried.  You use this note throughout the 
> document, so I think it would help to clarify exactly what is retried and why 
> that solves the particular problem.  I’m confused because, is it the refcount 
> increment that’s retried or the entire sequence of steps (i.e. do you go back 
> and reload the address of the item)?  Is there some kind of waiting period 
> before the retry?  I would infer that if you’re retrying the refcount 
> incrementing, it’s because you expect subsequent retries to transition from 
> zero to non-zero, but is that guaranteed?  Are there possibilities of 
> deadlocks or race conditions?
> 
> The entire operation is retried (not just the refcount). For "dict", this 
> means going back to step 1 and reloading the version tag and 
> PyDictKeysObject. The operation can fail (and need to be retried) only when 
> some other thread is concurrently modifying the dict. The reader needs to 
> perform the checks (and retry) to avoid returning inconsistent data, such as 
> an object that was never in the dict. With the checks and retry, returning 
> inconsistent or garbage data is not possible.
> 
> The retry is performed after locking the dict, so the operation is retried at 
> most once -- the read operation can't fail when it holds the dict's lock 
> because the lock prevents concurrent modifications. It would have also been 
> possible to retry the operation in a loop without locking the dict, but I was 
> concerned about reader starvation. (In the doc I wrote "livelock", but 
> "reader starvation" is more accurate.) In particular, I was concerned that a 
> thread repeatedly modifying a dict might prevent other threads reading the 
> dict from making progress. I hadn't seen this in practice, but I'm aware that 
> reader starvation can be an issue for similar designs like Linux's seqlock. 
> Acquiring the dict's lock when retrying avoids the reader starvation issue.
> 
> Deadlock isn't possible because the code does not acquire any other locks 
> while holding the dict's lock. For example, the code releases the dict's lock 
> before calling Py_DECREF or PyObject_RichCompareBool.
> 
> The race condition question is a bit harder to answer precisely. Concurrent 
> reads and modifications of a dict won't cause the program to segfault, return 
> garbage data, or items that were never in the dict.
> 
> Regards,
> Sam
> 
> 



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/65OZPNMIZ2INHEIKTW7H65SORRYDK4D4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Erik Demaine]

My apologies!  Must be an issue with my email client rendering.

Erik

On Mon, 11 Oct 2021, Guido van Rossum wrote:


On Mon, Oct 11, 2021 at 3:22 PM Erik Demaine  wrote:
  On Mon, 11 Oct 2021, Guido van Rossum wrote:


No, I didn't write that, Lukasz did.

  > I always found the following more obvious:
  >
  > def data_to_table(d: Iterable[Mapping[str, float]], *, sort:
  bool = False, reversed: bool = False) -> Table:
  >     ...___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RXMHRH55S2EBYBVFGPLFXKKVA2RWZIYL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Guido van Rossum]

On Mon, Oct 11, 2021 at 3:22 PM Erik Demaine  wrote:

> On Mon, 11 Oct 2021, Guido van Rossum wrote:
>

No, I didn't write that, Lukasz did.

> I always found the following more obvious:
> >
> > def data_to_table(d: Iterable[Mapping[str, float]], *, sort: bool =
> False, reversed: bool = False) -> Table:
> > ...
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WA37NRFZIOFIUTORVUT7RJ3HWYN7JXDH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Erik Demaine]

On Mon, 11 Oct 2021, Guido van Rossum wrote:


I always found the following more obvious:

def data_to_table(d: Iterable[Mapping[str, float]], *, sort: bool = False, 
reversed: bool = False) -> Table:
    ...

@dataclass
class Stream:
    converter: data_to_table | None

    def add_converter(self, converter: data_to_table) -> None:
        self.converter = converter


Another possibility would be that functions can't be used as their types 
directly, but need a casting operator like so:


```
    def add_converter(self, converter: typeof(data_to_table)) -> None:
        self.converter = converter
```

It's too bad `type` is already taken, but `typeof` is what TypeScript calls 
it: https://www.typescriptlang.org/docs/handbook/2/typeof-types.html -- 
perhaps there is a better distinguishing name, but I'll keep using `typeof` 
for the sake of this discussion.


For static analysis, `typing.typeof` should only be used for variables that 
are assigned (certain) constant values and aren't re-assigned.


Potentially this could be a much more general solution.  Maybe interesting 
return values for `typeof` could be defined for `dict`s, `namedtuple`s, etc. 
And it could co-exist with the (P, Q) -> R proposal (which, as you probably 
know, is how TypeScript expresses function types).



One disadvantage of this is that now arguments HAVE TO be named which
raises questions:
- should they be considered at type checking time?
- how to express "I don't care"?


I think it'd be natural for types constructed via `typeof` to include the 
argument names; I'd intend for an object "just like" this one.  But you could 
imagine another type operator that drops details like this, e.g. 
`typing.forget_argument_names(typing.typeof(data_table))`.


Erik
--
Erik Demaine  |  edema...@mit.edu  |  http://erikdemaine.org/___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QVW4I2CJVEETXM7I437WYIXLCYSOP4B4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Facundo Batista]

El lun, 11 de oct. de 2021 a la(s) 06:50, Simon Cross
(hodgestar+python...@gmail.com) escribió:

> Multiprocessing sort of added support for this via multiprocessing.Array -- 
> see 
> https://stackoverflow.com/questions/9754034/can-i-create-a-shared-multiarray-or-lists-of-lists-object-in-python-for-multipro.
>  I haven't looked at what multiprocessing.Array does under the hood.
>
> Summary of the StackOverflow answer for those who don't feel like clicking:
>
>mp_arr = mp.Array(c.c_double, size)
># then in each new process create a new numpy array using:
>arr = np.frombuffer(mp_arr.get_obj())

Right, this is very close to what I had in mind for the "example with
numpy.array" that I want to code next (as I just said in the response
to Gregory, this is a series of implementations for pedagogical
purposes trying to teach parallel processing).

Thanks!

-- 
.Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org.ar/
Twitter: @facundobatista
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YTEO6B3TBD6PHWKRMP6F46P25HFN4ZU5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Facundo Batista]

El dom, 10 de oct. de 2021 a la(s) 15:20, Gregory P. Smith
(g...@krypto.org) escribió:

>> Is that because my particular case is very uncommon? Or maybe we *do*
>> want this but we don't have it yet? Or do we already have a better way
>> of doing this?
>>
>> Thanks!
>>
>> [0] https://linkode.org/#95ZZtVCIVtBbx72dURK7a4
>
>
> My first reaction on seeing things like this is "Why not use a numpy.array?"

Yes, there is a plan to use `numpy.array` in another example (I'm
trying to just use stdlib for pedagogical purposes).

Thanks!

-- 
.Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org.ar/
Twitter: @facundobatista
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZSFAE3JIOLJ7KITLAPG6FTHF4SQ63D3G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Sam Gross]

I've updated the linked gists with the results from interpreters compiled
with PGO, so the numbers have slightly changed.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ESCEXN7HKL3GICHOHZMQTTHUDQN5WUYX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[gohanpra]

1. Barry Warsaw

> This means that any PEP which proposes syntactic changes to support typing 
> features must also address the implications of those syntax changes for the 
> general Python language.  PEP 646 (Variadic Generics) is a good example of 
> this.  Early on we recognized that the proposed syntax change had 
> implications for Python, and we asked the PEP authors to address those 
> implications for Python, which they added:

Yes, we definitely plan to include syntax implications in the PEP. (I'm 
co-author of PEP 646 :) )

This was one of the reasons we wanted to get high-level feedback here, since 
our proposal includes a syntax change for Python.

2. Łukasz

As Steven mentioned, your proposal of using a function name as its type is 
definitely something we're keeping in mind. It comes under the "alternative 
proposals" I talked about in part 2.

As a replacement for Callable itself (part 1), writing a separate function 
signature is much more verbose and inconvenient. Given how frequently we need 
to express simple positional-only callables (like `Callable[[Request], 
Response]`) and ParamSpec callables, the consensus was strongly in favor of 
having simple inline syntax.

However, when it comes to complex features like named parameters, default 
values, and `**kwargs` (part 2), your proposal might well be more readable than 
"hybrid" syntax. These use cases rarely come up, based on our stats, so a 
simple, syntax-free change like yours might be better than trying to duplicate 
everything that can be expressed in function signatures.

So, shorthand syntax + function-name-as-type could be a good overall solution 
for part 1 and part 2. This would give us simple and friendly syntax for the 
most common cases and readable syntax for the advanced cases.

What do you think?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4HDZ344W466RC7VPFD4MLUOJEOG7JDZJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Sam Gross]

On Mon, Oct 11, 2021 at 7:04 AM Antoine Pitrou  wrote:

> It's crude, but you can take a look at `ccbench` in the Tools directory.
>

Thanks, I wasn't familiar with this. The ccbench results look pretty good:
about 18.1x speed-up on "pi calculation" and 19.8x speed-up on "regular
expression" with 20 threads (turbo off). The latency and throughput results
look good too. With the GIL enabled (3.11), the compute intensive
background task increases latency and dramatically decreases throughput.
With the GIL disabled, latency remains low and throughput high.

Here are the full results for 20 threads without the GIL:
https://gist.github.com/colesbury/8479ee0246558fa1ab0f49e4c01caeed (nogil,
20 threads)

Here are the results for 4 threads (the default) for comparison with
upstream:
https://gist.github.com/colesbury/8479ee0246558fa1ab0f49e4c01caeed (nogil,
4 threads)
https://gist.github.com/colesbury/c0b89f82e51779670265fb7c7cd37114
(3.11/b108db63e0, 4 threads)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WCMIVNQ6DNOTZUUX4EX43LF2VJPF4ALW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Guido van Rossum]

On Mon, Oct 11, 2021 at 1:52 AM Łukasz Langa  wrote:

>
> On 7 Oct 2021, at 18:41, S Pradeep Kumar  wrote:
>
> Note that we considered and rejected using a full def-signature syntax
> like
> 
> (record: PurchaseRecord, permissions: List[AuthPermission], /) ->
> FormattedItem
> 
> because it would be more verbose for common cases and could lead to subtle
> bugs; more details in [3].
>
>
> Is this also why re-using an actual callable at a type was rejected?
>
> I always found the following more obvious:
>
>
> def data_to_table(d: Iterable[Mapping[str, float]], *, sort: bool = False,
> reversed: bool = False) -> Table:
> ...
>
>
> @dataclass
> class Stream:
> converter: data_to_table | None
>
> def add_converter(self, converter: data_to_table) -> None:
> self.converter = converter
>
>
> This solves the following problems with the `(P, Q) -> R` proposal:
> - how should this look like for "runtime" Python
> - how should we teach this
> - how can we express callables with complex signatures
>
> One disadvantage of this is that now arguments HAVE TO be named which
> raises questions:
> - should they be considered at type checking time?
> - how to express "I don't care"?
>
> To this I say:
> - yes, they should be considered at runtime (because kwargs have to be
> anyway)
> - ...unless they begin with an underscore
>
> This still leaves a minor problem that you can't have more than one
> argument literally named `_` so you'd have to do `_1`, `_2`, and so on. I
> don't think this is a big problem.
>
> In fact, forcing users to name callable arguments can be added as a fourth
> advantage to this design: making the annotations maximally informative to
> the human reader.
>
> The only remaining disadvantage that can't be addressed is that you can't
> create an *inline* callable type this way. I don't think this is a deal
> breaker as neither TypedDicts, Protocols, nor for this matter any
> PlainOldClasses can be defined inline inside a type annotation.
>

 Not everyone on typing-sig has the same preferences. For me personally,
your proposal is at best a better syntax for callback protocols, not for
inline callback types, precisely because it uses statement-level syntax and
requires you to think of a name for the type.

It also would cause some confusion because people aren't used for functions
to be used as type aliases. If I see

A = dict[str, list[int]]

it's easy enough to guess that A is a type alias.

But if I see

def Comparison(a: T, b: T) -> Literal[-1, 0, 1]:
...

my first thought is that it's a comparison function that someone hasn't
finished writing yet, not a function type -- since if it did have at least
one line of code in the body, it *would* be that.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TNX4KO4BWA2MJE5BNX5PHBY7QOZTYERX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Jim J. Jewett]

It isn't clear to me that reusing the inspect.Signature object was even 
considered.  If it was rejected as too complicated for most signatures, please 
put that in the PEP.  My own opinion, admittedly not as a typing fan, is that 
if you need enough details to want more than Callable, then you probably do 
want the full power of inspect.Signature, though you may also want some 
convenience methods to indicate that certain portions of the signature are 
undefined, rather than following the example function.

-jJ
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/PTG2WNWIFBBCSDSB4HC5QENXILD6LGUC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Thomas Grainger]

I have a PR to remove this FAQ entry: 
https://github.com/python/cpython/pull/28886
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YP3QZ7ZLMMQUAWVQRGAGNNETA6IDXP4P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[jack . jansen]

If I can make a wild suggestion: why not create a little language for type 
specifications?

If you look at other programming languages you’ll see that the “type definition 
sub-language” is often completely different from the “execution sub-language”, 
with only some symbols in common and used in vaguely related ways.  `bool 
(*myfuncptr)(int, float*)` uses a completely different set of syntactic rules 
than `rv = (*myfunptr)(*myintptr, &myfloat)`. So with some grains of salt you 
could say that C is comprised of a declarative typing sublanguage and an 
imperative execution sublanguage.

Python typing uses basically a subset of the execution expression syntax as its 
declarative typing language.

What if we created a little language that is clearly flagged, for example as 
t”….” or t’….’? Then we could simply define the typestring language to be 
readable, so you could indeed say t”(int, str) -> bool”. And we could even 
allow escapes (similar to f-strings) so that the previous expression could also 
be specified, if you really wanted to, as t”{typing.Callable[[int, str], bool}”.

Jack

> On 7 Oct 2021, at 18:41, S Pradeep Kumar  wrote:
> 
> Hello all,
> 
> Typing-sig has been discussing user-friendly syntax for the type used to 
> represent callables. [1] Since this affects the Python language syntax, we 
> wanted to get some high-level feedback from you before putting up a detailed 
> PEP.
> 
> TL;DR: We want to propose syntax for Callable types, e.g., `(int, str) -> 
> bool` instead of `typing.Callable[[int, str], bool]`. Questions: 1. Are there 
> concerns we need to keep in mind about such a syntax change? 2. Should we 
> propose syntax for additional features in the same PEP or in a future PEP?
> 
> 
> # Motivation
> 
> Guido has pointed out that `Callable` is one of the most frequently used type 
> forms but has the least readable syntax. For example, say we have a callback 
> `formatter` that accepts a record and a list of permissions:
> 
> ```python
> def print_purchases(
> user,
> formatter,  # <-- callback
> ):
> <...>
> output = formatter(record, permissions)
> print(output)
> ```
> 
> To give it a type, we currently have to write:
> 
> ```python
> from typing import Callable
> 
> def print_purchases(
> user: User,
> formatter: Callable[[PurchaseRecord, List[AuthPermission]], 
> FormattedItem]  # Hard to read.
> ) -> None:
> 
> <...>
> output = formatter(record, permissions)
> print(output)
> ```
> 
> `Callable` can be hard to understand for new users since it doesn't resemble 
> existing function signatures and there can be multiple square brackets. It 
> also requires an import from `typing`, which is inconvenient. Around 40% of 
> the time [2], users just give up on precisely typing the parameters and 
> return type of their callbacks and just leave it as a blank Callable. In 
> other cases, they don't add a type for their callback at all. Both mean that 
> they lose the safety guarantees from typing and leave room for bugs.
> 
> We believe that adding friendly syntax for Callable types will improve 
> readability and encourage users to type their callbacks more precisely. Other 
> modern, gradually-typed languages like TypeScript (JS), Hack (PHP), etc. have 
> special syntax for Callable types.
> 
> (As a precedent, PEP 604 recently added clean, user-friendly syntax for the 
> widely-used `Union` type. Instead of importing `Union` and writing `expr: 
> Union[AddExpr, SubtractExpr], we can just write `expr: AddExpr | 
> SubtractExpr`.)
> 
> 
> # Proposal and Questions
> 
> We have a two-part proposal and questions for you:
> 
> 1. Syntax to replace Callable
> 
> After a lot of discussion, there is strong consensus in typing-sig about 
> adding syntax to replace Callable. So, the above example would be written as:
> ```python
> def print_purchases(
> user: User,
> formatter: (PurchaseRecord, List[AuthPermission]) -> FormattedItem,
> ) -> None:
> 
> <...>
> output = formatter(record, permissions)
> print(output)
> ```
> This feels more natural and succinct. 
> 
> Async callbacks currently need to be written as 
> ```
> from typing import Callable
> 
> async_callback: Callable[[HttpRequest], Awaitable[HttpResponse]]
> ```
> 
> With the new syntax, they would be written as 
> ```
> async_callback: async (HttpRequest) -> HttpResponse
> ```
> which again seems more natural. There is similar syntax for the type of 
> decorators that pass on *args and **kwargs to the decorated function.
> 
> Note that we considered and rejected using a full def-signature syntax like 
> 
> (record: PurchaseRecord, permissions: List[AuthPermission], /) -> 
> FormattedItem
> 
> because it would be more verbose for common cases and could lead to subtle 
> bugs; more details in [3].
> 
> The Callable type is also usable as an expression, like in type aliases 
> `IntOperator = (int, int) -> int` and `cast((int) -> int, f)` calls.
> 
> **Question 1**: Are

[Python-Dev] Re: Python multithreading without the GIL

[Pablo Galindo Salgado]

As far as I understand we should get a smaller improvement on single thread
because some of the optimizations listed in this work are partially or
totally implemented.

This is excluding any non linear behaviour between the different
optimizations of course, and assuming that both versions yield the same
numbers.

On Mon, 11 Oct 2021, 20:28 Abdur-Rahmaan Janhangeer, 
wrote:

> When you mean "an order of magnitude less overhead than the current
> CPython implementation" do you mean compared with the main branch? We
> recently implemented already almost everything is listed in this paragraph:
>
> https://github.com/python/cpython/pull/27077
>
> We also pack some extra similar optimizations in this other PR, including
> stealing the frame arguments from python to python calls:
>
> https://github.com/python/cpython/pull/28488
>
> This could explain why the performance is closer to the current master
> branch as you indicate:
>
>
> This means that if we remove the GIL + add the 3.11 improvements we should
> get some more speed?
>
> (or if those are integrated in the POC?)
>
>
> Kind Regards,
>
> Abdur-Rahmaan Janhangeer
> about  | blog
> 
> github 
> Mauritius
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MSNUMB5L3KS55HHAEMQZLFOM6JL3RL2B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Sam Gross]

On Mon, Oct 11, 2021 at 12:58 PM Thomas Grainger  wrote:

> Is D1.update(D2) still atomic with this implementation?
> https://docs.python.org/3.11/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe
>

No. For example, another thread reading from the dict concurrently may
observe a partial update.

As Ronald Oussoren points out, dict.update isn't atomic in the general
case. CPython even includes some checks for concurrent modifications:
https://github.com/python/cpython/blob/2f92e2a590f0e5d2d3093549f5af9a4a1889eb5a/Objects/dictobject.c#L2582-L2586
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IAOCDDCJ653NBED3G2J2YBWD7HHPFHT6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Abdur-Rahmaan Janhangeer]

When you mean "an order of magnitude less overhead than the current CPython
implementation" do you mean compared with the main branch? We recently
implemented already almost everything is listed in this paragraph:

https://github.com/python/cpython/pull/27077

We also pack some extra similar optimizations in this other PR, including
stealing the frame arguments from python to python calls:

https://github.com/python/cpython/pull/28488

This could explain why the performance is closer to the current master
branch as you indicate:


This means that if we remove the GIL + add the 3.11 improvements we should
get some more speed?

(or if those are integrated in the POC?)


Kind Regards,

Abdur-Rahmaan Janhangeer
about  | blog

github 
Mauritius
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UKO7C45UCVVIJUUBOSPZLUNR4CC3WNIL/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Sam Gross]

>
> I’m unclear what is actually retried.  You use this note throughout the
> document, so I think it would help to clarify exactly what is retried and
> why that solves the particular problem.  I’m confused because, is it the
> refcount increment that’s retried or the entire sequence of steps (i.e. do
> you go back and reload the address of the item)?  Is there some kind of
> waiting period before the retry?  I would infer that if you’re retrying the
> refcount incrementing, it’s because you expect subsequent retries to
> transition from zero to non-zero, but is that guaranteed?  Are there
> possibilities of deadlocks or race conditions?


The entire operation is retried (not just the refcount). For "dict", this
means going back to step 1 and reloading the version tag and
PyDictKeysObject. The operation can fail (and need to be retried) only when
some other thread is concurrently modifying the dict. The reader needs to
perform the checks (and retry) to avoid returning inconsistent data, such
as an object that was never in the dict. With the checks and retry,
returning inconsistent or garbage data is not possible.

The retry is performed after locking the dict, so the operation is retried
at most once -- the read operation can't fail when it holds the dict's lock
because the lock prevents concurrent modifications. It would have also been
possible to retry the operation in a loop without locking the dict, but I
was concerned about reader starvation. (In the doc I wrote "livelock", but
"reader starvation" is more accurate.) In particular, I was concerned that
a thread repeatedly modifying a dict might prevent other threads reading
the dict from making progress. I hadn't seen this in practice, but I'm
aware that reader starvation can be an issue for similar designs like
Linux's seqlock. Acquiring the dict's lock when retrying avoids the reader
starvation issue.

Deadlock isn't possible because the code does not acquire any other
locks while holding the dict's lock. For example, the code releases the
dict's lock before calling Py_DECREF or PyObject_RichCompareBool.

The race condition question is a bit harder to answer precisely. Concurrent
reads and modifications of a dict won't cause the program to segfault,
return garbage data, or items that were never in the dict.

Regards,
Sam
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D37MQCDRXRVLDVZ65G5BJPJ6QEPSVLI4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Ronald Oussoren via Python-Dev]

> On 11 Oct 2021, at 18:58, Thomas Grainger  wrote:
> 
> Is D1.update(D2) still atomic with this implementation?  
> https://docs.python.org/3.11/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe
>  
> 
AFAIK this is already only atomic in specific circumstances, that are more 
limited than the FAQ appears to claim.

For dict.update to be atomic I’d expect that with two threads performing an 
update on the same keys you’d end up the update of either thread, but not a mix.

That is:

Thread 1:  d.update({“a”: 1, “b”: 1})
Thread 2:  d.update({“a”: 2, “b”: 2})

The result should have d[“a”] == d[“b”].

This can already end up with a mix of the two when “d” has keys that are 
objects that implement __eq__ in Python, because the interpreter could switch 
threads while interpreting __eq__. 

A pathological example:

# — start of script —

import threading
import time

stop = False
trigger = False
def runfunc():
while not stop:
if trigger:
d.update({"a": 2, "b": 2 })
print(d)
break

t = threading.Thread(target=runfunc)
t.start()


class X(str):
def __eq__(self, other):
if threading.current_thread() is t:
return str.__eq__(self, other)

global trigger
trigger = True
t.join()
return str.__eq__(self, other)

def __hash__(self):
return str.__hash__(self)


d = {X("b"):0}
print("before", d)
d.update({"a":1, "b": 1})
print("after", d)

stop = True
t.join()

# — end of script — 

This prints "after {'b': 1, 'a': 2}” on my machine.

Ronald


> 
> On Mon, 11 Oct 2021, 17:54 Sam Gross,  > wrote:
> On Fri, Oct 8, 2021 at 12:04 PM Nathaniel Smith  > wrote:
> I notice the fb.com  address -- is this a personal project or 
> something
> facebook is working on? what's the relationship to Cinder, if any?
> 
> It is a Facebook project, at least in the important sense that I work on it
> as an employee at Facebook. (I'm currently the only person working on it.)
> I keep in touch with some of the Cinder devs regularly and they've advised
> on the project, but otherwise the two projects are unrelated.
>  
> Regarding the tricky lock-free dict/list reads: I guess the more
> straightforward approach would be to use a plain ol' mutex that's
> optimized for this kind of fine-grained per-object lock with short
> critical sections and minimal contention, like WTF::Lock. Did you try
> alternatives like that? If so, I assume they didn't work well -- can
> you give more details?
> 
> I'm using WTF::Lock style locks for dict/list mutations. I did an experiment
> early on where I included locking around reads as well. I think it slowed down
> the pyperformance benchmarks by ~10% on average, but I can't find my notes
> so I plan to re-run the experiment.
> 
> Additionally, because dicts are used for things like global variables, I'd 
> expect
> that locks around reads prevent efficient scaling, but I haven't measured 
> this.
> 
> ___
> Python-Dev mailing list -- python-dev@python.org 
> 
> To unsubscribe send an email to python-dev-le...@python.org 
> 
> https://mail.python.org/mailman3/lists/python-dev.python.org/ 
> 
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/V76ZRBM6UMGYU7FTNENMOOW7OYEFYQ5Q/
>  
> 
> Code of Conduct: http://python.org/psf/codeofconduct/ 
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/5RKLUR2DYJ53OIRX74WVZCVRGW7VUXLF/
> Code of Conduct: http://python.org/psf/codeofconduct/

—

Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4ECNMYHYOOPNL4XHE4GBB5AQN6NPX7QX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Patrick Reader]

On 11/10/2021 16:45, Steven Troxler wrote:
> https://docs.google.com/document/d/1oCRBAAPs3efTYoIRSUwWLtOr1AUVqc8OCEYn4vKYjGI/edit?usp=sharing

I think ParamSpecs should not have any special syntax. `**P` looks way too much 
like kwargs, and any other syntax would be totally new precedent. I'd suggest 
simply `(P) -> bool`, because P is already clearly a ParamSpec because it was 
declared as such. If `Concatenate[T, P]` is clear enough without special syntax 
decorating the `P`, so is `(T, P) ->`.

(I posted this as a comment on the document, but I'm reposting it here as well 
for visibility)

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q6YI6DPSSKCGRBSZSFJMEYIQAORQEV5Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Patrick Reader]

On 11/10/2021 17:03, Steven Troxler wrote:
> Personally, I think we should both adopt function-as-a-type and shorthand 
> syntax, and reject both the XOR and hybrid syntax because function-as-a-type 
> is convenient for expressing the same things (named, default-value, and 
> variadic args).

+1

This combination of a shorthand syntax for simple callbacks and a longhand 
interface-style method for more complex signatures is the best option IMO.

It provides good flexibility, in the pragmatic sense of "there should be only 
one way to do it": there is a dead-simple way which is pretty good, and a more 
flexible way for more complex cases which is pretty easy to switch to if 
necessary.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XFE32UGDI4TAS6BREEHKLOYP3TZF6FFK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Sam Gross]

On Fri, Oct 8, 2021 at 11:35 AM Chris Jerdonek 
wrote:

> Is it also slower even when running with PYTHONGIL=1? If it could be made
> the same speed for single-threaded code when running in GIL-enabled mode,
> that might be an easier intermediate target while still adding value.
>

Running with PYTHONGIL=1 is a bit less than 1% faster (on pyperformance)
than with PYTHONGIL=0. It might be possible to improve PYTHONGIL=1 by
another 1-2% by adding runtime checks for the GIL before attempting to lock
dicts and lists during mutations. I think further optimizations specific to
the PYTHONGIL=1 use case would be tricky.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6YLZMVKWI77SSNUV5XOGBSRY44KJ76UQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Steven Troxler]

> Is this also why re-using an actual callable at a type was rejected?

The idea to use a function as a type (which Łukasz proposed at the PyCon typing 
summit) is still under consideration. Our thought was that it would probably be 
a separate PEP from a syntax change.

Personally, I think we should both adopt function-as-a-type and shorthand 
syntax, and reject both the XOR and hybrid syntax because function-as-a-type is 
convenient for expressing the same things (named, default-value, and variadic 
args).

We don't think function-as-a-type solves all cases for a few reasons:

(1) in existing code, Pradeep's data analysis shows that the use of named 
arguments in calls is quite rare, less than 5% of cases; positional arguments 
are at least 12 times more common.

As a result, we think it's important to have an option where positional 
arguments are convenient. We don't think the `/` or underscore conventions 
would be very convenient; `/` in particular would be easy to forget, which 
would lead to accidentally pinning argument names.

It's easy to imagine library authors not realizing they did this, particularly 
their unit tests use the same argument names.

(2)  Callable is one of the most heavily used types. So we do think the ability 
to inline it may be important.

(3) Around 20% of Callables in existing code Pradeep analyzed require a 
ParamSpec. That's 4 times the number of callables that made use of default 
arguments or named arguments. 

It is not easy to make use of ParamSpec in function-as-a-type, but it is easy 
to support with a new shorthand syntax.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MGJUXXBYLCZZUX3OTHYDOV5OUPGMEAQD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Thomas Grainger]

Is D1.update(D2) still atomic with this implementation?
https://docs.python.org/3.11/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe

On Mon, 11 Oct 2021, 17:54 Sam Gross,  wrote:

> On Fri, Oct 8, 2021 at 12:04 PM Nathaniel Smith  wrote:
>
>> I notice the fb.com address -- is this a personal project or something
>> facebook is working on? what's the relationship to Cinder, if any?
>>
>
> It is a Facebook project, at least in the important sense that I work on it
> as an employee at Facebook. (I'm currently the only person working on it.)
> I keep in touch with some of the Cinder devs regularly and they've advised
> on the project, but otherwise the two projects are unrelated.
>
>
>> Regarding the tricky lock-free dict/list reads: I guess the more
>> straightforward approach would be to use a plain ol' mutex that's
>> optimized for this kind of fine-grained per-object lock with short
>> critical sections and minimal contention, like WTF::Lock. Did you try
>> alternatives like that? If so, I assume they didn't work well -- can
>> you give more details?
>>
>
> I'm using WTF::Lock style locks for dict/list mutations. I did an
> experiment
> early on where I included locking around reads as well. I think it slowed
> down
> the pyperformance benchmarks by ~10% on average, but I can't find my notes
> so I plan to re-run the experiment.
>
> Additionally, because dicts are used for things like global variables, I'd
> expect
> that locks around reads prevent efficient scaling, but I haven't measured
> this.
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/V76ZRBM6UMGYU7FTNENMOOW7OYEFYQ5Q/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5RKLUR2DYJ53OIRX74WVZCVRGW7VUXLF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Steven Troxler]

We drew up a draft syntax to clarify the three styles that have been discussed 
in typing-sig:
- shorthand syntax, which supports exactly what Callable currently supports
- XOR syntax, which would allow a def-like style in addition to shorthand
- hybrid syntax, which extends shorthand to support named, default-value, and 
variadic args

https://docs.google.com/document/d/1oCRBAAPs3efTYoIRSUwWLtOr1AUVqc8OCEYn4vKYjGI/edit?usp=sharing

Within the discussion in typing-sig,
- everyone would like to see support for at least shorthand
- a little over half would like to have the more complex hybrid option
- I don't think anyone has XOR as their first choice, but several of us think 
it might be better than hybrid
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/P5R6P7NLFMCUOJHZL3K2VAQBFZMVGMR6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Sam Gross]

On Fri, Oct 8, 2021 at 12:04 PM Nathaniel Smith  wrote:

> I notice the fb.com address -- is this a personal project or something
> facebook is working on? what's the relationship to Cinder, if any?
>

It is a Facebook project, at least in the important sense that I work on it
as an employee at Facebook. (I'm currently the only person working on it.)
I keep in touch with some of the Cinder devs regularly and they've advised
on the project, but otherwise the two projects are unrelated.


> Regarding the tricky lock-free dict/list reads: I guess the more
> straightforward approach would be to use a plain ol' mutex that's
> optimized for this kind of fine-grained per-object lock with short
> critical sections and minimal contention, like WTF::Lock. Did you try
> alternatives like that? If so, I assume they didn't work well -- can
> you give more details?
>

I'm using WTF::Lock style locks for dict/list mutations. I did an experiment
early on where I included locking around reads as well. I think it slowed
down
the pyperformance benchmarks by ~10% on average, but I can't find my notes
so I plan to re-run the experiment.

Additionally, because dicts are used for things like global variables, I'd
expect
that locks around reads prevent efficient scaling, but I haven't measured
this.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/V76ZRBM6UMGYU7FTNENMOOW7OYEFYQ5Q/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Simon Cross]

On Sun, Oct 10, 2021 at 4:23 PM Facundo Batista 
wrote:
>
> struct.pack_into(format, buffer, offset, v1, v2, ...)

I've encountered this wart with pack and pack_into too.

The current interface makes sense when if v1, v2 are a small number of
items from a data record, but it becomes a bit silly when v1, v2, ... are
the elements of an array of 10 000 integers, for example.

The option to take a list-like argument like you suggest is a good idea,
but maybe this should have its own function or is just outside the scope of
the built-in struct module.

Multiprocessing sort of added support for this via multiprocessing.Array --
see
https://stackoverflow.com/questions/9754034/can-i-create-a-shared-multiarray-or-lists-of-lists-object-in-python-for-multipro.
I haven't looked at what multiprocessing.Array does under the hood.

Summary of the StackOverflow answer for those who don't feel like clicking:

   mp_arr = mp.Array(c.c_double, size)
   # then in each new process create a new numpy array using:
   arr = np.frombuffer(mp_arr.get_obj())

- Simon
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6S6Q437G3JS5TCKJ2SEKQ5DNRYMDZKIK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Antoine Pitrou]

On Sun, 10 Oct 2021 11:19:44 -0300
Facundo Batista  wrote:
> Hello everyone!
> 
> I need to pack a long list of numbers into shared memory, so I thought
> about using `struct.pack_into`.
> 
> Its signature is
> 
> struct.pack_into(format, buffer, offset, v1, v2, ...)
> 
> I have a long list of nums (several millions), ended up doing the following:
> 
> struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
> 
> However, passing all nums as `*args` is very inefficient [0]. So I
> started wondering why we don't have something like:
> 
> struct.pack_into(format, buffer, offset, values=values)
> 
> which would receive the list of values directly.

Just use `numpy.frombuffer` with your shared memory buffer and write
into the Numpy array?
https://numpy.org/doc/stable/reference/generated/numpy.frombuffer.html

When you're looking to efficiently handle large volumes of primitive
values such as integers, chances are Numpy already has the solution for
you.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/56TOXGB3T56KCE2UMGGGH3YAGHWQTWG3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Antoine Pitrou]

On Thu, 7 Oct 2021 15:52:56 -0400
Sam Gross  wrote:
> Hi,
> 
> I've been working on changes to CPython to allow it to run without the
> global interpreter lock. I'd like to share a working proof-of-concept that
> can run without the GIL. The proof-of-concept involves substantial changes
> to CPython internals, but relatively few changes to the C-API. It is
> compatible with many C extensions: extensions must be rebuilt, but usually
> require small or no modifications to source code. I've built compatible
> versions of packages from the scientific Python ecosystem, and they are
> installable through the bundled "pip".
> 
> Source code:
> https://github.com/colesbury/nogil
> 
> Design overview:
> https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit

Impressive work!

Just for the record:
"""
It’s harder to measure aggregate multi-threaded performance because
there aren’t any standard multi-threaded Python benchmarks, but the new
interpreter addresses many of the use cases that failed to scale
efficiently.
"""

It's crude, but you can take a look at `ccbench` in the Tools directory.

Regards

Antoine.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WRT7F2RHHCQ3N2TYEDC6JSIJ4T2ZM6F7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Serhiy Storchaka]

11.10.21 01:35, MRAB пише:
> Maybe what's needed is to add, say, '*' to the format string to indicate
> that multiple values should come from an iterable, e.g.:
> 
>     struct.pack_into(f'{len(nums)}*Q', buf, 0, nums)
> 
> in this case len(nums) from the nums argument.

See https://www.python.org/dev/peps/pep-3118/.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DLG3FLOADSMM2SLRIKDCS3EVW3XAZ3HM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: RFC on Callable Type Syntax

[Łukasz Langa]

> On 7 Oct 2021, at 18:41, S Pradeep Kumar  wrote:
> 
> Note that we considered and rejected using a full def-signature syntax like
> 
> (record: PurchaseRecord, permissions: List[AuthPermission], /) -> 
> FormattedItem
> 
> because it would be more verbose for common cases and could lead to subtle 
> bugs; more details in [3].

Is this also why re-using an actual callable at a type was rejected?

I always found the following more obvious:


def data_to_table(d: Iterable[Mapping[str, float]], *, sort: bool = False, 
reversed: bool = False) -> Table:
...


@dataclass
class Stream:
converter: data_to_table | None

def add_converter(self, converter: data_to_table) -> None:
self.converter = converter


This solves the following problems with the `(P, Q) -> R` proposal:
- how should this look like for "runtime" Python
- how should we teach this
- how can we express callables with complex signatures

One disadvantage of this is that now arguments HAVE TO be named which raises 
questions:
- should they be considered at type checking time?
- how to express "I don't care"?

To this I say:
- yes, they should be considered at runtime (because kwargs have to be anyway)
- ...unless they begin with an underscore

This still leaves a minor problem that you can't have more than one argument 
literally named `_` so you'd have to do `_1`, `_2`, and so on. I don't think 
this is a big problem.

In fact, forcing users to name callable arguments can be added as a fourth 
advantage to this design: making the annotations maximally informative to the 
human reader.

The only remaining disadvantage that can't be addressed is that you can't 
create an *inline* callable type this way. I don't think this is a deal breaker 
as neither TypedDicts, Protocols, nor for this matter any PlainOldClasses can 
be defined inline inside a type annotation.


- Ł


signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OIE3QG4HME4KX2JN7FF5OOZQTCTOX6AP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Serhiy Storchaka]

10.10.21 23:38, Guido van Rossum пише:
> What's exasperating to me about this whole discussion is that nobody has
> shown any reason why this kind of code would be occurring in user code.
> STORE_FAST x LOAD_FAST x seems that it must come from a user writing
> 
>     x = x

No, it comes from pretty common code:

   for x in a:
   y = x**2

   with open() as f:
   for x in f:
   ...

   x = get():
   if x:
   ...

But there is nothing to optimize here. This sequence is already fast. It
can be replaced with "DUP_TOP; STORE_FAST x", but it would not be
faster. Even if we introduce a new combined opcode, it would not be
significantly faster, and it can slow down the eval loop.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EUYIABPM3QKP34ZG3YTHWDRYGQL2EY4P/
Code of Conduct: http://python.org/psf/codeofconduct/