[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Guido van Rossum]

Sorry for the brainfart earlier. We've got a discussion in our "ideas"
database already about this, start reading here:
https://github.com/faster-cpython/ideas/issues/16#issuecomment-881439738

On Sun, Oct 10, 2021 at 2:55 PM Dennis Sweeney 
wrote:

> STORE_FAST can also be caused by the assignment to a loop variable, so
> STORE/LOAD pairs can come about with things like this:
>
> >>> def f():
> ... for x in stuff:
> ... x.work()
> ...
> ...
> >>> from dis import dis
> >>> dis(f)
> 2   0 LOAD_GLOBAL  0 (stuff)
> 2 GET_ITER
> >>4 FOR_ITER 6 (to 18)
> 6 STORE_FAST   0 (x)
>
> 3   8 LOAD_FAST0 (x)
> 10 LOAD_METHOD  1 (work)
> 12 CALL_METHOD  0
> 14 POP_TOP
> 16 JUMP_ABSOLUTE2 (to 4)
>
> 2 >>   18 LOAD_CONST   0 (None)
> 20 RETURN_VALUE
>
> I'd guess that they'd be somewhat common in comprehensions too:
>
> >>> dis(x**2 for x in range(1000))
> 0 GEN_START0
>
> 1   2 LOAD_FAST0 (.0)
> >>4 FOR_ITER 7 (to 20)
> 6 STORE_FAST   1 (x)
> 8 LOAD_FAST1 (x)
> 10 LOAD_CONST   0 (2)
> 12 BINARY_POWER
> 14 YIELD_VALUE
> 16 POP_TOP
> 18 JUMP_ABSOLUTE2 (to 4)
> >>   20 LOAD_CONST   1 (None)
> 22 RETURN_VALUE
>
>
> In fact, there's already a bpo issue from 2019:
> https://bugs.python.org/issue38381
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/73BMYW3TY7PJB7KRQ3Q3OROGU5UJVJAW/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BPDXGJYAMI7UCIDECB5XXQ6VQ4LGH7OG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[MRAB]

On 2021-10-10 19:20, Gregory P. Smith wrote:



On Sun, Oct 10, 2021 at 7:25 AM Facundo Batista 
mailto:facundobati...@gmail.com>> wrote:


Hello everyone!

I need to pack a long list of numbers into shared memory, so I thought
about using `struct.pack_into`.

Its signature is

     struct.pack_into(format, buffer, offset, v1, v2, ...)

I have a long list of nums (several millions), ended up doing the
following:

     struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)

However, passing all nums as `*args` is very inefficient [0]. So I
started wondering why we don't have something like:

     struct.pack_into(format, buffer, offset, values=values)

which would receive the list of values directly.

Is that because my particular case is very uncommon? Or maybe we *do*
want this but we don't have it yet? Or do we already have a better way
of doing this?

Thanks!

[0] https://linkode.org/#95ZZtVCIVtBbx72dURK7a4



My first reaction on seeing things like this is "Why not use a numpy.array?"

Does what you have really need to be a long list?  If so, that's already 
a huge amount of Python object storage as it is. Is it possible for your 
application to have kept that in a numpy array for the entirety of the 
data lifetime? 
https://numpy.org/doc/stable/reference/routines.array-creation.html 



I'm not saying the stdlib shouldn't have a better way to do this by not 
abusing *args as an API, just that other libraries solve the larger 
problem of data-memory-inefficiency in their own way already.


/(neat tricks from others regarding stdlib array, shm, & memoryview even 
if... not ideal)/


Maybe what's needed is to add, say, '*' to the format string to indicate 
that multiple values should come from an iterable, e.g.:


struct.pack_into(f'{len(nums)}*Q', buf, 0, nums)

in this case len(nums) from the nums argument.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6ZJ6B7ILRWH5BU4UAMNGOEJ5AIWRNFEJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Dennis Sweeney]

STORE_FAST can also be caused by the assignment to a loop variable, so 
STORE/LOAD pairs can come about with things like this:

>>> def f():
... for x in stuff:
... x.work()
... 
... 
>>> from dis import dis
>>> dis(f)
2   0 LOAD_GLOBAL  0 (stuff)
2 GET_ITER
>>4 FOR_ITER 6 (to 18)
6 STORE_FAST   0 (x)

3   8 LOAD_FAST0 (x)
10 LOAD_METHOD  1 (work)
12 CALL_METHOD  0
14 POP_TOP
16 JUMP_ABSOLUTE2 (to 4)

2 >>   18 LOAD_CONST   0 (None)
20 RETURN_VALUE

I'd guess that they'd be somewhat common in comprehensions too:

>>> dis(x**2 for x in range(1000))
0 GEN_START0

1   2 LOAD_FAST0 (.0)
>>4 FOR_ITER 7 (to 20)
6 STORE_FAST   1 (x)
8 LOAD_FAST1 (x)
10 LOAD_CONST   0 (2)
12 BINARY_POWER
14 YIELD_VALUE
16 POP_TOP
18 JUMP_ABSOLUTE2 (to 4)
>>   20 LOAD_CONST   1 (None)
22 RETURN_VALUE


In fact, there's already a bpo issue from 2019: 
https://bugs.python.org/issue38381
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/73BMYW3TY7PJB7KRQ3Q3OROGU5UJVJAW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Brandt Bucher]

Guido van Rossum wrote:
> What's exasperating to me about this whole discussion is that nobody has
> shown any reason why this kind of code would be occurring in user code.
> STORE_FAST x LOAD_FAST x seems that it must come from a user writing
> x = x

I think that would be LOAD_FAST(x) STORE_FAST(x). STORE_FAST(x) LOAD_FAST(x) 
could be caused by something like this:
 
def f():
x = …
y = x

Presumably this could be replaced with DUP_TOP() STORE_FAST(x)? But I don’t 
know if that’s any faster. It does require more stack space…

> Are we just a solution (a clever peephole
> trick) looking for a problem?

It seems so.

Brandt
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XGFBCVDIEXEBKFGUOHSXJD3GKO7IAYLM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Python multithreading without the GIL

[Barry Warsaw]

Congrats on this impressive work Sam.  I enjoyed the thorough write up of the 
design.  There’s one aspect that I don’t quite understand.  Maybe I missed the 
explanation.  For example:

```
• Load the address of the item
• Increment the reference count of the item, if it is non-zero 
(otherwise retry)
• Return the address of the item

(We typically acquire the per-collection mutex when retrying operations to 
avoid potential livelock issues.)
```

I’m unclear what is actually retried.  You use this note throughout the 
document, so I think it would help to clarify exactly what is retried and why 
that solves the particular problem.  I’m confused because, is it the refcount 
increment that’s retried or the entire sequence of steps (i.e. do you go back 
and reload the address of the item)?  Is there some kind of waiting period 
before the retry?  I would infer that if you’re retrying the refcount 
incrementing, it’s because you expect subsequent retries to transition from 
zero to non-zero, but is that guaranteed?  Are there possibilities of deadlocks 
or race conditions?

Can you go into some more detail (here or in the document) about how this works?

Cheers,
-Barry

> On Oct 7, 2021, at 12:52, Sam Gross  wrote:
> 
> Hi,
> 
> I've been working on changes to CPython to allow it to run without the global 
> interpreter lock. I'd like to share a working proof-of-concept that can run 
> without the GIL. The proof-of-concept involves substantial changes to CPython 
> internals, but relatively few changes to the C-API. It is compatible with 
> many C extensions: extensions must be rebuilt, but usually require small or 
> no modifications to source code. I've built compatible versions of packages 
> from the scientific Python ecosystem, and they are installable through the 
> bundled "pip".
> 
> Source code:
> https://github.com/colesbury/nogil
> 
> Design overview:
> https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsDFosB5e6BfnXLlejd9l0/edit
> 
> My goal with the proof-of-concept is to demonstrate that removing the GIL is 
> feasible and worthwhile, and that the technical ideas of the project could 
> serve as a basis of such an effort.
> 
> I'd like to start a discussion about these ideas and gauge the community's 
> interest in this approach to removing the GIL.
> 
> Regards,
> Sam Gross
> colesb...@gmail.com / sgr...@fb.com
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/ABR2L6BENNA6UPSPKV474HCS4LWT26GY/
> Code of Conduct: http://python.org/psf/codeofconduct/



signature.asc
Description: Message signed with OpenPGP
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GQOSFIG55U6LZKHO34DS55TGERPX5UVS/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Guido van Rossum]

On Sun, Oct 10, 2021 at 12:27 PM Patrick Reader <_...@pxeger.com> wrote:

> On 10/10/2021 18:33, Guido van Rossum wrote:
>
> On Sun, Oct 10, 2021 at 10:28 AM Brandt Bucher 
> wrote:
>
>> the peephole optimizer shouldn’t be tasked with “fixing” poorly-written
>> or uncommon code… just improving common code.
>>
> Good point.
>
>
> As soon as this pattern appears in a benchmark suite we'll optimize it
> away. :-) (That is, *if* the optimizer can prove that it can't raise a name
> error.)
>
> Perhaps a more realistic case that definitely doesn't require a check to
> prevent NameErrors would be optimising `STORE_FAST x LOAD_FAST x` for same
> x. But as Steven D'Aprano has just pointed out, they're FAST lookups
> anyway, so the benefit may well be minimal anyway.
>

What's exasperating to me about this whole discussion is that nobody has
shown any reason why this kind of code would be occurring in user code.
STORE_FAST x LOAD_FAST x seems that it must come from a user writing

x = x

But why would they write that? Are we just a solution (a clever peephole
trick) looking for a problem?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UH4B3J7UHPWETBZYZFDMPSOKHRSRY372/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Serhiy Storchaka]

10.10.21 19:05, Patrick Reader пише:
> This still isn't a completely direct write - you're still creating an array 
> in between which is then copied into shm, whereas struct.pack_into writes 
> directly into the shared memory with no intermediate buffer.
> 
> And unfortunately you can't do
> 
> shm.buf[l_offset:r_offset].cast('Q')[:] = nums
> 
> where nums is a plain Python iterable; it has to already be an array, so it 
> still requires a copy.

Yes, I tried this too. Maybe we will add a support of arbitrary
iterables. There may be problems with this because we don't know the
length of the iterable and can't guarantee that all items are
convertible without converting them and saving results in a temporary array.

If you do not want to spend additional memory, iterate nums and set
items one by one. Or iterate and set them by chunks

   target = shm.buf[l_offset:r_offset].cast('Q')
   for i in range(0, len(nums), chunksize):
   j = min(i + chunksize, len(nums))
   target[i:j] = array.array('Q', nums[i:j])

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4IYA25GXYMUQ5CHORD7YYYVUUUDCGWAF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Patrick Reader]

On 10/10/2021 18:33, Guido van Rossum wrote:
> On Sun, Oct 10, 2021 at 10:28 AM Brandt Bucher  wrote:
>
> the peephole optimizer shouldn’t be tasked with “fixing” poorly-written 
> or uncommon code… just improving common code.
>
Good point.
>
> As soon as this pattern appears in a benchmark suite we'll optimize it away. 
> :-) (That is, *if* the optimizer can prove that it can't raise a name error.)
Perhaps a more realistic case that definitely doesn't require a check to 
prevent NameErrors would be optimising `STORE_FAST x LOAD_FAST x` for same x. 
But as Steven D'Aprano has just pointed out, they're FAST lookups anyway, so 
the benefit may well be minimal anyway.___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JNDLOPUWPELH55WL4B5YFDGRIMHOIF2P/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Gregory P. Smith]

On Sun, Oct 10, 2021 at 7:25 AM Facundo Batista 
wrote:

> Hello everyone!
>
> I need to pack a long list of numbers into shared memory, so I thought
> about using `struct.pack_into`.
>
> Its signature is
>
> struct.pack_into(format, buffer, offset, v1, v2, ...)
>
> I have a long list of nums (several millions), ended up doing the
> following:
>
> struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
>
> However, passing all nums as `*args` is very inefficient [0]. So I
> started wondering why we don't have something like:
>
> struct.pack_into(format, buffer, offset, values=values)
>
> which would receive the list of values directly.
>
> Is that because my particular case is very uncommon? Or maybe we *do*
> want this but we don't have it yet? Or do we already have a better way
> of doing this?
>
> Thanks!
>
> [0] https://linkode.org/#95ZZtVCIVtBbx72dURK7a4


My first reaction on seeing things like this is "Why not use a numpy.array?"

Does what you have really need to be a long list?  If so, that's already a
huge amount of Python object storage as it is. Is it possible for your
application to have kept that in a numpy array for the entirety of the data
lifetime?
https://numpy.org/doc/stable/reference/routines.array-creation.html

I'm not saying the stdlib shouldn't have a better way to do this by not
abusing *args as an API, just that other libraries solve the larger problem
of data-memory-inefficiency in their own way already.

*(neat tricks from others regarding stdlib array, shm, & memoryview even
if... not ideal)*

-gps
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YD7WP6F3LEXFSEITVPZUNONVUBK3AKUW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Patrick Reader]

On 10/10/2021 18:26, Brandt Bucher wrote:
> The first reason is that this operation *does* have a side-effect: if a fast 
> local is unbound, the load will raise a NameError!
>
> def f():
>  x  # This should always raise.
>  x = None  # This makes x a fast local.
Ah, ok, that certainly explains it - thanks!
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GLQ4LXLZZMRLDQQJTDM2CZOFNJ62I6JE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Steven D'Aprano]

On Sun, Oct 10, 2021 at 09:15:30AM +, Patrick Reader wrote:

> - a LOAD_FAST cannot possibly have any side-effects outside the 
> interpreter stack [1]

> [1]: global variables are probably supposed to have the same 
> guarantee, but in practice this is not the case

I don't think that lookups are guaranteed to be side-effect free, since 
raising an exception is a side-effect.

def func():
x

can raise NameError if there is no global or builtin "x". So that cannot 
be optimized away. Similarly for locals:

def func(x):
if random.random() > 0.5:
del x
x

Will that raise or not? No way of telling without actually running the 
code.

Now I guess a sufficiently smart optimizer might be able to optimize 
away the lookup of a local variable if it cannot possibly be undefined, 
as in your initial example, but that brings us to how much benefit is 
gained for how much developer effort and code complexity.

If the benefit is small, or negligible, then the developer effort 
had better be even smaller.

In this case, the cost of the lookup is small (it is a *FAST* lookup). 
And furthermore, the likelihood that anyone will actually write that 
code is even smaller.

(There are times that we might test for the existence of a global, by 
looking up the name. There might even be times that we would do the same 
for a local. We might even, under some rare circumstances, do so without 
guarding it in a try...except block. But why on earth would we test for 
the existence of a local that cannot possibly fail to exist, which is 
the only case that is safe to optimize away?)

So the benefit is correspondingly tiny, and optimizing it away had 
better be effectively free in both developer effort and ongoing code 
complexity to make it worthwhile.


-- 
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/HWSHZLW64Q26EIUPX4S3TME5QXYBRLPY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Guido van Rossum]

On Sun, Oct 10, 2021 at 10:28 AM Brandt Bucher 
wrote:

> the peephole optimizer shouldn’t be tasked with “fixing” poorly-written or
> uncommon code… just improving common code.
>

As soon as this pattern appears in a benchmark suite we'll optimize it
away. :-) (That is, *if* the optimizer can prove that it can't raise a name
error.)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XBUGXARBK476SC5JXK3PXODQQ2LS7KTB/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Brandt Bucher]

I can think of two reasons.

The first reason is that this operation *does* have a side-effect: if a fast 
local is unbound, the load will raise a NameError!

def f():
 x  # This should always raise.
 x = None  # This makes x a fast local.

The second reason is one that Guido already alluded to: the peephole optimizer 
shouldn’t be tasked with “fixing” poorly-written or uncommon code… just 
improving common code.

If anything, we would probably just warn here. But even that seems like too 
much.

Brandt
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TJKKOVORTSM5QVRV5WXMUUKBUDYN2IQ7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Why doesn't peephole optimise away operations with fast locals?

[Guido van Rossum]

Since this is unrealistic code (why would you have a top-level expression
without any side effects), what does it matter whether this code is
optimized? Did you encounter this in a real use case?

On Sun, Oct 10, 2021 at 9:59 AM Patrick Reader <_...@pxeger.com> wrote:

> Consider sequences of bytecode operations on fast locals like the
> following:
>
> >>> def f(x):  # declare as parameter so that x is a fast local
> ... x
> ...
> >>> dis(f)
>   2   0 LOAD_FAST0 (x)
>   2 POP_TOP
>   4 LOAD_CONST   0 (None)
>   6 RETURN_VALUE
>
> Given the following assumptions:
>
> - a LOAD_FAST cannot possibly have any side-effects outside the
> interpreter stack [1]
>
> - a POP_TOP immediately undoes LOAD_FAST's effects on the interpreter stack
>
> I am wondering: why doesn't the peephole optimiser remove these opcode
> constructs?
>
> Is it maybe because of PEP 626 - an introspection tool needs to know that
> the variable is being used there? Although surely in that case the peephole
> optimiser could just replace it with a single NOP? (c.f.:
> https://bugs.python.org/issue42719)
>
> Or is one of my assumptions wrong?
>
> [1]: global variables are probably supposed to have the same guarantee,
> but in practice this is not the case, which is why I'm focusing on the
> _FAST ones; see for example
> https://ato.pxeger.com/run?1=m72soLIkIz9vwYKlpSVpuhY3JyXnJBYXK_hWumQml2ikAAlNKy4FIEhJTVOIj09PLcksSc2Nj9coTs1J01HITq2EyoNAQVFmXomGUnFmSqpCalpaanKJoqKSJly6KLWktChPobi0ILVIQ1MP2TSQOVxcqRWpyRpKFUo6MPsrbA01NSEugzoQ5lAA
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/767FGRV4ZL5IVBHWSAW5TJGQMGQS244Z/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WNFD6UCU6AND6TMA5M2LXNAXPXY577AU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Patrick Reader]

You can take a memory view of the array directly:

memoryview(array.array("Q", range(1000)))

If your exact use-case is writing to a SharedMemory, then I don't think there 
is any simple way to do it without creating some intermediate memory buffer. 
(other than using struct.pack_into, or packing the values completely manually)

On 10/10/2021 16:18, Facundo Batista wrote:
> El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka
> (storch...@gmail.com) escribió:
>> 10.10.21 17:19, Facundo Batista пише:
>>> I have a long list of nums (several millions), ended up doing the following:
>>>
>>> struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
>> Why not use array('Q', nums)?
> You mean `array` from the `array` module? The only way I see using it
> is like the following:
>
 shm = shared_memory.SharedMemory(create=True, size=total_size)
 a = array.array('Q', nums)
 shm.buf[l_offset:r_offset] = a.tobytes()
> But I don't like it because of the `tobytes` call, which will produce
> a huge bytearray only to insert it in the shared memory buffer.
>
> That's why I liked `pack_into`, because it will write directly into
> the memory view.
>
> Or I'm missing something?
>
> Thanks!
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ONWPTKL3P6VOZEHUDCL7B6CZ5TI3DER6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Why doesn't peephole optimise away operations with fast locals?

[Patrick Reader]

Consider sequences of bytecode operations on fast locals like the following:

>>> def f(x):  # declare as parameter so that x is a fast local
... x
...
>>> dis(f)
  2   0 LOAD_FAST    0 (x)
  2 POP_TOP
  4 LOAD_CONST   0 (None)
  6 RETURN_VALUE

Given the following assumptions:

- a LOAD_FAST cannot possibly have any side-effects outside the interpreter 
stack [1]

- a POP_TOP immediately undoes LOAD_FAST's effects on the interpreter stack

I am wondering: why doesn't the peephole optimiser remove these opcode 
constructs?

Is it maybe because of PEP 626 - an introspection tool needs to know that the 
variable is being used there? Although surely in that case the peephole 
optimiser could just replace it with a single NOP? (c.f.: 
https://bugs.python.org/issue42719)

Or is one of my assumptions wrong?

[1]: global variables are probably supposed to have the same guarantee, but in 
practice this is not the case, which is why I'm focusing on the _FAST ones; see 
for example 
https://ato.pxeger.com/run?1=m72soLIkIz9vwYKlpSVpuhY3JyXnJBYXK_hWumQml2ikAAlNKy4FIEhJTVOIj09PLcksSc2Nj9coTs1J01HITq2EyoNAQVFmXomGUnFmSqpCalpaanKJoqKSJly6KLWktChPobi0ILVIQ1MP2TSQOVxcqRWpyRpKFUo6MPsrbA01NSEugzoQ5lAA

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/767FGRV4ZL5IVBHWSAW5TJGQMGQS244Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Patrick Reader]

This still isn't a completely direct write - you're still creating an array in 
between which is then copied into shm, whereas struct.pack_into writes directly 
into the shared memory with no intermediate buffer.

And unfortunately you can't do

shm.buf[l_offset:r_offset].cast('Q')[:] = nums

where nums is a plain Python iterable; it has to already be an array, so it 
still requires a copy.

On 10/10/2021 16:57, Serhiy Storchaka wrote:
> 10.10.21 18:18, Facundo Batista пише:
>> You mean `array` from the `array` module? The only way I see using it
>> is like the following:
>>
> shm = shared_memory.SharedMemory(create=True, size=total_size)
> a = array.array('Q', nums)
> shm.buf[l_offset:r_offset] = a.tobytes()
>> But I don't like it because of the `tobytes` call, which will produce
>> a huge bytearray only to insert it in the shared memory buffer.
>>
>> That's why I liked `pack_into`, because it will write directly into
>> the memory view.
>>
>> Or I'm missing something?
>shm.buf[l_offset:r_offset].cast('Q')[:] = a
>
> or
>
>shm.buf[l_offset:r_offset] = memoryview(a).cast('B')
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/A2XKNKG4C55VKII6VOXRR3XEP6VHIU43/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IAOHC7LGAKNXMIIGAZI3ALJ6MJVFEEBI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Serhiy Storchaka]

10.10.21 18:18, Facundo Batista пише:
> You mean `array` from the `array` module? The only way I see using it
> is like the following:
> 
 shm = shared_memory.SharedMemory(create=True, size=total_size)
 a = array.array('Q', nums)
 shm.buf[l_offset:r_offset] = a.tobytes()
> 
> But I don't like it because of the `tobytes` call, which will produce
> a huge bytearray only to insert it in the shared memory buffer.
> 
> That's why I liked `pack_into`, because it will write directly into
> the memory view.
> 
> Or I'm missing something?

   shm.buf[l_offset:r_offset].cast('Q')[:] = a

or

   shm.buf[l_offset:r_offset] = memoryview(a).cast('B')


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/A2XKNKG4C55VKII6VOXRR3XEP6VHIU43/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Guido van Rossum]

Maybe instead of tobytes() you can use memoryview().

On Sun, Oct 10, 2021 at 08:21 Facundo Batista 
wrote:

> El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka
> (storch...@gmail.com) escribió:
> >
> > 10.10.21 17:19, Facundo Batista пише:
> > > I have a long list of nums (several millions), ended up doing the
> following:
> > >
> > > struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
> >
> > Why not use array('Q', nums)?
>
> You mean `array` from the `array` module? The only way I see using it
> is like the following:
>
> >>> shm = shared_memory.SharedMemory(create=True, size=total_size)
> >>> a = array.array('Q', nums)
> >>> shm.buf[l_offset:r_offset] = a.tobytes()
>
> But I don't like it because of the `tobytes` call, which will produce
> a huge bytearray only to insert it in the shared memory buffer.
>
> That's why I liked `pack_into`, because it will write directly into
> the memory view.
>
> Or I'm missing something?
>
> Thanks!
>
> --
> .Facundo
>
> Blog: http://www.taniquetil.com.ar/plog/
> PyAr: http://www.python.org.ar/
> Twitter: @facundobatista
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/52NGDRMD3BR4ZTT3KGF6FNNGQIIPXBWY/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-- 
--Guido (mobile)
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5NTTZC54FNPSO3VDCF6NIJ75NKNTFIJY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Facundo Batista]

El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka
(storch...@gmail.com) escribió:
>
> 10.10.21 17:19, Facundo Batista пише:
> > I have a long list of nums (several millions), ended up doing the following:
> >
> > struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
>
> Why not use array('Q', nums)?

You mean `array` from the `array` module? The only way I see using it
is like the following:

>>> shm = shared_memory.SharedMemory(create=True, size=total_size)
>>> a = array.array('Q', nums)
>>> shm.buf[l_offset:r_offset] = a.tobytes()

But I don't like it because of the `tobytes` call, which will produce
a huge bytearray only to insert it in the shared memory buffer.

That's why I liked `pack_into`, because it will write directly into
the memory view.

Or I'm missing something?

Thanks!

-- 
.Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org.ar/
Twitter: @facundobatista
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/52NGDRMD3BR4ZTT3KGF6FNNGQIIPXBWY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Packing a long list of numbers into memory

[Serhiy Storchaka]

10.10.21 17:19, Facundo Batista пише:
> I have a long list of nums (several millions), ended up doing the following:
> 
> struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)

Why not use array('Q', nums)?

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OWZXJRTQPZZKBUM4KMZWDWTBFELRAI46/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Packing a long list of numbers into memory

[Facundo Batista]

Hello everyone!

I need to pack a long list of numbers into shared memory, so I thought
about using `struct.pack_into`.

Its signature is

struct.pack_into(format, buffer, offset, v1, v2, ...)

I have a long list of nums (several millions), ended up doing the following:

struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)

However, passing all nums as `*args` is very inefficient [0]. So I
started wondering why we don't have something like:

struct.pack_into(format, buffer, offset, values=values)

which would receive the list of values directly.

Is that because my particular case is very uncommon? Or maybe we *do*
want this but we don't have it yet? Or do we already have a better way
of doing this?

Thanks!

[0] https://linkode.org/#95ZZtVCIVtBbx72dURK7a4

-- 
.Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org.ar/
Twitter: @facundobatista
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4WYEIBOXPQPQ5EQL5WRYI6X7W36I6UQJ/
Code of Conduct: http://python.org/psf/codeofconduct/