Hello everyone!
I need to pack a long list of numbers into shared memory, so I thought
about using `struct.pack_into`.
Its signature is
struct.pack_into(format, buffer, offset, v1, v2, ...)
I have a long list of nums (several millions), ended up doing the following:
struct.pack_into(f'
10.10.21 17:19, Facundo Batista пише:
> I have a long list of nums (several millions), ended up doing the following:
>
> struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
Why not use array('Q', nums)?
___
Python-Dev mailing list -- python-dev@python
El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka
(storch...@gmail.com) escribió:
>
> 10.10.21 17:19, Facundo Batista пише:
> > I have a long list of nums (several millions), ended up doing the following:
> >
> > struct.pack_into(f'{len(nums)}Q', buf, 0, *nums)
>
> Why not use array('Q
Maybe instead of tobytes() you can use memoryview().
On Sun, Oct 10, 2021 at 08:21 Facundo Batista
wrote:
> El dom, 10 de oct. de 2021 a la(s) 11:50, Serhiy Storchaka
> (storch...@gmail.com) escribió:
> >
> > 10.10.21 17:19, Facundo Batista пише:
> > > I have a long list of nums (several million
10.10.21 18:18, Facundo Batista пише:
> You mean `array` from the `array` module? The only way I see using it
> is like the following:
>
shm = shared_memory.SharedMemory(create=True, size=total_size)
a = array.array('Q', nums)
shm.buf[l_offset:r_offset] = a.tobytes()
>
> But I don'
This still isn't a completely direct write - you're still creating an array in
between which is then copied into shm, whereas struct.pack_into writes directly
into the shared memory with no intermediate buffer.
And unfortunately you can't do
shm.buf[l_offset:r_offset].cast('Q')[:] = nums
where
Consider sequences of bytecode operations on fast locals like the following:
>>> def f(x): # declare as parameter so that x is a fast local
... x
...
>>> dis(f)
2 0 LOAD_FAST 0 (x)
2 POP_TOP
4 LOAD_CONST 0 (None)
You can take a memory view of the array directly:
memoryview(array.array("Q", range(1000)))
If your exact use-case is writing to a SharedMemory, then I don't think there
is any simple way to do it without creating some intermediate memory buffer.
(other than using struct.pack_into, or packing t
Since this is unrealistic code (why would you have a top-level expression
without any side effects), what does it matter whether this code is
optimized? Did you encounter this in a real use case?
On Sun, Oct 10, 2021 at 9:59 AM Patrick Reader <_...@pxeger.com> wrote:
> Consider sequences of bytec
I can think of two reasons.
The first reason is that this operation *does* have a side-effect: if a fast
local is unbound, the load will raise a NameError!
def f():
x # This should always raise.
x = None # This makes x a fast local.
The second reason is one that Guido already allude
On Sun, Oct 10, 2021 at 10:28 AM Brandt Bucher
wrote:
> the peephole optimizer shouldn’t be tasked with “fixing” poorly-written or
> uncommon code… just improving common code.
>
As soon as this pattern appears in a benchmark suite we'll optimize it
away. :-) (That is, *if* the optimizer can prov
On Sun, Oct 10, 2021 at 09:15:30AM +, Patrick Reader wrote:
> - a LOAD_FAST cannot possibly have any side-effects outside the
> interpreter stack [1]
> [1]: global variables are probably supposed to have the same
> guarantee, but in practice this is not the case
I don't think that lookups
On 10/10/2021 18:26, Brandt Bucher wrote:
> The first reason is that this operation *does* have a side-effect: if a fast
> local is unbound, the load will raise a NameError!
>
> def f():
> x # This should always raise.
> x = None # This makes x a fast local.
Ah, ok, that certainly expl
On Sun, Oct 10, 2021 at 7:25 AM Facundo Batista
wrote:
> Hello everyone!
>
> I need to pack a long list of numbers into shared memory, so I thought
> about using `struct.pack_into`.
>
> Its signature is
>
> struct.pack_into(format, buffer, offset, v1, v2, ...)
>
> I have a long list of nums (
On 10/10/2021 18:33, Guido van Rossum wrote:
> On Sun, Oct 10, 2021 at 10:28 AM Brandt Bucher wrote:
>
> the peephole optimizer shouldn’t be tasked with “fixing” poorly-written
> or uncommon code… just improving common code.
>
Good point.
>
> As soon as this pattern appears in a benchmark sui
10.10.21 19:05, Patrick Reader пише:
> This still isn't a completely direct write - you're still creating an array
> in between which is then copied into shm, whereas struct.pack_into writes
> directly into the shared memory with no intermediate buffer.
>
> And unfortunately you can't do
>
> sh
On Sun, Oct 10, 2021 at 12:27 PM Patrick Reader <_...@pxeger.com> wrote:
> On 10/10/2021 18:33, Guido van Rossum wrote:
>
> On Sun, Oct 10, 2021 at 10:28 AM Brandt Bucher
> wrote:
>
>> the peephole optimizer shouldn’t be tasked with “fixing” poorly-written
>> or uncommon code… just improving comm
Congrats on this impressive work Sam. I enjoyed the thorough write up of the
design. There’s one aspect that I don’t quite understand. Maybe I missed the
explanation. For example:
```
• Load the address of the item
• Increment the reference count of the item, if it is non-zer
Guido van Rossum wrote:
> What's exasperating to me about this whole discussion is that nobody has
> shown any reason why this kind of code would be occurring in user code.
> STORE_FAST x LOAD_FAST x seems that it must come from a user writing
> x = x
I think that would be LOAD_FAST(x) STORE_FAST(
STORE_FAST can also be caused by the assignment to a loop variable, so
STORE/LOAD pairs can come about with things like this:
>>> def f():
... for x in stuff:
... x.work()
...
...
>>> from dis import dis
>>> dis(f)
2 0 LOAD_GLOBAL
On 2021-10-10 19:20, Gregory P. Smith wrote:
On Sun, Oct 10, 2021 at 7:25 AM Facundo Batista
mailto:facundobati...@gmail.com>> wrote:
Hello everyone!
I need to pack a long list of numbers into shared memory, so I thought
about using `struct.pack_into`.
Its signature is
Sorry for the brainfart earlier. We've got a discussion in our "ideas"
database already about this, start reading here:
https://github.com/faster-cpython/ideas/issues/16#issuecomment-881439738
On Sun, Oct 10, 2021 at 2:55 PM Dennis Sweeney
wrote:
> STORE_FAST can also be caused by the assignment
22 matches
Mail list logo
