[Numpy-discussion] Re: Performance mystery

2022-01-18 Thread Stefan van der Walt
On Tue, Jan 18, 2022, at 21:55, Warren Weckesser wrote: > expr = 'z.real**2 + z.imag**2' > > z = generate_sample(n, rng) 🤔 If I duplicate the `z = ...` line, I get the fast result throughout. If, however, I use `generate_sample(1, rng)` (or any other value than `n`), it does not improve matters

[Numpy-discussion] Re: Performance mystery

2022-01-18 Thread Andrew Nelson
Might be a 'warmup' of the timeit module? ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address

[Numpy-discussion] Re: Performance mystery

2022-01-19 Thread Francesc Alted
On Wed, Jan 19, 2022 at 7:33 AM Stefan van der Walt wrote: > On Tue, Jan 18, 2022, at 21:55, Warren Weckesser wrote: > > expr = 'z.real**2 + z.imag**2' > > > > z = generate_sample(n, rng) > > 🤔 If I duplicate the `z = ...` line, I get the fast result throughout. > If, however, I use `generate_sam

[Numpy-discussion] Re: Performance mystery

2022-01-19 Thread Andras Deak
On Wed, Jan 19, 2022 at 11:50 AM Francesc Alted wrote: > > > On Wed, Jan 19, 2022 at 7:33 AM Stefan van der Walt > wrote: > >> On Tue, Jan 18, 2022, at 21:55, Warren Weckesser wrote: >> > expr = 'z.real**2 + z.imag**2' >> > >> > z = generate_sample(n, rng) >> >> 🤔 If I duplicate the `z = ...` li

[Numpy-discussion] Re: Performance mystery

2022-01-19 Thread Sebastian Berg
On Wed, 2022-01-19 at 11:49 +0100, Francesc Alted wrote: > On Wed, Jan 19, 2022 at 7:33 AM Stefan van der Walt > > wrote: > > > On Tue, Jan 18, 2022, at 21:55, Warren Weckesser wrote: > > > expr = 'z.real**2 + z.imag**2' > > > > > > z = generate_sample(n, rng) > > > > 🤔 If I duplicate the `z =

[Numpy-discussion] Re: Performance mystery

2022-01-19 Thread Francesc Alted
On Wed, Jan 19, 2022 at 6:58 PM Stanley Seibert wrote: > Given that this seems to be Linux only, is this related to how glibc does > large allocations (>128kB) using mmap()? > > https://stackoverflow.com/a/33131385 > That's a good point. As MMAP_THRESHOLD is 128 KB, and the size of `z` is almost

[Numpy-discussion] Re: Performance mystery

2022-01-19 Thread Jerome Kieffer
On Wed, 19 Jan 2022 19:48:32 +0100 Francesc Alted wrote: > What puzzles me is that the timeit loops access `z` data 3*1 > times, which is plenty of time for doing the allocation (just should > require just a single iteration). Hi all, Very interesting discussion ... IIRC, timeit does some c

[Numpy-discussion] Re: Performance mystery

2022-01-19 Thread Sebastian Berg
On Wed, 2022-01-19 at 19:48 +0100, Francesc Alted wrote: > On Wed, Jan 19, 2022 at 6:58 PM Stanley Seibert > > wrote: > > > Given that this seems to be Linux only, is this related to how > > glibc does > > large allocations (>128kB) using mmap()? > > > > https://stackoverflow.com/a/33131385 > >

[Numpy-discussion] Re: Performance mystery

2022-01-19 Thread Sebastian Berg
On Wed, 2022-01-19 at 20:49 +0100, Jerome Kieffer wrote: > On Wed, 19 Jan 2022 19:48:32 +0100 > Francesc Alted wrote: > > > What puzzles me is that the timeit loops access `z` data 3*1 > > times, which is plenty of time for doing the allocation (just > > should > > require just a single itera

[Numpy-discussion] Re: Performance mystery

2022-01-20 Thread Francesc Alted
On Wed, Jan 19, 2022 at 7:48 PM Francesc Alted wrote: > On Wed, Jan 19, 2022 at 6:58 PM Stanley Seibert > wrote: > >> Given that this seems to be Linux only, is this related to how glibc does >> large allocations (>128kB) using mmap()? >> >> https://stackoverflow.com/a/33131385 >> > > That's a g

[Numpy-discussion] Re: Performance mystery

2022-01-20 Thread Sebastian Berg
On Thu, 2022-01-20 at 14:41 +0100, Francesc Alted wrote: > On Wed, Jan 19, 2022 at 7:48 PM Francesc Alted > wrote: > > > On Wed, Jan 19, 2022 at 6:58 PM Stanley Seibert > > > > wrote: > > > > > Given that this seems to be Linux only, is this related to how > > > glibc does > > > large allocatio

[Numpy-discussion] Re: Performance mystery

2022-01-20 Thread Francesc Alted
On Thu, Jan 20, 2022 at 4:10 PM Sebastian Berg wrote: > Thanks for figuring this out! It has been bugging me a lot before. So > it rather depends on how `malloc` works, and not the kernel. It is > surprising how "random" this can look, but I suppose some examples just > happen to sit almost e

[Numpy-discussion] Re: Performance mystery

2022-01-23 Thread Warren Weckesser
On 1/20/22, Francesc Alted wrote: > On Wed, Jan 19, 2022 at 7:48 PM Francesc Alted wrote: > >> On Wed, Jan 19, 2022 at 6:58 PM Stanley Seibert >> wrote: >> >>> Given that this seems to be Linux only, is this related to how glibc >>> does >>> large allocations (>128kB) using mmap()? >>> >>> https

[Numpy-discussion] Re: Performance mystery

2022-01-24 Thread Francesc Alted
On Mon, Jan 24, 2022 at 2:15 AM Warren Weckesser wrote: > Thanks Sebastian for pointing out that the issue is (minor) page > faults, and thanks Francesc for providing the links and the analysis > of the situation. After reading those links and experimenting with > malloc in a couple C programs,

[Numpy-discussion] Re: Performance mystery

2022-01-24 Thread Jonathan Fine
On Mon, Jan 24, 2022 at 10:02 AM Francesc Alted wrote: > Thanks for going down to the rabbit hole. It is now much clearer what's > going on. > > All in all, hats off to Warren for this entertaining piece of > investigation! > I can't resist. You'd expect someone called https://en.wikipedia.or

[Numpy-discussion] Re: Performance mystery

2022-01-24 Thread Francesc Alted
On Mon, Jan 24, 2022 at 11:01 AM Francesc Alted wrote: > On Mon, Jan 24, 2022 at 2:15 AM Warren Weckesser < > warren.weckes...@gmail.com> wrote: > >> Thanks Sebastian for pointing out that the issue is (minor) page >> faults, and thanks Francesc for providing the links and the analysis >> of the

[Numpy-discussion] Re: Performance mystery

2022-01-25 Thread Francesc Alted
On Mon, Jan 24, 2022 at 1:42 PM Francesc Alted wrote: > On Mon, Jan 24, 2022 at 11:01 AM Francesc Alted wrote: > >> On Mon, Jan 24, 2022 at 2:15 AM Warren Weckesser < >> warren.weckes...@gmail.com> wrote: >> >>> Thanks Sebastian for pointing out that the issue is (minor) page >>> faults, and tha