[issue10227] Improve performance of MemoryView slicing

Stefan Krah added the comment: Great. I removed the dependency since it's fixed in both cpython and pep-3118. -- dependencies: -Problems with Py_buffer management in memoryobject.c (and elsewhere?) stage: -> committed/rejected status: open -> closed _

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: Sure. Flagging this as fixed. Can´t close it until 10181 is closed due to some dependency thing. (perhaps someone else knows what to do?) -- resolution: -> fixed ___ Python tracker

[issue10227] Improve performance of MemoryView slicing

Stefan Krah added the comment: Kristján, I ran the benchmarks from http://bugs.python.org/issue10227#msg143731 in the current cpython and pep-3118 repos. In both cases the differences between Linux and Windows are far less pronounced than they used to be. All benchmarks were run with the x64 bui

[issue10227] Improve performance of MemoryView slicing

Antoine Pitrou added the comment: Thanks Stefan. I'm leaving the issue open since the original topic is a bit different. -- ___ Python tracker ___ _

[issue10227] Improve performance of MemoryView slicing

Roundup Robot added the comment: New changeset fa2f8dd077e0 by Antoine Pitrou in branch 'default': Issue #10227: Add an allocation cache for a single slice object. http://hg.python.org/cpython/rev/fa2f8dd077e0 -- nosy: +python-dev ___ Python tracker

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: Updated single slice caching patch for latest Py3.3 hg tip. -- Added file: http://bugs.python.org/file23727/slice-object-cache.patch ___ Python tracker

[issue10227] Improve performance of MemoryView slicing

Stefan Krah added the comment: With Stefan Behnel's slice-object-cache.patch, I get this (PEP-3118 branch): Linux: bytes: 0.097 usec bytearray: 0.127 usec memoryview: 0.12 usec Windows: bytes: 0.11 usec bytearray: 0,184 usec memoryview: 0.139 usec On Linux, that's quite a nice spee

[issue10227] Improve performance of MemoryView slicing

Stefan Krah added the comment: I see. I thought this was mainly about memoryview performance, so I did not specifically look at bytearray. The poor performance seems to be Windows specific: C:\Users\stefan\hg\pep-3118\PCbuild>amd64\python.exe -m timeit -n 1000 -s "x = ((b'x'*1))" "x[:1

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: I'm afraid I had put this matter _far_ out of my head :) Seeing the amount of discussion on that other defect (stuff I had already come across and scrathced my head over) I think there is a lot of catching up that I'd need to do and I am unable to gi

[issue10227] Improve performance of MemoryView slicing

Changes by Stefan Krah : -- dependencies: +Problems with Py_buffer management in memoryobject.c (and elsewhere?) ___ Python tracker ___ _

[issue10227] Improve performance of MemoryView slicing

Stefan Krah added the comment: Kristján, could you check out the new implementation over at #10181? I have trouble reproducing a big speed difference between bytearray and memoryview (Linux, 64-bit). Here are the timings I get for the current and the new version: Slicing --- 1) ./python -

[issue10227] Improve performance of MemoryView slicing

Changes by Mark Dickinson : -- assignee: mark.dickinson -> ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: htt

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: Created follow-up issue 11107 for caching constant slice objects. -- ___ Python tracker ___ ___ Pyth

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: A quick test against the py3k stdlib: find -name "*.py" | while read file; do egrep '\[[-0-9]*:[-0-9]*\]' "$file"; done | wc -l This finds 2096 lines in 393 files. -- ___ Python tracker

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: > of course, this will not help for other common cases such as l[x:x+2] ... which is exactly what this slice caching patch is there for. ;-) -- ___ Python tracker

[issue10227] Improve performance of MemoryView slicing

Antoine Pitrou added the comment: > I find it interesting that the base line is way below the other > timings. That makes me think it's actually worth caching constant > slice instances, as CPython already does for tuples. Indeed. I have never touched it, but I suppose it needs an upgrade of th

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: Here's another base line test: slicing an empty list patched: $ ./python -m timeit -s 'l = []' 'l[:]' 1000 loops, best of 3: 0.0847 usec per loop original: $ ./python -m timeit -s 'l = []' 'l[:]' 1000 loops, best of 3: 0.0977 usec per loop That's abo

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: Here are some real micro benchmarks (note that the pybench benchmarks actually do lots of other stuff besides slicing): base line: $ ./python -m timeit -s 'l = list(range(100)); s=slice(None)' 'l[s]' 100 loops, best of 3: 0.464 usec per loop $ ./python -m

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: There's a "PyObject_Del(obj)" in all code paths. -- ___ Python tracker ___ ___ Python-bugs-list mail

[issue10227] Improve performance of MemoryView slicing

Antoine Pitrou added the comment: > I ran the list tests in pybench and got this: > > Test minimum run-timeaverage run-time > thisother diffthisother diff > > List

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: > Any benchmark numbers for the slice cache? I ran the list tests in pybench and got this: Test minimum run-timeaverage run-time thisother diffthisother diff

[issue10227] Improve performance of MemoryView slicing

Antoine Pitrou added the comment: Any benchmark numbers for the slice cache? Also, is the call to PyObject_INIT necessary? -- ___ Python tracker ___

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: I've extracted and fixed the part of this patch that implements the slice object cache. In particular, PySlice_Fini() was incorrectly implemented. This patch applies cleanly for me against the latest py3k branch. -- Added file: http://bugs.python.org/f

[issue10227] Improve performance of MemoryView slicing

Changes by Antoine Pitrou : -- assignee: -> mark.dickinson nosy: +mark.dickinson versions: +Python 3.3 -Python 3.2 ___ Python tracker ___ ___

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: In case I'm not clear enough: The patch is for 3.2, the benchmarks are 3.2, but it was created based on 2.7 results, which may not fully apply for 3.2 -- ___ Python tracker

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: The benchmarks are from 3.2 Also, I'll do a more relevant profiling session for 3.2. This patch is based on profiling results from 2.7 so there might be more relevant optimization cases in 3.2 -- ___ Pytho

[issue10227] Improve performance of MemoryView slicing

Stefan Behnel added the comment: I find it a lot easier to appreciate patches that implement a single change than those that mix different changes. There are three different things in your patch, which I would like to see in at least three different commits. I'd be happy if you could separate

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: Well then, its back to the profiler for 3.2. I did all of the profiling with 2.7 for practical reasons (it was the only version I had available at the time) and then ported the change to 3.2 today. But obviously there are different rules in 3.2 :)

[issue10227] Improve performance of MemoryView slicing

Antoine Pitrou added the comment: > I'd have to take another look with the profiler to figure out how > bytes slicing in 3.0 works, but I suspect that it is somehow > fasttracked passed the creation of slice objects, etc. I don't think it is fasttracked at all. Even plain indexing is not fastt

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: In 2.x, strings are sliced using PySequence_GetSlice(). ceval.c in 3.0 is different, there is no apply_slice there (despite comments to that effect). I'd have to take another look with the profiler to figure out how bytes slicing in 3.0 works, but I

[issue10227] Improve performance of MemoryView slicing

Antoine Pitrou added the comment: > As an additional point: the PyMemoryObject has a "base" member that I > think is redundant. the "view.obj" should be sufficient. Yes, that's what I think as well. -- ___ Python tracker

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: As an additional point: the PyMemoryObject has a "base" member that I think is redundant. the "view.obj" should be sufficient. -- ___ Python tracker ___

[issue10227] Improve performance of MemoryView slicing

Antoine Pitrou added the comment: The sequence protocol (if I'm not confused) only work with a PyObject ** array. -- ___ Python tracker ___ _

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: But then, perhaps implementing the sequence protocol for memoryviews might be more efficient still. -- ___ Python tracker ___

[issue10227] Improve performance of MemoryView slicing

Kristján Valur Jónsson added the comment: Oh dear. Here it is. -- Added file: http://bugs.python.org/file19410/memoryobj.patch ___ Python tracker ___ __

[issue10227] Improve performance of MemoryView slicing

Antoine Pitrou added the comment: You forgot to attach your patch. -- ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubs

[issue10227] Improve performance of MemoryView slicing

New submission from Kristján Valur Jónsson : In a recent email exchange on python-dev, Antoine Pitrou mentioned that slicing memoryview objects (lazy slices) wasn't necessarily very efficient when dealing with short slices. The data he posted was: $ ./python -m timeit -s "x = b'x'*1" "x[