On Fri, Jun 6, 2014 at 11:33 PM, Sturla Molden <sturla.mol...@gmail.com> wrote: > Nathaniel Smith <n...@pobox.com> wrote: > >> The proposal in my initial email requires zero pthreads, and is >> substantially more effective. (Your proposal reduces only the alloc >> overhead for large arrays; mine reduces both alloc and memory access >> overhead for boyh large and small arrays.) > > My suggestion prevents the kernel from zeroing pages in the middle of a > computation, which is an important part. It would also be an optimiation > the Python interpreter could benefit from indepently of NumPy, by allowing > reuse of allocated memory pages within CPU bound portions of the Python > code. And no, the method I suggested does not only work for large arrays.
Small allocations are already recycled within process and don't touch the kernel, so your method doesn't affect them at all. My guess is that PyMalloc is unlikely to start spawning background threads any time soon, but if you'd like to propose it maybe you should start a new thread for that? > If we really want to take out the memory access overhead, we need to > consider lazy evaluation. E.g. a context manager that collects a symbolic > expression and triggers evaluation on exit: > > with numpy.accelerate: > x = <expression> > y = <expression> > z = <expression> > # evaluation of x,y,z happens here Using an alternative evaluation engine is indeed another way to optimize execution, which is why projects like numexpr, numba, theano, etc. exist. But this is basically switching to a different language in a different VM. I think people will keep running plain-old-CPython code for some time yet, and the point of this thread is that there's some low-hanging fruit for making all *that* code faster. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com