Dag Sverre Seljebotn wrote: > Stefan Behnel wrote: >>> There are three levels: >>> 1) Memory-holding object. When Cython allocates, we need a new type >>> (probably inheriting directly from object), which allocates memory and >>> stores shape/stride information, and returns the right information in >>> tp_getbuffer. Would be a 20-liner in Cython unless we want to go with >>> PyVarObject for allocation in which case I suppose it is a 200-liner in C. >> I'm all for keeping such an implementation in Cython, at least for now. We >> can always optimise/reimplement later. > > Sounds right. Would motive doing something about Cython support for > PyVarObject too (perhaps a magic type you'd inherit from, which has a > constructor with size and a void* field pointing to the allocated > buffer? So #152 would get us 90% of the way.) > >>> 2) The acquired Py_buffer on that object. >> Erm, buffers are not memory managed, right? They are just the plain structs >> that I think of? > > Yes, they need to be wrapped in an object of some kind. > >>> That would happen the same way >>> as with every array; preferably by using memoryview, or if not, another >>> custom Python type (need to backport memoryview anyway) or if we can't >>> seem to avoid it a custom refcounted struct. >> If we implement the object ourselves, what do we need memoryview for? Would >> it not just implement the buffer protocol? Without any ref-counting? And >> when coercing to an object, why not just return the object itself? It could >> just have a C-level interface and a Python interface. > > I don't follow you quite. But some points I think are relevant: > > a) The memory will often live in objects from 3rd party libraries > accessed through PEP 3118. It seems cleanest to have Cython-allocated > memory just be another such object, treated in the same fashion. > > b) If you load 10 GB of data into memory in one array (yes some numerics > users do that), copying is usually not an option; copying must always be > easily predictable and user-managed. > > c) When taking slices, you get a new view to the same data, however the > original Py_buffer, likely stored in a memoryview object, cannot be > modified by protocol. I.e. the new slice needs to be "stored" elsewhere. > And the original object can't be told to reallocate/reslice.
I have a respectful request to make here: Can we just let whoever will implement this (i.e. me and to some degree Kurt) do whatever we feel is right regarding implementation, and keep the discussion to syntax impact and project impact of the CEPs? For instance, I already had a long thread in here with Kurt working through a lot of the finer details here already (some of that is likely outdated, due in fact to something Stefan wrote, but still...). While I realize that you and Robert has helped me avoid some poor design decisions in the past, something I'm really grateful for, it's not a scalable solution in the long run. Sometimes having to redo stuff later is easier than discussing everything on the mailing list. Language semantics is something else, that needs to be discussed -- and I'm really glad the point about slice-as-a-view came up. -- Dag Sverre _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
