Thanks for your reply, you've given me plenty to think about On Sep 29, 11:51 pm, Antoine Pitrou <solip...@pitrou.net> wrote: > > > My original plan was to have the data processing and data acquisition > > functions running in separate processes, with a multiprocessing.Queue > > for passing the raw data packets. The raw data is read in as a char*, > > with a non constant length, hence I have allocated memory using > > PyMem_Malloc and I am returning from the acquisition function a > > PyCObject containing a pointer to this char* buffer, along with a > > destructor. > > That sounds overkill, and I also wonder how you plan to pass that > object in a multiprocessing Queue (which relies on objects being > pickleable). Why don't you simply create a PyString object instead?
Could you elaborate on why you feel this is overkill? Also, your right about passing the PyCObjects through a Queue, something which I hadn't really considered, so I've switched to using python strings as you suggested, an overhead I hoped to avoid but you can't win them all I suppose. > > So if I call these functions in a loop, e.g. The following will > > generate ~10GB of data > > > x = MyClass() > > for i in xrange(0, 10 * 2**20): > > c = x.malloc_buffer() > > x.retrieve_buffer(c) > > > All my memory disapears, until python crashes with a MemoryError. By > > placing a print in the destructor function I know it's being called, > > however it's not actually freeing the memory. So in short, what am I > > doing wrong? > > Python returns memory to the OS by calling free(). Not all OSes > actually relinquish memory when free() is called; some will simply set > it aside for the next allocation. > Another possible (and related) issue is memory fragmentation. Again, it > depends on the memory allocator. Yes, I know that's the case but the "freed" memory should be used for the next allocation, or atleast at some point before python runs out of memory. Anyway, this is besides the point as I've switched to using strings. Again thanks for taking the time to help me out, Tom Conneely -- http://mail.python.org/mailman/listinfo/python-list
