Martin v. Löwis wrote: > Travis Oliphant wrote: > >> So, I now believe that his code (plus the array scalar extension >> type) was actually exposing a real bug in the memory manager itself. >> In theory, the Python memory manager should have been able to re-use >> the memory for the array-scalar instances because they are always the >> same size. In practice, the memory was apparently not being re-used >> but instead new blocks were being allocated to handle the load. > > > That is really very hard to believe. Most people on this list would > probably agree that obmalloc certain *will* reuse deallocated memory > if the next request is for the very same size (number of bytes) that > the previously-release object had.
Yes, I see that it does. This became more clear as all the simple tests I tried failed to reproduce the problem (and I spent some time looking at the code and reading its comments). I just can't figure out another explanation for why the problem went away when I went to using the system malloc other than some kind of corner-case in the Python memory allocator. > >> His code is quite complicated and it is difficult to replicate the >> problem. > > > That the code is complex would not so much be a problem: we often > analyse complex code here. It is a problem that the code is not > available, and it would be a problem if the problem was not > reproducable even if you had the code (i.e. if the problem would > sometimes occur, but not the next day when you ran it again). > The problem was definitely reproducible. On his machine, and on the two machines I tried to run it on. It without fail rapidly consumed all available memory. > So if you can, please post the code somewhere, and add a bugreport > on sf.net/projects/python. > I'll try to do this at some point. I'll have to get permission from him for the actual Python code. The extension modules he used are all publically available (PyMC). I changed the memory allocator in scipy --- which eliminated the problem --- so you'd have to check out an older version of the code from SVN to see the problem. Thanks for the tips. -Travis _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com