2011/10/28 Gilles Louppe <[email protected]>: >> Still, almost 4 minutes just to extend the python heap and reallocate >> a bunch of already allocated objects seems unlikely. Also I don't >> understand why the Python interpreter would need to "move" allocated >> object: it can just grow the heap, reallocate a larger buffer list (if >> needed, with just 20 items this is not likely), and add a reference to >> the new item in the list buffer. > > Because the heap probably get fragmented (forests are ensembles of > trees, which are ensembles of nodes, which means many many many objects > stored in non-contiguous parts of the heap). Hence, to reduce > fragmentation, objects need to be moved. Again, this is just an > hypothesis, but I wouldn't be so surprised if the memory manager > actually worked in a similar way (this is a classic algorithm for a > memory manager).
Indeed... It might be interesting to invest more time in memory contiguous data-structures based on numpy arrays as recommended initially by Gael after all. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ------------------------------------------------------------------------------ The demand for IT networking professionals continues to grow, and the demand for specialized networking skills is growing even more rapidly. Take a complimentary Learning@Cisco Self-Assessment and learn about Cisco certifications, training, and career opportunities. http://p.sf.net/sfu/cisco-dev2dev _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
