[Bf-committers] need help on vgroup/mdisp allocation
A while back, testers of bmesh were complaining of performance issues related to vgroup and mdisp allocation. I wrote a temporary fix (BLI_cellalloc.h/c) but it's too evil for trunk. The basic problem is how vgroups/mdisps overuse the system allocator. The optimal solution would be to compile with jemalloc on all platforms, but I dunno how hard that would be (I think Firefox does it). Anyway, help on other solutions would be great. MDisps should be easy; I can add a mempool to CustomDataLayer, and use it for allocation (though it would touch a lot of code). Vgroups have a variable allocation size, however, and I'm unsure how to deal with them (they really can cause significant performance problems). Using jemalloc on all platforms might be the best approach, plugged into guardedalloc. Integrating something like mempool or memarena in CustomData is possible, but not ideal (given how much code it would affect, and how close we are to trunk integration). Joe ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
[Bf-committers] need help on vgroup/mdisp allocation
Hi jemalloc looks like a good solution, still, to compare - few other links which deal with memory allocators with variable size allocations. http://wkaras.webs.com/heapmm/heapmm.html http://g.oswego.edu/dl/html/malloc.html http://shm-arena.sourceforge.net/ http://www.sqlite.org/malloc.html http://www.boost.org/doc/libs/1_46_1/doc/html/interprocess/memory_algorithms.html#interprocess.memory_algorithms.rbtree_best_fit Regards Sergey ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] need help on vgroup/mdisp allocation
Here are some useful jemalloc links http://www.canonware.com/jemalloc/ http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/ LetterRip On Wed, May 11, 2011 at 12:32 PM, joe wrote: > Using jemalloc on all platforms might be the best approach, plugged > into guardedalloc. Integrating something like mempool or memarena in > CustomData is possible, but not ideal (given how much code it would > affect, and how close we are to trunk integration). > > Joe > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] need help on vgroup/mdisp allocation
Of additional interest is that since firefox facebook has made a number of improvements http://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919 LetterRip On Wed, May 11, 2011 at 12:41 PM, Tom M wrote: > Here are some useful jemalloc links > > http://www.canonware.com/jemalloc/ > http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/ > > LetterRip > > On Wed, May 11, 2011 at 12:32 PM, joe wrote: > >> Using jemalloc on all platforms might be the best approach, plugged >> into guardedalloc. Integrating something like mempool or memarena in >> CustomData is possible, but not ideal (given how much code it would >> affect, and how close we are to trunk integration). >> >> Joe >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] need help on vgroup/mdisp allocation
+1 for using jemalloc, for durian we used it for all systems + the render farm & it saved a lot of ram. see: http://www.sintel.org/development/memory-jemalloc/ for *nix we can LD_PRELOAD jemalloc, so no special requirements other then starting blender from a shell script, windows needs some investigation though. One of the main things to decide is weather jemalloc replaces all alloc's (as with LD_PRELOAD), or only MEM_* functions since this changes how its linked/bundled. How about make this a build option and give some time for platform maintainers to support, then default to ON when its working well? On Wed, May 11, 2011 at 9:47 PM, Tom M wrote: > Of additional interest is that since firefox facebook has made a > number of improvements > > http://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919 > > LetterRip > > > On Wed, May 11, 2011 at 12:41 PM, Tom M wrote: >> Here are some useful jemalloc links >> >> http://www.canonware.com/jemalloc/ >> http://blog.pavlov.net/2008/03/11/firefox-3-memory-usage/ >> >> LetterRip >> >> On Wed, May 11, 2011 at 12:32 PM, joe wrote: >> >>> Using jemalloc on all platforms might be the best approach, plugged >>> into guardedalloc. Integrating something like mempool or memarena in >>> CustomData is possible, but not ideal (given how much code it would >>> affect, and how close we are to trunk integration). >>> >>> Joe >>> ___ >>> Bf-committers mailing list >>> Bf-committers@blender.org >>> http://lists.blender.org/mailman/listinfo/bf-committers >>> >> > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > -- - Campbell ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
