Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
http://blog.pavlov.net/2007/12/04/vlad-and-analysis-of-dtrace-was-used/ On Sat, Jan 23, 2010 at 5:56 PM, Erwin Coumans wrote: > Dough Lea's malloc was updated a few months ago. I think Unreal Engine > 3 is using it too. > > Do you have a URL of this allocator benchmark? > > Thanks, > Erwin > > Sent from my iPhone > > On Jan 23, 2010, at 15:16, joe wrote: > >> jcmalloc seems the best, at least from the profiling results the >> firefox people got with it. Doug lea's malloc is a bit dated, iirc of >> the three or four good ones, it was the worst? can't remember >> precisely, I could easily be wrong. >> >> Joe >> >> On Sat, Jan 23, 2010 at 1:06 PM, Erwin Coumans > > wrote: >>> Have you tried the very fast and popular Doug Lea Malloc, or >>> dlmalloc? >>> >>> ftp://g.oswego.edu/pub/misc/ >>> http://g.oswego.edu/dl/html/malloc.html >>> >>> Cheers, >>> Erwin >>> >>> >>> On 23 January 2010 10:49, Campbell Barton >>> wrote: did you try one of the malloc replacements? - should be able to do it without rebuilding blender even. I tried 3 popular malloc replacements (benchmarked with the game engine) jemalloc, nedmalloc and hoard IIRC None made much of a difference for me though perhaps the BGE isnt a good test case, was also trying on linux with an optimized build. On Sat, Jan 23, 2010 at 5:21 PM, joe wrote: > This is also something I've had to deal with in the bmesh branch, > and > the code I wrote there should never see the light of day in trunk > (which motivates me to tackle this now :) ). It's really quite > horrible what I wrote to deal with vgroups performance problems. > > Joe > > On Sat, Jan 23, 2010 at 8:20 AM, joe wrote: >> The purpose was simple experimentation, since we need to do >> *something* and at the time I didn't think people would go for >> using >> jcmalloc. Vgroups really are a bad source of performance loss, >> mostly >> in debug builds (which we need to be usable but aren't in some >> cases). >> I was hoping to elicit ideas from other people, and go from >> there. >> Anyway, it was silly of me to ignore the possibility of using >> jcmalloc, which we can probably drop in as a replacement for >> malloc >> within guardedalloc itself (and even have compile time options >> to have >> guardedalloc go straight through to jcmalloc). >> >> Joe >> >> On Sat, Jan 23, 2010 at 6:53 AM, Brecht Van Lommel > > wrote: >>> Hi Joe, >>> >>> Right, I replied to the wrong mail, I was talking about the >>> guardedalloc changes. I understand this is experimental, but I >>> don't >>> think some more experimentation will be prove this to be the >>> right >>> thing to do. It may well lead to a speedup in simple test >>> cases, but a >>> simple use of pooling can lead to much wasted memory and make >>> problems >>> worse when running Blender for a while. So it is not clear to >>> me what >>> the purpose is here, if this is the start of writing an advanced >>> memory allocator then I don't think we should try to do that >>> ourselves, and if not then I don't think this can be good >>> enough to >>> put in a release. >>> >>> Brecht. >>> >>> On Sat, Jan 23, 2010 at 3:19 PM, joe wrote: What are you talking about specifically? It helps with ghash, because each bucket node was being allocated individually, causing a significant speed problem. This particular solution was very appropriate; it's why we have the mempool library in the first place. Now, the experimental code I committed (#ifdef'd out) to guardedalloc is different (and was a different commit even). This particular commit has nothing to do with that. On that topic, OSX has (or had, anyway) a reputation for having a system allocator almost as slow as windows; linux is the only OS as far as I know (other then the BSDs I guess) that doesn't suffer from this. So it's hardly simply a windows issue. The overhead we get from guardedalloc isn't all that bad, really. I wasn't talking about that in the slightest. What I was talking about was the significant performance loss we get from overusing the system allocator, which has caused significant problems for me and others. I committed the code #ifdef'd out, so people who need it can play around with it but not cause problems for others. There's a reason it's *experimental*. Joe >>> ___ >>> Bf-committers mailing list >>> Bf-committers@blender.org >>> http://lists.blender
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
Dough Lea's malloc was updated a few months ago. I think Unreal Engine 3 is using it too. Do you have a URL of this allocator benchmark? Thanks, Erwin Sent from my iPhone On Jan 23, 2010, at 15:16, joe wrote: > jcmalloc seems the best, at least from the profiling results the > firefox people got with it. Doug lea's malloc is a bit dated, iirc of > the three or four good ones, it was the worst? can't remember > precisely, I could easily be wrong. > > Joe > > On Sat, Jan 23, 2010 at 1:06 PM, Erwin Coumans > wrote: >> Have you tried the very fast and popular Doug Lea Malloc, or >> dlmalloc? >> >> ftp://g.oswego.edu/pub/misc/ >> http://g.oswego.edu/dl/html/malloc.html >> >> Cheers, >> Erwin >> >> >> On 23 January 2010 10:49, Campbell Barton >> wrote: >>> did you try one of the malloc replacements? - should be able to do >>> it >>> without rebuilding blender even. >>> I tried 3 popular malloc replacements (benchmarked with the game >>> engine) >>> jemalloc, nedmalloc and hoard IIRC None made much of a difference >>> for >>> me though perhaps the BGE isnt a good test case, was also trying on >>> linux with an optimized build. >>> >>> >>> On Sat, Jan 23, 2010 at 5:21 PM, joe wrote: This is also something I've had to deal with in the bmesh branch, and the code I wrote there should never see the light of day in trunk (which motivates me to tackle this now :) ). It's really quite horrible what I wrote to deal with vgroups performance problems. Joe On Sat, Jan 23, 2010 at 8:20 AM, joe wrote: > The purpose was simple experimentation, since we need to do > *something* and at the time I didn't think people would go for > using > jcmalloc. Vgroups really are a bad source of performance loss, > mostly > in debug builds (which we need to be usable but aren't in some > cases). > I was hoping to elicit ideas from other people, and go from > there. > Anyway, it was silly of me to ignore the possibility of using > jcmalloc, which we can probably drop in as a replacement for > malloc > within guardedalloc itself (and even have compile time options > to have > guardedalloc go straight through to jcmalloc). > > Joe > > On Sat, Jan 23, 2010 at 6:53 AM, Brecht Van Lommel > wrote: >> Hi Joe, >> >> Right, I replied to the wrong mail, I was talking about the >> guardedalloc changes. I understand this is experimental, but I >> don't >> think some more experimentation will be prove this to be the >> right >> thing to do. It may well lead to a speedup in simple test >> cases, but a >> simple use of pooling can lead to much wasted memory and make >> problems >> worse when running Blender for a while. So it is not clear to >> me what >> the purpose is here, if this is the start of writing an advanced >> memory allocator then I don't think we should try to do that >> ourselves, and if not then I don't think this can be good >> enough to >> put in a release. >> >> Brecht. >> >> On Sat, Jan 23, 2010 at 3:19 PM, joe wrote: >>> What are you talking about specifically? It helps with ghash, >>> because >>> each bucket node was being allocated individually, causing a >>> significant speed problem. This particular solution was very >>> appropriate; it's why we have the mempool library in the first >>> place. >>> >>> Now, the experimental code I committed (#ifdef'd out) to >>> guardedalloc >>> is different (and was a >>> different commit even). This particular commit has nothing to >>> do with >>> that. On that topic, OSX has (or had, anyway) a reputation >>> for having >>> a system allocator almost as slow as windows; linux is the >>> only OS as >>> far as I know (other then the BSDs I guess) that doesn't >>> suffer from >>> this. So it's hardly simply a windows issue. >>> >>> The overhead we get from guardedalloc isn't all that bad, >>> really. I >>> wasn't talking about that in the slightest. What I was >>> talking about >>> was the significant performance loss we get from overusing the >>> system >>> allocator, which has caused significant problems for me and >>> others. I >>> committed the code #ifdef'd out, so people who need it can >>> play around >>> with it but not cause problems for others. There's a reason >>> it's >>> *experimental*. >>> >>> Joe >>> >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/list
[Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
>This isn't a problem any*one* has had time to tackle, and I need to for >bmesh. I'm certainly not going to add anything that's actually enabled >*until* after much testing and developer discussion, this was more of an >effort to get the dialoge started. I'm still unclear on why this issue is a b-mesh related one. Unless something big has changed, b-mesh performance with vertex groups should be pretty much the same as with edit-mesh (or even mesh/derived-mesh when dealing with modifiers). So again I have to ask where the identified bottlenecks are. Cheers, Geoff ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
jcmalloc seems the best, at least from the profiling results the firefox people got with it. Doug lea's malloc is a bit dated, iirc of the three or four good ones, it was the worst? can't remember precisely, I could easily be wrong. Joe On Sat, Jan 23, 2010 at 1:06 PM, Erwin Coumans wrote: > Have you tried the very fast and popular Doug Lea Malloc, or dlmalloc? > > ftp://g.oswego.edu/pub/misc/ > http://g.oswego.edu/dl/html/malloc.html > > Cheers, > Erwin > > > On 23 January 2010 10:49, Campbell Barton wrote: >> did you try one of the malloc replacements? - should be able to do it >> without rebuilding blender even. >> I tried 3 popular malloc replacements (benchmarked with the game engine) >> jemalloc, nedmalloc and hoard IIRC None made much of a difference for >> me though perhaps the BGE isnt a good test case, was also trying on >> linux with an optimized build. >> >> >> On Sat, Jan 23, 2010 at 5:21 PM, joe wrote: >>> This is also something I've had to deal with in the bmesh branch, and >>> the code I wrote there should never see the light of day in trunk >>> (which motivates me to tackle this now :) ). It's really quite >>> horrible what I wrote to deal with vgroups performance problems. >>> >>> Joe >>> >>> On Sat, Jan 23, 2010 at 8:20 AM, joe wrote: The purpose was simple experimentation, since we need to do *something* and at the time I didn't think people would go for using jcmalloc. Vgroups really are a bad source of performance loss, mostly in debug builds (which we need to be usable but aren't in some cases). I was hoping to elicit ideas from other people, and go from there. Anyway, it was silly of me to ignore the possibility of using jcmalloc, which we can probably drop in as a replacement for malloc within guardedalloc itself (and even have compile time options to have guardedalloc go straight through to jcmalloc). Joe On Sat, Jan 23, 2010 at 6:53 AM, Brecht Van Lommel wrote: > Hi Joe, > > Right, I replied to the wrong mail, I was talking about the > guardedalloc changes. I understand this is experimental, but I don't > think some more experimentation will be prove this to be the right > thing to do. It may well lead to a speedup in simple test cases, but a > simple use of pooling can lead to much wasted memory and make problems > worse when running Blender for a while. So it is not clear to me what > the purpose is here, if this is the start of writing an advanced > memory allocator then I don't think we should try to do that > ourselves, and if not then I don't think this can be good enough to > put in a release. > > Brecht. > > On Sat, Jan 23, 2010 at 3:19 PM, joe wrote: >> What are you talking about specifically? It helps with ghash, because >> each bucket node was being allocated individually, causing a >> significant speed problem. This particular solution was very >> appropriate; it's why we have the mempool library in the first place. >> >> Now, the experimental code I committed (#ifdef'd out) to guardedalloc >> is different (and was a >> different commit even). This particular commit has nothing to do with >> that. On that topic, OSX has (or had, anyway) a reputation for having >> a system allocator almost as slow as windows; linux is the only OS as >> far as I know (other then the BSDs I guess) that doesn't suffer from >> this. So it's hardly simply a windows issue. >> >> The overhead we get from guardedalloc isn't all that bad, really. I >> wasn't talking about that in the slightest. What I was talking about >> was the significant performance loss we get from overusing the system >> allocator, which has caused significant problems for me and others. I >> committed the code #ifdef'd out, so people who need it can play around >> with it but not cause problems for others. There's a reason it's >> *experimental*. >> >> Joe >> > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > >>> ___ >>> Bf-committers mailing list >>> Bf-committers@blender.org >>> http://lists.blender.org/mailman/listinfo/bf-committers >>> >> >> >> >> -- >> - Campbell >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
Somehow I proof read that without catching the grammar mistakes. To reiterate my position. I committed this on an experimental basis, and I originally planned to get some other people to run some tests and review the code accordingly. I never, ever, *ever* wanted this code *as it is now* in a release, or enabled by default. I'm certainly open to the idea of trashing my code, which again, was *experimental*, and playing with plugging in jcmalloc or whatever. This isn't a problem any*one* has had time to tackle, and I need to for bmesh. I'm certainly not going to add anything that's actually enabled *until* after much testing and developer discussion, this was more of an effort to get the dialoge started. Joe On Sat, Jan 23, 2010 at 3:13 PM, joe wrote: > To reiterate my position. I committed this on an experimental basis, > and I originally planned to get some other people to run some tests > and review the code accordingly. I never, ever, *ever* wanted this > code *as it is now* in a release, or enabled by default. I'm > certainly open to the idea of trashing my code, which again, was > *experimental*, and playing with plugging in jcmalloc or whatever. > > This isn't a problem anyway has had time to tackle, and I need to for > bmesh. I'm certainly not going add anything that's actually enabled > after much testing and developer discussion, this was more of an > effort to get the dialoge started. > > Joe > > On Sat, Jan 23, 2010 at 10:53 AM, Geoffrey Bantle wrote: >> Apologies for the lack of line breaks in the last email, >> I had forgotten that yahoo mail doesn't insert them >> automatically. Below is the original message formatted >> sanely (I hope). >> >> == >> >> I feel like I should comment on this >> >> I originally wrote the mem-pool joeedh is using because >> b-mesh performance on windows was *significantly* slowed >> down by the amount of small allocations it made. It's >> actually much worse than edit-mesh in those regards, and >> I wanted to do something a *little* bit more sophisticated >> than the editmesh fast calloc functions/callbacks. That >> being said, the mem-pool is not particularly clever or >> complicated code and it should be remembered that it >> was written to solve a very specific problem. >> >> Anyway, if I recall correct it was someone on IRC who >> suggested I put it in Blenlib in case someone else might >> find it useful. Despite this, I think it's only really >> appropriate for a narrow range of applications (edit-mesh >> and b-mesh being the obvious choices.) I question the >> wisdom of tying it to something like guarded alloc where >> it has the potential to have unintended and adverse effects >> on the rest of the system. >> >>>This is also something I've had to deal with in the bmesh branch, and >>>the code I wrote there should never see the light of day in trunk >>>(which motivates me to tackle this now :) ). It's really quite >>>horrible what I wrote to deal with vgroups performance problems. >>> >>>Joe >> >> When you mention 'horrible performance problems' associated >> with vertex groups I'm curious whether or not you measured >> this. Did you do profiling? What did you pinpoint as the >> biggest bottlenecks? >> >> Cheers, >> Geoff >> >> >> >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
To reiterate my position. I committed this on an experimental basis, and I originally planned to get some other people to run some tests and review the code accordingly. I never, ever, *ever* wanted this code *as it is now* in a release, or enabled by default. I'm certainly open to the idea of trashing my code, which again, was *experimental*, and playing with plugging in jcmalloc or whatever. This isn't a problem anyway has had time to tackle, and I need to for bmesh. I'm certainly not going add anything that's actually enabled after much testing and developer discussion, this was more of an effort to get the dialoge started. Joe On Sat, Jan 23, 2010 at 10:53 AM, Geoffrey Bantle wrote: > Apologies for the lack of line breaks in the last email, > I had forgotten that yahoo mail doesn't insert them > automatically. Below is the original message formatted > sanely (I hope). > > == > > I feel like I should comment on this > > I originally wrote the mem-pool joeedh is using because > b-mesh performance on windows was *significantly* slowed > down by the amount of small allocations it made. It's > actually much worse than edit-mesh in those regards, and > I wanted to do something a *little* bit more sophisticated > than the editmesh fast calloc functions/callbacks. That > being said, the mem-pool is not particularly clever or > complicated code and it should be remembered that it > was written to solve a very specific problem. > > Anyway, if I recall correct it was someone on IRC who > suggested I put it in Blenlib in case someone else might > find it useful. Despite this, I think it's only really > appropriate for a narrow range of applications (edit-mesh > and b-mesh being the obvious choices.) I question the > wisdom of tying it to something like guarded alloc where > it has the potential to have unintended and adverse effects > on the rest of the system. > >>This is also something I've had to deal with in the bmesh branch, and >>the code I wrote there should never see the light of day in trunk >>(which motivates me to tackle this now :) ). It's really quite >>horrible what I wrote to deal with vgroups performance problems. >> >>Joe > > When you mention 'horrible performance problems' associated > with vertex groups I'm curious whether or not you measured > this. Did you do profiling? What did you pinpoint as the > biggest bottlenecks? > > Cheers, > Geoff > > > > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
Have you tried the very fast and popular Doug Lea Malloc, or dlmalloc? ftp://g.oswego.edu/pub/misc/ http://g.oswego.edu/dl/html/malloc.html Cheers, Erwin On 23 January 2010 10:49, Campbell Barton wrote: > did you try one of the malloc replacements? - should be able to do it > without rebuilding blender even. > I tried 3 popular malloc replacements (benchmarked with the game engine) > jemalloc, nedmalloc and hoard IIRC None made much of a difference for > me though perhaps the BGE isnt a good test case, was also trying on > linux with an optimized build. > > > On Sat, Jan 23, 2010 at 5:21 PM, joe wrote: >> This is also something I've had to deal with in the bmesh branch, and >> the code I wrote there should never see the light of day in trunk >> (which motivates me to tackle this now :) ). It's really quite >> horrible what I wrote to deal with vgroups performance problems. >> >> Joe >> >> On Sat, Jan 23, 2010 at 8:20 AM, joe wrote: >>> The purpose was simple experimentation, since we need to do >>> *something* and at the time I didn't think people would go for using >>> jcmalloc. Vgroups really are a bad source of performance loss, mostly >>> in debug builds (which we need to be usable but aren't in some cases). >>> I was hoping to elicit ideas from other people, and go from there. >>> Anyway, it was silly of me to ignore the possibility of using >>> jcmalloc, which we can probably drop in as a replacement for malloc >>> within guardedalloc itself (and even have compile time options to have >>> guardedalloc go straight through to jcmalloc). >>> >>> Joe >>> >>> On Sat, Jan 23, 2010 at 6:53 AM, Brecht Van Lommel >>> wrote: Hi Joe, Right, I replied to the wrong mail, I was talking about the guardedalloc changes. I understand this is experimental, but I don't think some more experimentation will be prove this to be the right thing to do. It may well lead to a speedup in simple test cases, but a simple use of pooling can lead to much wasted memory and make problems worse when running Blender for a while. So it is not clear to me what the purpose is here, if this is the start of writing an advanced memory allocator then I don't think we should try to do that ourselves, and if not then I don't think this can be good enough to put in a release. Brecht. On Sat, Jan 23, 2010 at 3:19 PM, joe wrote: > What are you talking about specifically? It helps with ghash, because > each bucket node was being allocated individually, causing a > significant speed problem. This particular solution was very > appropriate; it's why we have the mempool library in the first place. > > Now, the experimental code I committed (#ifdef'd out) to guardedalloc > is different (and was a > different commit even). This particular commit has nothing to do with > that. On that topic, OSX has (or had, anyway) a reputation for having > a system allocator almost as slow as windows; linux is the only OS as > far as I know (other then the BSDs I guess) that doesn't suffer from > this. So it's hardly simply a windows issue. > > The overhead we get from guardedalloc isn't all that bad, really. I > wasn't talking about that in the slightest. What I was talking about > was the significant performance loss we get from overusing the system > allocator, which has caused significant problems for me and others. I > committed the code #ifdef'd out, so people who need it can play around > with it but not cause problems for others. There's a reason it's > *experimental*. > > Joe > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers >>> >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > > > > -- > - Campbell > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
[Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
Apologies for the lack of line breaks in the last email, I had forgotten that yahoo mail doesn't insert them automatically. Below is the original message formatted sanely (I hope). == I feel like I should comment on this I originally wrote the mem-pool joeedh is using because b-mesh performance on windows was *significantly* slowed down by the amount of small allocations it made. It's actually much worse than edit-mesh in those regards, and I wanted to do something a *little* bit more sophisticated than the editmesh fast calloc functions/callbacks. That being said, the mem-pool is not particularly clever or complicated code and it should be remembered that it was written to solve a very specific problem. Anyway, if I recall correct it was someone on IRC who suggested I put it in Blenlib in case someone else might find it useful. Despite this, I think it's only really appropriate for a narrow range of applications (edit-mesh and b-mesh being the obvious choices.) I question the wisdom of tying it to something like guarded alloc where it has the potential to have unintended and adverse effects on the rest of the system. >This is also something I've had to deal with in the bmesh branch, and >the code I wrote there should never see the light of day in trunk >(which motivates me to tackle this now :) ). It's really quite >horrible what I wrote to deal with vgroups performance problems. > >Joe When you mention 'horrible performance problems' associated with vertex groups I'm curious whether or not you measured this. Did you do profiling? What did you pinpoint as the biggest bottlenecks? Cheers, Geoff ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
did you try one of the malloc replacements? - should be able to do it without rebuilding blender even. I tried 3 popular malloc replacements (benchmarked with the game engine) jemalloc, nedmalloc and hoard IIRC None made much of a difference for me though perhaps the BGE isnt a good test case, was also trying on linux with an optimized build. On Sat, Jan 23, 2010 at 5:21 PM, joe wrote: > This is also something I've had to deal with in the bmesh branch, and > the code I wrote there should never see the light of day in trunk > (which motivates me to tackle this now :) ). It's really quite > horrible what I wrote to deal with vgroups performance problems. > > Joe > > On Sat, Jan 23, 2010 at 8:20 AM, joe wrote: >> The purpose was simple experimentation, since we need to do >> *something* and at the time I didn't think people would go for using >> jcmalloc. Vgroups really are a bad source of performance loss, mostly >> in debug builds (which we need to be usable but aren't in some cases). >> I was hoping to elicit ideas from other people, and go from there. >> Anyway, it was silly of me to ignore the possibility of using >> jcmalloc, which we can probably drop in as a replacement for malloc >> within guardedalloc itself (and even have compile time options to have >> guardedalloc go straight through to jcmalloc). >> >> Joe >> >> On Sat, Jan 23, 2010 at 6:53 AM, Brecht Van Lommel >> wrote: >>> Hi Joe, >>> >>> Right, I replied to the wrong mail, I was talking about the >>> guardedalloc changes. I understand this is experimental, but I don't >>> think some more experimentation will be prove this to be the right >>> thing to do. It may well lead to a speedup in simple test cases, but a >>> simple use of pooling can lead to much wasted memory and make problems >>> worse when running Blender for a while. So it is not clear to me what >>> the purpose is here, if this is the start of writing an advanced >>> memory allocator then I don't think we should try to do that >>> ourselves, and if not then I don't think this can be good enough to >>> put in a release. >>> >>> Brecht. >>> >>> On Sat, Jan 23, 2010 at 3:19 PM, joe wrote: What are you talking about specifically? It helps with ghash, because each bucket node was being allocated individually, causing a significant speed problem. This particular solution was very appropriate; it's why we have the mempool library in the first place. Now, the experimental code I committed (#ifdef'd out) to guardedalloc is different (and was a different commit even). This particular commit has nothing to do with that. On that topic, OSX has (or had, anyway) a reputation for having a system allocator almost as slow as windows; linux is the only OS as far as I know (other then the BSDs I guess) that doesn't suffer from this. So it's hardly simply a windows issue. The overhead we get from guardedalloc isn't all that bad, really. I wasn't talking about that in the slightest. What I was talking about was the significant performance loss we get from overusing the system allocator, which has caused significant problems for me and others. I committed the code #ifdef'd out, so people who need it can play around with it but not cause problems for others. There's a reason it's *experimental*. Joe >>> ___ >>> Bf-committers mailing list >>> Bf-committers@blender.org >>> http://lists.blender.org/mailman/listinfo/bf-committers >>> >> > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > -- - Campbell ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
[Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
I feel like I should comment on this I originally wrote the mem-pool joeedh is using because b-mesh performance on windows was *significantly* slowed down by the amount of small allocations it made. It's actually much worse than edit-mesh in those regards, and I wanted to do something a *little* bit more sophisticated than the editmesh fast calloc functions/callbacks. That being said, the mem-pool is not particularly clever or complicated code and it should be remembered that it was written to solve a very specific problem. Anyway, if I recall correct it was someone on IRC who suggested I put it in Blenlib in case someone else might find it useful. Despite this, I think it's only really appropriate for a narrow range of applications (edit-mesh and b-mesh being the obvious choices.) I question the wisdom of tying it to something like guarded alloc where it has the potential to have unintended and adverse effects on the rest of the system. >This is also something I've had to deal with in the bmesh branch, and >the code I wrote there should never see the light of day in trunk >(which motivates me to tackle this now :) ). It's really quite >horrible what I wrote to deal with vgroups performance problems. > >Joe When you mention 'horrible performance problems' associated with vertex groups I'm curious whether or not you measured this. Did you do profiling? What did you pinpoint as the biggest bottlenecks? Cheers, Geoff ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
This is also something I've had to deal with in the bmesh branch, and the code I wrote there should never see the light of day in trunk (which motivates me to tackle this now :) ). It's really quite horrible what I wrote to deal with vgroups performance problems. Joe On Sat, Jan 23, 2010 at 8:20 AM, joe wrote: > The purpose was simple experimentation, since we need to do > *something* and at the time I didn't think people would go for using > jcmalloc. Vgroups really are a bad source of performance loss, mostly > in debug builds (which we need to be usable but aren't in some cases). > I was hoping to elicit ideas from other people, and go from there. > Anyway, it was silly of me to ignore the possibility of using > jcmalloc, which we can probably drop in as a replacement for malloc > within guardedalloc itself (and even have compile time options to have > guardedalloc go straight through to jcmalloc). > > Joe > > On Sat, Jan 23, 2010 at 6:53 AM, Brecht Van Lommel wrote: >> Hi Joe, >> >> Right, I replied to the wrong mail, I was talking about the >> guardedalloc changes. I understand this is experimental, but I don't >> think some more experimentation will be prove this to be the right >> thing to do. It may well lead to a speedup in simple test cases, but a >> simple use of pooling can lead to much wasted memory and make problems >> worse when running Blender for a while. So it is not clear to me what >> the purpose is here, if this is the start of writing an advanced >> memory allocator then I don't think we should try to do that >> ourselves, and if not then I don't think this can be good enough to >> put in a release. >> >> Brecht. >> >> On Sat, Jan 23, 2010 at 3:19 PM, joe wrote: >>> What are you talking about specifically? It helps with ghash, because >>> each bucket node was being allocated individually, causing a >>> significant speed problem. This particular solution was very >>> appropriate; it's why we have the mempool library in the first place. >>> >>> Now, the experimental code I committed (#ifdef'd out) to guardedalloc >>> is different (and was a >>> different commit even). This particular commit has nothing to do with >>> that. On that topic, OSX has (or had, anyway) a reputation for having >>> a system allocator almost as slow as windows; linux is the only OS as >>> far as I know (other then the BSDs I guess) that doesn't suffer from >>> this. So it's hardly simply a windows issue. >>> >>> The overhead we get from guardedalloc isn't all that bad, really. I >>> wasn't talking about that in the slightest. What I was talking about >>> was the significant performance loss we get from overusing the system >>> allocator, which has caused significant problems for me and others. I >>> committed the code #ifdef'd out, so people who need it can play around >>> with it but not cause problems for others. There's a reason it's >>> *experimental*. >>> >>> Joe >>> >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
The purpose was simple experimentation, since we need to do *something* and at the time I didn't think people would go for using jcmalloc. Vgroups really are a bad source of performance loss, mostly in debug builds (which we need to be usable but aren't in some cases). I was hoping to elicit ideas from other people, and go from there. Anyway, it was silly of me to ignore the possibility of using jcmalloc, which we can probably drop in as a replacement for malloc within guardedalloc itself (and even have compile time options to have guardedalloc go straight through to jcmalloc). Joe On Sat, Jan 23, 2010 at 6:53 AM, Brecht Van Lommel wrote: > Hi Joe, > > Right, I replied to the wrong mail, I was talking about the > guardedalloc changes. I understand this is experimental, but I don't > think some more experimentation will be prove this to be the right > thing to do. It may well lead to a speedup in simple test cases, but a > simple use of pooling can lead to much wasted memory and make problems > worse when running Blender for a while. So it is not clear to me what > the purpose is here, if this is the start of writing an advanced > memory allocator then I don't think we should try to do that > ourselves, and if not then I don't think this can be good enough to > put in a release. > > Brecht. > > On Sat, Jan 23, 2010 at 3:19 PM, joe wrote: >> What are you talking about specifically? It helps with ghash, because >> each bucket node was being allocated individually, causing a >> significant speed problem. This particular solution was very >> appropriate; it's why we have the mempool library in the first place. >> >> Now, the experimental code I committed (#ifdef'd out) to guardedalloc >> is different (and was a >> different commit even). This particular commit has nothing to do with >> that. On that topic, OSX has (or had, anyway) a reputation for having >> a system allocator almost as slow as windows; linux is the only OS as >> far as I know (other then the BSDs I guess) that doesn't suffer from >> this. So it's hardly simply a windows issue. >> >> The overhead we get from guardedalloc isn't all that bad, really. I >> wasn't talking about that in the slightest. What I was talking about >> was the significant performance loss we get from overusing the system >> allocator, which has caused significant problems for me and others. I >> committed the code #ifdef'd out, so people who need it can play around >> with it but not cause problems for others. There's a reason it's >> *experimental*. >> >> Joe >> > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
Hi Joe, Right, I replied to the wrong mail, I was talking about the guardedalloc changes. I understand this is experimental, but I don't think some more experimentation will be prove this to be the right thing to do. It may well lead to a speedup in simple test cases, but a simple use of pooling can lead to much wasted memory and make problems worse when running Blender for a while. So it is not clear to me what the purpose is here, if this is the start of writing an advanced memory allocator then I don't think we should try to do that ourselves, and if not then I don't think this can be good enough to put in a release. Brecht. On Sat, Jan 23, 2010 at 3:19 PM, joe wrote: > What are you talking about specifically? It helps with ghash, because > each bucket node was being allocated individually, causing a > significant speed problem. This particular solution was very > appropriate; it's why we have the mempool library in the first place. > > Now, the experimental code I committed (#ifdef'd out) to guardedalloc > is different (and was a > different commit even). This particular commit has nothing to do with > that. On that topic, OSX has (or had, anyway) a reputation for having > a system allocator almost as slow as windows; linux is the only OS as > far as I know (other then the BSDs I guess) that doesn't suffer from > this. So it's hardly simply a windows issue. > > The overhead we get from guardedalloc isn't all that bad, really. I > wasn't talking about that in the slightest. What I was talking about > was the significant performance loss we get from overusing the system > allocator, which has caused significant problems for me and others. I > committed the code #ifdef'd out, so people who need it can play around > with it but not cause problems for others. There's a reason it's > *experimental*. > > Joe > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
What are you talking about specifically? It helps with ghash, because each bucket node was being allocated individually, causing a significant speed problem. This particular solution was very appropriate; it's why we have the mempool library in the first place. Now, the experimental code I committed (#ifdef'd out) to guardedalloc is different (and was a different commit even). This particular commit has nothing to do with that. On that topic, OSX has (or had, anyway) a reputation for having a system allocator almost as slow as windows; linux is the only OS as far as I know (other then the BSDs I guess) that doesn't suffer from this. So it's hardly simply a windows issue. The overhead we get from guardedalloc isn't all that bad, really. I wasn't talking about that in the slightest. What I was talking about was the significant performance loss we get from overusing the system allocator, which has caused significant problems for me and others. I committed the code #ifdef'd out, so people who need it can play around with it but not cause problems for others. There's a reason it's *experimental*. Joe On Sat, Jan 23, 2010 at 5:46 AM, Brecht Van Lommel wrote: > Hi Joe, > > I'm trying to understand when using a memory pool for small > allocations would help here. I doesn't avoid any of the overhead of > our own guardedalloc code, it just replaces malloc. So it's basically > expected to be a better malloc, in case the one provided by the system > doesn't work well? To my knowledge, this is only applicable to > Windows. Perhaps a better solution would be to link in a library like > tcmalloc or jemalloc on Windows (like e.g. Google Chrome does), rather > than trying to do this ourselves. It seems to me this is quite a > complicated thing to do right, and it's already been done better. > > Brecht. > > On Sat, Jan 23, 2010 at 1:16 PM, joe wrote: >> I'll look into it. >> >> On Sat, Jan 23, 2010 at 4:02 AM, Thomas Dinges wrote: >>> Hi, >>> this breaks compiling again. >>> scons/msvc >>> >>> http://www.pasteall.org/10530 >>> >>> Some unresolved stuff in the linking process. >>> >>> >>> >>> >>> Am 23.01.2010 12:25, schrieb Joseph Eagar: Revision: 26206 http://projects.blender.org/plugins/scmsvn/viewcvs.php?view=rev&root=bf-blender&revision=26206 Author: joeedh Date: 2010-01-23 12:25:20 +0100 (Sat, 23 Jan 2010) Log Message: --- Initial results of my profiling of the animation system. Basically two simple changes, changes, I pulled in the faster ghash in bmesh (which uses mempools for allocation, providing a substanstial speedup in some cases, and also I inlined some of the functions), and I changed __inline to __forceinline for inlining of math functions. I also removed the timer in the view3d zoom op (ctrl-middlemouse) that was making it nonfunctional. Why was that there? >>> ___ >>> Bf-committers mailing list >>> Bf-committers@blender.org >>> http://lists.blender.org/mailman/listinfo/bf-committers >>> >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> >> > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
Hi Joe, I'm trying to understand when using a memory pool for small allocations would help here. I doesn't avoid any of the overhead of our own guardedalloc code, it just replaces malloc. So it's basically expected to be a better malloc, in case the one provided by the system doesn't work well? To my knowledge, this is only applicable to Windows. Perhaps a better solution would be to link in a library like tcmalloc or jemalloc on Windows (like e.g. Google Chrome does), rather than trying to do this ourselves. It seems to me this is quite a complicated thing to do right, and it's already been done better. Brecht. On Sat, Jan 23, 2010 at 1:16 PM, joe wrote: > I'll look into it. > > On Sat, Jan 23, 2010 at 4:02 AM, Thomas Dinges wrote: >> Hi, >> this breaks compiling again. >> scons/msvc >> >> http://www.pasteall.org/10530 >> >> Some unresolved stuff in the linking process. >> >> >> >> >> Am 23.01.2010 12:25, schrieb Joseph Eagar: >>> Revision: 26206 >>> >>> http://projects.blender.org/plugins/scmsvn/viewcvs.php?view=rev&root=bf-blender&revision=26206 >>> Author: joeedh >>> Date: 2010-01-23 12:25:20 +0100 (Sat, 23 Jan 2010) >>> >>> Log Message: >>> --- >>> Initial results of my profiling of the animation system. >>> Basically two simple changes, changes, I pulled in the faster >>> ghash in bmesh (which uses mempools for allocation, providing >>> a substanstial speedup in some cases, and also I inlined some >>> of the functions), and I changed __inline to __forceinline for inlining >>> of math functions. >>> >>> I also removed the timer in the view3d zoom op (ctrl-middlemouse) >>> that was making it nonfunctional. Why was that there? >>> >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
I'll look into it. On Sat, Jan 23, 2010 at 4:02 AM, Thomas Dinges wrote: > Hi, > this breaks compiling again. > scons/msvc > > http://www.pasteall.org/10530 > > Some unresolved stuff in the linking process. > > > > > Am 23.01.2010 12:25, schrieb Joseph Eagar: >> Revision: 26206 >> >> http://projects.blender.org/plugins/scmsvn/viewcvs.php?view=rev&root=bf-blender&revision=26206 >> Author: joeedh >> Date: 2010-01-23 12:25:20 +0100 (Sat, 23 Jan 2010) >> >> Log Message: >> --- >> Initial results of my profiling of the animation system. >> Basically two simple changes, changes, I pulled in the faster >> ghash in bmesh (which uses mempools for allocation, providing >> a substanstial speedup in some cases, and also I inlined some >> of the functions), and I changed __inline to __forceinline for inlining >> of math functions. >> >> I also removed the timer in the view3d zoom op (ctrl-middlemouse) >> that was making it nonfunctional. Why was that there? >> > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] [Bf-blender-cvs] [26206] Compiling issue
Hi, this breaks compiling again. scons/msvc http://www.pasteall.org/10530 Some unresolved stuff in the linking process. Am 23.01.2010 12:25, schrieb Joseph Eagar: > Revision: 26206 > > http://projects.blender.org/plugins/scmsvn/viewcvs.php?view=rev&root=bf-blender&revision=26206 > Author: joeedh > Date: 2010-01-23 12:25:20 +0100 (Sat, 23 Jan 2010) > > Log Message: > --- > Initial results of my profiling of the animation system. > Basically two simple changes, changes, I pulled in the faster > ghash in bmesh (which uses mempools for allocation, providing > a substanstial speedup in some cases, and also I inlined some > of the functions), and I changed __inline to __forceinline for inlining > of math functions. > > I also removed the timer in the view3d zoom op (ctrl-middlemouse) > that was making it nonfunctional. Why was that there? > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
