Re: [Scons-dev] SCons speedup and profiling results...
On Sun, Sep 29, 2013 at 9:06 PM, Dirk Bächle tshor...@gmx.de wrote: On 26.09.2013 02:08, Gary Oberbrunner wrote: [...] I think this is excellent work! Solid analysis. I know there's been some thought given to caching subst() before; it's trickier than one might think but in many cases it should work, and it definitely speeds things up. I'm also impressed by a 30% memory reduction -- interested to hear how that comes out. I continued my work on reducing the overall memory consumption in SCons. By combining my old branch (where I switched the core to using slots) with some additional patches, I am now able to save up to 50% memory...depending on the project. Please find some results attached (a comparison between the current default tip and my experimental branch), the code can be cloned from: hg clone http://bitbucket.org/**dirkbaechle/scons_experimentalhttp://bitbucket.org/dirkbaechle/scons_experimental-r reduced_memory_updated I also achieved up to 20% speed improvements on updates, by a first version of a caching for the env.subst() method. Best regards, Dirk Hi! I tried your branch reduced_memory_updated and noticed that when I alternate between using that branch and the default branch from the main SCons repository, some files in my project are rebuilt. Has the signature calculation in your branch been changed so I should expect this to happen? I use: - f4f09fbddf29 from default branch in main SCons repo - 3b330d4bc8dc from reduced_memory_updated in your scons_experimental repo Apart from this I see the memory consumption decrease from 720MB -- 450MB, and the up-to-date check time drop from 2m20s -- 1m57s. /Johan Holmberg ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons speedup and profiling results...
On 26.09.2013 02:08, Gary Oberbrunner wrote: [...] I think this is excellent work! Solid analysis. I know there's been some thought given to caching subst() before; it's trickier than one might think but in many cases it should work, and it definitely speeds things up. I'm also impressed by a 30% memory reduction -- interested to hear how that comes out. I continued my work on reducing the overall memory consumption in SCons. By combining my old branch (where I switched the core to using slots) with some additional patches, I am now able to save up to 50% memory...depending on the project. Please find some results attached (a comparison between the current default tip and my experimental branch), the code can be cloned from: hg clone http://bitbucket.org/dirkbaechle/scons_experimental -r reduced_memory_updated I also achieved up to 20% speed improvements on updates, by a first version of a caching for the env.subst() method. Best regards, Dirk Title: Comparing default to lowmem wonderbuild Times Runrun [s]update [s]update_implicit [s] Previous1172.325.619.4 Current1131.622.015.5 Factor0.970.860.80 Memory Runrun [MByte]update [MByte] Previous451.4424.2 Current251.5197.8 Factor0.560.47 sconsbld Times Runrun [s]update [s]update_implicit [s] Previous440.335.026.1 Current343.828.719.8 Factor0.780.820.76 Memory Runrun [MByte]update [MByte] Previous538.6554.4 Current231.6238.7 Factor0.430.43 questfperf Times Runrun [s]update [s]update_implicit [s] Previous1022.424.216.9 Current984.820.412.9 Factor0.960.840.77 Memory Runrun [MByte]update [MByte] Previous378.9391.1 Current210.9196.3 Factor0.560.50 mapnik Times Runrun [s]update [s]update_implicit [s] Previous867.212.54.7 Current867.59.44.1 Factor1.000.750.87 Memory Runrun [MByte]update [MByte] Previous151.5144.1 Current110.9110.7 Factor0.730.77 ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons speedup and profiling results...
Dirk, That's awesome!! Very impressive work. Brady On Sep 29, 2013 9:06 PM, Dirk Bächle tshor...@gmx.de wrote: On 26.09.2013 02:08, Gary Oberbrunner wrote: [...] I think this is excellent work! Solid analysis. I know there's been some thought given to caching subst() before; it's trickier than one might think but in many cases it should work, and it definitely speeds things up. I'm also impressed by a 30% memory reduction -- interested to hear how that comes out. I continued my work on reducing the overall memory consumption in SCons. By combining my old branch (where I switched the core to using slots) with some additional patches, I am now able to save up to 50% memory...depending on the project. Please find some results attached (a comparison between the current default tip and my experimental branch), the code can be cloned from: hg clone http://bitbucket.org/**dirkbaechle/scons_experimentalhttp://bitbucket.org/dirkbaechle/scons_experimental-r reduced_memory_updated I also achieved up to 20% speed improvements on updates, by a first version of a caching for the env.subst() method. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
[Scons-dev] SCons speedup and profiling results...
Hi there, a few minutes ago I added a new page to our Wiki. It's called http://scons.org/wiki/WhySconsIsNotSlow and shows a few results of the speedup and profiling experiments that I did recently. As mentioned in the Repositories section, you can also download the full set of results and the little test framework that I used from the Bitbucket repos: http://www.bitbucket.org/dirkbaechle/scons_testresults http://www.bitbucket.org/dirkbaechle/scons_testsuite So just dive in if you're interested, and let me hear what you think... By the way, I'm sorry for certain parts of the text being quite terse. The language could also be improved a little, I guess...but right now I'm lacking the time and patience for it (I just wanted this project off my desk ;) ). So if any native speakers out there should feel the itch to correct and extend my writings, just go ahead please. I'd highly appreciate it...and it's a Wiki after all. Best regards, Dirk ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
Re: [Scons-dev] SCons speedup and profiling results...
On Wed, Sep 25, 2013 at 5:51 PM, Dirk Bächle tshor...@gmx.de wrote: Hi there, a few minutes ago I added a new page to our Wiki. It's called http://scons.org/wiki/**WhySconsIsNotSlowhttp://scons.org/wiki/WhySconsIsNotSlow and shows a few results of the speedup and profiling experiments that I did recently. As mentioned in the Repositories section, you can also download the full set of results and the little test framework that I used from the Bitbucket repos: http://www.bitbucket.org/**dirkbaechle/scons_testresultshttp://www.bitbucket.org/dirkbaechle/scons_testresults http://www.bitbucket.org/**dirkbaechle/scons_testsuitehttp://www.bitbucket.org/dirkbaechle/scons_testsuite So just dive in if you're interested, and let me hear what you think... By the way, I'm sorry for certain parts of the text being quite terse. The language could also be improved a little, I guess...but right now I'm lacking the time and patience for it (I just wanted this project off my desk ;) ). So if any native speakers out there should feel the itch to correct and extend my writings, just go ahead please. I'd highly appreciate it...and it's a Wiki after all. I think this is excellent work! Solid analysis. I know there's been some thought given to caching subst() before; it's trickier than one might think but in many cases it should work, and it definitely speeds things up. I'm also impressed by a 30% memory reduction -- interested to hear how that comes out. Have you seen the speed and memory tests the buildbots used to run? Not sure what their status is these days but perhaps your tests could be integrated into them at some point. -- Gary ___ Scons-dev mailing list Scons-dev@scons.org http://two.pairlist.net/mailman/listinfo/scons-dev
