[Bf-committers] Blender performance measuring instrumentation improvement proposal
Hello everyone, Blender is evolving at a rather nice pace currently but I feel we could use better measurements to guide its the performance aspect of this a bit better. Some 3d games (like quake) have a feature called “timedemo” this basically runs the game as fast as the hardware will let it run. I tried doing something similar in blender and found a number of little things that maybe could be tweaked to make this possible. Overview of how this would work: The user runs the command: “blender --factory-startup --python-expr="import bpy; bpy.ops.debug_timer(type='FULLTEST', report=True); sys.exit(0);" demo_scene.blend” The following happens: 1. Blender starts and loads demo_scene.blend 2. Blender executes the python expression passed to it 3. debug_timer() is invoked: 1. Move animation counter to the start of the range 2. unlock max fps ( if vsync is disabled this allows very high framerates) 3. Gets an accurate time measurement 4. Run the animation and render 1 OpenGL frame per frame of animation 5. At the end of the range, capture time again 6. Print this time ( to stdout ?) 4. sys.exit(0), exit blender What we need for this to work: 1. implement bpy.ops.debug_timer() 2. Allow higher then 120 fps for animation playback 3. Not sure, if we allow gl frames to be locked to blender animation frames 4. (optional) implement ‘python-expr’ command line argument I might have missed stuff but the purpose of this message is to inform and discus the possibility as well as map out the bits and pieces that need to be done in order to get this functionality. I think something like this could also be instrumental in achieving high performance in the viewport project. Martijn ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
It doesn't seem to be so much an improvement to be honest, such a test will be more or less about nothing. It shouldn't be full testing, it should be per-area time measuring. For example, we could speedup viewport opengl display but accidentally slowdown dependency graph and having just an aggregated number will tell you simply nothing. it is also important to setup a strictly controlled environment for such tests if we want the numbers to mean anything. Otherwise we'll end up with folks reporting speed regressions in blender while it might be just new antivirus application being installed on their desktop. Surely if we'll have such an environment setup then it'll be handy, but the proposal for it needs much more work i'm afraid. On Wed, Jun 10, 2015 at 12:09 PM, Martijn Berger wrote: > Hello everyone, > > > Blender is evolving at a rather nice pace currently but I feel we could use > better measurements to guide its the performance aspect of this a bit > better. > > Some 3d games (like quake) have a feature called “timedemo” this basically > runs the game as fast as the hardware will let it run. I tried doing > something similar in blender and found a number of little things that maybe > could be tweaked to make this possible. > > Overview of how this would work: > > The user runs the command: > > “blender --factory-startup --python-expr="import bpy; > bpy.ops.debug_timer(type='FULLTEST', report=True); sys.exit(0);" > demo_scene.blend” > > The following happens: > > >1. > >Blender starts and loads demo_scene.blend >2. > >Blender executes the python expression passed to it >3. > >debug_timer() is invoked: >1. > > Move animation counter to the start of the range > 2. > > unlock max fps ( if vsync is disabled this allows very high > framerates) > 3. > > Gets an accurate time measurement > 4. > > Run the animation and render 1 OpenGL frame per frame of animation > 5. > > At the end of the range, capture time again > 6. > > Print this time ( to stdout ?) > 4. > >sys.exit(0), exit blender > > > What we need for this to work: > > >1. > >implement bpy.ops.debug_timer() >2. > >Allow higher then 120 fps for animation playback >3. > >Not sure, if we allow gl frames to be locked to blender animation frames >4. (optional) implement ‘python-expr’ command line argument > > > > I might have missed stuff but the purpose of this message is to inform and > discus the possibility as well as map out the bits and pieces that need to > be done in order to get this functionality. > > I think something like this could also be instrumental in achieving high > performance in the viewport project. > > > Martijn > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > -- With best regards, Sergey Sharybin ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
Hi, We actually have something like this already bpy.ops.wm.redraw_timer()(i.e. Ctrl-Alt-T) This can be used for testing how long it takes to do things like redrawing the screen/area/region(?) 10 times, and also for things like animation playback and undo/redo times. If you need anything else specific, it could probably be added there I guess... Joshua On Wed, Jun 10, 2015 at 10:09 PM, Martijn Berger wrote: > Hello everyone, > > > Blender is evolving at a rather nice pace currently but I feel we could use > better measurements to guide its the performance aspect of this a bit > better. > > Some 3d games (like quake) have a feature called “timedemo” this basically > runs the game as fast as the hardware will let it run. I tried doing > something similar in blender and found a number of little things that maybe > could be tweaked to make this possible. > > Overview of how this would work: > > The user runs the command: > > “blender --factory-startup --python-expr="import bpy; > bpy.ops.debug_timer(type='FULLTEST', report=True); sys.exit(0);" > demo_scene.blend” > > The following happens: > > >1. > >Blender starts and loads demo_scene.blend >2. > >Blender executes the python expression passed to it >3. > >debug_timer() is invoked: >1. > > Move animation counter to the start of the range > 2. > > unlock max fps ( if vsync is disabled this allows very high > framerates) > 3. > > Gets an accurate time measurement > 4. > > Run the animation and render 1 OpenGL frame per frame of animation > 5. > > At the end of the range, capture time again > 6. > > Print this time ( to stdout ?) > 4. > >sys.exit(0), exit blender > > > What we need for this to work: > > >1. > >implement bpy.ops.debug_timer() >2. > >Allow higher then 120 fps for animation playback >3. > >Not sure, if we allow gl frames to be locked to blender animation frames >4. (optional) implement ‘python-expr’ command line argument > > > > I might have missed stuff but the purpose of this message is to inform and > discus the possibility as well as map out the bits and pieces that need to > be done in order to get this functionality. > > I think something like this could also be instrumental in achieving high > performance in the viewport project. > > > Martijn > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
Hi Sergey, On Wed, Jun 10, 2015 at 12:47 PM, Sergey Sharybin wrote: > It doesn't seem to be so much an improvement to be honest, such a test will > be more or less about nothing. It shouldn't be full testing, it should be > per-area time measuring. For example, we could speedup viewport opengl > display but accidentally slowdown dependency graph and having just an > aggregated number will tell you simply nothing. > Yes this could be an issue. But I still think it is also useful to measure the whole application performance. > > it is also important to setup a strictly controlled environment for such > tests if we want the numbers to mean anything. Otherwise we'll end up with > folks reporting speed regressions in blender while it might be just new > antivirus application being installed on their desktop. > This argument could be made about all possible ways to do this. The numbers produced by some random user and posted on a forum somewhere have little meaning always no matter how you look at it. > Surely if we'll have such an environment setup then it'll be handy, but the > proposal for it needs much more work i'm afraid. > What exactly you think is difficult? Unlocking the max fps or making interface frames map to animation frames on a 1 to 1 basis ? It could maybe help if you could provide additional technical oriented feedback to that? > On Wed, Jun 10, 2015 at 12:09 PM, Martijn Berger > > wrote: > > > Hello everyone, > > > > > > Blender is evolving at a rather nice pace currently but I feel we could > use > > better measurements to guide its the performance aspect of this a bit > > better. > > > > Some 3d games (like quake) have a feature called “timedemo” this > basically > > runs the game as fast as the hardware will let it run. I tried doing > > something similar in blender and found a number of little things that > maybe > > could be tweaked to make this possible. > > > > Overview of how this would work: > > > > The user runs the command: > > > > “blender --factory-startup --python-expr="import bpy; > > bpy.ops.debug_timer(type='FULLTEST', report=True); sys.exit(0);" > > demo_scene.blend” > > > > The following happens: > > > > > >1. > > > >Blender starts and loads demo_scene.blend > >2. > > > >Blender executes the python expression passed to it > >3. > > > >debug_timer() is invoked: > >1. > > > > Move animation counter to the start of the range > > 2. > > > > unlock max fps ( if vsync is disabled this allows very high > > framerates) > > 3. > > > > Gets an accurate time measurement > > 4. > > > > Run the animation and render 1 OpenGL frame per frame of animation > > 5. > > > > At the end of the range, capture time again > > 6. > > > > Print this time ( to stdout ?) > > 4. > > > >sys.exit(0), exit blender > > > > > > What we need for this to work: > > > > > >1. > > > >implement bpy.ops.debug_timer() > >2. > > > >Allow higher then 120 fps for animation playback > >3. > > > >Not sure, if we allow gl frames to be locked to blender animation > frames > >4. (optional) implement ‘python-expr’ command line argument > > > > > > > > I might have missed stuff but the purpose of this message is to inform > and > > discus the possibility as well as map out the bits and pieces that need > to > > be done in order to get this functionality. > > > > I think something like this could also be instrumental in achieving high > > performance in the viewport project. > > > > > > Martijn > > ___ > > Bf-committers mailing list > > Bf-committers@blender.org > > http://lists.blender.org/mailman/listinfo/bf-committers > > > > > > -- > With best regards, Sergey Sharybin > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
Hi Joshua, On Wed, Jun 10, 2015 at 12:54 PM, Joshua Leung wrote: > Hi, > > We actually have something like this already bpy.ops.wm.redraw_timer()(i.e. Ctrl-Alt-T) > I was aware of that and this indeed feel like it could belong there. But this specific combination of settings is currently now there. Also there is the issue of allowing more then 120 fps for playback > > This can be used for testing how long it takes to do things like redrawing > the screen/area/region(?) 10 times, and also for things like animation > playback and undo/redo times. If you need anything else specific, it could > probably be added there I guess... > > Joshua > > Martijn ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
Hi Martijn, think its fine to have more comprehensive performance tests (which can include extending, bpy.ops.wm.redraw_timer or add related operators to give more comprehensive info). This seems like 2 separate projects, - ability to perform tests and collect feedback. - extend existing performance tests. Second needs review of each kind of test added, but that can be done on case-by-case basis, to see whats really useful to measure. Ability to execute operators directly from command line is handy, uploaded patch. https://developer.blender.org/D1347 ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
I didn't say it's difficult, i only mentioned that it is to be thought much more careful defining which exact information is useful for us and how to gather it. On Wed, Jun 10, 2015 at 1:25 PM, Campbell Barton wrote: > Hi Martijn, think its fine to have more comprehensive performance tests > (which can include extending, bpy.ops.wm.redraw_timer or add related > operators to give more comprehensive info). > > This seems like 2 separate projects, > > - ability to perform tests and collect feedback. > - extend existing performance tests. > > Second needs review of each kind of test added, but that can be done > on case-by-case basis, to see whats really useful to measure. > > > Ability to execute operators directly from command line is handy, > uploaded patch. > https://developer.blender.org/D1347 > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > -- With best regards, Sergey Sharybin ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
It would be nice to be able to separate this total performance time to sub-costs, otherwise it would indeed difficult to make out something out of that number apart from "performance changed" - which indeed says little, performance changes with every change. On 10 June 2015 at 13:38, Sergey Sharybin wrote: > I didn't say it's difficult, i only mentioned that it is to be thought much > more careful defining which exact information is useful for us and how to > gather it. > > On Wed, Jun 10, 2015 at 1:25 PM, Campbell Barton > wrote: > >> Hi Martijn, think its fine to have more comprehensive performance tests >> (which can include extending, bpy.ops.wm.redraw_timer or add related >> operators to give more comprehensive info). >> >> This seems like 2 separate projects, >> >> - ability to perform tests and collect feedback. >> - extend existing performance tests. >> >> Second needs review of each kind of test added, but that can be done >> on case-by-case basis, to see whats really useful to measure. >> >> >> Ability to execute operators directly from command line is handy, >> uploaded patch. >> https://developer.blender.org/D1347 >> ___ >> Bf-committers mailing list >> Bf-committers@blender.org >> http://lists.blender.org/mailman/listinfo/bf-committers >> > > > > -- > With best regards, Sergey Sharybin > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
Hi Sergey, I agree that the overall timer + playback has limited use cases compared to proper full instrumentation of all parts of blender. But I don't really suggest it does. What things would be good to be able to make this thing do in order to extract better information? Martijn On Wed, Jun 10, 2015 at 1:38 PM, Sergey Sharybin wrote: > I didn't say it's difficult, i only mentioned that it is to be thought much > more careful defining which exact information is useful for us and how to > gather it. > > On Wed, Jun 10, 2015 at 1:25 PM, Campbell Barton > wrote: > > > Hi Martijn, think its fine to have more comprehensive performance tests > > (which can include extending, bpy.ops.wm.redraw_timer or add related > > operators to give more comprehensive info). > > > > This seems like 2 separate projects, > > > > - ability to perform tests and collect feedback. > > - extend existing performance tests. > > > > Second needs review of each kind of test added, but that can be done > > on case-by-case basis, to see whats really useful to measure. > > > > > > Ability to execute operators directly from command line is handy, > > uploaded patch. > > https://developer.blender.org/D1347 > > ___ > > Bf-committers mailing list > > Bf-committers@blender.org > > http://lists.blender.org/mailman/listinfo/bf-committers > > > > > > -- > With best regards, Sergey Sharybin > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
Re: [Bf-committers] Blender performance measuring instrumentation improvement proposal
Martijn, thinking of something similar to current Debug Timer operator (or even an extension of it) which will perform some operation N times (N >= 10) and report total/average time on it. The operations could be: - Viewport draw (which we already have) - Full scene depsgraph evaluation - Scene depsgraph evaluation caused by timesource changes - Specified object evaluation - Internal datastructures performance testing (similar to ghash performance tests) - More extremes, which i'm not really sure how to do in a nice way: Cycles scene export time, Cycles BVH build time - Some more stuff which we can define in the future. IMO goal should be to make all those tests as reproachable as possible and much independent from external factors as possible. That's why idea of fps-based thing is something i consider unusable for reliable testing (since video driver might easily ruing your day with that). The work from Campbell about making it possible to run given code from a command line argument is handy, but not totally crucial for performance framework (since it could all be done in a .py file) On Wed, Jun 10, 2015 at 1:44 PM, Martijn Berger wrote: > Hi Sergey, > > I agree that the overall timer + playback has limited use cases compared to > proper full instrumentation of all parts of blender. > But I don't really suggest it does. > > What things would be good to be able to make this thing do in order to > extract better information? > > Martijn > > On Wed, Jun 10, 2015 at 1:38 PM, Sergey Sharybin > wrote: > > > I didn't say it's difficult, i only mentioned that it is to be thought > much > > more careful defining which exact information is useful for us and how to > > gather it. > > > > On Wed, Jun 10, 2015 at 1:25 PM, Campbell Barton > > wrote: > > > > > Hi Martijn, think its fine to have more comprehensive performance tests > > > (which can include extending, bpy.ops.wm.redraw_timer or add related > > > operators to give more comprehensive info). > > > > > > This seems like 2 separate projects, > > > > > > - ability to perform tests and collect feedback. > > > - extend existing performance tests. > > > > > > Second needs review of each kind of test added, but that can be done > > > on case-by-case basis, to see whats really useful to measure. > > > > > > > > > Ability to execute operators directly from command line is handy, > > > uploaded patch. > > > https://developer.blender.org/D1347 > > > ___ > > > Bf-committers mailing list > > > Bf-committers@blender.org > > > http://lists.blender.org/mailman/listinfo/bf-committers > > > > > > > > > > > -- > > With best regards, Sergey Sharybin > > ___ > > Bf-committers mailing list > > Bf-committers@blender.org > > http://lists.blender.org/mailman/listinfo/bf-committers > > > ___ > Bf-committers mailing list > Bf-committers@blender.org > http://lists.blender.org/mailman/listinfo/bf-committers > -- With best regards, Sergey Sharybin ___ Bf-committers mailing list Bf-committers@blender.org http://lists.blender.org/mailman/listinfo/bf-committers
