Re: [osg-users] Rewrite of DatabasePager::removeExpiredSubgraphs(double)
Hi Robert, Thanks for these changes, it was really needed. I added the method setMaximumNumOfRemovedChildPagedLODs for this purpose some time ago, allowing to unload a lot more quickly PagedLODs. For the DatabasePager::requestNodeFile() problem, my current workaround is to limite the number of files in queue (with a custom PagedLOD which check the number of files in queue before calling it). It works but I hope you'll provide a better solution. :) Cheers, On Sat, Mar 29, 2008 at 2:10 PM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Maciej, On Sat, Mar 29, 2008 at 10:41 AM, Maciej Krol [EMAIL PROTECTED] wrote: Great improvement! Memory usage dropped almost 6 fold for our paged scenes, from oscillating around 350MB to rock solid 60MB (WinXP). Wow, far better than I was even expecting, great to hear that thing a now load balancing far better - this is one of the key points of doing paging. Thanks for the feedback, Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Rewrite of DatabasePager::removeExpiredSubgraphs(double)
Hi Serge, On Mon, Mar 31, 2008 at 9:02 AM, Serge Lages [EMAIL PROTECTED] wrote: Thanks for these changes, it was really needed. I added the method setMaximumNumOfRemovedChildPagedLODs for this purpose some time ago, allowing to unload a lot more quickly PagedLODs. Have you had a chance to test the new version? For the DatabasePager::requestNodeFile() problem, my current workaround is to limite the number of files in queue (with a custom PagedLOD which check the number of files in queue before calling it). It works but I hope you'll provide a better solution. :) I haven't profiled the issue too closely yet but my guess is that the O(n) look up in the pending database requests lists is what is slowing things down. If this is the case then providing a more efficient search based on prior sorted lists would be the way forward. One could take the opposite tack and attach the DatabaseRequestObject to the PagedLOD as something like UserData. Multi-threading inside DatabasePager is also one of these items that is begging to be addressed. I may not have a chance before 2.4 to roll such changes in. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Rewrite of DatabasePager::removeExpiredSubgraphs(double)
Hi Robert,
Great improvement! Memory usage dropped almost 6 fold for our paged scenes,
from oscillating around 350MB to rock solid 60MB (WinXP).
Many thanks,
Maciej
2008/3/28, Robert Osfield [EMAIL PROTECTED]:
Hi All,
Testing that I've been carrying out on some new big terrain databases
has shown some real weaknesses in the DatabasePager ability to load
balance - with memory footprint growing significantly when the camera
moves fast over wide areas of high res data. Memory usage was growing
to over 4GB and blowing our test computers, hardly something that is
desired for a database paging system that is supposed to maintain a
modest memory footprint.
After much investigation the cause of the rather unconstrained memory
usage has been traced to the slow speed that the DatabasePager was
expiring (removing) no longer visible subgraphs - subgraphs were
correctly ear marked for removal but idsyncrosies in the threading and
original DatabasePager::removeExpiredSubgraphs(..) method meant that
it would take many frames to finally let go expired subgraphs. It in
fact was taking long enough that new data was coming in at a far
faster rate than it was been removed at hence the dramatic growth in
memory footprint.
The solution has been to rewrite
theDatabasePager::removeExpiredSubgraphs(..) method that is
responsible for expiring subgraph and setting them aside for deletion.
I've now got this new implementation working and sees a memory
footprint halved which not affecting the visual quality on screen.
Performance is now much more stable and crashes due to excessive
memory consumption are gone on my test machine. With this rewrite a
couple member variables and their associated set/getters are no longer
required, and rather leave them around for potential confusion I've
removed them completely so an svn update if you get an compile error
just remove the below methods and recompile.
-/** Set the maximum number of PagedLOD child to remove per frame
*/
-void setMaximumNumOfRemovedChildPagedLODs(unsigned int
number) { _maximumNumOfRemovedChildPagedLODs = number; }
-
-/** Get the maximum number of PagedLOD child to remove per frame
*/
-unsigned int getMaximumNumOfRemovedChildPagedLODs() const {
return _maximumNumOfRemovedChildPagedLODs; }
-
-/** Set the minimum number of inactive PagedLOD child to keep */
-void setMinimumNumOfInactivePagedLODs(unsigned int number) {
_minimumNumOfInactivePagedLODs = number; }
-
-/** Get the minimum number of inactive PagedLOD child to keep */
-unsigned int getMinimumNumOfInactivePagedLODs() const {
return _minimumNumOfInactivePagedLODs; }
I would like to improve this method further still as I've already come
up with a better load balancing algorithm to implement in this method,
however, it'll take a while longer to implement, so rather than wait
for this polished implementation I'm checking in what is working
effectively, albeit not perfect it is far better than what was there
before.
There are other aspects of the DatabasePager that need attention too,
such as the high cost of DatabasePager::requestNodeFile() method
during cull traversal when many file requested are queued up. This is
something that has been raised on the list before by others, so it's a
bit of common theme when we really start to push the size of paged
databases we are dealing with. I will be spending more time on this
code over the coming month, so feel free to chip in with your own
experiences.
Cheers,
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] Rewrite of DatabasePager::removeExpiredSubgraphs(double)
Hi Maciej, On Sat, Mar 29, 2008 at 10:41 AM, Maciej Krol [EMAIL PROTECTED] wrote: Great improvement! Memory usage dropped almost 6 fold for our paged scenes, from oscillating around 350MB to rock solid 60MB (WinXP). Wow, far better than I was even expecting, great to hear that thing a now load balancing far better - this is one of the key points of doing paging. Thanks for the feedback, Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
[osg-users] Rewrite of DatabasePager::removeExpiredSubgraphs(double)
Hi All,
Testing that I've been carrying out on some new big terrain databases
has shown some real weaknesses in the DatabasePager ability to load
balance - with memory footprint growing significantly when the camera
moves fast over wide areas of high res data. Memory usage was growing
to over 4GB and blowing our test computers, hardly something that is
desired for a database paging system that is supposed to maintain a
modest memory footprint.
After much investigation the cause of the rather unconstrained memory
usage has been traced to the slow speed that the DatabasePager was
expiring (removing) no longer visible subgraphs - subgraphs were
correctly ear marked for removal but idsyncrosies in the threading and
original DatabasePager::removeExpiredSubgraphs(..) method meant that
it would take many frames to finally let go expired subgraphs. It in
fact was taking long enough that new data was coming in at a far
faster rate than it was been removed at hence the dramatic growth in
memory footprint.
The solution has been to rewrite
theDatabasePager::removeExpiredSubgraphs(..) method that is
responsible for expiring subgraph and setting them aside for deletion.
I've now got this new implementation working and sees a memory
footprint halved which not affecting the visual quality on screen.
Performance is now much more stable and crashes due to excessive
memory consumption are gone on my test machine. With this rewrite a
couple member variables and their associated set/getters are no longer
required, and rather leave them around for potential confusion I've
removed them completely so an svn update if you get an compile error
just remove the below methods and recompile.
-/** Set the maximum number of PagedLOD child to remove per frame */
-void setMaximumNumOfRemovedChildPagedLODs(unsigned int
number) { _maximumNumOfRemovedChildPagedLODs = number; }
-
-/** Get the maximum number of PagedLOD child to remove per frame */
-unsigned int getMaximumNumOfRemovedChildPagedLODs() const {
return _maximumNumOfRemovedChildPagedLODs; }
-
-/** Set the minimum number of inactive PagedLOD child to keep */
-void setMinimumNumOfInactivePagedLODs(unsigned int number) {
_minimumNumOfInactivePagedLODs = number; }
-
-/** Get the minimum number of inactive PagedLOD child to keep */
-unsigned int getMinimumNumOfInactivePagedLODs() const {
return _minimumNumOfInactivePagedLODs; }
I would like to improve this method further still as I've already come
up with a better load balancing algorithm to implement in this method,
however, it'll take a while longer to implement, so rather than wait
for this polished implementation I'm checking in what is working
effectively, albeit not perfect it is far better than what was there
before.
There are other aspects of the DatabasePager that need attention too,
such as the high cost of DatabasePager::requestNodeFile() method
during cull traversal when many file requested are queued up. This is
something that has been raised on the list before by others, so it's a
bit of common theme when we really start to push the size of paged
databases we are dealing with. I will be spending more time on this
code over the coming month, so feel free to chip in with your own
experiences.
Cheers,
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
