Re: [Flightgear-devel] OSG caching and other OSG issues
Am 01.04.2012 18:25, schrieb Mathias Fröhlich: > If you work with weak pointers/observer_ptr you need to use the lock method. Ok, you're absolutely right. I wasn't aware of the "lock" method before, indeed it has to be used here. I'll push a change. Thanks for reviewing this! cheers, Thorsten -- This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching and other OSG issues
Hi, I try to catch up on things past an other busy week. On Thursday, March 29, 2012 00:22:46 ThorstenB wrote: > Mathias, maybe you can have a look at the simgear changes. Maybe you see > a way to improve this even further - i.e. do we still need all these > cache maps? Or could we simply rely on some OSG cache in some places? Just looking quick over what you checked in, I think that this needs some improovements: The change just returns pointers to objects you get from the observer_ptr. In general, this is not exactly thread safe. I do not see currently if you can ensure from a higher level that this is not a problem or not. In general consider the following situation: Thread 1 has an observer_ptr to A. Thread 2 owns the last reference to A. Thread 1 gets the still valid raw pointer to A from the observer_ptr. Thread 2 releases the last reference to A and deletes it. Thread 1 assigns the raw pointer to a ref_ptr. This probably ends in a segfault. If you work with weak pointers/observer_ptr you need to use the lock method. That one gives you a atomically a dead or alive ref_ptr to A. If you have this ref_ptr in our hands, you own a thread safe reference which makes sure that nobody deletes the object. If your ref_ptr is invalid. An other one was faster on unreferencing. In effect this groups step 3 and 5 into an atomic operation which can not be intercepted by step 4 in the example above. So, if you use this kind of weak stuff, you need to ensure that all the call chain from getting the ref_ptr from any cache down to adding this object to the scenegraph does not use raw pointers. Also the initial ref_ptr in this chain must be gained by observer_ptr::lock(). Regarding the effects, I don't think we can rely on any caching in osg since we do not currently use any osg based loader mechanism to gain these objects. At least nothing that I know of. Greetings Mathias -- This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching and other OSG issues
Hi, I've pushed a fix for a larger memory issue, mainly affecting effects/texture images, but also other objects. Once loaded, these stuck in memory until shutting down flightgear - so memory consumption was continually growing when flying larger distances. Probably wasn't much of an issue initially, when we only had few or generic textures. Now, with lots of custom models and eye candy, we obviously should be able to drop stuff again. We've discussed this about a year ago. Tim suggested using "observer pointers" instead of "references" in the cache maps. The discussed patch was never pushed to sg/next though, since I was running into stability issues with OSG 2.8.3. Since we don't support OSG 2.8.3 any longer, I've revived this again and changes are in simgear now. Keeps memory consumption a lot lower. I'm still seeing a slight memory growth when moving - but by far, it's not as bad as it was. Simple test is to "relocate-on-ground" to a series of airport locations and watch process memory. Mathias, maybe you can have a look at the simgear changes. Maybe you see a way to improve this even further - i.e. do we still need all these cache maps? Or could we simply rely on some OSG cache in some places? cheers, Thorsten Am 03.05.2011 21:49, schrieb ThorstenB: > On Mon, Apr 18, 2011 at 9:55 AM, Tim Moore wrote: >> On Sun, Apr 17, 2011 at 6:10 PM, ThorstenB wrote: >>> There is >>> a global cache in simgear/makeEffect.cxx (effectMap), and it has no >>> condition to ever drop anything. So, created effects always stay in >>> memory until shutdown - hence also their textures. >> >> It seemed like a good idea at the time :) >> The Effect cache could be changed to use osg::observer_ptr, which is a >> weak pointer. > > Good idea. Here's a patch using observers instead of references for > caching. Puts some live into the effect/texture destructors - so these > can finally free up some memory once the objects are no longer used > anywhere: > http://www.gitorious.org/~thorsten/fg/thorstenbs-simgear/commit/71d14629076b3a56c40cc0309b42b3a22d579bec -- This SF email is sponsosred by: Try Windows Azure free for 90 days Click Here http://p.sf.net/sfu/sfd2d-msazure ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
[Flightgear-devel] OSG caching and other OSG issues
On Mon, Apr 18, 2011 at 9:55 AM, Tim Moore wrote: > On Sun, Apr 17, 2011 at 6:10 PM, ThorstenB wrote: >> There is >> a global cache in simgear/makeEffect.cxx (effectMap), and it has no >> condition to ever drop anything. So, created effects always stay in >> memory until shutdown - hence also their textures. If that's used for >> scenery/MP aircraft a lot, it _might_ accumulate a lot of memory. Tim >> might know more about this :). > > It seemed like a good idea at the time :) > > The Effect cache could be changed to use osg::observer_ptr, which is a > weak pointer. Good idea. Here's a patch using observers instead of references for caching. Puts some live into the effect/texture destructors - so these can finally free up some memory once the objects are no longer used anywhere: http://www.gitorious.org/~thorsten/fg/thorstenbs-simgear/commit/71d14629076b3a56c40cc0309b42b3a22d579bec Patch is in my local branch (not in next). Any comments? It indeed improves memory consumption when switching (or flying :) ) between airports. However, I'm still seeing memory growth/leaks (and I'm really starting to dislike all these "smart" auto pointers - searching the last reference causing an object to leak, is a really annoying job...). I also noticed a bad leak in older OSG versions - including the stable release (2.8.3). It's fixed by now (OSG svn), but I haven't checked for which version it was fixed first. However, we've lost 3D cloud support with latest OSG trunk (not a single 3D cloud visible, everything fine with recent OSG devel releases though). Might be temporary OSG trunk issue... Also, osgText still isn't working for us with latest trunk (see http://code.google.com/p/flightgear-bugs/issues/detail?id=268 ). There were several fixes and updates to osgText just before we started seeing the issues. Since it still isn't fixed on the OSG side, it might also be our problem after all: maybe we're relying on an old osgText bug - or we need to adapt something. We'll probably need to look into this... cheers, Thorsten -- WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On Sun, Apr 17, 2011 at 6:10 PM, ThorstenB wrote: > On 16.04.2011 21:16, Anders Gidenstam wrote: >> If I'm not mistaken the particles issue has been around since we got >> particles, so it is apparently not that bad (leak and race >> condition) in practice. > Ok, thanks! I've create a bug issue as a reminder, in case someone else > noticed the issue some day. > http://code.google.com/p/flightgear-bugs/issues/detail?id=305 > >>> Not sure, but I guess the issues with "Effect" objects is probably worse >>> than with particles. There's a larger number of Effect objects - and >>> each is connected to a texture (.rgb/.png images) - which may occupy a >>> lot of memory. Just by starting at KSFO, loads of KSFO terminal textures >>> - and textures of 15 different MP aircraft immediately stick in memory... >> Yes, the big textures will eat memory fast. Particle systems usually use >> small textures (the effect of the accumulation of particle system updaters >> is noticeable with wildfire, though - but you'd need a lot of MP aircraft >> passing through to generate anywhere near those numbers of particle >> systems). > The effect/texture "mystery" is also solved - alas - explained. There is > a global cache in simgear/makeEffect.cxx (effectMap), and it has no > condition to ever drop anything. So, created effects always stay in > memory until shutdown - hence also their textures. If that's used for > scenery/MP aircraft a lot, it _might_ accumulate a lot of memory. Tim > might know more about this :). It seemed like a good idea at the time :) The Effect cache could be changed to use osg::observer_ptr, which is a weak pointer. Also, there may be some places where effects are not being found in the cache and are being recreated. Tim > > cheers, > Thorsten > > -- > Benefiting from Server Virtualization: Beyond Initial Workload > Consolidation -- Increasing the use of server virtualization is a top > priority.Virtualization can reduce costs, simplify management, and improve > application availability and disaster protection. Learn more about boosting > the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev > ___ > Flightgear-devel mailing list > Flightgear-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/flightgear-devel > -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On 17 Apr 2011, at 18:10, ThorstenB wrote: > The effect/texture "mystery" is also solved - alas - explained. There is > a global cache in simgear/makeEffect.cxx (effectMap), and it has no > condition to ever drop anything. So, created effects always stay in > memory until shutdown - hence also their textures. If that's used for > scenery/MP aircraft a lot, it _might_ accumulate a lot of memory. Tim > might know more about this :). > Interesting. After much trial and error, I managed to get a basic visualization system working for tracking AI traffic (see my question yesterday). My current implementation is taking a similar approach as what is used for generating runway and taxiway signs and also uses a simgear effects class. Because I use this system for plotting routes that are active, the system is quite dynamic and involves a lot of regenerations of the corresponding scenegraph node. During some of the early tests, regenerating the entire node on a frame to frame basis caused a dramatic slowdown of FlightGear, even though I am 99% sure that the old versions were appropriately removed from the scene graph and deleted from memory. I wonder whether the slow-down is related to the aforementioned effectMap bug. Anyways, my code is still a long way away from being production ready, and the visualization part is one section that probably needs a second opinion. All 'n' all, the AI/ATC interaction is progressing nicely (as long as I don't get distracted by launching Vostok's into space that is). :-) Cheers, Durk -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On 16.04.2011 21:16, Anders Gidenstam wrote: > If I'm not mistaken the particles issue has been around since we got > particles, so it is apparently not that bad (leak and race > condition) in practice. Ok, thanks! I've create a bug issue as a reminder, in case someone else noticed the issue some day. http://code.google.com/p/flightgear-bugs/issues/detail?id=305 >> Not sure, but I guess the issues with "Effect" objects is probably worse >> than with particles. There's a larger number of Effect objects - and >> each is connected to a texture (.rgb/.png images) - which may occupy a >> lot of memory. Just by starting at KSFO, loads of KSFO terminal textures >> - and textures of 15 different MP aircraft immediately stick in memory... > Yes, the big textures will eat memory fast. Particle systems usually use > small textures (the effect of the accumulation of particle system updaters > is noticeable with wildfire, though - but you'd need a lot of MP aircraft > passing through to generate anywhere near those numbers of particle > systems). The effect/texture "mystery" is also solved - alas - explained. There is a global cache in simgear/makeEffect.cxx (effectMap), and it has no condition to ever drop anything. So, created effects always stay in memory until shutdown - hence also their textures. If that's used for scenery/MP aircraft a lot, it _might_ accumulate a lot of memory. Tim might know more about this :). cheers, Thorsten -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On Sat, 16 Apr 2011, ThorstenB wrote: > Ok, thanks a lot Anders for these hints! I'll open a tracker issue for > the particle issue - so we won't forget these details. Can you estimate > on how bad this issue could get? Does it only mean a minor memory leak - > or could it get really bad over time (maybe eat some gigabytes in a 6 > hour TGA shift? :) ). If I'm not mistaken the particles issue has been around since we got particles, so it is apparently not that bad (leak and race condition) in practice. > Not sure, but I guess the issues with "Effect" objects is probably worse > than with particles. There's a larger number of Effect objects - and > each is connected to a texture (.rgb/.png images) - which may occupy a > lot of memory. Just by starting at KSFO, loads of KSFO terminal textures > - and textures of 15 different MP aircraft immediately stick in memory... Yes, the big textures will eat memory fast. Particle systems usually use small textures (the effect of the accumulation of particle system updaters is noticeable with wildfire, though - but you'd need a lot of MP aircraft passing through to generate anywhere near those numbers of particle systems). Cheers, Anders -- --- Anders Gidenstam WWW: http://www.gidenstam.org/FlightGear/ -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On 16.04.2011 16:24, Anders Gidenstam wrote: > On Sat, 16 Apr 2011, Anders Gidenstam wrote: > > Hmm.. I should have added that I looked at this in early September last > year so it is not fresh in my memory. > > IIRC the current code already does the fundamentaly unsafe operation of > adding items to a STL vector (in the loader thread) while concurrent > traversals of the same vector are possible (by various parts of OSG > rendering). That is NOT SAFE but a lot less risky than deleting from an > STL vector with concurrent traversal (consider the most likely internal > representation of a STL vector to see why). Ok, thanks a lot Anders for these hints! I'll open a tracker issue for the particle issue - so we won't forget these details. Can you estimate on how bad this issue could get? Does it only mean a minor memory leak - or could it get really bad over time (maybe eat some gigabytes in a 6 hour TGA shift? :) ). Not sure, but I guess the issues with "Effect" objects is probably worse than with particles. There's a larger number of Effect objects - and each is connected to a texture (.rgb/.png images) - which may occupy a lot of memory. Just by starting at KSFO, loads of KSFO terminal textures - and textures of 15 different MP aircraft immediately stick in memory... cheers, Thorsten -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On Sat, 16 Apr 2011, Anders Gidenstam wrote:
> My failed attempt to unlink the particle systems might provide some insight
> into that issue. The attempt is broken because the
> methods calls
>if (!Particles::getCommonRoot()->removeChild(item)) {
> or
>if (!Particles::getCommonGeode()->removeDrawable(item)) {
> can get called while another OSG thread traverses or inserts into the vector
> of children to the common root/geod.
Hmm.. I should have added that I looked at this in early September last
year so it is not fresh in my memory.
IIRC the current code already does the fundamentaly unsafe operation of
adding items to a STL vector (in the loader thread) while concurrent
traversals of the same vector are possible (by various parts of OSG
rendering). That is NOT SAFE but a lot less risky than deleting from an
STL vector with concurrent traversal (consider the most likely internal
representation of a STL vector to see why).
Cheers,
Anders
--
---
Anders Gidenstam
WWW: http://www.gidenstam.org/FlightGear/
--
Benefiting from Server Virtualization: Beyond Initial Workload
Consolidation -- Increasing the use of server virtualization is a top
priority.Virtualization can reduce costs, simplify management, and improve
application availability and disaster protection. Learn more about boosting
the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev
___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On Sat, 16 Apr 2011, Anders Gidenstam wrote:
I think that is a different leak from the effects. IIRC each particle
system is attached to the scene graph in (at least) two places: at the
emitter's location in the graph and in a global vector of particle system
updaters. IIRC they are never removed from the global list of updaters
(a bug - but a hard one to fix due to concurrency issues).
Tim knows more about this.
My failed attempt to unlink the particle systems might provide some
insight into that issue. The attempt is broken because the
methods calls
if (!Particles::getCommonRoot()->removeChild(item)) {
or
if (!Particles::getCommonGeode()->removeDrawable(item)) {
can get called while another OSG thread traverses or inserts into the
vector of children to the common root/geod.
Basically the ~CustomMatrixTransform() destructor is called on the wrong
thread for this attempt to work. The clean up action needs either to be
communicated to the right thread or synchronization added to
removeChild/Drawable() method (which would be against the OSG design).
(Note: the preprocessor variable OSG_PARTICLE_FIX is unrelated to my
changes.)
Cheers,
Anders
--
---
Anders Gidenstam
WWW: http://www.gidenstam.org/FlightGear/diff --git a/simgear/scene/model/particles.cxx b/simgear/scene/model/particles.cxx
index acac7df..4d49ac3 100644
--- a/simgear/scene/model/particles.cxx
+++ b/simgear/scene/model/particles.cxx
@@ -51,6 +51,57 @@
namespace simgear
{
+
+class CustomMatrixTransform : public osg::MatrixTransform
+{
+public:
+ CustomMatrixTransform(osgParticle::ParticleSystem* particleSystem) :
+ item(0) {
+ this->particleSystem = particleSystem;
+ };
+
+#ifdef OSG_PARTICLE_FIX
+void setCleanupItem(osg::Group* item)
+#else
+void setCleanupItem(osg::Drawable* item)
+#endif
+{
+this->item = item;
+}
+
+protected:
+~CustomMatrixTransform()
+{
+if (!Particles::getPSU()->removeParticleSystem(particleSystem)) {
+SG_LOG(SG_GENERAL, SG_ALERT,
+ "particles.cxx: ~CustomMatrixTransform() "
+ "Cleanup of particle system updater failed" << std::endl);
+}
+if (item) {
+#ifdef OSG_PARTICLE_FIX
+if (!Particles::getCommonRoot()->removeChild(item)) {
+SG_LOG(SG_GENERAL, SG_ALERT,
+ "particles.cxx: ~CustomMatrixTransform() "
+ "Cleanup of common root failed" << std::endl);
+}
+#else
+if (!Particles::getCommonGeode()->removeDrawable(item)) {
+SG_LOG(SG_GENERAL, SG_ALERT,
+ "particles.cxx: ~CustomMatrixTransform() "
+ "Cleanup of common geode failed" << std::endl);
+}
+#endif
+}
+}
+
+osg::ref_ptr particleSystem;
+#ifdef OSG_PARTICLE_FIX
+osg::ref_ptr item;
+#else
+osg::ref_ptr item;
+#endif
+};
+
void GlobalParticleCallback::operator()(osg::Node* node, osg::NodeVisitor* nv)
{
enabled = !enabledNode || enabledNode->getBoolValue();
@@ -151,7 +202,7 @@ osg::Group * Particles::appendParticles(const SGPropertyNode* configNode,
//may not be used depending on the configuration
PointerGuard callback;
-getPSU()->addParticleSystem(particleSys);
+getPSU()->addParticleSystem(particleSys);
getPSU()->setUpdateCallback(new GlobalParticleCallback(modelRoot));
//contains counter, placer and shooter by default
osgParticle::ModularEmitter* emitter = new osgParticle::ModularEmitter;
@@ -160,7 +211,7 @@ osg::Group * Particles::appendParticles(const SGPropertyNode* configNode,
// Set up the alignment node ("stolen" from animation.cxx)
// XXX Order of rotations is probably not correct.
-osg::MatrixTransform *align = new osg::MatrixTransform;
+CustomMatrixTransform* align = new CustomMatrixTransform(particleSys);
osg::Matrix res_matrix;
res_matrix.makeRotate(
configNode->getFloatValue("offsets/pitch-deg", 0.0)*SG_DEGREES_TO_RADIANS,
@@ -202,8 +253,10 @@ osg::Group * Particles::appendParticles(const SGPropertyNode* configNode,
g->addDrawable(particleSys);
callback()->particleFrame->addChild(g);
getCommonRoot()->addChild(callback()->particleFrame.get());
+align->setCleanupItem(callback()->particleFrame.get());
#else
getCommonGeode()->addDrawable(particleSys);
+align->setCleanupItem(particleSys);
#endif
}
std::string textureFile;
--
Benefiting from Server Virtualization: Beyond Initial Workload
Consolidation -- Increasing the use of server virtualization is a top
priority.Virtualization can reduce costs, simplify management, and improve
application availability and disaster protection. Learn more about boosting
the value of server vir
Re: [Flightgear-devel] OSG caching
On Sat, 16 Apr 2011, ThorstenB wrote: > Another observation: I started an MP session at KSFO (lots of MP > aircraft), then warped to the middle of nowhere (no MP aircraft). > After about 30min I dumped the scene graph. Surprisingly, loads of > osg::particles were still *in* the scene graph, referring to aircraft > specific smoke and contrail textures. I would have expected them to be > dropped from the scene graph after some time - or when we move to a > very distant position. Maybe there's another issue. I think that is a different leak from the effects. IIRC each particle system is attached to the scene graph in (at least) two places: at the emitter's location in the graph and in a global vector of particle system updaters. IIRC they are never removed from the global list of updaters (a bug - but a hard one to fix due to concurrency issues). Tim knows more about this. Cheers, Anders -- --- Anders Gidenstam WWW: http://www.gidenstam.org/FlightGear/ -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
Hi,
had another look at memory consumption. The FG multiplayer (=AI)
aircraft classes seem fine - they are created/removed as expected. But
there are problems with some of our OSG-based simgear classes: they
are never removed at run-time - hence memory is eaten up.
Problems start at simgear::Effect. Their parent objects
(simgear::EffectGeode) are still removed when the related model
disappears from the scene graph. But simgear::Effect objects stick in
memory, until exiting FG. Their reference counter shows that there
must be (forgotten) references to each of them somewhere (but they are
no longer members of the scene graph). Attached is a simple patch
logging Effect constructor/destructor calls. Shows that not a single
Effect::~Effect destructor is called at run-time - though we
continuously create new objects. Effect objects own 2D textures
(images), techniques, etc - so they have considerable memory impact.
Any suggestions on how to trace the cause?
New Effect objects are created when new MP aircraft join or new
scenery areas are loaded. So, no surprise there are issues with large
MP events/long distance flights.
Another observation: I started an MP session at KSFO (lots of MP
aircraft), then warped to the middle of nowhere (no MP aircraft).
After about 30min I dumped the scene graph. Surprisingly, loads of
osg::particles were still *in* the scene graph, referring to aircraft
specific smoke and contrail textures. I would have expected them to be
dropped from the scene graph after some time - or when we move to a
very distant position. Maybe there's another issue.
For me, this is all independent from enabling/disabling the OSG cache.
But maybe it's worse now, since we have more advanced aircraft, making
more use of effects and particles. And I guess the pretty local
weather clouds also make use of these.
cheers,
Thorsten
diff --git a/simgear/scene/material/Effect.cxx b/simgear/scene/material/Effect.cxx
index 6517bd2..314d131 100644
--- a/simgear/scene/material/Effect.cxx
+++ b/simgear/scene/material/Effect.cxx
@@ -83,15 +83,19 @@ using namespace osgUtil;
using namespace effect;
+static unsigned long EffectCount=0;
+
Effect::Effect()
: _cache(0), _isRealized(false)
{
+printf("%4lu Effect::Effect()\n",++EffectCount);
}
Effect::Effect(const Effect& rhs, const CopyOp& copyop)
: root(rhs.root), parametersProp(rhs.parametersProp), _cache(0),
_isRealized(rhs._isRealized)
{
+printf("%4lu Effect::Effect(..)\n",++EffectCount);
typedef vector > TechniqueList;
for (TechniqueList::const_iterator itr = rhs.techniques.begin(),
end = rhs.techniques.end();
@@ -149,6 +153,7 @@ void Effect::releaseGLObjects(osg::State* state) const
Effect::~Effect()
{
+printf("%4lu Effect::~Effect(..)\n",--EffectCount);
delete _cache;
}
--
Benefiting from Server Virtualization: Beyond Initial Workload
Consolidation -- Increasing the use of server virtualization is a top
priority.Virtualization can reduce costs, simplify management, and improve
application availability and disaster protection. Learn more about boosting
the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev___
Flightgear-devel mailing list
Flightgear-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On 13.04.2011 22:52, Csaba Halász wrote: > On Wed, Apr 13, 2011 at 9:16 PM, ThorstenB wrote: >> I never ran FG for 6 hours with clouds& MP enabled though. But are we >> sure that the OSG cache itself really made a major difference now? > > Yes, pretty sure. I fly FG during the monthly TGA events, and it was > never even half as bad as this time. Except for the one massive leak > you fixed, but luckily I didn't try to run that version during an > event :) I looked into OSG's cache behaviour when running FlightGear. Worked as expected though: I see the objects being dropped once they are no longer referenced and have expired. However, I saw the same issue which my memory debugger had reported before: some scenery elements, mainly related to MP aircraft, don't (always) get deleted. Indeed, these elements stick in the OSG cache now. But they stick since their reference count proves that they are still referenced elsewhere - so the cache holds on to them. It wouldn't help to drop referenced objects from the cache - they wouldn't be deleted anyway. So, yes, there's a leak related to MP aircraft - but the cause seems to be independent of the OSG cache. I'll be doing some more tests though. I've also pushed an update to clear the entire OSG cache when reloading scenery (see menu "Debug" => "Reload Scenery"). If memory consumption explodes, try triggering a scenery reload to wipe the cache. I doubt this would reduce memory though - but maybe there's a surprise. Csaba, which OSG version are you running anyway? cheers, Thorsten -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On Wed, Apr 13, 2011 at 9:16 PM, ThorstenB wrote: > * or it's all our fault, our leak alone - more or less unrelated to caching. > > I think the latter is (still) the case. I already posted some weeks ago > that I was seeing loads of leaked objects. I managed to fix a few things > at the time > > I never ran FG for 6 hours with clouds & MP enabled though. But are we > sure that the OSG cache itself really made a major difference now? Yes, pretty sure. I fly FG during the monthly TGA events, and it was never even half as bad as this time. Except for the one massive leak you fixed, but luckily I didn't try to run that version during an event :) -- Csaba/Jester -- Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On 13.04.2011 13:46, Frederic Bouvier wrote: > To reduce memory footprint caused by model caching, I added this code to the > begining of the main function of fgrun : > > osgDB::Registry* registry = osgDB::Registry::instance(); > registry->setExpiryDelay( 0. ); > > I suspect we can control the amount of caching the same way in FG The default expiration delay is 10 seconds. Any texture/image/... which isn't referenced any longer should automatically be purged after 10 seconds. Reducing it to 0 shouldn't help too much - just save 10 seconds worth of memory. But it would void performance benefits when stuff get's dropped and then reloaded immediately, e.g. you do a sim reset or the MP connection has a brief drop-out (I sometimes see MP models disappear and reappear after a few seconds). The real problem should be elsewhere: * either the whole OSG caching system just doesn't work at all, i.e. it never purges unreferenced and expired data (I somehow doubt this). * we somehow haven't implemented the caching thing correctly (no idea what we could be missing) * or it's all our fault, our leak alone - more or less unrelated to caching. I think the latter is (still) the case. I already posted some weeks ago that I was seeing loads of leaked objects. I managed to fix a few things at the time http://www.gitorious.org/fg/simgear/commit/44f27b23d0209d2ee9f508c43def5636564bb302 http://www.gitorious.org/fg/flightgear/commit/4761a3cdcf04771819b3a8de9d018fca20c90ca6 http://www.gitorious.org/fg/flightgear/commit/8962477cfa10850cb459d7de754c9a435cc91293 but that wasn't all of it by far. Still looked "leaky as a sieve" to me :| and mainly affected MP models, clouds (global weather based) and SG animation stuff - and thousands of properties connected to these. The longer FG was running (or the more players joined and left) the more objects were leaked. And that was long before we enabled the OSG cache. I never ran FG for 6 hours with clouds & MP enabled though. But are we sure that the OSG cache itself really made a major difference now? The thing is that we do have a lot of "automatic pointers" (objects with reference counting) in our code. It's all our classes based on OSG and all the SGShared... properties/classes. It's very tricky to find out if everything is working as expected here, since it's hard to tell when something should be removed. A simple mistake somewhere (like a non-virtual destructor in a base class), a single object which doesn't get disconnected - and a huge object tree connected to it never get's removed. It'd certainly be worth to investigate further and fix more leaks - but debugging these issues is painful and requires a long and tricky hunt. Any help would be appreciated... cheers, Thorsten -- Forrester Wave Report - Recovery time is now measured in hours and minutes not days. Key insights are discussed in the 2010 Forrester Wave Report as part of an in-depth evaluation of disaster recovery service providers. Forrester found the best-in-class provider in terms of services and vision. Read this report now! http://p.sf.net/sfu/ibm-webcastpromo ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
> > You can say THAT again! I wonder if OSG people know that they should > > also throw out stuff from cache, not just put things in! > > I have been bold enough to use the CACHE_ALL for today's TGA > > multiplayer event ... at the end of the 6th hour or so, FG's memory > > usage reached 7.5GB, with the extensive swapping negating any eventual > > advantage of the caching. > > > > So, I certainly do not recommend setting this option on by default, > > until the behavior is fixed (such as by introducing a memory limit). To reduce memory footprint caused by model caching, I added this code to the begining of the main function of fgrun : osgDB::Registry* registry = osgDB::Registry::instance(); registry->setExpiryDelay( 0. ); I suspect we can control the amount of caching the same way in FG Regards, -Fred -- Frédéric Bouvier http://www.youtube.com/user/fgfred64 Videos -- Forrester Wave Report - Recovery time is now measured in hours and minutes not days. Key insights are discussed in the 2010 Forrester Wave Report as part of an in-depth evaluation of disaster recovery service providers. Forrester found the best-in-class provider in terms of services and vision. Read this report now! http://p.sf.net/sfu/ibm-webcastpromo ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On Wed, Apr 13, 2011 at 1:29 PM, wrote: > > I've got a stupid question - maybe someone who knows these things can > educate me. > > If I do not use texture caching, textures are loaded from harddisk every > time I need them, for each and every model. > > If I do use texture caching but my memory runs over, swapping takes place. > Sometimes I may get lucky and the texture I need is currently in memory, > sometimes it is on harddisk and needs to be loaded. > > Naively, it seems to me the second case is still faster than the first, > because I always save at least some harddisk operations. I haven't flown > for 6 hours straight with CACHE_ALL, but for two hours something, and it > was still way faster than with CACHE_NONE. Depending on the format of the textures on disk and in memory, reading and converting a possibly smaller texture from the original file may be faster than reading back the already converted texture from swap (if it is not in memory at the given time). Additionally, I was almost at the limit of available swap space, I had to add a temporary swap file for fear of running out of virtual memory and thus getting a crash. (Would have sucked after 6 hours of flight). As I said, I was up to 7GB but I use a 64 bit system. People with 32 bit machines would run into trouble much earlier, typically at 2 or 3GB already. -- Csaba/Jester -- Forrester Wave Report - Recovery time is now measured in hours and minutes not days. Key insights are discussed in the 2010 Forrester Wave Report as part of an in-depth evaluation of disaster recovery service providers. Forrester found the best-in-class provider in terms of services and vision. Read this report now! http://p.sf.net/sfu/ibm-webcastpromo ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
> You can say THAT again! I wonder if OSG people know that they should > also throw out stuff from cache, not just put things in! > I have been bold enough to use the CACHE_ALL for today's TGA > multiplayer event ... at the end of the 6th hour or so, FG's memory > usage reached 7.5GB, with the extensive swapping negating any eventual > advantage of the caching. > > So, I certainly do not recommend setting this option on by default, > until the behavior is fixed (such as by introducing a memory limit). I've got a stupid question - maybe someone who knows these things can educate me. If I do not use texture caching, textures are loaded from harddisk every time I need them, for each and every model. If I do use texture caching but my memory runs over, swapping takes place. Sometimes I may get lucky and the texture I need is currently in memory, sometimes it is on harddisk and needs to be loaded. Naively, it seems to me the second case is still faster than the first, because I always save at least some harddisk operations. I haven't flown for 6 hours straight with CACHE_ALL, but for two hours something, and it was still way faster than with CACHE_NONE. So, do I really lose anything here? Perhaps there's something I'm not seeing... Cheers, * Thorsten -- Forrester Wave Report - Recovery time is now measured in hours and minutes not days. Key insights are discussed in the 2010 Forrester Wave Report as part of an in-depth evaluation of disaster recovery service providers. Forrester found the best-in-class provider in terms of services and vision. Read this report now! http://p.sf.net/sfu/ibm-webcastpromo ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching (was: Texture cache)
On Sun, Apr 3, 2011 at 11:39 PM, Tim Moore wrote: > On Sun, Apr 3, 2011 at 3:41 PM, ThorstenB wrote: >> I've also been using CACHE_ALL since then - not seeing any problems. But >> I haven't checked memory consumption. So, what's the status about the >> OSG caching options, should we enable these? Tim? > There are two issues I can think of. One is that animations might not > work correctly with caching enabled, but I think the copying we do > elsewhere should take care of that. The other is that memory usage > will be higher with caching enabled. You can say THAT again! I wonder if OSG people know that they should also throw out stuff from cache, not just put things in! I have been bold enough to use the CACHE_ALL for today's TGA multiplayer event ... at the end of the 6th hour or so, FG's memory usage reached 7.5GB, with the extensive swapping negating any eventual advantage of the caching. So, I certainly do not recommend setting this option on by default, until the behavior is fixed (such as by introducing a memory limit). -- Cheers, Csaba/Jester -- Xperia(TM) PLAY It's a major breakthrough. An authentic gaming smartphone on the nation's most reliable network. And it wants your games. http://p.sf.net/sfu/verizon-sfdev ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching (was: Texture cache)
On Sun, Apr 3, 2011 at 3:41 PM, ThorstenB wrote: Maybe someone could do some tests when changing the setting >> (SGPagedLOD.hxx:56) from "CACHE_NONE" to "CACHE_IMAGES" or even to >> "CACHE_ALL" (then recompile/install sg+fg). Would be interesting to >> know >> how this changed loading times, run-time fps and memory consumption. >> After 30 minutes more of testing: It also works in practice. I have seen >> no averse side effects, and the performance drop from loading new clouds >> is now almost completely gone - the framerate drops I still see are mainly >> associated with terrain sampling. >> >> The actual rendering performance is, as far as I can see, not changed, >> i.e. once everything is there, it is there. But all in all this makes >> things way smoother. It should be most pronounced on single CPU machines. > I've also been using CACHE_ALL since then - not seeing any problems. But > I haven't checked memory consumption. So, what's the status about the > OSG caching options, should we enable these? Tim? There are two issues I can think of. One is that animations might not work correctly with caching enabled, but I think the copying we do elsewhere should take care of that. The other is that memory usage will be higher with caching enabled. Tim > > cheers, > Thorsten > > -- > Create and publish websites with WebMatrix > Use the most popular FREE web apps or write code yourself; > WebMatrix provides all the features you need to develop and > publish your website. http://p.sf.net/sfu/ms-webmatrix-sf > ___ > Flightgear-devel mailing list > Flightgear-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/flightgear-devel > -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching
On 03.04.2011 16:18, Torsten Dreyer wrote: > Me, too. I had a few coredumps with the 4-nvidia-cards, 8 monitors > (multithreading=automatic) setup since I had that enabled. I was not able to > backtrace and blame it to the CACHING-Option, however. So this might just be a > random correlation. In fact I haven't seen any crash for a very long time now (yay!). But I'm not using multi-threading, as that never worked for me. FG starts, but the scenery doesn't look right: the surface is all black and many models are only visible partially. Does this work for everyone else - maybe depends on newer OSG versions (>2.9.x)? > How about checking a /sim/rendering/cache property at startup? Good idea. I'll be adding this and enable it by default. Let's see if we get any user reports... (of course I'm thinking about the "usual" thank-you-messages from people being extremely happy with the improvements :) ). cheers, Thorsten -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
Re: [Flightgear-devel] OSG caching (was: Texture cache)
> I've also been using CACHE_ALL since then - not seeing any problems. But > I haven't checked memory consumption. So, what's the status about the > OSG caching options, should we enable these? Tim? Me, too. I had a few coredumps with the 4-nvidia-cards, 8 monitors (multithreading=automatic) setup since I had that enabled. I was not able to backtrace and blame it to the CACHING-Option, however. So this might just be a random correlation. How about checking a /sim/rendering/cache property at startup? Torsten -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
[Flightgear-devel] OSG caching (was: Texture cache)
>>> Maybe someone could do some tests when changing the setting >>> >> (SGPagedLOD.hxx:56) from "CACHE_NONE" to "CACHE_IMAGES" or even to >>> >> "CACHE_ALL" (then recompile/install sg+fg). Would be interesting to know >>> >> how this changed loading times, run-time fps and memory consumption. > After 30 minutes more of testing: It also works in practice. I have seen > no averse side effects, and the performance drop from loading new clouds > is now almost completely gone - the framerate drops I still see are mainly > associated with terrain sampling. > > The actual rendering performance is, as far as I can see, not changed, > i.e. once everything is there, it is there. But all in all this makes > things way smoother. It should be most pronounced on single CPU machines. I've also been using CACHE_ALL since then - not seeing any problems. But I haven't checked memory consumption. So, what's the status about the OSG caching options, should we enable these? Tim? cheers, Thorsten -- Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf ___ Flightgear-devel mailing list Flightgear-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/flightgear-devel
