Re: [osg-users] readFontFile possible bug
HI Sherman, Thanks for the log. Boy its hard to make sense of multi-threaded console warnings. A couple of the non interleaved parts are readable and tracable: read or simply that it is broken..d comes from: src/osgPlugins/freetype/FreeTypeLibrary.cpp - FreeTypeLibrary::getFace(..) method Which suggests the FT_New_Face call has failed, which in the past I've found is down to calls to FreeType being multi-threaded - and FreeType isn't thread safe. There are mutexes in osgText to which serialize access to the freetype, awkwardly it has to be done at a relatively high level in osgText as its not just individual calls to FreeType that aren't safe - freetype itself has a current state that must be maintained between associated calls. The error : Warning: Could not find plugin to read objects from file c:/system/earth/earth_L2_X0_Y1_subtile.ive Suggest to me that either the osgDB::Registry has got muddled or that the plugins themselves have. I don't see an obvious cause for this. Both font reading and 3d model reading does get channelled through osgDB::Registry but once the plugins are loaded the areas that might cause overlap and problems are mostly out of the way. Perhaps its just a case of one part failing then overwriting memory then knocking out other innocent modules. In this spirit might fixing the multi-threading issues with the freetype plugin fix the other issues? Right now this is the avenue I'd take. W.r.t fixing freetype I wonder if we need to change the Freetype plugin so that operations don't be it are done in operation blocks with these blocks being seariliazed by the freetype plugin. These operation blocks would do everything required to service the current needs of the calling class and remove the need for the freetype plugin being state sensitive like it now. I think the area of state that needs to be addressed is that to get a font face you first need to set the sizes required etc. then get the font face. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
Hi Sherman, I have tried out your modified osgtext example with the --mt 5 etc option along with a 8GB whole earth database and find the frame rate stalling, especially when zooming in to new high res regions. Running with on screen stats on suggests that its the GPU that is stalling things, with a single frame taking a very long time, then once its freed up the app runs freely once more with millisceond GPU times. I've put timing code into the UpdateTextOperation and neither the load nor update methods take very long most of the time, and there isn't a big hit here when the stall happen so it doesn't seem to be directly associated with the extra threads running. I haven't been able to establish the exact cause of stalls, but it looks to be an issue with the newly introduced subgraphs forcing compile of texture objects/display lists in such a way that the OpenGL driver/GPU is stalling. The texture sizes associated with osgText shouldn't be very large so the bandwidth and memory footprint shouldn't be a bottleneck, clearly something is happening down in the OpenGL fifo/GPU that is causing major problems and stalling things. One thing it doesn't look like is a threading issue, my guess running the UpdateTextOperation synchronously i.e. from the main loop would probably see the same big hits on the GPU, although perhaps doing it synchronously would regulate the amount of new data added in a subgraph per frame. Getting to the bottom of this issue will take digging down into draw dispatch, and probably adding things like GPU stats collection to the osgText::Text/Font rendering methods as well as osg::Texture texture subloading. I would suggest opening a different thread into investigations of these GPU stall issues, as it would seem so far the multi-threaded font crashes/bugs look resolved and this thread can be closed. Robert, On Dec 14, 2007 12:58 AM, sherman wilcox [EMAIL PROTECTED] wrote: OK, grabbed the latest copy of OSG out of SVN (latest as of about 2PM today) and ran my test app again. With the latest OSG the app doesn't crash, but there are still major issues. So, the multi-threading issue is still there. The database pager simply stops working, I get all manner of errors in the console window, etc. I've attached the test app. Should compile in linux, windows, etc. If you run this app, you will need a paged database to really stress things. I have one (about 148MB compressed) that I can provide if anyone is interested. Just email me and I'll provide a link to download. Oh, and be sure to pass the app an argument of --mt 5, where 5 is the number of test threads to launch, can be any reasonable number. On Dec 12, 2007 10:21 PM, sherman wilcox [EMAIL PROTECTED] wrote: I spent a bit of time this evening digging up the the code that reproduces the crash. I'll update to the SVN snapshot of OSG and give it a try. I was able to reproduce the crash on 2.2.0. On Dec 12, 2007 4:22 AM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Sherman, On Dec 11, 2007 11:31 PM, sherman wilcox [EMAIL PROTECTED] wrote: I can still reliably reproduce the multithreaded osgText / databasepager crash issue :) Could you explain how your recreate the crash, and the circumstances of the crash. It may be exactly the same problem as Serge has, but also just as easily could be another weakness that needs solving. Aaa you gotta love debugging multi-threaded apps remotely :-) Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
Robert, I'm not so sure the multi-threading issue is resolved. I wasn't getting simple GPU stalls. The database pager would stop working after several of these stalls. By stop working I don't mean interruptions with resumption. That database pager would not come back at all. Also, the log messages to screen was strange. Some were garbled messages and mentioned that it could no longer find plugins, etc. The inability to find plugins could explain the database pager, but the question then becomes why does running background threads dump plugins and trash log output? On Dec 14, 2007 5:25 AM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Sherman, I have tried out your modified osgtext example with the --mt 5 etc option along with a 8GB whole earth database and find the frame rate stalling, especially when zooming in to new high res regions. Running with on screen stats on suggests that its the GPU that is stalling things, with a single frame taking a very long time, then once its freed up the app runs freely once more with millisceond GPU times. I've put timing code into the UpdateTextOperation and neither the load nor update methods take very long most of the time, and there isn't a big hit here when the stall happen so it doesn't seem to be directly associated with the extra threads running. I haven't been able to establish the exact cause of stalls, but it looks to be an issue with the newly introduced subgraphs forcing compile of texture objects/display lists in such a way that the OpenGL driver/GPU is stalling. The texture sizes associated with osgText shouldn't be very large so the bandwidth and memory footprint shouldn't be a bottleneck, clearly something is happening down in the OpenGL fifo/GPU that is causing major problems and stalling things. One thing it doesn't look like is a threading issue, my guess running the UpdateTextOperation synchronously i.e. from the main loop would probably see the same big hits on the GPU, although perhaps doing it synchronously would regulate the amount of new data added in a subgraph per frame. Getting to the bottom of this issue will take digging down into draw dispatch, and probably adding things like GPU stats collection to the osgText::Text/Font rendering methods as well as osg::Texture texture subloading. I would suggest opening a different thread into investigations of these GPU stall issues, as it would seem so far the multi-threaded font crashes/bugs look resolved and this thread can be closed. Robert, On Dec 14, 2007 12:58 AM, sherman wilcox [EMAIL PROTECTED] wrote: OK, grabbed the latest copy of OSG out of SVN (latest as of about 2PM today) and ran my test app again. With the latest OSG the app doesn't crash, but there are still major issues. So, the multi-threading issue is still there. The database pager simply stops working, I get all manner of errors in the console window, etc. I've attached the test app. Should compile in linux, windows, etc. If you run this app, you will need a paged database to really stress things. I have one (about 148MB compressed) that I can provide if anyone is interested. Just email me and I'll provide a link to download. Oh, and be sure to pass the app an argument of --mt 5, where 5 is the number of test threads to launch, can be any reasonable number. On Dec 12, 2007 10:21 PM, sherman wilcox [EMAIL PROTECTED] wrote: I spent a bit of time this evening digging up the the code that reproduces the crash. I'll update to the SVN snapshot of OSG and give it a try. I was able to reproduce the crash on 2.2.0. On Dec 12, 2007 4:22 AM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Sherman, On Dec 11, 2007 11:31 PM, sherman wilcox [EMAIL PROTECTED] wrote: I can still reliably reproduce the multithreaded osgText / databasepager crash issue :) Could you explain how your recreate the crash, and the circumstances of the crash. It may be exactly the same problem as Serge has, but also just as easily could be another weakness that needs solving. Aaa you gotta love debugging multi-threaded apps remotely :-) Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
I performed my tests using the SVN trunk of OSG (as about 2 hours ago) Here are some of the log messages: Warning: Could not find plugin to read objects from file c:/system/earth/earth_L2_X1_Y0_subtile.ive. Error r e.a.d.i.n ga nfoitlhee:r DeartraoIrn pcuotdSet rmeanesa mt:h:arte atdhUeI nfto(n)t: fiFlaei lceodu ltdo nroet add unsign ed in t. .v..a lbuee . opened, read or simply that it is broken..d Warning: Could not find plugin to read objects from file c:/system/earth/earth_L2_X0_Y1_subtile.ive. WarnEinrgr:o rC oruelda dniontg ffiinlde :p lDuagtianI ntpou trSetarde aomb:j:ercetasd UfIrnotm( )f:i lFea ilCe:d\ tWo irnedaodw su\Fonnstisg\nteidme s.itnttf v.alue . Warning: Could not find plugin to read objects from file c:/system/earth/earth_L3_X2_Y2_subtile.ive. Error reading file: D.a.t.a.I napnuottShterre aemr:r:orre acdoUdIen tm(e)a:n sF atihlaetd tthoe rfeoandt ufnislieg n ceodu lidn tn ovtadlue. be opened, read or simply that it is broken..d Warning: Could not find plugin to read objects from file c:/system/earth/earth_L2_X0_Y1_subtile.ive. WarniEnrgr:o rC orueladd innogt ffiilned: pDlautgaiInnp uttoS trreeaadm :o:brjeeacdtUsI nftr(o)m: fFialiel edC :t\oW rienaddo wusn\sFiognntesd\ tiinmte sv.atltufe.. Warning: Could not find plugin to read objects from file c:/system/earth/earth_L3_X2_Y2_subtile.ive. E.r.r..o ra nroetahdeirn ge rfriolre :co dDea tmaeIannpsu ttShtarte atmh:e: rfeoandtU Ifnitl(e) : cFoauillde dn ottod r ead unsigned i n.t. .v.a lbuee .o pened, read or simply that it is broken..d Warning: Could not find plugin to read objects from file c:/system/earth/earth_L2_X1_Y0_subtile.ive. WarniEnrgr:o rC oreualddi nngo tf iflien:d DpaltuagIinn ptutoS rteraeda mo:b:jreecatdsU Ifnrto(m) :f iFlaei leCd: \tWo i nrdeowasd\ Fuonnstisg\nteidm eisn.tt tvfal.ue. Warning: Could not find plugin to read objects from file c:/system/earth/earth_L2_X0_Y1_subtile.ive. Error readin g. .f.i.l ea:n oDtahtearI neprurtoSrt rceoadme: :meraenasd UtIhnatt( )t:h eF afiolnetd ftiol er ecaodu lud n sniotgdn ed int v a.l.u.e. be o.pen ed, read or simply that it is broken..d Warning: Could not find plugin to read objects from file c:/system/earth/earth_L3_X2_Y2_subtile.ive. WarnEirnrg:o rC orueladd innogt ffiilned: p lDuagtianI ntpou trSead torbejaemc:t:sr efarodmU Ifnitl(e) : FCa:i\lWeidn dtowos \rFeoadn tusn\stiigmneesd. titnft .va lue. On Dec 14, 2007 2:13 PM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Sherman, On Dec 14, 2007 3:15 PM, sherman wilcox [EMAIL PROTECTED] wrote: Robert, I'm not so sure the multi-threading issue is resolved. I wasn't getting simple GPU stalls. The database pager would stop working after several of these stalls. By stop working I don't mean interruptions with resumption. That database pager would not come back at all. Also, the log messages to screen was strange. Some were garbled messages and mentioned that it could no longer find plugins, etc. The inability to find plugins could explain the database pager, but the question then becomes why does running background threads dump plugins and trash log output? OK. This does sounds like more than just a plain stall, all I was able to do was recreate the stall. Could you capture the error messages and post them? As for a thread dumping plugins etc. This suggest to be that the Registry is getting corrupted for some reason, but who knows, this is all pretty new in style of problem than we've seen before. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
OK, grabbed the latest copy of OSG out of SVN (latest as of about 2PM
today) and ran my test app again. With the latest OSG the app doesn't
crash, but there are still major issues. So, the multi-threading issue
is still there. The database pager simply stops working, I get all
manner of errors in the console window, etc.
I've attached the test app. Should compile in linux, windows, etc.
If you run this app, you will need a paged database to really stress
things. I have one (about 148MB compressed) that I can provide if
anyone is interested. Just email me and I'll provide a link to
download. Oh, and be sure to pass the app an argument of --mt 5, where
5 is the number of test threads to launch, can be any reasonable
number.
On Dec 12, 2007 10:21 PM, sherman wilcox [EMAIL PROTECTED] wrote:
I spent a bit of time this evening digging up the the code that
reproduces the crash. I'll update to the SVN snapshot of OSG and give
it a try. I was able to reproduce the crash on 2.2.0.
On Dec 12, 2007 4:22 AM, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Sherman,
On Dec 11, 2007 11:31 PM, sherman wilcox [EMAIL PROTECTED] wrote:
I can still reliably reproduce the multithreaded osgText /
databasepager crash issue :)
Could you explain how your recreate the crash, and the circumstances
of the crash. It may be exactly the same problem as Serge has, but
also just as easily could be another weakness that needs solving.
Aaa you gotta love debugging multi-threaded apps remotely :-)
Robert.
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
/* OpenSceneGraph example, osgtext.
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the Software), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
* THE SOFTWARE.
*/
#include osgUtil/Optimizer
#include osgDB/ReadFile
#include osgDB/WriteFile
#include osgDB/Registry
#include osgViewer/Viewer
#include osgViewer/ViewerEventHandlers
#include osg/Geode
#include osg/Camera
#include osg/ShapeDrawable
#include osg/Sequence
#include osg/PolygonMode
#include osg/io_utils
#include osgText/Font
#include osgText/Text
osg::Group* createHUDText()
{
osg::Group* rootNode = new osg::Group;
osgText::Font* font = osgText::readFontFile(fonts/arial.ttf);
osg::Geode* geode = new osg::Geode;
rootNode-addChild(geode);
float windowHeight = 1024.0f;
float windowWidth = 1280.0f;
float margin = 50.0f;
//
// Examples of how to set up different text layout
//
osg::Vec4 layoutColor(1.0f,1.0f,0.0f,1.0f);
float layoutCharacterSize = 20.0f;
{
osgText::Text* text = new osgText::Text;
text-setFont(font);
text-setColor(layoutColor);
text-setCharacterSize(layoutCharacterSize);
text-setPosition(osg::Vec3(margin,windowHeight-margin,0.0f));
// the default layout is left to right, typically used in languages
// originating from europe such as English, French, German, Spanish etc..
text-setLayout(osgText::Text::LEFT_TO_RIGHT);
text-setText(text-setLayout(osgText::Text::LEFT_TO_RIGHT););
geode-addDrawable(text);
}
{
osgText::Text* text = new osgText::Text;
text-setFont(font);
text-setColor(layoutColor);
text-setCharacterSize(layoutCharacterSize);
text-setPosition(osg::Vec3(windowWidth-margin,windowHeight-margin,0.0f));
// right to left layouts would be used for hebrew or arabic fonts.
text-setLayout(osgText::Text::RIGHT_TO_LEFT);
text-setAlignment(osgText::Text::RIGHT_BASE_LINE);
text-setText(text-setLayout(osgText::Text::RIGHT_TO_LEFT););
geode-addDrawable(text);
}
{
osgText::Text* text = new osgText::Text;
text-setFont(font);
text-setColor(layoutColor);
text-setPosition(osg::Vec3(margin,windowHeight-margin,0.0f));
text-setCharacterSize(layoutCharacterSize);
// vertical font layout would be used for asian fonts.
text-setLayout(osgText::Text::VERTICAL);
Re: [osg-users] readFontFile possible bug
Hi Robert, On Dec 11, 2007 8:59 PM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Serge, How reliably can you recreate the crash? Does it happen with osgviewer or other OSG examples? Could you explain whats happening in your app before the crash. Here is the context of the crash : DatabasePager configuration : pager-setExpiryDelay(5); pager-setUnrefImageDataAfterApplyPolicy(true, true); pager-setDeleteRemovedSubgraphsInDatabaseThread(false); pager-setMaximumNumOfRemovedChildPagedLODs(50); pager-setMinimumNumOfInactivePagedLODs(0); Registry configuration : osgDB::Registry::instance()-getOptions()-setObjectCacheHint(osgDB::ReaderWriter::Options::CACHE_ALL); To make it crash, I have a custom manipulator which randomely move into a very high paged database (lots of giga). The database contains nodes with simple geometries but also texts. Sometime it crashs after hours of randomely moves, sometime after just 1 or 2 minutes. The crash happens in two locations : 1 - into osgText::Text::setFont (Text.cpp line 119) because the variable font is a corrupted pointer. 2 - into databasepager.cpp (line 604), during the initialization of databaseRequest-_loadedModel, because the pointer returned by readNodeFile is corrupted. These crashs happen into the pager thread, and the main thread is always into the removeExpiredSubgraphs method at this moment. In both cases, we have a very short laps of time where a C pointer is used for an object instead of a ref_ptr. For the crash number 2, it seems that switching readNodeFile by a readNode fix it. But I think that the better solution will be to prevent any cache clear during the readNodeFile call into databasepager.cpp, I will investigate further this morning and see if I can fix it. If we can work out the conditions that the crash happens we'll have a much better chance of isolation. It could be as your describe an object be deleted just when another thread is about to take a ref to it, I would have thought this would be a pretty rare condition though. Robert. On Dec 11, 2007 3:55 PM, Serge Lages [EMAIL PROTECTED] wrote: I have another theory about the crash, let's say that : osg::Object* object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); read the object from the cache, and between this moment and the one where the font object is set to a ref_ptr into setFont, the cache is cleared (into another thread). With this scenario the pointer points to a deleted object or is there anything to prevent it ? In my case this scenario is possible as I load my IVE with the databasepager and that I remove the expired objects into the main thread. -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
Hi Sherman, On Dec 11, 2007 11:31 PM, sherman wilcox [EMAIL PROTECTED] wrote: I can still reliably reproduce the multithreaded osgText / databasepager crash issue :) Could you explain how your recreate the crash, and the circumstances of the crash. It may be exactly the same problem as Serge has, but also just as easily could be another weakness that needs solving. Aaa you gotta love debugging multi-threaded apps remotely :-) Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
Hi Serge, Thanks for the explanation, this really does help. You suggest change to DatabasePager.cpp to use ReadResult rather than osgDB::readNodeFile() is an interesting one. While the getting the timing to reproduce the crash seems pretty rare it is certainly possible, but often any little opening for a multi-threading crash can come out and bite, even if its seems really unlikely. My current though is the weakness you've highlights is that osgDB::readNodeFile() doesn't return a ref_ptr, opening the door to the object being unref'd at the same time as the C pointer is be passed back from readNodeFile. Changing readNodeFile to pass back a ref_ptr is a possibility, but it'd break a lot of user code - this might be the right thing to do in terms of writing robust C++, but it is a heavy hit to force on users that don't code stuff in ways that would be sensitive to this issue. Perhaps a readRefNodeFile that passes back a ref_ptr would be a workaround. I guess one might be able to write a little class to work as adapter too so one could pass this object back and it work for C* as well as ref_ptr but this would be a pretty obscure use of C++. Thoughts? Robert. On Dec 12, 2007 9:28 AM, Serge Lages [EMAIL PROTECTED] wrote: Hi Robert, On Dec 11, 2007 8:59 PM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Serge, How reliably can you recreate the crash? Does it happen with osgviewer or other OSG examples? Could you explain whats happening in your app before the crash. Here is the context of the crash : DatabasePager configuration : pager-setExpiryDelay(5); pager-setUnrefImageDataAfterApplyPolicy(true, true); pager-setDeleteRemovedSubgraphsInDatabaseThread(false); pager-setMaximumNumOfRemovedChildPagedLODs(50); pager-setMinimumNumOfInactivePagedLODs(0); Registry configuration : osgDB::Registry::instance()-getOptions()-setObjectCacheHint(osgDB::ReaderWriter::Options::CACHE_ALL); To make it crash, I have a custom manipulator which randomely move into a very high paged database (lots of giga). The database contains nodes with simple geometries but also texts. Sometime it crashs after hours of randomely moves, sometime after just 1 or 2 minutes. The crash happens in two locations : 1 - into osgText::Text::setFont (Text.cpp line 119) because the variable font is a corrupted pointer. 2 - into databasepager.cpp (line 604), during the initialization of databaseRequest-_loadedModel, because the pointer returned by readNodeFile is corrupted. These crashs happen into the pager thread, and the main thread is always into the removeExpiredSubgraphs method at this moment. In both cases, we have a very short laps of time where a C pointer is used for an object instead of a ref_ptr. For the crash number 2, it seems that switching readNodeFile by a readNode fix it. But I think that the better solution will be to prevent any cache clear during the readNodeFile call into databasepager.cpp, I will investigate further this morning and see if I can fix it. If we can work out the conditions that the crash happens we'll have a much better chance of isolation. It could be as your describe an object be deleted just when another thread is about to take a ref to it, I would have thought this would be a pretty rare condition though. Robert. On Dec 11, 2007 3:55 PM, Serge Lages [EMAIL PROTECTED] wrote: I have another theory about the crash, let's say that : osg::Object* object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); read the object from the cache, and between this moment and the one where the font object is set to a ref_ptr into setFont, the cache is cleared (into another thread). With this scenario the pointer points to a deleted object or is there anything to prevent it ? In my case this scenario is possible as I load my IVE with the databasepager and that I remove the expired objects into the main thread. -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
Hi Serge, On Dec 12, 2007 11:08 AM, Serge Lages [EMAIL PROTECTED] wrote: The readRefNodeFile seems a good option, moving the criticals readNodeFile to it. The osgText part will need a little bit more of refactoring, moving all the part where a C pointer is passed to a ref_ptr. If you want I can do the changes this afternoon if we decide to go this way. :) A volunteer, great :-) This would really help, as you are able to reproduce the crash, doing these changes will hopefully allow you to test things directly. Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
On Dec 12, 2007 11:34 AM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Serge, Thanks for the explanation, this really does help. You suggest change to DatabasePager.cpp to use ReadResult rather than osgDB::readNodeFile() is an interesting one. While the getting the timing to reproduce the crash seems pretty rare it is certainly possible, but often any little opening for a multi-threading crash can come out and bite, even if its seems really unlikely. My current though is the weakness you've highlights is that osgDB::readNodeFile() doesn't return a ref_ptr, opening the door to the object being unref'd at the same time as the C pointer is be passed back from readNodeFile. Changing readNodeFile to pass back a ref_ptr is a possibility, but it'd break a lot of user code - this might be the right thing to do in terms of writing robust C++, but it is a heavy hit to force on users that don't code stuff in ways that would be sensitive to this issue. Perhaps a readRefNodeFile that passes back a ref_ptr would be a workaround. I guess one might be able to write a little class to work as adapter too so one could pass this object back and it work for C* as well as ref_ptr but this would be a pretty obscure use of C++. Thoughts? The readRefNodeFile seems a good option, moving the criticals readNodeFile to it. The osgText part will need a little bit more of refactoring, moving all the part where a C pointer is passed to a ref_ptr. If you want I can do the changes this afternoon if we decide to go this way. :) -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
I spent a bit of time this evening digging up the the code that reproduces the crash. I'll update to the SVN snapshot of OSG and give it a try. I was able to reproduce the crash on 2.2.0. On Dec 12, 2007 4:22 AM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Sherman, On Dec 11, 2007 11:31 PM, sherman wilcox [EMAIL PROTECTED] wrote: I can still reliably reproduce the multithreaded osgText / databasepager crash issue :) Could you explain how your recreate the crash, and the circumstances of the crash. It may be exactly the same problem as Serge has, but also just as easily could be another weakness that needs solving. Aaa you gotta love debugging multi-threaded apps remotely :-) Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
Hi Serge, Creating objects with and storing them locally with a C pointer rather than a ref_ptr is usually just fine if the pointer just lives in local scope, but there are gotchas lurking if your do certain operations. Most places where a C pointer is used within the OSG code base is because its fine to do in that context. Problems typically occur when you try to use a C pointer as global. Your own intermittent crash is more likely to be something like a thread sensitivity. Is your app multi-thread? In what ways? What is the nature of the database setup? Robert. On Dec 11, 2007 1:33 PM, Serge Lages [EMAIL PROTECTED] wrote: Hi all, I am currently having a random crash with osgText, here is the context : I read some IVE files with texts inside, when loading the text it crashs into osgText::Text::setFont (Text.cpp line 119) because the variable font is a corrupted pointer (not setted to NULL, but typically a variable already deleted). My investigations bring me to osgText::readFontFile into the file Font.cpp. Line 110 we can see : osg::Object* object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); I always thought that doing this may lead to a crash, and that we must do : osg::ref_ptrosg::Object object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); To be sure that readObjectFile will not delete the object by deleting its ReaderWriter::ReadResult. And line 117, the test object-referenceCount()==0 makes me also perplex, if the object's reference count is set to 0, shouldn't it be already deleted ? My crash is really a random one, but always in the same place, that's why I think that sometime the font object is deleted before being assigned to a ref_ptr into setFont. -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
I have discovered the takeObject method, and now I understand better why the object is not deleted, But it doesn't explain my crash... :/ On Dec 11, 2007 2:33 PM, Serge Lages [EMAIL PROTECTED] wrote: Hi all, I am currently having a random crash with osgText, here is the context : I read some IVE files with texts inside, when loading the text it crashs into osgText::Text::setFont (Text.cpp line 119) because the variable font is a corrupted pointer (not setted to NULL, but typically a variable already deleted). My investigations bring me to osgText::readFontFile into the file Font.cpp. Line 110 we can see : osg::Object* object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); I always thought that doing this may lead to a crash, and that we must do : osg::ref_ptrosg::Object object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); To be sure that readObjectFile will not delete the object by deleting its ReaderWriter::ReadResult. And line 117, the test object-referenceCount()==0 makes me also perplex, if the object's reference count is set to 0, shouldn't it be already deleted ? My crash is really a random one, but always in the same place, that's why I think that sometime the font object is deleted before being assigned to a ref_ptr into setFont. -- Serge Lages http://www.tharsis-software.com -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
I have another theory about the crash, let's say that : osg::Object* object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); read the object from the cache, and between this moment and the one where the font object is set to a ref_ptr into setFont, the cache is cleared (into another thread). With this scenario the pointer points to a deleted object or is there anything to prevent it ? In my case this scenario is possible as I load my IVE with the databasepager and that I remove the expired objects into the main thread. -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
Hi Robert,
Look back my last message where I emmit a possible scenario. It seems that
this scenario is happening because when it crashs, the main thread is into
the removeExpiredSubgraphs method.
And I am having the same problem with the readNodeFile call into
databasepager.cpp (line 604), it seems that between the moment where
takeNode is called and the pointer becomes nude and the moment where it is
assigned again to a ref_ptr my app can crash. It is again because of a
cache's clear at the same time into the main thread.
I have remplaced this call by :
ReaderWriter::ReadResult rr =
Registry::instance()-readNode(databaseRequest-_fileName,
databaseRequest-_loadOptions.get());
if (rr.success())
{
databaseRequest-_loadedModel = rr.getNode();
}
I am currently testing but it seems to not crash anymore.
Another possibility (which can also resolve the osgText crash), is to block
the access to removeExpiredSubgraphs during the nodes loadings (only when
_deleteRemovedSubgraphsInDatabaseThread is set to false).
On Dec 11, 2007 3:49 PM, Robert Osfield [EMAIL PROTECTED] wrote:
Hi Serge,
Creating objects with and storing them locally with a C pointer rather
than a ref_ptr is usually just fine if the pointer just lives in
local scope, but there are gotchas lurking if your do certain
operations. Most places where a C pointer is used within the OSG code
base is because its fine to do in that context. Problems typically
occur when you try to use a C pointer as global.
Your own intermittent crash is more likely to be something like a
thread sensitivity. Is your app multi-thread? In what ways? What is
the nature of the database setup?
Robert.
On Dec 11, 2007 1:33 PM, Serge Lages [EMAIL PROTECTED] wrote:
Hi all,
I am currently having a random crash with osgText, here is the context :
I
read some IVE files with texts inside, when loading the text it crashs
into
osgText::Text::setFont (Text.cpp line 119) because the variable font
is a
corrupted pointer (not setted to NULL, but typically a variable already
deleted).
My investigations bring me to osgText::readFontFile into the file
Font.cpp.
Line 110 we can see :
osg::Object* object = osgDB::readObjectFile(foundFile, userOptions ?
userOptions : localOptions.get());
I always thought that doing this may lead to a crash, and that we must
do :
osg::ref_ptrosg::Object object = osgDB::readObjectFile(foundFile,
userOptions ? userOptions : localOptions.get());
To be sure that readObjectFile will not delete the object by deleting
its
ReaderWriter::ReadResult.
And line 117, the test object-referenceCount()==0 makes me also
perplex, if
the object's reference count is set to 0, shouldn't it be already
deleted ?
My crash is really a random one, but always in the same place, that's
why I
think that sometime the font object is deleted before being assigned to
a
ref_ptr into setFont.
--
Serge Lages
http://www.tharsis-software.com
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
--
Serge Lages
http://www.tharsis-software.com
___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
Hi Serge, How reliably can you recreate the crash? Does it happen with osgviewer or other OSG examples? Could you explain whats happening in your app before the crash. If we can work out the conditions that the crash happens we'll have a much better chance of isolation. It could be as your describe an object be deleted just when another thread is about to take a ref to it, I would have thought this would be a pretty rare condition though. Robert. On Dec 11, 2007 3:55 PM, Serge Lages [EMAIL PROTECTED] wrote: I have another theory about the crash, let's say that : osg::Object* object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); read the object from the cache, and between this moment and the one where the font object is set to a ref_ptr into setFont, the cache is cleared (into another thread). With this scenario the pointer points to a deleted object or is there anything to prevent it ? In my case this scenario is possible as I load my IVE with the databasepager and that I remove the expired objects into the main thread. -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] readFontFile possible bug
I can still reliably reproduce the multithreaded osgText / databasepager crash issue :) On Dec 11, 2007 1:59 PM, Robert Osfield [EMAIL PROTECTED] wrote: Hi Serge, How reliably can you recreate the crash? Does it happen with osgviewer or other OSG examples? Could you explain whats happening in your app before the crash. If we can work out the conditions that the crash happens we'll have a much better chance of isolation. It could be as your describe an object be deleted just when another thread is about to take a ref to it, I would have thought this would be a pretty rare condition though. Robert. On Dec 11, 2007 3:55 PM, Serge Lages [EMAIL PROTECTED] wrote: I have another theory about the crash, let's say that : osg::Object* object = osgDB::readObjectFile(foundFile, userOptions ? userOptions : localOptions.get()); read the object from the cache, and between this moment and the one where the font object is set to a ref_ptr into setFont, the cache is cleared (into another thread). With this scenario the pointer points to a deleted object or is there anything to prevent it ? In my case this scenario is possible as I load my IVE with the databasepager and that I remove the expired objects into the main thread. -- Serge Lages http://www.tharsis-software.com ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
