Re: [osg-users] osg::notify crash, not thread safe
HI David, I have compiled your test code with the attached CmakeLists.txt and can confirm I get the crash on my Kubuntu 13.04 + 64bit system. With running gdb: (gdb) run 48 100 2 2 The stack track I'm getting is: *** Error in `/home/robert/Contributors/DavidFries/threadednotify': double free or corruption (top): 0x7fffd8000b40 *** *** Error in `/home/robert/Contributors/DavidFries/threadednotify[New Thread 0x7fffe0ff9700 (LWP 6024)] Program received signal SIGABRT, Aborted. [Switching to Thread 0x70ee7700 (LWP 6017)] 0x76dee037 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) where #0 0x76dee037 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x76df1698 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x76e2b5ab in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x76e37a46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x7742f2a9 in std::basic_stringbuf, std::allocator >::overflow(int) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x77433976 in std::basic_streambuf >::xsputn(char const*, long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x77426b83 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x77427dc4 in std::ostreambuf_iterator > std::num_put > >::_M_insert_int(std::ostreambuf_iterator >, std::ios_base&, char, long) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #8 0x77427f2d in std::num_put > >::do_put(std::ostreambuf_iterator >, std::ios_base&, char, long) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #9 0x7742adf6 in std::ostream& std::ostream::_M_insert(long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #10 0x0040299a in Spam::run (this=0x60d9b8) at /home/robert/Contributors/DavidFries/ThreadedNotify.cpp:88 #11 0x7769c7d0 in OpenThreads::ThreadPrivateActions::StartThread(void*) () from /home/robert/OpenSceneGraph/lib/libOpenThreads.so.14 #12 0x76ba1f8e in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #13 0x76eb0e1d in clone () from /lib/x86_64-linux-gnu/libc.so.6 This suggests the ostream being shared between threads is not thread safe. I don't believe it's the Notify.cpp code itself that is directly causing the problem, rather it looks to be that ostream implementation is the problem. Solving may not be easy without having a ostream per thread. Thoughts? Robert. ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] osg::notify crash, not thread safe
On Mon, Aug 05, 2013 at 04:34:17PM +, Robert Osfield wrote: > HI David, > > I have my brain focused on some big topics today without much scope > with thinking other complicated areas so won't dive into this one > right now. > > I recall a discussion about thread safety of Notify from last year > with a user who was heavily using notification in a multi-threaded > environement and found issues like yourself, I don't know if they are > the same though,I don't recall the details. He may of submitted > changes or proposed some, but if they had been straight forward and > without a performance compromise I would have merged them. It might > be searching the archives on osg-users and osg-submissions to look at > the discussions, it might help. I did merge some changes w.r.t > threading though: Date: Nov 28 06:08:02 2011 From: George Bekos Subject: [osg-users] Custom osg::NotifyHandler problems and multi-threading It sounded like the same issue, no changes were posted on the thread. It sounded to me to show an example that reproduced it on Linux which is what I posted. For now we'll just make sure are logging levels are set low so we don't have much chance of hitting it. > r13094 | robert | 2012-06-22 16:21:08 + (Fri, 22 Jun 2012) | 2 lines > > Restructed the way that the global notify variables are initialized to > avoid problems with multi-threaded initialization of these variables. > > This is at start up though. As for the assumptions that cerr and cout > are thread safe, normally I'd stick with this assumption, but I'm not > sure it's always the case across all compilers etc. > > Robert. > > On 5 August 2013 16:10, David Fries wrote: > > Hi Robert, > > > > I tested against the svn 3.2 and trunk branches today on Linux and get > > the same results with the test program I posted. > > > > ./ThreadedNotify 48 1000 2 2 > > Segmentation fault > > ./ThreadedNotify 48 1000 2 2 > > *** glibc detected *** ./ThreadedNotify: free(): invalid pointer: > > 0x00d81500 *** > > Segmentation fault > > ./ThreadedNotify 48 1000 2 2 > > *** glibc detected *** ./ThreadedNotify: double free or corruption (!prev): > > 0x0189b500 *** > > Segmentation fault > > > > On Sun, Aug 04, 2013 at 11:48:56AM +, Robert Osfield wrote: > >> Hi David, > >> > >> There were various changes to osg::Notify during the 3.1.x dev cycle. > >> Could you try out OSG-3.2 and see how you get on. > >> > >> Robert. > >> > >> On 2 August 2013 22:54, David Fries wrote: > >> > I spent a few days tracking down a crash I was having generally when a > >> > terrapage terrain was paging. This is using OSG 3.1, when I finally > >> > caught the corruption I found that both the DatabasePager thread and > >> > the Draw thread were calling OSG_NOTIFY. > >> > > >> > I wrote up a test that does little more than start up a bunch of > >> > threads and print messages and found it too crashes. I am testing on > >> > both Linux x86-64 gcc 4.7.2, and Windows 7 Visual Studio 10 and both > >> > crash, though it crashes much quicker on Windows. > >> > > >> > It will crash pretty quickly with the default (no arguments) on Windows. > >> > > >> > The following arguments are working for me to crash on Linux on a 12 > >> > core (24 with hyperthreading), x86-64 system. > >> > ./ThreadedNotify 48 1000 2 2 > >> > > >> > It has been pointed out that cout/cerr are thread safe, so it should > >> > be possible to make osg::notify thread safe as well. In this program > >> > it helps when I reduce the notification level, at least with the low > >> > number of messages being printed it seems to be unlikely. > >> > > >> > -- > >> > David Fries PGP pub CB1EE8F0 > >> > http://fries.net/~david/ > >> > > >> > ___ > >> > osg-users mailing list > >> > osg-users@lists.openscenegraph.org > >> > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org > >> > > >> ___ > >> osg-users mailing list > >> osg-users@lists.openscenegraph.org > >> http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org > > > > -- > > David Fries PGP pub CB1EE8F0 > > http://fries.net/~david/ > > ___ > > osg-users mailing list > > osg-users@lists.openscenegraph.org > > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org > ___ > osg-users mailing list > osg-users@lists.openscenegraph.org > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org -- David Fries PGP pub CB1EE8F0 http://fries.net/~david/ ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] osg::notify crash, not thread safe
HI David, I have my brain focused on some big topics today without much scope with thinking other complicated areas so won't dive into this one right now. I recall a discussion about thread safety of Notify from last year with a user who was heavily using notification in a multi-threaded environement and found issues like yourself, I don't know if they are the same though,I don't recall the details. He may of submitted changes or proposed some, but if they had been straight forward and without a performance compromise I would have merged them. It might be searching the archives on osg-users and osg-submissions to look at the discussions, it might help. I did merge some changes w.r.t threading though: r13094 | robert | 2012-06-22 16:21:08 + (Fri, 22 Jun 2012) | 2 lines Restructed the way that the global notify variables are initialized to avoid problems with multi-threaded initialization of these variables. This is at start up though. As for the assumptions that cerr and cout are thread safe, normally I'd stick with this assumption, but I'm not sure it's always the case across all compilers etc. Robert. On 5 August 2013 16:10, David Fries wrote: > Hi Robert, > > I tested against the svn 3.2 and trunk branches today on Linux and get > the same results with the test program I posted. > > ./ThreadedNotify 48 1000 2 2 > Segmentation fault > ./ThreadedNotify 48 1000 2 2 > *** glibc detected *** ./ThreadedNotify: free(): invalid pointer: > 0x00d81500 *** > Segmentation fault > ./ThreadedNotify 48 1000 2 2 > *** glibc detected *** ./ThreadedNotify: double free or corruption (!prev): > 0x0189b500 *** > Segmentation fault > > On Sun, Aug 04, 2013 at 11:48:56AM +, Robert Osfield wrote: >> Hi David, >> >> There were various changes to osg::Notify during the 3.1.x dev cycle. >> Could you try out OSG-3.2 and see how you get on. >> >> Robert. >> >> On 2 August 2013 22:54, David Fries wrote: >> > I spent a few days tracking down a crash I was having generally when a >> > terrapage terrain was paging. This is using OSG 3.1, when I finally >> > caught the corruption I found that both the DatabasePager thread and >> > the Draw thread were calling OSG_NOTIFY. >> > >> > I wrote up a test that does little more than start up a bunch of >> > threads and print messages and found it too crashes. I am testing on >> > both Linux x86-64 gcc 4.7.2, and Windows 7 Visual Studio 10 and both >> > crash, though it crashes much quicker on Windows. >> > >> > It will crash pretty quickly with the default (no arguments) on Windows. >> > >> > The following arguments are working for me to crash on Linux on a 12 >> > core (24 with hyperthreading), x86-64 system. >> > ./ThreadedNotify 48 1000 2 2 >> > >> > It has been pointed out that cout/cerr are thread safe, so it should >> > be possible to make osg::notify thread safe as well. In this program >> > it helps when I reduce the notification level, at least with the low >> > number of messages being printed it seems to be unlikely. >> > >> > -- >> > David Fries PGP pub CB1EE8F0 >> > http://fries.net/~david/ >> > >> > ___ >> > osg-users mailing list >> > osg-users@lists.openscenegraph.org >> > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org >> > >> ___ >> osg-users mailing list >> osg-users@lists.openscenegraph.org >> http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org > > -- > David Fries PGP pub CB1EE8F0 > http://fries.net/~david/ > ___ > osg-users mailing list > osg-users@lists.openscenegraph.org > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] osg::notify crash, not thread safe
Hi Robert, I tested against the svn 3.2 and trunk branches today on Linux and get the same results with the test program I posted. ./ThreadedNotify 48 1000 2 2 Segmentation fault ./ThreadedNotify 48 1000 2 2 *** glibc detected *** ./ThreadedNotify: free(): invalid pointer: 0x00d81500 *** Segmentation fault ./ThreadedNotify 48 1000 2 2 *** glibc detected *** ./ThreadedNotify: double free or corruption (!prev): 0x0189b500 *** Segmentation fault On Sun, Aug 04, 2013 at 11:48:56AM +, Robert Osfield wrote: > Hi David, > > There were various changes to osg::Notify during the 3.1.x dev cycle. > Could you try out OSG-3.2 and see how you get on. > > Robert. > > On 2 August 2013 22:54, David Fries wrote: > > I spent a few days tracking down a crash I was having generally when a > > terrapage terrain was paging. This is using OSG 3.1, when I finally > > caught the corruption I found that both the DatabasePager thread and > > the Draw thread were calling OSG_NOTIFY. > > > > I wrote up a test that does little more than start up a bunch of > > threads and print messages and found it too crashes. I am testing on > > both Linux x86-64 gcc 4.7.2, and Windows 7 Visual Studio 10 and both > > crash, though it crashes much quicker on Windows. > > > > It will crash pretty quickly with the default (no arguments) on Windows. > > > > The following arguments are working for me to crash on Linux on a 12 > > core (24 with hyperthreading), x86-64 system. > > ./ThreadedNotify 48 1000 2 2 > > > > It has been pointed out that cout/cerr are thread safe, so it should > > be possible to make osg::notify thread safe as well. In this program > > it helps when I reduce the notification level, at least with the low > > number of messages being printed it seems to be unlikely. > > > > -- > > David Fries PGP pub CB1EE8F0 > > http://fries.net/~david/ > > > > ___ > > osg-users mailing list > > osg-users@lists.openscenegraph.org > > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org > > > ___ > osg-users mailing list > osg-users@lists.openscenegraph.org > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org -- David Fries PGP pub CB1EE8F0 http://fries.net/~david/ ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
Re: [osg-users] osg::notify crash, not thread safe
Hi David, There were various changes to osg::Notify during the 3.1.x dev cycle. Could you try out OSG-3.2 and see how you get on. Robert. On 2 August 2013 22:54, David Fries wrote: > I spent a few days tracking down a crash I was having generally when a > terrapage terrain was paging. This is using OSG 3.1, when I finally > caught the corruption I found that both the DatabasePager thread and > the Draw thread were calling OSG_NOTIFY. > > I wrote up a test that does little more than start up a bunch of > threads and print messages and found it too crashes. I am testing on > both Linux x86-64 gcc 4.7.2, and Windows 7 Visual Studio 10 and both > crash, though it crashes much quicker on Windows. > > It will crash pretty quickly with the default (no arguments) on Windows. > > The following arguments are working for me to crash on Linux on a 12 > core (24 with hyperthreading), x86-64 system. > ./ThreadedNotify 48 1000 2 2 > > It has been pointed out that cout/cerr are thread safe, so it should > be possible to make osg::notify thread safe as well. In this program > it helps when I reduce the notification level, at least with the low > number of messages being printed it seems to be unlikely. > > -- > David Fries PGP pub CB1EE8F0 > http://fries.net/~david/ > > ___ > osg-users mailing list > osg-users@lists.openscenegraph.org > http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org > ___ osg-users mailing list osg-users@lists.openscenegraph.org http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
[osg-users] osg::notify crash, not thread safe
I spent a few days tracking down a crash I was having generally when a
terrapage terrain was paging. This is using OSG 3.1, when I finally
caught the corruption I found that both the DatabasePager thread and
the Draw thread were calling OSG_NOTIFY.
I wrote up a test that does little more than start up a bunch of
threads and print messages and found it too crashes. I am testing on
both Linux x86-64 gcc 4.7.2, and Windows 7 Visual Studio 10 and both
crash, though it crashes much quicker on Windows.
It will crash pretty quickly with the default (no arguments) on Windows.
The following arguments are working for me to crash on Linux on a 12
core (24 with hyperthreading), x86-64 system.
./ThreadedNotify 48 1000 2 2
It has been pointed out that cout/cerr are thread safe, so it should
be possible to make osg::notify thread safe as well. In this program
it helps when I reduce the notification level, at least with the low
number of messages being printed it seems to be unlikely.
--
David Fries PGP pub CB1EE8F0
http://fries.net/~david/
#include
#include
#include
#include
#include
#include
#include
#include
#include
#ifdef WIN32
#include
#endif
#include
#include
#include
#include
#include
using namespace std;
// When enabled send OSG errors to stderr and if Windows OutputDebugStringA.
class OutputSplitNotifyHandler : public osg::NotifyHandler
{
public:
void notify(osg::NotifySeverity severity, const char *message)
{
fputs(message, stderr);
#if WIN32
OutputDebugStringA(message);
#endif
}
};
// Exercise the code to here, but don't print anything because output can
// be slow.
class SilentNotifyHandler : public osg::NotifyHandler
{
public:
void notify(osg::NotifySeverity severity, const char *message)
{
// Sanity check string length instead of doing absolutly nothing with
// it. A bad length could indicate a corrupted buffer.
// Extra sanity check, make sure two runs of strlen come back with the
// same length.
int len = strlen(message);
if(!len || len > 100 || len != strlen(message))
{
//fprintf(stderr, "unexpected message length %d\n", len);
}
}
};
class Spam : public OpenThreads::Thread
{
public:
Spam() : Count(0), Test(0) {}
virtual void run();
void SetCount(int c) { Count = c; }
void SetTest(int t) { Test = t; }
private:
int Count;
int Test;
};
void Spam::run()
{
stringstream ss;
ss << setw(3) << getThreadId() << " running output concurrency test ";
string base = ss.str();
for(int i=0; i= 4)
{
tcount = atoi(argv[1]);
operations = atoi(argv[2]);
test = atoi(argv[3]);
}
if(argc == 5)
{
osg_handler = atoi(argv[4]);
}
// Select OSG output method
osg::ref_ptr outputNotify;
switch(osg_handler)
{
case 0:
// keep default handler
break;
case 1:
outputNotify = new OutputSplitNotifyHandler;
break;
case 2:
outputNotify = new SilentNotifyHandler;
break;
default:
cerr << "osg_handler " << osg_handler << " not recognized\n";
exit(1);
}
if(outputNotify)
{
// Calling get is required to initialize the stream before setting the
// handler.
osg::getNotifyHandler();
osg::setNotifyHandler(outputNotify);
}
Spam *t = new Spam[tcount];
osg::Timer timer;
for(i=0; i___
osg-users mailing list
osg-users@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org
