Assertion error in librados

2014-02-25 Thread Filippos Giannakos
Hello all, We recently bumped into the following assertion error in librados on our production service: common/Mutex.cc: In function 'void Mutex::Lock(bool)' thread 7fa2c2ccf700 time 2014-02-21 07:23:26.340791 common/Mutex.cc: 93: FAILED assert(r == 0) ceph vers

Re: Assertion error in librados

2014-02-25 Thread Gregory Farnum
ain), but we'll need more details to track it down. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Feb 25, 2014 at 6:49 AM, Filippos Giannakos wrote: > Hello all, > > We recently bumped into the following assertion error in librados on our > producti

Re: Assertion error in librados

2014-02-25 Thread Yehuda Sadeh
http://inktank.com | http://ceph.com > > > On Tue, Feb 25, 2014 at 6:49 AM, Filippos Giannakos > wrote: >> Hello all, >> >> We recently bumped into the following assertion error in librados on our >> production service: >> >> >> common/Mu

Re: Assertion error in librados

2014-02-25 Thread Filippos Giannakos
Hi Greg, Unfortunately we don't keep any Ceph related logs on the client side. On the server side, we kept the default log settings to avoid overlogging. Do you think that there might be something usefull on the OSD side ? On Tue, Feb 25, 2014 at 07:28:30AM -0800, Gregory Farnum wrote: > Do you h

Re: Assertion error in librados

2014-02-25 Thread Noah Watkins
Perhaps using gtest-style asserts (ASSERT_EQ(r, 0)) in Ceph would be useful so we can see parameter values to the assertion in the log. In this case, the return value from pthread_mutex_lock is almost certainly EINVAL, but it'd be informative to know for sure. On Tue, Feb 25, 2014 at 7:58 AM, Fili

Re: Assertion error in librados

2014-02-25 Thread Gregory Farnum
Nope; it's an entirely local problem. I'm kind of surprised there wasn't more logging available in the same location you got the core dump, but it's possible the log generation is turned off (in addition to the log dumping). The Dispatch lock and the messenger lock are distinct, Yehuda, and the me

Re: Assertion error in librados

2014-02-25 Thread Josh Durgin
That's a good idea. This particular assert in a Mutex is almost always a use-after-free of the Mutex or structure containing it though. On 02/25/2014 09:33 AM, Noah Watkins wrote: Perhaps using gtest-style asserts (ASSERT_EQ(r, 0)) in Ceph would be useful so we can see parameter values to the as

Re: Assertion error in librados

2014-02-25 Thread Noah Watkins
On Tue, Feb 25, 2014 at 9:51 AM, Josh Durgin wrote: > That's a good idea. This particular assert in a Mutex is almost always > a use-after-free of the Mutex or structure containing it though. I think that a use-after-free will also throw an EINVAL (assuming it isn't a pathalogical case) as pthrea

Re: Assertion error in librados

2014-03-28 Thread Filippos Giannakos
Hello, We recently bumped again into the same assertion error. Do you have any indications or update regarding the cause ? On Tue, Feb 25, 2014 at 11:26:15AM -0800, Noah Watkins wrote: > On Tue, Feb 25, 2014 at 9:51 AM, Josh Durgin wrote: > > That's a good idea. This particular assert in a Mutex

Re: Assertion error in librados

2014-03-31 Thread Gregory Farnum
Nope, I don't think anybody's looked into it. If you have core dumps you could get a backtrace and the return value referenced. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Fri, Mar 28, 2014 at 2:54 AM, Filippos Giannakos wrote: > Hello, > > We recently bumped again into