Re: [ceph-users] MDS Crashing 14.2.1
I ended up backing up the journals of the MDS ranks, recover_dentries for both
of them, resetting the journals and session table. It is back up. The recover
dentries stage didn't show any errors, so I'm not even sure why the MDS was
asserting about duplicate inodes.
--
Adam
On Thu, May 16, 2019, 13:52 Adam Tygart mailto:mo...@ksu.edu>>
wrote:
Hello all,
The rank 0 mds is still asserting. Is this duplicate inode situation
one that I should be considering using the cephfs-journal-tool to
export, recover dentries and reset?
Thanks,
Adam
On Thu, May 16, 2019 at 12:51 AM Adam Tygart
mailto:mo...@ksu.edu>> wrote:
>
> Hello all,
>
> I've got a 30 node cluster serving up lots of CephFS data.
>
> We upgraded to Nautilus 14.2.1 from Luminous 12.2.11 on Monday earlier
> this week.
>
> We've been running 2 MDS daemons in an active-active setup. Tonight
> one of the metadata daemons crashed with the following several times:
>
> -1> 2019-05-16 00:20:56.775 7f9f22405700 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/CInode.h:
> In function 'void CIn
> ode::set_primary_parent(CDentry*)' thread 7f9f22405700 time 2019-05-16
> 00:20:56.775021
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/CInode.h:
> 1114: FAILED ceph_assert(parent == 0 || g_conf().get_val("mds_h
> ack_allow_loading_invalid_metadata"))
>
> I made a quick decision to move to a single MDS because I saw
> set_primary_parent, and I thought it might be related to auto
> balancing between the metadata servers.
>
> This caused one MDS to fail, the other crashed, and now rank 0 loads,
> goes active and then crashes with the following:
> -1> 2019-05-16 00:29:21.151 7fe315e8d700 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/MDCache.cc:
> In function 'void M
> DCache::add_inode(CInode*)' thread 7fe315e8d700 time 2019-05-16
> 00:29:21.149531
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/MDCache.cc:
> 258: FAILED ceph_assert(!p)
>
> It now looks like we somehow have a duplicate inode in the MDS journal?
>
> https://people.cs.ksu.edu/~mozes/ceph-mds.melinoe.log <- was rank 0
> then became rank one after the crash and attempted drop to one active
> MDS
> https://people.cs.ksu.edu/~mozes/ceph-mds.mormo.log <- current rank 0
> and crashed
>
> Anyone have any thoughts on this?
>
> Thanks,
> Adam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] MDS Crashing 14.2.1
Hello all,
The rank 0 mds is still asserting. Is this duplicate inode situation
one that I should be considering using the cephfs-journal-tool to
export, recover dentries and reset?
Thanks,
Adam
On Thu, May 16, 2019 at 12:51 AM Adam Tygart wrote:
>
> Hello all,
>
> I've got a 30 node cluster serving up lots of CephFS data.
>
> We upgraded to Nautilus 14.2.1 from Luminous 12.2.11 on Monday earlier
> this week.
>
> We've been running 2 MDS daemons in an active-active setup. Tonight
> one of the metadata daemons crashed with the following several times:
>
> -1> 2019-05-16 00:20:56.775 7f9f22405700 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/CInode.h:
> In function 'void CIn
> ode::set_primary_parent(CDentry*)' thread 7f9f22405700 time 2019-05-16
> 00:20:56.775021
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/CInode.h:
> 1114: FAILED ceph_assert(parent == 0 || g_conf().get_val("mds_h
> ack_allow_loading_invalid_metadata"))
>
> I made a quick decision to move to a single MDS because I saw
> set_primary_parent, and I thought it might be related to auto
> balancing between the metadata servers.
>
> This caused one MDS to fail, the other crashed, and now rank 0 loads,
> goes active and then crashes with the following:
> -1> 2019-05-16 00:29:21.151 7fe315e8d700 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/MDCache.cc:
> In function 'void M
> DCache::add_inode(CInode*)' thread 7fe315e8d700 time 2019-05-16
> 00:29:21.149531
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/MDCache.cc:
> 258: FAILED ceph_assert(!p)
>
> It now looks like we somehow have a duplicate inode in the MDS journal?
>
> https://people.cs.ksu.edu/~mozes/ceph-mds.melinoe.log <- was rank 0
> then became rank one after the crash and attempted drop to one active
> MDS
> https://people.cs.ksu.edu/~mozes/ceph-mds.mormo.log <- current rank 0
> and crashed
>
> Anyone have any thoughts on this?
>
> Thanks,
> Adam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] MDS Crashing 14.2.1
Hello all,
I've got a 30 node cluster serving up lots of CephFS data.
We upgraded to Nautilus 14.2.1 from Luminous 12.2.11 on Monday earlier
this week.
We've been running 2 MDS daemons in an active-active setup. Tonight
one of the metadata daemons crashed with the following several times:
-1> 2019-05-16 00:20:56.775 7f9f22405700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/CInode.h:
In function 'void CIn
ode::set_primary_parent(CDentry*)' thread 7f9f22405700 time 2019-05-16
00:20:56.775021
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/CInode.h:
1114: FAILED ceph_assert(parent == 0 || g_conf().get_val("mds_h
ack_allow_loading_invalid_metadata"))
I made a quick decision to move to a single MDS because I saw
set_primary_parent, and I thought it might be related to auto
balancing between the metadata servers.
This caused one MDS to fail, the other crashed, and now rank 0 loads,
goes active and then crashes with the following:
-1> 2019-05-16 00:29:21.151 7fe315e8d700 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/MDCache.cc:
In function 'void M
DCache::add_inode(CInode*)' thread 7fe315e8d700 time 2019-05-16 00:29:21.149531
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/mds/MDCache.cc:
258: FAILED ceph_assert(!p)
It now looks like we somehow have a duplicate inode in the MDS journal?
https://people.cs.ksu.edu/~mozes/ceph-mds.melinoe.log <- was rank 0
then became rank one after the crash and attempted drop to one active
MDS
https://people.cs.ksu.edu/~mozes/ceph-mds.mormo.log <- current rank 0
and crashed
Anyone have any thoughts on this?
Thanks,
Adam
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
I am afraid, I hit the same bug. Giant worked fine, but after upgrading to
hammer (0.94.1) and putting some load on it, the MDSs eventually crashed and
now I am stuck in clientreplay most of the time. I am also using the cephfs
kernel client (3.18.y). As I didn't find a corresponding tracker entry .. is
there already a patch available?
Markus
From: mo...@ksu.edu
Date: Thu, 16 Apr 2015 08:04:13 -0500
To: uker...@gmail.com
CC: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] mds crashing
(Adding back to the list)
We've not seen any slow requests near that badly behind. Leading up to
the crash, the furthest behind I saw any request was ~90 seconds. Here
is the cluster log leading up to the mds crashes.
http://people.beocat.cis.ksu.edu/~mozes/ceph-mds-crashes-20150415.log
--
Adam
On Thu, Apr 16, 2015 at 1:35 AM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 10:44 AM, Adam Tygart mo...@ksu.edu wrote:
We did that just after Kyle responded to John Spray above. I am
rebuilding the kernel now to include dynamic printk support.
Maybe the first crash was caused by hang request in MDS. Is there
warnings like cluster [WRN] slow request [several thousands or more ]
seconds old, received at ...: client_request(client.734537:23 getattr
pAsLsXsFs ...) in your ceph cluster log.
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 9:37 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 10:24 AM, Adam Tygart mo...@ksu.edu wrote:
I don't have dynamic_debug enabled in the currently running kernel,
so I can't bump the verbosity of the ceph functions. I can rebuild the
kernel and reboot it to enable dynamic_debug, but then we'll have to
wait for when we re-trigger the bug. Attached is the mdsc file.
As for getting the mds back running, we put a route in the faulty
client to redirect ceph traffic to the loopback device. Started the
mds again, waited for the full startup sequence to finish for the mds
and re-set the normal routing. That seemed to cleanup the existing
session and allow the mds to live and the client to reconnect. With
the above mds requests still pending/hung, of course.
did you do the trick before? the trick leaves the client in ill state.
MDS will crash again after the client sends another 3M requests to it.
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 9:04 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 9:48 AM, Adam Tygart mo...@ksu.edu wrote:
What is significantly smaller? We have 67 requests in the 16,400,000
range and 250 in the 18,900,000 range.
that explains the crash. could you help me to debug this issue.
send /sys/kernel/debug/ceph/*/mdsc to me.
run echo module ceph +p /sys/kernel/debug/dynamic_debug/control
on the cephfs mount machine
restart the mds and wait until it crash again
run echo module ceph -p /sys/kernel/debug/dynamic_debug/control
on the cephfs mount machine
send kernel message of the cephfs mount machine to me (should in
/var/log/kerne.log or /var/log/message)
to recover from the crash. you can either force reset the machine
contains cephfs mount or add mds wipe sessions = 1 to mds section of
ceph.conf
Regards
Yan, Zheng
Thanks,
Adam
On Wed, Apr 15, 2015 at 8:38 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 9:07 AM, Adam Tygart mo...@ksu.edu wrote:
We are using 3.18.6-gentoo. Based on that, I was hoping that the
kernel bug referred to in the bug report would have been fixed.
The bug was supposed to be fixed, but you hit the bug again. could you
check if the kernel client has any hang mds request. (check
/sys/kernel/debug/ceph/*/mdsc on the machine that contain cephfs
mount. If there is any request whose ID is significant smaller than
other requests' IDs)
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 8:02 PM, Yan, Zheng uker...@gmail.com
wrote:
On Thu, Apr 16, 2015 at 5:29 AM, Kyle Hutson kylehut...@ksu.edu
wrote:
Thank you, John!
That was exactly the bug we were hitting. My Google-fu didn't lead
me to
this one.
here is the bug report http://tracker.ceph.com/issues/10449. It's a
kernel client bug which causes the session map size increase
infinitely. which version of linux kernel are using?
Regards
Yan, Zheng
On Wed, Apr 15, 2015 at 4:16 PM, John Spray
john.sp...@redhat.com wrote:
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had
been going
pretty well.
Then, about noon today, we had an mds crash. And then the
failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1),
it appears
to be OK for a little while. ceph -w goes through 'replay'
'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly
Re: [ceph-users] mds crashing
Here are some logs and the infos from the mdsc files. But I am afraid that there might not be much info in the logs, since I had a very low log level. Look for example at 2015-05-18T21:28:33+02:00. The mdsc files are concatenated from all of the clients. Date: Tue, 19 May 2015 16:45:12 +0800 Subject: Re: [ceph-users] mds crashing From: uker...@gmail.com To: bur...@muenster.de CC: mo...@ksu.edu; ceph-users@lists.ceph.com On Tue, May 19, 2015 at 4:31 PM, Markus Blank-Burian bur...@muenster.de wrote: I am afraid, I hit the same bug. Giant worked fine, but after upgrading to hammer (0.94.1) and putting some load on it, the MDSs eventually crashed and now I am stuck in clientreplay most of the time. I am also using the cephfs kernel client (3.18.y). As I didn't find a corresponding tracker entry .. is there already a patch available? Please send mds log and /sys/kernel/debug/ceph/*/mdsc on client machine to us. Besides, Is there warnings like cluster [WRN] slow request [several thousands or more ] seconds old, received at ...: client_request(client.734537:23 ...) in your ceph cluster log. Regards Yan, Zheng ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
On Tue, May 19, 2015 at 4:31 PM, Markus Blank-Burian bur...@muenster.de wrote: I am afraid, I hit the same bug. Giant worked fine, but after upgrading to hammer (0.94.1) and putting some load on it, the MDSs eventually crashed and now I am stuck in clientreplay most of the time. I am also using the cephfs kernel client (3.18.y). As I didn't find a corresponding tracker entry .. is there already a patch available? Please send mds log and /sys/kernel/debug/ceph/*/mdsc on client machine to us. Besides, Is there warnings like cluster [WRN] slow request [several thousands or more ] seconds old, received at ...: client_request(client.734537:23 ...) in your ceph cluster log. Regards Yan, Zheng ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
On 2015年05月19日 17:07, Markus Blank-Burian wrote: Here are some logs and the infos from the mdsc files. But I am afraid that there might not be much info in the logs, since I had a very low log level. Look for example at2015-05-18T21:28:33+02:00. The mdsc files are concatenated from all of the clients. You can increase log level such as 'ceph tell mds.* injectargs --debug_mds 20 --debug_rados 10 --debug_ms 1' ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
could you try the attached patch On Tue, May 19, 2015 at 5:10 PM, Markus Blank-Burian bur...@muenster.de wrote: Forgot the attachments. Besides, is there any way to get the cluster running again without restarting all client nodes? On Tue, May 19, 2015 at 10:45 AM, Yan, Zheng uker...@gmail.com wrote: On Tue, May 19, 2015 at 4:31 PM, Markus Blank-Burian bur...@muenster.de wrote: I am afraid, I hit the same bug. Giant worked fine, but after upgrading to hammer (0.94.1) and putting some load on it, the MDSs eventually crashed and now I am stuck in clientreplay most of the time. I am also using the cephfs kernel client (3.18.y). As I didn't find a corresponding tracker entry .. is there already a patch available? Please send mds log and /sys/kernel/debug/ceph/*/mdsc on client machine to us. Besides, Is there warnings like cluster [WRN] slow request [several thousands or more ] seconds old, received at ...: client_request(client.734537:23 ...) in your ceph cluster log. Regards Yan, Zheng 3.18.13.patch Description: Binary data ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
I actually managed to reboot everything today and it ran smoothly for the last few minutes. MDS failover also worked without problems. If anything bad happens in the next days, I will let you know. Markus On Tue, May 19, 2015 at 1:12 PM, Markus Blank-Burian bur...@muenster.de wrote: Thanks for the patch! Testing might take up to a week, since I have to reboot all the client nodes in the computing cluster. On Tue, May 19, 2015 at 12:27 PM, Yan, Zheng uker...@gmail.com wrote: could you try the attached patch On Tue, May 19, 2015 at 5:10 PM, Markus Blank-Burian bur...@muenster.de wrote: Forgot the attachments. Besides, is there any way to get the cluster running again without restarting all client nodes? On Tue, May 19, 2015 at 10:45 AM, Yan, Zheng uker...@gmail.com wrote: On Tue, May 19, 2015 at 4:31 PM, Markus Blank-Burian bur...@muenster.de wrote: I am afraid, I hit the same bug. Giant worked fine, but after upgrading to hammer (0.94.1) and putting some load on it, the MDSs eventually crashed and now I am stuck in clientreplay most of the time. I am also using the cephfs kernel client (3.18.y). As I didn't find a corresponding tracker entry .. is there already a patch available? Please send mds log and /sys/kernel/debug/ceph/*/mdsc on client machine to us. Besides, Is there warnings like cluster [WRN] slow request [several thousands or more ] seconds old, received at ...: client_request(client.734537:23 ...) in your ceph cluster log. Regards Yan, Zheng ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
(Adding back to the list)
We've not seen any slow requests near that badly behind. Leading up to
the crash, the furthest behind I saw any request was ~90 seconds. Here
is the cluster log leading up to the mds crashes.
http://people.beocat.cis.ksu.edu/~mozes/ceph-mds-crashes-20150415.log
--
Adam
On Thu, Apr 16, 2015 at 1:35 AM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 10:44 AM, Adam Tygart mo...@ksu.edu wrote:
We did that just after Kyle responded to John Spray above. I am
rebuilding the kernel now to include dynamic printk support.
Maybe the first crash was caused by hang request in MDS. Is there
warnings like cluster [WRN] slow request [several thousands or more ]
seconds old, received at ...: client_request(client.734537:23 getattr
pAsLsXsFs ...) in your ceph cluster log.
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 9:37 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 10:24 AM, Adam Tygart mo...@ksu.edu wrote:
I don't have dynamic_debug enabled in the currently running kernel,
so I can't bump the verbosity of the ceph functions. I can rebuild the
kernel and reboot it to enable dynamic_debug, but then we'll have to
wait for when we re-trigger the bug. Attached is the mdsc file.
As for getting the mds back running, we put a route in the faulty
client to redirect ceph traffic to the loopback device. Started the
mds again, waited for the full startup sequence to finish for the mds
and re-set the normal routing. That seemed to cleanup the existing
session and allow the mds to live and the client to reconnect. With
the above mds requests still pending/hung, of course.
did you do the trick before? the trick leaves the client in ill state.
MDS will crash again after the client sends another 3M requests to it.
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 9:04 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 9:48 AM, Adam Tygart mo...@ksu.edu wrote:
What is significantly smaller? We have 67 requests in the 16,400,000
range and 250 in the 18,900,000 range.
that explains the crash. could you help me to debug this issue.
send /sys/kernel/debug/ceph/*/mdsc to me.
run echo module ceph +p /sys/kernel/debug/dynamic_debug/control
on the cephfs mount machine
restart the mds and wait until it crash again
run echo module ceph -p /sys/kernel/debug/dynamic_debug/control
on the cephfs mount machine
send kernel message of the cephfs mount machine to me (should in
/var/log/kerne.log or /var/log/message)
to recover from the crash. you can either force reset the machine
contains cephfs mount or add mds wipe sessions = 1 to mds section of
ceph.conf
Regards
Yan, Zheng
Thanks,
Adam
On Wed, Apr 15, 2015 at 8:38 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 9:07 AM, Adam Tygart mo...@ksu.edu wrote:
We are using 3.18.6-gentoo. Based on that, I was hoping that the
kernel bug referred to in the bug report would have been fixed.
The bug was supposed to be fixed, but you hit the bug again. could you
check if the kernel client has any hang mds request. (check
/sys/kernel/debug/ceph/*/mdsc on the machine that contain cephfs
mount. If there is any request whose ID is significant smaller than
other requests' IDs)
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 8:02 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 5:29 AM, Kyle Hutson kylehut...@ksu.edu
wrote:
Thank you, John!
That was exactly the bug we were hitting. My Google-fu didn't lead
me to
this one.
here is the bug report http://tracker.ceph.com/issues/10449. It's a
kernel client bug which causes the session map size increase
infinitely. which version of linux kernel are using?
Regards
Yan, Zheng
On Wed, Apr 15, 2015 at 4:16 PM, John Spray john.sp...@redhat.com
wrote:
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been
going
pretty well.
Then, about noon today, we had an mds crash. And then the failover
mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it
appears
to be OK for a little while. ceph -w goes through 'replay'
'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly immediately after
getting to
'active', it crashes again.
I have the mds log at
http://people.beocat.cis.ksu.edu/~kylehutson/ceph-mds.hobbit01.log
http://people.beocat.cis.ksu.edu/%7Ekylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both
pg_num and
pgp_num from 2048 to 4096. We still have several objects misplaced
(~17%),
but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old
(non-ceph)
filesystem to this filesystem.
- Before we realized the mds crashes, we had
[ceph-users] mds crashing
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been going
pretty well.
Then, about noon today, we had an mds crash. And then the failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it appears
to be OK for a little while. ceph -w goes through 'replay' 'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly immediately after getting
to 'active', it crashes again.
I have the mds log at
http://people.beocat.cis.ksu.edu/~kylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both pg_num and
pgp_num from 2048 to 4096. We still have several objects misplaced (~17%),
but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old (non-ceph)
filesystem to this filesystem.
- Before we realized the mds crashes, we had just changed the size of our
metadata pool from 2 to 4.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
Thank you, John!
That was exactly the bug we were hitting. My Google-fu didn't lead me to
this one.
On Wed, Apr 15, 2015 at 4:16 PM, John Spray john.sp...@redhat.com wrote:
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been going
pretty well.
Then, about noon today, we had an mds crash. And then the failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it appears
to be OK for a little while. ceph -w goes through 'replay' 'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly immediately after getting
to 'active', it crashes again.
I have the mds log at http://people.beocat.cis.ksu.
edu/~kylehutson/ceph-mds.hobbit01.log http://people.beocat.cis.ksu.
edu/%7Ekylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both pg_num and
pgp_num from 2048 to 4096. We still have several objects misplaced (~17%),
but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old (non-ceph)
filesystem to this filesystem.
- Before we realized the mds crashes, we had just changed the size of our
metadata pool from 2 to 4.
It looks like you're seeing http://tracker.ceph.com/issues/10449, which
is a situation where the SessionMap object becomes too big for the MDS to
save.The cause of it in that case was stuck requests from a misbehaving
client running a slightly older kernel.
Assuming you're using the kernel client and having a similar problem, you
could try to work around this situation by forcibly unmounting the clients
while the MDS is offline, such that during clientreplay the MDS will remove
them from the SessionMap after timing out, and then next time it tries to
save the map it won't be oversized. If that works, you could then look
into getting newer kernels on the clients to avoid hitting the issue again
-- the #10449 ticket has some pointers about which kernel changes were
relevant.
Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
What is significantly smaller? We have 67 requests in the 16,400,000
range and 250 in the 18,900,000 range.
Thanks,
Adam
On Wed, Apr 15, 2015 at 8:38 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 9:07 AM, Adam Tygart mo...@ksu.edu wrote:
We are using 3.18.6-gentoo. Based on that, I was hoping that the
kernel bug referred to in the bug report would have been fixed.
The bug was supposed to be fixed, but you hit the bug again. could you
check if the kernel client has any hang mds request. (check
/sys/kernel/debug/ceph/*/mdsc on the machine that contain cephfs
mount. If there is any request whose ID is significant smaller than
other requests' IDs)
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 8:02 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 5:29 AM, Kyle Hutson kylehut...@ksu.edu wrote:
Thank you, John!
That was exactly the bug we were hitting. My Google-fu didn't lead me to
this one.
here is the bug report http://tracker.ceph.com/issues/10449. It's a
kernel client bug which causes the session map size increase
infinitely. which version of linux kernel are using?
Regards
Yan, Zheng
On Wed, Apr 15, 2015 at 4:16 PM, John Spray john.sp...@redhat.com wrote:
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been going
pretty well.
Then, about noon today, we had an mds crash. And then the failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it appears
to be OK for a little while. ceph -w goes through 'replay' 'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly immediately after
getting to
'active', it crashes again.
I have the mds log at
http://people.beocat.cis.ksu.edu/~kylehutson/ceph-mds.hobbit01.log
http://people.beocat.cis.ksu.edu/%7Ekylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both pg_num and
pgp_num from 2048 to 4096. We still have several objects misplaced
(~17%),
but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old (non-ceph)
filesystem to this filesystem.
- Before we realized the mds crashes, we had just changed the size of our
metadata pool from 2 to 4.
It looks like you're seeing http://tracker.ceph.com/issues/10449, which is
a situation where the SessionMap object becomes too big for the MDS to
save.The cause of it in that case was stuck requests from a misbehaving
client running a slightly older kernel.
Assuming you're using the kernel client and having a similar problem, you
could try to work around this situation by forcibly unmounting the clients
while the MDS is offline, such that during clientreplay the MDS will
remove
them from the SessionMap after timing out, and then next time it tries to
save the map it won't be oversized. If that works, you could then look
into
getting newer kernels on the clients to avoid hitting the issue again --
the
#10449 ticket has some pointers about which kernel changes were relevant.
Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
We are using 3.18.6-gentoo. Based on that, I was hoping that the
kernel bug referred to in the bug report would have been fixed.
--
Adam
On Wed, Apr 15, 2015 at 8:02 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 5:29 AM, Kyle Hutson kylehut...@ksu.edu wrote:
Thank you, John!
That was exactly the bug we were hitting. My Google-fu didn't lead me to
this one.
here is the bug report http://tracker.ceph.com/issues/10449. It's a
kernel client bug which causes the session map size increase
infinitely. which version of linux kernel are using?
Regards
Yan, Zheng
On Wed, Apr 15, 2015 at 4:16 PM, John Spray john.sp...@redhat.com wrote:
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been going
pretty well.
Then, about noon today, we had an mds crash. And then the failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it appears
to be OK for a little while. ceph -w goes through 'replay' 'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly immediately after getting
to
'active', it crashes again.
I have the mds log at
http://people.beocat.cis.ksu.edu/~kylehutson/ceph-mds.hobbit01.log
http://people.beocat.cis.ksu.edu/%7Ekylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both pg_num and
pgp_num from 2048 to 4096. We still have several objects misplaced (~17%),
but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old (non-ceph)
filesystem to this filesystem.
- Before we realized the mds crashes, we had just changed the size of our
metadata pool from 2 to 4.
It looks like you're seeing http://tracker.ceph.com/issues/10449, which is
a situation where the SessionMap object becomes too big for the MDS to
save.The cause of it in that case was stuck requests from a misbehaving
client running a slightly older kernel.
Assuming you're using the kernel client and having a similar problem, you
could try to work around this situation by forcibly unmounting the clients
while the MDS is offline, such that during clientreplay the MDS will remove
them from the SessionMap after timing out, and then next time it tries to
save the map it won't be oversized. If that works, you could then look into
getting newer kernels on the clients to avoid hitting the issue again -- the
#10449 ticket has some pointers about which kernel changes were relevant.
Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
On Thu, Apr 16, 2015 at 9:48 AM, Adam Tygart mo...@ksu.edu wrote:
What is significantly smaller? We have 67 requests in the 16,400,000
range and 250 in the 18,900,000 range.
that explains the crash. could you help me to debug this issue.
send /sys/kernel/debug/ceph/*/mdsc to me.
run echo module ceph +p /sys/kernel/debug/dynamic_debug/control
on the cephfs mount machine
restart the mds and wait until it crash again
run echo module ceph -p /sys/kernel/debug/dynamic_debug/control
on the cephfs mount machine
send kernel message of the cephfs mount machine to me (should in
/var/log/kerne.log or /var/log/message)
to recover from the crash. you can either force reset the machine
contains cephfs mount or add mds wipe sessions = 1 to mds section of
ceph.conf
Regards
Yan, Zheng
Thanks,
Adam
On Wed, Apr 15, 2015 at 8:38 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 9:07 AM, Adam Tygart mo...@ksu.edu wrote:
We are using 3.18.6-gentoo. Based on that, I was hoping that the
kernel bug referred to in the bug report would have been fixed.
The bug was supposed to be fixed, but you hit the bug again. could you
check if the kernel client has any hang mds request. (check
/sys/kernel/debug/ceph/*/mdsc on the machine that contain cephfs
mount. If there is any request whose ID is significant smaller than
other requests' IDs)
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 8:02 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 5:29 AM, Kyle Hutson kylehut...@ksu.edu wrote:
Thank you, John!
That was exactly the bug we were hitting. My Google-fu didn't lead me to
this one.
here is the bug report http://tracker.ceph.com/issues/10449. It's a
kernel client bug which causes the session map size increase
infinitely. which version of linux kernel are using?
Regards
Yan, Zheng
On Wed, Apr 15, 2015 at 4:16 PM, John Spray john.sp...@redhat.com wrote:
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been going
pretty well.
Then, about noon today, we had an mds crash. And then the failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it
appears
to be OK for a little while. ceph -w goes through 'replay' 'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly immediately after
getting to
'active', it crashes again.
I have the mds log at
http://people.beocat.cis.ksu.edu/~kylehutson/ceph-mds.hobbit01.log
http://people.beocat.cis.ksu.edu/%7Ekylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both pg_num and
pgp_num from 2048 to 4096. We still have several objects misplaced
(~17%),
but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old (non-ceph)
filesystem to this filesystem.
- Before we realized the mds crashes, we had just changed the size of
our
metadata pool from 2 to 4.
It looks like you're seeing http://tracker.ceph.com/issues/10449, which
is
a situation where the SessionMap object becomes too big for the MDS to
save.The cause of it in that case was stuck requests from a misbehaving
client running a slightly older kernel.
Assuming you're using the kernel client and having a similar problem, you
could try to work around this situation by forcibly unmounting the
clients
while the MDS is offline, such that during clientreplay the MDS will
remove
them from the SessionMap after timing out, and then next time it tries to
save the map it won't be oversized. If that works, you could then look
into
getting newer kernels on the clients to avoid hitting the issue again --
the
#10449 ticket has some pointers about which kernel changes were relevant.
Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
On Thu, Apr 16, 2015 at 9:07 AM, Adam Tygart mo...@ksu.edu wrote:
We are using 3.18.6-gentoo. Based on that, I was hoping that the
kernel bug referred to in the bug report would have been fixed.
The bug was supposed to be fixed, but you hit the bug again. could you
check if the kernel client has any hang mds request. (check
/sys/kernel/debug/ceph/*/mdsc on the machine that contain cephfs
mount. If there is any request whose ID is significant smaller than
other requests' IDs)
Regards
Yan, Zheng
--
Adam
On Wed, Apr 15, 2015 at 8:02 PM, Yan, Zheng uker...@gmail.com wrote:
On Thu, Apr 16, 2015 at 5:29 AM, Kyle Hutson kylehut...@ksu.edu wrote:
Thank you, John!
That was exactly the bug we were hitting. My Google-fu didn't lead me to
this one.
here is the bug report http://tracker.ceph.com/issues/10449. It's a
kernel client bug which causes the session map size increase
infinitely. which version of linux kernel are using?
Regards
Yan, Zheng
On Wed, Apr 15, 2015 at 4:16 PM, John Spray john.sp...@redhat.com wrote:
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been going
pretty well.
Then, about noon today, we had an mds crash. And then the failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it appears
to be OK for a little while. ceph -w goes through 'replay' 'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly immediately after getting
to
'active', it crashes again.
I have the mds log at
http://people.beocat.cis.ksu.edu/~kylehutson/ceph-mds.hobbit01.log
http://people.beocat.cis.ksu.edu/%7Ekylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both pg_num and
pgp_num from 2048 to 4096. We still have several objects misplaced (~17%),
but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old (non-ceph)
filesystem to this filesystem.
- Before we realized the mds crashes, we had just changed the size of our
metadata pool from 2 to 4.
It looks like you're seeing http://tracker.ceph.com/issues/10449, which is
a situation where the SessionMap object becomes too big for the MDS to
save.The cause of it in that case was stuck requests from a misbehaving
client running a slightly older kernel.
Assuming you're using the kernel client and having a similar problem, you
could try to work around this situation by forcibly unmounting the clients
while the MDS is offline, such that during clientreplay the MDS will remove
them from the SessionMap after timing out, and then next time it tries to
save the map it won't be oversized. If that works, you could then look
into
getting newer kernels on the clients to avoid hitting the issue again --
the
#10449 ticket has some pointers about which kernel changes were relevant.
Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
On Thu, Apr 16, 2015 at 5:29 AM, Kyle Hutson kylehut...@ksu.edu wrote:
Thank you, John!
That was exactly the bug we were hitting. My Google-fu didn't lead me to
this one.
here is the bug report http://tracker.ceph.com/issues/10449. It's a
kernel client bug which causes the session map size increase
infinitely. which version of linux kernel are using?
Regards
Yan, Zheng
On Wed, Apr 15, 2015 at 4:16 PM, John Spray john.sp...@redhat.com wrote:
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been going
pretty well.
Then, about noon today, we had an mds crash. And then the failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it appears
to be OK for a little while. ceph -w goes through 'replay' 'reconnect'
'rejoin' 'clientreplay' and 'active' but nearly immediately after getting to
'active', it crashes again.
I have the mds log at
http://people.beocat.cis.ksu.edu/~kylehutson/ceph-mds.hobbit01.log
http://people.beocat.cis.ksu.edu/%7Ekylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both pg_num and
pgp_num from 2048 to 4096. We still have several objects misplaced (~17%),
but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old (non-ceph)
filesystem to this filesystem.
- Before we realized the mds crashes, we had just changed the size of our
metadata pool from 2 to 4.
It looks like you're seeing http://tracker.ceph.com/issues/10449, which is
a situation where the SessionMap object becomes too big for the MDS to
save.The cause of it in that case was stuck requests from a misbehaving
client running a slightly older kernel.
Assuming you're using the kernel client and having a similar problem, you
could try to work around this situation by forcibly unmounting the clients
while the MDS is offline, such that during clientreplay the MDS will remove
them from the SessionMap after timing out, and then next time it tries to
save the map it won't be oversized. If that works, you could then look into
getting newer kernels on the clients to avoid hitting the issue again -- the
#10449 ticket has some pointers about which kernel changes were relevant.
Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] mds crashing
On 15/04/2015 20:02, Kyle Hutson wrote:
I upgraded to 0.94.1 from 0.94 on Monday, and everything had been
going pretty well.
Then, about noon today, we had an mds crash. And then the failover mds
crashed. And this cascaded through all 4 mds servers we have.
If I try to start it ('service ceph start mds' on CentOS 7.1), it
appears to be OK for a little while. ceph -w goes through 'replay'
'reconnect' 'rejoin' 'clientreplay' and 'active' but nearly
immediately after getting to 'active', it crashes again.
I have the mds log at
http://people.beocat.cis.ksu.edu/~kylehutson/ceph-mds.hobbit01.log
http://people.beocat.cis.ksu.edu/%7Ekylehutson/ceph-mds.hobbit01.log
For the possibly, but not necessarily, useful background info.
- Yesterday we took our erasure coded pool and increased both pg_num
and pgp_num from 2048 to 4096. We still have several objects misplaced
(~17%), but those seem to be continuing to clean themselves up.
- We are in the midst of a large (300+ TB) rsync from our old
(non-ceph) filesystem to this filesystem.
- Before we realized the mds crashes, we had just changed the size of
our metadata pool from 2 to 4.
It looks like you're seeing http://tracker.ceph.com/issues/10449, which
is a situation where the SessionMap object becomes too big for the MDS
to save.The cause of it in that case was stuck requests from a
misbehaving client running a slightly older kernel.
Assuming you're using the kernel client and having a similar problem,
you could try to work around this situation by forcibly unmounting the
clients while the MDS is offline, such that during clientreplay the MDS
will remove them from the SessionMap after timing out, and then next
time it tries to save the map it won't be oversized. If that works, you
could then look into getting newer kernels on the clients to avoid
hitting the issue again -- the #10449 ticket has some pointers about
which kernel changes were relevant.
Cheers,
John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
