Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

On Thu, May 24, 2018 at 12:00 AM, Sean Sullivan wrote: > Thanks Yan! I did this for the bug ticket and missed these replies. I hope I > did it correctly. Here are the pastes of the dumps: > > https://pastebin.com/kw4bZVZT -- primary > https://pastebin.com/sYZQx0ER --

Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

Thanks Yan! I did this for the bug ticket and missed these replies. I hope I did it correctly. Here are the pastes of the dumps: https://pastebin.com/kw4bZVZT -- primary https://pastebin.com/sYZQx0ER -- secondary they are not that long here is the output of one: 1. Thread 17

Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

On Wed, May 2, 2018 at 7:19 AM, Sean Sullivan wrote: > Forgot to reply to all: > > Sure thing! > > I couldn't install the ceph-mds-dbg packages without upgrading. I just > finished upgrading the cluster to 12.2.5. The issue still persists in 12.2.5 > > From here I'm not

Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

Most of this is over my head but the last line of the logs on both mds servers show something similar to: 0> 2018-05-01 15:37:46.871932 7fd10163b700 -1 *** Caught signal (Segmentation fault) ** in thread 7fd10163b700 thread_name:mds_rank_progr When I search for this in ceph user and devel

Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

Forgot to reply to all: Sure thing! I couldn't install the ceph-mds-dbg packages without upgrading. I just finished upgrading the cluster to 12.2.5. The issue still persists in 12.2.5 >From here I'm not really sure how to do generate the backtrace so I hope I did it right. For others on Ubuntu

Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

Hello Sean, On Mon, Apr 30, 2018 at 2:32 PM, Sean Sullivan wrote: > I was creating a new user and mount point. On another hardware node I > mounted CephFS as admin to mount as root. I created /aufstest and then > unmounted. From there it seems that both of my mds nodes

Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

I forgot that I left my VM mount command running. It hangs my VM but more alarming is that it crashes my MDS servers on the ceph cluster. The ceph cluster is all hardware nodes and the openstack vm does not have an admin keyring (although the cephX keyring for cephfs generated does have write

Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

So I think I can reliably reproduce this crash from a ceph client. ``` root@kh08-8:~# ceph -s cluster: id: 9f58ee5a-7c5d-4d68-81ee-debe16322544 health: HEALTH_OK services: mon: 3 daemons, quorum kh08-8,kh09-8,kh10-8 mgr: kh08-8(active) mds: cephfs-1/1/1 up

Re: [ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

I had 2 MDS servers (one active one standby) and both were down. I took a dumb chance and marked the active as down (it said it was up but laggy). Then started the primary again and now both are back up. I have never seen this before I am also not sure of what I just did. On Mon, Apr 30, 2018 at

[ceph-users] 12.2.4 Both Ceph MDS nodes crashed. Please help.

I was creating a new user and mount point. On another hardware node I mounted CephFS as admin to mount as root. I created /aufstest and then unmounted. From there it seems that both of my mds nodes crashed for some reason and I can't start them any more. https://pastebin.com/1ZgkL9fa -- my mds