Re: [ceph-users] mon server down

2014-04-16 Thread Jonathan Gowar
On Wed, 2014-04-16 at 03:02 +0100, Joao Eduardo Luis wrote: > You didn't recreate mon.ceph-3. > > The following should take care of that: > > 1. stop mon.ceph-3 > 2. ceph mon remove ceph-3 > 3. mv /var/lib/ceph/mon/ceph-3 /someplace/ceph-3 > 4. ceph mon getmap -o /tmp/monmap > 5. ceph-mon -i ceph

Re: [ceph-users] mon server down

2014-04-15 Thread Joao Eduardo Luis
On 04/16/2014 01:04 AM, Jonathan Gowar wrote: On Tue, 2014-04-15 at 18:02 +0100, Joao Eduardo Luis wrote: Ahah! You got bit by #5804: http://tracker.ceph.com/issues/5804 Best solution for your issue: - shutdown 'mon.ceph-3' - remove 'mon.ceph-3' from the cluster - recreate 'mon.ceph-3' - add

Re: [ceph-users] mon server down

2014-04-15 Thread Jonathan Gowar
On Tue, 2014-04-15 at 18:02 +0100, Joao Eduardo Luis wrote: > Ahah! You got bit by #5804: http://tracker.ceph.com/issues/5804 > > Best solution for your issue: > > - shutdown 'mon.ceph-3' > - remove 'mon.ceph-3' from the cluster > - recreate 'mon.ceph-3' > - add 'mon.ceph-3' to the cluster > >

Re: [ceph-users] mon server down

2014-04-15 Thread Joao Eduardo Luis
On 04/15/2014 05:20 PM, Jonathan Gowar wrote: On Tue, 2014-04-15 at 18:11 +0200, Udo Lembke wrote: is the mon-process running? netstat -an | grep 6789 | grep -i listen The process is running, but none in a listening state, instead it's "established". is the filesystem nearly full? df -k no

Re: [ceph-users] mon server down

2014-04-15 Thread Joao Eduardo Luis
On 04/15/2014 04:41 PM, Jonathan Gowar wrote: On Tue, 2014-04-15 at 15:47 +0100, Joao Eduardo Luis wrote: Well, logs would be nice. Set 'debug mon = 10' and 'debug ms = 1' on the monitor, rerun it, share the log. That might be helpful to diagnose the problem. -Joao Thanks, Joao. Seem

Re: [ceph-users] mon server down

2014-04-15 Thread Jonathan Gowar
On Tue, 2014-04-15 at 18:11 +0200, Udo Lembke wrote: > is the mon-process running? > netstat -an | grep 6789 | grep -i listen The process is running, but none in a listening state, instead it's "established". > is the filesystem nearly full? > df -k no. > any error output if you start the mon i

Re: [ceph-users] mon server down

2014-04-15 Thread Udo Lembke
Hi, is the mon-process running? netstat -an | grep 6789 | grep -i listen is the filesystem nearly full? df -k any error output if you start the mon in the foreground (here mon "b") ceph-mon -i b -d -c /etc/ceph/ceph.conf Udo Am 15.04.2014 16:11, schrieb Jonathan Gowar: > Hi, > > I had an OSD

[ceph-users] mon server down

2014-04-15 Thread Jonathan Gowar
Hi, I had an OSD fail, I replaced the drive, and that part of the array is now optimal. But in the process there's developed a problem with the mon array. I have 3 mon servers and 1 is marked down. I checked there's a mon process running, and have tried restarting the mon server. worked my w