Re: Can't start ceph mon

2012-11-26 Thread Gregory Farnum
On Fri, Nov 23, 2012 at 12:25 AM, Dave (Bob) wrote: > Greg, > > Is there any news on this? > > David Hurray, there is! Your "latest" stashed version of the PGMap got corrupted; knowing that helped me to track down a bug — we weren't checking the return value of an fsync! Fixes to prevent this sit

Re: Can't start ceph mon

2012-11-20 Thread Gregory Farnum
On Tue, Nov 20, 2012 at 10:17 AM, Dave (Bob) wrote: >>Do you have other monitors in working order? The easiest way to handle >>it if that's the case is just to remove this monitor from the cluster >>and add it back in as a new monitor with a fresh store. If not we can >>look into reconstructing it

Re: Can't start ceph mon

2012-11-20 Thread Dave (Bob)
>Do you have other monitors in working order? The easiest way to handle >it if that's the case is just to remove this monitor from the cluster >and add it back in as a new monitor with a fresh store. If not we can >look into reconstructing it. >-Greg >Also, if you still have it, could you zip up

Re: Can't start ceph mon

2012-11-19 Thread Gregory Farnum
Also, if you still have it, could you zip up your monitor data directory and put it somewhere accessible to us? (I can provide you a drop point if necessary.) We'd like to look at the file layouts a bit since we thought we were properly handling ENOSPC-style issues. -Greg On Mon, Nov 19, 2012 at 1

Re: Can't start ceph mon

2012-11-19 Thread Gregory Farnum
On Mon, Nov 19, 2012 at 1:08 PM, Dave Humphreys (Datatone) wrote: > > I have a problem in which I can't start my ceph monitor. The log is shown > below. > > The log shows version 0.54. I was running 0.52 when the problem arose, and I > moved to the latest in case the newer version fixed the prob

Can't Start Ceph Mon

2012-11-19 Thread Dave Humphreys (Bob)
I can't start my ceph monitor, the log is attached below. Whilst the log shows 0.54, the problem arose with 0.52, and may have been caused when disk space ran out as a result of a huge set of ceph log files. Is there a way to recover? Ragards, David bash-4.1# cat /var/log/ceph/mon.vault01.

Can't start ceph mon

2012-11-19 Thread Dave Humphreys (Datatone)
I have a problem in which I can't start my ceph monitor. The log is shown below. The log shows version 0.54. I was running 0.52 when the problem arose, and I moved to the latest in case the newer version fixed the problem. The original failure happened a week or so ago, and could have been as a