Re: [ceph-users] one pg stuck with 2 unfound pieces

2013-09-23 Thread Jens-Christian Fischer
Hi Sam in the meantime, the output of ceph pg 0.cfa query has become quite a bit longer (for better or worse) - see: http://pastebin.com/0Jxmm353 I have restarted osd.23 with the debug log settings and have extracted these 0.cfa related log lines - I can't interpret them. There might be more,

Re: [ceph-users] one pg stuck with 2 unfound pieces

2013-08-14 Thread Samuel Just
Try restarting the two osd processes with debug osd = 20, debug ms = 1, debug filestore = 20. Restarting the osds may clear the problem, but if it recurs, the logs should help explain what's going on. -Sam On Wed, Aug 14, 2013 at 12:17 AM, Jens-Christian Fischer jens-christian.fisc...@switch.ch

[ceph-users] one pg stuck with 2 unfound pieces

2013-08-13 Thread Jens-Christian Fischer
We have a cluster with 10 servers, 64 OSDs and 5 Mons on them. The OSDs are 3TB disk, formatted with btrfs and the servers are either on Ubuntu 12.10 or 13.04. Recently one of the servers (13.04) stood still (due to problems with btrfs - something we have seen a few times). I decided to not try

Re: [ceph-users] one pg stuck with 2 unfound pieces

2013-08-13 Thread Samuel Just
You can run 'ceph pg 0.cfa mark_unfound_lost revert'. (Revert Lost section of http://ceph.com/docs/master/rados/operations/placement-groups/). -Sam On Tue, Aug 13, 2013 at 6:50 AM, Jens-Christian Fischer jens-christian.fisc...@switch.ch wrote: We have a cluster with 10 servers, 64 OSDs and 5