Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-28 Thread Chris Kitzmiller
On Oct 28, 2014, at 5:20 PM, Lincoln Bryant wrote: > Hi Greg, Loic, > > I think we have seen this as well (sent a mail to the list a week or so ago > about incomplete pgs). I ended up giving up on the data and doing a > force_create_pgs after doing a find on my OSDs and deleting the relevant pg

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-28 Thread Loic Dachary
On 28/10/2014 22:20, Lincoln Bryant wrote: > Hi Greg, Loic, > > I think we have seen this as well (sent a mail to the list a week or so ago > about incomplete pgs). I ended up giving up on the data and doing a > force_create_pgs after doing a find on my OSDs and deleting the relevant pg > dir

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-28 Thread Loic Dachary
Hi Chris, Would you be so kind as to attach to http://tracker.ceph.com/issues/9752 the osdmaps that are relevant ( 4663 and 4685 would be helpfull) ? If my request is unclear I can guide you. Please note that I'm in Paris France and about to disconnect (11pm here ;-). I'll read you in the morni

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-28 Thread Lincoln Bryant
Hi Greg, Loic, I think we have seen this as well (sent a mail to the list a week or so ago about incomplete pgs). I ended up giving up on the data and doing a force_create_pgs after doing a find on my OSDs and deleting the relevant pg dirs. If there are any logs etc you'd like to see for debugg

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-28 Thread Gregory Farnum
On Thu, Oct 23, 2014 at 6:41 AM, Chris Kitzmiller wrote: > On Oct 22, 2014, at 8:22 PM, Craig Lewis wrote: > > Shot in the dark: try manually deep-scrubbing the PG. You could also try > marking various osd's OUT, in an attempt to get the acting set to include > osd.25 again, then do the deep-scru

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-23 Thread Chris Kitzmiller
On Oct 22, 2014, at 8:22 PM, Craig Lewis wrote: > Shot in the dark: try manually deep-scrubbing the PG. You could also try > marking various osd's OUT, in an attempt to get the acting set to include > osd.25 again, then do the deep-scrub again. That probably won't help though, > because the pg

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-22 Thread Craig Lewis
Shot in the dark: try manually deep-scrubbing the PG. You could also try marking various osd's OUT, in an attempt to get the acting set to include osd.25 again, then do the deep-scrub again. That probably won't help though, because the pg query says it probed osd.25 already... actually , it doesn

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-22 Thread Chris Kitzmiller
On Oct 22, 2014, at 7:51 PM, Craig Lewis wrote: > On Wed, Oct 22, 2014 at 3:09 PM, Chris Kitzmiller > wrote: >> On Oct 22, 2014, at 1:50 PM, Craig Lewis wrote: >>> Incomplete means "Ceph detects that a placement group is missing a >>> necessary period of history from its log. If you see this sta

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-22 Thread Craig Lewis
On Wed, Oct 22, 2014 at 3:09 PM, Chris Kitzmiller wrote: > On Oct 22, 2014, at 1:50 PM, Craig Lewis wrote: > > Incomplete means "Ceph detects that a placement group is missing a > necessary period of history from its log. If you see this state, report a > bug, and try to start any failed OSDs tha

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-22 Thread Chris Kitzmiller
On Oct 22, 2014, at 1:50 PM, Craig Lewis wrote: > Incomplete means "Ceph detects that a placement group is missing a necessary > period of history from its log. If you see this state, report a bug, and try > to start any failed OSDs that may contain the needed information". > > In the PG query,

Re: [ceph-users] Troubleshooting Incomplete PGs

2014-10-22 Thread Craig Lewis
Incomplete means "Ceph detects that a placement group is missing a necessary period of history from its log. If you see this state, report a bug, and try to start any failed OSDs that may contain the needed information". In the PG query, it lists some OSDs that it's trying to probe: "pr

[ceph-users] Troubleshooting Incomplete PGs

2014-10-21 Thread Chris Kitzmiller
I've gotten myself into the position of having ~100 incomplete PGs. All of my OSDs are up+in (and I've restarted them all one by one). I was in the process of rebalancing after altering my CRUSH map when I lost an OSD backing disk. I replaced that OSD and it seemed to be backfilling well. Durin