Re: [ceph-users] problem starting osd ; PGLog.cc: 984: FAILED assert hammer 0.94.9

2016-09-19 Thread Kostis Fardelas
So it seems that there is more than 1 PG with problems and something not-normal occured to the cluster. Taken as granted that your underlying storage/filesystems/networking work as expected you should check the timestamps/md5sums/attrs of the PGs' objects across the cluster and if you conclude

Re: [ceph-users] problem starting osd ; PGLog.cc: 984: FAILED assert hammer 0.94.9

2016-09-18 Thread Ronny Aasen
added debug journal = 20 and got some new lines in the log. that i added to the end of this email. any of you can make something out of them ? kind regards Ronny Aasen On 18.09.2016 18:59, Kostis Fardelas wrote: If you are aware of the problematic PGs and they are exportable, then

Re: [ceph-users] problem starting osd ; PGLog.cc: 984: FAILED assert hammer 0.94.9

2016-09-18 Thread Kostis Fardelas
If you are aware of the problematic PGs and they are exportable, then ceph-objectstore-tool is a viable solution. If not, then running gdb and/or higher debug osd level logs may prove useful (to understand more about the problem or collect info to ask for more in ceph-devel). On 13 September 2016

Re: [ceph-users] problem starting osd ; PGLog.cc: 984: FAILED assert hammer 0.94.9

2016-09-13 Thread Henrik Korkuc
On 16-09-13 11:13, Ronny Aasen wrote: I suspect this must be a difficult question since there have been no replies on irc or mailinglist. assuming it's impossible to get these osd's running again. Is there a way to recover objects from the disks. ? they are mounted and data is readable. I

Re: [ceph-users] problem starting osd ; PGLog.cc: 984: FAILED assert hammer 0.94.9

2016-09-13 Thread Ronny Aasen
I suspect this must be a difficult question since there have been no replies on irc or mailinglist. assuming it's impossible to get these osd's running again. Is there a way to recover objects from the disks. ? they are mounted and data is readable. I have pg's down since they want to probe

[ceph-users] problem starting osd ; PGLog.cc: 984: FAILED assert hammer 0.94.9

2016-09-12 Thread Ronny Aasen
after adding more osd's and having a big backfill running 2 of my osd's keep on stopping. We also ercently upgraded from 0.94.7 to 0.94.9 but i do not know if that is related. the log say. 0> 2016-09-12 10:31:08.288858 7f8749125880 -1 osd/PGLog.cc: In function 'static void