subject:"\[ceph\-users\] PGs stuck active\+remapped and osds lose data\?\!"

[ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Marcus Müller

Hi all,

Recently I added a new node with new osds to my cluster, which, of course 
resulted in backfilling. At the end, there are 4 pgs left in the state 4 
active+remapped and I don’t know what to do. 

Here is how my cluster looks like currently: 

ceph -s
 health HEALTH_WARN
4 pgs stuck unclean
recovery 3586/58734009 objects degraded (0.006%)
recovery 420074/58734009 objects misplaced (0.715%)
noscrub,nodeep-scrub flag(s) set
 monmap e9: 5 mons at 
{ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
election epoch 478, quorum 0,1,2,3,4 ceph1,ceph2,ceph3,ceph4,ceph5
 osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
flags noscrub,nodeep-scrub
  pgmap v9970276: 320 pgs, 3 pools, 4831 GB data, 19119 kobjects
15152 GB used, 40719 GB / 55872 GB avail
3586/58734009 objects degraded (0.006%)
420074/58734009 objects misplaced (0.715%)
 316 active+clean
   4 active+remapped
  client io 643 kB/s rd, 7 op/s

# ceph osd df
ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  
 0 1.28899  1.0  3724G  1697G  2027G 45.57 1.68 
 1 1.57899  1.0  3724G  1706G  2018G 45.81 1.69 
 2 1.68900  1.0  3724G  1794G  1929G 48.19 1.78 
 3 6.78499  1.0  7450G  1240G  6209G 16.65 0.61 
 4 8.3  1.0  7450G  1226G  6223G 16.47 0.61 
 5 9.51500  1.0  7450G  1237G  6212G 16.62 0.61 
 6 7.66499  1.0  7450G  1264G  6186G 16.97 0.63 
 7 9.75499  1.0  7450G  2494G  4955G 33.48 1.23 
 8 9.32999  1.0  7450G  2491G  4958G 33.45 1.23 
  TOTAL 55872G 15152G 40719G 27.12  
MIN/MAX VAR: 0.61/1.78  STDDEV: 13.54

# ceph health detail
HEALTH_WARN 4 pgs stuck unclean; recovery 3586/58734015 objects degraded 
(0.006%); recovery 420074/58734015 objects misplaced (0.715%); 
noscrub,nodeep-scrub flag(s) set
pg 9.7 is stuck unclean for 512936.160212, current state active+remapped, last 
acting [7,3,0]
pg 7.84 is stuck unclean for 512623.894574, current state active+remapped, last 
acting [4,8,1]
pg 8.1b is stuck unclean for 513164.616377, current state active+remapped, last 
acting [4,7,2]
pg 7.7a is stuck unclean for 513162.316328, current state active+remapped, last 
acting [7,4,2]
recovery 3586/58734015 objects degraded (0.006%)
recovery 420074/58734015 objects misplaced (0.715%)
noscrub,nodeep-scrub flag(s) set

# ceph osd tree
ID WEIGHT   TYPE NAME  UP/DOWN REWEIGHT PRIMARY-AFFINITY 
-1 56.00693 root default 
-2  1.28899 host ceph1   
 0  1.28899 osd.0   up  1.0  1.0 
-3  1.57899 host ceph2   
 1  1.57899 osd.1   up  1.0  1.0 
-4  1.68900 host ceph3   
 2  1.68900 osd.2   up  1.0  1.0 
-5 32.36497 host ceph4   
 3  6.78499 osd.3   up  1.0  1.0 
 4  8.3 osd.4   up  1.0  1.0 
 5  9.51500 osd.5   up  1.0  1.0 
 6  7.66499 osd.6   up  1.0  1.0 
-6 19.08498 host ceph5   
 7  9.75499 osd.7   up  1.0  1.0 
 8  9.32999 osd.8   up  1.0  1.0 

I’m using a customized crushmap because as you can see this cluster is not very 
optimal. Ceph1, ceph2 and ceph3 are vms on one physical host - Ceph4 and Ceph5 
are both separate physical hosts. So the idea is to spread 33% of the data to 
ceph1, ceph2 and ceph3 and the other 66% to each ceph4 and ceph5.

Everything went fine with the backfilling but now I see those 4 pgs stuck 
active+remapped since 2 days while the degrades objects increase. 

I did a restart of all osds after and after but this helped not really. It 
first showed me no degraded objects and then increased again.

What can I do in order to get those pgs to active+clean state again? My idea 
was to increase the weight of a osd a little bit in order to let ceph calculate 
the map again, is this a good idea?

---

On the other side I saw something very strange too: After the backfill was done 
(2 days ago), my ceph osd df looked like this:

# ceph osd df
ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  
 0 1.28899  1.0  3724G  1924G  1799G 51.67 1.79 
 1 1.57899  1.0  3724G  2143G  1580G 57.57 2.00 
 2 1.68900  1.0  3724G  2114G  1609G 56.78 1.97 
 3 6.78499  1.0  7450G  1234G  6215G 16.57 0.58 
 4 8.3  1.0  7450G  1221G  6228G 16.40 0.57 
 5 9.51500  1.0  7450G  1232G  6217G 16.54 0.57 
 6 7.66499  1.0  7450G  1258G  6191G 16.89 0.59 
 7 9.75499  1.0  7450G  2482G  4967G 33.33 1.16 
 8 9.32999  1.0  7450G  2480G  4969G 33.30 1.16 
  TOTAL 55872G 160

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Christian Wuerdig

On Tue, Jan 10, 2017 at 8:23 AM, Marcus Müller 
wrote:

> Hi all,
>
> Recently I added a new node with new osds to my cluster, which, of course
> resulted in backfilling. At the end, there are 4 pgs left in the state 4
> active+remapped and I don’t know what to do.
>
> Here is how my cluster looks like currently:
>
> ceph -s
>  health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 3586/58734009 objects degraded (0.006%)
> recovery 420074/58734009 objects misplaced (0.715%)
> noscrub,nodeep-scrub flag(s) set
>  monmap e9: 5 mons at {ceph1=192.168.10.3:6789/0,
> ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,
> ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
> election epoch 478, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
> flags noscrub,nodeep-scrub
>   pgmap v9970276: 320 pgs, 3 pools, 4831 GB data, 19119 kobjects
> 15152 GB used, 40719 GB / 55872 GB avail
> 3586/58734009 objects degraded (0.006%)
> 420074/58734009 objects misplaced (0.715%)
>  316 active+clean
>4 active+remapped
>   client io 643 kB/s rd, 7 op/s
>
> # ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>  0 1.28899  1.0  3724G  1697G  2027G 45.57 1.68
>  1 1.57899  1.0  3724G  1706G  2018G 45.81 1.69
>  2 1.68900  1.0  3724G  1794G  1929G 48.19 1.78
>  3 6.78499  1.0  7450G  1240G  6209G 16.65 0.61
>  4 8.3  1.0  7450G  1226G  6223G 16.47 0.61
>  5 9.51500  1.0  7450G  1237G  6212G 16.62 0.61
>  6 7.66499  1.0  7450G  1264G  6186G 16.97 0.63
>  7 9.75499  1.0  7450G  2494G  4955G 33.48 1.23
>  8 9.32999  1.0  7450G  2491G  4958G 33.45 1.23
>   TOTAL 55872G 15152G 40719G 27.12
> MIN/MAX VAR: 0.61/1.78  STDDEV: 13.54
>
> # ceph health detail
> HEALTH_WARN 4 pgs stuck unclean; recovery 3586/58734015 objects degraded
> (0.006%); recovery 420074/58734015 objects misplaced (0.715%);
> noscrub,nodeep-scrub flag(s) set
> pg 9.7 is stuck unclean for 512936.160212, current state active+remapped,
> last acting [7,3,0]
> pg 7.84 is stuck unclean for 512623.894574, current state active+remapped,
> last acting [4,8,1]
> pg 8.1b is stuck unclean for 513164.616377, current state active+remapped,
> last acting [4,7,2]
> pg 7.7a is stuck unclean for 513162.316328, current state active+remapped,
> last acting [7,4,2]
> recovery 3586/58734015 objects degraded (0.006%)
> recovery 420074/58734015 objects misplaced (0.715%)
> noscrub,nodeep-scrub flag(s) set
>
> # ceph osd tree
> ID WEIGHT   TYPE NAME  UP/DOWN REWEIGHT PRIMARY-AFFINITY
> -1 56.00693 root default
> -2  1.28899 host ceph1
>  0  1.28899 osd.0   up  1.0  1.0
> -3  1.57899 host ceph2
>  1  1.57899 osd.1   up  1.0  1.0
> -4  1.68900 host ceph3
>  2  1.68900 osd.2   up  1.0  1.0
> -5 32.36497 host ceph4
>  3  6.78499 osd.3   up  1.0  1.0
>  4  8.3 osd.4   up  1.0  1.0
>  5  9.51500 osd.5   up  1.0  1.0
>  6  7.66499 osd.6   up  1.0  1.0
> -6 19.08498 host ceph5
>  7  9.75499 osd.7   up  1.0  1.0
>  8  9.32999 osd.8   up  1.0  1.0
>
> I’m using a customized crushmap because as you can see this cluster is not
> very optimal. Ceph1, ceph2 and ceph3 are vms on one physical host - Ceph4
> and Ceph5 are both separate physical hosts. So the idea is to spread 33% of
> the data to ceph1, ceph2 and ceph3 and the other 66% to each ceph4 and
> ceph5.
>
> Everything went fine with the backfilling but now I see those 4 pgs stuck
> active+remapped since 2 days while the degrades objects increase.
>
> I did a restart of all osds after and after but this helped not really. It
> first showed me no degraded objects and then increased again.
>
> What can I do in order to get those pgs to active+clean state again? My
> idea was to increase the weight of a osd a little bit in order to let ceph
> calculate the map again, is this a good idea?
>

Trying google with "ceph pg stuck in active and remapped" points to a
couple of post on this ML typically indicating that it's a problem with the
CRUSH map and ceph being unable to satisfy the mapping rules. Your ceph -s
output indicates that your using replication of size 3 in your pools. You
also said you had a custom CRUSH map - can you post it?


>
> ---
>
> On the other side I saw something very strange too: After the backfill was
> done (2 days ago), my ceph osd df looked like this:
>
> # ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>  0 1.28899  1.0  3724G  1924G  1799G 51.67 1.79
>  1 1.57899  1.0  3724G  2143G  1580G 57.57 2.00
>  2 1.68900  1.0  3724G  2114G  1609G 56.78 1.97
>  3

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Marcus Müller

> Trying google with "ceph pg stuck in active and remapped" points to a couple 
> of post on this ML typically indicating that it's a problem with the CRUSH 
> map and ceph being unable to satisfy the mapping rules. Your ceph -s output 
> indicates that your using replication of size 3 in your pools. You also said 
> you had a custom CRUSH map - can you post it?

I’ve sent the file to you, since I’m not sure if it contains sensitive data. 
Yes I have replication of 3 and I did not customize the map by me.


> I might be missing something here but I don't quite see how you come to this 
> statement. ceph osd df and ceph -s both show 16093 GB used and 39779 GB out 
> of 55872 GB available. The sum of the first 3 OSDs used space is, as you 
> stated, 6181 GB which is approx 38.4% so quite close to your target of 33%

Maybe I have to explain it another way:

Directly after finishing the backfill I received this output:

 health HEALTH_WARN
4 pgs stuck unclean
recovery 1698/58476648 objects degraded (0.003%)
recovery 418137/58476648 objects misplaced (0.715%)
noscrub,nodeep-scrub flag(s) set
 monmap e9: 5 mons at 
{ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
election epoch 464, quorum 0,1,2,3,4 ceph1,ceph2,ceph3,ceph4,ceph5
 osdmap e3086: 9 osds: 9 up, 9 in; 4 remapped pgs
flags noscrub,nodeep-scrub
  pgmap v9928160: 320 pgs, 3 pools, 4809 GB data, 19035 kobjects
16093 GB used, 39779 GB / 55872 GB avail
1698/58476648 objects degraded (0.003%)
418137/58476648 objects misplaced (0.715%)
 316 active+clean
   4 active+remapped
  client io 757 kB/s rd, 1 op/s

# ceph osd df
ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  
 0 1.28899  1.0  3724G  1924G  1799G 51.67 1.79 
 1 1.57899  1.0  3724G  2143G  1580G 57.57 2.00 
 2 1.68900  1.0  3724G  2114G  1609G 56.78 1.97 
 3 6.78499  1.0  7450G  1234G  6215G 16.57 0.58 
 4 8.3  1.0  7450G  1221G  6228G 16.40 0.57 
 5 9.51500  1.0  7450G  1232G  6217G 16.54 0.57 
 6 7.66499  1.0  7450G  1258G  6191G 16.89 0.59 
 7 9.75499  1.0  7450G  2482G  4967G 33.33 1.16 
 8 9.32999  1.0  7450G  2480G  4969G 33.30 1.16 
  TOTAL 55872G 16093G 39779G 28.80  
MIN/MAX VAR: 0.57/2.00  STDDEV: 17.54

Here we can see, that the cluster is using 4809 GB data and has raw used 
16093GB. Or the other way, only 39779G available.

Two days later I saw:

 health HEALTH_WARN
4 pgs stuck unclean
recovery 3486/58726035 objects degraded (0.006%)
recovery 420024/58726035 objects misplaced (0.715%)
noscrub,nodeep-scrub flag(s) set
 monmap e9: 5 mons at 
{ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
election epoch 478, quorum 0,1,2,3,4 ceph1,ceph2,ceph3,ceph4,ceph5
 osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
flags noscrub,nodeep-scrub
  pgmap v9969059: 320 pgs, 3 pools, 4830 GB data, 19116 kobjects
15150 GB used, 40722 GB / 55872 GB avail
3486/58726035 objects degraded (0.006%)
420024/58726035 objects misplaced (0.715%)
 316 active+clean
   4 active+remapped

# ceph osd df
ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  
 0 1.28899  1.0  3724G  1696G  2027G 45.56 1.68 
 1 1.57899  1.0  3724G  1705G  2018G 45.80 1.69 
 2 1.68900  1.0  3724G  1794G  1929G 48.19 1.78 
 3 6.78499  1.0  7450G  1239G  6210G 16.64 0.61 
 4 8.3  1.0  7450G  1226G  6223G 16.46 0.61 
 5 9.51500  1.0  7450G  1237G  6212G 16.61 0.61 
 6 7.66499  1.0  7450G  1263G  6186G 16.96 0.63 
 7 9.75499  1.0  7450G  2493G  4956G 33.47 1.23 
 8 9.32999  1.0  7450G  2491G  4958G 33.44 1.23 
  TOTAL 55872G 15150G 40722G 27.12  
MIN/MAX VAR: 0.61/1.78  STDDEV: 13.54


As you can see now, we are using 4830 GB data BUT raw used is only 15150 GB or 
as said the other way, we have now 40722 GB free. You can see the change on the 
%USE of the osds. For me this looks like there is some data lost, since ceph 
did not do any backfill or other operation. That’s the problem...


> Am 09.01.2017 um 21:55 schrieb Christian Wuerdig 
> :
> 
> 
> 
> On Tue, Jan 10, 2017 at 8:23 AM, Marcus Müller  > wrote:
> Hi all,
> 
> Recently I added a new node with new osds to my cluster, which, of course 
> resulted in backfilling. At the end, there are 4 pgs left in the state 4 
> active+remapped and I don’t know what to do. 
> 
> Here is how my cluster looks like currently: 
> 
> ceph -s
>  health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 3586/58734009 objects degraded (0.006%)
> recovery 420074/587

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Brad Hubbard

There is currently a thread about this very issue on the ceph-devel
mailing list (check archives for "PG stuck unclean after
rebalance-by-weight" in the last few days.

Have a read of 
http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
and try bumping choose_total_tries up to 100 to begin with.

On Tue, Jan 10, 2017 at 7:22 AM, Marcus Müller  wrote:
> Trying google with "ceph pg stuck in active and remapped" points to a couple
> of post on this ML typically indicating that it's a problem with the CRUSH
> map and ceph being unable to satisfy the mapping rules. Your ceph -s output
> indicates that your using replication of size 3 in your pools. You also said
> you had a custom CRUSH map - can you post it?
>
>
> I’ve sent the file to you, since I’m not sure if it contains sensitive data.
> Yes I have replication of 3 and I did not customize the map by me.
>
>
> I might be missing something here but I don't quite see how you come to this
> statement. ceph osd df and ceph -s both show 16093 GB used and 39779 GB out
> of 55872 GB available. The sum of the first 3 OSDs used space is, as you
> stated, 6181 GB which is approx 38.4% so quite close to your target of 33%
>
>
> Maybe I have to explain it another way:
>
> Directly after finishing the backfill I received this output:
>
>  health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 1698/58476648 objects degraded (0.003%)
> recovery 418137/58476648 objects misplaced (0.715%)
> noscrub,nodeep-scrub flag(s) set
>  monmap e9: 5 mons at
> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
> election epoch 464, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>  osdmap e3086: 9 osds: 9 up, 9 in; 4 remapped pgs
> flags noscrub,nodeep-scrub
>   pgmap v9928160: 320 pgs, 3 pools, 4809 GB data, 19035 kobjects
> 16093 GB used, 39779 GB / 55872 GB avail
> 1698/58476648 objects degraded (0.003%)
> 418137/58476648 objects misplaced (0.715%)
>  316 active+clean
>4 active+remapped
>   client io 757 kB/s rd, 1 op/s
>
> # ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>  0 1.28899  1.0  3724G  1924G  1799G 51.67 1.79
>  1 1.57899  1.0  3724G  2143G  1580G 57.57 2.00
>  2 1.68900  1.0  3724G  2114G  1609G 56.78 1.97
>  3 6.78499  1.0  7450G  1234G  6215G 16.57 0.58
>  4 8.3  1.0  7450G  1221G  6228G 16.40 0.57
>  5 9.51500  1.0  7450G  1232G  6217G 16.54 0.57
>  6 7.66499  1.0  7450G  1258G  6191G 16.89 0.59
>  7 9.75499  1.0  7450G  2482G  4967G 33.33 1.16
>  8 9.32999  1.0  7450G  2480G  4969G 33.30 1.16
>   TOTAL 55872G 16093G 39779G 28.80
> MIN/MAX VAR: 0.57/2.00  STDDEV: 17.54
>
> Here we can see, that the cluster is using 4809 GB data and has raw used
> 16093GB. Or the other way, only 39779G available.
>
> Two days later I saw:
>
>  health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 3486/58726035 objects degraded (0.006%)
> recovery 420024/58726035 objects misplaced (0.715%)
> noscrub,nodeep-scrub flag(s) set
>  monmap e9: 5 mons at
> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
> election epoch 478, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
> flags noscrub,nodeep-scrub
>   pgmap v9969059: 320 pgs, 3 pools, 4830 GB data, 19116 kobjects
> 15150 GB used, 40722 GB / 55872 GB avail
> 3486/58726035 objects degraded (0.006%)
> 420024/58726035 objects misplaced (0.715%)
>  316 active+clean
>4 active+remapped
>
> # ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>  0 1.28899  1.0  3724G  1696G  2027G 45.56 1.68
>  1 1.57899  1.0  3724G  1705G  2018G 45.80 1.69
>  2 1.68900  1.0  3724G  1794G  1929G 48.19 1.78
>  3 6.78499  1.0  7450G  1239G  6210G 16.64 0.61
>  4 8.3  1.0  7450G  1226G  6223G 16.46 0.61
>  5 9.51500  1.0  7450G  1237G  6212G 16.61 0.61
>  6 7.66499  1.0  7450G  1263G  6186G 16.96 0.63
>  7 9.75499  1.0  7450G  2493G  4956G 33.47 1.23
>  8 9.32999  1.0  7450G  2491G  4958G 33.44 1.23
>   TOTAL 55872G 15150G 40722G 27.12
> MIN/MAX VAR: 0.61/1.78  STDDEV: 13.54
>
>
> As you can see now, we are using 4830 GB data BUT raw used is only 15150 GB
> or as said the other way, we have now 40722 GB free. You can see the change
> on the %USE of the osds. For me this looks like there is some data lost,
> since ceph did not do any backfill or other operation. That’s the problem...
>
>
> Am 09.01.2017 um 21:55 schrieb Christian Wuerdig
> :
>
>
>
> On Tue, Jan 10, 2017 at 8:23 AM, Marc

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Christian Wuerdig

On Tue, Jan 10, 2017 at 10:22 AM, Marcus Müller 
wrote:

> Trying google with "ceph pg stuck in active and remapped" points to a
> couple of post on this ML typically indicating that it's a problem with the
> CRUSH map and ceph being unable to satisfy the mapping rules. Your ceph -s
> output indicates that your using replication of size 3 in your pools. You
> also said you had a custom CRUSH map - can you post it?
>
>
> I’ve sent the file to you, since I’m not sure if it contains sensitive
> data. Yes I have replication of 3 and I did not customize the map by me.
>

I received your map but I'm not familiar enough with the details to give
any particular advise on this - I just suggested to post your map in case
someone more familiar with the CRUSH details might be able to spot
something. Brad just provided a pointer so that would be useful to try.


>
>
> I might be missing something here but I don't quite see how you come to
> this statement. ceph osd df and ceph -s both show 16093 GB used and 39779
> GB out of 55872 GB available. The sum of the first 3 OSDs used space is, as
> you stated, 6181 GB which is approx 38.4% so quite close to your target of
> 33%
>
>
> Maybe I have to explain it another way:
>
> Directly after finishing the backfill I received this output:
>
>  health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 1698/58476648 objects degraded (0.003%)
> recovery 418137/58476648 objects misplaced (0.715%)
> noscrub,nodeep-scrub flag(s) set
>  monmap e9: 5 mons at {ceph1=192.168.10.3:6789/0,
> ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,
> ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
> election epoch 464, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>  osdmap e3086: 9 osds: 9 up, 9 in; 4 remapped pgs
> flags noscrub,nodeep-scrub
>   pgmap v9928160: 320 pgs, 3 pools, 4809 GB data, 19035 kobjects
> 16093 GB used, 39779 GB / 55872 GB avail
> 1698/58476648 objects degraded (0.003%)
> 418137/58476648 objects misplaced (0.715%)
>  316 active+clean
>4 active+remapped
>   client io 757 kB/s rd, 1 op/s
>
> # ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>  0 1.28899  1.0  3724G  1924G  1799G 51.67 1.79
>  1 1.57899  1.0  3724G  2143G  1580G 57.57 2.00
>  2 1.68900  1.0  3724G  2114G  1609G 56.78 1.97
>  3 6.78499  1.0  7450G  1234G  6215G 16.57 0.58
>  4 8.3  1.0  7450G  1221G  6228G 16.40 0.57
>  5 9.51500  1.0  7450G  1232G  6217G 16.54 0.57
>  6 7.66499  1.0  7450G  1258G  6191G 16.89 0.59
>  7 9.75499  1.0  7450G  2482G  4967G 33.33 1.16
>  8 9.32999  1.0  7450G  2480G  4969G 33.30 1.16
>   TOTAL 55872G 16093G 39779G 28.80
> MIN/MAX VAR: 0.57/2.00  STDDEV: 17.54
>
> Here we can see, that the cluster is using 4809 GB data and has raw used
> 16093GB. Or the other way, only 39779G available.
>
> Two days later I saw:
>
>  health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 3486/58726035 objects degraded (0.006%)
> recovery 420024/58726035 objects misplaced (0.715%)
> noscrub,nodeep-scrub flag(s) set
>  monmap e9: 5 mons at {ceph1=192.168.10.3:6789/0,
> ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,
> ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
> election epoch 478, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
> flags noscrub,nodeep-scrub
>   pgmap v9969059: 320 pgs, 3 pools, 4830 GB data, 19116 kobjects
> 15150 GB used, 40722 GB / 55872 GB avail
> 3486/58726035 objects degraded (0.006%)
> 420024/58726035 objects misplaced (0.715%)
>  316 active+clean
>4 active+remapped
>
> # ceph osd df
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>  0 1.28899  1.0  3724G  1696G  2027G 45.56 1.68
>  1 1.57899  1.0  3724G  1705G  2018G 45.80 1.69
>  2 1.68900  1.0  3724G  1794G  1929G 48.19 1.78
>  3 6.78499  1.0  7450G  1239G  6210G 16.64 0.61
>  4 8.3  1.0  7450G  1226G  6223G 16.46 0.61
>  5 9.51500  1.0  7450G  1237G  6212G 16.61 0.61
>  6 7.66499  1.0  7450G  1263G  6186G 16.96 0.63
>  7 9.75499  1.0  7450G  2493G  4956G 33.47 1.23
>  8 9.32999  1.0  7450G  2491G  4958G 33.44 1.23
>   TOTAL 55872G 15150G 40722G 27.12
> MIN/MAX VAR: 0.61/1.78  STDDEV: 13.54
>
>
> As you can see now, we are using 4830 GB data BUT raw used is only 15150
> GB or as said the other way, we have now 40722 GB free. You can see the
> change on the %USE of the osds. For me this looks like there is some data
> lost, since ceph did not do any backfill or other operation. That’s the
> problem...
>
>
Ok that output is indeed a bit different. However as you should note the
actual data stored in the cluster goes from

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Shinobu Kinjo

> pg 9.7 is stuck unclean for 512936.160212, current state active+remapped, 
> last acting [7,3,0]
> pg 7.84 is stuck unclean for 512623.894574, current state active+remapped, 
> last acting [4,8,1]
> pg 8.1b is stuck unclean for 513164.616377, current state active+remapped, 
> last acting [4,7,2]
> pg 7.7a is stuck unclean for 513162.316328, current state active+remapped, 
> last acting [7,4,2]

Please execute:

for pg in 9.7 7.84 8.1b 7.7a;do ceph pg $pg query; done

Regards,

On Tue, Jan 10, 2017 at 7:31 AM, Christian Wuerdig
 wrote:
>
>
> On Tue, Jan 10, 2017 at 10:22 AM, Marcus Müller 
> wrote:
>>
>> Trying google with "ceph pg stuck in active and remapped" points to a
>> couple of post on this ML typically indicating that it's a problem with the
>> CRUSH map and ceph being unable to satisfy the mapping rules. Your ceph -s
>> output indicates that your using replication of size 3 in your pools. You
>> also said you had a custom CRUSH map - can you post it?
>>
>>
>> I’ve sent the file to you, since I’m not sure if it contains sensitive
>> data. Yes I have replication of 3 and I did not customize the map by me.
>
>
> I received your map but I'm not familiar enough with the details to give any
> particular advise on this - I just suggested to post your map in case
> someone more familiar with the CRUSH details might be able to spot
> something. Brad just provided a pointer so that would be useful to try.
>
>>
>>
>>
>> I might be missing something here but I don't quite see how you come to
>> this statement. ceph osd df and ceph -s both show 16093 GB used and 39779 GB
>> out of 55872 GB available. The sum of the first 3 OSDs used space is, as you
>> stated, 6181 GB which is approx 38.4% so quite close to your target of 33%
>>
>>
>> Maybe I have to explain it another way:
>>
>> Directly after finishing the backfill I received this output:
>>
>>  health HEALTH_WARN
>> 4 pgs stuck unclean
>> recovery 1698/58476648 objects degraded (0.003%)
>> recovery 418137/58476648 objects misplaced (0.715%)
>> noscrub,nodeep-scrub flag(s) set
>>  monmap e9: 5 mons at
>> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>> election epoch 464, quorum 0,1,2,3,4
>> ceph1,ceph2,ceph3,ceph4,ceph5
>>  osdmap e3086: 9 osds: 9 up, 9 in; 4 remapped pgs
>> flags noscrub,nodeep-scrub
>>   pgmap v9928160: 320 pgs, 3 pools, 4809 GB data, 19035 kobjects
>> 16093 GB used, 39779 GB / 55872 GB avail
>> 1698/58476648 objects degraded (0.003%)
>> 418137/58476648 objects misplaced (0.715%)
>>  316 active+clean
>>4 active+remapped
>>   client io 757 kB/s rd, 1 op/s
>>
>> # ceph osd df
>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>>  0 1.28899  1.0  3724G  1924G  1799G 51.67 1.79
>>  1 1.57899  1.0  3724G  2143G  1580G 57.57 2.00
>>  2 1.68900  1.0  3724G  2114G  1609G 56.78 1.97
>>  3 6.78499  1.0  7450G  1234G  6215G 16.57 0.58
>>  4 8.3  1.0  7450G  1221G  6228G 16.40 0.57
>>  5 9.51500  1.0  7450G  1232G  6217G 16.54 0.57
>>  6 7.66499  1.0  7450G  1258G  6191G 16.89 0.59
>>  7 9.75499  1.0  7450G  2482G  4967G 33.33 1.16
>>  8 9.32999  1.0  7450G  2480G  4969G 33.30 1.16
>>   TOTAL 55872G 16093G 39779G 28.80
>> MIN/MAX VAR: 0.57/2.00  STDDEV: 17.54
>>
>> Here we can see, that the cluster is using 4809 GB data and has raw used
>> 16093GB. Or the other way, only 39779G available.
>>
>> Two days later I saw:
>>
>>  health HEALTH_WARN
>> 4 pgs stuck unclean
>> recovery 3486/58726035 objects degraded (0.006%)
>> recovery 420024/58726035 objects misplaced (0.715%)
>> noscrub,nodeep-scrub flag(s) set
>>  monmap e9: 5 mons at
>> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>> election epoch 478, quorum 0,1,2,3,4
>> ceph1,ceph2,ceph3,ceph4,ceph5
>>  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>> flags noscrub,nodeep-scrub
>>   pgmap v9969059: 320 pgs, 3 pools, 4830 GB data, 19116 kobjects
>> 15150 GB used, 40722 GB / 55872 GB avail
>> 3486/58726035 objects degraded (0.006%)
>> 420024/58726035 objects misplaced (0.715%)
>>  316 active+clean
>>4 active+remapped
>>
>> # ceph osd df
>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>>  0 1.28899  1.0  3724G  1696G  2027G 45.56 1.68
>>  1 1.57899  1.0  3724G  1705G  2018G 45.80 1.69
>>  2 1.68900  1.0  3724G  1794G  1929G 48.19 1.78
>>  3 6.78499  1.0  7450G  1239G  6210G 16.64 0.61
>>  4 8.3  1.0  7450G  1226G  6223G 16.46 0.61
>>  5 9.51500  1.0  7450G  1237G  6212G 16.61 0.61
>>  6 7.66499  1.0  7450G  1263G  6186G 16.

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Shinobu Kinjo

Looking at ``ceph -s`` you originally provided, all OSDs are up.

> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs

But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
like related to ?:

> Ceph1, ceph2 and ceph3 are vms on one physical host

Are those OSDs running on vm instances?

# 9.7
 
>"state": "active+remapped",
>"snap_trimq": "[]",
>"epoch": 3114,
>"up": [
>7,
>3
>],
>"acting": [
>7,
>3,
>0
>],
 

# 7.84
 
>"state": "active+remapped",
>"snap_trimq": "[]",
>"epoch": 3114,
>   "up": [
>4,
>8
>],
>"acting": [
>4,
>8,
>1
>],
 

# 8.1b
 
>"state": "active+remapped",
>"snap_trimq": "[]",
>"epoch": 3114,
>"up": [
>4,
>7
>],
>"acting": [
>4,
>7,
>2
>],
 

# 7.7a
 
>"state": "active+remapped",
>"snap_trimq": "[]",
>"epoch": 3114,
>"up": [
>7,
>4
>],
>"acting": [
>7,
>4,
>2
>],
 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Marcus Müller

All osds are currently up:

 health HEALTH_WARN
4 pgs stuck unclean
recovery 4482/58798254 objects degraded (0.008%)
recovery 420522/58798254 objects misplaced (0.715%)
noscrub,nodeep-scrub flag(s) set
 monmap e9: 5 mons at 
{ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
election epoch 478, quorum 0,1,2,3,4 ceph1,ceph2,ceph3,ceph4,ceph5
 osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
flags noscrub,nodeep-scrub
  pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
15070 GB used, 40801 GB / 55872 GB avail
4482/58798254 objects degraded (0.008%)
420522/58798254 objects misplaced (0.715%)
 316 active+clean
   4 active+remapped
  client io 56601 B/s rd, 45619 B/s wr, 0 op/s

This did not chance for two days or so.


By the way, my ceph osd df now looks like this:

ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR  
 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69 
 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70 
 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69 
 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62 
 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61 
 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62 
 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63 
 7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24 
 8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24 
  TOTAL 55872G 15071G 40801G 26.97  
MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16

As you can see, now osd2 also went down to 45% Use and „lost“ data. But I also 
think this is no problem and ceph just clears everything up after backfilling.


> Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
> 
> Looking at ``ceph -s`` you originally provided, all OSDs are up.
> 
>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
> 
> But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
> like related to ?:
> 
>> Ceph1, ceph2 and ceph3 are vms on one physical host
> 
> Are those OSDs running on vm instances?
> 
> # 9.7
> 
>>   "state": "active+remapped",
>>   "snap_trimq": "[]",
>>   "epoch": 3114,
>>   "up": [
>>   7,
>>   3
>>   ],
>>   "acting": [
>>   7,
>>   3,
>>   0
>>   ],
> 
> 
> # 7.84
> 
>>   "state": "active+remapped",
>>   "snap_trimq": "[]",
>>   "epoch": 3114,
>>  "up": [
>>   4,
>>   8
>>   ],
>>   "acting": [
>>   4,
>>   8,
>>   1
>>   ],
> 
> 
> # 8.1b
> 
>>   "state": "active+remapped",
>>   "snap_trimq": "[]",
>>   "epoch": 3114,
>>   "up": [
>>   4,
>>   7
>>   ],
>>   "acting": [
>>   4,
>>   7,
>>   2
>>   ],
> 
> 
> # 7.7a
> 
>>   "state": "active+remapped",
>>   "snap_trimq": "[]",
>>   "epoch": 3114,
>>   "up": [
>>   7,
>>   4
>>   ],
>>   "acting": [
>>   7,
>>   4,
>>   2
>>   ],
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Shinobu Kinjo

On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller  wrote:
> All osds are currently up:
>
>  health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 4482/58798254 objects degraded (0.008%)
> recovery 420522/58798254 objects misplaced (0.715%)
> noscrub,nodeep-scrub flag(s) set
>  monmap e9: 5 mons at
> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
> election epoch 478, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
> flags noscrub,nodeep-scrub
>   pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
> 15070 GB used, 40801 GB / 55872 GB avail
> 4482/58798254 objects degraded (0.008%)
> 420522/58798254 objects misplaced (0.715%)
>  316 active+clean
>4 active+remapped
>   client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>
> This did not chance for two days or so.
>
>
> By the way, my ceph osd df now looks like this:
>
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>  0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
>  1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
>  2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
>  3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
>  4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
>  5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
>  6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
>  7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24
>  8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24
>   TOTAL 55872G 15071G 40801G 26.97
> MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16
>
> As you can see, now osd2 also went down to 45% Use and „lost“ data. But I
> also think this is no problem and ceph just clears everything up after
> backfilling.
>
>
> Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
>
> Looking at ``ceph -s`` you originally provided, all OSDs are up.
>
> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>
>
> But looking at ``pg query``, OSD.0 / 1 are not up. Are they something

That's not perfectly correct.

OSD.0/1/2 seem to be down.

> like related to ?:
>
> Ceph1, ceph2 and ceph3 are vms on one physical host
>
>
> Are those OSDs running on vm instances?
>
> # 9.7
> 
>
>   "state": "active+remapped",
>   "snap_trimq": "[]",
>   "epoch": 3114,
>   "up": [
>   7,
>   3
>   ],
>   "acting": [
>   7,
>   3,
>   0
>   ],
>
> 
>
> # 7.84
> 
>
>   "state": "active+remapped",
>   "snap_trimq": "[]",
>   "epoch": 3114,
>  "up": [
>   4,
>   8
>   ],
>   "acting": [
>   4,
>   8,
>   1
>   ],
>
> 
>
> # 8.1b
> 
>
>   "state": "active+remapped",
>   "snap_trimq": "[]",
>   "epoch": 3114,
>   "up": [
>   4,
>   7
>   ],
>   "acting": [
>   4,
>   7,
>   2
>   ],
>
> 
>
> # 7.7a
> 
>
>   "state": "active+remapped",
>   "snap_trimq": "[]",
>   "epoch": 3114,
>   "up": [
>   7,
>   4
>   ],
>   "acting": [
>   7,
>   4,
>   2
>   ],
>
> 
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Marcus Müller

> 
> That's not perfectly correct.
> 
> OSD.0/1/2 seem to be down.


Sorry but where do you see this? I think this indicates that they are up:   
osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?


> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
> 
> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller  
> wrote:
>> All osds are currently up:
>> 
>> health HEALTH_WARN
>>4 pgs stuck unclean
>>recovery 4482/58798254 objects degraded (0.008%)
>>recovery 420522/58798254 objects misplaced (0.715%)
>>noscrub,nodeep-scrub flag(s) set
>> monmap e9: 5 mons at
>> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>>election epoch 478, quorum 0,1,2,3,4
>> ceph1,ceph2,ceph3,ceph4,ceph5
>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>>flags noscrub,nodeep-scrub
>>  pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
>>15070 GB used, 40801 GB / 55872 GB avail
>>4482/58798254 objects degraded (0.008%)
>>420522/58798254 objects misplaced (0.715%)
>> 316 active+clean
>>   4 active+remapped
>>  client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>> 
>> This did not chance for two days or so.
>> 
>> 
>> By the way, my ceph osd df now looks like this:
>> 
>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>> 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
>> 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
>> 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
>> 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
>> 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
>> 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
>> 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
>> 7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24
>> 8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24
>>  TOTAL 55872G 15071G 40801G 26.97
>> MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16
>> 
>> As you can see, now osd2 also went down to 45% Use and „lost“ data. But I
>> also think this is no problem and ceph just clears everything up after
>> backfilling.
>> 
>> 
>> Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
>> 
>> Looking at ``ceph -s`` you originally provided, all OSDs are up.
>> 
>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>> 
>> 
>> But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
> 
> That's not perfectly correct.
> 
> OSD.0/1/2 seem to be down.
> 
>> like related to ?:
>> 
>> Ceph1, ceph2 and ceph3 are vms on one physical host
>> 
>> 
>> Are those OSDs running on vm instances?
>> 
>> # 9.7
>> 
>> 
>>  "state": "active+remapped",
>>  "snap_trimq": "[]",
>>  "epoch": 3114,
>>  "up": [
>>  7,
>>  3
>>  ],
>>  "acting": [
>>  7,
>>  3,
>>  0
>>  ],
>> 
>> 
>> 
>> # 7.84
>> 
>> 
>>  "state": "active+remapped",
>>  "snap_trimq": "[]",
>>  "epoch": 3114,
>> "up": [
>>  4,
>>  8
>>  ],
>>  "acting": [
>>  4,
>>  8,
>>  1
>>  ],
>> 
>> 
>> 
>> # 8.1b
>> 
>> 
>>  "state": "active+remapped",
>>  "snap_trimq": "[]",
>>  "epoch": 3114,
>>  "up": [
>>  4,
>>  7
>>  ],
>>  "acting": [
>>  4,
>>  7,
>>  2
>>  ],
>> 
>> 
>> 
>> # 7.7a
>> 
>> 
>>  "state": "active+remapped",
>>  "snap_trimq": "[]",
>>  "epoch": 3114,
>>  "up": [
>>  7,
>>  4
>>  ],
>>  "acting": [
>>  7,
>>  4,
>>  2
>>  ],
>> 
>> 
>> 
>> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Shinobu Kinjo

e.g.,
OSD7 / 3 / 0 are in the same acting set. They should be up, if they
are properly running.

# 9.7
 
>"up": [
>7,
>3
>],
>"acting": [
>7,
>3,
>0
>],
 

Here is an example:

  "up": [
1,
0,
2
  ],
  "acting": [
1,
0,
2
   ],

Regards,


On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller  wrote:
>>
>> That's not perfectly correct.
>>
>> OSD.0/1/2 seem to be down.
>
>
> Sorry but where do you see this? I think this indicates that they are up:   
> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
>
>
>> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
>>
>> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller  
>> wrote:
>>> All osds are currently up:
>>>
>>> health HEALTH_WARN
>>>4 pgs stuck unclean
>>>recovery 4482/58798254 objects degraded (0.008%)
>>>recovery 420522/58798254 objects misplaced (0.715%)
>>>noscrub,nodeep-scrub flag(s) set
>>> monmap e9: 5 mons at
>>> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>>>election epoch 478, quorum 0,1,2,3,4
>>> ceph1,ceph2,ceph3,ceph4,ceph5
>>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>>>flags noscrub,nodeep-scrub
>>>  pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
>>>15070 GB used, 40801 GB / 55872 GB avail
>>>4482/58798254 objects degraded (0.008%)
>>>420522/58798254 objects misplaced (0.715%)
>>> 316 active+clean
>>>   4 active+remapped
>>>  client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>>>
>>> This did not chance for two days or so.
>>>
>>>
>>> By the way, my ceph osd df now looks like this:
>>>
>>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>>> 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
>>> 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
>>> 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
>>> 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
>>> 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
>>> 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
>>> 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
>>> 7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24
>>> 8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24
>>>  TOTAL 55872G 15071G 40801G 26.97
>>> MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16
>>>
>>> As you can see, now osd2 also went down to 45% Use and „lost“ data. But I
>>> also think this is no problem and ceph just clears everything up after
>>> backfilling.
>>>
>>>
>>> Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
>>>
>>> Looking at ``ceph -s`` you originally provided, all OSDs are up.
>>>
>>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>>>
>>>
>>> But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
>>
>> That's not perfectly correct.
>>
>> OSD.0/1/2 seem to be down.
>>
>>> like related to ?:
>>>
>>> Ceph1, ceph2 and ceph3 are vms on one physical host
>>>
>>>
>>> Are those OSDs running on vm instances?
>>>
>>> # 9.7
>>> 
>>>
>>>  "state": "active+remapped",
>>>  "snap_trimq": "[]",
>>>  "epoch": 3114,
>>>  "up": [
>>>  7,
>>>  3
>>>  ],
>>>  "acting": [
>>>  7,
>>>  3,
>>>  0
>>>  ],
>>>
>>> 
>>>
>>> # 7.84
>>> 
>>>
>>>  "state": "active+remapped",
>>>  "snap_trimq": "[]",
>>>  "epoch": 3114,
>>> "up": [
>>>  4,
>>>  8
>>>  ],
>>>  "acting": [
>>>  4,
>>>  8,
>>>  1
>>>  ],
>>>
>>> 
>>>
>>> # 8.1b
>>> 
>>>
>>>  "state": "active+remapped",
>>>  "snap_trimq": "[]",
>>>  "epoch": 3114,
>>>  "up": [
>>>  4,
>>>  7
>>>  ],
>>>  "acting": [
>>>  4,
>>>  7,
>>>  2
>>>  ],
>>>
>>> 
>>>
>>> # 7.7a
>>> 
>>>
>>>  "state": "active+remapped",
>>>  "snap_trimq": "[]",
>>>  "epoch": 3114,
>>>  "up": [
>>>  7,
>>>  4
>>>  ],
>>>  "acting": [
>>>  7,
>>>  4,
>>>  2
>>>  ],
>>>
>>> 
>>>
>>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-09 Thread Marcus Müller

Ok, i understand but how can I debug why they are not running as they should? 
For me I thought everything is fine because ceph -s said they are up and 
running. 

I would think of a problem with the crush map. 

> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
> 
> e.g.,
> OSD7 / 3 / 0 are in the same acting set. They should be up, if they
> are properly running.
> 
> # 9.7
> 
>>   "up": [
>>   7,
>>   3
>>   ],
>>   "acting": [
>>   7,
>>   3,
>>   0
>>   ],
> 
> 
> Here is an example:
> 
>  "up": [
>1,
>0,
>2
>  ],
>  "acting": [
>1,
>0,
>2
>   ],
> 
> Regards,
> 
> 
> On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller  
> wrote:
>>> 
>>> That's not perfectly correct.
>>> 
>>> OSD.0/1/2 seem to be down.
>> 
>> 
>> Sorry but where do you see this? I think this indicates that they are up:   
>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
>> 
>> 
>>> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
>>> 
>>> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller  
>>> wrote:
 All osds are currently up:
 
health HEALTH_WARN
   4 pgs stuck unclean
   recovery 4482/58798254 objects degraded (0.008%)
   recovery 420522/58798254 objects misplaced (0.715%)
   noscrub,nodeep-scrub flag(s) set
monmap e9: 5 mons at
 {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
   election epoch 478, quorum 0,1,2,3,4
 ceph1,ceph2,ceph3,ceph4,ceph5
osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
   flags noscrub,nodeep-scrub
 pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
   15070 GB used, 40801 GB / 55872 GB avail
   4482/58798254 objects degraded (0.008%)
   420522/58798254 objects misplaced (0.715%)
316 active+clean
  4 active+remapped
 client io 56601 B/s rd, 45619 B/s wr, 0 op/s
 
 This did not chance for two days or so.
 
 
 By the way, my ceph osd df now looks like this:
 
 ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
 7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24
 8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24
 TOTAL 55872G 15071G 40801G 26.97
 MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16
 
 As you can see, now osd2 also went down to 45% Use and „lost“ data. But I
 also think this is no problem and ceph just clears everything up after
 backfilling.
 
 
 Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
 
 Looking at ``ceph -s`` you originally provided, all OSDs are up.
 
 osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
 
 
 But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
>>> 
>>> That's not perfectly correct.
>>> 
>>> OSD.0/1/2 seem to be down.
>>> 
 like related to ?:
 
 Ceph1, ceph2 and ceph3 are vms on one physical host
 
 
 Are those OSDs running on vm instances?
 
 # 9.7
 
 
 "state": "active+remapped",
 "snap_trimq": "[]",
 "epoch": 3114,
 "up": [
 7,
 3
 ],
 "acting": [
 7,
 3,
 0
 ],
 
 
 
 # 7.84
 
 
 "state": "active+remapped",
 "snap_trimq": "[]",
 "epoch": 3114,
 "up": [
 4,
 8
 ],
 "acting": [
 4,
 8,
 1
 ],
 
 
 
 # 8.1b
 
 
 "state": "active+remapped",
 "snap_trimq": "[]",
 "epoch": 3114,
 "up": [
 4,
 7
 ],
 "acting": [
 4,
 7,
 2
 ],
 
 
 
 # 7.7a
 
 
 "state": "active+remapped",
 "snap_trimq": "[]",
 "epoch": 3114,
 "up": [
 7,
 4
 ],
 "acting": [
 7,
 4,
 2
 ],
 
 
 
 
>> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-10 Thread Samuel Just

Shinobu isn't correct, you have 9/9 osds up and running.  up does not
equal acting because crush is having trouble fulfilling the weights in
your crushmap and the acting set is being padded out with an extra osd
which happens to have the data to keep you up to the right number of
replicas.  Please refer back to Brad's post.
-Sam

On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller  wrote:
> Ok, i understand but how can I debug why they are not running as they should? 
> For me I thought everything is fine because ceph -s said they are up and 
> running.
>
> I would think of a problem with the crush map.
>
>> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
>>
>> e.g.,
>> OSD7 / 3 / 0 are in the same acting set. They should be up, if they
>> are properly running.
>>
>> # 9.7
>> 
>>>   "up": [
>>>   7,
>>>   3
>>>   ],
>>>   "acting": [
>>>   7,
>>>   3,
>>>   0
>>>   ],
>> 
>>
>> Here is an example:
>>
>>  "up": [
>>1,
>>0,
>>2
>>  ],
>>  "acting": [
>>1,
>>0,
>>2
>>   ],
>>
>> Regards,
>>
>>
>> On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller  
>> wrote:

 That's not perfectly correct.

 OSD.0/1/2 seem to be down.
>>>
>>>
>>> Sorry but where do you see this? I think this indicates that they are up:   
>>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
>>>
>>>
 Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :

 On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller  
 wrote:
> All osds are currently up:
>
>health HEALTH_WARN
>   4 pgs stuck unclean
>   recovery 4482/58798254 objects degraded (0.008%)
>   recovery 420522/58798254 objects misplaced (0.715%)
>   noscrub,nodeep-scrub flag(s) set
>monmap e9: 5 mons at
> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>   election epoch 478, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>   flags noscrub,nodeep-scrub
> pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
>   15070 GB used, 40801 GB / 55872 GB avail
>   4482/58798254 objects degraded (0.008%)
>   420522/58798254 objects misplaced (0.715%)
>316 active+clean
>  4 active+remapped
> client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>
> This did not chance for two days or so.
>
>
> By the way, my ceph osd df now looks like this:
>
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
> 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
> 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
> 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
> 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
> 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
> 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
> 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
> 7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24
> 8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24
> TOTAL 55872G 15071G 40801G 26.97
> MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16
>
> As you can see, now osd2 also went down to 45% Use and „lost“ data. But I
> also think this is no problem and ceph just clears everything up after
> backfilling.
>
>
> Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
>
> Looking at ``ceph -s`` you originally provided, all OSDs are up.
>
> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>
>
> But looking at ``pg query``, OSD.0 / 1 are not up. Are they something

 That's not perfectly correct.

 OSD.0/1/2 seem to be down.

> like related to ?:
>
> Ceph1, ceph2 and ceph3 are vms on one physical host
>
>
> Are those OSDs running on vm instances?
>
> # 9.7
> 
>
> "state": "active+remapped",
> "snap_trimq": "[]",
> "epoch": 3114,
> "up": [
> 7,
> 3
> ],
> "acting": [
> 7,
> 3,
> 0
> ],
>
> 
>
> # 7.84
> 
>
> "state": "active+remapped",
> "snap_trimq": "[]",
> "epoch": 3114,
> "up": [
> 4,
> 8
> ],
> "acting": [
> 4,
> 8,
> 1
> ],
>
> 
>
> # 8.1b
> 
>
> "state": "active+remapped",
> "snap_trimq": "[]",
> "epoch": 3114,
> "up": [
> 4,
> 7
> ],
> "acting": [
> 4,
> 7,
> 2
> ],
>
> 
>
> # 7.7a
> 
>
> "state": "active+remapped",
> "snap_trimq": "[]",
> "epoch": 3114,
> "up": [
> 7,
> 4
> ],
> "acting": [
> 7,
> 4,
> 2
> ],
>
> 
>
>
>>>
>
> ___
> ceph-us

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-10 Thread Marcus Müller

Ok, thanks. Then I will change the tunables.

As far as I see, this would already help me: ceph osd crush tunables bobtail

Even if we run ceph hammer this would work according to the documentation, am I 
right? 

And: I’m using librados for our clients (hammer too) could this change create 
problems (even on older kernels)?


> Am 10.01.2017 um 17:50 schrieb Samuel Just :
> 
> Shinobu isn't correct, you have 9/9 osds up and running.  up does not
> equal acting because crush is having trouble fulfilling the weights in
> your crushmap and the acting set is being padded out with an extra osd
> which happens to have the data to keep you up to the right number of
> replicas.  Please refer back to Brad's post.
> -Sam
> 
>> On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller  
>> wrote:
>> Ok, i understand but how can I debug why they are not running as they 
>> should? For me I thought everything is fine because ceph -s said they are up 
>> and running.
>> 
>> I would think of a problem with the crush map.
>> 
>>> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
>>> 
>>> e.g.,
>>> OSD7 / 3 / 0 are in the same acting set. They should be up, if they
>>> are properly running.
>>> 
>>> # 9.7
>>> 
 "up": [
 7,
 3
 ],
 "acting": [
 7,
 3,
 0
 ],
>>> 
>>> 
>>> Here is an example:
>>> 
>>> "up": [
>>>  1,
>>>  0,
>>>  2
>>> ],
>>> "acting": [
>>>  1,
>>>  0,
>>>  2
>>> ],
>>> 
>>> Regards,
>>> 
>>> 
>>> On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller  
>>> wrote:
> 
> That's not perfectly correct.
> 
> OSD.0/1/2 seem to be down.
 
 
 Sorry but where do you see this? I think this indicates that they are up:  
  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
 
 
> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
> 
> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller  
> wrote:
>> All osds are currently up:
>> 
>>  health HEALTH_WARN
>> 4 pgs stuck unclean
>> recovery 4482/58798254 objects degraded (0.008%)
>> recovery 420522/58798254 objects misplaced (0.715%)
>> noscrub,nodeep-scrub flag(s) set
>>  monmap e9: 5 mons at
>> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>> election epoch 478, quorum 0,1,2,3,4
>> ceph1,ceph2,ceph3,ceph4,ceph5
>>  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>> flags noscrub,nodeep-scrub
>>   pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
>> 15070 GB used, 40801 GB / 55872 GB avail
>> 4482/58798254 objects degraded (0.008%)
>> 420522/58798254 objects misplaced (0.715%)
>>  316 active+clean
>>4 active+remapped
>> client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>> 
>> This did not chance for two days or so.
>> 
>> 
>> By the way, my ceph osd df now looks like this:
>> 
>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>> 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
>> 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
>> 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
>> 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
>> 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
>> 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
>> 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
>> 7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24
>> 8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24
>>   TOTAL 55872G 15071G 40801G 26.97
>> MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16
>> 
>> As you can see, now osd2 also went down to 45% Use and „lost“ data. But I
>> also think this is no problem and ceph just clears everything up after
>> backfilling.
>> 
>> 
>> Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
>> 
>> Looking at ``ceph -s`` you originally provided, all OSDs are up.
>> 
>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>> 
>> 
>> But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
> 
> That's not perfectly correct.
> 
> OSD.0/1/2 seem to be down.
> 
>> like related to ?:
>> 
>> Ceph1, ceph2 and ceph3 are vms on one physical host
>> 
>> 
>> Are those OSDs running on vm instances?
>> 
>> # 9.7
>> 
>> 
>> "state": "active+remapped",
>> "snap_trimq": "[]",
>> "epoch": 3114,
>> "up": [
>>   7,
>>   3
>> ],
>> "acting": [
>>   7,
>>   3,
>>   0
>> ],
>> 
>> 
>> 
>> # 7.84
>> 
>> 
>> "state": "active+remapped",
>> "snap_trimq": "[]",
>> "epoch": 3114,
>> "up": [
>>   4,
>>   8
>> ],
>> "acting": [
>>   4,
>>   8,
>>   1
>> ],
>> 
>> 
>> 
>> # 8.1b
>> 
>> 
>> "

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-10 Thread Marcus Müller

Hi Sam,

another idea: I have two HDDs here and already wanted to add them to ceph5, so 
that I would need a new crush map. Could this problem be solved by doing this?


> Am 10.01.2017 um 17:50 schrieb Samuel Just :
> 
> Shinobu isn't correct, you have 9/9 osds up and running.  up does not
> equal acting because crush is having trouble fulfilling the weights in
> your crushmap and the acting set is being padded out with an extra osd
> which happens to have the data to keep you up to the right number of
> replicas.  Please refer back to Brad's post.
> -Sam
> 
> On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller  
> wrote:
>> Ok, i understand but how can I debug why they are not running as they 
>> should? For me I thought everything is fine because ceph -s said they are up 
>> and running.
>> 
>> I would think of a problem with the crush map.
>> 
>>> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
>>> 
>>> e.g.,
>>> OSD7 / 3 / 0 are in the same acting set. They should be up, if they
>>> are properly running.
>>> 
>>> # 9.7
>>> 
  "up": [
  7,
  3
  ],
  "acting": [
  7,
  3,
  0
  ],
>>> 
>>> 
>>> Here is an example:
>>> 
>>> "up": [
>>>   1,
>>>   0,
>>>   2
>>> ],
>>> "acting": [
>>>   1,
>>>   0,
>>>   2
>>>  ],
>>> 
>>> Regards,
>>> 
>>> 
>>> On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller  
>>> wrote:
> 
> That's not perfectly correct.
> 
> OSD.0/1/2 seem to be down.
 
 
 Sorry but where do you see this? I think this indicates that they are up:  
  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
 
 
> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
> 
> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller  
> wrote:
>> All osds are currently up:
>> 
>>   health HEALTH_WARN
>>  4 pgs stuck unclean
>>  recovery 4482/58798254 objects degraded (0.008%)
>>  recovery 420522/58798254 objects misplaced (0.715%)
>>  noscrub,nodeep-scrub flag(s) set
>>   monmap e9: 5 mons at
>> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>>  election epoch 478, quorum 0,1,2,3,4
>> ceph1,ceph2,ceph3,ceph4,ceph5
>>   osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>>  flags noscrub,nodeep-scrub
>>pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
>>  15070 GB used, 40801 GB / 55872 GB avail
>>  4482/58798254 objects degraded (0.008%)
>>  420522/58798254 objects misplaced (0.715%)
>>   316 active+clean
>> 4 active+remapped
>> client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>> 
>> This did not chance for two days or so.
>> 
>> 
>> By the way, my ceph osd df now looks like this:
>> 
>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>> 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
>> 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
>> 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
>> 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
>> 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
>> 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
>> 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
>> 7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24
>> 8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24
>>TOTAL 55872G 15071G 40801G 26.97
>> MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16
>> 
>> As you can see, now osd2 also went down to 45% Use and „lost“ data. But I
>> also think this is no problem and ceph just clears everything up after
>> backfilling.
>> 
>> 
>> Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
>> 
>> Looking at ``ceph -s`` you originally provided, all OSDs are up.
>> 
>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>> 
>> 
>> But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
> 
> That's not perfectly correct.
> 
> OSD.0/1/2 seem to be down.
> 
>> like related to ?:
>> 
>> Ceph1, ceph2 and ceph3 are vms on one physical host
>> 
>> 
>> Are those OSDs running on vm instances?
>> 
>> # 9.7
>> 
>> 
>> "state": "active+remapped",
>> "snap_trimq": "[]",
>> "epoch": 3114,
>> "up": [
>>7,
>>3
>> ],
>> "acting": [
>>7,
>>3,
>>0
>> ],
>> 
>> 
>> 
>> # 7.84
>> 
>> 
>> "state": "active+remapped",
>> "snap_trimq": "[]",
>> "epoch": 3114,
>> "up": [
>>4,
>>8
>> ],
>> "acting": [
>>4,
>>8,
>>1
>> ],
>> 
>> 
>> 
>> # 8.1b
>> 
>> 
>> "state": "active+remapped",
>> "snap_trimq": "[]",
>> "epoch": 3114,
>> "up": [
>>4,
>>7
>>>

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-10 Thread Shinobu Kinjo

Yeah, Sam is correct. I've not looked at crushmap. But I should have
noticed what troublesome is with looking at `ceph osd tree`. That's my
bad, sorry for that.

Again please refer to:

http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/

Regards,


On Wed, Jan 11, 2017 at 1:50 AM, Samuel Just  wrote:
> Shinobu isn't correct, you have 9/9 osds up and running.  up does not
> equal acting because crush is having trouble fulfilling the weights in
> your crushmap and the acting set is being padded out with an extra osd
> which happens to have the data to keep you up to the right number of
> replicas.  Please refer back to Brad's post.
> -Sam
>
> On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller  
> wrote:
>> Ok, i understand but how can I debug why they are not running as they 
>> should? For me I thought everything is fine because ceph -s said they are up 
>> and running.
>>
>> I would think of a problem with the crush map.
>>
>>> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
>>>
>>> e.g.,
>>> OSD7 / 3 / 0 are in the same acting set. They should be up, if they
>>> are properly running.
>>>
>>> # 9.7
>>> 
   "up": [
   7,
   3
   ],
   "acting": [
   7,
   3,
   0
   ],
>>> 
>>>
>>> Here is an example:
>>>
>>>  "up": [
>>>1,
>>>0,
>>>2
>>>  ],
>>>  "acting": [
>>>1,
>>>0,
>>>2
>>>   ],
>>>
>>> Regards,
>>>
>>>
>>> On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller  
>>> wrote:
>
> That's not perfectly correct.
>
> OSD.0/1/2 seem to be down.


 Sorry but where do you see this? I think this indicates that they are up:  
  osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?


> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
>
> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller  
> wrote:
>> All osds are currently up:
>>
>>health HEALTH_WARN
>>   4 pgs stuck unclean
>>   recovery 4482/58798254 objects degraded (0.008%)
>>   recovery 420522/58798254 objects misplaced (0.715%)
>>   noscrub,nodeep-scrub flag(s) set
>>monmap e9: 5 mons at
>> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>>   election epoch 478, quorum 0,1,2,3,4
>> ceph1,ceph2,ceph3,ceph4,ceph5
>>osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>>   flags noscrub,nodeep-scrub
>> pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
>>   15070 GB used, 40801 GB / 55872 GB avail
>>   4482/58798254 objects degraded (0.008%)
>>   420522/58798254 objects misplaced (0.715%)
>>316 active+clean
>>  4 active+remapped
>> client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>>
>> This did not chance for two days or so.
>>
>>
>> By the way, my ceph osd df now looks like this:
>>
>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>> 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
>> 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
>> 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
>> 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
>> 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
>> 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
>> 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
>> 7 9.75499  1.0  7450G  2497G  4952G 33.52 1.24
>> 8 9.32999  1.0  7450G  2495G  4954G 33.49 1.24
>> TOTAL 55872G 15071G 40801G 26.97
>> MIN/MAX VAR: 0.61/1.70  STDDEV: 13.16
>>
>> As you can see, now osd2 also went down to 45% Use and „lost“ data. But I
>> also think this is no problem and ceph just clears everything up after
>> backfilling.
>>
>>
>> Am 10.01.2017 um 07:29 schrieb Shinobu Kinjo :
>>
>> Looking at ``ceph -s`` you originally provided, all OSDs are up.
>>
>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>>
>>
>> But looking at ``pg query``, OSD.0 / 1 are not up. Are they something
>
> That's not perfectly correct.
>
> OSD.0/1/2 seem to be down.
>
>> like related to ?:
>>
>> Ceph1, ceph2 and ceph3 are vms on one physical host
>>
>>
>> Are those OSDs running on vm instances?
>>
>> # 9.7
>> 
>>
>> "state": "active+remapped",
>> "snap_trimq": "[]",
>> "epoch": 3114,
>> "up": [
>> 7,
>> 3
>> ],
>> "acting": [
>> 7,
>> 3,
>> 0
>> ],
>>
>> 
>>
>> # 7.84
>> 
>>
>> "state": "active+remapped",
>> "snap_trimq": "[]",
>> "epoch": 3114,
>> "up": [
>> 4,
>> 8
>> ],
>> "acting": [
>> 4,
>> 8,
>> 1
>> ],
>>
>> 
>>
>> # 8.1b
>> 
>>
>> "state": "ac

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-10 Thread Marcus Müller

I have to thank you all. You give free support and this already helps me. I’m 
not the one who knows ceph that good, but everyday it’s getting better and 
better ;-)

According to the article Brad posted I have to change the ceph osd crush 
tunables. But there are two questions left as I already wrote:

- According to 
http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables 
 there 
are a few profiles. My needed profile would be BOBTAIL (CRUSH_TUNABLES2) wich 
would set choose_total_tries to 50. For the beginning better than 19. There I 
also see: "You can select a profile on a running cluster with the command: ceph 
osd crush tunables {PROFILE}“. My question on this is: Even if I run hammer, is 
it good and possible to set it to bobtail?

- We can also read: 
  WHICH CLIENT VERSIONS SUPPORT CRUSH_TUNABLES2 
  - v0.55 or later, including bobtail series (v0.56.x) 
  - Linux kernel version v3.9 or later (for the file system and RBD kernel 
clients)

And here my question is: If my clients use librados (version hammer), do I need 
to have this required kernel version on the clients or the ceph nodes? 

I don’t want to have troubles at the end with my clients. Can someone answer me 
this, before I change the settings?


> Am 11.01.2017 um 06:47 schrieb Shinobu Kinjo :
> 
> Yeah, Sam is correct. I've not looked at crushmap. But I should have
> noticed what troublesome is with looking at `ceph osd tree`. That's my
> bad, sorry for that.
> 
> Again please refer to:
> 
> http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
> 
> Regards,
> 
> 
> On Wed, Jan 11, 2017 at 1:50 AM, Samuel Just  wrote:
>> Shinobu isn't correct, you have 9/9 osds up and running.  up does not
>> equal acting because crush is having trouble fulfilling the weights in
>> your crushmap and the acting set is being padded out with an extra osd
>> which happens to have the data to keep you up to the right number of
>> replicas.  Please refer back to Brad's post.
>> -Sam
>> 
>> On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller  
>> wrote:
>>> Ok, i understand but how can I debug why they are not running as they 
>>> should? For me I thought everything is fine because ceph -s said they are 
>>> up and running.
>>> 
>>> I would think of a problem with the crush map.
>>> 
 Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
 
 e.g.,
 OSD7 / 3 / 0 are in the same acting set. They should be up, if they
 are properly running.
 
 # 9.7
 
>  "up": [
>  7,
>  3
>  ],
>  "acting": [
>  7,
>  3,
>  0
>  ],
 
 
 Here is an example:
 
 "up": [
   1,
   0,
   2
 ],
 "acting": [
   1,
   0,
   2
  ],
 
 Regards,
 
 
 On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller  
 wrote:
>> 
>> That's not perfectly correct.
>> 
>> OSD.0/1/2 seem to be down.
> 
> 
> Sorry but where do you see this? I think this indicates that they are up: 
>   osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
> 
> 
>> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
>> 
>> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller 
>>  wrote:
>>> All osds are currently up:
>>> 
>>>   health HEALTH_WARN
>>>  4 pgs stuck unclean
>>>  recovery 4482/58798254 objects degraded (0.008%)
>>>  recovery 420522/58798254 objects misplaced (0.715%)
>>>  noscrub,nodeep-scrub flag(s) set
>>>   monmap e9: 5 mons at
>>> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>>>  election epoch 478, quorum 0,1,2,3,4
>>> ceph1,ceph2,ceph3,ceph4,ceph5
>>>   osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>>>  flags noscrub,nodeep-scrub
>>>pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
>>>  15070 GB used, 40801 GB / 55872 GB avail
>>>  4482/58798254 objects degraded (0.008%)
>>>  420522/58798254 objects misplaced (0.715%)
>>>   316 active+clean
>>> 4 active+remapped
>>> client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>>> 
>>> This did not chance for two days or so.
>>> 
>>> 
>>> By the way, my ceph osd df now looks like this:
>>> 
>>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
>>> 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
>>> 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
>>> 2 1.68900  1.0  3724G  1695G  2028G 45.54 1.69
>>> 3 6.78499  1.0  7450G  1241G  6208G 16.67 0.62
>>> 4 8.3  1.0  7450G  1228G  6221G 16.49 0.61
>>> 5 9.51500  1.0  7450G  1239G  6210G 16.64 0.62
>>> 6 7.66499  1.0  7450G  1265G  6184G 16.99 0.63
>>> 7 9.75499  1.000

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-11 Thread Brad Hubbard

Your current problem has nothing to do with clients and neither does
choose_total_tries.

Try setting just this value to 100 and see if your situation improves.

Ultimately you need to take a good look at your cluster configuration
and how your crush map is configured to deal with that configuration
but start with choose_total_tries as it has the highest probability of
helping your situation. Your clients should not be affected.

Could you explain the reasoning behind having three hosts with one ods
each, one host with two osds and one with four?

You likely need to tweak your crushmap to handle this configuration
better or, preferably, move to a more uniform configuration.


On Wed, Jan 11, 2017 at 5:38 PM, Marcus Müller  wrote:
> I have to thank you all. You give free support and this already helps me.
> I’m not the one who knows ceph that good, but everyday it’s getting better
> and better ;-)
>
> According to the article Brad posted I have to change the ceph osd crush
> tunables. But there are two questions left as I already wrote:
>
> - According to
> http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables there
> are a few profiles. My needed profile would be BOBTAIL (CRUSH_TUNABLES2)
> wich would set choose_total_tries to 50. For the beginning better than 19.
> There I also see: "You can select a profile on a running cluster with the
> command: ceph osd crush tunables {PROFILE}“. My question on this is: Even if
> I run hammer, is it good and possible to set it to bobtail?
>
> - We can also read:
>   WHICH CLIENT VERSIONS SUPPORT CRUSH_TUNABLES2
>   - v0.55 or later, including bobtail series (v0.56.x)
>   - Linux kernel version v3.9 or later (for the file system and RBD kernel
> clients)
>
> And here my question is: If my clients use librados (version hammer), do I
> need to have this required kernel version on the clients or the ceph nodes?
>
> I don’t want to have troubles at the end with my clients. Can someone answer
> me this, before I change the settings?
>
>
> Am 11.01.2017 um 06:47 schrieb Shinobu Kinjo :
>
>
> Yeah, Sam is correct. I've not looked at crushmap. But I should have
> noticed what troublesome is with looking at `ceph osd tree`. That's my
> bad, sorry for that.
>
> Again please refer to:
>
> http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
>
> Regards,
>
>
> On Wed, Jan 11, 2017 at 1:50 AM, Samuel Just  wrote:
>
> Shinobu isn't correct, you have 9/9 osds up and running.  up does not
> equal acting because crush is having trouble fulfilling the weights in
> your crushmap and the acting set is being padded out with an extra osd
> which happens to have the data to keep you up to the right number of
> replicas.  Please refer back to Brad's post.
> -Sam
>
> On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller 
> wrote:
>
> Ok, i understand but how can I debug why they are not running as they
> should? For me I thought everything is fine because ceph -s said they are up
> and running.
>
> I would think of a problem with the crush map.
>
> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
>
> e.g.,
> OSD7 / 3 / 0 are in the same acting set. They should be up, if they
> are properly running.
>
> # 9.7
> 
>
>  "up": [
>  7,
>  3
>  ],
>  "acting": [
>  7,
>  3,
>  0
>  ],
>
> 
>
> Here is an example:
>
> "up": [
>   1,
>   0,
>   2
> ],
> "acting": [
>   1,
>   0,
>   2
>  ],
>
> Regards,
>
>
> On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller 
> wrote:
>
>
> That's not perfectly correct.
>
> OSD.0/1/2 seem to be down.
>
>
>
> Sorry but where do you see this? I think this indicates that they are up:
> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
>
>
> Am 10.01.2017 um 07:50 schrieb Shinobu Kinjo :
>
> On Tue, Jan 10, 2017 at 3:44 PM, Marcus Müller 
> wrote:
>
> All osds are currently up:
>
>   health HEALTH_WARN
>  4 pgs stuck unclean
>  recovery 4482/58798254 objects degraded (0.008%)
>  recovery 420522/58798254 objects misplaced (0.715%)
>  noscrub,nodeep-scrub flag(s) set
>   monmap e9: 5 mons at
> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
>  election epoch 478, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>   osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs
>  flags noscrub,nodeep-scrub
>pgmap v9981077: 320 pgs, 3 pools, 4837 GB data, 19140 kobjects
>  15070 GB used, 40801 GB / 55872 GB avail
>  4482/58798254 objects degraded (0.008%)
>  420522/58798254 objects misplaced (0.715%)
>   316 active+clean
> 4 active+remapped
> client io 56601 B/s rd, 45619 B/s wr, 0 op/s
>
> This did not chance for two days or so.
>
>
> By the way, my ceph osd df now looks like this:
>
> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE  VAR
> 0 1.28899  1.0  3724G  1699G  2024G 45.63 1.69
> 1 1.57899  1.0  3724G  1708G  2015G 45.87 1.70
> 2 1.68900  1.000

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-11 Thread Jens Dueholm Christensen

Hi

I'm the author of the mentioned thread on ceph-devel.

The second to last reply in that thread 
(http://marc.info/?l=ceph-devel&m=148396739308208&w=2 ) mentions what I 
suspected was the cause: 

Improper balance of the entire cluster (2 new nodes had double the capacity of 
the original cluster) and an attempt to reweigh usage caused the stuck+unclean 
PGs because CRUSH could no longer find an OSD to place the PG on once the OSD 
weight became too low.

On IRC, that suspicion was confirmed by Samuel Just (who replied to the thread) 
as indeed beeing the cause.

Marcus Müller:

A very quick look at the output from "ceph osd df" in your first mail shows 
some _very_ strange CRUSH weights:
osd.0 has a CRUSH weight of 1.28889 while osd.5 has a CRUSH weight of 9.51500.

You do realise that the CRUSH weight should be a representation of the OSDs 
capacity in TB, right?

That means that osd.0 should be able to hold 1.2TB data while osd.5 should be 
able to hold 9.5TB data - or at least that how CRUSH sees it - and that's it's 
basis for attempting to balance things out (and fails to do..).

You should change the CRUSH weight to reflect the real capacity of you OSDs 
across all nodes and OSDs.
Mind that this WILL - and I cannot stress this enough! - make a lot of data 
move around!!
It could also kill any client IO unless you lower the amount of allowed 
backfills to run from the same OSDs at once (google it..)

Lowering the CRUSH weight from a high number to a lower number can and WILL 
also raise your used capacity - be carefull you do not end up with a full 
cluster or risk ending up with a single OSD failure causing a full or 
over-capacity cluster.

A good explanation for what CRUSH weight and OSD weight represents for CRUSH 
can be seen here:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-June/040961.html

Once your CRUSH weights are correct you can try and lower the OSD weight a bit 
here and there, but too low a setting can again cause stuck+unclean PGs, but at 
least you will hopefully have a better balance for all your OSDs.

You MIGHT have to raise the tunable choose_total_tries a bit up from 50 once 
you start fiddling with the OSD weight, but for starters I REALLY REALLY 
_REALLY_ recommend you to fix those strange CRUSH weights before trying to do 
anything else but just to make sure you are using a somewhat same set of 
tunables appropriate for the version of Ceph you are running.

Regards,
Jens Dueholm Christensen 
Rambøll Survey IT

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Brad 
Hubbard
Sent: Monday, January 09, 2017 10:51 PM
To: Marcus Müller
Cc: Ceph Users
Subject: Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

There is currently a thread about this very issue on the ceph-devel
mailing list (check archives for "PG stuck unclean after
rebalance-by-weight" in the last few days.

Have a read of 
http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
and try bumping choose_total_tries up to 100 to begin with.

On Tue, Jan 10, 2017 at 7:22 AM, Marcus Müller  wrote:
> Trying google with "ceph pg stuck in active and remapped" points to a couple
> of post on this ML typically indicating that it's a problem with the CRUSH
> map and ceph being unable to satisfy the mapping rules. Your ceph -s output
> indicates that your using replication of size 3 in your pools. You also said
> you had a custom CRUSH map - can you post it?
>
>
> I’ve sent the file to you, since I’m not sure if it contains sensitive data.
> Yes I have replication of 3 and I did not customize the map by me.
>
>
> I might be missing something here but I don't quite see how you come to this
> statement. ceph osd df and ceph -s both show 16093 GB used and 39779 GB out
> of 55872 GB available. The sum of the first 3 OSDs used space is, as you
> stated, 6181 GB which is approx 38.4% so quite close to your target of 33%
>
>
> Maybe I have to explain it another way:
>
> Directly after finishing the backfill I received this output:
>
>  health HEALTH_WARN
> 4 pgs stuck unclean
> recovery 1698/58476648 objects degraded (0.003%)
> recovery 418137/58476648 objects misplaced (0.715%)
> noscrub,nodeep-scrub flag(s) set
>  monmap e9: 5 mons at
> {ceph1=192.168.10.3:6789/0,ceph2=192.168.10.4:6789/0,ceph3=192.168.10.5:6789/0,ceph4=192.168.60.6:6789/0,ceph5=192.168.60.11:6789/0}
> election epoch 464, quorum 0,1,2,3,4
> ceph1,ceph2,ceph3,ceph4,ceph5
>  osdmap e3086: 9 osds: 9 up, 9 in; 4 remapped pgs
> flags noscrub,nodeep-scrub
>   pgmap v9928160: 320 pgs, 3 pools, 4809 GB data, 19035 kobjects
> 16093 GB used, 39779 GB / 55872 GB avail
> 1698/58476648 objects degrade

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-11 Thread Marcus Müller

Ok, thank you. I thought I have to set ceph to a tunables profile. If I’m 
right, then I just have to export the current crush map, edit it and import it 
again, like:

ceph osd getcrushmap -o /tmp/crush
crushtool -i /tmp/crush --set-choose-total-tries 100 -o /tmp/crush.new
ceph osd setcrushmap -i /tmp/crush.new

Is this right or not?

I started this cluster with these 3 nodes and each 3 osds. They are vms. I knew 
that this cluster would expand very big, that’s the reason for my choice for 
ceph. Now I can’t add more HDDs to the vm hypervisor and I want to separate the 
nodes physically too. I bought a new node with these 4 drives and now another 
node with only 2 drives. As I hear now from several people this was not a good 
idea. For this reason, I bought now additional HDDs for the new node, so I have 
two with the same amount of HDDs and size. In the next 1-2 months I will get 
the third physical node and then everything should be fine. But at this time I 
have no other option. 

May it help to solve this problem by adding the 2 new HDDs to the new ceph node?



> Am 11.01.2017 um 12:00 schrieb Brad Hubbard :
> 
> Your current problem has nothing to do with clients and neither does
> choose_total_tries.
> 
> Try setting just this value to 100 and see if your situation improves.
> 
> Ultimately you need to take a good look at your cluster configuration
> and how your crush map is configured to deal with that configuration
> but start with choose_total_tries as it has the highest probability of
> helping your situation. Your clients should not be affected.
> 
> Could you explain the reasoning behind having three hosts with one ods
> each, one host with two osds and one with four?
> 
> You likely need to tweak your crushmap to handle this configuration
> better or, preferably, move to a more uniform configuration.
> 
> 
> On Wed, Jan 11, 2017 at 5:38 PM, Marcus Müller  
> wrote:
>> I have to thank you all. You give free support and this already helps me.
>> I’m not the one who knows ceph that good, but everyday it’s getting better
>> and better ;-)
>> 
>> According to the article Brad posted I have to change the ceph osd crush
>> tunables. But there are two questions left as I already wrote:
>> 
>> - According to
>> http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables there
>> are a few profiles. My needed profile would be BOBTAIL (CRUSH_TUNABLES2)
>> wich would set choose_total_tries to 50. For the beginning better than 19.
>> There I also see: "You can select a profile on a running cluster with the
>> command: ceph osd crush tunables {PROFILE}“. My question on this is: Even if
>> I run hammer, is it good and possible to set it to bobtail?
>> 
>> - We can also read:
>>  WHICH CLIENT VERSIONS SUPPORT CRUSH_TUNABLES2
>>  - v0.55 or later, including bobtail series (v0.56.x)
>>  - Linux kernel version v3.9 or later (for the file system and RBD kernel
>> clients)
>> 
>> And here my question is: If my clients use librados (version hammer), do I
>> need to have this required kernel version on the clients or the ceph nodes?
>> 
>> I don’t want to have troubles at the end with my clients. Can someone answer
>> me this, before I change the settings?
>> 
>> 
>> Am 11.01.2017 um 06:47 schrieb Shinobu Kinjo :
>> 
>> 
>> Yeah, Sam is correct. I've not looked at crushmap. But I should have
>> noticed what troublesome is with looking at `ceph osd tree`. That's my
>> bad, sorry for that.
>> 
>> Again please refer to:
>> 
>> http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
>> 
>> Regards,
>> 
>> 
>> On Wed, Jan 11, 2017 at 1:50 AM, Samuel Just  wrote:
>> 
>> Shinobu isn't correct, you have 9/9 osds up and running.  up does not
>> equal acting because crush is having trouble fulfilling the weights in
>> your crushmap and the acting set is being padded out with an extra osd
>> which happens to have the data to keep you up to the right number of
>> replicas.  Please refer back to Brad's post.
>> -Sam
>> 
>> On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller 
>> wrote:
>> 
>> Ok, i understand but how can I debug why they are not running as they
>> should? For me I thought everything is fine because ceph -s said they are up
>> and running.
>> 
>> I would think of a problem with the crush map.
>> 
>> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
>> 
>> e.g.,
>> OSD7 / 3 / 0 are in the same acting set. They should be up, if they
>> are properly running.
>> 
>> # 9.7
>> 
>> 
>> "up": [
>> 7,
>> 3
>> ],
>> "acting": [
>> 7,
>> 3,
>> 0
>> ],
>> 
>> 
>> 
>> Here is an example:
>> 
>> "up": [
>>  1,
>>  0,
>>  2
>> ],
>> "acting": [
>>  1,
>>  0,
>>  2
>> ],
>> 
>> Regards,
>> 
>> 
>> On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller 
>> wrote:
>> 
>> 
>> That's not perfectly correct.
>> 
>> OSD.0/1/2 seem to be down.
>> 
>> 
>> 
>> Sorry but where do you see this? I think this indicates that they are up:
>> osdmap e3114: 9 osds: 9 up, 9 in; 4 remapped pgs?
>> 
>> 
>> Am 10.01

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-11 Thread Shinobu Kinjo

Please refer to Jens's message.

Regards,

On Wed, Jan 11, 2017 at 8:53 PM, Marcus Müller  wrote:
> Ok, thank you. I thought I have to set ceph to a tunables profile. If I’m 
> right, then I just have to export the current crush map, edit it and import 
> it again, like:
>
> ceph osd getcrushmap -o /tmp/crush
> crushtool -i /tmp/crush --set-choose-total-tries 100 -o /tmp/crush.new
> ceph osd setcrushmap -i /tmp/crush.new
>
> Is this right or not?
>
> I started this cluster with these 3 nodes and each 3 osds. They are vms. I 
> knew that this cluster would expand very big, that’s the reason for my choice 
> for ceph. Now I can’t add more HDDs to the vm hypervisor and I want to 
> separate the nodes physically too. I bought a new node with these 4 drives 
> and now another node with only 2 drives. As I hear now from several people 
> this was not a good idea. For this reason, I bought now additional HDDs for 
> the new node, so I have two with the same amount of HDDs and size. In the 
> next 1-2 months I will get the third physical node and then everything should 
> be fine. But at this time I have no other option.
>
> May it help to solve this problem by adding the 2 new HDDs to the new ceph 
> node?
>
>
>
>> Am 11.01.2017 um 12:00 schrieb Brad Hubbard :
>>
>> Your current problem has nothing to do with clients and neither does
>> choose_total_tries.
>>
>> Try setting just this value to 100 and see if your situation improves.
>>
>> Ultimately you need to take a good look at your cluster configuration
>> and how your crush map is configured to deal with that configuration
>> but start with choose_total_tries as it has the highest probability of
>> helping your situation. Your clients should not be affected.
>>
>> Could you explain the reasoning behind having three hosts with one ods
>> each, one host with two osds and one with four?
>>
>> You likely need to tweak your crushmap to handle this configuration
>> better or, preferably, move to a more uniform configuration.
>>
>>
>> On Wed, Jan 11, 2017 at 5:38 PM, Marcus Müller  
>> wrote:
>>> I have to thank you all. You give free support and this already helps me.
>>> I’m not the one who knows ceph that good, but everyday it’s getting better
>>> and better ;-)
>>>
>>> According to the article Brad posted I have to change the ceph osd crush
>>> tunables. But there are two questions left as I already wrote:
>>>
>>> - According to
>>> http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables there
>>> are a few profiles. My needed profile would be BOBTAIL (CRUSH_TUNABLES2)
>>> wich would set choose_total_tries to 50. For the beginning better than 19.
>>> There I also see: "You can select a profile on a running cluster with the
>>> command: ceph osd crush tunables {PROFILE}“. My question on this is: Even if
>>> I run hammer, is it good and possible to set it to bobtail?
>>>
>>> - We can also read:
>>>  WHICH CLIENT VERSIONS SUPPORT CRUSH_TUNABLES2
>>>  - v0.55 or later, including bobtail series (v0.56.x)
>>>  - Linux kernel version v3.9 or later (for the file system and RBD kernel
>>> clients)
>>>
>>> And here my question is: If my clients use librados (version hammer), do I
>>> need to have this required kernel version on the clients or the ceph nodes?
>>>
>>> I don’t want to have troubles at the end with my clients. Can someone answer
>>> me this, before I change the settings?
>>>
>>>
>>> Am 11.01.2017 um 06:47 schrieb Shinobu Kinjo :
>>>
>>>
>>> Yeah, Sam is correct. I've not looked at crushmap. But I should have
>>> noticed what troublesome is with looking at `ceph osd tree`. That's my
>>> bad, sorry for that.
>>>
>>> Again please refer to:
>>>
>>> http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
>>>
>>> Regards,
>>>
>>>
>>> On Wed, Jan 11, 2017 at 1:50 AM, Samuel Just  wrote:
>>>
>>> Shinobu isn't correct, you have 9/9 osds up and running.  up does not
>>> equal acting because crush is having trouble fulfilling the weights in
>>> your crushmap and the acting set is being padded out with an extra osd
>>> which happens to have the data to keep you up to the right number of
>>> replicas.  Please refer back to Brad's post.
>>> -Sam
>>>
>>> On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller 
>>> wrote:
>>>
>>> Ok, i understand but how can I debug why they are not running as they
>>> should? For me I thought everything is fine because ceph -s said they are up
>>> and running.
>>>
>>> I would think of a problem with the crush map.
>>>
>>> Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
>>>
>>> e.g.,
>>> OSD7 / 3 / 0 are in the same acting set. They should be up, if they
>>> are properly running.
>>>
>>> # 9.7
>>> 
>>>
>>> "up": [
>>> 7,
>>> 3
>>> ],
>>> "acting": [
>>> 7,
>>> 3,
>>> 0
>>> ],
>>>
>>> 
>>>
>>> Here is an example:
>>>
>>> "up": [
>>>  1,
>>>  0,
>>>  2
>>> ],
>>> "acting": [
>>>  1,
>>>  0,
>>>  2
>>> ],
>>>
>>> Regards,
>>>
>>>
>>> On Tue, Jan 10, 2017 at 3:52 PM, Marcus Müller 
>>> wrote:
>>>
>>>
>>

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-11 Thread Marcus Müller

Yes, but everything i want to know is, if my way to change the tunables is 
right or not?



> Am 11.01.2017 um 13:11 schrieb Shinobu Kinjo :
> 
> Please refer to Jens's message.
> 
> Regards,
> 
>> On Wed, Jan 11, 2017 at 8:53 PM, Marcus Müller  
>> wrote:
>> Ok, thank you. I thought I have to set ceph to a tunables profile. If I’m 
>> right, then I just have to export the current crush map, edit it and import 
>> it again, like:
>> 
>> ceph osd getcrushmap -o /tmp/crush
>> crushtool -i /tmp/crush --set-choose-total-tries 100 -o /tmp/crush.new
>> ceph osd setcrushmap -i /tmp/crush.new
>> 
>> Is this right or not?
>> 
>> I started this cluster with these 3 nodes and each 3 osds. They are vms. I 
>> knew that this cluster would expand very big, that’s the reason for my 
>> choice for ceph. Now I can’t add more HDDs to the vm hypervisor and I want 
>> to separate the nodes physically too. I bought a new node with these 4 
>> drives and now another node with only 2 drives. As I hear now from several 
>> people this was not a good idea. For this reason, I bought now additional 
>> HDDs for the new node, so I have two with the same amount of HDDs and size. 
>> In the next 1-2 months I will get the third physical node and then 
>> everything should be fine. But at this time I have no other option.
>> 
>> May it help to solve this problem by adding the 2 new HDDs to the new ceph 
>> node?
>> 
>> 
>> 
>>> Am 11.01.2017 um 12:00 schrieb Brad Hubbard :
>>> 
>>> Your current problem has nothing to do with clients and neither does
>>> choose_total_tries.
>>> 
>>> Try setting just this value to 100 and see if your situation improves.
>>> 
>>> Ultimately you need to take a good look at your cluster configuration
>>> and how your crush map is configured to deal with that configuration
>>> but start with choose_total_tries as it has the highest probability of
>>> helping your situation. Your clients should not be affected.
>>> 
>>> Could you explain the reasoning behind having three hosts with one ods
>>> each, one host with two osds and one with four?
>>> 
>>> You likely need to tweak your crushmap to handle this configuration
>>> better or, preferably, move to a more uniform configuration.
>>> 
>>> 
 On Wed, Jan 11, 2017 at 5:38 PM, Marcus Müller  
 wrote:
 I have to thank you all. You give free support and this already helps me.
 I’m not the one who knows ceph that good, but everyday it’s getting better
 and better ;-)
 
 According to the article Brad posted I have to change the ceph osd crush
 tunables. But there are two questions left as I already wrote:
 
 - According to
 http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables there
 are a few profiles. My needed profile would be BOBTAIL (CRUSH_TUNABLES2)
 wich would set choose_total_tries to 50. For the beginning better than 19.
 There I also see: "You can select a profile on a running cluster with the
 command: ceph osd crush tunables {PROFILE}“. My question on this is: Even 
 if
 I run hammer, is it good and possible to set it to bobtail?
 
 - We can also read:
 WHICH CLIENT VERSIONS SUPPORT CRUSH_TUNABLES2
 - v0.55 or later, including bobtail series (v0.56.x)
 - Linux kernel version v3.9 or later (for the file system and RBD kernel
 clients)
 
 And here my question is: If my clients use librados (version hammer), do I
 need to have this required kernel version on the clients or the ceph nodes?
 
 I don’t want to have troubles at the end with my clients. Can someone 
 answer
 me this, before I change the settings?
 
 
 Am 11.01.2017 um 06:47 schrieb Shinobu Kinjo :
 
 
 Yeah, Sam is correct. I've not looked at crushmap. But I should have
 noticed what troublesome is with looking at `ceph osd tree`. That's my
 bad, sorry for that.
 
 Again please refer to:
 
 http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
 
 Regards,
 
 
 On Wed, Jan 11, 2017 at 1:50 AM, Samuel Just  wrote:
 
 Shinobu isn't correct, you have 9/9 osds up and running.  up does not
 equal acting because crush is having trouble fulfilling the weights in
 your crushmap and the acting set is being padded out with an extra osd
 which happens to have the data to keep you up to the right number of
 replicas.  Please refer back to Brad's post.
 -Sam
 
 On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller 
 wrote:
 
 Ok, i understand but how can I debug why they are not running as they
 should? For me I thought everything is fine because ceph -s said they are 
 up
 and running.
 
 I would think of a problem with the crush map.
 
 Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
 
 e.g.,
 OSD7 / 3 / 0 are in the same acting set. They should be up, if they
 are properly running.
 
 # 9.7

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-11 Thread Jens Dueholm Christensen

Hi Marcus

Please refer to the documentation:
http://docs.ceph.com/docs/master/rados/operations/crush-map/#editing-a-crush-map

I belive your suggestion only modifies the in-memory map and you never get a 
changed version written in the outfile, but it could easily be tested by 
decompiling the new version and checking the clear text version, but why not 
just do as the documentation suggests?


But you really should just set some default sane usable values (depending on 
kernel versions and your clients) and NOT create your own settings, allow the 
cluster to remap after the new profiles has been applied and then change CRUSH 
weights to correct values before you attempt any customization of tunables..


Regards,
Jens Dueholm Christensen 
Rambøll Survey IT

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Marcus 
Müller
Sent: Wednesday, January 11, 2017 2:50 PM
To: Shinobu Kinjo
Cc: Ceph Users
Subject: Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Yes, but everything i want to know is, if my way to change the tunables is 
right or not?



> Am 11.01.2017 um 13:11 schrieb Shinobu Kinjo :
> 
> Please refer to Jens's message.
> 
> Regards,
> 
>> On Wed, Jan 11, 2017 at 8:53 PM, Marcus Müller  
>> wrote:
>> Ok, thank you. I thought I have to set ceph to a tunables profile. If I’m 
>> right, then I just have to export the current crush map, edit it and import 
>> it again, like:
>> 
>> ceph osd getcrushmap -o /tmp/crush
>> crushtool -i /tmp/crush --set-choose-total-tries 100 -o /tmp/crush.new
>> ceph osd setcrushmap -i /tmp/crush.new
>> 
>> Is this right or not?
>> 
>> I started this cluster with these 3 nodes and each 3 osds. They are vms. I 
>> knew that this cluster would expand very big, that’s the reason for my 
>> choice for ceph. Now I can’t add more HDDs to the vm hypervisor and I want 
>> to separate the nodes physically too. I bought a new node with these 4 
>> drives and now another node with only 2 drives. As I hear now from several 
>> people this was not a good idea. For this reason, I bought now additional 
>> HDDs for the new node, so I have two with the same amount of HDDs and size. 
>> In the next 1-2 months I will get the third physical node and then 
>> everything should be fine. But at this time I have no other option.
>> 
>> May it help to solve this problem by adding the 2 new HDDs to the new ceph 
>> node?
>> 
>> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

2017-01-11 Thread Marcus Müller

OK, I changed the setting and it seems to work as expected. I have to say thank 
you to all you guys!

I was just not sure how to do this properly.



> Am 11.01.2017 um 13:11 schrieb Shinobu Kinjo :
> 
> Please refer to Jens's message.
> 
> Regards,
> 
> On Wed, Jan 11, 2017 at 8:53 PM, Marcus Müller  
> wrote:
>> Ok, thank you. I thought I have to set ceph to a tunables profile. If I’m 
>> right, then I just have to export the current crush map, edit it and import 
>> it again, like:
>> 
>> ceph osd getcrushmap -o /tmp/crush
>> crushtool -i /tmp/crush --set-choose-total-tries 100 -o /tmp/crush.new
>> ceph osd setcrushmap -i /tmp/crush.new
>> 
>> Is this right or not?
>> 
>> I started this cluster with these 3 nodes and each 3 osds. They are vms. I 
>> knew that this cluster would expand very big, that’s the reason for my 
>> choice for ceph. Now I can’t add more HDDs to the vm hypervisor and I want 
>> to separate the nodes physically too. I bought a new node with these 4 
>> drives and now another node with only 2 drives. As I hear now from several 
>> people this was not a good idea. For this reason, I bought now additional 
>> HDDs for the new node, so I have two with the same amount of HDDs and size. 
>> In the next 1-2 months I will get the third physical node and then 
>> everything should be fine. But at this time I have no other option.
>> 
>> May it help to solve this problem by adding the 2 new HDDs to the new ceph 
>> node?
>> 
>> 
>> 
>>> Am 11.01.2017 um 12:00 schrieb Brad Hubbard :
>>> 
>>> Your current problem has nothing to do with clients and neither does
>>> choose_total_tries.
>>> 
>>> Try setting just this value to 100 and see if your situation improves.
>>> 
>>> Ultimately you need to take a good look at your cluster configuration
>>> and how your crush map is configured to deal with that configuration
>>> but start with choose_total_tries as it has the highest probability of
>>> helping your situation. Your clients should not be affected.
>>> 
>>> Could you explain the reasoning behind having three hosts with one ods
>>> each, one host with two osds and one with four?
>>> 
>>> You likely need to tweak your crushmap to handle this configuration
>>> better or, preferably, move to a more uniform configuration.
>>> 
>>> 
>>> On Wed, Jan 11, 2017 at 5:38 PM, Marcus Müller  
>>> wrote:
 I have to thank you all. You give free support and this already helps me.
 I’m not the one who knows ceph that good, but everyday it’s getting better
 and better ;-)
 
 According to the article Brad posted I have to change the ceph osd crush
 tunables. But there are two questions left as I already wrote:
 
 - According to
 http://docs.ceph.com/docs/master/rados/operations/crush-map/#tunables there
 are a few profiles. My needed profile would be BOBTAIL (CRUSH_TUNABLES2)
 wich would set choose_total_tries to 50. For the beginning better than 19.
 There I also see: "You can select a profile on a running cluster with the
 command: ceph osd crush tunables {PROFILE}“. My question on this is: Even 
 if
 I run hammer, is it good and possible to set it to bobtail?
 
 - We can also read:
 WHICH CLIENT VERSIONS SUPPORT CRUSH_TUNABLES2
 - v0.55 or later, including bobtail series (v0.56.x)
 - Linux kernel version v3.9 or later (for the file system and RBD kernel
 clients)
 
 And here my question is: If my clients use librados (version hammer), do I
 need to have this required kernel version on the clients or the ceph nodes?
 
 I don’t want to have troubles at the end with my clients. Can someone 
 answer
 me this, before I change the settings?
 
 
 Am 11.01.2017 um 06:47 schrieb Shinobu Kinjo :
 
 
 Yeah, Sam is correct. I've not looked at crushmap. But I should have
 noticed what troublesome is with looking at `ceph osd tree`. That's my
 bad, sorry for that.
 
 Again please refer to:
 
 http://www.anchor.com.au/blog/2013/02/pulling-apart-cephs-crush-algorithm/
 
 Regards,
 
 
 On Wed, Jan 11, 2017 at 1:50 AM, Samuel Just  wrote:
 
 Shinobu isn't correct, you have 9/9 osds up and running.  up does not
 equal acting because crush is having trouble fulfilling the weights in
 your crushmap and the acting set is being padded out with an extra osd
 which happens to have the data to keep you up to the right number of
 replicas.  Please refer back to Brad's post.
 -Sam
 
 On Mon, Jan 9, 2017 at 11:08 PM, Marcus Müller 
 wrote:
 
 Ok, i understand but how can I debug why they are not running as they
 should? For me I thought everything is fine because ceph -s said they are 
 up
 and running.
 
 I would think of a problem with the crush map.
 
 Am 10.01.2017 um 08:06 schrieb Shinobu Kinjo :
 
 e.g.,
 OSD7 / 3 / 0 are in the same acting set. They should be up, if t

[ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

Re: [ceph-users] PGs stuck active+remapped and osds lose data?!

24 matches

Site Navigation

Mail list logo

Footer information