Re: [ceph-users] Inconsistent pgs after update to 0.73 - 0.74

2014-01-13 Thread Mark Kirkwood 

On 10/01/14 17:16, Mark Kirkwood wrote:

On 10/01/14 16:18, David Zafman wrote:
With pool size of 1 the scrub can still do some consistency 
checking.  These are things like missing attributes, on-disk size 
doesn’t match attribute size, non-clone without a head, expected 
clone.  You could check the osd logs to see what they were.


The pg below only had 1 object in error and it was detected after 
2013-12-13 15:38:13.283741 which was the last clean scrub.





Right - this is what I'm seeing in the log (not immediately clear why 
there is a problem, and why it needs to be seemingly trivially fixed 
by a repair:


2014-01-10 15:53:52.654065 7f17ff94f700  0 log [ERR] : 2.3f scrub stat 
mismatch, got 44/44 objects, 0/0 clones, 44/0 dirty, 0/0 whiteouts, 
180355088/180355088 bytes.
2014-01-10 15:53:52.654079 7f17ff94f700  0 log [ERR] : 2.3f scrub 1 
errors





I think I see why this message (and hence the error) is appearing in the 
log, In commit 43465d (merge pull request #950 from wip-pg-stat) extra 
checks were added to osd/replicatedPG.cc for dirty and whiteout. In the 
log I have dirty stat mismatch - so that explains why I didn't see any 
errors until I updated to this year (merge was Jan 1).  This still 
leaves the question of why the stat counts were wrong to start with...


Regards

Mark
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Inconsistent pgs after update to 0.73 - 0.74

2014-01-09 Thread Mark Kirkwood 
I've noticed this on 2 (development) clusters that I have with pools 
having size 1. I guess my first question would be - is this expected?


Here's some detail from one of the clusters:

$ ceph -v
ceph version 0.74-621-g6fac2ac (6fac2acc5e6f77651ffcd7dc7aa833713517d8a6)

$ ceph osd dump
epoch 104
fsid 4e8548e8-dfe4-46d0-a2e8-fb4a9beadff2
created 2013-11-08 11:01:57.051773
modified 2014-01-10 14:55:07.353514
flags

pool 0 'data' replicated size 1 min_size 1 crush_ruleset 0 object_hash 
rjenkins pg_num 64 pgp_num 64 last_change 47 owner 0 
crash_replay_interval 45
pool 1 'metadata' replicated size 1 min_size 1 crush_ruleset 1 
object_hash rjenkins pg_num 64 pgp_num 64 last_change 49 owner 0
pool 2 'rbd' replicated size 1 min_size 1 crush_ruleset 2 object_hash 
rjenkins pg_num 64 pgp_num 64 last_change 51 owner 0


max_osd 2
osd.0 up   in  weight 1 up_from 102 up_thru 102 down_at 101 
last_clean_interval [97,100) 192.168.2.63:6800/27931 
192.168.2.63:6801/27931 192.168.2.63:6802/27931 192.168.2.63:6803/27931 
exists,up 3c23570a-8a46-4ff2-9dab-cbbe82138bf7
osd.1 up   in  weight 1 up_from 103 up_thru 103 down_at 100 
last_clean_interval [98,99) 192.168.2.63:6805/28070 
192.168.2.63:6806/28070 192.168.2.63:6807/28070 192.168.2.63:6808/28070 
exists,up 73470b19-cd55-4881-a070-55efa06f3df3


$ ceph -s
cluster 4e8548e8-dfe4-46d0-a2e8-fb4a9beadff2
 health HEALTH_ERR 62 pgs inconsistent; 62 scrub errors; crush map 
has non-optimal tunables
 monmap e1: 1 mons at {vedavec=192.168.2.63:6789/0}, election epoch 
1, quorum 0 vedavec

 osdmap e104: 2 osds: 2 up, 2 in
  pgmap v4101: 192 pgs, 3 pools, 9691 MB data, 2439 objects
9833 MB used, 179 GB / 198 GB avail
 130 active+clean
  62 active+clean+inconsistent


There are no dmesg errors, and performing a repair on each inconsistent 
pg removes the inconsistent state. I note with this cluster above that I 
have non optimal tunables, however (I think) I sorted that on the other 
one, which had no effect on the inconsistent pgs.


Here's a query from one inconsistent pg:

$ ceph pg 2.3f query
{ state: active+clean+inconsistent,
  epoch: 104,
  up: [
0],
  acting: [
0],
  actingbackfill: [
0],
  info: { pgid: 2.3f,
  last_update: 57'195468,
  last_complete: 57'195468,
  log_tail: 57'192467,
  last_user_version: 195468,
  last_backfill: MAX,
  purged_snaps: [],
  history: { epoch_created: 1,
  last_epoch_started: 103,
  last_epoch_clean: 103,
  last_epoch_split: 0,
  same_up_since: 102,
  same_interval_since: 102,
  same_primary_since: 102,
  last_scrub: 57'195468,
  last_scrub_stamp: 2014-01-10 14:59:23.195403,
  last_deep_scrub: 57'195468,
  last_deep_scrub_stamp: 2014-01-08 11:48:47.559227,
  last_clean_scrub_stamp: 2013-12-13 15:38:13.283741},
  stats: { version: 57'195468,
  reported_seq: 531759,
  reported_epoch: 104,
  state: active+clean+inconsistent,
  last_fresh: 2014-01-10 14:59:23.195452,
  last_change: 2014-01-10 14:59:23.195452,
  last_active: 2014-01-10 14:59:23.195452,
  last_clean: 2014-01-10 14:59:23.195452,
  last_became_active: 0.00,
  last_unstale: 2014-01-10 14:59:23.195452,
  mapping_epoch: 101,
  log_start: 57'192467,
  ondisk_log_start: 57'192467,
  created: 1,
  last_epoch_clean: 103,
  parent: 0.0,
  parent_split_bits: 0,
  last_scrub: 57'195468,
  last_scrub_stamp: 2014-01-10 14:59:23.195403,
  last_deep_scrub: 57'195468,
  last_deep_scrub_stamp: 2014-01-08 11:48:47.559227,
  last_clean_scrub_stamp: 2013-12-13 15:38:13.283741,
  log_size: 3001,
  ondisk_log_size: 3001,
  stats_invalid: 0,
  stat_sum: { num_bytes: 180355088,
  num_objects: 44,
  num_object_clones: 0,
  num_object_copies: 44,
  num_objects_missing_on_primary: 0,
  num_objects_degraded: 0,
  num_objects_unfound: 0,
  num_objects_dirty: 0,
  num_whiteouts: 0,
  num_read: 143836,
  num_read_kb: 3135807,
  num_write: 195467,
  num_write_kb: 2994821,
  num_scrub_errors: 1,
  num_shallow_scrub_errors: 1,
  num_deep_scrub_errors: 0,
  num_objects_recovered: 0,
  num_bytes_recovered: 0,
  num_keys_recovered: 0},
  stat_cat_sum: {},
  up: [
0],
  acting: [
0]},
  empty: 0,
  dne: 0,
  incomplete: 0,
  last_epoch_started: 103,
  hit_set_history: { current_last_update: 0'0,
  current_last_stamp: 0.00,
  current_info: { begin: 0.00,
  end: 0.00,

Re: [ceph-users] Inconsistent pgs after update to 0.73 - 0.74

2014-01-09 Thread David Zafman 

With pool size of 1 the scrub can still do some consistency checking.  These 
are things like missing attributes, on-disk size doesn’t match attribute size, 
non-clone without a head, expected clone.  You could check the osd logs to see 
what they were.

The pg below only had 1 object in error and it was detected after 2013-12-13 
15:38:13.283741 which was the last clean scrub.

David Zafman
Senior Developer
http://www.inktank.com




On Jan 9, 2014, at 6:36 PM, Mark Kirkwood mark.kirkw...@catalyst.net.nz wrote:

 I've noticed this on 2 (development) clusters that I have with pools having 
 size 1. I guess my first question would be - is this expected?
 
 Here's some detail from one of the clusters:
 
 $ ceph -v
 ceph version 0.74-621-g6fac2ac (6fac2acc5e6f77651ffcd7dc7aa833713517d8a6)
 
 $ ceph osd dump
 epoch 104
 fsid 4e8548e8-dfe4-46d0-a2e8-fb4a9beadff2
 created 2013-11-08 11:01:57.051773
 modified 2014-01-10 14:55:07.353514
 flags
 
 pool 0 'data' replicated size 1 min_size 1 crush_ruleset 0 object_hash 
 rjenkins pg_num 64 pgp_num 64 last_change 47 owner 0 crash_replay_interval 45
 pool 1 'metadata' replicated size 1 min_size 1 crush_ruleset 1 object_hash 
 rjenkins pg_num 64 pgp_num 64 last_change 49 owner 0
 pool 2 'rbd' replicated size 1 min_size 1 crush_ruleset 2 object_hash 
 rjenkins pg_num 64 pgp_num 64 last_change 51 owner 0
 
 max_osd 2
 osd.0 up   in  weight 1 up_from 102 up_thru 102 down_at 101 
 last_clean_interval [97,100) 192.168.2.63:6800/27931 192.168.2.63:6801/27931 
 192.168.2.63:6802/27931 192.168.2.63:6803/27931 exists,up 
 3c23570a-8a46-4ff2-9dab-cbbe82138bf7
 osd.1 up   in  weight 1 up_from 103 up_thru 103 down_at 100 
 last_clean_interval [98,99) 192.168.2.63:6805/28070 192.168.2.63:6806/28070 
 192.168.2.63:6807/28070 192.168.2.63:6808/28070 exists,up 
 73470b19-cd55-4881-a070-55efa06f3df3
 
 $ ceph -s
cluster 4e8548e8-dfe4-46d0-a2e8-fb4a9beadff2
 health HEALTH_ERR 62 pgs inconsistent; 62 scrub errors; crush map has 
 non-optimal tunables
 monmap e1: 1 mons at {vedavec=192.168.2.63:6789/0}, election epoch 1, 
 quorum 0 vedavec
 osdmap e104: 2 osds: 2 up, 2 in
  pgmap v4101: 192 pgs, 3 pools, 9691 MB data, 2439 objects
9833 MB used, 179 GB / 198 GB avail
 130 active+clean
  62 active+clean+inconsistent
 
 
 There are no dmesg errors, and performing a repair on each inconsistent pg 
 removes the inconsistent state. I note with this cluster above that I have 
 non optimal tunables, however (I think) I sorted that on the other one, which 
 had no effect on the inconsistent pgs.
 
 Here's a query from one inconsistent pg:
 
 $ ceph pg 2.3f query
 { state: active+clean+inconsistent,
  epoch: 104,
  up: [
0],
  acting: [
0],
  actingbackfill: [
0],
  info: { pgid: 2.3f,
  last_update: 57'195468,
  last_complete: 57'195468,
  log_tail: 57'192467,
  last_user_version: 195468,
  last_backfill: MAX,
  purged_snaps: [],
  history: { epoch_created: 1,
  last_epoch_started: 103,
  last_epoch_clean: 103,
  last_epoch_split: 0,
  same_up_since: 102,
  same_interval_since: 102,
  same_primary_since: 102,
  last_scrub: 57'195468,
  last_scrub_stamp: 2014-01-10 14:59:23.195403,
  last_deep_scrub: 57'195468,
  last_deep_scrub_stamp: 2014-01-08 11:48:47.559227,
  last_clean_scrub_stamp: 2013-12-13 15:38:13.283741},
  stats: { version: 57'195468,
  reported_seq: 531759,
  reported_epoch: 104,
  state: active+clean+inconsistent,
  last_fresh: 2014-01-10 14:59:23.195452,
  last_change: 2014-01-10 14:59:23.195452,
  last_active: 2014-01-10 14:59:23.195452,
  last_clean: 2014-01-10 14:59:23.195452,
  last_became_active: 0.00,
  last_unstale: 2014-01-10 14:59:23.195452,
  mapping_epoch: 101,
  log_start: 57'192467,
  ondisk_log_start: 57'192467,
  created: 1,
  last_epoch_clean: 103,
  parent: 0.0,
  parent_split_bits: 0,
  last_scrub: 57'195468,
  last_scrub_stamp: 2014-01-10 14:59:23.195403,
  last_deep_scrub: 57'195468,
  last_deep_scrub_stamp: 2014-01-08 11:48:47.559227,
  last_clean_scrub_stamp: 2013-12-13 15:38:13.283741,
  log_size: 3001,
  ondisk_log_size: 3001,
  stats_invalid: 0,
  stat_sum: { num_bytes: 180355088,
  num_objects: 44,
  num_object_clones: 0,
  num_object_copies: 44,
  num_objects_missing_on_primary: 0,
  num_objects_degraded: 0,
  num_objects_unfound: 0,
  num_objects_dirty: 0,
  num_whiteouts: 0,
  num_read: 143836,
  num_read_kb: 3135807,
  num_write: 195467,
  num_write_kb: 2994821,
  num_scrub_errors: 1,

Re: [ceph-users] Inconsistent pgs after update to 0.73 - 0.74

2014-01-09 Thread Mark Kirkwood 

On 10/01/14 16:18, David Zafman wrote:

With pool size of 1 the scrub can still do some consistency checking.  These 
are things like missing attributes, on-disk size doesn’t match attribute size, 
non-clone without a head, expected clone.  You could check the osd logs to see 
what they were.

The pg below only had 1 object in error and it was detected after 2013-12-13 
15:38:13.283741 which was the last clean scrub.




Right - this is what I'm seeing in the log (not immediately clear why 
there is a problem, and why it needs to be seemingly trivially fixed by 
a repair:


2014-01-10 15:53:52.654065 7f17ff94f700  0 log [ERR] : 2.3f scrub stat 
mismatch, got 44/44 objects, 0/0 clones, 44/0 dirty, 0/0 whiteouts, 
180355088/180355088 bytes.

2014-01-10 15:53:52.654079 7f17ff94f700  0 log [ERR] : 2.3f scrub 1 errors

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com