Re: [ceph-users] [PG] Slow request *** seconds old，v4 currently waiting for pg to exist locally
Hi,
looks that some osds are down?!
What is the output of ceph osd tree
Udo
Am 25.09.2014 04:29, schrieb Aegeaner:
The cluster healthy state is WARN:
health HEALTH_WARN 118 pgs degraded; 8 pgs down; 59 pgs
incomplete; 28 pgs peering; 292 pgs stale; 87 pgs stuck inactive;
292 pgs stuck stale; 205 pgs stuck unclean; 22 requests are blocked
32 sec; recovery 12474/46357 objects degraded (26.909%)
monmap e3: 3 mons at
{CVM-0-mon01=172.18.117.146:6789/0,CVM-0-mon02=172.18.117.152:6789/0,CVM-0-mon03=172.18.117.153:6789/0},
election epoch 24, quorum 0,1,2 CVM-0-mon01,CVM-0-mon02,CVM-0-mon03
osdmap e421: 9 osds: 9 up, 9 in
pgmap v2261: 292 pgs, 4 pools, 91532 MB data, 23178 objects
330 MB used, 3363 GB / 3363 GB avail
12474/46357 objects degraded (26.909%)
20 stale+peering
87 stale+active+clean
8 stale+down+peering
59 stale+incomplete
118 stale+active+degraded
What does these errors mean? Can these PGs be recovered?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [PG] Slow request *** seconds old，v4 currently waiting for pg to exist locally
Hi again,
sorry - forgot my post... see
osdmap e421: 9 osds: 9 up, 9 in
shows that all your 9 osds are up!
Do you have trouble with your journal/filesystem?
Udo
Am 25.09.2014 08:01, schrieb Udo Lembke:
Hi,
looks that some osds are down?!
What is the output of ceph osd tree
Udo
Am 25.09.2014 04:29, schrieb Aegeaner:
The cluster healthy state is WARN:
health HEALTH_WARN 118 pgs degraded; 8 pgs down; 59 pgs
incomplete; 28 pgs peering; 292 pgs stale; 87 pgs stuck inactive;
292 pgs stuck stale; 205 pgs stuck unclean; 22 requests are blocked
32 sec; recovery 12474/46357 objects degraded (26.909%)
monmap e3: 3 mons at
{CVM-0-mon01=172.18.117.146:6789/0,CVM-0-mon02=172.18.117.152:6789/0,CVM-0-mon03=172.18.117.153:6789/0},
election epoch 24, quorum 0,1,2 CVM-0-mon01,CVM-0-mon02,CVM-0-mon03
osdmap e421: 9 osds: 9 up, 9 in
pgmap v2261: 292 pgs, 4 pools, 91532 MB data, 23178 objects
330 MB used, 3363 GB / 3363 GB avail
12474/46357 objects degraded (26.909%)
20 stale+peering
87 stale+active+clean
8 stale+down+peering
59 stale+incomplete
118 stale+active+degraded
What does these errors mean? Can these PGs be recovered?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [PG] Slow request *** seconds old，v4 currently waiting for pg to exist locally
osd_op(client.4625.1:9005787)
.
This is due to external factors. For example, the network settings.
2014-09-25 10:05 GMT+04:00 Udo Lembke ulem...@polarzone.de:
Hi again,
sorry - forgot my post... see
osdmap e421: 9 osds: 9 up, 9 in
shows that all your 9 osds are up!
Do you have trouble with your journal/filesystem?
Udo
Am 25.09.2014 08:01, schrieb Udo Lembke:
Hi,
looks that some osds are down?!
What is the output of ceph osd tree
Udo
Am 25.09.2014 04:29, schrieb Aegeaner:
The cluster healthy state is WARN:
health HEALTH_WARN 118 pgs degraded; 8 pgs down; 59 pgs
incomplete; 28 pgs peering; 292 pgs stale; 87 pgs stuck inactive;
292 pgs stuck stale; 205 pgs stuck unclean; 22 requests are blocked
32 sec; recovery 12474/46357 objects degraded (26.909%)
monmap e3: 3 mons at
{CVM-0-mon01=
172.18.117.146:6789/0,CVM-0-mon02=172.18.117.152:6789/0,CVM-0-mon03=172.18.117.153:6789/0
},
election epoch 24, quorum 0,1,2 CVM-0-mon01,CVM-0-mon02,CVM-0-mon03
osdmap e421: 9 osds: 9 up, 9 in
pgmap v2261: 292 pgs, 4 pools, 91532 MB data, 23178 objects
330 MB used, 3363 GB / 3363 GB avail
12474/46357 objects degraded (26.909%)
20 stale+peering
87 stale+active+clean
8 stale+down+peering
59 stale+incomplete
118 stale+active+degraded
What does these errors mean? Can these PGs be recovered?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
С уважением, Фасихов Ирек Нургаязович
Моб.: +79229045757
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [PG] Slow request *** seconds old，v4 currently waiting for pg to exist locally
Yeah, three of nine OSDs went down but I recreated them, but the pgs
cannot be recovered.
I don't know how to erase all the pgs, so I deleted all the osd pools,
including data and metadata … Now all pgs are active and clean...
I'm not sure if there are more elegant ways to deal with this.
===
Aegeaner
在 2014-09-25 14:11, Irek Fasikhov 写道:
osd_op(client.4625.1:9005787)
.
This is due to external factors. For example, the network settings.
2014-09-25 10:05 GMT+04:00 Udo Lembke ulem...@polarzone.de
mailto:ulem...@polarzone.de:
Hi again,
sorry - forgot my post... see
osdmap e421: 9 osds: 9 up, 9 in
shows that all your 9 osds are up!
Do you have trouble with your journal/filesystem?
Udo
Am 25.09.2014 08:01, schrieb Udo Lembke:
Hi,
looks that some osds are down?!
What is the output of ceph osd tree
Udo
Am 25.09.2014 04:29, schrieb Aegeaner:
The cluster healthy state is WARN:
health HEALTH_WARN 118 pgs degraded; 8 pgs down; 59 pgs
incomplete; 28 pgs peering; 292 pgs stale; 87 pgs stuck
inactive;
292 pgs stuck stale; 205 pgs stuck unclean; 22 requests are
blocked
32 sec; recovery 12474/46357 objects degraded (26.909%)
monmap e3: 3 mons at
{CVM-0-mon01=172.18.117.146:6789/0,CVM-0-mon02=172.18.117.152:6789/0,CVM-0-mon03=172.18.117.153:6789/0
http://172.18.117.146:6789/0,CVM-0-mon02=172.18.117.152:6789/0,CVM-0-mon03=172.18.117.153:6789/0},
election epoch 24, quorum 0,1,2
CVM-0-mon01,CVM-0-mon02,CVM-0-mon03
osdmap e421: 9 osds: 9 up, 9 in
pgmap v2261: 292 pgs, 4 pools, 91532 MB data, 23178
objects
330 MB used, 3363 GB / 3363 GB avail
12474/46357 objects degraded (26.909%)
20 stale+peering
87 stale+active+clean
8 stale+down+peering
59 stale+incomplete
118 stale+active+degraded
What does these errors mean? Can these PGs be recovered?
___
ceph-users mailing list
ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
С уважением, Фасихов Ирек Нургаязович
Моб.: +79229045757
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] [PG] Slow request *** seconds old，v4 currently waiting for pg to exist locally
The cluster healthy state is WARN:
health HEALTH_WARN 118 pgs degraded; 8 pgs down; 59 pgs
incomplete; 28 pgs peering; 292 pgs stale; 87 pgs stuck inactive;
292 pgs stuck stale; 205 pgs stuck unclean; 22 requests are blocked
32 sec; recovery 12474/46357 objects degraded (26.909%)
monmap e3: 3 mons at
{CVM-0-mon01=172.18.117.146:6789/0,CVM-0-mon02=172.18.117.152:6789/0,CVM-0-mon03=172.18.117.153:6789/0},
election epoch 24, quorum 0,1,2 CVM-0-mon01,CVM-0-mon02,CVM-0-mon03
osdmap e421: 9 osds: 9 up, 9 in
pgmap v2261: 292 pgs, 4 pools, 91532 MB data, 23178 objects
330 MB used, 3363 GB / 3363 GB avail
12474/46357 objects degraded (26.909%)
20 stale+peering
87 stale+active+clean
8 stale+down+peering
59 stale+incomplete
118 stale+active+degraded
What does these errors mean? Can these PGs be recovered?
2014-09-25 10:23:51.408102 mon.0 [INF] mdsmap e1: 0/0/1 up
2014-09-25 10:23:41.060536 osd.3 [WRN] 2 slow requests, 1 included
below; oldest blocked for 240.436614 secs
2014-09-25 10:23:41.060553 osd.3 [WRN] slow request 240.436614
seconds old, received at 2014-09-25 10:19:40.623834:
osd_op(client.4625.1:9005787 kvtest.rbd [watch add cookie 1 ver 0]
2.eb238611 reassert_version=81'5 ondisk+write e414) v4 currently
waiting for pg to exist locally
2014-09-25 10:23:39.042514 osd.7 [WRN] 4 slow requests, 1 included
below; oldest blocked for 240.121452 secs
2014-09-25 10:23:39.042529 osd.7 [WRN] slow request 240.121452
seconds old, received at 2014-09-25 10:19:38.921007:
osd_op(client.4625.1:9005791 kvtest.rbd [watch add cookie 1 ver 0]
2.eb238611 reassert_version=81'5 ondisk+write e418) v4 currently
waiting for pg to exist locally
2014-09-25 10:23:53.064475 osd.3 [WRN] 2 slow requests, 1 included
below; oldest blocked for 252.440597 secs
2014-09-25 10:23:53.064484 osd.3 [WRN] slow request 240.871862
seconds old, received at 2014-09-25 10:19:52.192569:
osd_op(client.4625.1:9005790 kvtest.rbd [watch add cookie 1 ver 0]
2.eb238611 reassert_version=81'5 ondisk+write e417) v4 currently
waiting for pg to exist locally
2014-09-25 10:24:32.135858 osd.8 [WRN] 6 slow requests, 6 included
below; oldest blocked for 240.211169 secs
2014-09-25 10:24:32.135867 osd.8 [WRN] slow request 240.211169
seconds old, received at 2014-09-25 10:20:31.924624:
osd_op(client.4625.1:9005132 rb.0.120e.2ae8944a.09ba [write
180224~16384] 2.72dfe69d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:32.135872 osd.8 [WRN] slow request 240.210789
seconds old, received at 2014-09-25 10:20:31.925004:
osd_op(client.4625.1:9005188 rb.0.120e.2ae8944a.0371 [write
1916928~16384] 2.3445541d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:32.135876 osd.8 [WRN] slow request 240.210747
seconds old, received at 2014-09-25 10:20:31.925046:
osd_op(client.4625.1:9005367 rb.0.120e.2ae8944a.0427 [write
3817472~16384] 2.654ab5d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:32.135880 osd.8 [WRN] slow request 240.210667
seconds old, received at 2014-09-25 10:20:31.925126:
osd_op(client.4625.1:9005444 rb.0.120e.2ae8944a.0bc6 [write
622592~16384] 2.2f36185d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:32.135886 osd.8 [WRN] slow request 240.210488
seconds old, received at 2014-09-25 10:20:31.925305:
osd_op(client.4625.1:9005581 rb.0.120e.2ae8944a.0753 [write
884736~16384] 2.be66e1d RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
2014-09-25 10:24:33.136321 osd.8 [WRN] 8 slow requests, 3 included
below; oldest blocked for 241.211660 secs
2014-09-25 10:24:33.136330 osd.8 [WRN] slow request 241.210921
seconds old, received at 2014-09-25 10:20:31.925363:
osd_op(client.4625.1:9005606 rb.0.120e.2ae8944a.03f0 [write
3391488~16384] 2.fdfdcfdd RETRY=1 ondisk+retry+write e421) v4
currently waiting for pg to exist locally
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
