[ceph-users] 转发: how to fix the mds damaged issue

2016-07-03 Thread Lihang 
root@BoreNode2:~# ceph -v
ceph version 10.2.0

发件人: lihang 12398 (RD)
发送时间: 2016年7月3日 14:47
收件人: ceph-users@lists.ceph.com
抄送: Ceph Development; 'uker...@gmail.com'; zhengbin 08747 (RD); xusangdi 11976 
(RD)
主题: how to fix the mds damaged issue

Hi, my ceph cluster mds is damaged and the cluster is degraded after our 
machines library power down suddenly. then the cluster is “HEALTH_ERR” and 
cann’t be recovered to health by itself after my
Reboot the storage node system or restart the ceph cluster yet. After that I 
also use the following command to remove the damaged mds, but the damaged mds 
be removed failed and the issue exist still. The another two mds state is 
standby. Who can tell me how to fix this issue and find out what happened in my 
cluter?
the remove damaged mds process in my storage node as follows.

1> Execute ”stop ceph-mds-all” command  in the damaged mds node

2>  ceph mds rmfailed 0 --yes-i-really-mean-it

3>  root@BoreNode2:~# ceph  mds rm 0

mds gid 0 dne

The detailed status of my cluster as following:
root@BoreNode2:~# ceph -s
  cluster 98edd275-5df7-414f-a202-c3d4570f251c
 health HEALTH_ERR
mds rank 0 is damaged
mds cluster is degraded
 monmap e1: 3 mons at 
{BoreNode2=172.16.65.141:6789/0,BoreNode3=172.16.65.142:6789/0,BoreNode4=172.16.65.143:6789/0}
election epoch 1010, quorum 0,1,2 BoreNode2,BoreNode3,BoreNode4
  fsmap e168: 0/1/1 up, 3 up:standby, 1 damaged
 osdmap e338: 8 osds: 8 up, 8 in
flags sortbitwise
  pgmap v17073: 1560 pgs, 5 pools, 218 kB data, 32 objects
423 MB used, 3018 GB / 3018 GB avail
1560 active+clean
root@BoreNode2:~# ceph mds dump
dumped fsmap epoch 168
fs_name TudouFS
epoch   156
flags   0
created 2016-04-02 02:48:11.150539
modified2016-04-03 03:04:57.347064
tableserver 0
root0
session_timeout 60
session_autoclose   300
max_file_size   1099511627776
last_failure0
last_failure_osd_epoch  83
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable 
ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses 
versioned encoding,6=dirfrag is stored in omap,8=file layout v2}
max_mds 1
in  0
up  {}
failed
damaged 0
stopped
data_pools  4
metadata_pool   3
inline_data disabled
-
本邮件及其附件含有杭州华三通信技术有限公司的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from H3C, 
which is
intended only for the person or entity whose address is listed above. Any use 
of the
information contained herein in any way (including, but not limited to, total 
or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify 
the sender
by phone or email immediately and delete it!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How many nodes/OSD can fail

2016-07-03 Thread Willi Fehler 

Hello David,

so in a 3 node Cluster how should I set min_size if I want that 2 nodes 
could fail?


Regards - Willi

Am 28.06.16 um 13:07 schrieb David:

Hi,

This is probably the min_size on your cephfs data and/or metadata 
pool. I believe the default is 2, if you have less than 2 replicas 
available I/O will stop. See: 
http://docs.ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas


On Tue, Jun 28, 2016 at 10:23 AM, willi.feh...@t-online.de 
 > wrote:


Hello,

I'm still very new to Ceph. I've created a small test Cluster.

ceph-node1

osd0

osd1

osd2

ceph-node2

osd3

osd4

osd5

ceph-node3

osd6

osd7

osd8

My pool for CephFS has a replication count of 3. I've powered of 2
nodes(6 OSDs went down) and my cluster status became critical and
my ceph clients(cephfs) run into a timeout. My data(I had only one
file on my pool) was still on one of the active OSDs. Is this the
expected behaviour that the Cluster status became critical and my
Clients run into a timeout?

Many thanks for your feedback.

Regards - Willi



___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How many nodes/OSD can fail

2016-07-03 Thread Sean Redmond 
It would need to be set to 1
On 3 Jul 2016 8:17 a.m., "Willi Fehler"  wrote:

> Hello David,
>
> so in a 3 node Cluster how should I set min_size if I want that 2 nodes
> could fail?
>
> Regards - Willi
>
> Am 28.06.16 um 13:07 schrieb David:
>
> Hi,
>
> This is probably the min_size on your cephfs data and/or metadata pool. I
> believe the default is 2, if you have less than 2 replicas available I/O
> will stop. See:
> http://docs.ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas
>
> On Tue, Jun 28, 2016 at 10:23 AM, willi.feh...@t-online.de <
> willi.feh...@t-online.de> wrote:
>
>> Hello,
>>
>> I'm still very new to Ceph. I've created a small test Cluster.
>>
>>
>>
>> ceph-node1
>>
>> osd0
>>
>> osd1
>>
>> osd2
>>
>> ceph-node2
>>
>> osd3
>>
>> osd4
>>
>> osd5
>>
>> ceph-node3
>>
>> osd6
>>
>> osd7
>>
>> osd8
>>
>>
>>
>> My pool for CephFS has a replication count of 3. I've powered of 2
>> nodes(6 OSDs went down) and my cluster status became critical and my ceph
>> clients(cephfs) run into a timeout. My data(I had only one file on my pool)
>> was still on one of the active OSDs. Is this the expected behaviour that
>> the Cluster status became critical and my Clients run into a timeout?
>>
>>
>>
>> Many thanks for your feedback.
>>
>>
>>
>> Regards - Willi
>>
>>
>> 
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How many nodes/OSD can fail

2016-07-03 Thread Willi Fehler 

Hello Sean,

I've powered down 2 nodes. So 6 of 9 OSD are down. But my client can't 
write and read anymore from my Ceph mount. Also 'ceph -s' hangs.


pool 1 'cephfs_data' replicated size 3 min_size 1 crush_ruleset 0 
object_hash rjenkins pg_num 300 pgp_num 300 last_change 447 flags 
hashpspool crash_replay_interval 45 stripe_width 0
pool 2 'cephfs_metadata' replicated size 3 min_size 1 crush_ruleset 0 
object_hash rjenkins pg_num 300 pgp_num 300 last_change 445 flags 
hashpspool stripe_width 0


2016-07-03 09:49:40.695953 7f3da56f9700  0 -- 192.168.0.5:0/2773396901 
>> 192.168.0.7:6789/0 pipe(0x7f3da0001f50 sd=3 :0 s=1 pgs=0 cs=0 l=1 
c=0x7f3daf20).fault
2016-07-03 09:49:44.195029 7f3da57fa700  0 -- 192.168.0.5:0/2773396901 
>> 192.168.0.6:6789/0 pipe(0x7f3da0005500 sd=4 :0 s=1 pgs=0 cs=0 l=1 
c=0x7f3da00067c0).fault
2016-07-03 09:49:50.205788 7f3da55f8700  0 -- 192.168.0.5:0/2773396901 
>> 192.168.0.6:6789/0 pipe(0x7f3da0005500 sd=3 :0 s=1 pgs=0 cs=0 l=1 
c=0x7f3da0004c40).fault
2016-07-03 09:49:52.720116 7f3da57fa700  0 -- 192.168.0.5:0/2773396901 
>> 192.168.0.7:6789/0 pipe(0x7f3da00023f0 sd=4 :0 s=1 pgs=0 cs=0 l=1 
c=0x7f3da00036b0).fault


Regards - Willi

Am 03.07.16 um 09:36 schrieb Sean Redmond:


It would need to be set to 1

On 3 Jul 2016 8:17 a.m., "Willi Fehler" > wrote:


Hello David,

so in a 3 node Cluster how should I set min_size if I want that 2
nodes could fail?

Regards - Willi

Am 28.06.16 um 13:07 schrieb David:

Hi,

This is probably the min_size on your cephfs data and/or metadata
pool. I believe the default is 2, if you have less than 2
replicas available I/O will stop. See:

http://docs.ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas

On Tue, Jun 28, 2016 at 10:23 AM, willi.feh...@t-online.de
 mailto:willi.feh...@t-online.de>> wrote:

Hello,

I'm still very new to Ceph. I've created a small test Cluster.

ceph-node1

osd0

osd1

osd2

ceph-node2

osd3

osd4

osd5

ceph-node3

osd6

osd7

osd8

My pool for CephFS has a replication count of 3. I've powered
of 2 nodes(6 OSDs went down) and my cluster status became
critical and my ceph clients(cephfs) run into a timeout. My
data(I had only one file on my pool) was still on one of the
active OSDs. Is this the expected behaviour that the Cluster
status became critical and my Clients run into a timeout?

Many thanks for your feedback.

Regards - Willi



___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How many nodes/OSD can fail

2016-07-03 Thread Tu Holmes 
Where are your mon nodes?

Were you mixing mon and OSD together?

Are 2 of the mon nodes down as well?
On Jul 3, 2016 12:53 AM, "Willi Fehler"  wrote:

> Hello Sean,
>
> I've powered down 2 nodes. So 6 of 9 OSD are down. But my client can't
> write and read anymore from my Ceph mount. Also 'ceph -s' hangs.
>
> pool 1 'cephfs_data' replicated size 3 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 300 pgp_num 300 last_change 447 flags
> hashpspool crash_replay_interval 45 stripe_width 0
> pool 2 'cephfs_metadata' replicated size 3 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 300 pgp_num 300 last_change 445 flags
> hashpspool stripe_width 0
>
> 2016-07-03 09:49:40.695953 7f3da56f9700  0 -- 192.168.0.5:0/2773396901 >>
> 192.168.0.7:6789/0 pipe(0x7f3da0001f50 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f3daf20).fault
> 2016-07-03 09:49:44.195029 7f3da57fa700  0 -- 192.168.0.5:0/2773396901 >>
> 192.168.0.6:6789/0 pipe(0x7f3da0005500 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f3da00067c0).fault
> 2016-07-03 09:49:50.205788 7f3da55f8700  0 -- 192.168.0.5:0/2773396901 >>
> 192.168.0.6:6789/0 pipe(0x7f3da0005500 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f3da0004c40).fault
> 2016-07-03 09:49:52.720116 7f3da57fa700  0 -- 192.168.0.5:0/2773396901 >>
> 192.168.0.7:6789/0 pipe(0x7f3da00023f0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f3da00036b0).fault
>
> Regards - Willi
>
> Am 03.07.16 um 09:36 schrieb Sean Redmond:
>
> It would need to be set to 1
> On 3 Jul 2016 8:17 a.m., "Willi Fehler"  wrote:
>
>> Hello David,
>>
>> so in a 3 node Cluster how should I set min_size if I want that 2 nodes
>> could fail?
>>
>> Regards - Willi
>>
>> Am 28.06.16 um 13:07 schrieb David:
>>
>> Hi,
>>
>> This is probably the min_size on your cephfs data and/or metadata pool. I
>> believe the default is 2, if you have less than 2 replicas available I/O
>> will stop. See:
>> http://docs.ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas
>>
>> On Tue, Jun 28, 2016 at 10:23 AM, willi.feh...@t-online.de <
>> willi.feh...@t-online.de> wrote:
>>
>>> Hello,
>>>
>>> I'm still very new to Ceph. I've created a small test Cluster.
>>>
>>>
>>>
>>> ceph-node1
>>>
>>> osd0
>>>
>>> osd1
>>>
>>> osd2
>>>
>>> ceph-node2
>>>
>>> osd3
>>>
>>> osd4
>>>
>>> osd5
>>>
>>> ceph-node3
>>>
>>> osd6
>>>
>>> osd7
>>>
>>> osd8
>>>
>>>
>>>
>>> My pool for CephFS has a replication count of 3. I've powered of 2
>>> nodes(6 OSDs went down) and my cluster status became critical and my ceph
>>> clients(cephfs) run into a timeout. My data(I had only one file on my pool)
>>> was still on one of the active OSDs. Is this the expected behaviour that
>>> the Cluster status became critical and my Clients run into a timeout?
>>>
>>>
>>>
>>> Many thanks for your feedback.
>>>
>>>
>>>
>>> Regards - Willi
>>>
>>>
>>> 
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How many nodes/OSD can fail

2016-07-03 Thread Willi Fehler 

Hello Tu,

yes that's correct. The mon nodes run as well on the OSD nodes. So I have

3 nodes in total. OSD, MDS and Mon on each Node.

Regards - Willi

Am 03.07.16 um 09:56 schrieb Tu Holmes:


Where are your mon nodes?

Were you mixing mon and OSD together?

Are 2 of the mon nodes down as well?

On Jul 3, 2016 12:53 AM, "Willi Fehler" > wrote:


Hello Sean,

I've powered down 2 nodes. So 6 of 9 OSD are down. But my client
can't write and read anymore from my Ceph mount. Also 'ceph -s' hangs.

pool 1 'cephfs_data' replicated size 3 min_size 1 crush_ruleset 0
object_hash rjenkins pg_num 300 pgp_num 300 last_change 447 flags
hashpspool crash_replay_interval 45 stripe_width 0
pool 2 'cephfs_metadata' replicated size 3 min_size 1
crush_ruleset 0 object_hash rjenkins pg_num 300 pgp_num 300
last_change 445 flags hashpspool stripe_width 0

2016-07-03 09:49:40.695953 7f3da56f9700  0 --
192.168.0.5:0/2773396901  >>
192.168.0.7:6789/0  pipe(0x7f3da0001f50
sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3daf20).fault
2016-07-03 09:49:44.195029 7f3da57fa700  0 --
192.168.0.5:0/2773396901  >>
192.168.0.6:6789/0  pipe(0x7f3da0005500
sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3da00067c0).fault
2016-07-03 09:49:50.205788 7f3da55f8700  0 --
192.168.0.5:0/2773396901  >>
192.168.0.6:6789/0  pipe(0x7f3da0005500
sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3da0004c40).fault
2016-07-03 09:49:52.720116 7f3da57fa700  0 --
192.168.0.5:0/2773396901  >>
192.168.0.7:6789/0  pipe(0x7f3da00023f0
sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f3da00036b0).fault

Regards - Willi

Am 03.07.16 um 09:36 schrieb Sean Redmond:


It would need to be set to 1

On 3 Jul 2016 8:17 a.m., "Willi Fehler" mailto:willi.feh...@t-online.de>> wrote:

Hello David,

so in a 3 node Cluster how should I set min_size if I want
that 2 nodes could fail?

Regards - Willi

Am 28.06.16 um 13:07 schrieb David:

Hi,

This is probably the min_size on your cephfs data and/or
metadata pool. I believe the default is 2, if you have less
than 2 replicas available I/O will stop. See:

http://docs.ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas

On Tue, Jun 28, 2016 at 10:23 AM, willi.feh...@t-online.de
 mailto:willi.feh...@t-online.de>> wrote:

Hello,

I'm still very new to Ceph. I've created a small test
Cluster.

ceph-node1

osd0

osd1

osd2

ceph-node2

osd3

osd4

osd5

ceph-node3

osd6

osd7

osd8

My pool for CephFS has a replication count of 3. I've
powered of 2 nodes(6 OSDs went down) and my cluster
status became critical and my ceph clients(cephfs) run
into a timeout. My data(I had only one file on my pool)
was still on one of the active OSDs. Is this the
expected behaviour that the Cluster status became
critical and my Clients run into a timeout?

Many thanks for your feedback.

Regards - Willi



___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How many nodes/OSD can fail

2016-07-03 Thread Sean Redmond 
Hi,

You will need 2 mons to be online.

Thanks
On 3 Jul 2016 8:58 a.m., "Willi Fehler"  wrote:

> Hello Tu,
>
> yes that's correct. The mon nodes run as well on the OSD nodes. So I have
>
> 3 nodes in total. OSD, MDS and Mon on each Node.
>
> Regards - Willi
>
> Am 03.07.16 um 09:56 schrieb Tu Holmes:
>
> Where are your mon nodes?
>
> Were you mixing mon and OSD together?
>
> Are 2 of the mon nodes down as well?
> On Jul 3, 2016 12:53 AM, "Willi Fehler"  wrote:
>
>> Hello Sean,
>>
>> I've powered down 2 nodes. So 6 of 9 OSD are down. But my client can't
>> write and read anymore from my Ceph mount. Also 'ceph -s' hangs.
>>
>> pool 1 'cephfs_data' replicated size 3 min_size 1 crush_ruleset 0
>> object_hash rjenkins pg_num 300 pgp_num 300 last_change 447 flags
>> hashpspool crash_replay_interval 45 stripe_width 0
>> pool 2 'cephfs_metadata' replicated size 3 min_size 1 crush_ruleset 0
>> object_hash rjenkins pg_num 300 pgp_num 300 last_change 445 flags
>> hashpspool stripe_width 0
>>
>> 2016-07-03 09:49:40.695953 7f3da56f9700  0 -- 192.168.0.5:0/2773396901
>> >> 192.168.0.7:6789/0 pipe(0x7f3da0001f50 sd=3 :0 s=1 pgs=0 cs=0 l=1
>> c=0x7f3daf20).fault
>> 2016-07-03 09:49:44.195029 7f3da57fa700  0 -- 192.168.0.5:0/2773396901
>> >> 192.168.0.6:6789/0 pipe(0x7f3da0005500 sd=4 :0 s=1 pgs=0 cs=0 l=1
>> c=0x7f3da00067c0).fault
>> 2016-07-03 09:49:50.205788 7f3da55f8700  0 -- 192.168.0.5:0/2773396901
>> >> 192.168.0.6:6789/0 pipe(0x7f3da0005500 sd=3 :0 s=1 pgs=0 cs=0 l=1
>> c=0x7f3da0004c40).fault
>> 2016-07-03 09:49:52.720116 7f3da57fa700  0 -- 192.168.0.5:0/2773396901
>> >> 192.168.0.7:6789/0 pipe(0x7f3da00023f0 sd=4 :0 s=1 pgs=0 cs=0 l=1
>> c=0x7f3da00036b0).fault
>>
>> Regards - Willi
>>
>> Am 03.07.16 um 09:36 schrieb Sean Redmond:
>>
>> It would need to be set to 1
>> On 3 Jul 2016 8:17 a.m., "Willi Fehler"  wrote:
>>
>>> Hello David,
>>>
>>> so in a 3 node Cluster how should I set min_size if I want that 2 nodes
>>> could fail?
>>>
>>> Regards - Willi
>>>
>>> Am 28.06.16 um 13:07 schrieb David:
>>>
>>> Hi,
>>>
>>> This is probably the min_size on your cephfs data and/or metadata pool.
>>> I believe the default is 2, if you have less than 2 replicas available I/O
>>> will stop. See:
>>> http://docs.ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas
>>>
>>> On Tue, Jun 28, 2016 at 10:23 AM, willi.feh...@t-online.de <
>>> willi.feh...@t-online.de> wrote:
>>>
 Hello,

 I'm still very new to Ceph. I've created a small test Cluster.



 ceph-node1

 osd0

 osd1

 osd2

 ceph-node2

 osd3

 osd4

 osd5

 ceph-node3

 osd6

 osd7

 osd8



 My pool for CephFS has a replication count of 3. I've powered of 2
 nodes(6 OSDs went down) and my cluster status became critical and my ceph
 clients(cephfs) run into a timeout. My data(I had only one file on my pool)
 was still on one of the active OSDs. Is this the expected behaviour that
 the Cluster status became critical and my Clients run into a timeout?



 Many thanks for your feedback.



 Regards - Willi


 

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How many nodes/OSD can fail

2016-07-03 Thread Willi Fehler 

Hello Sean,

great. Thank you for your feedback.
Have a nice sunday.

Regards - Willi

Am 03.07.16 um 10:00 schrieb Sean Redmond:


Hi,

You will need 2 mons to be online.

Thanks

On 3 Jul 2016 8:58 a.m., "Willi Fehler" > wrote:


Hello Tu,

yes that's correct. The mon nodes run as well on the OSD nodes. So
I have

3 nodes in total. OSD, MDS and Mon on each Node.

Regards - Willi

Am 03.07.16 um 09:56 schrieb Tu Holmes:


Where are your mon nodes?

Were you mixing mon and OSD together?

Are 2 of the mon nodes down as well?

On Jul 3, 2016 12:53 AM, "Willi Fehler" mailto:willi.feh...@t-online.de>> wrote:

Hello Sean,

I've powered down 2 nodes. So 6 of 9 OSD are down. But my
client can't write and read anymore from my Ceph mount. Also
'ceph -s' hangs.

pool 1 'cephfs_data' replicated size 3 min_size 1
crush_ruleset 0 object_hash rjenkins pg_num 300 pgp_num 300
last_change 447 flags hashpspool crash_replay_interval 45
stripe_width 0
pool 2 'cephfs_metadata' replicated size 3 min_size 1
crush_ruleset 0 object_hash rjenkins pg_num 300 pgp_num 300
last_change 445 flags hashpspool stripe_width 0

2016-07-03 09:49:40.695953 7f3da56f9700  0 --
192.168.0.5:0/2773396901  >>
192.168.0.7:6789/0 
pipe(0x7f3da0001f50 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7f3daf20).fault
2016-07-03 09:49:44.195029 7f3da57fa700  0 --
192.168.0.5:0/2773396901  >>
192.168.0.6:6789/0 
pipe(0x7f3da0005500 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7f3da00067c0).fault
2016-07-03 09:49:50.205788 7f3da55f8700  0 --
192.168.0.5:0/2773396901  >>
192.168.0.6:6789/0 
pipe(0x7f3da0005500 sd=3 :0 s=1 pgs=0 cs=0 l=1
c=0x7f3da0004c40).fault
2016-07-03 09:49:52.720116 7f3da57fa700  0 --
192.168.0.5:0/2773396901  >>
192.168.0.7:6789/0 
pipe(0x7f3da00023f0 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7f3da00036b0).fault

Regards - Willi

Am 03.07.16 um 09:36 schrieb Sean Redmond:


It would need to be set to 1

On 3 Jul 2016 8:17 a.m., "Willi Fehler"
mailto:willi.feh...@t-online.de>>
wrote:

Hello David,

so in a 3 node Cluster how should I set min_size if I
want that 2 nodes could fail?

Regards - Willi

Am 28.06.16 um 13:07 schrieb David:

Hi,

This is probably the min_size on your cephfs data
and/or metadata pool. I believe the default is 2, if
you have less than 2 replicas available I/O will stop.
See:

http://docs.ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas

On Tue, Jun 28, 2016 at 10:23 AM,
willi.feh...@t-online.de

mailto:willi.feh...@t-online.de>> wrote:

Hello,

I'm still very new to Ceph. I've created a small
test Cluster.

ceph-node1

osd0

osd1

osd2

ceph-node2

osd3

osd4

osd5

ceph-node3

osd6

osd7

osd8

My pool for CephFS has a replication count of 3.
I've powered of 2 nodes(6 OSDs went down) and my
cluster status became critical and my ceph
clients(cephfs) run into a timeout. My data(I had
only one file on my pool) was still on one of the
active OSDs. Is this the expected behaviour that
the Cluster status became critical and my Clients
run into a timeout?

Many thanks for your feedback.

Regards - Willi



___
ceph-users mailing list
ceph-users@lists.ceph.com

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com





___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-use

Re: [ceph-users] How many nodes/OSD can fail

2016-07-03 Thread Tu Holmes 
I am kind of a newbie but I thought you needed 2 mons working at a minimum.

You should split those away onto some really budget hardware.

//Tu
Hello Tu,

yes that's correct. The mon nodes run as well on the OSD nodes. So I have

3 nodes in total. OSD, MDS and Mon on each Node.

Regards - Willi

Am 03.07.16 um 09:56 schrieb Tu Holmes:

Where are your mon nodes?

Were you mixing mon and OSD together?

Are 2 of the mon nodes down as well?
On Jul 3, 2016 12:53 AM, "Willi Fehler"  wrote:

> Hello Sean,
>
> I've powered down 2 nodes. So 6 of 9 OSD are down. But my client can't
> write and read anymore from my Ceph mount. Also 'ceph -s' hangs.
>
> pool 1 'cephfs_data' replicated size 3 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 300 pgp_num 300 last_change 447 flags
> hashpspool crash_replay_interval 45 stripe_width 0
> pool 2 'cephfs_metadata' replicated size 3 min_size 1 crush_ruleset 0
> object_hash rjenkins pg_num 300 pgp_num 300 last_change 445 flags
> hashpspool stripe_width 0
>
> 2016-07-03 09:49:40.695953 7f3da56f9700  0 -- 192.168.0.5:0/2773396901 >>
> 192.168.0.7:6789/0 pipe(0x7f3da0001f50 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f3daf20).fault
> 2016-07-03 09:49:44.195029 7f3da57fa700  0 -- 192.168.0.5:0/2773396901 >>
> 192.168.0.6:6789/0 pipe(0x7f3da0005500 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f3da00067c0).fault
> 2016-07-03 09:49:50.205788 7f3da55f8700  0 -- 192.168.0.5:0/2773396901 >>
> 192.168.0.6:6789/0 pipe(0x7f3da0005500 sd=3 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f3da0004c40).fault
> 2016-07-03 09:49:52.720116 7f3da57fa700  0 -- 192.168.0.5:0/2773396901 >>
> 192.168.0.7:6789/0 pipe(0x7f3da00023f0 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f3da00036b0).fault
>
> Regards - Willi
>
> Am 03.07.16 um 09:36 schrieb Sean Redmond:
>
> It would need to be set to 1
> On 3 Jul 2016 8:17 a.m., "Willi Fehler"  wrote:
>
>> Hello David,
>>
>> so in a 3 node Cluster how should I set min_size if I want that 2 nodes
>> could fail?
>>
>> Regards - Willi
>>
>> Am 28.06.16 um 13:07 schrieb David:
>>
>> Hi,
>>
>> This is probably the min_size on your cephfs data and/or metadata pool. I
>> believe the default is 2, if you have less than 2 replicas available I/O
>> will stop. See:
>> http://docs.ceph.com/docs/master/rados/operations/pools/#set-the-number-of-object-replicas
>>
>> On Tue, Jun 28, 2016 at 10:23 AM, willi.feh...@t-online.de <
>> willi.feh...@t-online.de> wrote:
>>
>>> Hello,
>>>
>>> I'm still very new to Ceph. I've created a small test Cluster.
>>>
>>>
>>>
>>> ceph-node1
>>>
>>> osd0
>>>
>>> osd1
>>>
>>> osd2
>>>
>>> ceph-node2
>>>
>>> osd3
>>>
>>> osd4
>>>
>>> osd5
>>>
>>> ceph-node3
>>>
>>> osd6
>>>
>>> osd7
>>>
>>> osd8
>>>
>>>
>>>
>>> My pool for CephFS has a replication count of 3. I've powered of 2
>>> nodes(6 OSDs went down) and my cluster status became critical and my ceph
>>> clients(cephfs) run into a timeout. My data(I had only one file on my pool)
>>> was still on one of the active OSDs. Is this the expected behaviour that
>>> the Cluster status became critical and my Clients run into a timeout?
>>>
>>>
>>>
>>> Many thanks for your feedback.
>>>
>>>
>>>
>>> Regards - Willi
>>>
>>>
>>> 
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Rebalance Issue

2016-07-03 Thread Roozbeh Shafiee 
Hi list,

A few days ago one of my OSDs failed and I dropped out that but afterwards I got
HEALTH_WARN until now. After turing off the OSD, the self-healing system started
to rebalance data between other OSDs.

My question is: At the end of rebalancing, the process doesn’t complete and I 
get this message
at the end of “ceph -s” output:

recovery io 1456 KB/s, 0 object/s

how can I back to HEALTH_OK situation again?

My cluster details are:

- 27 OSDs
- 3 MONs
- 2048 pg/pgs
- Each OSD has 4 TB of space
- CentOS 7.2 with 3.10 linux kernel
- Ceph Hammer version

Thank you,
Roozbeh___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Rebalance Issue

2016-07-03 Thread Wido den Hollander 

> Op 3 juli 2016 om 10:34 schreef Roozbeh Shafiee :
> 
> 
> Hi list,
> 
> A few days ago one of my OSDs failed and I dropped out that but afterwards I 
> got
> HEALTH_WARN until now. After turing off the OSD, the self-healing system 
> started
> to rebalance data between other OSDs.
> 
> My question is: At the end of rebalancing, the process doesn’t complete and I 
> get this message
> at the end of “ceph -s” output:
> 
> recovery io 1456 KB/s, 0 object/s
> 

Could you post the exact output of 'ceph -s'?

There is something more which needs to be shown.

'ceph health detail' also might tell you more.

Wido

> how can I back to HEALTH_OK situation again?
> 
> My cluster details are:
> 
> - 27 OSDs
> - 3 MONs
> - 2048 pg/pgs
> - Each OSD has 4 TB of space
> - CentOS 7.2 with 3.10 linux kernel
> - Ceph Hammer version
> 
> Thank you,
> Roozbeh___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Rebalance Issue

2016-07-03 Thread Roozbeh Shafiee 
Thanks for quick response, Wido

the "ceph -s" output has pasted here:
http://pastie.org/10897747

and this is output of “ceph health detail”:
http://pastebin.com/vMeURWC9

Thank you

> On Jul 3, 2016, at 1:10 PM, Wido den Hollander  wrote:
> 
> 
>> Op 3 juli 2016 om 10:34 schreef Roozbeh Shafiee :
>> 
>> 
>> Hi list,
>> 
>> A few days ago one of my OSDs failed and I dropped out that but afterwards I 
>> got
>> HEALTH_WARN until now. After turing off the OSD, the self-healing system 
>> started
>> to rebalance data between other OSDs.
>> 
>> My question is: At the end of rebalancing, the process doesn’t complete and 
>> I get this message
>> at the end of “ceph -s” output:
>> 
>> recovery io 1456 KB/s, 0 object/s
>> 
> 
> Could you post the exact output of 'ceph -s'?
> 
> There is something more which needs to be shown.
> 
> 'ceph health detail' also might tell you more.
> 
> Wido
> 
>> how can I back to HEALTH_OK situation again?
>> 
>> My cluster details are:
>> 
>> - 27 OSDs
>> - 3 MONs
>> - 2048 pg/pgs
>> - Each OSD has 4 TB of space
>> - CentOS 7.2 with 3.10 linux kernel
>> - Ceph Hammer version
>> 
>> Thank you,
>> Roozbeh___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Rebalance Issue

2016-07-03 Thread Wido den Hollander 

> Op 3 juli 2016 om 10:50 schreef Roozbeh Shafiee :
> 
> 
> Thanks for quick response, Wido
> 
> the "ceph -s" output has pasted here:
> http://pastie.org/10897747
> 
> and this is output of “ceph health detail”:
> http://pastebin.com/vMeURWC9
> 

It seems the cluster is still backfilling PGs and you 'ceph -s' shows so: 
'recovery io 62375 kB/s, 15 objects/s'

It will just take some time before it finishes.

Wido

> Thank you
> 
> > On Jul 3, 2016, at 1:10 PM, Wido den Hollander  wrote:
> > 
> > 
> >> Op 3 juli 2016 om 10:34 schreef Roozbeh Shafiee 
> >> :
> >> 
> >> 
> >> Hi list,
> >> 
> >> A few days ago one of my OSDs failed and I dropped out that but afterwards 
> >> I got
> >> HEALTH_WARN until now. After turing off the OSD, the self-healing system 
> >> started
> >> to rebalance data between other OSDs.
> >> 
> >> My question is: At the end of rebalancing, the process doesn’t complete 
> >> and I get this message
> >> at the end of “ceph -s” output:
> >> 
> >> recovery io 1456 KB/s, 0 object/s
> >> 
> > 
> > Could you post the exact output of 'ceph -s'?
> > 
> > There is something more which needs to be shown.
> > 
> > 'ceph health detail' also might tell you more.
> > 
> > Wido
> > 
> >> how can I back to HEALTH_OK situation again?
> >> 
> >> My cluster details are:
> >> 
> >> - 27 OSDs
> >> - 3 MONs
> >> - 2048 pg/pgs
> >> - Each OSD has 4 TB of space
> >> - CentOS 7.2 with 3.10 linux kernel
> >> - Ceph Hammer version
> >> 
> >> Thank you,
> >> Roozbeh___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Rebalance Issue

2016-07-03 Thread Roozbeh Shafiee 
Yes, you’re right but I have 0 object/s recovery last night. when I changed 
pg/pgp from 1400
to 2048, rebalancing speeded up but the percentage of rebalancing backed to 53%.

I have this situation again n again since I dropped out failed OSD when I 
increase pg/pgp but 
each time rebalancing stopped at 0 objects/s and low speed transfer.

Thanks

> On Jul 3, 2016, at 1:25 PM, Wido den Hollander  wrote:
> 
> 
>> Op 3 juli 2016 om 10:50 schreef Roozbeh Shafiee :
>> 
>> 
>> Thanks for quick response, Wido
>> 
>> the "ceph -s" output has pasted here:
>> http://pastie.org/10897747
>> 
>> and this is output of “ceph health detail”:
>> http://pastebin.com/vMeURWC9
>> 
> 
> It seems the cluster is still backfilling PGs and you 'ceph -s' shows so: 
> 'recovery io 62375 kB/s, 15 objects/s'
> 
> It will just take some time before it finishes.
> 
> Wido
> 
>> Thank you
>> 
>>> On Jul 3, 2016, at 1:10 PM, Wido den Hollander  wrote:
>>> 
>>> 
 Op 3 juli 2016 om 10:34 schreef Roozbeh Shafiee 
 :
 
 
 Hi list,
 
 A few days ago one of my OSDs failed and I dropped out that but afterwards 
 I got
 HEALTH_WARN until now. After turing off the OSD, the self-healing system 
 started
 to rebalance data between other OSDs.
 
 My question is: At the end of rebalancing, the process doesn’t complete 
 and I get this message
 at the end of “ceph -s” output:
 
 recovery io 1456 KB/s, 0 object/s
 
>>> 
>>> Could you post the exact output of 'ceph -s'?
>>> 
>>> There is something more which needs to be shown.
>>> 
>>> 'ceph health detail' also might tell you more.
>>> 
>>> Wido
>>> 
 how can I back to HEALTH_OK situation again?
 
 My cluster details are:
 
 - 27 OSDs
 - 3 MONs
 - 2048 pg/pgs
 - Each OSD has 4 TB of space
 - CentOS 7.2 with 3.10 linux kernel
 - Ceph Hammer version
 
 Thank you,
 Roozbeh___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Rebalance Issue

2016-07-03 Thread Wido den Hollander 

> Op 3 juli 2016 om 11:02 schreef Roozbeh Shafiee :
> 
> 
> Yes, you’re right but I have 0 object/s recovery last night. when I changed 
> pg/pgp from 1400
> to 2048, rebalancing speeded up but the percentage of rebalancing backed to 
> 53%.
> 

Why did you change that? I would not change that value while a cluster is still 
in recovery.

> I have this situation again n again since I dropped out failed OSD when I 
> increase pg/pgp but 
> each time rebalancing stopped at 0 objects/s and low speed transfer.
> 

Hard to judge at this point. You might want to try and restart osd.27 and see 
if that gets things going again. It seems to be involved in many PGs which are 
in 'backfilling' state.

Wido

> Thanks
> 
> > On Jul 3, 2016, at 1:25 PM, Wido den Hollander  wrote:
> > 
> > 
> >> Op 3 juli 2016 om 10:50 schreef Roozbeh Shafiee 
> >> :
> >> 
> >> 
> >> Thanks for quick response, Wido
> >> 
> >> the "ceph -s" output has pasted here:
> >> http://pastie.org/10897747
> >> 
> >> and this is output of “ceph health detail”:
> >> http://pastebin.com/vMeURWC9
> >> 
> > 
> > It seems the cluster is still backfilling PGs and you 'ceph -s' shows so: 
> > 'recovery io 62375 kB/s, 15 objects/s'
> > 
> > It will just take some time before it finishes.
> > 
> > Wido
> > 
> >> Thank you
> >> 
> >>> On Jul 3, 2016, at 1:10 PM, Wido den Hollander  wrote:
> >>> 
> >>> 
>  Op 3 juli 2016 om 10:34 schreef Roozbeh Shafiee 
>  :
>  
>  
>  Hi list,
>  
>  A few days ago one of my OSDs failed and I dropped out that but 
>  afterwards I got
>  HEALTH_WARN until now. After turing off the OSD, the self-healing system 
>  started
>  to rebalance data between other OSDs.
>  
>  My question is: At the end of rebalancing, the process doesn’t complete 
>  and I get this message
>  at the end of “ceph -s” output:
>  
>  recovery io 1456 KB/s, 0 object/s
>  
> >>> 
> >>> Could you post the exact output of 'ceph -s'?
> >>> 
> >>> There is something more which needs to be shown.
> >>> 
> >>> 'ceph health detail' also might tell you more.
> >>> 
> >>> Wido
> >>> 
>  how can I back to HEALTH_OK situation again?
>  
>  My cluster details are:
>  
>  - 27 OSDs
>  - 3 MONs
>  - 2048 pg/pgs
>  - Each OSD has 4 TB of space
>  - CentOS 7.2 with 3.10 linux kernel
>  - Ceph Hammer version
>  
>  Thank you,
>  Roozbeh___
>  ceph-users mailing list
>  ceph-users@lists.ceph.com
>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> 
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Rebalance Issue

2016-07-03 Thread Roozbeh Shafiee 
Actually I tried all the ways which I found them on Ceph Docs and mailing lists 
but
non of them had no effect. As a last resort I changed pg/pgp.

Anyway… What can I do as the best way to solve this problem?

Thanks

> On Jul 3, 2016, at 1:43 PM, Wido den Hollander  wrote:
> 
> 
>> Op 3 juli 2016 om 11:02 schreef Roozbeh Shafiee :
>> 
>> 
>> Yes, you’re right but I have 0 object/s recovery last night. when I changed 
>> pg/pgp from 1400
>> to 2048, rebalancing speeded up but the percentage of rebalancing backed to 
>> 53%.
>> 
> 
> Why did you change that? I would not change that value while a cluster is 
> still in recovery.
> 
>> I have this situation again n again since I dropped out failed OSD when I 
>> increase pg/pgp but 
>> each time rebalancing stopped at 0 objects/s and low speed transfer.
>> 
> 
> Hard to judge at this point. You might want to try and restart osd.27 and see 
> if that gets things going again. It seems to be involved in many PGs which 
> are in 'backfilling' state.
> 
> Wido
> 
>> Thanks
>> 
>>> On Jul 3, 2016, at 1:25 PM, Wido den Hollander  wrote:
>>> 
>>> 
 Op 3 juli 2016 om 10:50 schreef Roozbeh Shafiee 
 :
 
 
 Thanks for quick response, Wido
 
 the "ceph -s" output has pasted here:
 http://pastie.org/10897747
 
 and this is output of “ceph health detail”:
 http://pastebin.com/vMeURWC9
 
>>> 
>>> It seems the cluster is still backfilling PGs and you 'ceph -s' shows so: 
>>> 'recovery io 62375 kB/s, 15 objects/s'
>>> 
>>> It will just take some time before it finishes.
>>> 
>>> Wido
>>> 
 Thank you
 
> On Jul 3, 2016, at 1:10 PM, Wido den Hollander  wrote:
> 
> 
>> Op 3 juli 2016 om 10:34 schreef Roozbeh Shafiee 
>> :
>> 
>> 
>> Hi list,
>> 
>> A few days ago one of my OSDs failed and I dropped out that but 
>> afterwards I got
>> HEALTH_WARN until now. After turing off the OSD, the self-healing system 
>> started
>> to rebalance data between other OSDs.
>> 
>> My question is: At the end of rebalancing, the process doesn’t complete 
>> and I get this message
>> at the end of “ceph -s” output:
>> 
>> recovery io 1456 KB/s, 0 object/s
>> 
> 
> Could you post the exact output of 'ceph -s'?
> 
> There is something more which needs to be shown.
> 
> 'ceph health detail' also might tell you more.
> 
> Wido
> 
>> how can I back to HEALTH_OK situation again?
>> 
>> My cluster details are:
>> 
>> - 27 OSDs
>> - 3 MONs
>> - 2048 pg/pgs
>> - Each OSD has 4 TB of space
>> - CentOS 7.2 with 3.10 linux kernel
>> - Ceph Hammer version
>> 
>> Thank you,
>> Roozbeh___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
>> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RADOSGW buckets via NFS?

2016-07-03 Thread Sean Redmond 
Hi,

I noticed in the jewel release notes:

"You can now access radosgw buckets via NFS (experimental)."

Are there any docs that explain the configuration of NFS to access RADOSGW
buckets?

Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cluster failing to recover

2016-07-03 Thread Matyas Koszik 

Hi,

I recently upgraded to jewel (10.2.2) and now I'm confronted with a rather
strange behavior: recovey does not progress in the way it should. If I
restart the osds on a host, it'll get a bit better (or worse), like this:

50 pgs undersized
recovery 43775/7057285 objects degraded (0.620%)
recovery 87980/7057285 objects misplaced (1.247%)

[restart osds on node1]

44 pgs undersized
recovery 39623/7061519 objects degraded (0.561%)
recovery 92142/7061519 objects misplaced (1.305%)

[restart osds on node1]

43 pgs undersized
1116 requests are blocked > 32 sec
recovery 38181/7061529 objects degraded (0.541%)
recovery 90617/7061529 objects misplaced (1.283%)

...

The current state is this:

 osdmap e38804: 53 osds: 51 up, 51 in; 66 remapped pgs
  pgmap v14797137: 4388 pgs, 8 pools, 13626 GB data, 3434 kobjects
27474 GB used, 22856 GB / 50330 GB avail
38172/7061565 objects degraded (0.541%)
90617/7061565 objects misplaced (1.283%)
8/3517300 unfound (0.000%)
4202 active+clean
 109 active+recovery_wait+degraded
  38 active+undersized+degraded+remapped+wait_backfill
  15 active+remapped+wait_backfill
  11 active+clean+inconsistent
   8 active+recovery_wait+degraded+remapped
   3 active+recovering+undersized+degraded+remapped
   2 active+recovery_wait+undersized+degraded+remapped


All the pools have size=2 min_size=1.

(All the unfound blocks are on undersized pgs, and I cannot seem to be
able to fix them without having replicas (?). They exist, but are
outdated, from an earlier problem.)



Matyas


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cluster failing to recover

2016-07-03 Thread Oliver Dzombic 
Hi,

please provide:

ceph health detail

ceph osd tree

-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
IP-Interactive

mailto:i...@ip-interactive.de

Anschrift:

IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen

HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic

Steuer Nr.: 35 236 3622 1
UST ID: DE274086107


Am 03.07.2016 um 21:36 schrieb Matyas Koszik:
> 
> Hi,
> 
> I recently upgraded to jewel (10.2.2) and now I'm confronted with a rather
> strange behavior: recovey does not progress in the way it should. If I
> restart the osds on a host, it'll get a bit better (or worse), like this:
> 
> 50 pgs undersized
> recovery 43775/7057285 objects degraded (0.620%)
> recovery 87980/7057285 objects misplaced (1.247%)
> 
> [restart osds on node1]
> 
> 44 pgs undersized
> recovery 39623/7061519 objects degraded (0.561%)
> recovery 92142/7061519 objects misplaced (1.305%)
> 
> [restart osds on node1]
> 
> 43 pgs undersized
> 1116 requests are blocked > 32 sec
> recovery 38181/7061529 objects degraded (0.541%)
> recovery 90617/7061529 objects misplaced (1.283%)
> 
> ...
> 
> The current state is this:
> 
>  osdmap e38804: 53 osds: 51 up, 51 in; 66 remapped pgs
>   pgmap v14797137: 4388 pgs, 8 pools, 13626 GB data, 3434 kobjects
> 27474 GB used, 22856 GB / 50330 GB avail
> 38172/7061565 objects degraded (0.541%)
> 90617/7061565 objects misplaced (1.283%)
> 8/3517300 unfound (0.000%)
> 4202 active+clean
>  109 active+recovery_wait+degraded
>   38 active+undersized+degraded+remapped+wait_backfill
>   15 active+remapped+wait_backfill
>   11 active+clean+inconsistent
>8 active+recovery_wait+degraded+remapped
>3 active+recovering+undersized+degraded+remapped
>2 active+recovery_wait+undersized+degraded+remapped
> 
> 
> All the pools have size=2 min_size=1.
> 
> (All the unfound blocks are on undersized pgs, and I cannot seem to be
> able to fix them without having replicas (?). They exist, but are
> outdated, from an earlier problem.)
> 
> 
> 
> Matyas
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cluster failing to recover

2016-07-03 Thread Oliver Dzombic 
Hi,

did you already do something ( replacing drives or changing something ) ?

You have 11 scrub errors, and ~ 11x inconsistent pg's

The inconsistent pg's, for example:

pg 4.3a7 is stuck unclean for 629.766502, current state
active+recovery_wait+degraded+inconsistent, last acting [10,21]

are not on the down osd's 1 and 22

neighter of them.

So the should not be missing. But they are.

Anyway, i think the next step would be to start a pg_repair command and
see where the road goes.

-- 
Mit freundlichen Gruessen / Best regards

Oliver Dzombic
IP-Interactive

mailto:i...@ip-interactive.de

Anschrift:

IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhausen

HRB 93402 beim Amtsgericht Hanau
Geschäftsführung: Oliver Dzombic

Steuer Nr.: 35 236 3622 1
UST ID: DE274086107


Am 03.07.2016 um 23:59 schrieb Matyas Koszik:
> 
> Hi,
> 
> I've continued restarting osds in the meantime, and it got somewhat
> better, but still very far from optimal.
> 
> Here're the details you requested:
> 
> http://pastebin.com/Vqgadz24
> 
> http://pastebin.com/vCL6BRvC
> 
> Matyas
> 
> 
> On Sun, 3 Jul 2016, Oliver Dzombic wrote:
> 
>> Hi,
>>
>> please provide:
>>
>> ceph health detail
>>
>> ceph osd tree
>>
>> --
>> Mit freundlichen Gruessen / Best regards
>>
>> Oliver Dzombic
>> IP-Interactive
>>
>> mailto:i...@ip-interactive.de
>>
>> Anschrift:
>>
>> IP Interactive UG ( haftungsbeschraenkt )
>> Zum Sonnenberg 1-3
>> 63571 Gelnhausen
>>
>> HRB 93402 beim Amtsgericht Hanau
>> Geschäftsführung: Oliver Dzombic
>>
>> Steuer Nr.: 35 236 3622 1
>> UST ID: DE274086107
>>
>>
>> Am 03.07.2016 um 21:36 schrieb Matyas Koszik:
>>>
>>> Hi,
>>>
>>> I recently upgraded to jewel (10.2.2) and now I'm confronted with a rather
>>> strange behavior: recovey does not progress in the way it should. If I
>>> restart the osds on a host, it'll get a bit better (or worse), like this:
>>>
>>> 50 pgs undersized
>>> recovery 43775/7057285 objects degraded (0.620%)
>>> recovery 87980/7057285 objects misplaced (1.247%)
>>>
>>> [restart osds on node1]
>>>
>>> 44 pgs undersized
>>> recovery 39623/7061519 objects degraded (0.561%)
>>> recovery 92142/7061519 objects misplaced (1.305%)
>>>
>>> [restart osds on node1]
>>>
>>> 43 pgs undersized
>>> 1116 requests are blocked > 32 sec
>>> recovery 38181/7061529 objects degraded (0.541%)
>>> recovery 90617/7061529 objects misplaced (1.283%)
>>>
>>> ...
>>>
>>> The current state is this:
>>>
>>>  osdmap e38804: 53 osds: 51 up, 51 in; 66 remapped pgs
>>>   pgmap v14797137: 4388 pgs, 8 pools, 13626 GB data, 3434 kobjects
>>> 27474 GB used, 22856 GB / 50330 GB avail
>>> 38172/7061565 objects degraded (0.541%)
>>> 90617/7061565 objects misplaced (1.283%)
>>> 8/3517300 unfound (0.000%)
>>> 4202 active+clean
>>>  109 active+recovery_wait+degraded
>>>   38 active+undersized+degraded+remapped+wait_backfill
>>>   15 active+remapped+wait_backfill
>>>   11 active+clean+inconsistent
>>>8 active+recovery_wait+degraded+remapped
>>>3 active+recovering+undersized+degraded+remapped
>>>2 active+recovery_wait+undersized+degraded+remapped
>>>
>>>
>>> All the pools have size=2 min_size=1.
>>>
>>> (All the unfound blocks are on undersized pgs, and I cannot seem to be
>>> able to fix them without having replicas (?). They exist, but are
>>> outdated, from an earlier problem.)
>>>
>>>
>>>
>>> Matyas
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fwd: Ceph installation and integration with Openstack

2016-07-03 Thread Gaurav Goyal 
Dear All,

I need your kind help please. I am new and want to understand the ceph
installation concept as per my lab setup.

Regards
Gaurav Goyal
On 02-Jul-2016 7:27 pm, "Gaurav Goyal"  wrote:

> Dear Ceph Users,
>
> I am very new to Ceph product and want to gain some knowledge for my lab
> setup.
>
> Situation is --> I have installed openstack setup (Liberty) for my lab.
>
> Host 1 --> Controller + Compute1
> Host 2  --> Compute 2
>
> DELL SAN storage is attached to both hosts as
>
> [root@OSKVM1 ~]# iscsiadm -m node
>
> 10.35.0.3:3260,1
> iqn.2001-05.com.equallogic:0-1cb196-07a83c107-4770018575af-vol1
>
> 10.35.0.8:3260,1
> iqn.2001-05.com.equallogic:0-1cb196-07a83c107-4770018575af-vol1
>
> 10.35.0.*:3260,-1
> iqn.2001-05.com.equallogic:0-1cb196-20d83c107-729002157606-vol2
>
> 10.35.0.8:3260,1
> iqn.2001-05.com.equallogic:0-1cb196-20d83c107-729002157606-vol2
>
> 10.35.0.*:3260,-1
> iqn.2001-05.com.equallogic:0-1cb196-f0783c107-70a00245761a-vol3
>
> 10.35.0.8:3260,1
> iqn.2001-05.com.equallogic:0-1cb196-f0783c107-70a00245761a-vol3
>
> 10.35.0.*:3260,-1
> iqn.2001-05.com.equallogic:0-1cb196-fda83c107-92700275761a-vol4
> 10.35.0.8:3260,1
> iqn.2001-05.com.equallogic:0-1cb196-fda83c107-92700275761a-vol4
>
> I need to configure this SAN storage as CEPH. SO i want to know how to
> install Ceph Hammer version.  I need to use ceph as block storage for my
> openstack environment.
>
> I was following this link for ceph installation  "
> http://docs.ceph.com/docs/master/start/quick-ceph-deploy/";
> Please help me with correct link for ceph installation and integration
> with Openstack.
>
> 1.  I think ceph-deploy, Mon0 , OSD0 would be host 1
>   OSD1 would be Host 2
>   Is it ok?
>
> 2.  What should be filesystem for ceph?
> 3. Basically, this is first time so want to have detailed
> information/guidance for confidence.
>
>
> Regards
> Gaurav Goyal
> +1647-685-3000
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Fwd: Ceph installation and integration with Openstack

2016-07-03 Thread Gaurav Goyal 
Dear Ceph Users,

I am very new to Ceph product and want to gain some knowledge for my lab
setup.

Situation is --> I have installed openstack setup (Liberty) for my lab.

Host 1 --> Controller + Compute1
Host 2  --> Compute 2

DELL SAN storage is attached to both hosts as

[root@OSKVM1 ~]# iscsiadm -m node

10.35.0.3:3260,1
iqn.2001-05.com.equallogic:0-1cb196-07a83c107-4770018575af-vol1

10.35.0.8:3260,1
iqn.2001-05.com.equallogic:0-1cb196-07a83c107-4770018575af-vol1

10.35.0.*:3260,-1
iqn.2001-05.com.equallogic:0-1cb196-20d83c107-729002157606-vol2

10.35.0.8:3260,1
iqn.2001-05.com.equallogic:0-1cb196-20d83c107-729002157606-vol2

10.35.0.*:3260,-1
iqn.2001-05.com.equallogic:0-1cb196-f0783c107-70a00245761a-vol3

10.35.0.8:3260,1
iqn.2001-05.com.equallogic:0-1cb196-f0783c107-70a00245761a-vol3

10.35.0.*:3260,-1
iqn.2001-05.com.equallogic:0-1cb196-fda83c107-92700275761a-vol4
10.35.0.8:3260,1
iqn.2001-05.com.equallogic:0-1cb196-fda83c107-92700275761a-vol4

I need to configure this SAN storage as CEPH. SO i want to know how to
install Ceph Hammer version.  I need to use ceph as block storage for my
openstack environment.

I was following this link for ceph installation  "
http://docs.ceph.com/docs/master/start/quick-ceph-deploy/";
Please help me with correct link for ceph installation and integration with
Openstack.

1.  I think ceph-deploy, Mon0 , OSD0 would be host 1
  OSD1 would be Host 2
  Is it ok?

2.  What should be filesystem for ceph?
3. Basically, this is first time so want to have detailed
information/guidance for confidence.


Regards
Gaurav Goyal
+1647-685-3000
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RADOSGW buckets via NFS?

2016-07-03 Thread Brad Hubbard 
On Sun, Jul 3, 2016 at 9:07 PM, Sean Redmond  wrote:
> Hi,
>
> I noticed in the jewel release notes:
>
> "You can now access radosgw buckets via NFS (experimental)."
>
> Are there any docs that explain the configuration of NFS to access RADOSGW
> buckets?

Here's what I found.

http://tracker.ceph.com/projects/ceph/wiki/RGW_-_NFS
https://github.com/nfs-ganesha/nfs-ganesha/tree/next/src/FSAL/FSAL_RGW
https://www.youtube.com/watch?v=zWURdwudAUI

It looks like the information in the video and docs about s3fs-fuse is
no longer relevant.

The file src/test/librgw_file_nfsns.cc in the source tree gives a
little insight.

It looks to me as though the NFS-Ganesha FSAL mounts a bucket using
LibRGW and leverages this library to perform needed operations.

Configuration would involve setting up the FSAL to *point* to the
relevant bucket correctly.
It's very likely the generic Ganesha docs can help here but I have no
experience with that
I'm afraid.

I can't find much more info at this time, which doesn't mean it doesn't exist :)

HTH,
Brad

>
> Thanks
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Active MON aborts on Jewel 10.2.2 with FAILED assert(info.state == MDSMap::STATE_STANDBY

2016-07-03 Thread Bill Sharer 
I was working on  a rolling upgrade on Gentoo to Jewel 10.2.2 from 
10.2.0.  However now I can't get a monitor quorum going again because as 
soon as I get one, the mon which wins the election blows out with an 
assertion failure.  Here's my status at the moment


kroll110.2.2ceph mon.0 and ceph osd.0 normally my lead mon
kroll210.2.2ceph mon 1 and ceph osd 2
kroll310.2.2ceph osd 1
kroll410.2.2ceph mon 3 and ceph osd 3
kroll510.2.2ceph mon 4 and ceph mds 2 normally my active mds
kroll610.2.0ceph mon 5 and ceph mds B normally standby mds

I had done rolling upgrade of everything but kroll6 and had rebooted the 
first three osd and mon servers.  mds 2 went down during gentoo update 
of kroll5 because of memory scarcity so mds B was the active mds 
server.  After rebooting kroll4 I found that mon 0 had gone done with 
the assertion failure.  I ended up stopping all ceph processes but 
desktops with client mounts were all still up for the moment and 
basically would be stuck on locks if I tried to access cephfs.


After trying to restart mons only beginning with mon 0 initially, the 
following happened to mon.0 after enough mons were up for a quorum:


2016-07-03 16:34:26.555728 7fbff22f8480  1 leveldb: Recovering log #2592390
2016-07-03 16:34:26.555762 7fbff22f8480  1 leveldb: Level-0 table 
#2592397: started
2016-07-03 16:34:26.558788 7fbff22f8480  1 leveldb: Level-0 table 
#2592397: 192 bytes OK

2016-07-03 16:34:26.562263 7fbff22f8480  1 leveldb: Delete type=3 #2592388

2016-07-03 16:34:26.562364 7fbff22f8480  1 leveldb: Delete type=0 #2592390

2016-07-03 16:34:26.563126 7fbff22f8480 -1 wrote monmap to 
/etc/ceph/tmpmonmap
2016-07-03 17:09:25.753729 7f8291dff480  0 ceph version 10.2.2 
(45107e21c568dd033c2f0a3107dec8f0b0e58374), pro

cess ceph-mon, pid 20842
2016-07-03 17:09:25.762588 7f8291dff480  1 leveldb: Recovering log #2592398
2016-07-03 17:09:25.767722 7f8291dff480  1 leveldb: Delete type=0 #2592398

2016-07-03 17:09:25.767803 7f8291dff480  1 leveldb: Delete type=3 #2592396

2016-07-03 17:09:25.768600 7f8291dff480  0 starting mon.0 rank 0 at 
192.168.2.1:6789/0 mon_data /var/lib/ceph/mon/ceph-0 fsid 
1798897a-f0c9-422d-86b3-d4933a12c7ac
2016-07-03 17:09:25.769066 7f8291dff480  1 mon.0@-1(probing) e10 preinit 
fsid 1798897a-f0c9-422d-86b3-d4933a12c7ac
2016-07-03 17:09:25.769923 7f8291dff480  1 
mon.0@-1(probing).paxosservice(pgmap 17869652..17870289) refresh 
upgraded, format 0 -> 1
2016-07-03 17:09:25.769947 7f8291dff480  1 mon.0@-1(probing).pg v0 
on_upgrade discarding in-core PGMap
2016-07-03 17:09:25.776148 7f8291dff480  0 mon.0@-1(probing).mds e1532 
print_map

e1532
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable 
ranges,3=default file layouts on dirs,4=dir inode in separate 
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no 
anchor table}


Filesystem 'cephfs' (0)
fs_name cephfs
epoch   1530
flags   0
modified2016-05-19 01:21:31.953710
tableserver 0
root0
session_timeout 60
session_autoclose   300
max_file_size   1099511627776
last_failure1478
last_failure_osd_epoch  26431
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable 
ranges,3=default file layouts on dirs,4=dir inode in separate 
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no 
anchor table}

max_mds 1
in  0
up  {0=1190233}
failed
damaged
stopped
data_pools  0
metadata_pool   1
inline_data disabled
1190233:192.168.2.6:6800/5437 'B' mds.0.1526 up:active seq 103145


Standby daemons:

1190222:192.168.2.5:6801/5871 '2' mds.-1.0 up:standby seq 135114

2016-07-03 17:09:25.776444 7f8291dff480  0 mon.0@-1(probing).osd e26460 
crush map has features 2200130813952, adjusting msgr requires
2016-07-03 17:09:25.776450 7f8291dff480  0 mon.0@-1(probing).osd e26460 
crush map has features 2200130813952, adjusting msgr requires
2016-07-03 17:09:25.776453 7f8291dff480  0 mon.0@-1(probing).osd e26460 
crush map has features 2200130813952, adjusting msgr requires
2016-07-03 17:09:25.776454 7f8291dff480  0 mon.0@-1(probing).osd e26460 
crush map has features 2200130813952, adjusting msgr requires
2016-07-03 17:09:25.776696 7f8291dff480  1 
mon.0@-1(probing).paxosservice(auth 19251..19344) refresh upgraded,

 format 0 -> 1
2016-07-03 17:09:25.777650 7f8291dff480  0 mon.0@-1(probing) e10  my 
rank is now 0 (was -1)
2016-07-03 17:09:25.778191 7f8281e89700  0 -- 192.168.2.1:6789/0 >> 
192.168.2.5:6789/0 pipe(0x556e0ab1d360 sd=15 :0 s=1 pgs=0 cs=0 l=0 
c=0x556e0ab1e620).fault
2016-07-03 17:09:25.778257 7f828208b700  0 -- 192.168.2.1:6789/0 >> 
192.168.2.2:6789/0 pipe(0x556e0ab17c80 sd=13 :0 s=1 pgs=0 cs=0 l=0 
c=0x556e0ab18f40).fault
2016-07-03 17:09:25.778314 7f8281d88700  0 -- 192.168.2.1:6789/0 >> 
192.168.2.6:6789/0 pipe(0x556e0ab1fe50 sd=16 :0 s=1 pgs=0 cs=0 l=0 
c=0x556e0ab21160).fault
2016-07-03 17:09:25.778364 7f8281f8a700

[ceph-users] Radosgw performance degradation

2016-07-03 Thread Andrey Komarov 
Hi guys.
I am currently encountering strange problem with radosgw.

My setup is:
3 mons, 40OSD  4Tb hdd + 8G SSD journal each on 4 servers and 2 RGW.
I have rgw_override_bucket_index_max_shards = 2048 in config and 100 buckets 
with ~3M objects each. Objects are relatively small: from 1k to 100k.
So my problem is big performance degradation for PUT operations with objects 
number growing. When I started upload, I had ~ 500 r/s for PUT operations. Now 
I have ~60 r/s.

I have these lines in ceph -w output:
slow request 30.190198 seconds old, received at 2016-07-04 03:12:23.354199: 
osd_op(client.316897.0:3317502521 82.78329d6 
3f8804ed-d15b-4a1b-a9ec-e95027dbb8c3.207388.28_22501/1-gallery.webp [create 0~0 
[excl],setxattr user.rgw.idtag (54),writefull 0~420,setxattr user.rgw.manifest 
(738),setxattr user.rgw.acl (175),setxattr user.rgw.content_type (11),setxattr 
user.rgw.etag (33),call rgw.obj_store_pg_ver,setxattr user.rgw.source_zone (4)] 
snapc 0=[] ondisk+write+known_if_redirected e4051) currently commit_sent

 I’ve found similar issue here:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-December/006838.html 

But unfortunately, I can’t find any replies to this question.
I suspect what it may be related to objects attributes saving, so, my question 
is: 

Anyone else here had similar problems? If so, is there any guidances to solve 
it?
Thanks.


Andrey Komarov | system administrator 
mailto: andrey.koma...@lazada.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com