date:20200321

[ceph-users] Maximum limit of lifecycle rule length

2020-03-21 Thread Amit Ghadge

He All,

We set rgw_lc_max_rules to 1 but we seen the issue while the xml rule
length are > 1MB It return InvalidRange, the format is below,

http://s3.amazonaws.com/doc/2006-03-01/;>

Enabledtest_1/1.
.
.
.


Any reason why Ceph not allowed lc rule length > 1MB.


Thanks,
AmitG
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs mount error 1 = Operation not permitted

2020-03-21 Thread Dungan, Scott A.

Zitat, thanks for the tips.

I tried appending the key directly in the mount command 
(secret=) and that produced the same error.

I took a look at the thread you suggested and I ran the commands that Paul at 
Croit suggested even though I the ceph dashboard showed "cephs" as already set 
as the application on both my data and metadata pools:

[root@ceph-n4 ~]# ceph osd pool application set data cephfs data cephfs
set application 'cephfs' key 'data' to 'cephfs' on pool 'data'
[root@ceph-n4 ~]# ceph osd pool application set meta_data cephfs metadata cephfs
set application 'cephfs' key 'metadata' to 'cephfs' on pool 'meta_data'

No change. I get the "mount error 1 = Operation not permitted" error the same 
as before.

I also tried manually editing the caps osd pool tags for my client.1, to allow 
rw to both the data pool as well as the metadata pool, as suggested further in 
the thread:

[client.1]
key = ***
caps mds = "allow rw path=all"
caps mon = "allow r"
caps osd = "allow rw tag cephfs pool=meta_data, allow rw pool=data"

No change.


From: Eugen Block 
Sent: Saturday, March 21, 2020 1:16 PM
To: ceph-users@ceph.io 
Subject: [ceph-users] Re: Cephfs mount error 1 = Operation not permitted

I just remembered there was a thread [1] about that a couple of weeks
ago. Seems like you need to add the capabilities to the client.

[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDVMYGCUTALACPFAJYITLOHJ/#I6LJR72AJGOCGINVOVEVSCKRIWV5TTZ2


Zitat von Eugen Block :

> Hi,
>
> have you tried to mount with the secret only instead of a secret file?
>
> mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=
>
> If that works your secret file is not right. If not you should check
> if the client actually has access to the cephfs pools ('ceph auth
> list').
>
>
>
> Zitat von "Dungan, Scott A." :
>
>> I am still very new to ceph and I have just set up my first small
>> test cluster. I have Cephfs enabled (named cephfs) and everything
>> is good in the dashboard. I added an authorized user key for cephfs
>> with:
>>
>> ceph fs authorize cephfs client.1 / r / rw
>>
>> I then copied the key to a file with:
>>
>> ceph auth get-key client.1 > /tmp/client.1.secret
>>
>> Copied the file over to the client and then attempt mount witth the
>> kernel driver:
>>
>> mount -t ceph ceph-n4:6789:/ /ceph -o
>> name=client.1,secretfile=/root/client.1.secret
>> mount error 1 = Operation not permitted
>>
>> I looked in the logs on the mds (which is also the mgr and mon for
>> the cluster) and I don't see any events logged for this. I also
>> tried the mount command with verbose and I didn't get any further
>> detail. Any tips would be most appreciated.
>>
>> --
>>
>> Scott Dungan
>> California Institute of Technology
>> Office: (626) 395-3170
>> sdun...@caltech.edu
>>
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs mount error 1 = Operation not permitted

2020-03-21 Thread Eugen Block

I just remembered there was a thread [1] about that a couple of weeks
ago. Seems like you need to add the capabilities to the client.

[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDVMYGCUTALACPFAJYITLOHJ/#I6LJR72AJGOCGINVOVEVSCKRIWV5TTZ2

Zitat von Eugen Block :

Hi,

have you tried to mount with the secret only instead of a secret file?

mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=

If that works your secret file is not right. If not you should check
if the client actually has access to the cephfs pools ('ceph auth
list').

Zitat von "Dungan, Scott A." :

I am still very new to ceph and I have just set up my first small
test cluster. I have Cephfs enabled (named cephfs) and everything
is good in the dashboard. I added an authorized user key for cephfs
with:

ceph fs authorize cephfs client.1 / r / rw

I then copied the key to a file with:

ceph auth get-key client.1 > /tmp/client.1.secret

Copied the file over to the client and then attempt mount witth the
kernel driver:

mount -t ceph ceph-n4:6789:/ /ceph -o
name=client.1,secretfile=/root/client.1.secret

mount error 1 = Operation not permitted

I looked in the logs on the mds (which is also the mgr and mon for
the cluster) and I don't see any events logged for this. I also
tried the mount command with verbose and I didn't get any further
detail. Any tips would be most appreciated.

Scott Dungan
California Institute of Technology
Office: (626) 395-3170
sdun...@caltech.edu

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephfs mount error 1 = Operation not permitted

2020-03-21 Thread Eugen Block


Hi,

have you tried to mount with the secret only instead of a secret file?

mount -t ceph ceph-n4:6789:/ /ceph -o name=client.1,secret=

If that works your secret file is not right. If not you should check  
if the client actually has access to the cephfs pools ('ceph auth  
list').




Zitat von "Dungan, Scott A." :

I am still very new to ceph and I have just set up my first small  
test cluster. I have Cephfs enabled (named cephfs) and everything is  
good in the dashboard. I added an authorized user key for cephfs with:


ceph fs authorize cephfs client.1 / r / rw

I then copied the key to a file with:

ceph auth get-key client.1 > /tmp/client.1.secret

Copied the file over to the client and then attempt mount witth the  
kernel driver:


mount -t ceph ceph-n4:6789:/ /ceph -o  
name=client.1,secretfile=/root/client.1.secret

mount error 1 = Operation not permitted

I looked in the logs on the mds (which is also the mgr and mon for  
the cluster) and I don't see any events logged for this. I also  
tried the mount command with verbose and I didn't get any further  
detail. Any tips would be most appreciated.


--

Scott Dungan
California Institute of Technology
Office: (626) 395-3170
sdun...@caltech.edu

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

2020-03-21 Thread Jan Pekař - Imatic

I understand, so I expected slow requests (like X slow requests are blocked > 32 sec) but I was not expecting that heartbeats are missed or
OSD's were restarted.

Maybe this "hard recovery" was not tested enough.

Also I'm concerned, that this OSD restart caused data degradation and recovery - cluster should be clean immediately after OSD up when no
client was uploading/modifying data during my tests.

With regards
Jan Pekar

On 21/03/2020 15.56, Anthony D'Atri wrote:

This is an expensive operation. You want to slow it down, not burden the OSDs.

On Mar 21, 2020, at 5:46 AM, Jan Pekař - Imatic wrote:

Each node has 64GB RAM so it should be enough (12 OSD's = 48GB used).

On 21/03/2020 13.14, XuYun wrote:
Bluestore requires more than 4G memory per OSD, do you have enough memory?

2020年3月21日 下午8:09，Jan Pekař - Imatic 写道：

Hello,

I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8)
nautilus (stable)

4 nodes - each node 11 HDD, 1 SSD, 10Gbit network

Cluster was empty, fresh install. We filled cluster with data (small blocks)
using RGW.

Cluster is now used for testing so no client was using it during my admin
operations mentioned below

After a while (7TB of data / 40M objects uploaded) we decided, that we increase
pg_num from 128 to 256 to better spread data and to speedup this operation,
I've set

ceph config set mgr target_max_misplaced_ratio 1

so that whole cluster rebalance as quickly as it can.

I have 3 issues/questions below:

I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to
restart with logged

heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had suicide
timed out after 150

after a while OSD's were back so I continued after a while with my tests.

My question - increasing number of PG with maximal target_max_misplaced_ratio
was too much for that OSDs? It is not recommended to do it this way? I had no
problem with this increase before, but configuration of cluster was slightly
different and it was luminous version.

Rebuild was still slow so I increased number of backfills

ceph tell osd.* injectargs "--osd-max-backfills 10"

and reduced recovery sleep time

ceph tell osd.* injectargs "--osd-recovery-sleep-hdd 0.01"

and after few hours I noticed, that some of my OSD's were restarted during
recovery, in log I can see

...

|2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21
06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread
0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 1
heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed
out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log
[WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21
06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 wrongly
marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574
start_waiting_for_healthy |

I observed network graph usage and network utilization was low during recovery
(10Gbit was not saturated).

So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought that
OSD is using threads and HDD timeouts are not influencing heartbeats to other
OSD's and MON. It looks like it is not true.

After OSD was wrongly marked down I can see that cluster has object degraded.
There were no degraded object before that.

Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs
degraded, 8 pgs undersized

It means that this OSD disconnection causes data degraded? How is it possible,
when no OSD was lost. Data should be on that OSD and after peering should be
everything OK. With luminous I had no problem, after OSD up degraded objects
where recovered/found during few seconds and cluster was healthy within seconds.

Thank you very much for additional info. I can perform additional tests you
recommend because cluster is used for testing purpose now.

With regards
Jan Pekar

Ing. Jan Pekař
jan.pe...@imatic.cz

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz | +420326555326

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Ing. Jan Pekař
jan.pe...@imatic.cz

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz | +420326555326

--
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Ing. Jan Pekař
jan.pe...@imatic.cz

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz | +420326555326

--
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send

[ceph-users] Cephfs mount error 1 = Operation not permitted

2020-03-21 Thread Dungan, Scott A.

I am still very new to ceph and I have just set up my first small test cluster. 
I have Cephfs enabled (named cephfs) and everything is good in the dashboard. I 
added an authorized user key for cephfs with:

ceph fs authorize cephfs client.1 / r / rw

I then copied the key to a file with:

ceph auth get-key client.1 > /tmp/client.1.secret

Copied the file over to the client and then attempt mount witth the kernel 
driver:

mount -t ceph ceph-n4:6789:/ /ceph -o 
name=client.1,secretfile=/root/client.1.secret
mount error 1 = Operation not permitted

I looked in the logs on the mds (which is also the mgr and mon for the cluster) 
and I don't see any events logged for this. I also tried the mount command with 
verbose and I didn't get any further detail. Any tips would be most appreciated.

--

Scott Dungan
California Institute of Technology
Office: (626) 395-3170
sdun...@caltech.edu

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

2020-03-21 Thread Anthony D'Atri

This is an expensive operation.  You want to slow it down, not burden the OSDs. 

> On Mar 21, 2020, at 5:46 AM, Jan Pekař - Imatic  wrote:
> 
> Each node has 64GB RAM so it should be enough (12 OSD's = 48GB used).
> 
>> On 21/03/2020 13.14, XuYun wrote:
>> Bluestore requires more than 4G memory per OSD, do you have enough memory?
>> 
>>> 2020年3月21日 下午8:09，Jan Pekař - Imatic  写道：
>>> 
>>> Hello,
>>> 
>>> I have ceph cluster version 14.2.7 
>>> (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)
>>> 
>>> 4 nodes - each node 11 HDD, 1 SSD, 10Gbit network
>>> 
>>> Cluster was empty, fresh install. We filled cluster with data (small 
>>> blocks) using RGW.
>>> 
>>> Cluster is now used for testing so no client was using it during my admin 
>>> operations mentioned below
>>> 
>>> After a while (7TB of data / 40M objects uploaded) we decided, that we 
>>> increase pg_num from 128 to 256 to better spread data and to speedup this 
>>> operation, I've set
>>> 
>>>  ceph config set mgr target_max_misplaced_ratio 1
>>> 
>>> so that whole cluster rebalance as quickly as it can.
>>> 
>>> I have 3 issues/questions below:
>>> 
>>> 1)
>>> 
>>> I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to 
>>> restart with logged
>>> 
>>> heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had 
>>> suicide timed out after 150
>>> 
>>> after a while OSD's were back so I continued after a while with my tests.
>>> 
>>> My question - increasing number of PG with maximal 
>>> target_max_misplaced_ratio was too much for that OSDs? It is not 
>>> recommended to do it this way? I had no problem with this increase before, 
>>> but configuration of cluster was slightly different and it was luminous 
>>> version.
>>> 
>>> 2)
>>> 
>>> Rebuild was still slow so I increased number of backfills
>>> 
>>>  ceph tell osd.*  injectargs "--osd-max-backfills 10"
>>> 
>>> and reduced recovery sleep time
>>> 
>>>  ceph tell osd.*  injectargs "--osd-recovery-sleep-hdd 0.01"
>>> 
>>> and after few hours I noticed, that some of my OSD's were restarted during 
>>> recovery, in log I can see
>>> 
>>> ...
>>> 
>>> |2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 
>>> 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 
>>> 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
>>> 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 
>>> 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had 
>>> timed out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 
>>> log_channel(cluster) log [WRN] : Monitor daemon marked osd.7 down, but it 
>>> is still running 2020-03-21 06:41:36.888 7fe1e7769700 0 
>>> log_channel(cluster) log [DBG] : map e3574 wrongly marked me down at e3573 
>>> 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574 start_waiting_for_healthy 
>>> |
>>> 
>>> I observed network graph usage and network utilization was low during 
>>> recovery (10Gbit was not saturated).
>>> 
>>> So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought 
>>> that OSD is using threads and HDD timeouts are not influencing heartbeats 
>>> to other OSD's and MON. It looks like it is not true.
>>> 
>>> 3)
>>> 
>>> After OSD was wrongly marked down I can see that cluster has object 
>>> degraded. There were no degraded object before that.
>>> 
>>>  Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 
>>> pgs degraded, 8 pgs undersized
>>> 
>>> It means that this OSD disconnection causes data degraded? How is it 
>>> possible, when no OSD was lost. Data should be on that OSD and after 
>>> peering should be everything OK. With luminous I had no problem, after OSD 
>>> up degraded objects where recovered/found during few seconds and cluster 
>>> was healthy within seconds.
>>> 
>>> Thank you very much for additional info. I can perform additional tests you 
>>> recommend because cluster is used for testing purpose now.
>>> 
>>> With regards
>>> Jan Pekar
>>> 
>>> -- 
>>> 
>>> Ing. Jan Pekař
>>> jan.pe...@imatic.cz
>>> 
>>> Imatic | Jagellonská 14 | Praha 3 | 130 00
>>> http://www.imatic.cz | +420326555326
>>> 
>>> --
>>> 
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> -- 
> 
> Ing. Jan Pekař
> jan.pe...@imatic.cz
> 
> Imatic | Jagellonská 14 | Praha 3 | 130 00
> http://www.imatic.cz | +420326555326
> 
> --
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

2020-03-21 Thread XuYun

We had a similar problem that caused by insufficient RAM: we have 6 OSDs and 
32G RAM per host, and somehow swap partition was used by OS, which lead 
sporadic performance problem.

> 2020年3月21日 下午8:45，Jan Pekař - Imatic  写道：
> 
> Each node has 64GB RAM so it should be enough (12 OSD's = 48GB used).
> 
> On 21/03/2020 13.14, XuYun wrote:
>> Bluestore requires more than 4G memory per OSD, do you have enough memory?
>> 
>>> 2020年3月21日 下午8:09，Jan Pekař - Imatic  写道：
>>> 
>>> Hello,
>>> 
>>> I have ceph cluster version 14.2.7 
>>> (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable)
>>> 
>>> 4 nodes - each node 11 HDD, 1 SSD, 10Gbit network
>>> 
>>> Cluster was empty, fresh install. We filled cluster with data (small 
>>> blocks) using RGW.
>>> 
>>> Cluster is now used for testing so no client was using it during my admin 
>>> operations mentioned below
>>> 
>>> After a while (7TB of data / 40M objects uploaded) we decided, that we 
>>> increase pg_num from 128 to 256 to better spread data and to speedup this 
>>> operation, I've set
>>> 
>>>  ceph config set mgr target_max_misplaced_ratio 1
>>> 
>>> so that whole cluster rebalance as quickly as it can.
>>> 
>>> I have 3 issues/questions below:
>>> 
>>> 1)
>>> 
>>> I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to 
>>> restart with logged
>>> 
>>> heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had 
>>> suicide timed out after 150
>>> 
>>> after a while OSD's were back so I continued after a while with my tests.
>>> 
>>> My question - increasing number of PG with maximal 
>>> target_max_misplaced_ratio was too much for that OSDs? It is not 
>>> recommended to do it this way? I had no problem with this increase before, 
>>> but configuration of cluster was slightly different and it was luminous 
>>> version.
>>> 
>>> 2)
>>> 
>>> Rebuild was still slow so I increased number of backfills
>>> 
>>>  ceph tell osd.*  injectargs "--osd-max-backfills 10"
>>> 
>>> and reduced recovery sleep time
>>> 
>>>  ceph tell osd.*  injectargs "--osd-recovery-sleep-hdd 0.01"
>>> 
>>> and after few hours I noticed, that some of my OSD's were restarted during 
>>> recovery, in log I can see
>>> 
>>> ...
>>> 
>>> |2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 
>>> 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 
>>> 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
>>> 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 
>>> 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had 
>>> timed out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 
>>> log_channel(cluster) log [WRN] : Monitor daemon marked osd.7 down, but it 
>>> is still running 2020-03-21 06:41:36.888 7fe1e7769700 0 
>>> log_channel(cluster) log [DBG] : map e3574 wrongly marked me down at e3573 
>>> 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574 start_waiting_for_healthy 
>>> |
>>> 
>>> I observed network graph usage and network utilization was low during 
>>> recovery (10Gbit was not saturated).
>>> 
>>> So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought 
>>> that OSD is using threads and HDD timeouts are not influencing heartbeats 
>>> to other OSD's and MON. It looks like it is not true.
>>> 
>>> 3)
>>> 
>>> After OSD was wrongly marked down I can see that cluster has object 
>>> degraded. There were no degraded object before that.
>>> 
>>>  Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 
>>> pgs degraded, 8 pgs undersized
>>> 
>>> It means that this OSD disconnection causes data degraded? How is it 
>>> possible, when no OSD was lost. Data should be on that OSD and after 
>>> peering should be everything OK. With luminous I had no problem, after OSD 
>>> up degraded objects where recovered/found during few seconds and cluster 
>>> was healthy within seconds.
>>> 
>>> Thank you very much for additional info. I can perform additional tests you 
>>> recommend because cluster is used for testing purpose now.
>>> 
>>> With regards
>>> Jan Pekar
>>> 
>>> -- 
>>> 
>>> Ing. Jan Pekař
>>> jan.pe...@imatic.cz
>>> 
>>> Imatic | Jagellonská 14 | Praha 3 | 130 00
>>> http://www.imatic.cz | +420326555326
>>> 
>>> --
>>> 
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> -- 
> 
> Ing. Jan Pekař
> jan.pe...@imatic.cz
> 
> Imatic | Jagellonská 14 | Praha 3 | 130 00
> http://www.imatic.cz | +420326555326
> 
> --
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Questions on Ceph cluster without OS disks

2020-03-21 Thread Marc Roos


I would say it is not a 'proven technology' otherwise you would see a 
wide spread implementation and adaptation of this method. However if you 
really need the physical disk space, it is a solution. Although I also 
would have questions on creating an extra redundant environment to 
service remote booting, just to spare a os disk position. Maybe this 
makes more sence in really big environments.





-Original Message-
From: huxia...@horebdata.cn [mailto:huxia...@horebdata.cn] 
Sent: 21 March 2020 13:54
To: Martin Verges; ceph-users
Subject: [ceph-users] Questions on Ceph cluster without OS disks

Hello， Martin，

I notice that Croit advocate the use of ceph cluster without OS disks, 
but with PXE boot. 

Do you use a NFS server to serve the root file system for each node? 
such as hosting configuration files, user and password, log files, etc. 
My question is, will the NFS server be a single point of failure? If the 
NFS server goes down, the network experience any outage, ceph nodes may 
not be able to write to the local file systems, possibly leading to 
service outage.

How do you deal with the above potential issues in production? I am a 
bit worried...

best regards,

samuel


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Questions on Ceph cluster without OS disks

2020-03-21 Thread huxia...@horebdata.cn

Hello， Martin，

I notice that Croit advocate the use of ceph cluster without OS disks, but with 
PXE boot. 

Do you use a NFS server to serve the root file system for each node? such as 
hosting configuration files, user and password, log files, etc. My question is, 
will the NFS server be a single point of failure? If the NFS server goes down, 
the network experience any outage, ceph nodes may not be able to write to the 
local file systems, possibly leading to service outage.

How do you deal with the above potential issues in production? I am a bit 
worried...

best regards,

samuel






huxia...@horebdata.cn

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

2020-03-21 Thread Jan Pekař - Imatic


Each node has 64GB RAM so it should be enough (12 OSD's = 48GB used).

On 21/03/2020 13.14, XuYun wrote:

Bluestore requires more than 4G memory per OSD, do you have enough memory?


2020年3月21日 下午8:09，Jan Pekař - Imatic  写道：

Hello,

I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) 
nautilus (stable)

4 nodes - each node 11 HDD, 1 SSD, 10Gbit network

Cluster was empty, fresh install. We filled cluster with data (small blocks) 
using RGW.

Cluster is now used for testing so no client was using it during my admin 
operations mentioned below

After a while (7TB of data / 40M objects uploaded) we decided, that we increase 
pg_num from 128 to 256 to better spread data and to speedup this operation, 
I've set

  ceph config set mgr target_max_misplaced_ratio 1

so that whole cluster rebalance as quickly as it can.

I have 3 issues/questions below:

1)

I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to 
restart with logged

heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had suicide 
timed out after 150

after a while OSD's were back so I continued after a while with my tests.

My question - increasing number of PG with maximal target_max_misplaced_ratio 
was too much for that OSDs? It is not recommended to do it this way? I had no 
problem with this increase before, but configuration of cluster was slightly 
different and it was luminous version.

2)

Rebuild was still slow so I increased number of backfills

  ceph tell osd.*  injectargs "--osd-max-backfills 10"

and reduced recovery sleep time

  ceph tell osd.*  injectargs "--osd-recovery-sleep-hdd 0.01"

and after few hours I noticed, that some of my OSD's were restarted during 
recovery, in log I can see

...

|2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 
'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 
06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 1 
heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed 
out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log 
[WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21 
06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 wrongly 
marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574 
start_waiting_for_healthy |

I observed network graph usage and network utilization was low during recovery 
(10Gbit was not saturated).

So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought that 
OSD is using threads and HDD timeouts are not influencing heartbeats to other 
OSD's and MON. It looks like it is not true.

3)

After OSD was wrongly marked down I can see that cluster has object degraded. 
There were no degraded object before that.

  Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs 
degraded, 8 pgs undersized

It means that this OSD disconnection causes data degraded? How is it possible, 
when no OSD was lost. Data should be on that OSD and after peering should be 
everything OK. With luminous I had no problem, after OSD up degraded objects 
where recovered/found during few seconds and cluster was healthy within seconds.

Thank you very much for additional info. I can perform additional tests you 
recommend because cluster is used for testing purpose now.

With regards
Jan Pekar

--

Ing. Jan Pekař
jan.pe...@imatic.cz

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz | +420326555326

--

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--

Ing. Jan Pekař
jan.pe...@imatic.cz

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz | +420326555326

--
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

2020-03-21 Thread XuYun

Bluestore requires more than 4G memory per OSD, do you have enough memory?

> 2020年3月21日 下午8:09，Jan Pekař - Imatic  写道：
> 
> Hello,
> 
> I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) 
> nautilus (stable)
> 
> 4 nodes - each node 11 HDD, 1 SSD, 10Gbit network
> 
> Cluster was empty, fresh install. We filled cluster with data (small blocks) 
> using RGW.
> 
> Cluster is now used for testing so no client was using it during my admin 
> operations mentioned below
> 
> After a while (7TB of data / 40M objects uploaded) we decided, that we 
> increase pg_num from 128 to 256 to better spread data and to speedup this 
> operation, I've set
> 
>  ceph config set mgr target_max_misplaced_ratio 1
> 
> so that whole cluster rebalance as quickly as it can.
> 
> I have 3 issues/questions below:
> 
> 1)
> 
> I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to 
> restart with logged
> 
> heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had 
> suicide timed out after 150
> 
> after a while OSD's were back so I continued after a while with my tests.
> 
> My question - increasing number of PG with maximal target_max_misplaced_ratio 
> was too much for that OSDs? It is not recommended to do it this way? I had no 
> problem with this increase before, but configuration of cluster was slightly 
> different and it was luminous version.
> 
> 2)
> 
> Rebuild was still slow so I increased number of backfills
> 
>  ceph tell osd.*  injectargs "--osd-max-backfills 10"
> 
> and reduced recovery sleep time
> 
>  ceph tell osd.*  injectargs "--osd-recovery-sleep-hdd 0.01"
> 
> and after few hours I noticed, that some of my OSD's were restarted during 
> recovery, in log I can see
> 
> ...
> 
> |2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 
> 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 
> 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
> 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780 7fe1da154700 1 
> heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed 
> out after 15 2020-03-21 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log 
> [WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21 
> 06:41:36.888 7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 
> wrongly marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 
> 3574 start_waiting_for_healthy |
> 
> I observed network graph usage and network utilization was low during 
> recovery (10Gbit was not saturated).
> 
> So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought 
> that OSD is using threads and HDD timeouts are not influencing heartbeats to 
> other OSD's and MON. It looks like it is not true.
> 
> 3)
> 
> After OSD was wrongly marked down I can see that cluster has object degraded. 
> There were no degraded object before that.
> 
>  Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs 
> degraded, 8 pgs undersized
> 
> It means that this OSD disconnection causes data degraded? How is it 
> possible, when no OSD was lost. Data should be on that OSD and after peering 
> should be everything OK. With luminous I had no problem, after OSD up 
> degraded objects where recovered/found during few seconds and cluster was 
> healthy within seconds.
> 
> Thank you very much for additional info. I can perform additional tests you 
> recommend because cluster is used for testing purpose now.
> 
> With regards
> Jan Pekar
> 
> -- 
> 
> Ing. Jan Pekař
> jan.pe...@imatic.cz
> 
> Imatic | Jagellonská 14 | Praha 3 | 130 00
> http://www.imatic.cz | +420326555326
> 
> --
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Problem with OSD::osd_op_tp thread had timed out and other connected issues

2020-03-21 Thread Jan Pekař - Imatic

Hello,

I have ceph cluster version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8)
nautilus (stable)

4 nodes - each node 11 HDD, 1 SSD, 10Gbit network

Cluster was empty, fresh install. We filled cluster with data (small blocks)
using RGW.

Cluster is now used for testing so no client was using it during my admin
operations mentioned below

After a while (7TB of data / 40M objects uploaded) we decided, that we increase pg_num from 128 to 256 to better spread data and to speedup
this operation, I've set

ceph config set mgr target_max_misplaced_ratio 1

so that whole cluster rebalance as quickly as it can.

I have 3 issues/questions below:

I noticed, that manual increase from 128 to 256 caused approx. 6 OSD's to
restart with logged

heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7f8c84b8b700' had suicide
timed out after 150

after a while OSD's were back so I continued after a while with my tests.

My question - increasing number of PG with maximal target_max_misplaced_ratio was too much for that OSDs? It is not recommended to do it
this way? I had no problem with this increase before, but configuration of cluster was slightly different and it was luminous version.

Rebuild was still slow so I increased number of backfills

ceph tell osd.* injectargs "--osd-max-backfills 10"

and reduced recovery sleep time

ceph tell osd.* injectargs "--osd-recovery-sleep-hdd 0.01"

and after few hours I noticed, that some of my OSD's were restarted during
recovery, in log I can see

...

|2020-03-21 06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21
06:41:28.343 7fe1f8bee700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.780
7fe1da154700 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fe1da154700' had timed out after 15 2020-03-21 06:41:36.888
7fe1e7769700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.7 down, but it is still running 2020-03-21 06:41:36.888
7fe1e7769700 0 log_channel(cluster) log [DBG] : map e3574 wrongly marked me down at e3573 2020-03-21 06:41:36.888 7fe1e7769700 1 osd.7 3574
start_waiting_for_healthy |

I observed network graph usage and network utilization was low during recovery
(10Gbit was not saturated).

So lot of IOPS on OSD causes also hartbeat operation to timeout? I thought that OSD is using threads and HDD timeouts are not influencing
heartbeats to other OSD's and MON. It looks like it is not true.

After OSD was wrongly marked down I can see that cluster has object degraded.
There were no degraded object before that.

Degraded data redundancy: 251754/117225048 objects degraded (0.215%), 8 pgs
degraded, 8 pgs undersized

It means that this OSD disconnection causes data degraded? How is it possible, when no OSD was lost. Data should be on that OSD and after
peering should be everything OK. With luminous I had no problem, after OSD up degraded objects where recovered/found during few seconds and
cluster was healthy within seconds.

Thank you very much for additional info. I can perform additional tests you
recommend because cluster is used for testing purpose now.

With regards
Jan Pekar

Ing. Jan Pekař
jan.pe...@imatic.cz

Imatic | Jagellonská 14 | Praha 3 | 130 00
http://www.imatic.cz | +420326555326

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph objecy storage client gui

2020-03-21 Thread Ignazio Cassano

Thanks Konstantin. I presume owncloud solution can use ceph as storage
backend.
Ignazio

Il Sab 21 Mar 2020, 05:45 Konstantin Shalygin  ha scritto:

> On 3/18/20 7:06 PM, Ignazio Cassano wrote:
> > Hello All,
> > I am looking for  object storage freee/opensource client gui (linux and
> > windows) for end users .
> > I tried swiftstack but it is only for personal use.
> > Help, please ?
> > Ignazio
>
>
> You should try to look for S3Browser [1]
>
> AFAIK, this is Windows only solution. For all platforms you can setup
> some "Own Cloud Storage", like nextCloud.
>
>
> [1] https://s3browser.com/
>
> k
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Maximum limit of lifecycle rule length

[ceph-users] Re: Cephfs mount error 1 = Operation not permitted

[ceph-users] Re: Cephfs mount error 1 = Operation not permitted

[ceph-users] Re: Cephfs mount error 1 = Operation not permitted

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

[ceph-users] Cephfs mount error 1 = Operation not permitted

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

[ceph-users] Re: Questions on Ceph cluster without OS disks

[ceph-users] Questions on Ceph cluster without OS disks

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

[ceph-users] Re: Problem with OSD::osd_op_tp thread had timed out and other connected issues

[ceph-users] Problem with OSD::osd_op_tp thread had timed out and other connected issues

[ceph-users] Re: ceph objecy storage client gui

14 matches

Site Navigation

Mail list logo

Footer information