Re: [ceph-users] CephFS+NFS For VMWare

[Paul Emmerich]

VMWare can be quite picky about NFS servers.
Some things that you should test before deploying anything with that in
production:

* failover
* reconnects after NFS reboots or outages
* NFS3 vs NFS4
* Kernel NFS (which kernel version? cephfs-fuse or cephfs-kernel?) vs NFS
Ganesha (VFS FSAL vs. Ceph FSAL)
* Stress tests with lots of VMWare clients - we had a setup than ran fine
with 5 big VMWare hypervisors but started to get random deadlocks once we
added 5 more

We are running CephFS + NFS + VMWare in production but we've encountered *a
lot* of problems until we got that stable for a few configurations.
Be prepared to debug NFS problems at a low level with tcpdump and a careful
read of the RFC and NFS server source ;)

Paul

2018-06-29 18:48 GMT+02:00 Nick Fisk :

> This is for us peeps using Ceph with VMWare.
>
>
>
> My current favoured solution for consuming Ceph in VMWare is via RBD’s
> formatted with XFS and exported via NFS to ESXi. This seems to perform
> better than iSCSI+VMFS which seems to not play nicely with Ceph’s PG
> contention issues particularly if working with thin provisioned VMDK’s.
>
>
>
> I’ve still been noticing some performance issues however, mainly
> noticeable when doing any form of storage migrations. This is largely due
> to the way vSphere transfers VM’s in 64KB IO’s at a QD of 32. vSphere does
> this so Arrays with QOS can balance the IO easier than if larger IO’s were
> submitted. However Ceph’s PG locking means that only one or two of these
> IO’s can happen at a time, seriously lowering throughput. Typically you
> won’t be able to push more than 20-25MB/s during a storage migration
>
>
>
> There is also another issue in that the IO needed for the XFS journal on
> the RBD, can cause contention and effectively also means every NFS write IO
> sends 2 down to Ceph. This can have an impact on latency as well. Due to
> possible PG contention caused by the XFS journal updates when multiple IO’s
> are in flight, you normally end up making more and more RBD’s to try and
> spread the load. This normally means you end up having to do storage
> migrations…..you can see where I’m getting at here.
>
>
>
> I’ve been thinking for a while that CephFS works around a lot of these
> limitations.
>
>
>
> 1.   It supports fancy striping, so should mean there is less per
> object contention
>
> 2.   There is no FS in the middle to maintain a journal and other
> associated IO
>
> 3.   A single large NFS mount should have none of the disadvantages
> seen with a single RBD
>
> 4.   No need to migrate VM’s about because of #3
>
> 5.   No need to fstrim after deleting VM’s
>
> 6.   Potential to do away with pacemaker and use LVS to do
> active/active NFS as ESXi does its own locking with files
>
>
>
> With this in mind I exported a CephFS mount via NFS and then mounted it to
> an ESXi host as a test.
>
>
>
> Initial results are looking very good. I’m seeing storage migrations to
> the NFS mount going at over 200MB/s, which equates to several thousand IO’s
> and seems to be writing at the intended QD32.
>
>
>
> I need to do more testing to make sure everything works as intended, but
> like I say, promising initial results.
>
>
>
> Further testing needs to be done to see what sort of MDS performance is
> required, I would imagine that since we are mainly dealing with large
> files, it might not be that critical. I also need to consider the stability
> of CephFS, RBD is relatively simple and is in use by a large proportion of
> the Ceph community. CephFS is a lot easier to “upset”.
>
>
>
> Nick
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw multizone not syncing large bucket completly to other zone

[Enrico Kern]

hmm that also pops up right away when i restart all radosgw instances. But
i will check further and see if i can find something. Maybe doing the
upgrade to mimic too.

That bucket is basically under load on the master zone all the time as we
use it as historical storage for druid, so there is constantly data written
to it. I just dont get why disabling/enabling sync on the bucket flawless
syncs everything while if i just keep it enabled it stops syncing at all.
For the last days i was just running disabling/enabling for the bucket in a
while loop with 30 minute interval, but thats no persistent fix ;)

On Fri, Jun 29, 2018 at 6:15 PM Yehuda Sadeh-Weinraub 
wrote:

>
>
> On Fri, Jun 29, 2018 at 8:48 AM, Enrico Kern 
> wrote:
>
>> also when i try to sync the bucket manual i get this error:
>>
>> ERROR: sync.run() returned ret=-16
>> 2018-06-29 15:47:50.137268 7f54b7e4ecc0  0 data sync: ERROR: failed to
>> read sync status for
>> bucketname:6a9448d2-bdba-4bec-aad6-aba72cd8eac6.27150814.1
>>
>> it works flawless with all other buckets.
>>
>
> error 16 is EBUSY: meaning it can't take a lease to do work on the bucket.
> This usually happens when another entity (e.g., a running radosgw process)
> is working on it at the same time. Either something took the lease and
> never gave it back (leases shouldn't be indefinite, usually are being taken
> for a short period but are renewed periodically), or there might be some
> other bug related to the lease itself. I would start by first figuring out
> whether it's the first case or the second one. On the messenger log there
> should be a message prior to that that shows the operation that got the -16
> as a response (should have something like "...=-16 (Device or resource
> busy)" in it). The same line would also contain the name of the rados
> object that is used to manage the lease. Try to look at the running radosgw
> log at the same time when this happens, and check whether there are other
> operations on that object.
> One thing to note is that if you run a sync on a bucket and stop it
> uncleanly in the middle (e.g., like killing the process), the leak will
> stay locked for a period of time (Something in the order of 1 to 2 minutes).
>
> Yehuda
>
>>
>>
>> On Fri, Jun 29, 2018 at 5:39 PM Enrico Kern 
>> wrote:
>>
>>> Hello,
>>>
>>> thanks for the reply.
>>>
>>> We have around 200k objects in the bucket. It is not automatic resharded
>>> (is that even supported in multisite?)
>>>
>>> What i see when i run a complete data sync with the debug logs after a
>>> while i see alot of informations that it is unable to perform some log and
>>> also some device or resource busy (also with alot of different osds,
>>> restarting the osds also doesnt make this error going away):
>>>
>>>
>>> 018-06-29 15:18:30.391085 7f38bf882cc0 20
>>> cr:s=0x55de55700b20:op=0x55de55717010:20RGWContinuousLeaseCR: couldn't lock
>>> amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.59:sync_lock:
>>> retcode=-16
>>>
>>> 2018-06-29 15:18:30.391094 7f38bf882cc0 20
>>> cr:s=0x55de55732750:op=0x55de5572d970:20RGWContinuousLeaseCR: couldn't lock
>>> amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.10:sync_lock:
>>> retcode=-16
>>>
>>> 2018-06-29 15:22:01.618744 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604
>>> <== osd.43 10.30.3.44:6800/29982 13272  osd_op_reply(258628
>>> datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.52 [call]
>>> v14448'24265315 uv24265266 ondisk = -16 ((16) Device or resource busy)) v8
>>>  209+0+0 (2379682838 0 0) 0x7f38a8005110 con 0x7f3868003380
>>>
>>> 2018-06-29 15:22:01.618829 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604
>>> <== osd.43 10.30.3.44:6800/29982 13273  osd_op_reply(258629
>>> datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.105 [call]
>>> v14448'24265316 uv24265256 ondisk = -16 ((16) Device or resource busy)) v8
>>>  210+0+0 (4086289880 0 0) 0x7f38a8005110 con 0x7f3868003380
>>>
>>>
>>> There are no issues with the OSDs all other stuff in the cluster works
>>> (rbd, images to openstack etc.)
>>>
>>>
>>> Also that command with appending debug never finishes.
>>>
>>> On Tue, Jun 26, 2018 at 5:45 PM Yehuda Sadeh-Weinraub 
>>> wrote:
>>>


 On Sun, Jun 24, 2018 at 12:59 AM, Enrico Kern <
 enrico.k...@glispamedia.com> wrote:

> Hello,
>
> We have two ceph luminous clusters (12.2.5).
>
> recently one of our big buckets stopped syncing properly. We have a
> one specific bucket which is around 30TB in size consisting of alot of
> directories with each one having files of 10-20MB.
>
> The secondary zone is often completly missing multiple days of data in
> this bucket, while all other smaller buckets sync just fine.
>
> Even with the complete data missing radosgw-admin sync status always
> says everything is fine.
>
> the sync error log doesnt show anything for those days.
>
> Running

[ceph-users] CephFS+NFS For VMWare

[Nick Fisk]

This is for us peeps using Ceph with VMWare.

 

My current favoured solution for consuming Ceph in VMWare is via RBD's 
formatted with XFS and exported via NFS to ESXi. This seems
to perform better than iSCSI+VMFS which seems to not play nicely with Ceph's PG 
contention issues particularly if working with thin
provisioned VMDK's.

 

I've still been noticing some performance issues however, mainly noticeable 
when doing any form of storage migrations. This is
largely due to the way vSphere transfers VM's in 64KB IO's at a QD of 32. 
vSphere does this so Arrays with QOS can balance the IO
easier than if larger IO's were submitted. However Ceph's PG locking means that 
only one or two of these IO's can happen at a time,
seriously lowering throughput. Typically you won't be able to push more than 
20-25MB/s during a storage migration

 

There is also another issue in that the IO needed for the XFS journal on the 
RBD, can cause contention and effectively also means
every NFS write IO sends 2 down to Ceph. This can have an impact on latency as 
well. Due to possible PG contention caused by the XFS
journal updates when multiple IO's are in flight, you normally end up making 
more and more RBD's to try and spread the load. This
normally means you end up having to do storage migrations...you can see where 
I'm getting at here.

 

I've been thinking for a while that CephFS works around a lot of these 
limitations. 

 

1.   It supports fancy striping, so should mean there is less per object 
contention

2.   There is no FS in the middle to maintain a journal and other 
associated IO

3.   A single large NFS mount should have none of the disadvantages seen 
with a single RBD

4.   No need to migrate VM's about because of #3

5.   No need to fstrim after deleting VM's

6.   Potential to do away with pacemaker and use LVS to do active/active 
NFS as ESXi does its own locking with files

 

With this in mind I exported a CephFS mount via NFS and then mounted it to an 
ESXi host as a test.

 

Initial results are looking very good. I'm seeing storage migrations to the NFS 
mount going at over 200MB/s, which equates to
several thousand IO's and seems to be writing at the intended QD32.

 

I need to do more testing to make sure everything works as intended, but like I 
say, promising initial results. 

 

Further testing needs to be done to see what sort of MDS performance is 
required, I would imagine that since we are mainly dealing
with large files, it might not be that critical. I also need to consider the 
stability of CephFS, RBD is relatively simple and is in
use by a large proportion of the Ceph community. CephFS is a lot easier to 
"upset".

 

Nick

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems setting up iSCSI

[Bernhard Dick]

Am 29.06.2018 um 17:45 schrieb Jason Dillaman:
The 7d0023e73855a42ac25038403387ab41ca10753a version should be fine, 
it's what I use in my test environment. I really cannot explain why 
tcmu-runner is missing the rbd handler. I can only assume you have 
restarted the daemon after installing it.
That did the trick. I didn't do any reboot or daemon restarting when I 
installed the other (newer) tcmu-runner versions. So the old version 
still ran.

Now it seems to be working.
Thank you very much for your help.

  Best regards
Bernhard
Any other log messages from 
tcmu-runner? Do you have a "/usr/lib64/tcmu-runner/handler_rbd.so" file? 
Perhaps enable debug-level logging in "/etc/tcmu/tcmu.conf" and see if 
that helps.


On Fri, Jun 29, 2018 at 11:40 AM Bernhard Dick > wrote:


Am 29.06.2018 um 17:26 schrieb Jason Dillaman:
 > OK, so your tcmu-runner doesn't have support for rbd images for some
 > reason. Where did you get your copy of tcmu-runner?
hm originally I took it from the gluster40 repository from the Centos
Storage SIG.
 > You should build it
 > from upstream or pull a copy from here [1].
 >
 > [1] https://shaman.ceph.com/repos/tcmu-runner/
I tried the version:

https://1.chacra.ceph.com/r/tcmu-runner/master/7d0023e73855a42ac25038403387ab41ca10753a/centos/7/flavors/default/x86_64/tcmu-runner-1.3.0-89.g7d0023e.el7.x86_64.rpm
(because the version number from the storage SIG was a 1.3.0-rc4 which
seemed a bit low). That resulted also in the error above.

Now I tried the version from

https://1.chacra.ceph.com/r/tcmu-runner/master/b7498c0857a77f61616b8ad86c775ae39f9c119d/centos/7/flavors/default/x86_64/
but the error persists.

I am however able to map and unmap the image on the host using rbd map
and unmap commands.

    Regards
      Bernhard
 >
 > On Fri, Jun 29, 2018 at 11:12 AM Bernhard Dick mailto:bernh...@bdick.de>
 > >> wrote:
 >
 >     Hi,
 >
 >     Am 29.06.2018 um 17:04 schrieb Jason Dillaman:
 >      > Is 'tcmu-runner' running on that node?
 >     yes it is running
 >
 >      > Any errors in dmesg or
 >     here are no errors
 >      > /var/log/tcmu-runner.log?
 >     the following error is shown:
 >     [ERROR] add_device:436: could not find handler for uio0
 >
 >         Regards
 >           Bernhard
 >      >
 >      > On Fri, Jun 29, 2018 at 10:43 AM Bernhard Dick
mailto:bernh...@bdick.de>
 >     >
 >      > 
      >
 >      >     Hi,
 >      >
 >      >     Am 28.06.2018 um 18:09 schrieb Jason Dillaman:
 >      >      > Do you have the ansible backtrace from the
"ceph-iscsi-gw :
 >      >     igw_lun |
 >      >      > configureluns (create/map rbds and add to lio)]" step?
 >      >     I assume you mean the following (from running with
verbosity
 >     3, after
 >      >     running the purge-iscsi-gateways playbook beforehand)
 >      >
 >      >     Using module file
/home/dick/git/ceph-ansible/library/igw_lun.py
 >      >      ESTABLISH SSH CONNECTION FOR USER: root
 >      >      SSH: EXEC ssh -o ControlMaster=auto -o
 >      >     ControlPersist=600s -o StrictHostKeyChecking=no -o
 >      >     KbdInteractiveAuthentication=no -o
 >      >     PreferredAuthentications=gssapi-with-mic,g
 >      >     ssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o
 >      >     User=root -o ConnectTimeout=30 -o
 >      >     ControlPath=/home/dick/.ansible/cp/%h-%r-%p ovirt-test04
 >     '/bin/sh -c
 >      >     '"'"'/usr/bin/p
 >      >     ython && sleep 0'"'"''
 >      >      (1, '', 'Traceback (most recent call
last):\n
 >     File
 >      >     "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line
166, in
 >      >     \n
 >      >          ansible_main()\n  File "/tmp/ansible_F3
 >      >     sf6j/ansible_module_igw_lun.py", line 135, in
ansible_main\n
 >      >     lun.manage(desired_state)\n  File
 >      >   
  "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",

 >     line 365,
 >      >     in manage\n    self.allocate()\n  File "/u
 >      >     sr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
 >     line 490, in
 >      >     allocate\n    lun = self.add_dev_to_lio()\n  File
 >      >   
  "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",

 >     line 683,
 >      >     in add_dev_to_lio\n
 >      >           wwn=in_wwn, control=control_string)\n  File
 >      >     "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py",

[ceph-users] Performance tuning for SAN SSD config

[Matthew Stroud]

We back some of our ceph clusters with SAN SSD disk, particularly VSP G/F and 
Purestorage. I’m curious what are some settings we should look into modifying 
to take advantage of our SAN arrays. We had to manually set the class for the 
luns to SSD class which was a big improvement. However we still see situations 
where we get slow requests and the underlying disks and network are 
underutilized.

More info about our setup. We are running centos 7 with Luminous as our ceph 
release. We have 4 osd nodes that have 5x2TB disks each and they are setup as 
bluestore. Our ceph.conf is attached with some information removed for security 
reasons.

Thanks ahead of time.

Thanks,
Matthew Stroud



CONFIDENTIALITY NOTICE: This message is intended only for the use and review of 
the individual or entity to which it is addressed and may contain information 
that is privileged and confidential. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message solely to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this communication in error, please notify 
sender immediately by telephone or return email. Thank you.


ceph.conf
Description: ceph.conf
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph snapshots

[Paul Emmerich]

IIRC it can be changed/takes effect immediately. The message is only an
implementation detail:
there is no observer registered that explicitly takes some action when it's
changed, but the value is re-read
anyways. But it's some time since I had to change this value on run-time
but I'm pretty sure it worked
(same with recovery sleep)


Paul
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph snapshots

[Marc Schöchlin]

It seems that this might interesting - unfortunately this cannot be
changed dynamically:

# ceph tell osd.* injectargs '--osd_snap_trim_sleep 0.025'
osd.0: osd_snap_trim_sleep = '0.025000' (not observed, change may
require restart)
osd.1: osd_snap_trim_sleep = '0.025000' (not observed, change may
require restart)
osd.2: osd_snap_trim_sleep = '0.025000' (not observed, change may
require restart)


Am 29.06.2018 um 17:36 schrieb Paul Emmerich:
> It's usually the snapshot deletion that triggers slowness. Are you
> also deleting/rotating old snapshots when creating new ones?
>
> In this case: try to increase osd_snap_trim_sleep a little bit. Even
> to 0.025 can help a lot with a lot of concurrent snapshot deletions.
> (That's what we set as default for exactly this reason - users see
> snapshot deletion as instant and cheap, but it can be quite expensive)
>
> Paul
>
>
> 2018-06-29 17:28 GMT+02:00 Marc Schöchlin  >:
>
> Hi Gregory,
>
> thanks for the link - very interesting talk.
> You mentioned the following settings in your talk, but i was not
> able to find some documentation in the osd config reference:
> (http://docs.ceph.com/docs/luminous/rados/configuration/osd-config-ref/
> )
>
> My clusters settings look like this (luminous/12.2.5):
>
> osd_snap_trim_cost = 1048576
> osd_snap_trim_priority = 5
> osd_snap_trim_sleep = 0.00
> mon_osd_snap_trim_queue_warn_on = 32768
>
> I currently experience messages like this:
>
> 2018-06-29 12:17:47.230028 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534846 : cluster
> [INF] Health check cleared: REQUEST_SLOW (was: 22 slow requests
> are blocked > 32 sec)
> 2018-06-29 12:17:47.230069 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534847 : cluster
> [INF] Cluster is now healthy
> 2018-06-29 12:18:03.287947 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534876 : cluster
> [WRN] Health check failed: 24 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:18:08.307626 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534882 : cluster
> [WRN] Health check update: 70 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:18:14.325471 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534889 : cluster
> [WRN] Health check update: 79 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:18:24.502586 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534900 : cluster
> [WRN] Health check update: 84 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:18:34.489700 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534911 : cluster
> [WRN] Health check update: 17 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:18:39.489982 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534917 : cluster
> [WRN] Health check update: 19 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:18:44.490274 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534923 : cluster
> [WRN] Health check update: 40 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:18:52.620025 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534932 : cluster
> [WRN] Health check update: 92 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:18:58.641621 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534939 : cluster
> [WRN] Health check update: 32 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:19:02.653015 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534948 : cluster
> [INF] Health check cleared: REQUEST_SLOW (was: 32 slow requests
> are blocked > 32 sec)
> 2018-06-29 12:19:02.653048 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534949 : cluster
> [INF] Cluster is now healthy
> 2018-06-29 12:19:08.674106 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534956 : cluster
> [WRN] Health check failed: 15 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:19:14.491798 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534963 : cluster
> [WRN] Health check update: 14 slow requests are blocked > 32 sec
> (REQUEST_SLOW)
> 2018-06-29 12:19:19.492129 mon.ceph-mon-s43 mon.0
> 10.23.27.153:6789/0  1534969 : cluster
> [WRN] Health check update: 32 slow requests are blocked > 32 

Re: [ceph-users] radosgw multizone not syncing large bucket completly to other zone

[Yehuda Sadeh-Weinraub]

On Fri, Jun 29, 2018 at 8:48 AM, Enrico Kern  wrote:

> also when i try to sync the bucket manual i get this error:
>
> ERROR: sync.run() returned ret=-16
> 2018-06-29 15:47:50.137268 7f54b7e4ecc0  0 data sync: ERROR: failed to
> read sync status for bucketname:6a9448d2-bdba-4bec-
> aad6-aba72cd8eac6.27150814.1
>
> it works flawless with all other buckets.
>

error 16 is EBUSY: meaning it can't take a lease to do work on the bucket.
This usually happens when another entity (e.g., a running radosgw process)
is working on it at the same time. Either something took the lease and
never gave it back (leases shouldn't be indefinite, usually are being taken
for a short period but are renewed periodically), or there might be some
other bug related to the lease itself. I would start by first figuring out
whether it's the first case or the second one. On the messenger log there
should be a message prior to that that shows the operation that got the -16
as a response (should have something like "...=-16 (Device or resource
busy)" in it). The same line would also contain the name of the rados
object that is used to manage the lease. Try to look at the running radosgw
log at the same time when this happens, and check whether there are other
operations on that object.
One thing to note is that if you run a sync on a bucket and stop it
uncleanly in the middle (e.g., like killing the process), the leak will
stay locked for a period of time (Something in the order of 1 to 2 minutes).

Yehuda

>
>
> On Fri, Jun 29, 2018 at 5:39 PM Enrico Kern 
> wrote:
>
>> Hello,
>>
>> thanks for the reply.
>>
>> We have around 200k objects in the bucket. It is not automatic resharded
>> (is that even supported in multisite?)
>>
>> What i see when i run a complete data sync with the debug logs after a
>> while i see alot of informations that it is unable to perform some log and
>> also some device or resource busy (also with alot of different osds,
>> restarting the osds also doesnt make this error going away):
>>
>>
>> 018-06-29 15:18:30.391085 7f38bf882cc0 20 cr:s=0x55de55700b20:op=
>> 0x55de55717010:20RGWContinuousLeaseCR: couldn't lock
>> amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-
>> bdba-4bec-aad6-aba72cd8eac6.59:sync_lock: retcode=-16
>>
>> 2018-06-29 15:18:30.391094 7f38bf882cc0 20 cr:s=0x55de55732750:op=
>> 0x55de5572d970:20RGWContinuousLeaseCR: couldn't lock
>> amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-
>> bdba-4bec-aad6-aba72cd8eac6.10:sync_lock: retcode=-16
>>
>> 2018-06-29 15:22:01.618744 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604
>> <== osd.43 10.30.3.44:6800/29982 13272  osd_op_reply(258628
>> datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.52 [call]
>> v14448'24265315 uv24265266 ondisk = -16 ((16) Device or resource busy)) v8
>>  209+0+0 (2379682838 0 0) 0x7f38a8005110 con 0x7f3868003380
>>
>> 2018-06-29 15:22:01.618829 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604
>> <== osd.43 10.30.3.44:6800/29982 13273  osd_op_reply(258629
>> datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.105
>> [call] v14448'24265316 uv24265256 ondisk = -16 ((16) Device or resource
>> busy)) v8  210+0+0 (4086289880 0 0) 0x7f38a8005110 con 0x7f3868003380
>>
>>
>> There are no issues with the OSDs all other stuff in the cluster works
>> (rbd, images to openstack etc.)
>>
>>
>> Also that command with appending debug never finishes.
>>
>> On Tue, Jun 26, 2018 at 5:45 PM Yehuda Sadeh-Weinraub 
>> wrote:
>>
>>>
>>>
>>> On Sun, Jun 24, 2018 at 12:59 AM, Enrico Kern <
>>> enrico.k...@glispamedia.com> wrote:
>>>
 Hello,

 We have two ceph luminous clusters (12.2.5).

 recently one of our big buckets stopped syncing properly. We have a one
 specific bucket which is around 30TB in size consisting of alot of
 directories with each one having files of 10-20MB.

 The secondary zone is often completly missing multiple days of data in
 this bucket, while all other smaller buckets sync just fine.

 Even with the complete data missing radosgw-admin sync status always
 says everything is fine.

 the sync error log doesnt show anything for those days.

 Running

 radosgw-admin metadata sync and data sync also doesnt solve the issue.
 The only way of making it sync again is to disable and re-eanble the sync.
 That needs to be done as often as like 10 times in an hour to make it sync
 properly.

 radosgw-admin bucket sync disable
 radosgw-admin bucket sync enable

 when i run data init i sometimes get this:

  radosgw-admin data sync init --source-zone berlin
 2018-06-24 07:55:46.337858 7fe7557fa700  0 ERROR: failed to distribute
 cache for amsterdam.rgw.log:datalog.sync-status.6a9448d2-bdba-
 4bec-aad6-aba72cd8eac6

 Sometimes when really alot of data is missing (yesterday it was more
 then 1 month) this helps making them get in sync again when run on the
 secondary 

Re: [ceph-users] radosgw multizone not syncing large bucket completly to other zone

[Enrico Kern]

also when i try to sync the bucket manual i get this error:

ERROR: sync.run() returned ret=-16
2018-06-29 15:47:50.137268 7f54b7e4ecc0  0 data sync: ERROR: failed to read
sync status for bucketname:6a9448d2-bdba-4bec-aad6-aba72cd8eac6.27150814.1

it works flawless with all other buckets.


On Fri, Jun 29, 2018 at 5:39 PM Enrico Kern  wrote:

> Hello,
>
> thanks for the reply.
>
> We have around 200k objects in the bucket. It is not automatic resharded
> (is that even supported in multisite?)
>
> What i see when i run a complete data sync with the debug logs after a
> while i see alot of informations that it is unable to perform some log and
> also some device or resource busy (also with alot of different osds,
> restarting the osds also doesnt make this error going away):
>
>
> 018-06-29 15:18:30.391085 7f38bf882cc0 20
> cr:s=0x55de55700b20:op=0x55de55717010:20RGWContinuousLeaseCR: couldn't lock
> amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.59:sync_lock:
> retcode=-16
>
> 2018-06-29 15:18:30.391094 7f38bf882cc0 20
> cr:s=0x55de55732750:op=0x55de5572d970:20RGWContinuousLeaseCR: couldn't lock
> amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.10:sync_lock:
> retcode=-16
>
> 2018-06-29 15:22:01.618744 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604 <==
> osd.43 10.30.3.44:6800/29982 13272  osd_op_reply(258628
> datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.52 [call]
> v14448'24265315 uv24265266 ondisk = -16 ((16) Device or resource busy)) v8
>  209+0+0 (2379682838 0 0) 0x7f38a8005110 con 0x7f3868003380
>
> 2018-06-29 15:22:01.618829 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604 <==
> osd.43 10.30.3.44:6800/29982 13273  osd_op_reply(258629
> datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.105 [call]
> v14448'24265316 uv24265256 ondisk = -16 ((16) Device or resource busy)) v8
>  210+0+0 (4086289880 0 0) 0x7f38a8005110 con 0x7f3868003380
>
>
> There are no issues with the OSDs all other stuff in the cluster works
> (rbd, images to openstack etc.)
>
>
> Also that command with appending debug never finishes.
>
> On Tue, Jun 26, 2018 at 5:45 PM Yehuda Sadeh-Weinraub 
> wrote:
>
>>
>>
>> On Sun, Jun 24, 2018 at 12:59 AM, Enrico Kern <
>> enrico.k...@glispamedia.com> wrote:
>>
>>> Hello,
>>>
>>> We have two ceph luminous clusters (12.2.5).
>>>
>>> recently one of our big buckets stopped syncing properly. We have a one
>>> specific bucket which is around 30TB in size consisting of alot of
>>> directories with each one having files of 10-20MB.
>>>
>>> The secondary zone is often completly missing multiple days of data in
>>> this bucket, while all other smaller buckets sync just fine.
>>>
>>> Even with the complete data missing radosgw-admin sync status always
>>> says everything is fine.
>>>
>>> the sync error log doesnt show anything for those days.
>>>
>>> Running
>>>
>>> radosgw-admin metadata sync and data sync also doesnt solve the issue.
>>> The only way of making it sync again is to disable and re-eanble the sync.
>>> That needs to be done as often as like 10 times in an hour to make it sync
>>> properly.
>>>
>>> radosgw-admin bucket sync disable
>>> radosgw-admin bucket sync enable
>>>
>>> when i run data init i sometimes get this:
>>>
>>>  radosgw-admin data sync init --source-zone berlin
>>> 2018-06-24 07:55:46.337858 7fe7557fa700  0 ERROR: failed to distribute
>>> cache for
>>> amsterdam.rgw.log:datalog.sync-status.6a9448d2-bdba-4bec-aad6-aba72cd8eac6
>>>
>>> Sometimes when really alot of data is missing (yesterday it was more
>>> then 1 month) this helps making them get in sync again when run on the
>>> secondary zone:
>>>
>>> radosgw-admin bucket check --fix --check-objects
>>>
>>> how can i debug that problem further? We have so many requests on the
>>> cluster that is is hard to dig something out of the log files..
>>>
>>> Given all the smaller buckets are perfectly in sync i suspect some
>>> problem because of the size of the bucket.
>>>
>>
>> How many objects in the bucket? Is it getting automatically resharded?
>>
>>
>>>
>>> Any points to the right direction are greatly appreciated.
>>>
>>
>> A few things to look at that might help identify the issue.
>>
>> What does this show (I think the luminous command is as follows):
>>
>> $ radosgw-admin bucket sync status --source-zone=
>>
>> You can try manually syncing the bucket, and get specific logs for that
>> operation:
>>
>> $ radosgw-admin bucket sync run -source-zone= --debug-rgw=20
>> --debug-ms=1
>>
>> And you can try getting more info from the sync trace module:
>>
>> $ ceph --admin-daemon  sync trace history
>> 
>>
>> You can also try the 'sync trace show' command.
>>
>>
>> Yehuda
>>
>>
>>
>>>
>>> Regards,
>>>
>>> Enrico
>>>
>>> --
>>>
>>> *Enrico Kern*
>>> VP IT Operations
>>>
>>> enrico.k...@glispa.com
>>> +49 (0) 30 555713017 / +49 (0) 152 26814501
>>> skype: flyersa
>>> LinkedIn Profile 
>>>

Re: [ceph-users] Problems setting up iSCSI

[Jason Dillaman]

The 7d0023e73855a42ac25038403387ab41ca10753a version should be fine, it's
what I use in my test environment. I really cannot explain why tcmu-runner
is missing the rbd handler. I can only assume you have restarted the daemon
after installing it. Any other log messages from tcmu-runner? Do you have a
"/usr/lib64/tcmu-runner/handler_rbd.so" file? Perhaps enable debug-level
logging in "/etc/tcmu/tcmu.conf" and see if that helps.

On Fri, Jun 29, 2018 at 11:40 AM Bernhard Dick  wrote:

> Am 29.06.2018 um 17:26 schrieb Jason Dillaman:
> > OK, so your tcmu-runner doesn't have support for rbd images for some
> > reason. Where did you get your copy of tcmu-runner?
> hm originally I took it from the gluster40 repository from the Centos
> Storage SIG.
> > You should build it
> > from upstream or pull a copy from here [1].
> >
> > [1] https://shaman.ceph.com/repos/tcmu-runner/
> I tried the version:
>
> https://1.chacra.ceph.com/r/tcmu-runner/master/7d0023e73855a42ac25038403387ab41ca10753a/centos/7/flavors/default/x86_64/tcmu-runner-1.3.0-89.g7d0023e.el7.x86_64.rpm
> (because the version number from the storage SIG was a 1.3.0-rc4 which
> seemed a bit low). That resulted also in the error above.
>
> Now I tried the version from
>
> https://1.chacra.ceph.com/r/tcmu-runner/master/b7498c0857a77f61616b8ad86c775ae39f9c119d/centos/7/flavors/default/x86_64/
> but the error persists.
>
> I am however able to map and unmap the image on the host using rbd map
> and unmap commands.
>
>Regards
>  Bernhard
> >
> > On Fri, Jun 29, 2018 at 11:12 AM Bernhard Dick  > > wrote:
> >
> > Hi,
> >
> > Am 29.06.2018 um 17:04 schrieb Jason Dillaman:
> >  > Is 'tcmu-runner' running on that node?
> > yes it is running
> >
> >  > Any errors in dmesg or
> > here are no errors
> >  > /var/log/tcmu-runner.log?
> > the following error is shown:
> > [ERROR] add_device:436: could not find handler for uio0
> >
> > Regards
> >   Bernhard
> >  >
> >  > On Fri, Jun 29, 2018 at 10:43 AM Bernhard Dick  > 
> >  > >> wrote:
> >  >
> >  > Hi,
> >  >
> >  > Am 28.06.2018 um 18:09 schrieb Jason Dillaman:
> >  >  > Do you have the ansible backtrace from the "ceph-iscsi-gw :
> >  > igw_lun |
> >  >  > configureluns (create/map rbds and add to lio)]" step?
> >  > I assume you mean the following (from running with verbosity
> > 3, after
> >  > running the purge-iscsi-gateways playbook beforehand)
> >  >
> >  > Using module file
> /home/dick/git/ceph-ansible/library/igw_lun.py
> >  >  ESTABLISH SSH CONNECTION FOR USER: root
> >  >  SSH: EXEC ssh -o ControlMaster=auto -o
> >  > ControlPersist=600s -o StrictHostKeyChecking=no -o
> >  > KbdInteractiveAuthentication=no -o
> >  > PreferredAuthentications=gssapi-with-mic,g
> >  > ssapi-keyex,hostbased,publickey -o PasswordAuthentication=no
> -o
> >  > User=root -o ConnectTimeout=30 -o
> >  > ControlPath=/home/dick/.ansible/cp/%h-%r-%p ovirt-test04
> > '/bin/sh -c
> >  > '"'"'/usr/bin/p
> >  > ython && sleep 0'"'"''
> >  >  (1, '', 'Traceback (most recent call last):\n
> > File
> >  > "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in
> >  > \n
> >  >  ansible_main()\n  File "/tmp/ansible_F3
> >  > sf6j/ansible_module_igw_lun.py", line 135, in ansible_main\n
> >  > lun.manage(desired_state)\n  File
> >  > "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> > line 365,
> >  > in manage\nself.allocate()\n  File "/u
> >  > sr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> > line 490, in
> >  > allocate\nlun = self.add_dev_to_lio()\n  File
> >  > "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> > line 683,
> >  > in add_dev_to_lio\n
> >  >   wwn=in_wwn, control=control_string)\n  File
> >  > "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line
> 818, in
> >  > __init__\nself._configure(config, size, wwn,
> hw_max_sectors,
> >  > control)\n  File "/usr/lib/pytho
> >  > n2.7/site-packages/rtslib_fb/tcm.py", line 836, in
> _configure\n
> >  > self._enable()\n  File
> >  > "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line
> 172, in
> >  > _enable\nfwrite(path, "1\\n")\n  File "/usr/lib/pyt
> >  > hon2.7/site-packages/rtslib_fb/utils.py", line 79, in fwrite\n
> >  > file_fd.write(str(string))\nIOError: [Errno 2] No such file or
> >  > directory\n')
> >  > The full traceback is:
> >  > Traceback (most recent call last):
> >  > File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py",
> > 

Re: [ceph-users] Problems setting up iSCSI

[Bernhard Dick]

Am 29.06.2018 um 17:26 schrieb Jason Dillaman:
OK, so your tcmu-runner doesn't have support for rbd images for some 
reason. Where did you get your copy of tcmu-runner? 
hm originally I took it from the gluster40 repository from the Centos 
Storage SIG.
You should build it 
from upstream or pull a copy from here [1].


[1] https://shaman.ceph.com/repos/tcmu-runner/
I tried the version: 
https://1.chacra.ceph.com/r/tcmu-runner/master/7d0023e73855a42ac25038403387ab41ca10753a/centos/7/flavors/default/x86_64/tcmu-runner-1.3.0-89.g7d0023e.el7.x86_64.rpm
(because the version number from the storage SIG was a 1.3.0-rc4 which 
seemed a bit low). That resulted also in the error above.


Now I tried the version from 
https://1.chacra.ceph.com/r/tcmu-runner/master/b7498c0857a77f61616b8ad86c775ae39f9c119d/centos/7/flavors/default/x86_64/

but the error persists.

I am however able to map and unmap the image on the host using rbd map 
and unmap commands.


  Regards
Bernhard


On Fri, Jun 29, 2018 at 11:12 AM Bernhard Dick > wrote:


Hi,

Am 29.06.2018 um 17:04 schrieb Jason Dillaman:
 > Is 'tcmu-runner' running on that node?
yes it is running

 > Any errors in dmesg or
here are no errors
 > /var/log/tcmu-runner.log?
the following error is shown:
[ERROR] add_device:436: could not find handler for uio0

    Regards
      Bernhard
 >
 > On Fri, Jun 29, 2018 at 10:43 AM Bernhard Dick mailto:bernh...@bdick.de>
 > >> wrote:
 >
 >     Hi,
 >
 >     Am 28.06.2018 um 18:09 schrieb Jason Dillaman:
 >      > Do you have the ansible backtrace from the "ceph-iscsi-gw :
 >     igw_lun |
 >      > configureluns (create/map rbds and add to lio)]" step?
 >     I assume you mean the following (from running with verbosity
3, after
 >     running the purge-iscsi-gateways playbook beforehand)
 >
 >     Using module file /home/dick/git/ceph-ansible/library/igw_lun.py
 >      ESTABLISH SSH CONNECTION FOR USER: root
 >      SSH: EXEC ssh -o ControlMaster=auto -o
 >     ControlPersist=600s -o StrictHostKeyChecking=no -o
 >     KbdInteractiveAuthentication=no -o
 >     PreferredAuthentications=gssapi-with-mic,g
 >     ssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o
 >     User=root -o ConnectTimeout=30 -o
 >     ControlPath=/home/dick/.ansible/cp/%h-%r-%p ovirt-test04
'/bin/sh -c
 >     '"'"'/usr/bin/p
 >     ython && sleep 0'"'"''
 >      (1, '', 'Traceback (most recent call last):\n 
File

 >     "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in
 >     \n
 >          ansible_main()\n  File "/tmp/ansible_F3
 >     sf6j/ansible_module_igw_lun.py", line 135, in ansible_main\n
 >     lun.manage(desired_state)\n  File
 >     "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
line 365,
 >     in manage\n    self.allocate()\n  File "/u
 >     sr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
line 490, in
 >     allocate\n    lun = self.add_dev_to_lio()\n  File
 >     "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
line 683,
 >     in add_dev_to_lio\n
 >           wwn=in_wwn, control=control_string)\n  File
 >     "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 818, in
 >     __init__\n    self._configure(config, size, wwn, hw_max_sectors,
 >     control)\n  File "/usr/lib/pytho
 >     n2.7/site-packages/rtslib_fb/tcm.py", line 836, in _configure\n
 >     self._enable()\n  File
 >     "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 172, in
 >     _enable\n    fwrite(path, "1\\n")\n  File "/usr/lib/pyt
 >     hon2.7/site-packages/rtslib_fb/utils.py", line 79, in fwrite\n
 >     file_fd.write(str(string))\nIOError: [Errno 2] No such file or
 >     directory\n')
 >     The full traceback is:
 >     Traceback (most recent call last):
 >         File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py",
line 166, in
 >     
 >           ansible_main()
 >         File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py",
line 135, in
 >     ansible_main
 >           lun.manage(desired_state)
 >         File
"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
 >     line 365, in manage
 >           self.allocate()
 >         File
"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
 >     line 490, in allocate
 >           lun = self.add_dev_to_lio()
 >         File
"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
 >     line 683, in add_dev_to_lio
 >           wwn=in_wwn, control=control_string)
 >         File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py",
line 818,
 >     in __init__
 >           

Re: [ceph-users] radosgw multizone not syncing large bucket completly to other zone

[Enrico Kern]

Hello,

thanks for the reply.

We have around 200k objects in the bucket. It is not automatic resharded
(is that even supported in multisite?)

What i see when i run a complete data sync with the debug logs after a
while i see alot of informations that it is unable to perform some log and
also some device or resource busy (also with alot of different osds,
restarting the osds also doesnt make this error going away):


018-06-29 15:18:30.391085 7f38bf882cc0 20
cr:s=0x55de55700b20:op=0x55de55717010:20RGWContinuousLeaseCR: couldn't lock
amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.59:sync_lock:
retcode=-16

2018-06-29 15:18:30.391094 7f38bf882cc0 20
cr:s=0x55de55732750:op=0x55de5572d970:20RGWContinuousLeaseCR: couldn't lock
amsterdam.rgw.log:datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.10:sync_lock:
retcode=-16

2018-06-29 15:22:01.618744 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604 <==
osd.43 10.30.3.44:6800/29982 13272  osd_op_reply(258628
datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.52 [call]
v14448'24265315 uv24265266 ondisk = -16 ((16) Device or resource busy)) v8
 209+0+0 (2379682838 0 0) 0x7f38a8005110 con 0x7f3868003380

2018-06-29 15:22:01.618829 7f38ad4c7700  1 -- 10.30.3.67:0/3390890604 <==
osd.43 10.30.3.44:6800/29982 13273  osd_op_reply(258629
datalog.sync-status.shard.6a9448d2-bdba-4bec-aad6-aba72cd8eac6.105 [call]
v14448'24265316 uv24265256 ondisk = -16 ((16) Device or resource busy)) v8
 210+0+0 (4086289880 0 0) 0x7f38a8005110 con 0x7f3868003380


There are no issues with the OSDs all other stuff in the cluster works
(rbd, images to openstack etc.)


Also that command with appending debug never finishes.

On Tue, Jun 26, 2018 at 5:45 PM Yehuda Sadeh-Weinraub 
wrote:

>
>
> On Sun, Jun 24, 2018 at 12:59 AM, Enrico Kern  > wrote:
>
>> Hello,
>>
>> We have two ceph luminous clusters (12.2.5).
>>
>> recently one of our big buckets stopped syncing properly. We have a one
>> specific bucket which is around 30TB in size consisting of alot of
>> directories with each one having files of 10-20MB.
>>
>> The secondary zone is often completly missing multiple days of data in
>> this bucket, while all other smaller buckets sync just fine.
>>
>> Even with the complete data missing radosgw-admin sync status always says
>> everything is fine.
>>
>> the sync error log doesnt show anything for those days.
>>
>> Running
>>
>> radosgw-admin metadata sync and data sync also doesnt solve the issue.
>> The only way of making it sync again is to disable and re-eanble the sync.
>> That needs to be done as often as like 10 times in an hour to make it sync
>> properly.
>>
>> radosgw-admin bucket sync disable
>> radosgw-admin bucket sync enable
>>
>> when i run data init i sometimes get this:
>>
>>  radosgw-admin data sync init --source-zone berlin
>> 2018-06-24 07:55:46.337858 7fe7557fa700  0 ERROR: failed to distribute
>> cache for
>> amsterdam.rgw.log:datalog.sync-status.6a9448d2-bdba-4bec-aad6-aba72cd8eac6
>>
>> Sometimes when really alot of data is missing (yesterday it was more then
>> 1 month) this helps making them get in sync again when run on the secondary
>> zone:
>>
>> radosgw-admin bucket check --fix --check-objects
>>
>> how can i debug that problem further? We have so many requests on the
>> cluster that is is hard to dig something out of the log files..
>>
>> Given all the smaller buckets are perfectly in sync i suspect some
>> problem because of the size of the bucket.
>>
>
> How many objects in the bucket? Is it getting automatically resharded?
>
>
>>
>> Any points to the right direction are greatly appreciated.
>>
>
> A few things to look at that might help identify the issue.
>
> What does this show (I think the luminous command is as follows):
>
> $ radosgw-admin bucket sync status --source-zone=
>
> You can try manually syncing the bucket, and get specific logs for that
> operation:
>
> $ radosgw-admin bucket sync run -source-zone= --debug-rgw=20
> --debug-ms=1
>
> And you can try getting more info from the sync trace module:
>
> $ ceph --admin-daemon  sync trace history
> 
>
> You can also try the 'sync trace show' command.
>
>
> Yehuda
>
>
>
>>
>> Regards,
>>
>> Enrico
>>
>> --
>>
>> *Enrico Kern*
>> VP IT Operations
>>
>> enrico.k...@glispa.com
>> +49 (0) 30 555713017 / +49 (0) 152 26814501
>> skype: flyersa
>> LinkedIn Profile 
>>
>>
>>  
>>
>> *Glispa GmbH* | Berlin Office
>> Sonnenburger Straße 73
>> 
>> 10437 Berlin
>> 

Re: [ceph-users] RBD gets resized when used as iSCSI target

[Jason Dillaman]

gwcli doesn't allow you to shrink images (it silently ignores you).  Use
'rbd resize' and restart the GWs to pick up the new size.

On Fri, Jun 29, 2018 at 11:36 AM Wladimir Mutel  wrote:

> Wladimir Mutel wrote:
>
> > it back to gwcli/disks), I discover that its size is rounded up to 3
> > TiB, i.e. 3072 GiB or 786432*4M Ceph objects. As we know, GPT is stored
>
> > 'targetcli ls /' (there, it is still 3.0T). Also, when I restart
> > rbd-target-gw.service, it gets resized back up to 3.0T as shown by
>
> Well, I see this size in /etc/target/saveconfig.json
> And I see how the RBD is extended in /var/log/tcmu-runner.log
> And I remember that once I lazily added 2.7T RBD specifying its
> size as
> 3T in gwcli. Now trying to fix that wihout deleting/recreating the RBD...
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] crusmap show wrong osd for PGs (EC-Pool)

[ulembke]

Hi all,
I had an issue on an hammer-cluster (0.94.9 - ugraded from 0.94.7 
today).


There are three PGs incomplete:
root@ceph-06:~# ceph health detail
HEALTH_WARN 3 pgs incomplete; 3 pgs stuck inactive; 3 pgs stuck unclean
pg 24.cc is stuck inactive for 595902.285007, current state incomplete, 
last acting [8,111,12,88,128,44,56]
pg 24.59 is stuck inactive for 596201.505599, current state 
remapped+incomplete, last acting [2147483647,77,55,2147483647,26,12,9]
pg 16.142 is stuck inactive for 596297.260651, current state 
remapped+incomplete, last acting [74,12,2147483647,2147483647,5]
pg 24.cc is stuck unclean for 599519.131339, current state incomplete, 
last acting [8,111,12,88,128,44,56]
pg 24.59 is stuck unclean for 604346.612898, current state 
remapped+incomplete, last acting [2147483647,77,55,2147483647,26,12,9]
pg 16.142 is stuck unclean for 662937.231308, current state 
remapped+incomplete, last acting [74,12,2147483647,2147483647,5]
pg 16.142 is remapped+incomplete, acting [74,12,2147483647,2147483647,5] 
(reducing pool ecarchiv min_size from 3 may help; search ceph.com/docs 
for 'incomplete')
pg 24.cc is incomplete, acting [8,111,12,88,128,44,56] (reducing pool 
ec7archiv min_size from 5 may help; search ceph.com/docs for 
'incomplete')
pg 24.59 is remapped+incomplete, acting 
[2147483647,77,55,2147483647,26,12,9] (reducing pool ec7archiv min_size 
from 5 may help; search ceph.com/docs for 'incomplete')


osd-12 was rebuildet some days ago - but the location of the other 
slices are not all right:


24.cc crushmap: [8,111,12,88,128,44,56]
real live:  [8,121, X,88,130,44,56] - due the new osd-12 and the 
wrong searchlist (osd-121 + osd-130) the PG is undersized!


/var/lib/ceph/osd/ceph-8/current/24.ccs0_head
/var/lib/ceph/osd/ceph-44/current/24.ccs5_head
/var/lib/ceph/osd/ceph-56/current/24.ccs6_head
/var/lib/ceph/osd/ceph-88/current/24.ccs3_head
/var/lib/ceph/osd/ceph-121/current/24.ccs1_head
/var/lib/ceph/osd/ceph-130/current/24.ccs4_head

the same with the other PGs (two EC-5+2 + one EC-3+2)

/var/lib/ceph/osd/ceph-9/current/24.59s6_head
/var/lib/ceph/osd/ceph-26/current/24.59s4_head
/var/lib/ceph/osd/ceph-55/current/24.59s2_head
/var/lib/ceph/osd/ceph-77/current/24.59s1_head
/var/lib/ceph/osd/ceph-110/current/24.59s3_head
/var/lib/ceph/osd/ceph-122/current/24.59s0_head

/var/lib/ceph/osd/ceph-123/current/16.142s2_head
/var/lib/ceph/osd/ceph-5/current/16.142s4_head
/var/lib/ceph/osd/ceph-74/current/16.142s0_head
/var/lib/ceph/osd/ceph-115/current/16.142s3_head

set_choose_tries is set to 200.

ceph pg repair don't work.

Howto let calculate ceph the "right" osds?

Any hints?

Udo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph snapshots

[Paul Emmerich]

It's usually the snapshot deletion that triggers slowness. Are you also
deleting/rotating old snapshots when creating new ones?

In this case: try to increase osd_snap_trim_sleep a little bit. Even to
0.025 can help a lot with a lot of concurrent snapshot deletions.
(That's what we set as default for exactly this reason - users see snapshot
deletion as instant and cheap, but it can be quite expensive)

Paul


2018-06-29 17:28 GMT+02:00 Marc Schöchlin :

> Hi Gregory,
>
> thanks for the link - very interesting talk.
> You mentioned the following settings in your talk, but i was not able to
> find some documentation in the osd config reference:
> (http://docs.ceph.com/docs/luminous/rados/configuration/osd-config-ref/)
>
> My clusters settings look like this (luminous/12.2.5):
>
> osd_snap_trim_cost = 1048576
> osd_snap_trim_priority = 5
> osd_snap_trim_sleep = 0.00
> mon_osd_snap_trim_queue_warn_on = 32768
>
> I currently experience messages like this:
>
> 2018-06-29 12:17:47.230028 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534846 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 22 slow
> requests are blocked > 32 sec)
> 2018-06-29 12:17:47.230069 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534847 : cluster [INF] Cluster is now healthy
> 2018-06-29 12:18:03.287947 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534876 : cluster [WRN] Health check failed: 24 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:18:08.307626 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534882 : cluster [WRN] Health check update: 70 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:18:14.325471 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534889 : cluster [WRN] Health check update: 79 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:18:24.502586 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534900 : cluster [WRN] Health check update: 84 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:18:34.489700 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534911 : cluster [WRN] Health check update: 17 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:18:39.489982 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534917 : cluster [WRN] Health check update: 19 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:18:44.490274 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534923 : cluster [WRN] Health check update: 40 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:18:52.620025 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534932 : cluster [WRN] Health check update: 92 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:18:58.641621 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534939 : cluster [WRN] Health check update: 32 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:19:02.653015 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534948 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 32 slow
> requests are blocked > 32 sec)
> 2018-06-29 12:19:02.653048 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534949 : cluster [INF] Cluster is now healthy
> 2018-06-29 12:19:08.674106 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534956 : cluster [WRN] Health check failed: 15 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:19:14.491798 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534963 : cluster [WRN] Health check update: 14 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:19:19.492129 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534969 : cluster [WRN] Health check update: 32 slow requests are blocked >
> 32 sec (REQUEST_SLOW)
> 2018-06-29 12:19:22.726667 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534973 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 32 slow
> requests are blocked > 32 sec)
> 2018-06-29 12:19:22.726697 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1534974 : cluster [INF] Cluster is now healthy
> 2018-06-29 13:00:00.000121 mon.ceph-mon-s43 mon.0 10.23.27.153:6789/0
> 1537844 : cluster [INF] overall HEALTH_OK
>
> Is that related to snap trimming?
>
> I am currently migrating 250 virtual machines to my new an shiny 2448 PGs,
> 72 OSD (48 HDD, 24 SSD, 5 osd nodes) cluster and these messages appear with
> some delay after the daily rbd snapshot creation
>
> Regards
>
> Marc
>
> Am 29.06.2018 um 04:27 schrieb Gregory Farnum:
>
> You may find my talk at OpenStack Boston’s Ceph day last year to be
> useful: https://www.youtube.com/watch?v=rY0OWtllkn8
> -Greg
> On Wed, Jun 27, 2018 at 9:06 AM Marc Schöchlin  wrote:
>
>> Hello list,
>>
>> i currently hold 3 snapshots per rbd image for my virtual systems.
>>
>> What i miss in the current documentation:
>>
>>   * details about the implementation of snapshots
>>   o implementation details
>>   o which scenarios create high overhead per snapshot
>>   o what causes the really short performance degration on snapshot
>> creation/deletion
>>   o why 

Re: [ceph-users] RBD gets resized when used as iSCSI target

[Wladimir Mutel]

Wladimir Mutel wrote:

it back to gwcli/disks), I discover that its size is rounded up to 3 
TiB, i.e. 3072 GiB or 786432*4M Ceph objects. As we know, GPT is stored 


'targetcli ls /' (there, it is still 3.0T). Also, when I restart 
rbd-target-gw.service, it gets resized back up to 3.0T as shown by 


Well, I see this size in /etc/target/saveconfig.json
And I see how the RBD is extended in /var/log/tcmu-runner.log
	And I remember that once I lazily added 2.7T RBD specifying its size as 
3T in gwcli. Now trying to fix that wihout deleting/recreating the RBD...


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems setting up iSCSI

[Jason Dillaman]

OK, so your tcmu-runner doesn't have support for rbd images for some
reason. Where did you get your copy of tcmu-runner? You should build it
from upstream or pull a copy from here [1].

[1] https://shaman.ceph.com/repos/tcmu-runner/

On Fri, Jun 29, 2018 at 11:12 AM Bernhard Dick  wrote:

> Hi,
>
> Am 29.06.2018 um 17:04 schrieb Jason Dillaman:
> > Is 'tcmu-runner' running on that node?
> yes it is running
>
> > Any errors in dmesg or
> here are no errors
> > /var/log/tcmu-runner.log?
> the following error is shown:
> [ERROR] add_device:436: could not find handler for uio0
>
>Regards
>  Bernhard
> >
> > On Fri, Jun 29, 2018 at 10:43 AM Bernhard Dick  > > wrote:
> >
> > Hi,
> >
> > Am 28.06.2018 um 18:09 schrieb Jason Dillaman:
> >  > Do you have the ansible backtrace from the "ceph-iscsi-gw :
> > igw_lun |
> >  > configureluns (create/map rbds and add to lio)]" step?
> > I assume you mean the following (from running with verbosity 3, after
> > running the purge-iscsi-gateways playbook beforehand)
> >
> > Using module file /home/dick/git/ceph-ansible/library/igw_lun.py
> >  ESTABLISH SSH CONNECTION FOR USER: root
> >  SSH: EXEC ssh -o ControlMaster=auto -o
> > ControlPersist=600s -o StrictHostKeyChecking=no -o
> > KbdInteractiveAuthentication=no -o
> > PreferredAuthentications=gssapi-with-mic,g
> > ssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o
> > User=root -o ConnectTimeout=30 -o
> > ControlPath=/home/dick/.ansible/cp/%h-%r-%p ovirt-test04 '/bin/sh -c
> > '"'"'/usr/bin/p
> > ython && sleep 0'"'"''
> >  (1, '', 'Traceback (most recent call last):\n  File
> > "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in
> > \n
> >  ansible_main()\n  File "/tmp/ansible_F3
> > sf6j/ansible_module_igw_lun.py", line 135, in ansible_main\n
> > lun.manage(desired_state)\n  File
> > "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line
> 365,
> > in manage\nself.allocate()\n  File "/u
> > sr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 490,
> in
> > allocate\nlun = self.add_dev_to_lio()\n  File
> > "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line
> 683,
> > in add_dev_to_lio\n
> >   wwn=in_wwn, control=control_string)\n  File
> > "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 818, in
> > __init__\nself._configure(config, size, wwn, hw_max_sectors,
> > control)\n  File "/usr/lib/pytho
> > n2.7/site-packages/rtslib_fb/tcm.py", line 836, in _configure\n
> > self._enable()\n  File
> > "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 172, in
> > _enable\nfwrite(path, "1\\n")\n  File "/usr/lib/pyt
> > hon2.7/site-packages/rtslib_fb/utils.py", line 79, in fwrite\n
> > file_fd.write(str(string))\nIOError: [Errno 2] No such file or
> > directory\n')
> > The full traceback is:
> > Traceback (most recent call last):
> > File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166,
> in
> > 
> >   ansible_main()
> > File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 135,
> in
> > ansible_main
> >   lun.manage(desired_state)
> > File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> > line 365, in manage
> >   self.allocate()
> > File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> > line 490, in allocate
> >   lun = self.add_dev_to_lio()
> > File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> > line 683, in add_dev_to_lio
> >   wwn=in_wwn, control=control_string)
> > File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line
> 818,
> > in __init__
> >   self._configure(config, size, wwn, hw_max_sectors, control)
> > File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line
> 836,
> > in _configure
> >   self._enable()
> > File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line
> 172,
> > in _enable
> >   fwrite(path, "1\n")
> > File "/usr/lib/python2.7/site-packages/rtslib_fb/utils.py", line
> > 79,
> > in fwrite
> >   file_fd.write(str(string))
> > IOError: [Errno 2] No such file or directory
> >
> > failed: [ovirt-test04] (item={u'host': u'ovirt-test04', u'image':
> > u'iscsi2342', u'state': u'present', u'pool': u'rbd', u'size':
> > u'1G'}) => {
> >   "changed": false,
> >   "item": {
> >   "host": "ovirt-test04",
> >   "image": "iscsi2342",
> >   "pool": "rbd",
> >   "size": "1G",
> >   "state": "present"
> >   },
> >   "module_stderr": "Traceback (most recent call last):\n  File
> > \"/tmp/ansible_F3sf6j/ansible_module_igw_lun.py\", line 166, in
> > \n  

Re: [ceph-users] Problems setting up iSCSI

[Bernhard Dick]

Hi,

Am 29.06.2018 um 17:04 schrieb Jason Dillaman:
Is 'tcmu-runner' running on that node? 

yes it is running

Any errors in dmesg or 

here are no errors

/var/log/tcmu-runner.log?

the following error is shown:
[ERROR] add_device:436: could not find handler for uio0

  Regards
Bernhard


On Fri, Jun 29, 2018 at 10:43 AM Bernhard Dick > wrote:


Hi,

Am 28.06.2018 um 18:09 schrieb Jason Dillaman:
 > Do you have the ansible backtrace from the "ceph-iscsi-gw :
igw_lun |
 > configureluns (create/map rbds and add to lio)]" step?
I assume you mean the following (from running with verbosity 3, after
running the purge-iscsi-gateways playbook beforehand)

Using module file /home/dick/git/ceph-ansible/library/igw_lun.py
 ESTABLISH SSH CONNECTION FOR USER: root
 SSH: EXEC ssh -o ControlMaster=auto -o
ControlPersist=600s -o StrictHostKeyChecking=no -o
KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,g
ssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o
User=root -o ConnectTimeout=30 -o
ControlPath=/home/dick/.ansible/cp/%h-%r-%p ovirt-test04 '/bin/sh -c
'"'"'/usr/bin/p
ython && sleep 0'"'"''
 (1, '', 'Traceback (most recent call last):\n  File
"/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in
\n
     ansible_main()\n  File "/tmp/ansible_F3
sf6j/ansible_module_igw_lun.py", line 135, in ansible_main\n
lun.manage(desired_state)\n  File
"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 365,
in manage\n    self.allocate()\n  File "/u
sr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 490, in
allocate\n    lun = self.add_dev_to_lio()\n  File
"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 683,
in add_dev_to_lio\n
      wwn=in_wwn, control=control_string)\n  File
"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 818, in
__init__\n    self._configure(config, size, wwn, hw_max_sectors,
control)\n  File "/usr/lib/pytho
n2.7/site-packages/rtslib_fb/tcm.py", line 836, in _configure\n
self._enable()\n  File
"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 172, in
_enable\n    fwrite(path, "1\\n")\n  File "/usr/lib/pyt
hon2.7/site-packages/rtslib_fb/utils.py", line 79, in fwrite\n
file_fd.write(str(string))\nIOError: [Errno 2] No such file or
directory\n')
The full traceback is:
Traceback (most recent call last):
    File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in

      ansible_main()
    File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 135, in
ansible_main
      lun.manage(desired_state)
    File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
line 365, in manage
      self.allocate()
    File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
line 490, in allocate
      lun = self.add_dev_to_lio()
    File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
line 683, in add_dev_to_lio
      wwn=in_wwn, control=control_string)
    File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 818,
in __init__
      self._configure(config, size, wwn, hw_max_sectors, control)
    File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 836,
in _configure
      self._enable()
    File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 172,
in _enable
      fwrite(path, "1\n")
    File "/usr/lib/python2.7/site-packages/rtslib_fb/utils.py", line
79,
in fwrite
      file_fd.write(str(string))
IOError: [Errno 2] No such file or directory

failed: [ovirt-test04] (item={u'host': u'ovirt-test04', u'image':
u'iscsi2342', u'state': u'present', u'pool': u'rbd', u'size':
u'1G'}) => {
      "changed": false,
      "item": {
          "host": "ovirt-test04",
          "image": "iscsi2342",
          "pool": "rbd",
          "size": "1G",
          "state": "present"
      },
      "module_stderr": "Traceback (most recent call last):\n  File
\"/tmp/ansible_F3sf6j/ansible_module_igw_lun.py\", line 166, in
\n    ansible_main()\n  File
\"/tmp/ansible_F3sf6j/ansible_module_igw_lun.
py\", line 135, in ansible_main\n    lun.manage(desired_state)\n  File
\"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py\", line
365,
in manage\n    self.allocate()\n  File \"/usr/lib/python2.7/site-pa
ckages/ceph_iscsi_config/lun.py\", line 490, in allocate\n    lun =
self.add_dev_to_lio()\n  File
\"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py\", line
683,
in add_dev_to_lio\n    wwn=in_wwn, contr
ol=control_string)\n  File
\"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py\", line 818, in
__init__\n    self._configure(config, size, wwn, 

[ceph-users] RBD gets resized when used as iSCSI target

[Wladimir Mutel]

Dear all,

	I create an RBD to be used as iSCSI target, with size close to the most 
popular 3TB HDD size, 5860533168 512-byte sectors, or 715398*4M Ceph 
objects (2.7 TB or 2794.4 GB). Then I add it into gwcli/disks (having to 
specify the same size, 2861592M), and then, after some manipulations 
which I do not remember exactly (like, remove it from gwcli conf, use it 
for some time as RBD target in QEMU VM, then re-add it back to 
gwcli/disks), I discover that its size is rounded up to 3 TiB, i.e. 3072 
GiB or 786432*4M Ceph objects. As we know, GPT is stored at the end of 
block device, so when we increase its size in this way, GPT becomes 
inaccessible and partition limits need to be guessed anew in some other way.


	I can shrink this gratuitously-increased RBD by 'rbd resize', and this 
is reflected in 'gwcli ls /' (3.0T becomes 2.7T). But not in 'targetcli 
ls /' (there, it is still 3.0T). Also, when I restart 
rbd-target-gw.service, it gets resized back up to 3.0T as shown by 
'gwcli ls /' and to 786432 objects in 'rbd info'. I look into 
rbd/gateway.conf RADOS object, and don't see any explicit size specified 
there. Where does it take this 3.0T size from ? My last suspicion is RBD 
name which is 'tower-prime-e-3tb'. Can its '3tb' suffix be the culprit ?


Thank you in advance for your replies.
I am getting lost and slowly infuriated with this behavior.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems setting up iSCSI

[Jason Dillaman]

Is 'tcmu-runner' running on that node? Any errors in dmesg or
/var/log/tcmu-runner.log?

On Fri, Jun 29, 2018 at 10:43 AM Bernhard Dick  wrote:

> Hi,
>
> Am 28.06.2018 um 18:09 schrieb Jason Dillaman:
> > Do you have the ansible backtrace from the "ceph-iscsi-gw : igw_lun |
> > configureluns (create/map rbds and add to lio)]" step?
> I assume you mean the following (from running with verbosity 3, after
> running the purge-iscsi-gateways playbook beforehand)
>
> Using module file /home/dick/git/ceph-ansible/library/igw_lun.py
>  ESTABLISH SSH CONNECTION FOR USER: root
>  SSH: EXEC ssh -o ControlMaster=auto -o
> ControlPersist=600s -o StrictHostKeyChecking=no -o
> KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,g
> ssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o
> User=root -o ConnectTimeout=30 -o
> ControlPath=/home/dick/.ansible/cp/%h-%r-%p ovirt-test04 '/bin/sh -c
> '"'"'/usr/bin/p
> ython && sleep 0'"'"''
>  (1, '', 'Traceback (most recent call last):\n  File
> "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in \n
> ansible_main()\n  File "/tmp/ansible_F3
> sf6j/ansible_module_igw_lun.py", line 135, in ansible_main\n
> lun.manage(desired_state)\n  File
> "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 365,
> in manage\nself.allocate()\n  File "/u
> sr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 490, in
> allocate\nlun = self.add_dev_to_lio()\n  File
> "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 683,
> in add_dev_to_lio\n
>  wwn=in_wwn, control=control_string)\n  File
> "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 818, in
> __init__\nself._configure(config, size, wwn, hw_max_sectors,
> control)\n  File "/usr/lib/pytho
> n2.7/site-packages/rtslib_fb/tcm.py", line 836, in _configure\n
> self._enable()\n  File
> "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 172, in
> _enable\nfwrite(path, "1\\n")\n  File "/usr/lib/pyt
> hon2.7/site-packages/rtslib_fb/utils.py", line 79, in fwrite\n
> file_fd.write(str(string))\nIOError: [Errno 2] No such file or
> directory\n')
> The full traceback is:
> Traceback (most recent call last):
>File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in
> 
>  ansible_main()
>File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 135, in
> ansible_main
>  lun.manage(desired_state)
>File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> line 365, in manage
>  self.allocate()
>File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> line 490, in allocate
>  lun = self.add_dev_to_lio()
>File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py",
> line 683, in add_dev_to_lio
>  wwn=in_wwn, control=control_string)
>File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 818,
> in __init__
>  self._configure(config, size, wwn, hw_max_sectors, control)
>File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 836,
> in _configure
>  self._enable()
>File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 172,
> in _enable
>  fwrite(path, "1\n")
>File "/usr/lib/python2.7/site-packages/rtslib_fb/utils.py", line 79,
> in fwrite
>  file_fd.write(str(string))
> IOError: [Errno 2] No such file or directory
>
> failed: [ovirt-test04] (item={u'host': u'ovirt-test04', u'image':
> u'iscsi2342', u'state': u'present', u'pool': u'rbd', u'size': u'1G'}) => {
>  "changed": false,
>  "item": {
>  "host": "ovirt-test04",
>  "image": "iscsi2342",
>  "pool": "rbd",
>  "size": "1G",
>  "state": "present"
>  },
>  "module_stderr": "Traceback (most recent call last):\n  File
> \"/tmp/ansible_F3sf6j/ansible_module_igw_lun.py\", line 166, in
> \nansible_main()\n  File
> \"/tmp/ansible_F3sf6j/ansible_module_igw_lun.
> py\", line 135, in ansible_main\nlun.manage(desired_state)\n  File
> \"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py\", line 365,
> in manage\nself.allocate()\n  File \"/usr/lib/python2.7/site-pa
> ckages/ceph_iscsi_config/lun.py\", line 490, in allocate\nlun =
> self.add_dev_to_lio()\n  File
> \"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py\", line 683,
> in add_dev_to_lio\nwwn=in_wwn, contr
> ol=control_string)\n  File
> \"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py\", line 818, in
> __init__\nself._configure(config, size, wwn, hw_max_sectors,
> control)\n  File \"/usr/lib/python2.7/site-packages
> /rtslib_fb/tcm.py\", line 836, in _configure\nself._enable()\n  File
> \"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py\", line 172, in
> _enable\nfwrite(path, \"1\\n\")\n  File \"/usr/lib/python2.7/site-
> packages/rtslib_fb/utils.py\", line 79, in fwrite\n
> file_fd.write(str(string))\nIOError: [Errno 2] No such file or
> directory\n",
>  "module_stdout": "",
>  "msg": "MODULE 

Re: [ceph-users] Problems setting up iSCSI

[Bernhard Dick]

Hi,

Am 28.06.2018 um 18:09 schrieb Jason Dillaman:
Do you have the ansible backtrace from the "ceph-iscsi-gw : igw_lun | 
configureluns (create/map rbds and add to lio)]" step? 
I assume you mean the following (from running with verbosity 3, after 
running the purge-iscsi-gateways playbook beforehand)


Using module file /home/dick/git/ceph-ansible/library/igw_lun.py
 ESTABLISH SSH CONNECTION FOR USER: root
 SSH: EXEC ssh -o ControlMaster=auto -o 
ControlPersist=600s -o StrictHostKeyChecking=no -o 
KbdInteractiveAuthentication=no -o 
PreferredAuthentications=gssapi-with-mic,g
ssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 
User=root -o ConnectTimeout=30 -o 
ControlPath=/home/dick/.ansible/cp/%h-%r-%p ovirt-test04 '/bin/sh -c 
'"'"'/usr/bin/p

ython && sleep 0'"'"''
 (1, '', 'Traceback (most recent call last):\n  File 
"/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in \n 
   ansible_main()\n  File "/tmp/ansible_F3
sf6j/ansible_module_igw_lun.py", line 135, in ansible_main\n 
lun.manage(desired_state)\n  File 
"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 365, 
in manage\nself.allocate()\n  File "/u
sr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 490, in 
allocate\nlun = self.add_dev_to_lio()\n  File 
"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 683, 
in add_dev_to_lio\n
wwn=in_wwn, control=control_string)\n  File 
"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 818, in 
__init__\nself._configure(config, size, wwn, hw_max_sectors, 
control)\n  File "/usr/lib/pytho
n2.7/site-packages/rtslib_fb/tcm.py", line 836, in _configure\n 
self._enable()\n  File 
"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 172, in 
_enable\nfwrite(path, "1\\n")\n  File "/usr/lib/pyt
hon2.7/site-packages/rtslib_fb/utils.py", line 79, in fwrite\n 
file_fd.write(str(string))\nIOError: [Errno 2] No such file or directory\n')

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 166, in 


ansible_main()
  File "/tmp/ansible_F3sf6j/ansible_module_igw_lun.py", line 135, in 
ansible_main

lun.manage(desired_state)
  File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", 
line 365, in manage

self.allocate()
  File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", 
line 490, in allocate

lun = self.add_dev_to_lio()
  File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", 
line 683, in add_dev_to_lio

wwn=in_wwn, control=control_string)
  File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 818, 
in __init__

self._configure(config, size, wwn, hw_max_sectors, control)
  File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 836, 
in _configure

self._enable()
  File "/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py", line 172, 
in _enable

fwrite(path, "1\n")
  File "/usr/lib/python2.7/site-packages/rtslib_fb/utils.py", line 79, 
in fwrite

file_fd.write(str(string))
IOError: [Errno 2] No such file or directory

failed: [ovirt-test04] (item={u'host': u'ovirt-test04', u'image': 
u'iscsi2342', u'state': u'present', u'pool': u'rbd', u'size': u'1G'}) => {

"changed": false,
"item": {
"host": "ovirt-test04",
"image": "iscsi2342",
"pool": "rbd",
"size": "1G",
"state": "present"
},
"module_stderr": "Traceback (most recent call last):\n  File 
\"/tmp/ansible_F3sf6j/ansible_module_igw_lun.py\", line 166, in 
\nansible_main()\n  File 
\"/tmp/ansible_F3sf6j/ansible_module_igw_lun.
py\", line 135, in ansible_main\nlun.manage(desired_state)\n  File 
\"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py\", line 365, 
in manage\nself.allocate()\n  File \"/usr/lib/python2.7/site-pa
ckages/ceph_iscsi_config/lun.py\", line 490, in allocate\nlun = 
self.add_dev_to_lio()\n  File 
\"/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py\", line 683, 
in add_dev_to_lio\nwwn=in_wwn, contr
ol=control_string)\n  File 
\"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py\", line 818, in 
__init__\nself._configure(config, size, wwn, hw_max_sectors, 
control)\n  File \"/usr/lib/python2.7/site-packages
/rtslib_fb/tcm.py\", line 836, in _configure\nself._enable()\n  File 
\"/usr/lib/python2.7/site-packages/rtslib_fb/tcm.py\", line 172, in 
_enable\nfwrite(path, \"1\\n\")\n  File \"/usr/lib/python2.7/site-
packages/rtslib_fb/utils.py\", line 79, in fwrite\n 
file_fd.write(str(string))\nIOError: [Errno 2] No such file or 
directory\n",

"module_stdout": "",
"msg": "MODULE FAILURE",
"rc": 1
}

Have you tried 
using the stock v4.16 kernel (no need to use the one on shaman)?
I have also used the v4.17.2-1.el7.elrepo.x86_64 elrepo kernel, having 
the same result.


  Regards
Bernhard


On Thu, Jun 28, 2018 at 11:29 AM Bernhard Dick > wrote:


Hi Jason,

Am 

Re: [ceph-users] cephfs compression?

[Youzhong Yang]

Thanks Richard. Yes, it seems working by perf dump:

osd.6
"bluestore_compressed":   62622444,
"bluestore_compressed_allocated": 186777600,
"bluestore_compressed_original":373555200,

It's very interesting that  bluestore_compressed_allocated is approximately
50% of  bluestore_compressed_original across all OSDs, just curious - why?

On Fri, Jun 29, 2018 at 1:15 AM, Richard Bade  wrote:

> Oh, also because the compression is at the osd level you don't see it
> in ceph df. You just see that your RAW is not increasing as much as
> you'd expect. E.g.
> $ sudo ceph df
> GLOBAL:
> SIZE AVAIL RAW USED %RAW USED
> 785T  300T 485T 61.73
> POOLS:
> NAMEID USED %USED MAX AVAIL OBJECTS
> cephfs-metadata 11 185M 068692G   178
> cephfs-data 12 408T 75.26  134T 132641159
>
> You can see that we've used 408TB in the pool but only 485TB RAW -
> Rather than ~600TB RAW that I'd expect for my k4, m2 pool settings.
> On Fri, 29 Jun 2018 at 17:08, Richard Bade  wrote:
> >
> > I'm using compression on a cephfs-data pool in luminous. I didn't do
> > anything special
> >
> > $ sudo ceph osd pool get cephfs-data all | grep ^compression
> > compression_mode: aggressive
> > compression_algorithm: zlib
> >
> > You can check how much compression you're getting on the osd's
> > $ for osd in `seq 0 11`; do echo osd.$osd; sudo ceph daemon osd.$osd
> > perf dump | grep 'bluestore_compressed'; done
> > osd.0
> > "bluestore_compressed": 686487948225,
> > "bluestore_compressed_allocated": 788659830784,
> > "bluestore_compressed_original": 1660064620544,
> > 
> > osd.11
> > "bluestore_compressed": 700999601387,
> > "bluestore_compressed_allocated": 808854355968,
> > "bluestore_compressed_original": 1752045551616,
> >
> > I can't say for mimic, but definitely for luminous v12.2.5 compression
> > is working well with mostly default options.
> >
> > -Rich
> >
> > > For RGW, compression works very well. We use rgw to store crash dumps,
> in
> > > most cases, the compression ratio is about 2.0 ~ 4.0.
> >
> > > I tried to enable compression for cephfs data pool:
> >
> > > # ceph osd pool get cephfs_data all | grep ^compression
> > > compression_mode: force
> > > compression_algorithm: lz4
> > > compression_required_ratio: 0.95
> > > compression_max_blob_size: 4194304
> > > compression_min_blob_size: 4096
> >
> > > (we built ceph packages and enabled lz4.)
> >
> > > It doesn't seem to work. I copied a 8.7GB folder to cephfs, ceph df
> says it
> > > used 8.7GB:
> >
> > > root at ceph-admin:~# ceph df
> > > GLOBAL:
> > > SIZE   AVAIL  RAW USED %RAW USED
> > > 16 TiB 16 TiB  111 GiB  0.69
> > > POOLS:
> > > NAMEID USED%USED MAX AVAIL
>  OBJECTS
> > > cephfs_data 1  8.7 GiB  0.17   5.0 TiB
> 360545
> > > cephfs_metadata 2  221 MiB 0   5.0 TiB
>  77707
> >
> > > I know this folder can be compressed to ~4.0GB under zfs lz4
> compression.
> >
> > > Am I missing anything? how to make cephfs compression work? is there
> any
> > trick?
> >
> > > By the way, I am evaluating ceph mimic v13.2.0.
> >
> > > Thanks in advance,
> > > --Youzhong
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] VMWARE and RBD

[Steven Vacaroaia]

Hi Horace

Thanks

Would you be willing to share instructions for using SCST instead of
ceph-iscsi ?

Thanks
Steven

On Thu, 28 Jun 2018 at 23:59, Horace  wrote:

> Seems there's no plan for that and the vmware kernel documentation will
> only share to partners. You would better off to use iscsi. By the way, i
> found that the performance is much better for SCST than ceph-iscsi. I don't
> think ceph-iscsi is production-ready?
>
> Regards,
> Horace Ng
>
> --
> *From: *"Steven Vacaroaia" 
> *To: *"ceph-users" 
> *Sent: *Tuesday, June 19, 2018 12:08:40 AM
> *Subject: *[ceph-users] VMWARE and RBD
>
> Hi,
> I read somewhere that VMWare is planning to support RBD directly
>
> Anyone here know more about this ..maybe a tentative / date / version  ?
>
> Thanks
> Steven
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous Bluestore performance, bcache

[Andrei Mikhailovsky]

Thanks Richard,

That sounds impressive, especially the around 30% hit ratio. That would be 
ideal for me, but we were only getting single digit results during my trials. I 
think around 5% was the figure if I remember correctly.

However, most of our vms were created a bit chaotically (not using predefined 
vm templates), which could be the reason for the low cache hit rate. If most of 
your vms are created from templates, you should get a far better cache 
performance figures I think.

But the main factor is the data type, which is very random for our case as we 
have different applications and services.

Would be interested to learn how you are achieving your 30-35% hit ratio on 700 
vms.

Cheers

- Original Message -
> From: "Richard Bade" 
> To: "ceph-users" 
> Sent: Thursday, 28 June, 2018 23:56:59
> Subject: Re: [ceph-users] Luminous Bluestore performance, bcache

> Hi Andrei,
> These are good questions. We have another cluster with filestore and
> bcache but for this particular one I was interested in testing out
> bluestore. So I have used bluestore both with and without bcache.
> For my synthetic load on the vm's I'm using this fio command:
> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4k --iodepth=64 --size=4G
> --readwrite=randwrite --rate_iops=50
> 
> Currently on bluestore with my synthetic load I'm getting 7% hit ratio
> (cat /sys/block/bcache*/bcache/stats_total/cache_hit_ratio)
> On our filestore cluster with ~700 vm's of varied workload we're
> geting about 30-35% hit ratio.
> In the hourly hit ratio I have as high as 50% on some osd's in our
> filestore cluster. Only 25% on my synthetic load on bluestore so far,
> but I hadn't actually been checking this stat until now.
> 
> I hope that helps.
> Regards,
> Richard
> 
>> Hi Richard,
>> It is an interesting test for me too as I am planning to migrate to
>> Bluestore storage and was considering repurposing the ssd disks
>> that we currently use for journals.
>> I was wondering if you are using the Filestore or the bluestone
>> for the osds?
>> Also, when you perform your testing, how good is the hit ratio
>> that you have on the bcache?
>> Are you using a lot of random data for your benchmarks? How
>> large is your test file for each vm?
>> We have been playing around with a few caching scenarios a
>> few years back (enchanceio and a few more which I can't
>> remember now) and we have seen a very poor hit ratio on the
>> caching system. Was wondering if you see a different picture?
>> Cheers
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to secure Prometheus endpoints (mgr plugin and node_exporter)

[Martin Palma]

Since Prometheus uses a pull model over HTTP for collecting metrics.
What are the best practices to secure these HTTP endpoints?

- With a reverse proxy with authentication?
- Export the node_exporter only on the cluster network? (not usable
for the mgr plugin and for nodes like mons, mdss,...)
- No security at all?

Best,
Martin
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-volume: failed to activate some bluestore osds

[Dan van der Ster]

On Thu, Jun 7, 2018 at 8:40 PM Dan van der Ster  wrote:
>
> On Thu, Jun 7, 2018 at 6:33 PM Sage Weil  wrote:
> >
> > On Thu, 7 Jun 2018, Dan van der Ster wrote:
> > > > > Wait, we found something!!!
> > > > >
> > > > > In the 1st 4k on the block we found the block.db pointing at the wrong
> > > > > device (/dev/sdc1 instead of /dev/sdaa1)
> > > > >
> > > > > 0130  6b 35 79 2b 67 3d 3d 0d  00 00 00 70 61 74 68 5f  
> > > > > |k5y+g==path_|
> > > > > 0140  62 6c 6f 63 6b 2e 64 62  09 00 00 00 2f 64 65 76  
> > > > > |block.db/dev|
> > > > > 0150  2f 73 64 63 31 05 00 00  00 72 65 61 64 79 05 00  
> > > > > |/sdc1ready..|
> > > > > 0160  00 00 72 65 61 64 79 06  00 00 00 77 68 6f 61 6d  
> > > > > |..readywhoam|
> > > > > 0170  69 02 00 00 00 34 38 eb  c2 d7 d6 00 00 00 00 00  
> > > > > |i48.|
> > > > > 0180  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  
> > > > > ||
> > > > >
> > > > > It is similarly wrong for another broken osd.53 (block.db is /dev/sdc2
> > > > > instead of /dev/sdaa2).
> > > > > And for the osds that are running, that block.db is correct!
> >
> > Also, note that you can fix your OSDs by changing the path to a stable
> > name for the same device (/dev/disk/by-partuuid/something?) with
> > 'ceph-bluestore-tool set-label-key ...'.
>
> Good to know, thanks!
> I understand your (3) earlier now...  Yes, ceph-volume should call
> that to fix the OSD if the device changes.

FTR the fixes for this issue have been merged into luminous for
12.2.6: https://github.com/ceph/ceph/pull/22716

If someone runs into this before 12.2.6 is out, here's a quick fix:

for db in /var/lib/ceph/osd/ceph-*/block.db
do
  dev=`readlink -f $db`
  osd=`dirname $db`
  echo "ceph-bluestore-tool set-label-key --key path_block.db --value
${dev} --dev ${osd}/block"
done

Cheers, Dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pre-sharding s3 buckets

[Sean Purdy]

On Wed, 27 Jun 2018, Matthew Vernon said:
> Hi,
> 
> On 27/06/18 11:18, Thomas Bennett wrote:
> 
> > We have a particular use case that we know that we're going to be
> > writing lots of objects (up to 3 million) into a bucket. To take
> > advantage of sharding, I'm wanting to shard buckets, without the
> > performance hit of resharding.
> 
> I assume you're running Jewel (Luminous has dynamic resharding); you can
> set rgw_override_bucket_index_max_shards = X in your ceph.conf, which
> will cause all new buckets to have X shards for the indexes.
> 
> HTH,
> 
> Matthew

But watch out if you are running Luminous - manual and automatic
resharding breaks if you have versioning or lifecycles on your bucket.
Fix in next stable release 12.2.6 apparently.

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-January/023968.html
http://tracker.ceph.com/issues/23886


Sean
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph FS (kernel driver) - Unable to set extended file attributed

[Yu Haiyang]

Problem solved.

Seems like I can’t set stripe_unit to a value larger than object_size.
I should increase object_size attribute before increasing stripe_unit attribute.

Hope this would help someone. :)

On Jun 29, 2018, at 12:20 PM, Yu Haiyang 
mailto:haiya...@moqi.ai>> wrote:

Hi,

I want to play around with my ceph.file.layout attributes such as stripe_unit 
and object_size to see how it affected my Ceph FS performance.
However I’ve been unable to set any attribute with below error.

$ setfattr -n ceph.file.layout.stripe_unit -v 41943040 file1
setfattr: file1: Invalid argument

Using strace I can see it failed at something related to missing locale 
language pack.
Any suggestion how to resolve this?

$ strace setfattr -n ceph.file.layout.stripe_unit -v 41943040 file
open("/usr/share/locale/en_HK/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No 
such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such 
file or directory)
open("/usr/share/locale-langpack/en_HK/LC_MESSAGES/libc.mo", O_RDONLY) = -1 
ENOENT (No such file or directory)
open("/usr/share/locale-langpack/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT 
(No such file or directory)
write(2, "setfattr: file: Invalid argument"..., 33setfattr: file: Invalid 
argument
) = 33
exit_group(1)   = ?

Many thanks,
Haiyang
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com