Re: [ceph-users] problem w libvirt version 4.5 and 12.2.7

2019-01-03 Thread Tomasz Płaza
Konstantin, Thanks for reply. I've managed to unravel it partially. Somehow (did not look into srpm) starting from this version libvirt started to calculate real allocation if fastdiff feature is present on image. Doing "rbd object-map rebuild" on every image helped (do not know why it was

Re: [ceph-users] upgrade from jewel 10.2.10 to 10.2.11 broke anonymous swift

2019-01-03 Thread Johan Guldmyr
> > Does anybody have a suggestion of what I could try to troubleshoot this? Upgrading to Luminous also "solves the issue". I'll look into that :) // Johan ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Help Ceph Cluster Down

2019-01-03 Thread Arun POONIA
Hi Chris, Indeed that's what happened. I didn't set noout flag either and I did zapped disk on new server every time. In my cluster status fre201 is only new server. Current Status after enabling 3 OSDs on fre201 host. [root@fre201 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS

[ceph-users] cephfs : rsync backup create cache pressure on clients, filling caps

2019-01-03 Thread Alexandre DERUMIER
Hi, I'm currently doing cephfs backup, through a dedicated clients mounting the whole filesystem at root. others clients are mounting part of the filesystem. (kernel cephfs clients) I have around 22millions inodes, before backup, I have around 5M caps loaded by clients #ceph daemonperf

Re: [ceph-users] Help Ceph Cluster Down

2019-01-03 Thread Chris
If you added OSDs and then deleted them repeatedly without waiting for replication to finish as the cluster attempted to re-balance across them, its highly likely that you are permanently missing PGs (especially if the disks were zapped each time). If those 3 down OSDs can be revived there is

[ceph-users] Help Ceph Cluster Down

2019-01-03 Thread Arun POONIA
Hi, Recently I tried adding a new node (OSD) to ceph cluster using ceph-deploy tool. Since I was experimenting with tool and ended up deleting OSD nodes on new server couple of times. Now since ceph OSDs are running on new server cluster PGs seems to be inactive (10-15%) and they are not

Re: [ceph-users] Compacting omap data

2019-01-03 Thread Brad Hubbard
Nautilus will make this easier. https://github.com/ceph/ceph/pull/18096 On Thu, Jan 3, 2019 at 5:22 AM Bryan Stillwell wrote: > > Recently on one of our bigger clusters (~1,900 OSDs) running Luminous > (12.2.8), we had a problem where OSDs would frequently get restarted while >

Re: [ceph-users] CephFS client df command showing raw space after adding second pool to mds

2019-01-03 Thread Yan, Zheng
On Fri, Jan 4, 2019 at 1:53 AM David C wrote: > > Hi All > > Luminous 12.2.12 > Single MDS > Replicated pools > > A 'df' on a CephFS kernel client used to show me the usable space (i.e the > raw space with the replication overhead applied). This was when I just had a > single cephfs data pool.

Re: [ceph-users] Omap issues - metadata creating too many

2019-01-03 Thread J. Eric Ivancich
If you can wait a few weeks until the next release of luminous there will be tooling to do this safely. Abhishek Lekshmanan of SUSE contributed the PR. It adds some sub-commands to radosgw-admin: radosgw-admin reshard stale-instances list radosgw-admin reshard stale-instances rm If you

Re: [ceph-users] Mimic 13.2.3?

2019-01-03 Thread Alex Litvak
It is true for all distros. It doesn't happen the first time either. I think it is a bit dangerous. On 1/3/19 12:25 AM, Ashley Merrick wrote: Have just run an apt update and have noticed there are some CEPH packages now available for update on my mimic cluster / ubuntu. Have yet to install

[ceph-users] CephFS client df command showing raw space after adding second pool to mds

2019-01-03 Thread David C
Hi All Luminous 12.2.12 Single MDS Replicated pools A 'df' on a CephFS kernel client used to show me the usable space (i.e the raw space with the replication overhead applied). This was when I just had a single cephfs data pool. After adding a second pool to the mds and using file layouts to

Re: [ceph-users] Omap issues - metadata creating too many

2019-01-03 Thread Bryan Stillwell
Josef, I've noticed that when dynamic resharding is on it'll reshard some of our bucket indices daily (sometimes more). This causes a lot of wasted space in the .rgw.buckets.index pool which might be what you are seeing. You can get a listing of all the bucket instances in your cluster with

Re: [ceph-users] Help with setting device-class rule on pool without causing data to move

2019-01-03 Thread David C
Thanks, Sage! That did the trick. Wido, seems like an interesting approach but I wasn't brave enough to attempt it! Eric, I suppose this does the same thing that the crushtool reclassify feature does? Thank you both for your suggestions. For posterity: - I grabbed some 14.0.1 packages,

[ceph-users] upgrade from jewel 10.2.10 to 10.2.11 broke anonymous swift

2019-01-03 Thread Johan Guldmyr
Hello, This is with RDO CentOS7, keystone and swift_account_in_url. The CEPH cluster runs luminous. curl 'https://object.example.org/swift/v1/AUTH_12345qhexvalue/test20_segments' this list the contents of the public bucket (Read ACL is .r:* according to swift stat test20_segments ) with

Re: [ceph-users] problem w libvirt version 4.5 and 12.2.7

2019-01-03 Thread Konstantin Shalygin
After update to CentOS 7.6, libvirt was updated from 3.9 to 4.5. Executing: "virsh vol-list ceph --details" makes libvirtd using 300% CPU for 2 minutes to show volumes on rbd. Quick pick at tcpdump shows accessing rbd_data.* which previous version of libvirtd did not need. Ceph version is 12.2.7.

Re: [ceph-users] cephfs kernel client instability

2019-01-03 Thread Andras Pataki
I wonder if anyone could offer any insight on the issue below, regarding the CentOS 7.6 kernel cephfs client connecting to a Luminous cluster.  I have since tried a much newer 4.19.13 kernel, which did not show the same issue (but unfortunately for various reasons unrelated to ceph, we can't

Re: [ceph-users] Omap issues - metadata creating too many

2019-01-03 Thread Josef Zelenka
Hi, i had the default - so it was on(according to ceph kb). turned it off, but the issue persists. i noticed Bryan Stillwell(cc-ing him) had the same issue (reported about it yesterday) - tried his tips about compacting, but it doesn't do anything, however i have to add to his last point, this