Re: [ceph-users] Changing the release cadence

2019-06-17 Thread Daniel Baumann
Hi, I didn't bother to create a twitter account just to be able to participate in the poll.. so.. please count me in for October. Regards, Daniel ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.co

[ceph-users] Ceph Clients Upgrade?

2019-06-17 Thread Pardhiv Karri
Hi, What is the easiest and fastest way to upgrade Ceph clients which are Openstack instances. We have a lot of Openstack Instances and rebooting them will take a lot of time as we need to communicate with the end users. Is there any other way like restarting a cinder service to get those clients

Re: [ceph-users] obj_size_info_mismatch error handling

2019-06-17 Thread Brad Hubbard
Can you open a tracker for this Dan and provide scrub logs with debug_osd=20 and rados list-inconsistent-obj output? On Mon, Jun 3, 2019 at 10:44 PM Dan van der Ster wrote: > > Hi Reed and Brad, > > Did you ever learn more about this problem? > We currently have a few inconsistencies arriving wit

Re: [ceph-users] How to see the ldout log?

2019-06-17 Thread huang jun
you should add this to your ceph.conf [client] log file = /var/log/ceph/$name.$pid.log debug client = 20 ?? ?? 于2019年6月18日周二 上午11:18写道: > > I am a student new to cephfs. I want see the ldout log in > /src/client/Client.cc (for example, ldout(cct, 20) << " no cap on " << > dn->inode->vino() << d

[ceph-users] How to see the ldout log?

2019-06-17 Thread ?? ??
I am a student new to cephfs. I want see the ldout log in /src/client/Client.cc (for example, ldout(cct, 20) << " no cap on " << dn->inode->vino() << dendl;). Can anyone teach me? The /var/log/ceph dir on the client is empty. ___ ceph-users mailing list

Re: [ceph-users] Shell Script For Flush and Evicting Objects from Cache Tier

2019-06-17 Thread Romit Misra
Usage is as follows:- sh flush_recovery.sh filename.txt & For Logging:- sh flush_recovery.sh filename.txt >> /var/path/flush.log & Thanks Romit On Tue, Jun 18, 2019 at 7:07 AM Romit Misra wrote: > The Script Reads a File, (Object Listing from the cache pool via the rados > ls -p), and starts f

[ceph-users] Shell Script For Flush and Evicting Objects from Cache Tier

2019-06-17 Thread Romit Misra
The Script Reads a File, (Object Listing from the cache pool via the rados ls -p), and starts flushing, and evicting to the Base Tier. It respects any blocked requests on the Cluster, and toggles recovery throttles based on the same (This is the case if the Flush + Evict) is going in at the Same Ti

[ceph-users] Upgrade Documentation: Wait for recovery

2019-06-17 Thread Richard Bade
Hi Everyone, Recently we moved a bunch of our servers from one rack to another. In the late stages of this we hit a point when some requests were blocked due to one pg being in "peered" state. This was unexpected to us, but on discussion with Wido we understand why this happened. However it's brou

[ceph-users] Adding and removing monitors with Mimic's new centralized configuration

2019-06-17 Thread Robert Stanford
Is it possible to add and remove monitors in Mimic, using the new centralized configuration method? Regards R ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Protecting against catastrophic failure of host filesystem

2019-06-17 Thread Eitan Mosenkis
Hi. I'm running a small single-host Ceph cluster on Proxmox (as my home NAS). I want to encrypt my OSDs but I don't want the host's SSD to be a single point of failure. What Ceph config/keyring/secret keys do I need to make safe [encrypted] copies of to ensure that even if the host running the Cep

Re: [ceph-users] Even more objects in a single bucket?

2019-06-17 Thread Paul Emmerich
On Mon, Jun 17, 2019 at 5:02 PM Harald Staub wrote: > There are customers asking for 500 million objects in a single object > storage bucket (i.e. 5000 shards), but also more. But we found some > places that say that there is a limit in the number of shards per > bucket, e.g. > We've got a clust

Re: [ceph-users] Changing the release cadence

2019-06-17 Thread David Turner
This was a little long to respond with on Twitter, so I thought I'd share my thoughts here. I love the idea of a 12 month cadence. I like October because admins aren't upgrading production within the first few months of a new release. It gives it plenty of time to be stable for the OS distros as we

Re: [ceph-users] ceph fs: stat fails on folder

2019-06-17 Thread Frank Schilder
Please ignore the message below, it has nothing to do with ceph. Sorry for the spam. Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: ceph-users on behalf of Frank Schilder Sent: 17 June 2019 20:33 To: ceph-use

[ceph-users] ceph fs: stat fails on folder

2019-06-17 Thread Frank Schilder
We observe the following on ceph fs clients with identical ceph fs mounts: [frans@sophia1 ~]$ ls -l ../neda ls: cannot access ../neda/NEWA_TEST: Permission denied total 5 drwxrwxr-x 1 neda neda1 May 17 19:30 ffpy_test -rw-rw-r-- 1 neda neda 135 May 17 21:06 mount_newa drwxrwxr-x 1 neda neda

[ceph-users] Pool configuration for RGW on multi-site cluster

2019-06-17 Thread Frank Schilder
We have a two-site cluster with OSDs and pools localised in two locations. I was now looking into setting up a rados gateway with the following properties: - Pools should be EC pools whenever possible. - Pools for specific buckets should be localised at OSDs on only one of the two locations (mea

Re: [ceph-users] Changing the release cadence

2019-06-17 Thread Sage Weil
On Wed, 5 Jun 2019, Sage Weil wrote: > That brings us to an important decision: what time of year should we > release? Once we pick the timing, we'll be releasing at that time *every > year* for each release (barring another schedule shift, which we want to > avoid), so let's choose carefully!

Re: [ceph-users] Even more objects in a single bucket?

2019-06-17 Thread Matthew Vernon
Hi, On 17/06/2019 16:00, Harald Staub wrote: > There are customers asking for 500 million objects in a single object > storage bucket (i.e. 5000 shards), but also more. But we found some > places that say that there is a limit in the number of shards per > bucket, e.g. Our largest bucket was abou

[ceph-users] Ceph Scientific Computing User Group

2019-06-17 Thread Kevin Hrpcek
Hey all, At cephalocon some of us who work in scientific computing got together for a BoF and had a good conversation. There was some interest in finding a way to continue the conversation focused on ceph in scientific computing and htc/hpc environments. We are considering putting together mont

Re: [ceph-users] Even more objects in a single bucket?

2019-06-17 Thread Casey Bodley
Hi Harry, When dynamic resharding was introduced for luminous, this limit on the number of bucket index shards was increased from 7877 to 65521. However, you're likely to have problems with bucket listing performance before you get to 7877 shards, because every listing request has to read from

Re: [ceph-users] Even more objects in a single bucket?

2019-06-17 Thread EDH - Manuel Rios Fernandez
Hi Harald , We saw in our internal Veeam repo that only 4TB used created more than 10M objects. I don't know if Veeam need to list content inside the bucket, that will make a 500 millions bucket not a good solution at least in our experience with sharding. I read someone in the IRC telling that

[ceph-users] Even more objects in a single bucket?

2019-06-17 Thread Harald Staub
There are customers asking for 500 million objects in a single object storage bucket (i.e. 5000 shards), but also more. But we found some places that say that there is a limit in the number of shards per bucket, e.g. https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/obj

Re: [ceph-users] Weird behaviour of ceph-deploy

2019-06-17 Thread Brian Topping
I don’t have an answer for you, but it’s going to help others to have shown: Versions of all nodes involved and multi-master configuration Confirm forward and reverse DNS and SSH / remote sudo since you are using deploy Specific steps that did not behave properly > On Jun 17, 2019, at 6:29 AM, CUZA

Re: [ceph-users] Weird behaviour of ceph-deploy

2019-06-17 Thread CUZA Frédéric
I'll keep updating this until I find a solution so if anyone faces the same problem he might have solution. Atm : I install the new osd node with ceph-deploy and nothing change, node is still not present in the cluster nor in the crushmap. I decided to manually add it to the crush map : ceph osd

Re: [ceph-users] rocksdb corruption, stale pg, rebuild bucket index

2019-06-17 Thread Harald Staub
Our resharding was directly from unsharded to 1024 shards. One could imagine that an intermediate step would help, but I have no idea. About bluefs size I would not know of something bad. Well maybe it was a problem that we had a very small (20 GB) device (SSD LV) for the DB which means spillo

Re: [ceph-users] bluestore_allocated vs bluestore_stored

2019-06-17 Thread Mark Nelson
Earlier in bluestore's life, we couldn't handle a 4K min_alloc size on NVMe without incurring pretty significant slowdowns (and also generally higher amounts of metadata in the DB).  Lately I've been seeing some indications that we've improved the stack to the point where 4K min_alloc no longer

Re: [ceph-users] strange osd beacon

2019-06-17 Thread Rafał Wądołowski
Yes, but look into pgs array. It shouldn't be empty. That should by addressed by this PR: https://tracker.ceph.com/issues/40377 Best Regards, Rafał Wądołowski On 16.06.2019 07:06, huang jun wrote: > osd send osd beacons every 300s, and it's used to let mon know that > osd is alive, > for some

Re: [ceph-users] rocksdb corruption, stale pg, rebuild bucket index

2019-06-17 Thread Dan van der Ster
We have resharded a bucket with 60 million objects from 32 to 64 shards without any problem. (Though there were several slow ops at the "stalls after counting the objects phase", so I set nodown as a precaution). We're now resharding that bucket from 64 to 1024. In your case I wonder if it was the

Re: [ceph-users] rocksdb corruption, stale pg, rebuild bucket index

2019-06-17 Thread Harald Staub
We received the large omap warning before, but for some reasons we could not react quickly. We accepted the risk of the bucket becoming slow, but had not thought of further risks ... On 17.06.19 10:15, Dan van der Ster wrote: Nice to hear this was resolved in the end. Coming back to the begin

Re: [ceph-users] out of date python-rtslib repo on https://shaman.ceph.com/

2019-06-17 Thread Matthias Leopold
thank you very much for updating python-rtslib!! could you maybe also do this for tcmu-runner (version 1.4.1)? shaman repos are very convenient for installing and updating the ceph iscsi stack, I would be very happy if I could continue using it matthias Am 14.06.19 um 18:08 schrieb Matthias Le

Re: [ceph-users] bluestore_allocated vs bluestore_stored

2019-06-17 Thread Igor Fedotov
Hi Maged, min_alloc_size determines allocation granularity hence if object size isn't aligned with its value allocation overhead still takes place. E.g. with min_alloc_size = 16K and object size = 24K total allocation (i.e. bluestore_allocated) would be 32K. And yes, this overhead is perman

Re: [ceph-users] rocksdb corruption, stale pg, rebuild bucket index

2019-06-17 Thread Dan van der Ster
Nice to hear this was resolved in the end. Coming back to the beginning -- is it clear to anyone what was the root cause and how other users can avoid this from happening? Maybe some better default configs to warn users earlier about too-large omaps? Cheers, Dan On Thu, Jun 13, 2019 at 7:36 PM H