Re: [ceph-users] Increasing number of PGs by not a factor of two?

Great summary David. Wouldn't this be worth a blog post? On 17.05.2018 20:36, David Turner wrote: > By sticking with PG numbers as a base 2 number (1024, 16384, etc) all > of your PGs will be the same size and easier to balance and manage.  > What happens when you have a non base 2 number is some

Re: [ceph-users] Poor CentOS 7.5 client performance

On Wed, May 16, 2018 at 8:27 PM, Donald "Mac" McCarthy wrote: > CephFS. 8 core atom C2758, 16 GB ram, 256GB ssd, 2.5 GB NIC (supermicro > microblade node). > > Read test: > dd if=/ceph/1GB.test of=/dev/null bs=1M Yup, looks like a kcephfs regression. The performance of the above command is hig

Re: [ceph-users] Increasing number of PGs by not a factor of two?

You would actually need to go through one last time to get to your target PGs, but anyway, like all commands you come across online, test them and make sure they do what you intend. On Thu, May 17, 2018 at 2:36 PM David Turner wrote: > By sticking with PG numbers as a base 2 number (1024, 16384,

Re: [ceph-users] Increasing number of PGs by not a factor of two?

By sticking with PG numbers as a base 2 number (1024, 16384, etc) all of your PGs will be the same size and easier to balance and manage. What happens when you have a non base 2 number is something like this. Say you have 4 PGs that are all 2GB in size. If you increase pg(p)_num to 6, then you w

[ceph-users] Metadata sync fails after promoting new zone to master - mdlog buffer read issue

I have a ​3 zone multi-site setup using Ceph luminous (12.2.4) on Ubuntu 18.04. I used ceph-deploy to build each cluster and followed https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html-single/object_gateway_guide_for_red_hat_enterprise_linux/index#multi_site for multi-site se

Re: [ceph-users] A question about HEALTH_WARN and monitors holding onto cluster maps

Generally they clean up slowly by deleting 30 maps every time the maps update. You can speed that up by creating false map updates with something like updating a pool setting to what it already is. What it sounds like happened to you is that your mon crashed and restarted. If it crashed and has

Re: [ceph-users] A question about HEALTH_WARN and monitors holding onto cluster maps

That seems like a sane way to do it, thanks for the clarification Wido. As a follow-up, do you have any feeling as to whether the trimming a particularly intensive task? We just had a fun afternoon where the monitors became unresponsive (no ceph status etc) for several hours, seemingly due to t

Re: [ceph-users] Blocked requests activating+remapped afterextendingpg(p)_num

Hi! @Paul Thanks! I know, I read the whole topic about size 2 some months ago. But this has not been my decision, I had to set it up like that. In the meantime, I did a reboot of node1001 and node1002 with flag "noout" set and now peering has finished and only 0.0x% are rebalanced. IO is flowing

[ceph-users] Help/advice with crush rules

I've been trying to wrap my head around crush rules, and I need some help/advice.  I'm thinking of using erasure coding instead of replication, and trying to understand the possibilities for planning for failure cases. For a simplified example, consider a 2 level topology, OSDs live on hosts,

[ceph-users] loaded dup inode

I am seeing ceph-mds error with "loaded dup inode" 2018-05-17 09:33:25.141358 [ERR] loaded dup inode 0x1000212acba [2,head] v7879 at /static/x/A/B/A3AF99016CC90CA60CEFBA1A0696/20180515, but inode 0x1000212acba.head v3911 already exists at /static/X/A/20180515/014 How to clean?. I try to delet

Re: [ceph-users] Blocked requests activating+remapped afterextendingpg(p)_num

Check ceph pg query, it will (usually) tell you why something is stuck inactive. Also: never do min_size 1. Paul 2018-05-17 15:48 GMT+02:00 Kevin Olbrich : > I was able to obtain another NVMe to get the HDDs in node1004 into the > cluster. > The number of disks (all 1TB) is now balanced betwe

Re: [ceph-users] A question about HEALTH_WARN and monitors holding onto cluster maps

On 05/17/2018 04:37 PM, Thomas Byrne - UKRI STFC wrote: > Hi all, > >   > > As far as I understand, the monitor stores will grow while not HEALTH_OK > as they hold onto all cluster maps. Is this true for all HEALTH_WARN > reasons? Our cluster recently went into HEALTH_WARN due to a few weeks >

[ceph-users] A question about HEALTH_WARN and monitors holding onto cluster maps

Hi all, As far as I understand, the monitor stores will grow while not HEALTH_OK as they hold onto all cluster maps. Is this true for all HEALTH_WARN reasons? Our cluster recently went into HEALTH_WARN due to a few weeks of backfilling onto new hardware pushing the monitors data stores over the

Re: [ceph-users] Blocked requests activating+remapped afterextendingpg(p)_num

I was able to obtain another NVMe to get the HDDs in node1004 into the cluster. The number of disks (all 1TB) is now balanced between racks, still some inactive PGs: data: pools: 2 pools, 1536 pgs objects: 639k objects, 2554 GB usage: 5167 GB used, 14133 GB / 19300 GB avail p

Re: [ceph-users] Increasing number of PGs by not a factor of two?

Hi Oliver, a good value is 100-150 PGs per OSD. So in your case between 20k and 30k. You can increase your PGs, but keep in mind that this will keep the cluster quite busy for some while. That said I would rather increase in smaller steps than in one large move. Kai On 17.05.2018 01:29, Oliver

Re: [ceph-users] Poor CentOS 7.5 client performance

Dan, You are correct, that is some important information. I am using the kernel client. Mounts from fstab. Mac Please excuse any typos. Autocorrect is evil! > On May 17, 2018, at 08:30, Dan van der Ster wrote: > > Hi, > > It still isn't clear if you're using the fuse or kernel client.

Re: [ceph-users] Poor CentOS 7.5 client performance

Hi, It still isn't clear if you're using the fuse or kernel client. Do you `mount -t ceph` or something else? -- Dan On Wed, May 16, 2018 at 8:28 PM Donald "Mac" McCarthy wrote: > CephFS. 8 core atom C2758, 16 GB ram, 256GB ssd, 2.5 GB NIC (supermicro microblade node). > Read test: > dd if

Re: [ceph-users] Blocked requests activating+remapped afterextendingpg(p)_num

Ok, I just waited some time but I still got some "activating" issues: data: pools: 2 pools, 1536 pgs objects: 639k objects, 2554 GB usage: 5194 GB used, 11312 GB / 16506 GB avail pgs: 7.943% pgs not active 5567/1309948 objects degraded (0.425%) 1

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

On 05/16/2018 03:34 PM, Wido den Hollander wrote: > > > On 05/16/2018 01:22 PM, Blair Bethwaite wrote: >> On 15 May 2018 at 08:45, Wido den Hollander > > wrote: >> >> > We've got some Skylake Ubuntu based hypervisors that we can look at to >> > compare tomorrow...

Re: [ceph-users] Blocked requests activating+remapped afterextendingpg(p)_num

PS: Cluster currently is size 2, I used PGCalc on Ceph website which, by default, will place 200 PGs on each OSD. I read about the protection in the docs and later noticed that I better had only placed 100 PGs. 2018-05-17 13:35 GMT+02:00 Kevin Olbrich : > Hi! > > Thanks for your quick reply. > B

Re: [ceph-users] Blocked requests activating+remapped afterextendingpg(p)_num

Hi! Thanks for your quick reply. Before I read your mail, i applied the following conf to my OSDs: ceph tell 'osd.*' injectargs '--osd_max_pg_per_osd_hard_ratio 32' Status is now: data: pools: 2 pools, 1536 pgs objects: 639k objects, 2554 GB usage: 5211 GB used, 11295 GB / 16506

Re: [ceph-users] Blocked requests activating+remapped afterextendingpg(p)_num

Hi, On 05/17/2018 01:09 PM, Kevin Olbrich wrote: Hi! Today I added some new OSDs (nearly doubled) to my luminous cluster. I then changed pg(p)_num from 256 to 1024 for that pool because it was complaining about to few PGs. (I noticed that should better have been small changes). This is the cu

[ceph-users] Blocked requests activating+remapped after extending pg(p)_num

Hi! Today I added some new OSDs (nearly doubled) to my luminous cluster. I then changed pg(p)_num from 256 to 1024 for that pool because it was complaining about to few PGs. (I noticed that should better have been small changes). This is the current status: health: HEALTH_ERR 336

Re: [ceph-users] [SUSPECTED SPAM]Re: RBD features and feature journaling performance

On Thu, May 17, 2018 at 11:03 AM, Jorge Pinilla López wrote: > Thanks for the info!, I absolutely agree that it should be documented > > Any further info about why journaling feature is so slow? Because everything is written twice: first to the journal and then to the actual data objects. journa

Re: [ceph-users] [SUSPECTED SPAM]Re: RBD features and feature journaling performance

Thanks for the info!, I absolutely agree that it should be documented Any further info about why journaling feature is so slow? El jueves, 17 de mayo de 2018 5:04:51 (CEST) Konstantin Shalygin escribió: > > I'm trying to better understand rbd features but I have only found the > > information on

Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?

Brad, thanks for the bug report. This is exactly the problem I am having (log-wise). >>> >>> You don't give any indication what version you are running but see >>> https://tracker.ceph.com/issues/23205 >> >> >> the cluster is an Proxmox installation which is based on an Ubuntu kernel. >> >> # ceph

Re: [ceph-users] in retrospect get OSD for "slow requests are blocked" ? / get detailed health status via librados?

On Thu, May 17, 2018 at 4:16 PM, Uwe Sauter wrote: > Hi, > >>> I'm currently chewing on an issue regarding "slow requests are blocked". >>> I'd like to identify the OSD that is causing those events >>> once the cluster is back to HEALTH_OK (as I have no monitoring yet that >>> would get this info

Re: [ceph-users] Question to avoid service stop when osd is full

Thank you, David. I found "ceph osd pool set-quota" command. I think using this command to SSD pool is useful to avoid the problem in quotation, isn't it? best regards On 2018/04/10 5:22, David Turner wrote: The proper way to prevent this is to set your full ratios safe and monitor your disk