[ceph-users] Recovery throughput inversely linked with rbd_cache_xyz?

Hello, A couple days ago I increased the rbd cache size from the default to 256MB/osd on a small 4 node, 6 osd/node setup in a test/lab setting.  The rbd volumes are all vm images with writeback cache parameters and steady if only a few mb/sec writes going on. Logging mostly.    I noticed

[ceph-users] active+remapped+backfilling keeps going .. and going

Hello, I have a Proxmox ceph cluster with 5 nodes and 3 OSDs each (total 15 OSDs), on a 10G network. The cluster started small, and I’ve progressively added OSDs over time. Problem is…. The cluster never rebalances completely. There is always progress on backfilling, but PGs that used to be

[ceph-users] v13.2.10 Mimic released

We're happy to announce the availability of the tenth bugfix release of Ceph Mimic, this release fixes a RGW vulnerability affecting mimic, and we recommend that all mimic users upgrade. Notable Changes --- * CVE 2020 12059: Fix an issue with Post Object Requests with Tagging

[ceph-users] Re: Upgrading to Octopus

Hello Simon, I think that Khodayar is right. I managed to install a new Ceph cluster on CentOS 8.1. Therefore you will need the ceph-el8.repo for the time being. For some reason, "they" left the py3 packages you mentioned out of EPEL (as with leveldb, but this package appeared luckily last week

[ceph-users] Re: adding block.db to OSD

Hi, if the OSDs are idle the difference is even more worse: # ceph tell osd.0 bench { "bytes_written": 1073741824, "blocksize": 4194304, "elapsed_sec": 15.39670787501, "bytes_per_sec": 69738403.346825853, "iops": 16.626931034761871 } # ceph tell osd.38 bench {

[ceph-users] Re: How to debug ssh: ceph orch host add ceph01 10.10.1.1

Can anyone help me here? :-/ On Wed, Apr 22, 2020 at 10:36 PM Ml Ml wrote: > > Hello List, > > i did: > root@ceph01:~# ceph cephadm set-ssh-config -i /tmp/ssh_conf > > root@ceph01:~# cat /tmp/ssh_conf > Host * > User root > StrictHostKeyChecking no > UserKnownHostsFile /dev/null > >

[ceph-users] Re: Upgrading to Octopus

Yes! That was what I was going to paste here On Thu, Apr 23, 2020 at 7:18 PM Adam Tygart wrote: > The release notes [1] specify only partial support for CentOS 7. > > "Note that the dashboard, prometheus, and restful manager modules will > not work on the CentOS 7 build due to Python 3 module

[ceph-users] Re: Increase number of read and writes

Hi Janne, Thanks a lot ! I should have checked it earlier..I got it :-) Basically I would like to compile the client read and write C/C++ codes and then later profile the executables with valgrind and other profiling tools. The reason being I want to see the function calls, execution time etc.

[ceph-users] Dear Abby: Why Is Architecting CEPH So Hard?

Hello, Looks like the original chain got deleted, but thank you to everyone who responded. Just to keep any new-comers in the loop, I have pasted the original positing below. To all the original contributors to this chain, I feel much more confident in my design theory for the storage nodes.

[ceph-users] Re: Upgrading to Octopus

The release notes [1] specify only partial support for CentOS 7. "Note that the dashboard, prometheus, and restful manager modules will not work on the CentOS 7 build due to Python 3 module dependencies that are missing in CentOS 7." You will need to move to CentOS 8, or potentially containerize

[ceph-users] Increase number of read and writes

Hi, I am using Ceph in developer mode. Currently I am implementing Librados examples which are also available in Introduction to Librados section https://docs.ceph.com/docs/master/rados/api/librados-intro/#step-3-creating-an-i-o-context. It says once your app has a cluster handle and a connection

[ceph-users] Re: Upgrading to Octopus

Simon, You can try to search for the exact package name, you can try these repos as well: yum -y install epel-release centos-release-ceph-nautilus centos-release-openstack-stein On Thu, Apr 23, 2020 at 11:57 AM Simon Sutter wrote: >

[ceph-users] Re: adding block.db to OSD

Hi, Am 23.04.20 um 14:06 schrieb Igor Fedotov: I don't recall any additional tuning to be applied to new DB volume. And assume the hardware is pretty the same... Do you still have any significant amount of data spilled over for these updated OSDs? If not I don't have any valid explanation for

[ceph-users] Re: adding block.db to OSD

I don't recall any additional tuning to be applied to new DB volume. And assume the hardware is pretty the same... Do you still have any significant amount of data spilled over for these updated OSDs? If not I don't have any valid explanation for the phenomena. You might want to try "ceph

[ceph-users] Re: Dear Abby: Why Is Architecting CEPH So Hard?

Hi Maritin, How is the performance of d120-c21 hdd cluster? Can it utilize the full performance of the 16 hdd？ linyunfan Martin Verges 于2020年4月23日周四 下午6:12写道： > > Hello, > > simpler systems tend to be cheaper to buy per TB storage, not on a > theoretical but practical quote. > > For example 1U

[ceph-users] Re: Dear Abby: Why Is Architecting CEPH So Hard?

On Thu, 2020-04-23 at 09:08 +0200, Janne Johansson wrote: > Den tors 23 apr. 2020 kl 08:49 skrev Darren Soothill < > darren.sooth...@suse.com>: > > > If you want the lowest cost per TB then you will be going with > > larger nodes in your cluster but it does mean you minimum cluster > > size is

[ceph-users] Re: Dear Abby: Why Is Architecting CEPH So Hard?

Hello, simpler systems tend to be cheaper to buy per TB storage, not on a theoretical but practical quote. For example 1U Gigabyte 16bay D120-C21 systems with a density of 64 disks per 4U are quite ok for most users. On 40 Nodes per rack + 2 switches you have 10PB raw space for around 350k€.

[ceph-users] Healthy objects trapped in incomplete pgs

Dear Cephers, A few days ago disaster struck the Ceph cluster (erasure-coded) I am administrating, as the UPS power was pull from the cluster causing a power outage. After rebooting the system, 6 osds were lost (spread over 5 osd nodes) as they could not mount anymore, several others had

[ceph-users] Re: Dear Abby: Why Is Architecting CEPH So Hard?

I can think of 1 vendor who has made some of the compromises that you talk of although memory and CPU is not one of them they are limited on slots and NVME capacity. But there are plenty of other vendors out there who use the same model of motherboard across the whole chassis range so there

[ceph-users] Re: missing amqp-exchange on bucket-notification with AMQP endpoint

HTTP/1.1 > Content-Type: application/x-www-form-urlencoded; charset=utf-8 > Accept-Encoding: identity > Date: Tue, 23 Apr 2020 05:00:35 GMT > X-Amz-Content-Sha256: > e8d828552b412fde2cd686b0a984509bc485693a02e8c53ab84cf36d1dbb961a > Host: s3.example.com > X-Amz-Date: 2 as02004

[ceph-users] Re: Upgrading to Octopus

Hello Khodayar Of cours I tried installing them with yum... They are not available in the centos base or epel repos, here are the ones, which are available: [root@node1 ~]# yum list | egrep "cherrypy|jwt|routes" python-cherrypy.noarch 3.2.2-4.el7@base

[ceph-users] Re: Dear Abby: Why Is Architecting CEPH So Hard?

Den tors 23 apr. 2020 kl 08:49 skrev Darren Soothill < darren.sooth...@suse.com>: > If you want the lowest cost per TB then you will be going with larger > nodes in your cluster but it does mean you minimum cluster size is going to > be many PB’s in size. > Now the question is what is the tax

[ceph-users] Re: Dear Abby: Why Is Architecting CEPH So Hard?

If you want the lowest cost per TB then you will be going with larger nodes in your cluster but it does mean you minimum cluster size is going to be many PB’s in size. There are a number of fixed costs associated with a node. So Motherboard, Network cards, disk controllers, the more disks you