[ceph-users] maximum numbers of monitor

hi all: I have a question about monitor node, what is the maximum numbers of monitor node and how many monitor node need in a production environment? The official docs only recommends that monitor need odd number, so how do I calculate the numbers of monitor in a cluster, for example, a

[ceph-users] optimization for write when object map feature enabled

Hi Jason, Sorry to bother you. A question about object map. object map tracks which blocks of the image are actually allocated. I can see one use case is for read optimization. For read, it checks the object existence first,if the object doesn't exist locally, it sends read directly to parent. so

Re: [ceph-users] Performance counters oddities, cache tier and otherwise

Hello Nick, On Thu, 7 Apr 2016 10:03:27 +0100 Nick Fisk wrote: > Hi Christian, > > > > Hello, > > > > Ceph 0.94.5 for the record. > > > > As some may remember, I phased in a 2TB cache tier 5 weeks ago. > > > > About now it has reached about 60% usage, which is what I have the > >

Re: [ceph-users] rebalance near full osd

There was a discussion before regarding to the situation where you are facing now. [1] Would you have a look, if it's helpful or not for you. [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007622.html Cheers, Shinobu ___

Re: [ceph-users] rebalance near full osd

Hi Christian, thanks for your input. I don't think the PG count is my issue. I've actually have too many PGs for the size of my cluster consisting of just 18 osds spread amongst 2 osd servers and 3 mons: root@arh-ibstorage1-ib:~# ceph -s health HEALTH_WARN 1 near full osd(s)

Re: [ceph-users] Safely reboot nodes in a Ceph Cluster

There was a discussion regarding to rebooting the ceph cluster before. [1] [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-December/036397.html Cheers, Shinobu On Fri, Apr 8, 2016 at 7:30 AM, Mad Th wrote: > We have a 3 node ceph cluster. How do we safely

Re: [ceph-users] Ceph InfiniBand Cluster - Jewel - Performance

On Thu, Apr 7, 2016 at 10:09 PM, German Anders wrote: > also jewel does not supposed to get more 'performance', since it used > bluestore in order to store metadata. Or do I need to specify during install > to use bluestore? Do the words "enable experimental unrecoverable

Re: [ceph-users] Ceph InfiniBand Cluster - Jewel - Performance

On 04/07/2016 02:43 PM, German Anders wrote: Hi Cephers, I've setup a production environment Ceph cluster with the Jewel release (10.1.0 (96ae8bd25f31862dbd5302f304ebf8bf1166aba6)) consisting of 3 MON Servers and 6 OSD Servers: 3x MON Servers: 2x Intel Xeon E5-2630v3@2.40Ghz 384GB RAM 2x 200G

Re: [ceph-users] Ceph InfiniBand Cluster - Jewel - Performance

also jewel does not supposed to get more 'performance', since it used bluestore in order to store metadata. Or do I need to specify during install to use bluestore? Thanks, *German* 2016-04-07 16:55 GMT-03:00 Robert LeBlanc : > -BEGIN PGP SIGNED MESSAGE- > Hash:

Re: [ceph-users] Ceph InfiniBand Cluster - Jewel - Performance

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ceph is not able to use native Infiniband protocols yet and so it is only leveraging IPoIB at the moment. The most likely reason you are only getting ~10 Gb performance is that IPoIB heavily leverages multicast in Infiniband (if you do so research

[ceph-users] Ceph InfiniBand Cluster - Jewel - Performance

Hi Cephers, I've setup a production environment Ceph cluster with the Jewel release (10.1.0 (96ae8bd25f31862dbd5302f304ebf8bf1166aba6)) consisting of 3 MON Servers and 6 OSD Servers: 3x MON Servers: 2x Intel Xeon E5-2630v3@2.40Ghz 384GB RAM 2x 200G Intel DC3700 in RAID-1 for OS 1x InfiniBand

[ceph-users] ceph_assert_fail after upgrade from hammer to infernalis

All, I have 4 nodes each with 5 OSDs. I recently upgraded to infernalis via ceph-deploy. It went mostly ok but one of my nodes cannot mount any OSDs. When I look at the status of the service, I see: Apr 07 12:22:06 borg02 ceph-osd[3868]: 9: (ceph::__ceph_assert_fail(char const*, char const*,

Re: [ceph-users] adding cache tier in productive hammer environment

Hi Chris, thank you very much for your advice ! Currently, we have already running: osd_op_threads = 8 osd_max_backfills = 1 osd_recovery_max_active = 1 I will add your suggestions ! For sure there is a lot of space in tweaking the config, which is basically very basic. -- Mit freundlichen

Re: [ceph-users] Creating new user to mount cephfs

You are correct sir. I modified the user capabilities by adding the mds cap with the 'allow r' permission using the following command. *ceph auth caps client.cephfs mon 'allow r' mds 'allow r' osd 'allow rwx pool=cephfs_metadata,allow rwx pool=cephfs_data'* ​Thanks,​ *Nate Curry* On Thu, Apr

[ceph-users] ceph striping

Hi All, first I wanted to say hello, as I am new to the list. Secondly, we want to use ceph for VM disks and cephfs for our source code, image data, login directories, etc. I would like to know, if striping would improve performance if we would set something like the following and move away

Re: [ceph-users] Ceph performance expectations

Hi Sergio, yes I think you have also answered most of your own points – The main thing is to try and avoid excessive seeks on the HDDS, it would help to separate the journal and data but since HDDs are heavily dependent on seek and latency delays, it would not help to have multiple journals on

Re: [ceph-users] Ceph performance expectations

Thanks, Mark. Yes, we're using XFS and 3-replication, although we might switch to 2-replication since we're not too worried about resiliency. I did some test on single disks with dd, and am able to get about 152 MB/s writes and 191 MB/s reads from a single disk. I also run the same test on all

Re: [ceph-users] Ceph performance expectations

If you're at all concerned with performance and the budget is set, drop a storage node and replace some osds with SSDs in the other nodes. We use 32x 4TB + 4x SSDs storage nodes and have 192GB of memory; 128GB wasn't enough. If you try to do this setup without SSD journals then you are going

Re: [ceph-users] adding cache tier in productive hammer environment

Hi Oliver, Have you tried tuning some of the cluster settings to fix the IO errors in the VMs? We found some of the same issues when reweighting, backfilling and removing large snapshots. By minimizing the number of concurrent backfills and prioritizing client IO we can now add/remove OSDs

Re: [ceph-users] Ceph performance expectations

Thanks, Alan. Unfortunately, we currently don't have much flexibility in terms of the hardware we can get so adding SSDs might not be possible in the near future. What is the best practice here, allocating, for each OSD, one disk just for data and one disk just for thd journal? Since the journals

[ceph-users] v10.1.1 Jewel candidate released

Hi all, We've pushed 10.1.1, a second release candidate for Jewel. This fixes another round of bugs, and we are getting pretty close to a final release. There are a few known issues to watch out for: - Old CephFS clusters will mangle the layouts with this release; the fix was committed

Re: [ceph-users] Creating new user to mount cephfs

On Thu, Apr 7, 2016 at 10:02 AM, Nate Curry wrote: > I am trying to get cephfs mounted as a user other than client.admin in order > to avoid putting the admin keyring on the client host. When I try to > connect with that user and keyring I am getting the following error: > >

Re: [ceph-users] Ceph performance expectations

Hi Sergio On 04/07/2016 07:00 AM, Sergio A. de Carvalho Jr. wrote: Hi all, I've setup a testing/development Ceph cluster consisting of 5 Dell PowerEdge R720xd servers (256GB RAM, 2x 8-core Xeon E5-2650 @ 2.60 GHz, dual-port 10Gb Ethernet, 2x 900GB + 12x 4TB disks) running CentOS 6.5 and Ceph

[ceph-users] Creating new user to mount cephfs

I am trying to get cephfs mounted as a user other than client.admin in order to avoid putting the admin keyring on the client host. When I try to connect with that user and keyring I am getting the following error: *mount error 13 = Permission denied* I am following the exact instructions that

Re: [ceph-users] IO wait high on XFS

We use ceph on a 3 server Debian proxmox cluster that has 4 x4TB disks each in a shared Ceph cluster. The io wait is much too high (around 9%). The default file system is XFS and I found a suggestion that by adding this line osd mount options xfs = rw,noatime,inode64 into the global area of

[ceph-users] 800TB - Ceph Physical Architecture Proposal

I'd appreciate any critique on the following plan. Before I detail the plan, here are my current questions. --- 1) Am I under-powering the CPU on the proposed OSD node configuration? 2) Will latency of roughly 300 micro seconds introduced

[ceph-users] Ceph performance expectations

Hi all, I've setup a testing/development Ceph cluster consisting of 5 Dell PowerEdge R720xd servers (256GB RAM, 2x 8-core Xeon E5-2650 @ 2.60 GHz, dual-port 10Gb Ethernet, 2x 900GB + 12x 4TB disks) running CentOS 6.5 and Ceph Hammer 0.94.6. All servers use one 900GB disk for the root partition

Re: [ceph-users] adding cache tier in productive hammer environment

Hi Christian, thank you for answering, i appriciate your time ! --- Its used for RBD hosted vm's and also cephfs hosted vm's. Well the basic problem is/was that single OSD's simply go out/down. Ending in SATA BUS error's for the VM's which have to be rebooted, if they anyway can, because as

Re: [ceph-users] How can I monitor current ceph operation at cluster

Hi. I use this for my zabbix environment: https://github.com/thelan/ceph-zabbix/ It works really well for me. Regards David On 07.04.2016 11:20, Nick Fisk wrote: Hi. I have small question about monitoring performance at ceph cluster. We have cluster with 5 nodes and 8 drives on each

Re: [ceph-users] How can I monitor current ceph operation at cluster

> Hi. > I have small question about monitoring performance at ceph cluster. > > We have cluster with 5 nodes and 8 drives on each node, and 5 monitor on > every node. For monitoring cluster we use zabbix. It asked every node for 30 > second about current ceph operation and get different result

[ceph-users] How can I monitor current ceph operation at cluster

Hi. I have small question about monitoring performance at ceph cluster. We have cluster with 5 nodes and 8 drives on each node, and 5 monitor on every node. For monitoring cluster we use zabbix. It asked every node for 30 second about current ceph operation and get different result from every

Re: [ceph-users] Performance counters oddities, cache tier and otherwise

Hi Christian, > Hello, > > Ceph 0.94.5 for the record. > > As some may remember, I phased in a 2TB cache tier 5 weeks ago. > > About now it has reached about 60% usage, which is what I have the > cache_target_dirty_ratio set to. > > And for the last 3 days I could see some writes