[ceph-users] CephFS : Kernel/Fuse technical differences

Hi everyone, We successfully use Ceph here for several years now, and since recently, CephFS. From the same CephFS server, I notice a big difference between a fuse mount and a kernel mount (10 times faster for kernel mount). It makes sense to me (an additional fuse library versus a direct ac

Re: [ceph-users] getting pg inconsistent periodly

Hi all, For information, I updated my Luminous cluster to the latest version 12.2.12 two weeks ago and, since then, I no longer encounter any problems of inconsistent pgs :) Regards, rv Le 03/05/2019 à 11:54, Hervé Ballans a écrit : Le 24/04/2019 à 10:06, Janne Johansson a écrit : Den ons

Re: [ceph-users] getting pg inconsistent periodly

Le 24/04/2019 à 10:06, Janne Johansson a écrit : Den ons 24 apr. 2019 kl 08:46 skrev Zhenshi Zhou >: Hi, I'm running a cluster for a period of time. I find the cluster usually run into unhealthy state recently. With 'ceph health detail', one or t

[ceph-users] Ceph cluster available to clients with 2 different VLANs ?

Hi all, I have a Ceph cluster on Luminous 12.2.10 with 3 mon and 6 osd servers. My current network settings is a separated public and cluster (private IP) network. I would like my cluster available to clients on another VLAN than the default one (which is the public network on ceph.conf) Is

Re: [ceph-users] Possible data damage: 1 pg inconsistent

Hi Frank, I encounter exactly the same issue with the same disks than yours. Every day, after a batch of deep scrubbing operation, ther are generally between 1 and 3 inconsistent pgs, and that, on different OSDs. It could confirm a problem on these disks, but : - it concerns only the pgs of

Re: [ceph-users] Luminous or Mimic client on Debian Testing (Buster)

Hi, OK I hadn't thought that the Ubuntu packages were very close to Debian anymore ! The solution given by Martin works and my issue is solved, but I keep this option as an alternative... Thanks, rv Le 13/11/2018 à 18:30, vita...@yourcmc.ru a écrit : Use Ubuntu bionic repository, Mimic inst

Re: [ceph-users] Luminous or Mimic client on Debian Testing (Buster)

/MartinVerges croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io YouTube: https://goo.gl/PGE1Bx 2018-11-13 16:42 GMT+01:00 Hervé Ballans : Hi, On my CephFS production cluster (Luminous 12.2.8), I

[ceph-users] Luminous or Mimic client on Debian Testing (Buster)

Hi, On my CephFS production cluster (Luminous 12.2.8), I would like to add a CephFS client from a server installed with Debian Buster (Testing release). But, the default proposed Ceph packages in this release are still Jewel : # cat /etc/debian_version buster/sid # apt search ceph-common Sor

[ceph-users] Mimic and Debian 9

Hi, I just wanted to know if we had a chance soon to install Mimic on Debian 9 ?! ;) I know there is a problem with the required version gcc (compatible with c++17) that is not yet backported on current stable version of Debian, but is there any news on this side ? Regards, Hervé

[ceph-users] mds_cache_memory_limit value

Hi all, I have just configured a new value for 'mds_cache_memory_limit'. The output message tells "not observed, change may require restart". So I'm not really sure, has the new value been taken into account directly or do I have to restart the mds daemons on each MDS node ? $ sudo ceph tell

Re: [ceph-users] NVMe SSD not assigned "nvme" device class

Hi, You can easily configure it manually, e.g. : $ sudo ceph osd crush rm-device-class osd.xx $ sudo ceph osd crush set-device-class nvme osd.xx Indeed, it may be useful when you want to create custom rules on this type of device. Hervé Le 01/10/2018 à 23:25, Vladimir Brik a écrit : Hello,

Re: [ceph-users] Bluestore DB showing as ssd

Hi, By testing the command on my side, it gives me the right information (modulo the fact that the disk is a nvme and not ssd) : # ceph osd metadata 1 |grep bluefs_db     "bluefs_db_access_mode": "blk",     "bluefs_db_block_size": "4096",     "bluefs_db_dev": "259:3",     "bluefs_db_dev_node":

Re: [ceph-users] Proxmox/ceph upgrade and addition of a new node/OSDs

od luck with your very next migration ! Hervé Le 13/09/2018 à 22:04, mj a écrit : Hi Hervé, No answer from me, but just to say that I have exactly the same upgrade path ahead of me. :-) Please report here any tips, trics, or things you encountered doing the upgrades. It could potentially

[ceph-users] Proxmox/ceph upgrade and addition of a new node/OSDs

Dear list, I am currently in the process of upgrading Proxmox 4/Jewel to Proxmox5/Luminous. I also have a new node to add to my Proxmox cluster. What I plan to do is the following (from https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous): * upgrade Jewel to Luminous * let the "ceph osd c

Re: [ceph-users] Get supported features of all connected clients

$ sudo ceph tell mds.0 client ls ? Le 11/09/2018 à 13:00, Tobias Florek a écrit : Hi! I have a cluster serving RBDs and CephFS that has a big number of clients I don't control. I want to know what feature flags I can safely set without locking out clients. Is there a command analogous to `ce

[ceph-users] CephFS : fuse client vs kernel driver

Hi all, I just finished setting up a new Ceph cluster (Luminous 12.2.7, 3xMON nodes and 6xOSDs nodes, BlueStore OSD on sata hdd with WAL/DB on separated NVMe devices, 2x10 Gbs network per node, 3 replicas by pool) I created a CephFS pool : data pool uses hdd OSDs and metadata pool uses dedic

Re: [ceph-users] Shared WAL/DB device partition for multiple OSDs?

Le 23/08/2018 à 18:44, Alfredo Deza a écrit : ceph-volume-systemd.log (extract) [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: lvm-6-ba351d69-5c48-418e-a377-4034f503af93 [2018-08-20 11:26:26,386][systemd][INFO ] raw systemd input received: lvm-3-9380cd27-c0fe-4ede-9ed3-d0

Re: [ceph-users] Shared WAL/DB device partition for multiple OSDs?

Le 23/08/2018 à 16:13, Alfredo Deza a écrit : What you mean is that, at this stage, I must directly declare the UUID paths in value of --block.db (i.e. replace /dev/nvme0n1p1 with its PARTUUID), that is ? No, this all looks correct. How does the ceph-volume.log and ceph-volume-systemd.log look w

Re: [ceph-users] Shared WAL/DB device partition for multiple OSDs?

Le 23/08/2018 à 15:20, Alfredo Deza a écrit : Thanks Alfredo for your reply. I'm using the very last version of Luminous (12.2.7) and ceph-deploy (2.0.1). I have no problem in creating my OSD, that's work perfectly. My issue only concerns the problem of the mount names of the NVMe partitions whic

Re: [ceph-users] Shared WAL/DB device partition for multiple OSDs?

Le 23/08/2018 à 12:51, Alfredo Deza a écrit : On Thu, Aug 23, 2018 at 5:42 AM, Hervé Ballans wrote: Hello all, I would like to continue a thread that dates back to last May (sorry if this is not a good practice ?..) Thanks David for your usefil tips on this thread. In my side, I created my

Re: [ceph-users] Shared WAL/DB device partition for multiple OSDs?

Hello all, I would like to continue a thread that dates back to last May (sorry if this is not a good practice ?..) Thanks David for your usefil tips on this thread. In my side, I created my OSDs with ceph-deploy (in place of ceph-volume) [1], but this is exactly the same context as this ment

Re: [ceph-users] Bluestore : how to check where the WAL is stored ?

symlink is good enough symptom that DB and WAL are merged . But you can also inspect OSD startup log or check bluefs perf counters after some load - corresponding WAL counters (total/used) should be zero. Thanks, Igor On 8/16/2018 4:55 PM, Hervé Ballans wrote: Hi all, I'm setting u

[ceph-users] Bluestore : how to check where the WAL is stored ?

Hi all, I'm setting up my Ceph cluster (last release of Luminous) and I'm currently configuring OSD with WAL and DB on NVMe disk. OSD data are on a SATA disk and Both WAL and DB are on the same partition of the NVMe disk. After creating partitions on the NVMe (block partitions, without fil

Re: [ceph-users] IO to OSD with librados

Le 19/06/2018 à 09:02, Dan van der Ster a écrit : The storage arrays are Nexsan E60 arrays having two active-active redundant controllers, 60 3 TB disk drives. The disk drives are organized into six 8+2 Raid 6 LUNs of 24 TB each. This is not the ideal Ceph hardware. Ceph is designed to use disk

Re: [ceph-users] Crush maps : split the root in two parts on an OSD node with same disks ?

à 16:52, Janne Johansson a écrit : Den tis 12 juni 2018 kl 15:06 skrev Hervé Ballans mailto:herve.ball...@ias.u-psud.fr>>: Hi all, I have a cluster with 6 OSD nodes, each has 20 disks, all of the 120 disks are strictly identical (model and size). (The cluster is also composed

[ceph-users] Crush maps : split the root in two parts on an OSD node with same disks ?

Hi all, I have a cluster with 6 OSD nodes, each has 20 disks, all of the 120 disks are strictly identical (model and size). (The cluster is also composed of 3 MON servers on 3 other machines) For design reason, I would like to separate my cluster storage into 2 pools of 60 disks. My idea is

Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

Hi, Just a little question regarding this operation : [root@osdhost osd]# ceph-volume lvm prepare --bluestore --data /dev/sdc --block.wal /dev/sda2 --block.db /dev/sda1 On a previous post, I understood that if both wal and db are stored on the same separate device, then we could use a single

Re: [ceph-users] Separate BlueStore WAL/DB : best scenario ?

Le 21/03/2018 à 11:48, Ronny Aasen a écrit : On 21. mars 2018 11:27, Hervé Ballans wrote: Hi all, I have a question regarding a possible scenario to put both wal and db in a separate SSD device for an OSD node composed by 22 OSDs (HDD SAS 10k 1,8 To). I'm thinking of 2 options (at

[ceph-users] Separate BlueStore WAL/DB : best scenario ?

Hi all, I have a question regarding a possible scenario to put both wal and db in a separate SSD device for an OSD node composed by 22 OSDs (HDD SAS 10k 1,8 To). I'm thinking of 2 options (at about the same price) : - add 2 SSD SAS Write Intensive (10DWPD) - or add a unique SSD NVMe 800 Go

Re: [ceph-users] Case where a separate Bluestore WAL/DB device crashes...

Thu, Mar 01, 2018 at 04:57:59PM +0100, Hervé Ballans wrote: :Can we find recent benchmarks on this performance issue related to the :location of WAL/DBs ? I don't have benchmarks but I have some anecdotes. we previously had 4T NLSAS (7.2k) filestore data drives with journals on SSD

Re: [ceph-users] Case where a separate Bluestore WAL/DB device crashes...

SSDs. If the SSD fails, all osds using it aren't lost and need to be removed from the cluster and recreated with a new drive. You can never guarantee data integrity on bluestore or filestore if any media of the osd fails completely. On Thu, Mar 1, 2018, 10:24 AM Hervé Ba

[ceph-users] Case where a separate Bluestore WAL/DB device crashes...

Hello, With Bluestore, I have a couple of questions regarding the case of separate partitions for block.wal and block.db. Let's take the case of an OSD node that contains several OSDs (HDDs) and also contains one SSD drive for storing WAL partitions and an another one for storing DB partitio

[ceph-users] Hardware considerations on setting up a new Luminous Ceph cluster

Hi all, I'm managing since 3 years now an high-availability Ceph clusters for our virtualization infrastructure (Proxmox VE). We use Jewel with rbd. It works perfectly well and meets our data integrity and performance needs. In parallel, we want to add a new Ceph cluster for data storage with

Re: [ceph-users] Very slow start of osds after reboot

Hi Piotr, Just to verify one point, how are connected your disks (physically), in a NON-RAID or RAID0 mode ? rv Le 31/08/2017 à 16:24, Piotr Dzionek a écrit : For a last 3 weeks I have been running latest LTS Luminous Ceph release on CentOS7. It started with 4th RC and now I have Stable Rele

Re: [ceph-users] BlueStore WAL or DB devices on a distant SSD ?

ock.db /dev/sdd) ? I ask this because the online documentation on "BlueStore Config Reference" states that adding these devices are useful only if the device used is faster than the primary device ? Thanks again, Hervé On Wed, Aug 16, 2017, 6:04 AM Hervé Ballans mailto:herve.ball

[ceph-users] BlueStore WAL or DB devices on a distant SSD ?

Hi, We are currently running two Proxmox/ceph clusters that work perfectly (since 2014) and thank to this succesful experience, we plan to install a new Ceph cluster for storage of our computing cluster. Until now, we only used RBD (virtualization context) but now we want to use CephFS for t