Re: [ceph-users] RBD snapshots cause disproportionate performance degradation

Yes, it's a expected case. Actually if you use Hammer, you can enable filestore_fiemap to use sparse copy which especially useful for rbd snapshot copy. But keep in mind some old kernel are *broken* in fiemap. CentOS 7 is only the distro I verfied fine to this feature. On Wed, Nov 18, 2015 at 12:

[ceph-users] RBD snapshots cause disproportionate performance degradation

Hi, We’ve been running an all-SSD Ceph cluster for a few months now and generally are very happy with it. However, we’ve noticed that if we create a snapshot of an RBD device, then writing to the RBD goes massively slower than before we took the snapshot. Similarly, we get poor performance if

Re: [ceph-users] about PG_Number

By the way, here is a useful tool to calculate pg. http://ceph.com/pgcalc/ Best wishes, Mika 2015-11-18 11:46 GMT+08:00 Vickie ch : > Hi wah peng, > Hope you don't mind. Just for reference. > A extreme case. If your ceph cluster have 3 osd disks on different osd > server. > Set pg number is

Re: [ceph-users] about PG_Number

Hi wah peng, Hope you don't mind. Just for reference. A extreme case. If your ceph cluster have 3 osd disks on different osd server. Set pg number is 10240.(Just example) That's mean all these pg will create on 3 disks. Lost one OSD also means a lot of pg lost too. It may bring some trouble for r

Re: [ceph-users] can not create rbd image

Hi , Looks like your cluster have warning message of "2 near full osd(s)". ​Maybe ​try to extend osds first？ ​ Best wishes, Mika 2015-11-12 23:05 GMT+08:00 min fang : > Hi cepher, I tried to use the following command to create a img, but > unfortunately, the command hung for a long time un

Re: [ceph-users] rbd create => seg fault

On 13/11/15 21:21, Ilya Dryomov wrote: > On Fri, Nov 13, 2015 at 5:25 AM, Mark Kirkwood > wrote: >> When you do: >> >> $ rbd create >> > > This seg fault is not in any way tied to the kernel version. > > The kernel isn't involved in creating rbd images or using them through > librbd (or anything

[ceph-users] OSD Recovery Delay Start

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 We are having a lot of trouble with the SSD OSDs for our cache tier when they reboot. It causes massive blocked I/O when booting the OSD and the entire cluster I/O nearly stalls even when the OSD is only down for 60 seconds. I have noticed that when

Re: [ceph-users] rbd create => seg fault

This was resolved when I found more versioned libs that were also out of sync in the Python neighborhood: /usr/local/lib/python2.7/site-packages This exercise motivated me to install multiple versions of Ceph in versioned app directories, such as /usr/local/ceph-hammer and /usr/local/ceph-infernal

Re: [ceph-users] ms crc header: seeking info?

Thank you very much, Haomai, and others, too! ms_crc_header was set consistently across nodes at all times. :) Root cause of my problems was the mismatched libraries being picked up when shared libs load. Plain & simple. Next time, I should not allow errant input from other "cooks in the kitchen"

[ceph-users] SSD Caching Mode Question

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 We are inserting an SSD tier into our very busy cluster and I have a question regrading writeback and forward modes. Write back is the "normal" mode for RBD with VMs. When we put the tier in writeback mode we see objects are being promoted and once

Re: [ceph-users] SL6/Centos6 rebuild question

Dear All... I was able to build el6 rpms for infernalis but they requires GCC 4.8.3. If you are interested in the details on how to do it, let me know. There were two major issues: - The build complains about a parsing configuration error on ceph-9.2.0/selinux/ceph.te /usr/bin/checkmod

Re: [ceph-users] rados_aio_cancel

On Monday, November 16, 2015, min fang wrote: > Is this function used in detach rx buffer, and complete IO back to the > caller? From the code, I think this function will not interact with OSD or > MON side, which means, we just cancel IO from client side. Am I right? > > Thanks. > Right. If th

Re: [ceph-users] SSD pool and SATA pool

I prefer to use the crush hook location functionality to call a script like this so OSD's are dynamically placed in the correct crush root on startup https://gist.github.com/wido/5d26d88366e28e25e23d From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Michael Kuriger

Re: [ceph-users] SSD pool and SATA pool

Many thanks! [yp] Michael Kuriger Sr. Unix Systems Engineer • mk7...@yp.com |• 818-649-7235 From: Sean Redmond mailto:sean.redmo...@gmail.com>> Date: Tuesday, November 17, 2015 at 2:00 PM To: Nikola Ciprich mailto:nikola.cipr...@linuxbox.cz>> Cc: Michael Kuriger mailto

Re: [ceph-users] SSD pool and SATA pool

Hi, The below should help you: http://www.sebastien-han.fr/blog/2014/08/25/ceph-mix-sata-and-ssd-within-the-same-box/ Thanks On Tue, Nov 17, 2015 at 9:58 PM, Nikola Ciprich wrote: > I'm not an ceph expert, but I needed to use > > osd crush update on start = false > > in [osd] config section..

Re: [ceph-users] SSD pool and SATA pool

I'm not an ceph expert, but I needed to use osd crush update on start = false in [osd] config section.. BR nik On Tue, Nov 17, 2015 at 08:53:37PM +, Michael Kuriger wrote: > Hey everybody, > I have 10 servers, each with 2 SSD drives, and 8 SATA drives. Is it possible > to create 2 pools

[ceph-users] SSD pool and SATA pool

Hey everybody, I have 10 servers, each with 2 SSD drives, and 8 SATA drives. Is it possible to create 2 pools, one made up of SSD and one made up of SATA? I tried manually editing the crush map to do it, but the configuration doesn’t seem to persist reboots. Any help would be very appreciated

[ceph-users] Performance output con Ceph IB with fio examples

Hi cephers, Is there anyone out there using Ceph (any version) with Infiniband FDR topology network (both public and cluster), that could share some performance results? To be more specific, running something like this on a RBD volume mapped to a IB host: # fio --rw=randread --bs=4m --numjobs=4 -

Re: [ceph-users] Disaster recovery of monitor

Hi guys ... Thanks a lot for your support. I discovered what happened. I had 2 monitors, osnode01 and osnode02. I tried do add a 3rd by using ceph-deploy. ceph-deploy was using a key different from the one used by my monitor cluster. So, I added osnode08 to the monitor cluster and it did not be

Re: [ceph-users] Problem with infernalis el7 package

You're right stijn, I apologize that we did not bump the release number in this case. That would have been the correct thing to do, but our build system simply isn't set up to do that easily, and we wanted to get a fix out as soon as possible. - Ken On Wed, Nov 11, 2015 at 1:34 AM, Stijn De Weird

Re: [ceph-users] can't stop ceph

The version of the documentation you were browsing was for "argonaut", which is very old, and predates Upstart integration. Here's the version of the docs for firefly (0.80.z), the version that you're using on Ubuntu: http://docs.ceph.com/docs/firefly/rados/operations/operating/ This version has

[ceph-users] Bcache and Ceph Question

Hi all, Is there any way to use bcache in an already configured Ceph cluster? I've both OSD and Journals inside the same OSD daemon, and I want to try bcache in front of the OSD daemon and also move in the bcache device the journal, so for example I got: /dev/sdc --> SSD disk /dev/sdc1 --> 1st

Re: [ceph-users] Disaster recovery of monitor

Now I tried to inject the latest map I had. Also, I created a second monitor on osnode02, like I had before, using the same map. I started both monitors ... Logs from osnode01 show my content ... and then it started to show lines like 2015-11-17 10:56:26.515069 7fc73af67700 0 mon.osnode01@0(prob

Re: [ceph-users] Disaster recovery of monitor

On 11/17/2015 12:27 PM, Jose Tavares wrote: > My concern is about this log line > > 2015-11-17 10:11:16.143864 7f81e14aa700 0 > mon.osnode01@0(probing).data_health(0) update_stats avail 19% total 220 > GB, used 178 GB, avail 43194 MB > > I use to have 7TB of available space with 263G of con

Re: [ceph-users] Disaster recovery of monitor

On Tue, Nov 17, 2015 at 7:27 AM, Joao Eduardo Luis wrote: > On 11/17/2015 03:56 AM, Jose Tavares wrote: > > The problem is that I think I don't have any good monitor anymore. > > How do I know if the map I am trying is ok? > > > > I also saw in the logs that the primary mon was trying to contact

Re: [ceph-users] Disaster recovery of monitor

On Tue, Nov 17, 2015 at 6:32 AM, Wido den Hollander wrote: > On 11/17/2015 04:56 AM, Jose Tavares wrote: > > The problem is that I think I don't have any good monitor anymore. > > How do I know if the map I am trying is ok? > > > > How do you mean there is no good monitor? Did you encounter a dis

Re: [ceph-users] Cannot mount CephFS after irreversible OSD lost

On Tue, Nov 17, 2015 at 12:17 PM, Mykola Dvornik wrote: > Dear John, > > Thanks for such a prompt reply! > > Seems like something happens on the mon side, since there are no > mount-specific requests logged on the mds side (see below). > FYI, some hours ago I've disabled auth completely, but it di

Re: [ceph-users] Cannot mount CephFS after irreversible OSD lost

Dear John, Thanks for such a prompt reply! Seems like something happens on the mon side, since there are no mount-specific requests logged on the mds side (see below). FYI, some hours ago I've disabled auth completely, but it didn't help. The serialized metadata pool is 9.7G. I can try to compre

Re: [ceph-users] restart all nodes

On 17-11-15 11:07, Patrik Plank wrote: > Hi, > > > maybe a trivial question :-|| > > I have to shut down all my ceph nodes. > > What's the best way to do this. > > Can I just shut down all nodes or should i > > first shut down the ceph process? > First, set the noout flag in the monitors:

Re: [ceph-users] Cannot mount CephFS after irreversible OSD lost

On Tue, Nov 17, 2015 at 10:08 AM, Mykola Dvornik wrote: > However when I've brought the mds back online the CephFS cannot be mounted > anymore complaining on the client side 'mount error 5 = Input/output error'. > Since mds was running just fine without any suspicious messages in its log, > I've d

[ceph-users] restart all nodes

Hi, maybe a trivial question :-|| I have to shut down all my ceph nodes. What's the best way to do this. Can I just shut down all nodes or should i first shut down the ceph process? best regards ___ ceph-users mailing list ceph-users@lists.cep

[ceph-users] Cannot mount CephFS after irreversible OSD lost

Dear ceph experts, I've built and administrating 12 OSD ceph cluster (spanning over 3 nodes) with replication count of 2. The ceph version is ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) The cluster hosts two pools (data and metadata) that are exported over CephFS. At some

Re: [ceph-users] Ceph Openstack deployment

Hi, This is an old one. never used (I just delete it)...I have checked that ntp is running fine. Binary Host Zone StatusState Updated At cinder-volume cloudvolume01@iscsi-cloudvolume01nova enabled:-) 2015-11-17 09:

Re: [ceph-users] Disaster recovery of monitor

On 11/17/2015 03:56 AM, Jose Tavares wrote: > The problem is that I think I don't have any good monitor anymore. > How do I know if the map I am trying is ok? > > I also saw in the logs that the primary mon was trying to contact a > removed mon at IP .112 .. So, I added .112 again ... and it didn'

[ceph-users] next ceph breizh camp

Hi, The next ceph breizh camp will take place the 26th November at University of Nantes and begin at 10.00 AM, at: IGARUN (institut de géographie), Salle réunion (991/992), 1er étage Chemin de la Censive du Tertre, sur le campus Tertre de Nantes. You can enroll yourself at : http://doodle.com

Re: [ceph-users] Disaster recovery of monitor

On 11/17/2015 04:56 AM, Jose Tavares wrote: > The problem is that I think I don't have any good monitor anymore. > How do I know if the map I am trying is ok? > How do you mean there is no good monitor? Did you encounter a disk failure or something? > I also saw in the logs that the primary mon