Re: [ceph-users] reproducable rbd-nbd crashes

Hi Mike, Am 22.07.19 um 16:48 schrieb Mike Christie: > On 07/22/2019 06:00 AM, Marc Schöchlin wrote: >>> With older kernels no timeout would be set for each command by default, >>> so if you were not running that tool then you would not see the nbd >>> disconnect+io_errors+xfs issue. You would

Re: [ceph-users] Iscsi in the nautilus Dashboard

Hi Brent,As far as I know version 3.0 (which I assume is version 9) is the minimum required for the dashboard.I would go with the latest from Shaman; it won't break the actual iSCSI part of the setup, only maybe the iSCSI support in the dashboard. I haven't tried it myself, I'm still at version

Re: [ceph-users] reproducable rbd-nbd crashes

Hi Mike, Am 22.07.19 um 17:01 schrieb Mike Christie: > On 07/19/2019 02:42 AM, Marc Schöchlin wrote: >> We have ~500 heavy load rbd-nbd devices in our xen cluster (rbd-nbd 12.2.5, >> kernel 4.4.0+10, centos clone) and ~20 high load krbd devices (kernel >> 4.15.0-45, ubuntu 16.04) - we never

Re: [ceph-users] Iscsi in the nautilus Dashboard

I posted to the ceph-iscsi github but Dillaman noted that 3.2 was version 10. Which means that wouldn’t solve the issue with the version 9 requirement of the current 14.2.2 nautilus. Paul noted 3.1 is “pretty broken”, soo which version is version 9? Or should I hack/patch the dashboard in

[ceph-users] Mark CephFS inode as lost

We have a Luminous cluster which has filled up to 100% multiple times and this causes an inode to be left in a bad state. Doing anything to these files causes the client to hang which requires evicting the client and failing over the MDS. Usually we move the parent directory out of the way and

Re: [ceph-users] MON crashing when upgrading from Hammer to Luminous

First link should be this one http://docs.ceph.com/docs/jewel/install/upgrading-ceph/#upgrade-procedures rather than http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures

Re: [ceph-users] MON crashing when upgrading from Hammer to Luminous

Hi you’ll have to go from Hammer to Jewel then from Jewel to Luminous for a smooth upgrade. - http://docs.ceph.com/docs/mimic/install/upgrading-ceph/#upgrade-procedures -

Re: [ceph-users] Failed to get omap key when mirroring of image is enabled

On Mon, Jul 22, 2019 at 3:26 PM Ajitha Robert wrote: > > Thanks for your reply > > 1) In scenario 1, I didnt attempt to delete the cinder volume. Please find > the cinder volume log. > http://paste.openstack.org/show/754731/ It might be better to ping Cinder folks about that one. It doesn't

Re: [ceph-users] MON / MDS Storage Location

Hi, mon: /var/lib/ceph/mon/* mds: inside the cephfs_data and cephfs_metadata rados pools On 07/22/2019 09:25 PM, dhils...@performair.com wrote: > All; > > Where, in the filesystem, do MONs and MDSs store their data? > > Thank you, > > Dominic L. Hilsbos, MBA > Director - Information

[ceph-users] MON / MDS Storage Location

All; Where, in the filesystem, do MONs and MDSs store their data? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com ___ ceph-users mailing list

[ceph-users] MON crashing when upgrading from Hammer to Luminous

Dear Everyone, First of all, guys, seriously, Thank you for Ceph. now to the problem, upgrading ceph from 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403) to 12.2.12-218-g9fd889f (9fd889fe09c652512ca78854702d5ad9bf3059bb), ceph-mon seems unable to upgrade it's database, problem is gone if i

Re: [ceph-users] Failed to get omap key when mirroring of image is enabled

On Mon, Jul 22, 2019 at 10:49 AM Ajitha Robert wrote: > > No error log in rbd-mirroring except some connection timeout came once, > Scenario 1: > when I create a bootable volume of 100 GB with a glance image.Image get > downloaded and from cinder, volume log throws with "volume is busy

Re: [ceph-users] which tool to use for benchmarking rgw s3, yscb or cosbench

I have had good luck with YCSB as an initial assessment of different storage systems. Typically I'll use this first when I am playing with a new system, but I like to switch to the more native tools (rados bench, cassandra-stress, etc etc) as soon as I am more comfortable. And I can definitely

Re: [ceph-users] reproducable rbd-nbd crashes

On 07/19/2019 02:42 AM, Marc Schöchlin wrote: > We have ~500 heavy load rbd-nbd devices in our xen cluster (rbd-nbd 12.2.5, > kernel 4.4.0+10, centos clone) and ~20 high load krbd devices (kernel > 4.15.0-45, ubuntu 16.04) - we never experienced problems like this. For this setup, do you have

Re: [ceph-users] reproducable rbd-nbd crashes

On 07/22/2019 06:00 AM, Marc Schöchlin wrote: >> With older kernels no timeout would be set for each command by default, >> so if you were not running that tool then you would not see the nbd >> disconnect+io_errors+xfs issue. You would just see slow IOs. >> >> With newer kernels, like 4.15,

Re: [ceph-users] Iscsi in the nautilus Dashboard

Hi all,That was not the most recent. This is it (3.2.4): https://2.chacra.ceph.com/r/ceph-iscsi/master/8a3967698257e1b49a9d554847b84418c15da902/centos/7/flavors/default/KasparOp 22 juli 2019 om 14:01 schreef Kaspar Bosma : Hi Brent,You may want to have a look at the repos at shaman.ceph.com.The

Re: [ceph-users] Failed to get omap key when mirroring of image is enabled

On Sun, Jul 21, 2019 at 8:25 PM Ajitha Robert wrote: > > I have a rbd mirroring setup with primary and secondary clusters as peers > and I have a pool enabled image mode.., In this i created a rbd image , > enabled with journaling. > > But whenever i enable mirroring on the image, I m getting

Re: [ceph-users] Iscsi in the nautilus Dashboard

Hi Brent,You may want to have a look at the repos at shaman.ceph.com.The latest (3.2.2) packaged version of Ceph iSCSI is located here:https://4.chacra.ceph.com/r/ceph-iscsi/master/ff5e6873c43ab6828d3f7264526100b95a7e3954/centos/7/flavors/default/noarch/You can also find related package repos for

Re: [ceph-users] New best practices for osds???

OK, I meant "it may help performance" :) the main point is that we had at least one case of data loss due to some Adaptec controller in RAID0 mode discussed recently in our ceph chat... -- With best regards, Vitaliy Filippov ___ ceph-users

Re: [ceph-users] reproducable rbd-nbd crashes

Hello Mike, i attached inline comments. Am 19.07.19 um 22:20 schrieb Mike Christie: > >> We have ~500 heavy load rbd-nbd devices in our xen cluster (rbd-nbd 12.2.5, >> kernel 4.4.0+10, centos clone) and ~20 high load krbd devices (kernel >> 4.15.0-45, ubuntu 16.04) - we never experienced

Re: [ceph-users] New best practices for osds???

On Mon, Jul 22, 2019 at 12:52 PM Vitaliy Filippov wrote: > It helps performance, Not necessarily, I've seen several setups where disabling the cache increases performance Paul > but it can also lead to data loss if the raid > controller is crap (not flushing data correctly) > > -- > With

Re: [ceph-users] Iscsi in the nautilus Dashboard

Version 9 is the fqdn stuff which was introduced in 3.1. Use 3.2 as 3.1 is pretty broken. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Jul 22, 2019 at 3:24

Re: [ceph-users] New best practices for osds???

It helps performance, but it can also lead to data loss if the raid controller is crap (not flushing data correctly) -- With best regards, Vitaliy Filippov ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Future of Filestore?

Linear reads, `hdparm -t /dev/vda`. Check if you have `cache=writeback` enabled in your VM options. If it's enabled but you still get 5mb/s then try to benchmark your cluster with fio -ioengine=rbd from outside a VM. Like fio -ioengine=rbd -name=test -bs=4M -iodepth=16 -rw=read

Re: [ceph-users] Future of Filestore?

On 22/7/19 7:39 pm, Vitaliy Filippov wrote: > 5MB/s in what mode? Linear reads, `hdparm -t /dev/vda`. > For linear writes, that definitely means some kind of misconfiguration. > For random writes... there's a handbrake in Bluestore which makes random > writes run at half speed in HDD-only setups

Re: [ceph-users] Future of Filestore?

On 22/7/19 7:13 pm, Marc Roos wrote: > > >> Reverting back to filestore is quite a lot of work and time again. > >> Maybe see first if with some tuning of the vms you can get better > results? > > > >None of the VMs are particularly disk-intensive. There's two users > accessing the system

Re: [ceph-users] which tool to use for benchmarking rgw s3, yscb or cosbench

On 2019-07-21T23:51:41, Wei Zhao wrote: > Hi: > I found cosbench is a very convenient tool for benchmaring rgw. But > when I read papers , I found YCSB tool, > https://github.com/brianfrankcooper/YCSB/tree/master/s3 . It seems > that this is used for test cloud service , and seems a right

Re: [ceph-users] Future of Filestore?

5MB/s in what mode? For linear writes, that definitely means some kind of misconfiguration. For random writes... there's a handbrake in Bluestore which makes random writes run at half speed in HDD-only setups :) https://github.com/ceph/ceph/pull/26909 And if you push that handbrake down

Re: [ceph-users] Future of Filestore?

>> Reverting back to filestore is quite a lot of work and time again. >> Maybe see first if with some tuning of the vms you can get better results? > >None of the VMs are particularly disk-intensive. There's two users accessing the system over a WiFi network for email, and some HTTP/SMTP

Re: [ceph-users] how to debug slow requests

Hello, Your issue look like mine - I had op stuck with the same status: check "Random slow requests without any load" on this month list archive. Bests, Le 7/20/19 à 6:06 PM, Wei Zhao a écrit : Hi ceph users: I was doing write benchmark, and found some io will be blocked for a very long