Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Le lundi 23 juillet 2018 à 11:07 +0700, Konstantin Shalygin a écrit : > > I even have no fancy kernel or device, just real standard Debian. > > The > > uptime was 6 days since the upgrade from 12.2.6... > > Nicolas, you should upgrade your 12.2.6 to 12.2.7 due bugs in this > release. That was don

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Ceph doesn't shut down systems as in kill or reboot the box if that's what you're saying? On Mon, Jul 23, 2018 at 5:04 PM, Nicolas Huillard wrote: > Le lundi 23 juillet 2018 à 11:07 +0700, Konstantin Shalygin a écrit : >> > I even have no fancy kernel or device, just real standard Debian. >> > Th

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Do you have any hardware watchdog running in the system? A watchdog could trigger a powerdown if it meets some value. Any event logs from the chassis itself? Kind regards, Caspar 2018-07-21 10:31 GMT+02:00 Nicolas Huillard : > Hi all, > > One of my server silently shutdown last night, with no e

[ceph-users] Converting to multisite

I already have a set of default.rgw.* pools. They are in use. I want to convert to multisite. The tutorials show to create new pools (zone.rgw.*). Do I have to destroy my old pools and lose all data, in order to convert to multisite? ___ ceph-users m

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Le lundi 23 juillet 2018 à 18:23 +1000, Brad Hubbard a écrit : > Ceph doesn't shut down systems as in kill or reboot the box if that's > what you're saying? That's the first part of what I was saying, yes. I was pretty sure Ceph doesn't reboot/shutdown/reset, but now it's 100% sure, thanks. Maybe

[ceph-users] Checksum verification of BlueStore superblock using Python

Hi, I try to use Python (3) to verify the checksum at the end of a BlueStore superblock, but I cannot figure it out how to do it. In my test scenario, the superblock is 0x158 bytes long (starting with "bluestore block device\n\n"), then 4 bytes of CRC32 follow. In my case the checksum is 0xb75

Re: [ceph-users] "CPU CATERR Fault" Was: Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Le lundi 23 juillet 2018 à 10:28 +0200, Caspar Smit a écrit : > Do you have any hardware watchdog running in the system? A watchdog > could > trigger a powerdown if it meets some value. Any event logs from the > chassis > itself? Nice suggestions ;-) I see some [watchdog/N] and one [watchdogd] ke

[ceph-users] ceph bluestore data cache on osd

Hi Team, We need a mechanism to have some data cache on OSD build on bluestore . Is there an option available to enable data cache? With default configurations , OSD logs state that data cache is disabled by default, bluestore(/var/lib/ceph/osd/ceph-66) _set_cache_sizes cache_size 1073741824

Re: [ceph-users] [Ceph-deploy] Cluster Name

How very timely, I am facing the exact same issue. Kind regards, Glen Baars -Original Message- From: ceph-users On Behalf Of Thode Jocelyn Sent: Monday, 23 July 2018 1:42 PM To: Vasu Kulkarni Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users] [Ceph-deploy] Cluster Name Hi, Yes my

Re: [ceph-users] Why lvm is recommended method for bleustore

On 22-7-2018 15:51, Satish Patel wrote: I read that post and that's why I open this thread for few more questions and clearence, When you said OSD doesn't come up what actually that means? After reboot of node or after service restart or installation of new disk? You said we are using manual

Re: [ceph-users] Why lvm is recommended method for bleustore

Le dimanche 22 juillet 2018 à 09:51 -0400, Satish Patel a écrit : > I read that post and that's why I open this thread for few more > questions and clearence, > > When you said OSD doesn't come up what actually that means?  After > reboot of node or after service restart or installation of new dis

Re: [ceph-users] CFP: linux.conf.au 2019 (Christchurch, New Zealand)

Just a friendly reminder, the linux.conf.au 2019 CFP closes next Monday, July 30. Don't miss out! :-) On 07/02/2018 04:10 PM, Tim Serong wrote: > Hi All, > > As happened last year, I forwarded the LCA CFP to ceph-users and > ceph-devel, but it didn't make it to ceph-devel due to some alleged sp

Re: [ceph-users] ceph bluestore data cache on osd

Firstly I'd suggest to inspect bluestore performance counters before and after adjusting cache parameters (and after running the same test suite). Namely: "bluestore_buffer_bytes" "bluestore_buffer_hit_bytes" "bluestore_buffer_miss_bytes" Is hit ratio (bluestore_buffer_hit_bytes) much diffe

Re: [ceph-users] Error bluestore doesn't support lvm

Hi, On 21/07/18 04:24, Satish Patel wrote: > I am using openstack-ansible with ceph-ansible to deploy my Ceph > custer and here is my config in yml file You might like to know that there's a dedicated (if quiet!) list for ceph-ansible - ceph-ansi...@lists.ceph.com Regards, Matthew -- The We

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Hi, > One of my server silently shutdown last night, with no explanation > whatsoever in any logs. According to the existing logs, the shutdown We have seen similar things with our SuperMicro servers; our current best theory is that it's related to CPU power management. Disabling it in BIOS seems

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Am 23.07.2018 um 11:18 schrieb Nicolas Huillard: > Le lundi 23 juillet 2018 à 18:23 +1000, Brad Hubbard a écrit : >> Ceph doesn't shut down systems as in kill or reboot the box if that's >> what you're saying? > > That's the first part of what I was saying, yes. I was pretty sure Ceph > doesn't re

Re: [ceph-users] "CPU CATERR Fault" Was: Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Am 23.07.2018 um 11:39 schrieb Nicolas Huillard: > Le lundi 23 juillet 2018 à 10:28 +0200, Caspar Smit a écrit : >> Do you have any hardware watchdog running in the system? A watchdog >> could >> trigger a powerdown if it meets some value. Any event logs from the >> chassis >> itself? > > Nice sug

Re: [ceph-users] Mimic 13.2.1 release date

Any news on this yet? 13.2.1 would be very welcome! :-) Wido On 07/09/2018 05:11 PM, Wido den Hollander wrote: > Hi, > > Is there a release date for Mimic 13.2.1 yet? > > There are a few issues which currently make deploying with Mimic 13.2.0 > a bit difficult, for example: > > - https://track

Re: [ceph-users] Add Partitions to Ceph Cluster

Hi Dimitri, what is the output of - ceph osd tree? Perhaps you have a initials crush weight of 0 and in this case there wouldnt be any change in the PGs till you change the weight. - Mehmet Am 2018-07-10 11:58, schrieb Dimitri Roschkowski: Hi, is it possible to use just a partition instea

Re: [ceph-users] "CPU CATERR Fault" Was: Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Le lundi 23 juillet 2018 à 12:43 +0200, Oliver Freyermuth a écrit : > There ARE chassis/BMC/IPMI level events, one of which is "CPU > > CATERR > > Fault", with a timestamp matching the timestamps below, and no more > > information. > > If this kind of failure (or a less severe one) also happens at

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Le lundi 23 juillet 2018 à 11:40 +0100, Matthew Vernon a écrit : > One of my server silently shutdown last night, with no explanation > > whatsoever in any logs. According to the existing logs, the > > shutdown > > We have seen similar things with our SuperMicro servers; our current > best theory

[ceph-users] Insane CPU utilization in ceph.fuse

Hello, I've created a Ceph cluster of 3 nodes (3 mons, 3 osd, 3 mgr and 3 mds with two active). This cluster is for mainly for server a webpage (small files) and is configured to have three copies of files (a copy on every OSD). My question is about ceph.fuse clients: I've noticed an insane CPU us

Re: [ceph-users] Insane CPU utilization in ceph.fuse

Hi, do you happen to have a relatively large number of clients and a relatively small cache size on the MDS? Paul 2018-07-23 13:16 GMT+02:00 Daniel Carrasco : > Hello, > > I've created a Ceph cluster of 3 nodes (3 mons, 3 osd, 3 mgr and 3 mds > with two active). This cluster is for mainly for

Re: [ceph-users] Cephfs kernel driver availability

On Sun, Jul 22, 2018 at 9:03 PM Bryan Henderson wrote: > > Is there some better place to get a filesystem driver for the longterm > stable Linux kernel (3.16) than the regular kernel.org source distribution? The general advice[1] on this is not to try and use a 3.x kernel with CephFS. The only e

Re: [ceph-users] Why lvm is recommended method for bleustore

On Mon, Jul 23, 2018 at 6:09 AM, Nicolas Huillard wrote: > Le dimanche 22 juillet 2018 à 09:51 -0400, Satish Patel a écrit : >> I read that post and that's why I open this thread for few more >> questions and clearence, >> >> When you said OSD doesn't come up what actually that means? After >> re

Re: [ceph-users] Insane CPU utilization in ceph.fuse

Hi, thanks for your response. Clients are about 6, and 4 of them are the most of time on standby. Only two are active servers that are serving the webpage. Also we've a varnish on front, so are not getting all the load (below 30% in PHP is not much). About the MDS cache, now I've the mds_cache_mem

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Le lundi 23 juillet 2018 à 12:40 +0200, Oliver Freyermuth a écrit : > Am 23.07.2018 um 11:18 schrieb Nicolas Huillard: > > Le lundi 23 juillet 2018 à 18:23 +1000, Brad Hubbard a écrit : > > > Ceph doesn't shut down systems as in kill or reboot the box if > > > that's > > > what you're saying? > >

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

Am 23.07.2018 um 14:59 schrieb Nicolas Huillard: > Le lundi 23 juillet 2018 à 12:40 +0200, Oliver Freyermuth a écrit : >> Am 23.07.2018 um 11:18 schrieb Nicolas Huillard: >>> Le lundi 23 juillet 2018 à 18:23 +1000, Brad Hubbard a écrit : Ceph doesn't shut down systems as in kill or reboot the

[ceph-users] alert conditions

Hi community, the topic of alerting conditions for a ceph cluster comes up in various contexts. Some folks use prometheus or grafana, (I believe) sopme people would like snmp traps from ceph, the mgr dashboard could provide basic alerting capabilities and there is of course ceph -s. Also see "

Re: [ceph-users] Omap warning in 12.2.6

Thanks for the heads up. I upgraded the cluster to 12.2.7 and the message went away. No CRC errors luckily. -Brent From: Brady Deetz [mailto:bde...@gmail.com] Sent: Thursday, July 19, 2018 3:26 PM To: Brent Kennedy Cc: ceph-users Subject: Re: [ceph-users] Omap warning in 12.2.6 12

Re: [ceph-users] Cephfs kernel driver availability

If you're using CentOS/RHEL you can try the elrepo kernels Mike Kuriger -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of John Spray Sent: Monday, July 23, 2018 5:07 AM To: Bryan Henderson Cc: ceph-users@lists.ceph.com Subject: Re: [ceph-users

Re: [ceph-users] Why lvm is recommended method for bleustore

This is great explanation, based on your details look like when reboot machine (OSD node) it will take longer time to initialize all number of OSDs but if we use LVM in that case it shorten that time. There is a good chance that LVM impact some performance because of extra layer, Does anyone has a

Re: [ceph-users] Why lvm is recommended method for bleustore

On Mon, Jul 23, 2018 at 1:56 PM, Satish Patel wrote: > This is great explanation, based on your details look like when reboot > machine (OSD node) it will take longer time to initialize all number > of OSDs but if we use LVM in that case it shorten that time. That is one aspect, yes. Most importa

Re: [ceph-users] Insane CPU utilization in ceph.fuse

On Mon, Jul 23, 2018 at 5:48 AM, Daniel Carrasco wrote: > Hi, thanks for your response. > > Clients are about 6, and 4 of them are the most of time on standby. Only two > are active servers that are serving the webpage. Also we've a varnish on > front, so are not getting all the load (below 30% in

Re: [ceph-users] Why lvm is recommended method for bleustore

Alfredo, Thanks, I think i should go with LVM then :) I have question here, I have 4 physical SSD per server, some reason i am using ceph-ansible 3.0.8 version which doesn't create LVM volume itself so i have to create LVM volume manually. I am using bluestore ( want to keep WAL/DB on same DATA

Re: [ceph-users] JBOD question

I am planning to buy "LSI SAS 9207-8i" does anyone know it support both RAID & JBOD mode together so i can do RAID-1 on OS disk and other disk for JBOD On Sat, Jul 21, 2018 at 11:16 AM, Willem Jan Withagen wrote: > On 21/07/2018 01:45, Oliver Freyermuth wrote: >> >> Hi Satish, >> >> that really c

Re: [ceph-users] Insane CPU utilization in ceph.fuse

Thanks!, It's true that I've seen a continuous memory growth, but I've not thought in a memory leak. I don't remember exactly how many hours were neccesary to fill the memory, but I calculate that were about 14h. With the new configuration looks like memory grows slowly and when it reaches 5-6 GB

Re: [ceph-users] Fwd: MDS memory usage is very high

Hi, I forgot to say that maybe the Diff is lower than real (8Mb), because the memory usage was still high and i've prepared a new configuration with lower limit (5Mb). I've not reloaded the daemons for now, but maybe the configuration was loaded again today and that's the reason why is using less

Re: [ceph-users] Insane CPU utilization in ceph.fuse

Hi, I forgot to say that maybe the Diff is lower than real (8Mb), because the memory usage was still high and i've prepared a new configuration with lower limit (5Mb). I've not reloaded the daemons for now, but maybe the configuration was loaded again today and that's the reason why is using less

[ceph-users] Technical Writer - Red Hat Ceph Storage

Helloposting for greater visibility of this opportunity, thank you. Technical Writer - Red Hat Ceph Storage US-MA-Boston Posting date (7/19/2018 2:14 AM) Job ID: 64257 Category: Product Documentation URL: https://us-redhat.icims.com/jobs/64257/technical-writer---red-hat-ceph-storage/job?hu

[ceph-users] Reclaim free space on RBD images that use Bluestore?????

I have XenServers that connect via iSCSI to Ceph gateway servers that use lrbd and targetcli. On my ceph cluster the RBD images I create are used as storage repositories in Xenserver for the virtual machine vdisks. Whenever I delete a virtual machine, XenServer shows that the repository size h

Re: [ceph-users] Reclaim free space on RBD images that use Bluestore?????

On 23.07.2018 22:18, Sean Bolding wrote: I have XenServers that connect via iSCSI to Ceph gateway servers that use lrbd and targetcli. On my ceph cluster the RBD images I create are used as storage repositories in Xenserver for the virtual machine vdisks. Whenever I delete a virtual machine,

Re: [ceph-users] Insane CPU utilization in ceph.fuse

On Mon, Jul 23, 2018 at 11:08 AM Patrick Donnelly wrote: > On Mon, Jul 23, 2018 at 5:48 AM, Daniel Carrasco > wrote: > > Hi, thanks for your response. > > > > Clients are about 6, and 4 of them are the most of time on standby. Only > two > > are active servers that are serving the webpage. Also

Re: [ceph-users] Mimic 13.2.1 release date

Looks we're not getting it soon. http://tracker.ceph.com/issues/24981 > On 23.07.2018, at 13:45, Wido den Hollander wrote: > > Any news on this yet? 13.2.1 would be very welcome! :-) > > Wido > > On 07/09/2018 05:11 PM, Wido den Hollander wrote: >> Hi, >

Re: [ceph-users] Insane CPU utilization in ceph.fuse

Yeah, is also my thread. This thread was created before lower the cache size from 512Mb to 8Mb. I thought that maybe was my fault and I did a misconfiguration, so I've ignored the problem until now. Greetings! El mar., 24 jul. 2018 1:00, Gregory Farnum escribió: > On Mon, Jul 23, 2018 at 11:08

Re: [ceph-users] Reclaim free space on RBD images that use Bluestore?????

I have same issue, i just build new Ceph cluster for my Openstack VMs workload using rbd and i have created bunch of VM did some dd test to create big big file to test performance now i deleted all dd file but ceph still showing USED space. I tried to do from guest VM [root@c7-vm ~]# sudo fstrim

Re: [ceph-users] Insane CPU utilization in ceph.fuse

could you profile memory allocation of mds http://docs.ceph.com/docs/mimic/rados/troubleshooting/memory-profiling/ On Tue, Jul 24, 2018 at 7:54 AM Daniel Carrasco wrote: > > Yeah, is also my thread. This thread was created before lower the cache size > from 512Mb to 8Mb. I thought that maybe was

Re: [ceph-users] Reclaim free space on RBD images that use Bluestore?????

Forgive me found this post which solved my issue: https://www.sebastien-han.fr/blog/2015/02/02/openstack-and-ceph-rbd-discard/ On Mon, Jul 23, 2018 at 11:22 PM, Satish Patel wrote: > I have same issue, i just build new Ceph cluster for my Openstack VMs > workload using rbd and i have created bun

[ceph-users] ceph cluster monitoring tool

My 5 node ceph cluster is ready for production, now i am looking for good monitoring tool (Open source), what majority of folks using in their production? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-