[ceph-users] Mon scrub errors
Hi,
Im' having a cluster with three moitors, two mds and nine osd. Lately I've
been getting scrub errors from the monitors;
2018-04-05 07:26:52.147185 [ERR] mon.2 ScrubResult(keys
{osd_pg_creating=1,osdmap=99} crc
{osd_pg_creating=1404726104,osdmap=3323124730})
2018-04-05 07:26:52.147167 [ERR] mon.0 ScrubResult(keys
{osd_metadata=5,osd_pg_creating=1,osdmap=94} crc
{osd_metadata=477302505,osd_pg_creating=1404726104,osdmap=2387598890})
2018-04-05 07:26:52.147139 [ERR] scrub mismatch
2018-04-05 07:26:52.144378 [ERR] mon.2 ScrubResult(keys
{mgrstat=92,monmap=5,osd_pg_creating=1,osdmap=2} crc
{mgrstat=2630742218,monmap=4118007020,osd_pg_creating=1404726104,osdmap=940126788})
2018-04-05 07:26:52.144360 [ERR] mon.0 ScrubResult(keys
{mgrstat=92,monmap=5,osd_metadata=3} crc
{mgrstat=2630742218,monmap=4118007020,osd_metadata=3256871745})
2018-04-05 07:26:52.144334 [ERR] scrub mismatch
2018-04-05 07:26:52.140213 [ERR] mon.2 ScrubResult(keys
{mgr=67,mgrstat=33} crc {mgr=1823433831,mgrstat=205032})
2018-04-05 07:26:52.140193 [ERR] mon.0 ScrubResult(keys
{mgr=67,mgr_command_descs=1,mgr_metadata=2,mgrstat=30} crc
{mgr=1823433831,mgr_command_descs=2758154725,mgr_metadata=2776211204,mgrstat=779157107})
2018-04-05 07:26:52.140165 [ERR] scrub mismatch
2018-04-05 07:26:52.120025 [ERR] mon.2 ScrubResult(keys
{logm=23,mdsmap=77} crc {logm=1176883332,mdsmap=2557424777})
2018-04-05 07:26:52.120008 [ERR] mon.0 ScrubResult(keys
{logm=23,mds_health=4,mds_metadata=1,mdsmap=72} crc
{logm=1176883332,mds_health=984298828,mds_metadata=395047004,mdsmap=809526521})
2018-04-05 07:26:52.119982 [ERR] scrub mismatch
What can I do to correct the errors?
Ceph version 12.2.3
Best regards
//Rickard
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Rados bucket issues, default.rgw.buckets.index growing every day
On Wed, 2018-04-04 at 09:38 +0200, Mark Schouten wrote: > I have some issues with my bucket index. As you can see in the > attachment, everyday around 16:30, the amount of objects in the > default.rgw.buckets.index increases. This happens since upgrading > from > 12.2.2 to 12.2.4. I disabled dynamic resharding, which seems to have stopped the increasing number of objects.. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | i...@tuxis.nl smime.p7s Description: S/MIME cryptographic signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Use trimfs on already mounted RBD image
Ok, thanks for Your reply. 2018-04-05 8:15 GMT+02:00 Wido den Hollander : > > > On 04/04/2018 07:30 PM, Damian Dabrowski wrote: > > Hello, > > > > I wonder if it is any way to run `trimfs` on rbd image which is > > currently used by the KVM process? (when I don't have access to VM) > > > > I know that I can do this by qemu-guest-agent but not all VMs have it > > installed. > > > > I can't use rbdmap too, because most images don't have distributed > > filesystems. It's mostly ext4/xfs so I can't mount them in two places at > > same time. > > > > I would be grateful for any help. > > > > No, you can not. If you map and mount a filesystem twice you will > corrupt the data on there. > > Same goes when you are going to fstrim, you will probably break/corrupt > data on the system. > > Do not attempt it. Shut down the VM and then do the fstrim. > > Wido > > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] bluestore OSD did not start at system-boot
hi folks,
i just figured out that my ODS's did not start because the filsystem
is not mounted.
So i wrote a script to Hack my way around it
#
#! /usr/bin/env bash
DATA=( $(ceph-volume lvm list | grep -e 'osd id\|osd fsid' | awk
'{print $3}' | tr '\n' ' ') )
OSDS=$(( ${#DATA[@]}/2 ))
for OSD in $(seq 0 $(($OSDS-1))); do
ceph-volume lvm activate "${DATA[( $OSD*2 )]}" "${DATA[( $OSD*2+1 )]}"
done
#
i'am sure that this is not the way it should be!? so any help i
welcome to figure out why my BlueStore-OSD is not mounted at
boot-time.
Thanks,
Ansgar
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] bluestore OSD did not start at system-boot
Hey Ansgar,
we have a similar "problem": in our case all servers are wiped on
reboot, as they boot their operating system from the network into
initramfs.
While the OS configuration is done with cdist [0], we consider ceph osds
more dynamic data and just re-initialise all osds on boot using the
ungleich-tools [1] suite, which we created to work with ceph clusters
mostly.
Especially [2] might be of interest for you.
HTH,
Nico
[0] https://www.nico.schottelius.org/software/cdist/
[1] https://github.com/ungleich/ungleich-tools
[2] https://github.com/ungleich/ungleich-tools/blob/master/ceph-osd-activate-all
Ansgar Jazdzewski writes:
> hi folks,
>
> i just figured out that my ODS's did not start because the filsystem
> is not mounted.
>
> So i wrote a script to Hack my way around it
> #
> #! /usr/bin/env bash
>
> DATA=( $(ceph-volume lvm list | grep -e 'osd id\|osd fsid' | awk
> '{print $3}' | tr '\n' ' ') )
>
> OSDS=$(( ${#DATA[@]}/2 ))
>
> for OSD in $(seq 0 $(($OSDS-1))); do
> ceph-volume lvm activate "${DATA[( $OSD*2 )]}" "${DATA[( $OSD*2+1 )]}"
> done
> #
>
> i'am sure that this is not the way it should be!? so any help i
> welcome to figure out why my BlueStore-OSD is not mounted at
> boot-time.
>
> Thanks,
> Ansgar
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rgw make container private again
There is an existing bug related to the impossibility to delete the read ACL of Swift. Check https://tracker.ceph.com/issues/22897 V. On 30/03/18, 18:11 , "ceph-users on behalf of Vladimir Prokofev" wrote: As usual, I found solution after a while. Metadata field is not deleting as it should by API docs, but it can be changed. So I just changed it with curl -X POST -i -H "X-Auth-Token: -H "X-Container-Read: :*" https://endpoint.url/swift/v1/containername and now metadata field looks like this X-Container-Read: :* Essentialy this behaves the same as when there's no X-Container-Read at all. But overall this is still an issue - what should've taken 5 seconds to just uncheck a box in Horizon interface turned into couple hours of debugging. Can anyone who uses same version check if this issue is reproducible? If so - this seems to be a ticket-worthy. 2018-03-30 17:40 GMT+03:00 Vladimir Prokofev : CEPH 12.2.2, RGW. I'm using it as an object storage endpoint for Openstack. Recently while browsing an object storage from Horizon, I accidently marked container as public. The issue is - I can't make it private again! Docs state that to do it I should simply delete X-Container-Read metadata, but I just can't! Examples: private container headers(only relevant output, some other empty container): X-Container-Bytes-Used-Actual: 0 X-Storage-Policy: default-placement public container headers(only relevant output): X-Container-Bytes-Used-Actual: 114688 X-Container-Read: .r:*,.rlistings X-Storage-Policy: default-placement As you can see, there's now an X-Container-Read header. I've tried to make it back private with swift client and curl, but to no success. Here're some curl examples. Updating works! If I do curl -X POST -i -H "X-Auth-Token: " -H "X-Container-Read: .r:test" https://endpoint.url/swift/v1/containername metadata will become X-Container-Read: .r:test But if I do curl -X POST -i -H "X-Auth-Token: " -H "X-Remove-Container-Read: x" https://endpoint.url/swift/v1/containername nothing happens, metadata field will remain there. So is this a broken API in RGW, or am I missing something? Maybe there's some explicit warning that after becoming public you can't make container private again? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Cephfs hardlink snapshot
'Hard links do not interact well with snapshots' is this still an issue? Because I am using rsync and hardlinking. And it would be nice if I can snapshot the directory, instead of having to copy it. http://docs.ceph.com/docs/master/dev/cephfs-snapshots/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Mon scrub errors
On Thu, Apr 5, 2018 at 3:10 PM, Rickard Nilsson
wrote:
> Hi,
>
> Im' having a cluster with three moitors, two mds and nine osd. Lately I've
> been getting scrub errors from the monitors;
>
> 2018-04-05 07:26:52.147185 [ERR] mon.2 ScrubResult(keys
> {osd_pg_creating=1,osdmap=99} crc
> {osd_pg_creating=1404726104,osdmap=3323124730})
> 2018-04-05 07:26:52.147167 [ERR] mon.0 ScrubResult(keys
> {osd_metadata=5,osd_pg_creating=1,osdmap=94} crc
> {osd_metadata=477302505,osd_pg_creating=1404726104,osdmap=2387598890})
> 2018-04-05 07:26:52.147139 [ERR] scrub mismatch
> 2018-04-05 07:26:52.144378 [ERR] mon.2 ScrubResult(keys
> {mgrstat=92,monmap=5,osd_pg_creating=1,osdmap=2} crc
> {mgrstat=2630742218,monmap=4118007020,osd_pg_creating=1404726104,osdmap=940126788})
> 2018-04-05 07:26:52.144360 [ERR] mon.0 ScrubResult(keys
> {mgrstat=92,monmap=5,osd_metadata=3} crc
> {mgrstat=2630742218,monmap=4118007020,osd_metadata=3256871745})
> 2018-04-05 07:26:52.144334 [ERR] scrub mismatch
> 2018-04-05 07:26:52.140213 [ERR] mon.2 ScrubResult(keys {mgr=67,mgrstat=33}
> crc {mgr=1823433831,mgrstat=205032})
> 2018-04-05 07:26:52.140193 [ERR] mon.0 ScrubResult(keys
> {mgr=67,mgr_command_descs=1,mgr_metadata=2,mgrstat=30} crc
> {mgr=1823433831,mgr_command_descs=2758154725,mgr_metadata=2776211204,mgrstat=779157107})
> 2018-04-05 07:26:52.140165 [ERR] scrub mismatch
> 2018-04-05 07:26:52.120025 [ERR] mon.2 ScrubResult(keys {logm=23,mdsmap=77}
> crc {logm=1176883332,mdsmap=2557424777})
> 2018-04-05 07:26:52.120008 [ERR] mon.0 ScrubResult(keys
> {logm=23,mds_health=4,mds_metadata=1,mdsmap=72} crc
> {logm=1176883332,mds_health=984298828,mds_metadata=395047004,mdsmap=809526521})
> 2018-04-05 07:26:52.119982 [ERR] scrub mismatch
>
> What can I do to correct the errors?
Rickard, probably you can nuke mon.2, and re-add it ? so its monstore
can be sync'ed from other two monitor (for instance, mon.0)? for how
to remove/add monitors , see
http://docs.ceph.com/docs/kraken/rados/operations/add-or-rm-mons/ .
>
> Ceph version 12.2.3
>
> Best regards
> //Rickard
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
--
Regards
Kefu Chai
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%
Hi, I have a strange issue - OSDs from a specific server are introducing huge performance issue This is a brand new installation on 3 identical servers - DELL R620 with PERC H710 , bluestore DB and WAL on SSD, 10GB dedicated private/public networks When I add the OSD I see gaps like below and huge latency atop provides no clear culprit EXCEPT very low network and specific disk utilization BUT 100% DSK for ceph-osd process which stay like that ( 100%) for the duration of the test ( see below) Not sure why ceph-osd process DSK stays at 100% while all the specific DSK ( for sdb, sde ..etc) are 1% busy ? Any help/ instructions for how to troubleshooting this will be appreciated (apologies if the format is not being kept) CPU | sys 4% | user 1% | | irq 1% | | idle794% | wait 0% | | | steal 0% | guest 0% | curf 2.20GHz | | curscal ?% | CPL | avg10.00 | | avg50.00 | avg15 0.00 | | | | csw547/s | | intr 832/s | | | numcpu 8 | | MEM | tot62.9G | free 61.4G | cache 520.6M | dirty 0.0M | buff 7.5M | slab 98.9M | slrec 64.8M | shmem 8.8M | shrss 0.0M | shswp 0.0M | vmbal 0.0M | | hptot 0.0M | hpuse 0.0M | SWP | tot 6.0G | free6.0G | | | | | | | | | | vmcom 1.5G | | vmlim 37.4G | LVM | dm-0 | busy 1% | | read 0/s | write 54/s | | KiB/r 0 | KiB/w455 | MBr/s 0.0 | | MBw/s 24.0 | avq 3.69 | | avio 0.14 ms | DSK | sdb | busy 1% | | read 0/s | write 102/s | | KiB/r 0 | KiB/w240 | MBr/s 0.0 | | MBw/s 24.0 | avq 6.69 | | avio 0.08 ms | DSK | sda | busy 0% | | read 0/s | write 12/s | | KiB/r 0 | KiB/w 4 | MBr/s 0.0 | | MBw/s0.1 | avq 1.00 | | avio 0.05 ms | DSK | sde | busy 0% | | read 0/s | write0/s | | KiB/r 0 | KiB/w 0 | MBr/s 0.0 | | MBw/s0.0 | avq 1.00 | | avio 2.50 ms | NET | transport | tcpi 718/s | tcpo 972/s | udpi 0/s | | udpo 0/s | tcpao0/s | tcppo0/s | tcprs 21/s | tcpie0/s | tcpor0/s | | udpnp0/s | udpie 0/s | NET | network | ipi719/s | | ipo399/s | ipfrw0/s | | deliv 719/s | | | | | icmpi0/s | | icmpo 0/s | NET | eth5 1% | pcki 2214/s | pcko 939/s | | sp 10 Gbps | si 154 Mbps | so 52 Mbps | | coll 0/s | mlti 0/s | erri 0/s | erro 0/s | drpi 0/s | drpo 0/s | NET | eth4 0% | pcki 712/s | pcko54/s | | sp 10 Gbps | si 50 Mbps | so 90 Kbps | | coll 0/s | mlti 0/s | erri 0/s | erro 0/s | drpi 0/s | drpo 0/s | PID TID RDDSK WRDSK WCANCL DSK CMD 1/21 2067 - 0K/s 0.0G/s 0K/s 100% ceph-osd 2018-04-05 10:55:24.316549 min lat: 0.0203278 max lat: 10.7501 avg lat: 0.496822 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 1096 1080 107.988 0 - 0.496822 41 16 1096 1080 105.354 0 - 0.496822 42 16 1096 1080 102.846 0 - 0.496822 43 16 1096 1080 100.454 0 - 0.496822 44 16 1205 1189 108.079 48. 0.0430396 0.588127 45 16 1234 1218 108.255 116 0.0318717 0.575485 46 16 1234 1218 105.901 0 - 0.575485 47 16 1234 1218 103.648 0 - 0.575485 48 16 1234 1218 101.489 0 - 0.575485 49 16 1261 1245 101.622270.157469 0.604268 50 16 1335 1319 105.508 2960.191907 0.604862 51 16 1418 1402 109.949 332 0.0367004 0.573429 52 16 1437 1421 109.296760.031818 0.566289 53 16 1481 1465 110.554 176 0.0405567 0.564885 54 16 1516 1500 111.099 140 0.0272873 0.5
Re: [ceph-users] ceph-deploy: recommended?
... we use (only!) ceph-deploy in all our environments, tools and scripts. If I look in the efforts went into ceph-volume and all the related issues, "manual LVM" overhead and/or still missing features, PLUS the in the same discussions mentioned recommendations to use something like ceph-ansible in parallel for the missing stuff, I can only hope we will find a (full time?!) maintainer for ceph-deploy and keep it alive. PLEASE ;) Gesendet: Donnerstag, 05. April 2018 um 08:53 Uhr Von: "Wido den Hollander" An: ceph-users@lists.ceph.com Betreff: Re: [ceph-users] ceph-deploy: recommended? On 04/04/2018 08:58 PM, Robert Stanford wrote: > > I read a couple of versions ago that ceph-deploy was not recommended > for production clusters. Why was that? Is this still the case? We > have a lot of problems automating deployment without ceph-deploy. > > In the end it is just a Python tool which deploys the daemons. It is not active in any way. Stability of the cluster is not determined by the use of ceph-deploy, but by the runnings daemons. I use ceph-deploy sometimes in very large deployments to make my life a bit easier. Wido > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com] ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Luminous and Bluestore: low load and high latency on RBD
I am seeing a strange behavior with Luminous 12.2.4 and Bluestore, Ubuntu kernel 4.14 and 108 x 10K RPM Seagate drives, no SSD WAL/DB and 8GB Areca controllers, 10 GbE networking on OSD nodes and 56GbE on clients. Single stream IOs to RD volumes return with 50-1000 ms latency, while atop shows any of the physical drives respond with single ms of latency. No obvious errors - I am wondering if we are not working the OSD daemons hard enough. In the past, there was the osd_op_threads, which seemed to help when > 20 - this parameter seems to be gone now. I looked at the osd_op_queue, but not sure if it would do similar things. Any ideas? https://vendor2.nginfotpdx.net/gitlab/upstream/ceph/commit/8aa159befa58cd9059ad99c752146f3a5dbfcb8b -- Alex Gorbachev Storcium ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Cephfs hardlink snapshot
Hi Marc, On Wed, Apr 4, 2018 at 11:21 PM, Marc Roos wrote: > > 'Hard links do not interact well with snapshots' is this still an issue? > Because I am using rsync and hardlinking. And it would be nice if I can > snapshot the directory, instead of having to copy it. Hardlink handling for snapshots will be in Mimic. -- Patrick Donnelly ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] EC related osd crashes (luminous 12.2.4)
Hello all, I'm having some stability issues with my ceph cluster at the moment. Using CentOS 7, and Ceph 12.2.4. I have osds that are segfaulting regularly. roughly every minute or so, and it seems to be getting worse, now with cascading failures. Backtraces look like this: ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable) 1: (()+0xa3c611) [0x55cb9249c611] 2: (()+0xf5e0) [0x7eff83b495e0] 3: (std::list, std::allocator > >::list(std::list, std::allocator > > const&)+0x3e) [0x55cb9225562e] 4: (ECBackend::send_all_remaining_reads(hobject_t const&, ECBackend::ReadOp&)+0x33b) [0x55cb92243bab] 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0x1074) [0x55cb92245184] 6: (ECBackend::_handle_message(boost::intrusive_ptr)+0x1af) [0x55cb9224fa2f] 7: (PGBackend::handle_message(boost::intrusive_ptr)+0x50) [0x55cb921545f0] 8: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x59c) [0x55cb920c004c] 9: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x3f9) [0x55cb91f45f69] 10: (PGQueueable::RunVis::operator()(boost::intrusive_ptr const&)+0x57) [0x55cb921c2b57] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xfce) [0x55cb91f749de] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x839) [0x55cb924e1089] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55cb924e3020] 14: (()+0x7e25) [0x7eff83b41e25] 15: (clone()+0x6d) [0x7eff82c3534d] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. When I start a crashed osd, it seems to cause cascading crashes in other osds with the same backtrace. This is making it problematic to keep my placement groups up and active. A full (start to finish) log file is available here: http://people.cs.ksu.edu/~mozes/ceph-osd.44.log Anyone have any thoughts, or workarounds? -- Adam ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] EC related osd crashes (luminous 12.2.4)
Well, the cascading crashes are getting worse. I'm routinely seeing 8-10 of my 518 osds crash. I cannot start 2 of them without triggering 14 or so of them to crash repeatedly for more than an hour. I've ran another one of them with more logging, debug osd = 20; debug ms = 1 (definitely more than one crash in there): http://people.cs.ksu.edu/~mozes/ceph-osd.422.log Anyone have any thoughts? My cluster feels like it is getting more and more unstable by the hour... -- Adam ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] EC related osd crashes (luminous 12.2.4)
On 04/05/2018 06:15 PM, Adam Tygart wrote: Well, the cascading crashes are getting worse. I'm routinely seeing 8-10 of my 518 osds crash. I cannot start 2 of them without triggering 14 or so of them to crash repeatedly for more than an hour. I've ran another one of them with more logging, debug osd = 20; debug ms = 1 (definitely more than one crash in there): http://people.cs.ksu.edu/~mozes/ceph-osd.422.log Anyone have any thoughts? My cluster feels like it is getting more and more unstable by the hour... Thanks to your logs, I think I've found the root cause. It looks like a bug in the EC recovery code that's triggered by EC overwrites. I'm working on a fix. For now I'd suggest setting the noout and norecover flags to avoid hitting this bug any more by avoiding recovery. Backfilling with no client I/O would also avoid the bug. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] EC related osd crashes (luminous 12.2.4)
On 04/05/2018 08:11 PM, Josh Durgin wrote: On 04/05/2018 06:15 PM, Adam Tygart wrote: Well, the cascading crashes are getting worse. I'm routinely seeing 8-10 of my 518 osds crash. I cannot start 2 of them without triggering 14 or so of them to crash repeatedly for more than an hour. I've ran another one of them with more logging, debug osd = 20; debug ms = 1 (definitely more than one crash in there): http://people.cs.ksu.edu/~mozes/ceph-osd.422.log Anyone have any thoughts? My cluster feels like it is getting more and more unstable by the hour... Thanks to your logs, I think I've found the root cause. It looks like a bug in the EC recovery code that's triggered by EC overwrites. I'm working on a fix. For now I'd suggest setting the noout and norecover flags to avoid hitting this bug any more by avoiding recovery. Backfilling with no client I/O would also avoid the bug. I forgot to mention the tracker ticket for this bug is: http://tracker.ceph.com/issues/23195 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] EC related osd crashes (luminous 12.2.4)
Thank you! Setting norecover has seemed to work in terms of keeping the osds up. I am glad my logs were of use to tracking this down. I am looking forward to future updates. Let me know if you need anything else. -- Adam On Thu, Apr 5, 2018 at 10:13 PM, Josh Durgin wrote: > On 04/05/2018 08:11 PM, Josh Durgin wrote: >> >> On 04/05/2018 06:15 PM, Adam Tygart wrote: >>> >>> Well, the cascading crashes are getting worse. I'm routinely seeing >>> 8-10 of my 518 osds crash. I cannot start 2 of them without triggering >>> 14 or so of them to crash repeatedly for more than an hour. >>> >>> I've ran another one of them with more logging, debug osd = 20; debug >>> ms = 1 (definitely more than one crash in there): >>> http://people.cs.ksu.edu/~mozes/ceph-osd.422.log >>> >>> Anyone have any thoughts? My cluster feels like it is getting more and >>> more unstable by the hour... >> >> >> Thanks to your logs, I think I've found the root cause. It looks like a >> bug in the EC recovery code that's triggered by EC overwrites. I'm working >> on a fix. >> >> For now I'd suggest setting the noout and norecover flags to avoid >> hitting this bug any more by avoiding recovery. Backfilling with no client >> I/O would also avoid the bug. > > > I forgot to mention the tracker ticket for this bug is: > http://tracker.ceph.com/issues/23195 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
