Hi Sage, Thanks for chiming in. I can't image how busy you are. Sorry guys. I reprovisioned the offending osd right after this email and a conversation on #ceph. I do have the output from '/usr/bin/ceph daemon osd.5 perf dump | /usr/bin/jq .' saved. I'll be happy to add it to the issue tracker.
/C On Fri, Mar 22, 2019 at 7:01 PM Sage Weil <s...@newdream.net> wrote: > I have a ticket open for this: > > http://tracker.ceph.com/issues/38745 > > Please comment there with the health warning you're seeing and any other > details so we can figure out why it's happening. > > I wouldn't reprovision those OSDs yet, until we know why it happens. > Also, it's likely that ceph-bluestore-tool can sort it out be > adding/removing the db volume. > > Thanks! > sage > > > On Fri, 22 Mar 2019, Mazzystr wrote: > > > I am also seeing BlueFS spill since updating to Nautilus. I also see > high > > slow_used_bytes and slow_total_bytes metrics. It sure looks to me that > the > > only solution is to zap and rebuilt the osd. I had to manually check 36 > > osds some of them traditional processes and some containerized. The lack > > of tooling here is underwhelming... As soon as I rebuilt the osd the > > "BlueFS spill..." warning went away. > > > > I use 50Gb db partitions on an nvme with 3 or 6 Tb spinning disks. I > don't > > understand the spillover. > > > > > > On Fri, Feb 15, 2019 at 12:33 PM David Turner <drakonst...@gmail.com> > wrote: > > > > > The answer is probably going to be in how big your DB partition is vs > how > > > big your HDD disk is. From your output it looks like you have a 6TB > HDD > > > with a 28GB Blocks.DB partition. Even though the DB used size isn't > > > currently full, I would guess that at some point since this OSD was > created > > > that it did fill up and what you're seeing is the part of the DB that > > > spilled over to the data disk. This is why the official recommendation > > > (that is quite cautious, but cautious because some use cases will use > this > > > up) for a blocks.db partition is 4% of the data drive. For your 6TB > disks > > > that's a recommendation of 240GB per DB partition. Of course the > actual > > > size of the DB needed is dependent on your use case. But pretty much > every > > > use case for a 6TB disk needs a bigger partition than 28GB. > > > > > > On Thu, Feb 14, 2019 at 11:58 PM Konstantin Shalygin <k0...@k0ste.ru> > > > wrote: > > > > > >> Wrong metadata paste of osd.73 in previous message. > > >> > > >> > > >> { > > >> > > >> "id": 73, > > >> "arch": "x86_64", > > >> "back_addr": "10.10.10.6:6804/175338", > > >> "back_iface": "vlan3", > > >> "bluefs": "1", > > >> "bluefs_db_access_mode": "blk", > > >> "bluefs_db_block_size": "4096", > > >> "bluefs_db_dev": "259:22", > > >> "bluefs_db_dev_node": "nvme2n1", > > >> "bluefs_db_driver": "KernelDevice", > > >> "bluefs_db_model": "INTEL SSDPEDMD400G4 ", > > >> "bluefs_db_partition_path": "/dev/nvme2n1p11", > > >> "bluefs_db_rotational": "0", > > >> "bluefs_db_serial": "CVFT4324002Q400BGN ", > > >> "bluefs_db_size": "30064771072", > > >> "bluefs_db_type": "nvme", > > >> "bluefs_single_shared_device": "0", > > >> "bluefs_slow_access_mode": "blk", > > >> "bluefs_slow_block_size": "4096", > > >> "bluefs_slow_dev": "8:176", > > >> "bluefs_slow_dev_node": "sdl", > > >> "bluefs_slow_driver": "KernelDevice", > > >> "bluefs_slow_model": "TOSHIBA HDWE160 ", > > >> "bluefs_slow_partition_path": "/dev/sdl2", > > >> "bluefs_slow_rotational": "1", > > >> "bluefs_slow_size": "6001069199360", > > >> "bluefs_slow_type": "hdd", > > >> "bluefs_wal_access_mode": "blk", > > >> "bluefs_wal_block_size": "4096", > > >> "bluefs_wal_dev": "259:22", > > >> "bluefs_wal_dev_node": "nvme2n1", > > >> "bluefs_wal_driver": "KernelDevice", > > >> "bluefs_wal_model": "INTEL SSDPEDMD400G4 ", > > >> "bluefs_wal_partition_path": "/dev/nvme2n1p12", > > >> "bluefs_wal_rotational": "0", > > >> "bluefs_wal_serial": "CVFT4324002Q400BGN ", > > >> "bluefs_wal_size": "1073741824", > > >> "bluefs_wal_type": "nvme", > > >> "bluestore_bdev_access_mode": "blk", > > >> "bluestore_bdev_block_size": "4096", > > >> "bluestore_bdev_dev": "8:176", > > >> "bluestore_bdev_dev_node": "sdl", > > >> "bluestore_bdev_driver": "KernelDevice", > > >> "bluestore_bdev_model": "TOSHIBA HDWE160 ", > > >> "bluestore_bdev_partition_path": "/dev/sdl2", > > >> "bluestore_bdev_rotational": "1", > > >> "bluestore_bdev_size": "6001069199360", > > >> "bluestore_bdev_type": "hdd", > > >> "ceph_version": "ceph version 12.2.10 > > >> (177915764b752804194937482a39e95e0ca3de94) luminous (stable)", > > >> "cpu": "Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz", > > >> "default_device_class": "hdd", > > >> "distro": "centos", > > >> "distro_description": "CentOS Linux 7 (Core)", > > >> "distro_version": "7", > > >> "front_addr": "172.16.16.16:6803/175338", > > >> "front_iface": "vlan4", > > >> "hb_back_addr": "10.10.10.6:6805/175338", > > >> "hb_front_addr": "172.16.16.16:6805/175338", > > >> "hostname": "ceph-osd5", > > >> "journal_rotational": "0", > > >> "kernel_description": "#1 SMP Tue Aug 14 21:49:04 UTC 2018", > > >> "kernel_version": "3.10.0-862.11.6.el7.x86_64", > > >> "mem_swap_kb": "0", > > >> "mem_total_kb": "65724256", > > >> "os": "Linux", > > >> "osd_data": "/var/lib/ceph/osd/ceph-73", > > >> "osd_objectstore": "bluestore", > > >> "rotational": "1" > > >> } > > >> _______________________________________________ > > >> ceph-users mailing list > > >> ceph-users@lists.ceph.com > > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >> > > > _______________________________________________ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
