Sorry, dont know where to find mds logs. I just found some logs in
/var/log/ceph/ceph-volume.log from mds pod and here it is(just a piece):
...
[2023-02-15 12:09:07,460][ceph_volume.main][INFO ] Running command:
ceph-volume inventory --format json /dev/sda3
[2023-02-15 12:09:07,461][ceph_volume.process][INFO ] Running command:
/usr/bin/lsblk -plno KNAME,NAME,TYPE
[2023-02-15 12:09:07,471][ceph_volume.process][INFO ] stdout /dev/sda
/dev/sda
disk
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda1
/dev/sda1
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda2
/dev/sda2
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda3
/dev/sda3
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sda4
/dev/sda4
part
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sdb
/dev/sdb
disk
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/sr0
/dev/sr0
rom
[2023-02-15 12:09:07,472][ceph_volume.process][INFO ] stdout /dev/dm-0
/dev/mapper/ceph--893e62b0--dc66--4db6--9487--404cad5907d6-osd--block--19ef4373--d828--4259--9159--86f2e89f4397
lvm
[2023-02-15 12:09:07,479][ceph_volume.process][INFO ] Running command:
/usr/sbin/lvs --noheadings --readonly --separator=";" -a --units=b --nosuffix
-S lv_path=/dev/sda3 -o lv_tags,lv_path,lv_name,vg_name,lv_uuid,lv_size
[2023-02-15 12:09:07,525][ceph_volume.process][INFO ] stderr Udev is running
and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will
manage logical volume symlinks in device directory.
[2023-02-15 12:09:07,526][ceph_volume.process][INFO ] stderr Udev is running
and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will
obtain device list by scanning device directory.
[2023-02-15 12:09:07,526][ceph_volume.process][INFO ] Running command:
/usr/bin/lsblk --nodeps -P -o
NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL
/dev/sda3
[2023-02-15 12:09:07,544][ceph_volume.process][INFO ] stdout NAME="sda3"
KNAME="sda3" MAJ:MIN="8:3" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0"
RM="0" MODEL="" SIZE="1M" STATE="" OWNER="root" GROUP="disk" MODE="brw-rw----"
ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="1" SCHED="bfq" TYPE="part"
DISC-ALN="0" DISC-GRAN="4K" DISC-MAX="1G" DISC-ZERO="0" PKNAME="sda"
PARTLABEL="BIOS-BOOT"
[2023-02-15 12:09:07,544][ceph_volume.process][INFO ] Running command:
/usr/sbin/blkid -c /dev/null -p /dev/sda3
[2023-02-15 12:09:07,553][ceph_volume.process][INFO ] stdout /dev/sda3:
PART_ENTRY_SCHEME="gpt" PART_ENTRY_NAME="BIOS-BOOT"
PART_ENTRY_UUID="860433f4-8158-432c-88aa-e42901086299"
PART_ENTRY_TYPE="21686148-6449-6e6f-744e-656564454649" PART_ENTRY_NUMBER="3"
PART_ENTRY_OFFSET="1048576" PART_ENTRY_SIZE="2048" PART_ENTRY_DISK="8:0"
[2023-02-15 12:09:07,554][ceph_volume.process][INFO ] Running command:
/usr/sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o
vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size
/dev/sda3
[2023-02-15 12:09:07,595][ceph_volume.process][INFO ] stderr Udev is running
and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will
manage logical volume symlinks in device directory.
[2023-02-15 12:09:07,595][ceph_volume.process][INFO ] stderr Udev is running
and DM_DISABLE_UDEV environment variable is set. Bypassing udev, LVM will
obtain device list by scanning device directory.
[2023-02-15 12:09:07,596][ceph_volume.process][INFO ] stderr Failed to find
device for physical volume "/dev/sda3".
[2023-02-15 12:09:07,596][ceph_volume.process][INFO ] Running command:
/usr/bin/ceph-bluestore-tool show-label --dev /dev/sda3
[2023-02-15 12:09:07,630][ceph_volume.process][INFO ] stderr unable to read
label for /dev/sda3: (2) No such file or directory
[2023-02-15 12:09:07,630][ceph_volume.process][INFO ] Running command:
/usr/bin/ceph-bluestore-tool show-label --dev /dev/sda3
[2023-02-15 12:09:07,672][ceph_volume.process][INFO ] stderr unable to read
label for /dev/sda3: (2) No such file or directory
[2023-02-15 12:09:07,673][ceph_volume.process][INFO ] Running command:
/usr/sbin/udevadm info --query=property /dev/sda3
[2023-02-15 12:09:07,687][ceph_volume.process][INFO ] stdout
DEVLINKS=/dev/disk/by-partuuid/860433f4-8158-432c-88aa-e42901086299
/dev/disk/by-id/scsi-SQEMU_QEMU_HARDDISK_6ef7d571-c0dc-4394-bcc0-4625a857778b-part3
/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_6ef7d571-c0dc-4394-bcc0-4625a857778b-part3
/dev/disk/by-path/pci-0000:00:06.0-scsi-0:0:0:0-part3
/dev/disk/by-partlabel/BIOS-BOOT
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout DEVNAME=/dev/sda3
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
DEVPATH=/devices/pci0000:00/0000:00:06.0/virtio2/host2/target2:0:0/2:0:0:0/block/sda/sda3
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout DEVTYPE=partition
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout ID_BUS=scsi
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_MODEL=QEMU_HARDDISK
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_MODEL_ENC=QEMU\x20HARDDISK\x20\x20\x20
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_PART_ENTRY_DISK=8:0
[2023-02-15 12:09:07,688][ceph_volume.process][INFO ] stdout
ID_PART_ENTRY_NAME=BIOS-BOOT
....
Also after some manipulation our filesystem now fine(few steps later we create
new metadata pool and restore current from new). But mds still read only. Our
ceph shows only this issue :
cluster:
id: 9213604e-b0b6-49d5-bcb3-f55ab3d79119
health: HEALTH_WARN
1 MDSs are read only
21 daemons have recently crashed
services:
mon: 5 daemons, quorum bd,bj,bm,bn,bo (age 8h)
mgr: a(active, since 102m)
mds: 1/1 daemons up, 1 hot standby
osd: 9 osds: 9 up (since 8h), 9 in (since 7d)
rgw: 1 daemon active (1 hosts, 1 zones)
Thank you
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
