Hello, On Sun, 10 Jul 2016 12:46:39 +0000 (UTC) William Josefsson wrote:
> Hi everyone, > > I have problem with swapping drive and partition names on reboot. My > Ceph is Hammer on CentOS7, Dell R730 6xSSD (2xSSD OS RAID1 PERC, > 4xSSD=Journal drives), 18x1.8T SAS for OSDs. > > Whenever I reboot, drives randomly seem to change names. This is > extremely dangerous and frustrating when I've initially setup CEPH with > ceph-deploy, zap, prepare and activate. It has happened that I've > accidentally erased wrong disk too when e.g. /dev/sdX had > become /dev/sdY. > This isn't a Ceph specific question per se and you could probably keep things from moving around by enforcing module loads in a particular order. But that of course still wouldn't help if something else changed or a drive totally failed. So in the context of Ceph, it doesn't (shouldn't) care if the OSD (HDD) changes names, especially since you did set it up with ceph-deploy. And to avoid the journals getting jumbled up, do what everybody does (outside of Ceph as well), use /dev/disk/by-id or uuid. Like: --- # ls -la /var/lib/ceph/osd/ceph-28/ journal -> /dev/disk/by-id/wwn-0x55cd2e404b73d569-part3 --- Christian > Please see an output below of how this drive swapping below appears SDC > is shifted, indexes and drive names got shuffled. Ceph OSDs didn't come > up properly. > > Please advice on how to get this corrected, with no more drive name > shuffling. Can this be due to the PERC HW raid? thx will > > > > POST REBOOT 2 (expected outcome.. with sda,sdb,sdc,sdd as journal. sdw > is a perc raid1) > > > [cephnode3][INFO ] Running command: sudo /usr/sbin/ceph-disk list > [cephnode3][DEBUG ] /dev/sda : > [cephnode3][DEBUG ] /dev/sda1 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda2 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sda3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sda4 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda5 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdb : [cephnode3][DEBUG ] /dev/sdb1 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb2 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdb3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sdb4 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb5 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdc : [cephnode3][DEBUG ] /dev/sdc1 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdc2 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdc3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sdc4 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdc5 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdd : [cephnode3][DEBUG ] /dev/sdd1 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd2 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdd3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sdd4 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd5 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sde : [cephnode3][DEBUG ] /dev/sde1 ceph data, active, cluster > ceph, osd.0 [cephnode3][DEBUG ] /dev/sdf : [cephnode3][DEBUG > ] /dev/sdf1 ceph data, active, cluster ceph, osd.1 [cephnode3][DEBUG > ] /dev/sdg : [cephnode3][DEBUG ] /dev/sdg1 ceph data, active, cluster > ceph, osd.2 [cephnode3][DEBUG ] /dev/sdh : [cephnode3][DEBUG > ] /dev/sdh1 ceph data, active, cluster ceph, osd.3 [cephnode3][DEBUG > ] /dev/sdi : [cephnode3][DEBUG ] /dev/sdi1 ceph data, active, cluster > ceph, osd.4 [cephnode3][DEBUG ] /dev/sdj : [cephnode3][DEBUG > ] /dev/sdj1 ceph data, active, cluster ceph, osd.5 [cephnode3][DEBUG > ] /dev/sdk : [cephnode3][DEBUG ] /dev/sdk1 ceph data, active, cluster > ceph, osd.6 [cephnode3][DEBUG ] /dev/sdl : [cephnode3][DEBUG > ] /dev/sdl1 ceph data, active, cluster ceph, osd.7 [cephnode3][DEBUG > ] /dev/sdm : [cephnode3][DEBUG ] /dev/sdm1 other, xfs > [cephnode3][DEBUG ] /dev/sdn : > [cephnode3][DEBUG ] /dev/sdn1 ceph data, active, cluster ceph, osd.9 > [cephnode3][DEBUG ] /dev/sdo : > [cephnode3][DEBUG ] /dev/sdo1 ceph data, active, cluster ceph, osd.10 > [cephnode3][DEBUG ] /dev/sdp : > [cephnode3][DEBUG ] /dev/sdp1 ceph data, active, cluster ceph, osd.11 > [cephnode3][DEBUG ] /dev/sdq : > [cephnode3][DEBUG ] /dev/sdq1 ceph data, active, cluster ceph, osd.12 > [cephnode3][DEBUG ] /dev/sdr : > [cephnode3][DEBUG ] /dev/sdr1 ceph data, active, cluster ceph, osd.13 > [cephnode3][DEBUG ] /dev/sds : > [cephnode3][DEBUG ] /dev/sds1 ceph data, active, cluster ceph, osd.14 > [cephnode3][DEBUG ] /dev/sdt : > [cephnode3][DEBUG ] /dev/sdt1 ceph data, active, cluster ceph, osd.15 > [cephnode3][DEBUG ] /dev/sdu : > [cephnode3][DEBUG ] /dev/sdu1 ceph data, active, cluster ceph, osd.16 > [cephnode3][DEBUG ] /dev/sdv : > [cephnode3][DEBUG ] /dev/sdv1 ceph data, active, cluster ceph, osd.17 > [cephnode3][DEBUG ] /dev/sdw : > [cephnode3][DEBUG ] /dev/sdw1 other, xfs, mounted on / > [cephnode3][DEBUG ] /dev/sdw2 swap, swap > > > POST REBOOT 1: > > > [cephnode3][DEBUG ] /dev/sda : > [cephnode3][DEBUG ] /dev/sda1 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda2 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sda3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sda4 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sda5 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdb : [cephnode3][DEBUG ] /dev/sdb1 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb2 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdb3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sdb4 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdb5 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdc : [cephnode3][DEBUG ] /dev/sdc1 ceph data, active, cluster > ceph, osd.13 [cephnode3][DEBUG ] /dev/sdd : [cephnode3][DEBUG > ] /dev/sdd1 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sdd2 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sdd3 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdd4 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sdd5 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde : > [cephnode3][DEBUG ] /dev/sde1 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde2 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sde3 other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 > [cephnode3][DEBUG ] /dev/sde4 other, > ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG ] /dev/sde5 > other, ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 [cephnode3][DEBUG > ] /dev/sdf : [cephnode3][DEBUG ] /dev/sdf1 ceph data, active, cluster > ceph, osd.0 [cephnode3][DEBUG ] /dev/sdg : [cephnode3][DEBUG > ] /dev/sdg1 ceph data, active, cluster ceph, osd.14 [cephnode3][DEBUG > ] /dev/sdh : [cephnode3][DEBUG ] /dev/sdh1 ceph data, active, cluster > ceph, osd.1 [cephnode3][DEBUG ] /dev/sdi : [cephnode3][DEBUG > ] /dev/sdi1 ceph data, active, cluster ceph, osd.2 [cephnode3][DEBUG > ] /dev/sdj : [cephnode3][DEBUG ] /dev/sdj1 ceph data, active, cluster > ceph, osd.3 [cephnode3][DEBUG ] /dev/sdk : [cephnode3][DEBUG > ] /dev/sdk1 ceph data, active, cluster ceph, osd.4 [cephnode3][DEBUG > ] /dev/sdl : [cephnode3][DEBUG ] /dev/sdl1 ceph data, active, cluster > ceph, osd.15 [cephnode3][DEBUG ] /dev/sdm : [cephnode3][DEBUG > ] /dev/sdm1 ceph data, active, cluster ceph, osd.5 [cephnode3][DEBUG > ] /dev/sdn : [cephnode3][DEBUG ] /dev/sdn1 ceph data, active, cluster > ceph, osd.6 [cephnode3][DEBUG ] /dev/sdo : > [cephnode3][DEBUG ] /dev/sdo1 ceph data, active, cluster ceph, osd.7 > [cephnode3][DEBUG ] /dev/sdp : > [cephnode3][DEBUG ] /dev/sdp1 other, xfs > [cephnode3][DEBUG ] /dev/sdq : > [cephnode3][DEBUG ] /dev/sdq1 ceph data, active, cluster ceph, osd.9 > [cephnode3][DEBUG ] /dev/sdr : > [cephnode3][DEBUG ] /dev/sdr1 ceph data, active, cluster ceph, osd.16 > [cephnode3][DEBUG ] /dev/sds : > [cephnode3][DEBUG ] /dev/sds1 ceph data, active, cluster ceph, osd.10 > [cephnode3][DEBUG ] /dev/sdt : > [cephnode3][DEBUG ] /dev/sdt1 ceph data, active, cluster ceph, osd.11 > [cephnode3][DEBUG ] /dev/sdu : > [cephnode3][DEBUG ] /dev/sdu1 ceph data, active, cluster ceph, osd.12 > [cephnode3][DEBUG ] /dev/sdv : > [cephnode3][DEBUG ] /dev/sdv1 ceph data, active, cluster ceph, osd.17 > [cephnode3][DEBUG ] /dev/sdw : > [cephnode3][DEBUG ] /dev/sdw1 other, xfs, mounted on / > [cephnode3][DEBUG ] /dev/sdw2 swap, swap > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Christian Balzer Network/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/ _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
