Hi, On 17/12/2015 07:53, Jesper Thorhauge wrote: > Hi, > > Some more information showing in the boot.log; > > 2015-12-16 07:35:33.289830 7f1b990ad800 -1 > filestore(/var/lib/ceph/tmp/mnt.aWZTcE) mkjournal error creating journal on > /var/lib/ceph/tmp/mnt.aWZTcE/journal: (22) Invalid argument > 2015-12-16 07:35:33.289842 7f1b990ad800 -1 OSD::mkfs: ObjectStore::mkfs > failed with error -22 > 2015-12-16 07:35:33.289883 7f1b990ad800 -1 ** ERROR: error creating empty > object store in /var/lib/ceph/tmp/mnt.aWZTcE: (22) Invalid argument > ERROR:ceph-disk:Failed to activate > ceph-disk: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', > '--mkkey', '-i', '7', '--monmap', > '/var/lib/ceph/tmp/mnt.aWZTcE/activate.monmap', '--osd-data', > '/var/lib/ceph/tmp/mnt.aWZTcE', '--osd-journal', > '/var/lib/ceph/tmp/mnt.aWZTcE/journal', '--osd-uuid', > 'c83b5aa5-fe77-42f6-9415-25ca0266fb7f', '--keyring', > '/var/lib/ceph/tmp/mnt.aWZTcE/keyring']' returned non-zero exit status 1 > ceph-disk: Error: One or more partitions failed to activate > > Maybe related to the "(22) Invalid argument" part..?
After a reboot the symlinks are reconstructed and if they are still incorrect, it means there is an inconsistency somewhere else. To debug the problem, could you mount /dev/sda1 and verify the symlink of the journal file ? Then verify the content of /dev/disk/by-partuuid. And also display the partition information with sgdisk -i 1 /dev/sda and sgdisk -i 2 /dev/sda. Are you collocating your journal with the data, on the same disk ? Or are they on two different disks ? git log --no-merges --oneline tags/v0.94.3..tags/v0.94.5 udev shows nothing, meaning there has been no change to udev rules. There is one change related to the installation of the udev rules https://github.com/ceph/ceph/commit/4eb58ad2027148561d94bb43346b464b55d041a6. Could you double check 60-ceph-partuuid-workaround.rules is installed where it should ? Cheers > > /Jesper > > ********************* > > Hi, > > I have done several reboots, and it did not lead to healthy symlinks :-( > > /Jesper > > ************ > > Hi, > > On 16/12/2015 07:39, Jesper Thorhauge wrote: >> Hi, >> >> A fresh server install on one of my nodes (and yum update) left me with >> CentOS 6.7 / Ceph 0.94.5. All the other nodes are running Ceph 0.94.2. >> >> "ceph-disk prepare /dev/sda /dev/sdc" seems to work as expected, but >> "ceph-disk activate / dev/sda1" fails. I have traced the problem to >> "/dev/disk/by-partuuid", where the journal symlinks are broken; >> >> -rw-r--r-- 1 root root 0 Dec 16 07:35 1e9d527f-0866-4284-b77c-c1cb04c5a168 >> -rw-r--r-- 1 root root 0 Dec 16 07:35 c34d4694-b486-450d-b57f-da24255f0072 >> lrwxrwxrwx 1 root root 10 Dec 16 07:35 c83b5aa5-fe77-42f6-9415-25ca0266fb7f >> -> ../../sdb1 >> lrwxrwxrwx 1 root root 10 Dec 16 07:35 e85f4d92-c8f1-4591-bd2a-aa43b80f58f6 >> -> ../../sda1 >> >> Re-creating them manually wont survive a reboot. Is this a problem with the >> udev rules in Ceph 0.94.3+? > > This usually is a symptom of something else going wrong (i.e. it is possible > to confuse the kernel into creating the wrong symbolic links). The correct > symlinks should be set when you reboot. > >> Hope that somebody can help me :-) > > Please let us know if rebooting leads to healthy symlinks. > > Cheers >> >> Thanks! >> >> Best regards, >> Jesper >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > -- > Loïc Dachary, Artisan Logiciel Libre > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Loïc Dachary, Artisan Logiciel Libre
signature.asc
Description: OpenPGP digital signature
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
