Thanks Dan, I have raised the tracker for this issue http://tracker.ceph.com/issues/15176
On Thu, Mar 17, 2016 at 10:47 AM, Dan van der Ster <d...@vanderster.com> wrote: > Hi, > > It's true, partprobe works intermittently. I extracted the key > commands to show the problem: > > [18:44]# /usr/sbin/sgdisk --new=2:0:20480M --change-name=2:'ceph > journal' --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d > --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- > /dev/sdc > The operation has completed successfully. > [18:44]# partprobe /dev/sdc > Error: Error informing the kernel about modifications to partition > /dev/sdc2 -- Device or resource busy. This means Linux won't know > about any changes you made to /dev/sdc2 until you reboot -- so you > shouldn't mount it or use it in any way before rebooting. > Error: Failed to add partition 2 (Device or resource busy) > [18:44]# partprobe /dev/sdc > [18:44]# partprobe /dev/sdc > Error: Error informing the kernel about modifications to partition > /dev/sdc2 -- Device or resource busy. This means Linux won't know > about any changes you made to /dev/sdc2 until you reboot -- so you > shouldn't mount it or use it in any way before rebooting. > Error: Failed to add partition 2 (Device or resource busy) > [18:44]# partprobe /dev/sdc > Error: Error informing the kernel about modifications to partition > /dev/sdc2 -- Device or resource busy. This means Linux won't know > about any changes you made to /dev/sdc2 until you reboot -- so you > shouldn't mount it or use it in any way before rebooting. > Error: Failed to add partition 2 (Device or resource busy) > > But partx works every time: > > [18:46]# /usr/sbin/sgdisk --new=2:0:20480M --change-name=2:'ceph > journal' --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d > --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- > /dev/sdd > The operation has completed successfully. > [18:46]# partx -u /dev/sdd > [18:46]# partx -u /dev/sdd > [18:46]# partx -u /dev/sdd > [18:46]# > > -- Dan > > On Thu, Mar 17, 2016 at 6:31 PM, Vasu Kulkarni <vakul...@redhat.com> > wrote: > > I can raise a tracker for this issue since it looks like an intermittent > > issue and mostly dependent on specific hardware or it would be better if > you > > add all the hardware/os details in tracker.ceph.com, also from your > logs it > > looks like you have > > Resource busy issue: Error: Failed to add partition 2 (Device or > resource > > busy) > > > > From my test run logs on centos 7.2 , 10.0.5 ( > > > http://qa-proxy.ceph.com/teuthology/vasu-2016-03-15_15:34:41-selinux-master---basic-mira/62626/teuthology.log > > ) > > > > 2016-03-15T18:49:56.305 > > INFO:teuthology.orchestra.run.mira041.stderr:[ceph_deploy.osd][DEBUG ] > > Preparing host mira041 disk /dev/sdb journal None activate True > > 2016-03-15T18:49:56.305 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][DEBUG ] find the > > location of an executable > > 2016-03-15T18:49:56.309 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][INFO ] Running > > command: sudo /usr/sbin/ceph-disk -v prepare --cluster ceph --fs-type > xfs -- > > /dev/sdb > > 2016-03-15T18:49:56.546 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-osd --cluster=ceph > --show-config-value=fsid > > 2016-03-15T18:49:56.611 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --cluster > > ceph > > 2016-03-15T18:49:56.643 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --cluster > ceph > > 2016-03-15T18:49:56.708 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --cluster > ceph > > 2016-03-15T18:49:56.708 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:56.709 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] set_type: > > Will colocate journal with data on /dev/sdb > > 2016-03-15T18:49:56.709 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-osd --cluster=ceph > > --show-config-value=osd_journal_size > > 2016-03-15T18:49:56.774 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:56.774 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:56.775 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:56.775 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup > > osd_mkfs_options_xfs > > 2016-03-15T18:49:56.777 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup > > osd_fs_mkfs_options_xfs > > 2016-03-15T18:49:56.809 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup > > osd_mount_options_xfs > > 2016-03-15T18:49:56.841 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup > > osd_fs_mount_options_xfs > > 2016-03-15T18:49:56.857 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:56.858 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:56.858 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > ptype_tobe_for_name: name = journal > > 2016-03-15T18:49:56.859 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:56.859 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > create_partition: Creating journal partition num 2 size 5120 on /dev/sdb > > 2016-03-15T18:49:56.859 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M > > --change-name=2:ceph journal > > --partition-guid=2:d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198 > > --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdb > > 2016-03-15T18:49:57.927 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][DEBUG ] The > operation > > has completed successfully. > > 2016-03-15T18:49:57.927 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > update_partition: Calling partprobe on created device /dev/sdb > > 2016-03-15T18:49:57.928 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > command_check_call: Running command: /usr/bin/udevadm settle > --timeout=600 > > 2016-03-15T18:49:58.393 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] command: > > Running command: /sbin/partprobe /dev/sdb > > 2016-03-15T18:49:58.393 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > command_check_call: Running command: /usr/bin/udevadm settle > --timeout=600 > > 2016-03-15T18:49:59.109 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:59.203 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:59.203 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb2 uuid path is /sys/dev/block/8:18/dm/uuid > > 2016-03-15T18:49:59.204 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > prepare_device: Journal is GPT partition > > /dev/disk/by-partuuid/d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198 > > 2016-03-15T18:49:59.204 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > prepare_device: Journal is GPT partition > > /dev/disk/by-partuuid/d4b2fa8d-3f2a-4ce9-a2fe-2a3872d7e198 > > 2016-03-15T18:49:59.204 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:59.205 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > set_data_partition: Creating osd partition on /dev/sdb > > 2016-03-15T18:49:59.205 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:59.205 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > ptype_tobe_for_name: name = data > > 2016-03-15T18:49:59.206 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > get_dm_uuid: > > get_dm_uuid /dev/sdb uuid path is /sys/dev/block/8:16/dm/uuid > > 2016-03-15T18:49:59.206 > > INFO:teuthology.orchestra.run.mira041.stderr:[mira041][WARNING] > > create_partition: Creating data partition num 1 size 0 on /dev/sdb > > > > > > > > On Thu, Mar 17, 2016 at 8:06 AM, Dan van der Ster <d...@vanderster.com> > > wrote: > >> > >> Hi, > >> > >> Is there a tracker for this? We just hit the same problem on 10.0.5. > >> > >> Cheers, Dan > >> > >> # rpm -q ceph > >> ceph-10.0.5-0.el7.x86_64 > >> > >> # cat /etc/redhat-release > >> CentOS Linux release 7.2.1511 (Core) > >> > >> # ceph-disk -v prepare /dev/sdc > >> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > >> /sys/dev/block/8:32/dm/uuid > >> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > >> /sys/dev/block/8:32/dm/uuid > >> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > >> /sys/dev/block/8:32/dm/uuid > >> INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph > >> --show-config-value=fsid > >> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > >> --name=osd. --lookup osd_mkfs_type > >> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > >> --name=osd. --lookup osd_mkfs_options_xfs > >> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > >> --name=osd. --lookup osd_fs_mkfs_options_xfs > >> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > >> --name=osd. --lookup osd_mount_options_xfs > >> INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph > >> --show-config-value=osd_journal_size > >> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > >> --name=osd. --lookup osd_cryptsetup_parameters > >> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > >> --name=osd. --lookup osd_dmcrypt_key_size > >> INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph > >> --name=osd. --lookup osd_dmcrypt_type > >> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > >> /sys/dev/block/8:32/dm/uuid > >> INFO:ceph-disk:Will colocate journal with data on /dev/sdc > >> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > >> /sys/dev/block/8:32/dm/uuid > >> DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is > >> /sys/dev/block/8:32/dm/uuid > >> DEBUG:ceph-disk:Creating journal partition num 2 size 20480 on /dev/sdc > >> INFO:ceph-disk:Running command: /usr/sbin/sgdisk --new=2:0:20480M > >> --change-name=2:ceph journal > >> --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d > >> --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- > >> /dev/sdc > >> The operation has completed successfully. > >> DEBUG:ceph-disk:Calling partprobe on prepared device /dev/sdc > >> INFO:ceph-disk:Running command: /usr/bin/udevadm settle > >> INFO:ceph-disk:Running command: /usr/sbin/partprobe /dev/sdc > >> Error: Error informing the kernel about modifications to partition > >> /dev/sdc2 -- Device or resource busy. This means Linux won't know > >> about any changes you made to /dev/sdc2 until you reboot -- so you > >> shouldn't mount it or use it in any way before rebooting. > >> Error: Failed to add partition 2 (Device or resource busy) > >> Traceback (most recent call last): > >> File "/usr/sbin/ceph-disk", line 3528, in <module> > >> main(sys.argv[1:]) > >> File "/usr/sbin/ceph-disk", line 3482, in main > >> args.func(args) > >> File "/usr/sbin/ceph-disk", line 1817, in main_prepare > >> luks=luks > >> File "/usr/sbin/ceph-disk", line 1447, in prepare_journal > >> return prepare_journal_dev(data, journal, journal_size, > >> journal_uuid, journal_dm_keypath, cryptsetup_parameters, luks) > >> File "/usr/sbin/ceph-disk", line 1401, in prepare_journal_dev > >> raise Error(e) > >> __main__.Error: Error: Command '['/usr/sbin/partprobe', '/dev/sdc']' > >> returned non-zero exit status 1 > >> > >> On Tue, Mar 15, 2016 at 8:38 PM, Vasu Kulkarni <vakul...@redhat.com> > >> wrote: > >> > Thanks for the steps that should be enough to test it out, I hope you > >> > got > >> > the latest ceph-deploy either from pip or throught github. > >> > > >> > On Tue, Mar 15, 2016 at 12:29 PM, Stephen Lord < > steve.l...@quantum.com> > >> > wrote: > >> >> > >> >> I would have to nuke my cluster right now, and I do not have a spare > >> >> one.. > >> >> > >> >> The procedure though is literally this, given a 3 node redhat 7.2 > >> >> cluster, > >> >> ceph00, ceph01 and ceph02 > >> >> > >> >> ceph-deploy install --testing ceph00 ceph01 ceph02 > >> >> ceph-deploy new ceph00 ceph01 ceph02 > >> >> > >> >> ceph-deploy mon create ceph00 ceph01 ceph02 > >> >> ceph-deploy gatherkeys ceph00 > >> >> > >> >> ceph-deploy osd create ceph00:sdb:/dev/sdi > >> >> ceph-deploy osd create ceph00:sdc:/dev/sdi > >> >> > >> >> All devices have their partition tables wiped before this. They are > all > >> >> just SATA devices, no special devices in the way. > >> >> > >> >> sdi is an ssd and it is being carved up for journals. The first osd > >> >> create > >> >> works, the second one gets stuck in a loop in the update_partition > call > >> >> in > >> >> ceph_disk for the 5 iterations before it gives up. When I look in > >> >> /sys/block/sdi the partition for the first osd is visible, the one > for > >> >> the > >> >> second is not. However looking at /proc/partitions it sees the > correct > >> >> thing. So something about partprobe is not kicking udev into doing > the > >> >> right > >> >> thing when the second partition is added I suspect. > >> >> > >> >> If I do not use the separate journal device then it usually works, > but > >> >> occasionally I see a single retry in that same loop. > >> >> > >> >> There is code in ceph_deploy which uses partprobe or partx depending > on > >> >> which distro it detects, that is how I worked out what to change > here. > >> >> > >> >> If I have to tear things down again I will reproduce and post here. > >> >> > >> >> Steve > >> >> > >> >> > On Mar 15, 2016, at 2:12 PM, Vasu Kulkarni <vakul...@redhat.com> > >> >> > wrote: > >> >> > > >> >> > Do you mind giving the full failed logs somewhere in fpaste.org > along > >> >> > with some os version details? > >> >> > There are some known issues on RHEL, If you use 'osd prepare' and > >> >> > 'osd > >> >> > activate'(specifying just the journal partition here) it might work > >> >> > better. > >> >> > > >> >> > On Tue, Mar 15, 2016 at 12:05 PM, Stephen Lord > >> >> > <steve.l...@quantum.com> > >> >> > wrote: > >> >> > Not multipath if you mean using the multipath driver, just trying > to > >> >> > setup OSDs which use a data disk and a journal ssd. If I run just a > >> >> > disk > >> >> > based OSD and only specify one device to ceph-deploy then it > usually > >> >> > works > >> >> > although sometimes has to retry. In the case where I am using it to > >> >> > carve an > >> >> > SSD into several partitions for journals it fails on the second > one. > >> >> > > >> >> > Steve > >> >> > > >> >> > >> >> > >> >> > ---------------------------------------------------------------------- > >> >> The information contained in this transmission may be confidential. > Any > >> >> disclosure, copying, or further distribution of confidential > >> >> information is > >> >> not permitted unless such privilege is explicitly granted in writing > by > >> >> Quantum. Quantum reserves the right to have electronic > communications, > >> >> including email and attachments, sent across its networks filtered > >> >> through > >> >> anti virus and spam software programs and retain such messages in > order > >> >> to > >> >> comply with applicable data security and retention requirements. > >> >> Quantum is > >> >> not responsible for the proper and complete transmission of the > >> >> substance of > >> >> this communication or for any delay in its receipt. > >> > > >> > > >> > > >> > _______________________________________________ > >> > ceph-users mailing list > >> > ceph-users@lists.ceph.com > >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
