On Thu, Jul 6, 2017 at 2:43 PM, Ilya Dryomov <idryo...@gmail.com> wrote:
> On Thu, Jul 6, 2017 at 2:23 PM, Stanislav Kopp <stask...@gmail.com> wrote:
>> 2017-07-06 14:16 GMT+02:00 Ilya Dryomov <idryo...@gmail.com>:
>>> On Thu, Jul 6, 2017 at 1:28 PM, Stanislav Kopp <stask...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> 2017-07-05 20:31 GMT+02:00 Ilya Dryomov <idryo...@gmail.com>:
>>>>> On Wed, Jul 5, 2017 at 7:55 PM, Stanislav Kopp <stask...@gmail.com> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I have problem that sometimes I can't unmap rbd device, I get "sysfs
>>>>>> write failed rbd: unmap failed: (16) Device or resource busy", there
>>>>>> is no open files and "holders" directory is empty. I saw on the
>>>>>> mailling list that you can "force" unmapping the device, but I cant
>>>>>> find how does it work. "man rbd" only mentions "force" as "KERNEL RBD
>>>>>> (KRBD) OPTION", but "modinfo rbd" doesn't show this option. Did I miss
>>>>>> something?
>>>>>
>>>>> Forcing unmap on an open device is not a good idea. I'd suggest
>>>>> looking into what's holding the device and fixing that instead.
>>>>
>>>> We use pacemaker's resource agent for rbd mount/unmount
>>>> (https://github.com/ceph/ceph/blob/master/src/ocf/rbd.in)
>>>> I've reproduced the failure again and now saw in ps output that there
>>>> is still unmout fs process in D state:
>>>>
>>>> root 29320 0.0 0.0 21980 1272 ? D 09:18 0:00
>>>> umount /export/rbd1
>>>>
>>>> this explains rbd unmap problem, but strange enough I don't see this
>>>> mount in /proc/mounts, so it looks like it was successfully unmounted,
>>>> if I try to strace the "umount" procces it hung (the strace, with no
>>>> output), looks like kernel problem? Do you have some tips for further
>>>> debugging?
>>>
>>> Check /sys/kernel/debug/ceph/<cluster-fsid.client-id>/osdc. It lists
>>> in-flight requests, that's what umount is blocked on.
>>
>> I see this in my output, but don't know what does it means honestly:
>>
>> root@nfs-test01:~# cat
>> /sys/kernel/debug/ceph/4f23f683-21e6-49f3-ae2c-c95b150b9dc6.client138566/osdc
>> REQUESTS 2 homeless 0
>> 658 osd9 0.75514984 [9,1,6]/9 [9,1,6]/9
>> rbd_data.6e28c6b8b4567.0000000000000000 0x400024 10'0
>> set-alloc-hint,write
>> 659 osd15 0.40f1ea02 [15,7,9]/15 [15,7,9]/15
>> rbd_data.6e28c6b8b4567.0000000000000001 0x400024 10'0
>> set-alloc-hint,write
>
> It means you have two pending writes (OSD requests), to osd9 and osd15.
> What is the output of
>
> $ ceph -s
> $ ceph pg dump pgs_brief
Stanislav and I tracked this down to a pacemaker misconfiguration:
"... the problem was wrong netmask, we use /22 for this network, but
primary interface of machine was configured with /23 and the VIP even
with /24, because of that, the VIP was often the first interface
which caused unmount problem, because it was stopped as first resource
in fail-over situation. After fixing netmask for both VIP and
machine's IP it's working without issues (at least it never fails
since)."
Thanks,
Ilya
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
