[ceph-users] [Troubleshooting] I have a watcher I can't get rid of...
I'm having a hard time removing an RBD that I no longer need. # rbd rm / 2016-08-03 15:00:01.085784 7ff9dfc997c0 -1 librbd: image has watchers - not removing Removing image: 0% complete...failed. rbd: error: image still has watchers This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. So, I use `rbd status` to identify the watcher: # rbd status / Watchers: watcher=:0/705293879 client.1076985 cookie=1 I log onto that host, and did # rbd showmapped which returns nothing I don't use snapshot and I don't use cloning, so, there shouldn't be any image sharing. I ended up rebooting that host and the watcher is still around, and my problem persist: I can't remove the RBD. At this point, I'm all out of ideas on how to troubleshoot this problem. I'm running infernalis: # ceph --version ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd) in my set up, on CentOS 7.2 hosts # uname -r 3.10.0-327.22.2.el7.x86_64 I appreciate any assistance, -kc K.C. Wong kcw...@verseon.com 4096R/B8995EDE E527 CBE8 023E 79EA 8BBB 5C77 23A6 92E9 B899 5EDE hkps://hkps.pool.sks-keyservers.net signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [Troubleshooting] I have a watcher I can't get rid of...
If the client is no longer running the watch should expire within 30 seconds. If you are still experiencing this issue, you can blacklist the mystery client via "ceph osd blacklist add". On Wed, Aug 3, 2016 at 6:06 PM, K.C. Wong wrote: > I'm having a hard time removing an RBD that I no longer need. > > # rbd rm / > 2016-08-03 15:00:01.085784 7ff9dfc997c0 -1 librbd: image has watchers - not > removing > Removing image: 0% complete...failed. > rbd: error: image still has watchers > This means the image is still open or the client using it crashed. Try again > after closing/unmapping it or waiting 30s for the crashed client to timeout. > > So, I use `rbd status` to identify the watcher: > > # rbd status / > Watchers: > watcher=:0/705293879 client.1076985 cookie=1 > > I log onto that host, and did > > # rbd showmapped > > which returns nothing > > I don't use snapshot and I don't use cloning, so, there shouldn't > be any image sharing. I ended up rebooting that host and the > watcher is still around, and my problem persist: I can't remove > the RBD. > > At this point, I'm all out of ideas on how to troubleshoot this > problem. I'm running infernalis: > > # ceph --version > ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd) > > in my set up, on CentOS 7.2 hosts > > # uname -r > 3.10.0-327.22.2.el7.x86_64 > > I appreciate any assistance, > > -kc > > K.C. Wong > kcw...@verseon.com > 4096R/B8995EDE E527 CBE8 023E 79EA 8BBB 5C77 23A6 92E9 B899 5EDE > hkps://hkps.pool.sks-keyservers.net > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Jason ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [Troubleshooting] I have a watcher I can't get rid of...
Thank you, Jason. While I can't find the culprit for the watcher (the watcher never expired, and survived a reboot. udev, maybe?), blacklisting the host did allow me to remove the device. Much appreciated, -kc > On Aug 4, 2016, at 4:50 AM, Jason Dillaman wrote: > > If the client is no longer running the watch should expire within 30 > seconds. If you are still experiencing this issue, you can blacklist > the mystery client via "ceph osd blacklist add". > > On Wed, Aug 3, 2016 at 6:06 PM, K.C. Wong wrote: >> I'm having a hard time removing an RBD that I no longer need. >> >> # rbd rm / >> 2016-08-03 15:00:01.085784 7ff9dfc997c0 -1 librbd: image has watchers - not >> removing >> Removing image: 0% complete...failed. >> rbd: error: image still has watchers >> This means the image is still open or the client using it crashed. Try again >> after closing/unmapping it or waiting 30s for the crashed client to timeout. >> >> So, I use `rbd status` to identify the watcher: >> >> # rbd status / >> Watchers: >>watcher=:0/705293879 client.1076985 cookie=1 >> >> I log onto that host, and did >> >> # rbd showmapped >> >> which returns nothing >> >> I don't use snapshot and I don't use cloning, so, there shouldn't >> be any image sharing. I ended up rebooting that host and the >> watcher is still around, and my problem persist: I can't remove >> the RBD. >> >> At this point, I'm all out of ideas on how to troubleshoot this >> problem. I'm running infernalis: >> >> # ceph --version >> ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd) >> >> in my set up, on CentOS 7.2 hosts >> >> # uname -r >> 3.10.0-327.22.2.el7.x86_64 >> >> I appreciate any assistance, >> >> -kc >> >> K.C. Wong >> kcw...@verseon.com >> 4096R/B8995EDE E527 CBE8 023E 79EA 8BBB 5C77 23A6 92E9 B899 5EDE >> hkps://hkps.pool.sks-keyservers.net >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > > -- > Jason K.C. Wong kcw...@verseon.com 4096R/B8995EDE E527 CBE8 023E 79EA 8BBB 5C77 23A6 92E9 B899 5EDE hkps://hkps.pool.sks-keyservers.net signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] [Troubleshooting] I have a watcher I can't get rid of...
On Thu, Aug 4, 2016 at 10:44 PM, K.C. Wong wrote: > Thank you, Jason. > > While I can't find the culprit for the watcher (the watcher never expired, > and survived a reboot. udev, maybe?), blacklisting the host did allow me > to remove the device. It survived a reboot because watch state is persisted on the OSDs. This sounds like an OSD bug where the watcher wasn't cleaned up or timed out after 30 seconds. Was that image unmapped gracefully? Thanks, Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com