On Thu, Feb 1, 2018 at 11:19 AM, Andre Goree <an...@drenet.net> wrote: > On 2018/02/01 10:58 am, Alfredo Deza wrote: >> >> On Thu, Feb 1, 2018 at 10:44 AM, Andre Goree <an...@drenet.net> wrote: >>> >>> Does ceph-volume support lvm+Bluestore? >> >> >> Yes it does. Since version 12.2.2 which seems to be what you have. >> >>> I ask this, bc I'm trying to use >>> ceph-ansible to provision an OSD node, and am getting a hang when the >>> 'ceph-volume create' command is trying to be run. I'm also using >>> ceph-ansible/master (not stable-3.0 or any other stable version), which >>> has >>> the parameters for bluestore+lvm, but may not actually be implemented in >>> 12.2.2. ceph-ansible stable-3.0 branch does NOT contain a section for >>> bluestore in the ceph-volume/lvm section. >> >> >> Correct, stable-3.0 doesn't have it. >> >>> >>> Here you can see the full command(s) being run (via ansible), and an >>> strace >>> on that process. I've left that process running overnight and it appears >>> it >>> is still timing out (apologies if I should've used pastebin): >> >> >> This is not really ceph-volume. It is trying to communicate with the >> cluster but it is not going through. Something in your configuration >> is not quite right, so the command is hanging. >> >> This is the actual command: >> >> /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring >> /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new >> a2ee64a4-b5ba-4ca9-8528-4205f3ad8c99 >> >> What that command is trying to do is to tell the monitor about the >> newly created OSD. It is easy to replicate this "hanging" problem if >> you modify your ceph.conf to point to an invalid IP for >> the monitors. >> >> > > Thank you for confirming that and pointing me in the right direction! > > It would appear my network configuration is certainly correct (from my > understanding; "public" network is 172.16.238.0/24, cluster network is > 172.16.239.0/24 -- a configuration that works for the other OSDs built with > ceph-ansible/ceph-disk) and I can reach port 6789 on my MON node: > > ~# ping -c4 172.16.238.11 && ping -c4 172.16.239.11 > PING 172.16.238.11 (172.16.238.11) 56(84) bytes of data. > 64 bytes from 172.16.238.11: icmp_seq=1 ttl=64 time=0.141 ms > 64 bytes from 172.16.238.11: icmp_seq=2 ttl=64 time=0.102 ms > 64 bytes from 172.16.238.11: icmp_seq=3 ttl=64 time=0.107 ms > 64 bytes from 172.16.238.11: icmp_seq=4 ttl=64 time=0.096 ms > > --- 172.16.238.11 ping statistics --- > 4 packets transmitted, 4 received, 0% packet loss, time 2999ms > rtt min/avg/max/mdev = 0.096/0.111/0.141/0.020 ms > PING 172.16.239.11 (172.16.239.11) 56(84) bytes of data. > 64 bytes from 172.16.239.11: icmp_seq=1 ttl=64 time=0.252 ms > 64 bytes from 172.16.239.11: icmp_seq=2 ttl=64 time=0.133 ms > 64 bytes from 172.16.239.11: icmp_seq=3 ttl=64 time=0.098 ms > 64 bytes from 172.16.239.11: icmp_seq=4 ttl=64 time=0.103 ms > > --- 172.16.239.11 ping statistics --- > 4 packets transmitted, 4 received, 0% packet loss, time 2998ms > rtt min/avg/max/mdev = 0.098/0.146/0.252/0.063 ms > ~# telnet 172.16.238.11 6789 > Trying 172.16.238.11... > Connected to 172.16.238.11. > Escape character is '^]'. > ceph v027???^?^]quit > > telnet> quit > Connection closed. > > > Is there a command you'd recommend I use to try to ensure connectivity to > the MON node from this new OSD node to perhaps help troubleshoot this issue > I'm having?
You need to make sure you are correlating your network interactions with the same values Ceph is configured with. Like in my example before, it is easy to replicate if you have an incorrect IP in your ceph.conf This might be 10.0.0.1 and you are pinging 10.0.1.0 and it works, but ceph is using the incorrect one :) I don't have a specific command that might get you closer. I would go through the mon and osd troubleshooting guides http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-mon/ http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/ > > In the meantime, I'm going over my ceph-ansible configuration(s) again to > ensure I didn't miss something. > > Thank you for your guidance Alfredo, it's been invaluable. > > > > -- > Andre Goree > -=-=-=-=-=- > Email - andre at drenet.net > Website - http://blog.drenet.net > PGP key - http://www.drenet.net/pubkey.html > -=-=-=-=-=- _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
