Re: [ceph-users] stuck ceph-deploy mon create-initial / giant
Hi Loic, this is the content of our ceph.conf [global] fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 mon_initial_members = ceph1, ceph2, ceph3 mon_host = 192.168.10.107,192.168.10.108,192.168.10.109 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd pool default size = 2 public network = 192.168.10.0/24 cluster networt = 192.168.108.0/24 where public network is 10GBE native and cluster network a tagged VLAN on a lacp/802.3ad bond. Am Montag, den 23.02.2015, 23:24 +0100 schrieb Loic Dachary: Hi Stephan, Could you share the /etc/ceph/ceph.conf content ? Maybe ceph-create-keys cannot reach the monitor ? Cheers On 23/02/2015 22:53, Stephan Seitz wrote: Hi all, I'm currently facing a strange problem when deploying giant on Ubuntu 14.04. Following the docs, and reaching to ceph-deploy mon create-initial on the three mon hosts leads to every single mon stuck with a companion ceph-create-keys waiting forever. ceph-deploy quits after waiting for mon-qorum without success. Say: with error: for each mon. An strace shows repetively the very same bulk, with one line stating the mon state in pending. Network has been double-checked. netstat / lsof shows established TCP connection between the mons. Trying to fiddle around the quorum and starting the cluster with only one mon, didn't help either. It behaves in the very same way: Having the mon stuck and the ceph-create-keys waiting forever. Could someone please shed some light, what this problem could be caused by? Except for deploying the first time giant, I didn't face that problem on former installations. Thanks in advance, - Stephan -- Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-44 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] stuck ceph-deploy mon create-initial / giant
On 24/02/2015 09:58, Stephan Seitz wrote: Hi Loic, this is the content of our ceph.conf [global] fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 mon_initial_members = ceph1, ceph2, ceph3 mon_host = 192.168.10.107,192.168.10.108,192.168.10.109 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd pool default size = 2 public network = 192.168.10.0/24 cluster networt = 192.168.108.0/24 s/networt/network/ ? where public network is 10GBE native and cluster network a tagged VLAN on a lacp/802.3ad bond. Am Montag, den 23.02.2015, 23:24 +0100 schrieb Loic Dachary: Hi Stephan, Could you share the /etc/ceph/ceph.conf content ? Maybe ceph-create-keys cannot reach the monitor ? Cheers On 23/02/2015 22:53, Stephan Seitz wrote: Hi all, I'm currently facing a strange problem when deploying giant on Ubuntu 14.04. Following the docs, and reaching to ceph-deploy mon create-initial on the three mon hosts leads to every single mon stuck with a companion ceph-create-keys waiting forever. ceph-deploy quits after waiting for mon-qorum without success. Say: with error: for each mon. An strace shows repetively the very same bulk, with one line stating the mon state in pending. Network has been double-checked. netstat / lsof shows established TCP connection between the mons. Trying to fiddle around the quorum and starting the cluster with only one mon, didn't help either. It behaves in the very same way: Having the mon stuck and the ceph-create-keys waiting forever. Could someone please shed some light, what this problem could be caused by? Except for deploying the first time giant, I didn't face that problem on former installations. Thanks in advance, - Stephan -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] stuck ceph-deploy mon create-initial / giant
On Tue, 24 Feb 2015 11:17:22 +0100 Loic Dachary wrote: On 24/02/2015 09:58, Stephan Seitz wrote: Hi Loic, this is the content of our ceph.conf [global] fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 mon_initial_members = ceph1, ceph2, ceph3 mon_host = 192.168.10.107,192.168.10.108,192.168.10.109 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd pool default size = 2 public network = 192.168.10.0/24 cluster networt = 192.168.108.0/24 s/networt/network/ ? If this really should turn out to be the case, it is another painfully obvious reason why I proposed to provide full config parser output in the logs when in default debugging level. Stuff like this or the lack of picking up configuration statements would be immediately visible. Christian where public network is 10GBE native and cluster network a tagged VLAN on a lacp/802.3ad bond. Am Montag, den 23.02.2015, 23:24 +0100 schrieb Loic Dachary: Hi Stephan, Could you share the /etc/ceph/ceph.conf content ? Maybe ceph-create-keys cannot reach the monitor ? Cheers On 23/02/2015 22:53, Stephan Seitz wrote: Hi all, I'm currently facing a strange problem when deploying giant on Ubuntu 14.04. Following the docs, and reaching to ceph-deploy mon create-initial on the three mon hosts leads to every single mon stuck with a companion ceph-create-keys waiting forever. ceph-deploy quits after waiting for mon-qorum without success. Say: with error: for each mon. An strace shows repetively the very same bulk, with one line stating the mon state in pending. Network has been double-checked. netstat / lsof shows established TCP connection between the mons. Trying to fiddle around the quorum and starting the cluster with only one mon, didn't help either. It behaves in the very same way: Having the mon stuck and the ceph-create-keys waiting forever. Could someone please shed some light, what this problem could be caused by? Except for deploying the first time giant, I didn't face that problem on former installations. Thanks in advance, - Stephan -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] stuck ceph-deploy mon create-initial / giant
On 24/02/2015 12:00, Christian Balzer wrote: On Tue, 24 Feb 2015 11:17:22 +0100 Loic Dachary wrote: On 24/02/2015 09:58, Stephan Seitz wrote: Hi Loic, this is the content of our ceph.conf [global] fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 mon_initial_members = ceph1, ceph2, ceph3 mon_host = 192.168.10.107,192.168.10.108,192.168.10.109 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd pool default size = 2 public network = 192.168.10.0/24 cluster networt = 192.168.108.0/24 s/networt/network/ ? If this really should turn out to be the case, it is another painfully obvious reason why I proposed to provide full config parser output in the logs when in default debugging level. I agree. However, it is non trivial to implement because there is not a central place where all valid values are defined. It is also likely that third party scripts rely on the fact that arbitrary key/values can be stored in the configuration file. Stuff like this or the lack of picking up configuration statements would be immediately visible. That's probably not the reason why it fails. Do you confirm ? Christian where public network is 10GBE native and cluster network a tagged VLAN on a lacp/802.3ad bond. Am Montag, den 23.02.2015, 23:24 +0100 schrieb Loic Dachary: Hi Stephan, Could you share the /etc/ceph/ceph.conf content ? Maybe ceph-create-keys cannot reach the monitor ? Cheers On 23/02/2015 22:53, Stephan Seitz wrote: Hi all, I'm currently facing a strange problem when deploying giant on Ubuntu 14.04. Following the docs, and reaching to ceph-deploy mon create-initial on the three mon hosts leads to every single mon stuck with a companion ceph-create-keys waiting forever. ceph-deploy quits after waiting for mon-qorum without success. Say: with error: for each mon. An strace shows repetively the very same bulk, with one line stating the mon state in pending. Network has been double-checked. netstat / lsof shows established TCP connection between the mons. Trying to fiddle around the quorum and starting the cluster with only one mon, didn't help either. It behaves in the very same way: Having the mon stuck and the ceph-create-keys waiting forever. Could someone please shed some light, what this problem could be caused by? Except for deploying the first time giant, I didn't face that problem on former installations. Thanks in advance, - Stephan -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] stuck ceph-deploy mon create-initial / giant
On 02/24/2015 09:06 PM, Loic Dachary wrote: On 24/02/2015 12:00, Christian Balzer wrote: On Tue, 24 Feb 2015 11:17:22 +0100 Loic Dachary wrote: On 24/02/2015 09:58, Stephan Seitz wrote: Hi Loic, this is the content of our ceph.conf [global] fsid = 719f14b2-7475-4b25-8c5f-3ffbcf594d13 mon_initial_members = ceph1, ceph2, ceph3 mon_host = 192.168.10.107,192.168.10.108,192.168.10.109 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd pool default size = 2 public network = 192.168.10.0/24 cluster networt = 192.168.108.0/24 s/networt/network/ ? If this really should turn out to be the case, it is another painfully obvious reason why I proposed to provide full config parser output in the logs when in default debugging level. I agree. However, it is non trivial to implement because there is not a central place where all valid values are defined. It is also likely that third party scripts rely on the fact that arbitrary key/values can be stored in the configuration file. This could be implemented as a warning then? Possible invalid or 3rd party entry found: X. Please check your ceph.conf file. People can then report false positives and they an be added to the list of known values? Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] stuck ceph-deploy mon create-initial / giant
Hi Stephan, Could you share the /etc/ceph/ceph.conf content ? Maybe ceph-create-keys cannot reach the monitor ? Cheers On 23/02/2015 22:53, Stephan Seitz wrote: Hi all, I'm currently facing a strange problem when deploying giant on Ubuntu 14.04. Following the docs, and reaching to ceph-deploy mon create-initial on the three mon hosts leads to every single mon stuck with a companion ceph-create-keys waiting forever. ceph-deploy quits after waiting for mon-qorum without success. Say: with error: for each mon. An strace shows repetively the very same bulk, with one line stating the mon state in pending. Network has been double-checked. netstat / lsof shows established TCP connection between the mons. Trying to fiddle around the quorum and starting the cluster with only one mon, didn't help either. It behaves in the very same way: Having the mon stuck and the ceph-create-keys waiting forever. Could someone please shed some light, what this problem could be caused by? Except for deploying the first time giant, I didn't face that problem on former installations. Thanks in advance, - Stephan -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] stuck ceph-deploy mon create-initial / giant
Hi Stephan, I've just had a very similar problem today. It turned out that the problem was the mgmt network which I use to manage the nodes is different to the public network. I had created host entries resolving to the mgmt network ip's, so when initial create was running it was trying to connect on the mgmt IP and not the public network IP. I'm wondering if you are experiencing a similar problem? Nick -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Loic Dachary Sent: 23 February 2015 22:25 To: Stephan Seitz; ceph-users@lists.ceph.com Subject: Re: [ceph-users] stuck ceph-deploy mon create-initial / giant Hi Stephan, Could you share the /etc/ceph/ceph.conf content ? Maybe ceph-create-keys cannot reach the monitor ? Cheers On 23/02/2015 22:53, Stephan Seitz wrote: Hi all, I'm currently facing a strange problem when deploying giant on Ubuntu 14.04. Following the docs, and reaching to ceph-deploy mon create-initial on the three mon hosts leads to every single mon stuck with a companion ceph-create-keys waiting forever. ceph-deploy quits after waiting for mon-qorum without success. Say: with error: for each mon. An strace shows repetively the very same bulk, with one line stating the mon state in pending. Network has been double-checked. netstat / lsof shows established TCP connection between the mons. Trying to fiddle around the quorum and starting the cluster with only one mon, didn't help either. It behaves in the very same way: Having the mon stuck and the ceph-create-keys waiting forever. Could someone please shed some light, what this problem could be caused by? Except for deploying the first time giant, I didn't face that problem on former installations. Thanks in advance, - Stephan -- Loïc Dachary, Artisan Logiciel Libre ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
