On Sat, 14 Mar 2015, Georgios Dimitrakakis wrote: > This is the message that is flooding the ceph-mon.log now: > > > 2015-03-14 08:16:39.286823 7f9f6920b700 1 > mon.fu@0(electing).elector(1) init, last seen epoch 1 > 2015-03-14 08:16:42.736674 7f9f6880a700 1 mon.fu@0(electing) e2 > adding peer 15.12.6.21:6789/0 to list of hints > 2015-03-14 08:16:42.737891 7f9f6880a700 1 > mon.fu@0(electing).elector(1) discarding election message: > 15.12.6.21:6789/0 > not in my monmap e2: 2 mons at > {fu=192.168.1.100:6789/0,jin=192.168.1.101:6789/0}
It sounds like you need to follow some variation of this procedure: http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/#removing-monitors-from-an-unhealthy-cluster ..although it may be that simply killing the daemon running on 15.12.6.21 and restarting the other mon daemons will be enough. If not, the procedure linked above will let tyou remove all traces of it and get things up again. Not quite sure where things went awry but I assume the priority is to get things working first and figure that out later! sage > > > > George > > > > This is the log for monitor (ceph-mon.log) when I try to restart the > > monitor: > > > > > > 2015-03-14 07:47:26.384561 7f1f1dc0f700 -1 mon.fu@0(probing) e2 *** > > Got Signal Terminated *** > > 2015-03-14 07:47:26.384593 7f1f1dc0f700 1 mon.fu@0(probing) e2 > > shutdown > > 2015-03-14 07:47:26.384654 7f1f1dc0f700 0 quorum service shutdown > > 2015-03-14 07:47:26.384657 7f1f1dc0f700 0 > > mon.fu@0(shutdown).health(0) HealthMonitor::service_shutdown 1 > > services > > 2015-03-14 07:47:26.384665 7f1f1dc0f700 0 quorum service shutdown > > 2015-03-14 07:47:27.620670 7fc04b4437a0 0 ceph version 0.80.9 > > (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-mon, pid > > 17050 > > 2015-03-14 07:47:27.703151 7fc04b4437a0 0 starting mon.fu rank 0 at > > 192.168.1.100:6789/0 mon_data /var/lib/ceph/mon/ceph-fu fsid > > a1132ec2-7104-4e8e-a3d5-95965cae9138 > > 2015-03-14 07:47:27.703421 7fc04b4437a0 1 mon.fu@-1(probing) e2 > > preinit fsid a1132ec2-7104-4e8e-a3d5-95965cae9138 > > 2015-03-14 07:47:27.704504 7fc04b4437a0 1 > > mon.fu@-1(probing).paxosservice(pgmap 897493..898204) refresh > > upgraded, format 0 -> 1 > > 2015-03-14 07:47:27.704525 7fc04b4437a0 1 mon.fu@-1(probing).pg v0 > > on_upgrade discarding in-core PGMap > > 2015-03-14 07:47:27.837060 7fc04b4437a0 0 mon.fu@-1(probing).mds > > e104 print_map > > epoch 104 > > flags 0 > > created 2014-11-30 01:58:17.176938 > > modified 2015-03-14 06:07:05.683239 > > tableserver 0 > > root 0 > > session_timeout 60 > > session_autoclose 300 > > max_file_size 1099511627776 > > last_failure 0 > > last_failure_osd_epoch 1760 > > compat compat={},rocompat={},incompat={1=base v0.20,2=client > > writeable ranges,3=default file layouts on dirs,4=dir inode in > > separate object,5=mds uses versioned encoding,6=dirfrag is stored in > > omap} > > max_mds 1 > > in 0 > > up {0=59315} > > failed > > stopped > > data_pools 3 > > metadata_pool 4 > > inline_data disabled > > 59315: 15.12.6.21:6800/26628 'fu' mds.0.21 up:active seq 9 > > > > 2015-03-14 07:47:27.837972 7fc04b4437a0 0 mon.fu@-1(probing).osd > > e1768 crush map has features 1107558400, adjusting msgr requires > > 2015-03-14 07:47:27.837990 7fc04b4437a0 0 mon.fu@-1(probing).osd > > e1768 crush map has features 1107558400, adjusting msgr requires > > 2015-03-14 07:47:27.837996 7fc04b4437a0 0 mon.fu@-1(probing).osd > > e1768 crush map has features 1107558400, adjusting msgr requires > > 2015-03-14 07:47:27.838003 7fc04b4437a0 0 mon.fu@-1(probing).osd > > e1768 crush map has features 1107558400, adjusting msgr requires > > 2015-03-14 07:47:27.839054 7fc04b4437a0 1 > > mon.fu@-1(probing).paxosservice(auth 2751..2829) refresh upgraded, > > format 0 -> 1 > > 2015-03-14 07:47:27.840052 7fc04b4437a0 0 mon.fu@-1(probing) e2 my > > rank is now 0 (was -1) > > 2015-03-14 07:47:27.840512 7fc045ef5700 0 -- 192.168.1.100:6789/0 >> > > 192.168.1.101:6789/0 pipe(0x3958780 sd=13 :0 s=1 pgs=0 cs=0 l=0 > > c=0x38c0dc0).fault > > > > > > > > > > > >> I can no longer start my OSDs :-@ > >> > >> > >> failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf > >> --name=osd.6 > >> --keyring=/var/lib/ceph/osd/ceph-6/keyring osd crush create-or-move > >> -- > >> 6 3.63 host=fu root=default' > >> > >> > >> Please help!!! > >> > >> George > >> > >>> ceph mon add stops at this: > >>> > >>> > >>> [jin][INFO ] Running command: sudo ceph mon getmap -o > >>> /var/lib/ceph/tmp/ceph.raijin.monmap > >>> > >>> > >>> and never gets over it!!!!! > >>> > >>> > >>> Any help?? > >>> > >>> Thanks, > >>> > >>> > >>> George > >>> > >>>> Guyn any help much appreciated because my cluster is down :-( > >>>> > >>>> After trying ceph mon add which didn't complete since it was stuck > >>>> for ever here: > >>>> > >>>> [jin][WARNIN] 2015-03-14 07:07:14.964265 7fb4be6f5700 0 > >>>> monclient: > >>>> hunting for new mon > >>>> ^CKilled by signal 2. > >>>> [ceph_deploy][ERROR ] KeyboardInterrupt > >>>> > >>>> > >>>> the previously healthy node is now down completely :-( > >>>> > >>>> $ ceph mon stat > >>>> 2015-03-14 07:21:37.782360 7ff2545b1700 0 -- > >>>> 192.168.1.100:0/1042061 > >>>> >> 192.168.1.101:6789/0 pipe(0x7ff248000c00 sd=4 :0 s=1 pgs=0 cs=0 > >>>> l=1 > >>>> c=0x7ff248000e90).fault > >>>> ^CError connecting to cluster: InterruptedOrTimeoutError > >>>> > >>>> > >>>> Any ideas?? > >>>> > >>>> > >>>> All the best, > >>>> > >>>> George > >>>> > >>>> > >>>> > >>>>> Georgeos > >>>>> > >>>>> , you need to have "deployment server" and cd into folder that > >>>>> you > >>>>> used originaly while deploying CEPH - in this folder you should > >>>>> already have ceph.conf, admin.client keyring and other stuff - > >>>>> which > >>>>> is required to to connect to cluster...and provision new MONs or > >>>>> OSDs, > >>>>> etc. > >>>>> > >>>>> Message: > >>>>> [ceph_deploy][ERROR ] RuntimeError: mon keyring not found; run > >>>>> new to > >>>>> create a new cluster... > >>>>> > >>>>> ...means (if Im not mistaken) that you are runnign ceph-deploy > >>>>> from > >>>>> NOT original folder... > >>>>> > >>>>> On 13 March 2015 at 23:03, Georgios Dimitrakakis wrote: > >>>>> > >>>>>> Not a firewall problem!! Firewall is disabled ... > >>>>>> > >>>>>> Loic I ve tried mon create because of this: > >>>>>> > >>>>> > >>>>> http://ceph.com/docs/v0.80.5/start/quick-ceph-deploy/#adding-monitors > >>>>>> [4] > >>>>>> > >>>>>> Should I first create and then add?? What is the proper order??? > >>>>>> Should I do it from the already existing monitor node or can I > >>>>>> run > >>>>>> it from the new one? > >>>>>> > >>>>>> If I try add from the beginning I am getting this: > >>>>>> > >>>>>> ceph_deploy.conf][DEBUG ] found configuration file at: > >>>>>> /home/.cephdeploy.conf > >>>>>> [ceph_deploy.cli][INFO ] Invoked (1.5.22): /usr/bin/ceph-deploy > >>>>>> mon add jin > >>>>>> [ceph_deploy][ERROR ] RuntimeError: mon keyring not found; run > >>>>>> new > >>>>>> to create a new cluster > >>>>>> > >>>>>> Regards, > >>>>>> > >>>>>> George > >>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> I think ceph-deploy mon add (instead of create) is what you > >>>>>>> should be using. > >>>>>>> > >>>>>>> Cheers > >>>>>>> > >>>>>>> On 13/03/2015 22:25, Georgios Dimitrakakis wrote: > >>>>>>> > >>>>>>>> On an already available cluster I ve tried to add a new > >>>>>>>> monitor! > >>>>>>>> > >>>>>>>> I have used ceph-deploy mon create {NODE} > >>>>>>>> > >>>>>>>> where {NODE}=the name of the node > >>>>>>>> > >>>>>>>> and then I restarted the /etc/init.d/ceph service with a > >>>>>>>> success at the node > >>>>>>>> where it showed that the monitor is running like: > >>>>>>>> > >>>>>>>> # /etc/init.d/ceph restart > >>>>>>>> === mon.jin === > >>>>>>>> === mon.jin === > >>>>>>>> Stopping Ceph mon.jin on jin...kill 36388...done > >>>>>>>> === mon.jin === > >>>>>>>> Starting Ceph mon.jin on jin... > >>>>>>>> Starting ceph-create-keys on jin... > >>>>>>>> > >>>>>>>> But checking the quorum it doesnt show the newly added > >>>>>>>> monitor! > >>>>>>>> > >>>>>>>> Plus ceph mon stat gives out only 1 monitor!!! > >>>>>>>> > >>>>>>>> # ceph mon stat > >>>>>>>> e1: 1 mons at {fu=MAILSCANNER WARNING: NUMERICAL LINKS ARE > >>>>>>>> OFTEN MALICIOUS: 192.168.1.100:6789/0 [1]}, election epoch 1, > >>>>>>>> quorum 0 fu > >>>>>>>> > >>>>>>>> Any ideas on what have I done wrong??? > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> > >>>>>>>> George > >>>>>>>> _______________________________________________ > >>>>>>>> ceph-users mailing list > >>>>>>>> ceph-users@lists.ceph.com [2] > >>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [3] > >>>>>> _______________________________________________ > >>>>>> ceph-users mailing list > >>>>>> ceph-users@lists.ceph.com [5] > >>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [6] > >>>> _______________________________________________ > >>>> ceph-users mailing list > >>>> ceph-users@lists.ceph.com > >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >>> _______________________________________________ > >>> ceph-users mailing list > >>> ceph-users@lists.ceph.com > >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com