Hi Zheng, Sorry for the late reply. For sure, I will try this again after we completely verifying all content in the file system. Hopefully all will be good. And, please confirm this, I will set debug_mds=10 for the ceph-mds, and do you want me to send the ceph-mon log too?
BTW, how to confirm that the mds has passed the beacon to mon or not? Thank you so much Zheng! Bazli -----Original Message----- From: Yan, Zheng [mailto:uker...@gmail.com] Sent: Tuesday, April 29, 2014 10:13 PM To: Mohd Bazli Ab Karim Cc: Luke Jing Yuan; Wong Ming Tat Subject: Re: [ceph-users] Ceph mds laggy and failed assert in function replay mds/journal.cc On Tue, Apr 29, 2014 at 5:30 PM, Mohd Bazli Ab Karim <bazli.abka...@mimos.my> wrote: > Hi Zheng, > > The another issue that Luke mentioned just now was like this. > At first, we ran one mds (mon01) with the new compiled ceph-mds. It works > fine with only one MDS running at that time. However, when we ran two more > MDSes mon02 mon03 with the new compiled ceph-mds, it started acting weird. > Mon01 which was became active at first, will have the error and started to > respawning. Once respawning happened, mon03 will take over from mon01 as > master mds, and replay happened again. > Again, when mon03 became active, it will have the same error like below, and > respawning again. So, it seems to me that replay will continue to happen from > one mds to another when they got respawned. > > 2014-04-29 15:36:24.917798 7f5c36476700 1 mds.0.server > reconnect_clients -- 1 sessions > 2014-04-29 15:36:24.919620 7f5c2fb3e700 0 -- 10.4.118.23:6800/26401 > >> 10.1.64.181:0/1558263174 pipe(0x2924f5780 sd=41 :6800 s=0 pgs=0 > cs=0 l=0 c=0x37056e0).accept peer addr is really > 10.1.64.181:0/1558263174 (socket is 10.1.64.181:57649/0) > 2014-04-29 15:36:24.921661 7f5c36476700 0 log [DBG] : reconnect by > client.884169 10.1.64.181:0/1558263174 after 0.003774 > 2014-04-29 15:36:24.921786 7f5c36476700 1 mds.0.12858 reconnect_done > 2014-04-29 15:36:25.109391 7f5c36476700 1 mds.0.12858 handle_mds_map > i am now mds.0.12858 > 2014-04-29 15:36:25.109413 7f5c36476700 1 mds.0.12858 handle_mds_map > state change up:reconnect --> up:rejoin > 2014-04-29 15:36:25.109417 7f5c36476700 1 mds.0.12858 rejoin_start > 2014-04-29 15:36:26.918067 7f5c36476700 1 mds.0.12858 > rejoin_joint_start > 2014-04-29 15:36:33.520985 7f5c36476700 1 mds.0.12858 rejoin_done > 2014-04-29 15:36:36.252925 7f5c36476700 1 mds.0.12858 handle_mds_map > i am now mds.0.12858 > 2014-04-29 15:36:36.252927 7f5c36476700 1 mds.0.12858 handle_mds_map > state change up:rejoin --> up:active > 2014-04-29 15:36:36.252932 7f5c36476700 1 mds.0.12858 recovery_done -- > successful recovery! > 2014-04-29 15:36:36.745833 7f5c36476700 1 mds.0.12858 active_start > 2014-04-29 15:36:36.987854 7f5c36476700 1 mds.0.12858 cluster recovered. > 2014-04-29 15:36:40.182604 7f5c36476700 0 mds.0.12858 > handle_mds_beacon no longer laggy > 2014-04-29 15:36:57.947441 7f5c2fb3e700 0 -- 10.4.118.23:6800/26401 > >> 10.1.64.181:0/1558263174 pipe(0x2924f5780 sd=41 :6800 s=2 pgs=156 > cs=1 l=0 c=0x37056e0).fault with nothing to send, going to standby > 2014-04-29 15:37:10.534593 7f5c36476700 1 mds.-1.-1 handle_mds_map i > (10.4.118.23:6800/26401) dne in the mdsmap, respawning myself > 2014-04-29 15:37:10.534604 7f5c36476700 1 mds.-1.-1 respawn > 2014-04-29 15:37:10.534609 7f5c36476700 1 mds.-1.-1 e: '/usr/bin/ceph-mds' > 2014-04-29 15:37:10.534612 7f5c36476700 1 mds.-1.-1 0: '/usr/bin/ceph-mds' > 2014-04-29 15:37:10.534616 7f5c36476700 1 mds.-1.-1 1: '--cluster=ceph' > 2014-04-29 15:37:10.534619 7f5c36476700 1 mds.-1.-1 2: '-i' > 2014-04-29 15:37:10.534621 7f5c36476700 1 mds.-1.-1 3: 'mon03' > 2014-04-29 15:37:10.534623 7f5c36476700 1 mds.-1.-1 4: '-f' > 2014-04-29 15:37:10.534641 7f5c36476700 1 mds.-1.-1 cwd / > 2014-04-29 15:37:12.155458 7f8907c8b780 0 ceph version (), process > ceph-mds, pid 26401 > 2014-04-29 15:37:12.249780 7f8902d10700 1 mds.-1.0 handle_mds_map > standby > > p/s. we ran ceph-mon and ceph-mds on same servers, (mon01,mon02,mon03) > > I sent to you two log files, mon01 and mon03 where the scenario of mon03 have > state->standby->replay->active->respawned. And also, mon01 which is now > running as active as a single MDS at this moment. > After the MDS became ative, it did not send beacon to the monitor. It seems like the MDS was busy doing something else. If this issue still happen, set debug_mds=10 and send the log to me. Regards Yan, Zheng > Regards, > Bazli > -----Original Message----- > From: Luke Jing Yuan > Sent: Tuesday, April 29, 2014 4:46 PM > To: Yan, Zheng > Cc: Mohd Bazli Ab Karim; Wong Ming Tat > Subject: RE: [ceph-users] Ceph mds laggy and failed assert in function > replay mds/journal.cc > > Hi Zheng, > > Thanks for the information. Actually we encounter another issue, in our > original setup, we have 3 MDS running (say mon01, mon02 and mon03), when we > do the replay/recovery we did it on mon01. After we completed, we restarted > the mds again on mon02 and mon03 (without the mds_wipe_sessions and using the > patched binary) but there was when we noticed something else, my colleague > Mr. Bazli who started the original thread probably can explain a bit more on > the observations made. > > Regards, > Luke > > -----Original Message----- > From: Yan, Zheng [mailto:uker...@gmail.com] > Sent: Tuesday, 29 April, 2014 4:26 PM > To: Luke Jing Yuan > Subject: Re: [ceph-users] Ceph mds laggy and failed assert in function > replay mds/journal.cc > > On Tue, Apr 29, 2014 at 3:43 PM, Luke Jing Yuan <jyl...@mimos.my> wrote: >> Hi, >> >> MDS did finish the replay and working after that but we are wondering should >> we leave the mds_wipe_sessions in ceph.conf or remove it. >> > > should disable mds_wipe_sessions after mds starts working > >> Regards, >> Luke >> >> -----Original Message----- >> From: ceph-users-boun...@lists.ceph.com >> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Yan, Zheng >> Sent: Tuesday, 29 April, 2014 3:36 PM >> To: Jingyuan Luke >> Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com >> Subject: Re: [ceph-users] Ceph mds laggy and failed assert in >> function replay mds/journal.cc >> >> On Tue, Apr 29, 2014 at 3:13 PM, Jingyuan Luke <jyl...@gmail.com> wrote: >>> Hi, >>> >>> Assuming we got MDS working back on track, should we still leave the >>> mds_wipe_sessions in the ceph.conf or remove it and restart MDS. >>> Thanks. >> >> No. >> >> It has been several hours. the MDS still does not finish replaying the >> journal? >> >> Regards >> Yan, Zheng >> >> >> >> ________________________________ >> DISCLAIMER: >> >> >> This e-mail (including any attachments) is for the addressee(s) only and may >> be confidential, especially as regards personal data. If you are not the >> intended recipient, please note that any dealing, review, distribution, >> printing, copying or use of this e-mail is strictly prohibited. If you have >> received this email in error, please notify the sender immediately and >> delete the original message (including any attachments). >> >> >> MIMOS Berhad is a research and development institution under the purview of >> the Malaysian Ministry of Science, Technology and Innovation. Opinions, >> conclusions and other information in this e-mail that do not relate to the >> official business of MIMOS Berhad and/or its subsidiaries shall be >> understood as neither given nor endorsed by MIMOS Berhad and/or its >> subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts >> responsibility for the same. All liability arising from or in connection >> with computer viruses and/or corrupted e-mails is excluded to the fullest >> extent permitted by law. >> >> ------------------------------------------------------------------ >> - >> - >> DISCLAIMER: >> >> This e-mail (including any attachments) is for the addressee(s) only >> and may contain confidential information. If you are not the intended >> recipient, please note that any dealing, review, distribution, >> printing, copying or use of this e-mail is strictly prohibited. If >> you have received this email in error, please notify the sender >> immediately and delete the original message. >> MIMOS Berhad is a research and development institution under the >> purview of the Malaysian Ministry of Science, Technology and >> Innovation. Opinions, conclusions and other information in this e- >> mail that do not relate to the official business of MIMOS Berhad >> and/or its subsidiaries shall be understood as neither given nor >> endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS >> Berhad nor its subsidiaries accepts responsibility for the same. All >> liability arising from or in connection with computer viruses and/or >> corrupted e-mails is excluded to the fullest extent permitted by law. >> >> > > ________________________________ > DISCLAIMER: > > > This e-mail (including any attachments) is for the addressee(s) only and may > be confidential, especially as regards personal data. If you are not the > intended recipient, please note that any dealing, review, distribution, > printing, copying or use of this e-mail is strictly prohibited. If you have > received this email in error, please notify the sender immediately and delete > the original message (including any attachments). > > > MIMOS Berhad is a research and development institution under the purview of > the Malaysian Ministry of Science, Technology and Innovation. Opinions, > conclusions and other information in this e-mail that do not relate to the > official business of MIMOS Berhad and/or its subsidiaries shall be understood > as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and > neither MIMOS Berhad nor its subsidiaries accepts responsibility for the > same. All liability arising from or in connection with computer viruses > and/or corrupted e-mails is excluded to the fullest extent permitted by law. > > > ------------------------------------------------------------------ > - > - > DISCLAIMER: > > This e-mail (including any attachments) is for the addressee(s) only > and may contain confidential information. If you are not the intended > recipient, please note that any dealing, review, distribution, > printing, copying or use of this e-mail is strictly prohibited. If you > have received this email in error, please notify the sender > immediately and delete the original message. > MIMOS Berhad is a research and development institution under the > purview of the Malaysian Ministry of Science, Technology and > Innovation. Opinions, conclusions and other information in this e- > mail that do not relate to the official business of MIMOS Berhad > and/or its subsidiaries shall be understood as neither given nor > endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS > Berhad nor its subsidiaries accepts responsibility for the same. All > liability arising from or in connection with computer viruses and/or > corrupted e-mails is excluded to the fullest extent permitted by law. > ________________________________ DISCLAIMER: This e-mail (including any attachments) is for the addressee(s) only and may be confidential, especially as regards personal data. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message (including any attachments). MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law. ------------------------------------------------------------------ - - DISCLAIMER: This e-mail (including any attachments) is for the addressee(s) only and may contain confidential information. If you are not the intended recipient, please note that any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited. If you have received this email in error, please notify the sender immediately and delete the original message. MIMOS Berhad is a research and development institution under the purview of the Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions and other information in this e- mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability arising from or in connection with computer viruses and/or corrupted e-mails is excluded to the fullest extent permitted by law. _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
