Re: [vpp-dev] vppctl makes VPP hang-up occasionally
Glad to hear it. By the way, the Jira ticket you created was so well done that I was able to diagnose the problem in a few minutes. Thanks for taking the time to create such a useful bug report. Dabe From: Yusuke Tatsumi Sent: Friday, July 12, 2019 11:02 AM To: Luke, Chris ; Dave Barach (dbarach) ; vpp-dev@lists.fd.io Subject: Re: vppctl makes VPP hang-up occasionally Hi Dave, Chris, Thank you for your kind support! With your patch, our VPPs works fine "without" hang-up so far in 24h. https://gerrit.fd.io/r/#/c/20573/ Regards, Tatsumi. 差出人: Luke, Chris mailto:chris_l...@comcast.com>> 送信日時: 2019年7月11日 6:38 宛先: Dave Barach (dbarach); 立見 祐介; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> 件名: RE: vppctl makes VPP hang-up occasionally Sure, and in 99.99% of cases nobody would ever notice, too. For the curious, the most straightforward way to tickle this particular feature is something like “socat /run/vpp/cli.sock stdio” to get a CLI connection without any TELNET negotiation; there’s a ~1s pause while VPP waits for the negotiation to complete before it gives up and starts the session regardless. Chris. From: Dave Barach (dbarach) mailto:dbar...@cisco.com>> Sent: Wednesday, July 10, 2019 17:04 To: Luke, Chris mailto:chris_l...@cable.comcast.com>>; Yusuke Tatsumi mailto:ytats...@yahoo-corp.jp>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [EXTERNAL] RE: vppctl makes VPP hang-up occasionally +1, commenting out the timer_call(...) was a quick hack to make the pain go away... From: Luke, Chris mailto:chris_l...@comcast.com>> Sent: Wednesday, July 10, 2019 4:55 PM To: Dave Barach (dbarach) mailto:dbar...@cisco.com>>; Yusuke Tatsumi mailto:ytats...@yahoo-corp.jp>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: vppctl makes VPP hang-up occasionally For the archives, https://gerrit.fd.io/r/#/c/20573/ was merged earlier to resolve this issue. We are hoping Tatsumi is able to confirm this, but since it is a better approach regardless. Chris. From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Dave Barach via Lists.Fd.Io Sent: Tuesday, July 9, 2019 08:06 To: Yusuke Tatsumi mailto:ytats...@yahoo-corp.jp>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally The root cause is obvious from the backtrace in VPP-1711: memory allocator takes a spin-lock, timer interrupt fires, back into the memory allocator, deadlock. The [sic] timer interrupt function in .../src/vppinfra/timers.c is a mistake. You managed to find the only place in all of vpp which uses it. As a quick 1-line workaround, comment out the “timer_call” shown below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an “if(0)”... /* In case the client doesn't negotiate terminal type, use * a timer to kick off the initial prompt. */ if (0) timer_call (unix_cli_file_welcome_timer, cf_index, 1); HTH... Dave From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Yusuke Tatsumi Sent: Tuesday, July 9, 2019 1:11 AM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] vppctl makes VPP hang-up occasionally Hi all, We found that VPP itself freeze once a day under using "vppctl" command repeatedly. I think this is a kind of spin-lock problem but I can't understand the root cause. See here for details. https://jira.fd.io/browse/VPP-1711 If this goes on, It's hard to operate VPP as production service. So I need help to fix this issue. Could anyone help/suggest to this issue? Thanks, Tatsumi. - 立見 祐介 ヤフー株式会社 テクノロジーグループ システム統括本部 クラウドプラットフォーム本部 技術1部 コンピュート TEL: 03-6898-3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> - Yusuke Tatsumi Compute team, Cloud Platform Division, System Management Group Yahoo Japan Corporation Direct: +81 (3) 6898 3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13504): https://lists.fd.io/g/vpp-dev/message/13504 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vppctl makes VPP hang-up occasionally
Hi Dave, Chris, Thank you for your kind support! With your patch, our VPPs works fine "without" hang-up so far in 24h. https://gerrit.fd.io/r/#/c/20573/ Regards, Tatsumi. 差出人: Luke, Chris 送信日�r: 2019年7月11日 6:38 宛先: Dave Barach (dbarach); 立�� �v介; vpp-dev@lists.fd.io 件名: RE: vppctl makes VPP hang-up occasionally Sure, and in 99.99% of cases nobody would ever notice, too. For the curious, the most straightforward way to tickle this particular feature is something like “socat /run/vpp/cli.sock stdio” to get a CLI connection without any TELNET negotiation; there’s a ~1s pause while VPP waits for the negotiation to complete before it gives up and starts the session regardless. Chris. From: Dave Barach (dbarach) Sent: Wednesday, July 10, 2019 17:04 To: Luke, Chris ; Yusuke Tatsumi ; vpp-dev@lists.fd.io Subject: [EXTERNAL] RE: vppctl makes VPP hang-up occasionally +1, commenting out the timer_call(...) was a quick hack to make the pain go away... From: Luke, Chris mailto:chris_l...@comcast.com>> Sent: Wednesday, July 10, 2019 4:55 PM To: Dave Barach (dbarach) mailto:dbar...@cisco.com>>; Yusuke Tatsumi mailto:ytats...@yahoo-corp.jp>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: vppctl makes VPP hang-up occasionally For the archives, https://gerrit.fd.io/r/#/c/20573/ was merged earlier to resolve this issue. We are hoping Tatsumi is able to confirm this, but since it is a better approach regardless. Chris. From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Dave Barach via Lists.Fd.Io Sent: Tuesday, July 9, 2019 08:06 To: Yusuke Tatsumi mailto:ytats...@yahoo-corp.jp>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally The root cause is obvious from the backtrace in VPP-1711: memory allocator takes a spin-lock, timer interrupt fires, back into the memory allocator, deadlock. The [sic] timer interrupt function in .../src/vppinfra/timers.c is a mistake. You managed to find the only place in all of vpp which uses it. As a quick 1-line workaround, comment out the “timer_call” shown below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an “if(0)”... /* In case the client doesn't negotiate terminal type, use * a timer to kick off the initial prompt. */ if (0) timer_call (unix_cli_file_welcome_timer, cf_index, 1); HTH... Dave From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Yusuke Tatsumi Sent: Tuesday, July 9, 2019 1:11 AM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] vppctl makes VPP hang-up occasionally Hi all, We found that VPP itself freeze once a day under using "vppctl" command repeatedly. I think this is a kind of spin-lock problem but I can't understand the root cause. See here for details. https://jira.fd.io/browse/VPP-1711 If this goes on, It's hard to operate VPP as production service. So I need help to fix this issue. Could anyone help/suggest to this issue? Thanks, Tatsumi. ― 立�� �v介 ヤフ�`株式会社 テクノロジ�`グル�`プ システム�y括本部 クラウドプラットフォ�`ム本部 技�g1部 コンピュ�`ト TEL: 03-6898-3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> ― Yusuke Tatsumi Compute team, Cloud Platform Division, System Management Group Yahoo Japan Corporation Direct: +81 (3) 6898 3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13503): https://lists.fd.io/g/vpp-dev/message/13503 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally
Get you now, thanks. Ray K On 11/07/2019 11:58, Luke, Chris wrote: > His output is from the stdio console; it's the vppctl session that causes a > trigger to be loaded. > > Chris. > > -Original Message- > From: vpp-dev@lists.fd.io On Behalf Of Ray Kinsella > Sent: Thursday, July 11, 2019 06:42 > To: dbar...@cisco.com; Yusuke Tatsumi ; > vpp-dev@lists.fd.io > Subject: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally > > Good work > > Bit confused - if we are waiting for a timer to trigger here to display a > welcome and show a prompt. > > How does Tatsumi have a prompt in the first place, to type the initial > command? > > My understanding is the initial memory allocation is happening in response to > Tatsumi (or anyone) typing in a command. > > Ray K > > On 09/07/2019 13:05, Dave Barach via Lists.Fd.Io wrote: >> The root cause is obvious from the backtrace in VPP-1711: memory >> allocator takes a spin-lock, timer interrupt fires, back into the >> memory allocator, deadlock. >> >> >> >> The [sic] timer interrupt function in .../src/vppinfra/timers.c is a >> mistake. You managed to find the only place in all of vpp which uses it. >> >> >> >> As a quick 1-line workaround, comment out the “timer_call” shown >> below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an >> “if(0)”... >> >> >> >> /* In case the client doesn't negotiate terminal type, use >> >> * a timer to kick off the initial prompt. */ >> >> if (0) >> >> timer_call (unix_cli_file_welcome_timer, cf_index, 1); >> >> >> >> HTH... Dave >> >> >> >> *From:*vpp-dev@lists.fd.io *On Behalf Of *Yusuke >> Tatsumi >> *Sent:* Tuesday, July 9, 2019 1:11 AM >> *To:* vpp-dev@lists.fd.io >> *Subject:* [vpp-dev] vppctl makes VPP hang-up occasionally >> >> >> >> Hi all, >> >> >> >> We found that VPP itself freeze once a day under using "vppctl" >> command repeatedly. >> >> I think this is a kind of spin-lock problem but I can't understand the >> root cause. >> >> See here for details. >> >> https://jira.fd.io/browse/VPP-1711 >> >> >> >> If this goes on, It's hard to operate VPP as production service. So I >> need help to fix this issue. >> >> Could anyone help/suggest to this issue? >> >> >> >> Thanks, >> >> Tatsumi. >> >> >> >> — >> >> 立見祐介 >> >> ヤフー株式会社 >> >> テクノロジーグループ システム統括本部クラウドプラットフォーム本部技術1 >> 部 コンピュート >> >> TEL: 03-6898-3081 >> >> mail: ytats...@yahoo-corp.jp <mailto:ytats...@yahoo-corp.jp> >> >> >> >> — >> >> Yusuke Tatsumi >> >> Compute team, >> >> Cloud Platform Division, >> >> System Management Group >> >> Yahoo Japan Corporation >> >> Direct: +81 (3) 6898 3081 >> >> mail: ytats...@yahoo-corp.jp <mailto:ytats...@yahoo-corp.jp> >> >> >> >> >> -=-=-=-=-=-=-=-=-=-=-=- >> Links: You receive all messages sent to this group. >> >> View/Reply Online (#13463): >> https://lists.fd.io/g/vpp-dev/message/13463 >> Mute This Topic: >> https://protect2.fireeye.com/url?k=d480be3758b37aec.d4809983-2f088dfce >> 1d69bd3=https://lists.fd.io/mt/32402610/675355 >> Group Owner: vpp-dev+ow...@lists.fd.io >> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [m...@ashroe.eu] >> -=-=-=-=-=-=-=-=-=-=-=- >> > signature.asc Description: OpenPGP digital signature -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13488): https://lists.fd.io/g/vpp-dev/message/13488 Mute This Topic: https://lists.fd.io/mt/32428896/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally
His output is from the stdio console; it's the vppctl session that causes a trigger to be loaded. Chris. -Original Message- From: vpp-dev@lists.fd.io On Behalf Of Ray Kinsella Sent: Thursday, July 11, 2019 06:42 To: dbar...@cisco.com; Yusuke Tatsumi ; vpp-dev@lists.fd.io Subject: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally Good work Bit confused - if we are waiting for a timer to trigger here to display a welcome and show a prompt. How does Tatsumi have a prompt in the first place, to type the initial command? My understanding is the initial memory allocation is happening in response to Tatsumi (or anyone) typing in a command. Ray K On 09/07/2019 13:05, Dave Barach via Lists.Fd.Io wrote: > The root cause is obvious from the backtrace in VPP-1711: memory > allocator takes a spin-lock, timer interrupt fires, back into the > memory allocator, deadlock. > > > > The [sic] timer interrupt function in .../src/vppinfra/timers.c is a > mistake. You managed to find the only place in all of vpp which uses it. > > > > As a quick 1-line workaround, comment out the “timer_call” shown > below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an > “if(0)”... > > > > /* In case the client doesn't negotiate terminal type, use > > * a timer to kick off the initial prompt. */ > > if (0) > > timer_call (unix_cli_file_welcome_timer, cf_index, 1); > > > > HTH... Dave > > > > *From:*vpp-dev@lists.fd.io *On Behalf Of *Yusuke > Tatsumi > *Sent:* Tuesday, July 9, 2019 1:11 AM > *To:* vpp-dev@lists.fd.io > *Subject:* [vpp-dev] vppctl makes VPP hang-up occasionally > > > > Hi all, > > > > We found that VPP itself freeze once a day under using "vppctl" > command repeatedly. > > I think this is a kind of spin-lock problem but I can't understand the > root cause. > > See here for details. > > https://jira.fd.io/browse/VPP-1711 > > > > If this goes on, It's hard to operate VPP as production service. So I > need help to fix this issue. > > Could anyone help/suggest to this issue? > > > > Thanks, > > Tatsumi. > > > > — > > 立見祐介 > > ヤフー株式会社 > > テクノロジーグループ システム統括本部クラウドプラットフォーム本部技術1 > 部 コンピュート > > TEL: 03-6898-3081 > > mail: ytats...@yahoo-corp.jp <mailto:ytats...@yahoo-corp.jp> > > > > — > > Yusuke Tatsumi > > Compute team, > > Cloud Platform Division, > > System Management Group > > Yahoo Japan Corporation > > Direct: +81 (3) 6898 3081 > > mail: ytats...@yahoo-corp.jp <mailto:ytats...@yahoo-corp.jp> > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#13463): > https://lists.fd.io/g/vpp-dev/message/13463 > Mute This Topic: > https://protect2.fireeye.com/url?k=d480be3758b37aec.d4809983-2f088dfce > 1d69bd3=https://lists.fd.io/mt/32402610/675355 > Group Owner: vpp-dev+ow...@lists.fd.io > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [m...@ashroe.eu] > -=-=-=-=-=-=-=-=-=-=-=- > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13487): https://lists.fd.io/g/vpp-dev/message/13487 Mute This Topic: https://lists.fd.io/mt/32428896/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vppctl makes VPP hang-up occasionally
Good work Bit confused - if we are waiting for a timer to trigger here to display a welcome and show a prompt. How does Tatsumi have a prompt in the first place, to type the initial command? My understanding is the initial memory allocation is happening in response to Tatsumi (or anyone) typing in a command. Ray K On 09/07/2019 13:05, Dave Barach via Lists.Fd.Io wrote: > The root cause is obvious from the backtrace in VPP-1711: memory > allocator takes a spin-lock, timer interrupt fires, back into the memory > allocator, deadlock. > > > > The [sic] timer interrupt function in .../src/vppinfra/timers.c is a > mistake. You managed to find the only place in all of vpp which uses it. > > > > As a quick 1-line workaround, comment out the “timer_call” shown below, > around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an “if(0)”... > > > > /* In case the client doesn't negotiate terminal type, use > > * a timer to kick off the initial prompt. */ > > if (0) > > timer_call (unix_cli_file_welcome_timer, cf_index, 1); > > > > HTH... Dave > > > > *From:*vpp-dev@lists.fd.io *On Behalf Of *Yusuke > Tatsumi > *Sent:* Tuesday, July 9, 2019 1:11 AM > *To:* vpp-dev@lists.fd.io > *Subject:* [vpp-dev] vppctl makes VPP hang-up occasionally > > > > Hi all, > > > > We found that VPP itself freeze once a day under using "vppctl" command > repeatedly. > > I think this is a kind of spin-lock problem but I can't understand the > root cause. > > See here for details. > > https://jira.fd.io/browse/VPP-1711 > > > > If this goes on, It's hard to operate VPP as production service. So I > need help to fix this issue. > > Could anyone help/suggest to this issue? > > > > Thanks, > > Tatsumi. > > > > — > > 立見祐介 > > ヤフー株式会社 > > テクノロジーグループ システム統括本部クラウドプラットフォーム本部技術1 > 部 コンピュート > > TEL: 03-6898-3081 > > mail: ytats...@yahoo-corp.jp <mailto:ytats...@yahoo-corp.jp> > > > > — > > Yusuke Tatsumi > > Compute team, > > Cloud Platform Division, > > System Management Group > > Yahoo Japan Corporation > > Direct: +81 (3) 6898 3081 > > mail: ytats...@yahoo-corp.jp <mailto:ytats...@yahoo-corp.jp> > > > > > -=-=-=-=-=-=-=-=-=-=-=- > Links: You receive all messages sent to this group. > > View/Reply Online (#13463): https://lists.fd.io/g/vpp-dev/message/13463 > Mute This Topic: https://lists.fd.io/mt/32402610/675355 > Group Owner: vpp-dev+ow...@lists.fd.io > Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [m...@ashroe.eu] > -=-=-=-=-=-=-=-=-=-=-=- > signature.asc Description: OpenPGP digital signature -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13486): https://lists.fd.io/g/vpp-dev/message/13486 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vppctl makes VPP hang-up occasionally
Sure, and in 99.99% of cases nobody would ever notice, too. For the curious, the most straightforward way to tickle this particular feature is something like “socat /run/vpp/cli.sock stdio” to get a CLI connection without any TELNET negotiation; there’s a ~1s pause while VPP waits for the negotiation to complete before it gives up and starts the session regardless. Chris. From: Dave Barach (dbarach) Sent: Wednesday, July 10, 2019 17:04 To: Luke, Chris ; Yusuke Tatsumi ; vpp-dev@lists.fd.io Subject: [EXTERNAL] RE: vppctl makes VPP hang-up occasionally +1, commenting out the timer_call(...) was a quick hack to make the pain go away... From: Luke, Chris mailto:chris_l...@comcast.com>> Sent: Wednesday, July 10, 2019 4:55 PM To: Dave Barach (dbarach) mailto:dbar...@cisco.com>>; Yusuke Tatsumi mailto:ytats...@yahoo-corp.jp>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: RE: vppctl makes VPP hang-up occasionally For the archives, https://gerrit.fd.io/r/#/c/20573/ was merged earlier to resolve this issue. We are hoping Tatsumi is able to confirm this, but since it is a better approach regardless. Chris. From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Dave Barach via Lists.Fd.Io Sent: Tuesday, July 9, 2019 08:06 To: Yusuke Tatsumi mailto:ytats...@yahoo-corp.jp>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally The root cause is obvious from the backtrace in VPP-1711: memory allocator takes a spin-lock, timer interrupt fires, back into the memory allocator, deadlock. The [sic] timer interrupt function in .../src/vppinfra/timers.c is a mistake. You managed to find the only place in all of vpp which uses it. As a quick 1-line workaround, comment out the “timer_call” shown below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an “if(0)”... /* In case the client doesn't negotiate terminal type, use * a timer to kick off the initial prompt. */ if (0) timer_call (unix_cli_file_welcome_timer, cf_index, 1); HTH... Dave From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Yusuke Tatsumi Sent: Tuesday, July 9, 2019 1:11 AM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] vppctl makes VPP hang-up occasionally Hi all, We found that VPP itself freeze once a day under using "vppctl" command repeatedly. I think this is a kind of spin-lock problem but I can't understand the root cause. See here for details. https://jira.fd.io/browse/VPP-1711 If this goes on, It's hard to operate VPP as production service. So I need help to fix this issue. Could anyone help/suggest to this issue? Thanks, Tatsumi. - 立見 祐介 ヤフー株式会社 テクノロジーグループ システム統括本部 クラウドプラットフォーム本部 技術1部 コンピュート TEL: 03-6898-3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> - Yusuke Tatsumi Compute team, Cloud Platform Division, System Management Group Yahoo Japan Corporation Direct: +81 (3) 6898 3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13485): https://lists.fd.io/g/vpp-dev/message/13485 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vppctl makes VPP hang-up occasionally
+1, commenting out the timer_call(...) was a quick hack to make the pain go away... From: Luke, Chris Sent: Wednesday, July 10, 2019 4:55 PM To: Dave Barach (dbarach) ; Yusuke Tatsumi ; vpp-dev@lists.fd.io Subject: RE: vppctl makes VPP hang-up occasionally For the archives, https://gerrit.fd.io/r/#/c/20573/ was merged earlier to resolve this issue. We are hoping Tatsumi is able to confirm this, but since it is a better approach regardless. Chris. From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Dave Barach via Lists.Fd.Io Sent: Tuesday, July 9, 2019 08:06 To: Yusuke Tatsumi mailto:ytats...@yahoo-corp.jp>>; vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally The root cause is obvious from the backtrace in VPP-1711: memory allocator takes a spin-lock, timer interrupt fires, back into the memory allocator, deadlock. The [sic] timer interrupt function in .../src/vppinfra/timers.c is a mistake. You managed to find the only place in all of vpp which uses it. As a quick 1-line workaround, comment out the “timer_call” shown below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an “if(0)”... /* In case the client doesn't negotiate terminal type, use * a timer to kick off the initial prompt. */ if (0) timer_call (unix_cli_file_welcome_timer, cf_index, 1); HTH... Dave From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Yusuke Tatsumi Sent: Tuesday, July 9, 2019 1:11 AM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] vppctl makes VPP hang-up occasionally Hi all, We found that VPP itself freeze once a day under using "vppctl" command repeatedly. I think this is a kind of spin-lock problem but I can't understand the root cause. See here for details. https://jira.fd.io/browse/VPP-1711 If this goes on, It's hard to operate VPP as production service. So I need help to fix this issue. Could anyone help/suggest to this issue? Thanks, Tatsumi. - 立見 祐介 ヤフー株式会社 テクノロジーグループ システム統括本部 クラウドプラットフォーム本部 技術1部 コンピュート TEL: 03-6898-3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> - Yusuke Tatsumi Compute team, Cloud Platform Division, System Management Group Yahoo Japan Corporation Direct: +81 (3) 6898 3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13484): https://lists.fd.io/g/vpp-dev/message/13484 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vppctl makes VPP hang-up occasionally
For the archives, https://gerrit.fd.io/r/#/c/20573/ was merged earlier to resolve this issue. We are hoping Tatsumi is able to confirm this, but since it is a better approach regardless. Chris. From: vpp-dev@lists.fd.io On Behalf Of Dave Barach via Lists.Fd.Io Sent: Tuesday, July 9, 2019 08:06 To: Yusuke Tatsumi ; vpp-dev@lists.fd.io Cc: vpp-dev@lists.fd.io Subject: [EXTERNAL] Re: [vpp-dev] vppctl makes VPP hang-up occasionally The root cause is obvious from the backtrace in VPP-1711: memory allocator takes a spin-lock, timer interrupt fires, back into the memory allocator, deadlock. The [sic] timer interrupt function in .../src/vppinfra/timers.c is a mistake. You managed to find the only place in all of vpp which uses it. As a quick 1-line workaround, comment out the “timer_call” shown below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an “if(0)”... /* In case the client doesn't negotiate terminal type, use * a timer to kick off the initial prompt. */ if (0) timer_call (unix_cli_file_welcome_timer, cf_index, 1); HTH... Dave From: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> mailto:vpp-dev@lists.fd.io>> On Behalf Of Yusuke Tatsumi Sent: Tuesday, July 9, 2019 1:11 AM To: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> Subject: [vpp-dev] vppctl makes VPP hang-up occasionally Hi all, We found that VPP itself freeze once a day under using "vppctl" command repeatedly. I think this is a kind of spin-lock problem but I can't understand the root cause. See here for details. https://jira.fd.io/browse/VPP-1711 If this goes on, It's hard to operate VPP as production service. So I need help to fix this issue. Could anyone help/suggest to this issue? Thanks, Tatsumi. - 立見 祐介 ヤフー株式会社 テクノロジーグループ システム統括本部 クラウドプラットフォーム本部 技術1部 コンピュート TEL: 03-6898-3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> - Yusuke Tatsumi Compute team, Cloud Platform Division, System Management Group Yahoo Japan Corporation Direct: +81 (3) 6898 3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13483): https://lists.fd.io/g/vpp-dev/message/13483 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] vppctl makes VPP hang-up occasionally
The root cause is obvious from the backtrace in VPP-1711: memory allocator takes a spin-lock, timer interrupt fires, back into the memory allocator, deadlock. The [sic] timer interrupt function in .../src/vppinfra/timers.c is a mistake. You managed to find the only place in all of vpp which uses it. As a quick 1-line workaround, comment out the “timer_call” shown below, around line 2837 in .../src/vlib/unix/cli.c, e.g. by adding an “if(0)”... /* In case the client doesn't negotiate terminal type, use * a timer to kick off the initial prompt. */ if (0) timer_call (unix_cli_file_welcome_timer, cf_index, 1); HTH... Dave From: vpp-dev@lists.fd.io On Behalf Of Yusuke Tatsumi Sent: Tuesday, July 9, 2019 1:11 AM To: vpp-dev@lists.fd.io Subject: [vpp-dev] vppctl makes VPP hang-up occasionally Hi all, We found that VPP itself freeze once a day under using "vppctl" command repeatedly. I think this is a kind of spin-lock problem but I can't understand the root cause. See here for details. https://jira.fd.io/browse/VPP-1711 If this goes on, It's hard to operate VPP as production service. So I need help to fix this issue. Could anyone help/suggest to this issue? Thanks, Tatsumi. - 立見 祐介 ヤフー株式会社 テクノロジーグループ システム統括本部 クラウドプラットフォーム本部 技術1部 コンピュート TEL: 03-6898-3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> - Yusuke Tatsumi Compute team, Cloud Platform Division, System Management Group Yahoo Japan Corporation Direct: +81 (3) 6898 3081 mail: ytats...@yahoo-corp.jp<mailto:ytats...@yahoo-corp.jp> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13463): https://lists.fd.io/g/vpp-dev/message/13463 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
[vpp-dev] vppctl makes VPP hang-up occasionally
Hi all, We found that VPP itself freeze once a day under using "vppctl" command repeatedly. I think this is a kind of spin-lock problem but I can't understand the root cause. See here for details. https://jira.fd.io/browse/VPP-1711 If this goes on, It's hard to operate VPP as production service. So I need help to fix this issue. Could anyone help/suggest to this issue? Thanks, Tatsumi. ― 立�� �v介 ヤフ�`株式会社 テクノロジ�`グル�`プ システム�y括本部 クラウドプラットフォ�`ム本部 技�g1部 コンピュ�`ト TEL: 03-6898-3081 mail: ytats...@yahoo-corp.jp ― Yusuke Tatsumi Compute team, Cloud Platform Division, System Management Group Yahoo Japan Corporation Direct: +81 (3) 6898 3081 mail: ytats...@yahoo-corp.jp -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13461): https://lists.fd.io/g/vpp-dev/message/13461 Mute This Topic: https://lists.fd.io/mt/32402610/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
