[asterisk-users] Exceptionally long queue length queuing . . . .
I wonder if anyone out there has a perspective on this. There are a welter of tickets out there on the matter, most of them closed. This problem began for me over a year ago, and continues up to the latest versions I've installed (1.6.2.13). It happens randomly, and the suggestion on one of the bug tracker tickets that it is instigated by a small network leg looks to be on point to me, because while it happens way often, it doesn't always happen. My ITSPs have all dropped IAX, and if they're experiencing this problem I can see why. Once the first of these messages has occurred, it's "goodbye audio" for the rest of the call. If anyone has a perspective on this longstanding problem, I'd sure be glad to hear it. Thanks. b. -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
[asterisk-users] Exceptionally long queue length queuing
Hi, We have a box up and we are starting to see a lot of "Exceptionally long queue length queuing" in the logs. From all the research so far it seems like this leads to their systems crashing and being unreachable. In our case the box remains up and takes calls. We are running Asterisk 16.6.1. We are using MusicOnHold to play online music streams via ffmpeg. Any idea on how to troubleshoot this further to see why this is happening? TIA. Regards, Dovid -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Exceptionally long queue length queuing . . . .
31 okt 2010 kl. 13.43 skrev Paul Belanger: > On Sat, Oct 30, 2010 at 6:22 PM, Brian Capouch wrote: >> I wonder if anyone out there has a perspective on this. There are a >> welter of tickets out there on the matter, most of them closed. >> > I'm actually able to reproduce this pretty often, for me using IAX2 > with IMAP voicemail (google apps) is how. I haven't had much time to > debug it, but plan to play more with it the coming weeks. Any update, Paul? /O -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Exceptionally long queue length queuing . . . .
On Wed, Nov 10, 2010 at 2:42 AM, Olle E. Johansson wrote: > Any update, Paul? > Yes, still reproducible. I'm going to do some homework first before creating a new issue on the tracker. Specifically, if anybody else can reproduce it. -- Paul Belanger | dCAP Polybeacon | Consultant Jabber: paul.belan...@polybeacon.com | IRC: pabelanger (Freenode) | Blog: http://blog.polybeacon.com | Twitter: http://twitter.com/pabelanger -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Exceptionally long queue length queuing . . . .
I have the same problem, once in a while. Curiously though, it occurs on a dedicated 100Mbps switched local network. I'm running 1.4.31 * servers. Vieri --- On Sat, 10/30/10, Brian Capouch wrote: > I wonder if anyone out there has a > perspective on this. There are a > welter of tickets out there on the matter, most of them > closed. > > This problem began for me over a year ago, and continues up > to the > latest versions I've installed (1.6.2.13). > > It happens randomly, and the suggestion on one of the bug > tracker > tickets that it is instigated by a small network leg looks > to be on > point to me, because while it happens way often, it doesn't > always happen. > > My ITSPs have all dropped IAX, and if they're experiencing > this problem > I can see why. Once the first of these messages has > occurred, it's > "goodbye audio" for the rest of the call. > > If anyone has a perspective on this longstanding problem, > I'd sure be > glad to hear it. > > Thanks. > > b. -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Exceptionally long queue length queuing . . . .
On Sat, Oct 30, 2010 at 6:22 PM, Brian Capouch wrote: > I wonder if anyone out there has a perspective on this. There are a > welter of tickets out there on the matter, most of them closed. > I'm actually able to reproduce this pretty often, for me using IAX2 with IMAP voicemail (google apps) is how. I haven't had much time to debug it, but plan to play more with it the coming weeks. -- Paul Belanger | dCAP Polybeacon | Consultant Jabber: paul.belan...@polybeacon.com | IRC: pabelanger (Freenode) | Blog: http://blog.polybeacon.com | Twitter: http://twitter.com/pabelanger -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- New to Asterisk? Join us for a live introductory webinar every Thurs: http://www.asterisk.org/hello asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Exceptionally long queue length queuing
On Sun, Jun 28, 2020 at 2:26 PM Dovid Bender wrote: > Hi, > > We have a box up and we are starting to see a lot of "Exceptionally long > queue length queuing" in the logs. From all the research so far it seems > like this leads to their systems crashing and being unreachable. In our > case the box remains up and takes calls. We are running Asterisk 16.6.1. We > are using MusicOnHold to play online music streams via ffmpeg. Any idea on > how to troubleshoot this further to see why this is happening? > The message occurs when either a channel is deadlocked or the processing time to handle what a channel is doing is substantial, and things build up. Getting a backtrace[1] at the time it occurs will show what the system is actually doing and can provide insight into what is causing it. [1] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-Runningast_coredumperfordeadlocks,taskprocessorbackups,etc . -- Joshua C. Colp Asterisk Technical Lead Sangoma Technologies Check us out at www.sangoma.com and www.asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Exceptionally long queue length queuing
On Mon, Jun 29, 2020 at 6:46 AM Joshua C. Colp wrote: > On Sun, Jun 28, 2020 at 2:26 PM Dovid Bender wrote: > >> Hi, >> >> We have a box up and we are starting to see a lot of "Exceptionally long >> queue length queuing" in the logs. From all the research so far it seems >> like this leads to their systems crashing and being unreachable. In our >> case the box remains up and takes calls. We are running Asterisk 16.6.1. We >> are using MusicOnHold to play online music streams via ffmpeg. Any idea on >> how to troubleshoot this further to see why this is happening? >> > > The message occurs when either a channel is deadlocked or the processing > time to handle what a channel is doing is substantial, and things build up. > Getting a backtrace[1] at the time it occurs will show what the system is > actually doing and can provide insight into what is causing it. > > [1] > https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-Runningast_coredumperfordeadlocks,taskprocessorbackups,etc > . > > -- > I ended up re-writing my code to use less Local channels and most of the errors went away. I also noticed that my load average and CPU usage is way down. I will open a ticket since it seems that it may be a bug that others are experiencing. > -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Exceptionally long queue length queuing
On Fri, Jul 3, 2020 at 3:32 AM Dovid Bender wrote: > > > On Mon, Jun 29, 2020 at 6:46 AM Joshua C. Colp wrote: > >> On Sun, Jun 28, 2020 at 2:26 PM Dovid Bender wrote: >> >>> Hi, >>> >>> We have a box up and we are starting to see a lot of "Exceptionally long >>> queue length queuing" in the logs. From all the research so far it seems >>> like this leads to their systems crashing and being unreachable. In our >>> case the box remains up and takes calls. We are running Asterisk 16.6.1. We >>> are using MusicOnHold to play online music streams via ffmpeg. Any idea on >>> how to troubleshoot this further to see why this is happening? >>> >> >> The message occurs when either a channel is deadlocked or the processing >> time to handle what a channel is doing is substantial, and things build up. >> Getting a backtrace[1] at the time it occurs will show what the system is >> actually doing and can provide insight into what is causing it. >> >> [1] >> https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-Runningast_coredumperfordeadlocks,taskprocessorbackups,etc >> . >> >> -- >> > I ended up re-writing my code to use less Local channels and most of the > errors went away. I also noticed that my load average and CPU usage is way > down. I will open a ticket since it seems that it may be a bug that others > are experiencing. > A backtrace would still be needed for the issue, otherwise there is no way to know what exactly is going on our where things are getting held up. -- Joshua C. Colp Asterisk Technical Lead Sangoma Technologies Check us out at www.sangoma.com and www.asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- Check out the new Asterisk community forum at: https://community.asterisk.org/ New to Asterisk? Start here: https://wiki.asterisk.org/wiki/display/AST/Getting+Started asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Exceptionally long queue length queuing
Hello list, Hope you are all doing well! Sorry for the long email but I tried to explain all I've seen regarding this issue I am going to open a ticket for this issue but I found it useful to already explain it here. I've also recently faced this "Exceptionally long queue length queuing" error in some servers running 16.8.0 and after A LOT of investigation, I've discovered what is causing it in my case (not sure if is the same case as initially reported by Dovid) So, what I discovered is that in case the diaplan has a Wait() and during this wait period many "deferrable frames" are received, the final piece of the Wait function (ast_safe_sleep_conditional) can throw the "Exceptionally long queue length queuing" error message. It is not really common to get a lot of "deferrable frames" (see ast_is_deferrable_frame) but we can have that by simply putting on hold and off hold while the channel is on "wait". My production case is a little more complex and happens while on Wait and with the AST_CONTROL_SRCCHANGE event. This happens when an old Asterisk 13.13.0 receives a hold REINVITE and starts a new RTP stream (with different SSRC) towards some other victim Asterisk with the Music on Hold while still sending the original caller RTP (so at this point 2 RTP streams with different SSRCs are sent to the destination that happens to be a channel currently on Wait). I could not find any ticket for such bug, but version 16.8.0 does not have this problem and instead seems to inject the Music on Hold in the existing RTP stream so there aren't 2 streams at the same time to the same destination. The problem with 2 streams is that each RTP package of the new stream generates a source change frame (AST_CONTROL_SRCCHANGE), and at 50pps this builds up a lot of events fast and so if the Wait is high the issue pops up (I agree having a channel on Wait for too long is not a good practice but noone is perfect...). Also if multiple channels are facing this situation then the problem just escalates very bad. When the "Exceptionally long queue length queuing" is happening all AMI commands seem to fail, BYE messages seem to be missed/delayed, internal timeouts expire and all sort of weird things happen to all channels in the server, apparently the whole Asterisk process gets locked in this loop affecting everything else... and this is what makes this issue really bad because Asterisk can become completely unresponsive while the error message is happening... if it only could throw the error and keep working fine I guess it would be reasonable leaving it up to the dialplan logic to avoid a long Wait() Anyway, to replicate the issue is enough a simple dialplan like this: exten => ,1,Answer() same => n,Wait(30) same => n,NoOp(After wait) same => n,Playback(goodbye) same => n,Hangup() After calling in and while in the Wait(30), repeatedly press Hold/Unhold in the telephone (with Linphone just press the hold once and then press and hold the spacebar which will repeatedly do the hold and unhold several times). After Wait is finished the "Exceptionally long queue length queuing" will show up but gets resolved very fast just because I think there weren't enough frames queued to really freeze Asterisk for too long. Thank you, Kind regards, Patrick Wakano On Fri, 3 Jul 2020 at 19:29, Joshua C. Colp wrote: > On Fri, Jul 3, 2020 at 3:32 AM Dovid Bender wrote: > >> >> >> On Mon, Jun 29, 2020 at 6:46 AM Joshua C. Colp wrote: >> >>> On Sun, Jun 28, 2020 at 2:26 PM Dovid Bender >>> wrote: >>> Hi, We have a box up and we are starting to see a lot of "Exceptionally long queue length queuing" in the logs. From all the research so far it seems like this leads to their systems crashing and being unreachable. In our case the box remains up and takes calls. We are running Asterisk 16.6.1. We are using MusicOnHold to play online music streams via ffmpeg. Any idea on how to troubleshoot this further to see why this is happening? >>> >>> The message occurs when either a channel is deadlocked or the processing >>> time to handle what a channel is doing is substantial, and things build up. >>> Getting a backtrace[1] at the time it occurs will show what the system is >>> actually doing and can provide insight into what is causing it. >>> >>> [1] >>> https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-Runningast_coredumperfordeadlocks,taskprocessorbackups,etc >>> . >>> >>> -- >>> >> I ended up re-writing my code to use less Local channels and most of the >> errors went away. I also noticed that my load average and CPU usage is way >> down. I will open a ticket since it seems that it may be a bug that others >> are experiencing. >> > > A backtrace would still be needed for the issue, otherwise there is no way > to know what exactly is going on our where things are getting held up. > > -- > Joshua C. Colp > Asterisk Technical Lead > Sangoma Technologies > Check us out at