Re: [asterisk-users] stucked calls in asterisk 1.4
On Thu, May 28, 2009 at 02:15:08PM +0200, Stefan Schmidt wrote: > > > Alex Samad schrieb: > > Hi > > Hi Alex, > > > > > I am new to asterisk so my suggestions might be a bit silly. > > > > Why not setup a iax2 connection bettween the asterisk servers, because > > its a lower overhear and more efficient. > > We had changed from iax connections to sip connections cause we had > timing problems with iax. Also in meetme and when a client send a fax > over the connection. This problems has gone after changing to sip connects. okay > [snip] > > its not a network packet loss its in the kernel or asterisk itself. okay [snip] > > > > the other thought is keep going the way youwere add another nic card to > > server a such that you have > > > > eth0 for your sip clients > > eth1 for router server b > > eth2 for router server c > > i´ve allready tried this, and the problem still occurs. i thought you had only add 1 extra nic. The thought was that 2 servers could push 1 nic over the limit, where as a 1 nic per server possibly not. But if you are sure its not a networking packet loss issue then its mute > > [snip] > > > > Try linux-kernel mailing list or do a google for udp buffer tunning > > from memory there is the kernel option which is the abs max, but the > > program must also select a buffer amount which is usually the default > > value which is not the max value. > > increase the max udp buffer size was just something i´ve found to try > but i´ve set the default value to 512kB by now, maybe this will help. At work there was a thread similiar to this, but different application problem with udp. Some who new said (if I can remember it rightly). there is a max/default/min for the system and an application can ask for a bigger buff, but by default it gets default, so changing the max isn't going to help by itself. > > > another thing to look for (you haven't mentioned you nic type or kernel > > version), but I had a realtek 8168b which would sort of work with the in > > kernel driver (8111c) on any kernel < 2.6.28. After 2.6.28 the in kernel > > driver worked really well. system would be under load packets would be > > come corrupt > > system info: > HP Proliant DL380 G5 2x Intel Xeon 2,3 GHZ quad core > 6GB Ram > Debian Lenny ver. 5.0.1 > kernel: 2.6.26-1-amd64 > > I´ve recompiled with dont optimize and debug threads and when the > asterisk hangs or lag i see in core show lock the following: > [snip] > > maybe this is a bug? I am not a asterisk person so can't comment on that. But another thought came to mind, maybe run 2 instances of asterisk - it seems like you have plenty of horse power here it looks like it should be able to cope (I am presume no translating ?) The other thing is HP have a PSP pack for there servers - they have just started to release debian based ones as well - current alien'ed rpm packages but its a start :) alex > > best regards > > steve > -- There is an old custom among my people. When a woman saves a man's life, he is grateful. -- Nona, the Kanuto witch woman, "A Private Little War", stardate 4211.8. signature.asc Description: Digital signature ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
Stefan, I'm not sure if you've considered the underlying hardware, firmware, and device drivers yet, but I was brought in to evaluate a site under heavy load and was able to stabilize things by applying vendor supplied drivers and firmware to the system. What is the underlying hardware (E.g. Dell PowerEdge 2950 Series 3, HP DL380G5, etc.) and OS distribution? I've seen plenty of timing issues (and resulting kernel panics) due to outdated firmware or unsupported/inappropriate drivers. To be clear, in most cases of brand named hardware, these will come from the manufacturer's repository (or website), not the OS distribution's. For example, if you're on HP hardware with RHEL 5.X, you might want to check HP's website for the latest firmware and kernel modules (e.g. drivers) for your system. Just a thought. Regards, Elliot -Original Message- From: asterisk-users-boun...@lists.digium.com [mailto:asterisk-users-boun...@lists.digium.com] On Behalf Of Stefan Schmidt Sent: Thursday, May 28, 2009 4:50 AM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] stucked calls in asterisk 1.4 David Backeberg schrieb: > On Wed, May 27, 2009 at 1:49 PM, Stefan Schmidt wrote: >> all server are in one rack in our datacenter and are connected to an HP >> Procurve 2650 switch, which has been setup around 3 months ago, cause of >> the old switch died silent in the night. >> >> all server had two interfaces and i have allready tried to route the >> traffic between the pbx and the routing server over the second >> interface, where database requests normally run. But this didnt solved >> the problem too. >> >> i will try to increase the UDP buffer size in the linux kernel, maybe >> this will take some affect. > > I will say that asterisk-1.6 is supposed to have a better SIP stack > than 1.4. Perhaps the difference in performance will help you. > Specifically, check out: > > http://svn.digium.com/svn/asterisk/branches/1.6.0/CHANGES > http://svn.digium.com/svn/asterisk/branches/1.6.1/CHANGES > > I'd recommend you go to at least 1.6.1.* series i will have a look at 1.6.1 version but the problem is a network/kernel problem. i´ve tried to set the udp rcv buffer to 1mb but that didnt change the problem. what i can see is that the asterisk itself sometimes lag so no call can be started, but running calls wont be broken. in netstat -su i see a big amount of udp packet receive errors when it lags. Like 600 to 800 pakets per second. in syslog i cant see anything that would cause asterisk to react like this. anyone an idea what can cause this problem? best regards steve ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users This message is intended only for the use of the individual (s) or entity to which it is addressed and may contain information that is privileged, confidential, and/or proprietary to Calling Circles LLC and its affiliates. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, forwarding or copying of this communication is prohibited without the express permission of the sender. If you have received this communication in error, please notify the sender immediately and delete the original message. ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
Alex Samad schrieb: > Hi Hi Alex, > > I am new to asterisk so my suggestions might be a bit silly. > > Why not setup a iax2 connection bettween the asterisk servers, because > its a lower overhear and more efficient. We had changed from iax connections to sip connections cause we had timing problems with iax. Also in meetme and when a client send a fax over the connection. This problems has gone after changing to sip connects. >> what i can see is that the asterisk itself sometimes lag so no call can >> be started, but running calls wont be broken. >> >> in netstat -su i see a big amount of udp packet receive errors when it >> lags. Like 600 to 800 pakets per second. > > if you are seeing packet loss, have you check the stats on the cards and > the ports on the switch any sort of loss might indicate a faulty > card/port its not a network packet loss its in the kernel or asterisk itself. > you could also try some network testing with iperf by default it tests > udp. > > the other thought is keep going the way youwere add another nic card to > server a such that you have > > eth0 for your sip clients > eth1 for router server b > eth2 for router server c i´ve allready tried this, and the problem still occurs. > it would just take a bit of routing magic. The thing to watch then is > the bus these card are on if they share the same pci bus. But you are > not really moving around much data. > > Try linux-kernel mailing list or do a google for udp buffer tunning > from memory there is the kernel option which is the abs max, but the > program must also select a buffer amount which is usually the default > value which is not the max value. increase the max udp buffer size was just something i´ve found to try but i´ve set the default value to 512kB by now, maybe this will help. > another thing to look for (you haven't mentioned you nic type or kernel > version), but I had a realtek 8168b which would sort of work with the in > kernel driver (8111c) on any kernel < 2.6.28. After 2.6.28 the in kernel > driver worked really well. system would be under load packets would be > come corrupt system info: HP Proliant DL380 G5 2x Intel Xeon 2,3 GHZ quad core 6GB Ram Debian Lenny ver. 5.0.1 kernel: 2.6.26-1-amd64 I´ve recompiled with dont optimize and debug threads and when the asterisk hangs or lag i see in core show lock the following: === === Currently Held Locks == === === === (times locked) === === Thread ID: 1092675920 (do_monitor started at [16595] chan_sip.c restart_monitor()) === ---> Lock #0 (chan_sip.c): MUTEX 16264 sipsock_read &netlock 0x7f6d0e9d0700 (1) === --- and a second before everything comes fine again: === === Currently Held Locks == === === === (times locked) === === Thread ID: 1092675920 (do_monitor started at [16595] chan_sip.c restart_monitor()) === ---> Lock #0 (chan_sip.c): MUTEX 16264 sipsock_read &netlock 0x7f6d0e9d0700 (1) === ---> Lock #1 (chan_sip.c): MUTEX 4728 find_call &p->lock 0x374ffd0 (1) === ---> Lock #2 (sched.c): MUTEX 220 ast_sched_add_variable &con->lock 0x1b2eee0 (1) === --- === === maybe this is a bug? best regards steve ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
On Thu, May 28, 2009 at 10:49:38AM +0200, Stefan Schmidt wrote: > > David Backeberg schrieb: > > On Wed, May 27, 2009 at 1:49 PM, Stefan Schmidt wrote: > >> all server are in one rack in our datacenter and are connected to an HP > >> Procurve 2650 switch, which has been setup around 3 months ago, cause of > >> the old switch died silent in the night. > >> > >> all server had two interfaces and i have allready tried to route the > >> traffic between the pbx and the routing server over the second > >> interface, where database requests normally run. But this didnt solved > >> the problem too. > >> > >> i will try to increase the UDP buffer size in the linux kernel, maybe > >> this will take some affect. > > > > I will say that asterisk-1.6 is supposed to have a better SIP stack > > than 1.4. Perhaps the difference in performance will help you. > > Specifically, check out: > > > > http://svn.digium.com/svn/asterisk/branches/1.6.0/CHANGES > > http://svn.digium.com/svn/asterisk/branches/1.6.1/CHANGES > > > > I'd recommend you go to at least 1.6.1.* series > > i will have a look at 1.6.1 version but the problem is a network/kernel > problem. > > i´ve tried to set the udp rcv buffer to 1mb but that didnt change the > problem. Hi I am new to asterisk so my suggestions might be a bit silly. Why not setup a iax2 connection bettween the asterisk servers, because its a lower overhear and more efficient. > > what i can see is that the asterisk itself sometimes lag so no call can > be started, but running calls wont be broken. > > in netstat -su i see a big amount of udp packet receive errors when it > lags. Like 600 to 800 pakets per second. if you are seeing packet loss, have you check the stats on the cards and the ports on the switch any sort of loss might indicate a faulty card/port you could also try some network testing with iperf by default it tests udp. the other thought is keep going the way youwere add another nic card to server a such that you have eth0 for your sip clients eth1 for router server b eth2 for router server c it would just take a bit of routing magic. The thing to watch then is the bus these card are on if they share the same pci bus. But you are not really moving around much data. Try linux-kernel mailing list or do a google for udp buffer tunning from memory there is the kernel option which is the abs max, but the program must also select a buffer amount which is usually the default value which is not the max value. another thing to look for (you haven't mentioned you nic type or kernel version), but I had a realtek 8168b which would sort of work with the in kernel driver (8111c) on any kernel < 2.6.28. After 2.6.28 the in kernel driver worked really well. system would be under load packets would be come corrupt for example ethtool -S eth0 NIC statistics: tx_packets: 75371387 rx_packets: 970026891 tx_errors: 0 rx_errors: 0 rx_missed: 0 align_errors: 0 tx_single_collisions: 0 tx_multi_collisions: 0 unicast: 969467810 broadcast: 201337 multicast: 357744 tx_aborted: 0 tx_underrun: 0 > > in syslog i cant see anything that would cause asterisk to react like this. > > anyone an idea what can cause this problem? > > best regards > > steve > > ___ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: >http://lists.digium.com/mailman/listinfo/asterisk-users > -- Documentation: Instructions translated from Swedish by Japanese for English speaking persons. signature.asc Description: Digital signature ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
David Backeberg schrieb: > On Wed, May 27, 2009 at 1:49 PM, Stefan Schmidt wrote: >> all server are in one rack in our datacenter and are connected to an HP >> Procurve 2650 switch, which has been setup around 3 months ago, cause of >> the old switch died silent in the night. >> >> all server had two interfaces and i have allready tried to route the >> traffic between the pbx and the routing server over the second >> interface, where database requests normally run. But this didnt solved >> the problem too. >> >> i will try to increase the UDP buffer size in the linux kernel, maybe >> this will take some affect. > > I will say that asterisk-1.6 is supposed to have a better SIP stack > than 1.4. Perhaps the difference in performance will help you. > Specifically, check out: > > http://svn.digium.com/svn/asterisk/branches/1.6.0/CHANGES > http://svn.digium.com/svn/asterisk/branches/1.6.1/CHANGES > > I'd recommend you go to at least 1.6.1.* series i will have a look at 1.6.1 version but the problem is a network/kernel problem. i´ve tried to set the udp rcv buffer to 1mb but that didnt change the problem. what i can see is that the asterisk itself sometimes lag so no call can be started, but running calls wont be broken. in netstat -su i see a big amount of udp packet receive errors when it lags. Like 600 to 800 pakets per second. in syslog i cant see anything that would cause asterisk to react like this. anyone an idea what can cause this problem? best regards steve ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
On Wed, May 27, 2009 at 1:49 PM, Stefan Schmidt wrote: > all server are in one rack in our datacenter and are connected to an HP > Procurve 2650 switch, which has been setup around 3 months ago, cause of > the old switch died silent in the night. > > all server had two interfaces and i have allready tried to route the > traffic between the pbx and the routing server over the second > interface, where database requests normally run. But this didnt solved > the problem too. > > i will try to increase the UDP buffer size in the linux kernel, maybe > this will take some affect. I will say that asterisk-1.6 is supposed to have a better SIP stack than 1.4. Perhaps the difference in performance will help you. Specifically, check out: http://svn.digium.com/svn/asterisk/branches/1.6.0/CHANGES http://svn.digium.com/svn/asterisk/branches/1.6.1/CHANGES I'd recommend you go to at least 1.6.1.* series ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
David Backeberg schrieb: > Now that I better understand your problem, I'm out of ideas. thats the point where i stand ;) > You are correct that if a BYE sip packet gets lost, > a) it won't get retransmitted if it's UDP > b) the side that's waiting for the hangup will think the call is still active > > I've seen this in my system where the network switch went down while > calls were active. The system where the call was happening caught the > hangup, but the trunk system never got the bye as the network was > down. > > My only questions are: > Are you using a quality network switch, or maybe you can use a > cross-over cable to eliminate collisions with other systems? > all server are in one rack in our datacenter and are connected to an HP Procurve 2650 switch, which has been setup around 3 months ago, cause of the old switch died silent in the night. all server had two interfaces and i have allready tried to route the traffic between the pbx and the routing server over the second interface, where database requests normally run. But this didnt solved the problem too. i will try to increase the UDP buffer size in the linux kernel, maybe this will take some affect. best regards steve ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
On Wed, May 27, 2009 at 10:30 AM, Stefan Schmidt wrote: > as i said the routing server also handles calls from an ser proxy and > another asterisk server where iax accounts terminates and this problem > is only on the pbx server. > > Maybe it is a network problem but the quality of the rtp streams is ok > but i think that there are too much sip pakets for the system. > > there are around 1600 sip users registerd, 300 - 400 sip channels > (register, options, notifys and invite) and 600 - 700 subscriptions so > there is much sip traffic. > > this server also does rtp handling and have 50 to 100 calls (active > ones) and in peek time there is around 10mbit of traffic with 5 to 6 kpps. > > maybe a problem with udp buffer size?? Now that I better understand your problem, I'm out of ideas. You are correct that if a BYE sip packet gets lost, a) it won't get retransmitted if it's UDP b) the side that's waiting for the hangup will think the call is still active I've seen this in my system where the network switch went down while calls were active. The system where the call was happening caught the hangup, but the trunk system never got the bye as the network was down. My only questions are: Are you using a quality network switch, or maybe you can use a cross-over cable to eliminate collisions with other systems? ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
David Backeberg schrieb: > On Wed, May 27, 2009 at 9:26 AM, Stefan Schmidt wrote: >> Server A call it PBX there are the sip clients connected >> A call comes from server B or C to server A and then to a client, gets >> stucked on Server A when PSTN side hangs up. On server B or C the call > > You may not have properly configured your card for the way to detect > hangups. If this is going to a proprietary PBX, you may need to change > around the line signaling. Some kinds of line signaling reverse the > polarity on the line voltage to signal a hangup. Other lines don't. > The cheap trick is to try your settings both ways and when one way > works to use that. its not a proprietary pbx its just a self developed asterisk and the server where the card is recognize the hangup, but the bye from the server (b or c) to the pbx dont work. as i said the routing server also handles calls from an ser proxy and another asterisk server where iax accounts terminates and this problem is only on the pbx server. Maybe it is a network problem but the quality of the rtp streams is ok but i think that there are too much sip pakets for the system. there are around 1600 sip users registerd, 300 - 400 sip channels (register, options, notifys and invite) and 600 - 700 subscriptions so there is much sip traffic. this server also does rtp handling and have 50 to 100 calls (active ones) and in peek time there is around 10mbit of traffic with 5 to 6 kpps. maybe a problem with udp buffer size?? thanks for your help so far david! best regards steve ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
On Wed, May 27, 2009 at 9:26 AM, Stefan Schmidt wrote: > Server A call it PBX there are the sip clients connected > A call comes from server B or C to server A and then to a client, gets > stucked on Server A when PSTN side hangs up. On server B or C the call You may not have properly configured your card for the way to detect hangups. If this is going to a proprietary PBX, you may need to change around the line signaling. Some kinds of line signaling reverse the polarity on the line voltage to signal a hangup. Other lines don't. The cheap trick is to try your settings both ways and when one way works to use that. ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
hello David Backeberg schrieb: > On Wed, May 27, 2009 at 7:32 AM, Stefan Schmidt wrote: >> i have a problem with stucked or hanging calls in asterisk 1.4.25 >> only appears on this server and not on the routing server. Even if i > > I'm confused. So the server where the calls get stuck has both SIP and > DAHDI/Zaptel channel calls? There are 3 servers. Server A call it PBX there are the sip clients connected Server B call it gateway1 has an Sangoma Card in it Server C call it gateway2 also has an sangoma Card in it A call comes from server B or C to server A and then to a client, gets stucked on Server A when PSTN side hangs up. On server B or C the call is closed. This also happens in other direction, if client dials out over Server a to server b or c to the pstn net. > And the SIP side of those calls isn't getting hungup? thats correct. > It is true that if your server doesn't receive a BYE packet it will > think that the call is still there. Does your dialplan that attaches > to the routing server do a hangup at the end of the call? on the routing server the call is closed every time. There we didnt have this problem. On this server (B+C) also terminate calls from a ser proxy and another asterisk server but the call stucks only on server A. best regards steve ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] stucked calls in asterisk 1.4
On Wed, May 27, 2009 at 7:32 AM, Stefan Schmidt wrote: > i have a problem with stucked or hanging calls in asterisk 1.4.25 > only appears on this server and not on the routing server. Even if i I'm confused. So the server where the calls get stuck has both SIP and DAHDI/Zaptel channel calls? And the SIP side of those calls isn't getting hungup? It is true that if your server doesn't receive a BYE packet it will think that the call is still there. Does your dialplan that attaches to the routing server do a hangup at the end of the call? ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
[asterisk-users] stucked calls in asterisk 1.4
hello, i have a problem with stucked or hanging calls in asterisk 1.4.25 we had this problem before and so we upgradet from 1.2.32 to 1.4.25 but it still exists and as i could see, happens even more. on this server there are > 1500 clients registered all with qualify on and we had 2 routing server with 4 E1 pstn connects on each. The connect between this server and the 2 routing server runs over sip. This problem only appears on this server and not on the routing server. Even if i soft hangup a stucked channel i get the sip response 481 call/leg transaction doesnt exists back from the routing server, so one call leg had allready send a bye but this server hasnt closed the call. i think there could be a network problem but also the server itself and the switch where the 3 servers are connected is younger than 3 months and we had the same problem with this system on an older server. sometimes i see that most of the sip peers get unreachable or too lagged so i think that there could be a problem for asterisk to handle that amount of pakets. i´ve just splittet up the traffic on 2 interfaces, so that normal traffic from the clients comes to eth0 and the traffic from and to the routing servers runs over eth1 but the problem still occurs. do you have any ideas what i could do to solve this problem? best regards steve smith ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
