Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Mon, Sep 7, 2009 at 4:19 PM, Benny Amorsen > wrote: > "Olle E. Johansson" writes: > > > Imaging my surprise this Monday when I installed a plain old Asterisk > > 1.4 on a new HP server, a DL380 G6, and could run in circles around > > the old IBM servers. > > The G6 series is pure magic for everything I've let it touch > network-wise. > > I have three guesses as to why: > > 1) Lots and lots of bandwidth between CPU and I/O, plus built-in memory > controller so any packet copying runs wicked fast. > > 2) MSI-X seems to really help, at least when combined with modern > ethernet chipsets (the original PRO/1000 is looking a bit dated now, but > more modern PRO/1000 should still be a good choice). > > 3) Multi-queue NIC. This should REALLY help when you have lots of cores > and CPU threads. Depends on fairly new kernels. > > I'm not sure which is the answer though. > > > /Benny > > Packets per second is going to be the eventual bottleneck no matter if it is Asterisk, FreeSwitch, or whatever. Using multiple switches will help but a backplane or interface. can only take so many PPS. http://en.wikipedia.org/wiki/Throughput ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
"Olle E. Johansson" writes: > Imaging my surprise this Monday when I installed a plain old Asterisk > 1.4 on a new HP server, a DL380 G6, and could run in circles around > the old IBM servers. The G6 series is pure magic for everything I've let it touch network-wise. I have three guesses as to why: 1) Lots and lots of bandwidth between CPU and I/O, plus built-in memory controller so any packet copying runs wicked fast. 2) MSI-X seems to really help, at least when combined with modern ethernet chipsets (the original PRO/1000 is looking a bit dated now, but more modern PRO/1000 should still be a good choice). 3) Multi-queue NIC. This should REALLY help when you have lots of cores and CPU threads. Depends on fairly new kernels. I'm not sure which is the answer though. /Benny ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
After dial.
I have put this in my hangup context as:
exten => h,1,Noop(QOS=${RTPAUDIOQOS})
-Original Message-
From: asterisk-users-boun...@lists.digium.com
[mailto:asterisk-users-boun...@lists.digium.com] On Behalf Of Klaus
Darilion
Sent: Thursday, August 27, 2009 13:04
To: Asterisk Users Mailing List - Non-Commercial Discussion
Subject: Re: [asterisk-users] Breaking news,but what happened? 11.000
channels on one server
John Todd wrote:
> 5) Any summary stats on RTP packet loss, etc? (from
> "CHANNEL(rtpqos,audio,all)") on channels?
I wonder how to retrieve those stats:
- after Dial()?
- during Dial()? (how?)
regards
klaus
___
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
AstriCon 2009 - October 13 - 15 Phoenix, Arizona
Register Now: http://www.astricon.net
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
John Todd wrote: > 5) Any summary stats on RTP packet loss, etc? (from > "CHANNEL(rtpqos,audio,all)") on channels? I wonder how to retrieve those stats: - after Dial()? - during Dial()? (how?) regards klaus ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
What's the big deal I can get 11.000 (11) calls on an Acer netbook ;-). 11,000 Wow! My testing has shown that multiple core really didn't do much for asterisk scaling.. My tests show a very busy first core and the rest idle. However I am a big fan of HP servers I run 8 core DL360's VMware hosts and they crank. If you are indeed seeing 11,000 calls that is very encouraging. I have had a sinking feeling I that I was going to have to move from asterisk to something else more scalable.. Doug -Original Message- From: asterisk-users-boun...@lists.digium.com [mailto:asterisk-users-boun...@lists.digium.com] On Behalf Of John Todd Sent: Thursday, August 27, 2009 10:54 AM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] Breaking news,but what happened? 11.000 channels on one server On Aug 25, 2009, at 5:59 AM, Olle E. Johansson wrote: > Hello Asterisk users around the world! > > Recently, I have been working with pretty large Asterisk > installations. 300 servers running Asterisk and Kamailio (OpenSER). > Replacing large Nortel systems with just a few tiny boxes and other > interesting solutions. Testing has been a large part of these > projects. How much can we put into one Asterisk box? Calls per euro > invested matters. > > So far, we've been able to reach about 2000 channels of G.711 with > quad core CPU's and Intel Pro/1000 network cards in IBM servers. At > that point, we see that IRQ balancer gives up and goes to bed, and all > the traffic is directed to one core and the system gives up. We've > been running these tests on several systems, with different NICs and > have been working hard to tweak capacity. New drivers, new cards, new > stuff. But all indications told us that the problem was the CPU > context switching between handling network traffic (RTP traffic) and > Asterisk. This was also confirmed from a few different independent > software development teams. > > Imaging my surprise this Monday when I installed a plain old Asterisk > 1.4 on a new HP server, a DL380 G6, and could run in circles around > the old IBM servers. Three servers looping calls between them and we > bypassed 10.000 channels without any issues. SIP to SIP calls, the > p2p RTP bridge, basically running a media proxy. At that point, our > cheap gigabit switch gave up, and of course the NICs. Pushing 850 Mbit > was more than enough. The CPU's (we had 16 of them with > hyperthreading) was not very stressed. Asterisk was occupying a few of > them in a nice way, but we had a majority of them idling around > looking for something to do.I'll > > So, please help me. I need answers to John Todds questions while he's > treating me with really good expensive wine at Astricon. How did this > happen? Was it the Broadcom NICs? Was it the Intel 5530 Xeon CPU's? Or > a combination? Or maybe just the cheap Netgear switch... > > I hope to get more access to these boxes, three of them, to run tests > with the latest code. In that version we have the new hashtables, all > the refcounters and fancy stuff that the Digium team has reworked on > the inside of Asterisk. The trunk version will propably behave much, > much better than 1.4 when it comes to heavy loads and high call setup > rates. > > We're on our way to build a new generation of Asterisk, far away from > the 1.0 platform. At the same time, the hardware guys have obviously > not been asleep. They're giving us inexpensive hardware that makes our > software shine. Now we need to test other things and see how the rest > of Asterisk scales, apart from the actual calls. Manager, events, > musiconhold, agi/fastagi... New interesting challenges. > > So take one of these standard rack servers from HP and run a telco for > a small city on one box. While you're at it, buy a spare one, hardware > can fail ( ;-) ). > But don't say that Asterisk does not scale well. Those times are gone. > > /Olle > > --- > * Olle E Johansson - o...@edvina.net > * Open Unified Communication - SIP & XMPP projects > Your dinner awaits, along with a very nice bottle of wine. (or port, or whatever it is you prefer.) But, just a few questions.. ;-) Moving back away from Layer 3 discussion that erupted on this thread, let's get back to what you actually did. It seems odd to me (though I'm hopeful!) that this would "just work" without any changes. Especially on 1.4. So being somewhat scientific about it, I'd say that the first thing to do is to examine your measurements with an assumption that there is a flaw in your observations. If you find no errors with that hypothesis, then you've deduced that things are as they seem. :-) 1) Are you certain that the medi
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Aug 25, 2009, at 5:59 AM, Olle E. Johansson wrote: > Hello Asterisk users around the world! > > Recently, I have been working with pretty large Asterisk > installations. 300 servers running Asterisk and Kamailio (OpenSER). > Replacing large Nortel systems with just a few tiny boxes and other > interesting solutions. Testing has been a large part of these > projects. How much can we put into one Asterisk box? Calls per euro > invested matters. > > So far, we've been able to reach about 2000 channels of G.711 with > quad core CPU's and Intel Pro/1000 network cards in IBM servers. At > that point, we see that IRQ balancer gives up and goes to bed, and all > the traffic is directed to one core and the system gives up. We've > been running these tests on several systems, with different NICs and > have been working hard to tweak capacity. New drivers, new cards, new > stuff. But all indications told us that the problem was the CPU > context switching between handling network traffic (RTP traffic) and > Asterisk. This was also confirmed from a few different independent > software development teams. > > Imaging my surprise this Monday when I installed a plain old Asterisk > 1.4 on a new HP server, a DL380 G6, and could run in circles around > the old IBM servers. Three servers looping calls between them and we > bypassed 10.000 channels without any issues. SIP to SIP calls, the > p2p RTP bridge, basically running a media proxy. At that point, our > cheap gigabit switch gave up, and of course the NICs. Pushing 850 Mbit > was more than enough. The CPU's (we had 16 of them with > hyperthreading) was not very stressed. Asterisk was occupying a few of > them in a nice way, but we had a majority of them idling around > looking for something to do.I'll > > So, please help me. I need answers to John Todds questions while he's > treating me with really good expensive wine at Astricon. How did this > happen? Was it the Broadcom NICs? Was it the Intel 5530 Xeon CPU's? Or > a combination? Or maybe just the cheap Netgear switch... > > I hope to get more access to these boxes, three of them, to run tests > with the latest code. In that version we have the new hashtables, all > the refcounters and fancy stuff that the Digium team has reworked on > the inside of Asterisk. The trunk version will propably behave much, > much better than 1.4 when it comes to heavy loads and high call setup > rates. > > We're on our way to build a new generation of Asterisk, far away from > the 1.0 platform. At the same time, the hardware guys have obviously > not been asleep. They're giving us inexpensive hardware that makes our > software shine. Now we need to test other things and see how the rest > of Asterisk scales, apart from the actual calls. Manager, events, > musiconhold, agi/fastagi... New interesting challenges. > > So take one of these standard rack servers from HP and run a telco for > a small city on one box. While you're at it, buy a spare one, hardware > can fail ( ;-) ). > But don't say that Asterisk does not scale well. Those times are gone. > > /Olle > > --- > * Olle E Johansson - o...@edvina.net > * Open Unified Communication - SIP & XMPP projects > Your dinner awaits, along with a very nice bottle of wine. (or port, or whatever it is you prefer.) But, just a few questions.. ;-) Moving back away from Layer 3 discussion that erupted on this thread, let's get back to what you actually did. It seems odd to me (though I'm hopeful!) that this would "just work" without any changes. Especially on 1.4. So being somewhat scientific about it, I'd say that the first thing to do is to examine your measurements with an assumption that there is a flaw in your observations. If you find no errors with that hypothesis, then you've deduced that things are as they seem. :-) 1) Are you certain that the media was actually being routed? I know you said that the switch and NICs gave up because of traffic, but did you choose a random channel and record the media between two servers? In other words, are you 100% certain that there was valid RTP being exchanged? (I typically make one call out of every 100 or 1000 a "monkeys" call, and record it instead of just routing to Echo() directly, then play back later to ensure media was actually happening.) 2) Were the SIP transactions completing normally? What was the rate of ramp-up? 3) Can you post your dialplan and a few snippets of "core show channels" at peak usage? 4) What was the sampling rate for the media? 20ms? 30ms? 40ms? 5) Any summary stats on RTP packet loss, etc? (from "CHANNEL(rtpqos,audio,all)") on channels? JT --- John Todd email:jt...@digium.com Digium, Inc. | Asterisk Open Source Community Director 445 Jan Davis Drive NW - Huntsville AL 35806 - USA direct: +1-256-428-6083 http://www.digium.com/ ___ -- Bandwidth and Colocation Provided by http://www.api-d
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Wed, Aug 26, 2009 at 11:35 AM, Peder wrote: > On a side note, why do people buy high-end servers like IBM or HP and then > put in crappy switches, like Dell or Netgear and then wonder why performance > is bad? That's like buying a BMW 7 series and then using the cheapest 87 > octane gas you can find. If you want to build a good network, buy good > network equipment like Cisco, Extreme, Foundry or some of the other high end > manufacturers. DLink, Linksys, Netgear and Dell are all low end consumer > grade. No matter how they may try and sell it, that's what it is. It is > fine for that, but it is not enterprise grade equipment. I just ran across > a customer that had a Cisco Catalyst 4000 with an uptime of 1500 days (4+ > years). Try and get that with Linksys or Netgear. I actually agree with your argument, and I've personally told people before to chuck their Dell switch overboard and replace it with a crossover cable. However, I sure hope that four-years-ago IOS Cisco switch isn't on the edge. That's way too long to not upgrade your IOS. ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
You are thinking IP (layer3), not mac address (layer2 - ethernet switching). Bonding is general a poor choice of wording for multiple Ethernet connections as an individual connection won't use both links. The way most NIC's and switches do bonding is that they hash the source and destination mac address and odd packets go over one link and even packets go over the other (assuming two links). So if there are two machines talking, they will flood one link and the other will be empty. If there are 100 machines talking to one machine, then it will be fairly even balancing. If the PBX is behind a router or firewall, then it will see all external IPs as one mac (the router or firewall), so you again will get one link saturated and one empty. Your best bet is 10G if you can afford it. On a side note, why do people buy high-end servers like IBM or HP and then put in crappy switches, like Dell or Netgear and then wonder why performance is bad? That's like buying a BMW 7 series and then using the cheapest 87 octane gas you can find. If you want to build a good network, buy good network equipment like Cisco, Extreme, Foundry or some of the other high end manufacturers. DLink, Linksys, Netgear and Dell are all low end consumer grade. No matter how they may try and sell it, that's what it is. It is fine for that, but it is not enterprise grade equipment. I just ran across a customer that had a Cisco Catalyst 4000 with an uptime of 1500 days (4+ years). Try and get that with Linksys or Netgear. -Original Message- From: asterisk-users-boun...@lists.digium.com [mailto:asterisk-users-boun...@lists.digium.com] On Behalf Of David Backeberg Sent: Tuesday, August 25, 2009 8:58 PM To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server On Tue, Aug 25, 2009 at 12:50 PM, John A. Sullivan III wrote: > You don't necessarily need a switch to support it. One can use alb mode > in Linux on any old switch and it works reasonably well other than for > some excessive ARP traffic. However, as we found out the hard way when > building our Nexenta SAN, bonding works very well with many-to-many > traffic but does very little to boost one-to-one network flows. They > will all collapse to the same pair of NICs in most scenarios and, in the > one mode where they do not, packet sequencing issues will reduce the > bandwidth to much less than the sum of the connections. Take care - Your claims make sense for a typical Machine A has one IP address Machine B has one IP address And there is only one route between A and B. In this scenario, yes, all calls take same route. But what about giving each machine two addresses, two routes. And halve your calls between the two paths between the same systems. Doesn't this get around your problem, and allow you a chance to saturate double the number of interfaces? If you have four interfaces (as my new boxes do), lather, rinse, repeat. Anybody have any reason why spreading the bandwidth across multiple routes wouldn't get around this problem? ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Wed, 2009-08-26 at 06:18 +1000, Alex Samad wrote: > > any thoughts of different media like 10G ethernet or infiniband ? > For another project i had a look at 10G. Prices of the nic's were reasonable, but even an 5-ports nice were mind blowing (>>35,000 Euro) So i opted for multiple nic's and back-to-back (for interconncting backbones) hw ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Tue, 2009-08-25 at 21:57 -0400, David Backeberg wrote: > On Tue, Aug 25, 2009 at 12:50 PM, John A. Sullivan > III wrote: > > You don't necessarily need a switch to support it. One can use alb mode > > in Linux on any old switch and it works reasonably well other than for > > some excessive ARP traffic. However, as we found out the hard way when > > building our Nexenta SAN, bonding works very well with many-to-many > > traffic but does very little to boost one-to-one network flows. They > > will all collapse to the same pair of NICs in most scenarios and, in the > > one mode where they do not, packet sequencing issues will reduce the > > bandwidth to much less than the sum of the connections. Take care - > > Your claims make sense for a typical > Machine A has one IP address > Machine B has one IP address > > And there is only one route between A and B. In this scenario, yes, > all calls take same route. > > But what about giving each machine two addresses, two routes. And > halve your calls between the two paths between the same systems. > Doesn't this get around your problem, and allow you a chance to > saturate double the number of interfaces? > > If you have four interfaces (as my new boxes do), lather, rinse, > repeat. Anybody have any reason why spreading the bandwidth across > multiple routes wouldn't get around this problem? Yes, that's correct and exactly what we did in our SAN environment. There are some issues. You will generally not want them all on the same IP network - the inbound traffic may spread across the four addresses if told to do so but the reply traffic will likely go out the default interface. If they are four distinct IP networks, it means dividing the end users among the multiple networks. In the case of our SAN, we did it without a router using logical networks on the same physical medium. With iproute2 and secondary routing tables, one can be even more creative. In fact, having many phones going to one Asterisk device will probably work well with bonding because it is many to one and each combination of MAC addresses will be treated as a different traffic stream. However, if I recall, the testing environment was two or three asterisk systems talking to each other, wasn't it? - John -- John A. Sullivan III Open Source Development Corporation +1 207-985-7880 jsulli...@opensourcedevel.com http://www.spiritualoutreach.com Making Christianity intelligible to secular society ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Tue, Aug 25, 2009 at 12:50 PM, John A. Sullivan III wrote: > You don't necessarily need a switch to support it. One can use alb mode > in Linux on any old switch and it works reasonably well other than for > some excessive ARP traffic. However, as we found out the hard way when > building our Nexenta SAN, bonding works very well with many-to-many > traffic but does very little to boost one-to-one network flows. They > will all collapse to the same pair of NICs in most scenarios and, in the > one mode where they do not, packet sequencing issues will reduce the > bandwidth to much less than the sum of the connections. Take care - Your claims make sense for a typical Machine A has one IP address Machine B has one IP address And there is only one route between A and B. In this scenario, yes, all calls take same route. But what about giving each machine two addresses, two routes. And halve your calls between the two paths between the same systems. Doesn't this get around your problem, and allow you a chance to saturate double the number of interfaces? If you have four interfaces (as my new boxes do), lather, rinse, repeat. Anybody have any reason why spreading the bandwidth across multiple routes wouldn't get around this problem? ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Wed, 2009-08-26 at 06:18 +1000, Alex Samad wrote: > On Tue, Aug 25, 2009 at 07:30:08PM +0200, Olle E. Johansson wrote: > > > > 25 aug 2009 kl. 18.50 skrev John A. Sullivan III: > > > > > On Tue, 2009-08-25 at 18:28 +0200, Olle E. Johansson wrote: > > >> 25 aug 2009 kl. 16.20 skrev Olivier: > > [snip] > > > > mode > > > in Linux on any old switch and it works reasonably well other than for > > > some excessive ARP traffic. However, as we found out the hard way > > > when > > > building our Nexenta SAN, bonding works very well with many-to-many > > > traffic but does very little to boost one-to-one network flows. They > > > will all collapse to the same pair of NICs in most scenarios and, in > > > the > > > one mode where they do not, packet sequencing issues will reduce the > > > bandwidth to much less than the sum of the connections. Take care - > > > John > > > > That is very good feedback - thanks, John! > > > > Which means that my plan B needs to be put in action. Well, I did > > create a new branch for it yesterday... ;-) > > any thoughts of different media like 10G ethernet or infiniband ? > > > Yes, this is drifting a little off-topic but good network design does provide the foundation for good Asterisk design. If we have lots of servers talking to lots of servers, bonding over Gig links works very well. But as we build fewer very big servers via virtualization or, as in this case, trying to make a single large server do the work previously handled by several, the network bandwidth becomes a huge issue. Because almost all bonding algorithms choose a single path for a flow of data (usually based upon MAC address but sometimes on IP address or even socket), bonding becomes less useful in these scenarios. In fact, it is even worse - even in cases where the OS stack (e.g., Linux) will support bonding based upon data above the MAC layer, the switches frequently do not and will again collapse several paths into one as soon as the data crosses the switch. Thus, for few-to-few traffic patterns, bigger pipes such as 10G are better than bonded pipes. Specifically, 10 bonded 1 Gbps links will effectively yield 1 Gbps throughput as opposed to 1 10Gbps link yielding 10 Gpbs throughput. As an aside, in our iSCSI work, we found latency to be a huge issue if the file block size was small (e.g., Linux files - 4K block size). Thus, the lower latency of faster protocols is a huge performance booster. This will not be so much of an issue with Asterisk where the difference between 100 usecs and 10 usecs in negligible. -- John A. Sullivan III Open Source Development Corporation +1 207-985-7880 jsulli...@opensourcedevel.com http://www.spiritualoutreach.com Making Christianity intelligible to secular society ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Tue, Aug 25, 2009 at 07:30:08PM +0200, Olle E. Johansson wrote: > > 25 aug 2009 kl. 18.50 skrev John A. Sullivan III: > > > On Tue, 2009-08-25 at 18:28 +0200, Olle E. Johansson wrote: > >> 25 aug 2009 kl. 16.20 skrev Olivier: [snip] > > mode > > in Linux on any old switch and it works reasonably well other than for > > some excessive ARP traffic. However, as we found out the hard way > > when > > building our Nexenta SAN, bonding works very well with many-to-many > > traffic but does very little to boost one-to-one network flows. They > > will all collapse to the same pair of NICs in most scenarios and, in > > the > > one mode where they do not, packet sequencing issues will reduce the > > bandwidth to much less than the sum of the connections. Take care - > > John > > That is very good feedback - thanks, John! > > Which means that my plan B needs to be put in action. Well, I did > create a new branch for it yesterday... ;-) any thoughts of different media like 10G ethernet or infiniband ? > > /O > > ___ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > AstriCon 2009 - October 13 - 15 Phoenix, Arizona > Register Now: http://www.astricon.net > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: >http://lists.digium.com/mailman/listinfo/asterisk-users > -- "I mean, these good folks are revolutionizing how businesses conduct their business. And, like them, I am very optimistic about our position in the world and about its influence on the United States. We're concerned about the short-term economic news, but long-term I'm optimistic. And so, I hope investors, you know -secondly, I hope investors hold investments for periods of time -that I've always found the best investments are those that you salt away based on economics. " - George W. Bush 01/04/2001 Austin, TX signature.asc Description: Digital signature ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
25 aug 2009 kl. 19.42 skrev Steve Totaro: > > > On Tue, Aug 25, 2009 at 12:28 PM, Olle E. Johansson > wrote: > > 25 aug 2009 kl. 16.20 skrev Olivier: > > > I would be curious to know if bonding 2 Ethernet ports together > > would help to push the upper limit a bit further ... > > (by the way, this limit is 11000 channels or 5500 calls, isn't it ? > > Yes, this is 11.000 channels. > > Bonding is good advice, provided we have a switch that can handle > that. Gotta find a place to borrow such a switch. > > /Olle > > Even the el cheapo Dell web managed switches can do this and port > mirroring. Cool. Will look into that. Regardless, I need to find a non-el cheapo customer that wants to sponsor further work on this if I'm going to get my hands on a switch and get some time for more tests :-) Mail me off-list if you're interested! /O ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Tue, Aug 25, 2009 at 12:28 PM, Olle E. Johansson wrote: > > 25 aug 2009 kl. 16.20 skrev Olivier: > > > I would be curious to know if bonding 2 Ethernet ports together > > would help to push the upper limit a bit further ... > > (by the way, this limit is 11000 channels or 5500 calls, isn't it ? > > Yes, this is 11.000 channels. > > Bonding is good advice, provided we have a switch that can handle > that. Gotta find a place to borrow such a switch. > > /Olle > Even the el cheapo Dell web managed switches can do this and port mirroring. -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
25 aug 2009 kl. 18.50 skrev John A. Sullivan III: > On Tue, 2009-08-25 at 18:28 +0200, Olle E. Johansson wrote: >> 25 aug 2009 kl. 16.20 skrev Olivier: >> >>> I would be curious to know if bonding 2 Ethernet ports together >>> would help to push the upper limit a bit further ... >>> (by the way, this limit is 11000 channels or 5500 calls, isn't it ? >> >> Yes, this is 11.000 channels. >> >> Bonding is good advice, provided we have a switch that can handle >> that. Gotta find a place to borrow such a switch. >> >> /Olle > You don't necessarily need a switch to support it. One can use alb > mode > in Linux on any old switch and it works reasonably well other than for > some excessive ARP traffic. However, as we found out the hard way > when > building our Nexenta SAN, bonding works very well with many-to-many > traffic but does very little to boost one-to-one network flows. They > will all collapse to the same pair of NICs in most scenarios and, in > the > one mode where they do not, packet sequencing issues will reduce the > bandwidth to much less than the sum of the connections. Take care - > John That is very good feedback - thanks, John! Which means that my plan B needs to be put in action. Well, I did create a new branch for it yesterday... ;-) /O ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Tue, 2009-08-25 at 18:28 +0200, Olle E. Johansson wrote: > 25 aug 2009 kl. 16.20 skrev Olivier: > > > I would be curious to know if bonding 2 Ethernet ports together > > would help to push the upper limit a bit further ... > > (by the way, this limit is 11000 channels or 5500 calls, isn't it ? > > Yes, this is 11.000 channels. > > Bonding is good advice, provided we have a switch that can handle > that. Gotta find a place to borrow such a switch. > > /Olle You don't necessarily need a switch to support it. One can use alb mode in Linux on any old switch and it works reasonably well other than for some excessive ARP traffic. However, as we found out the hard way when building our Nexenta SAN, bonding works very well with many-to-many traffic but does very little to boost one-to-one network flows. They will all collapse to the same pair of NICs in most scenarios and, in the one mode where they do not, packet sequencing issues will reduce the bandwidth to much less than the sum of the connections. Take care - John -- John A. Sullivan III Open Source Development Corporation +1 207-985-7880 jsulli...@opensourcedevel.com http://www.spiritualoutreach.com Making Christianity intelligible to secular society ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
25 aug 2009 kl. 16.20 skrev Olivier: > I would be curious to know if bonding 2 Ethernet ports together > would help to push the upper limit a bit further ... > (by the way, this limit is 11000 channels or 5500 calls, isn't it ? Yes, this is 11.000 channels. Bonding is good advice, provided we have a switch that can handle that. Gotta find a place to borrow such a switch. /Olle ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
I would be curious to know if bonding 2 Ethernet ports together would help to push the upper limit a bit further ... (by the way, this limit is 11000 channels or 5500 calls, isn't it ? ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
On Tue, Aug 25, 2009 at 8:59 AM, Olle E. Johansson wrote: > Hello Asterisk users around the world! > > Recently, I have been working with pretty large Asterisk > installations. 300 servers running Asterisk and Kamailio (OpenSER). > Replacing large Nortel systems with just a few tiny boxes and other > interesting solutions. Testing has been a large part of these > projects. How much can we put into one Asterisk box? Calls per euro > invested matters. > > So far, we've been able to reach about 2000 channels of G.711 with > quad core CPU's and Intel Pro/1000 network cards in IBM servers. At > that point, we see that IRQ balancer gives up and goes to bed, and all > the traffic is directed to one core and the system gives up. We've > been running these tests on several systems, with different NICs and > have been working hard to tweak capacity. New drivers, new cards, new > stuff. But all indications told us that the problem was the CPU > context switching between handling network traffic (RTP traffic) and > Asterisk. This was also confirmed from a few different independent > software development teams. > > Imaging my surprise this Monday when I installed a plain old Asterisk > 1.4 on a new HP server, a DL380 G6, and could run in circles around > the old IBM servers. Three servers looping calls between them and we > bypassed 10.000 channels without any issues. SIP to SIP calls, the > p2p RTP bridge, basically running a media proxy. At that point, our > cheap gigabit switch gave up, and of course the NICs. Pushing 850 Mbit > was more than enough. The CPU's (we had 16 of them with > hyperthreading) was not very stressed. Asterisk was occupying a few of > them in a nice way, but we had a majority of them idling around > looking for something to do. > > So, please help me. I need answers to John Todds questions while he's > treating me with really good expensive wine at Astricon. How did this > happen? Was it the Broadcom NICs? Was it the Intel 5530 Xeon CPU's? Or > a combination? Or maybe just the cheap Netgear switch... > > I hope to get more access to these boxes, three of them, to run tests > with the latest code. In that version we have the new hashtables, all > the refcounters and fancy stuff that the Digium team has reworked on > the inside of Asterisk. The trunk version will propably behave much, > much better than 1.4 when it comes to heavy loads and high call setup > rates. > > We're on our way to build a new generation of Asterisk, far away from > the 1.0 platform. At the same time, the hardware guys have obviously > not been asleep. They're giving us inexpensive hardware that makes our > software shine. Now we need to test other things and see how the rest > of Asterisk scales, apart from the actual calls. Manager, events, > musiconhold, agi/fastagi... New interesting challenges. > > So take one of these standard rack servers from HP and run a telco for > a small city on one box. While you're at it, buy a spare one, hardware > can fail ( ;-) ). > But don't say that Asterisk does not scale well. Those times are gone. > > /Olle > > --- > * Olle E Johansson - o...@edvina.net > * Open Unified Communication - SIP & XMPP projects > > I always was a fan and recommended IBM DL380s if not 360s (dual power supply). I would like to see some benchmarking on the AMI. Not sure how to do it but that used to be a very weak link. I wonder if, and how much it has improved over 1.2.x -- Thanks, Steve Totaro +18887771888 (Toll Free) +12409381212 (Cell) +12024369784 (Skype) ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [asterisk-users] Breaking news, but what happened? 11.000 channels on one server
Hi, what machines where the IBM Servers? I would be really interested in this as we have currently IBM Hardware deployed and, well, maybe it's time to investigate in different hardware, best -- Raimund Sacherer - RunSolutions Open Source It Consulting - Email: r...@runsolutions.com tel: 625 40 32 08 Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 25, 2009, at 2:59 PM, Olle E. Johansson wrote: Hello Asterisk users around the world! Recently, I have been working with pretty large Asterisk installations. 300 servers running Asterisk and Kamailio (OpenSER). Replacing large Nortel systems with just a few tiny boxes and other interesting solutions. Testing has been a large part of these projects. How much can we put into one Asterisk box? Calls per euro invested matters. So far, we've been able to reach about 2000 channels of G.711 with quad core CPU's and Intel Pro/1000 network cards in IBM servers. At that point, we see that IRQ balancer gives up and goes to bed, and all the traffic is directed to one core and the system gives up. We've been running these tests on several systems, with different NICs and have been working hard to tweak capacity. New drivers, new cards, new stuff. But all indications told us that the problem was the CPU context switching between handling network traffic (RTP traffic) and Asterisk. This was also confirmed from a few different independent software development teams. Imaging my surprise this Monday when I installed a plain old Asterisk 1.4 on a new HP server, a DL380 G6, and could run in circles around the old IBM servers. Three servers looping calls between them and we bypassed 10.000 channels without any issues. SIP to SIP calls, the p2p RTP bridge, basically running a media proxy. At that point, our cheap gigabit switch gave up, and of course the NICs. Pushing 850 Mbit was more than enough. The CPU's (we had 16 of them with hyperthreading) was not very stressed. Asterisk was occupying a few of them in a nice way, but we had a majority of them idling around looking for something to do. So, please help me. I need answers to John Todds questions while he's treating me with really good expensive wine at Astricon. How did this happen? Was it the Broadcom NICs? Was it the Intel 5530 Xeon CPU's? Or a combination? Or maybe just the cheap Netgear switch... I hope to get more access to these boxes, three of them, to run tests with the latest code. In that version we have the new hashtables, all the refcounters and fancy stuff that the Digium team has reworked on the inside of Asterisk. The trunk version will propably behave much, much better than 1.4 when it comes to heavy loads and high call setup rates. We're on our way to build a new generation of Asterisk, far away from the 1.0 platform. At the same time, the hardware guys have obviously not been asleep. They're giving us inexpensive hardware that makes our software shine. Now we need to test other things and see how the rest of Asterisk scales, apart from the actual calls. Manager, events, musiconhold, agi/fastagi... New interesting challenges. So take one of these standard rack servers from HP and run a telco for a small city on one box. While you're at it, buy a spare one, hardware can fail ( ;-) ). But don't say that Asterisk does not scale well. Those times are gone. /Olle --- * Olle E Johansson - o...@edvina.net * Open Unified Communication - SIP & XMPP projects ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
[asterisk-users] Breaking news, but what happened? 11.000 channels on one server
Hello Asterisk users around the world! Recently, I have been working with pretty large Asterisk installations. 300 servers running Asterisk and Kamailio (OpenSER). Replacing large Nortel systems with just a few tiny boxes and other interesting solutions. Testing has been a large part of these projects. How much can we put into one Asterisk box? Calls per euro invested matters. So far, we've been able to reach about 2000 channels of G.711 with quad core CPU's and Intel Pro/1000 network cards in IBM servers. At that point, we see that IRQ balancer gives up and goes to bed, and all the traffic is directed to one core and the system gives up. We've been running these tests on several systems, with different NICs and have been working hard to tweak capacity. New drivers, new cards, new stuff. But all indications told us that the problem was the CPU context switching between handling network traffic (RTP traffic) and Asterisk. This was also confirmed from a few different independent software development teams. Imaging my surprise this Monday when I installed a plain old Asterisk 1.4 on a new HP server, a DL380 G6, and could run in circles around the old IBM servers. Three servers looping calls between them and we bypassed 10.000 channels without any issues. SIP to SIP calls, the p2p RTP bridge, basically running a media proxy. At that point, our cheap gigabit switch gave up, and of course the NICs. Pushing 850 Mbit was more than enough. The CPU's (we had 16 of them with hyperthreading) was not very stressed. Asterisk was occupying a few of them in a nice way, but we had a majority of them idling around looking for something to do. So, please help me. I need answers to John Todds questions while he's treating me with really good expensive wine at Astricon. How did this happen? Was it the Broadcom NICs? Was it the Intel 5530 Xeon CPU's? Or a combination? Or maybe just the cheap Netgear switch... I hope to get more access to these boxes, three of them, to run tests with the latest code. In that version we have the new hashtables, all the refcounters and fancy stuff that the Digium team has reworked on the inside of Asterisk. The trunk version will propably behave much, much better than 1.4 when it comes to heavy loads and high call setup rates. We're on our way to build a new generation of Asterisk, far away from the 1.0 platform. At the same time, the hardware guys have obviously not been asleep. They're giving us inexpensive hardware that makes our software shine. Now we need to test other things and see how the rest of Asterisk scales, apart from the actual calls. Manager, events, musiconhold, agi/fastagi... New interesting challenges. So take one of these standard rack servers from HP and run a telco for a small city on one box. While you're at it, buy a spare one, hardware can fail ( ;-) ). But don't say that Asterisk does not scale well. Those times are gone. /Olle --- * Olle E Johansson - o...@edvina.net * Open Unified Communication - SIP & XMPP projects ___ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- AstriCon 2009 - October 13 - 15 Phoenix, Arizona Register Now: http://www.astricon.net asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
