Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Waldo, Thanks for the information. If you don't mind answering: are you guys developing this solution for your internal needs (meaning serving UAs from within your enterprise) or are you planning on offering services to the public? This solution is being developed for our internal needs. It's not that I'm really interested in your business or business model. I'm mainly curious to know how you will deal with potential UAs that are behind external NATs. Will you Asterisk farm stand behind a NAT or will it all be publicly accessible where no NAT translation or port forwarding will exist? I read the section on Asterisk and NAT on the wiki but still left me with some open questions. Our SIP traffic will never leave our internal network. There will be no NAT/firewalls to traverse. Calls to/from the PSTN will pass through a Cisco AS5400HPX Universal Gateway that handles the TDM/VoIP translation. In my particular setup, I work in a small call center. I have Asterisk behind one NAT with port forwarding on port 5060 and ports 1-2, only because I have 2 remote agents. The rest of the agents are in-house. The remote agents themselves are behind other NATs (behind their DSL service provider). Some times my Asterisk queues have trouble contacting the remote agents. At first, I thought I could simply put a SER server on the public edge, but I'm not sure if that will really solve the problem. I question this setup in terms of stability and security. Even worse, what would happen if my boss decides to increase the remote agents? I spoke to you privately about this and suggested using the IAX protocol with IAXy devices, but you indicated you needed to use SIP. Since we are not dealing with remote agents in our implementation, that is really all I can offer. I hope that the list members will be able to help you solve your problem. Sincerely, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Matt,Thanks for the information. If you don't mind answering: are you guys developing this solution for your internal needs (meaning serving UAs from within your enterprise) or are you planning on offering services to the public?It's not that I'm really interested in your business or business model. I'm mainly curious to know how you will deal with potential UAs that are behind external NATs. Will you Asterisk "farm" stand behind a NAT or will it all be "publicly" accessible where no NAT translation or port forwarding will exist? I read the section on Asterisk and NAT on the wiki but still left me with some open questions.In my particular setup, I work in a small call center. I have Asterisk behind one NAT with port forwarding on port 5060 and ports 1-2, only because I have 2 remote agents. The rest of the agents are in-house. The remote agents themselves are behind other NATs (behind their DSL service provider). Some times my Asterisk queues have trouble contacting the remote agents. At first, I thought I could simply put a SER server on the public edge, but I'm not sure if that will really solve the problem. I question this setup in terms of stability and security. Even worse, what would happen if my boss decides to increase the remote agents?Thanks again,WaldoOn Sep 22, 2005, at 7:24 PM, Matt Roth wrote: Waldo, - Is your solution 100% Asterisk or are you using other "helpers" such as SER or XXXproxy or whatever? We are not using SER or XXXproxy. We are using a Cisco AS5400HPX Universal Gateway to terminate our Ts. The gateway sends SIP traffic to the Asterisk server, from which point we are 100% Asterisk. We are also a 100% open implementation, from hardware specs to software configurations. Right now I'm a little bogged down with development and I have a deadline looming over my head, but in general if you want to know something just ask and I'll do my best to provide a quick response. ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Hello Matt, very interesting setup! are you using asteriak queues for inbound or not at all? Bye l. In data Thu, 22 Sep 2005 06:25:44 +0200, Matt Florell [EMAIL PROTECTED] ha scritto: We wrote VICIDIAL(part of the GPL astGUIclient suite http://astguiclient.sf.net) for our call center operations. Yes it does use a central queue and does not use Asterisk queues or agents. The system is based on a MySQL database and meetme rooms for the agents that use a web-client app for lead information and call control. We mostly do outbound and the volume is split across several servers, and for inbound we do have forwarding to other servers if the defined capacity is exceeded a certain point. As for our distributed recording approach, it's easy with meetme rooms to record a call on one server from another server, you just drop a call into the meetme room that is a monitor exten on another server over IAX2, TDMoE or a crossover Zap T1 connection. As for phone calls at one time: for inbound we almost never exceed 50 agents on a single server with no more than 72 incoming lines live at once. Our average is actually much less than that. For outbound we usually have about 15-40 agents per server with upto 96 lines dialing out concurrently. At our main office location we've had over 100 agents on at one time across 6 Asterisk servers handling over 350 calls at once with a total of more than 550 live channels on our Asterisk servers(includes recording, client and trunk channels). -- Assum est, versa et manduca. ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On Sep 21, 2005, at 1:46 AM, Zoa wrote: True... But i tried several brands of cards, and several drivers, the dual nic gigabit intel card was a lot better than all the other combinations i tried. Fun fact: short of buying a 10gigE card, your best performance will probably be from a cheap desktop board with Intel's on-board GigE chip connected via the CSA bus. It bypasses the PCI bus entirely and drops packets straight onto the north bridge; this gives it a surprisingly large performance boost. Only Intel would make its cheapest board outperform its more expensive server-grade solutions. :-) In general, Intel's GigE cards are pretty good, and they're under $50. It's hard to justify buying anything cheaper when you care about performance and reliability. Scott ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Hi Matt,Is your solution 100% Asterisk or are you using other "helpers" such as SER or XXXproxy or whatever?Thanks,WaldoOn Sep 21, 2005, at 12:45 PM, Matt Roth wrote:All, This message has generated a lot of responses, so I'm going to address each of them here in an attempt to consolidate the thread. ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
We are 100% Asterisk on the VOIP side. We use SIP, IAX and Zap(channelbanks) for phones and Zap T1s for telco termination. MATT---On 9/22/05, Waldo Rubinstein [EMAIL PROTECTED] wrote: Hi Matt,Is your solution 100% Asterisk or are you using other helpers such as SER or XXXproxy or whatever?Thanks,Waldo On Sep 21, 2005, at 12:45 PM, Matt Roth wrote: All, This message has generated a lot of responses, so I'm going to address each of them here in an attempt to consolidate the thread. ___--Bandwidth and Colocation sponsored by Easynews.com --Asterisk-Users mailing listAsterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-usersTo UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
- The reason i recommended you to use a ramdisk is because i think the - problem with recording to disk is saving 20ms of stream 1, then 20 ms of - stream 2, then 20ms of stream 3 etc etc meaning you write everytime - very small things. (with a lot of seeking). I was thinking about this idly in the car yesterday, and imagined that we could send all the Monitor audio out in a single multiplexed stream into a single file. Then we can either dumultiplex after, or reassemble the audio stream on the fly during playback. Generally more is recorded than listened to. Steve ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Hi everyone, This is just another attempt to address everybody in one place and consolidate the thread. Zoa, - I think its the best you can do. If something pops into your head, even if it's off the wall, don't hesitate to share it. Any idea is better than no idea if things go south. - Maybe there should be some option to be set for the monitor command to - buffer, with a warning that it will eat memory. - Its also not needed to buffer the complete call at once, just buffering - and writing to disk every 10 seconds would already be a big improvement, - and since the calls won't all start at the same time, the writing to - disk will be spread over time. This would be a nice option. res_monitor.c is using ast_writefile() to handle writing each call leg to file. Maybe a more specialized file writing function that includes buffering or threading is all that is needed. - Good luck with the implementation, let us know how it turns out! - Try to find out what happens if the nfs share is gone, or overloaded too. Thanks. The information provided by members of this list is invaluable, so I'll be sure to keep you informed of our progress. The plan is to have the application that is called via the MONITOR_EXEC hook verify the NFS mount before the copy. If it's not there it'll have to write locally until the problem is resolved. The NFS mount will have it's own dedicated network interface (an Intel Pro 1000 MT) communicating to the digital recording server over a Gigabit connection. That should be sufficient. - If there is no zaptel hardware involved, i don't think you have to be - too concerned about the network cards interrupts. Nope, we're terminating our Ts in a Cisco AS5400HPX Universal Gateway and using ztdummy as a timing source for music on hold. Our Asterisk server (a Dell PowerEdge 6850) has configurable IRQs, so I can fix any conflicts that may arise with other hardware relatively easily. Matt, - Have you tried recording directly to GSM format? It will help reduce the - bottleneck on disk IO although it will use more CPU cycles(in your case - on a RAM drive this may not help at all) We don't want to do any transcoding on the Asterisk server because of the drain on CPU. The RAM disk seems to be the best solution for our scenario. - (Regarding Avoided deadlock warnings and their effects) - There aren't always audio skips but they do happen more when you get more - ast_channel_walk warnings. The audio gaps are usually less than a quarter - second in our experience but can be upto a second depending on the - severity of the IO problem at that instance. It's very hard to test for - until you get into production and you have real conversations and real - people listening to them that can hear the audio skips. Thank you for this information. It seems like the warnings won't cause any unacceptable results, but we'll be listening to the audio regularly after we go into production. - We have sevaral call centers as well, and we just restrict a single server - to 50 recordings at once and then we would pass the next recording as an - IAX2 channel to another recording server. It's a scalable system for us - that is relatively cheap and works well since we can mix and gsm-encode - the audio on these multiple servers at night when not in production - leaving the NSF server just for storage and not audio processing. That is a smart solution. Thanks for sharing it. If we have any problems with our setup we may start looking in this direction. - (Regarding queue management, call volume, and system architecture of - his Asterisk setup) - We wrote VICIDIAL(part of the GPL astGUIclient suite - http://astguiclient.sf.net) for our call center operations. Yes it does - use a central queue and does not use Asterisk queues or agents. The - system is based on a MySQL database and meetme rooms for the agents that - use a web-client app for lead information and call control. We'll have to take a look at your work. We have to manage multiple queues and plan on using Asterisk queues and agents on our single Asterisk server. - We mostly do outbound and the volume is split across several servers, and - for inbound we do have forwarding to other servers if the defined - capacity is exceeded a certain point. How are you calculating this capacity? - As for our distributed recording approach, it's easy with meetme rooms to - record a call on one server from another server, you just drop a call into - the meetme room that is a monitor exten on another server over IAX2, TDMoE - or a crossover Zap T1 connection. Great idea. Once again, if we have problems with our implementation it's great to have this alternative path to explore. - As for phone calls at one time: for inbound we almost never exceed 50 - agents on a single server with no more than 72 incoming lines live at
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On 9/22/05, Matt Roth [EMAIL PROTECTED] wrote: Matt,- Have you tried recording directly to GSM format? It will help reduce the - bottleneck on disk IO although it will use more CPU cycles(in your case- on a RAM drive this may not help at all)We don't want to do any transcoding on the Asterisk server because ofthe drain on CPU. The RAM disk seems to be the best solution for our scenario. That's pretty much what I though, wanted to mention it though. - (Regarding Avoided deadlock warnings and their effects)- There aren't always audio skips but they do happen more when you get more - ast_channel_walk warnings. The audio gaps are usually less than a quarter- second in our experience but can be upto a second depending on the- severity of the IO problem at that instance. It's very hard to test for - until you get into production and you have real conversations and real- people listening to them that can hear the audio skips.Thank you for this information. It seems like the warnings won't causeany unacceptable results, but we'll be listening to the audio regularly after we go into production. When we traced the recording skips back to an exact time there is always a ast_channel_walk warning in the logs, although there is not always an audio skip if the warning appears, but the more of those warnings, the more skips we got. - We mostly do outbound and the volume is split across several servers, and- for inbound we do have forwarding to other servers if the defined - capacity is exceeded a certain point.How are you calculating this capacity? Trial and error, astGUIclient/VICIDIAL has a very detailed system performance logging utility built into it that tracks open asterisk channels, ram, CPU use %, system load and a few other things and we were best able to find a good and safe capacity at which to limit our systems to ensure reliability: http://astguiclient.sourceforge.net/VDreports/performance_new.gif - As for phone calls at one time: for inbound we almost never exceed 50- agents on a single server with no more than 72 incoming lines live at once.- Our average is actually much less than that. For outbound we usually have- about 15-40 agents per server with upto 96 lines dialing outconcurrently.- At our main office location we've had over 100 agents on at one time across- 6 Asterisk servers handling over 350 calls at once with a total ofmore than- 550 live channels on our Asterisk servers(includes recording, client and- trunk channels).I hope you don't mind answering some questions about your setup. It seems like you have really solid numbers to work with. We'reimplementing our own reporting (both real-time and next-day) usingAsterisk's call detail records (stored in a MySQL database) andinformation captured from the management interface (via AstManProxy). Are we reinventing the wheel? Do you have any tips for us in regards toour reporting software? We don't use any Asterisk-generated logs, they didn't offer enough information for us, so we created several of our own MySQL logs for calls and Manager actions that we use to track every action that happens across all systems as well as all live calls that are occuring in real-time on all systems. This allows VICIDIAL to have real-time status screens and datetime-level statistical performance reports available any time you want to pull them. As for tips, just log everything, you may not think you need it now but you will eventually want just about every little piece of call and action information that you can get your hands on for reporting. I'm assuming you are using VICIDIAL as a predictive dialer inconjunction with Asterisk for your outbound calling. We are looking for an outbound dialing solution. Could you please provide a list of theabilities and limitations of VICIDIAL? Well, since we wrote VICIDIAL we are a little biased :) it is free and GPL so you won't get many of the pretty features you get with those expensive dialer solutions, but you can tinker with it all you like. We've been using it for about 2 years in production and now have it on over 200 seats across 4 locations for our company. There are also over 50 production VICIDIAL installations in over a dozen countries that we know about. It works better for us than any of the dialer systems we used to use and we have total control over it which makes it easier to change things on the fly. There are limitations and things that we have not gotten around to writing yet, but there is an active community around it and we are developing new features for it all the time and releasing a new version every 2-8 weeks. One limitation is Answering Machine detection. We haven't found a good method of doing this and we don't like the delay that all commercial systems have when doing AM detection so we just do a dial timeout by default, about 4-5 rings will eliminate 90% of the answering machines you receive without any delay in the calls that are answered by a human. For other features see the product page:
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Hey ho, I suppose you are the person from the digium forum :) The reason i recommended you to use a ramdisk is because i think the problem with recording to disk is saving 20ms of stream 1, then 20 ms of stream 2, then 20ms of stream 3 etc etc meaning you write everytime very small things. (with a lot of seeking). Our best test results were with: - buffering the recordings to a ramdisk, then - on low load (at night) copy the files over the network (easy to shape the pipe, so that you dont overload anything), This way, the memory buffer will take care of the 'fragmentation' and not your harddisk. - on the remove server, do all the mixing / indexing etc. (i really don't mixing or converting between audio formats on the same server as asterisk). If you want to go even freakier, run asterisk (or you complete distro) from a ramdisk. Oh, another thing, for the people trying this the performance of hdparm is not linear with the quality of your calls, tweaking your disks to be faster will not help for asterisk when you do a copy. (in general). I thought over your suggestion to use a sniffer to do the recordings, you might pull it off, but will have to write your own to do so. (or go to the expensive version of commercial sniffer applications). Zoa. - www.asteriskguru.com Matt Florell wrote: On 9/20/05, *Matt Roth* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Patrick, Thank you for your suggestions. Our initial runs were recording directly to an NFS mount and they experienced the same problems as recording to the local disk. In our final setup, the copy will be done to an NFS mount as long as it exists, falling back to local disk only when the NFS server is down. The theory that we're running on is that any I/O bottlenecks (or network latencies in the case of NFS) only matter when they are bound to a call in progress. In that scenario, the bottleneck would introduce a latency in Asterisk's handling of the RTP packets causing call degradation and drops. By decoupling I/O from live calls and performing the copies (a very lightweight operation) in a separate process, we hope to not affect Asterisk's real-time handling of the RTP packets. Because of limited access to the test equipment, we were only able to test up to storing the digital recordings on a RAM disk. Please shoot holes in this setup if you see any weaknesses. Better today than on our go-live date. Thanks, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer Hello, I'm very interested in the specifics of your setup. How much space is on the RAM disk? What kind of RAM drive is it? What format are you recording to? What codec are the SIP calls being placed over? We've run into the Avoided deadlock recording issues several times when trying to do more than 50 concurrent recordings. Changing the ast_channel_lock loop from 10 to 20 has helped somewhat reduce the warnings and reduce audio gaps on the recordings, but what is really needed for more robust recording is a configurable recording buffer that wouldn't freak out if a 10ms delay occurs. Good luck and please keep us updated on your progress, MATT--- ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users signature.asc Description: OpenPGP digital signature ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On Wed, 2005-09-21 at 10:07 +0300, Zoa wrote: The reason i recommended you to use a ramdisk is because i think the problem with recording to disk is saving 20ms of stream 1, then 20 ms of stream 2, then 20ms of stream 3 etc etc meaning you write everytime very small things. (with a lot of seeking). Our best test results were with: filesystems are also a consideration with larger scale projects. Different filesystems add different amounts of overheads on different types of operations. Some are faster at moving small files around others faster with large files. This adds to the disk latency. Removing the disk latency itself is a good thing, since that is typically slower, but to crank out that last little bit of performance some research into the different filesystems under the specific kernel that you are using could also be a consideration. The most obvious (and easiest to update a running system) is to remove things like atime, whih with most linux distros is on by default. This causes a write operation for the read of a file to update the last time accessed. A couple little things can add up to a few percent improvement and generally make the cost go down. - buffering the recordings to a ramdisk, then - on low load (at night) copy the files over the network (easy to shape the pipe, so that you dont overload anything), Or have a seperate network set up (dual nic card for example) where the 2nd network is used just for NFS traffic. Although NFS generally is ugly network wise, it is standard and makes things easier. Just gotta watch the IO on the system given that the network card itself will cause cpu cycles to be used, but lets face it cpu is cheap now. Different drivers also work differently, and then with the 2.6 series kernels you can use device polling instead of interupts which can help a little. If you want to go even freakier, run asterisk (or you complete distro) from a ramdisk. When you say ramdisk here I assume you mean using conventional ram, its cheap yes but its volatile, do you have any plans for failure of the system or ram? Or is the data integrity itself not as critical? The reason that people like hard drives is because most of the time if the system goes down for any reason the data is still intact. I thought over your suggestion to use a sniffer to do the recordings, you might pull it off, but will have to write your own to do so. (or go to the expensive version of commercial sniffer applications). isnt vomit free? It was a voip sniffer that worked with some codecs many years ago (I wanna say mid-late 90s but I may be thinking of another back then). http://vomit.xtdnet.nl/ does G.711 only. The bigger prIoblem that I see is that sniffers dont always get all the traffic that is on a network particularly when the network has more traffic on it. While this generally isnt a concern and I would like to think that even a poorly configured network could allow for 512 calls, it is a factor to implement this type of a solution. -- Trixter http://www.0xdecafbad.com Bret McDanel UK +44 870 340 4605 Germany +49 801 777 555 3402 US +1 360 207 0479 or +1 516 687 5200 FreeWorldDialup: 635378 signature.asc Description: This is a digitally signed message part ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Also when you do things over the network, disable your onboard network card, and go for some more expensive network card. In our tests with small packets, we could increase the throughput with a factor 2. (related to cpu load). Zoa. -- www.asteriskguru.com trixter http://www.0xdecafbad.com wrote: On Wed, 2005-09-21 at 10:07 +0300, Zoa wrote: The reason i recommended you to use a ramdisk is because i think the problem with recording to disk is saving 20ms of stream 1, then 20 ms of stream 2, then 20ms of stream 3 etc etc meaning you write everytime very small things. (with a lot of seeking). Our best test results were with: filesystems are also a consideration with larger scale projects. Different filesystems add different amounts of overheads on different types of operations. Some are faster at moving small files around others faster with large files. This adds to the disk latency. Removing the disk latency itself is a good thing, since that is typically slower, but to crank out that last little bit of performance some research into the different filesystems under the specific kernel that you are using could also be a consideration. The most obvious (and easiest to update a running system) is to remove things like atime, whih with most linux distros is on by default. This causes a write operation for the read of a file to update the last time accessed. A couple little things can add up to a few percent improvement and generally make the cost go down. - buffering the recordings to a ramdisk, then - on low load (at night) copy the files over the network (easy to shape the pipe, so that you dont overload anything), Or have a seperate network set up (dual nic card for example) where the 2nd network is used just for NFS traffic. Although NFS generally is ugly network wise, it is standard and makes things easier. Just gotta watch the IO on the system given that the network card itself will cause cpu cycles to be used, but lets face it cpu is cheap now. Different drivers also work differently, and then with the 2.6 series kernels you can use device polling instead of interupts which can help a little. If you want to go even freakier, run asterisk (or you complete distro) from a ramdisk. When you say ramdisk here I assume you mean using conventional ram, its cheap yes but its volatile, do you have any plans for failure of the system or ram? Or is the data integrity itself not as critical? The reason that people like hard drives is because most of the time if the system goes down for any reason the data is still intact. I thought over your suggestion to use a sniffer to do the recordings, you might pull it off, but will have to write your own to do so. (or go to the expensive version of commercial sniffer applications). isnt vomit free? It was a voip sniffer that worked with some codecs many years ago (I wanna say mid-late 90s but I may be thinking of another back then). http://vomit.xtdnet.nl/ does G.711 only. The bigger prIoblem that I see is that sniffers dont always get all the traffic that is on a network particularly when the network has more traffic on it. While this generally isnt a concern and I would like to think that even a poorly configured network could allow for 512 calls, it is a factor to implement this type of a solution. ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users signature.asc Description: OpenPGP digital signature ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On Wed, 2005-09-21 at 11:11 +0300, Zoa wrote: Also when you do things over the network, disable your onboard network card, and go for some more expensive network card. In our tests with small packets, we could increase the throughput with a factor 2. (related to cpu load). I wonder how much of that is a poorly written driver and not the card itself. I have seen some fairly poor drivers performance wise. :/ -- Trixter http://www.0xdecafbad.com Bret McDanel UK +44 870 340 4605 Germany +49 801 777 555 3402 US +1 360 207 0479 or +1 516 687 5200 FreeWorldDialup: 635378 signature.asc Description: This is a digitally signed message part ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
True... But i tried several brands of cards, and several drivers, the dual nic gigabit intel card was a lot better than all the other combinations i tried. zoa trixter http://www.0xdecafbad.com wrote: On Wed, 2005-09-21 at 11:11 +0300, Zoa wrote: Also when you do things over the network, disable your onboard network card, and go for some more expensive network card. In our tests with small packets, we could increase the throughput with a factor 2. (related to cpu load). I wonder how much of that is a poorly written driver and not the card itself. I have seen some fairly poor drivers performance wise. :/ ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users signature.asc Description: OpenPGP digital signature ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
All, This message has generated a lot of responses, so I'm going to address each of them here in an attempt to consolidate the thread. Matt, - I'm very interested in the specifics of your setup. - How much space is on the RAM disk? Currently it is 10 GB. We are upgrading it to 16 GB. - What kind of RAM drive is it? The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs. The details for each 1 GB DIMM can be seen here: http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm The upgrade will involve adding 2 GB DIMMs to the system, but I don't have the details on these yet. The RAM disk is setup by adding the following kernel command line option to grub.conf: ramdisk_size=10485760 We are running Fedora Core 3 with the most up to date 2.6 SMP kernel. By default the RAM disk's block size is 1024 bytes, so we are formatting it as an ext2 file system with a block size of 1024 bytes using the following command: mke2fs -b 1024 -m 0 /dev/ram0 The block size can easily be changed from the kernel's view (using the kernel command line option ramdisk_blocksize=) or from mke2fs's view (using the -b argument), so please let me know if I can make an easy optimization here. Finally, the RAM disk is mounted using the command: mount /dev/ram0 /digrec A good RAMDISK howto exists at: http://www.vanemery.com/Linux/Ramdisk/ramdisk.html - What format are you recording to? - What codec are the SIP calls being placed over? We are recording to the PCM format and using the G711 uLaw codec. High voice quality is essential to our application (we are a call center) so we partnered with MCI to configure our network for the required bandwidth and chose the highest quality, zero compression codec. We noload all other codecs in order to avoid transcoding on the switch, so we must record to PCM. Later (on a separate server) the recordings are mixed to GSM which provides a 5 to 1 compression ratio with very little artifacts. - We've run into the Avoided deadlock recording issues several times when trying to do - more than 50 concurrent recordings. Changing the ast_channel_lock loop from 10 to 20 has - helped somewhat reduce the warnings and reduce audio gaps on the recordings, but what is - really needed for more robust recording is a configurable recording buffer that wouldn't - freak out if a 10ms delay occurs. Are you saying that these messages indicate a gap in a digital recording? If so, what is the duration of the gap? If it's comparable to a CD skip, I think we can deal with it until a buffer or another solution is implemented. - Good luck and please keep us updated on your progress Thank you. I'll be keeping the list informed of our progress. Zoa, - I suppose you are the person from the digium forum That was actually my boss's boss. We thank you all the way up and down the line for your suggestion. - The reason i recommended you to use a ramdisk is because i think the - problem with recording to disk is saving 20ms of stream 1, then 20 ms of - stream 2, then 20ms of stream 3 etc etc meaning you write everytime - very small things. (with a lot of seeking). Agreed. This is why we hope that decoupling the copy (memory to disk) from Asterisk itself and, more importantly, Asterisk's real-time handling of the call being recorded will be sufficient. For the record, when recording 512 simultaneous calls to the local disk we saw a peek of about 13,000 blocks written per second. - Our best test results were with: - - - buffering the recordings to a ramdisk, then We're doing that, as per your suggestion. - - on low load (at night) copy the files over the network (easy to shape - the pipe, so that you dont overload anything), This way, the memory - buffer will take care of the 'fragmentation' and not your harddisk. If you'll note the format of the recordings and that we'll be recording up to 200,000 minutes of calls a day, with a little quick math you'll realize that it would take 80 to 100 GBs of memory for us to buffer a full day's recordings. Combined with the fact that a server failure late in the day would cause us to lose them all, this is not a desirable solution. Instead, we plan to write an application to call from the MONITOR_EXEC hook that will be executed at the end of each call. This application will be niced down to the lowest priority, and simply copy the leg files from memory to disk. Under normal conditions (ie. our NFS server is up) this will actually be a copy to a remote disk using an asynchronous NFS transfer. All actual disk I/O will occur on our digital recording server and any handling of the digital recordings will occur only after the call they are bound to is completed. Do you have any suggestions regarding this
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
In light of the I/O bottleneck problem I'd have to ask why asterisk can't just buffer incoming audio and then flush a complete audio file to disk.. I'm assuming that recordings vary in length.. the problem with this idea is what happens if 50 recordings all complete at the same time.. a dump like that might not be very pretty (a fast drive plus a little scheduler or limiter so that only x number of files get written to disk at a time would probably help out there) but I can imagine that a single file being written is much more efficient and more disk-friendly.. perhaps I don't know what the heck I'm talking about but at least in my mind the theory sounds better than the current 'stream-to-file' method employed by asterisk. Matt Roth wrote: All, This message has generated a lot of responses, so I'm going to address each of them here in an attempt to consolidate the thread. Matt, - I'm very interested in the specifics of your setup. - How much space is on the RAM disk? Currently it is 10 GB. We are upgrading it to 16 GB. - What kind of RAM drive is it? The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs. The details for each 1 GB DIMM can be seen here: http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm The upgrade will involve adding 2 GB DIMMs to the system, but I don't have the details on these yet. The RAM disk is setup by adding the following kernel command line option to grub.conf: ramdisk_size=10485760 We are running Fedora Core 3 with the most up to date 2.6 SMP kernel. By default the RAM disk's block size is 1024 bytes, so we are formatting it as an ext2 file system with a block size of 1024 bytes using the following command: mke2fs -b 1024 -m 0 /dev/ram0 The block size can easily be changed from the kernel's view (using the kernel command line option ramdisk_blocksize=) or from mke2fs's view (using the -b argument), so please let me know if I can make an easy optimization here. Finally, the RAM disk is mounted using the command: mount /dev/ram0 /digrec A good RAMDISK howto exists at: http://www.vanemery.com/Linux/Ramdisk/ramdisk.html - What format are you recording to? - What codec are the SIP calls being placed over? We are recording to the PCM format and using the G711 uLaw codec. High voice quality is essential to our application (we are a call center) so we partnered with MCI to configure our network for the required bandwidth and chose the highest quality, zero compression codec. We noload all other codecs in order to avoid transcoding on the switch, so we must record to PCM. Later (on a separate server) the recordings are mixed to GSM which provides a 5 to 1 compression ratio with very little artifacts. - We've run into the Avoided deadlock recording issues several times when trying to do - more than 50 concurrent recordings. Changing the ast_channel_lock loop from 10 to 20 has - helped somewhat reduce the warnings and reduce audio gaps on the recordings, but what is - really needed for more robust recording is a configurable recording buffer that wouldn't - freak out if a 10ms delay occurs. Are you saying that these messages indicate a gap in a digital recording? If so, what is the duration of the gap? If it's comparable to a CD skip, I think we can deal with it until a buffer or another solution is implemented. - Good luck and please keep us updated on your progress Thank you. I'll be keeping the list informed of our progress. Zoa, - I suppose you are the person from the digium forum That was actually my boss's boss. We thank you all the way up and down the line for your suggestion. - The reason i recommended you to use a ramdisk is because i think the - problem with recording to disk is saving 20ms of stream 1, then 20 ms of - stream 2, then 20ms of stream 3 etc etc meaning you write everytime - very small things. (with a lot of seeking). Agreed. This is why we hope that decoupling the copy (memory to disk) from Asterisk itself and, more importantly, Asterisk's real-time handling of the call being recorded will be sufficient. For the record, when recording 512 simultaneous calls to the local disk we saw a peek of about 13,000 blocks written per second. - Our best test results were with: - - - buffering the recordings to a ramdisk, then We're doing that, as per your suggestion. - - on low load (at night) copy the files over the network (easy to shape - the pipe, so that you dont overload anything), This way, the memory - buffer will take care of the 'fragmentation' and not your harddisk. If you'll note the format of the recordings and that we'll be recording up to 200,000 minutes of calls a day, with a little quick math you'll realize that it would take 80 to 100 GBs of
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
I would think memory would be the limiting factor. A 3-4 minute wav file is what, 30Meg or so? And there is one for each end of the call, so that's 60Meg. Now let's say it's a 15 minute call and then are 10 of them at once. That's 30Meg x 5 (5 times the length of my estimate) x 2 (each leg) x 10 simultaneous callsequals 3 Gig of RAM. Once you run out of RAM, then what does it do? It would have to try and dump it all to disk at once and you are back where you started. I think the average * implementation doesn't have nearly enough free RAM to do this. The RAMdisk solution seems to be pretty elegant in it's simplicity. Matt Hess wrote: In light of the I/O bottleneck problem I'd have to ask why asterisk can't just buffer incoming audio and then flush a complete audio file to disk.. I'm assuming that recordings vary in length.. the problem with this idea is what happens if 50 recordings all complete at the same time.. a dump like that might not be very pretty (a fast drive plus a little scheduler or limiter so that only x number of files get written to disk at a time would probably help out there) but I can imagine that a single file being written is much more efficient and more disk-friendly.. perhaps I don't know what the heck I'm talking about but at least in my mind the theory sounds better than the current 'stream-to-file' method employed by asterisk. Matt Roth wrote: All, This message has generated a lot of responses, so I'm going to address each of them here in an attempt to consolidate the thread. Matt, - I'm very interested in the specifics of your setup. - How much space is on the RAM disk? Currently it is 10 GB. We are upgrading it to 16 GB. - What kind of RAM drive is it? The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs. The details for each 1 GB DIMM can be seen here: http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm The upgrade will involve adding 2 GB DIMMs to the system, but I don't have the details on these yet. The RAM disk is setup by adding the following kernel command line option to grub.conf: ramdisk_size=10485760 We are running Fedora Core 3 with the most up to date 2.6 SMP kernel. By default the RAM disk's block size is 1024 bytes, so we are formatting it as an ext2 file system with a block size of 1024 bytes using the following command: mke2fs -b 1024 -m 0 /dev/ram0 The block size can easily be changed from the kernel's view (using the kernel command line option ramdisk_blocksize=) or from mke2fs's view (using the -b argument), so please let me know if I can make an easy optimization here. Finally, the RAM disk is mounted using the command: mount /dev/ram0 /digrec A good RAMDISK howto exists at: http://www.vanemery.com/Linux/Ramdisk/ramdisk.html - What format are you recording to? - What codec are the SIP calls being placed over? We are recording to the PCM format and using the G711 uLaw codec. High voice quality is essential to our application (we are a call center) so we partnered with MCI to configure our network for the required bandwidth and chose the highest quality, zero compression codec. We noload all other codecs in order to avoid transcoding on the switch, so we must record to PCM. Later (on a separate server) the recordings are mixed to GSM which provides a 5 to 1 compression ratio with very little artifacts. - We've run into the Avoided deadlock recording issues several times when trying to do - more than 50 concurrent recordings. Changing the ast_channel_lock loop from 10 to 20 has - helped somewhat reduce the warnings and reduce audio gaps on the recordings, but what is - really needed for more robust recording is a configurable recording buffer that wouldn't - freak out if a 10ms delay occurs. Are you saying that these messages indicate a gap in a digital recording? If so, what is the duration of the gap? If it's comparable to a CD skip, I think we can deal with it until a buffer or another solution is implemented. - Good luck and please keep us updated on your progress Thank you. I'll be keeping the list informed of our progress. Zoa, - I suppose you are the person from the digium forum That was actually my boss's boss. We thank you all the way up and down the line for your suggestion. - The reason i recommended you to use a ramdisk is because i think the - problem with recording to disk is saving 20ms of stream 1, then 20 ms of - stream 2, then 20ms of stream 3 etc etc meaning you write everytime - very small things. (with a lot of seeking). Agreed. This is why we hope that decoupling the copy (memory to disk) from Asterisk itself and, more importantly, Asterisk's real-time handling of the call being recorded will be sufficient. For
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
The problem is that then it won't work on systems with little memory. 50 streams would eat memory like crazy. Zoa Matt Hess wrote: In light of the I/O bottleneck problem I'd have to ask why asterisk can't just buffer incoming audio and then flush a complete audio file to disk.. I'm assuming that recordings vary in length.. the problem with this idea is what happens if 50 recordings all complete at the same time.. a dump like that might not be very pretty (a fast drive plus a little scheduler or limiter so that only x number of files get written to disk at a time would probably help out there) but I can imagine that a single file being written is much more efficient and more disk-friendly.. perhaps I don't know what the heck I'm talking about but at least in my mind the theory sounds better than the current 'stream-to-file' method employed by asterisk. Matt Roth wrote: All, This message has generated a lot of responses, so I'm going to address each of them here in an attempt to consolidate the thread. Matt, - I'm very interested in the specifics of your setup. - How much space is on the RAM disk? Currently it is 10 GB. We are upgrading it to 16 GB. - What kind of RAM drive is it? The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs. The details for each 1 GB DIMM can be seen here: http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm The upgrade will involve adding 2 GB DIMMs to the system, but I don't have the details on these yet. The RAM disk is setup by adding the following kernel command line option to grub.conf: ramdisk_size=10485760 We are running Fedora Core 3 with the most up to date 2.6 SMP kernel. By default the RAM disk's block size is 1024 bytes, so we are formatting it as an ext2 file system with a block size of 1024 bytes using the following command: mke2fs -b 1024 -m 0 /dev/ram0 The block size can easily be changed from the kernel's view (using the kernel command line option ramdisk_blocksize=) or from mke2fs's view (using the -b argument), so please let me know if I can make an easy optimization here. Finally, the RAM disk is mounted using the command: mount /dev/ram0 /digrec A good RAMDISK howto exists at: http://www.vanemery.com/Linux/Ramdisk/ramdisk.html - What format are you recording to? - What codec are the SIP calls being placed over? We are recording to the PCM format and using the G711 uLaw codec. High voice quality is essential to our application (we are a call center) so we partnered with MCI to configure our network for the required bandwidth and chose the highest quality, zero compression codec. We noload all other codecs in order to avoid transcoding on the switch, so we must record to PCM. Later (on a separate server) the recordings are mixed to GSM which provides a 5 to 1 compression ratio with very little artifacts. - We've run into the Avoided deadlock recording issues several times when trying to do - more than 50 concurrent recordings. Changing the ast_channel_lock loop from 10 to 20 has - helped somewhat reduce the warnings and reduce audio gaps on the recordings, but what is - really needed for more robust recording is a configurable recording buffer that wouldn't - freak out if a 10ms delay occurs. Are you saying that these messages indicate a gap in a digital recording? If so, what is the duration of the gap? If it's comparable to a CD skip, I think we can deal with it until a buffer or another solution is implemented. - Good luck and please keep us updated on your progress Thank you. I'll be keeping the list informed of our progress. Zoa, - I suppose you are the person from the digium forum That was actually my boss's boss. We thank you all the way up and down the line for your suggestion. - The reason i recommended you to use a ramdisk is because i think the - problem with recording to disk is saving 20ms of stream 1, then 20 ms of - stream 2, then 20ms of stream 3 etc etc meaning you write everytime - very small things. (with a lot of seeking). Agreed. This is why we hope that decoupling the copy (memory to disk) from Asterisk itself and, more importantly, Asterisk's real-time handling of the call being recorded will be sufficient. For the record, when recording 512 simultaneous calls to the local disk we saw a peek of about 13,000 blocks written per second. - Our best test results were with: - - - buffering the recordings to a ramdisk, then We're doing that, as per your suggestion. - - on low load (at night) copy the files over the network (easy to shape - the pipe, so that you dont overload anything), This way, the memory - buffer will take care of the 'fragmentation' and not your harddisk. If you'll note the format of the recordings and that we'll be recording up to 200,000 minutes of calls a
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
It's true that the average Asterisk implementation doesn't have enough RAM, but we are replacing a legacy NorTel switch in a call center. If you look at the cost of traditional PBXs, the cost of additional memory starts to look a little better. = ) Now for some quick math: 1 minute of PCM audio = 480 KB * 2 leg files ~= 1 MB/minute Avail. Mem. = 10 GB = 10,000 MB = 10,000 minutes of digital recordings Peak calling = 200,000 minutes/day 10,000 / 200,000 = 5% So our buffer is 5% of our total calling for the day. We'll be bumping the RAM disk up to 16 GB yielding an 8% buffer. We'll be moving calls out of memory to a remote disk (via NFS) as soon as they are finished, so I think we'll be okay. We'll be monitoring memory usage and sharing our data. It's my hope that this is the solution for large scale (250-500 simultaneous calls) Asterisk installations. I hope my math was right, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer Zoa wrote: - The problem is that then it won't work on systems with little memory. 50 - streams would eat memory like crazy. - - Zoa [EMAIL PROTECTED] wrote: I would think memory would be the limiting factor. A 3-4 minute wav file is what, 30Meg or so? And there is one for each end of the call, so that's 60Meg. Now let's say it's a 15 minute call and then are 10 of them at once. That's 30Meg x 5 (5 times the length of my estimate) x 2 (each leg) x 10 simultaneous callsequals 3 Gig of RAM. Once you run out of RAM, then what does it do? It would have to try and dump it all to disk at once and you are back where you started. I think the average * implementation doesn't have nearly enough free RAM to do this. The RAMdisk solution seems to be pretty elegant in it's simplicity. Matt Hess wrote: In light of the I/O bottleneck problem I'd have to ask why asterisk can't just buffer incoming audio and then flush a complete audio file to disk.. I'm assuming that recordings vary in length.. the problem with this idea is what happens if 50 recordings all complete at the same time.. a dump like that might not be very pretty (a fast drive plus a little scheduler or limiter so that only x number of files get written to disk at a time would probably help out there) but I can imagine that a single file being written is much more efficient and more disk-friendly.. perhaps I don't know what the heck I'm talking about but at least in my mind the theory sounds better than the current 'stream-to-file' method employed by asterisk. Matt Roth wrote: All, This message has generated a lot of responses, so I'm going to address each of them here in an attempt to consolidate the thread. Matt, - I'm very interested in the specifics of your setup. - How much space is on the RAM disk? Currently it is 10 GB. We are upgrading it to 16 GB. - What kind of RAM drive is it? The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs. The details for each 1 GB DIMM can be seen here: http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm The upgrade will involve adding 2 GB DIMMs to the system, but I don't have the details on these yet. The RAM disk is setup by adding the following kernel command line option to grub.conf: ramdisk_size=10485760 We are running Fedora Core 3 with the most up to date 2.6 SMP kernel. By default the RAM disk's block size is 1024 bytes, so we are formatting it as an ext2 file system with a block size of 1024 bytes using the following command: mke2fs -b 1024 -m 0 /dev/ram0 The block size can easily be changed from the kernel's view (using the kernel command line option ramdisk_blocksize=) or from mke2fs's view (using the -b argument), so please let me know if I can make an easy optimization here. Finally, the RAM disk is mounted using the command: mount /dev/ram0 /digrec A good RAMDISK howto exists at: http://www.vanemery.com/Linux/Ramdisk/ramdisk.html - What format are you recording to? - What codec are the SIP calls being placed over? We are recording to the PCM format and using the G711 uLaw codec. High voice quality is essential to our application (we are a call center) so we partnered with MCI to configure our network for the required bandwidth and chose the highest quality, zero compression codec. We noload all other codecs in order to avoid transcoding on the switch, so we must record to PCM. Later (on a separate server) the recordings are mixed to GSM which provides a 5 to 1 compression ratio with very little artifacts. - We've run into the Avoided deadlock recording issues several times when trying to do - more than 50 concurrent recordings. Changing the ast_channel_lock loop from 10 to 20 has - helped somewhat reduce the warnings and reduce audio gaps on the recordings, but what is - really needed
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
I think its the best you can do. Maybe there should be some option to be set for the monitor command to buffer, with a warning that it will eat memory. Its also not needed to buffer the complete call at once, just buffering and writing to disk every 10 seconds would already be a big improvement, and since the calls won't all start at the same time, the writing to disk will be spread over time. Good luck with the implementation, let us know how it turns out! Try to find out what happens if the nfs share is gone, or overloaded too. If there is no zaptel hardware involved, i don't think you have to be too concerned about the network cards interrupts. greetings, Zoa. Matt Roth wrote: It's true that the average Asterisk implementation doesn't have enough RAM, but we are replacing a legacy NorTel switch in a call center. If you look at the cost of traditional PBXs, the cost of additional memory starts to look a little better. = ) Now for some quick math: 1 minute of PCM audio = 480 KB * 2 leg files ~= 1 MB/minute Avail. Mem. = 10 GB = 10,000 MB = 10,000 minutes of digital recordings Peak calling = 200,000 minutes/day 10,000 / 200,000 = 5% So our buffer is 5% of our total calling for the day. We'll be bumping the RAM disk up to 16 GB yielding an 8% buffer. We'll be moving calls out of memory to a remote disk (via NFS) as soon as they are finished, so I think we'll be okay. We'll be monitoring memory usage and sharing our data. It's my hope that this is the solution for large scale (250-500 simultaneous calls) Asterisk installations. I hope my math was right, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer Zoa wrote: - The problem is that then it won't work on systems with little memory. 50 - streams would eat memory like crazy. - - Zoa [EMAIL PROTECTED] wrote: I would think memory would be the limiting factor. A 3-4 minute wav file is what, 30Meg or so? And there is one for each end of the call, so that's 60Meg. Now let's say it's a 15 minute call and then are 10 of them at once. That's 30Meg x 5 (5 times the length of my estimate) x 2 (each leg) x 10 simultaneous callsequals 3 Gig of RAM. Once you run out of RAM, then what does it do? It would have to try and dump it all to disk at once and you are back where you started. I think the average * implementation doesn't have nearly enough free RAM to do this. The RAMdisk solution seems to be pretty elegant in it's simplicity. Matt Hess wrote: In light of the I/O bottleneck problem I'd have to ask why asterisk can't just buffer incoming audio and then flush a complete audio file to disk.. I'm assuming that recordings vary in length.. the problem with this idea is what happens if 50 recordings all complete at the same time.. a dump like that might not be very pretty (a fast drive plus a little scheduler or limiter so that only x number of files get written to disk at a time would probably help out there) but I can imagine that a single file being written is much more efficient and more disk-friendly.. perhaps I don't know what the heck I'm talking about but at least in my mind the theory sounds better than the current 'stream-to-file' method employed by asterisk. Matt Roth wrote: All, This message has generated a lot of responses, so I'm going to address each of them here in an attempt to consolidate the thread. Matt, - I'm very interested in the specifics of your setup. - How much space is on the RAM disk? Currently it is 10 GB. We are upgrading it to 16 GB. - What kind of RAM drive is it? The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked DIMMs. The details for each 1 GB DIMM can be seen here: http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm The upgrade will involve adding 2 GB DIMMs to the system, but I don't have the details on these yet. The RAM disk is setup by adding the following kernel command line option to grub.conf: ramdisk_size=10485760 We are running Fedora Core 3 with the most up to date 2.6 SMP kernel. By default the RAM disk's block size is 1024 bytes, so we are formatting it as an ext2 file system with a block size of 1024 bytes using the following command: mke2fs -b 1024 -m 0 /dev/ram0 The block size can easily be changed from the kernel's view (using the kernel command line option ramdisk_blocksize=) or from mke2fs's view (using the -b argument), so please let me know if I can make an easy optimization here. Finally, the RAM disk is mounted using the command: mount /dev/ram0 /digrec A good RAMDISK howto exists at: http://www.vanemery.com/Linux/Ramdisk/ramdisk.html - What format are you recording to? - What codec are the SIP calls being placed over? We are recording to the PCM format and using the G711 uLaw codec. High voice quality is essential to our application (we are a call center) so we
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On 9/21/05, Matt Roth [EMAIL PROTECTED] wrote: - What format are you recording to?- What codec are the SIP calls being placed over? We are recording to the PCM format and using the G711 uLaw codec.Highvoice quality is essential to our application (we are a call center) sowe partnered with MCI to configure our network for the required bandwidth and chose the highest quality, zero compression codec.Wenoload all other codecs in order to avoid transcoding on the switch, sowe must record to PCM. Later (on a separate server) the recordings are mixed to GSM which provides a 5 to 1 compression ratio with very littleartifacts. Have you tried recording directly to GSM format? It will help reduce the bottleneck on disk IO although it will use more CPU cycles(in your case on a RAM drive this may not help at all) - We've run into the Avoided deadlock recording issues several times when trying to do- more than 50 concurrent recordings. Changing the ast_channel_lock loopfrom 10 to 20 has- helped somewhat reduce the warnings and reduce audio gaps on therecordings, but what is- really needed for more robust recording is a configurable recording buffer that wouldn't- freak out if a 10ms delay occurs.Are you saying that these messages indicate a gap in a digitalrecording?If so, what is the duration of the gap? If it's comparableto a CD skip, I think we can deal with it until a buffer or another solution is implemented.There aren't always audio skips but they do happen more when you get more ast_channel_walk warnings. The audio gaps are usually less than a quarter second in our experience but can be upto a second depending on the severity of the IO problem at that instance. It's very hard to test for until you get into production and you have real conversations and real people listening to them that can hear the audio skips. We have sevaral call centers as well, and we just restrict a single server to 50 recordings at once and then we would pass the next recording as an IAX2 channel to another recording server. It's a scalable system for us that is relatively cheap and works well since we can mix and gsm-encode the audio on these multiple servers at night when not in production leaving the NSF server just for storage and not audio processing. MATT--- ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On 9/21/05, Matt Florell [EMAIL PROTECTED] wrote: We have sevaral call centers as well, and we just restrict a single server to 50 recordings at once and then we would pass the next recording as an IAX2 channel to another recording server. It's a scalable system for us that is relatively cheap and works well since we can mix and gsm-encode the audio on these multiple servers at night when not in production leaving the NSF server just for storage and not audio processing. Interesting approach, How do you deal with queue management ? Do you have central server that manages all queues ? How many phone calls u have at same time ? Inbound or outbound ? Could you share a little bit more on the architecture of your system ? regards m. ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On 9/21/05, izo [EMAIL PROTECTED] wrote: On 9/21/05, Matt Florell [EMAIL PROTECTED] wrote:We have sevaral call centers as well, and we just restrict a single server to 50 recordings at once and then we would pass the next recording as an IAX2 channel to another recording server. It's a scalable system for us that is relatively cheap and works well since we can mix and gsm-encode the audio on these multiple servers at night when not in production leaving the NSF server just for storage and not audio processing.Interesting approach,How do you deal with queue management ?Do you have central server that manages all queues ?How many phone calls u have at same time ? Inbound or outbound ? Could you share a little bit more on the architecture of your system ?regardsm We wrote VICIDIAL(part of the GPL astGUIclient suite http://astguiclient.sf.net) for our call center operations. Yes it does use a central queue and does not use Asterisk queues or agents. The system is based on a MySQL database and meetme rooms for the agents that use a web-client app for lead information and call control. We mostly do outbound and the volume is split across several servers, and for inbound we do have forwarding to other servers if the defined capacity is exceeded a certain point. As for our distributed recording approach, it's easy with meetme rooms to record a call on one server from another server, you just drop a call into the meetme room that is a monitor exten on another server over IAX2, TDMoE or a crossover Zap T1 connection. As for phone calls at one time: for inbound we almost never exceed 50 agents on a single server with no more than 72 incoming lines live at once. Our average is actually much less than that. For outbound we usually have about 15-40 agents per server with upto 96 lines dialing out concurrently. At our main office location we've had over 100 agents on at one time across 6 Asterisk servers handling over 350 calls at once with a total of more than 550 live channels on our Asterisk servers(includes recording, client and trunk channels). We mostly use cheaper single CPU systems instead of more expensive multi-CPU systems which is more cost effective in a larger setup and allows for greater flexibility, redundancy and scalability. I'll also be presenting a case-study on our whole setup at Astricon next month. The presentation file will be available on the astGUIclient project website after I present it for those who are interested. MATT--- ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
[Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
List users, Over the last few days we have been working with MCI's development lab to test our Asterisk setup. We were using a piece of hardware called an Abacus 5000 that is capable of creating and terminating thousands of SIP calls. Initially, we could not get past 64 simultaneous digitally recorded calls without having call quality issues including dropped calls. We identified an I/O bottleneck and rectified it by digitally recording to a RAM disk. Using this method, we were able to digitally record 512 simultaneous SIP-to-SIP calls with 100% call completion. Our plan is to use the MONITOR_EXEC hook to call a custom program that will copy files to the hard disk at call completion. This should be no problem when calling Monitor directly from the dialplan, but I need to know if there are any complications when digitally recording from app_queue or chan_agent. If anyone has experience with using MONITOR_EXEC and MONITOR_EXEC_ARGS with these applications, your input would be greatly appreciated. Digital recording seems to be the limiting factor when scaling an Asterisk system and we will share any advances we make with the community. We also noticed some Avoided initial deadlock and Avoided deadlock warnings in the messages file that Asterisk generates. Our 128 and 256 simultaneous call tests each produced one of these warnings. Our 512 simultaneous call test produced 18 of them all within a 10 second span, each on a separate SIP channel. Can anyone shed light on what these warnings mean, how they can be avoided, and if they are something to worry about. They did not seem to affect the outcome of the testing in any way. Details of our testing (hardware configuration, OS tweaks, system resource usage, reports from the Abacus 5000, etc.) will be made available over time. Please bear with me, as I am working on a deadline. Thank you, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On Tue, 2005-09-20 at 18:37 -0400, Matt Roth wrote: List users, Over the last few days we have been working with MCI's development lab to test our Asterisk setup. We were using a piece of hardware called an Abacus 5000 that is capable of creating and terminating thousands of SIP calls. Initially, we could not get past 64 simultaneous digitally recorded calls without having call quality issues including dropped calls. We identified an I/O bottleneck and rectified it by digitally recording to a RAM disk. Using this method, we were able to digitally record 512 simultaneous SIP-to-SIP calls with 100% call completion. Our plan is to use the MONITOR_EXEC hook to call a custom program that will copy files to the hard disk at call completion. Matt, Interesting stuff. Did you test copying of the recordings on the ramdisk to a local harddisk also during that 512 call load? Just wondering if that copy action wouldn't also create an I/O bottleneck and cause call quality issues under load. Did you consider using remote storage e.g. via nfs, a fibre channel or iSCSI link to a SAN? Regards, Patrick ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Patrick wrote: On Tue, 2005-09-20 at 18:37 -0400, Matt Roth wrote: List users, Over the last few days we have been working with MCI's development lab to test our Asterisk setup. We were using a piece of hardware called an Abacus 5000 that is capable of creating and terminating thousands of SIP calls. Initially, we could not get past 64 simultaneous digitally recorded calls without having call quality issues including dropped calls. We identified an I/O bottleneck and rectified it by digitally recording to a RAM disk. Using this method, we were able to digitally record 512 simultaneous SIP-to-SIP calls with 100% call completion. Our plan is to use the MONITOR_EXEC hook to call a custom program that will copy files to the hard disk at call completion. Matt, Interesting stuff. Did you test copying of the recordings on the ramdisk to a local harddisk also during that 512 call load? Just wondering if that copy action wouldn't also create an I/O bottleneck and cause call quality issues under load. Did you consider using remote storage e.g. via nfs, a fibre channel or iSCSI link to a SAN? I use 3Ware RAID cards in my systems with write cache turned on. I'm wondering if this presents a reduced interrupt load on the system than directly-attached hard drives? ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
Patrick, Thank you for your suggestions. Our initial runs were recording directly to an NFS mount and they experienced the same problems as recording to the local disk. In our final setup, the copy will be done to an NFS mount as long as it exists, falling back to local disk only when the NFS server is down. The theory that we're running on is that any I/O bottlenecks (or network latencies in the case of NFS) only matter when they are bound to a call in progress. In that scenario, the bottleneck would introduce a latency in Asterisk's handling of the RTP packets causing call degradation and drops. By decoupling I/O from live calls and performing the copies (a very lightweight operation) in a separate process, we hope to not affect Asterisk's real-time handling of the RTP packets. Because of limited access to the test equipment, we were only able to test up to storing the digital recordings on a RAM disk. Please shoot holes in this setup if you see any weaknesses. Better today than on our go-live date. Thanks, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer Patrick wrote: On Tue, 2005-09-20 at 18:37 -0400, Matt Roth wrote: List users, Over the last few days we have been working with MCI's development lab to test our Asterisk setup. We were using a piece of hardware called an Abacus 5000 that is capable of creating and terminating thousands of SIP calls. Initially, we could not get past 64 simultaneous digitally recorded calls without having call quality issues including dropped calls. We identified an I/O bottleneck and rectified it by digitally recording to a RAM disk. Using this method, we were able to digitally record 512 simultaneous SIP-to-SIP calls with 100% call completion. Our plan is to use the MONITOR_EXEC hook to call a custom program that will copy files to the hard disk at call completion. Matt, Interesting stuff. Did you test copying of the recordings on the ramdisk to a local harddisk also during that 512 call load? Just wondering if that copy action wouldn't also create an I/O bottleneck and cause call quality issues under load. Did you consider using remote storage e.g. via nfs, a fibre channel or iSCSI link to a SAN? Regards, Patrick ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
Re: [Asterisk-Users] SUCCESS - 512 Simultaneous Calls with Digital Recording
On 9/20/05, Matt Roth [EMAIL PROTECTED] wrote: Patrick, Thank you for your suggestions. Our initial runs were recording directly to an NFS mount and they experienced the same problems as recording to the local disk. In our final setup, the copy will be done to an NFS mount as long as it exists, falling back to local disk only when the NFS server is down. The theory that we're running on is that any I/O bottlenecks (or network latencies in the case of NFS) only matter when they are bound to a call in progress. In that scenario, the bottleneck would introduce a latency in Asterisk's handling of the RTP packets causing call degradation and drops. By decoupling I/O from live calls and performing the copies (a very lightweight operation) in a separate process, we hope to not affect Asterisk's real-time handling of the RTP packets. Because of limited access to the test equipment, we were only able to test up to storing the digital recordings on a RAM disk. Please shoot holes in this setup if you see any weaknesses. Better today than on our go-live date. Thanks, Matthew Roth InterMedia Marketing Solutions Software Engineer and Systems Developer Hello, I'm very interested in the specifics of your setup. How much space is on the RAM disk? What kind of RAM drive is it? What format are you recording to? What codec are the SIP calls being placed over? We've run into the Avoided deadlock recording issues several times when trying to do more than 50 concurrent recordings. Changing the ast_channel_lock loop from 10 to 20 has helped somewhat reduce the warnings and reduce audio gaps on the recordings, but what is really needed for more robust recording is a configurable recording buffer that wouldn't freak out if a 10ms delay occurs. Good luck and please keep us updated on your progress, MATT--- ___ --Bandwidth and Colocation sponsored by Easynews.com -- Asterisk-Users mailing list Asterisk-Users@lists.digium.com http://lists.digium.com/mailman/listinfo/asterisk-users To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users
