Tracked this down (or more to the point found the issue causing it), it was high levels of bursty disk activity.
The iowait went through the roof (30-40%). The disks are scsi serviced by an MPT-Fusion controller in a Dell Poweredge 2850. We're using LVM to bind the disks into a JBOD set. Steve -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steve Hanselman Sent: 11 June 2007 10:40 To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: RE: [asterisk-users] Bridged PRI calls - processor involvement? I checked for BIOS upgrades the other week and there were none. I'm starting to suspect kernel changes as being the reason for this so I guess I'm going to have to remove some of the patchy disk activity to smooth the load and then start researching!!! Steve -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Gordon Henderson Sent: 11 June 2007 09:53 To: Asterisk Users Mailing List - Non-Commercial Discussion Subject: RE: [asterisk-users] Bridged PRI calls - processor involvement? On Mon, 11 Jun 2007, Steve Hanselman wrote: > This is the io wait figure from vmstat. > > If I run a vmstat 2 whilst I'm on a call I can see that the "wa" figure > gets very high when the missing audio problem occurs. I once looked after a Dell 2850 that exhibited some odd behaviour that I never got to the bottom of. It would seem to lock-up or just crawl for 2-3 seconds every now & then. Nothing logged, noting on the console. It had 6 SCSI drives fitted. I rebuilt the server twice, rebuilt the s/w RAID arrays twice, even put all 6 drives in another box (which appeared towork OK), but never got to the bottom of it. Each disk would benchmark really fast individually, Ethernet performance was good, but overall, when everything was used together, it just didn't feel right. (compared to other Dells and other servers, biger & smaller that I've built and used over the years). I'd see processes hung in a "D" state (waiting for IO to complete) for what seemed like an overly long time, (waiting on disk), but ... I suspected a BIOS pproblem, but never had a chance to get to the bottom of it. (It was a live server doing *everything* for a small company - DNS, NIS, NFS, Intranet/WiKi, Samba, etc, etc, etc,... so taking it offline for tests was problematic) So I wonder if looking at the BIOS and seeing if there are any Dell upgrades avalable for it might help? Gordon > > Steve > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Matthew > Fredrickson > Sent: 08 June 2007 19:38 > To: Asterisk Users Mailing List - Non-Commercial Discussion > Subject: Re: [asterisk-users] Bridged PRI calls - processor involvement? > > iowait time? I'm not familiar with that. Where are you seeing that? > Also, is it a reproducible problem? > > --- > Matthew Fredrickson > Software Engineer > Digium, Inc. > > On Jun 8, 2007, at 11:23 AM, Steve Hanselman wrote: > >> It probably did but we run in updates every week and nobody can state >> exactly when the problem started only "a few weeks ago" - not very >> helpful. >> >> I can see that when I hear the issue the iowait time is high on the >> processor. >> >> Steve >> -----Original Message----- >> From: [EMAIL PROTECTED] >> [mailto:[EMAIL PROTECTED] On Behalf Of Matthew >> Fredrickson >> Sent: 08 June 2007 15:43 >> To: Asterisk Users Mailing List - Non-Commercial Discussion >> Subject: Re: [asterisk-users] Bridged PRI calls - processor >> involvement? >> >> Did it accompany an update you made? If you can find out what version >> the problem started occurring, that would help in fixing the problem, >> >> Matthew Fredrickson >> Software/Hardware Engineer >> Digium, Inc. >> >> On Jun 8, 2007, at 2:59 AM, Steve Hanselman wrote: >> >>> The setup. >>> >>> Asterisk is on a 3G Zeon Dell 2850 running Fedora Core 5/6 (all yum >>> updates applied), the TE410 lives on it's own interrupt. >>> Asterisk sits between our telco and a PRI enabled PBX. >>> These are the relevant versions installed: >>> >>> Linux: 2.6.20-1.2316.fc5smp >>> Zaptel: 1:1.4.2.1-34.fc5 >>> Asterisk: 1:1.4.0-34.fc5.at >>> Libpri: 1:1.4.0-16.fc5.at >>> Wildcard details: >>> Found TE4XXP at base address fe3ffc00, remapped to f88bec00 >>> TE4XXP version c01a016a, burst OFF, slip debug: OFF >>> Octasic optimized! >>> FALC version: 00000005, Board ID: 00 >>> Reg 0: 0x377bb400 >>> Reg 1: 0x377bb000 >>> Reg 2: 0xffffffff >>> Reg 3: 0x00000000 >>> Reg 4: 0x00000001 >>> Reg 5: 0x00000000 >>> Reg 6: 0xc01a016a >>> Reg 7: 0x00001f00 >>> Reg 8: 0x00000000 >>> Reg 9: 0x00ff0000 >>> Reg 10: 0x0000004a >>> TTE4XXP: Launching card: 0 >>> TE4XXP: Setting up global serial parameters >>> Found a Wildcard: Wildcard TE410P (3rd Gen) >>> TE4XXP: Span 1 configured for CCS/HDB3/CRC4 >>> TE4XXP: Span 2 configured for CCS/HDB3/CRC4 >>> >>> >>> >>> The problem: >>> >>> At random points during calls we lose 1-3 seconds of speech (both > ways >>> both callee and caller), this can be replicated (or at least a very >>> good >>> approximation!) by generating a high level of interrupt/cpu activity >>> (for instance copying data from a USB caddy as we tried the other day >>> in >>> an attempt to reproduce this more reliably). >>> >>> The calls are bridged PRI:PRI calls, no VOIP involvement. >>> >>> This was not a problem until approx 3-4 weeks ago, but I can't tie it >>> down to an exact date. >>> >>> Steve >>> >>> >>>> Interrupt sharing is not a problem anymore with those cards. What >>>> version of zaptel did you try installing? Can you explain more > about >>>> your problems? Also, your configuration and setup would help out as >>>> well. >>>> >>>> --- >>>> Matthew Fredrickson >>>> Digium, Inc. >>> >>> >>> The information contained in this email is intended for the personal >>> and confidential use >>> of the addressee only. It may also be privileged information. If you >>> are not the intended >>> recipient then you are hereby notified that you have received this >>> document in error and >>> that any review, distribution or copying of this document is strictly >>> prohibited. If you have >>> received this communication in error, please notify Brendata >>> immediately on: >>> >>> +44 (0)1268 466100, or email '[EMAIL PROTECTED]' >>> >>> Brendata (UK) Ltd >>> Nevendon Hall, Nevendon Road, Basildon, Essex. SS13 1BX UK >>> Registered Office as above. Registered in England No. 2764339 >>> >>> See our current vacancies at www.brendata.co.uk >>> _______________________________________________ >>> --Bandwidth and Colocation provided by Easynews.com -- >>> >>> asterisk-users mailing list >>> To UNSUBSCRIBE or update options visit: >>> http://lists.digium.com/mailman/listinfo/asterisk-users >> >> _______________________________________________ >> --Bandwidth and Colocation provided by Easynews.com -- >> >> asterisk-users mailing list >> To UNSUBSCRIBE or update options visit: >> http://lists.digium.com/mailman/listinfo/asterisk-users >> >> >> The information contained in this email is intended for the personal >> and confidential use >> of the addressee only. It may also be privileged information. If you >> are not the intended >> recipient then you are hereby notified that you have received this >> document in error and >> that any review, distribution or copying of this document is strictly >> prohibited. If you have >> received this communication in error, please notify Brendata >> immediately on: >> >> +44 (0)1268 466100, or email '[EMAIL PROTECTED]' >> >> Brendata (UK) Ltd >> Nevendon Hall, Nevendon Road, Basildon, Essex. SS13 1BX UK >> Registered Office as above. Registered in England No. 2764339 >> >> See our current vacancies at www.brendata.co.uk >> _______________________________________________ >> --Bandwidth and Colocation provided by Easynews.com -- >> >> asterisk-users mailing list >> To UNSUBSCRIBE or update options visit: >> http://lists.digium.com/mailman/listinfo/asterisk-users > > _______________________________________________ > --Bandwidth and Colocation provided by Easynews.com -- > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users > > > The information contained in this email is intended for the personal and confidential use > of the addressee only. It may also be privileged information. If you are not the intended > recipient then you are hereby notified that you have received this document in error and > that any review, distribution or copying of this document is strictly prohibited. If you have > received this communication in error, please notify Brendata immediately on: > > +44 (0)1268 466100, or email '[EMAIL PROTECTED]' > > Brendata (UK) Ltd > Nevendon Hall, Nevendon Road, Basildon, Essex. SS13 1BX UK > Registered Office as above. Registered in England No. 2764339 > > See our current vacancies at www.brendata.co.uk > _______________________________________________ > --Bandwidth and Colocation provided by Easynews.com -- > > asterisk-users mailing list > To UNSUBSCRIBE or update options visit: > http://lists.digium.com/mailman/listinfo/asterisk-users > _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users The information contained in this email is intended for the personal and confidential use of the addressee only. It may also be privileged information. If you are not the intended recipient then you are hereby notified that you have received this document in error and that any review, distribution or copying of this document is strictly prohibited. If you have received this communication in error, please notify Brendata immediately on: +44 (0)1268 466100, or email '[EMAIL PROTECTED]' Brendata (UK) Ltd Nevendon Hall, Nevendon Road, Basildon, Essex. SS13 1BX UK Registered Office as above. Registered in England No. 2764339 See our current vacancies at www.brendata.co.uk _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users The information contained in this email is intended for the personal and confidential use of the addressee only. It may also be privileged information. If you are not the intended recipient then you are hereby notified that you have received this document in error and that any review, distribution or copying of this document is strictly prohibited. If you have received this communication in error, please notify Brendata immediately on: +44 (0)1268 466100, or email '[EMAIL PROTECTED]' Brendata (UK) Ltd Nevendon Hall, Nevendon Road, Basildon, Essex. SS13 1BX UK Registered Office as above. Registered in England No. 2764339 See our current vacancies at www.brendata.co.uk _______________________________________________ --Bandwidth and Colocation provided by Easynews.com -- asterisk-users mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-users