Related fun: black hole routers http://support.microsoft.com/kb/314825
On Thu, Jun 9, 2011 at 4:26 PM, Peter Maddin <petermad...@iinet.net.au>wrote: > For what’s is worth I had a similar problem with sending data via FTP using > a FTP control. > > The firewall would drop packets if it did not like the negotiated data port > . > > > > When this happened the application (this happened when the command line FTP > in a windows shell was used) would hang. > > No time out or exception. > > > > I had to use WireShark to observe the FTP communication dialog. > > As the negotiated ports vary, most of the time it would work fine but every > now and then this would happen. > > > > The Firewall did log the fact that it was dropping packets but the > administrator did not think it important enough to let me know ( I am only > user among many). > > > > If you are using a desktop application one can kill it. > > If it is a service that operates continuously then it’s a real headache. > > I had to write another service that monitored this service and kill it if > was hung. > > The operating system would then start a new instance. > > > > Regards Peter > > *From:* ozdotnet-boun...@ozdotnet.com [mailto: > ozdotnet-boun...@ozdotnet.com] *On Behalf Of *Michael Lyons > *Sent:* Thursday, 9 June 2011 1:46 PM > > *To:* 'ozDotNet' > *Subject:* RE: Website request slow performance / timeouts > > > > We think we have narrowed it down to a firewall issue and are getting the > vendor involved. From what we could see we were randomly getting a bunch of > packets which seem to be being mysteriously dropped when transversing the > trust/untrust zones of the firewall, nothing in the logs about it though. > Only way we figured it out was via Wireshark at the start of this week; > fiddler offered a hint via recreating SSL connections randomly. 99% sure > it’s the firewall but there’s still a little room for it to be something > else. > > > > Thanks guys for your help, definitely using the failed request tracing on a > permanent basis from now on. > > > > ----- > > Michael > > > > *From:* ozdotnet-boun...@ozdotnet.com [mailto: > ozdotnet-boun...@ozdotnet.com] *On Behalf Of *Paul Glavich > *Sent:* Tuesday, 7 June 2011 1:17 PM > *To:* 'ozDotNet' > *Subject:* RE: Website request slow performance / timeouts > > > > Sorry for late response. I hardly get onto this list anymore due to time > constraints. > > > > It won’t be easy, but perhaps grap Wireshark/Ethereal and monitor the > traffic looking for TCP/IP packets getting dropped, if using SSL, excessive > SSL negotiation, certificate revocation paths taking a long time and things > like that. Have seen similar issues where the TCP packets were getting > dropped/retried due to (if I remember correctly as it has been a while) bad > TCP settings 9something to do with the header, framesize I think) on an > internal firewall. That particular problem took ages to find though. > > > > > > - Glav > > > > *From:* ozdotnet-boun...@ozdotnet.com [mailto: > ozdotnet-boun...@ozdotnet.com] *On Behalf Of *Michael Lyons > *Sent:* Tuesday, 31 May 2011 3:09 PM > *To:* 'ozDotNet' > *Subject:* RE: Website request slow performance / timeouts > > > > No maximums have been set, only minimums. IIS VM has 2 cores and 1Ghz > reserved for it with a maximum of about 4Ghz. > > > > I’ve had failed request tracing on for anything that takes longer than 5 > seconds and so far it has caught only one instance out of a number of times > I’ve seen it happen. Total time for the failed instance was just over 5 > seconds. > > Strangest thing though was the duration for the ManagedPipelineHandler it > has “NO_END”. The HttpRedirectionModule started before it, so I assume it > ran a response redirect and aborted the thread causing the NO_END > > > > I’ve also checked that garbage collection isn’t interfering and it’s > definitely not an issue. > > > > --- > > Michael > > > > *From:* ozdotnet-boun...@ozdotnet.com [mailto: > ozdotnet-boun...@ozdotnet.com] *On Behalf Of *djones...@gmail.com > *Sent:* Monday, 30 May 2011 4:00 PM > *To:* ozDotNet > *Subject:* Re: Website request slow performance / timeouts > > > > Hi > > Recently, they moved my production website and database on to virtual > machines. > > We experianced slow downs, timeouts on the website and blocking table locks > on the database. > > The only thing was that each Morning, the first person to connect > recalculates the cache for everyone else ( batch job before start of day ). > > Before the migration, the page took 22 seconds to load and pegged out all 8 > cpu's on the db machine. > > After migration, it timed out after 15 min, using 2 cpu cores and 25% cpu > capacity. > > The team that did the move took the average processor usage and set that as > the max available cpu usage on the db and websites. > > It's fixed now. > > Davy > > "When all you have is a hammer, every problem looks like a nail." I feel > much the same way about xml > ------------------------------ > > *From: *"Michael Lyons" <maill...@ittworx.com> > > *Sender: *ozdotnet-boun...@ozdotnet.com > > *Date: *Mon, 30 May 2011 14:23:51 +1000 > > *To: *'ozDotNet'<ozdotnet@ozdotnet.com> > > *ReplyTo: *ozDotNet <ozdotnet@ozdotnet.com> > > *Subject: *RE: Website request slow performance / timeouts > > > > Noonie, > > > > No impersonation and connection pooling is on. > > > > Thanks for the suggestions was worth a double check. > > > > *From:* ozdotnet-boun...@ozdotnet.com [mailto: > ozdotnet-boun...@ozdotnet.com] *On Behalf Of *noonie > *Sent:* Thursday, 26 May 2011 10:39 PM > *To:* ozDotNet > *Subject:* Re: Website request slow performance / timeouts > > > > Michael, > > Just a long-shot... > > Are you impersonating the users when connecting to the database? > > Is connection pooling on? > > -- > (mobile) noonie > > On 24/05/2011 2:29 PM, "Michael Lyons" <maill...@ittworx.com> wrote: > > I’ve been working on an ASP.Net solution which has a slow performance > issues > > and it has got me baffled. > > > > > > > > Problem: > > > > The production server randomly slows down when serving asp.net requests > and > > even times out. > > > > > > > > System architecture: > > > > The solution is hosted on a dedicated box which is running VmWares ESXi > with > > 4 VM servers sitting on it (1 per core). Each VM is on its own network. > > > > All network communication is done through a dedicated hardware firewall, > > even between VM’s (unfortunately the auditor has to have it this way). > > > > Database is on 1 VM while another has the web server. > > > > ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all > on > > top of Windows Server 2008 R2 > > > > > > > > Analysis to date: > > > > I’ve run a profiler over the solution and so far come up with nothing > that > > really needs to be optimised. > > > > Our staging environment is running the same way as our production system > > architecture minus the hardware firewall and has a lot lower hardware > specs > > but performs better than the production environment. When I’m talking > > slower, I’m talking ¼ of the memory and a 7 year old CPU. > > > > Production IIS logs show some randomly high request execution times. > > > > > > > > Theories to date: > > > > ESXi is doing something weird and causing VM’s to run slow. > > > > Firewall is blocking requests randomly or is having performance issues, > > although I don’t see it. > > > > IIS is randomly running slow. > > > > Sql Server is randomly running slow > > > > > > > > > > > > My questions: > > > > What would Windows performance counters would you watch? Besides the > typical > > CPU, Disk, memory and ASP.Net 4.0 counters? > > > > Does the IIS logs request execution times include the time to send the > > network data? Eg. From time of socket open to time of socket closed? Or > is > > it just the pipeline without the TCP time included – eg. Serving a > straight > > html file would just really be time to read the file from disk. > > > > What else would you look at? > > > > > > > > > > > > ------ > > > > Michael Lyons > > > -- w: http://jcooney.net t: @josephcooney
