For whats is worth I had a similar problem with sending data via FTP using a FTP control.
The firewall would drop packets if it did not like the negotiated data port . When this happened the application (this happened when the command line FTP in a windows shell was used) would hang. No time out or exception. I had to use WireShark to observe the FTP communication dialog. As the negotiated ports vary, most of the time it would work fine but every now and then this would happen. The Firewall did log the fact that it was dropping packets but the administrator did not think it important enough to let me know ( I am only user among many). If you are using a desktop application one can kill it. If it is a service that operates continuously then its a real headache. I had to write another service that monitored this service and kill it if was hung. The operating system would then start a new instance. Regards Peter From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Michael Lyons Sent: Thursday, 9 June 2011 1:46 PM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts We think we have narrowed it down to a firewall issue and are getting the vendor involved. From what we could see we were randomly getting a bunch of packets which seem to be being mysteriously dropped when transversing the trust/untrust zones of the firewall, nothing in the logs about it though. Only way we figured it out was via Wireshark at the start of this week; fiddler offered a hint via recreating SSL connections randomly. 99% sure its the firewall but theres still a little room for it to be something else. Thanks guys for your help, definitely using the failed request tracing on a permanent basis from now on. ----- Michael From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Paul Glavich Sent: Tuesday, 7 June 2011 1:17 PM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts Sorry for late response. I hardly get onto this list anymore due to time constraints. It wont be easy, but perhaps grap Wireshark/Ethereal and monitor the traffic looking for TCP/IP packets getting dropped, if using SSL, excessive SSL negotiation, certificate revocation paths taking a long time and things like that. Have seen similar issues where the TCP packets were getting dropped/retried due to (if I remember correctly as it has been a while) bad TCP settings 9something to do with the header, framesize I think) on an internal firewall. That particular problem took ages to find though. - Glav From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Michael Lyons Sent: Tuesday, 31 May 2011 3:09 PM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts No maximums have been set, only minimums. IIS VM has 2 cores and 1Ghz reserved for it with a maximum of about 4Ghz. Ive had failed request tracing on for anything that takes longer than 5 seconds and so far it has caught only one instance out of a number of times Ive seen it happen. Total time for the failed instance was just over 5 seconds. Strangest thing though was the duration for the ManagedPipelineHandler it has NO_END. The HttpRedirectionModule started before it, so I assume it ran a response redirect and aborted the thread causing the NO_END Ive also checked that garbage collection isnt interfering and its definitely not an issue. --- Michael From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of djones...@gmail.com Sent: Monday, 30 May 2011 4:00 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Hi Recently, they moved my production website and database on to virtual machines. We experianced slow downs, timeouts on the website and blocking table locks on the database. The only thing was that each Morning, the first person to connect recalculates the cache for everyone else ( batch job before start of day ). Before the migration, the page took 22 seconds to load and pegged out all 8 cpu's on the db machine. After migration, it timed out after 15 min, using 2 cpu cores and 25% cpu capacity. The team that did the move took the average processor usage and set that as the max available cpu usage on the db and websites. It's fixed now. Davy "When all you have is a hammer, every problem looks like a nail." I feel much the same way about xml _____ From: "Michael Lyons" <maill...@ittworx.com> Sender: ozdotnet-boun...@ozdotnet.com Date: Mon, 30 May 2011 14:23:51 +1000 To: 'ozDotNet'<ozdotnet@ozdotnet.com> ReplyTo: ozDotNet <ozdotnet@ozdotnet.com> Subject: RE: Website request slow performance / timeouts Noonie, No impersonation and connection pooling is on. Thanks for the suggestions was worth a double check. From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of noonie Sent: Thursday, 26 May 2011 10:39 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Michael, Just a long-shot... Are you impersonating the users when connecting to the database? Is connection pooling on? -- (mobile) noonie On 24/05/2011 2:29 PM, "Michael Lyons" <maill...@ittworx.com> wrote: > Ive been working on an ASP.Net solution which has a slow performance issues > and it has got me baffled. > > > > Problem: > > The production server randomly slows down when serving asp.net requests and > even times out. > > > > System architecture: > > The solution is hosted on a dedicated box which is running VmWares ESXi with > 4 VM servers sitting on it (1 per core). Each VM is on its own network. > > All network communication is done through a dedicated hardware firewall, > even between VMs (unfortunately the auditor has to have it this way). > > Database is on 1 VM while another has the web server. > > ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all on > top of Windows Server 2008 R2 > > > > Analysis to date: > > Ive run a profiler over the solution and so far come up with nothing that > really needs to be optimised. > > Our staging environment is running the same way as our production system > architecture minus the hardware firewall and has a lot lower hardware specs > but performs better than the production environment. When Im talking > slower, Im talking ¼ of the memory and a 7 year old CPU. > > Production IIS logs show some randomly high request execution times. > > > > Theories to date: > > ESXi is doing something weird and causing VMs to run slow. > > Firewall is blocking requests randomly or is having performance issues, > although I dont see it. > > IIS is randomly running slow. > > Sql Server is randomly running slow > > > > > > My questions: > > What would Windows performance counters would you watch? Besides the typical > CPU, Disk, memory and ASP.Net 4.0 counters? > > Does the IIS logs request execution times include the time to send the > network data? Eg. From time of socket open to time of socket closed? Or is > it just the pipeline without the TCP time included eg. Serving a straight > html file would just really be time to read the file from disk. > > What else would you look at? > > > > > > ------ > > Michael Lyons >
