RE: Website request slow performance / timeouts
For whats is worth I had a similar problem with sending data via FTP using a FTP control. The firewall would drop packets if it did not like the negotiated data port . When this happened the application (this happened when the command line FTP in a windows shell was used) would hang. No time out or exception. I had to use WireShark to observe the FTP communication dialog. As the negotiated ports vary, most of the time it would work fine but every now and then this would happen. The Firewall did log the fact that it was dropping packets but the administrator did not think it important enough to let me know ( I am only user among many). If you are using a desktop application one can kill it. If it is a service that operates continuously then its a real headache. I had to write another service that monitored this service and kill it if was hung. The operating system would then start a new instance. Regards Peter From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Michael Lyons Sent: Thursday, 9 June 2011 1:46 PM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts We think we have narrowed it down to a firewall issue and are getting the vendor involved. From what we could see we were randomly getting a bunch of packets which seem to be being mysteriously dropped when transversing the trust/untrust zones of the firewall, nothing in the logs about it though. Only way we figured it out was via Wireshark at the start of this week; fiddler offered a hint via recreating SSL connections randomly. 99% sure its the firewall but theres still a little room for it to be something else. Thanks guys for your help, definitely using the failed request tracing on a permanent basis from now on. - Michael From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Paul Glavich Sent: Tuesday, 7 June 2011 1:17 PM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts Sorry for late response. I hardly get onto this list anymore due to time constraints. It wont be easy, but perhaps grap Wireshark/Ethereal and monitor the traffic looking for TCP/IP packets getting dropped, if using SSL, excessive SSL negotiation, certificate revocation paths taking a long time and things like that. Have seen similar issues where the TCP packets were getting dropped/retried due to (if I remember correctly as it has been a while) bad TCP settings 9something to do with the header, framesize I think) on an internal firewall. That particular problem took ages to find though. - Glav From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Michael Lyons Sent: Tuesday, 31 May 2011 3:09 PM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts No maximums have been set, only minimums. IIS VM has 2 cores and 1Ghz reserved for it with a maximum of about 4Ghz. Ive had failed request tracing on for anything that takes longer than 5 seconds and so far it has caught only one instance out of a number of times Ive seen it happen. Total time for the failed instance was just over 5 seconds. Strangest thing though was the duration for the ManagedPipelineHandler it has NO_END. The HttpRedirectionModule started before it, so I assume it ran a response redirect and aborted the thread causing the NO_END Ive also checked that garbage collection isnt interfering and its definitely not an issue. --- Michael From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of djones...@gmail.com Sent: Monday, 30 May 2011 4:00 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Hi Recently, they moved my production website and database on to virtual machines. We experianced slow downs, timeouts on the website and blocking table locks on the database. The only thing was that each Morning, the first person to connect recalculates the cache for everyone else ( batch job before start of day ). Before the migration, the page took 22 seconds to load and pegged out all 8 cpu's on the db machine. After migration, it timed out after 15 min, using 2 cpu cores and 25% cpu capacity. The team that did the move took the average processor usage and set that as the max available cpu usage on the db and websites. It's fixed now. Davy When all you have is a hammer, every problem looks like a nail. I feel much the same way about xml _ From: Michael Lyons maill...@ittworx.com Sender: ozdotnet-boun...@ozdotnet.com Date: Mon, 30 May 2011 14:23:51 +1000 To: 'ozDotNet'ozdotnet@ozdotnet.com ReplyTo: ozDotNet ozdotnet@ozdotnet.com Subject: RE: Website request slow performance / timeouts Noonie, No impersonation and connection pooling is on. Thanks for the suggestions was worth a double check. From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun
Re: Website request slow performance / timeouts
Related fun: black hole routers http://support.microsoft.com/kb/314825 On Thu, Jun 9, 2011 at 4:26 PM, Peter Maddin petermad...@iinet.net.auwrote: For what’s is worth I had a similar problem with sending data via FTP using a FTP control. The firewall would drop packets if it did not like the negotiated data port . When this happened the application (this happened when the command line FTP in a windows shell was used) would hang. No time out or exception. I had to use WireShark to observe the FTP communication dialog. As the negotiated ports vary, most of the time it would work fine but every now and then this would happen. The Firewall did log the fact that it was dropping packets but the administrator did not think it important enough to let me know ( I am only user among many). If you are using a desktop application one can kill it. If it is a service that operates continuously then it’s a real headache. I had to write another service that monitored this service and kill it if was hung. The operating system would then start a new instance. Regards Peter *From:* ozdotnet-boun...@ozdotnet.com [mailto: ozdotnet-boun...@ozdotnet.com] *On Behalf Of *Michael Lyons *Sent:* Thursday, 9 June 2011 1:46 PM *To:* 'ozDotNet' *Subject:* RE: Website request slow performance / timeouts We think we have narrowed it down to a firewall issue and are getting the vendor involved. From what we could see we were randomly getting a bunch of packets which seem to be being mysteriously dropped when transversing the trust/untrust zones of the firewall, nothing in the logs about it though. Only way we figured it out was via Wireshark at the start of this week; fiddler offered a hint via recreating SSL connections randomly. 99% sure it’s the firewall but there’s still a little room for it to be something else. Thanks guys for your help, definitely using the failed request tracing on a permanent basis from now on. - Michael *From:* ozdotnet-boun...@ozdotnet.com [mailto: ozdotnet-boun...@ozdotnet.com] *On Behalf Of *Paul Glavich *Sent:* Tuesday, 7 June 2011 1:17 PM *To:* 'ozDotNet' *Subject:* RE: Website request slow performance / timeouts Sorry for late response. I hardly get onto this list anymore due to time constraints. It won’t be easy, but perhaps grap Wireshark/Ethereal and monitor the traffic looking for TCP/IP packets getting dropped, if using SSL, excessive SSL negotiation, certificate revocation paths taking a long time and things like that. Have seen similar issues where the TCP packets were getting dropped/retried due to (if I remember correctly as it has been a while) bad TCP settings 9something to do with the header, framesize I think) on an internal firewall. That particular problem took ages to find though. - Glav *From:* ozdotnet-boun...@ozdotnet.com [mailto: ozdotnet-boun...@ozdotnet.com] *On Behalf Of *Michael Lyons *Sent:* Tuesday, 31 May 2011 3:09 PM *To:* 'ozDotNet' *Subject:* RE: Website request slow performance / timeouts No maximums have been set, only minimums. IIS VM has 2 cores and 1Ghz reserved for it with a maximum of about 4Ghz. I’ve had failed request tracing on for anything that takes longer than 5 seconds and so far it has caught only one instance out of a number of times I’ve seen it happen. Total time for the failed instance was just over 5 seconds. Strangest thing though was the duration for the ManagedPipelineHandler it has “NO_END”. The HttpRedirectionModule started before it, so I assume it ran a response redirect and aborted the thread causing the NO_END I’ve also checked that garbage collection isn’t interfering and it’s definitely not an issue. --- Michael *From:* ozdotnet-boun...@ozdotnet.com [mailto: ozdotnet-boun...@ozdotnet.com] *On Behalf Of *djones...@gmail.com *Sent:* Monday, 30 May 2011 4:00 PM *To:* ozDotNet *Subject:* Re: Website request slow performance / timeouts Hi Recently, they moved my production website and database on to virtual machines. We experianced slow downs, timeouts on the website and blocking table locks on the database. The only thing was that each Morning, the first person to connect recalculates the cache for everyone else ( batch job before start of day ). Before the migration, the page took 22 seconds to load and pegged out all 8 cpu's on the db machine. After migration, it timed out after 15 min, using 2 cpu cores and 25% cpu capacity. The team that did the move took the average processor usage and set that as the max available cpu usage on the db and websites. It's fixed now. Davy When all you have is a hammer, every problem looks like a nail. I feel much the same way about xml -- *From: *Michael Lyons maill...@ittworx.com *Sender: *ozdotnet-boun...@ozdotnet.com *Date: *Mon, 30 May 2011 14:23:51 +1000 *To: *'ozDotNet'ozdotnet@ozdotnet.com
RE: Website request slow performance / timeouts
We think we have narrowed it down to a firewall issue and are getting the vendor involved. From what we could see we were randomly getting a bunch of packets which seem to be being mysteriously dropped when transversing the trust/untrust zones of the firewall, nothing in the logs about it though. Only way we figured it out was via Wireshark at the start of this week; fiddler offered a hint via recreating SSL connections randomly. 99% sure its the firewall but theres still a little room for it to be something else. Thanks guys for your help, definitely using the failed request tracing on a permanent basis from now on. - Michael From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Paul Glavich Sent: Tuesday, 7 June 2011 1:17 PM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts Sorry for late response. I hardly get onto this list anymore due to time constraints. It wont be easy, but perhaps grap Wireshark/Ethereal and monitor the traffic looking for TCP/IP packets getting dropped, if using SSL, excessive SSL negotiation, certificate revocation paths taking a long time and things like that. Have seen similar issues where the TCP packets were getting dropped/retried due to (if I remember correctly as it has been a while) bad TCP settings 9something to do with the header, framesize I think) on an internal firewall. That particular problem took ages to find though. - Glav From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Michael Lyons Sent: Tuesday, 31 May 2011 3:09 PM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts No maximums have been set, only minimums. IIS VM has 2 cores and 1Ghz reserved for it with a maximum of about 4Ghz. Ive had failed request tracing on for anything that takes longer than 5 seconds and so far it has caught only one instance out of a number of times Ive seen it happen. Total time for the failed instance was just over 5 seconds. Strangest thing though was the duration for the ManagedPipelineHandler it has NO_END. The HttpRedirectionModule started before it, so I assume it ran a response redirect and aborted the thread causing the NO_END Ive also checked that garbage collection isnt interfering and its definitely not an issue. --- Michael From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of djones...@gmail.com Sent: Monday, 30 May 2011 4:00 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Hi Recently, they moved my production website and database on to virtual machines. We experianced slow downs, timeouts on the website and blocking table locks on the database. The only thing was that each Morning, the first person to connect recalculates the cache for everyone else ( batch job before start of day ). Before the migration, the page took 22 seconds to load and pegged out all 8 cpu's on the db machine. After migration, it timed out after 15 min, using 2 cpu cores and 25% cpu capacity. The team that did the move took the average processor usage and set that as the max available cpu usage on the db and websites. It's fixed now. Davy When all you have is a hammer, every problem looks like a nail. I feel much the same way about xml _ From: Michael Lyons maill...@ittworx.com Sender: ozdotnet-boun...@ozdotnet.com Date: Mon, 30 May 2011 14:23:51 +1000 To: 'ozDotNet'ozdotnet@ozdotnet.com ReplyTo: ozDotNet ozdotnet@ozdotnet.com Subject: RE: Website request slow performance / timeouts Noonie, No impersonation and connection pooling is on. Thanks for the suggestions was worth a double check. From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of noonie Sent: Thursday, 26 May 2011 10:39 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Michael, Just a long-shot... Are you impersonating the users when connecting to the database? Is connection pooling on? -- (mobile) noonie On 24/05/2011 2:29 PM, Michael Lyons maill...@ittworx.com wrote: Ive been working on an ASP.Net solution which has a slow performance issues and it has got me baffled. Problem: The production server randomly slows down when serving asp.net requests and even times out. System architecture: The solution is hosted on a dedicated box which is running VmWares ESXi with 4 VM servers sitting on it (1 per core). Each VM is on its own network. All network communication is done through a dedicated hardware firewall, even between VMs (unfortunately the auditor has to have it this way). Database is on 1 VM while another has the web server. ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all on top of Windows Server 2008 R2 Analysis to date: Ive run a profiler over the solution and so far come up with nothing
Re: Website request slow performance / timeouts
Hi Recently, they moved my production website and database on to virtual machines. We experianced slow downs, timeouts on the website and blocking table locks on the database. The only thing was that each Morning, the first person to connect recalculates the cache for everyone else ( batch job before start of day ). Before the migration, the page took 22 seconds to load and pegged out all 8 cpu's on the db machine. After migration, it timed out after 15 min, using 2 cpu cores and 25% cpu capacity. The team that did the move took the average processor usage and set that as the max available cpu usage on the db and websites. It's fixed now. Davy When all you have is a hammer, every problem looks like a nail. I feel much the same way about xml -Original Message- From: Michael Lyons maill...@ittworx.com Sender: ozdotnet-boun...@ozdotnet.com Date: Mon, 30 May 2011 14:23:51 To: 'ozDotNet'ozdotnet@ozdotnet.com Reply-To: ozDotNet ozdotnet@ozdotnet.com Subject: RE: Website request slow performance / timeouts Noonie, No impersonation and connection pooling is on. Thanks for the suggestions was worth a double check. From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of noonie Sent: Thursday, 26 May 2011 10:39 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Michael, Just a long-shot... Are you impersonating the users when connecting to the database? Is connection pooling on? -- (mobile) noonie On 24/05/2011 2:29 PM, Michael Lyons maill...@ittworx.com wrote: I’ve been working on an ASP.Net solution which has a slow performance issues and it has got me baffled. Problem: The production server randomly slows down when serving asp.net requests and even times out. System architecture: The solution is hosted on a dedicated box which is running VmWares ESXi with 4 VM servers sitting on it (1 per core). Each VM is on its own network. All network communication is done through a dedicated hardware firewall, even between VM’s (unfortunately the auditor has to have it this way). Database is on 1 VM while another has the web server. ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all on top of Windows Server 2008 R2 Analysis to date: I’ve run a profiler over the solution and so far come up with nothing that really needs to be optimised. Our staging environment is running the same way as our production system architecture minus the hardware firewall and has a lot lower hardware specs but performs better than the production environment. When I’m talking slower, I’m talking ¼ of the memory and a 7 year old CPU. Production IIS logs show some randomly high request execution times. Theories to date: ESXi is doing something weird and causing VM’s to run slow. Firewall is blocking requests randomly or is having performance issues, although I don’t see it. IIS is randomly running slow. Sql Server is randomly running slow My questions: What would Windows performance counters would you watch? Besides the typical CPU, Disk, memory and ASP.Net 4.0 counters? Does the IIS logs request execution times include the time to send the network data? Eg. From time of socket open to time of socket closed? Or is it just the pipeline without the TCP time included – eg. Serving a straight html file would just really be time to read the file from disk. What else would you look at? -- Michael Lyons
RE: Website request slow performance / timeouts
No maximums have been set, only minimums. IIS VM has 2 cores and 1Ghz reserved for it with a maximum of about 4Ghz. Ive had failed request tracing on for anything that takes longer than 5 seconds and so far it has caught only one instance out of a number of times Ive seen it happen. Total time for the failed instance was just over 5 seconds. Strangest thing though was the duration for the ManagedPipelineHandler it has NO_END. The HttpRedirectionModule started before it, so I assume it ran a response redirect and aborted the thread causing the NO_END Ive also checked that garbage collection isnt interfering and its definitely not an issue. --- Michael From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of djones...@gmail.com Sent: Monday, 30 May 2011 4:00 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Hi Recently, they moved my production website and database on to virtual machines. We experianced slow downs, timeouts on the website and blocking table locks on the database. The only thing was that each Morning, the first person to connect recalculates the cache for everyone else ( batch job before start of day ). Before the migration, the page took 22 seconds to load and pegged out all 8 cpu's on the db machine. After migration, it timed out after 15 min, using 2 cpu cores and 25% cpu capacity. The team that did the move took the average processor usage and set that as the max available cpu usage on the db and websites. It's fixed now. Davy When all you have is a hammer, every problem looks like a nail. I feel much the same way about xml _ From: Michael Lyons maill...@ittworx.com Sender: ozdotnet-boun...@ozdotnet.com Date: Mon, 30 May 2011 14:23:51 +1000 To: 'ozDotNet'ozdotnet@ozdotnet.com ReplyTo: ozDotNet ozdotnet@ozdotnet.com Subject: RE: Website request slow performance / timeouts Noonie, No impersonation and connection pooling is on. Thanks for the suggestions was worth a double check. From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of noonie Sent: Thursday, 26 May 2011 10:39 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Michael, Just a long-shot... Are you impersonating the users when connecting to the database? Is connection pooling on? -- (mobile) noonie On 24/05/2011 2:29 PM, Michael Lyons maill...@ittworx.com wrote: Ive been working on an ASP.Net solution which has a slow performance issues and it has got me baffled. Problem: The production server randomly slows down when serving asp.net requests and even times out. System architecture: The solution is hosted on a dedicated box which is running VmWares ESXi with 4 VM servers sitting on it (1 per core). Each VM is on its own network. All network communication is done through a dedicated hardware firewall, even between VMs (unfortunately the auditor has to have it this way). Database is on 1 VM while another has the web server. ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all on top of Windows Server 2008 R2 Analysis to date: Ive run a profiler over the solution and so far come up with nothing that really needs to be optimised. Our staging environment is running the same way as our production system architecture minus the hardware firewall and has a lot lower hardware specs but performs better than the production environment. When Im talking slower, Im talking ¼ of the memory and a 7 year old CPU. Production IIS logs show some randomly high request execution times. Theories to date: ESXi is doing something weird and causing VMs to run slow. Firewall is blocking requests randomly or is having performance issues, although I dont see it. IIS is randomly running slow. Sql Server is randomly running slow My questions: What would Windows performance counters would you watch? Besides the typical CPU, Disk, memory and ASP.Net 4.0 counters? Does the IIS logs request execution times include the time to send the network data? Eg. From time of socket open to time of socket closed? Or is it just the pipeline without the TCP time included eg. Serving a straight html file would just really be time to read the file from disk. What else would you look at? -- Michael Lyons
RE: Website request slow performance / timeouts
Noonie, No impersonation and connection pooling is on. Thanks for the suggestions was worth a double check. From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of noonie Sent: Thursday, 26 May 2011 10:39 PM To: ozDotNet Subject: Re: Website request slow performance / timeouts Michael, Just a long-shot... Are you impersonating the users when connecting to the database? Is connection pooling on? -- (mobile) noonie On 24/05/2011 2:29 PM, Michael Lyons maill...@ittworx.com wrote: Ive been working on an ASP.Net solution which has a slow performance issues and it has got me baffled. Problem: The production server randomly slows down when serving asp.net requests and even times out. System architecture: The solution is hosted on a dedicated box which is running VmWares ESXi with 4 VM servers sitting on it (1 per core). Each VM is on its own network. All network communication is done through a dedicated hardware firewall, even between VMs (unfortunately the auditor has to have it this way). Database is on 1 VM while another has the web server. ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all on top of Windows Server 2008 R2 Analysis to date: Ive run a profiler over the solution and so far come up with nothing that really needs to be optimised. Our staging environment is running the same way as our production system architecture minus the hardware firewall and has a lot lower hardware specs but performs better than the production environment. When Im talking slower, Im talking ¼ of the memory and a 7 year old CPU. Production IIS logs show some randomly high request execution times. Theories to date: ESXi is doing something weird and causing VMs to run slow. Firewall is blocking requests randomly or is having performance issues, although I dont see it. IIS is randomly running slow. Sql Server is randomly running slow My questions: What would Windows performance counters would you watch? Besides the typical CPU, Disk, memory and ASP.Net 4.0 counters? Does the IIS logs request execution times include the time to send the network data? Eg. From time of socket open to time of socket closed? Or is it just the pipeline without the TCP time included eg. Serving a straight html file would just really be time to read the file from disk. What else would you look at? -- Michael Lyons
RE: Website request slow performance / timeouts
Glav, · Actually staging is 32bit and production 64bit. I just swapped production to 32bit to see if that will make a difference. So far the issue still exists though even with the switch, hopefully there is a slight performance improvement in general. · Ive looked at request execution time and its a bit bizarre. Most of the time it is 15ms on a request that takes longer than it should, but sometimes it suddenly jumps up to over 10 seconds for a rather simple page. Monitoring the firewall shows query execution time between the web server and the database server is done in less than 2 seconds (firewall doesnt measure less than 1 second and includes TCP socket open and close). · The firewall shows that the TCP socket is getting reset on the times when these issues occur, meaning that there is no TCP end frame. Yet those times I can clearly see that from the IIS logs that request execution time is less than a seconds to return the page. Why would IIS not return/close the socket? · Havent noticed any app restarts in the event logs, but Ill put this under a 48 hour performance log. · No havent noticed any memory leaks as yet. Although Ill turn the GC counter on in the performance log as well. Thanks - Michael From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Paul Glavich Sent: Thursday, 26 May 2011 8:37 AM To: 'ozDotNet' Subject: RE: Website request slow performance / timeouts A few things you can look at are: · Is staging a 32 bit environment or is everything all 64bit? If memory requirements are not large you can look at running IIS in 32bit mode in the VMs which can yield good CPU utilisation benefits. Generally, VMs suck IMHO compared to physical. I have personally used 32bit mode IIS within a 64bit VM and seen very good CPU improvement. · Do you use stored procedures and if so, do many of the stored procedures have conditional logic in there that cause different SQL statements to be executed based in differing input parameters (not talking about constructing SQL strings and using sp_executesql tho)? This kind of thing can cause SQL to use incorrect query plans for a proc and cause really long execution times of that proc until it is recompiled/query plan removed from cache. · Look at the Asp.NET request execution time counter to see if ASP.NET itself is taking a long time to process any requests. If not, then you know its outside of the ASP.NET app space. If yes, it could be doing either a long bit of processing or waiting on something like a SQL query to execute · Also look at Asp.NET Application Restart and worker process restarts. App restarts may be causing app recompilation and thus making requests take longer. · I assume you have looked at the memory counter to ensure no memory leaks are present (ever increasing memory usage). If you suspect this, you may also want to look at the GC counts in the Asp.Net Memory counters to ensure an excessive amount of garbage collection is not holding up processing (although u would see this on staging as well). - Glav From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Michael Lyons Sent: Tuesday, 24 May 2011 2:29 PM To: 'ozDotNet' Subject: Website request slow performance / timeouts Ive been working on an ASP.Net solution which has a slow performance issues and it has got me baffled. Problem: The production server randomly slows down when serving asp.net requests and even times out. System architecture: The solution is hosted on a dedicated box which is running VmWares ESXi with 4 VM servers sitting on it (1 per core). Each VM is on its own network. All network communication is done through a dedicated hardware firewall, even between VMs (unfortunately the auditor has to have it this way). Database is on 1 VM while another has the web server. ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all on top of Windows Server 2008 R2 Analysis to date: Ive run a profiler over the solution and so far come up with nothing that really needs to be optimised. Our staging environment is running the same way as our production system architecture minus the hardware firewall and has a lot lower hardware specs but performs better than the production environment. When Im talking slower, Im talking ¼ of the memory and a 7 year old CPU. Production IIS logs show some randomly high request execution times. Theories to date: ESXi is doing something weird and causing VMs to run slow. Firewall is blocking requests randomly or is having performance issues, although I dont see it. IIS is randomly running slow. Sql Server is randomly running slow My questions: What would Windows performance counters would you watch? Besides the typical CPU, Disk, memory and ASP.Net 4.0 counters? Does the IIS logs request
RE: Website request slow performance / timeouts
A few things you can look at are: · Is staging a 32 bit environment or is everything all 64bit? If memory requirements are not large you can look at running IIS in 32bit mode in the VMs which can yield good CPU utilisation benefits. Generally, VMs suck IMHO compared to physical. I have personally used 32bit mode IIS within a 64bit VM and seen very good CPU improvement. · Do you use stored procedures and if so, do many of the stored procedures have conditional logic in there that cause different SQL statements to be executed based in differing input parameters (not talking about constructing SQL strings and using sp_executesql tho)? This kind of thing can cause SQL to use incorrect query plans for a proc and cause really long execution times of that proc until it is recompiled/query plan removed from cache. · Look at the Asp.NET request execution time counter to see if ASP.NET itself is taking a long time to process any requests. If not, then you know its outside of the ASP.NET app space. If yes, it could be doing either a long bit of processing or waiting on something like a SQL query to execute · Also look at Asp.NET Application Restart and worker process restarts. App restarts may be causing app recompilation and thus making requests take longer. · I assume you have looked at the memory counter to ensure no memory leaks are present (ever increasing memory usage). If you suspect this, you may also want to look at the GC counts in the Asp.Net Memory counters to ensure an excessive amount of garbage collection is not holding up processing (although u would see this on staging as well). - Glav From: ozdotnet-boun...@ozdotnet.com [mailto:ozdotnet-boun...@ozdotnet.com] On Behalf Of Michael Lyons Sent: Tuesday, 24 May 2011 2:29 PM To: 'ozDotNet' Subject: Website request slow performance / timeouts Ive been working on an ASP.Net solution which has a slow performance issues and it has got me baffled. Problem: The production server randomly slows down when serving asp.net requests and even times out. System architecture: The solution is hosted on a dedicated box which is running VmWares ESXi with 4 VM servers sitting on it (1 per core). Each VM is on its own network. All network communication is done through a dedicated hardware firewall, even between VMs (unfortunately the auditor has to have it this way). Database is on 1 VM while another has the web server. ASP.Net is v4 running on IIS 7.5 while database is SQL Server 2008R2 all on top of Windows Server 2008 R2 Analysis to date: Ive run a profiler over the solution and so far come up with nothing that really needs to be optimised. Our staging environment is running the same way as our production system architecture minus the hardware firewall and has a lot lower hardware specs but performs better than the production environment. When Im talking slower, Im talking ¼ of the memory and a 7 year old CPU. Production IIS logs show some randomly high request execution times. Theories to date: ESXi is doing something weird and causing VMs to run slow. Firewall is blocking requests randomly or is having performance issues, although I dont see it. IIS is randomly running slow. Sql Server is randomly running slow My questions: What would Windows performance counters would you watch? Besides the typical CPU, Disk, memory and ASP.Net 4.0 counters? Does the IIS logs request execution times include the time to send the network data? Eg. From time of socket open to time of socket closed? Or is it just the pipeline without the TCP time included eg. Serving a straight html file would just really be time to read the file from disk. What else would you look at? -- Michael Lyons
