Great stuff. Glad to hear you found the issue.
Wireshark has helped me many times. Sometimes you have to get down to that layer to see what's really going on. From: gswe...@acts360.com [mailto:gswe...@acts360.com] Sent: Thursday, 17 March 2011 10:28 AM To: NT System Admin Issues Subject: RE: Seriously Wierd Issue Ok here is what we found... All of the workstation traffic for port 80 was getting timed out connecting to an IP address. Turns out there is an agent on every workstation that does a heartbeat check every 5 secs back to the main IP and every 5 mins to update processor, memory, disk stats, etc.. It does this over port 80. The remote side apparently did not accept as many connections as was needed so these workstations hung on port 80 requests... waiting for them to release I am guessing. In between a site would eventually pull up and we had full speed to download or whatever, but any time an additional http connection needed to be made it was essentially waiting for this service to release the http request. Once we turned off this service on all the workstations, http traffic died, btw it was like 30 bytes or something but 100+ workstations making an http every 5 secs apparently did it on connections rather than bandwidth. Now to deal with the company that is utilizing this software to get them to deal with their firewall or application to accept more http connections. I tried 2 different firewalls at the main site to ensure it wasn't theirs and same problem on each. Learned a lot about wireshark today.... Fin, ack, rst.... Greg Sweers CEO ACTS360.com <http://www.acts360.com/> P.O. Box 1193 Brandon, FL 33509 813-657-0849 Office 813-758-6850 Cell 813-341-1270 Fax From: Ziots, Edward [mailto:ezi...@lifespan.org] Sent: Wednesday, March 16, 2011 9:32 AM To: NT System Admin Issues Subject: RE: Seriously Wierd Issue You can also install Fiddler on the PC Client, and look at the http traffic being sent and the HTTP error codes, but I agree looking at Layer 3 traffic is first, before looking at Layer 7. Z Edward E. Ziots CISSP, Network +, Security + Network Engineer Lifespan Organization Email:ezi...@lifespan.org Cell:401-639-3505 From: gswe...@acts360.com [mailto:gswe...@acts360.com] Sent: Wednesday, March 16, 2011 9:17 AM To: NT System Admin Issues Subject: RE: Seriously Wierd Issue Good idea.. Limits the traffic down so I can see it easier. Will let you know. Thanks! Greg Sweers CEO ACTS360.com <http://www.acts360.com/> P.O. Box 1193 Brandon, FL 33509 813-657-0849 Office 813-758-6850 Cell 813-341-1270 Fax From: James Hill [mailto:j.h...@coffeeclub.com.au] Sent: Wednesday, March 16, 2011 2:04 AM To: NT System Admin Issues Subject: RE: Seriously Wierd Issue I'd run it on a PC first. Capture the traffic when you try to browse to a web page then have a look through the capture. You'll be able to see when it resolves the dns address and basically what happens after that. It's all time stamped so something obvious may appear. Also compare two captures, one during the day when the problem occurs and one after 4 or 5 pm. From: gswe...@acts360.com [mailto:gswe...@acts360.com] Sent: Wednesday, 16 March 2011 1:38 PM To: NT System Admin Issues Subject: RE: Seriously Wierd Issue Yeah I am going to connect one to the main switch and turn on the monitoring port tomorrow to see what we can see. Not sure what I am even looking for though. Greg Sweers CEO ACTS360.com <http://www.acts360.com/> P.O. Box 1193 Brandon, FL 33509 813-657-0849 Office 813-758-6850 Cell 813-341-1270 Fax From: James Hill [mailto:j.h...@coffeeclub.com.au] Sent: Tuesday, March 15, 2011 11:04 PM To: NT System Admin Issues Subject: RE: Seriously Wierd Issue Have you run wireshark on one of the pc's? It might help. Might not too but it's an avenue to try. From: gswe...@acts360.com [mailto:gswe...@acts360.com] Sent: Wednesday, 16 March 2011 12:40 PM To: NT System Admin Issues Subject: RE: Seriously Wierd Issue Yeah we have plugged in directly to the router, no issue, Plugged in behind firewall, no issue Behind iprism, no issue.. Its possible load, but when the issue is happening there is virtually zero on the pipe, and once the speedtest page loads, I can get 50/5 no issue. Its simply a delay in opening the page, IE cycles for 10 to 15 secs. I am going to test Firefox tomorrow to see if its something in IE, or a GP that's doing something weird, but I don't see that only happening during the day, but being fine at night. I am more thinking that there is some device on the network that is doing something...I just don't know how to identify weird traffic or responses that would cause the delays. Like I said, its just weird.. I am out of ideas. We are going over tomorrow during the day, and pulling each network segment off to see if we can isolate it to a specific network segment and progressively work it back until we find the culprit. Greg Sweers CEO ACTS360.com <http://www.acts360.com/> P.O. Box 1193 Brandon, FL 33509 813-657-0849 Office 813-758-6850 Cell 813-341-1270 Fax From: Kim Longenbaugh [mailto:k...@colonialsavings.com] Sent: Tuesday, March 15, 2011 5:27 PM To: NT System Admin Issues Subject: RE: Seriously Wierd Issue I think you're on the right track looking at the firewall, since the problem doesn't happen going to internal sites, which don't go across it. Maybe the firewall is doing some sort of packet inspection which is taking some time to complete but doesn't put much load on it. It's odd that the behavior changes later in the day, though. From: gswe...@acts360.com [mailto:gswe...@acts360.com] Sent: Tuesday, March 15, 2011 4:09 PM To: NT System Admin Issues Subject: Seriously Wierd Issue School of about 110 workstations. They have a Watchguard firewall doing no proxy's, connection limiting etc. We had an IPRISM in the picture, pulled it out for testing Verified all duplex speed settings. One main HP 2848 switch at the core, all other switches connected via Fiber are unmanaged. Had Brighthouse rerun the cable and replace the cable modem. I am able to connect to their test site and get 50meg down / 5 meg up consistently. Here is where it gets strange. During the day, it takes 10 secs for every website to pull up, DNS resolves immediately... and during this wait you can ping anything internally/externally without any issue. Once the page pulls up you are fine, any speedtest site for example, will run and zing...it full speed. Internally this is not an issue, any website, program , application runs fine. No speed issues. After 4 or 5 pm...this goes away completely. No issues resolving sites, pages display instantly or within a sec or so.. Looking at the firewall during this time, there is virtually no traffic on the pipe, low number of connections, under 100 usually passing through the firewall. But again, once I get to a speedtest I can download full speed.. Switch is sitting at 7 or 8 % CPU utilization. Any idea where to look, what to look for, or what could cause such a HUGE drop in performance for web, ftp, etc.. It has to be something on the network or a workstation, but I cannot figure out for the life of me what. I can spin up wireshark and dump the monitoring port to it, but I wouldn't know what to look for causing this. Greg Sweers CEO ACTS360.com <http://www.acts360.com/> P.O. Box 1193 Brandon, FL 33509 813-657-0849 Office 813-758-6850 Cell 813-341-1270 Fax ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe ntsysadmin
