Re: Network scan tool/appliance horror stories
* Jones, Barry (bejo...@semprautilities.com) wrote: I can share with you several stories personnel (both IT or vendors), who have scanned Electric Utility environments with or without permission; and hence caused multiple failures - including electro-mechanical systems and related applications. Utilities typically utilize many industrial controllers - some of which many IT personnel have no knowledge, and some are not robust enough to weather the storm. 1. Know your environment. 2. Know your tools. 3. Communicate. Second that. First agree on what rate they are allowed to scan your network, then let them come back with what they find before they point other tools at the found nodes. Then inform the owners of said nodes of what is going to happen. In a previous life I found an publicly available SQL server on a network belonging to a medical institution that I was pen testing. I pointed Nessus at it and it just died... BR /Joakim
RE: Network scan tool/appliance horror stories
I can share with you several stories personnel (both IT or vendors), who have scanned Electric Utility environments with or without permission; and hence caused multiple failures - including electro-mechanical systems and related applications. Utilities typically utilize many industrial controllers - some of which many IT personnel have no knowledge, and some are not robust enough to weather the storm. 1. Know your environment. 2. Know your tools. 3. Communicate. -Original Message- From: Dan White [mailto:dwh...@olp.net] Sent: Monday, October 29, 2012 12:47 PM To: Pedersen, Sean Cc: nanog@nanog.org Subject: Re: Network scan tool/appliance horror stories On 10/29/12 12:10 -0700, Pedersen, Sean wrote: We're evaluating several tools at the moment, and one vendor wants to dynamically scan our network to pick up hosts - SNMP, port-scans, WMI, the works. I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok. http://www.tulsaworld.com/news/article.aspx?subjectid=334articleid=20121002_11_A1_CUTLIN325691 A layer 7 failure. Make sure all members of your organization are aware of your plans. -- Dan White
Re: Network scan tool/appliance horror stories
We have had ncircle scans unexpectedly crash alcatel-lucent omni-switches. On Mon, Oct 29, 2012 at 3:10 PM, Pedersen, Sean sean.peder...@usairways.com wrote: We're evaluating several tools at the moment, and one vendor wants to dynamically scan our network to pick up hosts - SNMP, port-scans, WMI, the works. I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok.
RE: Network scan tool/appliance horror stories
Network scan tools are a great way to verify what important protocols you left out of your control plane policing non-default policies. Had a scanner totally clog up our 6500 core router DHCP relay (ip helper) function once. Uggghhh, security people Chuck
Re: Network scan tool/appliance horror stories
On Mon, 29 Oct 2012, Pedersen, Sean wrote: We're evaluating several tools at the moment, and one vendor wants to dynamically scan our network to pick up hosts - SNMP, port-scans, WMI, the works. I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok. If you have any overloaded/under-powered network gear, such as stateful firewalls and routers that do lots of NAT, you might find them very quickly, depending on how aggressive the scanning tool is. There might also be devices out there that, while possibly lightly loaded, can reach some minimally documented resource threshold under a very aggressive scan, and subsequently tip over. Also, if you're doing IPv6, the performance metrics for many network devices can be a bit more of a moving target. jms
Re: Network scan tool/appliance horror stories
It all depends on what tools they are using and how you have your system setup. Both NMAP and Nessus can check system\service to see if common accounts have default or non password at all. This can cause these accounts to be locked out. There are other exploits that can cause systems\services to be DOS'd but these normally have to be enabled. Best to get a statement of works from them which should list all the tools including options they will be using. They also should be able to hand over a raw dump of ALL commands run during the testing. On 29 October 2012 19:25, Justin M. Streiner strei...@cluebyfour.orgwrote: On Mon, 29 Oct 2012, Pedersen, Sean wrote: We're evaluating several tools at the moment, and one vendor wants to dynamically scan our network to pick up hosts - SNMP, port-scans, WMI, the works. I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok. If you have any overloaded/under-powered network gear, such as stateful firewalls and routers that do lots of NAT, you might find them very quickly, depending on how aggressive the scanning tool is. There might also be devices out there that, while possibly lightly loaded, can reach some minimally documented resource threshold under a very aggressive scan, and subsequently tip over. Also, if you're doing IPv6, the performance metrics for many network devices can be a bit more of a moving target. jms -- ??? BaconZombie LOAD *,8,1
Re: Network scan tool/appliance horror stories
On 29/10/2012 19:25, Justin M. Streiner wrote: Also, if you're doing IPv6, the performance metrics for many network devices can be a bit more of a moving target. I'd almost be tempted to set up a few machines doing v6 only on the LAN, with some trivial to exploit telnet/SNMP access then invite them to scan the LAN and see if said machines are picked up. My experience of these things a year or two ago was that most of these companies thought everyone had an internal flat IPv4 network in RFC1918 space and that was that. YMMV of course. Paul. -- Paul Thornton
Re: Network scan tool/appliance horror stories
I heard a story in the past year of someone that had a system get scanned and it opened a ticket with their IT department for each time they scanned them. Eventually the IT department system crashed due to the excessive number of tickets being opened by their scanning tool. The network was properly exempted from the future scans after the system had to be recovered from backup. - Jared LOAD *,8,1 ^ yay
Re: Network scan tool/appliance horror stories
On 10/29/12 12:10 -0700, Pedersen, Sean wrote: We're evaluating several tools at the moment, and one vendor wants to dynamically scan our network to pick up hosts - SNMP, port-scans, WMI, the works. I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok. http://www.tulsaworld.com/news/article.aspx?subjectid=334articleid=20121002_11_A1_CUTLIN325691 A layer 7 failure. Make sure all members of your organization are aware of your plans. -- Dan White
RE: Network scan tool/appliance horror stories
During scans at various times in the past (and depending on throttling and settings of that scan) we've seen: 1) small remote site firewalls doing site to site vpns drop a small number of packets 2) locally installed remote control service popup a 'user has been disconnected' error on PCs when port scanned 3) some devices send alerts like 'Unauthorized attempt to gain access' when their SNMP ports are hit with non-standard community strings 4) logging on some devices that causes concern for the admin of that device (Is someone hacking my device?) 5) out of date/non-patched (yet critical) applications and/or web servers crashing/locking up (this occurred on specific nessus scans, not a generic port/snmp scan) 6) large stacks of 3750s (six or more members) have issues around CPU during certain SNMP commands (I want to say some sort of getbulk type of command) The first four were pretty minor although #3 could generate a lot of calls to the support center. #5 was a big deal due to the nature of the application. #6 was impactful because we dropped routing neighbors for about 10 seconds but this was a couple of years ago so may have been an old IOS bug. -Original Message- From: Pedersen, Sean [mailto:sean.peder...@usairways.com] Sent: Monday, October 29, 2012 12:11 PM To: nanog@nanog.org Subject: Network scan tool/appliance horror stories We're evaluating several tools at the moment, and one vendor wants to dynamically scan our network to pick up hosts - SNMP, port-scans, WMI, the works. I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok.
Re: Network scan tool/appliance horror stories
On Mon, Oct 29, 2012 at 2:10 PM, Pedersen, Sean sean.peder...@usairways.com wrote: I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok. A particular model of ShoreTel voice switches I used to administer (running VxWorks, IIRC) would reliably lock up hard when hit with nmap's OS/service detection on a particular port. Required pulling the plug to restore service. The truly odd thing was that it didn't seem like a resource exhaustion issue, it could be triggered with a single well-crafted probe or two. After several long nights of painful troubleshooting with their level III support, we came to the conclusion that if it hurts, you probably shouldn't do it, and mitigating ACLs were put in place. -n
Re: Network scan tool/appliance horror stories
On Oct 29, 2012, at 3:55 PM, Rutis, Cameron 6) large stacks of 3750s (six or more members) have issues around CPU during certain SNMP commands (I want to say some sort of getbulk type of command) The first four were pretty minor although #3 could generate a lot of calls to the support center. #5 was a big deal due to the nature of the application. #6 was impactful because we dropped routing neighbors for about 10 seconds but this was a couple of years ago so may have been an old IOS bug. Saw the same. All of our 3750 stacks (which are small) committed suicide during a trial of Foglight. We had discovery timings turned way down, but it still caused a reload on a mix of the last supposedly really stable releases of 12.x. Not confidence inspiring. TAC was useless and suggested a v15 upgrade despite no known fix. The proposed v15 upgrade sent our lab boxes into continuous reload unless you broke the stack and manually wiped each switch. Oh, and port 28 was invisible on each switch after upgrade, and Gi2/0/28 would throw a syntax error. Wait for new releases, lather, rinse, repeat. Total time to resolution in production was several man-weeks on our side, and a few months calendar time, all because the discovery scan revealed how great a software company Cisco has become.
Re: Network scan tool/appliance horror stories
On Mon, Oct 29, 2012 at 12:10:40PM -0700, Pedersen, Sean wrote: We're evaluating several tools at the moment, and one vendor wants to dynamically scan our network to pick up hosts - SNMP, port-scans, WMI, the works. I was curious if anyone had any particularly gruesome horror stories of scanning tools run amok. Check your netmask on the to-be-discovered network and what the rate of discovery is. I have seen internal systems attempt to scan and discover nodes in a /16 and promptly set off a flood of alarms on all PDUs (6 per rack) and plenty of other devices that thought they are being attacked. -andreas
