Re: Mitigating human error in the SP
On Thu, 04 Feb 2010 17:22:23 -0600 Larry Sheldon larryshel...@cox.net wrote: On 2/4/2010 5:13 PM, Larry Sheldon wrote: On 2/4/2010 3:30 PM, Scott Weeks wrote: A recent organizational change at my company has put someone in charge who is determined to make things perfect. We are a service provider, isn't a common occurrence, and the engineer in question has a pristine track record. This outage, of a high profile customer, triggered upper management to react by calling a meeting just days after. Put bluntly, we've been told Human errors are unacceptable, and they will be completely eliminated. One is too many. From experience... At one point this will become overwhelming. You'll wake up every morning dreading going to work instead of looking forward to it. Chain shot will be put in the 'blame cannon' and blasted regularly and at everyone. Update your resume and get everything in place just in case it gets to the point you can't take it anymore sooner than you expect. ;-) This is a golden opportunity. Prepare a pLan for building the lab necessary to pre-test EVERYTHING. Plan. Prepare a plan. Cost it out. Present the cost and the plan in a public forum or widely distributed memorandum (including as a minimum everybody that was at the meeting and everybody in the chain(s) of command between you and the edict giver. Problem is, when the inevitable human error does occur, the expensive lab then just looks like it was a huge waste of money, and that the networking people took advantage of the situation to build a play ground. They'll then likely be shown the door. The only way to completely eliminate human error is to eliminate the humans - from everything - hardware design, software design, deployment and maintenance. Actually, come to think of it, there is a way to eliminate human error. Staff netops with monkeys, and as long as management don't work out that they've now got monkey error, they'll be happy. When they cotton on to that, replace monkey error with goat error. Then sheep error. Then hamster error. etc. etc. By the time they run out of types of animal error, they'll realise how wonderful human error really is! -- Government big enough to supply everything you need is big enough to take everything you have. Remember: The Ark was built by amateurs, the Titanic by professionals. Requiescas in pace o email Ex turpi causa non oritur actio Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml
Re: Mitigating human error in the SP
On 2/6/2010 8:12 AM, Mark Smith wrote: On Thu, 04 Feb 2010 17:22:23 -0600 Larry Sheldonlarryshel...@cox.net wrote: Present the cost and the plan in a public forum or widely distributed memorandum (including as a minimum everybody that was at the meeting and everybody in the chain(s) of command between you and the edict giver. Problem is, when the inevitable human error does occur, the expensive lab then just looks like it was a huge waste of money, and that the networking people took advantage of the situation to build a play ground. They'll then likely be shown the door. I can't imagine wanting to work at a place like that anyway. The only way to completely eliminate human error is to eliminate the humans - from everything - hardware design, software design, deployment and maintenance. This may be a little heavy on the philosophy, but there has to be a human in there somewhere. And, will I don't have any statistics at all, I sense that machine failures probably exceed human failures in rate and severity. Certainly there are assists that make sense. And by the way, what difference does it make if you get fired because a machine replaced you and getting fired because somebody made a mistake? -- Government big enough to supply everything you need is big enough to take everything you have. Remember: The Ark was built by amateurs, the Titanic by professionals. Requiescas in pace o email Ex turpi causa non oritur actio Eppure si rinfresca ICBM Targeting Information: http://tinyurl.com/4sqczs http://tinyurl.com/7tp8ml
Re: google contact? why is google hosting/supporting/encouraging spammers?
Hello nanog, I was pointed to the recent thread about Google Groups spammers and thought I'd throw out a few comments. Keeping Google free of spam, and keeping the wider internet community on our side is important to us. By the time I looked at it, the account named by Jim was already shut down as part of a systematic anti-abuse sweep. We automatically identify and shut down many accounts every day and this account was no different. The nature of providing free email services at scale is that abuse is inevitable and the only way to tackle it is with automation, so hopefully you can understand that we may not always be as responsive to ad hoc manual reports as would be ideal. Google Groups spam specifically is something we started seriously tackling only recently. Doing it as well as GMail required a rewrite of the rather old groups backend. It's already improved dramatically in the past few months and it is continuing to get better rapidly as we close the remaining holes. On the question of caring, we do care about spam of all kinds, both inbound and outbound, and have a large anti abuse team dedicated to stopping spammers. This is a difficult problem - there is a dedicated underground economy of people who spend all day reverse engineering our systems so they can sell accounts and mail relay services to other, less capable individuals. But I think it's fair to say we're ahead of the game here - for example, Google is the only major webmail provider I know of that has aggressively deployed SMS verification. Thanks for the feedback. Google wants to be the best netizen possible, and we know getting spam from google IPs is frustrating - we're on it. /mike -- GMail Engineering Google Switzerland GmbH
Re: How polluted is 1/8?
Hi Jared, Merit would be happy to sink and collected this traffic. Perhaps even the entire /8 depending on the traffic level. Ideally we would want to do the entire /8. We have disk and bandwidth in place for our other research activities and this would fit in nicely. We could probably do a full pcap capture for a week or so and make it publicly available to the broader research community. -manish There was a call on the apnic list for someone to sink some of the traffic. I'd like to see someone capture the data and post pcaps/netflow analysis, and possibly just run a http server on that /24 so people can test if their network is broken. I've taken a peek at the traffic, and I don't think it's 100's of megs, but without a global view who knows. - Jared
Small Network Equipment Shipping boxes
Gurus, Where I work we ship our own gear. That being said, we've run out of 1U shipping boxes for cisco gear. I've searched the list archives, uline, as well as a few other shipping material websites and cannot find an acceptable shipping box with foam inserts. I would like a corrugated cardboard box with foam supports for shipping 1U cisco gear (3550s, 4948s, etc..) I was wondering if anyone has sourced a good shipping box? I apologize if this topic has been brought up before and my search didn't find the thread. Thanks, -a
Small Network Equipment Shipping boxes
Gurus, Where I work we ship our own gear. That being said, we've run out of 1U shipping boxes for cisco gear. I've searched the list archives, uline, as well as a few other shipping material websites and cannot find an acceptable shipping box with foam inserts. I would like a corrugated cardboard box with foam supports for shipping 1U cisco gear (3550s, 4948s, etc..) I was wondering if anyone has sourced a good shipping box? I apologize if this topic has been brought up before and my search didn't find the thread. Thanks, -a
Re: fiber plant management?
On Fri, Feb 5, 2010 at 5:38 AM, Martin Hannigan mar...@theicelandguy.com wrote: Honestly? A spreadsheet will do it. Let me translate that into plain English for you. He said that a barebones database will do it and he happens to use a simple one that he built himself. Clearly there are scaling issues with his technology choice, but other than that, his solution is probably the most common one you will find out there. Most ISPs use a straightforward database (usually a full-blown relational one) and build their own application around it. A company that I worked at 10 years ago built a system around a CRM tool called Vantive. At first glance, CRM tools are not the kind of thing you would normally choose for tracking circuits and physical plant. But since this CRM tool allowed customization with VBA, and since VBA allowed access to full-blown relational databases, we ended up with a very nice tool that not only tracked circuits, fibres, etc. but also allowed us to link it all to customers so if there was a specific fibre route cut, we could immediately get a list of contact names and numbers for all customers whose services were affected. My advice is to find out what in-house database skills you have available, and get them to build a simple records system using Oracle, PostreSQL, DB2, SQL Server or similar, and to make sure that they understand that the intent is to evolve it step by step into a full-blown system. This last is important so that they think about how to make a long-term design and make fewer mistakes that need to be fixed later. --Michael Dillon
Re: Regular Expression for IPv6 addresses
Folks, Thanks for all the comments on the IPv6 regex... - Jeroen Massar jer...@unfix.org wrote: The only proper way of testing if an address is a valid IPv6 address is to feed it to getaddrinfo() and then use it through that API. Good point. One of the reasons to do this was for environments where getaddrinfo() might not be availble. (For example, in a Javascript - see the page on the InterMapper site: http://intermapper.com/ipv6validator ) Of course, there should be only limited places where a user can enter or see IP addresses in the first place. There is this great thing called DNS which is what most people should be using. Another good point. But look at Seiichi's note (below)... - Seiichi Kawamura kawamu...@mesh.ad.jp also wrote: This might be of some interest to you. http://tools.ietf.org/html/draft-ietf-6man-text-addr-representation-04 We believe this correctly recognizes all the cases specified by RFC4291. But it's a great idea to update the Javascript page above to reformat to this recommendation. - Carsten Bormann c...@tzi.org I was looking at this regexp for my Ruby course as a great example of how not to use Regexps. But thank you for this textbook example! And, of course, an error did creep in. You're quite welcome! :-) But seriously... Do you have an example of an address that causes the RE to fail? If you really need an RE, the right approach here is to write a small program to generate the RE. Absolutely. The Perl program cited includes code much like your example. Rich Brownrichard.e.br...@dartware.com Dartware, LLC http://www.dartware.com 66-7 Benning Street Telephone: 603-643-9600 West Lebanon, NH 03784-3407 Fax: 603-643-2289 PS Dartware is sponsoring a table displaying our InterMapper network monitoring software at the NANOG48 Beer 'n Gear. Please come by to introduce yourself and take a look...
Re: Insecure Cable networks ?
Yes this is a huge security hole. Management networks should always be restricted to some extent and the fact that default passwords allow you into VoIP gateways provides an avenue for call fraud. At a very minimum the devices should restrict which addresses can talk to them (ie. management servers in the MSO) and passwords should be non-default. If I were them or involved in the operation of their network I should start with an audit. Obviously I didn't change or tried to change anything, the few cases I tried to gain access to some randomly selected devices/locations were just to confirm that imho there is a big exposure here. For example, I found devices such as an integrated modem and wireless router where if I wanted I would have been able to enable WiFi guest access or change the existing WiFi configuration such as SSID, keys, etc. Some modems don't seem to provide access via port 80, I didn't scan for any other potential ports or back doors (such as SNMP ports,etc), they simple show the message Access to this web page is currently unavailable.. The most popular/used device, just for the number of times I've got the same interface for the few (less than a 100) IP I tried seems to be the Ambit modem, the main page shows sort of general modem information, something like: Cable Modem Information Cable Modem : DOCSIS 1.0/1.1/2.0 Compliant MAC Address : 00:1F:XX:XX:XX:XX Serial Number : REMOVED Boot Code Version : 2.1.6d Software Version : 2.105.1008 Hardware Version : 1.20 CA Key : Installed Gaining access to the modem is quite simple, on the left there is a frame that has a Login link and says Factory default username/password isuser , which actually worked on all the ones I found and tried, on the right hand corner there are two links one that says Modem and other that says Tools, if I click on Tools I see at least two options, one that takes me to a form page to change the password, and the other one to change the Frequency Scanning Plan. Again I didn't try to change anything to confirm that it is actually possible but I've the hunch that it is possible. Another case could be integrated modem/router with VoIP features such as Motorola's SurfBoard, the standard management interface without even login in to the thing provides plenty of information, don't know how useful but, there is a link that says Advanced which requires you to enter a password, don't waste much of your brain, the password is simply motorola, with that you get access to more information including MGCP Logs, I didn't analyze the logs in detail but it didn't take much effort to find out that a guy was being called by a collection department of Wells Fargo Bank from an Oregon (503) number. In another case I saw a log entry that could be interpreted as a dialed out number. In summary, I don't believe that any customer should have access to any other customer device in such a way that you can alter the provisioning of a service or snoop and see how the service is being used, this raises not only security but privacy concerns. I didn't use any scripts or tried any heavy tools or hacking, mine is a very minuscule sample of what seems to be a widespread bad practice or mismanaged network configuration. Ryan thanks for your message, I checked and saw that you work for TWC in the Albany area, but no offense, I've no problems to share more details and cooperate, only if being contacted by a grownup honcho in charge of networking/security. I promise, I won't break anything ... Cheers Jorge
Re: Regular Expression for IPv6 addresses
On Fri, Feb 5, 2010 at 12:15 AM, sth...@nethelp.no wrote: And now for the trick question. Is :::077.077.077.077 a legal mapped address and if it, does it match 077.077.077.077? Wasn't there an internet draft on that subject, recently? http://tools.ietf.org/html/draft-ietf-6man-text-addr-representation-04 077.077.077.077 is equivalent to 77.77.77.77 if valid at all RFC 4038 is very clear that the text representation of a mapped IPv4 address is Base 10. http://tools.ietf.org/html/rfc4038#section-5.1 This is a bit like asking if :::10.1.2 is a valid IP address though. And is it the same as the ip address 10.1.2 ? (Which of course expands to 10.1.0.2, on common implementations of inet_pton, inet_aton, and getaddrinfo) Or :::0xA010002 I would say these are perfectly valid _shorthands_ and abbreviations for entering an IP address, which may be provided by some systems, but that they are non-canonical text representations for displaying publishing or sharing IP addresses. -- -J
Re: Regular Expression for IPv6 addresses
In message 6eb799ab1002061452s51f9cf61p303d36130291...@mail.gmail.com, James Hess writes: On Fri, Feb 5, 2010 at 12:15 AM, sth...@nethelp.no wrote: And now for the trick question. =A0Is :::077.077.077.077 a legal mapped address and if it, does it match 077.077.077.077? Wasn't there an internet draft on that subject, recently? http://tools.ietf.org/html/draft-ietf-6man-text-addr-representation-04 077.077.077.077 is equivalent to 77.77.77.77 if valid at all RFC 4038 is very clear that the text representation of a mapped IPv4 address is Base 10. http://tools.ietf.org/html/rfc4038#section-5.1 But 077.077.077.077 is octal dotted quad. Decimal dotted quad does *not* have leading zeros. The point of allowing for dotted quad is to allow for easy mapping between IPv4 representation and IPv6 with encoded IPv4 representations. Accepting a octal representation as decimal is a bad thing and leads to none obvious failures. % ping 077.077.077.077 PING 077.077.077.077 (63.63.63.63): 56 data bytes ^C --- 077.077.077.077 ping statistics --- 4 packets transmitted, 0 packets received, 100% packet loss % ping :::077.077.077.077 would not get to same box if my ping accepted that as a address literal which luckily it doesn't. This is a bit like asking if :::10.1.2 is a valid IP address though. Except it clearly isn't as there are not 4 components. And is it the same as the ip address 10.1.2 ? (Which of course expands to 10.1.0.2, on common implementations of inet_pton, inet_aton, and getaddrinfo) Or :::0xA010002 inet_pton() did not accept 10.1.2 when it was originally written. This was a *deliberate* decision. Some vendors have changed it to accept it but they are wrong. I can say that because I was involved in making that decision. I would say these are perfectly valid _shorthands_ and abbreviations for entering an IP address, which may be provided by some systems, but that they are non-canonical text representations for displaying publishing or sharing IP addresses. -- -J -- Mark Andrews, ISC 1 Seymour St., Dundas Valley, NSW 2117, Australia PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org
Re: How common are wide open SIP gateways?
On Feb 5, 2010, at 1:27 PM, Scott Howard wrote: On Fri, Feb 5, 2010 at 9:45 AM, David Birnbaum dav...@pins.net wrote: We have noticed a lot of issues with Asterisk 1.2 and some 1.4 rollouts. FreePBX had some truck-sized holes in it. Most/all of the big issues that existed in previous version of Asterisk/FreePBX have been resolved in later releases. The majority of the stolen SIP cases I've heard of have come down to brute forcing of often very insecure passwords - quite often stupid insecure passwords like the same as the username. And of course the username itself is normally the extension, which makes is relatively easy to guess (if 100 doesn't exist, then 200 or 1000 probably does, etc). Then there's the issue of unencrypted/unsecured phone provisioning files, complete with SIP usernames/passwords, hosted on internet webservers - often with the only security being your ability to guess the MAC address... On our relatively small client base, we are seing SIP probing on more or less a non-stop basis, and some of our customers have been hacked over the Presuming you're running Asterisk, fail2ban can help. The only real issue I've had with it is that many softphones will repeated try to register if you get the password wrong, so a user entering their username/password even only once will get them blocked for X minutes. Scott I'll second Scott's comments, and add a few. SIP servers aren't much good unless they're wide open, if you're serving to a large number of diverse users whose networks you do not control with a VPN or a customized client. This invites probing to determine identity choice weakness. It seems that new SIP servers are discovered within about 5 days of being put up without filtering, at least looking at my logs. The most commonly-available toolset for such attacks seems to have moved SIP attacks into script-kiddie land about a year and a half ago. The tool has three functions: scan for SIP servers (UDP 5060), identify SIP identities via login failure or other error message information leakage, and lastly guess passwords in brute-force manners on those identified SIP extensions. The attacks seem to be geographically diverse - I've seen originations both in North America as well as non-NA origins, though the ultimate origin is often a mystery due to compromised servers being used for probe sweeps. The attacks also seem to have a variety of purposes. The four that I've most commonly seen are: 1) Experimenting, joy riders. 2) Attacking to obtain free international long distance 3) Attacking to obtain access into the PBX network with fraudulent identity to perform fraudulent internal activity (This is Bob from accounting...) 4) Attacking to create large numbers of domestic calls for phishing scams (This is your bank. Please enter your credit card number now.) Of these, #4 seems to be the only one that gets significant attention of LEA resources. I wrote some notes for security basics on this a while back as it pertains to Asterisk in particular, but the problem remains with some very old installations that accept inbound calls into the default Asterisk context (which is not a bug, but really a configuration error) or it crops up anew with administrators who do not adequately create sufficiently random SIP identities and passwords. Asterisk is fairly robust against such attacks, but often the flexibility of a complex system allows administrators to inadvertently expose themselves in ways they wouldn't ordinarily think about. More here: http://blogs.digium.com/2009/03/28/sip-security/ As far as network impacts: some of these probes are fairly significant in bandwidth consumption (3-5 mbps, from what I've seen) and may cause problems with whatever your SIP authentication method is due to the volume of requests. A distributed attack at higher volumes has less chance of success because most SIP platforms do not have the ability to respond to high volumes of requests anyway. Fail2Ban can be implemented on most SIP platforms at the application level, and works quite well against most probe methods. I can't even comment on the issue of unencrypted/unauthenticated provisioning servers. If you're provisioning in an unauthenticated way across the big internet, then... well, you takes yer chances. Lastly: SIP is very flexible in handling alternate ports for communications in URIs or other pointers, though I have never seen a SIP server using anything other than 5060/5061. Perhaps related, I've never seen a suspicious system probing on 5060 looking at any other ports. Maybe changing ports would siipmly solve problems pretty quickly for people seeing attacks who have some ability to influence/ configure their end devices or trunking peers. (At least, for a little while. Remember: when chased by a bear, you just need to be faster than the guy behind you.) JT
