Re: qwest backbone
On Mon, May 21, 2007 at 12:27:59PM -0700, Philip Lavine wrote: Any issues on the qwest backbone http://www.internetpulse.net/ shows problems with Qwest and with Verizon/MCI. I can confirm MCI problems, but have no details other than yes something is seriously broken. Possible backbone fibre cut? -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: outage
On Fri, Mar 23, 2007 at 05:44:03PM -0400, Koch, Christian wrote: I peer with google at a bunch of spots so I can't see anything, and everything looks ok.. Looks like global crossing issue route-server.phx1trace 64.233.175.98 Type escape sequence to abort. Tracing the route to 64.233.175.98 1 ge4-1-0-226-1000M.ar4.PHX1.gblx.net (67.17.64.89) 0 msec 0 msec 0 msec 2 so3-0-0-2488M.ar3.PAO2.gblx.net (67.17.94.97) 20 msec 16 msec 20 msec 3 * * * 4 * * * 5 * * * 6 * * * Another post on NANOG, shortly before your reply: From: Nathan March [EMAIL PROTECTED] To: nanog@merit.edu Date: Fri, 23 Mar 2007 14:43:08 -0700 Subject: Extended GBLX outage in Atlanta / Miami? Anyone know what is going on with global crossing in atlanta? Connectivity to one of our sites has been down for over 2 hours now with no sign of routing around or fixing the problem: 835 ms38 ms40 ms p4-0.core01.sjc01.atlas.cogentco.com [66.28.4.94] 988 ms78 ms87 ms p14-0.core01.iah01.atlas.cogentco.com [66.28.4.237] 10 103 ms 122 ms 116 ms p3-0.core01.atl01.atlas.cogentco.com [154.54.5.90] 11 * 275 ms 149 ms ge4-1-0-390-1000M.ar4.ATL1.gblx.net [64.208.110.97] 12 *** Request timed out. 13 *** Request timed out. - Nathan -- | Jeremy Chadwickjdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: NOC Personel Question (Possibly OT)
On Thu, Mar 15, 2007 at 09:49:36AM -0400, Donald Stahl wrote: We call our level 1 NOC people Operators. We reserve Network Analyst for the level 2 people who also do some small amount of scripting and other more advanced troubleshooting. Network Analyst makes me think of Stock Analysts, though. The problem is that they aren't very good at telling me what kind of returns I can expect on my equipment and what the future holds for the network :) Has anyone thought to clearly define these titles somewhere so that everyone can standardize on them? Because each NOC is different, depending upon what hiring practises are deployed by the company populating aforementioned NOC. Here's two NOCs for you (yes, both are real): 1) Expected to have above-average UNIX skills, above-average exposure to DNS (understanding SOAs, must have familiarity with dig, etc.), familiarity with HTTP (manual fetches/form queries, etc.), SSH and related aspects (tunnels, keys, etc.), decent networking troubleshooting skills (more than just exposure to ping, exposure to BGP is good, knowledge of the OSI layer is highly respected, etc.). Of course the standard crap also applies: extensive ticket work, answering phone calls + dealing with clients, escalation procedures, 24x7 operations (e.g. graveyard guys getting little sleep), etc.. NOC employees have root and enable on systems and networking devices, and are encouraged to use them to track -- and solve -- issues if they feel comfortable/know how (otherwise escalate/check with someone else). Hiring practises also require personable people (read: no ego, are willing to teach others), and do not hire people who tote themselves as superior or too proud to work in a NOC. 2) Anyone with the least bit of any IT experience at all (I know how to install Windows Server and plug in PCI cards, is that OK?) is hired. Don't know anything about UNIX? No problem, here's some documentation you can read that'll teach you. Don't know how TCP/IP works? That's fine, it's not part of the job, just escalate according to this procedure. If the individual has extensive experience(s), great, hire them. If they have very few skills but have more than none, hire them. NOC employees do not have root/enable; all issues are escalated. Management adheres to Rules must be followed Do not try to be different We like robots ideals. NOC #2 is what most people think of, and that's understandable, because there's a lot of NOCs which are completely chaotic and borderline useless (read: getting in the way of engineers solving problems more so than helping them solve the problem). My point is, people working in NOC #1 would be generally disappointed to have to put NOC Analyst on their business card when most of their time was spent doing SA or NA-related things. NOC #2 individuals probably won't care (the skilled ones might care, but might not make a big deal about it, because maybe they're just happy that they have a job at all.) So, my recommendation to the OP would be to pick titles appropriate to the individuals' skill set. The term NOCster or NOCling or other such terms are *not* terms of endearment -- they're borderline insulting, unless your NOC is like #2, which in that case you might as well just put down Emotionless Robot, because that's eventually how people become in those environments. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: [funsec] Not so fast, broadband providers tell big users (fwd)
On Tue, Mar 13, 2007 at 03:52:57PM +, Peter Corlett wrote: On Tue, Mar 13, 2007 at 08:27:04AM -0700, Roland Dobbins wrote: On Mar 13, 2007, at 8:17 AM, Chris L. Morrow wrote: [...] what business drivers are there to put more bits on the wire to the end user? BitTorrent. The download speed is however limited by the upload speed of the peers, which acts as its own rate-limit given that the bandwidth on broadband connections is somewhat asymmetric. Ideally that's how it's supposed to work, but isn't how it works as of present-day. Speaking solely about the BitTorrent protocol, upstream does not affect downstream speed. In fact, there's a BitTorrent client out there which specifically *does not* share any of the data being downloaded (thus acting as a pure leeching client): http://dcg.ethz.ch/projects/bitthief/ -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: what the heck do i do now?
On Thu, Feb 01, 2007 at 08:45:52PM +, Paul Vixie wrote:
2) maps.vix.com.604800 IN NS u1.vix.com.
maps.vix.com. 604800 IN NS u2.vix.com.
maps.vix.com. 604800 IN NS u3.vix.com.
... [as many as you like]
u1.vix.com. 604800 IN A 192.0.2.1
u2.vix.com. 604800 IN A 192.0.2.2
u3.vix.com. 604800 IN A 192.0.2.3
... [as many as you like]
i hadn't thought of that. i'll think seriously about it, thanks.
The caveats to this are:
1) DNS servers which are not configured to blackhole IANA-reserved
network blocks (read: the majority) will blindly try to reach
192.0.0.0/17 and friends.
2) Some people (like myself) have ipfw/pf rules which block and
log outbound packets to reserved blocks. We log these because
usually it's the sign of broken software or possibly some weird
IP routing (read: OS IP stack) problem. In the case of ipfw (I
haven't tested pf), the block gets reported to underlying layers
as EACCES, which can be incredibly confusing for admins.
Of course, this isn't an issue as long as every service on the
face of the planet has some form of blackholing. In the case of
someone pointing an MX record to something that has an A record
of 127.255.255.255, for example, and lacks the blackholing
capability, the service (postfix, sendmail, whatever) will
blindly try to communicate with that destination, thus spewing
out nasties on the console or in logfiles. Example:
00200 1234 93513 deny ip from { 127.0.0.0/8 or 192.168.0.0/16 or
172.16.0.0/12 or 10.0.0.0/8 } to any in recv fxp0
002013 180 deny log ip from any to { 127.0.0.0/8 or dst-ip
192.168.0.0/16 or dst-ip 172.16.0.0/12 or dst-ip 10.0.0.0/8 } out xmit fxp0
As you can see, there's still many routers which don't have
outbound blocks in place. Since I do not trust my ISP to do
this for me, and I believe in taking the initiative, I block
them locally on the system. And as you can see (thus case
in point), I don't have a very up-to-date list of IANA-reserved
blocks in my list.
Until I recently added the blackholing rules in BIND, I kept
seeing the resolver try to contact 127.0.0.0/8 hosts, or
10.0.0.0/8 hosts. People who had things like localhost as
auth. NS entries -- hey, isn't this what you did?! ;-)
My vote is to simply remove the NS and A records for maps.vix.com
and let people utilise search engines and mailing list archives to
figure out where to go (mail-abuse).
--
| Jeremy Chadwick jdc at parodius.com |
| Parodius Networkinghttp://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: what the heck do i do now?
On Mon, Feb 05, 2007 at 10:13:08PM -0500, Jon Lewis wrote: On Mon, 5 Feb 2007, Jeremy Chadwick wrote: 1) DNS servers which are not configured to blackhole IANA-reserved network blocks (read: the majority) will blindly try to reach 192.0.0.0/17 and friends. 192.0.2.0/24 - This block is assigned as TEST-NET for use in documentation and example code. It is often used in conjunction with domain names example.com or example.net in vendor and protocol documentation. Addresses within this block should not appear on the public Internet. I was going purely off of what ARIN reports: Internet Assigned Numbers Authority RESERVED-192 (NET-192-0-0-0-1) 192.0.0.0 - 192.0.127.255 Internet Assigned Numbers Authority IANA (NET-192-0-2-0-1) 192.0.2.0 - 192.0.2.255 If there is something magical about 192.0.2.0/24, then I'd love to know what it is (please do educate me!). But from my perspective, it just looks like another IANA-reserved netblock. That /24 doesn't show up in BGP unless something is broken or you have a cymru bogon feed. Either way, worst case is you're default routing to an ISP/NSP and the packets get a few hops before someone drops them as unroutable. Right, so the mentality here is that someone will eventually filter the packets or they'll be dropped due to a null route BGP rule. This I understand, but IMHO it's better to filter such packets before they ever reach someone else's networking gear. (Sorry if I'm not phrasing this as eloquently as possible.) In my case, I simply purchase co-lo space from providers and rely on their routing configurations, hoping they're doing things properly. But as one can see from the ipfw stats I pasted, some aren't. Understand where I'm coming from? 2) Some people (like myself) have ipfw/pf rules which block and log outbound packets to reserved blocks. We log these because usually it's the sign of broken software or possibly some weird IP routing (read: OS IP stack) problem. In the case of ipfw (I haven't tested pf), the block gets reported to underlying layers as EACCES, which can be incredibly confusing for admins. If it means they get noticed, mission accomplished. That's exactly what Paul wants. In that case, it's a win-win situation. My vote is to simply remove the NS and A records for maps.vix.com and let people utilise search engines and mailing list archives to figure out where to go (mail-abuse). The vix.com NS's will get slammed with all the DNSBL queries then. The suggestions I made at least get some of the queriers (assuming they have properly functioning caches) off your back for a while. Hmm, yes, you're absolutely correct. But I'm curious why you picked 192.0.2.0/24 rather than some other reserved block? (I've also sent a copy of this discussion to an associate of mine at Nominum, who's now wondering the same thing I am...) I've found this thread immensely educational so far! -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Google wants to be your Internet
On Sat, Jan 20, 2007 at 05:55:49PM -0600, Gadi Evron wrote: Some examples may be: -. Working on establishing new standards and topologies to enable both vendors and providers to adopt them. Keep this point in mind while reading my below comment. For now, the P2P folks who are not in most cases eveel Internet Pirates are mostly allied, whether in name or in practice with illegal activities. The technology isn't illegal and can be quite good for all of us to save quite a bit of bandwidth rather than waste it (quite a bit of redudndancy there!). A paper put together by the authors of a download-only free riding BitTorrent client, called BitThief. The paper is worth reading: http://dcg.ethz.ch/publications/hotnets06.pdf http://dcg.ethz.ch/projects/bitthief/ (client is here) The part that saddens me the most about this project isn't the complete disregard for the give back what you take moral (though that part does sadden me personally) , but what this is going to do to the protocol and the clients. Chances are that other torrent client authors are going to see the project as major defiance and start implementing things like filtering what client can connect to who based on the client name/ID string (ex. uTorrent, Azureus, MainLine), which as we all know, is going to last maybe 3 weeks. This in turn will solicit the BitThief authors implementing a feature that allows the client to either spoof its client name or use randomly- generated ones. Rinse lather repeat, until everyone is fighting rather than cooperating. Will the BT protocol be reformed to address this? 50/50 chance. So, instead of fighting it and seeing it left in the hands of the pirates and the privacy folks trying to bypass the Firewall of [insert evil regime here], why not utilize it? I think Adrian Chadd's mail addresses this indirectly: it's not being utilised because of the bandwidth requirements. ISPs probably don't have an interest in BT caching because of 1) cost of ownership, 2) legal concerns (if an ISP cached a publicly distributed copy of some pirated software, who's then responsible?), and most of all, 3) it's easier to buy a content-sniffing device that rate-limits, or just start hard-limiting users who use too much bandwidth (a phrase ISPs use as justification for shutting off customers' connections, but never provide numbers of just what's too much). The result of these items already been shown: BT encryption. I personally know of 3 individuals who have their client to use en- cryption only (disabling non-encrypted connection support). For security? Nope -- solely because their ISP uses a rate limiting device. Bram Cohen's official statement is that using encryption to get around this is silly because not many ISPs are implementing such devices (maybe not *right now*, Bram, but in the next year or two, they likely will): http://bramcohen.livejournal.com/29886.html ISPs will go with implementing the above device *before* implementing something like a BT caching box. Adrian probably knows this too, and chances are it's probably because of the 3 above items I listed. So my question is this: how exactly do we (as administrators of systems or networks) get companies, managers, and even other administrators, to think differently about solving this? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: FW: [cacti-announce] Cacti 0.8.6j Released (fwd)
On Thu, Jan 18, 2007 at 11:40:06AM -0600, Gadi Evron wrote: Many of us run cacti. FYI. Thanks for posting this, even though it's slightly OT. Not to start an opinion war, but those who do run Cacti should really consider removing this software from their boxes permanently. http://secunia.com/advisories/23528/ For those who don't have the time/care enough to go look at the Secunia report, I'll summarise it: 1) cmd.php and copy_cacti_user.php both blindly pass arguments passed in the URL to system(). This, IMHO, is reason enough to not run this software. 2) cmd.php and copy_cacti_user.php both blindly pass arguments passed in the URL to whatever SQL back-end is used (MySQL most commonly); no escaping or sanitising is done. Otherwise known as an SQL injection flaw. There are other flaws mentioned, but they're simply subsets of the above two. Also, register_argc_argv is enabled (rightfully so) by default in PHP, so don't let that decrease the severity of this atrocity. (I can forgive SQL injections, but cannot blindly calling system()). I'd been considering (off and on for about a year) using Cacti for statistics gathering, and now I'm glad I didn't. This kind-of flaw reflects directly on the programming ethics and of the authors behind this software. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: comcast routing issue question
On Thu, Nov 30, 2006 at 12:06:29AM -0500, Jim Popovitch wrote: $ mtr 69.61.40.34 HOST: blueLoss% Snt Last Avg Best Wrst 1. 192.168.3.1 0.0% 11.1 1.1 1.1 1.1 2. 73.62.48.10.0% 19.9 9.9 9.9 9.9 3. 68.86.108.25 0.0% 19.3 9.3 9.3 9.3 4. 68.86.106.54 0.0% 19.6 9.6 9.6 9.6 5. 68.86.106.9 0.0% 19.0 9.0 9.0 9.0 6. 68.86.90.121 0.0% 1 18.2 18.2 18.2 18.2 7. 68.86.84.70 0.0% 1 23.9 23.9 23.9 23.9 8. ??? 100.0 10.0 0.0 0.0 0.0 Taking the 69.61.40.33/28 subnet a bit further, .36 drops at 68.86.84.70 but .37 - .39 make it. .40 drops at 68.86.84.70, but .41 makes it. You're not the only one who noticed this. http://www.dslreports.com/forum/remark,17368208 -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: OT: How to stop UltraDNS sales people calling
On Tue, Nov 28, 2006 at 05:56:19PM -0600, Gadi Evron wrote: Okay, this was fun and I am all for OT fun. But can we please stop putting down a part of our community? Especially one which contributes to NANOG so much? We all have sale trolls to live with. I both agree and disagree. I agree that the put-downs are a bit excessive (I laughed more than once :-) ), but I disagree with the sale trolls comment. UltraDNS's sales staff *does not* have to behave like this. Guerrilla marketing is not an effective way of selling a product, nor does it help establish an initial good impression of a vendor, regardless of who they are, or what their track record is. In fact, at least when it comes to me, the less aggressive a salesman is, the better chance he/she has of selling me whatever it is they're selling. (In general, they're best off not contacting me in any way, instead letting me come to them if I spawn interest in their services.) Based purely on the OP's description, it sounds as if UltraDNS is bordering on unsolicited telemarketing. This should really be reported to both someone within UltraDNS's mid-tier or upper management; their sales tactic reflects very badly on the company as a whole (this entire NANOG thread is proof of that; chances are many of us now have a lesser opinion of UltraDNS than we did prior to the initial post). If that doesn't work, it's time to talk to the FCC, who may do nothing. But they would be more than willing to tell you if there have been previous complaints about UltraDNS's solicitations. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: IP adresss management verification
On Mon, Nov 13, 2006 at 09:20:38AM -0800, chuck goolsbee wrote: It pisses me off to no end when a sales guy comes to me with a request from a customer for a /20 for a half-rack of web servers. The justification ALWAYS comes down to this inane search engine optimization pipe dream. =\ Interesting. Most of the time I've seen customers ask for a /24 or larger blocks is solely for IRC vanity hosts. Is anyone keeping statistics for this? If not, they should. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: register.com down sev0?
On Sat, Oct 28, 2006 at 12:39:31AM -0500, Chris Owen wrote: The spam I got was directly from register.com. It came with a register.com return email address, pointed to a register.com web site and came from an IP address the resolved to *.register.com (I will admit I didn't confirm the netblock belonged to them). I've never done any business with them and the spam was for a domain name renewal for a domain registered elsewhere. In other words, it was a classic whois scrapped spam. Some clarification: the information is probably not being scraped via WHOIS. You're not allowed to scrape via WHOIS. Deceptive companies who want to get around this simply buy the WHOIS records (I should be more precise: the data that would appear in a WHOIS lookup) from the registrar directly. I can point you to an Email thread discussing this find, which includes couple statements from OpenSRS's Product Manager (who in a roundabout way admitted that anyone can buy their WHOIS database), if you'd like. This doesn't explain the spam, but it I really do not see any purpose to buying a registrar's copy of customer WHOIS records other than for mass-marketing. This is bad business in general. As I've previously said, this isn't like its some sort of borderline case where someone in one part of the company is doing something that someone else doesn't know about. These guys are pretty hard core. I'd say I get 20-30 emails a year from them for various domain names I'm a contact on. I've also received USPS spam which is another story but no less unethical since they are all these BS renewal type letters. They might not be Domain Registry of America but they are hardly innocent. I've mentioned this on NANOG before. See the thread about why I refuse to put legitimate contact information (Email contact information is always valid; just not the address or phone number) in our domain WHOIS records. The DROA is half of the reason; the other half is what I described above. The entire situation is depressing, solely because ICANN is doing absolutely nothing to try and stop this sort-of behaviour (both what the DROA does, and registrars selling their customers' WHOIS records to whoever bids the most for it). -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Extreme Slowness
On Thu, Oct 26, 2006 at 06:01:43PM -0400, Elijah Savage wrote: For FYI :) I realize that ICMP is not the best way to test and it is not a true indication of slowness or the presence of a problem. Which begs the same question I've asked in the recent past: then what *is* a good diagnostic tool? If ICMP is not the best way to test, then what is? What other globally-implemented layer 3 or below protocols do we have available for troubleshooting? Sure, UDP-based traceroute still relies on ICMP TTL exceeded responses to work. I've no idea what TCP traceroute relies on, as I haven't looked at it. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: GBLX issues
On Thu, Oct 19, 2006 at 02:40:01PM -0400, J. Oquendo wrote: Anyone else seeing issues with GBLX, DC area? Yes. There appears to be a fibre cut of some kind either around Virginia or Washington DC. [EMAIL PROTECTED] may be a better place to discuss. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Refusing Pings on Core Routers??? A new trend?
On Thu, Oct 19, 2006 at 07:18:01PM -0400, Deepak Jain wrote: 1 NOC (that will remain nameless even though they should really be shamed) said the following in response to the question -- when we were trying to diagnose +50ms jumps in their latency within a single POP. Q: As part of this, can you tell me why your router is prohibiting packets being sent to our interface? A: The reason you cannot hit your interface is it is blocked for security reasons. I've heard this response before, albeit not from the company you're referring to. The most common response -- which is at this point a template response -- I hear is Well, you can't rely on traceroute because of ICMP prioritisation. When you start to explain how traceroute actually works (both ICMP-based and UDP-based (which still relies on ICMP responses, of course!)), and that ICMP prio should only affect the IP of which the router listens on (and not hops beyond or at the dest), most NOCs fire back with another template of their choice (We're not aware of any issues, No that's incorrect, I'll check with engineers, or the ever-so-amusing traceroute and ping aren't reliable, you need to use a different method of testing) -- but the most common is: can you send us traceroutes of what you're seeing? But! You just said. argh!! I happen to work in a NOC, and I have never -- nor will I ever -- spout off that template response. When a client or customer calls about something, I give them the benefit of the doubt. If it turns out they're wrong later, they at least (hopefully) learned something. I just happen to believe in getting things done, rather than arguing against doing investigative work. When an issue occurs, look at it quickly, not 24-48 hours later. I am absolutely fine with ICMP being prioritised last, but those scenarios induce more questions; so ICMP is prio'd last, which would mean the router is busy processing other packets, which could mean your router is over-utilised either CPU-wise or iface-wise since we're seeing 250ms at your hop and beyond. 48 hours later, a network technician looks at the router and either finds absolutely nothing (It must've gone away on it's own) or finds something conclusive (but only when the issue re-occurs, is still occurring, or if they keep historic data). Did I miss the conspiracy?? I know my membership dues are all paid up. If this has been going on a while, I apologize I guess I've just noticed the trend in our shift reports. Yes, this has been going on for awhile. Well, not ICMP_UNREACH_NET (from your example) but general ICMP prioritisation or the explicit dropping of either ICMP_TIMXCEED (traceroute) or ICMP_ECHOREPLY (ping). A real-life example, from my own (residential) ISP. Try to imagine reporting an issue at hop 6 to a technician (who will always insist the problem is somewhere prior). Here's an example of a working network (no sarcasm; I'm serious!): 1. 192.168.1.1 0.0%3030 0.5 0.5 0.5 0.6 2. ??? 100.030 0 0.0 0.0 0.0 0.0 3. 68.87.198.129 0.0%3030 8.5 9.9 7.5 21.1 4. 68.87.192.34 30.0%3021 9.2 11.9 9.2 20.5 5. 68.87.226.13466.7%3010 10.9 12.9 10.2 25.9 6. 12.116.188.13 0.0%3030 10.6 12.6 10.1 25.0 7. 12.123.12.126 0.0%3030 12.9 12.3 10.2 15.3 And an example of when things are broken: 1. 192.168.1.1 0.0%3030 0.5 0.5 0.4 0.6 2. ??? 100.030 0 0.0 0.0 0.0 0.0 3. 68.87.198.129 0.0%3030 12.7 12.7 8.1 28.4 4. 68.87.192.34 20.0%3024 13.4 11.5 9.5 14.5 5. 68.87.226.13496.7%30 1 12.6 12.6 12.6 12.6 6. 12.116.188.1350.0%3015 15.1 11.8 10.3 15.1 7. 12.123.12.12250.0%3015 11.6 17.5 11.1 60.5 Since I'm not a network administrator, I'll ask point blank: why exactly do your netadmins filter and rate ICMP like this, and what are you gaining from it? Most kiddies stick with pure TCP or UDP these days -- the goal is to saturate the pipe, not cause a literal service DoS (e.g. crashing Apache, etc.) Additionally, I'll ask another question: exactly what tool are NOCs (or even network administrators) supposed to use to diagnose network path problems via layer 3 and 4? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Anyone from Comcast or ATT Worldnet on list?
Please contact me off-list. You (whether that be Comcast or ATT) have a networking (either circuit or BGP) issue in the northern California Bay Area which has been going on for numerous days now with no resolution. Thanks. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: [Fwd: Important ICANN Notice Regarding Your Domain Name(s)]
On Wed, Oct 04, 2006 at 03:38:02PM -0600, Chris Stone wrote: On Wed, 2006-10-04 at 14:27 -0700, Thomas Leavitt wrote: Is this a GoDaddy specific thing? I've owned and/or managed an untold number domain names since 1995 and never seen a notification of this sort before (primary registrar to this date was Gandi.net, and before that Network Solutions back in the bad old days). While, of course, the message is worded a bit different, we get the same thing for our domains registered under OpenSRS also every year. I receive these sorts-of notices from our OpenSRS-based registrar numerous times a year (usually once a month, for multiple domains). It may have something to do with the fact that I refuse to comply with ICANN's mandatory regulation demanding legitimate public contact information in WHOIS records. (If someone wants to hear my reasoning, I'll be more than happy to share.) -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: West Coast Fiber Cut?
On Fri, Sep 29, 2006 at 12:29:28PM -0700, Rick Kunkel wrote: Anyone know much about this major west coast fiber between Los Angeles and Washington that was supposed cut this morning? Our network is having gnarly problems through one of our providers and lesser ones through the other. Investigation went on for about 2 hours, whereupon i finally received an email from InterNAP talking about the problems starting at 9:45AM PDT, and being rooted in this fiber cut. My other provider has since told me that it was a Qwest fiber, and that most major transit providers were using it. Anyone heard anything else about this? I haven't heard anything, although we don't use InterNAP for IP nor Qwest for transit. Some of our eastern-bound IP traffic does head south (from norcal to socal). Do you have general timestamps of when the issue began? And two hours to determine a cut is pretty absurd, if you ask me. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Zimbabwe satellite Internet link restored
On Thu, Sep 28, 2006 at 03:15:44PM +0100, Alexander Harrowell wrote: Any chance of a moderator de-subscribing [EMAIL PROTECTED] from nanog? Every time anyone posts it kicks back a DSN, either failed or mail-loop More importantly: how did this address get subscribed to NANOG when manual verification is required to subscribe? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Qwest event 70 min ago?
On Tue, Sep 12, 2006 at 08:07:57PM -0600, Charlie Watts wrote: Did anybody see a Qwest event ~70 minutes ago? I'm not a direct customer so they won't talk to me, but we lost connectivity to a number of Qwest-connected sites for about 12 minutes. The data is falling off of the 1hr report, but you can still see it now: http://www.internetpulse.net/ http://www.internetpulse.net/Main.aspx?OriginValue=QwestOriginLevel=1 Thanks! All we got was this, from one of our clients: DATE OF EVENT: 9/12/06 TIME OF EVENT: 18:59 MDT LOCATION: Network Outage - Multiple CyberCenters EVENT DESCRIPTION: This is to notify you that the Qwest Hosting Services has experienced core routing conflicts that may have impacted your service. This is the final notification of this event. An RFO will be available within 48 hours upon request. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Earthlink playing Sitefinder
On Mon, Aug 28, 2006 at 10:20:50AM -0400, David Lesher wrote: I see discussion on DSLReports that Earthlink is deploying a Sitefinder-ish DNS scheme. It bounces people to a barefruit.com that then sprays ads back at you. The news article itself: http://www.dslreports.com/shownews/77566 The actual discussion thread, which has applicable details: http://www.dslreports.com/forum/remark,16763566 -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Amazon?
On Mon, Aug 21, 2006 at 03:21:40PM -0400, Jon R. Kibler wrote: Hi, Anyone know what is up with Amazon? They appear to be down. Things should be working again. Details and brief intro: The company I work for handles their automated call processing for them, amongst other things. We monitor a portion of their network externally. Amazon, from what I understand, was aware of the issue -- but have not provided any details as to what the problem was. Portions of Amazon-Target (www.target.com) may still be offline. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Wikipedia/Cogent
On Fri, Aug 18, 2006 at 03:02:45PM -0400, Richard A Steenbergen wrote: So, lets kick this Friday off right... I don't suppose anybody has noticed that Wikipedia is being blackholed by Cogent, and that it seems to be intentional? :) I'm not able to reach 207.142.136.174 (also via Cogent) either, which is forums.miranda-im.org. 207.142.136.0/24 *[BGP/170] 01:57:05, localpref 100 AS path: 701 174 ? For Wikipedia: 207.142.131.0/24 *[BGP/170] 01:58:02, localpref 100 AS path: 701 174 ? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Wikipedia/Cogent
Looks like some others may have noticed... 207.142.131.0/24 *[BGP/170] 00:26:46, localpref 100 AS path: 701 3356 30217 I -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: SORBS Contact
The thread was originally very benefitial (for me, as we use SORBS and provide some basic SMTP services), despite being somewhat off-topic for NANOG... but has now evolved into the Battle of Awful Analogies(tm). Discussions of this type always resort to the same analogy, for that matter: cars. It seems we've reached that point. Also, as I'm still fairly new here: why do so many NANOG threads go this route (pun intended)? Are some folks here unable to simply say what they mean? Just curious. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Detecting parked domains
On Wed, Aug 02, 2006 at 09:10:31PM +0200, Florian Weimer wrote: Has anyone come up with a quick method for detecting if a domain name is parked, but is not being used except displaying ads? AFAICT, the main challenge is to define what parked means in the context of your application. It seemed quite obvious to me: he's talking about domain squatting. Parking is just a euphemism. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Abovenet fibre cut
Has anyone heard of a fibre cut affecting Abovenet customers (particularly those utilising MPLS), possibly centralised around the east coast? I've managed to confirm with the Abovenet NOC that there is a cut which is likely the reason we're seeing increased latency between California and Virginia, and Arizona and Virginia, but have no other details about the problem, nor the location of the cut. Thanks. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: www.gigablast.com
On Wed, Jul 12, 2006 at 02:50:54PM -0700, Malcolm Staudinger wrote: Google is your friend? They're a search engine. robots.txt and forget it. Malcolm That's assuming whoever designed their software actually adheres to robots.txt. RFCs recommend people adhere to it, but there are some who don't; it's operationally optional. I can't find a single reference to what standards GigaBlast adheres to, or any technical data about how their engine works. The way their site is designed, it looks like a total fly-by-night operation. If GigaBlast is supposedly indexing his site, they have to be basing their GET requests on something (the equivalent of a normal browsers' Referer header; but again, who knows if they pass that along?). The requests Jim is seeing appear to be garbage, similar to spam composition, not based on actual references/indexes. I could be outright wrong here. Additionally, how does this solve the issue of Jim's bandwidth, CPU, memory, if not his time, being wasted for HTTP requests which shouldn't necessarily even be arriving at his boxes (which is what he's essentially complaining about)? So filter upstream, or on the machine itself. Okay, that's a solution, but it doesn't address incoming traffic (just responses). -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Best practices inquiry: tracking SSH host keys
On Thu, Jul 06, 2006 at 04:52:52PM -0400, Steven M. Bellovin wrote: On Thu, 29 Jun 2006 19:43:48 + (GMT), Christopher L. Morrow [EMAIL PROTECTED] wrote: apparently kerberos scares people... I'm not sure I 'get' that, but :( A corp security group once for a long time 'didnt believe in kerberos', some people 'get it' some don't :( Kerberos is a single point of failure; that scares people. You *know* you have to keep the Kerberos server locked down tight, highly available (very tricky for some ISP scenarios!), etc. Speaking purely from a system administration point of view, Kerberos is also a nightmare. Not only does the single-point-of-failure induce red flags in most SAs I know (myself included), but having to kerberise every authentication-oriented binary on the system that you have is also a total nightmare. Kerberos 4 is also completely incompatible with 5. Let's also not bring up the issue of globally-readable Kerberos tickets laying around /tmp on machines which use Kerberos, okay? ;-) Admittedly, the rebuttals to this are a) most things use PAM which can use Kerberos transparently and b) most network utilities these days support Kerberos. I run into things every day that don't support neither Kerberos or PAM. The bottom line is that SSH is easier, so more people will use it. That may not be the best attitude, I'll admit, but that's reality. At my current workplace, our SAs + developers wrote a distributed key system (client + daemon) that runs on all of our machines. It handles distribution and receiving of SSH keys, creating home dirs, and deciding who gets their public key stuck into /root/.ssh/authorized_keys as well. I haven't looked, but it wouldn't surprise me if something like this was already available via SourceForge or some other open-source publishing medium. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Nationwide Routing issues with Wiltel
On Mon, Jun 26, 2006 at 04:39:17PM -0400, Vincent India wrote: Anyone experiencing problems with Wiltel Backbone, or know of any issues with the Wiltel Backbone? I called their NOC and was told they are experiencing a nationwide routing problem that they are working on but couldn't get any further details? Was anyone able to get an RFO or post-mortem for this? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Nationwide Routing issues with Wiltel
On Mon, Jun 26, 2006 at 04:55:04PM -0400, david raistrick wrote: On Mon, 26 Jun 2006, Vincent India wrote: Anyone experiencing problems with Wiltel Backbone, or know of any issues with the Wiltel Backbone? I called their NOC and was told they are experiencing a nationwide routing problem that they are working on but couldn't get any further details? Told me it was related to the L3/Wiltel integration. Most of the breaks I've been seeing seem to be at the point where most of my traffic has be going from wiltel to l3 in DC or so. Oh, and that the former wiltel tier 1 guys had 40+ calls in the hold queue... I can confirm this as well, although have no proof (at this point) that Layer3 is necessarily to blame. We (or rather the company I work for) are seeing similar between MCI/UU/Verizon and WCG when reaching some of our clients: 4. 0.so-4-0-0.CL1.LAX15.ALTER.NET 0.0% 102 10.9 11.3 10.9 17.1 0.7 5. POS6-0.GW1.LAX15.ALTER.NET 0.0% 102 10.7 11.1 10.7 13.9 0.5 6. wcgGigELAX-gw.customer.alter.net 98.0% 102 135.4 147.0 135.4 158.5 16.3 7. anhmca1wcx2-pos6-1-oc48.wcg.net 98.0% 102 162.9 156.8 150.8 162.9 8.6 -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: Tor and network security/administration
On Wed, Jun 21, 2006 at 05:02:47PM -0400, Todd Vierling wrote: If the point of the technology is to add a degree of anonymity, you can be pretty sure that a marker expressly designed to state the message Hi, I'm anonymous! will never be a standard feature of said technology. That's a pretty obvious non-starter. Which begs the original question of this thread which I started: with that said, how exactly does one filter this technology? You can't doesn't make for a very practical solution, by the way. The same was said about BitTorrent (non-encrypted) when it came out, and the same is being said about encrypted BT (which has caused some ISPs to induce rate-limiting). I'm also left wondering something else, based on the Legalities Tor page. The justification seems to be that because no one's ever been sued for using Tor to, say, perform illegitimate transactions (Kevin's examples) or hack a server somewhere (via SSH or some other open service), that somehow that speaks for itself. I don't know about the rest of the folks on NANOG, but telling a court I run the Tor service by choice, but the packets that come out of my box aren't my responsibility, paraphrased, isn't going to save you from prison time (at least here in the US). Your box, your network port, your responsibility: period. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Tor and network security/administration
Apologies if this has been brought up before. Being as I'm not a network administrator myself (although I do filter some stuff using pf and ipfw on my severs), I'm curious what NAs think of the following technology: http://tor.eff.org/overview.html.en The problem I see is that this technology will be used (literally, not ideally) solely for harassment (especially via IRC). I do not see any other practical use for this technology other than that. The whole right to privacy/anonymity argument is legitimate, but I do not see people using* Tor for legitimate purposes. A colleague of mine stated his opinion of my opinion: Your problem with Tor is that you can't control it, isn't it? And he's right -- that's the exact problem I have with it. Comments/concerns? -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: who runs http://www.networkthinktank.com/ ?
On Fri, Jun 09, 2006 at 08:01:37PM +0200, William Waites wrote: Maybe there's someone in Hannover that knows who they are... Their telephone number is just an anonymous mailbox... Very mysterious indeed... FWIW, the WHOIS location is residential. Doesn't provide much information, nor does it imply anything... http://maps.google.com/maps?f=qhl=enq=1413+Gesna+Drive,+Hanover,+MD+21076ll=39.142443,-76.700792spn=0.011949,0.026779t=hom=1 -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Telia network degredation / POC
Does anyone have a contact number/POC of any sort for Telia that's within the United States? Jared's NOC list only contains a contact number in Sweden. It seems their network has been falling apart both within Sweden and the US since the raid on TPB (ThePirateBay.org) earlier today. Sure, it's probably kiddies as usual, but this is pretty major. Can anyone confirm/deny? Host Loss% Snt Rcv Last Avg Best Wrst StDev 1. gige-g6-0-19.gsr12012.fmt. 0.0% 119 1190.3 0.3 0.2 0.6 0.1 2. pos1-0.gsr12416.fmt.he.net 0.0% 119 119 51.5 19.6 0.4 266.9 50.7 3. pos10-0.gsr12416.sjc2.he.n 0.8% 119 1181.1 11.2 0.9 199.1 36.4 4. sjo-bb1-pos5-2-0.telia.net 19.3% 119961.0 5.2 0.9 145.5 18.3 5. las-bb1-pos7-0-0-0.telia.n 36.4% 11975 13.6 17.3 13.5 133.4 16.2 6. dsl-bb1-pos7-0-0.telia.net 56.4% 11851 80.9 51.7 48.1 114.5 12.0 7. nyk-bb2-link.telia.net 82.1% 11821 85.6 85.4 85.2 85.7 0.1 8. kbn-bb2-link.telia.net 53.8% 11854 165.7 166.1 165.6 177.5 1.6 9. s-bb2-link.telia.net 52.1% 11856 177.3 177.8 177.1 196.6 2.7 10. s-b4-pos5-0.telia.net 52.1% 11856 177.3 184.2 177.2 318.2 26.1 11. hy-peer1-pos4-0.se.telia.n 52.1% 11856 177.4 183.5 177.2 318.6 26.7 12. hy-c1-link.se.telia.net52.1% 11856 177.4 184.7 177.3 357.0 31.5 13. oes-b-c1-link.se.telia.net 53.0% 11855 190.4 190.4 190.2 191.0 0.1 14. ll-d6-link.se.telia.net52.1% 11856 195.3 195.3 194.8 203.7 1.2 15. bd-a13-link.se.telia.net 53.0% 11855 195.5 206.1 195.3 408.6 39.6 16. 213.65.248.233 52.1% 11856 193.9 194.1 193.8 196.9 0.4 Thanks. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networkinghttp://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
