Re: IBM report reviews Internet crime
JC Dill wrote: I'm really surprised that ISPs haven't banded together to sue Microsoft for negligently selling and distributing an insecure OS that is an Attractive Nuisance - causing the ISPs (who don't own the OS infected computers) harm from the network traffic the infected OSs send, and causing them untold support dollars to handle the problem. If every big ISP joined a class action lawsuit to force Microsoft to pay up for the time ISPs spend fixing viruses on Windows computer, Microsoft would get a LOT more proactive about solving this problem directly. The consumers have no redress against MS because of the EULA, but this doesn't extend to other computer owners (e.g. ISPs) who didn't agree to the EULA on the infected machine but who are impacted by the infection. jc I think I would rather see a class action against Symantec for the hundreds of hours ISP's waste fixing customers mail server settings that Symantec sees fit to screw up with every update. We can always tell when they have pushed a major update - hundreds of calls from mail users who can no longer send mail. It's 2008. How bloody hard is it to notice that the mail server SMTP port is 587 and authentication is turned on? Why do they mess with it? -- Mark Radabaugh Amplex 419.837.5015 x21 [EMAIL PROTECTED]
Re: An Attempt at Economically Rational Pricing: Time Warner Trial
Mikael Abrahamsson wrote: I'm also looking forward to the pricing, all the per-byte plans I have seen so far makes the ISP look extremely greedy by overpricing, as opposed to we want to charge fairly for use that is what they say in their press statements. I see it more as an experiment driven by the P2P issues and the net neutrality arguments. If we have to throw away the established flat rate / oversubscription models due to P2P upload then something has to give - either per byte pricing arrives, traffic shaping becomes more common, upstream rates are reduced, or the entire last mile is replaced. P2P is not going to go away and it's hiding itself more every day. Rate limiting hurts all the customers while per byte pricing hurts only a few users. It took 20 years to build the existing last mile. I don't see it being replaced en mass. I'm going out on a limb here but per-byte is going to be the answer in the end - and marketing is going to have some work ahead of them in selling it. Now it becomes a game of chicken to see who blinks first. Mark
Re: FW: ISPs slowing P2P traffic...
Joe Greco wrote:
As long as you fairly disclose to your end-users what limitations and
restrictions exist on your network, I don't see the problem.
You've set out a qualification that generally doesn't exist. For example,
this discussion included someone from a WISP, Amplex, I believe, that
listed certain conditions of use on their web site, and yet it seems like
they're un{willing,able} (not assigning blame/fault/etc here) to deliver
that level of service, and using their inability as a way to justify
possibly rate shaping P2P traffic above and beyond what they indicate on
their own documents.
Actually you misrepresent what I said versus what you said. It's
getting a little old.
I responded to the original question by Deepak Jain over why anyone
cared about P2P traffic rather then just using a hard limit with the
reasons why a Wireless ISP would want to shape P2P traffic.
You then took it upon yourself to post sections of our website to Nanog
and claim that your service was much superior because you happen to run
Metro Ethernet.
Our website pretty clearly spells out our practices and they are MUCH
more transparent than any other provider I know of.Can we do EXACTLY
what we say on our website if EVERY client wants to run P2P at the full
upload rate? No - but we can do it for the ones who care at this
point.At the moment the only people who seem to care about this are
holier than thou network engineers and content providers looking for
ways to avoid their own distribution costs. Neither one of them is
paying me a dime.
Mark
Re: ISPs slowing P2P traffic...
The vast majority of our last-mile connections are fixed wireless. The design of the system is essentially half-duplex with an adjustable ratio between download/upload traffic. PTP heavily stresses the upload channel and left unchecked results in poor performance for other customers. Bandwidth quotas don't help much since it just moves the problem to the 'start' of the quota time. Hard limits on upload bandwidth help considerably but do not solve the problem since only a few dozen customers running a steady 256k upload stream can saturate the channel. We still need a way to shape the upload traffic. It's easy to say put up more access points, sectors, etc.) but there are constraints due to RF spectrum, tower space, etc. Unfortunately there are no easy answers here. The network (at least ours) is designed to provide broadband download speeds to rural customers. It's not designed and is not capable of being a CDN for the rest of the world. I would be much happier creating a torrent server at the data center level that customers could seed/upload from rather than doing it over the last mile. I don't see this working from a legal standpoint though. -- Mark Radabaugh Amplex 419.837.5015 x21 [EMAIL PROTECTED]
Re: ISPs slowing P2P traffic...
Joe Greco wrote, There are lots of things that could heavily stress your upload channel. Things I've seen would include: 1) Sending a bunch of full-size pictures to all your friends and family, which might not seem too bad until it's a gig worth of 8-megapixel photos and 30 recipients, and you send to each recipient separately, 2) Having your corporate laptop get backed up to the company's backup server, 3) Many general-purpose VPN tasks (file copying, etc), 4) Online gaming (capable of creating a vast PPS load, along with fairly steady but low volumetraffic), etc. P2P is only one example of things that could be stressful. These things all happen - but they simply don't happen 24 hours a day, 7 days a week. A P2P client often does. snip for brevity The questions boil down to things like: 1) Given that you unable to provide unlimited upstream bandwidth to your end users, what amount of upstream bandwidth /can/ you afford to provide? Again - it depends. I could tell everyone they can have 56k upload continuous and there would be no problem from a network standpoint - but it would suck to be a customer with that restriction. It's a balance between providing good service to most customers while leaving us options. What Amplex won't do... Provide high burst speed if you insist on running peer-to-peer file sharing on a regular basis. Occasional use is not a problem. Peer-to-peer networks generate large amounts of upload traffic. This continuous traffic reduces the bandwidth available to other customers - and Amplex will rate limit your connection to the minimum rated speed if we feel there is a problem. So, the way I would read this, as a customer, is that my P2P traffic would most likely eventually wind up being limited to 256kbps up, unless I am on the business service, where it'd be 768kbps up. Depends on your catching our attention. As a 'smart' consumer you might choose to set the upload limit on your torrent client to 200k and the odds are pretty high we would never notice you. For those who play nicely we don't restrict upload bandwidth but leave it at the capacity of the equipment (somewhere between 768k and 1.5M). Yep - that's a rather subjective criteria. Sorry. This seems quite fair and equitable. It's clearly and unambiguously disclosed, it's still guaranteeing delivery of the minimum class of service being purchased, etc. If such an ISP were unable to meet the commitment that it's made to customers, then there's a problem - and it isn't the customer's problem, it's the ISP's. This ISP has said We guarantee our speeds will be as good or better than we specify - which is fairly clear. We try to do the right thing - but taking the high road costs us when our competitors don't. I would like to think that consumers are smart enough to see the difference but I'm becoming more and more jaded as time goes on One solution is to stop accepting new customers where a tower is already operating at a level which is effectively rendering it full. Unfortunately full is an ambiguous definition.Is it when: a) Number of Customers * 256k up = access point limit? b) Number of Customers * 768k down = access point limit? c) Peak upload traffic = access point limit? d) Peak download traffic = access point limit? (e) Average ping times start to increase? History shows (a) and (b) occur well before the AP is particularly loaded and would be wasteful of resources. (c) occurs quickly with a relatively small number of P2P clients. (e) Ping time variations occur slightly before (d) and is our usual signal to add capacity to a tower. We have not yet run into the situation where we can not either reduce sector size (beamwidth, change polarity, add frequencies, etc.) but that day will come and P2P accelerates that process without contributing the revenue to pay for additional capacity. As a small provider there is a much closer connect between revenue and cost. 100 'regular' customers pay the bills. 10 customers running P2P unchecked doesn't (and makes 90 others unhappy). Were upload costs insignificant I wouldn't have a problem with P2P - but that unfortunately is not the case. Mark
Re: ISPs slowing P2P traffic...
I would be much happier creating a torrent server at the data center level that customers could seed/upload from rather than doing it over the last mile. I don't see this working from a legal standpoint though. Why not? There's plenty of perfectly legal P2P content out there. Hum... maybe there is an idea here. I believe the bittorrent protocol rewards uploading users with faster downloading. Moving the upload content to a more appropriate point on the network (a central torrent server) breaks this model. How would a client get faster download speeds based on the uploads they made to a central server?To solve the inevitable legal issues there would also need to be a way to track how content ended up on the server as well. Are there any torrent clients that do this? Mark
Re: ISPs slowing P2P traffic...
P2P based CDN's are a current buzzword; Verilan even has a white paper on it https://www.verisign.com/cgi-bin/clearsales_cgi/leadgen.htm?form_id=9653toc=e20050314159653020ra=72.219.222.192email= Password protected link. I think we are going to see a lot more of this, and not just from kids. Regards Marshall This should prove to be interesting. The Video CDN model will be a threat to far more operators than P2P has been to the music industry. Cable companies make significant revenue from video content (ok - that was obvious).Since they are also IP Network operators they have a vested interest in seeing that video CDN's that bypass their primary revenue stream fail.The ILEC's are building out fiber mostly so that they can compete with the cable companies with a triple play solution. I can't see them being particularly supportive of this either. As a wireless network operator I'm not terribly interested in helping 3rd parties that cause issue on my network with upload traffic (rant away about how were getting paid by the end user to carry this traffic...). Mark
Re: ISPs slowing P2P traffic...
Geo. wrote: The vast majority of our last-mile connections are fixed wireless. The design of the system is essentially half-duplex with an adjustable ratio between download/upload traffic. This in a nutshell is the problem, the ratio between upload and download should be 1:1 and if it were then there would be no problems. Folks need to stop pretending they aren't part of the internet. Setting a ratio where upload:download is not 1:1 makes you a leech. It's a cheat designed to allow technology companies to claim their devices provide more bandwidth than they actually do. Bandwidth is 2 way, you should give as much as you get. Making the last mile a 18x unbalanced pipe (ie 6mb down and 384K up) is what has created this problem, not file sharing, not running backups, not any of the things that require up speed. For the entire internet up speed must equal down speed or it can't work. You can't leech and expect everyone else to pay for your unbalanced approach. Geo. Your back to the 'last mile access' problem. Most Cable, DSL, and Wireless is asymmetric and for good reason - making efficient use of limited overall bandwidth and providing customers the high download speeds they demand. You can posit that the Internet should be symmetric but it will take major financial and engineering investment to change that. Given that there is no incentive for network operators to assist 3rd party CDN's by increasing upload speeds I don't see this happening in the near future. I am not even remotely surprised that network operators would be interested in disrupting this traffic. Mark
Re: Why do we use facilities with EPO's?
Leo Bicknell wrote: I was complaining to some of the power designers during the building of a major facility that the EPO button represented a single point of failure, and effectively made all of the redundancy built into the power system useless. After all, what's the point of having two (or more) of anything, if there's one button somewhere that turns it all off? Seems like the EPO should be a logical AND with the fire alarm system - it only works AFTER you have an existing fire alarm in the building. -- Mark Radabaugh Amplex 419.837.5015 x21 [EMAIL PROTECTED]
Re: multiple-choice question of the day
Randy Bush wrote: No transition plan Declared victory before the hard part even started No real long term plan No realistic estimation of costs No real support for the folk on the front lines Victory will be next month Describes: a - The war in Iraq b - DNSsec c - IPv6 d - All of the above d. But what inspired this question? Mark
Re: Nationwide Routing issues with Wiltel
Steve Sobol wrote: On Mon, 26 Jun 2006, Vincent India wrote: Anyone experiencing problems with Wiltel Backbone, or know of any issues with the Wiltel Backbone? I called their NOC and was told they are experiencing a nationwide routing problem that they are working on but couldn't get any further details? I have a box sitting in a colo off a WCG circuit in Columbus, OH; traceroutes from the west coast were dying a few hops short of the colo facility, but I'm not a direct customer of WCG, so calling them for info would have been pointless... As a customer we were not able to get through to L3 on the phone. Apparently prefix filtering wasn't working so well either given that AS27251 was managing to announce 38/8, 64/8 and 67/8 with L3 happily passing it along. -- Mark Radabaugh Amplex [EMAIL PROTECTED] 419.837.5015
Re: How do you handle client contact for network abuse/malware compaints etc.?
Nicole Harrington wrote: Hello As a sort of addendum to the thread of Quarantine your infected users spreading malware I am curious how other handle contact to the users/clients for network security incidents. The question I have is; When someone reports an incident to you about one of your clients (a user or server owner) possibly being infected, having an owned box being used for hacking into other servers or being used to spread malware, how much information do you send/forward on to that user/client to support your case. Is it normal practice to simply forward on unaltered logs sent in by those complaining or do you sanitize them a bit to protect the people notifying you? Do you even send them at all at first or do you simply inform them that a complaint has been received. In short, how much information do you pass on to support yourself and when. Thanks Nicole Harrington All depends on the client and if I think the abuse is intentional or not. If the user knows what he/she is doing and I don't think they are being malicious then I will send them everything. If I think they are doing it on purpose I send enough to prove my case and tell them to knock it off - before I knock it off for them (or after - depends on how much damage they are causing). If they don't have a clue then sending them a bunch of information they won't understand is pointless. We either help them clean up the mess or refer them to someone who can. -- Mark Radabaugh Amplex [EMAIL PROTECTED] 419.837.5015
Re: The Domain Name Service as an IDS
Amongst others, I've developed the following services with it for my internal customers: Hi Chris, thanks for your reply. I was just told by the admin team to keep DNS operational issues off-list. Would you mind if we take this to the DNS operations mailing list run by the ISC OARC? Gadi. Let's see - a description of an interesting way to use DNS metrics to detect network abuse - network abuse that routinely causes headaches on our network and results in customer complaints. Seems pretty on topic for a network operations mailing list. -- Mark Radabaugh Amplex [EMAIL PROTECTED] 419.837.5015
Re: multi homing pressure
John Payne wrote: Hrm, people keep saying that BGP is hard and takes time. As well as my end-user-facing network responsibilities, I also have corporate network responsibilities here. All of our corporate hub locations are multi-homed (or soon will be)... and I honestly can't remember the last time I made any changes (besides IOS upgrades) to BGP configs for the 2 hubs in the US. (We're moving physical locations in the international hubs and taking new providers, so I'm discounting those changes as you'd have similar changes in a single homed statically routed move). If you don't have multihoming requirements other than availability then it really can be fire and forget. Except for those pesky bogon filters which corporations seem to like to fire and forget. -- Mark Radabaugh Amplex [EMAIL PROTECTED] 419.837.5015
IPv6 BGP Peering
What is the state of IPv6 BGP peering with US transit providers? Questions to sales / tech reps are generally met with I heard we were working on something and that's as far as I have made it so far. The routing table shows UUNet, Verio, Sprint and a few other transit providers but I am not having much luck finding contact or setup information for those providers specific to IPv6. -- Mark Radabaugh Amplex [EMAIL PROTECTED] 419.837.5015
Re: CAT5 surge/lightning strike protection recommendations?
Steven M. Bellovin wrote: In message [EMAIL PROTECTED], Aditya writes: The short-term solution seems to be using the APC PNET1s/Tripplite DNET1/etc. in each unit and tying them to the water main as an inexpensive, immediate step while funds are allocated for conduit, labor etc.. If I recall correctly, the National Electrical Code was change about a dozen years ago to bar grounding to water pipe unless it's within about 6' of where the pipe enters the house. --Steven M. Bellovin, http://www.cs.columbia.edu/~smb The change was in the 1993 NEC and restricts using water pipe as a ground to being within 5 feet of the point of entrance to the building. There is an additional requirement that the water pipe must be metallic and have a minimum of 10' buried in the ground. This can be hard to verify on existing installations due to the common use of plastic water pipe by utilities in recent years. The 5' from the point of entrance is also due to the common use of plastic water pipe in buildings. -- Mark Radabaugh Amplex [EMAIL PROTECTED] 419.837.5015
Re: CAT5 surge/lightning strike protection recommendations?
R.P. Aditya wrote: I have a bunch of cat5 buried about 1 ft below the surface connecting multiple buildings on a campus (short runs) and lightning strikes nearby have caused surges along one or more of the cables and burnt out switch ports. I would like to protect the switch ports -- there seem to be lots of products on the market. Anyone have recommendations (tested/practical is best :-)? The APC Protectnet PNET1 and PRM24 seem quite nice and not too expensive -- if they workpros? cons? Thanks, Adi I'll go with the fiber recommendation but that's not what you asked :-) We use quite a few of the Motorola 300SS surge suppressors. They are made for use with Motorola's fixed wireless Internet platform and go on the Ethernet cable before it enters the building. They do a good job of protecting the ports on near misses. Direct strikes and they are toast along with anything attached to them - but that's just the way it goes :-) http://www.tessco.com/products/headerProductSearch.do?searchType=1searchText=300sssearchField=1 -- Mark Radabaugh Amplex [EMAIL PROTECTED] 419.837.5015
Re: CAT5 surge/lightning strike protection recommendations?
David Lesher wrote: Surge protectors can not protect you from ground differential issues. True enough - but 10/100 Ethernet is normally isolated by the transformer on the Ethernet transceiver. AFAIK there is not a connection between the signal lines and ground. Isolation is 1500V for the magnetics I checked. Off course all bets are off when lightning strikes since the voltage tends to be just a tad higher than 1500 volts. Mark Radabaugh
Re: SWIP and Rwhois in the Real World
Are folks finding that public rwhois availability is a wide spread problem with ISPs who are using rwhois, or is level3 an isolated incident? Replies off-list are ok. Andrew (also an ARIN Advisory Council member) The person trying to implement Rwhois here reported that the ARIN recommended server is very difficult (impossible?) to configure to return the information that ARIN says they want from Rwhois. He gave up and resorted to SWIP. I suspect either the server implementation or the instructions need some help. -- Mark Radabaugh Amplex [EMAIL PROTECTED] 419.837.5015
Re: Microsoft SOHO router multicast problem? - or maybe it's just doing what it's supposed to be doing...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chance Whaley wrote: | Sorry about that. Didn't look in detail. Saw the UDP port 6257 and | stopped. | | The mcast is coming from someplace upstream from | fastethernet-0-0.genoa-gw.amplex.net (that is if I did my mcast | MAC to mcast IP conversion right). Without knowing your topology | and seeing more traffic it's kinda hard to figure out. | | | If you want to send more traffic captures I will be happy to look | at them. | | .chance | The destination mac address the routers start using is 01:00:5e:76:6c:7e. The 01:00:5e is the ethernet multicast header. The 76:6c:7e is supposed to be the lower 23 bits of the Ethernet multicast address - which translates to 118.108.126.With the 23 bits from the multicast spec for encoding the IP address 118 is the correct conversion of 246 with the high bit stripped off. The gateway on this subnet is 64.246.108.126 (netmask is 255.255.255.0 but originally was .128 - hence the odd spot for the gateway). The routers decided to convert a mangled unicast packet to a multicast packet - for them to then loop on it is even stranger. It makes for a pretty good DOS attack. 2 or more of these routers in a broadcast domain can get ugly in a hurry. Mark -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCX3F2g0PQSWMG2wsRAhYaAKCDeTpKF1QuDhX82rQIOpPTQW4xwACggXhd uHRnFxzmWbrfHSvZGS9ljrs= =IcaN -END PGP SIGNATURE-
Microsoft SOHO router multicast problem? - or maybe it's just doing what it's supposed to be doing...
So which one of the gods of Multicast would like to take a look at a short tcpdump and tell me if the multicast broadcast storm is a problem with the protocol, the Microsoft implementation, or just a really weird coincidence? We run a fixed wireless network that for various reasons is bridged. Yes - it's a crappy design and we are working on changing it but that's not really the point. I have been trying to track down a broadcast storm that shows up on the network intermittently. I finally managed to capture the start of one tonight. The process starts with a slightly mangled packet (intentional? - can't tell yet) with the 'multicast promiscuous bit' set. All of the customers with Microsoft routers (and one Belkin) then rewrite the mangled packet into a multicast packet, decriment the TTL, and forward it back out the interface it came in on. This process then repeats with each of the Microsoft routers responding to the packets from the other routers and sending them out again. With 4 of these routers it manages to generate 20,000+ packets before all of the TTL's drop to 0. Needless to say this results in a little bit of a performance hit. I have blocked Multicast at several points on the network so the problem should be gone for now. The tcpdump file is at http://www.amplex.net/images/multicast.cap Mark Radabaugh Amplex
Re: Router choice for medium size hosting provider
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Alex Campbell wrote: | | Hi everyone, | | I'm looking for a new router to connect our data center to our tier | 1 ISP via a 50mbps fibre link. Does anyone have any advice about | what level of Cisco router would be required to saturate this link? | | | We're looking at the 2811 but I can't get any real world data about | whether it can route packets at 50mbps - this seems doubtful | although unclear from the information on the Cisco data sheets. | | (I'm aware that a cheap PC running Linux could provide similar | throughput to a $2 Cisco router but for a variety of reasons | I'm reluctant to follow this path). | | Thanks, | | Alex | Do you need BGP? That's going to make a big difference in what you want to use. An idea on the number/type of interfaces you need would be helpful as well. Mark Radabaugh -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCVHtNg0PQSWMG2wsRArOOAKCWwM70hEx2dxGDBU/yWK1Jn+4AnACdFGpD 7fJ9wZFncJ2Mq4OJPDyqWPQ= =TQfK -END PGP SIGNATURE-
Re: Bandwidth Advisors - www.bandwidthadvisors.com
For those that don't know... I am now the COO of UnitedLayer. It sounds like, since I am not going to pay the extortion fee to Bandwidth Advisors, that their consultants won't know about our pricing and services. Even if I did pay the fee, that means that their clients can't get the best deal as I need to raise my fees to client to cover the small residual payment going to Bandwidth Advisors. Tim Tim, Your completely free to hire your own sales people and advertising agency - if they do a good enough job you won't need someone like Bandwidth Advisors.Of course those people are going to want some money - and nearly all sales people work on some type of base + commission basis. It's up to you to decide what is less expensive and better for you. Either one is going to cost you something. Sitting around waiting for people to come to you can also be pretty expensive if you have any kind of fixed expenses. Mark Radabaugh Amplex
Re: Fire Code/UFC Regs?
Perhaps someone who knows EE can enlighten me? Joe OK - my considered opinion as a BSEE is: It's a pile of BS designed to sell PDU's. but do not efficiently distribute the power, meaning that some equipment may be deprived of the necessary amperage it requires to run properly Yeah. Sure. Mark Radabaugh
Re: AOL scomp
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Joe Maimon wrote: | I believe one has an extra duty to be as strict as possible about | accepting email to be forwarded to external parties: | | Read: Setup for every usuable blocklist, including you own, which | rejects email outright. And spamassassin setup to reject any | reasonable low FP score threshold. And none of that tag em all | and let the user sort it out business. | | Its not legitimate to cover your eyes and forward probable garbage | to someone else. You want it on your system, thats your decision. | AOL blocklisting high percentage garbage senders, including those | merely forwarding, is perfectly valid in my book. | | To blocklist all servers in the path or just the most recent one is | a local decision Now here I would disagree. These are specific requests by individuals to forward mail to from one of their own accounts to another one of their own accounts. I do not think AOL (or anyone) should consider mail forwarded at the customers request as indicating that our mail servers are sending spam. As that is apparently not the case I have seriously considered as a matter of policy refusing to install mail forwards to AOL customers. Mark Radabaugh Amplex -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (MingW32) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCHjCqg0PQSWMG2wsRAnnfAJ9IE+GIuYnBrDKrE3OlpAvZIuuXbQCfSEAS GSSlg8c0AHPh044rMDauHyI= =OjDT -END PGP SIGNATURE-
Re: Verizon wins MCI
Alex Rubenstein wrote: But does anyone really know how big WorldCon is/was? First thing Verizon will have to do is fire the entire billing department and replace them with people/systems that can generate correct bills and send them to the correct customers. dripping with sarcasm Thats right! I forgot that Verizon was capable of billing correctly! /dripping with sarcasm So the real question is do we scream bloody murder about the current incorrect MCI billing before or after Verizon screws it up? Let's see - MCI already fired all the customer service people Mark
Re: Verizon wins MCI
Christopher L. Morrow wrote: On Tue, 15 Feb 2005, Mark Radabaugh wrote: Alex Rubenstein wrote: But does anyone really know how big WorldCon is/was? First thing Verizon will have to do is fire the entire billing department and replace them with people/systems that can generate correct bills and send them to the correct customers. dripping with sarcasm Thats right! I forgot that Verizon was capable of billing correctly! /dripping with sarcasm So the real question is do we scream bloody murder about the current incorrect MCI billing before or after Verizon screws it up? Let's see - MCI already fired all the customer service people some of customer service is still here... we didn't all get sent overseas yet. I think of you as Security rather than customer service - and your group still does a great job. I have absolutely no complaints in that regard. I like UUNet's network quite a bit (other than the billing department). Mark
Re: How to monitor BGP route stability ?
Joe Shen wrote: Hi, Is there any tool to monitor BGP route stablity? thanks Joe __ Do You Yahoo!? Log on to Messenger with your mobile phone! http://sg.messenger.yahoo.com http://www.renesys.com. Gradus is pretty effective. Mark Radabaugh Amplex
Newedge Networks meltdown?
Seems there is a major problem within NewEdge networks this morning. From the recording on the NOC line it sounds like they lost an internal database and PVC's at many of the switches. Anyone know what happened? Let the speculation begin :-) Mark Radabaugh Amplex
Re: AOL tarpitting?
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mark Jeftovic Sent: Friday, November 05, 2004 2:26 PM To: [EMAIL PROTECTED] Subject: AOL tarpitting? The new route was tarpitted within 24 hours and absolutely nothing was communicated to us about it via the feedback loop. Does that thing actually work? Any contacts, on or off list who could advise? -mark The feedback loop seems to work quite well though we see a lot of what I consider to be bogus spam reports (not mass mail - just the usual idiots forwarding jokes, mail from their own employers, etc.).. I suspect some AOL users think the 'report as spam' button is the delete key - it was pointed out that they are next to each other on the menu. The reports have helped us to find a few low volume exploits of scripts (100 messages /day) on the web servers that we probably would not have found otherwise. Mark Radabaugh Amplex
Re: Are AOL's MXs mass rejecting anyone else's emails?
Christopher X. Candreva wrote: On Tue, 7 Sep 2004, Jon Lewis wrote: Any network that doesn't already have it, I highly recommend signing up for AOL's feedback loop (aka scomp reports) at http://postmaster.aol.com/tools/fbl.html. This will give you a sort of early warning system notifying you of spam issues on your network. And you will also get random emails that your users have sent to AOL users, who then click on Report as spam seemingly at random. I've received Spam reports on e-mail asking when someone's kids should be picked up at school, giving directions for a job interview, CONGRATULATING that same person on being accepted for the job, and in once case received a 'spam complaint' on every mail my user sent as part of a conversation. As in, the AOL user replied, then clicked Report as spam. He received a reply to his reply, replied, and Reported as Spam. This was not a Stop e-mailing me conversation. It was a perfectly normal conversation between two people. Then there are the people who have mail forwarded from here to their AOL account, and can't get it through their thick skulls that Report as spam isn't doing a damn thin in this case. G. So it's a nice idea -- but IMHO fails in practice. It's still pretty handy but I agree lots of AOL users seem to think the 'report as spam' button must be the delete button or something. When somebody on our network gets infected with a spam trojan the feedback loop is pretty helpful in detecting it quickly. Mark Radabaugh Amplex
Re: sms messaging without a net?
Mark Radabaugh wrote: Dan Hollis wrote: Does anyone know of a way to send SMS messages without an internet connection? Having a network monitoring system send sms pages via email very quickly runs into chicken-egg scenario. How do you email a page to let the admins know their net has gone down. :-P ATT shut down their TAP dialup late last year. The only method that comes to mind is to buy a GSM modem which has SMS messaging capability. Has anyone done this? -Dan We use a standard modem and pestered Alltel until they gave us the secret modem-SMS gateway number. Nagios includes /usr/local/bin/sms_client. Fairly simple to use: Usage: sms_client -v sms_client -d sms_client [-q][-l loglevel] [service:]number|name[,name|[service:]number...] [message] ... As long as the monitoring server and the phone lines are working it works great. Mark Radabaugh Amplex Dan Hollis wrote: Does anyone know of a way to send SMS messages without an internet connection? Having a network monitoring system send sms pages via email very quickly runs into chicken-egg scenario. How do you email a page to let the admins know their net has gone down. :-P ATT shut down their TAP dialup late last year. The only method that comes to mind is to buy a GSM modem which has SMS messaging capability. Has anyone done this? -Dan We use a standard modem and pestered Alltel until they gave us the secret TAP gateway number. Nagios is configured to use sms_client: http://www.smsclient.org/ Works great other than being a royal pain in the rear when a lot of things go down and dependencies are not set up in Nagios... I'll fix that one of these days :-) Mark Radabaugh Amplex
Re: T1 short-haul vs. long-haul
David Lesher wrote: I have hard experience with SDSL vs straight DS1 service. SDSL has three-four different companies in the picture. You only get to talk to the first. Period. Everything beyond is 2nd hand. They all have finger splints from overuse; pointing to each other. I have had Verizonal leave a DS-3 feed to a DSLAM in loopback and go home when they ran out of OT budget. DS-1 service is a horse of a different color. Bells go off when they go down. You can raise hell, but I never have had to... Except in SBC territory where T1 service means they might get around to fixing it if they feel like it (and only during normal business hours). The usual method is to declare no trouble found and close the ticket (especially if the ticket is approaching 24 hours). Mark Radabaugh Amplex
Re: Akamai DNS Issue?
Workarounds and defences already exist, and have been in use for a long time. long list removed Failures in master servers can be mitigated by having several of them; simultaneous failure of all master servers can be managed to some degree using appropriate SOA timers, so that slave servers provide coverage while master servers are brought back into service. Different styles of attack can be mitigated by different DNS hosting strategies. A robustly-hosted zone will have an NS set that exhibits several or all of these approaches (and others too). The hosting of the root zone provides guidance, here. Joe But you don't say how to avoid failures caused by massive confusion when maintaining a excessively complicated system Mark
Re: Determining ownership of Internet routing problems
Now I'm wondering if that is even a valid assumption. Maybe the truth is more like this: [backbone provider A] [backbone provider B] / \ /\ [intermediate A] [intermediate C] [intermediate B] / \ [ISP A] [ISP B] and if the problems is with intermediate C, I'm probably SOL. Clearly, I would want my ISP to insist that his upstream providers not allow such unreliable topologies to be used. Ken Wallewein CDP,CNE,MCSE,CCA,CCNA This is a close approximation of the actual topography - though there may be a few more 'C's between A and B.The reality is that there are a couple of really large providers and the majority of networks connect either directly or are no more than a couple of hops off of the largest networks. See http://www.caida.org/analysis/topology/as_core_network/AS_Network.xml. End users trying to fix routing loops usually have fairly limited success due to the difficulty in notifying the affected providers - this is an issue that has been hashed to death and I don't think anyone is in the mood to discuss it again. Your 2 upstream providers are the ones you need to hold responsible - it's their job to get your packets where they need to go. It is entirely possible that various networks on the Internet will not be able to talk to other networks due to policy and peering decisions made by the various providers. Providers that do not provide access to the majority of the Internet usually find they loose market share quickly - this tends to be self correcting in fairly short order. Mark Radabaugh Amplex (419) 720-3635
Gammanetworks
Anyone know if GammaNetworking is black/whitehat (http://www.gammanetworking.com)? I don't find much on them but they are supporting at least one spam gang and unresponsive to inquiries. Mark Radabaugh Amplex (419) 720-3635
bad link to real stream
http://www.nanog.org/mtg-0310/network.html has To watch the meeting live, check the links on the main conference page. - the link is bad. The real real feed is at ??? Mark Radabaugh Amplex (419) 720-3635
Verisign to sell Network Solutions
This is interesting: Dear Valued Network Solutions® Customer, Today VeriSign, Inc. announced that it has entered into a definitive agreement to sell Network Solutions to a new entity formed by Pivotal Private Equity. Mark Radabaugh Amplex (419) 720-3635 begin 666 clear.gif K1TE.#EA`0`!`( ``'Y! $`+ `!``$```(1 $`.P`` ` end
Re: Verisign to sell Network Solutions
So...correct me if I'm wrong here...does this mean that the registry services operations and the GTLD maintenance operations for .com/.net will be owned by different companies? Yep. And it means that Verisign business is no longer based so much on serving customers but more on leveraging various monopoly rights that they have such as ownership of .com and ownership of the main root CAs whose certificates are bundled with Microsoft's OS. Isn't that what we wanted all along? This just gets more interesting. If the sale of the NetSol was in the works for some time (and these things usually take a while) it would mean that Verisign was likely counting on the revenue from Sitefinder to offset the loss of revenue from selling NetSol. They cut off an arm and then shot themselves in the foot. Maybe ICANN can put them out of our misery. It would help explain why Verisign continues to press so hard for Sitefinder despite the firestorm over it New Coke anyone? Mark
Re: Sitefinder fan - this guy needs a clue.
- Original Message - From: Robert Boyle [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 08, 2003 10:52 AM Subject: Sitefinder fan - this guy needs a clue. Wow. This guy is completely delusional. http://zdnet.com.com/2100-1107_2-5087746.html That 'guy' is Verisign's Senior VP - he is spouting the company line. It's pretty irresponsible of NZNet to not identify who the person is. I would have hoped (but been highly surprised if he did) that McLauglin would have identified himself as an principle of Verisign.Just more of the PR machine on a roll. Mark Radabaugh Amplex (419) 720-3635
Re: DoS Attacks
So here I am, asking if anyone here has any advice on dealing with these issues in the future? Its painfully apparent noone takes these situations seriously enough. What should we do when we are put in a position like this? Just sit back and hope it goes away itself? Also, any ideas on how to deal with these attacks on lower bandwidth connections? Right now, 2mbit.com / sosdg.org is sitting on a 1.5/256 business DSL line. I really can't afford to be buying T1s or T3s just to hold up to attacks like this. As always, thanks. -- Brian Bruns The Summit Open Source Development Group I think I would follow two avenues next time - the direct approach with FSU (or wherever the traffic is coming from) as well as with your DSL provider. Your upstream should be able to assist in at least keeping the traffic off of your dedicated line. Whether your DSL provider has the resources to sink the traffic may be another matter -- but they are at least in a position to help you and (since you are paying them) have an interest in dealing with you. Mark Radabaugh Amplex (419) 720-3635
Re: Massive sprintlink problems?
- Original Message - From: Michael Loftis [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, October 01, 2003 2:47 PM Subject: Massive sprintlink problems? Anyone else seeing this:: (1sec+ delay to my idle DSL line across sprintlink...) traceroute is definitely taking an asymmetric path, since pings and tcp connections are consistent 1sec plus RTT starting somewhere in seattle or tacoma.tok? tokyo? Anyway before I start rattling this around I wanted to see if anyone else is seeing this to/from other destinations. Sprint Naming Conventions: http://www.sprintlink.net/faq/namingconvention_sl.html TOK Tokyo, Japan 9 sl-bb22-sj-15-0.sprintlink.net (144.232.3.162) 10 sl-bb20-tok-10-0.sprintlink.net (144.232.9.243) 11 sl-bb21-tac-8-2.sprintlink.net (144.232.19.243) Why they are sending it through Tokyo is another question Mark
Re: Annoying dynamic DNS updates (was Re: someone from attbi please contact me ...)
Original Message - From: Owen DeLong [EMAIL PROTECTED] To: [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Monday, September 29, 2003 1:07 PM Subject: Re: Annoying dynamic DNS updates (was Re: someone from attbi please contact me ...) Think about Micr0$0ft trying to fight off thousands or better millions of small claims cases all over the country. Even if Micr0$0ft wins every one, they lose. Owen FWIW (and IANAL) in Ohio a corporation filing in small claims court requires a lawyer to represent the company. This increases the cost to at least $250 just to get in the door. I would be suprised to get out with under a $1000 in legal fees. Even though we have a Ohio spam law (yeah - I know we are talking about DNS here..) AFAIK no ISP has bothered trying to use it since the cost relative to the potential recovery is out of line. Makes it pretty impractical to use this method for dealing with annoying but economically minor issues. Mark
Nothing like viruses with bugs in them (Swen)
Seems like this virus/worm has a bug where it will occasionally send out 1 byte attachments rather than the correct worm payload. Since the virus is not truly attached it tends to pass through e-mail virus scanners. It's causing a fair amount of end user confusion today -- lots of 'why is your/my virus scanner not working?' questions. Mark Radabaugh Amplex (419) 720-3635
Re: Change to .com/.net behavior
In other news, Verisign has a press release on their website announcing something called Next Registration Rights Service, where you can place an order to have somebody else's domain transferred to you if they ever don't pay their bill. The press release goes on to say that this is a great way for holders of existing domain names to buy insurance to protect themselves from the loss of their domain names if their bill doesn't get paid, but apparrently only if nobody beats them to it. -Steve If you make the mistake of letting a domain reach the 'redemption' period Verisign holds it hostage and dead for a couple of weeks unless you pay them a $150 extortion fee to get it back. Apparently ICANN approved the redemption period and allows the registrar to set whatever fee they like. I can not prove but I suspect that Verislime is now leaving expired domain in the GTLD servers until they reach the redemption period in the hope that people will not notice the domain not resolving until it reaches the extortion period. Why are we still putting up with this garbage from Verisign and ICANN? Mark Radabaugh Amplex (419) 720-3635
Re: Cross-country shipping of large network/computer gear?
I was wondering if anyone could provide any advice or suggestions on shipping heavy/bulky equipment (~300 pounds, about a half-rack worth of gear) on short notice cross-country? We're obviously looking to minimize cost, but realistically it can't be in transit for more than two days. Are there any companies or methods people would recommend? Thanks in advance for the help. Thanks again, Matt This probably is too small of a load for this but we have had good luck moving high value industrial control panels using the special cargo division of carriers like United Van Lines (http://www.unitedvanlines.com/spec/highvalue.htm?gid=9). Basically standard household moving trucks with crews dedicated to moving high value electronics, exhibits, art, etc. around the country.With a 2 person crew in the truck you can go a hell of a long ways in 2 days though the cost may not be exactly pretty. Mark Radabaugh Amplex (419) 720-3635
Who uses RADB? [was BGP to doom us all]
No, the lazy operational implementations of how people deploy BGP in their networks will be the downfall of the Internet. I see on a daily basis, wrong announcements, route leaks tripping max-prefixes, RADB entries that are either totally out of date, completely wrong or for some large organisations they don't even have RADB entries. sBGP may [and probably will] help with some of that but its not a panacea. Regards, Neil. Who actually uses RADB to build filters other than Verio? While my experience with other providers is limited Verio is the only one (of the ones we have used) who used RADB entries for BGP peers. Overall it wasn't the best solution IMHO for a couple of reasons: - there was nothing to keep us from making bogus entries in the RADB - filters were only updated once a day making changes slow This is not meant as a complaint toward Verio - I'm simply trying to decide why we should go to the added expense of entering our routes in a RADB. To date I have seen no operational difference between using RADB and not using it. My view may very well be distorted by the fact that we are not a transit AS :-) Mark Radabaugh Amplex (419) 720-3635
Re: Who uses RADB? [was BGP to doom us all]
So, let's recap why no one uses them (as many have said already in the related thread): Laziness. The same laziness that results in the slew of other things many folks have pointed out not being addressed. -danny You forgot the other one - expense. AFAIK all of the registries have fees or require you to be a customer. If there is no operational value for me why would I want to spend the money? I realize most of you work for companies that consider a million dollars chump change but that is not the case everywhere. If you can give me a convincing reason to register my routes in a RADB I will - but at this point I have yet to see it. What does a RADB tell you about a non-transit network that you can't see from BGP and WHOIS? There is no more security in RADB than there is in our current method of notifying our peers of the netblocks we are announcing. Mark Radabaugh Amplex (419) 720-3635
Re: Who uses RADB? [was BGP to doom us all]
It doesnt cost a million dollars to have access to a RR, its somewhat less! You pay for your domains you pay for your IPs you pay for your ASN you pay for your SSL, so why be shocked you pay a little for this too? And if everyone filters your prefixes that will be operational value enough to join! Correct. We pay for lots and lots of things - and there are about 30 other things I need NOW that cost $500. You've been reading this thread right? Those were the reasons and they were pretty good, if you dont you may get filtered eventually or have your routes hijacked. Eventually is not now - and given that you have a horrendous chicken and egg problem I don't see it happening anytime in even the remote future. I'll grant you that it would be nice to have it so that my routes can't be hijacked - but we are back to the same chicken and egg problem. I'm contributing to one end of it - but I'm not the hard one to convince here. It's the many thousands of others who don't read NANOG. Well you cant arbitrarily register routes to them, you have to be a member, and have to match the authorisation criteria. I use RIPE and you have to be authorised on both the ASN and the INETNUM objects to register the route for it. True enough. And to get my BGP peers to accept my routes I have to do the exact same thing by communicating with them - not just changing entries in the RADB. If I want to launch a malicious attack both methods leave trails - but I'm willing to bet that it's a lot more likely that a person reviewing my request at a BGP peer will catch me before an automated system. Even if you compromise my routers it still doesn't allow you to announce anything interesting from me - you still have to convince my upstream providers to accept the announcements based on the current system of manually entered prefixes. We have had our routes registered in RADB in the past but despite the theory that it is laziness we dropped it due to expense and lack of relevence. I'll probably register our routes again but until RADB becomes a requirement of the RIR's or someone with authority I rather suspect this is a dead end. Steve Mark
SSL crack in the news
http://www.cnn.com/2003/TECH/internet/02/21/email.encryption.reut/index.html Very little real information... Mark Radabaugh Amplex (419) 720-3635
Public thanks to UUNet security
Since the good things so rarely get mentioned... I would like to publicly thank UUNet's network operations for dealing with a DOS attack quickly and efficiently yesterday. I am happy to say it only required one phone call of less than 15 minutes to get the appropriate filtering in place. Mark Radabaugh Amplex (419) 720-3635
Re: dontaing bgp config files [Re: Risk of Internet collapse grows]
if you run a network that has choices to make (more than one BGP speaking neighbor), you can help us by donating your bgp config files. abstracted or anonymized versions are ok. http://www.cs.washington.edu/research/networking/policy-inference/donation.h tml I'm not sure if you want the bgp tables or the configuration itself. In any case http://www.renesys.com/ currently has a project to collect bgp views - you may wish to talk to them. There are also many public route-servers that can give you a good view of the routing tables. http://www.caida.org has done a lot of work on analysis of BGP Mark Radabaugh Amplex (419) 720-3635
PM3's crashing
Anyone seeing odd crashes on Lucent PM3's tonight? We have boxes dying all over the network with hard lockups. The machines are in different physical locations with different telcos. It smells alot like a DOS of some sort. Mark Radabaugh Amplex (419) 720-3635
RE: routing table size
Until then, my money is on clueless redist connected/statics, large cable/dsl providers who announce a /24 per pop/city/whatever to their single transit provider, and general ignorance. Why attribute to functionality what can easily be explained by incomptence. :) -- Richard A Steenbergen [EMAIL PROTECTED] You forgot one of my favorite frustrations - slow start. Try this: a) start an ISP and tell your upstream you want a /21. They will tell you that you can only have a /24. b) Tell them that you understand they can't give you a /21 based on ARIN guidelines but you would like them to reserve it for you. Listen to them laugh. c) Keep requesting more space as you need it while you grow. Tell them you want contiguous space. Listen to them laugh. Your choice is take a new discontinuous block or renumber the whole network. This would be why we announce 2 /22's and 2 /23's even though given contiguous space we could make a single announcement. Add in the $2500 cost of obtaining a ARIN allocation versus what are 'free' addresses from our upstreams and we will probably continue as-is for a while. Why does ARIN need $2500 for an entry in a database anyway? End result is we would like to make a single announcement. By being truthful in requesting address space based on the guidelines we end up with address space that is fragmented - so we make the extra announcements. I have not seen a statistic for non-transit AS's announcing multiple discontinuous prefixes - I suspect that there are a lot of them for the same reason we do it. Obviously you can't keep leaving big 'reserved' holes in your allocations to downstreams for potential growth. You can't expect a network to renumber everytime they need more space. I don't have a good answer to this problem nor do I expect one - it's just another reason why we have additional growth in the routing tables. Mark Radabaugh Amplex (419) 833-3635
Re: wcom overbilling
On Fri, 5 Jul 2002, Chris Beggy wrote: Wcom's overbilling will be investigated: Is there a single wcom customer on nanog that *hasn't* been overbilled? -Dan I really really shouldn't do this to myself but... Our UUNet invoice has been correct every month since the T1 circuit was installed about 8 months ago. Mark Radabaugh Amplex
Cross Connect Cost at Exchange point
What is the range of pricing for rack to rack Ethernet crossconnects at a peering location? This would be from a colocation rack to a (small local) public peering point within the building. Private replies please - I will summarize for the list. Thanks,
RE: Sorta OT: IP Multicasting
3.) Are there any examples of audio feeds that use Multicast I can test with? The University of Oregon makes KWAX (classical radio) available via IP multicast; if you are using IP/TV or a compatible client, select the session UO Presents KWAX Classical Radio), and we also have a variety of video sources up for folks to try. Regards, Joe www.on-the-i.com has four channels that multicast music. I prefer channel 2 but that's just me :-) Mark
RE: Perspective on ARIN allocations to non-American entities
I've searched the IANA and ICANN sites, and have found no justification for what appear to be ARIN allocations to foreign entities within 66.231. Two serious UCE/hacking attempt offenders are as follows: 66.231.64.0/20 GIGA-BLK-1 66.231.128.0/20 ECON-BLK-1 Why have these blocks apparently been allocated via ARIN? Am I missing something? Cheers, Brian GIGA-BLK-1 is in Columbia which last time I checked was in South America. From the ARIN website: We at the American Registry for Internet Numbers manage the Internet numbering resources for North and South America, the Caribbean, and sub-Saharan Africa. ECON-BLK-1 has a French address. I'm going to take a wild guess that it's for a network in Africa or the Caribbean that is administered by a French company. Mark
RE: Verio as an DS3 upstream provider - comments?
Subject: Verio as an DS3 upstream provider - comments? I'm looking into possibly using Verio as an additional upstream provider. If you have any experience with Verio, can you give me any feedback on them? Thanks. - mz Given the rapidly shrinking size of the geographic area served, the sale of all of the dialup business to Earthlink and the closure of many of the data centers I would not be surprised to see significant depeering with Verio in the near future. We had service with Verio for several years. Don't expect to be able to resolve BGP issues any time other than 9-5pm. First level dedicated access tech support is sometimes difficult to reach and can be extremely clueless. Billing is an utter basket case. I really like the part where Verio closed the Toledo POP and left town but is still billing us for service. I'm waiting for them to call and threaten to shut us off - that should be a fairly humorous conversation. The network itself was decent. Good connectivity and low latency but I wouldn't expect it to last much longer. Mark Radabaugh Amplex (419) 833-3635
RE: Problems with a black hole list in the netherlands
Were it not referenced by http://Relays.OsiruSoft.com./, It's his own personal block list - it's not meant for use by other ISP's. Ignore it - unless you really need to send them mail. I don't know of any other ISP's actually using it as a block list. Mark Radabaugh Amplex (419) 833-3635
