Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
Well, a particular router doesn't get to set its dampening according to its 'view' today, and that view is going to vary depending on prefix. I would like to argue that how we define flapping today is simply a broken concept. We count up/down/path change transitions, but such transitions can exist due to connectivity or implementation differences and may have nothing to do with not complying with ettiquette. If people were damped for a single up/down, then yes, that's a problem. Dampening someone because they have a single path change is not the idea. That would sort of nullify the spirit of being multihomed with BGP, or even using BGP in any production or transit network for that matter. IF there's a connection problem, or implementation difference that makes a lot of up/down, then dampening could occur close to the problem but it will be contained close, and won't spread to the rest of the internet. I submit that a better way of measuring flap is to look at the period over which a particular prefix is flapping. Anti-social behavior is many changes occurring over long periods of time. However, many changes over a short period may well not be a problem. Regards, Tony So configure your route dampening that way. No one's twisting your arm to use the defaults. Want many flaps in the short term to not be damped? Simply increase your suppress limit. Way way up. Increase or decrease your half-life as you see fit. Just don't expect the rest of the world to do the same. But do expect to see others damp the flapping that you leak to their network. gut feeling If no one on the net ran dampening, routers wouldn't get anything done except processing constant churn. /gut feeling I'm grateful that routers out there in the world are doing route-damp so my router's CPU doesn't have to deal with needless route-churn, even if it does impact my connectivity (to poorly connected) end sites somewhat. The stability is worth it. Another item to consider, is this is a sentence from RFC2439 By damping their own routing information, providers can reduce their own need to make requests of other providers to clear damping state after correcting a problem. Using that logic, I'm also grateful for my transit providers dampening the advertisements to their peers, of my flapping transit links to them. That way, when they flap, (hey, it happens to everyone eventually) it doesn't get me damped by the rest of the internet, and my connections to different transit providers can do their job of carrying the traffic. Dampening, in it's current state, really is a good idea. -Jerry
Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
On 21-dec-04, at 9:16, Jerry Pasker wrote: IF there's a connection problem, or implementation difference that makes a lot of up/down, then dampening could occur close to the problem but it will be contained close, and won't spread to the rest of the internet. Today's AS hierarchy is quite flat, which severely limits the usefulness of dampening. If the link between ASes A and B flaps, then B doesn't get to dampen these flaps. C, connected to D, does, but if C is a small network that doesn't help much as flap dampening brings its own overhead. In a two or three router network there probably isn't any advantage in dampening. Only when you get to protect a larger number of routers from the update, it helps. Now of course D, connected to C, will be isolated from the instability. But in today's internet, there often isn't a D. According to the weekly routing table report the current average AS path length is 4.5. Subtract at least .5 for prepending, and there must be a significant number of 3 or even 2 AS hop paths to get the average at 4.
Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
Jerry, i've been wondering, since most people aren't using a 25xx class router for bgp anymore, and the forwarding planes are able to cope more when 'bad things(tm)' happen, what the value of dampening is these days. ie: does dampening cause more problems than it tries to solve/avoid these days. - jared I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it. another point to consider is the number of affected routers. Yakov.
Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
An even more important consideration is whether our current paradigm of flap dampening actually is the behavior that we want to penalize. If a single link bounces just once, then thanks to our mesh, confederations, differing MRAI's etc., we can see many many changes to the AS path, resulting in dampening. Do we really want to inflict pain over such an incident? Tony Of course not. Dampening should be set on a router according to that router's views. The more chances a router can see a single event multiplied, the more lax dampening should be on that router. I think Rob Thomas's saying of know your network really applies here. In my book, the threat of dampening to anyone not playing nice is the true value of route dampening. Automatic enforcement of etiquette. -Jerry .
Re: verizon.net and other email grief
Aside from the general discussion I seem to have provoked - most of which may not be relevant to my problem anyway :( Has anyone any idea if @verizon.net email accounts will accept email from the UK ever again? Enquiring postmasters want to know. Still seeing intermittent issues with hotmail and BTConnect, but at least email is flowing to them most of the time.
Re: verizon.net and other email grief
Or just register our-name. and be done with it? Not so farfetched http://www.2idi.com Seems to me that DNS already has n+12 redundancy available. But the world is big enough to believe that perhaps 12 extra servers is not enough for everyone everywhere and that it may not be a trivial problem to figure out the right placement of those 12 servers within the topology. So, anycast provides the ability to move from n+12 to n+x. But we can not only increase the number of extra servers, we can also move them around trying to find the right number and the right places, without impacting any DNS configuration. Seems to me that anycast is a worthwhile thing and certainly justifies moving from n+12 to n+11 or n+10 in the DNS. But anycast should not be considered a replacement for the tried and tested DNS protocol redundancy. Maybe in 5 years or so, but not today. --Michael Dillon
Re: verizon.net and other email grief
On 14.12 09:39, Todd Vierling wrote: That's definitely true, though it can be used successfully -- if there's a very reliable kill-switch to withdraw the advertisement in a moment, or some kind of fallback mechanism in place to handle gross failures. Using this as the *only* remedy for unavailability of an anycast instance is insufficient given the speed at which bad news travels in BGP. You want to have the service available at multiple addresses with each of those engineered as differently as possible. Daniel
Re: verizon.net and other email grief
On 16-dec-04, at 12:52, Daniel Karrenberg wrote: That's definitely true, though it can be used successfully -- if there's a very reliable kill-switch to withdraw the advertisement in a moment, or some kind of fallback mechanism in place to handle gross failures. Using this as the *only* remedy for unavailability of an anycast instance is insufficient given the speed at which bad news travels in BGP. You want to have the service available at multiple addresses with each of those engineered as differently as possible. And that's exactly why UltraDNS' treatment of .org is evil. I really don't understand why people with .org domains aren't complaining louder about this. There are also other risks associated with anycast. See discussions on the IETF list earlier this year: about anycast + per packet load balancing ([dnsop] Re: Root Anycast (fwd), early october) and about root anycast (13 Root Server Limitation, may).
Re: verizon.net and other email grief
On Thu, 16 Dec 2004 10:33:27 PST, just me said: and be done with it? Look. Some folks think that $technology is a good solution for $application. Some don't. The great thing about teh internat is that differing solutions to common problems are embraced. Better solutions reap their rewards, and generally survive. So the competing .org provider deploys their better solution and survives, how, exactly? pgpijSTkYZe0W.pgp Description: PGP signature
Re: verizon.net and other email grief
On Thu, 16 Dec 2004 [EMAIL PROTECTED] wrote: On Thu, 16 Dec 2004 12:24:56 PST, just me said: So the competing .org provider deploys their better solution and survives, how, exactly? Are there not a variety of other registries? It's not a registry problem. % dig org. ns and ponder all the competition. is org the sole delegation from . [EMAIL PROTECTED]darwin The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
Re: verizon.net and other email grief
On Thu, 16 Dec 2004 12:24:56 PST, just me said: So the competing .org provider deploys their better solution and survives, how, exactly? Are there not a variety of other registries? It's not a registry problem. % dig org. ns and ponder all the competition. pgputEARn6nGr.pgp Description: PGP signature
Re: verizon.net and other email grief
On Thu, 16 Dec 2004 12:35:09 PST, just me said:
is org the sole delegation from .
If you're trying to register in .org, yes.
If you want to claim but the organization looking to register under .org
can go register under .com or .net or .biz, ask yourself why we bother having
TLD's at all? Why don't we all just register our-name.%{3randchars} and be done
with it? Or just register our-name. and be done with it?
pgp1s2jR46Bom.pgp
Description: PGP signature
Re: verizon.net and other email grief
Steve Gibbard wrote: On Thu, 16 Dec 2004, Iljitsch van Beijnum wrote: Having just two addresses is the main problem, the fact that they're also anycast just makes it even worse under certain circumstances. How does anycast make it worse? If both anycast routes converges to the same broken pod somehow(damping?). And troublshooting that when it only happens in AS sounds like it would be a bit more difficult. -Steve
Re: verizon.net and other email grief
If both anycast routes converges to the same broken pod somehow(damping?). And troublshooting that when it only happens in AS sounds like it would be a bit more difficult. That's not an anycast problem, that's just a misconfiguration. -Bill
Dampening considered harmful? (Was: Re: verizon.net and other email grief)
On Thu, Dec 16, 2004 at 01:43:25PM -0800, Bill Woodcock wrote: If both anycast routes converges to the same broken pod somehow(damping?). And troublshooting that when it only happens in AS sounds like it would be a bit more difficult. That's not an anycast problem, that's just a misconfiguration. i've been wondering, since most people aren't using a 25xx class router for bgp anymore, and the forwarding planes are able to cope more when 'bad things(tm)' happen, what the value of dampening is these days. ie: does dampening cause more problems than it tries to solve/avoid these days. - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
i've been wondering, since most people aren't using a 25xx class router for bgp anymore, and the forwarding planes are able to cope more when 'bad things(tm)' happen, what the value of dampening is these days. ie: does dampening cause more problems than it tries to solve/avoid these days. - jared I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it. I don't know about the rest of the network operators, but the threat of ever being dampened makes me much more careful with my network in general. My general practice is that if a transit line has problems, it's BGP session is put in an admin down state ASAP until the problems can be isolated and corrected. It is my humble little opinion that the threat of being damped for some period of time is a great hammer to hold over network operator's heads. I think it's a 'good thing(tm)' for automatic penalties to go in to effect for those who are irresponsible with the global routing table. Even if that means that though no fault of my own, one of my transit lines flaps, and gets me somewhat damped for an hour. -Jerry
Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
On Fri, Dec 17, 2004 at 12:42:21AM +0100, Iljitsch van Beijnum wrote: On 17-dec-04, at 0:21, Jerry Pasker wrote: ie: does dampening cause more problems than it tries to solve/avoid these days. I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it. One reason to be careful with dampening is that flaps can be multiplied. (Connect to routeviews and see the different flap counts under different peers for the same flap at your end to observe this.) There have been numerous people who have spoken and released research on this topic. I think with the better routing code out there these days, that most people can quickly handle a large number of next-hop changes, etc.. in their hw/sw that disabling dampening would allow the networks to reconverge fairly quickly without (much) trouble. (going to respond to the streaming video/audio/whatnot issue seperately). - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
On Thu, Dec 16, 2004 at 11:43:12PM -0500, Jared Mauch wrote: On Fri, Dec 17, 2004 at 12:42:21AM +0100, Iljitsch van Beijnum wrote: On 17-dec-04, at 0:21, Jerry Pasker wrote: ie: does dampening cause more problems than it tries to solve/avoid these days. I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it. One reason to be careful with dampening is that flaps can be multiplied. (Connect to routeviews and see the different flap counts under different peers for the same flap at your end to observe this.) There have been numerous people who have spoken and released research on this topic. I think with the better routing code out there these days, that most people can quickly handle a large number of next-hop changes, etc.. in their hw/sw that disabling dampening would allow the networks to reconverge fairly quickly without (much) trouble. (going to respond to the streaming video/audio/whatnot issue seperately). oops, i thought their reply was cc:ed to the list, but i guess not. i'm going to summarize a private reply i got: a streaming video provider saw that when dampening was enabled, it reduced the number of complaints related to rebuffering and other similar issues by 15%. I think that having a more stable/reliable network switchover to a forwarding path should happen at a more moderate rate.. Obviously, if a prefix is flapping badly via one connection, it should be less prefered, but if it's the only path to that prefix, what harm is caused by reinstalling it fairly quickly? (eg: after 60 seconds of stability instead of a lot longer). This would possibly create a few transient cases of lack of reachability that would be a bit harder to diagnose if max dampening timers were a lot lower and a route was still oscilating.. In the case of lack of reachability to an anycast prefix (i know that .org and ultradns were being picked on, so i'll use them as an example.. sorry Rodney :) ) having the shorter times would increase availability. Obviously the ideal case is that you would exempt these prefixes from dampening, but creating such policies may actually be more work for the router in the long term than just disabling dampening totally. you might see more transient cpu usage as the network goes flappity-flap, but the cost of not evaluating a more complex dampening policy might see some savings as well. (considering the cpu time spent to evaluate it, allocate memory to store state, count flaps, etc..) - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: verizon.net and other email grief
On Thu, 16 Dec 2004, Iljitsch van Beijnum wrote: And that's exactly why UltraDNS' treatment of .org is evil. I really don't understand why people with .org domains aren't complaining louder about this. Instead of re-starting this particular perennial thread, can we please just abbreviate it to an URL such as ftp://ftp.rfc-editor.org/in-notes/internet-drafts/draft-kook-anycast-is-evil-01.txt and be done with it? Look. Some folks think that $technology is a good solution for $application. Some don't. The great thing about teh internat is that differing solutions to common problems are embraced. Better solutions reap their rewards, and generally survive. I wonder how many folks perpetually arguing this point have ever actually implemented anycasted DNS service? In any case, I cry uncle. Can we just agree to disagree? matt ghali [EMAIL PROTECTED]darwin The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
Re: verizon.net and other email grief
On Thu, 16 Dec 2004, Iljitsch van Beijnum wrote: Having just two addresses is the main problem, the fact that they're also anycast just makes it even worse under certain circumstances. How does anycast make it worse? -Steve
Re: verizon.net and other email grief
On Thu, 16 Dec 2004 [EMAIL PROTECTED] wrote: On Thu, 16 Dec 2004 10:33:27 PST, just me said: and be done with it? Look. Some folks think that $technology is a good solution for $application. Some don't. The great thing about teh internat is that differing solutions to common problems are embraced. Better solutions reap their rewards, and generally survive. So the competing .org provider deploys their better solution and survives, how, exactly? Are there not a variety of other registries? matt ghali [EMAIL PROTECTED]darwin The only thing necessary for the triumph of evil is for good men to do nothing. - Edmund Burke
Re: Dampening considered harmful? (Was: Re: verizon.net and other email grief)
On 17-dec-04, at 0:21, Jerry Pasker wrote: ie: does dampening cause more problems than it tries to solve/avoid these days. I don't know what takes more router resources; dampening enabled doing the dampening calculations, or no dampening and constantly churning the BGP table. I would assume dampening generally saves router resources, or operators wouldn't chose to enable it. I generally don't use dampening in most setups, and continuous churning is rare these days, as far as I can tell. I seem to remember that it was mostly caused by bad implementations in the days that it was a big issue. The trouble with dampening is that it only works on stuff that happens beyond the routers your AS talks to. When your neighbors or your own stuff flap you don't get to dampen that. So I guess it's still useful for large networks that have a significant number of views on the same stuff, but it's not really worth the trouble for smaller networks. One reason to be careful with dampening is that flaps can be multiplied. (Connect to routeviews and see the different flap counts under different peers for the same flap at your end to observe this.)
Re: verizon.net and other email grief
On Friday 10 Dec 2004 5:26 pm, Rich Kulawiec wrote: When an incoming SMTP connection is made to one of Verizon's MX's, they allow it to proceed until the putative sender is specified, i.e. they wait for this part of the SMTP transaction: I don't think this is the issue, as we aren't getting that far in the transaction. Inspection suggests that the anycast announcements in the UK were pointing to a server that wasn't accepting email. I believe here the problem is using anycast, and not providing a backup system not using anycast. The previous case I'm aware of was when bits of the NE USA lost .org because they only had anycast DNS servers (and still do AFAIK), and the announcement messed up. Whilst I plead ignorant of the technical details of anycast, strikes me that it is clearly more complex, and thus more prone to failure, and these failures are potentially less obvious.
Anycast reliability (was: Re: verizon.net and other email grief)
On Mon, 13 Dec 2004, Simon Waters wrote: Inspection suggests that the anycast announcements in the UK were pointing to a server that wasn't accepting email. I believe here the problem is using anycast, and not providing a backup system not using anycast. The previous case I'm aware of was when bits of the NE USA lost .org because they only had anycast DNS servers (and still do AFAIK), and the announcement messed up. Whilst I plead ignorant of the technical details of anycast, strikes me that it is clearly more complex, and thus more prone to failure, and these failures are potentially less obvious. (for anybody reading this who doesn't know, anycast is multiple servers in multiple locations announcing routes and accepting connections to the same IP address). Off the top of my head, there are two major forms of DNS failures: - DNS servers stop responding - DNS servers respond with incorrect answers. Both of these are problems that can affect Anycast systems, but neither is unique to Anycast. In the first case, the DNS protocol has its own failover mechanism, and good anycast implementations also provide a failover mechanism. If a DNS server doesn't get a response from the first IP address it sends a DNS query to, it will send its query to another IP address listed as a DNS server for the domain it's trying to query. This process gets repeated until a working DNS server is found, or until it runs out of IP addresses of DNS servers to try. Anycast adds a second failover mechanism, whereby if a DNS server knows it's stopped responding, it can withdraw the announcement of its IP address, and routing protocols will redirect queries that it would have rejected to a different (hopefully working) server. Since the anycast routing protocol based failover mechanism can fail (for example, if a server stops answering queries but doesn't withdraw its routing announcements), it is important for there to be multiple IP addresses to query when looking for information on a domain. This allows DNS's internal failover mechanism to be used. However, this doesn't mean that the other listed DNS server IP addresses for a domain shouldn't also be addresses of anycast networks; they just shouldn't be part of the same anycast network. If .org has had failures (I don't know whether it has, and it's not an argument I want to wander into the middle of), it may be instructive to look at the differences between .org's anycast setup and that of the root servers. .org has two IP addresses listed, which I believe are for two different anycast clouds. That means that to get .org responses from any given location, at least one of two DNS servers must be working properly. There are thirteen root server IP addresses, most of which are at least somewhat anycasted. So, to get a response from the roots from any given location, at least one of thirteen servers needs to be doing the right thing. Again, that's not an issue of anycast vs. non-anycast. It's an issue of whether n+1 redundancy works as well as n+12 redundancy. The case being talked about here was the other sort of DNS failure -- a DNS server responding with incorrect information. That's a problem that becomes harder to notice as the number of DNS servers for a domain increases, but has nothing to do with whether those servers are in a anycast configuration. Indeed, Mr. Waters' suggested non-anycast backup would just add yet another server that could potentially get out of sync. Instead, this is the standard monitoring issue for big networks -- when a network gets big enough that failures won't be immediately obvious to a network's operator, there needs to be a monitoring system to detect failures. In the DNS case, the monitoring system should probably be trying DNS queries against all the servers, and making sure it gets consistent results. -Steve
Re: Anycast reliability (was: Re: verizon.net and other email grief)
On 13 Dec 2004, at 15:27, Steve Gibbard wrote: On Mon, 13 Dec 2004, Simon Waters wrote: Inspection suggests that the anycast announcements in the UK were pointing to a server that wasn't accepting email. I believe here the problem is using anycast, and not providing a backup system not using anycast. The previous case I'm aware of was when bits of the NE USA lost .org because they only had anycast DNS servers (and still do AFAIK), and the announcement messed up. Whilst I plead ignorant of the technical details of anycast, strikes me that it is clearly more complex, and thus more prone to failure, and these failures are potentially less obvious. (for anybody reading this who doesn't know, anycast is multiple servers in multiple locations announcing routes and accepting connections to the same IP address). Distribution of a service (whether by anycast or by some other means) is bound to introduce complexity over that incurred by a single instance of a service running in just one place. In some cases, the cost of that complexity is offset by reduced costs (or risk) elsewhere, and anycast makes sense. For a discussion of some of the issues surrounding service distribution using anycast, see: http://www.ietf.org/internet-drafts/draft-kurtis-anycast-bcp-00.txt Flames and projectiles relating to that draft would be very gratefully received (either directly or on the GROW list, but probably not on NANOG). Joe
Re: verizon.net and other email grief
On Mon, 13 Dec 2004 08:50:20 +, Simon Waters [EMAIL PROTECTED] wrote: Inspection suggests that the anycast announcements in the UK were pointing to a server that wasn't accepting email. First I've heard of Verizon using anycast Or do you mean anycast lookups of .net were broken from your part of the internet? --srs -- Suresh Ramasubramanian ([EMAIL PROTECTED])
Re: verizon.net and other email grief
Reply (*long* reply) being sent off-list. If anyone else wants to see it, rattle my cage. ---Rsk
verizon.net and other email grief
Hi, trying to pin down why so much email isn't making it recently. We see issues with various big ISPs. The most obvious is none of the three UK ISPs I have ready access to can connect to port 25 on relay.verizon.net. (MX for all the verizon.net email addresses). We can ping it (I'm sure it isn't singular?), but we have no more luck delivering email than contacting verizon technical staff, logs suggests we are in day 3 of this. I'm now listening to hold music at International rates - ouch. Have a support call with GXN open on connectivity to hotmail, which seems to vary with UK ISP, but is in general pretty poor from all of them. In general I'd say off topic in Nanog, but there comes a point where you start wondering if the routing is messed up. Am I alone before posting more details. Simon
Re: verizon.net and other email grief
On Fri, Dec 10, 2004 at 02:43:21PM +, Simon Waters wrote: The most obvious is none of the three UK ISPs I have ready access to can connect to port 25 on relay.verizon.net. (MX for all the verizon.net email addresses). We can ping it (I'm sure it isn't singular?), but we have no more luck delivering email than contacting verizon technical staff, logs suggests we are in day 3 of this. I'm now listening to hold music at International rates - ouch. I think I can shine a little bit of light on what might be your Verizon problem. Summary: Verizon has put in place an exceedingly stupid anti-spam system which does not work, which facilitates DoS attacks, and which provides active assistance to spammers. Verizon has been told all of this, and it's been discussed on Spam-L. If there's been a response from Verizon, I haven't seen it: and AFAIK the practice continues. Anyone trying to deliver mail there might want to at least skim this to get an idea of the issues they may bump into. Please note that in places this is sketchy because it seems impossible to get Verizon to provide the information necessary to make it otherwise (or correct any errors). Details: When an incoming SMTP connection is made to one of Verizon's MX's, they allow it to proceed until the putative sender is specified, i.e. they wait for this part of the SMTP transaction: MAIL From:[EMAIL PROTECTED] Then they pause the incoming connection. And then they start up an outbound SMTP connection from somewhere else on Verizon's network, back to one of the MX's for example.com. They then attempt to verify that blah is a valid, deliverable address there. Since most people have long since disabled SMTP VRFY, they actually construct a fake message and attempt delivery with RCPT. If delivery looks like it's going to succeed, they hang up this connection (which is rude), and un-pause the incoming one, and allow it to proceed. If delivery looks like it's going to fail, then they also hang up their outbound connection (still rude), un-pause the incoming one, and reject the traffic. This also means that if the MX they try to connect to is (a) busy (b) down (c) unaware of all the deliverable addresses (d) something else, that they'll refuse the incoming message. It also means that if the address that's trying to send mail to Verizon is something like [EMAIL PROTECTED], which is the address that the people at Thule Racks emit support traffic from, but which doesn't accept traffic, that Verizon will deny the message. (Yeah, this isn't very bright on Thule's part, either.) Whoops. This is bad for a whole bunch of reasons: two of the more obvious ones are (a) it's a pathetic anti-spam measure because ANY forged address ANYWHERE will do, and (b) it doesn't scale. Add to that (c) it abuses RCPT because apparently Verizon is unwilling to use VRFY and to accept the decision of many mail server operators to disable it. Oh, and (d) the behavior of their probe systems is nearly indistinguishable from that of spam-spewing zombies, which don't obey the SMTP protocol either. [ (b) is also how it lends itself to DoS attacks. Sure, Verizon could rate-limit the rate at which they make outbound connections, but then attacker X could impose significant delay on mail from domain Y just by forging a boatload of messages purporting to be from addresses in Y to Verizon. If Verizon rate-limits their outbound connections, then any real messages from Y will be stuck in the verification queue along with a kazillion forgeries. And beyond that: other people are foolishly adopting this callback nonsense as well. Slashdot carried a note the other day about a program _designed_ to do this. This allows attacker X to forge messages from domain Y to idiots I1, I2... In, for a very large n, and then stand back as all of them simultaneously try to connect to the MX's for domain Y. General principle: any anti-spam measure that generates more junk SMTP traffic at a time when we're drowning in it is probably a bad idea. ] One thing that's not clear is whether or not Verizon caches any of this information. Doing so might help cut down on DoS attack methods that involve them, but of course it doesn't do anything about those which leverage everyone else who's doing callbacks. And this is unfortunately, not the end of it. A lot of people, including me, are blocking particularly problematic spammer-controlled networks at (a) our border routers (b) our firewalls or (c) our mail servers. In other words, we not only won't accept mail from them, we won't even allow them to connect: we're blocking all IP traffic from them. This prevents them from spamming (at least directly from their own network space); it also prevents them from using their resources to build lists of deliverable addresses to sell to other spammers by poking
RE: verizon.net and other email grief
While I can't speak to what Verizon is using, Both Exim and Postfix have the very same feature called address verification. Its in use at a number of ISPs. My systems reject 1000's of messages every day because of verification failures. Roy Engehausen -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Rich Kulawiec Sent: Friday, December 10, 2004 9:27 AM To: [EMAIL PROTECTED] Subject: Re: verizon.net and other email grief On Fri, Dec 10, 2004 at 02:43:21PM +, Simon Waters wrote: The most obvious is none of the three UK ISPs I have ready access to can connect to port 25 on relay.verizon.net. (MX for all the verizon.net email addresses). We can ping it (I'm sure it isn't singular?), but we have no more luck delivering email than contacting verizon technical staff, logs suggests we are in day 3 of this. I'm now listening to hold music at International rates - ouch. I think I can shine a little bit of light on what might be your Verizon problem. Summary: Verizon has put in place an exceedingly stupid anti-spam system which does not work, which facilitates DoS attacks, and which provides active assistance to spammers. Verizon has been told all of this, and it's been discussed on Spam-L. If there's been a response from Verizon, I haven't seen it: and AFAIK the practice continues. Anyone trying to deliver mail there might want to at least skim this to get an idea of the issues they may bump into. Please note that in places this is sketchy because it seems impossible to get Verizon to provide the information necessary to make it otherwise (or correct any errors). Details: When an incoming SMTP connection is made to one of Verizon's MX's, they allow it to proceed until the putative sender is specified, i.e. they wait for this part of the SMTP transaction: MAIL From:[EMAIL PROTECTED] Then they pause the incoming connection. And then they start up an outbound SMTP connection from somewhere else on Verizon's network, back to one of the MX's for example.com. They then attempt to verify that blah is a valid, deliverable address there. Since most people have long since disabled SMTP VRFY, they actually construct a fake message and attempt delivery with RCPT. If delivery looks like it's going to succeed, they hang up this connection (which is rude), and un-pause the incoming one, and allow it to proceed. If delivery looks like it's going to fail, then they also hang up their outbound connection (still rude), un-pause the incoming one, and reject the traffic. This also means that if the MX they try to connect to is (a) busy (b) down (c) unaware of all the deliverable addresses (d) something else, that they'll refuse the incoming message. It also means that if the address that's trying to send mail to Verizon is something like [EMAIL PROTECTED], which is the address that the people at Thule Racks emit support traffic from, but which doesn't accept traffic, that Verizon will deny the message. (Yeah, this isn't very bright on Thule's part, either.) Whoops. This is bad for a whole bunch of reasons: two of the more obvious ones are (a) it's a pathetic anti-spam measure because ANY forged address ANYWHERE will do, and (b) it doesn't scale. Add to that (c) it abuses RCPT because apparently Verizon is unwilling to use VRFY and to accept the decision of many mail server operators to disable it. Oh, and (d) the behavior of their probe systems is nearly indistinguishable from that of spam-spewing zombies, which don't obey the SMTP protocol either. [ (b) is also how it lends itself to DoS attacks. Sure, Verizon could rate-limit the rate at which they make outbound connections, but then attacker X could impose significant delay on mail from domain Y just by forging a boatload of messages purporting to be from addresses in Y to Verizon. If Verizon rate-limits their outbound connections, then any real messages from Y will be stuck in the verification queue along with a kazillion forgeries. And beyond that: other people are foolishly adopting this callback nonsense as well. Slashdot carried a note the other day about a program _designed_ to do this. This allows attacker X to forge messages from domain Y to idiots I1, I2... In, for a very large n, and then stand back as all of them simultaneously try to connect to the MX's for domain Y. General principle: any anti-spam measure that generates more junk SMTP traffic at a time when we're drowning in it is probably a bad idea. ] One thing that's not clear is whether or not Verizon caches any of this information. Doing so might help cut down on DoS attack methods that involve them, but of course it doesn't do anything about those which leverage everyone else who's doing callbacks. And this is unfortunately, not the end of it. A lot of people, including me, are blocking
Re: verizon.net and other email grief
On Fri, 10 Dec 2004, Rich Kulawiec wrote: Verizon has put in place an exceedingly stupid anti-spam system which does not work, which facilitates DoS attacks, and which provides active assistance to spammers. The technique discussed is called callback verification and I do not agree that the technique stupid or provides assistance to spammers. I do agree that some of the aspects in how this was implemented by Verizon is not correct and causing problems. Details: When an incoming SMTP connection is made to one of Verizon's MX's, they allow it to proceed until the putative sender is specified, i.e. they wait for this part of the SMTP transaction: MAIL From:[EMAIL PROTECTED] Then they pause the incoming connection. And then they start up an outbound SMTP connection from somewhere else on Verizon's network, back to one of the MX's for example.com. They then attempt to verify that blah is a valid, deliverable address there. Since most people have long since disabled SMTP VRFY, they actually construct a fake message and attempt delivery with RCPT. If delivery looks like it's going to succeed, they hang up this connection (which is rude) If I were them I'd do RSET first and then QUIT instead of direct hangup , and un-pause the incoming one, and allow it to proceed. If delivery looks like it's going to fail, then they also hang up their outbound connection (still rude), un-pause the incoming one, and reject the traffic. Rejection of traffic is not approriate in this case, the greylisting technique should instead be used and they should reply back with 4xx error code (most likely 421 or 451), this allows for retry of email later on instead of outright denial. This also means that if the MX they try to connect to is (a) busy (b) down (c) unaware of all the deliverable addresses (d) something else, that they'll refuse the incoming message. All good reasons explaining why error code should 4xx instead of 5xx as I noted above. It also means that if the address that's trying to send mail to Verizon is something like [EMAIL PROTECTED], which is the address that the people at Thule Racks emit support traffic from, but which doesn't accept traffic, that Verizon will deny the message. (Yeah, this isn't very bright on Thule's part, either.) They are correct in this case. The address entered in RFC2821 MAIL FROM is Bounces-To address and it must accept bounced email and as such it must accept incoming emails. If the address does not accept traffic as you indicated should not be used in MAIL FROM and different adddress from local machine should be used. Please read email RFCs and then you'll understand that Envelope MAIL FROM (despite its name) is not an address that indicate source of email, the source of email address is actually From: header (or in some cases Sender header - which one to use will depend on your situation) This is bad for a whole bunch of reasons: two of the more obvious ones are (a) it's a pathetic anti-spam measure because ANY forged address ANYWHERE will do I agree that it requires verification that connecting machine has right to use given mail-from address as part of its sending email. See SPF. But for current situation it does work just fine and causes number of emails with randomly generated emails to be stopped. , and (b) it doesn't scale. The scalability depends on implementation. Since we have Verizon implementing it, I'm guessing it scales just fine based on the size of their email network. I'll also note that a lot of email is sent from the same address and as long as they cache results of verification it'll not be necessary to do verification each and every time. Add to that (c) it abuses RCPT because apparently Verizon is unwilling to use VRFY and to accept the decision of many mail server operators to disable it. Oh, and (d) the behavior of their probe systems is nearly indistinguishable from that of spam-spewing zombies, which don't obey the SMTP protocol either. Sometimes spammers probe various address before sending email to compile new list of email addresses and they use similar technique, however those are in fact distinguishable because you'll see them probing multiple non-existing address usually in alphabetical order. If email is sent from real working addresses, the callback verification would not look like a spammer probing technique. [ (b) is also how it lends itself to DoS attacks. Sure, Verizon could rate-limit the rate at which they make outbound connections, but then attacker X could impose significant delay on mail from domain Y just by forging a boatload of messages purporting to be from addresses in Y to Verizon. If Verizon rate-limits their outbound connections, then any real messages from Y will be stuck in the verification queue along with a kazillion forgeries. As with many techniques trying to address problems of current email
Re: verizon.net and other email grief
- Original Message - From: Roy [EMAIL PROTECTED] To: Rich Kulawiec [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Friday, December 10, 2004 2:23 PM Subject: RE: verizon.net and other email grief While I can't speak to what Verizon is using, Both Exim and Postfix have the very same feature called address verification. Its in use at a number of ISPs. My systems reject 1000's of messages every day because of verification failures. i've never seen this done with postfix, but i know that exim's default 'address verification' for non-local addresses just checks that the domain in the from is valid and that an mx record exists for it. they also have what they call 'callout verification', which is equivalent to what is being discussed, but the documentation makes the drawbacks painfully clear and suggests that it only be used against hosts within the same organization. i'm not a fan of exim, but it appears that although they've given users the rope, they've been diligent enough to label it appropriately. -p --- paul galyinin
Re: verizon.net and other email grief
--On Friday, December 10, 2004 12:30 -0800 Paul Trebilco [EMAIL PROTECTED] wrote: Christopher X. Candreva wrote: That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net. This doesn't scale. How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting. The other part is that you CACHE the answer you get (good, bad, or indifferent). I think that SPF+sender address verification is a GOOD thing when properly implemented. Yes it can be a bit of a hassle, but you shouldn't be sending mail you're not prepared to bounce. That said, none of my sites are running a current enough version of Postfix to do this.
Re: verizon.net and other email grief
--On Friday, December 10, 2004 15:38 -0500 Paul G [EMAIL PROTECTED] wrote: - Original Message - From: Paul Trebilco [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, December 10, 2004 3:30 PM Subject: Re: verizon.net and other email grief How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting. oh, so you would be ok with someone joe-jobbing you on their 1 million messages/day spam run and getting 1 million 'verification' connections to your mailserver farm? Far less traffic than the bounces would create at both ends. Yes this doesn't prevent it from happening if the address is real, but that's why I mentioned SPF in my previous email..That helps to verify the sender can send email for a given domain, and if that passes, then you want to see if the sender exists, if both pass then you can go on to other methods. OF course I'd first check blacklists before any of this, but that's my personal preference.
Re: verizon.net and other email grief
- Original Message - From: Paul Trebilco [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, December 10, 2004 3:30 PM Subject: Re: verizon.net and other email grief How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting. oh, so you would be ok with someone joe-jobbing you on their 1 million messages/day spam run and getting 1 million 'verification' connections to your mailserver farm? -p --- paul galynin
Re: verizon.net and other email grief
Paul G [EMAIL PROTECTED] wrote: [...] they also have what they call 'callout verification', which is equivalent to what is being discussed, but the documentation makes the drawbacks painfully clear and suggests that it only be used against hosts within the same organization. No, that caveat is given for *recipient callforward verification* which is dangerous if turned on blindly. I know, I tried it for a very short while :) i'm not a fan of exim, but it appears that although they've given users the rope, they've been diligent enough to label it appropriately. Sender callback verification is a different beast and is highly effective against spam. It does of course not come without its price of false positives caused by misconfigured senders. Unlike other mechanisms, it at least doesn't inconvenience senders who haven't botched their mail system. The only false positives I see are things like web sites that mail from a webserver role account which doesn't have a mailbox. Even so, ecommerce sites are learning to not do this, and ordered goods usually turn up regardless of whether or not an automatically-generated confirmation email arrives. -- PGP key ID E85DC776 - finger [EMAIL PROTECTED] for full key
Re: verizon.net and other email grief
on Fri, Dec 10, 2004 at 12:36:12PM -0800, william(at)elan.net wrote: On Fri, 10 Dec 2004, Rich Kulawiec wrote: Verizon has put in place an exceedingly stupid anti-spam system which does not work, which facilitates DoS attacks, and which provides active assistance to spammers. The technique discussed is called callback verification and I do not agree that the technique stupid or provides assistance to spammers. I do agree that some of the aspects in how this was implemented by Verizon is not correct and causing problems. snip But for current situation it does work just fine and causes number of emails with randomly generated emails to be stopped. Erm. Yeah, it stops them from being delivered to Verizon by shifting half the cost of verification onto the victims. , and (b) it doesn't scale. The scalability depends on implementation. Since we have Verizon implementing it, I'm guessing it scales just fine based on the size of their email network. See above. It doesn't scale when /everyone else/ starts doing it. Callback verification if properly implemented will never generate more junk SMTP traffic as DATA part of SMTP transmission never happens. By the time Verizon's callback servers hang up on us they've already generated more junk SMTP traffic, wasted our resources to protect their customers, and aided spammers doing list validation. Your claim that dictionary attacks are always alphabetical is pretty weak and brings nothing to bear on the actual problem - that by rejecting mail from a given address because of (possibly spurious) verification, they are actually giving the spammers a tool they can use to cull bad addresses from their own lists. The only positive thing I have to say about Verizon's callback scheme is that so far it has not been seen here more than 6 times in a single day in the past two months. So they must be doing some caching, given that at least one of the domains we host has been under joe job outscatter attack for several months running now. -- hesketh.com/inc. v: +1(919)834-2552 f: +1(919)834-2554 w: http://hesketh.com join us! http://hesketh.com/about/careers/account_manager.htmljoin us!
Re: verizon.net and other email grief
On Fri, 10 Dec 2004 12:36:12 PST, william(at)elan.net said: They are correct in this case. The address entered in RFC2821 MAIL FROM is Bounces-To address and it must accept bounced email and as such it must accept incoming emails. If the address does not accept traffic as you indicated should not be used in MAIL FROM and different adddress from local machine should be used. Please read email RFCs and then you'll Yes - *you* know that the address you put in the MAIL FROM: should be one that you know is willing to accept any bounces you may be about to generate, and *I* know that. The problem is that the spammer (I include A/V companies that still generate rejection notices) knows that as well - and doesn't feel the same need to follow the rules and be a good netizen. Yes, it's a royal pain trying to follow a protocol when you know up front that there's a 70% to 90% chance that the other person is intentionally failing to follow it pgpOpxPsgi6Lm.pgp Description: PGP signature
Re: verizon.net and other email grief
Krzysztof Adamski wrote: On Fri, 10 Dec 2004, Jeffrey I. Schiller wrote: On Fri, Dec 10, 2004 at 12:26:59PM -0500, Rich Kulawiec wrote: One thing that's not clear is whether or not Verizon caches any of this information. It appears that they do some amount of caching. -Jeff It does not appear that they are caching it, here is a sample from my log file: Dec 6 19:18:15 white sm-mta[25976]: iB70IF6n025976: [EMAIL PROTECTED]... User unknown Dec 6 19:18:15 white sm-mta[25977]: iB70IF6n025977: [EMAIL PROTECTED]... User unknown Dec 6 19:18:16 white sm-mta[25976]: iB70IF6n025976: from=, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182] Dec 6 19:18:16 white sm-mta[25977]: iB70IF6n025977: from=, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68] Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: [EMAIL PROTECTED]... User unknown Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: [EMAIL PROTECTED]... User unknown Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: lost input channel from sc006pub.verizon.net [206.46.170.182] to MTA after rcpt Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: from=[EMAIL PROTECTED], size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182] Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: lost input channel from sc019pub.verizon.net [206.46.170.68] to MTA after rcpt Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: from=[EMAIL PROTECTED], size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68] What happens when verizon tries to send email to somebody who does the same type of check, does this not create an infinite loop? Not if Verizon treats the antispam[0-9]+ mailboxes in a special manner and answers without a check. And they have to answer that the box exists or things are gonna _really_ break. Doing a quick test using the last antispam[0-9]+ address in my SMTP logs, I got all 250 responses without a more recent call back. -- Crist J. Clark [EMAIL PROTECTED] Globalstar Communications(408) 933-4387
RE: verizon.net and other email grief
On Fri, 10 Dec 2004, Roy wrote: While I can't speak to what Verizon is using, Both Exim and Postfix have the very same feature called address verification. Its in use at a number of ISPs. My systems reject 1000's of messages every day because of verification failures. That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net. This doesn't scale. == Chris Candreva -- [EMAIL PROTECTED] -- (914) 967-7816 WestNet Internet Services of Westchester http://www.westnet.com/
Re: verizon.net and other email grief
On Fri, Dec 10, 2004 at 12:26:59PM -0500, Rich Kulawiec wrote: One thing that's not clear is whether or not Verizon caches any of this information. It appears that they do some amount of caching. -Jeff
Re: verizon.net and other email grief
Christopher X. Candreva wrote: That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net. This doesn't scale. How so? Are you maybe confusing reject with bounce? If address verification takes place while the SMTP connection is still up, no forged adresses get messaged, at least not by the server doing the rejecting. -- Best Regards, Paul D Trebilco Systems Administrator Server101.com -- Fast and Reliable Website Hosting! http://www.server101.com/ signature.asc Description: OpenPGP digital signature
RE: verizon.net and other email grief
On Fri, 10 Dec 2004, Christopher X. Candreva wrote: That would be 1000's of other people's servers getting traffic from you because someone forged their address in the spam. You are effectively doubleing the total load spam places on the net. That is already what happens when spammer forged your address - you see 1000's people sending you bounces and nastygrams. The real solution is to use some form of protection for envelope mail-from address so that it could not be so easily spoofed and forged. There are currently several proposals on the table on how to do it and some of the proposals are already being used on the internet in experimental ways: SPF (dns records listing ips of mail systems that can send mail with given envelope mail-from domain). For more information see: http://spf.pobox.com http://www.openspf.org http://www.spfhelp.org CSV with MPR records (similar SPF but provides list of mail-server hostnames that can use MAIL-FROM domain, the spoofing of mail-server names is protected based on EHLO by CSV): For more information see: http://www.csvmail.com/draft-otis-marid-mpr-00.html (and for CSV see http://mipassoc.org/csv/index.html) BATV (replacement of your real mail-from address with special private connection-specific address - this allows to /dev/null bad bounces if they come back to you and you did not send the email). For more information see: http://mipassoc.org/batv/index.html SES (predates BATV and similar technique, except that a HMAC encrypted address can confirmed by means of public server which allows email to be dropped at recepient instead of dropped at the source as being bad bounce as with BATV). For more information see: http://ses.codeshare.ca/ -- William Leibzon Elan Networks [EMAIL PROTECTED]
Re: verizon.net and other email grief
On Fri, 10 Dec 2004, Jeffrey I. Schiller wrote: On Fri, Dec 10, 2004 at 12:26:59PM -0500, Rich Kulawiec wrote: One thing that's not clear is whether or not Verizon caches any of this information. It appears that they do some amount of caching. -Jeff It does not appear that they are caching it, here is a sample from my log file: Dec 6 19:18:15 white sm-mta[25976]: iB70IF6n025976: [EMAIL PROTECTED]... User unknown Dec 6 19:18:15 white sm-mta[25977]: iB70IF6n025977: [EMAIL PROTECTED]... User unknown Dec 6 19:18:16 white sm-mta[25976]: iB70IF6n025976: from=, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182] Dec 6 19:18:16 white sm-mta[25977]: iB70IF6n025977: from=, size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68] Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: [EMAIL PROTECTED]... User unknown Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: [EMAIL PROTECTED]... User unknown Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: lost input channel from sc006pub.verizon.net [206.46.170.182] to MTA after rcpt Dec 6 19:18:16 white sm-mta[25976]: iB70IF6o025976: from=[EMAIL PROTECTED], size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc006pub.verizon.net [206.46.170.182] Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: lost input channel from sc019pub.verizon.net [206.46.170.68] to MTA after rcpt Dec 6 19:18:16 white sm-mta[25977]: iB70IF6o025977: from=[EMAIL PROTECTED], size=0, class=0, nrcpts=0, proto=SMTP, daemon=MTA, relay=sc019pub.verizon.net [206.46.170.68] What happens when verizon tries to send email to somebody who does the same type of check, does this not create an infinite loop? K
Re: verizon.net and other email grief
On Fri, Dec 10, 2004 at 06:03:11PM -0500, Krzysztof Adamski wrote: It does not appear that they are caching it, here is a sample from my log file: ... Well when I tested it (3 hours ago) I connected to them manually while watching my incoming milter log. Indeed they visited immediate and verified my address (with an rcpt to command) and when that succeeded, I got the sender ok in my telnet session. I then broke the connection and tried again. The second time they did not contact my server and gave me a sender ok. It is now about three hours later and I just tried again and they did not contact my server. Now this is the success case where the mailbox exists. I don't know if they cache the non-existence of a mailbox. -Jeff -- = Jeffrey I. Schiller MIT Network Manager Information Services and Technology Massachusetts Institute of Technology 77 Massachusetts Avenue Room W92-190 Cambridge, MA 02139-4307 617.253.0161 - Voice [EMAIL PROTECTED]
