Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
Hi, Further to the initial research sent to NANOG, after discussions with a number of operators, we have compiled some recommendations on the handling of invalid AS4_PATH attributes. Any feedback on these recommendations is appreciated: As discussed on the IETF IDR list last month, there are concerns relating to the treatment of AS_CONFED_SET/SEQUENCE in AS4_PATH as described in RFC4893 [0]. Since the last post to that thread the situation has been made more urgent with the release of Cisco IOS 12.0(32)S12, which responds to malformed AS4_PATH attributes by sending a NOTIFICATION to the neighbour, and tearing down the BGP adjacency. This behaviour seems to be required by RFC4721 section 6.3, as there is no alternative error handling defined in RFC4893. As posted last Friday [1], and discussed on the IDR list, this strict implementation introduces a new attack vector by which a BGP session can be torn down due to a an attribute populated by a distant BGP neighbour. These malformed attributes have already been seen in the wild as a result of a error in Juniper's implementation of RFC4893. Following discussions with a number of operators, we have attempted to generate some recommendations relating to the behaviour that would be operationally most useful when treating the invalid data in the AS4_PATH optional transitive attribute. There are two cases to consider when an invalid AS4_PATH is received: (1) A path to the prefix is not already known from that neighbour. (2) A path to the prefix has already been learnt from that neighbour; In case (1) we recommend that the BGP speaker should discard the UPDATE and log the fact. The log entry should include the received AS_PATH and AS4_PATH to aid in debugging. In case (2) we recommend that the BGP speaker should treat the UPDATE as a withdrawal of existing path to the prefix. As per case (1) a log entry should be raised to indicate that this has occurred. It is quite possible that in both cases this behaviour may result in the BGP speaker no longer having a valid path to the destination. We foresee that this lack of a prefix in a BGP speaker's routing table may cause some operational load initially, however, we feel that this is acceptable, considering the alternate behaviours. Should a prefix be injected into the global table with an invalid AS4_PATH, and should the newly advertised (invalid) path be selected by all upstreams available to a given ASN then this ASN will lose reachability to the prefix. Whilst this can be abused we do not see this as more serious than the existing possibility of malicious injection and blackholing of a prefix by a 3rd party. As long as the rejection of paths due to invalid AS4_PATHs is clearly reported to the administrator the source of the problem can be clearly identified. We consider that attempting to extract a valid AS4 or AS_PATH from the invalid UPDATE is a mistake since this allows the propagation of invalid BGP data. In addition, incorrect implementation of this comparatively complex mechanism by a vendor may result in loops. By explicitly not installing prefixes with invalid AS_PATH or AS4_PATH into the routing table, the possibility of loops caused by these invalid paths is avoided. The defined behaviour in RFC4893 and RFC4271 has significantly harmful effects and it seems only by virtue of the fact that the implementations of many vendors do not strictly comply with the RFCs that this problem has not had the same impact for every vendor. At the current time, however, one cannot deploy a 4-byte capable Cisco IOS device, or an OpenBGP (current stable release) router into the global table, without risking teardown of a every session via which a global table is learnt. Further discussion of this issue would be much appreciated, as a common and consistent approach to rectifying the problem will benefit network operators far more than individual vendor implementing their own solution. Should a consensus be reached an update to the RFC is required in order to ensure that future implementations do not exhibit this harmful behaviour. Kind regards, Andy Davidson (NetSumo), andy.david...@netsumo.com Jonathan Oddy (HostWay), jonathan.o...@hostway.co.uk Rob Shakir (GX Networks), r...@eng.gxn.net [0]: http://www.ietf.org/mail-archive/web/idr/current/msg03368.html [1]: http://www.merit.edu/mail.archives/nanog/msg14345.html Many thanks to David Freedman (Claranet) for assistance in developing the recommendations in this document.
Re: Inauguration streaming traffic
On Tue, Jan 20, 2009 at 12:38:11PM -0500, Christopher Morrow wrote: > On Tue, Jan 20, 2009 at 12:26 PM, Brian Wallingford wrote: > > On Tue, 20 Jan 2009, Jay Hennigan wrote: > > > > :We're a regional ISP, about 80% SMB 20% residential. We're seeing > > :almost double our normal downstream traffic right now. Anyone else? > > > > We're seeing traffic levels nearly 2x normal. On 9/11/01, we were > > probably only about 50% higher than the norm. Of course, lots has > > changed, so that's probably not a fair comparison. > > As an aside... thanks to BBC for streaming this, I couldn't find > another source that wasn't overloaded/jerky/ugly :( The Beeb's HD multicast feed is about 23Mbit/s to the host, and we received it at quite decent (subjective) quality here on a JANET-connected university site. The feed runs continuously (as far as any 'as-is' test stream does :) and this morning is pretty flawless. The interesting aspect of the HD stream was seeing how various systems coped with the CPU load. It was good to have some HD content available that encouraged people to install vlc, find out a little about multicast, and system issues in receiving it. Thanks again Beeb :) -- Tim
Re: DNS Amplification attack?
On 2009-01-21, Kameron Gasso wrote: > Christopher Morrow wrote: >> a point to bear in mind here is that... 'its working' is good enough >> for the bad folks :( no need to optimize when this works. Also, it's >> likely this isn't all of the problem the spoofed requestors are seeing >> these past few days :( > > Unfortunately, I can't restrict traffic to/from my authoritative > nameservers like I can with my recursive ones, since it will break DNS > resolution for outside visitors to domains we host. > > Fortunately, the spoofed queries are 60 bytes and my REFUSED responses > are only 59, so it's a terribly inefficient way to DoS someone. > However, I never said that the DDoS kiddies were smart - doesn't seem to > be stopping them from trying. :( > > Thanks, For you, yes. In many cases, there's either no amplification or a small decrease in traffic (though it makes it hard to shut off the true source). In a few cases (e.g. tinydns), there's no response, so the attackers traffic is wasted. But what about the people that happen to have misconfigured their authoritative DNS servers so that they're answering recursive queries from the world? 60 -> 520 bytes isn't bad, and I bet it's not _that_ uncommon...
Re: expectations for bgp peering?
On Tue, 20 Jan 2009, mike wrote: So I am just wondering what my expecations should be in a bgp peering scenario where I am multihomed with my own ASN and arin assigned ip space. At issue is the fact that my backup isp forced me to use ebgp multihop to peer with a router internal to their network and not the border router I am directly attached to, and secondly, that they say I am not allowed to prepend at all - they will do it for me, and from the looks of things they have established a route-map that just prepends their AS 6 times to my announcement. Assuming you're getting full routes from this provider, I wouldn't be surprised if the multihop is required because their router you're connected to doesn't have or can't handle full BGP routes. I've done this with customers in the past when they were connected to routers that didn't have the RAM for full routes. As for the "you're not allowed to prepend" thing, have you experimented to see what happens if you try? Unless they're giving you special pricing based on the idea that they're providing you with strictly backup transit, they shouldn't be doing the prepending (unless you've asked them to or used communities to tell them to). This smells of bad engineering. I have looked up the bgp report for my provider and they have 0 downstream AS's, and the week that this project has taken (and it's still not up and working) has left me with less than absolute confidence in the provider. I want to know if anyone has an opinion on ebgp Only a week? I've seen BGP turnups take much longer...not that they should. -- Jon Lewis | I route Senior Network Engineer | therefore you are Atlantic Net| _ http://www.lewis.org/~jlewis/pgp for PGP public key_
Re: DNS Amplification attack?
On Jan 20, 2009, at 6:31 PM, David W. Hankins wrote:
On Tue, Jan 20, 2009 at 12:54:32PM -0800, Wil Schultz wrote:
Anyone else noticing "." requests coming in to your DNS servers?
http://isc.sans.org/diary.html?storyid=5713
I was surprised to see 'amplification' in the subject line here, since
on my nameservers my replies are of equal length to the queries. A
little bit of asking around, and I see that it is an amplification
attack, preying on old software.
Let me sum up;
If you're running 9.4 or later, you will reply to these packets with
45 octet RCODE:Refused replies. 1:1. 9.4 has an "allow-query-cache"
directive that defaults to track allow-recursion, which you should
have set appropriately.
After reading this thread, I decided it was prudent to test my
authoritative nameservers & was surprised to discover I could retrieve
cached records from my nameserver even though I have "recursion no;"
in my options stanza in named.conf. Re-reading the ARM, I see that
behavior is expected. But is there some reason not to set "allow-
recursion { none; };" since I already have recursion disabled?
Thanks,
Dave Coulthart
WSJ on things to do in Santo Domingo
http://online.wsj.com/article/SB123240330058595471.html -- no idea if you have to be a subscriber or not. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: expectations for bgp peering?
On Jan 21, 2009, at 12:25 AM, mike wrote: Hello, So I am just wondering what my expecations should be in a bgp peering scenario where I am multihomed with my own ASN and arin assigned ip space. At issue is the fact that my backup isp forced me to use ebgp multihop to peer with a router internal to their network and not the border router I am directly attached to, and secondly, that they say I am not allowed to prepend at all - they will do it for me, and from the looks of things they have established a route- map that just prepends their AS 6 times to my announcement. Hmmm, this is distinctly unusual. I'd suspect that the person that you are talking to is a: very new to BGP and is just applying the wrong canned route-map or b: the person is a little less new to BGP and has reached the "Oooh, now I know that I'm doing and can twiddle the knobs with the best of them" stage. I'd suggest trying to find someone else there to talk to Unless you have specifically bought the service as a "backup" service (and they are clumsily (and poorly) trying to make sure that you don't use it as your primary path) I cannot think of why your ISP would do this. This also seems a bit worrying -- either they have enough capacity to carry your traffic when you need them to (and so should be happy to let you use them and bill you for the bits) or they don't and you will be unhappy when your primary goes away. Are they really really cheap? If you need a "backup" ISP for regulatory reasons and don't really care, thats fine. If however you want good performance when your primary goes away, I'd suggest looking into this more... This smells of bad engineering. I have looked up the bgp report for my provider and they have 0 downstream AS's, Yeah, that is worrying and the week that this project has taken (and it's still not up and working) has left me with less than absolute confidence in the provider. I want to know if anyone has an opinion on ebgp multihop for external customers, and wether I should really have an expectation to be able to assign my prepends as suits my needs? While multihop is not in itself an issue, it does give one pause and it is worth finding out the reason. But, yes, you should expect to be able to prepend at will... W Are there any conditions that could make this fail that I should be aware of? Mike- smime.p7s Description: S/MIME cryptographic signature
Re: expectations for bgp peering?
On Jan 21, 2009, at 8:17 AM, Jon Lewis wrote: As for the "you're not allowed to prepend" thing, have you experimented to see what happens if you try? Unless they're giving you special pricing based on the idea that they're providing you with strictly backup transit, they shouldn't be doing the prepending (unless you've asked them to or used communities to tell them to). See, this is why I like NANOG. Many eyes see things one pair does not. Hadn't even occurred to me that they were giving you special "backup transit only" pricing. In that case, makes perfect sense to force multiple prepends on their side. Thanx Jon. -- TTFN, patrick
Re: Inauguration streaming traffic
> The Beeb's HD multicast feed is about 23Mbit/s to the host, and we received > it at quite decent (subjective) quality here on a JANET-connected university > site. This is full broadcast HD, exactly the same as we have on satellite. We don't consider it generally usable, it's part of IPTV services not general Internet distribution. I expect we'll make a 1.5Mbit/s stream, not really HD just what people flog as Internet HD, some time. > The interesting aspect of the HD stream was seeing how various systems coped > with the CPU load. It was good to have some HD content available that > encouraged people to install vlc, find out a little about multicast, and > system issues in receiving it. That's why I leave it there but unpublicised as users would try getting it down their adsl and annoy their ISP. > Thanks again Beeb :) /me doffs cap brandon
Re: expectations for bgp peering?
Jon Lewis wrote: On Tue, 20 Jan 2009, mike wrote: Assuming you're getting full routes from this provider, I wouldn't be surprised if the multihop is required because their router you're connected to doesn't have or can't handle full BGP routes. There is a fairly large Tier 1 US provider who does exactly this. I imagine that this allows them to build out their network using cheap L3 (think 3550's or even 3560's) switches for their customer interfaces with a few "route servers" in core pops and a number of beefy border routers.
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
On Wed, Jan 21, 2009, Patrick W. Gilmore wrote: > Google is not the only company which will put caches into any provider > - or school (which is really just a special case provider) - with > enough traffic. A school with 30 machines probably would not > qualify. This is not being mean, this is just being rational. No way > those 30 machines save the company enough money to pay for the caches. > > Again, sux, but that's life. I'd love to hear your solution - besides > writing "magic" into squid to intentionally break or alter (some would > use much harsher language) content you do not own. Content others are > providing for free. Finding ways to force object revalidation by an intermediary cache (so the end origin server knows something has been fetched) and thus allowing the cache to serve the content on behalf of the content origintor, under their full control, but without the bits being served. I'm happy to work with content providers if they'd like to point out which bits of HTTP design and implementation fail them (eg, issues surrounding Variant object caching and invalidation/revalidation) and get them fixed in a public manner in Squid so it -can- be deployed by people to save on bandwidth in places where it still matters. Adrian
Re: expectations for bgp peering?
Patrick W. Gilmore wrote: > On Jan 21, 2009, at 8:17 AM, Jon Lewis wrote: > >> As for the "you're not allowed to prepend" thing, have you >> experimented to see what happens if you try? Unless they're giving >> you special pricing based on the idea that they're providing you >> with strictly backup transit, they shouldn't be doing the prepending >> (unless you've asked them to or used communities to tell them to). > > See, this is why I like NANOG. Many eyes see things one pair does not. > > Hadn't even occurred to me that they were giving you special "backup > transit only" pricing. In that case, makes perfect sense to force > multiple prepends on their side. > Good insight, Patrick. If I might suggest a point or two, it's that there's more than one "expectation" here, or perhaps should be: 1. Expectation on protocol/policy behavior 2. Expectation on service delivery and economics If breaking #1 doesn't break the basic functionality, but does achieve something under #2, it's worth clarifying. If #1 doesn't improve #2, there's a legitimate gripe. Means and ends aren't always locked together. >
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
Patrick W. Gilmore wrote: I do not work for GOOG or YouTube, I do not know why they do what they do. However, it is trivial to think up perfectly valid reasons for Google to intentionally break caches on YouTube content (e.g. paid advertising per download). Doesn't matter if you have small links or no infrastructure or whatever. Google has ever right, moral & legal, to serve content as they please. They are providing the content for free, but they want to do it on their own terms. Seems perfectly reasonable to me. Do you disagree? This brings me back the peering problem - if network A's customer sends network B's server a small packet, and network B's server sends back a video, why should Network A be forced to pay the lion's share of the bandwidth costs to deliver Network B's video (and ads) to the viewer? Networks which send large amounts of content should do their best to reduce the bandwidth load on end-user networks whenever and where ever possible, by hot-potato routing, by allowing the content to be cached, etc. They can't do otherwise and also claim they "do no harm". Adrian, what did your contacts at Google say when you asked them how this policy was consistent with their Do No Harm motto? If you didn't ask, I suggest you go ask! jc
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
> policy was consistent with their Do No Harm motto? Google's motto is Do No Evil, not Do No Harm.
Traffic metrics and cost analysis....
Hey all, Can anyone point to a good power-point template/presentation for metric and cost analysis for routing? I will not plagiarize unless; I am given copyright permission :-). Seriously, anyone have one handy, I am under the press to complete a presentation before Friday morning. Thanks a million, Jay Murphy IP Network Specialist NM Department of Health ITSD - IP Network Operations Santa Fé, New México 87502 "We move the information that moves your world." Confidentiality Notice: This e-mail, including all attachments is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited unless specifically provided under the New Mexico Inspection of Public Records Act. If you are not the intended recipient, please contact the sender and destroy all copies of this message. -- This email has been scanned by the Sybari - Antigen Email System.
Re: isprime DOS in progress
On Tue, 2009-01-20 at 14:55 -0600, Todd T. Fries forwarded: > From: ISPrime Support > These are the result of a spoofed dns recursion attack against our servers. > The actual packets in question (the ones reaching your servers) do NOT > originate from our network as such there is no way for us to filter things > from our end. > If you are receiving queries from 76.9.31.42/76.9.16.171 neither of these > machines make legitimate outbound dns requests so an inbound filter of > packets to udp/53 from either of these two sources is perfect. > If you are receiving queries from 66.230.128.15/66.230.160.1 these servers > are authoritative nameservers. Please do not blackhole either of these IPs as > they host many domains. However, these IPs do not make outbound DNS requests > so filtering requests to your IPs from these ips with a destination port of > 53 should block any illegitimate requests. I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question. Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away. Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic? Even if I used a REJECT policy, I'd expect the ICMP messages to go back to the appropriate - as in real - hosts, rather than the spoofing sources. Something here is very odd, very odd indeed... or I'm being dumb. It's happened before. Graeme
Re: BGP Session Teardown due to AS_CONFED_SEQUENCE in AS4_PATH
Hi, folks: We (IDR Chairs and co-authors) are working on updating RFC 4893 regarding the handling of the confed related segments in the AS4_PATH attribute. Expect to have the revised draft this week. Thanks.-- Enke Rob Shakir wrote: Hi, Further to the initial research sent to NANOG, after discussions with a number of operators, we have compiled some recommendations on the handling of invalid AS4_PATH attributes. Any feedback on these recommendations is appreciated: As discussed on the IETF IDR list last month, there are concerns relating to the treatment of AS_CONFED_SET/SEQUENCE in AS4_PATH as described in RFC4893 [0]. Since the last post to that thread the situation has been made more urgent with the release of Cisco IOS 12.0(32)S12, which responds to malformed AS4_PATH attributes by sending a NOTIFICATION to the neighbour, and tearing down the BGP adjacency. This behaviour seems to be required by RFC4721 section 6.3, as there is no alternative error handling defined in RFC4893. As posted last Friday [1], and discussed on the IDR list, this strict implementation introduces a new attack vector by which a BGP session can be torn down due to a an attribute populated by a distant BGP neighbour. These malformed attributes have already been seen in the wild as a result of a error in Juniper's implementation of RFC4893. Following discussions with a number of operators, we have attempted to generate some recommendations relating to the behaviour that would be operationally most useful when treating the invalid data in the AS4_PATH optional transitive attribute. There are two cases to consider when an invalid AS4_PATH is received: (1) A path to the prefix is not already known from that neighbour. (2) A path to the prefix has already been learnt from that neighbour; In case (1) we recommend that the BGP speaker should discard the UPDATE and log the fact. The log entry should include the received AS_PATH and AS4_PATH to aid in debugging. In case (2) we recommend that the BGP speaker should treat the UPDATE as a withdrawal of existing path to the prefix. As per case (1) a log entry should be raised to indicate that this has occurred. It is quite possible that in both cases this behaviour may result in the BGP speaker no longer having a valid path to the destination. We foresee that this lack of a prefix in a BGP speaker's routing table may cause some operational load initially, however, we feel that this is acceptable, considering the alternate behaviours. Should a prefix be injected into the global table with an invalid AS4_PATH, and should the newly advertised (invalid) path be selected by all upstreams available to a given ASN then this ASN will lose reachability to the prefix. Whilst this can be abused we do not see this as more serious than the existing possibility of malicious injection and blackholing of a prefix by a 3rd party. As long as the rejection of paths due to invalid AS4_PATHs is clearly reported to the administrator the source of the problem can be clearly identified. We consider that attempting to extract a valid AS4 or AS_PATH from the invalid UPDATE is a mistake since this allows the propagation of invalid BGP data. In addition, incorrect implementation of this comparatively complex mechanism by a vendor may result in loops. By explicitly not installing prefixes with invalid AS_PATH or AS4_PATH into the routing table, the possibility of loops caused by these invalid paths is avoided. The defined behaviour in RFC4893 and RFC4271 has significantly harmful effects and it seems only by virtue of the fact that the implementations of many vendors do not strictly comply with the RFCs that this problem has not had the same impact for every vendor. At the current time, however, one cannot deploy a 4-byte capable Cisco IOS device, or an OpenBGP (current stable release) router into the global table, without risking teardown of a every session via which a global table is learnt. Further discussion of this issue would be much appreciated, as a common and consistent approach to rectifying the problem will benefit network operators far more than individual vendor implementing their own solution. Should a consensus be reached an update to the RFC is required in order to ensure that future implementations do not exhibit this harmful behaviour. Kind regards, Andy Davidson (NetSumo), andy.david...@netsumo.com Jonathan Oddy (HostWay), jonathan.o...@hostway.co.uk Rob Shakir (GX Networks), r...@eng.gxn.net [0]: http://www.ietf.org/mail-archive/web/idr/current/msg03368.html [1]: http://www.merit.edu/mail.archives/nanog/msg14345.html Many thanks to David Freedman (Claranet) for assistance in developing the recommendations in this document.
Re: isprime DOS in progress
Hello, Representing ISPrime here. This attack has been ongoing on 66.230.128.15/66.230.160.1 for about 24 hours now, and we are receiving roughly 5Gbit of attack packets from roughly 750,000 hosts. It's somewhat absurd to suggest that we are attacking our own nameservers, I assure you, we didn't spend many hours looking for your specific nameserver to start sending 10 requests per second for the root zone, and our nameservers serve many popular domains. Given the attack is still in progress, I can't really say much more publicly, but suffice to say, we're working on the situation. -Phil AS23393 On Jan 21, 2009, at 12:08 PM, Graeme Fowler wrote: On Tue, 2009-01-20 at 14:55 -0600, Todd T. Fries forwarded: From: ISPrime Support These are the result of a spoofed dns recursion attack against our servers. The actual packets in question (the ones reaching your servers) do NOT originate from our network as such there is no way for us to filter things from our end. If you are receiving queries from 76.9.31.42/76.9.16.171 neither of these machines make legitimate outbound dns requests so an inbound filter of packets to udp/53 from either of these two sources is perfect. If you are receiving queries from 66.230.128.15/66.230.160.1 these servers are authoritative nameservers. Please do not blackhole either of these IPs as they host many domains. However, these IPs do not make outbound DNS requests so filtering requests to your IPs from these ips with a destination port of 53 should block any illegitimate requests. I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question. Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away. Something smells "not quite right" here - if the traffic is spoofed, and my "Refused" responses have been flying right back to the *real* IP addresses, how are the spoofing hosts to know that I'm dropping the traffic? Even if I used a REJECT policy, I'd expect the ICMP messages to go back to the appropriate - as in real - hosts, rather than the spoofing sources. Something here is very odd, very odd indeed... or I'm being dumb. It's happened before. Graeme
RE: isprime DOS in progress
-Original Message-
From: Graeme Fowler [mailto:gra...@graemef.net]
Sent: Wednesday, January 21, 2009 11:08 AM
To: Nanog Mailing list
Subject: Re: isprime DOS in progress
> I've been seeing a lot of noise from the latter two addresses after
> switching on query logging (and finishing an application of Team Cymru's
> excellent template) so I decided to DROP traffic from the addresses
> (with source port != 53) at the hosts in question.
> Well, blow me down if they didn't completely stop talking to me. Four
> dropped packets each, and they've gone away.
> Something smells "not quite right" here - if the traffic is spoofed, and
> my "Refused" responses have been flying right back to the *real* IP
> addresses, how are the spoofing hosts to know that I'm dropping the
> traffic?
>
> Even if I used a REJECT policy, I'd expect the ICMP messages to go back
> to the appropriate - as in real - hosts, rather than the spoofing
> sources.
>
> Something here is very odd, very odd indeed... or I'm being dumb. It's
> happened before.
>
> Graeme
In looking at my query logs I am seeing only requests from 66.230.160.1 and
66.230.128.15 so I've done the same thing with iptables and the rules are
resulting in an ever growing number of packets being dropped.
# iptables -nvL | grep -F -B 1 -A 1 66.230.160.1 | awk '{ print
$1,$2,$3,$8,$10,$11,$12 }'
pkts bytes target source
49517 2228K DROP 66.230.160.1 udp spt:!53 dpt:53
35905 1616K DROP 66.230.128.15 udp spt:!53 dpt:53
Re: 2 services have disappeared
On Sat, Jan 17, 2009 at 11:47 AM, Henry Linneweh wrote: > http://www.networkthinktank.com/ > http://www.completewhois.com > > are there any replacement services for these vanished services? > > -henry > I have been using http://www.robtex.com/ with success. -- Timothy G. O'Brien, CISSP, GSEC, NSA-IAM Irish dot MASMS at gmail dot com HTTP://www.linkedin.com/in/obrientg [snipped - FAQ for email trimming: http://kimihia.org.nz/articles/email/] Discussion Groups: http://www.hoax-slayer.com/discussion-groups.html Discussion List Etiquette: http://www.acfichapter.com/subscribe.htm Properly Formatted Email Replies for the Lazy: http://email.about.com/cs/netiquettetips/qt/et052704.htm Mailing List Etiquette FAQ: http://www.gweep.ca/~edmonds/usenet/ml-etiquette.html RFC 1855: Netiquette Guidelines: http://www.faqs.org/rfcs/rfc1855.html Miss Mailers Answers Your Questions on Mailing Lists: http://www.faqs.org/faqs/mail/miss-mailers/ and of course the all knowing Google: http://www.google.com/search?hl=en&q=mailing+list+etiquette -- A: No. Q: Should I include e-mail quotations after my reply? = An often repeated quote on news.admin.net-abuse.email: "Spam is not about content, it is about consent".
Re: isprime DOS in progress, and Re: DNS Amplification attack?
Can't some upstream provider find a source of the DNS NS . questions that is other than isprime?
Re: isprime DOS in progress
On Wed, 21 Jan 2009, Phil Rosenthal wrote: This attack has been ongoing on 66.230.128.15/66.230.160.1 for about 24 hours now, and we are receiving roughly 5Gbit of attack packets from roughly 750,000 hosts. I'm only receiving NS queries for "." from spoofed 66.230.128.15 and 66.230.160.1 via above.net (of my three transit providers) and none from peering. This usually indicates a single source, such as one rooted machine on non-BCP38 net spewing most of a gigabit. Given the attack is still in progress, I can't really say much more publicly, but suffice to say, we're working on the situation. Have you had any luck tracking back the source of the spoofed packets?If me talking to above.net sounds useful, let me know. -- Aaron
Re: isprime DOS in progress
Graeme Fowler wrote: On Tue, 2009-01-20 at 14:55 -0600, Todd T. Fries forwarded: I've been seeing a lot of noise from the latter two addresses after switching on query logging (and finishing an application of Team Cymru's excellent template) so I decided to DROP traffic from the addresses (with source port != 53) at the hosts in question. Well, blow me down if they didn't completely stop talking to me. Four dropped packets each, and they've gone away. I've seen that behaviour in the past, but not this time? I've seen a few of these attacks bouncing off my nameservers recently, and when I add "DROP" rules to my firewall, the incoming traffic disappears soon after. But the most recent set (66.230.160.1 and 66.230.128.15) are still hammering away... -- Harald
Re: inauguration streams review
On Tue, 20 Jan 2009, Jack Carrozzo wrote: Cell networks held up reasonably well for voice, though SMS and MMS delivery times approached an hour during the event. Switch load in almost the entire US was higher than midnight on New Years (which is generally the highest load of the year). Our network has been preparing since June, and I assume likewise for others. Unfortunately for me Sprint did not seem to prepare or have enough capacity for Voice, SMS or Data access. No live Twitter blogging! While I was able to get a few (maybe 5 between 10am and 2pm) text messages out while standing near the Washington Monument, calls and data were an impossibility, and SMS only seemed to have capacity available during lulls in the Inaugural activity. It was disappointing as a customer -- I'm sure that, had the capacity been there, the revenue from that single event would have made a significant impact on any of the carrier's revenue, at least for the month. -Jack Carrozzo (Engineer at $large cell company whose policy doesn't allow me to specify) (Google spills the beans!) I'm curious if you can find out -- did the record traffic positively affect revenue for that period compared to last year at the same time, or even last week on the same day? And from a more technical standpoint, did your $large cell company put up temporary towers? I'm curious as to how your company added capacity to handle the event, as well as how many "Network Busy" messages customers got, if any. I know I got more of those messages than I did successful communications. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: inauguration streams review
I can't comment on revenue-generation, though access as a whole was quite high. We hardly had any voice IAs (Ineffective Attempts, or 'Busy' messages). Since data can be queued, the only thing that would cause data IAs are bad RF conditions - we had a TON of 'cell on wheels' in the area for the event so we had enough carrier space to cover it. In-network data response times were hardly affected, with switch loads well below 50%. In-network SMS were still getting to their destinations in under 5 seconds for the most part I don't have any numbers on MMS or mobile IP data at the moment, though I would have heard if something horrible had happened. I'm told that the out-of-network SMS queue was piling pretty high at one point, to delivery times up to an hour, though they all still got there. We can't control other network's switches obviously. This isn't trying to sound like an advertisement - *I'm* not affected either way if people sign up with us as I'm not in sales, however from my point of view it looks like we had the most solid network... Our guys were planning and setting things up since June. Cheers, -Jack Carrozzo On Wed, Jan 21, 2009 at 1:29 PM, Peter Beckman wrote: > On Tue, 20 Jan 2009, Jack Carrozzo wrote: > >> Cell networks held up reasonably well for voice, though SMS and MMS >> delivery times approached an hour during the event. Switch load in >> almost the entire US was higher than midnight on New Years (which is >> generally the highest load of the year). >> >> Our network has been preparing since June, and I assume likewise for >> others. > > Unfortunately for me Sprint did not seem to prepare or have enough > capacity for Voice, SMS or Data access. No live Twitter blogging! > > While I was able to get a few (maybe 5 between 10am and 2pm) text messages > out while standing near the Washington Monument, calls and data were an > impossibility, and SMS only seemed to have capacity available during lulls > in the Inaugural activity. > > It was disappointing as a customer -- I'm sure that, had the capacity been > there, the revenue from that single event would have made a significant > impact on any of the carrier's revenue, at least for the month. > >> -Jack Carrozzo >> (Engineer at $large cell company whose policy doesn't allow me to specify) > > (Google spills the beans!) I'm curious if you can find out -- did the > record traffic positively affect revenue for that period compared to last > year at the same time, or even last week on the same day? > > And from a more technical standpoint, did your $large cell company put up > temporary towers? I'm curious as to how your company added capacity to > handle the event, as well as how many "Network Busy" messages customers > got, if any. I know I got more of those messages than I did successful > communications. > > Beckman > --- > Peter Beckman Internet Guy > beck...@angryox.com http://www.angryox.com/ > --- >
Re: inauguration streams review
How many simultaneous connections can each COW handle? What kind of backhaul connections do they have? -Mike On Wed, Jan 21, 2009 at 10:49 AM, Jack Carrozzo wrote: > I can't comment on revenue-generation, though access as a whole was quite > high. > > We hardly had any voice IAs (Ineffective Attempts, or 'Busy' > messages). Since data can be queued, the only thing that would cause > data IAs are bad RF conditions - we had a TON of 'cell on wheels' in > the area for the event so we had enough carrier space to cover it. > > In-network data response times were hardly affected, with switch loads > well below 50%. In-network SMS were still getting to their > destinations in under 5 seconds for the most part I don't have any > numbers on MMS or mobile IP data at the moment, though I would have > heard if something horrible had happened. > > I'm told that the out-of-network SMS queue was piling pretty high at > one point, to delivery times up to an hour, though they all still got > there. We can't control other network's switches obviously. > > This isn't trying to sound like an advertisement - *I'm* not affected > either way if people sign up with us as I'm not in sales, however from > my point of view it looks like we had the most solid network... Our > guys were planning and setting things up since June. > > Cheers, > > -Jack Carrozzo > > On Wed, Jan 21, 2009 at 1:29 PM, Peter Beckman > wrote: > > On Tue, 20 Jan 2009, Jack Carrozzo wrote: > > > >> Cell networks held up reasonably well for voice, though SMS and MMS > >> delivery times approached an hour during the event. Switch load in > >> almost the entire US was higher than midnight on New Years (which is > >> generally the highest load of the year). > >> > >> Our network has been preparing since June, and I assume likewise for > >> others. > > > > Unfortunately for me Sprint did not seem to prepare or have enough > > capacity for Voice, SMS or Data access. No live Twitter blogging! > > > > While I was able to get a few (maybe 5 between 10am and 2pm) text > messages > > out while standing near the Washington Monument, calls and data were an > > impossibility, and SMS only seemed to have capacity available during > lulls > > in the Inaugural activity. > > > > It was disappointing as a customer -- I'm sure that, had the capacity > been > > there, the revenue from that single event would have made a significant > > impact on any of the carrier's revenue, at least for the month. > > > >> -Jack Carrozzo > >> (Engineer at $large cell company whose policy doesn't allow me to > specify) > > > > (Google spills the beans!) I'm curious if you can find out -- did the > > record traffic positively affect revenue for that period compared to > last > > year at the same time, or even last week on the same day? > > > > And from a more technical standpoint, did your $large cell company put > up > > temporary towers? I'm curious as to how your company added capacity to > > handle the event, as well as how many "Network Busy" messages customers > > got, if any. I know I got more of those messages than I did successful > > communications. > > > > Beckman > > > --- > > Peter Beckman Internet > Guy > > beck...@angryox.com > http://www.angryox.com/ > > > --- > > >
RE: inauguration streams review
Just curious on that note with COW .. did you have much security related problems setting up stuff nearby? -Original Message- From: Mike Lyon [mailto:mike.l...@gmail.com] Sent: Wednesday, January 21, 2009 1:52 PM To: Jack Carrozzo Cc: nanog@nanog.org Subject: Re: inauguration streams review How many simultaneous connections can each COW handle? What kind of backhaul connections do they have? -Mike On Wed, Jan 21, 2009 at 10:49 AM, Jack Carrozzo wrote: > I can't comment on revenue-generation, though access as a whole was quite > high. > > We hardly had any voice IAs (Ineffective Attempts, or 'Busy' > messages). Since data can be queued, the only thing that would cause > data IAs are bad RF conditions - we had a TON of 'cell on wheels' in > the area for the event so we had enough carrier space to cover it. > > In-network data response times were hardly affected, with switch loads > well below 50%. In-network SMS were still getting to their > destinations in under 5 seconds for the most part I don't have any > numbers on MMS or mobile IP data at the moment, though I would have > heard if something horrible had happened. > > I'm told that the out-of-network SMS queue was piling pretty high at > one point, to delivery times up to an hour, though they all still got > there. We can't control other network's switches obviously. > > This isn't trying to sound like an advertisement - *I'm* not affected > either way if people sign up with us as I'm not in sales, however from > my point of view it looks like we had the most solid network... Our > guys were planning and setting things up since June. > > Cheers, > > -Jack Carrozzo > > On Wed, Jan 21, 2009 at 1:29 PM, Peter Beckman > wrote: > > On Tue, 20 Jan 2009, Jack Carrozzo wrote: > > > >> Cell networks held up reasonably well for voice, though SMS and MMS > >> delivery times approached an hour during the event. Switch load in > >> almost the entire US was higher than midnight on New Years (which is > >> generally the highest load of the year). > >> > >> Our network has been preparing since June, and I assume likewise for > >> others. > > > > Unfortunately for me Sprint did not seem to prepare or have enough > > capacity for Voice, SMS or Data access. No live Twitter blogging! > > > > While I was able to get a few (maybe 5 between 10am and 2pm) text > messages > > out while standing near the Washington Monument, calls and data were an > > impossibility, and SMS only seemed to have capacity available during > lulls > > in the Inaugural activity. > > > > It was disappointing as a customer -- I'm sure that, had the capacity > been > > there, the revenue from that single event would have made a significant > > impact on any of the carrier's revenue, at least for the month. > > > >> -Jack Carrozzo > >> (Engineer at $large cell company whose policy doesn't allow me to > specify) > > > > (Google spills the beans!) I'm curious if you can find out -- did the > > record traffic positively affect revenue for that period compared to > last > > year at the same time, or even last week on the same day? > > > > And from a more technical standpoint, did your $large cell company put > up > > temporary towers? I'm curious as to how your company added capacity to > > handle the event, as well as how many "Network Busy" messages customers > > got, if any. I know I got more of those messages than I did successful > > communications. > > > > Beckman > > > --- > > Peter Beckman Internet > Guy > > beck...@angryox.com > http://www.angryox.com/ > > > --- > > > "The information transmitted is intended only for the person or entity to which it is addressed and contains confidential and/or privileged material. If you received this in error, please contact the sender immediately and then destroy this transmission, including all attachments, without copying, distributing or disclosing same. Thank you."
Re: inauguration streams review
COWs are more or less full sites - so standard N concurrent voice calls per carrier (check out the CDMA standard if you're really interested), times the number of carriers. They can do 850+PCS all carrier if configured that way. If we can grab fiber from a nearby building that's best (hence why this takes so long to plan), however a lot of time we rely on OC3 microwave backhaul. I wasn't involved with the DC guys as I'm in Boston so I don't know specifics of this event. Re: security, I don't know since I wasn't involved though since all the planning started so far back I doubt there was much issue. -Jack Carrozzo On Wed, Jan 21, 2009 at 1:54 PM, Paul Stewart wrote: > Just curious on that note with COW .. did you have much security related > problems setting up stuff nearby? > > -Original Message- > From: Mike Lyon [mailto:mike.l...@gmail.com] > Sent: Wednesday, January 21, 2009 1:52 PM > To: Jack Carrozzo > Cc: nanog@nanog.org > Subject: Re: inauguration streams review > > How many simultaneous connections can each COW handle? What kind of > backhaul > connections do they have? > > -Mike > > > On Wed, Jan 21, 2009 at 10:49 AM, Jack Carrozzo > wrote: > >> I can't comment on revenue-generation, though access as a whole was > quite >> high. >> >> We hardly had any voice IAs (Ineffective Attempts, or 'Busy' >> messages). Since data can be queued, the only thing that would cause >> data IAs are bad RF conditions - we had a TON of 'cell on wheels' in >> the area for the event so we had enough carrier space to cover it. >> >> In-network data response times were hardly affected, with switch loads >> well below 50%. In-network SMS were still getting to their >> destinations in under 5 seconds for the most part I don't have any >> numbers on MMS or mobile IP data at the moment, though I would have >> heard if something horrible had happened. >> >> I'm told that the out-of-network SMS queue was piling pretty high at >> one point, to delivery times up to an hour, though they all still got >> there. We can't control other network's switches obviously. >> >> This isn't trying to sound like an advertisement - *I'm* not affected >> either way if people sign up with us as I'm not in sales, however from >> my point of view it looks like we had the most solid network... Our >> guys were planning and setting things up since June. >> >> Cheers, >> >> -Jack Carrozzo >> >> On Wed, Jan 21, 2009 at 1:29 PM, Peter Beckman >> wrote: >> > On Tue, 20 Jan 2009, Jack Carrozzo wrote: >> > >> >> Cell networks held up reasonably well for voice, though SMS and MMS >> >> delivery times approached an hour during the event. Switch load in >> >> almost the entire US was higher than midnight on New Years (which > is >> >> generally the highest load of the year). >> >> >> >> Our network has been preparing since June, and I assume likewise > for >> >> others. >> > >> > Unfortunately for me Sprint did not seem to prepare or have enough >> > capacity for Voice, SMS or Data access. No live Twitter blogging! >> > >> > While I was able to get a few (maybe 5 between 10am and 2pm) text >> messages >> > out while standing near the Washington Monument, calls and data > were an >> > impossibility, and SMS only seemed to have capacity available > during >> lulls >> > in the Inaugural activity. >> > >> > It was disappointing as a customer -- I'm sure that, had the > capacity >> been >> > there, the revenue from that single event would have made a > significant >> > impact on any of the carrier's revenue, at least for the month. >> > >> >> -Jack Carrozzo >> >> (Engineer at $large cell company whose policy doesn't allow me to >> specify) >> > >> > (Google spills the beans!) I'm curious if you can find out -- did > the >> > record traffic positively affect revenue for that period compared > to >> last >> > year at the same time, or even last week on the same day? >> > >> > And from a more technical standpoint, did your $large cell company > put >> up >> > temporary towers? I'm curious as to how your company added > capacity to >> > handle the event, as well as how many "Network Busy" messages > customers >> > got, if any. I know I got more of those messages than I did > successful >> > communications. >> > >> > Beckman >> > >> > > --- >> > Peter Beckman > Internet >> Guy >> > beck...@angryox.com >> http://www.angryox.com/ >> > >> > > --- >> > >> > > > > > > > "The information transmitted is intended only for the person or entity to > which it is addressed and contains confidential and/or privileged material. > If you received this in error, please contact the sender immediately and then > destroy this transmission, including all attachments, without copying, > distributing or disclosing same. Thank yo
Re: DNS Amplification attack?
>>> On 1/20/2009 at 7:23 PM, Mark Andrews wrote:
> In message <20090121140825.xwdzd4p64kgwo...@web1.nswh.com.au>,
> j...@miscreant.or
> g writes:
>> > On Tue, Jan 20, 2009 at 9:16 PM, Kameron Gasso wro=
>> te:
>>
>> > We're also seeing a great number of these, but the idiots spoofing the
>> > queries are hitting several non-recursive nameservers we host - and only
>> > generating 59-byte "REFUSED" replies.
>> >
>> > Looks like they probably just grabbed a bunch of DNS hosts out of WHOIS
>> > and hoped that they were recursive resolvers.
>>
>> First post to this list, play nice :)
>>
>> Are you sure about this? I'm seeing these requests on /every/ =20
>> (unrelated) NS I have access to, which numbers several dozen, in =20
>> various countries across the world, and from various registries (.net, =20
>> .org, .com.au). The spread of servers I've checked is so random that =20
>> I'm wondering just how many NS records they've laid their hands on.
>>
>> I've also noticed that on a server running BIND 9.3.4-P1 with =20
>> recursion disabled, they're still appear to be getting the list of =20
>> root NS's from cache, which is a 272-byte response to a 61-byte =20
>> request, which by my definition is an amplification.
>
> BIND 9.3.4-P1 is past end-of-life.
>
> You need to properly set allow-query at both the option/view
> level and at the zone level to prevent retrieving answers
> from the cache in 9.3.x.
>
> option/view level "allow-query { trusted; };"
> zone level "allow-query { any; };"
>
> BIND 9.4.x and later have allow-query-cache make the
> configuration job easier. It also defaults to directly
> connected networks.
Another BIND-specific question since we're on the topic. I see
some of our authorative servers being hit with these spoofs, and
yes, the 9.3.5-P1 (that's what Sun supports in Solaris these
days) were sending back answers from the cache... but wait...
what cache?
The view the Internet gets only has our authorative zones. There
is no declaration for the root zone, master, slave, or hints.
How does BIND have the root cached in that view? Where did it
get it from? I guess it's hard coded somewhere?
Blocking this in the firewall. 1:0 amplification better than the
BIND fix, 1:1. But I'll get to the BIND fix anyway.
Re: DNS Amplification attack?
Once upon a time, Crist Clark said: > Another BIND-specific question since we're on the topic. I see > some of our authorative servers being hit with these spoofs, and > yes, the 9.3.5-P1 (that's what Sun supports in Solaris these > days) were sending back answers from the cache... but wait... > what cache? > > The view the Internet gets only has our authorative zones. There > is no declaration for the root zone, master, slave, or hints. > How does BIND have the root cached in that view? Where did it > get it from? I guess it's hard coded somewhere? BIND has had the hints compiled in for some time as a fall-back, but for an auth-only server, "additional-from-cache no;" will kill such responses. -- Chris Adams Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Re: isprime DOS in progress
On Wed, 2009-01-21 at 12:27 -0500, Phil Rosenthal wrote: > Representing ISPrime here. Well... representing myself and nobody else, so if that stretches my credibility thin so be it. > It's somewhat absurd to suggest that we are attacking our own > nameservers, I assure you, we didn't spend many hours looking for your > specific nameserver to start sending 10 requests per second for the > root zone, and our nameservers serve many popular domains. I just checked to make sure I did not make that assertion. I did not. I observed something odd, and stated as much to see if anyone else did. I apologise if you read my message as insinuating what you stated, but I assure you that wasn't the intention. I did say "maybe I'm being dumb", and that is indeed the answer - I applied a temporary netfilter ruleset, then made it permanent - and it switched the DROP and LOG statements round so that... the packet got dropped first and the log statements never got hit. Schoolboy error (and interesting that someone else has observed this behaviour before!)... Normal service has been resumed. I should write a haiku here (sorry, MLC, poor joke). > Given the attack is still in progress, I can't really say much more > publicly, but suffice to say, we're working on the situation. In a previous job I've been on the receiving end of similar attacks so I have a large degree of understanding of the pressure you're under at the moment. I wish you the best of luck sorting it out. Graeme
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
On Jan 21, 2009, at 11:07 AM, Adrian Chadd wrote: On Wed, Jan 21, 2009, Patrick W. Gilmore wrote: Google is not the only company which will put caches into any provider - or school (which is really just a special case provider) - with enough traffic. A school with 30 machines probably would not qualify. This is not being mean, this is just being rational. No way those 30 machines save the company enough money to pay for the caches. Again, sux, but that's life. I'd love to hear your solution - besides writing "magic" into squid to intentionally break or alter (some would use much harsher language) content you do not own. Content others are providing for free. Finding ways to force object revalidation by an intermediary cache (so the end origin server knows something has been fetched) and thus allowing the cache to serve the content on behalf of the content origintor, under their full control, but without the bits being served. Excellent idea. It is a shame content owners do not see the utility in your idea. To bring this back to an operational topic, just because a content owner does not want to work with someone on this, does the lack of external bandwidth / infrastructure / whatever make it "OK" to install a proxy which will intentionally re-write the content? -- TTFN, patrick
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
> Excellent idea. It is a shame content owners do not see the utility > in your idea. > > To bring this back to an operational topic, just because a content > owner does not want to work with someone on this, does the lack of > external bandwidth / infrastructure / whatever make it "OK" to install > a proxy which will intentionally re-write the content? This really boils down to "who is more important? The content or the contents' eyeballs?" (Or the people having to deliver said content to said eyeballs, and aren't being paid by the content deliverer on their behalf.) Adrian
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
On 21/01/2009 21:30, Patrick W. Gilmore wrote: On Jan 21, 2009, at 11:07 AM, Adrian Chadd wrote: Finding ways to force object revalidation by an intermediary cache (so the end origin server knows something has been fetched) and thus allowing the cache to serve the content on behalf of the content origintor, under their full control, but without the bits being served. Excellent idea. It is a shame content owners do not see the utility in your idea. This doesn't provide feed-back to the content distributors on partial downloads, etc - which is useful information to content providers, if you're into data mining end-user browsing habits. In the specific case of Youtube, of course I don't know that they do this, but I'd be surprised if they didn't. Nick
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
On Jan 21, 2009, at 4:38 PM, Adrian Chadd wrote: Excellent idea. It is a shame content owners do not see the utility in your idea. To bring this back to an operational topic, just because a content owner does not want to work with someone on this, does the lack of external bandwidth / infrastructure / whatever make it "OK" to install a proxy which will intentionally re-write the content? This really boils down to "who is more important? The content or the contents' eyeballs?" (Or the people having to deliver said content to said eyeballs, and aren't being paid by the content deliverer on their behalf.) No, it does not. If I own something, it doesn't matter how "important" the people who want to buy it are. But I guess that's not operational. -- TTFN, patrick
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
On Wed, Jan 21, 2009, Nick Hilliard wrote: > This doesn't provide feed-back to the content distributors on partial > downloads, etc - which is useful information to content providers, if > you're into data mining end-user browsing habits. In the specific case of > Youtube, of course I don't know that they do this, but I'd be surprised if > they didn't. If they'd like that included as a side-channel for certain response types, then they could ask. Its not like caches don't store per-connection information like that already.. :) Adrian
Re: Inauguration streaming traffic
Interesting read on yesterday's streaming. My experiences seem to mirror a lot of what is written here: http://www.techcrunch.com/2009/01/21/the-day-live-web-video-streaming-failed-us/ -Jim P.
Re: DNS Amplification attack?
In message <497705bd.33e4.009...@globalstar.com>, "Crist Clark" writes:
> >>> On 1/20/2009 at 7:23 PM, Mark Andrews wrote:
>
> > In message <20090121140825.xwdzd4p64kgwo...@web1.nswh.com.au>,=20
> > j...@miscreant.or=20
> > g writes:
> >> > On Tue, Jan 20, 2009 at 9:16 PM, Kameron Gasso =
> wro=3D
> >> te:
> >>=20
> >> > We're also seeing a great number of these, but the idiots spoofing =
> the
> >> > queries are hitting several non-recursive nameservers we host - and =
> only
> >> > generating 59-byte "REFUSED" replies.
> >> >
> >> > Looks like they probably just grabbed a bunch of DNS hosts out of =
> WHOIS
> >> > and hoped that they were recursive resolvers.
> >>=20
> >> First post to this list, play nice :)
> >>=20
> >> Are you sure about this? I'm seeing these requests on /every/ =3D20
> >> (unrelated) NS I have access to, which numbers several dozen, in =3D20
> >> various countries across the world, and from various registries (.net, =
> =3D20
> >> .org, .com.au). The spread of servers I've checked is so random that =
> =3D20
> >> I'm wondering just how many NS records they've laid their hands on.
> >>=20
> >> I've also noticed that on a server running BIND 9.3.4-P1 with =3D20
> >> recursion disabled, they're still appear to be getting the list of =
> =3D20
> >> root NS's from cache, which is a 272-byte response to a 61-byte =3D20
> >> request, which by my definition is an amplification.
> >=20
> > BIND 9.3.4-P1 is past end-of-life.
> >=20
> > You need to properly set allow-query at both the option/view
> > level and at the zone level to prevent retrieving answers
> > from the cache in 9.3.x.
> >=20
> > option/view level "allow-query { trusted; };"
> > zone level "allow-query { any; };"
> >=20
> > BIND 9.4.x and later have allow-query-cache make the
> > configuration job easier. It also defaults to directly
> > connected networks.
>
> Another BIND-specific question since we're on the topic. I see
> some of our authorative servers being hit with these spoofs, and
> yes, the 9.3.5-P1 (that's what Sun supports in Solaris these
> days) were sending back answers from the cache... but wait...
> what cache?
Authoritative servers need a cache. Authoritative servers
need to ask queries. The DNS protocol has evolved since
RFC 1034 and RFC 1035 and authoritative servers need to
translate named to addresses for their own use.
See RFC 1996, A Mechanism for Prompt Notification of Zone
Changes (DNS NOTIFY).
> The view the Internet gets only has our authorative zones. There
> is no declaration for the root zone, master, slave, or hints.
> How does BIND have the root cached in that view? Where did it
> get it from? I guess it's hard coded somewhere?
>
> Blocking this in the firewall. 1:0 amplification better than the
> BIND fix, 1:1. But I'll get to the BIND fix anyway.
The real fix is to get BCP 38 deployed. Reflection
amplification attacks can be effective if BCP 38 measures
have not been deployed. Go chase down the offending
sources. BCP 38 is nearly 10 years old.
We all should be taking this as a opportunity to find where
the leaks are in the BCP 38 deployment and correct them.
Mark
--
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742 INTERNET: mark_andr...@isc.org
Re: DNS Amplification attack?
Mark Andrews writes: > Authoritative servers need a cache. Authoritative servers > need to ask queries. The DNS protocol has evolved since > RFC 1034 and RFC 1035 and authoritative servers need to > translate named to addresses for their own use. > > See RFC 1996, A Mechanism for Prompt Notification of Zone > Changes (DNS NOTIFY). if i had RFC 1996 to do over again i would either limit outbound notifies to in-zone servernames, or recommend that primary server operators configure stealth slaves for servername-containing zones, or (most likely) i would point out that the need to look up secondary servernames requires that an authority-only nameserver be able to act as a stub resolver and that such a server much have access to an independent recursive nameserver. it's not too late to implement it that way. no authority-only server should need a cache of any kind. the above text from marka represents a BIND implementatin detail, not a protocol requirement, evolved or not. > The real fix is to get BCP 38 deployed. Reflection > amplification attacks can be effective if BCP 38 measures > have not been deployed. Go chase down the offending > sources. BCP 38 is nearly 10 years old. my agreement with this statement is tempered by the fact that BCP38 deployment cannot be continuously assured, nor tested. therefore we will need protocols, implementations, and operational practices that take account of packet source address spoofing as an unduring property of the internet. > We all should be taking this as a opportunity to find where > the leaks are in the BCP 38 deployment and correct them. > > Mark yea, verily. and maybe track down rfc1918-sourced spew while you're at it. -- Paul Vixie
Re: Inauguration streaming traffic
Is there a general study done on the overall impact of inauguration streaming traffic ? any summary on what is the overall gain of bandwidth, etc.
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
Surely the whole point of this is that the end users (the eyeballs) get the best experience they can as they're the ultimate consumer. So working with everyone in the chain between the content owner and the eyeballs is important. If you're a content owner then you want the experience to be good so that either you sell more ads or that your "brand" (whatever that may mean) is well thought of. It's why content owners use CDNs - to ensure that it's delivered close to the end user. Surely that means, logically to me anyway, that working with caching providers local to the eyeballs is the next logical point. Certainly for the heavy bits that don't change (eg the video streams Adrian mentioned). It's a mutual benefit thing - if your content sucks for a school (for example) to use then they're not going to use it. That's not good for you as a content owner. I realise that CDNs probably aren't that keen on people caching as it reduces their revenue, but a level of being rational about helping the whole chain deliver means probably more traffic overall. MMC On 22/01/2009, at 8:13 AM, Patrick W. Gilmore wrote: (Or the people having to deliver said content to said eyeballs, and aren't being paid by the content deliverer on their behalf.) No, it does not. If I own something, it doesn't matter how "important" the people who want to buy it are. But I guess that's not operational. -- TTFN, patrick
Re: Inauguration streaming traffic
Arbor had a good writeup on the traffic that they saw. http://asert.arbornetworks.com/2009/01/the-great-obama-traffic-flood/ Regards, James Pleger On Jan 21, 2009, at 7:14 PM, Ong Beng Hui wrote: Is there a general study done on the overall impact of inauguration streaming traffic ? any summary on what is the overall gain of bandwidth, etc.
Re: "IP networks will feel traffic pain in 2009" (C|Net & Cisco)
On Thu, Jan 22, 2009, Matthew Moyle-Croft wrote: > I realise that CDNs probably aren't that keen on people caching as it > reduces their revenue, but a level of being rational about helping the > whole chain deliver means probably more traffic overall. I mean, I could extend an olive branch to all the CDNs out there and ask exactly what kind of extensions they'd like to see in intermediary caches so they can get the statistics they require, whilst allowing the edge to serve content on their behalf, but I doubt I'd get any bites. Oh well. Back to hacking on software so it can shuffle more bits. Adrian
