Re: [policy] When Tech Meets Policy...

[Sean Donelan]

On Mon, 13 Aug 2007, Chris L. Morrow wrote:

but today that provision is: If you buy a domain you have 5 days to
'return' it. The reason behind the return could be: "oops, I typo'd" or
"hurray, please refund me for the 1M domains I bought 4.99 days ago!". The
'protect the consumer' problem is what's enabling tasting.


So combine these ideas with the possibility that someone will claim 
various consumer protection laws apply to these transactions and want to 
cancel the contract within three days.


Instead, why don't we have a three day waiting period when the domain is
"reserved" but not active.   Grandma could notice her typo, credit card 
processor's could notice fake card numbers, and so on and rescind the 
registration.


After three days the sale is "final."  Only then the name is made active 
in the zone files.


Do people really not plan that far ahead, that they need brand new domain 
names to be active (not just reserved) within seconds?

Re: Industry best practices (was Re: large organization nameservers

[Sean Donelan]

Followups probably should go to the dnsops mailing list.

I got tired of things and went back to the original question, and put 
together my list of what the "minimum" packets needed for full DNS 
performance on the modern Internet.


It is the minimum, based on the security principle deny everything, allow
only what is needed. But "needed" is performance based. So it means not 
relying on fallbacks, timeouts or hoping no one complains. It does not 
include packets needed for diagnostic or troubleshooting information.
It is based on the "modern" Internet so does not included very deprecated 
packets like Source Quench or unimplemented functions like broadcast DNS

queries.

It does include current Internet practices for EDNS, Notify, global DNS 
load balancers and error handling I've seen in recent, i.e. less than 10 
years old, DNS, Router and OS software.


I didn't included TOS/DSCP and some military options, mainly because I'm 
not sure what "modern" military networks are currently using.  If you are
using TOS/DSCP or military options, there are some things you will need to 
add.

Industry best practices (was Re: large organization nameservers sending icmp packets to dns servers)

[Sean Donelan]

On Tue, 7 Aug 2007, Kevin Oberman wrote:

This has been a pain for me for years. I have tried to reason with
security people about this and, while they don't dispute my reasoning,
they always end up saying that it is the "standard" practice and that,
lacking any evidence of what it might be breaking, it will continue to
be blocked. And I don't mean small companies, either. One of the biggest
issues I have is with one of the countries largest government funded
research labs.


Having worked on both sides of the fence, i.e. I was a card-carrying 
member of both ASIS and NFPA, I used grumbled about the kooky things 
sysadmins and programmers did in the name of "security" as much as I 
grumbled about the kooky things security folks did in the name of 
"security." Heck, if programmers only produced bug-free software and 
sysadmins kept only well configured systems, security people would 
have a lot less work to do.


What are the industry best practices for keeping DNS servers secure?

CERT publishes a document on securing DNS:


NIST publishes a document on securing DNS:


CMYRU publishes a document on securing DNS:


Microsoft publishes a document on securing DNS:


IETF publishes a document on operational (including security) requirements 
for root DNS servers:



While there is a lot in common, they each also have variations and 
omissions.  Especially when it comes to some possibly obscure interactions
with many different protocols and applications. The relationships between 
IP, ICMP, TCP, UDP and DNS seems to be tough for many people to get 
right.  When you add undocumented "common knowledge" and other applications
leveraging DNS for all sorts of stuff besides name/address resolution, its 
the typical programmer generated pile of spaghetti.


Its often simplier to wait for something to break before you fix it. I 
know many sysadmins, programmers and even security people, that use that

philosphy to decide which things to work on today.


The good thing about security folks (and their cousins, the auditors) is 
most are compliance driven.  So if you get a new Industry Best Practice, 
often they will become your friend enforcing whatever that says.



So what should the Industry Best Practice(s) for DNS servers (root, 
authoritative and recursive) be?  And what should it say about the

interaction between IP/ICMP and TCP/UDP?  And maybe we'll even get
G-Root to follow it.

Re: large organization nameservers sending icmp packets to dns servers.

[Sean Donelan]

On Mon, 6 Aug 2007, Drew Weaver wrote:
Is it a fairly normal practice for large companies such as Yahoo! And 
Mozilla to send icmp/ping packets to DNS servers? If so, why? And a 
related question would be from a service provider standpoint is there 
any reason to deny ICMP/PING packets to name servers within your 
organization?


They use ICMP/Echo Request to calculate a rough surrogate latency estimate 
for potential users of that caching DNS server so they can return 
different DNS answers depending on your network topology.  Yes its an 
approximation, and can be wrong.  Some networks even re-route ICMP/Echo to 
a completely different box which just responsed to pings; so it may not
even go to the same place.  Given all those caveats, many times its still 
the best guess you can make.


ICMP/ECHO is a separate protocol which is easy to filter if you want to, 
without affecting "normal" TCP/UDP/etc packets. But then expect to get 
"worse" default DNS answers from those same sites attempting to optimize 
their DNS answers.


It would be cool if people ran NTP port 123 on their DNS servers, 
and then we could get extreme measurements.  But then I'm sure someone 
would point out 62 flaws with that.  In the end, its up to each 
network operator to make its own decision.  If your DNS servers aren't

being negatively impacted, and it helps your users get better answers,
you might keep it.  If the answers are reversed, you might drop them.

My IDS is badly tuned Well maybe there is a fix for that.

I35 bridge collapse in Minneapolis

[Sean Donelan]

Telephone call gapping by the major long distance carriers into the region 
seemed to be in effect for a while.  I don't believe this is one of the

five critical Mississippi River fiber crossing points, so Internet traffic
appears mostly unaffected.

365 Main reason for outage report published

[Sean Donelan]

The 365 Main San Francisco data center has published its report concerning 
the outage on July 24 after a utility problem.


http://www.365main.com/status_update.html

Other data centers using Hitec backup generators will want to review the 
365's report and those with specific Hitec controllers may want to update 
them.

DNSSEC deployment at IANA (was Re: DNS Hijacking by Cox)

[Sean Donelan]

On Sun, 22 Jul 2007, Steven M. Bellovin wrote:

And people wonder why I support DNSsec


Followups probably should go to the DNS mailing lists

At IEPG, IANA gave an update on the progress being made to implement 
signing of the root/infrastructure-tlds zones.


http://www.potaroo.net/iepg/2007-07-ietf69/notes.txt
https://ns.iana.org/dnssec/status.html

History of the EPO (Emergency Power Off)

[Sean Donelan]

The interesting thing about the EPO and data centers is it wasn't 
orginally for life-safety, but came out of a recommendation by IBM

to the NFPA for property protection.

But like many things, the original reasoning been lost to history, and 
the codes grew in different ways.



http://www.datacenterknowledge.com/archives/2007/May/07/averting_disaster_with_the_epo_button.html

  The history of the emergency power off switch dates back to 1959, when a
  fire in the Air Force's statistical division in the Pentagon caused $6.9
  million in property damage and destroyed three IBM mainframe computers.
  "Nothing gets the government.s attention like something that happens to
  government," said Sawyer. The National Fire Protection Agency (NFPA) was
  tasked to develop rules to address fire risks in IT environments.


Sometimes you need to revisit the rules.  For example, for folks
thought having automatic water sprinklers in data centers was a bad thing. 
Slowly folks have started to rethink it, and now automatic sprinklers are

found in more data centers.  I don't have hard data, but my experience
is there have been fewer outages from accidental sprinkler discharges
than from accidental EPO activations.

Re: Why do we use facilities with EPO's?

[Sean Donelan]

On Wed, 25 Jul 2007, Leo Bicknell wrote:

What I found interesting is that a single EPO is not a hard and
fast rule.  They walked me through a twisty maze of the national
electric code, the national fire code, and local regulations.
Through that journey, they left me with a rather interesting tidbit.


Well.

An Emergency Power Off button is a NEC (the electrical code adopted in 
most of the USA) trade-off for allowing more flexible wiring practices in 
computer rooms.  If you don't use any of those alternative wiring 
practices, you aren't required to install an EPO (modulo the rare local 
code variation). The problem is some people want to get rid of the EPO, 
but also want to keep using alternative wiring practices.


If you've looked in many computer rooms, you'll see some creative wiring 
practices in use.


You'll notice Telco central offices don't have building EPOs.  Likewise 
Equinix data centers don't have EPOs.  But they have limits on what 
wiring practices can be used in their facilities compared to other

data centers.

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Tue, 24 Jul 2007, Joe Greco wrote:

So I'm supposed to invent a solution that does WAY MORE than what Cox
was trying to accomplish, and then you'll listen?  Forget that (or
pay me).


Since it was a false positive, isn't the correct answer to not include
irc.vel.net in the Bot C&C list rather than trying to come up with more 
convoluted solutions?


Is it that much different than when a group makes a mistake implementing a 
USENET Death Penalty, SPAMHAUS DROP list, Bogon lists, Walled Gardens, 
even BCP38++, etc?  Anytime you expect ISPs to do more than forward 
packets (and right or wrong some vocal groups and politicians think ISPs 
should do even more to stop network abuse), there is always a chance 
someone or something will make a mistake.

Bots back to full throttle

[Sean Donelan]

The DNS entries for the EFnet and other mainline IRC servers previously 
affected appear to have timed-out/been removed from various ISP 
caching DNS resolvers I checked.  I didn't check if all the routing 
blackholes have cleared up.


User-based IRC servers should back to being pummled by Bot C&C requests
as normal now.

As I said, false positives suck.  Unfortunately there isn't an anti-abuse 
system that never makes a mistake; unless you do nothing.  I'm skeptical 
of people who claim they've never made a mistake.

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

Would it be better if ISPs just blackholed certain IP addresses associated
with Bot C&C servers instead of trying to give the user a message.  That
doesn't require examining the data content of any messages.  The user just
gets a connection timeout.


Compared to hijacking DNS and intercepting sessions?  Yes.  Absolutely.
See, it isn't that hard to come up with better ideas.


That's what Verizon was doing.  Guess what.  People complained about it 
too.



Interestingly enough, some of us care.  Some of us care enough to run clean
networks AND to make sure that what we're selling isn't compromised by
deliberate DNS hijackings and site redirections.


But do include things like patching servers to filter messages that 
contain certain strings which might accidently catch a legitimate message 
on occasion.  People probably complain about those things too.


It sucks when you are the one that gets caught by a false positive. 
Unfortunately, every attempt at anti-abuse systems have experienced it

at one time or another.  Probably even some of the things you've done
over the years trying to run a clean network has accidently made a 
mistake.

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

"Some privacy advocates" will be upset with ISP's doing what Cox is doing.
Maybe you missed that.  If we assume that it is okay for Cox to actually
intercept the IRC sessions of their users, we're wa far into that
mess anyways.  I'm saying "do it right" if you're going to do it at all.


Would it be better if ISPs just blackholed certain IP addresses associated 
with Bot C&C servers instead of trying to give the user a message.  That 
doesn't require examining the data content of any messages.  The user just

gets a connection timeout.


Personally, I'd prefer that they didn't do it, but that set of solutions
is more complex.


So it is better for ISPs to do nothing, than attempt something that isn't 
perfect. Thanks.  I'll remember that the next time someone complains about 
ISPs not caring about abuse or bots on networks.


Someone will find something to complain about no matter what ISPs do.

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

Please enlighten me.


Intercept and inspect IRC packets.  If they join a botnet channel, turn on
a flag in the user's account.  Place them in a garden (no IRC, no nothing,
except McAfee or your favorite AV/patch set).


Wow, you are recommending ISPs wiretap their subscribers.

I suspect some privacy advocates will be upset with ISPs doing that.

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

Although this seems to be the first bit mistake in over two years, does
that make the practice unacceptable as another tool to respond to Bots?


The practice of blocking public EFnet servers?


As I've said multiple times, sometimes mistakes happen and the wrong 
things end up on a list.  I doubt that was the intent.


Many people have suggested blocking C&C servers used by bots over the 
years.



Yes, when there are better solutions to the problem at hand.


Please enlighten me.

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Chris L. Morrow wrote:

So, to back this up and get off the original complaint, if a service
provider can protect a large portion of their customer base with some
decent intelligence gathering and security policy implementation is that a
good thing? keeping in mind that in this implementation users who know
enough and are willing to forgoe that 'protection' (for some value of
protection) can certainly circumvent/avoid it.


Joe St Sauver covers some of these topics.

http://www.uoregon.edu/~joe/zombies.pdf

Should ISPs attempt to block Bot Command and Control connections (which 
is more general than just IRC C&C Bots), assuming ISPs try to avoid 
"legitimate" servers although mistakes might happen?

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

So are you claiming no bots ever try to connect to that server?


I don't care if bots ever try to connect to that server.  I can effectively
stop the bots from connecting to servers by shutting down the Internet, but
that doesn't make that solution reasonable or correct.


Do spam block lists only block spam e-mails, or do they on occasion 
sometimes block both legitimate e-mail and spam?


Yes, your IRC was probably listed by mistake.  And more than likely 
someone will correct that mistake.


Yes, it is agravating to you personally because it is your server that
was blocked by mistake.

Although this seems to be the first bit mistake in over two years, does
that make the practice unacceptable as another tool to respond to Bots?

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

Hint: there is no bot.  My traffic is being redirected regardless.  Were I
a Cox customer (and I'm not), I'd be rather ticked off.


Hint: the bots are on computers connecting to the irc server, not the irc 
server.



Interfering with services in order to clean a bot would be a much more
plausible excuse if there was a bot.  There is no bot.


So are you claiming no bots ever try to connect to that server?

Re: DNS Hijacking by Cox

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

I can't help but notice you totally avoided responding to what I wrote;
I would have to take this to mean that you know that it is fundamentally
unreasonable to expect users to set up their own recursers to work around
ISP recurser brokenness (which is essentially what this is).


Its more resonable to expect users to know how to remove bots and fix 
their compromised computers?

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

So how do you connect to the real IRC server, then?  Remember that most
end users are not nslookup-wielding shell commandos who can figure out
whois and look up the IP.


If those users are so technically unsophisticated, do you really expect 
the other users with infected computers to figure out how to disinfect 
their computer and remove the Bots instead?


So you have potentially tens of thousands of infected computers with Bots 
making connections to an IRC server.  You know many of those bots are 
well-known, old bots that have built-in removal commands.  But 99% of 
those users don't have the technical knowledge to clean their machine 
themselves or know what a Bot is. On the other hand, you have 1% of users 
are sophisticated enough to use IRC servers.  And a few percentage of 
overlap between the two groups.


What do you do?
  a. nothing
  b. terminate tens of thousands of user accounts (of users who are mostly 
"innocent" except their computer was compromised)

  c. block all IRC
  d. redirect IRC connections to a few servers known to be used by Bots
  e. something else

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking by Cox)

[Sean Donelan]

On Mon, 23 Jul 2007, Suresh Ramasubramanian wrote:

What should be the official IETF recognized method for network operators
to asynchronously communicate with users/hosts connect to the network for
various reasons getting those machines cleaned up?


Most large carriers that are also MAAWG members seem to be pushing
walled gardens for this purpose.


Walled gardens also block access to external IRC servers.

On a network protocol level, walled gardens also contain things like fake 
DNS servers (what about DNSsec), fake http servers, fake (or forced) NAT 
re-writing IP addresses, access control lists and lots of stuff trying to 
respond to the user's traffic with alerts from the ISP.


Although there seems to be a contingent of folks who believe ISPs should
never block or redirect any Internet traffic for any reason, the reality 
is stepping into the middle of the user's traffic sometimes the only 
practical way for ISPs to reach some Internet users with infected 
computers.


But, like other attempts to respond to network abuse (e.g. various 
block lists), sometimes there are false positives and mistakes.  When
it happens, you tweak the filters and undue the wrong block. Demanding 
zero chance of error before ISPs doing anything just means ISPs won't do 
anything.

Re: DNS Hijacking by Cox

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

I'd prefer that ISP's tends towards taking no action when taking action
has a strong probability of backfiring.


Everything has a chance of backfiring.  So ISPs should take no action.

Please let me know how your next DDOS attack lasts.

Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

I think there's a bit of a difference, in that when you're using every
commercial WiFi hotspot and hotel login system, that they redirect
everything.  Would you truly consider that to be the same thing as one
of those services redirecting "www.cnn.com" to their own ad-filled news
page?


Let's get "real."  That's not what those ISPs are doing in this case.

They aren't pretending to be the real IRC server (the redirected IRC 
server indicates its not the real one).  The ISP isn't send ad-fill 
messages.  The irc.foonet.com server clearly sends several cleaning 
commands used by several well-known, and very old, Bots.  I might have 
given the server a different name, but its obviously not trying to 
impersonate the real irc server.


Do you prefer ISPs to break everything, including the users VOIP service 
(can't call 9-1-1), e-mail service (can't contact the help desk), web 
service (can't look for help)?  Or should the ISP only disrupt the minimum 
number of services needed to clean the Bot?

Re: DNS Hijacking by Cox

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

And, incidentally, I do consider this a false positive.  If any average
person might be tripped up by it, and we certainly have a lot of average
users on IRC, then it's bad.  So, the answer is, "at least one false
positive."


The only way any human activity will NEVER have a single false positive, 
i.e. mistake, is by never doing anything.


Do people really want ISPs not to do anything?

Re: DNS Hijacking by Cox

[Sean Donelan]

On Mon, 23 Jul 2007, Joe Greco wrote:

I'll accept that argument once you've explained to all your family
members how to do it - and they've actually done it, successfully.

Let's be real now.


If we're going to be "real now," consider how rarely ISPs have done this
over the last several years.

Its very hard to wake the dragon.  Yes, ISPs can do all sorts of awful 
things, but the reality is most of the big ISPs are extremely conservative 
at taking any steps that disrupts customers traffic.  While they sometimes 
make a mistake, it takes a lot to get big ISPs to do anything.  Since 2005

when ISPs started doing this, how many false positives have come up?

I don't think it is "real" to think big ISPs are going to redirect 
customer traffic in order to steal customer credit card numbers or destroy

a competitor.

Re: DNS Hijacking by Cox

[Sean Donelan]

On Tue, 24 Jul 2007, Perry Lorier wrote:
doing it[1].  If you're interested in finding people that Undernet detects as 
being open proxies or such like, put an IDS rule looking for ":[^ ]* 465 [^ 
]* :AUTO ".


Of course, then someone would flame the ISP for running a sniffer/wiretap 
on their network.


ISPs are pretty much damned no matter what they do or don't do.

How should ISPs notify customers about Bots (Was Re: DNS Hijacking by Cox)

[Sean Donelan]

On Sun, 22 Jul 2007, Joe Greco wrote:

We can break a lot of things in the name of "saving the Internet."  That
does not make it wise to do so.


Since the last time the subject of ISPs taking action and doing something 
about Bots, a lot of people came up with many ideas involving the ISP 
answering DNS queries with the addresses of ISP cleaning servers.


Just about every commercial WiFi hotspot and hotel login system uses a 
fake DNS server to redirect users to its login pages.  Many universities 
use a fake DNS server to redirect student computers to cleaning sites.


What should be the official IETF recognized method for network operators 
to asynchronously communicate with users/hosts connect to the network for

various reasons getting those machines cleaned up?

As far as I know, PPPOE is the only network access protocol that includes 
the feature of redirecting a host to a network operator's system; but 
Microsoft has decided not to implement it.

Multiple different ISPs respond to Bots (was RE: DNS Hijacking by Cox)

[Sean Donelan]

On Sun, 22 Jul 2007, Raymond L. Corbin wrote:

I agree. They are at least trying to clean up their network. If they are
having a lot of problems with zombie bots that DDoS / Spam then this is
a good way to stop it, for now. The small group of users can either use
other nameservers or something like psybnc to connect if they want to
get on IRC.


It doesn't seem to be rogue Cox engineers.  Several major ISPs have all 
taken action against these particular IRC servers (not! IRC in general).
They either re-direct the traffic to a cleaning server, or are blackholing 
the traffic completely.


Yes, it could have been some type of false positive; but when multiple 
ISPs all start re-acting to something, I think there might be more to the 
story.  Especially when those ISPs are noted for not responding to 
incidents.  One ISP, it might be the ISP.  Multiple ISPs, gotta start 
looking at what has them disturbed.


Its hard to wake those dragons.

Re: DNS Hijacking by Cox

[Sean Donelan]

On Sun, 22 Jul 2007, William Allen Simpson wrote:

Comcast still blocks port 25.  And last week, a locally well-known person
was blocked from sending outgoing port 25 email to their servers from her
home Comcast service.


MSA port 587 is only 9 years old.  I guess it takes some people longer 
than others to update their practices.  Based on what I know how 
comcast's abuse systems implement their port 25 restrictions, I think it 
is extremely unlikely it was based on other people having her e-mail 
address in their Outlook programs.


Some people complain ISPs refuse to take action about abuse and 
compromised computers on their networks.  On the other hand, people 
complain when ISPs take action about abuse and compromised computers on 
their networks.  ISPs are pretty much damned if they do, and damned if

they don't.

Several ISPs have been redirecting malware using IRC to "cleaning" 
servers for a couple of years trying to respond to the massive number
of bots.  On occasion they pick up C&C server which also contains some 
"legitimate" uses. Trying to come up with a good cleaning message for

each protocol can be a challenge.

Yes, false positives and false negatives are always an issue. People 
running sevaral famous block lists for spam and other abuse also 
made mistakes on occasion.

Re: DNS Hijacking by Cox

[Sean Donelan]

On Sun, 22 Jul 2007, Andrew Matthews wrote:

isn't there a law against hijacking dns? What can i do to persue this?


DNS is just another application protocol that runs over IP.  You don't 
have to use those DNS servers to resolve names.

Re: iPhone and Network Disruptions ...

[Sean Donelan]

On Sat, 21 Jul 2007, Prof. Robert Mathews (OSIA) wrote:
Cisco, Duke has now come to see the elimination of the problem,  see: "*Duke 
Resolves iPhone, Wi-Fi Outage Problems"* at 
http://www.eweek.com/article2/0,1895,2161065,00.asp


Since neither Apple, Cisco nor Duke seems willing to say exactly what the 
problem was or what they fixed; not very surprising; it was probably a 
"Duh" problem unique to Duke's network.


Otherwise it would be a shame for Apple, Cisco and Duke to not let other 
network operators that might have the same problem to know how to prevent

it from recurring elsewhere.

China Internet problems

[Sean Donelan]

Reuters is reporting that some traffic between China and other countries
is having some problems.  Sina.com and 263.com have notified its users
about problems with overseas e-mail.


http://ca.today.reuters.com/news/newsArticle.aspx?type=technologyNews&storyID=2007-07-18T124822Z_01_PEK91855_RTRIDST_0_TECH-CHINA-INTERNET-COL.XML&archived=False
BEIJING (Reuters) - Internet users and company officials in China on 
Wednesday blamed a series of disruptions to cross-border email traffic on 
adjustments to the country's vast Internet surveillance system.

Re: Belgian court rules that ISPs must block file-sharing

[Sean Donelan]

On Fri, 6 Jul 2007, Chris L. Morrow wrote:

On Thu, 5 Jul 2007, Steven M. Bellovin wrote:

http://www.pcworld.com/article/id,134159-c,internetlegalissues/article.html

Note that this is based on their interpretation of EU law.


and a hearty 'good luck' to them... :( I suppose someone could point the
Belgian's over to the Panamanians (who tried to block VoIP, thanks C&W
PTT for that 'fun'). Hurray, more clue- legislation...


Does anyone have an english language translation of the eleven methods
proposed by the "expert" to implement this order?

I don't think it is going to be pratical, especially since the NSA hasn't
solved the problem of covert channels in decades.  But maybe this "expert"
has come up with something novel.  Or maybe not.

But I'd like to see what was proposed before passing judgement on it.

Re: TransAtlantic Cable Break

[Sean Donelan]

On Mon, 25 Jun 2007, Chris L. Morrow wrote:

I suppose if you had some special traffic you could qos up that and down
everything else but that wasn't quite what Simon was getting at I don't
think.



Although we may think IP is everything, Internet traffic is not the 
only type of traffice carried by telecommunication transmission systems.
If you take the entire cable system from a transmission engineer's point 
of view, it looks different than from an IP engineer's point of view.


Remember last year during the earthquake near Tawain, air traffic control 
and pstn voice capacity came back faster than Internet capacity.


Convergence isn't all its cracked up to be.

Vietnam arrests alledged fiber-optic cable thieves

[Sean Donelan]

http://www.thanhniennews.com/society/?catid=3&newsid=29347

Police arrested two more people Saturday for stealing undersea fiber-optic 
cables off the southern coast of Vietnam, including the group.s suspected 
leader. Ten have been apprehended in the case so far.

[...]
Eleven kilometers of the TVH (Thailand-Vietnam-Hong Kong) line and 32 km 
of the APCN (Asia Pacific Cable Network), linking nine Asian countries, 
have been stolen.

Re: TransAtlantic Cable Break

[Sean Donelan]

On Fri, 22 Jun 2007, Hank Nussbacher wrote:

Tell that to the 10 gig wave customers who lost service. Very few cable
systems provide protection at the 10 gig wave level.


If you don't pay the extra amount for a protected circuit, why should your 
circuit get protection for free when others have to pay for it?  Now, if 
there are 10G customers with protected circuits who lost service, then 
hopefully they have in their contract hefty penalty clauses against the 
carrier.  If not, then they are just plain stupid.


Is paying for "protected circuits" actually worth it.  Or are you better 
off just buying two circuits and using both during normal conditions. 
Use switching at layer 3 to the remaining circuit during abnormal 
conditions.  Most of the time, you get twice the capacity for only twice
the price instead of a "protected circuit" where you only get the once 
the capacity for twice the price.


Of course, there is still the problem some facility provider will "groom" 
both your circuits on to the same cable.  If you are buying pre-emptable 
circuits, hopefully you understand what that means.

Re: Quarantining infected hosts (Was: FBI tells the public to call their ISP for help)

[Sean Donelan]

On Mon, 18 Jun 2007, Suresh Ramasubramanian wrote:

On 6/18/07, Sean Donelan <[EMAIL PROTECTED]> wrote:

Automation is a non-starter unless you have people to deal with the
exceptions.  If you don't deal with exceptions, eventually problems with
any automated system will overwhelm you.  You can only hid behind IVR
recordings "You call is very important to us" for so long.


You're preaching to the choir there.  That still doesnt underrate the
importance of automating this.  Throwing people at it simply doesnt
scale.


You need a both.  The mistake engineers make is thinking technology 
is the solution.  The mistake customer care makes is thinking a pleasent 
voice is the solution.  The mistake law enforcement makes is thinking an

arrest is the solution.  The mistake legislators make is thinking a law
is the solution.  And so on.

We need a mix of all those things, including people, technology, laws and 
physical arrests.  The problem is not a naturally occuring phenomena. 
The opponents are intelligent people who react to anything we do.


I've seen ISPs with very advanced automated systems that went unused 
becaused their customer care organizations couldn't cope with the scale 
of problem customers.  I was building infected customer sandboxes a long 
time ago.  Even if your automated systems handle 99% of the problem 
customers, that 1% can doom your plans if you don't understand it.


ISPs looking for automation may consider these vendors or several 
free/open source alternatives.


Simplicita: http://www.simplicita.com/
Bradbord: http://www.bradfordnetworks.com/
Motive: http://www.motive.com/
Cisco/Perfigo: http://www.cisco.com/en/US/products/ps6128/index.html
F-Secure Network Control: 
http://www.f-secure.co.uk/enterprises/products/fsnc.html
Trend Micro Intercloud: 
http://us.trendmicro.com/us/about/news/pr/article/20070123143622.html

Re: FBI tells the public to call their ISP for help

[Sean Donelan]

On Wed, 13 Jun 2007, Roland Dobbins wrote:
It seems to me that the larger inference is that law enforcement are taking 
the botnet problem more seriously, which is what a lot of folks in the 
operational community have been advocating for a long time.  While one aspect 
of the messaging is questionable, it seems to me that active national-level 
LEO involvement in this problem-space would be welcomed by many.


Its great to see FBI agents and the DOJ taking more interest in the 
problem of Bots and computer intrusions.  I'm especially happy to see 
some arrests.  The focus on home-grown bad guys was also good, instead

of pointing the finger at some random other country.  There are more than
enough bad guys in more than enough countries to go around.  If US law 
enforcement makes any progress at home, each other country can work on

their native bad guys.

Unfortunately, most FBI agents probably have about as much control over 
the FBI press office as most ISP security engineers have over their 
marketing departments.  While FBI agents may be working with ISP security 
engineers, I suspect the FBI press office didn't bothered to vet or 
coordinate its press release with ISPs before issuing it.  We've

all cringed at one time or another at what our respective marketing
teams come up with.

Re: FBI tells the public to call their ISP for help

[Sean Donelan]

On Wed, 13 Jun 2007, Brandon Butterworth wrote:

The fine people at the FBI are recommending people call their ISP for
home computer technical support, even though most ISPs don't sell
home computers, operating system software or application software.


Sounds like an opportunity, someone will want to buy that custom


I found it amusing the FBI saying "don't call us (either)."

AT&T: Support+ $99 for virus/spyware issues 1-866-294-3464
Best Buy: Geek Squad $99 to $349 1-800-GEEK-SQUAD (1-800-433-5778)
Circuit City: Firedog $79 to $159 1-800-FIRE-DOG (1-800-347-3364)
Microsoft PC security: No Charge for virus issues 1-866-PC-SAFETY


BTW, 1 million compromised computers is probably a low estimate.


100 aware users is probably a high estimate


If they are aware users, they probably don't have the problem.

FBI tells the public to call their ISP for help

[Sean Donelan]

If your call center volumes go up today...

The fine people at the FBI are recommending people call their ISP for 
home computer technical support, even though most ISPs don't sell 
home computers, operating system software or application software.


http://www.fbi.gov/page2/june07/botnet061307.htm
   First, if you believe your computer has been compromised, do not call
   the FBI directly. You should contact your Internet service provider.
   They can help you determine if your computer has been infected, and
   what steps to take to restore it. We are not in a position to provide
   technical assistance.

BTW, 1 million compromised computers is probably a low estimate.

CableLabs issues CALEA specification

[Sean Donelan]

CableLabs announced the release of its new Cable Broadband Intercept 
Specification.


http://www.cablemodem.com/downloads/specs/CM-SP-CBI2.0-I01-070611.pdf

While praising CableLabs, the FBI avoided saying the CableLabs 
specifications were acceptable.  The FBI committed to working to further 
refine and develop future comprehensive solutions.


http://www.fbi.gov/pressrel/pressrel07/calea061207.htm

UK ISPs v. US ISPs (was RE: Network Level Content Blocking)

[Sean Donelan]

On Fri, 8 Jun 2007, [EMAIL PROTECTED] wrote:

In this case I would suggest that it is in ISPs best interests to get
involved with network content blocking, so that ISPs collectively become
deep experts on the subject. We are then in a position to modify these
activities in a way that is beneficial to ISPs and their customers (who
happen to be voters too). And we are in a position to advise government
on future actions as well. If ISPs choose not to get involved, then they
are less likely to be listened to by government partly because they have
less credibility and partly because they simply don't understand the
issue and therefore fail to communicate effectively.


UK ISP associations have developed a centralized blocking solution with 
IWF providing the decision making of what to filter.  90% of the UK 
broadband users accept the same "voluntary" decisions about what to 
filter.


On the other hand, US ISP associations have advocated for decentralized 
blocking solutions, leaving the decision to parents and multiple content 
filtering companies.  US ISP associations have been active in this area

since the early 1990's, although US ISP associations seem to only last so
long before they disappear and a new association springs up.

Is a centralized filtering solution better or worse than a decentralized 
filtering solution?


Schools, libraries, families, etc in the US choose which content filter
product to use, which vary greatly how well they work and what they
choose to filter.  Since its "voluntary," some US families choose not to
have any content filters.  Other US families choose to filter much more
than other families.

Cisco, Juniper, Streamshield, NetNanny, etc sell identical products around 
the world.  If an ISP anywhere in the world wants to offer either a
centralized or decentralized filtering solution, the products are available. 
Likewise, if an individual is concerned about what his or her family sees,

they can use without their ISP, the products are available.

Re: Network Level Content Blocking (UK)

[Sean Donelan]

On Thu, 7 Jun 2007, Iljitsch van Beijnum wrote:
Its more than null routes, but not much more.  The router does a re-route 
on a list of network/IP address, and then for the protocols the redirector
box understands (i.e. pretty much only HTTP) it matches part of the 
application/URL pattern.


That's a cool way to implement monitoring of traffic towards random parts of 
the internet.


There are much easier, cheaper ways to do that.

And as another person pointed out, the IWF method is not very 
surreptitious so the bad guys can tell someone found them and

can improve their methods.

And did I mention the false positive problem of click-fraud and
embedded IMG URLs accessing those sites too.  Yes, your computer
may have been recorded accessing a bad site when you read a
spam mail.

Re: Network Level Content Blocking (UK)

[Sean Donelan]

On Thu, 7 Jun 2007, Sean Donelan wrote:

On Thu, 7 Jun 2007, Chris L. Morrow wrote:

Its not "content" blocking, its source/destination blocking.


oh, so null routes? I got the impression it was application-aware, or
atleast port-aware... If it's proxying or doing anything more than
port-level blocking it's likely it sees content as well, or COULD.

Either way, it's not like it's effective for anything except the m ost
casual of users :(


Its more than null routes, but not much more.  The router does a re-route on 
a list of network/IP address, and then for the protocols the redirector
box understands (i.e. pretty much only HTTP) it matches part of the 
application/URL pattern.


So IWF can block only one part of a sub-tree of a popular shared webhosting 
site *IF* is one of a few application protocols.


Sorry, clicked send before finishing.

BUT the important thing is the network operator and routers don't actually 
look at the content.  If the same bad content (picture, video, whatever) 
appears somewhere else that isn't on the IWF list, it won't be blocked.


And likewise if the content at the source/destination changes/removed, 
e.g. the picture disappears, the destination will continue to be blocked 
until IWF updates their bad list even though nothing bad still at the 
destination.

Re: Network Level Content Blocking (UK)

[Sean Donelan]

On Thu, 7 Jun 2007, Chris L. Morrow wrote:

Its not "content" blocking, its source/destination blocking.


oh, so null routes? I got the impression it was application-aware, or
atleast port-aware... If it's proxying or doing anything more than
port-level blocking it's likely it sees content as well, or COULD.

Either way, it's not like it's effective for anything except the m ost
casual of users :(


Its more than null routes, but not much more.  The router does a re-route 
on a list of network/IP address, and then for the protocols the redirector
box understands (i.e. pretty much only HTTP) it matches part of the 
application/URL pattern.


So IWF can block only one part of a sub-tree of a popular shared 
webhosting site *IF* is one of a few application protocols.

Re: Network Level Content Blocking (UK)

[Sean Donelan]

On Thu, 7 Jun 2007, James Blessing wrote:

1. Revocation of mere conduit status; by inspecting certain content and
preventing access to it the ISP is doing more that just passing packets
and is getting involved in the content.


Its not "content" blocking, its source/destination blocking.

While IWF may decide to list a particular source/destination based on its 
view of content, the network doesn't know look at or know what the 
content is and blocks anything at that source/destination address.  The 
"address" may be an application layer "address," i.e. a URL part rather 
than a network layer address.  But if the "address" is dynamically 
generated or changed, it may not have the same content.


Some cellular networks still have walled gardens, which only allow 
access to "approved" source/destinations. Again not based on content, but

based on business relationships with the cellular network operator.

Once you understand its the network isn't blocking "content" but rather 
an ever expanding list of sources/destinations, the real question is how 
can you be certain the bad stuff and good stuff will stay in separate 
places.  Or will the bad stuff continue to migrate elsewhere until you've

blocked most of the Internet, and only "approved" sources/destinations
remain?

Re: Which ISPs Are Spying on You?

[Sean Donelan]

On Thu, 31 May 2007, Hank Nussbacher wrote:

http://www.wired.com/politics/onlinerights/news/2007/05/isp_privacy


Pretty useless reporting, especially since the reporter apparently didn't 
even do even a little Google research to discover which ISP's (not the 
ones he queried) were "partners" with the the original company, COMPETE, 
that made the claim at the conference.


Marketing people tend to exagerate their capabilities, whatever the 
industry.  Reporters should be skeptical of marketing claims.

Re: Content provider plans

[Sean Donelan]

On Wed, 30 May 2007, Michal Krsek wrote:
Few weeks ago I had interesting discussion with *unnamed* Google VIP. His 
answer has been:
"Google engineers doesn't see need to spend money on building IPv6 
infrastructure. You, as user, can motivate them by sending request supporting 
this idea."


So did you write your e-mail to Google techies?


Why would Google techies care.  However you might point out to Google 
advertisers how many eyeballs no longer get their advertising because

they are on IPv6 connections.

Oh, that's right.  User's probably don't know or care whether its IPv6, 
IPv4 or IPX.  After a while, when the IPv4toIPV6 tunnels and translators 
are overloaded, some eyeballs and content will discover each other gets

better performance going "native."  And it will happen.  Trying to force
it before just makes it grumpy.

The Internet has always been an example of just-in-time engineering.

--
Caution buying from UltraDNS/Neustar. Its monthly rates aren't actually 
monthly.

Re: Slate Podcast on Estonian DOS atatck

[Sean Donelan]

On Thu, 24 May 2007, Bill Woodcock wrote:

   > First of it's kind that it targeted a country.
No, at the very least, Moonlight Maze and Titan Rain came before.  But by
today's standards, Moonlight Maze would have been trivially small.  I
don't have any numbers for Titan Rain.  Anyone know how it compared to the
4mpps of this attack?


Don't forget the Pakistan/India cyber-skimishes.  Substantial parts of
their Internet infrastructure was successfully attacked for months at
a time.

Periodically, China, Japan and South Korea seem to have rotating grudge
matches between their hacking groups.  And in wars with bullets, there 
was Yugoslavia and Radio B92 all-media attacks; and pro-Chinese groups
launched several attacks after the US bombed the Chincese embassy in 
Belgrade.


There have been so many cyber-protests of many different US policies 
for many years even keeping a list would be a lot of work.  And even

some pro-US groups launching cyber-attacks to protest policies of
other countries.

Re: Slate Podcast on Estonian DOS atatck

[Sean Donelan]

On Wed, 23 May 2007, Bill Woodcock wrote:

   > http://www.slate.com/id/2166749/fr/podcast/

Downloading it now.

John Markoff just called me for the NYT piece.  Odd that it's just hitting
the news now, two weeks later.


I wonder, does this mean Estonia is now more likely to act/re-act to its 
own homegrown miscreants which attack systems in other countries after 
seeing the impact it had in their own country?  Or is this going to remain 
a case of the "bad guys" are always in some other country, not mine.

Re: Broadband routers and botnets - being proactive

[Sean Donelan]

On Mon, 14 May 2007, Gadi Evron wrote:

Just a joke, Sean.
What would you consider from your experience, the best way to make these
third parties take responsibility?


First, you need to identify the ODM making the software used in the CPE.

--
Warning: Be careful signing up for UltraDNS services.  Their terms are
longer than they appear.

Re: Broadband routers and botnets - being proactive

[Sean Donelan]

On Sun, 13 May 2007, Gadi Evron wrote:

"Passing the buck! Buck passer!" (see below - skip to Dilbert link)


I guess you missed my attempts 3 or 4 years ago at trying to establish 
some standards for CPE concerning security.  I've been at this party for

a long time, I know how the song ends.


Not saying that you are wrong but... Ahh, these are out of our
control, nor will they do anything if we don't. Might as well tell users
not to patch their Windows systems as it's the responsibility of the store
who sold them the computer. Yes, it could help if the stores did
something.


I spent about a year of my life working with Microsoft on getting patched 
versions of Windows in the pipeline and getting OEMs to regularly update 
their manufacturing copy being pre-installed on machines as they leave

the factory.  So yes, it did help to improve things in the pipeline.

Re: Broadband routers and botnets - being proactive

[Sean Donelan]

On Sun, 13 May 2007, Florian Weimer wrote:

Fortunately, there is a simple solution to this kind of problem: ISPs
are very likely liable if they fail to alert customers about security
problems, and do not provide updates in a timely manner.  After a few
painful incidents, the ISPs will learn, and either ship better
software (unlikely) or implement some kind of patch management.  With
a bit of luck, the latter does not just shift back liability back to
the customer, but also helps to parly solve the problem (in the sense
that CPE attacks are less attractive).


It won't solve the problem.  ISPs will simply stop distributing CPE, and
tell customers to buy CPE from their nearest electronics store (Best Buy, 
Radio Shack, or the equivilent in other countries).  If you thought it

was hard getting ISPs to patch CPE, try getting electronics stores to
patch the CPE.  Look at the ancient bugs in D-Link, Linksys, Netgear boxes
that consumers haven't figured out how to patch for years.

You really need to identify the sources and fix it there.

Re: ISP CALEA compliance

[Sean Donelan]

On Fri, 11 May 2007, Steven M. Bellovin wrote:

As Bill Simpson has quite correctly pointed out, you're also not
required to roll over and play dead when someone from the government
asks you for some data. There are laws they're obligated to follow,
too.  Even if you want to look at it from a purely selfish position,
you and/or your company may be liable if you co-operate with an
improper or illegal request.  Have a look at
http://www4.law.cornell.edu/uscode/html/uscode18/usc_sec_18_2520000-.html
which provides for civil liability for illegal wiretaps.  You're clear,
under that statute, if you have good reason to believe the request is
legal under certain very specific sections of the wiretap law, but not
otherwise.


An important thing to remember in this discussion is CALEA does not 
expand, contract or otherwise change other laws concerning electronic 
survellance.  The government can not intercept anything under CALEA.

All interception orders must be authorized by some other statute
or some other lawful authority (e.g. claims of Executive Power).

You might never, ever receive an lawful interception order, but still
be in violation of CALEA.  Likewise you might be 100% CALEA compliant,
and still decline or be unable to perform some intercept orders.  CALEA
does enhance some monetary penalties for not being able to perform a 
lawful intercept authorized by some other statute or authority; but CALEA 
doesn't authorize the interception itself.


Despite attempts by some folks, CALEA compliance != Wiretap authority.

RE: ISP CALEA compliance

[Sean Donelan]

On Thu, 10 May 2007, Stasiniewicz, Adam wrote:

Anyway, here is what I have learned from my experience with our friends in
law enforcement (be it local, state, or federal).  First and foremost, they
like us are only humans trying to make a living.  They are not out to get us


The troublemakers are usually not the law enforcement agents, but the 
consultants and vendors who are just trying to make a living.

Re: ISP CALEA compliance

[Sean Donelan]

On Thu, 10 May 2007, Joe Provo wrote:

Highly likely for most old requests.  Your voice folks can tell you the
#1 CALEA request is neither kiddie pron nor terrrists, but rather DEA.


Remember, CALEA compliance is separate from any intercept orders you
receive.  If you ask your voice folks, you'll also find out very few 
current voice intercepts actually use CALEA compliant equipment or 
capabilities.


CALEA is primarily concerned with the interception of real-time 
communications, and doesn't included access to stored records.


http://www.access.gpo.gov/uscode/title47/chapter9_subchapteri_.html

Also if you talk to your voice guys who have been doing this for many
years, you'll discover everytime an telephone engineer opened his mouth
and said "what about this," the response from the government was "yes,
we want that too, even though we don't understand what it is."


Anyone concerned with broadband CALEA should check with their legal team
and officers to see who if anyone signed off on the securities manual
form 445 and form 105 SSI.  Dealines were in February and March, so if
your legal believes you are needing to comply, they should have already
handled the matter.


Yep, that's why you have lawyers and legal departments.  CALEA is not
an engineering problem, its a legal/budget problem.  Whose legal and 
budget is going to pay for it, and who doesn't.

Re: ISP CALEA compliance

[Sean Donelan]

On Thu, 10 May 2007, Daniel Senie wrote:
Just had this conversation with one of my clients, and it's a good question. 
Seems like the telco providing the ATM (or other) access cloud might be the 
responsible party. The ISP reselling that DSL is too far upstream anyway to 
capture traffic between users of the same DSL cloud, though they could 
capture traffic between those DSL users and other users of their network or 
the Internet at large.


Consult your attorney, of course.


The problem for the DOJ/FBI is CALEA doesn't apply to "private line" 
networks.  The underlying ATM carrier is just providing a private line
"emulation" between the ISP and the subscriber, like a T-1 circuit.  In 
the Voice world, CALEA generally applied to which ever carrier is 
operating the first voice switch connected to the subscriber.


But since CALEA was passed, the world changed.  The carrier providing
the facilities and the carrier providing the switching may not be the
same company.  So the phrase "facilities-based broadband Internet access"
is a mess, unless you happen to be a vertically integrated company.  For
vertically integrated carriers, its mostly a problem of which division
gets stuck with the bill.  But for unaffiliated carriers, I think there
is going to be a lot of finger pointing between the facilities-based,
broadband, and Internet companies.

Re: ISP CALEA compliance

[Sean Donelan]

On Thu, 10 May 2007, Patrick Muldoon wrote:
We've been under the impression that is *all* data.  So for us, things like 
PPPoE Sessions, just putting a tap/span port upstream of the aggregation 
router will not work as you would miss any traffic going from USER A <-> USER 
B, if they where on the same aggregation device.   Since the Intercept has to 
be invisible to the parties being tapped, you can't route their traffic back 
out and then in either, since the tap would change the flow.In that 
regard, we've been upgrading our older NPE's to newer ones in order to 
support SII,  All the while I keep having something a co-worker said stuck in 
my head.  "CALEA - Consultant And Lawyer Enrichment Act" :)


If you are doing PPPOE over another carrier's ATM network, are you really
a "facilities-based" provider?  Or is the CALEA compliance the 
responsibility of the underlying ATM network provider to give LEA access 
to the ATM VC of the subscriber under surviellance?

Re: ISP CALEA compliance

[Sean Donelan]

On Thu, 10 May 2007, Jason Frisvold wrote:

Here's a question that's come up around here.  Does a CALEA intercept
include "hairpining" or is it *only* traffic leaving your network?
I'm of the opinion that a CALEA intercept request includes every bit
of traffic being sent or received by the targeted individual, but
there is strong opposition here that thinks only internet-related
traffic counts.


The DOJ/FBI has been pretty consistent. They want it all and if there is 
a technicality in the law that doesn't give it to them they have 
consistently tried to expand the laws, regulations and court cases to 
give it to them. If you want to be the test case, talk to your lawyers 
about how little you can do.


But its also important to remember CALEA compliance and responding to a 
Title III intercept court order are not necessarily the same thing.


CALEA is only a subset of stuff some carriers have to be prepared to do 
for "Free." Other wiretaps requiring things above and beyond CALEA can be 
done for a time and materials billing to law enforcement after you get an 
lawful order (which can vary depending on what is demanded).  For 
example, a Title III, FISA or ECPA lawful order can apply to traffic and 
institutions not covered by CALEA.  ISPs have been responding to lawful 
orders for over a decade, even before CALEA was a law.  And the reality

is most of the stuff law enforcement actually wants from ISPs on a day to
day basis isn't covered by CALEA (i.e. stored communications and 
transaction records).


http://www.fcc.gov/calea/

  All facilities-based broadband Internet access providers and providers
  of interconnected VoIP service have until May 14, 2007 to come into
  compliance with CALEA.

So are you a

   Facilities-based? (DSL v. cable, dark fiber v. ATM?)
   Broadband? (< 200Kbps?)
   Internet? (VPN?)
   Access? (backbone v. access?)
   Provider? (freenets or paid?)

or are you a

   Provider?
   Interconnected?
   VoIP?
   Service?

If the answer is yes, talk to your lawyer before May 14.  If the answer is
maybe, talk to your lawer, if the answer is I don't know, talk to your 
lawyer.  And if the answer is no, you probably should still talk to your

lawyer.

Warning about UltraDNS terms

[Sean Donelan]

Although UltraDNS/Neustar gives month-to-month pricing, they actually
have a 1 year term even if you cancel.  So you may want to be
aware of it in case you are just testing their service for a few
months.

Re: IP Block 99/8 (DHS insanity - offtopic)

[Sean Donelan]

On Mon, 23 Apr 2007, Chris L. Morrow wrote:

I think the strawman proposals so far were something like:

1) iana has 'root' ca-cert
2) iana signs down certs for RIR's
3) RIR's sign down certs for LIR's
4) LIR's sign down certs for 'users' (where 'users' is probably
address-space users, like corporations or end-sites)

This seemed not-too-insane, and would give ISP/operator type folks that
ability to easily and quickly verify that:

157.242.0.0/16 is in point of fact permitted to originate by the org-id: LMU-1

with some level of authority... It's nothing really more than that.


You can do online or offline verification of a trust chain.  RSA, certs, 
etc are just the math.  But the math doesn't change the trust.  If the
LIR/RIR directories are poorly maintained, their signatures aren't going 
to be any better.


The problem in your trust chain above is the LIR's don't actually verify 
much about the 'users'; and its very easy to spoof the LIRs (i.e. I 
forgot my password) to change their directory information.  And the same

thing will probably be true when you ask LIRs to sign things.  I lost my
RSA cert, please sign a new one for "me".

An online chain of RWHOIS delegations or a offline chain of RSA 
certificates (which you will still need an online CRL check), doesn't

change the problems in the LIRs (or even RIRs or IANA).  A lot of math
won't make the answer more authoritative.

RE: IP Block 99/8

[Sean Donelan]

On Fri, 20 Apr 2007, Marcus H. Sachs wrote:

If we had "clean" registries and signed/verifiable advertisements this would
not be an issue.  Most of you know that DHS was pushing the Secure Protocols
for the Routing Infrastructure initiative
(http://www.cyber.st.dhs.gov/spri.html).  Due to budget cuts this program is
on the shelf for now.  However, we are still interested in making it happen.


The grass is always greener, which is closely related to don't watch 
sausage being made.


Telephone numbers are over 50 years old, but routing of telephone numbers 
isn't actually verifiable either especially with some international 
destinations.


Almost all multi-organizational identity systems have this problem.  If 
you can't trust the organizations involved, more math isn't going to help.

Paging Doug Humphrey (formerly Coloco)

[sean donelan]

Sorry about the general mail. Network operators
sometimes disappear into the ether.  If anyone knows
how to contact Coloco or Doug Humphrey I'd appreciate
finding out how to get my server back from Laurel Maryland.


 

Get your own web address.  
Have a HUGE year through Yahoo! Small Business.
http://smallbusiness.yahoo.com/domains/?p=BESTDEAL

Darned if they do or don't (was Re: SaidCom disconnected by Level 3)

[Sean Donelan]

On Thu, 15 Mar 2007, Jon Lewis wrote:
When we have a customer spamming, we don't call the police.  We either talk 
to, ACL, or shut off the customer.  The above suggests to me that SaidCom had 
spam issues that they were either unable or unwilling to remedy themselves.


Probably no one besides SaidCom and Level3 knows the whole story. If you 
do some more research you will find allegations, besides spam, that
would involve law enforcement. I don't repeat them here because I don't 
know if they are true or not.


Skipping the faces (or lack of facts ) in this particular case.

People criticize ISPs for not shutting down customers.  And  people 
criticize ISPs when they do shut down customers.  Whether it is a 
10Gig link or a DSL/Cable link, do bad guys get more leeway just because 
they pay for more expensive circuits?  Or should an ISP shutdown customers 
as soon as possible when they violate the AUP/TOS regardless who it is.

Three warnings, six warnings, six hundred warnings, how many is too
many?

Would you be more careful about your computers being compromised, 
copyright infringement, signing up downstream customers if you knew
violations of your ISP's AUP would result in disconnection? Everyone 
always wants the other guy's circuit terminated when something

bad happens, but never wants their own circuit terminated when they
screw up.

How many people thank the police officer for stopping them and giving
them a ticket for violating traffic rules?

Ethernet won (was: RE: [funsec] Not so fast, broadband...)

[Sean Donelan]

On Tue, 13 Mar 2007, [EMAIL PROTECTED] wrote:

Sure, as long as you're willing to fork over the cash for CPE capable of
handling OC-XX linecards.  The service cost is hardly the only cost
associated with buying that kind of bandwidth.  It's amusing to me that
we're worrying about FTTH when some of the largest carriers are still not
capable of delivering ethernet handoffs in some of those same top 30 cities.
Don't we need to get there first before we start wiring everyone's home with
fiber and a small router with an SFP?


Bell Atlantic had ethernet access since the early 1990's, along with FDDI, 
SMDS, ATM, etc, etc, etc and whatever else various government agencies 
wanted to buy around Maryland, Virginia and Washington DC.  Now AT&T, 
Qwest and Verizon have metro ethernet access tariffs in major cities in 
each of their territories.  Ethernet seems to have won for data access

especially for 10Gbps and greater.

If you've got the money, they've got the ethernet for you.

Unfortunately, "I want it" isn't a good business case.

Re: [funsec] Not so fast, broadband providers tell big users (fwd)

[Sean Donelan]

On Tue, 13 Mar 2007, Todd Vierling wrote:

Critical mass is approaching.  There's only so long that North
American consumers can be held back from bandwidth-hogging
applications and downloads while parts of the world have long since
upgraded to 10Mbit/s bidirectional (and beyond) consumer-grade access
speeds.


There is the "advertised speed" and then the "*" fine print conditions. 
Providers in some countries have high advertised speeds, but low usage 
caps, fair use policies, low actual speeds to different destinations,
expensive measured telephone usage charges (i.e. dialup) and various 
other things which aren't always included in the comparisons.


The advertised speeds vary widely around the world in different markets.
The actual average consumer speeds are more interesting.  Nevertheless, 
the US is still behind.



If many of US consumers were already buying the biggest pipe and were 
willing to pay even more for even higher speeds; would we be having
this discussion?  Or is the reality that US consumers are buying lower 
priced services even when bigger services are available.


Several US Providers are very happy to sell 1Gbps and even 10Gbps to 
anyone in major (i.e. NFL/top 30) cities, but not at $14.95/month.  45Mbps 
symetrical is readily available from most COs in the US, but again not at 
$14.95/month.


I don't know of any US provider who wants to turn away profitable 
business.  The question is how to make it profitable.

Re: Where are static bogon filters appropriate? was: 96.2.0.0/16 Bogons

[Sean Donelan]

On Tue, 6 Mar 2007, Mikael Abrahamsson wrote:
Customer gets hacked, one of their boxen starts spewing traffic with spoofed 
addresses. The way I understand your solution is to automatically shut their 
port and disrupt all their traffic, and have them call customer support to 
get any further.


Do you really think this is a good solution?

I don't see any customer with a choice continuing having a relationship with 
me if I treat them like that. It will cost me and them too much.


So instead I just drop their spoofed traffic and if they call and say that 
their line is slow, I'll just say it's full and they can themselves track 
down the offending machine and shut it off to solve the problem.


Compromised systems rarely have one thing wrong with them, and delaying
the pain just makes things worse.

Drop spoofed traffic, and they send non-spoofed packets.
Block port 25, and they send slammer on port 1434
Block messenger port 1025, and they send DNS DOS on port 53
Block irc bots port 6667, and they send VOIP spam port 5060
and so on and so on.


   The fast-spreading virus infected as many as 200 county computers
   Wednesday, and technicians shut down the entire network for Anne
   Arundel offices for more than 24 hours.

http://msmvps.com/blogs/donna/archive/2006/02/12/83332.aspx
   One day last year, things started going haywire at Northwest Hospital
   and Medical Center. Key cards would no longer open the operating-room
   doors; computers in the intensive-care unit shut down; doctors' pagers
   wouldn't work.

   It turns out the Seattle hospital's computers . along with up to 50,000
   others across the country . had been turned into an army of robots
   controlled by 20-year-old

Caused by "known" vulnerabilities with patches available, but the 
customers decided it wasn't "important" enough to take action before

they lost everything.

Is it really customer service to avoid the issue?

Re: Where are static bogon filters appropriate? was: 96.2.0.0/16 Bogons

[Sean Donelan]

On Tue, 6 Mar 2007, Mikael Abrahamsson wrote:
Also, all the examples you give implies a BGP transit customer. I am 
imagining all kinds of customers, from colo customers where I am their 
default gateway, to residential customers where it's the same way.


I tried to give examples upstream of a router, not a bridged/direct
connection which may have all sorts of unroutable junk which a router 
should not (and mostly doesn't) forward.  Although spoofing MAC addresses

is probably suspicious behaivor in most bridged networks too.

Disabling 
their port and punting them to customer support is NOT a cost efficient way 
of dealing with the problems, at least not in the market I am in.


Isn't this true of everything (bad source addresses, worms, abuse, etc). 
Does hiding/ignoring the problem just makes it worse because there is no 
incentive to fix the problem while it is still a small problem? If it 
isn't important enough to bother the customer, why bother to fix it?


How you stop forwarding bad stuff is a local decision.  As long as you 
stop it, no one will turn off your interface. If your network is 
forwarding so many packets with false source addresses that it would be a 
major customer support cost issue to fix, your network probably has other 
configuration problems.  You are probably just deferring those customer
service costs until an unpredictable time in the future when those 
misconfigurations disrupt other parts of your network.

Re: Where are static bogon filters appropriate? was: 96.2.0.0/16 Bogons

[Sean Donelan]

On Sun, 4 Mar 2007, Mikael Abrahamsson wrote:
Instead of dropping packets with unallocated sources addresses, perhaps 
backbones should shutdown interfaces they receive packets from unallocated 
address space.  Would this be more effective at both stopping the sources 
of unallocated addresses; as well as sources that spoof other addresses 
because the best way to prevent your interface from being shutdown by 
backbone operators is to be certain you only transmit packets with your 
source addresses.


uRPF or plain source-based filtering for the IP blocks allocated to the 
customer is enough. Shutting it down doesn't make any commercial sense, 
customers wont buy your service if their port is going to be shut down due to 
a single packet. They'll (likely) understand if you won't forward a packet 
from them which has a source address not not belonging to them, though.


When customers misconfigure their router, e.g. wrong BGP neighbor or ASN, 
wrong interface IP address, exceed max prefix limit, etc; don't they lose 
Internet connectivity until they fix it?


A properly configure router should never forward even a single bad 
packet.  If it does, isn't it likely to have configuration problems so

why continue to keep misconfigured routers connected?

Customers are unlikely to fix problems which don't cause them to lose
service.

Re: Where are static bogon filters appropriate? was: 96.2.0.0/16 Bogons

[Sean Donelan]

On Fri, 2 Mar 2007, Daniel Senie wrote:
How do you know, if you're the one being attacked and you have no idea if the 
originating network or their immediate upstream implemented BCP38? Shall we 
just discard ingress filtering? If few attacks are using it today, should we 
declare it no longer relevant? At the same time we should ask if we should be 
x-raying shoes at the airport, since there's only been one guy who tried to 
blow up his shoes. The larger security question is, "do you stop looking for 
old threats simply because they're not the most common threats?" How many 
CodeRed packets flow over the Internet on a typical day? I assure you it's 
not zero.


Show me the data.

How many CodeRed packets originate from unallocated addresses?

Is the proposal actually effective at detecting or protecting against the 
threat?  Or is it just a wasted effort for show?


http://www.tsa.gov/press/happenings/kip_hawley_x-ray_remarks.shtm

Instead of dropping packets with unallocated sources addresses, perhaps 
backbones should shutdown interfaces they receive packets from 
unallocated address space.   Would this be more effective at both 
stopping the sources of unallocated addresses; as well as sources that 
spoof other addresses because the best way to prevent your interface from 
being shutdown by backbone operators is to be certain you only transmit 
packets with your source addresses.

Re: Where are static bogon filters appropriate? was: 96.2.0.0/16 Bogons

[Sean Donelan]

On Fri, 2 Mar 2007, Roland Dobbins wrote:

Sometimes, network operators have to take the bull
by the horns and develop their own systems to do a job that vendors
simply don't understand.


Concur - but it seems that many seem to be looking for someone else to do 
this for them (or, perhaps, the lack of someone to do it for them as an 
excuse to do nothing at all).


How much of a problem is traffic from unallocated addresses?  Backbone 
operators probably have NetFlow data which they could mine to find out.

On the other hand, how much of a problem is obsolete bogon filters causing
everytime IANA delegates another block to an RIR?

Or by the way, how much spoofed traffic uses allocated addresses?

Re: Where are static bogon filters appropriate? was: 96.2.0.0/16 Bogons

[Sean Donelan]

On Thu, 1 Mar 2007, Chris L. Morrow wrote:

So... again, are bogon filters 'in the core' useful? (call 'core' some
network not yours) The cisco auto-secure feature sure showed some fun
effects for this too, eh?


We managed to fix that mis-application in later releases, but it has 
human dependency for that set of releases.


http://www.cisco.com/en/US/products/sw/iosswrel/ps5207/products_field_notice09186a00803e13e9.shtml


Like security "experts" the recommend blocking calls in the North American 
numbering plan to area code 809, or listing individual area codes in PBXs,

its not a good idea unless you have full-time telephone LERG engineers.

My rule of thumb is networks which use the "default" route probably should 
not implement bogon filters of reserved for future allocation addresses. 
Of course, they should still implement postive outbound traffic controls 
(i.e. BCP38++) on their own traffic.  Even if the current engineers are

careful, people change, but the new people may not know or forget about
updating miscellanous routers especially in networks with "default" 
routing.


Most RFC3330 Special Use Addresses which should not appear on the public 
Internet probably should be dropped crossing Internet provider boundaries 
unless specifically allowed, i.e. RFC1918, 127/8, 169.254/16, 224/4, 
240/4, etc. That does not include other addresses mentioned in RFC3330

like 14/8, 24/8, 223.255.255.0/24 which are/will be routable on the
public Internet.  Most backbones already apply some filter like this to
their BGP sessions and it can be optimized for hardware support even on
very high traffic links.  This is pretty low-risk, low-change traffic
hygiene.


So what about filtering reserved for future use "unallocated" IP addresses 
between "default-free" backbones?  Is it enough of a real problem to 
overcome the other hurdle of making operational changes/errors in major

backbone interconnects.  Or is the current practice of whacking the
traffic when it causes a problem, whether its from allocated or 
unallocated space sufficient?


Personally, I think the current practice of whacking problem traffic from
unallocated IP addresses seems sufficient.  If people decide its enough of
a problem that backbones must take the operational risk to change filters, 
then I think IANA must adopt a more predictable schedule.  For example, 
IANA announces future allocations at each IETF meeting which will become 
allocated for use after the next IETF meeting to give backbones 3-4 
months to schedule the change.

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Mon, 26 Feb 2007, Eric Gauthier wrote:

Generally, we've found that most end users don't even know that their systems
are infected - be it with spyware, bots, etc - and are happy when we can help
them clear things up as they usually aren't in a position to fix things on their
own.  I know that the really bad analogy of driving a car has been used a few
times in this thread, but I think part of the analogy is true.  If someone owns
and uses a car but the car has no indicator lights to say that something
is wrong, its hard to believe that the driver will be able to fix the problem
or even know to contact the repair shop.  We've tried to give our users
that "indicator" light and some help repairing it



You forgot a big difference.  Universities usually don't give tuition
refunds, so you have a $40,000 "penalty" hanging over the student's head
which gives students an incentive to listen and want to respond to your 
notices. It's similar to why public libraries have a much harder time 
getting people to return books than university libraries.


Ask car repair shops about people driving their cars after that indicator
light turns on with smoke belching out of it until the car grinds to a 
stop. While consumers might miss one notification method, after notifying 
people by e-mail, telephone, snail mail, web redirects, and any other way 
you can think of; consumers are very good at ignoring warnings until 
their computer stops working.


Detection or notification isn't the problem. Getting people to want to 
fix their computer is.


If there isn't a way to test if the computer is actually fixed, then you 
just repeatedly cycle around the consumer saying its fixed/nothing is 
wrong and the ISP claiming its broken.


What's the Turing test for a fixed computer?

Re: Counting tells you if you are making progress

[Sean Donelan]

On Wed, 21 Feb 2007, Todd Vierling wrote:

I'd say it's severely biased in the overestimation direction -- but
that's not to say it isn't a problem, because zombies Suck.


People with access to the ppp, dhcp or nat logs for a network can de-dup the 
counts based on IP addresses to come up with better surveys of infected 
computers.  They can further correlate the reports with contact
with the computer owners of how many computers were found with known or unknown 
malware. But we rarely hear data from them.


Although I disagree with some of the survey counts, finding zombies isn't 
a problem.  Figuring out if a computer is actually fixed and stays fixed 
is still the problem.  Sometimes it feels like an episode of "House." 
Except House wraps up the case in 60 minutes.

Counting tells you if you are making progress

[Sean Donelan]

If you can't measure a problem, its difficult to tell if you are
making things better or worse.

On Tue, 20 Feb 2007, Rich Kulawiec wrote:

I don't understand why you don't believe those numbers.  The estimates
that people are making are based on externally-observed known-hostile
behavior by the systems in question: they're sending spam, performing
SSH attacks, participating in botnets, controlling botnets, hosting
spamvertised web sites, handling phisher DNS, etc.  They're not based
on things like mere downloads or similar.  As Joe St. Sauver pointed
out to me, "a million compromised systems a day is quite reasonable,
actually (you can track it by rsync'ing copies of the CBL and cummulating
the dotted quads over time)".


Counting IP addresses tends to greatly overestimate and underestimate
the problem of compromised machines.

It tends to overestimate the problem in networks with large dynamic
pools of IP addresses as a few compromised machines re-appear across
multiple IP addresses.  It tends to underestimate the problem in
networks with small NAT pools with multiple machines sharing a few IP
addresses. Differences between networks may reflect different address
pool management algorithms rather than different infection rates.

How do you measure if changes are actually making a difference?

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Sean Donelan]

On Mon, 19 Feb 2007, Rich Kulawiec wrote:

Pop quiz, bonus round: how much does it cost Comcast to defend its
mail servers from Verizon's spam, and vice versa?  Heck, how much
does it cost Comcast to defend its mail servers from its own spam?


How much do they spend on abuse/customer security?  Is it more or less
than they spend on their mail servers?  Even if they shifted all the money 
they spent on the mail departments (opex and capex) to their 
abuse/customer security departments would it make a significant 
difference?


Getting money usually isn't that much of a problem, within reason. 
Figuring out what to spend it on that actually makes a difference is the 
problem.

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Sat, 17 Feb 2007, Gadi Evron wrote:

On Sat, 17 Feb 2007, Sean Donelan wrote:

On Sat, 17 Feb 2007, Gadi Evron wrote:

Is there a significant difference between the "many" ISPs implementing
walled gardens and other ISPs as far as infection rates?


Yes.


Then please share, many people would love to have that data.


Same goes for you with the sentence you removed above. :)


I have, many times over the last 5 years.  Doing research is an amazing 
thing.



I am working on this, and hopefully will have something in a few months
which can be measurable rather than jokes about "many".


Many people will be waiting for your data.

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Sat, 17 Feb 2007, Gadi Evron wrote:

Is there a significant difference between the "many" ISPs implementing
walled gardens and other ISPs as far as infection rates?


Yes.


Then please share, many people would love to have that data.

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Sat, 17 Feb 2007, Gadi Evron wrote:

Yes, but that is because the successful ISPs currently often implement
their own if they have the resources and R&D power. The really big ones
have it automated, the small ones have it limited to be "activated by an
abuse desk person".


And I also know "many" ISPs that developed home-grown systems and had to
abandoned them due to various problems.

Until you understand the differences and why various attempts haven't
worked, you are doomed to repeat the same mistakes; and unlikely to
be successfull beyond a few limited environments.

Is there a significant difference between the "many" ISPs implementing
walled gardens and other ISPs as far as infection rates?

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Sat, 17 Feb 2007, Gadi Evron wrote:

Public ISPs have been testing these types of systems for over 5 years.
What sorts of differences can you think of that would explain why public
ISPs have found them not very effective?

Public ISPs have been using walled gardens for a long time for user
registration and collecting credit card information.  So they know how to
implement walled gardens.  But what happens when public ISPs use it for
infected machines?



Many already do, successfully.

When I say many I actually mean I know of 6. 3 of them huge, 3 of them
relatively small.


Interesting use of the word "many."  Many people use Multics.

I know of "many" more that have tested it and returned it to various 
vendors.  There are several tough problems people are still trying

to solve.

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Sat, 17 Feb 2007, Petri Helenius wrote:
After all these years, I'm still surprised a consortium of ISP's haven't 
figured out a way to do something a-la Packet Fence for their clients where 
- whenever an infected machine is detected after logging in, that machine 
is thrown into say a VLAN with instructions on how to clean their machines 
before they're allowed to go further and stay online.
This has been commercially available for quite some time so it would be only 
up to the providers to implement it.


Public ISPs have been testing these types of systems for over 5 years. 
What sorts of differences can you think of that would explain why public

ISPs have found them not very effective?

Public ISPs have been using walled gardens for a long time for user 
registration and collecting credit card information.  So they know how to
implement walled gardens.  But what happens when public ISPs use it for 
infected machines?

RE: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Fri, 16 Feb 2007, Nicholas J. Shank wrote:

How is the "acceptable" infection rate for universities different than

the

infection rate of other types of networks?


Because other types of networks are expected (expected being the
keyword) to have competent administrators.


Expected by whom?

How many home networks or even small business networks have competent
administrators?

What is the infection rate for the network at a typical NANOG meeting full 
of Internet "experts?"  What was the infection rate at the RSA security 
conference network earlier this month?


Although some specific individual networks may have higher or lower 
infection rates, I haven't see a significant difference in infection

rates between types of networks or industries.  For universities with
low infection rates, there are just as many universities with high 
infection rates.  For government networks with low infection rates, there

are just as many government networks with high infection rates.

Would taking the practices from the specific individual networks with low 
infection rates and using them elsewhere change the infection rate of

other networks?

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Fri, 16 Feb 2007, Eric Gauthier wrote:

I run the network for a University with about 12,000 students and 12,000
computers in our dormitories.  We, like many other Universities, have spent the
last five or six years putting systems in place that are both reactive and
preventative.  From my perspective, the issues are still there but I'm not
sure that I agree with your implications.

Do we still have "compromised" systems?  Yes.
Is the number of "compromosed" systems at any time large?  No.
Is the situation out of control?  No.

Email me off-list if you want more details.  IMHO, Its too bad broadband
providers have not yet picked up on what the Universities have done.


Why do you claim broadband providers haven't picked up on what 
universities have done?


Couldn't broadband providers say the same thing
> Do we still have "compromised" systems?  Yes.
> Is the number of "compromosed" systems at any time large?  No.
> Is the situation out of control?  No.

If you compare infection rates of a broadband provider with 10 million 
subscribers, which probably translates to at least 30 million devices with 
NAT, WiFi and mobile devices; would its infection rate be significantly 
different from a university with 12,000 students with 1 computer each?


If your university's upstream ISP implemented a policy of cutting off the
university's Internet connection anytime a device in the university 
network was compromised; how many hours a year would the university

be down?  What if the university's ISP had a three-strikes policy, would
the university have used up all of its three-strikes?  What proof should
the univeristy's upstream ISP accept the problem is corrected?

Is there some infection rate of university networks that upstream ISPs 
should accept as "normal?"  Or should ISPs have a zero-tolorance policy

for universities becoming infected repeatedly?

How is the "acceptable" infection rate for universities different than the 
infection rate of other types of networks?

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Fri, 16 Feb 2007, [EMAIL PROTECTED] wrote:

I hear enough from people who *do* work at Some Other Place. :)


Hearing about it is not the same as experiencing it first-hand.



Never claimed *our* solution would work everywhere (heck, I even admit it
isn't 100% effective for *us*).  A very large chunk of what *we* do would be
doomed to failure at any organization where the problem set includes "make a
profit selling connectivity to cost-conscious general consumers".


The Other ISPs do all of the things you mentioned, except they don't give
their techs free rooms.  Instead they give out $50 or $100 gift cards for 
in-home or in-store techs from several consumer electronics chains to fix 
customer computers; which may be similar to the level of expertise you
would get from unpaid residential dorm techs.  However, the environment 
and populations aren't necessarily comparable.


Understanding why those things have been doomed to failure is an important 
difference. It isn't because ISPs unwilling to try them. But instead

its because ISPs have tried those things (and many other things).  They
fail not because of the cost side of the equation, but because they don't 
have much effect on the problem over the long-term in that environment and 
population.


If someone (vendor, academic, etc) comes up with something that works 
well for the environment and population facing the general public ISP,
there are a lot of ISPs with money constantly asking what can they 
buy/pay/do to fix it.  However, they are also very skeptical, because

this is a well-travelled road, and they've seen a lot of claims that
didn't pan out.

Re: botnets: web servers, end-systems and Vint Cerf

[Sean Donelan]

On Fri, 16 Feb 2007, [EMAIL PROTECTED] wrote:

And most ISPs don't provide in-house tech support and an orientation lecture
when you sign up - though some *do* provide the free A/V these days. :)


Working a day on the help desk at the *other* ISPs, which ever ISP you
want to point fingers at, is always an eye-opening experience.

Even when you think things should be the same, they sometimes have very
different problems to solve.

Re: RBL for bots?

[Sean Donelan]

On Thu, 15 Feb 2007, Drew Weaver wrote:

   Has anyone created an RBL, much like (possibly) the BOGON list which
includes the IP addresses of hosts which seem to be "infected" and are
attempting to brute-force SSH/HTTP, etc?


Bots are rarely single purpose engines.  If they have been detected doing 
bad things, they will probably appear in multiple RBLs for multiple
reasons.  If something is in multiple RBLs, even if it hasn't done the 
particular badness you are looking for, its probably just a matter of 
time.


Perhaps not surprising, some of the porn site vendors appear to have 
the most sophisticated systems for detecting brute force/password sharing

attacks.

Re: Every incident is an opportunity (was Re: Hackers hit key Internet traffic computers)

[Sean Donelan]

On Sun, 11 Feb 2007, Gadi Evron wrote:

Colin Powell mentioned at RSA in his extremely good, entertaining and
pointless talk something of relevance. During the cold war American kids
were trained to hide beneath their desktops in caseof a nuclear
attack. Much good that would have done.


The important lesson is you can educate people. The content may have been
bogus, but it was very effective at reaching most of the population. 
People who grew up during that era still remember it.


If you can come up with a few simple things to do, it is possible to
reach most of the public.  But we are our own worst enemies.  When we
have the opportunity, instead of giving the few simple things everyone
could do, we create a lot of confusion.

RE: Every incident is an opportunity (was Re: Hackers hit key Internet traffic computers)

[Sean Donelan]

On Sat, 10 Feb 2007, Stasiniewicz, Adam wrote:

Sean makes a good point, but there is one small problem with his
suggestions.  He is preaching to the choir.


Just trying to get the choir to sing on key.  Of course, I know the choir
will probably spin off singing 18 different songs.

Local interest.

The next security incident, can the security experts in the US talk about 
what US readers can do.  Experts in Europe talk about European readers can
do.  Experts in China, Australia, India, Brazil, Antarctica talk about 
what readers in those areas can do.


I have no idea when, where or what the next incident will be, but can 
guess it will involve the usual problems.


Turn on automatic update, turn off services you don't use, don't believe
everything you read on the net.

Every incident is an opportunity (was Re: Hackers hit key Internet traffic computers)

[Sean Donelan]

On Tue, 6 Feb 2007, Roy wrote:
Its amazing how reporters has to butcher technology information to make it 
understood by their editors


http://www.cnn.com/2007/TECH/internet/02/06/internet.attacks.ap/index.html?eref=rss_topstories


Do we keep missing opportunities?

Yes, it was a minor incident, just like a minor earthquake, the hurricane 
that doesn't hit, the fire that is exitinguished. But it was also an 
opportunity to get the message out to the public about the things they 
can do to take control.


We remind people what to do in a tornado, earthquake, flood, hurricane, 
etc.  This on-going education does help; even though some people still

drive their cars through moving water or go outside to watch the tornado.


Instead of pointing fingers at South Korea, China, etc, every country
with compromised computers (all of them) are the problem.  The United 
States may be slow as far as broadband, but it makes up for it in the 
number of compromised computers.


We may know the drill, but it doesn't hurt to repeat message everytime
we have the public's attention for 15 seconds.

1. Turn on Automatic Update if your computer isn't managed by a full-time 
IT group.


   Microsoft Windows, Apple MAC OS/X, and several versions of Linux
   have Automatic Update available.  Most vendors make security patches
   available to users whether or not the software is licensed or
   un-licensed.

   Zero day exploits may be sexy and get the press attention, but the
   long-term problem are the computers that never get patched.  The VML
   exploit on the football stadium websites was patched last month; but
   its not how fast a patch is released, its how fast people install it.

2. Use a hardware firewall/router for your broadband connection and turn 
on the software firewall on your computer in case you ever move your

computer to a different network.

Use Wireless security (WEP, WPA, VPN, SSL, etc) if using a WiFi access
point, or turn off the radio on both your home gateway and computer
if you are not using WiFi.

3. Even if your computer is secure, miscreants depend on your trust. Be 
suspicious of messages, files, software; even if it appears to come from a 
person or company you trust.


   Anti-spam, anti-spyware, anit-virus, anti-phishing tools can help.  But
   don't assume because you are using them, you can click on everything
   and still be safe.  The miscreants are always finding new ways around
   them.

   It may just be human nature, but people seem to engage in more risky
   behavior when they believe they are protected.

4. If your computer is compromised, unplug it until you can get it fixed.

Its not going to fix itself, and ignoring the problem is just going
to get worse.

SRI-NIC.ARPA 26.0.0.73

[Sean Donelan]

Do old packets ever go away on the Internet?  How many DNS packets still 
wander towards SRI-NIC.ARPA's old root server at 26.0.0.73?


At some point, regardless of what the lawyers say, you've got to make your
own decision and move on.  Things change on the Internet, if you don't
maintain your systems they will become obsolete.  Conversely, no matter
how many ways, how many times you try to inform people about changes
someone will miss it, ignore it, misunderstand it, etc.  And someone
may even sue you over it.

Internet alert plan to warn on failures / Cable repairs delayed

[Sean Donelan]

Repair Status

Au said today that the five cable systems that have been partially 
repaired are: Flag North Asia Loop, owned by India's Reliance 
Communications Ltd.; Reach North Asian Loop, owned by Hong Kong's PCCW 
Ltd. and Australia's Telstra Corp.; Se-Me-We3, owned by a group including 
Singapore Telecommunications Ltd. and France Telecom SA; and APCN and 
APCN2, owned by operators including China Telecom Corp. and Taiwan's 
Chunghwa Telecom Co.


A segment of Asia Netcom Corp.'s EAC cable system will be repaired today, 
Chan said. Ofta did not receive the repair status on the seventh cable, 
C2C, owned by C2C Group Ltd, Chan said.




http://www.thestandard.com.hk/news_detail.asp?pp_cat=11&art_id=37146&sid=11956652&con_type=1
Internet alert plan to warn on failures
Timothy Chui
Tuesday, January 30, 2007

A system designed to warn consumers of future breakdowns in Internet 
connections is being launched by the Office of the Telecommunications 
Authority next month.

Re: Colocation in the US.

[Sean Donelan]

On Thu, 25 Jan 2007, Bill Woodcock wrote:
Obviously convection is the best way, and I've gotten away with it a 
few times myself, but the usual answer to your "why not" question is 
"Fire codes."  Convection drives the intensity and spread of fires. 
Which is what furnace chimneys are for.  Thus all the controls on plenum 
spaces.  But when you can get away with it, it's great.


Lots of codes, energy conservation codes, fire codes, building codes, etc.

Although people may gripe about power and cooling, one fire can really 
ruin your day.  Energy can turn into heat in either controlled or 
uncontrolled ways.


Anyone who is interested should look at the conference proceedings and
papers published by ASHRAE.  There was a presentation a few years ago which
explained why Equinix, although I don't think it used the name, has those 
30foot high ceilings painted black :-)  Some of the energy conservation 
codes cost energy when applied to areas beyond their design assumptions.


I noticed AT&T changed its standards.  NEBS still applies in COs, but AT&T 
has published new standards for what they call Internet data spaces. 
Nothing earthshaking, but it was interesting to see some of AT&T's risks 
assessments change.


As I suggested years ago, equipment design, cooling design, power design, 
geography design are all inter-related.  Which is the premium for you?

Re: Google wants to be your Internet

[Sean Donelan]

On Tue, 23 Jan 2007, Chris L. Morrow wrote:

globally unique addresses

I have an electic company, it's got 2500 partners, all with the same
'internal ip addressing plan' (192.168.1.0/24) we need to communicate, is
NAT on both sides really efficient?


What do you do when the electric companies split up again, renumber the
meters into different network blocks?

Satellite set-top boxes don't need to be assigned unique phone numbers to 
report pay-per-view events back to Dish/DirecTV.  They just wake up every
few weeks, use the transport identifier already available on the 
customer's phone line and sends the data, with an embedded identifier

independent from the network transport.  If the satellite STB ever knew
its telephone number, it is probably out-of-date after a few area code
changes.  The same thing happens with burgerler alarm reporting, and lots 
of other things.


I think network engineers are too quick to use network identifiers for
applications.  Electric meters, set-top boxes, alarm systems, ice-boxes,
and whatever else you want to connect to the network don't need to have
the same permanent identifier for the application and the transport.  Most
of the time they don't need a permanent transport identifier.

Re: Google wants to be your Internet

[Sean Donelan]

On Mon, 22 Jan 2007, Daniel Golding wrote:
One interesting point - they plan to use Broadband over Power Line (BPL) 
technology to do this. Meter monitoring is the killer app for BPL, which can 
then also be used for home broadband, Meter reading is one of the top costs 
and trickiest problems for utilities.


Why is IP required, and even if you used IP for transport why must the 
meter identification be based on an IP address?  If meters only report 
information, they don't need a unique transport address and could put

the meter identifier in the application data.

Even if the intent is to include additional controls, e.g. cycle air 
conditioners during peak periods, you still don't need to use IP or

unique IP transport addresses.

Just because you have the hammer called IP, doesn't mean you must use
it on everything.

RE: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted

[Sean Donelan]

On Sun, 21 Jan 2007, Fergie wrote:

This really has more to do with analogies regarding organizations
such as DeBeers, and less with Murphy's Law. :-)


No, its not a scarcity argument.  You have the same problem regardless
of the number of carriers or fibers or routes.  There wasn't a lack of
alternate capacity in Asia. Almost all service was restored even though 
the cables are still being repaired.


Its an assurance problem, not an engineering problem.

RE: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted

[Sean Donelan]

On Sun, 21 Jan 2007, Rod Beck wrote:
Well, gentlemen, you have to ask for the fiber maps and have them 
placed in the contract as an exhibit.


Most of the large commercial banks are doing it. It's doable, but it 
does require effort.


Uhm, did you bother to read the NDAI report?  The Federal Reserve learned
several lessons.  Fiber maps are not sufficient.  If you are relying 
just on fiber maps, you are going to learn the same lesson again and 
again.


The FAA, Federal Reserve, SFTI and SMART are probably at the top as
far as trying to engineer their networks and maintain diversity 
assurances.  But even the Federal Reserve found the cost more than

it could afford. What commercial banks are doing is impressive,
but only in a "commercially reasonable" way. Some residual risk and 
outages are always going to exist.


No matter what the salesman tells you, Murphy still lives.

RE: Undersea fiber cut after Taiwan earthquake - PCCW / Singtel / KT e tc connectivity disrupted

[Sean Donelan]

On Sun, 21 Jan 2007, Rod Beck wrote:
Unfortunately it is news to the decision makers, the buyers of network 
capacity at many of the major IP backbones. Indeed, the Atlantic route 
has problems quite similar to the Pacific.


If this is news to them, perhaps its time to get new decision makers and 
buyers of network capacity at major IP backbones :-)


Unfotunately people have to learn the same lessons over and over again.

http://www.atis.org/ndai/

  "End-to-end multi-carrier circuit diversity assurance currently cannot
  be conducted in a scalable manner. The cost and level of manual effort
  required demonstrated that an ongoing program for end-to-end
  multi-carrier circuit diversity assurance cannot currently be widely
  offered."

http://www.atis.org/PRESS/pressreleases2006/031506.htm

  "The NDAI report confirmed our suspicions that diversity assurance is
  not for the meek," Malphrus added. "It is expensive and requires
  commitment by the customer to work closely with carriers in performing
  due diligence. Until the problem is solved, circuit route diversity
  should not be promoted as a general customer best practice."

RE: Pac Rim Cable Damage Defies Repair [was: AFP article on Taiwan cable repair effort]

[Sean Donelan]

On Wed, 17 Jan 2007, Frank Bulk wrote:

This article paints a rather dismal picture:

Despite optimistic estimates that it would take only three weeks
to repair the massive damage done to what are now said to be
eight submarine cables by the Dec. 26, 2006, magnitude-6.7
earthquake near Taiwan, reports today indicate that not one of
the cables is back in service.
http://www.telecomweb.com/tnd/21168.html


Earth is a single point of failure.

Telegeography has created a nice map showing the affected cables.
http://www.telegeography.com/wordpress/?p=45

NTT reports it has fully restored all its circuits through other routes. 
Other carriers in the region are reporting 80% to 95% of normal service

has been restored through other routes.

IP does not provide survivability UNLESS you have alternate, diverse
connectivity.  But alternate routes are often more expensive, higher
latency, i.e. less competitive.

Re: Comment spammers chewing blogger bandwidth like crazy

[Sean Donelan]

On Sun, 14 Jan 2007, Tony Finch wrote:

I would expect the lists of compromised hosts to be fairly effective -
open proxies of various kinds and perhaps botnet hosts. As for SMTP the
blacklists would only be a starting point that either provide a cheap
preliminary check or feed a more sophisticated filtering system.


If you allow anonymous, unauthenticated access to any system it will
be abused.  Auctions, blogs, chat, mail, phone, etc.  IP addresses
have never been good authenticators for applications. Sending 
confirmation E-mail addresses aren't that much better.  And blacklists 
will just continue to grow longer.


How do you know your user?

Re: Network end users to pull down 2 gigabytes a day, continuously?

[Sean Donelan]

On Fri, 12 Jan 2007, Stephen Sprunk wrote:
There is no technical challenge here; what the pirates are already doing 
works pretty well, and with a little UI work it'd even be ready for the mass 
market.  The challenges are figuring out how to pay for the pipes needed to 
deliver all these bits at consumer rates, and how to collect revenue from all 
the viewers to fairly compensate the producers -- both business problems, 
though for different folks.


Will the North American market change from using speed to volume for 
pricing Internet connections?  Web hosting and other markets around the

world already use GB/transferred packages instead of the port speed.

What happens if a 100Mbps port is $19.95/month with $1.95 per GB 
transferred up and down?  Are P2P swarms as attractive?