Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Fergie]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

- -- Gadi Evron <[EMAIL PROTECTED]> wrote:

>And this is before we get into the academic off-topic discussion of what a
>bot actually is, which after almost 11 years of dealing with these I find
>difficult to define. Is it an IP address? A computer? Perhaps an instance
>of a bot (and every machine could have even hundreds).
>
>Welcome to the realm of Internet security operations and the different
>groups and folks involved (and now industry). It is about Internet
>security rather than this or that network security or this and that sample
>detection.

Interestingly enough, I discovered during my trip to Tokyo this
week that the Japanese government is _mandating_ that the national
ISPs address the botnet problem, specifically.

I'm still gathering details on the framework -- which is still
being defined, if I'm not mistaken -- but I applaud them for
taking the lead in this regard.

If they are even marginally successful, I hope it will be an
example for others around the world to stop making excuses and
begin addressing the problem.

- - ferg

-BEGIN PGP SIGNATURE-
Version: PGP Desktop 9.5.3 (Build 5003)

wj8DBQFF2+dhq1pz9mNUZTMRAhd9AJ9FqULfYzAXzwlhSRdrU2a5Xd5frwCcDedO
XAQipmVgJwGfqq34fANSy7w=
=mAC6
-END PGP SIGNATURE-



--
"Fergie", a.k.a. Paul Ferguson
 Engineering Architecture for the Internet
 fergdawg(at)netzero.net
 ferg's tech blog: http://fergdawg.blogspot.com/

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Gadi Evron]

On Tue, 20 Feb 2007, Rich Kulawiec wrote:

Hi Rich,

 thanks for your input, Rich. As always, quite
interesting.

> 
> BTW #2: All of this leaves open an important and likely-unanswerable
> question: how many systems are compromised but not as yet manifesting
> any external sign of it?  Certainly any competent adversary would hold
> a considerable fraction of its forces in reserve.  (If it were me,
> that fraction would be at least "the majority".)

I stopped really counting bots a while back. I insisted, along with many
friends, that counting botnets was what matters. When we reached thousands
we gave that up.

We often quoted anti nuclear weapons proliferation sentiments from the
cold war, such as: "why be able to destroy the world a thousand times
over if once is more than enough?" we often also changed it to say "3
times" as redudancy could be important. :>

Today, it is clear the bad guys can get their hands on as many bots as
they need, or in a more scary scenario, want. They don't need that many.

As a prime example, I believe that VeriSign made it public that only 200
bots were used in the DNS amplification attacks against them last
year. Even if they missed one, two or even three zeroes, it speaks quite a
bit as to our fragile infrastructure.

If brute force alone can acheive this, what of application
attacks, perhaps even 0days? :)

Still, we keep surviving and we will still be here next year, too, with
bigger and bigger trucks and tubes to hold the Internet together, whether
for regular or malicious usage. eCommerce and online banking might not
survive in a few years if people such as us here don't keep doing what we
do, but that part of it is off topic to NANOG.

10 years ago, almost no one knew what botnets were. Counting and
measuring seemed to be very important 3 years ago, and to governments and
academics, and even a year ago. Today it is just what funding for botnet
research is based on ( :) ), still, I don't really see the
relevance. Botnets are a serious issue, but they are only a sympthom
of the problem called the Internet.

Sitting on different networks and testing them for how many malicious
scans happen every second/minute/hour/day and then checking that against
how many machines with trivially exploited vulnerabilities exist on these
networks can fill in some of the puzzlea, but the delta from what we may
see if we consider email attachments and malicious web sites...

The factor may be quite big.

We will never be able to count how many bots exist. We can count limited
parts of that pool such as those seen in spam. Those are several millions
every day (which should be scary enough) but not quite the right number.

And this is before we get into the academic off-topic discussion of what a
bot actually is, which after almost 11 years of dealing with these I find
difficult to define. Is it an IP address? A computer? Perhaps an instance
of a bot (and every machine could have even hundreds).

Welcome to the realm of Internet security operations and the different
groups and folks involved (and now industry). It is about Internet
security rather than this or that network security or this and that sample
detection.

> 
> ---Rsk
> 

Gadi.

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Rich Kulawiec]

On Mon, Feb 19, 2007 at 02:04:13PM +, Simon Waters wrote:
> I simply don't believe the higher figures bandied about in the discussion for 
> compromised hosts. Certainly Microsoft's malware team report a high level of 
> trojans around, but they include things like the Jar files downloaded onto 
> many PCs, that attempt to exploit a vulnerability that most people patched 
> several years ago. Simply identifying your computer downloaded (as designed), 
> but didn't run (because it was malformed), malware, isn't an infection, or of 
> especial interest (other than indicating something about the frequency with 
> which webservers attempt to deliver malware).

I don't understand why you don't believe those numbers.  The estimates
that people are making are based on externally-observed known-hostile
behavior by the systems in question: they're sending spam, performing
SSH attacks, participating in botnets, controlling botnets, hosting
spamvertised web sites, handling phisher DNS, etc.  They're not based
on things like mere downloads or similar.  As Joe St. Sauver pointed
out to me, "a million compromised systems a day is quite reasonable,
actually (you can track it by rsync'ing copies of the CBL and cummulating
the dotted quads over time)".

So I'm genuinely baffled.  I'd like someone to explain to me why this
seems implausible.

BTW #1: I'm not asserting that my little January experiment is the basis
for such an estimate.  It's not.  It wasn't intended to be, otherwise
I would have used a very different methodology.

BTW #2: All of this leaves open an important and likely-unanswerable
question: how many systems are compromised but not as yet manifesting
any external sign of it?  Certainly any competent adversary would hold
a considerable fraction of its forces in reserve.  (If it were me,
that fraction would be at least "the majority".)

---Rsk

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Sean Donelan]

On Mon, 19 Feb 2007, Rich Kulawiec wrote:

Pop quiz, bonus round: how much does it cost Comcast to defend its
mail servers from Verizon's spam, and vice versa?  Heck, how much
does it cost Comcast to defend its mail servers from its own spam?


How much do they spend on abuse/customer security?  Is it more or less
than they spend on their mail servers?  Even if they shifted all the money 
they spent on the mail departments (opex and capex) to their 
abuse/customer security departments would it make a significant 
difference?


Getting money usually isn't that much of a problem, within reason. 
Figuring out what to spend it on that actually makes a difference is the 
problem.

RE: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[michael.dillon]

 
> > Now, even those people have shifted to a hierarchical 
> architecture of
> > instant-messaging servers.
> 
> In what way is IM hierarchial? 
> Jabber/XMPP has a mesh-of-stars topology 

That is hierarchy. One level is a star topology, the next level is a
mesh.

> which is the same as email's
> modulo some simplifications (mainly owing to the lack of forwarding).

In other words, it is not the same as email's. Of course it may end up
that way but we can hope.

> ISTR that you were arguing in favour of a chain-of-trust 
> system for email
> back in November on the IETF list. I pointed out that the 
> architecture you
> are proposing is essentially the same as inter-domain routing 
> (IP & BGP)
> and Usenet, and you failed to explain how your ideas would solve the
> unwanted traffic problem for email given that the same architecture
> doesn't solve the unwanted traffic problem for IP or NNTP.

An abstract simplification of an architecture is not equal to the
architecture itself. The fact that you can simplify different
architectures into a similar abstract model, doesn't mean that they have
the same problems. Problems often arise in the details of
implementation, not in the theoretical models. I never claimed that my
proposed email model would solve the unwanted mail problem. It was
intended to carry authenticated sender info to the receiver, and to
provide an authenticated reverse path for complaints to postmaster. And
since it was based on negotiated bilateral email peering agreements, if
the chain of trust was subverted at some point in the chain, the peer
would have legal recourse to cut service.

--Michael Dillon

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[J. Oquendo]

[EMAIL PROTECTED] wrote:

And you'll need to de-install IE and Outlook,



  

This will not happen. Not even remotely.


Thus ensuring that Firefox/Thunderbird will be the main target of the
malware people. Is this necessarily any better? Note that Windows
provides an extensive series of hooks which can be used by an
application which wishes to subvert the normal operation of the OS. That
subversive application could be the security monitor which is required
by the ISP for Internet access because it is recommended in your
guidelines.

  
I concur with ISP's looking for IE as some form of guideline. Stupid 
story... So I call Cox because for the 8mb down I am supposed to be 
getting, I was maxing out at 2mb, not a big deal.


TechGirl: Can you go to your start menu...
Me: No I don't use Windows
TechGirl: Please hold
TechGirl: (five minutes later) Are you using OSX?
Me: No. Using Solaris, what would you like me to do?
TechGirl: Please hold
TechGirl: (minutes later) We don't support Solaris
Me: What does an operating system have to do with lousy bandwidth...
TechGirl: Please hold
TechGirl: (minutes later) I have to escalate this to my manager
TechGirl: Please hold
Manager: Please go to your start menu...
Me: No. As stated I'm not on Windows nor OSX. I use Solaris and I AM 
CONNECTED the service is horrible

Manager: Well we only support Windows and OSX
Me: (*ponders what this has to do with cruddy connectivity) Forget it... 
(Plugs in Windows laptop to make things easier).


ISP's have come to rely on the bane of their client's issues. Asking 
someone to remove IE only to have their support group look for it is a 
nightmare in itself. Too many people have become so overdependent on 
Windows.



We live in a complex world. Computers are more complex than they were.
OSes are more complex. Apps are more complex. Networks are more complex.
And SOLUTIONS are more complex. But if the designers of computers, OSes,
apps and networks can deal with the complexity, why can't security folks
do likewise?

  
The issue of security folks dealing with complexities is, they shouldn't 
have to when it comes to 65% of the problems which lead to incidents. 
Why should an ISP have to deal with issues that have nothing to do with 
their networks. I get calls day and night from VoIP customers: "My 
service is down your service sucks"


2007-02-19 00:23:36 '212XXX6428' at [EMAIL PROTECTED]:5060 for 3600
2007-02-19 07:59:43 '212XXX6428' at [EMAIL PROTECTED]:5060 for 3600
2007-02-19 10:58:44 '212XXX6428' at [EMAIL PROTECTED]:5060 for 3600
2007-02-19 12:58:05 '212XXX6428' at [EMAIL PROTECTED]:5060 for 3600

This client goes up and down like a see-saw at least 8 times a day. 
Their provider is horrible. Why should I spend resources trying to fix 
what has nothing to do with my company. Same applies to anyone in the 
security industry to a degree. A security engineer can only do so much 
given parameters most work with. "We're a Windows only shop!" touted the 
MCSE with glee as he wondered why he spent so much time rebooting.




That actually sounds like an answerable question, if a company took it
seriously enough. If the senders and receiver are both on your network,
your finance department should be able to come up with some cost
figures.
  


They won't because they haven't been pressed to do so, and it is rare 
that someone will take it upon themselves to do a good deed when it 
comes to situations like this.


Roland Dobbins wrote:

> NATting firewalls don't help at all with email-delivered malware, 
browser exploits, etc.


Antivirus and ad-aware like programs almost often do when used 
appropriately. It boils down to education which won't happen. If forced 
however it is a different story so again I will point to customer 
sandboxing.


And yes firewalls do help if configured properly on the business side of 
things. I use the same brute forcing script to create firewall rules to 
block IN AND OUT those offensive networks. So even if say a machine were 
to get infected, its only momentarily before I catch it, but this is my 
network(s) and those I manage/maintain. I have zero tolerance for junk 
and don't mind blocking a /8 if needed. People want to complain then I 
point out logfiles with information on why their entire class is blocked.


[EMAIL PROTECTED] wrote:


None of this is rocket science. The hardware available today can do
this. This hardware is not expensive. It does, however, require systems
vendors to have a bit of imagination and that seems to be in rather
short supply in the modern world.



Why would a vendor put all their eggs in one basket. "Brand New AntiVirus software... Guaranteed to stop 
hackers! Only $49.99 per year...", "Brand New AntiMalware software... Guaranteed to stop hackers! 
Only $19.99 a year!", "Brand New Intrusion Detection Prevention Dissemination Articulation 
software... Guaranteed to stop nuclear weapons of mass destruction... Guaranteed to keep you off of the 
Internet..."

A v

RE: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Tony Finch]

On Mon, 19 Feb 2007, [EMAIL PROTECTED] wrote:
>
> Now, even those people have shifted to a hierarchical architecture of
> instant-messaging servers.

In what way is IM hierarchial? The commercial IM systems have a star
topology with a tightly controlled core and basically no inter-domain
federation, so I don't know why you claim they are hierarchial.
Jabber/XMPP has a mesh-of-stars topology which is the same as email's
modulo some simplifications (mainly owing to the lack of forwarding).

ISTR that you were arguing in favour of a chain-of-trust system for email
back in November on the IETF list. I pointed out that the architecture you
are proposing is essentially the same as inter-domain routing (IP & BGP)
and Usenet, and you failed to explain how your ideas would solve the
unwanted traffic problem for email given that the same architecture
doesn't solve the unwanted traffic problem for IP or NNTP.

http://www1.ietf.org/mail-archive/web/ietf/current/msg44467.html

Tony.
-- 
f.a.n.finch  <[EMAIL PROTECTED]>  http://dotat.at/
HUMBER THAMES DOVER WIGHT PORTLAND: SOUTHERLY 4 OR 5, OCCASIONALLY 6 IN
PORTLAND. SLIGHT OR MODERATE, OCCASIONALLY ROUGH IN PORTLAND. DRIZZLE THEN
RAIN. MODERATE OR GOOD.

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Roland Dobbins]

I look forward to your paper on "the end to end concept, and
why it doesn't
apply to email" ;)


I think the problem here is that people invoke something they think  
of as 'the end-to-end principle', but actually isn't.


from :


-

 . . .  functions placed at low levels of a system may be redundant  
or of little

value when compared with the cost of providing them at that low level.

-

*That* is the actual 'end-to-end principle'.  The imposition of  
hierarchy in application-layer email routing (or DNS infrastructure,  
etc.) has nothing to do with the actual end-to-end principle, except  
as a good example of honoring it.


---
Roland Dobbins <[EMAIL PROTECTED]> // 408.527.6376 voice

  The telephone demands complete participation.

  -- Marshall McLuhan

RE: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[michael.dillon]

> I look forward to your paper on "the end to end concept, and 
> why it doesn't 
> apply to email" ;)

Clearly the answer is that it never has applied to email in the pasts.
Hosts don't email each other, people do. People have always relied on
Internet postmaster services to enable Internet email. Given that we
have already thrown out the end-to-end concept from day one, why must we
maintain such a brain-dead flat architecture. People who wanted the
end-to-end concept used to use "talk" on UNIX and Windows popup messages
until recently. Now, even those people have shifted to a hierarchical
architecture of instant-messaging servers.

> I'm not convinced there is an email architecture problem of 
> relevance to the 
> discussion. People mistake a security problem for its most 
> visible symptoms. 

There is more than one security problem here. A well-thought-out email
architecture will only address one of those security problems.

> The SMTP based email system has many faults, but it seems 
> only mildly stressed 
> under the onslaught of millions of hosts attempting to 
> subvert it. 

It depends where you measure that stress. The decline of Internet email
mindshare in favour of IM and Web forums indicates to me that it is
severely stressed at the user level.

> We may need a trust system to deal with identity within the 
> existing email 
> architecture, 

Bingo!

> but I see no reason why that need be 
> hierarchical, indeed 
> attempts to build such hierarchical systems have often failed 
> to gather a 
> critical mass, but peer to peer trust systems have worked 
> fine for decades 
> for highly sensitive types of data.

Peer-to-peer is a form of hierarchy. If you decide to trust X, Y, and Z
and also trust all the hosts that X, Y and Z trust, then you have a
trust hierarchy carved out of the peer-to-peer space. So if I trust AOL,
Earthlink and Verizon, and I also trust all those trusted by these
three, then you can't talk to my mail server until you arrange trust
with me, or with one of the three trusted mail systems. Fact is that the
email architecture does not include any form of trust and things like
Sender-ID and DKIM are only bandaids that don't solve the problem and
introduce additional insecurities.

Additionally, if we can introduce hierarchy into the mail flow, we also
introduce points at which cost-based models of spam prevention can be
tried. If you can pay a penny a message to guarantee that your mail gets
delivered quickly, bypassing any spam-filtering checkpoints, then that
is something that the majority of users would buy into and the money
provides grease for the wheels of the system, making it worthwhile to do
things like set up an email peering agreement.

Let's face it, the Internet of the early 90's is gone. It won't be
coming back either. The challenge now is to operate a network that is
capable of being *THE* global communications infrastructure. If the
public Internet doesn't adapt to this job, then other networks will
leverage the IETF's technology to do so.

--Michael Dillon

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Roland Dobbins]

On Feb 19, 2007, at 6:04 AM, Simon Waters wrote:

I look forward to your paper on "the end to end concept, and why it  
doesn't

apply to email"


The end-to-end principle has no bearing upon this discussion at all,  
unless you're referring to firewalls/NATs.


---
Roland Dobbins <[EMAIL PROTECTED]> // 408.527.6376 voice

  The telephone demands complete participation.

  -- Marshall McLuhan

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Simon Waters]

On Monday 19 February 2007 13:27, you wrote:
> 
> people consider this to be a Windows malware problem. I consider it to
> be an email architecture problem. We all know that you need hierarchy to
> scale networks and I submit that any email architecture without
> hierarchy is broken by design and no amount of ill-thought-out bandaids
> will fix it.

I look forward to your paper on "the end to end concept, and why it doesn't 
apply to email" ;)

I'm not convinced there is an email architecture problem of relevance to the 
discussion. People mistake a security problem for its most visible symptoms. 

The SMTP based email system has many faults, but it seems only mildly stressed 
under the onslaught of millions of hosts attempting to subvert it. Most of 
the attempts to "fix" the architecture problem so far have moved the problem 
from blacklisting IP addresses, to blacklisting domains, or senders, or other 
entities which occupy a larger potential space than the IPv4 addresses, which 
one can use to effectively deal with most of the symptom. In comparison, 
people controlling malware botnets, have demonstrated their ability to 
completely DDoS significant chunks of network, suggesting perhaps that other 
protocols are potentially more vulnerable than SMTP, or more approrpiate 
layers to address the problem at.

We may need a trust system to deal with identity within the existing email 
architecture, but I see no reason why that need be hierarchical, indeed 
attempts to build such hierarchical systems have often failed to gather a 
critical mass, but peer to peer trust systems have worked fine for decades 
for highly sensitive types of data.

I simply don't believe the higher figures bandied about in the discussion for 
compromised hosts. Certainly Microsoft's malware team report a high level of 
trojans around, but they include things like the Jar files downloaded onto 
many PCs, that attempt to exploit a vulnerability that most people patched 
several years ago. Simply identifying your computer downloaded (as designed), 
but didn't run (because it was malformed), malware, isn't an infection, or of 
especial interest (other than indicating something about the frequency with 
which webservers attempt to deliver malware).

RE: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[michael.dillon]

> But suppose you put such a firewall in place.  You'll need to
> configure the firewall properly -- paying as much attention to
> outbound rules as inbound. 

Sounds like a good thing to document in a best practices document that
can be used to certify firewall implementations. When trying to solve a
social problem, techniques like the Good Housekeeping seal of approval
are quite effective. As recommended by the editors of...

> You'll need to add anti-virus software.  And anti-spyware software.
> Then you need to make sure the "signature" databases for both of those
> are updated early and often,

What if the guidelines state that subscription and database oriented
techniques for virus detection are not adequate and therefore not
compliant. Only heuristic, capability-based systems are acceptable.

> And you'll need to de-install IE and Outlook,

Thus ensuring that Firefox/Thunderbird will be the main target of the
malware people. Is this necessarily any better? Note that Windows
provides an extensive series of hooks which can be used by an
application which wishes to subvert the normal operation of the OS. That
subversive application could be the security monitor which is required
by the ISP for Internet access because it is recommended in your
guidelines.

> Something which requires this much work just to make it through its
> first day online, while being used by J. Random Person, is hopelessly
> inadequate.  Which is why systems like this are routinely 
> compromised in
> huge numbers.  Which is why we have a large-scale problem on 
> our hands.

We live in a complex world. Computers are more complex than they were.
OSes are more complex. Apps are more complex. Networks are more complex.
And SOLUTIONS are more complex. But if the designers of computers, OSes,
apps and networks can deal with the complexity, why can't security folks
do likewise?

> This left me with >1.5M observed hosts seen in a month.  
> They're all sending
> spam.  (How do I know?  Because 100% of the mail traffic sent to that
> server is spam.) 

What you did sounds dumb except that you said this is an experiment.
Unfortunately, real live email servers do exactly the same, i.e. talk to
all comers, because the email architecture is flat like a pancake. Some
people consider this to be a Windows malware problem. I consider it to
be an email architecture problem. We all know that you need hierarchy to
scale networks and I submit that any email architecture without
hierarchy is broken by design and no amount of ill-thought-out bandaids
will fix it. 

> Pop quiz, bonus round: how much does it cost Comcast to defend its
> mail servers from Verizon's spam, and vice versa?  Heck, how much
> does it cost Comcast to defend its mail servers from its own spam?

That actually sounds like an answerable question, if a company took it
seriously enough. If the senders and receiver are both on your network,
your finance department should be able to come up with some cost
figures.

--Michael Dillon

Re: botnets: web servers, end-systems and Vint Cerf [LONG, sorry]

[Rich Kulawiec]

I really don't want to get into an OS debate here, but this does
have major operational impact, so I will anyway but will be as
brief as possible.  Please see second (whitespace-separated) section
for some sample hijacked system statistics which may or may not
reflect overall network population.

On Fri, Feb 16, 2007 at 04:27:55PM -, [EMAIL PROTECTED] wrote:
> I disagree. [...]
> 
> Therefore, I assert that securing systems adequately for use on the
> Internet is indeed a SOLVED PROBLEM in computing. However, it isn't yet
> solved in a social or business sense. 

I think I understand your point about the social and business sense of the
problem; if so, then we're probably in at least rough agreement on that.
People do stupid things with computers (like reading email with a web
browser, or replying to spam) and it's proven to be very difficult to
convince them to stop doing those things.

I'm reminded of Ranum's point (from
http://www.ranum.com/security/computer_security/editorials/dumb/ ) about
how if user education was going to work...it would have worked by now.
I think the ongoing success of phishing operations, including those run
by illiterate amateurs, in face of massive publicity via nearly every
communications channel society has to offer, illustrates it nicely.

But, and this may be where we disagree, it's not solved where Microsoft
operating systems are concerned -- and I don't accept the notion that
just putting such systems behind a firewall/NAT box is adequate.
(I'll also argue that any OS which *requires* an external firewall
to survive more than a few minutes' exposure is unsuitable for use
on the Internet.  *Not good enough*.)

But suppose you put such a firewall in place.  You'll need to
configure the firewall properly -- paying as much attention to
outbound rules as inbound.  (And how many people ever do that?  Even
on corporate networks, there are still people stunningly incompetent
enough to use default-permit policies on outbound traffic.  And
controlling outbound traffic from these systems is arguably more
important than controlling inbound -- inbound likely only abuses
the owner, outbound abuses the entire Internet.)

You'll need to add anti-virus software.  And anti-spyware software.
Then you need to make sure the "signature" databases for both of those
are updated early and often, keeping in mind that you have now elected
to play a game that you will inevitably lose the first time that new
malware propagates faster than the keepers of those databases can develop
and distribute signatures.  Vegas lives for suckers like this.

And you'll need to de-install IE and Outlook, since
everything else you've done will be defeated as soon as the next
IE/Outlook-remotely-exploitable-and-leading-directly-to-
full-system-compromise-here's-a-working-demo is published on
full-disclosure, which should be, oh, about three hours from now.

And this is before we even get to the licensing and DRM backdoors
*designed into* Vista.

Something which requires this much work just to make it through its
first day online, while being used by J. Random Person, is hopelessly
inadequate.  Which is why systems like this are routinely compromised in
huge numbers.  Which is why we have a large-scale problem on our hands.




Which brings me to the second point, and that is skepticism over the
100M ballpark figure that's been bandied about.  Personally, I wouldn't
even blink if someone produced convincing proof that the real number
was 300M.  I think that's completely plausible -- "plausible" but still,
I very much hope, unrealistically high.  So from my point of view, this
100M stuff is old news -- i.e., I'm telling you the ocean is wet.

A tiny example: some data (summarized below) from a small experiment last
month using a single test mail server.  I threw away all the data blocked
outright by the firewall in front of it.  I threw away all data that didn't
involve connections directed at port 25.  I threw away all the data for
connecting hosts without rDNS.  I threw away all the data for connecting hosts
with rDNS that looked even vaguely server-like.  I threw away repeat visits.
All of which means that my sampling method is akin to waving a thimble in
a hurricane and will thus provide a gross (and likely skewed) underestimate.

This left me with >1.5M observed hosts seen in a month.  They're all sending
spam.  (How do I know?  Because 100% of the mail traffic sent to that
server is spam.)  And they're all running Windows, except for a handful
which aren't or which were indeterminate.  Note that rDNS lookups were
from local long-lived cache, so rDNS may be well out-of-date in some cases.

Some random examples:

41.241.32.87dsl-241-32-87.telkomadsl.co.za
89.28.3.133 89-28-3-133.starnet.md
190.49.152.243  190-49-152-243.speedy.com.ar
218.178.50.40   softbank218178050040.bbtec.net
200.171.123.83  200-171-123-83.dsl.telesp.net.br
74.132.179.31   74-132-179-3