Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Yes, when there are better solutions to the problem at hand. Please enlighten me. Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner. Please also do this on encrypted control channels or channels not 'irc', also please stay 'cost effective'. Additionally, please do NOT require in-line placement unless you can do complete end-to-end telco-level testing (loops, bit pattern testing, etc), also it'd be a good idea to have a sensible management interface for these devices (serial port 9600 8n1 at least along with a scriptable ssh/telnet/text-ish cli). Looking at DPI (which is required for your solution to work) you are still talking about paying about 500k/edge-device for a carrier-grade DPI solution that can reliably do +2gbps line-rate inspection and actions. This quickly becomes non-cost-effective if your network is more than 1 edge device and less than 500k customers... Adding cost (operational cost you can only recover via increased user fees) is going to make this not deployable in any real network. Wow, I didn't even have to strain myself. sarcasim aside, this isn't a simple problem and at scale the solutions trim down quickly away from anything that seems 'great' :( using DNS and/or routing tricks to circumvent known bad behaviours are the only solutions that seem to fall out. Yes they aren't subscriber specific, but you can't get to subscriber specific capabilities without a fairly large cost outlay. -Chris
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Yes, when there are better solutions to the problem at hand. Please enlighten me. Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner. M... okay. Would you like solution #1 or solution #2? (You can pay for solutions above and beyond that) Solution #1: you know you need to intercept irc.vel.net, so you inject an IGP /32 route for the corresponding IP address, and run it through your IDS sensor. Now, you're not going to be able to claim that you actually have even 100Mbps of traffic bound for that destination, so that's well within the capabilities of modern IDS systems. This has the added benefit of not hijacking someone's namespace, AND not breaking normal communications with the remote site. Bonus points for doing it on Linux or FreeBSD and selectively port forwarding actual observed bot clients to a local cleansing IRCD, then mailing off the problem IP to support so they can contact the customer about AV and bot cleansing software, etc. Oh, you were going to claim that your routers can't handle a few extra /32 routes? I guess I have no help for you there. You win. So on to #2. Solution #2: You really can't handle a few extra /32 routes? Then go ahead, and hijack that DNS, but run it to a transparent proxy server that is in compliance with the ideas outlined above. Cost effective? One FreeBSD box, some freely available tools, and some time by some junior engineer with a little IRC experience, and you have a great tool available, which doesn't inconvenience legitimate users but does accomplish *MORE* than what Cox did. Please also do this on encrypted control channels or channels not 'irc', also please stay 'cost effective'. So I'm supposed to invent a solution that does WAY MORE than what Cox was trying to accomplish, and then you'll listen? Forget that (or pay me). Additionally, please do NOT require in-line placement unless you can do complete end-to-end telco-level testing (loops, bit pattern testing, etc), Ok. also it'd be a good idea to have a sensible management interface for these devices (serial port 9600 8n1 at least along with a scriptable ssh/telnet/text-ish cli). Ok. Looking at DPI (which is required for your solution to work) you are still talking about paying about 500k/edge-device for a carrier-grade DPI solution that can reliably do +2gbps line-rate inspection and actions. Yeah, I see that. Not. (I do see your blind spot, though.) This quickly becomes non-cost-effective if your network is more than 1 edge device and less than 500k customers... Adding cost (operational cost you can only recover via increased user fees) is going to make this not deployable in any real network. Uh huh. Wow, I didn't even have to strain myself. sarcasim aside, this isn't a simple problem and at scale the solutions trim down quickly away from anything that seems 'great' :( using DNS and/or routing tricks to circumvent known bad behaviours are the only solutions that seem to fall out. Yes they aren't subscriber specific, but you can't get to subscriber specific capabilities without a fairly large cost outlay. That's not true. The problem is isolating the traffic in question. Since you DO NOT HAVE GIGABITS OF TRAFFIC destined for IRC servers, this becomes a Networking 101-style question. A /32 host route is going to be effective. Manipulating DNS is definitely the less desirable method, because it has the potential for breaking more things. But, hey, it can be done, and with an amount of effort that isn't substantially different from the amount of work Cox would have had to do to accomplish what they did. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On 7/24/07, Chris L. Morrow [EMAIL PROTECTED] wrote: Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner. Please also do this on encrypted control channels or channels not 'irc', also please stay 'cost effective'. Additionally, Right. However one consolation is that all this is edge filtering, outbound. And some of it can be pushed off onto the CPE (eoe availability of patched CPE) Outbound traffic volumes wont be as horrendously high as those inbound, and should be a bit easier to categorize than inbound traffic DNS and routing tricks are the silliest, to some - but well, they work for a lot of low hanging fruit. And as you say, they are cost effective. srs
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Tue, 24 Jul 2007, Suresh Ramasubramanian wrote: On 7/24/07, Chris L. Morrow [EMAIL PROTECTED] wrote: Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner. Please also do this on encrypted control channels or channels not 'irc', also please stay 'cost effective'. Additionally, Right. However one consolation is that all this is edge filtering, outbound. And some of it can be pushed off onto the CPE (eoe availability of patched CPE) I'd love to see CPE dsl/cable-modem providers integrate with a 'service' that lists out 'bad' things. it'd be nice if the user could even tailor that list (just CC or CC + child-porn or CC older not than X days/hours/minutes) ... I think it might even help, and be vendor agnostic (from a provide and hardware) perspective. Outbound traffic volumes wont be as horrendously high as those inbound, and should be a bit easier to categorize than inbound traffic DNS and routing tricks are the silliest, to some - but well, they work for a lot of low hanging fruit. And as you say, they are cost effective. and they are at the control of the provider, which I think is also somewhat important, people don't know or don't want to police themselves (in general). Also, the false positive issue will still exist, regardless of where this 'service' exists. So we'll have to learn to deal with it and provide workarounds that grandma can do on her own without calling her vendor (equipment or service). -Chris
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On 7/24/07, Joe Greco [EMAIL PROTECTED] wrote: The problem is isolating the traffic in question. Since you DO NOT HAVE GIGABITS OF TRAFFIC destined for IRC servers, this becomes a Networking 101-style question. A /32 host route is going to be effective. Manipulating DNS is definitely the less desirable method, because it has the potential for breaking more things. But, hey, it can be done, and with an amount of effort that isn't substantially different from the amount of work Cox would have had to do to accomplish what they did. Yup - though I still dont see much point in specialcasing IRC. It would probably be much more cost effective in the long run to have something rather more comprehensive. Yes there are a few bots around still using IRC but a lot of them have moved to other, better things (and there's fun headless bots too, hardcoded with instructions and let loose so there's no CC, no centralized domain or dynamic dns for takedown.. you want to make a change? just release another bot into the wild).
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Jul 24, 2007, at 8:59 AM, Joe Greco wrote: But, hey, it can be done, and with an amount of effort that isn't substantially different from the amount of work Cox would have had to do to accomplish what they did. Actually, it's requires a bit more planning and effort, especially if one gets into sinkholing and then reinjecting, which necessitates breaking out of the /32 routing loop post-analysis/-proxy. It can and is done, but performing DNS poisoning with an irchoneyd setup is quite a bit easier. And in terms of the amount of traffic headed towards the IRC servers in question - the miscreants DDoS one another's CC servers all the time, so it pays to be careful what one sinkholes, backhauls, and re-injects not only in terms of current traffic, but likely traffic. In large networks, scale is also a barrier to deployment. Leveraging DNS can provide a pretty large footprint over the entire topology for less effort, IMHO. Also, it appears (I've no firsthand knowledge of this, only the same public discussions everyone else has seen) that the goal wasn't just to classify possibly-botted hosts, but to issue self-destruct commands for several bot variations which support this functionality. [Note: This is not intended as commentary as to whether or not the DNS poisoning in question was a Good or Bad Idea, just on the delta of effort and other operational considerations of DNS poisoning vs. sinkholing/re-injection.] Public reports that both Cox and Time-Warner performed this activity nearly simultaneously; was it a coordinated effort? Was this a one- time, short-term measure to try and de-bot some hosts? Does anyone have any insight as to whether this exercise has resulted in less undesirable activity on the networks in question? --- Roland Dobbins [EMAIL PROTECTED] // 408.527.6376 voice Culture eats strategy for breakfast. -- Ford Motor Company
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On 7/24/07, Joe Greco [EMAIL PROTECTED] wrote: The problem is isolating the traffic in question. Since you DO NOT HAVE GIGABITS OF TRAFFIC destined for IRC servers, this becomes a Networking 101-style question. A /32 host route is going to be effective. Manipulating DNS is definitely the less desirable method, because it has the potential for breaking more things. But, hey, it can be done, and with an amount of effort that isn't substantially different from the amount of work Cox would have had to do to accomplish what they did. Yup - though I still dont see much point in specialcasing IRC. This is probably true. However, in this case, apparently Cox felt there was some benefit to tackling this class of bot. My guess would have been that they were abandoned, and as such, there wouldn't have been much point to doing this. However, maybe that wasn't the case. It would probably be much more cost effective in the long run to have something rather more comprehensive. Sure, but that actually *is* more difficult. It isn't just a technical solution. It has to involve actual ongoing analysis of botnets, and how they operate, plus technical countermeasures. Are there ISP's who are willing to devote resources to that? Yes there are a few bots around still using IRC but a lot of them have moved to other, better things (and there's fun headless bots too, hardcoded with instructions and let loose so there's no CC, no centralized domain or dynamic dns for takedown.. you want to make a change? just release another bot into the wild). Hardly unexpected. The continuing evolution is likely to be pretty scary. Disposables are nice, but the trouble and slowness in seeding makes them less valuable. I'm expecting that we'll see compartmentalized bots, where each bot has a small number of neighbors, a pseudo-scripting command language, extensible communication ABI to facilitate the latest in detection avoidance, and some basic logic to seed/pick neighbors that aren't local. Build in some strong encryption, have them each repeat the encrypted orders to their neighbors, and you have a structure that would be exceedingly difficult to deal with. Considering how long ago that sort of model was proposed, it is actually remarkable that it doesn't seem to have been perfected by now, and that we're still blocking IRC. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Tue, 24 Jul 2007, Joe Greco wrote: So I'm supposed to invent a solution that does WAY MORE than what Cox was trying to accomplish, and then you'll listen? Forget that (or pay me). Since it was a false positive, isn't the correct answer to not include irc.vel.net in the Bot CC list rather than trying to come up with more convoluted solutions? Is it that much different than when a group makes a mistake implementing a USENET Death Penalty, SPAMHAUS DROP list, Bogon lists, Walled Gardens, even BCP38++, etc? Anytime you expect ISPs to do more than forward packets (and right or wrong some vocal groups and politicians think ISPs should do even more to stop network abuse), there is always a chance someone or something will make a mistake.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Jul 24, 2007, at 8:59 AM, Joe Greco wrote: But, hey, it can be done, and with an amount of effort that isn't substantially different from the amount of work Cox would have had to do to accomplish what they did. Actually, it's requires a bit more planning and effort, especially if one gets into sinkholing and then reinjecting, which necessitates breaking out of the /32 routing loop post-analysis/-proxy. Since what I'm talking about is mainly IDS-style inspection of packets, combined with optional redirection of candidate hosts to a local cleanser IRCD, the only real problem is dumping outbound packets somewhere where the /32 routing loop would be avoided. Presumably it isn't a substantial challenge for a network engineer to implement a policy route for packets from that box to the closest transit (even if it isn't an optimal path). It's only IRC, after all. ;-) It can and is done, but performing DNS poisoning with an irchoneyd setup is quite a bit easier. Similar in complexity, just without the networking angle. And in terms of the amount of traffic headed towards the IRC servers in question - the miscreants DDoS one another's CC servers all the time, so it pays to be careful what one sinkholes, backhauls, and re-injects not only in terms of current traffic, but likely traffic. I don't see how what I suggest could be anything other than a benefit to the Internet community, when considering this situation. If your network is generating a gigabit of traffic towards an IRC server, and is forced to run it through an IDS that only has 100Mbps ports, then you've decreased the attack by 90%. Your local clients break, because they're suddenly seeing 90% packet loss to the IRC server, and you now have a better incentive to fix the attack sources. Am I missing some angle there? I haven't spent a lot of time considering it. In large networks, scale is also a barrier to deployment. Leveraging DNS can provide a pretty large footprint over the entire topology for less effort, IMHO. Yes, there is some truth there, especially in networks made up of independent autonomous systems. DNS redirection to a host would favor port redirection, so an undesirable side effect would be that all Cox customers connecting to irc.vel.net would have appeared to be coming from the same host. It is less effort, but more invasive. Also, it appears (I've no firsthand knowledge of this, only the same public discussions everyone else has seen) that the goal wasn't just to classify possibly-botted hosts, but to issue self-destruct commands for several bot variations which support this functionality. The road to hell is paved with good intentions. The realities of the consumer broadband scene make it necessary to take certain steps to protect the network. I think everyone here realized what the goal of the exercise was. The point is that there are other ways to conduct such an exercise. In particular, I firmly believe that any time there is a decision to break legitimate services on the net, that we have an obligation to seriously consider the alternatives and the consequences. [Note: This is not intended as commentary as to whether or not the DNS poisoning in question was a Good or Bad Idea, just on the delta of effort and other operational considerations of DNS poisoning vs. sinkholing/re-injection.] Public reports that both Cox and Time-Warner performed this activity nearly simultaneously; was it a coordinated effort? Was this a one- time, short-term measure to try and de-bot some hosts? Does anyone have any insight as to whether this exercise has resulted in less undesirable activity on the networks in question? That's a very interesting question. I would have expected the bots in question to be idle and abandoned, but perhaps that is not the case. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Tue, 24 Jul 2007 12:00:40 CDT, Joe Greco said: Hardly unexpected. The continuing evolution is likely to be pretty scary. Disposables are nice, but the trouble and slowness in seeding makes them less valuable. I'm expecting that we'll see compartmentalized bots, where each bot has a small number of neighbors, a pseudo-scripting command language, extensible communication ABI to facilitate the latest in detection avoidance, and some basic logic to seed/pick neighbors that aren't local. Build in some strong encryption, have them each repeat the encrypted orders to their neighbors, and you have a structure that would be exceedingly difficult to deal with. Considering how long ago that sort of model was proposed, it is actually remarkable that it doesn't seem to have been perfected by now, and that we're still blocking IRC. Obviously, botnet authors are lazy, and not motivated to do all that work to do all that extra stuff, when we're still focusing on the *last* generation of use a well-known IRC net for CC bots, and haven't really address the *current* use a hijacked host running a private IRC net bots yet. Equally likely - somebody's already written the code, but is waiting for when it is actually *needed* before deploying. If you're the leading side of an arms race, tipping your hand regarding the next escalation is usually a bad idea pgpWFOiE2xIF3.pgp Description: PGP signature
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Tue, Jul 24, 2007 at 12:00:40PM -0500, Joe Greco wrote: Yes there are a few bots around still using IRC but a lot of them have moved to other, better things (and there's fun headless bots too, hardcoded with instructions and let loose so there's no CC, no centralized domain or dynamic dns for takedown.. you want to make a change? just release another bot into the wild). Hardly unexpected. The continuing evolution is likely to be pretty scary. Disposables are nice, but the trouble and slowness in seeding makes them less valuable. I'm expecting that we'll see compartmentalized bots, where each bot has a small number of neighbors, a pseudo-scripting command language, extensible communication ABI to facilitate the latest in detection avoidance, and some basic logic to seed/pick neighbors that aren't local. Build in some strong encryption, have them each repeat the encrypted orders to their neighbors, and you have a structure that would be exceedingly difficult to deal with. Considering how long ago that sort of model was proposed, it is actually remarkable that it doesn't seem to have been perfected by now, and that we're still blocking IRC. Thats because there is a huge world out there of badly protected hosts just waiting to become bots and a fairly basic set of tactics being deployed to prevent them. ie until globally it is somewhat more difficult to build a botnet there is no need to develop complicated solutions. the simpler ones are proven, easy to roll out, easy to modify. its just supply and demand... Steve
RE: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
Obviously, botnet authors are lazy, and not motivated to do all that work to do all that extra stuff, when we're still focusing on the *last* generation of use a well-known IRC net for CC bots, and haven't really address the *current* use a hijacked host running a private IRC net bots yet. Most 'large' botnets are run of off private IRC servers. Any good IRC admin would notice when more then 1k 'bots' started joining their servers. They can look at channel topics and see if it says something like .scan .advscan etc etc. Theres a whole list of commands the old RXBot use to do, I'm sure its more advanced then it was two years ago when I last used IRC. http://www.darksun.ws/phatrxbot/rxbot.html Typically it's the really new kiddies who setup botnets on public IRCD servers, as the IRC admins don't want the extra traffic caused by the bots, nor the problems the script kiddies cause. So adding a public EFNet server to their redirect list wasn't best, however it's simply a false positive. These bots are very simple to use, and you can simply find your better 'bots' by checking the ISP it's from and its uptime. Take that then make it download a preconfigured IRCD such as Unreal and make it run in the background and you have a private IRCD server to route your bots to. So it may not be as fruitful if the public IRC servers are in fact ensuring script kiddies don't live on their networks, but if they check the packets to see what FQDN they are using for their botnet then it wouldn't bother me that they change the DNS to their own 'cleansing' servers. But in doing this it may lead to false positives such as the problem when the EFNet server got blocked. Just my thoughts... Raymond Corbin Support Analyst HostMySite.com
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking )
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Christopher Morrow [EMAIL PROTECTED] wrote: I'd love to see CPE dsl/cable-modem providers integrate with a 'service' that lists out 'bad' things. it'd be nice if the user could even tailor that list (just CC or CC + child-porn or CC older not than X days/hours/minutes) ... I think it might even help, and be vendor agnostic (from a provide and hardware) perspective. Ironically, that is exactly part of a product announcement that we (Trend Micro) are making on 30 July. Since this topic arose, I saw Trend mentioned as a possible product culprit in this scenario, but it isn't. Yet. :-) The particular service to be announced on Monday (BIS, or Botnet Identification Service), is nothing more than a BGP feed of _known_ and _vetted_ botnet CCs as /32s, intended to be a black-hole feed. Interested folks should either e-mail me off-list, or just wait for the official announcement on 30 July. Cheers, - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.2 (Build 2014) wj8DBQFGplq5q1pz9mNUZTMRAnFzAKCicaHuvoTwJk92hPOOu2E/ofjhegCcCrMc XCA4rpUCimConxtKV/Qrsfs= =N2f1 -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Tue, 24 Jul 2007, Joe Greco wrote: So I'm supposed to invent a solution that does WAY MORE than what Cox was trying to accomplish, and then you'll listen? Forget that (or pay me). Since it was a false positive, Fact not in evidence, as much as it'd be good if it were so. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Tue, 24 Jul 2007, Joe Greco wrote: On Mon, 23 Jul 2007, Joe Greco wrote: Yes, when there are better solutions to the problem at hand. Please enlighten me. Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner. M... okay. Would you like solution #1 or solution #2? (You can pay for solutions above and beyond that) I tried to be nice and non-sarcastic. I outlined requirements from a real network security professional on a large transit IP network. You completely glossed over most of it and assumed a host of things that weren't in the requirements. I'm sorry that i didn't get my point across to you, please have a nice day. -Chris
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking )
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Chris L. Morrow [EMAIL PROTECTED] wrote: On Tue, 24 Jul 2007, Paul Ferguson wrote: The particular service to be announced on Monday (BIS, or Botnet Identification Service), is nothing more than a BGP feed of _known_ and _vetted_ botnet CCs as /32s, intended to be a black-hole feed. Interested folks should either e-mail me off-list, or just wait for the official announcement on 30 July. note that this will take out vhost systems... unless they are vetted off the list, which is certainly possible of course. But of course. :-) - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.2 (Build 2014) wj8DBQFGpm6Rq1pz9mNUZTMRAuVXAJ4jxpw0BRVlv8qmNXZQ9P++VHTjcwCg9JGy 591aYUFqou4ziRL5TQASDGg= =lQba -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Tue, 24 Jul 2007, Joe Greco wrote: On Mon, 23 Jul 2007, Joe Greco wrote: Yes, when there are better solutions to the problem at hand. Please enlighten me. Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Pleaes do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner. M... okay. Would you like solution #1 or solution #2? (You can pay for solutions above and beyond that) I tried to be nice and non-sarcastic. I outlined requirements from a real network security professional on a large transit IP network. You completely glossed over most of it and assumed a host of things that weren't in the requirements. I'm sorry that i didn't get my point across to you, please have a nice day. As far as Please enlighten me followed by Please do this at 1Gbps, really 2Gbps today and 20gbps shortly, in a cost effective manner goes, I don't consider that to be non-sarcastic. I consider it to be either very rude, or perhaps a challenge. I attempted to reply in an approximately equivalent tone. But, now, what exactly did I gloss over? And what things did I assume that weren't in the requirements? It's already been demonstrated that it doesn't need to handle 1Gbps, 2Gbps, or 20Gbps, so those requirements are irrelevant. You then said: Please also do this on encrypted control channels or channels not 'irc', also please stay 'cost effective'. But I'm not about to be trapped into building a solution that does WAY MORE than what Cox was trying to do. That it was a requirement from a real network security professional is not relevant, as we're discussing ways to accomplish what Cox was trying, without the related breakage. You further said: Additionally, please do NOT require in-line placement unless you can do complete end-to-end telco-level testing (loops, bit pattern testing, etc), To which I said: Ok., because my solution meets that measure. It does not require in-line placement, condition met. You went on to say: also it'd be a good idea to have a sensible management interface for these devices (serial port 9600 8n1 at least along with a scriptable ssh/telnet/text-ish cli). And again I said: Ok., because my solution can be built on a FreeBSD or Linux box, and as a result, you gain those features too. Condition met. And finally, you say: Looking at DPI (which is required for your solution to work) you are still talking about paying about 500k/edge-device for a carrier-grade DPI solution that can reliably do +2gbps line-rate inspection and actions. And I finally said: Yeah, I see that. Not. Because I don't fundamentally believe that you need to do deep packet inspection of all packets in order to accomplish what Cox was doing. So what exactly did I assume that wasn't in the requirements (and by that, I mean the requirements to do what Cox was attempting to do, not the requirements of some random real network security professional)? If you really think I glossed over something that was important, then by all means, point it out and call me on it. Don't just say HAND. Part of network engineering is being a little bit clever. Brute force is great when you really need to move 20Gbps of traffic. Any idiot can buy big iron to move traffic. However, putting your face in front of the firehose is a bad way to take a sip of water. With that in mind, I will once again point out that doing the equivalent what Cox was trying to do did not /require/ the ability to do deep packet inspection at 20 gigabits per second, and as a result, I'm exercising my option of being a clever network engineer, rather than one who simply throws money and big iron at the problem. You asked for enlightenment. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
RE: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Wow, you are recommending ISPs wiretap their subscribers. I suspect some privacy advocates will be upset with ISPs doing that. Suppose I add a firewall rule to my router to block traffic to a particular port. Does my router thereby wiretap every packet passing through it because it needs to find out its destination port in order to determine if the rule applies or not? It is sometimes a tricky issue when you filter through legitimate traffic to stop illegitimate traffic. But a rule that this is always wiretapping of anything subjected to the automated inspection leads to ridiculous results. DS
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking by Cox)
Hiya, Plenty of boxes can do redirection in the middle such as Redback, Ellacoya etc. We redirect customers who are infected to a web page when the first connect. Then every few hours they get re-directed again, just enough so it's a bit annoying. If they ignore this for a few weeks, they get redirected more frequently :) -- Leigh Sean Donelan wrote: On Sun, 22 Jul 2007, Joe Greco wrote: We can break a lot of things in the name of saving the Internet. That does not make it wise to do so. Since the last time the subject of ISPs taking action and doing something about Bots, a lot of people came up with many ideas involving the ISP answering DNS queries with the addresses of ISP cleaning servers. Just about every commercial WiFi hotspot and hotel login system uses a fake DNS server to redirect users to its login pages. Many universities use a fake DNS server to redirect student computers to cleaning sites. What should be the official IETF recognized method for network operators to asynchronously communicate with users/hosts connect to the network for various reasons getting those machines cleaned up? As far as I know, PPPOE is the only network access protocol that includes the feature of redirecting a host to a network operator's system; but Microsoft has decided not to implement it.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Sun, 22 Jul 2007, Joe Greco wrote: We can break a lot of things in the name of saving the Internet. That does not make it wise to do so. Since the last time the subject of ISPs taking action and doing something about Bots, a lot of people came up with many ideas involving the ISP answering DNS queries with the addresses of ISP cleaning servers. Just about every commercial WiFi hotspot and hotel login system uses a fake DNS server to redirect users to its login pages. I think there's a bit of a difference, in that when you're using every commercial WiFi hotspot and hotel login system, that they redirect everything. Would you truly consider that to be the same thing as one of those services redirecting www.cnn.com to their own ad-filled news page? While I'm not a fan of it, I know that when I go to a hotel, I should try to pull up www.cnn.com (which is actually what I use, because I so rarely use that URL, so it doesn't pollute my browser cache). If I get CNN, then I'm live. If I have to click a button and agree to some terms, then I'm live a bit later. However, if I were to go to a hotel, and they intercept random (to me) web sites, I'd consider that a very bad thing. Many universities use a fake DNS server to redirect student computers to cleaning sites. I'm not sure I entirely approve of that, either, but at least it is more like the hotel login scenario than the hotel random site redirection scenario. What should be the official IETF recognized method for network operators to asynchronously communicate with users/hosts connect to the network for various reasons getting those machines cleaned up? That's a good question. It would actually be good to have a system in place, something competent, instead of the mishmash of broken trash in use by hotels to log in users, etc. I'd see it as an overall benefit. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: I think there's a bit of a difference, in that when you're using every commercial WiFi hotspot and hotel login system, that they redirect everything. Would you truly consider that to be the same thing as one of those services redirecting www.cnn.com to their own ad-filled news page? Let's get real. That's not what those ISPs are doing in this case. They aren't pretending to be the real IRC server (the redirected IRC server indicates its not the real one). The ISP isn't send ad-fill messages. The irc.foonet.com server clearly sends several cleaning commands used by several well-known, and very old, Bots. I might have given the server a different name, but its obviously not trying to impersonate the real irc server. Do you prefer ISPs to break everything, including the users VOIP service (can't call 9-1-1), e-mail service (can't contact the help desk), web service (can't look for help)? Or should the ISP only disrupt the minimum number of services needed to clean the Bot?
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking by Cox)
On 7/23/07, Sean Donelan [EMAIL PROTECTED] wrote: What should be the official IETF recognized method for network operators to asynchronously communicate with users/hosts connect to the network for various reasons getting those machines cleaned up? Most large carriers that are also MAAWG members seem to be pushing walled gardens for this purpose. -- Suresh Ramasubramanian ([EMAIL PROTECTED])
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: I think there's a bit of a difference, in that when you're using every commercial WiFi hotspot and hotel login system, that they redirect everything. Would you truly consider that to be the same thing as one of those services redirecting www.cnn.com to their own ad-filled news page? Let's get real. That's not what those ISPs are doing in this case. I never said it was, but if you don't want to compare the situations using reasonable comparisons (redirecting one thing is different than redirecting all), then I have no interest in debating with you, and you win for some sucky definition of win. They aren't pretending to be the real IRC server (the redirected IRC server indicates its not the real one). The ISP isn't send ad-fill messages. The irc.foonet.com server clearly sends several cleaning commands used by several well-known, and very old, Bots. I might have given the server a different name, but its obviously not trying to impersonate the real irc server. So how do you connect to the real IRC server, then? Remember that most end users are not nslookup-wielding shell commandos who can figure out whois and look up the IP. And what happens when the ISP redirects by IP instead, if we're going to play that game? Do you prefer ISPs to break everything, including the users VOIP service (can't call 9-1-1), e-mail service (can't contact the help desk), web service (can't look for help)? Or should the ISP only disrupt the minimum number of services needed to clean the Bot? All right, here we go. Please explain the nature of the bot on my freshly installed (last night) FreeBSD 6.2R box. # ls -ld /; date; uname -r; uname -s drwxr-xr-x 28 root wheel 512 Jul 22 23:04 / Mon Jul 23 10:56:57 CDT 2007 6.2-RELEASE FreeBSD # echo nameserver 68.4.16.30 /etc/resolv.conf # host irc.vel.net irc.vel.net has address 70.168.71.144 Hint: there is no bot. My traffic is being redirected regardless. Were I a Cox customer (and I'm not), I'd be rather ticked off. Interfering with services in order to clean a bot would be a much more plausible excuse if there was a bot. There is no bot. So, to reiterate your own point: Or should the ISP only disrupt the minimum number of services needed to clean the Bot? Yes, exactly. And that's obviously not what Cox is doing. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking by Cox)
On Mon, 23 Jul 2007, Suresh Ramasubramanian wrote: What should be the official IETF recognized method for network operators to asynchronously communicate with users/hosts connect to the network for various reasons getting those machines cleaned up? Most large carriers that are also MAAWG members seem to be pushing walled gardens for this purpose. Walled gardens also block access to external IRC servers. On a network protocol level, walled gardens also contain things like fake DNS servers (what about DNSsec), fake http servers, fake (or forced) NAT re-writing IP addresses, access control lists and lots of stuff trying to respond to the user's traffic with alerts from the ISP. Although there seems to be a contingent of folks who believe ISPs should never block or redirect any Internet traffic for any reason, the reality is stepping into the middle of the user's traffic sometimes the only practical way for ISPs to reach some Internet users with infected computers. But, like other attempts to respond to network abuse (e.g. various block lists), sometimes there are false positives and mistakes. When it happens, you tweak the filters and undue the wrong block. Demanding zero chance of error before ISPs doing anything just means ISPs won't do anything.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: So how do you connect to the real IRC server, then? Remember that most end users are not nslookup-wielding shell commandos who can figure out whois and look up the IP. If those users are so technically unsophisticated, do you really expect the other users with infected computers to figure out how to disinfect their computer and remove the Bots instead? So you have potentially tens of thousands of infected computers with Bots making connections to an IRC server. You know many of those bots are well-known, old bots that have built-in removal commands. But 99% of those users don't have the technical knowledge to clean their machine themselves or know what a Bot is. On the other hand, you have 1% of users are sophisticated enough to use IRC servers. And a few percentage of overlap between the two groups. What do you do? a. nothing b. terminate tens of thousands of user accounts (of users who are mostly innocent except their computer was compromised) c. block all IRC d. redirect IRC connections to a few servers known to be used by Bots e. something else
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007 11:39:35 EDT, Sean Donelan said: messages. The irc.foonet.com server clearly sends several cleaning commands used by several well-known, and very old, Bots. Old and well-known bots. Remember that for a moment, and think 6 month old antivirus signatures for a bit service (can't look for help)? Or should the ISP only disrupt the minimum number of services needed to clean the Bot? Is there any indication that the commands actually pushed have a *significant* chance of actually wiping any resident bots, or is it That's an old worn-out magic word time? It's one thing if 95% of the time, hijacking the connection and pushing command strings actually cleans a bot up. It's another thing entirely if it only works 5 or 10% of the time because most of the bots currently out there are no longer susceptible to that cleaning method. pgpNFwvkqg6nN.pgp Description: PGP signature
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: On Sun, 22 Jul 2007, Joe Greco wrote: We can break a lot of things in the name of saving the Internet. That does not make it wise to do so. Since the last time the subject of ISPs taking action and doing something about Bots, a lot of people came up with many ideas involving the ISP answering DNS queries with the addresses of ISP cleaning servers. Just about every commercial WiFi hotspot and hotel login system uses a fake DNS server to redirect users to its login pages. I think there's a bit of a difference, in that when you're using every commercial WiFi hotspot and hotel login system, that they redirect everything. Would you truly consider that to be the same thing as one of those services redirecting www.cnn.com to their own ad-filled news page? That's only on initial login, prior to login I suppose. I'm fairly certain their servers could return other 'invalid' responses after login if they wanted, they might even see some revenue savings by redirecting a list of 'known bad things' off to 127.0.0.1 (for instance, pick your preferred place). However, if I were to go to a hotel, and they intercept random (to me) web sites, I'd consider that a very bad thing. What if it was things you didn't use, didn't know about and were there for some measure of your protection? (or your grandmother's protection even) Many universities use a fake DNS server to redirect student computers to cleaning sites. I'm not sure I entirely approve of that, either, but at least it is more like the hotel login scenario than the hotel random site redirection scenario. The problem is that there is very little difference... and it's very 'easy' to say (as a provider) hey, I can help my customers, and the Intertubes as a whole... (btw, how's this all different than opendns?) One of the highlights of this discussion is that people get upset when you mess with 'basic plumbing' in a non-obvious manner. I suppose if you KNOW that it's happening (change your resolv.conf to opendns servers) that's one thing, though do you know or can you config opendns to NOT redirect (example) irc.vel.net but DO irc.badguy.net? messing with DNS brings with it consequences, some good ones and some bad ones...
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking by Cox)
On 7/23/07, Sean Donelan [EMAIL PROTECTED] wrote: But, like other attempts to respond to network abuse (e.g. various block lists), sometimes there are false positives and mistakes. When it happens, you tweak the filters and undue the wrong block. Demanding zero chance of error before ISPs doing anything just means ISPs won't do anything. Running email abuse desks for about a decade now makes me tend to agree with you .. and completely unfiltered pipes to the internet for customer broadband are a pipe dream, most places. -- Suresh Ramasubramanian ([EMAIL PROTECTED])
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Hint: there is no bot. My traffic is being redirected regardless. Were I a Cox customer (and I'm not), I'd be rather ticked off. Hint: the bots are on computers connecting to the irc server, not the irc server. Interfering with services in order to clean a bot would be a much more plausible excuse if there was a bot. There is no bot. So are you claiming no bots ever try to connect to that server?
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On 7/23/07, Joe Greco [EMAIL PROTECTED] wrote: All right, here we go. Please explain the nature of the bot on my freshly installed (last night) FreeBSD 6.2R box. %age of freshly installed freebsd 6.2R boxes v/s random windows boxes on cox cable? Like anything else, its a numbers game.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007 12:42:22 EDT, Sean Donelan said: b. terminate tens of thousands of user accounts (of users who are mostly innocent except their computer was compromised) Given how often compromised computers have *multiple* installs of badware on them, just cleaning off *one* bot that happens to be old enough to respond to their cleaning script is not magically making their system actually safe. There's probably *other* stuff on the box as well. So just waving a mostly-ineffective magic wand at *part* of the problem isn't doing anybody any favors. Maybe you *should* be doing something drastic enough to make the user sit up and take notice and *do* something... (Disclaimer - I can get away with doing that, as user bails for another provider and takes his revenue with them instead of fixing the problem isn't an issue for my revenue stream. YMMV. :) pgpnpS5ILTFm2.pgp Description: PGP signature
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -- Sean Donelan [EMAIL PROTECTED] wrote: On Mon, 23 Jul 2007, Joe Greco wrote: So how do you connect to the real IRC server, then? Remember that most end users are not nslookup-wielding shell commandos who can figure out whois and look up the IP. If those users are so technically unsophisticated, do you really expect the other users with infected computers to figure out how to disinfect their computer and remove the Bots instead? I would imagine that if we're talking about unsophisticated users, the majority of them have no idea what IRC is anyway -- most of them are using AIM, or Yahoo! IM, or - - ferg -BEGIN PGP SIGNATURE- Version: PGP Desktop 9.6.2 (Build 2014) wj8DBQFGpO2Uq1pz9mNUZTMRArtXAKD/gF0YM9MYcLA6AZ2InaNBrlgaHACgngiP kzDzfUd+9BsdcbxDz1Z9xig= =OoHG -END PGP SIGNATURE- -- Fergie, a.k.a. Paul Ferguson Engineering Architecture for the Internet fergdawg(at)netzero.net ferg's tech blog: http://fergdawg.blogspot.com/
RE: How should ISPs notify customers about Bots (Was Re: DNS Hijacking by Cox)
Running email abuse desks for about a decade now makes me tend to agree with you .. and completely unfiltered pipes to the internet for customer broadband are a pipe dream, most places. If ISPs were able to standardize consumer Internet access services using a gateway box, then the necessary filtering could be done on the gateway which runs a secure OS. Of course its not too late to do this. Essentially all the consumer edge infrastructure needs to be upgraded to transition to IPv6. Rather than providing raw unfiltered Internet access over IPv6, ISPs could use a standard gateway box. When I say standardize, I mean that ISPs could collectively work out the specs for such an IPv6 Internet gateway in the IETF along with vendors and other interested parties. Once a standard spec is agreed upon, vendors will make such boxes at the price-point that you need. I would also expect that I can buy such a box and manage it myself if I choose, rather than having the ISP manage it for me as with most users. I would also expect the box to have no NAT, use real IPv6 addresses, and provide various firewall features to protect my home network better than an IPv4 NAT box without preventing me from using new peer-to-peer protocols like SIP. --Michael Dillon
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
I would imagine that if we're talking about unsophisticated users, the majority of them have no idea what IRC is anyway -- most of them are using AIM, or Yahoo! IM, or Quite true. I do know of a small fraction, however, that when Yahoo stopped supporting the chats for their groups, that went over to a Java IRC client. Granted, they still don't know that its IRC, but they'll still end up running into something totally unexplained. Tuc/TBOH
RE: How should ISPs notify customers about Bots (Was Re: DNS Hijacking by Cox)
On Mon, 23 Jul 2007 [EMAIL PROTECTED] wrote: Running email abuse desks for about a decade now makes me tend to agree with you .. and completely unfiltered pipes to the internet for customer broadband are a pipe dream, most places. If ISPs were able to standardize consumer Internet access services using a gateway box, then the necessary filtering could be done on the gateway which runs a secure OS. Of course its not too late to do this. Essentially all the consumer edge infrastructure needs to be upgraded to transition to IPv6. Rather than providing raw unfiltered Internet access over IPv6, ISPs could use a standard gateway box. would you like that in black plastic? with a nice dial on top to spin? :) When I say standardize, I mean that ISPs could collectively work out the specs for such an IPv6 Internet gateway in the IETF along with vendors and other interested parties. Once a standard spec is agreed upon, vendors will make such boxes at the price-point that you need. I think that was discussed in v6ops actually just 5 mins ago. I would also expect that I can buy such a box and manage it myself if I choose, rather than having the ISP manage it for me as with most users. but it connects to my network, and if you touch it you could damage my network... we could maybe get some legislation to fix this... I would also expect the box to have no NAT, use real IPv6 addresses, and provide various firewall features to protect my home network better than an IPv4 NAT box without preventing me from using new peer-to-peer protocols like SIP. See the v6ops draft on CPE security... maybe that's a step in the right direction? I'm sure the author would like some commentary.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On 7/23/07, Joe Greco [EMAIL PROTECTED] wrote: All right, here we go. Please explain the nature of the bot on my freshly installed (last night) FreeBSD 6.2R box. %age of freshly installed freebsd 6.2R boxes v/s random windows boxes on cox cable? That's fairly irrelevant. The fact is that this isn't targetting infected boxes, it's targetting everyone. Like anything else, its a numbers game. All of computing is a numbers game. That doesn't make it right to go around breaking random services just because it might fix some random problem. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: So how do you connect to the real IRC server, then? Remember that most end users are not nslookup-wielding shell commandos who can figure out whois and look up the IP. If those users are so technically unsophisticated, do you really expect the other users with infected computers to figure out how to disinfect their computer and remove the Bots instead? So you have potentially tens of thousands of infected computers with Bots making connections to an IRC server. You know many of those bots are well-known, old bots that have built-in removal commands. But 99% of those users don't have the technical knowledge to clean their machine themselves or know what a Bot is. On the other hand, you have 1% of users are sophisticated enough to use IRC servers. And a few percentage of overlap between the two groups. What do you do? a. nothing b. terminate tens of thousands of user accounts (of users who are mostly innocent except their computer was compromised) c. block all IRC d. redirect IRC connections to a few servers known to be used by Bots e. something else e. something else. Because: a. is wrong. b. doesn't fix the problem, merely shoots yourself in the foot. c. is impossible, stupid, and pointless to try. d. has been proven to be implemented incompetently by Cox. e. includes solutions known to work, including walled gardens, etc. Again, you do not need to hijack someone else's domain name and redirect portions of their namespace at one of your own servers in order to fix this problem. Well, of course, that assumes that you're technically competent. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: So are you claiming no bots ever try to connect to that server? I don't care if bots ever try to connect to that server. I can effectively stop the bots from connecting to servers by shutting down the Internet, but that doesn't make that solution reasonable or correct. Do spam block lists only block spam e-mails, or do they on occasion sometimes block both legitimate e-mail and spam? Yes, your IRC was probably listed by mistake. And more than likely someone will correct that mistake. Yes, it is agravating to you personally because it is your server that was blocked by mistake. Although this seems to be the first bit mistake in over two years, does that make the practice unacceptable as another tool to respond to Bots?
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Tuc at T-B-O-H.NET wrote: I would imagine that if we're talking about unsophisticated users, the majority of them have no idea what IRC is anyway -- most of them are using AIM, or Yahoo! IM, or Quite true. I do know of a small fraction, however, that when Yahoo stopped supporting the chats for their groups, that went over to a Java IRC client. Granted, they still don't know that its IRC, but they'll still end up running into something totally unexplained. and the sympton TODAY is 'irc', but in reality if cox spoke up I'd bet they are doing this with much more than just this one irc server (or set of irc servers)... So, to back this up and get off the original complaint, if a service provider can protect a large portion of their customer base with some decent intelligence gathering and security policy implementation is that a good thing? keeping in mind that in this implementation users who know enough and are willing to forgoe that 'protection' (for some value of protection) can certainly circumvent/avoid it. It's perfectly plausible that cox implemented some trend-micro-like (or maybe trend micro actual) device to do this work for them... just to pick on one vendor of solutions in this space. -Chris
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: So are you claiming no bots ever try to connect to that server? I don't care if bots ever try to connect to that server. I can effectively stop the bots from connecting to servers by shutting down the Internet, but that doesn't make that solution reasonable or correct. Do spam block lists only block spam e-mails, or do they on occasion sometimes block both legitimate e-mail and spam? It depends on the list. Yes, your IRC was probably listed by mistake. And more than likely someone will correct that mistake. Yes, it is agravating to you personally because it is your server that was blocked by mistake. Although this seems to be the first bit mistake in over two years, does that make the practice unacceptable as another tool to respond to Bots? The practice of blocking public EFnet servers? Yes, when there are better solutions to the problem at hand. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Chris L. Morrow wrote: So, to back this up and get off the original complaint, if a service provider can protect a large portion of their customer base with some decent intelligence gathering and security policy implementation is that a good thing? keeping in mind that in this implementation users who know enough and are willing to forgoe that 'protection' (for some value of protection) can certainly circumvent/avoid it. Joe St Sauver covers some of these topics. http://www.uoregon.edu/~joe/zombies.pdf Should ISPs attempt to block Bot Command and Control connections (which is more general than just IRC CC Bots), assuming ISPs try to avoid legitimate servers although mistakes might happen?
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, Jul 23, 2007 at 02:48:05PM -0500, Joe Greco wrote: On 7/23/07, Joe Greco [EMAIL PROTECTED] wrote: All right, here we go. Please explain the nature of the bot on my freshly installed (last night) FreeBSD 6.2R box. %age of freshly installed freebsd 6.2R boxes v/s random windows boxes on cox cable? That's fairly irrelevant. The fact is that this isn't targetting infected boxes, it's targetting everyone. its relevant because you specified freebsd and hence it becomes necessary to consider what % of users have freebsd boxes and how many of those are infected Like anything else, its a numbers game. All of computing is a numbers game. That doesn't make it right to go around breaking random services just because it might fix some random problem. right .. whats that then? you're buying a product, you have TCs, you are protected by consumer law.. what moral of society is being breached for it not to be right? and neither the services are random or the problem. they are quite specific and the solution has been calculated to be the path of least resistance for the whole. you sound a lot like a consumer more than a network operator.. i'm not saying i would like what cox do if i were a consumer of theirs but they are dealing with an issue on their subscription service and they dont seem to be doing anything particularly radical do you have a better suggestion for them? incidentally, if you are a consumer and a tech-savvy one, why dont you just circumvent the restriction? Steve
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Although this seems to be the first bit mistake in over two years, does that make the practice unacceptable as another tool to respond to Bots? The practice of blocking public EFnet servers? As I've said multiple times, sometimes mistakes happen and the wrong things end up on a list. I doubt that was the intent. Many people have suggested blocking CC servers used by bots over the years. Yes, when there are better solutions to the problem at hand. Please enlighten me.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Although this seems to be the first bit mistake in over two years, does that make the practice unacceptable as another tool to respond to Bots? The practice of blocking public EFnet servers? As I've said multiple times, sometimes mistakes happen and the wrong things end up on a list. I doubt that was the intent. Many people have suggested blocking CC servers used by bots over the years. There's a difference between blocking actual CC servers and blocking general IRC servers that are incidentally being used as CC servers. Yes, when there are better solutions to the problem at hand. Please enlighten me. Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Wow, I didn't even have to strain myself. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, Jul 23, 2007 at 02:48:05PM -0500, Joe Greco wrote: On 7/23/07, Joe Greco [EMAIL PROTECTED] wrote: All right, here we go. Please explain the nature of the bot on my freshly installed (last night) FreeBSD 6.2R box. %age of freshly installed freebsd 6.2R boxes v/s random windows boxes on cox cable? That's fairly irrelevant. The fact is that this isn't targetting infected boxes, it's targetting everyone. its relevant because you specified freebsd and hence it becomes necessary to consider what % of users have freebsd boxes and how many of those are infected No, it's not necessary to consider what % of users have FreeBSD boxes. I simply used that to indicate that the box in question /is/ /not/ /infected/, and yet I'm being redirected. The point here is that it is inappropriate to break legitimate services in the pursuit of the greater good. Like anything else, its a numbers game. All of computing is a numbers game. That doesn't make it right to go around breaking random services just because it might fix some random problem. right .. whats that then? you're buying a product, you have TCs, you are protected by consumer law.. what moral of society is being breached for it not to be right? If I'm buying Internet access, and I ask for irc.vel.net, I expect to be connected to that site. and neither the services are random or the problem. they are quite specific and the solution has been calculated to be the path of least resistance for the whole. you sound a lot like a consumer more than a network operator.. Every network operator is a consumer and a provider. i'm not saying i would like what cox do if i were a consumer of theirs but they are dealing with an issue on their subscription service and they dont seem to be doing anything particularly radical This isn't radical? do you have a better suggestion for them? Sure. Posted already. If they need some professional advice, there's a ton of people who could provide highly effective solutions. incidentally, if you are a consumer and a tech-savvy one, why dont you just circumvent the restriction? For the same reason I don't support having multiple incoherent DNS roots. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Hint: there is no bot. My traffic is being redirected regardless. Were I a Cox customer (and I'm not), I'd be rather ticked off. Hint: the bots are on computers connecting to the irc server, not the irc server. Hint: I know. As I said, for the challenged, THERE IS NO BOT. MY TRAFFIC IS BEING REDIRECTED REGARDLESS. Interfering with services in order to clean a bot would be a much more plausible excuse if there was a bot. There is no bot. So are you claiming no bots ever try to connect to that server? I don't care if bots ever try to connect to that server. I can effectively stop the bots from connecting to servers by shutting down the Internet, but that doesn't make that solution reasonable or correct. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Please enlighten me. Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Wow, you are recommending ISPs wiretap their subscribers. I suspect some privacy advocates will be upset with ISPs doing that.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Please enlighten me. Intercept and inspect IRC packets. If they join a botnet channel, turn on a flag in the user's account. Place them in a garden (no IRC, no nothing, except McAfee or your favorite AV/patch set). Wow, you are recommending ISPs wiretap their subscribers. I suspect some privacy advocates will be upset with ISPs doing that. Some privacy advocates will be upset with ISP's doing what Cox is doing. Maybe you missed that. If we assume that it is okay for Cox to actually intercept the IRC sessions of their users, we're wa far into that mess anyways. I'm saying do it right if you're going to do it at all. Personally, I'd prefer that they didn't do it, but that set of solutions is more complex. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Some privacy advocates will be upset with ISP's doing what Cox is doing. Maybe you missed that. If we assume that it is okay for Cox to actually intercept the IRC sessions of their users, we're wa far into that mess anyways. I'm saying do it right if you're going to do it at all. Would it be better if ISPs just blackholed certain IP addresses associated with Bot CC servers instead of trying to give the user a message. That doesn't require examining the data content of any messages. The user just gets a connection timeout. Personally, I'd prefer that they didn't do it, but that set of solutions is more complex. So it is better for ISPs to do nothing, than attempt something that isn't perfect. Thanks. I'll remember that the next time someone complains about ISPs not caring about abuse or bots on networks. Someone will find something to complain about no matter what ISPs do.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Some privacy advocates will be upset with ISP's doing what Cox is doing. Maybe you missed that. If we assume that it is okay for Cox to actually intercept the IRC sessions of their users, we're wa far into that mess anyways. I'm saying do it right if you're going to do it at all. Would it be better if ISPs just blackholed certain IP addresses associated with Bot CC servers instead of trying to give the user a message. That doesn't require examining the data content of any messages. The user just gets a connection timeout. Compared to hijacking DNS and intercepting sessions? Yes. Absolutely. See, it isn't that hard to come up with better ideas. Personally, I'd prefer that they didn't do it, but that set of solutions is more complex. So it is better for ISPs to do nothing, than attempt something that isn't perfect. Well, that's not what I said, now, is it. I did say that there's a set of solutions out there to deal with that. Thanks. I'll remember that the next time someone complains about ISPs not caring about abuse or bots on networks. Interestingly enough, some of us care. Some of us care enough to run clean networks AND to make sure that what we're selling isn't compromised by deliberate DNS hijackings and site redirections. Hmm. ... JG -- Joe Greco - sol.net Network Services - Milwaukee, WI - http://www.sol.net We call it the 'one bite at the apple' rule. Give me one chance [and] then I won't contact you again. - Direct Marketing Ass'n position on e-mail spam(CNN) With 24 million small businesses in the US alone, that's way too many apples.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On Mon, 23 Jul 2007, Joe Greco wrote: Would it be better if ISPs just blackholed certain IP addresses associated with Bot CC servers instead of trying to give the user a message. That doesn't require examining the data content of any messages. The user just gets a connection timeout. Compared to hijacking DNS and intercepting sessions? Yes. Absolutely. See, it isn't that hard to come up with better ideas. That's what Verizon was doing. Guess what. People complained about it too. Interestingly enough, some of us care. Some of us care enough to run clean networks AND to make sure that what we're selling isn't compromised by deliberate DNS hijackings and site redirections. But do include things like patching servers to filter messages that contain certain strings which might accidently catch a legitimate message on occasion. People probably complain about those things too. It sucks when you are the one that gets caught by a false positive. Unfortunately, every attempt at anti-abuse systems have experienced it at one time or another. Probably even some of the things you've done over the years trying to run a clean network has accidently made a mistake.
Re: How should ISPs notify customers about Bots (Was Re: DNS Hijacking
On 7/24/07, Chris L. Morrow [EMAIL PROTECTED] wrote: So, to back this up and get off the original complaint, if a service provider can protect a large portion of their customer base with some decent intelligence gathering and security policy implementation is that a good thing? keeping in mind that in this implementation users who know enough and are willing to forgoe that 'protection' (for some value of protection) can certainly circumvent/avoid it. Right. Let us get to best practices rather than debating ethics. So how would you keep your network clean of infected PCs? * Gather information (log parsers, darknet / honeynet traffic monitoring, feeds from XBL type blocklists) * Redirect common bot abused services like IRC by default either across your network or on whatever part of your network you see bot activity as evidenced from darknet etc observation (and run the risk that right after you get that IP information, the infected XP box on that IP is replaced not by another XP box but by a fully loaded geek install of freebsd, rather than by an infected win2k box, a patched vista etc) * Walled garden type outbound IDS to quarantine an IP completely when malware activity is noted. Yes, irc bots arent the only kind of bots - those are positively old fashioned, yes there can be multiple malware on a single PC, yes, port 25 blocking to stop bots is treating lung cancer with cough sirup (tip of the hat to Joe St.Sauver) .. etc etc etc. A good BCP would be a nice thing to have around. srs
