Re: Interesting new spam technique - getting a lot more popular.
On Jun 15, 2006, at 7:06 AM, Kristal, Jeremiah wrote: I don't think it was Extreme that filed it, or at least they didn't write it. It was the good folks at Qwest engineering who came up with the idea, which was implemented (for some low value of implemented) by Extreme. The authors had moved on by the time the RFC was published, but they were certainly Qwesties (and probably CSN before that). Nope, not at all CSN.. I *think* the same idea was floated to Cisco at the same time; their PVLAN was offered in beta not long after Extreme moved super/sub-VLANs into public release. Yep, pointer here, for folks interested in the current state of that work within the IETF: http://www.ietf.org/internet-drafts/draft-sanjib-private-vlan-05.txt Unfortunately for those of us who had to actually implement said abomination, it didn't quite work as well as promised. In fact I was just trying to decide which was more painful, taking over a hosting network with 90% of their hosts in one VLAN (VLAN2, they asked for free advice when they first started to attempt to migrate), or supporting super/sub-VLANs in an operational environment. Customers hated both, but at least they saw better performance once the hosting network was broken up per-customer VLANs. Indeed, as there's a discernible gap there from concept to implementation, implementation to network engineering, beta/EFT, and from network engineering & beta/EFT to deployment & operationalizing any such technology. If you disregard any of these phases, for whatever reason, it'll likely to come back to bite you. Customers hated it because of some very serious operational flaws. Some stuff was to be expected, like seeing broadcast traffic in all subs under a super-VLAN. As with any new technology, some amount of education is often required when change occurs. In this case, a reasonable response would be to first ask if anything was broken as a result, and if not, then to explain the benefits such a service model provides from a billing, Internet address conservation and security perspective. Customers hating something simply because they liked and are no longer seeing broadcast traffic seems a bit intractable to me. This is the perfect example where a small amount of education can go a long way. Now, if something is broken, OTOH, that's different. Some stuff was truly flawed, like having some small percentage of packets leaking across sub-VLANs. Residential customers don't mind, and probably would never notice. Large corporate clients who are putting important servers in a hosting environment get rather concerned when you start seeing traffic (including cleartext login info) from their neighbors on their interfaces. Indeed, and this is clearly an implementation bug, and potentially, a security vulnerability, if an attacker could determine how to elicit such a behavior. Trying to convince your vendor that this (and other) flaw exists when you're the only client using it in production, and you're pushing several orders of magnitude more traffic than their labs, can be slightly frustrating. Again, this is why it's important to be able to clearly articulate such a problem, and why phases 2 & 3 above are so critical, and why each new application of such a mechanism requires revisiting the entire process. I personally felt that this was a solution in search of a problem. The enterprise hosting division on an RBOC was probably not the best place to deploy it. IIRC, the "enterprise hosting solution of an RBOC" wasn't the intended initial application, though I do suspect such a solution would provide considerable advantages in a deployment such as that - again, assuming it was engineered and operationalized appropriately. The current low-end hosting environment is a problem that fits pretty well, but based on my experience in that segment, there is a much bigger return on investment in paying a couple of engineers well enough to manage your VLAN allocations correctly and use existing (generally secondary market) hardware and tools. While I'm not sure about any of the deployment details beyond the initial problem set, which falls pretty much squarely within your "that fits pretty well" category, I would be interested in hearing what your solution to such a problem is? Certainly, some amount of engineering needs to be applied, and customers that justify their own IP subnets should be handled as such, but in this day and age, I'd hope that most folks separate customers into different Link layer broadcast domains, and employ some bit of cognition WRT Internet address space conversation. -danny
Re: Interesting new spam technique - getting a lot more popular.
Once upon a time, chuck goolsbee <[EMAIL PROTECTED]> said: > * They lacked sufficient clue to grok name-based virtual hosting. Name-based virtual hosting is not a cure-all. Think about SSL and anonymous FTP uploads for starters. -- Chris Adams <[EMAIL PROTECTED]> Systems and Network Administrator - HiWAAY Internet Services I don't speak for anybody but myself - that's enough trouble.
Re: Interesting new spam technique - getting a lot more popular.
At 2:35 PM -0400 6/15/06, Matt Buford wrote: But how could this possibly be IP abuse or evil (except perhaps in the eyes of the search engines)? What difference does it make to ARIN if I give a customer 30 IPs from a single /24 or 30 IPs from 30 different /24s? How is that customer using those IPs? If the IPs are on a single server used for webhosting, it is in violation of ARIN's IPv4 allocation policy. In every case where we've seen people asking for outrageous amounts of IP space for webhosting it is either because: * They are trying to game the search engines due to this pervasive folklore. or * They lacked sufficient clue to grok name-based virtual hosting. The latter can be fixed quite easily. I wish I had some way of debunking the former. It makes little difference to me and is trivial to do in my topology since I already have 30+ /24s on the interface. Just becasue you can, doesn't mean that you should. But hey, your network, your rules I guess. It is slightly more work to document the IPs since they each have to be put into my database instead of a single range, but this is handled by the server people. I prefer to have our 'server people' and our 'network people' working together without annoying each other too much. While my use of the word "evil" was a smirking poke at the dominant search engine, I don't really think this behavior is malice so much as disregard for the ecosystem. We've done our best to be very conservative in our IP allocations to our customers, if nothing else to remain good neighbors to the rest of the Network. I wasn't even aware of this bizarre SEO/IP scheme until we made that acquisition two years ago. Now I look around and see operations a fraction of our size consuming large allocations for small installations. The pursuit of a page rank seems a pretty selfish reason to consume a limited resource. --chuck
Re: Interesting new spam technique - getting a lot more popular.
On 6/14/06, Florian Weimer <[EMAIL PROTECTED]> wrote: There are universal subscriber gateways that simply override all network configuration on the host, but they aren't marketed at datacenters AFAIK. After all, who would think that a datacenter needs a network security policy similar to that of a hotel offering Internet access in its rooms? That's the way we are using now... works very well... With a subscriber management equipment, you can put each customer in their own vlan. Each vlan is bound to a subscriber which has its ip addresses. When more addresses are requested, just add some to the subscriber. Thanks, Richard
RE: Interesting new spam technique - getting a lot more popular.
> At 7:03 PM -0400 6/14/06, Matt Buford wrote: > >There is also strong demand among web hosting customers to scatter > >sites across multiple /24's due to search engine optimization. > > I hear this line of thinking often, but to me it sounds like > bulls^X^X^X^X^X... um, "folklore". When our > customers/salesdroids ask for it, I (politely) refuse. We > acquired a hosting operation in 2004 that had blown a full > /20 on literally a rack and a half of hardware, and I was > aghast at what a nightmare that was. We're still untangling that mess. > > Anyway, if somebody could enlighten me to definitive proof, > or stated policy by Goo... er "search engines", that confirms > this "search engine result optimization by blatant abuse of > IP addresses" I'd appreciate it. I for one believe it is bunk > dreamt up by somebody trying to sell something. If it is true > though, I would have to say that it "is evil" and I would > imagine many folks here (and not to mention ARIN, RIPE, et > al) would agree. I think you're 100% right. AFAIK it *is* just folklore. But unfortunately, SEO's have to make their money somehow and all too often it seems they make their money making up crap like this. Then all the sheep that lap up every word that comes out of their favorite SEO's mouth start demanding whatever the latest craze in SEO is. This creates opposing pressures between the need to maintain a secure, reliable infrastructure and your salesdroids begging for whatever the clients are requesting. It's a tough balance to strike...best practices are all well and good, but rigid inflexibility is unlikely to win you many clients. (Especially when you consider that the vast majority of the webhosting clients out there couldn't care less about security until it affects them.) It's a shame, but the reality is I think market forces pressure most of us into making technology decisions against our better judgement from time to time. So does it surprise me in the least that there are datacenters out there running hundreds of customers out of one giant subnet? No, not one bit. Will it eventually come back to bite them, causing countless hours and $$$ to clean up the situation when it does? Inevitably. But I don't believe it's done out of ignorance in most cases. I honestly can't believe there is that much rampant incompetence out there. To me it's more likely to be a bunch of network geeks *who know better* kowtowing to pressures from management to deliver what customers are demanding, security risks be damned. But maybe that just highlights a niche market just waiting to be exploited. I imagine there's money to be made marketing security devices that allow for the convenience of being able to assign IP's on a one-by-one basis while still protecting against the various nonsense that can create, all with an easily manageable interface. Doesn't seem to far-fetched. The tools and technology already exist, just a matter of putting them all together and making it easy. Andrew Cruse
Re: Interesting new spam technique - getting a lot more popular.
"chuck goolsbee" <[EMAIL PROTECTED]> wrote: Anyway, if somebody could enlighten me to definitive proof, or stated policy by Goo... er "search engines", that confirms this "search engine result optimization by blatant abuse of IP addresses" I'd appreciate it. I for one believe it is bunk dreamt up by somebody trying to sell something. If it is true though, I would have to say that it "is evil" and I would imagine many folks here (and not to mention ARIN, RIPE, et al) would agree. Is it true? I don't know, but a quick google search returns everyone talking about it as if it is true. If it is true, is it sort of gaming the system? Yes, I suppose so. Instead of pulling 1 block of 30 from your IP allocation tool, you have to pull 30 blocks of 1. This is more administrative work and I can completely understand why someone might refuse to do it just because it is a silly hassle. But how could this possibly be IP abuse or evil (except perhaps in the eyes of the search engines)? What difference does it make to ARIN if I give a customer 30 IPs from a single /24 or 30 IPs from 30 different /24s? It makes little difference to me and is trivial to do in my topology since I already have 30+ /24s on the interface. So, I do so simply because I can't think of any reason not to. It is slightly more work to document the IPs since they each have to be put into my database instead of a single range, but this is handled by the server people.
Re: Interesting new spam technique - getting a lot more popular.
At 7:03 PM -0400 6/14/06, Matt Buford wrote: There is also strong demand among web hosting customers to scatter sites across multiple /24's due to search engine optimization. I hear this line of thinking often, but to me it sounds like bulls^X^X^X^X^X... um, "folklore". When our customers/salesdroids ask for it, I (politely) refuse. We acquired a hosting operation in 2004 that had blown a full /20 on literally a rack and a half of hardware, and I was aghast at what a nightmare that was. We're still untangling that mess. Anyway, if somebody could enlighten me to definitive proof, or stated policy by Goo... er "search engines", that confirms this "search engine result optimization by blatant abuse of IP addresses" I'd appreciate it. I for one believe it is bunk dreamt up by somebody trying to sell something. If it is true though, I would have to say that it "is evil" and I would imagine many folks here (and not to mention ARIN, RIPE, et al) would agree. --chuck
RE: Interesting new spam technique - getting a lot more popular.
Has anyone considered using sFlow to detect this type of bad behavior? Many layer 2 switches vendors mentioned in the discussion support sFlow (see http://www.sflow.org/products/network.php for a list). sFlow operates at layer 2 (think of it as a kind of remote sampled mirror port capability that lets you capture the first 128 bytes of Ethernet frames from every l2/l3 switch port in the data center). Information that you could get from sFlow that is relevant to the discussion include: ingress switch port, source and destination mac addresses, vlans, ip addresses, ARP targets and senders, layer 4 protocol and ports. Peter
RE: Interesting new spam technique - getting a lot more popular.
On Thu, 15 Jun 2006, Mikael Abrahamsson wrote: > advice when they first started to attempt to migrate), or supporting > super/sub-VLANs in an operational environment. Customers hated both, > but at least they saw better performance once the hosting network was > broken up per-customer VLANs. Why would customers hate it? We have deployed super/subvlan for residential DSL (1 static IP address per residential user) and we have no complaints afaik. Yes, if you want more flexiblity to put any IP in any vlan in any or alike, the implementation is lacking. Customers hated it because of some very serious operational flaws. Some stuff was to be expected, like seeing broadcast traffic in all subs under a super-VLAN. Some stuff was truly flawed, like having some small percentage of packets leaking across sub-VLANs. Residential customers don't mind, and probably would never notice. Large corporate clients who are putting important servers in a hosting environment get rather concerned when you start seeing traffic (including cleartext login info) from their neighbors on their interfaces. Trying to convince your vendor that this (and other) flaw exists when you're the only client using it in production, and you're pushing several orders of magnitude more traffic than their labs, can be slightly frustrating. I personally felt that this was a solution in search of a problem. The enterprise hosting division on an RBOC was probably not the best place to deploy it. The current low-end hosting environment is a problem that fits pretty well, but based on my experience in that segment, there is a much bigger return on investment in paying a couple of engineers well enough to manage your VLAN allocations correctly and use existing (generally secondary market) hardware and tools. Jeremiah
RE: Interesting new spam technique - getting a lot more popular.
On Thu, 15 Jun 2006, Kristal, Jeremiah wrote: advice when they first started to attempt to migrate), or supporting super/sub-VLANs in an operational environment. Customers hated both, but at least they saw better performance once the hosting network was broken up per-customer VLANs. Why would customers hate it? We have deployed super/subvlan for residential DSL (1 static IP address per residential user) and we have no complaints afaik. Yes, if you want more flexiblity to put any IP in any vlan in any or alike, the implementation is lacking. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
RE: Interesting new spam technique - getting a lot more popular.
On Thu, 15 Jun 2006, Mikael Abrahamsson wrote: Some ciscos can do this as well (recent IOS). IP unnumbered and static routes towards vlan interfaces means you can put customers in their own vlan and still have them be part of a larger IP subnet spanning several vlans. Since it was Extreme that filed RFC3069 I seriously doubt Cisco will ever implement it straight up. I don't think it was Extreme that filed it, or at least they didn't write it. It was the good folks at Qwest engineering who came up with the idea, which was implemented (for some low value of implemented) by Extreme. The authors had moved on by the time the RFC was published, but they were certainly Qwesties (and probably CSN before that). I *think* the same idea was floated to Cisco at the same time; their PVLAN was offered in beta not long after Extreme moved super/sub-VLANs into public release. Unfortunately for those of us who had to actually implement said abomination, it didn't quite work as well as promised. In fact I was just trying to decide which was more painful, taking over a hosting network with 90% of their hosts in one VLAN (VLAN2, they asked for free advice when they first started to attempt to migrate), or supporting super/sub-VLANs in an operational environment. Customers hated both, but at least they saw better performance once the hosting network was broken up per-customer VLANs. Jeremiah
Re: Interesting new spam technique - getting a lot more popular.
On Wed, 14 Jun 2006 11:59:51 -0700 Warren Kumari <[EMAIL PROTECTED]> wrote: > > > On Jun 14, 2006, at 2:18 AM, John van Oppen wrote: > > > > That being said, I know at least one of our transit customers does > > hosting exactly how you are describing. Coincidentally, this > > customer is also one of the customers that asked if we could "give > > them a class C block." > > Ok, I KNOW I am going to be slapped by a bunch of people here, but > > I often refer to a /24 (anywhere in the space) as a "class C". SLAP! Actually, we've recently seen an Internet service RFP requesting Class A addresses because they were "better" than Class Bs! At least they won't be asking for any Class Cs - too low rent for them ! Hmm, I've just realised that we've just been assigned a "Class A" /18, so maybe we can supply the customer "Class A, Number 1 Grade, Premium, Royal Quality IP addresses" after all. -- "Sheep are slow and tasty, and therefore must remain constantly alert." - Bruce Schneier, "Beyond Fear"
Re: Interesting new spam technique - getting a lot more popular.
On Thu, 15 Jun 2006, Chris Hills wrote: Unless I am missing something obvious, it seems like rfc 3069 (sub/super vlans) provides an easy (interim?) solution to this dilemma. Some ciscos can do this as well (recent IOS). IP unnumbered and static routes towards vlan interfaces means you can put customers in their own vlan and still have them be part of a larger IP subnet spanning several vlans. Since it was Extreme that filed RFC3069 I seriously doubt Cisco will ever implement it straight up. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: Interesting new spam technique - getting a lot more popular.
Bill Nash wrote: > Trying to migrate customers to their own vlan when they've been alloted > IPs, willy nilly, across one of the bajillion /24's secondaried on the > vlan interface drives me into an entire new dimension of pissed off. Unless I am missing something obvious, it seems like rfc 3069 (sub/super vlans) provides an easy (interim?) solution to this dilemma.
Re: Interesting new spam technique - getting a lot more popular
* A spamware daemon is installed on the dedicated server, to keep the network interface in promiscuous mode * The daemon determines which IP addresses on the local subnet are not in use. It also determines the addresses of the network routers. One or more unused IP addresses are commandeered for use by the spammer. * The perp server sends unrequested ARP responses to only the gateway routers, so that the routers never have to ask for a layer-3 to layer-2 association -- it's alway in the ARP cache of the routers. Nobody else sees this traffic in an EtherSwitch fabric, so ARPWATCH and its kin are defeated. Pings and traceroutes also fail with "host unreachable.". The daemon then only has to watch on the NIC, in promiscuous mode, for TCP packets to the hijacked address on port 80, and pass them down the tunnel to the remote Web server. * Finally, GRE and IPIP tunneling is used to connect the stolen IP addresses to the spammer's real servers hosted elsewhere. The end result is that the spammer has created a server at an IP address which not even the owners of the network are aware of. And if one went to http://www.senderbase.org/ and monitored their own IP block, wouldn't the spammer appear there? Or just plain monitoring spikes in outgoing port 25 traffic should alert someone that something is amiss. -Hank
Re: Interesting new spam technique - getting a lot more popular.
And let me tell you.. inheriting a network like that, knowing a better way to do it, will make you want to put a gun in your mouth. Two /19's worth of address space in VLAN1 (not just in one vlan, but in vlan *1*. Cisco nerds are slapping foreheads or spitting Coke right now.) Trying to migrate customers to their own vlan when they've been alloted IPs, willy nilly, across one of the bajillion /24's secondaried on the vlan interface drives me into an entire new dimension of pissed off. Don't even get me started on allocation and traffic accounting. - billn On Wed, 14 Jun 2006, Richard A Steenbergen wrote: On Wed, Jun 14, 2006 at 07:03:10PM -0400, Matt Buford wrote: As a hoster with many customers on large shared VLANs perhaps I can add a bit... Note that if you're reading this list, you have already identified yourself as a non-typical hoster. Go read WHT or GFY for 10 minutes for an example of typical hosters, and if you're not a drooling idiot in need of a brain transplant afterwards consider yourself lucky. :) And don't forget, there are hundreds of hosting networks like the ones I described, a lot of whom are in the 1 - 30Gbps traffic range, with absolutely no clue how to do better. -- Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Interesting new spam technique - getting a lot more popular.
On Wed, Jun 14, 2006 at 07:03:10PM -0400, Matt Buford wrote: > As a hoster with many customers on large shared VLANs perhaps I can add a > bit... Note that if you're reading this list, you have already identified yourself as a non-typical hoster. Go read WHT or GFY for 10 minutes for an example of typical hosters, and if you're not a drooling idiot in need of a brain transplant afterwards consider yourself lucky. :) And don't forget, there are hundreds of hosting networks like the ones I described, a lot of whom are in the 1 - 30Gbps traffic range, with absolutely no clue how to do better. -- Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Interesting new spam technique - getting a lot more popular.
As a hoster with many customers on large shared VLANs perhaps I can add a bit... "Richard A Steenbergen" <[EMAIL PROTECTED]> wrote: Simple: Subnets are hard, customers are stupid, and ARIN is not exactly a hosters best friend. When a hosting customer asks for 5 IPs today and 25 IPs tomorrow, it is infinitely easier for the hosting folks to just slap them into /24s and say "ok uhm you are now .69-.94" than to try and explain subnets, cidr, reserving IP space in cidr sized blocks etc to the customer. Hosters are also generally under-equipped in the paperwork and detailed documentation department, so they tend to run their IP allocations into the ground while attempting to explain their need for more space. CIDR allocations are "wasteful" to them, especially when a customer needs to expand from 30 IPs to 35 IPs and crosses a new boundry. I'm not sure documentation is really THAT much of it. I mean, we have subnets behind firewalls and manage subnet allocations there. It is a hassle compared to the simplicity of large shared VLANs, but we're certainly capable of tracking it. The problem is more for the customers with only a few servers or even just a single server. They tend to have no idea about future IP requirements. Ask any hoster CEO who isn't a hands-on networking person what his expectations are for near-future IPs and he'll generally give you some wild answer like "up to 50,000" because he wouldn't be CEO if he didn't expect his business to explode overnight. :) In general, customers with larger firewalled solutions (and thus a private subnet and private behind-firewall VLAN) tend to have a better handle on this. Also, have a requirement that I must be able to accept a customer server plugging into my network anywhere. If a customer buys 1 server today, then another server 6 months from now, that second server may be on a different floor of the datacenter, or even a few miles away in a nearby datacenter. A few months after setting these servers up, the customer might want to migrate a single IP from one server to another. So, since I must carry every VLAN everywhere, that can get tricky (not impossible, but messy) when you have thousands of customers, with say 2-5 VLANs each. With MSTP the spanning tree limit of a 6500 is 50,000 logical interfaces (ports*vlans). At 5000 VLANs that would mean I'm only allowed to use 10 all-vlan-carrying tagged ports on a 6500, which wouldn't make for a very efficient core. There is also strong demand among web hosting customers to scatter sites across multiple /24's due to search engine optimization. This is trivial to satisfy in the large shared subnet model. Finally, as we all know, there is the efficiency issue. Of course, as long as ARIN lets people get away with it, it isn't that big of a deal (other than being good netizens) so I won't go into this one much. Incase you've never seen hoster configs, they generally look a little something like this: ip address 1.1.1.1 255.255.255.0 ip address 1.1.2.1 255.255.255.0 secondary ip address 1.1.3.1 255.255.255.0 secondary ip address 1.1.4.1 255.255.255.0 secondary ip address 1.1.5.1 255.255.255.0 secondary Yep, the ip address section typically looks like that, plus all the usual stuff like GLBP, policy routing, etc... Anything else is quite honestly beyond 99% of hosters out there, they're still blissfully calling these things "class c's". I've seen some truly godawful thins configured by hosters, like chains of 3548s all linking back to a single router interface in ways you can't even imagine. I can't speak for other hosters, but for me it is more about different customer bases (some customers have no idea what their requirements are) and different business requirements. I think we are quite aware of the issues either way, and know exactly what the advantages and disadvantages are. If anything, I'd say we're very much experts in the field of large layer 2 networks. Oh, and there are no chains of 3548's here. All 6500s, each one directly attached to at least 2 cores. If you made it dirt simple for them they would probably be doing something better (I usually point folks who ask to pvlans, then take the opportunity to make a hasty retreat while they are distracted), but otherwise they don't see the benefit in it. Why bother configuring your router better when you can just send your $5/hr monkey over with a redhat cd and have them reinstall, right? :) We use pvlans successfully (though it has been a long bug-ridden journey). This mitigates just about all of the disadvantages of the big shared VLAN while maintaining all the advantages. The one disadvantage that pvlans does not mitigate is unused IPs. Thus, I have been lobbying Cisco since 2002 to fix all the pvlan bugs (almost done there!) and for the ability to add bulk static arps and stop even supporting dynamic ARPing for customer IPs (no real progress on this front). For now we have
RE: Interesting new spam technique - getting a lot more popular.
On Wed, 14 Jun 2006, Church, Chuck wrote: > > Since this technique requires a IPinIP or GRE tunnel, wouldn't blocking > these two protocols to/from the hosts be sufficient? Assuming of course > the customer's host isn't using that normally. sure, but those are probably just convenience things, what happens when they setup an ssl tunnel? or http tunnel or ping tunnel?
Re: Interesting new spam technique - getting a lot more popular.
On Jun 14, 2006, at 2:18 AM, John van Oppen wrote: That being said, I know at least one of our transit customers does hosting exactly how you are describing. Coincidentally, this customer is also one of the customers that asked if we could "give them a class C block." Ok, I KNOW I am going to be slapped by a bunch of people here, but I often refer to a /24 (anywhere in the space) as a "class C". I also call the thingie on my digital watch an LCD display, the thing that stops breaks from locking the ABS system and the number I type into the ATM machine my PIN number. Oh yeah, my DLT tape drive is connected to a SCSCI interface. Yup, all of the above are technically incorrect (ok, most of them are just redundant), but I do it anyway, and I am going to carry on doing it, so there! W -- "Working the ICANN process is like being nibbled to death by ducks, it takes forever, it doesn't make sense, and in the end we're still dead in the water." -- Tom Galvin, VeriSign's vice president for government relations.
Re: Interesting new spam technique - getting a lot more popular.
On Jun 14, 2006, at 1:53 PM, Church, Chuck wrote: Since this technique requires a IPinIP or GRE tunnel, wouldn't blocking these two protocols to/from the hosts be sufficient? Assuming of course the customer's host isn't using that normally. Unfortunately, that probably won't work for very long, if at all. First, it's kinda difficult to guarantee your customers will not use a protocol. Second, unless you have deep packet inspection, what is to stop the spammer from using, say, port 80 for their tunnel? Third, what's to stop them from using SSH tunnels? Etc., etc., etc -- TTFN, patrick
RE: Interesting new spam technique - getting a lot more popular.
Since this technique requires a IPinIP or GRE tunnel, wouldn't blocking these two protocols to/from the hosts be sufficient? Assuming of course the customer's host isn't using that normally. Chuck Netco Government Services has recently acquired Multimax and is changing its name to Multimax Inc. Visit http://www.multimax.com for more information.
Re: Interesting new spam technique - getting a lot more popular.
> "Mikael" == Mikael Abrahamsson <[EMAIL PROTECTED]> writes: > On Wed, 14 Jun 2006, Christopher L. Morrow wrote: >> is it really that hard to make your foudry/extreme/cisco l3 switch >> vlan and subnet??? Is this a education thing or a laziness thing? >> Is this perhaps covered in a 'bcp' (not even an official IETF >> thing, just a hosters bible sort of thing) ? Mikael> This problem is fixed by following the BCP regarding spoof Mikael> filtering, Only if you also filter _OUTGOING_ traffic, by port, to allow only the destination IPs that the customer equipment should be seeing. Filtering the ingress direction (customer equipment -> your network) does not help (until _everyone_ does it), since the spammer only needs to _receive_ traffic with the hijacked IP, not send it (that can be done from the other corner of the spammer's triangle route). -- Andrew, Supernews http://www.supernews.com
Re: Interesting new spam technique - getting a lot more popular.
* Christopher L. Morrow: > is it really that hard to make your foudry/extreme/cisco l3 switch vlan > and subnet??? Is this a education thing or a laziness thing? You need those L3 switches before you can do this. Obviously, L2 gear is much cheaper, and will work equally well until it is attacked. Even with L3 switches, adressing it is a bit tricky. Not all vendors support point-to-point adressing on Ethernet interfaces (certainly some host software doesn't). There are universal subscriber gateways that simply override all network configuration on the host, but they aren't marketed at datacenters AFAIK. After all, who would think that a datacenter needs a network security policy similar to that of a hotel offering Internet access in its rooms?
Re: Interesting new spam technique - getting a lot more popular.
* Christopher L. Morrow: > On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: >> >> http://thespamdiaries.blogspot.com/2006/02/new-host-cloaking-technique-used-by.html >> >> * Monitor your local network for interfaces transmitting ARP >> responses they shouldn't be. > > how about just mac security on switch ports? limit the number of mac's at > each port to 1 or some number 'valid' ? The attack is not visible at layer 2, so this won't help. You need static ARP tables on relevant hosts, but even that is only a stopgag measure. Better invest into one (virtual) router port per customer host. 8-/
Re: Interesting new spam technique - getting a lot more popular.
On Wed, 2006-06-14 at 05:28 +, Edward B. DREGER wrote: > CLM> Date: Wed, 14 Jun 2006 04:46:31 + (GMT) > CLM> From: Christopher L. Morrow > > CLM> is it really that hard to make your foudry/extreme/cisco l3 switch vlan > CLM> and subnet??? > > Of course not. > > > CLM> Is this a education thing or a laziness thing? > > Both. And in some cases even a nasty fincancial thing. Billing customers extra datatraffic due to a large amount of broadcast traffic (especially when running badly configured Win32 servers) inside a single /23 or even /22 in one large VLAN is sadly still the case for some hosters. -- --- Erik Haagsman Network Architect We Dare BV Tel: +31(0)10-7507008 Fax: +31(0)10-7507005 http://www.we-dare.nl
RE: Interesting new spam technique - getting a lot more popular.
> is it really that hard to make your foudry/extreme/cisco l3 switch vlan
> and subnet??? Is this a education thing or a laziness thing? Is this
> perhaps covered in a 'bcp' (not even an official IETF thing, just a
> hosters bible sort of thing) ?
Subnets aren't exactly good for address space usage.
For Cisco kit, there are numerous nerd knobs that can be deployed that would
seemingly mitigate this spam technique.
In short, IP Source Guard ("stop malicious people from using IP addresses
that weren't assigned to them"), Port Security ("limit # of mac addresses on
a given port to X") and Dynamic ARP Inspection ("discard bogus arp
packets").
Combined with things like Private VLANs (allow different customers to share
the same subnet but restrict them being able to talk/see one another), there
are ways of securing things.
Of course, just like everything its up to folks to deploy them. Many of
these knobs aren't safe or practical for "default" settings.
I'm sure other vendors have similar features also.
Yes, these have been presented on numerous times within Cisco forums (e.g.
Networkers) as best practice & are typically very well attended.
Not necessarily by the all the folk that need to, I guess. :(
cheers,
lincoln.
Re: Interesting new spam technique - getting a lot more popular.
On Wed, 14 Jun 2006, Christopher L. Morrow wrote: | how about just mac security on switch ports? limit the number of mac's at | each port to 1 or some number 'valid' ? Hi, Just to be clear, simple L2 mac security doesn't help here. This attack (arp spoofing on a shared subnet) does not involve more than one mac per switch port. Nor are there any changes in switch port / mac associations. You need to watch at the higher layers (arp, ip). Cheers -- Chris Edwards, Glasgow University Computing Service
RE: Interesting new spam technique - getting a lot more popular.
We end up with customers asking for more IPs too. We just add additional subnets to the interface, perhaps they started with a /30 but now need three more IPs, we just add an additional /29 to the interface leaving both blocks. It is not often that anything needs to be explained to the customer other than the correct subnet mask and gateway for the IPs. This makes our configs look like this for each customer vlan: ip address 2.2.2.9 255.255.255.252 ip address 3.3.2.129 255.255.255.224 secondary That being said, I know at least one of our transit customers does hosting exactly how you are describing. Coincidentally, this customer is also one of the customers that asked if we could "give them a class C block." Using this strategy has never been a problem with ARIN for us, in fact I have applied for and received more space at intervals between 6 and 14 months for the last four years without any issue at all. John :) -Ursprüngliche Nachricht- Von: Richard A Steenbergen [mailto:[EMAIL PROTECTED] Gesendet: Wednesday, June 14, 2006 12:18 AM An: Christopher L. Morrow Cc: NANOG Betreff: Re: Interesting new spam technique - getting a lot more popular. On Wed, Jun 14, 2006 at 04:46:31AM +, Christopher L. Morrow wrote: > > is it really that hard to make your foudry/extreme/cisco l3 switch vlan > and subnet??? Is this a education thing or a laziness thing? Is this > perhaps covered in a 'bcp' (not even an official IETF thing, just a > hosters bible sort of thing) ? Simple: Subnets are hard, customers are stupid, and ARIN is not exactly a hosters best friend. When a hosting customer asks for 5 IPs today and 25 IPs tomorrow, it is infinitely easier for the hosting folks to just slap them into /24s and say "ok uhm you are now .69-.94" than to try and explain subnets, cidr, reserving IP space in cidr sized blocks etc to the customer. Hosters are also generally under-equipped in the paperwork and detailed documentation department, so they tend to run their IP allocations into the ground while attempting to explain their need for more space. CIDR allocations are "wasteful" to them, especially when a customer needs to expand from 30 IPs to 35 IPs and crosses a new boundry. Incase you've never seen hoster configs, they generally look a little something like this: ip address 1.1.1.1 255.255.255.0 ip address 1.1.2.1 255.255.255.0 secondary ip address 1.1.3.1 255.255.255.0 secondary ip address 1.1.4.1 255.255.255.0 secondary ip address 1.1.5.1 255.255.255.0 secondary ... Anything else is quite honestly beyond 99% of hosters out there, they're still blissfully calling these things "class c's". I've seen some truly godawful thins configured by hosters, like chains of 3548s all linking back to a single router interface in ways you can't even imagine. If you made it dirt simple for them they would probably be doing something better (I usually point folks who ask to pvlans, then take the opportunity to make a hasty retreat while they are distracted), but otherwise they don't see the benefit in it. Why bother configuring your router better when you can just send your $5/hr monkey over with a redhat cd and have them reinstall, right? :) -- Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Interesting new spam technique - getting a lot more popular.
On Wed, Jun 14, 2006 at 04:46:31AM +, Christopher L. Morrow wrote: > > is it really that hard to make your foudry/extreme/cisco l3 switch vlan > and subnet??? Is this a education thing or a laziness thing? Is this > perhaps covered in a 'bcp' (not even an official IETF thing, just a > hosters bible sort of thing) ? Simple: Subnets are hard, customers are stupid, and ARIN is not exactly a hosters best friend. When a hosting customer asks for 5 IPs today and 25 IPs tomorrow, it is infinitely easier for the hosting folks to just slap them into /24s and say "ok uhm you are now .69-.94" than to try and explain subnets, cidr, reserving IP space in cidr sized blocks etc to the customer. Hosters are also generally under-equipped in the paperwork and detailed documentation department, so they tend to run their IP allocations into the ground while attempting to explain their need for more space. CIDR allocations are "wasteful" to them, especially when a customer needs to expand from 30 IPs to 35 IPs and crosses a new boundry. Incase you've never seen hoster configs, they generally look a little something like this: ip address 1.1.1.1 255.255.255.0 ip address 1.1.2.1 255.255.255.0 secondary ip address 1.1.3.1 255.255.255.0 secondary ip address 1.1.4.1 255.255.255.0 secondary ip address 1.1.5.1 255.255.255.0 secondary ... Anything else is quite honestly beyond 99% of hosters out there, they're still blissfully calling these things "class c's". I've seen some truly godawful thins configured by hosters, like chains of 3548s all linking back to a single router interface in ways you can't even imagine. If you made it dirt simple for them they would probably be doing something better (I usually point folks who ask to pvlans, then take the opportunity to make a hasty retreat while they are distracted), but otherwise they don't see the benefit in it. Why bother configuring your router better when you can just send your $5/hr monkey over with a redhat cd and have them reinstall, right? :) -- Richard A Steenbergen <[EMAIL PROTECTED]> http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Interesting new spam technique - getting a lot more popular.
CLM> Date: Wed, 14 Jun 2006 04:46:31 + (GMT) CLM> From: Christopher L. Morrow CLM> is it really that hard to make your foudry/extreme/cisco l3 switch vlan CLM> and subnet??? Of course not. CLM> Is this a education thing or a laziness thing? Both. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
RE: Interesting new spam technique - getting a lot more popular.
JvO> Date: Tue, 13 Jun 2006 21:35:14 -0700 JvO> From: John van Oppen JvO> It sure seems like this is a good demo of the best practice of JvO> having customers on their own VLANs with their own subnets. We JvO> have been doing this since we started offering colo services, is We actually go so far as to isolate certain services on their own subnet/VLAN. JvO> this less common than I thought? I'm afraid so. I've worked on a good many networks where everything is in one VLAN; a common argument for the practice is IP assignment granularity. Rarely do I find MAC ACLs in place at the switch. (I'm actually trying to remember a specific installation that had MAC filtering set up by a prior engineer... I'm _sure_ I've encountered at least a couple.) Note that these observations are for small- and mid-sized networks. Maybe things are better in the larger networks. YMMV. Eddy -- Everquick Internet - http://www.everquick.net/ A division of Brotsman & Dreger, Inc. - http://www.brotsman.com/ Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 785 865 5885 Lawrence and [inter]national Phone: +1 316 794 8922 Wichita DO NOT send mail to the following addresses: [EMAIL PROTECTED] -*- [EMAIL PROTECTED] -*- [EMAIL PROTECTED] Sending mail to spambait addresses is a great way to get blocked. Ditto for broken OOO autoresponders and foolish AV software backscatter.
Re: Interesting new spam technique - getting a lot more popular.
On Wed, 14 Jun 2006, Christopher L. Morrow wrote: is it really that hard to make your foudry/extreme/cisco l3 switch vlan and subnet??? Is this a education thing or a laziness thing? Is this perhaps covered in a 'bcp' (not even an official IETF thing, just a hosters bible sort of thing) ? This problem is fixed by following the BCP regarding spoof filtering, if needed, doing the IP source filtering at the switchport instead of at the router level. Treat your colo customers the same way you would residential customers with the same security level. Whatever the customer himself can change, control. IP spoof filtering, and if your platform supports it, even rewrite the MAC address so it's local to the access cable and not used in your aggregation network (some DSLAM vendors do this, for instance). I haven't seen any switch vendors that does this yet, unfortunately. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: Interesting new spam technique - getting a lot more popular.
That’s a very good question... I was also under the assumption that most providers would have adopted new practices rather then simply dumping customers on a single subnet/vlan... unless were going back in time :P As far as the "special daemon program" goes.. any packet sniffer will reveal all needed information to jack an ip. I'm actually surprised that its taken spammers this long to figure out and utilize such vulnerabilities in networks... seeing how spamming is a multi billion $ industry... few ways to limit ip jackings... keep your subnets small as possible, force the use of private vlans, as a provider... you should provide a way for your clients to be able to view their traffic patterns... in case of a hijack, they would notice the increased traffic and could bring it to the providers attention sooner then later... monitor your switch ports (snmp?) for bursts of outbound traffic (bandwidth / pps)... -- Payam Chychi John van Oppen wrote: It sure seems like this is a good demo of the best practice of having customers on their own VLANs with their own subnets. We have been doing this since we started offering colo services, is this less common than I thought? John -Ursprüngliche Nachricht- Von: Christopher L. Morrow [mailto:[EMAIL PROTECTED] Gesendet: Tuesday, June 13, 2006 9:23 PM An: Suresh Ramasubramanian Cc: NANOG Betreff: Re: Interesting new spam technique - getting a lot more popular. On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: That was not my advice btw - just forwarding on what I saw. oh,. apologies, i did cut the message down quite a bit :( I understood you were quoting from the spamdiaries website, I apologize to the other listeners (readers?) if it confused the issue. What you say does seem like a "must do" all right - but putting ARP filters in is actually a reasonable idea. Atleast it'd trim down the 'problem' to the single customer subnet, I assume that dedicated hosting folks don't just drop machines behind a switch on one big flat subnet? That's probably a naive assumption though :( Perhaps this is clue #12 that that is a 'less than good' option? :) On 6/14/06, Christopher L. Morrow <[EMAIL PROTECTED]> wrote: On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: http://thespamdiaries.blogspot.com/2006/02/new-host-cloaking-technique-used-by.html * Monitor your local network for interfaces transmitting ARP responses they shouldn't be. how about just mac security on switch ports? limit the number of mac's at each port to 1 or some number 'valid' ? -- Suresh Ramasubramanian ([EMAIL PROTECTED])
Re: Interesting new spam technique - getting a lot more popular.
On Wed, 14 Jun 2006, Adam Rothschild wrote: > On 2006-06-14-00:23:15, "Christopher L. Morrow" <[EMAIL PROTECTED]> wrote: > [...] > > I assume that dedicated hosting folks don't just drop machines > > behind a switch on one big flat subnet? That's probably a naive > > assumption though > > I've long been a proponent of a per-customer VLAN or L3 interface, > depending on what the topology allows for, but I'm afraid we're in the > minority. doh :( > > >From what I've seen, the overwhelming majority of "dedicated hosters" > do precisely what the article alludes to -- placing hundreds (if not > thousands!) of disparate hosts on the same broadcast domain, with no > safeguards in place to prevent ARP spoofing, IP hijacking, and other > forms of malfeasance... > is it really that hard to make your foudry/extreme/cisco l3 switch vlan and subnet??? Is this a education thing or a laziness thing? Is this perhaps covered in a 'bcp' (not even an official IETF thing, just a hosters bible sort of thing) ?
Re: Interesting new spam technique - getting a lot more popular.
On 2006-06-14-00:23:15, "Christopher L. Morrow" <[EMAIL PROTECTED]> wrote: [...] > I assume that dedicated hosting folks don't just drop machines > behind a switch on one big flat subnet? That's probably a naive > assumption though I've long been a proponent of a per-customer VLAN or L3 interface, depending on what the topology allows for, but I'm afraid we're in the minority. >From what I've seen, the overwhelming majority of "dedicated hosters" do precisely what the article alludes to -- placing hundreds (if not thousands!) of disparate hosts on the same broadcast domain, with no safeguards in place to prevent ARP spoofing, IP hijacking, and other forms of malfeasance... -a
Re: Interesting new spam technique - getting a lot more popular.
On 6/14/06, Christopher L. Morrow <[EMAIL PROTECTED]> wrote: Atleast it'd trim down the 'problem' to the single customer subnet, I assume that dedicated hosting folks don't just drop machines behind a switch on one big flat subnet? That's probably a naive assumption though :( Perhaps this is clue #12 that that is a 'less than good' option? :) Given the people who complained, and their traditionally spammer infested nature I wouldnt be surprised at all to find that they've put all their hosts on a flat subnet Various /24s of theirs keep getting complimentary upgrades from our filters after reaching a certain limit - based on a X IPs blocked per /24, Y /24s per /16 metric .. when that limit is reached, we automatically upgrade the blocks to cover infested /24s.
RE: Interesting new spam technique - getting a lot more popular.
It sure seems like this is a good demo of the best practice of having customers on their own VLANs with their own subnets. We have been doing this since we started offering colo services, is this less common than I thought? John -Ursprüngliche Nachricht- Von: Christopher L. Morrow [mailto:[EMAIL PROTECTED] Gesendet: Tuesday, June 13, 2006 9:23 PM An: Suresh Ramasubramanian Cc: NANOG Betreff: Re: Interesting new spam technique - getting a lot more popular. On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: > That was not my advice btw - just forwarding on what I saw. > oh,. apologies, i did cut the message down quite a bit :( I understood you were quoting from the spamdiaries website, I apologize to the other listeners (readers?) if it confused the issue. > What you say does seem like a "must do" all right - but putting ARP > filters in is actually a reasonable idea. > Atleast it'd trim down the 'problem' to the single customer subnet, I assume that dedicated hosting folks don't just drop machines behind a switch on one big flat subnet? That's probably a naive assumption though :( Perhaps this is clue #12 that that is a 'less than good' option? :) > On 6/14/06, Christopher L. Morrow > <[EMAIL PROTECTED]> wrote: > > > > On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: > > > > > > http://thespamdiaries.blogspot.com/2006/02/new-host-cloaking-technique-used-by.html > > > > > > * Monitor your local network for interfaces transmitting ARP > > > responses they shouldn't be. > > > > how about just mac security on switch ports? limit the number of mac's at > > each port to 1 or some number 'valid' ? > > > > > -- > Suresh Ramasubramanian ([EMAIL PROTECTED]) >
Re: Interesting new spam technique - getting a lot more popular.
On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: > That was not my advice btw - just forwarding on what I saw. > oh,. apologies, i did cut the message down quite a bit :( I understood you were quoting from the spamdiaries website, I apologize to the other listeners (readers?) if it confused the issue. > What you say does seem like a "must do" all right - but putting ARP > filters in is actually a reasonable idea. > Atleast it'd trim down the 'problem' to the single customer subnet, I assume that dedicated hosting folks don't just drop machines behind a switch on one big flat subnet? That's probably a naive assumption though :( Perhaps this is clue #12 that that is a 'less than good' option? :) > On 6/14/06, Christopher L. Morrow > <[EMAIL PROTECTED]> wrote: > > > > On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: > > > > > > http://thespamdiaries.blogspot.com/2006/02/new-host-cloaking-technique-used-by.html > > > > > > * Monitor your local network for interfaces transmitting ARP > > > responses they shouldn't be. > > > > how about just mac security on switch ports? limit the number of mac's at > > each port to 1 or some number 'valid' ? > > > > > -- > Suresh Ramasubramanian ([EMAIL PROTECTED]) >
Re: Interesting new spam technique - getting a lot more popular.
That was not my advice btw - just forwarding on what I saw. What you say does seem like a "must do" all right - but putting ARP filters in is actually a reasonable idea. On 6/14/06, Christopher L. Morrow <[EMAIL PROTECTED]> wrote: On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: > > http://thespamdiaries.blogspot.com/2006/02/new-host-cloaking-technique-used-by.html > > * Monitor your local network for interfaces transmitting ARP > responses they shouldn't be. how about just mac security on switch ports? limit the number of mac's at each port to 1 or some number 'valid' ? -- Suresh Ramasubramanian ([EMAIL PROTECTED])
Re: Interesting new spam technique - getting a lot more popular.
On Wed, 14 Jun 2006, Suresh Ramasubramanian wrote: > > http://thespamdiaries.blogspot.com/2006/02/new-host-cloaking-technique-used-by.html > > * Monitor your local network for interfaces transmitting ARP > responses they shouldn't be. how about just mac security on switch ports? limit the number of mac's at each port to 1 or some number 'valid' ?
Interesting new spam technique - getting a lot more popular.
http://thespamdiaries.blogspot.com/2006/02/new-host-cloaking-technique-used-by.html Does seem to have potential, because at least one large webhost says they got bit hard by this (when they asked me to unblock one of their /24s) - and I've been seeing the same type of spam for quite some time [pizzabox cpanel servers running exim, but emitting porn spam with completely different hostnames and qmail headers] So this brings us today to a new technique reported by Stephen Satchell of satchell.net last Thursday. It reads almost like a mystery novel, involving cloaking, promiscuous interfaces, stolen IP addresses, and tunneling. It gets a little tricky, so follow the bouncing ball: * The spammer obtains a dedicated server at the victim service provider. The server shares a subnet with other customers. * A spamware daemon is installed on the dedicated server, to keep the network interface in promiscuous mode * The daemon determines which IP addresses on the local subnet are not in use. It also determines the addresses of the network routers. One or more unused IP addresses are commandeered for use by the spammer. * The perp server sends unrequested ARP responses to only the gateway routers, so that the routers never have to ask for a layer-3 to layer-2 association -- it's alway in the ARP cache of the routers. Nobody else sees this traffic in an EtherSwitch fabric, so ARPWATCH and its kin are defeated. Pings and traceroutes also fail with "host unreachable.". The daemon then only has to watch on the NIC, in promiscuous mode, for TCP packets to the hijacked address on port 80, and pass them down the tunnel to the remote Web server. * Finally, GRE and IPIP tunneling is used to connect the stolen IP addresses to the spammer's real servers hosted elsewhere. The end result is that the spammer has created a server at an IP address which not even the owners of the network are aware of. There are a number of ways you can protect your own network from from this exploit: * Give each customer their own subnet. * Null-route unused IP addresses in your network space, or otherwise make sure that there's a legitimate server somewhere that will claim them. * Monitor your local network for interfaces transmitting ARP responses they shouldn't be. -- Suresh Ramasubramanian ([EMAIL PROTECTED])
