I hear ya... hate to be in reactive mode as
well. When sharing pipes you only have two choices, clamp bandwidth down
for any single customer to the point that one or more customers' abuse won't
impact others, or react via monitoring.
For the specific cases you outlined, it sounds like
IMGate might help. We don't use it, but from what I've read on the
lists, it sounds like it could be configured to protect against these
scenarios.
Darin.
----- Original Message -----
Sent: Wednesday, February 16, 2005 3:51 PM
Subject: Re: [Declude.JunkMail] OT: Switch to control
bandwidth
I've got a nice solution for this called IPcheck Server Monitor
from Paessler (http://www.paessler.com/products/ipcheck/?link=menu).
It is buggy however from the standpoint of the interface, though they have been
continually improving and fixing it. It has nice notification features
such as dependencies. I monitor from a separate network from the servers
themselves, giving a good impression of what my customers experience.
I
kind of feel tied down to monitoring and then researching, and then resolving
issues. For instance, in the last 24 hours I had the following
happen:
- Customer's mail server IP changed without
notifying us, causing lots of spooled messages. - One
person sent about 150 MB of E-mail (separate large messages) to a MailPure
protected account and that slowed down
our responsiveness on all services (HTTP and POP3 also affected. No fix
can be applied for this except for
limiting as the traffic was legit, just unusually bursty.
- Customer's remote Web server misbehaving and delivered 10,000 messages to
their own account through our
gateway. No immediate issues, but it bears watching for potential
escalation that could threaten our
performance.
I'm spending a lot of time with this stuff on a regular
basis and want to be a bit more proactive so that I don't need to feel tied
down. Almost all issues can be controlled by rate limiting, though some
require extensive granularity to achieve sufficient protection. QOS is
also at issue since I stay away from doing inexpensive services, concentrating
on value-added, and many of my customers do expect more.
Right now I'm
fine, but the more I grow, the more often these things will occur and I think
it's time to put something else in place that can stop issues from potentially
affecting every service/customer.
Matt
Darin Cox wrote:
Best solution is monitoring. Without
creating a system of dedicated circuits to each customer you can't guarantee
one customer will not adversely affect another. Rate-limiting at the
switch (or software "switch") will help, but still means a smaller pipe for
everyone else...and doesn't help with multiple customers
misbehaving.
With appropriate SNMP or RMON monitoring,
however, you can be notified as soon as traffic goes beyond a given threshold
and react accordingly.
Darin.
-----
Original Message -----
Sent: Wednesday, February 16, 2005 3:18 PM
Subject: Re: [Declude.JunkMail] OT: Switch to control
bandwidth
I just wanted to follow up on this thread. First, thanks
for all of the suggestions. Here's a summary of what caught my eye.
1) There are some decent choices out there, and seemingly a 3COM
SuperStack 3 3226 comes at a nice price point (around $500) and allows
limiting per port at 1 Mbps increments and also does 7 custom levels of
protocol prioritization. This was suggested to me off-list. It
seems like a good thing for colocation since you don't care for more
granularity among your customers, they can choose to do with their bandwidth
what they wish. I'm not into colocation yet and this probably falls
short of my needs otherwise.
2) I was also intrigued by the
NetEqualizer product, which seems to be a the commercial version of an open
source project called Linux Bandwidth Arbitrator (www.bandwidtharbitrator.com).
This might very well offer functionality beyond all of the switches, but
offers more complication in setup and management unless you go with the
for-profit version. This is of course not a switch, but that's ok
since cheap switches can be placed behind it.
3) Cisco is of course a
popular choice, but I'm not a fan of their ridiculous licensing schemes for
the software and high prices. Used, these things come fairly cheap,
but they are the 'Outlook' of routers and switches, and the most likely to
be targeted by exploits. For that reason, I am probably going to
migrate away from anything Cisco once I outgrow what I already have. I
may change my mind however.
4) I don't think I need a firewall, or
don't want to deal with the expense and limitations of it (concurrent
sessions, etc.). I have so few ports open that I'm fine with router
level protection and this is exclusively a DMZ with no client computers
behind it.
Despite what these products offer, I still
think that the switches generally come up short of being a perfect solution to
my needs (that of a Web hosting/E-mail provider). I essentially have 5
services that I need to support across 3 machines; HTTP, FTP, DNS, SMTP, and
POP3. It seems that by just simply bandwidth limiting a port, I won't be
able to slow down but a portion of the problematic bandwidth and there can be
other issues caused by that (such as limiting all HTTP because of one site
that is getting hammered). It would be best to limit HTTP by IP instead
of by port. I haven't tested it out yet, but it may be that IIS will
actually work when limiting in Windows 2003 unlike 2k, and that may solve my
issue on that front at least. FTP may or may not be covered by the same,
I'm not sure yet.
It seems however that some of the worst issues are
coming from fairly unique situations and specific IP addresses.
Conditions like E-mail loops can not only bring down a mail server, but also
bring down a whole network if all of your bandwidth is used. This of
course can also affect POP3 service. If a customer does a mass mailing with
huge images sourced from their site, the bandwidth could also bring us down
without limits. I even had a customer send 144 messages out the other
day with a 2.5 MB attachment, and if you do the math, you will find that this
was 400 MB of bandwidth that IMail naturally attempts to deliver ASAP.
I've also noted that IMail doesn't do well with response times under heavy
bandwidth load even if the CPU is fine while other services on the same box
have far less latency. This affects the quality of service to my
customers, and I like things to be responsive.
So what I am really
looking for is some way to protect Web hosting clients from another Web
hosting client's issue, protect POP3 service from having the bandwidth
bogarted by some SMTP loop, or FTP, or HTTP, etc. Since everyone shares
the same MX records, and the same outgoing SMTP and POP3, it's hard to find
decent separation unless I get down to the IP level and start limiting things
based on at least the destination IP if not the source IP also. To do
anything less would seem to be somewhat futile because I would continue to
have sporadic issues with the most problematic things which can be long-lived
to the point that they are resolved/blocked (DOS or loops for
instance).
I kind of get the feeling that a hardware based solution
living in a switch or firewall of some sort might not be appropriate because
it would be too expensive for me to justify. It seems that a Linux
solution such as Bandwidth Arbitrator/NetEqualizer would need to be added in
order to properly achieve the level of granularity that I desire without
enormous cost.
I have another qualification for this. I wish to
spend less that $1,000 and have my network be survivable with a failure of
this device. If I was using a switch based solution, I would need two
switches for redundancy (though maybe a backup cheap switch). A
firewall/router would likely be prohibitively expensive if you went for
redundancy. An in-line Linux solution could however be simply bypassed
in the event of an outage, though it would need to be very stable and probably
won't be as stable as a good switch...
Does anyone have any feelings on
this, and maybe some pointers to other in-line software solutions that might
fit the bill?
Thanks,
Matt
Markus Gufler
wrote:
It
might even be nice to do this on a per-IP basis instead of a
per-port basis, though that's not absolutely necessary.
Since this is a Web hosting segment and our bandwidth is
naturally limited going out, and very little intra-DMZ
traffic exists, something that is 10/100 is all that is necessary.
Maybe give a look to a Fortinet 50 or 60-series Firewall. You can manage
guaranted & max traffic and also priorize certain protocols. The price
shouldn't be higher then a manageable switch with traffic shapping
capabilities.
If you want to monitor each switch port with SNMP unfortunately the cheap
Syslink Switch has no SNMP support. At the moment I look for different
solutions. Certain Cisco Catalyst switches looks promising but also the good
old HP ProCurve 2512/2524.
Markus
---
[This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)]
---
This E-mail came from the Declude.JunkMail mailing list. To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type "unsubscribe Declude.JunkMail". The archives can be found
at http://www.mail-archive.com.
--
=====================================================
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=====================================================
--
=====================================================
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=====================================================
|