I've got a nice solution for this called IPcheck Server Monitor from
Paessler (http://www.paessler.com/products/ipcheck/?link=menu). It is
buggy however from the standpoint of the interface, though they have
been continually improving and fixing it. It has nice notification
features such as dependencies. I monitor from a separate network from
the servers themselves, giving a good impression of what my customers
experience.
I kind of feel tied down to monitoring and then researching, and then
resolving issues. For instance, in the last 24 hours I had the
following happen:
- Customer's mail server IP changed without notifying us, causing
lots of spooled messages.
- One person sent about 150 MB of E-mail (separate large messages)
to a MailPure protected account
and that slowed down our responsiveness on all services (HTTP
and POP3 also affected. No fix can
be applied for this except for limiting as the traffic was
legit, just unusually bursty.
- Customer's remote Web server misbehaving and delivered 10,000
messages to their own account through
our gateway. No immediate issues, but it bears watching for
potential escalation that could threaten our
performance.
I'm spending a lot of time with this stuff on a regular basis and want
to be a bit more proactive so that I don't need to feel tied down.
Almost all issues can be controlled by rate limiting, though some
require extensive granularity to achieve sufficient protection. QOS is
also at issue since I stay away from doing inexpensive services,
concentrating on value-added, and many of my customers do expect more.
Right now I'm fine, but the more I grow, the more often these things
will occur and I think it's time to put something else in place that
can stop issues from potentially affecting every service/customer.
Matt
Darin Cox wrote:
Best solution is monitoring.
Without creating a system of dedicated circuits to each customer you
can't guarantee one customer will not adversely affect another.
Rate-limiting at the switch (or software "switch") will help, but still
means a smaller pipe for everyone else...and doesn't help with multiple
customers misbehaving.
With appropriate SNMP or RMON
monitoring, however, you can be notified as soon as traffic goes beyond
a given threshold and react accordingly.
Darin.
-----
Original Message -----
Sent: Wednesday, February 16, 2005 3:18 PM
Subject: Re: [Declude.JunkMail] OT: Switch to control
bandwidth
I just wanted to follow up on this thread. First, thanks for all of
the suggestions. Here's a summary of what caught my eye.
1) There are some decent choices out there, and seemingly
a 3COM SuperStack 3 3226 comes at a nice price point (around $500) and
allows limiting per port at 1 Mbps increments and also does 7 custom
levels of protocol prioritization. This was suggested to me off-list.
It seems like a good thing for colocation since you don't care for more
granularity among your customers, they can choose to do with their
bandwidth what they wish. I'm not into colocation yet and this
probably falls short of my needs otherwise.
2) I was also intrigued by the NetEqualizer product, which seems to be
a the commercial version of an open source project called Linux
Bandwidth Arbitrator (www.bandwidtharbitrator.com).
This might very well offer functionality beyond all of the switches,
but offers more complication in setup and management unless you go with
the for-profit version. This is of course not a switch, but that's ok
since cheap switches can be placed behind it.
3) Cisco is of course a popular choice, but I'm not a fan of their
ridiculous licensing schemes for the software and high prices. Used,
these things come fairly cheap, but they are the 'Outlook' of routers
and switches, and the most likely to be targeted by exploits. For that
reason, I am probably going to migrate away from anything Cisco once I
outgrow what I already have. I may change my mind however.
4) I don't think I need a firewall, or don't want to deal with the
expense and limitations of it (concurrent sessions, etc.). I have so
few ports open that I'm fine with router level protection and this is
exclusively a DMZ with no client computers behind it.
Despite what these products offer, I still think that the switches
generally come up short of being a perfect solution to my needs (that
of a Web hosting/E-mail provider). I essentially have 5 services that
I need to support across 3 machines; HTTP, FTP, DNS, SMTP, and POP3.
It seems that by just simply bandwidth limiting a port, I won't be able
to slow down but a portion of the problematic bandwidth and there can
be other issues caused by that (such as limiting all HTTP because of
one site that is getting hammered). It would be best to limit HTTP by
IP instead of by port. I haven't tested it out yet, but it may be that
IIS will actually work when limiting in Windows 2003 unlike 2k, and
that may solve my issue on that front at least. FTP may or may not be
covered by the same, I'm not sure yet.
It seems however that some of the worst issues are coming from fairly
unique situations and specific IP addresses. Conditions like E-mail
loops can not only bring down a mail server, but also bring down a
whole network if all of your bandwidth is used. This of course can
also affect POP3 service. If a customer does a mass mailing with huge
images sourced from their site, the bandwidth could also bring us down
without limits. I even had a customer send 144 messages out the other
day with a 2.5 MB attachment, and if you do the math, you will find
that this was 400 MB of bandwidth that IMail naturally attempts to
deliver ASAP. I've also noted that IMail doesn't do well with response
times under heavy bandwidth load even if the CPU is fine while other
services on the same box have far less latency. This affects the
quality of service to my customers, and I like things to be responsive.
So what I am really looking for is some way to protect Web hosting
clients from another Web hosting client's issue, protect POP3 service
from having the bandwidth bogarted by some SMTP loop, or FTP, or HTTP,
etc. Since everyone shares the same MX records, and the same outgoing
SMTP and POP3, it's hard to find decent separation unless I get down to
the IP level and start limiting things based on at least the
destination IP if not the source IP also. To do anything less would
seem to be somewhat futile because I would continue to have sporadic
issues with the most problematic things which can be long-lived to the
point that they are resolved/blocked (DOS or loops for instance).
I kind of get the feeling that a hardware based solution living in a
switch or firewall of some sort might not be appropriate because it
would be too expensive for me to justify. It seems that a Linux
solution such as Bandwidth Arbitrator/NetEqualizer would need to be
added in order to properly achieve the level of granularity that I
desire without enormous cost.
I have another qualification for this. I wish to spend less that
$1,000 and have my network be survivable with a failure of this
device. If I was using a switch based solution, I would need two
switches for redundancy (though maybe a backup cheap switch). A
firewall/router would likely be prohibitively expensive if you went for
redundancy. An in-line Linux solution could however be simply bypassed
in the event of an outage, though it would need to be very stable and
probably won't be as stable as a good switch...
Does anyone have any feelings on this, and maybe some pointers to other
in-line software solutions that might fit the bill?
Thanks,
Matt
Markus Gufler wrote:
It
might even be nice to do this on a per-IP basis instead of a
per-port basis, though that's not absolutely necessary.
Since this is a Web hosting segment and our bandwidth is
naturally limited going out, and very little intra-DMZ
traffic exists, something that is 10/100 is all that is necessary.
Maybe give a look to a Fortinet 50 or 60-series Firewall. You can manage
guaranted & max traffic and also priorize certain protocols. The price
shouldn't be higher then a manageable switch with traffic shapping
capabilities.
If you want to monitor each switch port with SNMP unfortunately the cheap
Syslink Switch has no SNMP support. At the moment I look for different
solutions. Certain Cisco Catalyst switches looks promising but also the good
old HP ProCurve 2512/2524.
Markus
---
[This E-mail was scanned for viruses by Declude Virus (http://www.declude.com)]
---
This E-mail came from the Declude.JunkMail mailing list. To
unsubscribe, just send an E-mail to [EMAIL PROTECTED], and
type "unsubscribe Declude.JunkMail". The archives can be found
at http://www.mail-archive.com.
--
=====================================================
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=====================================================
--
=====================================================
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=====================================================
|
