On Thu, Aug 06, 2009 at 08:49:42PM -0700, Darren Reed wrote:
Hi Darren,
sorry for the late answer, but vacation ... So going through the mail
stack took a while as well.
> Jens Elkner wrote:
...
> >>>So the sun case engineer explained, if ipf can not insert an entry into
> >>>the state table, it just _continues_ evaluating the rules that follow.
...
> >>Well, what would you have it do?
> >
> >As said, at least notify the user about the problem. Since I'm not
...
> >do here. My guts say, drop and stop processing other rules.
> >
> >Another way could be, to lower the TTL of all entries in steps of N hours,
> >cleanup, and check, whether now the insert succeeds. This implies of course
> >a new threshold to be set by the OS/user and a user notification, that this
> >measure has been taken (documentation about what that means would be
> >probably a nice thing, too) ...
> It does trigger this to happen but in a slightly more detailed way.
> There's a high and low watermark for the table and it starts with
> the oldest and works its way forward to try and empty things out.
Not sure, whether you mean the same thing: I would do something like this
assuming default TTL is 120h and TTL.low is the lowest TTL allowed:
if table.load > 98% && TTL > TTL.low
TTL -= 1h
foreach con : table
if con.TTL <= 1h
con.kill && drop con
else
con.TTL -= 1h
else if TTL < 120h && table.load < 80%
TTL += 1h
I guess, this would be sufficient. Usually we've ~ 100..200 IMAP
connections, and ~ 1-2 SMTP conns/s - when bots aka spammers are active,
~ 5-10 (very seldom 15) conns/s for about 2-3 hours. Since most of the
delivery attemps get NACKd pretty fast, I would say, it is not that much.
However, not cleanly shutdowned connections probably pollute the state
table (just a guess) ...
> Buf if there are more new connection attempts than there are old
> things that can be removed, packets will get dropped. I'm not
> sure if the improvements in the cleaning were in S10U4, but they
> are in Nevada and the latest S10 updates.
I would guess not, since before U4 we never had such problems.
> >>It is not appropriate to have ipfilter automatically
> >>grow the tables as you suggest through "automatic
> >>tuning" because then your system becomes vulnerable
> >>to a denial of service attack from remote attackers.
> >
> >Isn't it already vunerable to DoS, when the state table is too small?
>
> Yes & no. But a different denial of service attack.
OK, but growing to a certain degree only should be ok. E.g. if fr_statemax
reached, start cleanup like described above and add some buckets temp.
Max. avail mem. might be the base for calculation (like ARC for ZFS).
> If the table just kept growing, it could consume all kernel memory
> and make the machine unresponse/crash. In this situtation, if the table
> fills up then certain packets will get dropped. If you're running a web
> server or some other kind of service where the security of the TCP
> connection itself isn't as important, maybe stateless filtering is a better
> choice?
Hmmm, problem machines are e-mail, subversion and sometimes web servers ....
> Or if there are multiple keep-state rules, you can apply individual
> limits to each. (see below)
Ohhh - an undocumented feature ...
> >Actually I've checked all of our production servers (most of them
> >already have fr_statemax=40129) and all show paket lost > 100,
> >usually > 5000 even on very very low traffic machines and who wonders, of
> >course on svn machines as well. So this implies, that's something wrong
> >with ipf or at least with its default settings...
>
> Over what time period?
Depends, not yet monitored (just writing a tool for it), but I would say
wrt. manual checks with 1-2 days.
...
> The primary purpose of the firewall is to provide security and part of that
> security comes through being able to audit what's going on. In that case,
> letting through packets that matched a "pass" rule but for which the state
> could not be created is arguably incorrect.
Exactly, and that's why ipfilter should not treat the rule in this case
like "not-existing", but stop, block and log.
Appropos log: IMHO it is pretty bad, that log entries are often useless,
since one can't determine, whether a packet 'p'assed or was 'b'locked (for
the normal human being, that's all, what finally counts). Not sure,
but the fact, that the log entry appears somewhere is a clear indication,
that it has been logged - so IMHO 'L' seems to be redundant. Wrt. 'n'
I think, if no rule was matched, one will see @-1:-1 , so redundant as well.
And 's' (I guess) indicates blocked, but no where explained/documented, so
it is useless as well ...
> >Actually can only rely on the things, which are
> >documented, and you probably know, there is a lot room for improvements:
> >E.g. It is documented, how one can obtain certain ipf statistics, but
> >nowhere, what these stats actually telling you/how to interprete those
> >stats.
>
> I'm aware of that and as we like to say, this is an open source
> project, so if something really bothers you please feel free to
> submit additions or corrections.
Well, I would do that, but I think, guessing games are not helpful and
waste even more valuable time ... And even so I would like to have a
closer look at the code, I'll probably get not the chance to do that
within the next 12 month :(
> We only have so much bandwidth.
Yepp, like every human ;-)
...
> >>syslog message from where?
...
> >What I know is, that e.g. on linux there is a klogd ...
...
> Yes, but that is Linux.
> There are different rules for OpenSolaris developers.
Yes, of course. But wondering, where I can find them ;-)
> >>Generating messages from within kernel modules
> >>is generally frowned upon.
But having a system, which goes mad and not knowing why, is even worse.
> >>Whilst we can send messages to/via it, we are not allowed to
> >>rely on them as being the only communication channel with the
> >>systems administrator.
> >
> >OK. But did I say that?
>
> No, but there are different rules for you than there are for me/us
> as a developer(s) of [Open]Solaris.
So you are actually saying, theory is more important than practical use
or a little bit overdone: if the theory is ok, why to care about endusers/
usability?
> >So all of our production machines (S10u7) even the low traffic ones
> >and fr_statemax=40129 have a problem and 'keep state' needs to be
> >considered harmful :(((
>
> Just to make sure, you have "flags S" with all of your TCP "keep state"
> rules?
Yes.
> The limit on how many state-table entries can be controlled per-rule:
>
> pass in on bge0 proto tcp from any to any port = 22 flags S keep state
> (limit 20)
>
> ...will always allow upto 20 and no more than 20 entries to be created
> because of that rule matching. Now if there is another rule, like this:
>
> pass in on bge0 proto tcp all flags S keep state
>
> ...then the 21st ssh connection will "fail" matching the first rule
> (limit is reached)
> but will successfully match the second and potentially then cause a new
> state
> table entry to be created.
>
> Similarly, the reverse is true also: if the rule without the limit fails
> to create
> a state table entry because the table is too full then the second, with the
> limit of 20 allows for a state table entry to be created so long as
> there are
> no more than 19 such entries already in the table.
Would be nice to doc this in ipf(4).
So if I understand that correctly, limit is like a reservation of table
entries for the given rule. The latter case would help at least to get not
locked out of the system, but I think does not really help.
> >>>BTW: Why shows 'ipfstat -t' so many entries with negative ttls. It
> >>> appears, that if the min? value of -59:-59 is reached, ttl gets
> >>> reset to 0:00 and restarts decrementing it ... - strange
> >>>
> >>That's a known bug... fixed in the current opensolaris
> >>source code tree and will be fixed in the next release.
> >>A fix is also being considered for Solaris 10.
>
> They get removed from the state table but not the list of state entries.
> Doing "ipf -Fs -FS" will clean it all out :-D
Still a little bit confused: Is this just another aka stat table?
If not, isn't this perhaps the root cause of fr_statemax reached?
Regards and thanx for your time,
jel.
--
Otto-von-Guericke University http://www.cs.uni-magdeburg.de/
Department of Computer Science Geb. 29 R 027, Universitaetsplatz 2
39106 Magdeburg, Germany Tel: +49 391 67 12768
_______________________________________________
networking-discuss mailing list
[email protected]
