I can hardly say whether it's your problem but in last months I came across a couple of servers where we also experienced strange problems. Some blocked and logged packets looked like part of the communication and should have been permitted because of held state. The problem was the limit on the maximum number of states IPF can hold at any given moment. If I understand the IPF sources correctly there is a fixed limit on number of several of the structures it allocates and when it is reached bad things happen.
The sign of you reaching the limit on states is non-zero counter maximum in the output of ipfstat -s. You can also watch the row active and if you see it growing and than suddenly dropping it will probably mean you reached the limit. If the limit is reached I think that all current state entries are dropped which can cause some returning packets to be dropped (which were being permitted by the deleted state entry). The limit is defined in ip_state.h in IPSTATE_MAX (if not overriden) and is 4013 (at least on FreeBSD). I successfully run ipfilter after I changed this to 41999 and IPSTATE_SIZE to 59999 (and recompiled the kernel). If I understand the problem correctly the limits are very conservative for todays machines and the default should be enlarged. It will also be nice if one didn't have to recompile and reboot to change them. Michal Jan Rockstedt wrote: > From: <[EMAIL PROTECTED]> > To: <[email protected]> > Sent: Friday, April 21, 2006 7:06 PM > Subject: IPF keep state, fails randomly? > > > > Our mailserver uses IPF. We have stated keep state rules as stated below. > > > > We have around 400-500 mails passed by our mailserver daily. > > > > Everything works fine beside that we get a lot of "lost connection with > > xxxxxx[x.x.x.x] while sending end of data" and "lost connection with > > xxxxxx[x.x.x.x] while sending message body" from certain ISP's/mailservers > > (I know that this is naturall sometimes, but here we get them like 10% of > > outgoing mails). > > > > > > "lost connection with xxxxxx[x.x.x.x] while sending end of data" occurs > > with > > small mails (below 1Mbyte and whom have no attachments), and "lost > > connection with xxxxxx[x.x.x.x] while sending message body" occurs when > > mails contain big attachments as in bigger then 1Mbyte. > > > > Really small mails goes thru without a problem (50Kb or smaller). > > > > > > When we turn off IPF, these problems disappear and everything works > > smoothly. > > > > The first thing I thought of was it maybe had to do with MTU Discovery, > > and > > I tried to ping -D -s 1472, and this goes thrue without a problem to > > receiving mailservers who gives those "lost connection" messages. I have > > also mailed with MTU 1492/1300/1200, but the result is the same. Same > > problems, but ones I turn off IPF everything looks fine. > > > > So the problem seems to definetly be in IPFilter. Its not in MTU och ICMP > > type 3, 4 packages as I can see. > > > > I have contacted admins of the receiving end and none of them recognizes > > the > > problem, and their mailservers get no errors when we get "lost connection > > with xxxxxx[x.x.x.x] while sending end of data" and they get "connection > > timed out" when we get lost connection with xxxxxx[x.x.x.x] while sending > > message body". > > > > I can not see anything common between these receiving servers (they use > > all > > from postfix/qmail/exim and MS-based smtpds) Beside that they all use > > firewalls/filters themselves and don't have these kind of problems with > > other servers beside us. And since these problems disappears when we > > disable > > IPfilter, I think that it should be the cause. > > > > But where does it fail then? Could it be a bug? Or have we configured it > > badly? Or something else? Does keep state fails randomly? Does the > > receiving > > end manipulate packets strangely so keep state fails? > > > > Sniffing the traffic with tcpdump I could not find anything of interest. > > > > I'm thankful for all tips and contributions that I can get. > > > > Our system contains of: > > FreeBSD 6.0 RELEASE-p4 (i386) [Pentum 4 2.8Ghz 1Gb RAM] IP Filter 4.1.8 > > Postfix postfix-2.2.9,1 MTU is set to 1500, MTU Discovery is enabled. > > My ruleset: > > > > pass out quick on lo0 > > pass in quick on lo0 > > pass in quick on xl0 proto tcp from any to any port = 21 flags S keep > > state > > pass in quick on xl0 proto tcp from any to any port = 22 flags S keep > > state > > pass in quick on xl0 proto tcp from any to any port = 110 flags S keep > > state > > pass in quick on xl0 proto tcp from any to any port = 143 flags S keep > > state > > pass in quick on xl0 proto tcp from any to any port = 993 flags S keep > > state > > pass in quick on xl0 proto tcp from any to any port = 995 flags S keep > > state > > pass in quick on xl0 proto tcp from any to any port = smtp keep state pass > > in quick on xl0 proto tcp from any to any port = 587 keep state pass in > > quick proto icmp from any to any icmp-type 3 keep state pass in quick > > proto > > icmp from any to any icmp-type 11 keep state pass in quick on xl0 proto > > tcp > > from any to any port xxxxx >< xxxxxx flags S keep state pass out quick on > > xl0 from any to any keep state block in on xl0 all block out on xl0 all > > > > Same problems also occurs with: > > pass out quick on lo0 > > pass in quick on lo0 > > pass in quick on xl0 proto tcp from any to any flags S keep state pass out > > quick on xl0 proto tcp from any to any flags S keep state pass in quick on > > xl0 proto udp from any to any keep state pass out quick on xl0 proto udp > > from any to any keep state pass in quick on xl0 proto icmp from any to any > > keep state pass out quick on xl0 proto icmp from any to any keep state > > block > > in on xl0 all block out on xl0 all > > > > Everything works with: > > pass in on any > > pass out on any > > And it also works with: > > pass in on any > > pass out on any > > block out quick on xl0 proto icmp from any to any block in quick on xl0 > > proto icmp from any to any > > > > With Best Regards Adde > > Perhapse this will help you? > > ------ > This could be a problem due to timeouts with the ident protocol, also > known as auth, which uses port 113. Most mailservers will try and do > an ident check on you when you connect to them. > ---- > > //Jan > >
