Re: High volume proxy server configuration.

[Mike Silbersack]

On Tue, 21 May 2002, Scott Hess wrote:

> Setup: 2x SMP server running FreeBSD4.5.  Apache 1.3.x.  2Gig of memory.
>
> When stress-testing, I am able to cause the kernel messages:
>
> m_clalloc failed, consider increase NMBCLUSTERS value
> fxp0: cluster allocation failed, packet dropped!
>
> Here's my theory: When the amount of space used for user processes and
> kernel usage fills all of memory, and a burst of packets are received from
> the backend servers, the kernel isn't able to allocate pages and drops the
> packets, with the message.  The sender resends, and things cascade away.
> Since this is a kernel vm issue, the console also locks up.  [Well, it's
> the best I have.]
>
> I've tried upping vm.v_free_min, vm.v_free_target, and vm.v_free_reserved.
> It doesn't appear to have any impact.

I think that your theory is probably close to what is happening.
Unfortunately, there's no easy way to address this yet.  Due to the
extensive use of zone allocators in 4.x, it's hard to size all allocations
correctly.  For this reason, there may be other subtle issues with 2 gig+
machines.

For now, I think your best option may be to run your mbuf allocation
program so that you have a certain amount of mbufs allocated and ready for
your application.  Along those lines, you might consider writing a kernel
patch which performs this function based on a configurable value; I would
be happy to commit such a feature if it was implemented well; other people
with busy servers might find it useful.

I've been pondering various methods to handle out of mbuf cluster
situations better, but handling your case seems especially difficult.
I'll have to think more.

Mike "Silby" Silbersack


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-net" in the body of the message

Re: Interface statistic

[Larry Sica]

On Tue, 21 May 2002 14:58:22 +0300, Ivailo Tanusheff wrote:
>Hi,
>
>Can you tell me a way to collect per network interface statistic on
>my FreeBSD box?
>At this moment I'm using IPFilter accounting to collect needed
>information, but I think this way I'm collecting only information
>related to tcp, udp and icmp traffic. My purpose is to visualize
>this data in MRTG.
>
 
You could capture traffic via tcpdump.  I think MRTG can read tcpdump output.  I am going by memory right now though so some research would be helpful.  
 



-- Larry
 
 
Larry Sica
 [EMAIL PROTECTED]
 

Question about Dummynet and Diffserv

[Craig Rodrigues]

Hi,

I am trying to set up a network testbed where I can offer different
levels of service to different streams of traffic marked with
different Diffserv codepoints.

I have two FreeBSD routers (4.6 RC1) in my testbed, compiled with
Dummynet, ALTQ, and IPFIREWALL.

Dummynet works great for changing characteristics such as delay,
packet loss rate, and bandwidth.  I have used the ipfw command to set
up rules like: "Set the delay to 800ms for packets with source
address foo and destination address bar."

However, ipfw support setting up filter rules based on the IP TOS
field (Diffserv byte).

The ipfw command seems to only support the capability for creating
firewall rules based on the following IP options: ssrr (strict source
route), lsrr (loose source route), rr (record packet route) and ts
(timestamp).  It doesn't let you create a filter rule based on
the TOS field.

Now, on the other hand, with ALTQ, it is possible to set up
filter rules which can deal with IP TOS.  However, ALTQ seems
to be separate from ipfw/Dummynet, and doesn't have the nice
features of being able to specify delay and packet loss rates.

Does anybody know how I can resolve this?  I basically
want to use ipfw/Dummynet, but set the filter rule based
on the TOS field.

Thanks.
-- 
Craig RodriguesDistributed Systems and Logistics, Office 6/304
[EMAIL PROTECTED]   BBN Technologies, a Verizon company
(617) 873-4725 Cambridge, MA


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-net" in the body of the message

Re: High volume proxy server configuration.

[Wilbert deGraaf]

Hello Scott,

> Here's my theory: When the amount of space used for user processes and
> kernel usage fills all of memory, and a burst of packets are received from
> the backend servers, the kernel isn't able to allocate pages and drops the
> packets, with the message.  The sender resends, and things cascade away.
> Since this is a kernel vm issue, the console also locks up.  [Well, it's
> the best I have.]

It sounds like the proxy doesn't implement flow control. What I mean is that
if a proxy reads all it can from a server, and writes it to the client as
fast it can, memory usage can easily explode if the connection to the client
is slower that the server connection.

Did you modify that part of the Apache proxy ?

If so, you probably want to stop reading from the server until you have been
able to send data to the client. That way, TCP takes care of it.

I expected you to run into user level memory allocation problems, but since
you ran out of NMBCLUSTERS, it could be that the proxy tries to write to the
client at the speed it reads itself from the servers. In this case, I
understand the problem that you describe.

Hope this helps, Wilbert


- Original Message -
From: "Scott Hess" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Tuesday, May 21, 2002 3:41 PM
Subject: High volume proxy server configuration.


> Background: I'm working on an intelligent Apache-based proxy server for
> backend servers running a custom Apache module.  The server does some
> inspection of the incoming request to determine how to direct it, and
> passes the reseponse directly back to the client.  Thus, I'd like to be
> able to set the TCP buffers fairly large, with the server merely acting as
> a conduit to transfer data between the backend server and the client.
> Upstream data is relatively small (a handful of kilobytes), downstream can
> be large (100k-2Meg).
>
> Setup: 2x SMP server running FreeBSD4.5.  Apache 1.3.x.  2Gig of memory.
>
> When stress-testing, I am able to cause the kernel messages:
>
> m_clalloc failed, consider increase NMBCLUSTERS value
> fxp0: cluster allocation failed, packet dropped!
>
> The system hangs for a perhaps five minutes, and then comes back and is
> able to continue operating.  pings work, but the console isn't responsive
> (I mean "no response until things clear a couple minutes later).  I've
> spent some time trying to tweak things, but I haven't been able to prevent
> the problem.  My /boot/loader.conf includes:
>
> kern.maxusers="512"
> kern.ipc.nmbclusters="65536"
>
> The problem can happen at various points.  I've seen it happen with the
> mbuf cluster count <1k.  Usually, the current/peak/max of netstat -m will
> have peak nowhere near 65536.  This usually happens when I have on the
> order of 2000 processes/connections running - the machine is 80% idle at
> this point, though.
>
> I wrote a program to specifically use up mbuf clusters (many servers write
> lots of data, many clients sleep), and it didn't cause any problems until
> hitting the maximum.  Even then, the machine wasn't locked up at the
> console.  So I think the message is a symptom of something else.
>
> Here's my theory: When the amount of space used for user processes and
> kernel usage fills all of memory, and a burst of packets are received from
> the backend servers, the kernel isn't able to allocate pages and drops the
> packets, with the message.  The sender resends, and things cascade away.
> Since this is a kernel vm issue, the console also locks up.  [Well, it's
> the best I have.]
>
> I've tried upping vm.v_free_min, vm.v_free_target, and vm.v_free_reserved.
> It doesn't appear to have any impact.
>
> I was also getting the message:
>
>pmap_collect: collecting pv entries -- suggest increasing
PMAP_SHPGPERPROC
>
> From what I can tell, this sounds like a direct results of running so many
> processes forked from the same parent.  Each process is small (SIZE ~4M).
> I increased PMAP_SHPGPERPROC to 400, now I don't seem to get this message.
> I've watched 'sysctl vm.zone', and the PV ENTRY line seems more
> reasonable, now.
>
> The last line of vmstat output when this happens (broadly similar to
> previous lines):
>
>  procs  memory  pagedisks faults  cpu
>  r b w avmfre  flt  re  pi  po  fr  sr da0 da1   in   sy  cs us sy
id
>  8 2 0 2141424  41184 8255  46   0   0 3982   0   0   0 3477 5416 1264 14
38 48
>
> This is consistent with top:
>
> last pid: 79636;  load averages:  3.51,  1.59,  0.83up 0+22:23:16
16:37:14
> 2268 processes:9 running, 2259 sleeping
> CPU states: 19.6% user,  0.0% nice, 19.6% system,  5.4% interrupt, 55.4%
idle
> Mem: 578M Active, 25M Inact, 361M Wired, 3528K Cache, 112M Buf, 37M Free
> Swap: 2048M Total, 35M Used, 2012M Free, 1% Inuse
>
> [Hmm, one note - I'm replicating this on a 1Gig machine, but we've also
> seen it in an extreme case on the 2Gig machine which is in production.]
>
> Hmm.  vmstat just came b

High volume proxy server configuration.

[Scott Hess]

Background: I'm working on an intelligent Apache-based proxy server for
backend servers running a custom Apache module.  The server does some
inspection of the incoming request to determine how to direct it, and
passes the reseponse directly back to the client.  Thus, I'd like to be
able to set the TCP buffers fairly large, with the server merely acting as
a conduit to transfer data between the backend server and the client.  
Upstream data is relatively small (a handful of kilobytes), downstream can
be large (100k-2Meg).

Setup: 2x SMP server running FreeBSD4.5.  Apache 1.3.x.  2Gig of memory.

When stress-testing, I am able to cause the kernel messages:

m_clalloc failed, consider increase NMBCLUSTERS value
fxp0: cluster allocation failed, packet dropped!

The system hangs for a perhaps five minutes, and then comes back and is
able to continue operating.  pings work, but the console isn't responsive
(I mean "no response until things clear a couple minutes later).  I've
spent some time trying to tweak things, but I haven't been able to prevent
the problem.  My /boot/loader.conf includes:

kern.maxusers="512"
kern.ipc.nmbclusters="65536"

The problem can happen at various points.  I've seen it happen with the
mbuf cluster count <1k.  Usually, the current/peak/max of netstat -m will
have peak nowhere near 65536.  This usually happens when I have on the
order of 2000 processes/connections running - the machine is 80% idle at
this point, though.

I wrote a program to specifically use up mbuf clusters (many servers write
lots of data, many clients sleep), and it didn't cause any problems until
hitting the maximum.  Even then, the machine wasn't locked up at the
console.  So I think the message is a symptom of something else.

Here's my theory: When the amount of space used for user processes and
kernel usage fills all of memory, and a burst of packets are received from
the backend servers, the kernel isn't able to allocate pages and drops the
packets, with the message.  The sender resends, and things cascade away.  
Since this is a kernel vm issue, the console also locks up.  [Well, it's
the best I have.]

I've tried upping vm.v_free_min, vm.v_free_target, and vm.v_free_reserved.  
It doesn't appear to have any impact.

I was also getting the message:

   pmap_collect: collecting pv entries -- suggest increasing PMAP_SHPGPERPROC

>From what I can tell, this sounds like a direct results of running so many
processes forked from the same parent.  Each process is small (SIZE ~4M).  
I increased PMAP_SHPGPERPROC to 400, now I don't seem to get this message.  
I've watched 'sysctl vm.zone', and the PV ENTRY line seems more
reasonable, now.

The last line of vmstat output when this happens (broadly similar to
previous lines):

 procs  memory  pagedisks faults  cpu
 r b w avmfre  flt  re  pi  po  fr  sr da0 da1   in   sy  cs us sy id
 8 2 0 2141424  41184 8255  46   0   0 3982   0   0   0 3477 5416 1264 14 38 48

This is consistent with top:

last pid: 79636;  load averages:  3.51,  1.59,  0.83up 0+22:23:16  16:37:14
2268 processes:9 running, 2259 sleeping
CPU states: 19.6% user,  0.0% nice, 19.6% system,  5.4% interrupt, 55.4% idle
Mem: 578M Active, 25M Inact, 361M Wired, 3528K Cache, 112M Buf, 37M Free
Swap: 2048M Total, 35M Used, 2012M Free, 1% Inuse

[Hmm, one note - I'm replicating this on a 1Gig machine, but we've also
seen it in an extreme case on the 2Gig machine which is in production.]

Hmm.  vmstat just came back, the first two lines:

 procs  memory  pagedisks faults  cpu
 r b w avmfre  flt  re  pi  po  fr  sr da0 da1   in   sy  cs us sy id
2268 2 0 2352192  62236 7308  59   0  32 3306 161397  28   0 2454 5111 1153 12 40 49
 0 2 0  266364  46240 292845 1517   9 608 38036 6843317   1   0 334730 253302 17192  0 
100  0

top shows increased space used in swap (42M, now), so it looks like we got
a bunch of swapping going on.  [Just to be clear - when the event happens,
things don't simple get a bad response time.  There's _no_ response, until
the problem clears and everything comes back.  Then it's all shiny-happy,
again.]

/etc/sysctl.conf has:

kern.ipc.somaxconn=4192
net.inet.ip.portrange.last=4
kern.ipc.maxsockbuf=2097152

We are definitely not using the full maxsockbuf range!  Actually, we've
left things at the default (sendspace=32k, recvspace=64k).

AFAICT, everything else is at default settings.

Thanks for any help,
scott


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-net" in the body of the message

hfa0 PCA200E more informations [Help]

[Christophe Prevotaux]

Hi,


Fore PCA-200E AAL 5 Statistics
CRC/Len  CRC   Proto  PDU
  Cells In   Cells Out  Errs   DropsPDUs In   PDUs Out   Errs  Errs   Drops
 147895872   220929747   4757  9   20005779   23507681  9  0  9


As can be seen I get 4757 CRC Erros, 9 PDU Dropped

Fore PCA-200E Device Statistics
Type 1  Type 1  Type 2  Type 2
Small Buff  Large Buff  Small Buff  Large Buff  Receive Receive
Alloc Fail  Alloc Fail  Alloc Fail  Alloc Fail  Queue Full  Carrier
  3222   0   0   0   0  On


Also 3222 Buffers Failed


Fore PCA-200E Device Driver Statistics
  No  Xmit   Max   Seg  NoNo  NoIQNo   CmdNo
 VCC Queue   Seg   Not   Seg   DMA   VCCNo  Mbuf  Full   DMA Queue   DMA
 Out  Full  Size Align   Pad   OutIn  BuffInIn   Sup  Full   Cmd
   0  1551827 0 0 0 0 0 0 0 0 0 0 0


And as previously stated (in the previous [EMAIL PROTECTED] posting) 
1551827 Xmit Queue Full errors 


InputInput  Input  Output   Output Output
Interface  VPI   VCI PDUsBytes   ErrsPDUsBytes   Errs
hfa0 033  20005779 2048761777  9 23507681 1247248987  14238

Also Output Errs seems to confirm it since I get 14238 Output Errs.


Can anyone enlighten me as to how I can fix this or what is needed 
to get rid of these errors. ? 


--
===
Christophe PrevotauxEmail: [EMAIL PROTECTED]
HEXANET SARLURL: http://www.hexanet.fr/
Z.A.C Les CharmillesTel: +33 (0)3 26 79 30 05 
3 Allée Thierry Sabine  Direct: +33 (0)3 26 79 08 02 
BP202   Fax: +33 (0)3 26 79 30 06
51686 Reims Cedex 2
FRANCE   HEXANET Network Operation Center 
===

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-net" in the body of the message

Re: Multicast problem with "wi" driver in promiscuous mode - any resolution?

[John Hay]

> I don't think anybody has applied fixes to the wi driver in that time
> frame for this purpose.  Have fun :-(.

The problem is that the wavelan/orinoco cards at least, only have space
for 16 multicast addresses and don't have an "all multicast" bit, so if
you go over 16 addresses or want to catch all multicast packets, you
have to enable promicious mode. I have a work in progress patch from
a while back that did work if I remember correctly. :-)

The reason I didn't go much further with it was because I found that
the Orinoco cards stayed at 2Mbit when promiscious mode was enabled,
so at the end I just tunneled the multicast stuff over the wireless
net.

John
-- 
John Hay -- [EMAIL PROTECTED] / [EMAIL PROTECTED]


Index: if_wi.c
===
RCS file: /home/ncvs/src/sys/i386/isa/Attic/if_wi.c,v
retrieving revision 1.18.2.14
diff -u -r1.18.2.14 if_wi.c
--- if_wi.c 31 Jan 2002 16:56:59 -  1.18.2.14
+++ if_wi.c 19 Feb 2002 09:04:22 -
@@ -1231,13 +1231,26 @@
 
bzero((char *)&mcast, sizeof(mcast));
 
-   mcast.wi_type = WI_RID_MCAST;
-   mcast.wi_len = (3 * 16) + 1;
-
-   if (ifp->if_flags & IFF_ALLMULTI || ifp->if_flags & IFF_PROMISC) {
+   if (ifp->if_flags & IFF_ALLMULTI &&
+   !(sc->wi_if_flags & IFF_ALLMULTI)) {
+#if 0
+   mcast.wi_type = WI_RID_MCAST;
+   mcast.wi_len = (3 * 16) + 1;
wi_write_record(sc, (struct wi_ltv_gen *)&mcast);
+#endif
+
+   sc->wi_if_flags |= IFF_ALLMULTI;
+   ifpromisc(ifp, 1);
return;
}
+   if (!(ifp->if_flags & IFF_ALLMULTI) &&
+   sc->wi_if_flags & IFF_ALLMULTI) {
+   printf("wi%d: switch of all multicast\n", ifp->if_unit);
+   sc->wi_if_flags &= ~IFF_ALLMULTI;
+   ifpromisc(ifp, 0);
+   if (ifp->if_flags & IFF_PROMISC)
+   return;
+   }
 
LIST_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) {
if (ifma->ifma_addr->sa_family != AF_LINK)
@@ -1247,11 +1260,13 @@
(char *)&mcast.wi_mcast[i], ETHER_ADDR_LEN);
i++;
} else {
-   bzero((char *)&mcast, sizeof(mcast));
-   break;
+   printf("wi%d: Oops too many multicast addresses\n",
+   ifp->if_unit);
+   return;
}
}
 
+   mcast.wi_type = WI_RID_MCAST;
mcast.wi_len = (i * 3) + 1;
wi_write_record(sc, (struct wi_ltv_gen *)&mcast);
 

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-net" in the body of the message

Re: Multicast problem with "wi" driver in promiscuous mode - anyresolution?

[M. Warner Losh]

I don't think anybody has applied fixes to the wi driver in that time
frame for this purpose.  Have fun :-(.

Warner

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-net" in the body of the message

RE: "dynamic" ipfw

[Frans ter Borg]

Scott must have meant to type http://www.bsdshell.net which does list the
EtherFirewall project.

Best regards,

Frans

On Tue, 21 May 2002, Mire, John wrote:

> nice project page, does it do anything?
>
> -Original Message-
> From: Scott Ullrich [mailto:[EMAIL PROTECTED]]
> Sent: Monday, May 20, 2002 5:23 PM
> To: 'John Angelmo'; [EMAIL PROTECTED]
> Subject: RE: "dynamic" ipfw
>
>
>
> Check out http://www.bsdshell.com  's
> EtherFirewall project.  It will allow you to maintain Mac addresses with
> your IPFW rules.


To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-net" in the body of the message

RE: "dynamic" ipfw

[Mire, John]

Title: RE: "dynamic" ipfw



a 
search on google did not turn up anything for me and the webpage is just a page 
with seiki on it and no other links: