Re: "No buffer space available" with a lot of queueing
Claudio Jeker wrote: On Thu, Feb 01, 2007 at 09:32:17AM +0100, Federico Giannici wrote: Claudio Jeker wrote: On Wed, Jan 31, 2007 at 06:21:57PM +0100, Adriaan wrote: On 1/31/07, Bret Lambert <[EMAIL PROTECTED]> wrote: [snip] Running and tuning OpenBSD network servers in a production environment: http://www.openbsd.org/papers/tuning-openbsd.ps may have the info you're looking for. IIRC Theo said he would throw Henning in the ocean for that paper ;) Actually Henning said he would throw Henning in the ocean for that paper. But back to the original problem with the no buffer space reports. This is in fact an expected behaviour for non stream oriented protocols like udp or icmp (ping). The problem here is that ENOBUF is returned by the queuing code when the interface queue is full. The error is then reported upwards and in case of datagram sockets is reported back to the userland process. A smart client/server would reduce the packet throughput in such a case. e.g. ping -f is not that smart. In TCP this error is handled by the kernel and so mostly invisible for the userland. Claudio, thanks for the explanation. Considering that the traffic is not GENERATED by the PC itself, and that it is a firewall/queuing/gateway with a lot of queues and traffic, how can I increase the "interface queue"? Are we talking about a NIC (hardware) queue, so I can try to change it with another model (currently a fxp)? Or is it some kind of kernel buffer, so there is some kernel option to increase? You are using many altq queues. Now altq implements a virtual limit on the interface queue via some sort of token bucket. Are you talking about the "tbrsize " parameter of the "altq on" declaration in "pf.conf"? Currently I'm not specifying it. Do you know how can I find the default value (to use as a starting point)? If the queue gets full packet need to be dropped -- like stuffing 10Mbps into a 2Mbps queue will not work without dropping at least some packets. Altq returns ENOBUF when packets need to be dropped because the limit has been reached. So when your pfctl -v -s q output shows many dropped packets you know where the ENOBUFs come from. The "pfctl -v -s q" commands show the "drops" of the single queues and the value of the "qlength" parameter, but I don't think this is what I'm looking for. Sometimes some queue reach it's limit and some packets are dropped, it's normal. But yesterday seemed a different case, because for a few times ALL traffic seemed to be blocked for a certain number of seconds, during a big flood of traffic. Even ICMPs were blocked and they have a certain amount of granted bandwidth. Thanks. -- ___ __ |- [EMAIL PROTECTED] |ederico Giannici http://www.neomedia.it ___
Re: "No buffer space available" with a lot of queueing
On Thu, Feb 01, 2007 at 09:32:17AM +0100, Federico Giannici wrote: > Claudio Jeker wrote: > >On Wed, Jan 31, 2007 at 06:21:57PM +0100, Adriaan wrote: > >>On 1/31/07, Bret Lambert <[EMAIL PROTECTED]> wrote: > >>[snip] > >>>Running and tuning OpenBSD network servers > >>>in a production environment: > >>> > >>>http://www.openbsd.org/papers/tuning-openbsd.ps > >>> > >>>may have the info you're looking for. > >>IIRC Theo said he would throw Henning in the ocean for that paper ;) > >> > > > >Actually Henning said he would throw Henning in the ocean for that paper. > > > >But back to the original problem with the no buffer space reports. > >This is in fact an expected behaviour for non stream oriented protocols > >like udp or icmp (ping). The problem here is that ENOBUF is returned by > >the queuing code when the interface queue is full. The error is then > >reported upwards and in case of datagram sockets is reported back to the > >userland process. A smart client/server would reduce the packet throughput > >in such a case. e.g. ping -f is not that smart. > >In TCP this error is handled by the kernel and so mostly invisible for the > >userland. > > Claudio, thanks for the explanation. > > Considering that the traffic is not GENERATED by the PC itself, and that > it is a firewall/queuing/gateway with a lot of queues and traffic, how > can I increase the "interface queue"? > > Are we talking about a NIC (hardware) queue, so I can try to change it > with another model (currently a fxp)? > Or is it some kind of kernel buffer, so there is some kernel option to > increase? > You are using many altq queues. Now altq implements a virtual limit on the interface queue via some sort of token bucket. If the queue gets full packet need to be dropped -- like stuffing 10Mbps into a 2Mbps queue will not work without dropping at least some packets. Altq returns ENOBUF when packets need to be dropped because the limit has been reached. So when your pfctl -v -s q output shows many dropped packets you know where the ENOBUFs come from. -- :wq Claudio
Re: "No buffer space available" with a lot of queueing
Claudio Jeker wrote: On Wed, Jan 31, 2007 at 06:21:57PM +0100, Adriaan wrote: On 1/31/07, Bret Lambert <[EMAIL PROTECTED]> wrote: [snip] Running and tuning OpenBSD network servers in a production environment: http://www.openbsd.org/papers/tuning-openbsd.ps may have the info you're looking for. IIRC Theo said he would throw Henning in the ocean for that paper ;) Actually Henning said he would throw Henning in the ocean for that paper. But back to the original problem with the no buffer space reports. This is in fact an expected behaviour for non stream oriented protocols like udp or icmp (ping). The problem here is that ENOBUF is returned by the queuing code when the interface queue is full. The error is then reported upwards and in case of datagram sockets is reported back to the userland process. A smart client/server would reduce the packet throughput in such a case. e.g. ping -f is not that smart. In TCP this error is handled by the kernel and so mostly invisible for the userland. Claudio, thanks for the explanation. Considering that the traffic is not GENERATED by the PC itself, and that it is a firewall/queuing/gateway with a lot of queues and traffic, how can I increase the "interface queue"? Are we talking about a NIC (hardware) queue, so I can try to change it with another model (currently a fxp)? Or is it some kind of kernel buffer, so there is some kernel option to increase? Thanks. -- ___ __ |- [EMAIL PROTECTED] |ederico Giannici http://www.neomedia.it ___
Re: "No buffer space available" with a lot of queueing
On Wed, Jan 31, 2007 at 06:21:57PM +0100, Adriaan wrote: > On 1/31/07, Bret Lambert <[EMAIL PROTECTED]> wrote: > [snip] > > > >Running and tuning OpenBSD network servers > >in a production environment: > > > >http://www.openbsd.org/papers/tuning-openbsd.ps > > > >may have the info you're looking for. > > IIRC Theo said he would throw Henning in the ocean for that paper ;) > Actually Henning said he would throw Henning in the ocean for that paper. But back to the original problem with the no buffer space reports. This is in fact an expected behaviour for non stream oriented protocols like udp or icmp (ping). The problem here is that ENOBUF is returned by the queuing code when the interface queue is full. The error is then reported upwards and in case of datagram sockets is reported back to the userland process. A smart client/server would reduce the packet throughput in such a case. e.g. ping -f is not that smart. In TCP this error is handled by the kernel and so mostly invisible for the userland. -- :wq Claudio
Re: "No buffer space available" with a lot of queueing
On 1/31/07, Federico Giannici <[EMAIL PROTECTED]> wrote: Adriaan wrote: > On 1/31/07, Bret Lambert <[EMAIL PROTECTED]> wrote: > [snip] >> >> Running and tuning OpenBSD network servers >> in a production environment: >> >> http://www.openbsd.org/papers/tuning-openbsd.ps >> >> may have the info you're looking for. > > IIRC Theo said he would throw Henning in the ocean for that paper ;) And it is FIVE years old. I think a lot a things changed since it was written... Read it before you assume - a lot of what it covers regards under the hood internals that have remained largely unchanged since the early days. I at least found some of the knowledge and explanations of the tools and output useful. DS
Re: "No buffer space available" with a lot of queueing
Adriaan wrote: On 1/31/07, Bret Lambert <[EMAIL PROTECTED]> wrote: [snip] Running and tuning OpenBSD network servers in a production environment: http://www.openbsd.org/papers/tuning-openbsd.ps may have the info you're looking for. IIRC Theo said he would throw Henning in the ocean for that paper ;) And it is FIVE years old. I think a lot a things changed since it was written... Bye. -- ___ __ |- [EMAIL PROTECTED] |ederico Giannici http://www.neomedia.it ___
Re: "No buffer space available" with a lot of queueing
On 1/31/07, Bret Lambert <[EMAIL PROTECTED]> wrote: [snip] Running and tuning OpenBSD network servers in a production environment: http://www.openbsd.org/papers/tuning-openbsd.ps may have the info you're looking for. IIRC Theo said he would throw Henning in the ocean for that paper ;) =Adriaan=
Re: "No buffer space available" with a lot of queueing
On Wed, 2007-01-31 at 15:46 +0100, Federico Giannici wrote: > We have a PC with OpenBSD 4.0-stable i386 that we use as a > firewall/gateway. It has a lot of HFSC queues. > > Today we had a flood if traffic and the outgoing interface started to > "loss" packets. > > If I tried to ping through the outgoing interface the "No buffer space > available" error occurred. > > When the traffic decreased the error disappeared. > > What "buffer space" it is talking about? > Is there some parameter (kernel, sysctl, ALTQ, etc...) I can increase to > avoid this problem? Running and tuning OpenBSD network servers in a production environment: http://www.openbsd.org/papers/tuning-openbsd.ps may have the info you're looking for. - Bert > > > Thanks.
"No buffer space available" with a lot of queueing
We have a PC with OpenBSD 4.0-stable i386 that we use as a firewall/gateway. It has a lot of HFSC queues. Today we had a flood if traffic and the outgoing interface started to "loss" packets. If I tried to ping through the outgoing interface the "No buffer space available" error occurred. When the traffic decreased the error disappeared. What "buffer space" it is talking about? Is there some parameter (kernel, sysctl, ALTQ, etc...) I can increase to avoid this problem? Thanks. -- ___ __ |- [EMAIL PROTECTED] |ederico Giannici http://www.neomedia.it ___
