On Tue, 2005-11-01 at 10:43 -0600, Fleming, John (ZeroChaos) wrote: > >Also I wrote when stall happens I can't telnet to port 80 on web server > >host - which means it is not just program causing stall. > Are you trying this from the same host as the benchmark program? I > wonder if a 2nd host would have the same problem.
I did not have an extra host for test. I've finally figured out it looks like client is running out of local ports as increasing ip_local_port_range allowed to get to the different point. Two things confused me here 1) For some reason it does not fail if firewall is disabled. Probably something is different with connect closure. 2) The error code reported by "ab" is connect timeout. for this kind of error it should be "Can't assign requested address" or something similar. I guess it could be apache runtime abstraction library does not report this error well enough. > > -----Original Message----- > From: Peter Zaitsev [mailto:[EMAIL PROTECTED] > Sent: Monday, October 31, 2005 3:53 PM > To: support@pfsense.com > Subject: Re: [pfSense Support] Network Device pooling > > On Mon, 2005-10-31 at 16:31 -0500, Scott Ullrich wrote: > > Are we absolutely sure this program works as intended? Personally I > > wouldn't trust anything like this but smartbits. > > Well... > > It works if filtering is disabled on pfsese - this is what worries me. > If the program would be broken it should not work in both cases. > > Also I wrote when stall happens I can't telnet to port 80 on web server > host - which means it is not just program causing stall. > > If it is protection on FreeBSD side from too much activity from same IP > (Ie as it limits response to flood ping) this would be good to know. > > I hope this problem is actually something like that - I know there are a > lot of FreeBSD based routers out where - if it would be broken for real > workloads something would scream already. > > One more interesting thing I noticed: > > Percentage of the requests served within a certain time (ms) > 50% 32 > 66% 33 > 75% 33 > 80% 33 > 90% 44 > 95% 295 > 98% 324 > 99% 330 > 100% 21285 (longest request) > > Even if apache benchmark does not timeout it often shows too long > response rate - (21 sec in this case) > > What I've noticed - it can be 3, 9 or 21 secs in this case - This > really look like the times at which SYN packets are resent by TCP/IP > stacks if no reply for previous one arrives. > > > Doing more experiments I also discovered I can increase chance of > passing benchmark (still not to 100%) if i reduce tcp_fin_timeout and > increase ip_local_port_range variables ob my test driver host. > > This still brings the question why with filtering and without behavior > is different but it makes me worry less :) > > > > > > Scott > > > > > > On 10/31/05, Peter Zaitsev <[EMAIL PROTECTED]> wrote: > > > On Mon, 2005-10-31 at 16:25 -0500, Scott Ullrich wrote: > > > > >apr_poll: The timeout specified has expired (70007) > > > > > > > > What is the above from? Your benchmark testing box? > > > > > > Yes. This is output from apache benchmark program. > > > > > > > > > Benchmarking 111.111.111.158 (be patient) > > > Completed 10000 requests > > > Completed 20000 requests > > > Completed 30000 requests > > > apr_poll: The timeout specified has expired (70007) > > > Total of 30517 requests completed > > > > > > > > > > > > > > > > > On 10/31/05, Peter Zaitsev <[EMAIL PROTECTED]> wrote: > > > > > On Mon, 2005-10-31 at 15:48 -0500, Scott Ullrich wrote: > > > > > > Are you viewing the traffic queue status? This would be > normal if you are... > > > > > > > > > > Heh, > > > > > > > > > > yes good quess. These were running in the other window. > > > > > > > > > > > > > > > So here is the output for "stalled" case > > > > > > > > > > # pfctl -ss | wc -l > > > > > 51898 > > > > > > > > > > I have number of states set to 100.000 in advanced page so it is > not > > > > > peak number. > > > > > > > > > > > > > > > Note what really surprises me is the number of request when if > fails: > > > > > > > > > > apr_poll: The timeout specified has expired (70007) > > > > > Total of 28217 requests completed > > > > > > > > > > This number of 28217 is seen so often... Sometimes it is a bit > more ot > > > > > less but it is very frequently withing +/- 100 of it. > > > > > > > > > > I was asked if I can connect to the remote box when this problem > happens > > > > > - yes. I can SSH to the same box which runs Apache, but I > can't > > > > > connect to the port 80 when this problem happens. > > > > > > > > > > So it looks like it does not like to see all these states > corresponding > > > > > to the same target port number. > > > > > > > > > > > > > > > > > > > > > > > > > > > Scott > > > > > > > > > > > > > > > > > > On 10/31/05, Peter Zaitsev <[EMAIL PROTECTED]> wrote: > > > > > > > On Mon, 2005-10-31 at 14:39 -0500, Scott Ullrich wrote: > > > > > > > > On 10/31/05, Fleming, John (ZeroChaos) > <[EMAIL PROTECTED]> wrote: > > > > > > > > > I wonder if part of the problem is PF isn't seeing the > TCP tear down. It > > > > > > > > > seems a little odd that the max gets hit and nothing > else gets through. > > > > > > > > > I guess it could be the benchmark isn't shutting down > the session right > > > > > > > > > after its down transferring data, but I would think it > would kill the > > > > > > > > > benchmark client to have 10K(ish) of open TCP sessions. > > > > > > > > > > > > > > > > One way to deterimine this would be to run pfctl -ss | wc > -l once > > > > > > > > pfSense stops responding? > > > > > > > > > > > > > > Very interesting.... > > > > > > > > > > > > > > I tried running this before the problems but it looks > strange already: > > > > > > > > > > > > > > # pfctl -ss | wc -l > > > > > > > 4893 > > > > > > > Killed > > > > > > > # pfctl -ss | wc -l > > > > > > > 23245 > > > > > > > Killed > > > > > > > > > > > > > > There is nothing in dmesg or system logs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
