Can we please let this thread die already? I'm tired about hearing of benchmarking the *WRONG* way.
Scott On 11/1/05, Peter Zaitsev <[EMAIL PROTECTED]> wrote: > On Tue, 2005-11-01 at 10:43 -0600, Fleming, John (ZeroChaos) wrote: > > >Also I wrote when stall happens I can't telnet to port 80 on web server > > >host - which means it is not just program causing stall. > > Are you trying this from the same host as the benchmark program? I > > wonder if a 2nd host would have the same problem. > > I did not have an extra host for test. > > I've finally figured out it looks like client is running out of local > ports as increasing ip_local_port_range allowed to get to the > different point. > > Two things confused me here > > 1) For some reason it does not fail if firewall is disabled. Probably > something is different with connect closure. > > 2) The error code reported by "ab" is connect timeout. for this kind of > error it should be "Can't assign requested address" or something > similar. I guess it could be apache runtime abstraction library does > not report this error well enough. > > > > > > > > -----Original Message----- > > From: Peter Zaitsev [mailto:[EMAIL PROTECTED] > > Sent: Monday, October 31, 2005 3:53 PM > > To: support@pfsense.com > > Subject: Re: [pfSense Support] Network Device pooling > > > > On Mon, 2005-10-31 at 16:31 -0500, Scott Ullrich wrote: > > > Are we absolutely sure this program works as intended? Personally I > > > wouldn't trust anything like this but smartbits. > > > > Well... > > > > It works if filtering is disabled on pfsese - this is what worries me. > > If the program would be broken it should not work in both cases. > > > > Also I wrote when stall happens I can't telnet to port 80 on web server > > host - which means it is not just program causing stall. > > > > If it is protection on FreeBSD side from too much activity from same IP > > (Ie as it limits response to flood ping) this would be good to know. > > > > I hope this problem is actually something like that - I know there are a > > lot of FreeBSD based routers out where - if it would be broken for real > > workloads something would scream already. > > > > One more interesting thing I noticed: > > > > Percentage of the requests served within a certain time (ms) > > 50% 32 > > 66% 33 > > 75% 33 > > 80% 33 > > 90% 44 > > 95% 295 > > 98% 324 > > 99% 330 > > 100% 21285 (longest request) > > > > Even if apache benchmark does not timeout it often shows too long > > response rate - (21 sec in this case) > > > > What I've noticed - it can be 3, 9 or 21 secs in this case - This > > really look like the times at which SYN packets are resent by TCP/IP > > stacks if no reply for previous one arrives. > > > > > > Doing more experiments I also discovered I can increase chance of > > passing benchmark (still not to 100%) if i reduce tcp_fin_timeout and > > increase ip_local_port_range variables ob my test driver host. > > > > This still brings the question why with filtering and without behavior > > is different but it makes me worry less :) > > > > > > > > > > Scott > > > > > > > > > On 10/31/05, Peter Zaitsev <[EMAIL PROTECTED]> wrote: > > > > On Mon, 2005-10-31 at 16:25 -0500, Scott Ullrich wrote: > > > > > >apr_poll: The timeout specified has expired (70007) > > > > > > > > > > What is the above from? Your benchmark testing box? > > > > > > > > Yes. This is output from apache benchmark program. > > > > > > > > > > > > Benchmarking 111.111.111.158 (be patient) > > > > Completed 10000 requests > > > > Completed 20000 requests > > > > Completed 30000 requests > > > > apr_poll: The timeout specified has expired (70007) > > > > Total of 30517 requests completed > > > > > > > > > > > > > > > > > > > > > > On 10/31/05, Peter Zaitsev <[EMAIL PROTECTED]> wrote: > > > > > > On Mon, 2005-10-31 at 15:48 -0500, Scott Ullrich wrote: > > > > > > > Are you viewing the traffic queue status? This would be > > normal if you are... > > > > > > > > > > > > Heh, > > > > > > > > > > > > yes good quess. These were running in the other window. > > > > > > > > > > > > > > > > > > So here is the output for "stalled" case > > > > > > > > > > > > # pfctl -ss | wc -l > > > > > > 51898 > > > > > > > > > > > > I have number of states set to 100.000 in advanced page so it is > > not > > > > > > peak number. > > > > > > > > > > > > > > > > > > Note what really surprises me is the number of request when if > > fails: > > > > > > > > > > > > apr_poll: The timeout specified has expired (70007) > > > > > > Total of 28217 requests completed > > > > > > > > > > > > This number of 28217 is seen so often... Sometimes it is a bit > > more ot > > > > > > less but it is very frequently withing +/- 100 of it. > > > > > > > > > > > > I was asked if I can connect to the remote box when this problem > > happens > > > > > > - yes. I can SSH to the same box which runs Apache, but I > > can't > > > > > > connect to the port 80 when this problem happens. > > > > > > > > > > > > So it looks like it does not like to see all these states > > corresponding > > > > > > to the same target port number. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Scott > > > > > > > > > > > > > > > > > > > > > On 10/31/05, Peter Zaitsev <[EMAIL PROTECTED]> wrote: > > > > > > > > On Mon, 2005-10-31 at 14:39 -0500, Scott Ullrich wrote: > > > > > > > > > On 10/31/05, Fleming, John (ZeroChaos) > > <[EMAIL PROTECTED]> wrote: > > > > > > > > > > I wonder if part of the problem is PF isn't seeing the > > TCP tear down. It > > > > > > > > > > seems a little odd that the max gets hit and nothing > > else gets through. > > > > > > > > > > I guess it could be the benchmark isn't shutting down > > the session right > > > > > > > > > > after its down transferring data, but I would think it > > would kill the > > > > > > > > > > benchmark client to have 10K(ish) of open TCP sessions. > > > > > > > > > > > > > > > > > > One way to deterimine this would be to run pfctl -ss | wc > > -l once > > > > > > > > > pfSense stops responding? > > > > > > > > > > > > > > > > Very interesting.... > > > > > > > > > > > > > > > > I tried running this before the problems but it looks > > strange already: > > > > > > > > > > > > > > > > # pfctl -ss | wc -l > > > > > > > > 4893 > > > > > > > > Killed > > > > > > > > # pfctl -ss | wc -l > > > > > > > > 23245 > > > > > > > > Killed > > > > > > > > > > > > > > > > There is nothing in dmesg or system logs. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
