Re: Threads not being used
On Tue, Jan 28, 2003 at 06:05:55PM -0800, Matt Scifo wrote: > I only see one radiusd dameon in top, and my cpu usage never gets higher > than 30%, and thats on one cpu. The other 3 cpu's never get touched. > Load average is negligible as well. Recent tops show only the main thread by default, under Linux. /fc - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
On Thu, 2003-01-30 at 04:15, Alan DeKok wrote: > Matt Scifo <[EMAIL PROTECTED]> wrote: > > Did you get a chance to review the info I posted? Any ideas/thoughts > > would be greatly appreciated. > > Yeah, it didn't look unreasonable to me. The only unusual thing was > that there were a lot of threads in the pool. But that shouldn't > cause problems... > > Try using 4-5 threads in the pool. If the problem still persists, > then you'll have to use more complicated tools to track down the > problem. > > e.g. strace, or ktrace. See what it's doing, and when. Maybe > something in that output would help. > > But I've run the server on some pretty slow systems, and still > gotten 100's of authentications per second. > Ok, thanks for the ideas. I'll let you know if I find anything. Matt - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
Matt Scifo <[EMAIL PROTECTED]> wrote: > Did you get a chance to review the info I posted? Any ideas/thoughts > would be greatly appreciated. Yeah, it didn't look unreasonable to me. The only unusual thing was that there were a lot of threads in the pool. But that shouldn't cause problems... Try using 4-5 threads in the pool. If the problem still persists, then you'll have to use more complicated tools to track down the problem. e.g. strace, or ktrace. See what it's doing, and when. Maybe something in that output would help. But I've run the server on some pretty slow systems, and still gotten 100's of authentications per second. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
Alan
Did you get a chance to review the info I posted? Any ideas/thoughts
would be greatly appreciated.
Matt
On Wed, 2003-01-29 at 10:08, Matt Scifo wrote:
> On Wed, 2003-01-29 at 02:11, Alan DeKok wrote:
> > Matt Scifo <[EMAIL PROTECTED]> wrote:
> > > I didn't even think to look in /proc. I found the same thing. The
> > > threads were spawned according to /proc, yet the daemon is not reporting
> > > thread info in the debug output. Though that still doesn't explain the
> > > horrid numbers I'm seeing.
> >
> > The horrid numbers are due to something else blocking the server
> > (back-end database, disk IO, DNS, etc)
> >
>
> I assumed that was what the issue had to be. Yet I have tuned and
> stripped the server down to the bare minimum and am still seeing
> disappointing numbers.
>
> Let me tell you in more detail exactly how my configuration is set up so
> you can get a better idea about my concerns. As you can see from my
> configuration below, I am still receiving low numbers even when I have
> no back-end database, added disk IO do to writing detail records, and
> hostname lookups are off. Even with no accounting/authentication
> processing, I can never get more than 60 requests per/sec, which is
> disappointing on my hardware and stripped down configuration.
>
> Hardware: Quad Xeon 550mhz with 2g ram and 8g scsi disk
> Software: Redhat 8.0 running Freeradius 0.8.1
> Network: Full Duplex 100mb network
> Configuration: (I removed commented out sections)
>
> BEGIN CONFIGURATION ##
> prefix = /usr/local
> exec_prefix = ${prefix}
> sysconfdir = /etc
> localstatedir = /var
> sbindir = ${exec_prefix}/sbin
> logdir = ${localstatedir}/log/radius
> raddbdir = ${sysconfdir}/raddb
> radacctdir = ${logdir}/radacct
> confdir = ${raddbdir}
> run_dir = ${localstatedir}/run/radiusd
> log_file = ${logdir}/radius.log
> libdir = ${exec_prefix}/lib
> pidfile = ${run_dir}/radiusd.pid
> max_request_time = 30
> delete_blocked_requests = no
> cleanup_delay = 5
> max_requests = 10
> bind_address = *
> port = 0
> hostname_lookups = no
> allow_core_dumps = no
> regular_expressions = yes
> extended_expressions = yes
> log_stripped_names = no
> log_auth = no
> log_auth_badpass = no
> log_auth_goodpass = no
> usercollide = no
> lower_user = no
> lower_pass = no
> nospace_user = no
> nospace_pass = no
> checkrad = ${sbindir}/checkrad
> security {
> max_attributes = 200
> reject_delay = 1
> status_server = no
> }
> proxy_requests = no
> $INCLUDE ${confdir}/proxy.conf
> $INCLUDE ${confdir}/clients.conf
> $INCLUDE ${confdir}/snmp.conf
> thread pool {
> start_servers = 100
> max_servers = 150
> min_spare_servers = 30
> max_spare_servers = 50
> max_requests_per_server = 0
> }
> modules {
> detail {
> detailfile = ${radacctdir}/detail-%Y%m%d
> detailperm = 0600
> }
> acct_unique {
> key = "User-Name, Acct-Session-Id, NAS-IP-Address, Client-IP-Address,
> NAS-Port-Id"
> }
> $INCLUDE ${confdir}/sql.conf
> expr {
> }
> }
> instantiate {
> expr
> }
>
> ## I have run tests with all of these enabled, a combination of them
> ## enabled, and even with none of them enabled.
> accounting {
> #acct_unique
> #detail
> #sql
> }
>
> post-auth {
> }
>
> END CONFIGURATION ##
>
>
>
> Here is debug output from one accounting request packet (with no
> accounting options enabled, hence the "Nothing to do" line)...
>
> rad_recv: Accounting-Request packet from host 66.81.1.206:46298, id=215,
> length=113
> Thread 33 assigned request 2362
> --- Walking the entire request list ---
> Thread 33 handling request 2362, (47 handled so far)
> Cleaning up request 2361 ID 214 with timestamp 3e3811f9
> Nothing to do. Sleeping until we see a request.
> User-Name = "mikem"
> Service-Type = Framed-User
> NAS-IP-Address = 203.63.154.1
> NAS-Port = 1234
> NAS-Port-Type = Async
> Acct-Session-Id = "2206"
> Acct-Status-Type = Stop
> Called-Station-Id = "123456789"
> Calling-Station-Id = "987654321"
> Acct-Delay-Time = 0
> Acct-Session-Time = 1972
> Acct-Input-Octets = 20972
> Acct-Output-Octets = 30972
> Sending Accounting-Response of id 215 to 66.81.1.206:46298
> Finished request 2362
> Going to the next request
> Thread 33 waiting to be assigned a request
>
>
>
>
> Results from "top" during test shows that radiusd never uses more than
> 20% cpu...
>
> 10:01am up 22:18, 2 users, load average: 0.05, 0.06, 0.00
> 94 processes: 93 sleeping, 1 running, 0 zombie, 0 stopped
> CPU0 states: 0.1% user, 4.0% system, 0.0% nice, 94.0% idle
> CPU1 states: 5.0% user, 1.0% system, 0.0% nice, 92.0% idle
> CPU2 states: 2.0% user, 0.0% system, 0.0% nice, 97.0% idle
> CPU3 states: 3.0% user, 0.0% system, 0.0% nice, 96.0% idle
> Mem: 2064712K av, 175380K u
Re: Threads not being used
On Wed, 2003-01-29 at 06:25, Alan DeKok wrote:
> Toni Mueller <[EMAIL PROTECTED]> wrote:
> > we can safely assume that you are able to hammer out enough packets
> > to load your server to begin with...
>
> ./radclient ... -c 1000 ...
>
> Sends a request 1000 times (not duplicated: 1000 unique, but similar
> requests)
>
yeah, it's a nice tool. radpwtst from radiator works the same way, it
justs let you do everything from command line, which makes it easy to
script. like...
time perl -e 'for (1..100) {`radpwtst -time -trace -s xx.xx.xx.xx
-acct_port 1813 -secret testing123 -noauth -iterations 100 \&`;}'
:)
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
Toni Mueller <[EMAIL PROTECTED]> wrote: > we can safely assume that you are able to hammer out enough packets > to load your server to begin with... ./radclient ... -c 1000 ... Sends a request 1000 times (not duplicated: 1000 unique, but similar requests) Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
Hi, On Tue, Jan 28, 2003 at 06:05:55PM -0800, Matt Scifo wrote: > I tested with just start packets and also with start/stop packets using > radpwtst v 1.52 from Radiator. we can safely assume that you are able to hammer out enough packets to load your server to begin with... Although I don't have this problem, I'm interested in the solution ;-) Best, --Toni++ - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
On Wed, 2003-01-29 at 02:11, Alan DeKok wrote:
> Matt Scifo <[EMAIL PROTECTED]> wrote:
> > I didn't even think to look in /proc. I found the same thing. The
> > threads were spawned according to /proc, yet the daemon is not reporting
> > thread info in the debug output. Though that still doesn't explain the
> > horrid numbers I'm seeing.
>
> The horrid numbers are due to something else blocking the server
> (back-end database, disk IO, DNS, etc)
>
I assumed that was what the issue had to be. Yet I have tuned and
stripped the server down to the bare minimum and am still seeing
disappointing numbers.
Let me tell you in more detail exactly how my configuration is set up so
you can get a better idea about my concerns. As you can see from my
configuration below, I am still receiving low numbers even when I have
no back-end database, added disk IO do to writing detail records, and
hostname lookups are off. Even with no accounting/authentication
processing, I can never get more than 60 requests per/sec, which is
disappointing on my hardware and stripped down configuration.
Hardware: Quad Xeon 550mhz with 2g ram and 8g scsi disk
Software: Redhat 8.0 running Freeradius 0.8.1
Network: Full Duplex 100mb network
Configuration: (I removed commented out sections)
BEGIN CONFIGURATION ##
prefix = /usr/local
exec_prefix = ${prefix}
sysconfdir = /etc
localstatedir = /var
sbindir = ${exec_prefix}/sbin
logdir = ${localstatedir}/log/radius
raddbdir = ${sysconfdir}/raddb
radacctdir = ${logdir}/radacct
confdir = ${raddbdir}
run_dir = ${localstatedir}/run/radiusd
log_file = ${logdir}/radius.log
libdir = ${exec_prefix}/lib
pidfile = ${run_dir}/radiusd.pid
max_request_time = 30
delete_blocked_requests = no
cleanup_delay = 5
max_requests = 10
bind_address = *
port = 0
hostname_lookups = no
allow_core_dumps = no
regular_expressions = yes
extended_expressions= yes
log_stripped_names = no
log_auth = no
log_auth_badpass = no
log_auth_goodpass = no
usercollide = no
lower_user = no
lower_pass = no
nospace_user = no
nospace_pass = no
checkrad = ${sbindir}/checkrad
security {
max_attributes = 200
reject_delay = 1
status_server = no
}
proxy_requests = no
$INCLUDE ${confdir}/proxy.conf
$INCLUDE ${confdir}/clients.conf
$INCLUDE ${confdir}/snmp.conf
thread pool {
start_servers = 100
max_servers = 150
min_spare_servers = 30
max_spare_servers = 50
max_requests_per_server = 0
}
modules {
detail {
detailfile = ${radacctdir}/detail-%Y%m%d
detailperm = 0600
}
acct_unique {
key = "User-Name, Acct-Session-Id, NAS-IP-Address, Client-IP-Address,
NAS-Port-Id"
}
$INCLUDE ${confdir}/sql.conf
expr {
}
}
instantiate {
expr
}
## I have run tests with all of these enabled, a combination of them
## enabled, and even with none of them enabled.
accounting {
#acct_unique
#detail
#sql
}
post-auth {
}
END CONFIGURATION ##
Here is debug output from one accounting request packet (with no
accounting options enabled, hence the "Nothing to do" line)...
rad_recv: Accounting-Request packet from host 66.81.1.206:46298, id=215,
length=113
Thread 33 assigned request 2362
--- Walking the entire request list ---
Thread 33 handling request 2362, (47 handled so far)
Cleaning up request 2361 ID 214 with timestamp 3e3811f9
Nothing to do. Sleeping until we see a request.
User-Name = "mikem"
Service-Type = Framed-User
NAS-IP-Address = 203.63.154.1
NAS-Port = 1234
NAS-Port-Type = Async
Acct-Session-Id = "2206"
Acct-Status-Type = Stop
Called-Station-Id = "123456789"
Calling-Station-Id = "987654321"
Acct-Delay-Time = 0
Acct-Session-Time = 1972
Acct-Input-Octets = 20972
Acct-Output-Octets = 30972
Sending Accounting-Response of id 215 to 66.81.1.206:46298
Finished request 2362
Going to the next request
Thread 33 waiting to be assigned a request
Results from "top" during test shows that radiusd never uses more than
20% cpu...
10:01am up 22:18, 2 users, load average: 0.05, 0.06, 0.00
94 processes: 93 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 0.1% user, 4.0% system, 0.0% nice, 94.0% idle
CPU1 states: 5.0% user, 1.0% system, 0.0% nice, 92.0% idle
CPU2 states: 2.0% user, 0.0% system, 0.0% nice, 97.0% idle
CPU3 states: 3.0% user, 0.0% system, 0.0% nice, 96.0% idle
Mem: 2064712K av, 175380K used, 1889332K free, 0K shrd, 40860K buff
Swap: 1052248K av, 0K used, 1052248K free91536K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
3536 root 15 0 1732 1732 776 S14.2 0.0 0:01 radiusd
I hope I have provided enough information to be useful. Are there any
other thoughts you can bring to light that could explain
Re: Threads not being used
At 09:40 AM 1/29/2003 -0800, Matt Scifo wrote: On Wed, 2003-01-29 at 02:08, Alan DeKok wrote: > Debug mode doesn't use threads. Edit the source code & rebuild to > enable threads in debugging mode. > > Look in src/main/radiusd.c for the string 'X' (with the quotes). > Delete one of the following lines which has 'spawn_flag' in it. > Rebuild & reinstall radiusd. > > Alan DeKok. I made this change and am now seeing thread information is debug mode. Alternatively, don't run 'radiusd -X' run 'radiusd -x -x -x' to get debugging info in threaded mode. -Chris -- \\\|||/// \ StarNet Inc. \ Chris Parker \ ~ ~ / \ WX *is* Wireless!\ Director, Engineering | @ @ |\ http://www.starnetwx.net \ (847) 963-0116 oOo---(_)---oOo--\-- \ Wholesale Internet Services - http://www.megapop.net - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
On Wed, 2003-01-29 at 02:08, Alan DeKok wrote: > Matt Scifo <[EMAIL PROTECTED]> wrote: > > I began to play around with my thread-pool settings and noticed that > > only one radiusd process was running despite the settings. > > That depends on your local system. Some systems show only one > process, even if multiple threads are running. > > > Running the daemon in debug mode failed to produce any output > > relating to the use of threads which I had seen in other posts to > > this list. > > Debug mode doesn't use threads. Edit the source code & rebuild to > enable threads in debugging mode. > > Look in src/main/radiusd.c for the string 'X' (with the quotes). > Delete one of the following lines which has 'spawn_flag' in it. > Rebuild & reinstall radiusd. > > Alan DeKok. I made this change and am now seeing thread information is debug mode. Thanks Matt Scifo - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
Matt Scifo <[EMAIL PROTECTED]> wrote: > I didn't even think to look in /proc. I found the same thing. The > threads were spawned according to /proc, yet the daemon is not reporting > thread info in the debug output. Though that still doesn't explain the > horrid numbers I'm seeing. The horrid numbers are due to something else blocking the server (back-end database, disk IO, DNS, etc) Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
Matt Scifo <[EMAIL PROTECTED]> wrote: > I began to play around with my thread-pool settings and noticed that > only one radiusd process was running despite the settings. That depends on your local system. Some systems show only one process, even if multiple threads are running. > Running the daemon in debug mode failed to produce any output > relating to the use of threads which I had seen in other posts to > this list. Debug mode doesn't use threads. Edit the source code & rebuild to enable threads in debugging mode. Look in src/main/radiusd.c for the string 'X' (with the quotes). Delete one of the following lines which has 'spawn_flag' in it. Rebuild & reinstall radiusd. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
On Tue, 2003-01-28 at 17:52, Kostas Kalevras wrote: > On Wed, 28 Jan 2003, Matt Scifo wrote: > > > I didn't even think to look in /proc. I found the same thing. The > > threads were spawned according to /proc, yet the daemon is not reporting > > thread info in the debug output. Though that still doesn't explain the > > horrid numbers I'm seeing. > > > > With only accounting enabled and authorization,authentication,session > > commented out, the number of requests per second should be much higher > > on my test box. > > > > Anyone have any ideas? > > > > Matt > > When you run top do you see one radius server process runing or more? > Also what is your cpu usage and especially your i/o (system) state percentage? > Try enabling only mysql accounting. Detail file accounting can be quite i/o > intensive so you shouldn't rely only on that to judge the server performance. > What are your accounting test packets? Are they start,stop records or both? > I only see one radiusd dameon in top, and my cpu usage never gets higher than 30%, and thats on one cpu. The other 3 cpu's never get touched. Load average is negligible as well. I have tried running the server using just detail accounting, local sql accounting, remote sql accounting (which was less than 1/3 the original number of requests per/sec, probably due to firewall bandwidth throttling), and a combination of detail files and sql accounting. I tested with just start packets and also with start/stop packets using radpwtst v 1.52 from Radiator. Matt - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
On Wed, 28 Jan 2003, Matt Scifo wrote: > I didn't even think to look in /proc. I found the same thing. The > threads were spawned according to /proc, yet the daemon is not reporting > thread info in the debug output. Though that still doesn't explain the > horrid numbers I'm seeing. > > With only accounting enabled and authorization,authentication,session > commented out, the number of requests per second should be much higher > on my test box. > > Anyone have any ideas? > > Matt When you run top do you see one radius server process runing or more? Also what is your cpu usage and especially your i/o (system) state percentage? Try enabling only mysql accounting. Detail file accounting can be quite i/o intensive so you shouldn't rely only on that to judge the server performance. What are your accounting test packets? Are they start,stop records or both? -- Kostas Kalevras Network Operations Center [EMAIL PROTECTED] National Technical University of Athens, Greece Work Phone: +30 210 7721861 'Go back to the shadow' Gandalf - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
I didn't even think to look in /proc. I found the same thing. The threads were spawned according to /proc, yet the daemon is not reporting thread info in the debug output. Though that still doesn't explain the horrid numbers I'm seeing. With only accounting enabled and authorization,authentication,session commented out, the number of requests per second should be much higher on my test box. Anyone have any ideas? Matt On Tue, 2003-01-28 at 17:22, Kevin Bonner wrote: > I see this as well, however, looking at the PID's in /proc shows that the > radiusd threads were spawned Haven't found the time to debug this, but > at least now I know that others are seeing similar things. > > Kevin > > On Tuesday 28 January 2003 19:44, Matt Scifo wrote: > > Hello > > > > I have freeradius-0.8.1 installed on a quad xeon 550mhz box with 2g ram > > running an updated/patched redhat 8.0 dist. > > > > I have configured freeradius to only do accounting, completely disabling > > authentication. I have performed tests using just a detail file as well > > as mysql logging. > > > > I noticed that during my initial testing of freeradius I was only able > > to reach 60 requests per/sec at most, which was rather disappointing. > > > > I began to play around with my thread-pool settings and noticed that > > only one radiusd process was running despite the settings. Running the > > daemon in debug mode failed to produce any output relating to the use of > > threads which I had seen in other posts to this list. > > > > I.E. > > > > Thread 1 assigned request 0 > > Thread 1 handling request 0, (1 handled so far) > > Threads: total/active/spare threads = 5/1/4 > > > > According to the INSTALL file, threads are enabled by default, however I > > recompiled and used --with-threads and --with-thread-pool and still had > > no luck. Output from the configure script shows that the check for > > pthreads was successful. > > > > checking for pthread.h... (cached) yes > > checking for pthread_create in -lpthread... (cached) yes > > > > Am I doing something wrong? If threads are indeed being used and it is > > just not indicated, why are my numbers so low? > > > > Matt Scifo > > > > > > - > > List info/subscribe/unsubscribe? See > > http://www.freeradius.org/list/users.html > > > - > List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Threads not being used
I see this as well, however, looking at the PID's in /proc shows that the radiusd threads were spawned Haven't found the time to debug this, but at least now I know that others are seeing similar things. Kevin On Tuesday 28 January 2003 19:44, Matt Scifo wrote: > Hello > > I have freeradius-0.8.1 installed on a quad xeon 550mhz box with 2g ram > running an updated/patched redhat 8.0 dist. > > I have configured freeradius to only do accounting, completely disabling > authentication. I have performed tests using just a detail file as well > as mysql logging. > > I noticed that during my initial testing of freeradius I was only able > to reach 60 requests per/sec at most, which was rather disappointing. > > I began to play around with my thread-pool settings and noticed that > only one radiusd process was running despite the settings. Running the > daemon in debug mode failed to produce any output relating to the use of > threads which I had seen in other posts to this list. > > I.E. > > Thread 1 assigned request 0 > Thread 1 handling request 0, (1 handled so far) > Threads: total/active/spare threads = 5/1/4 > > According to the INSTALL file, threads are enabled by default, however I > recompiled and used --with-threads and --with-thread-pool and still had > no luck. Output from the configure script shows that the check for > pthreads was successful. > > checking for pthread.h... (cached) yes > checking for pthread_create in -lpthread... (cached) yes > > Am I doing something wrong? If threads are indeed being used and it is > just not indicated, why are my numbers so low? > > Matt Scifo > > > - > List info/subscribe/unsubscribe? See > http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Threads not being used
Hello I have freeradius-0.8.1 installed on a quad xeon 550mhz box with 2g ram running an updated/patched redhat 8.0 dist. I have configured freeradius to only do accounting, completely disabling authentication. I have performed tests using just a detail file as well as mysql logging. I noticed that during my initial testing of freeradius I was only able to reach 60 requests per/sec at most, which was rather disappointing. I began to play around with my thread-pool settings and noticed that only one radiusd process was running despite the settings. Running the daemon in debug mode failed to produce any output relating to the use of threads which I had seen in other posts to this list. I.E. Thread 1 assigned request 0 Thread 1 handling request 0, (1 handled so far) Threads: total/active/spare threads = 5/1/4 According to the INSTALL file, threads are enabled by default, however I recompiled and used --with-threads and --with-thread-pool and still had no luck. Output from the configure script shows that the check for pthreads was successful. checking for pthread.h... (cached) yes checking for pthread_create in -lpthread... (cached) yes Am I doing something wrong? If threads are indeed being used and it is just not indicated, why are my numbers so low? Matt Scifo - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
