Re: over 30 radiusd processes - more information
Hi,
> 1) Could bad accounting packets cause the radiusd process to EXIT?
> 2) Could bad accounting packets result in hung child processes (as seen
> in the gdb output after the radius log file)?
I'd say yes. we ensure that bad packets dont hit our accounting servers eg
accounting {
# call SQL accouting function only if session-time is not zero
if (Acct-Session-Time != 0) {
sql
}
else {
ok
}
}
alan
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes - more information
> I think you may be 'jumping the gun' a wee bit. > > The system currently has over 13,000 active sessions. > > There were some odd accounting packets, but the vast majority were valid. > These could be configuration errors or hack attempts (investigating). Something broke at 2am. Before that you had a single malformed packet but after 2am there were dozens. > Questions: > > 1) Could bad accounting packets cause the radiusd process to EXIT? I don't think so, but I can't answer why did that happen. > 2) Could bad accounting packets result in hung child processes (as seen in > the gdb output after the radius log file)? You should look into the detail file and see what is wrong with the packets that were stuck (or did the queries fail for some reason), like: WARNING: Unresponsive child for request 165616, in module sql component accounting etc. Ivan Kalik Kalik Informatika ISP - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes - more information
I think you may be 'jumping the gun' a wee bit. The system currently has over 13,000 active sessions. There were some odd accounting packets, but the vast majority were valid. These could be configuration errors or hack attempts (investigating). Questions: 1) Could bad accounting packets cause the radiusd process to EXIT? 2) Could bad accounting packets result in hung child processes (as seen in the gdb output after the radius log file)? Thanks, -craig - Original Message - From: "Ivan Kalik" To: "FreeRadius users mailing list" Sent: Sunday, October 18, 2009 10:56 AM Subject: Re: over 30 radiusd processes - more information I've continued to try an investigate the root cause of this, and the last run behaved slightly differently - the parent process seems to have terminated, and there are more messages in the radius log. There were four (4) hung processes left over. I have attached the radius.log file below, as well as gdb sessions for the hung processes showing the results of the gd 'bt' and 'list' commands. It looks like Alan's initial idea that the hung processes are a result of running the acctstop.sh process are correct. I've tried looking at the code to see if anything 'leapt out' at me, but the logic is quite clever, and dissecting it from the middle is quite a challenge. I am hoping that the gdb output might prove helpful to someone already familiar with the logic flow. It seems I can reproduce this issue within 24 hours, so if there is any other information I could gather, please left me know. Thanks, -craig radiusd.log Fri Oct 16 11:15:56 2009 : Info: Exiting normally. Fri Oct 16 11:16:22 2009 : Info: rlm_sql (sql): Driver rlm_sql_mysql (module rlm_sql_mysql) loaded and linked Fri Oct 16 11:16:22 2009 : Info: rlm_sql (sql): Attempting to connect to radi...@localhost:/radius Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #0 Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #1 Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #2 Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #3 Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #4 Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server inner-tunnel Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server copy-acct-to-home-server Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server copy-acct-to-radius-c Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server Fri Oct 16 11:16:22 2009 : Info: Ready to process requests. Fri Oct 16 17:29:12 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm4tl', nas '192.168.1.101'] Sat Oct 17 02:00:18 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:18 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] etc. /* * If stop but zero session length AND no previous * session found, drop it as in invalid packet * This is to fix CISCO's aaa from filling our * table with bogus crap */ Your NAS is broken. Fix it so it sends proper accounting packets. Ivan Kalik Kalik Informatika ISP - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html __ Information from ESET Smart Security, version of virus signature database 4519 (20091018) __ The message was checked by ESET Smart Security. http://www.eset.com __ Information from ESET Smart Security, version of virus signature database 4519 (20091018) __ The message was checked by ESET Smart Security. http://www.eset.com - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes - more information
Craig Campbell wrote: > I have attached the radius.log file below, as well as gdb sessions for > the hung processes showing the results of the gd 'bt' and 'list' commands. The log is interesting. > Sat Oct 17 02:01:25 2009 : Error: WARNING: Unresponsive child for > request 165616, in module sql component accounting It looks like the SQL module is being blocked somehow. Why? > (gdb) bt > #0 0x003acf4dee6e in __lll_lock_wait_private () from /lib64/libc.so.6 > #1 0x003acf48c75d in _L_lock_1685 () from /lib64/libc.so.6 > #2 0x003acf48c4a7 in __tz_convert () from /lib64/libc.so.6 > #3 0x2b794fa6e39f in vp_prints_value (out=0x41ced6c0 "", > outlen=1008, vp=, >delimitst=) at print.c:267 Hmm... OK. It's a pthread lock / fork issue. I think the solution is to print the "env" variables *before* forking. I'll see what I can do. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes - more information
> I've continued to try an investigate the root cause of this, and the last > run behaved slightly differently - the parent process seems to have > terminated, and there are more messages in the radius log. > > There were four (4) hung processes left over. > > I have attached the radius.log file below, as well as gdb sessions for the > hung processes showing the results of the gd 'bt' and 'list' commands. > > It looks like Alan's initial idea that the hung processes are a result of > running the acctstop.sh process are correct. I've tried looking at the > code > to see if anything 'leapt out' at me, but the logic is quite clever, and > dissecting it from the middle is quite a challenge. > > I am hoping that the gdb output might prove helpful to someone already > familiar with the logic flow. > > It seems I can reproduce this issue within 24 hours, so if there is any > other information I could gather, please left me know. > > Thanks, > -craig > > radiusd.log > > Fri Oct 16 11:15:56 2009 : Info: Exiting normally. > Fri Oct 16 11:16:22 2009 : Info: rlm_sql (sql): Driver rlm_sql_mysql > (module > rlm_sql_mysql) loaded and linked > Fri Oct 16 11:16:22 2009 : Info: rlm_sql (sql): Attempting to connect to > radi...@localhost:/radius > Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL > server for #0 > Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL > server for #1 > Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL > server for #2 > Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL > server for #3 > Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL > server for #4 > Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server inner-tunnel > Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server > copy-acct-to-home-server > Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server > copy-acct-to-radius-c > Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server > Fri Oct 16 11:16:22 2009 : Info: Ready to process requests. > Fri Oct 16 17:29:12 2009 : Error: [sql] stop packet with zero session > length. [user 'use...@realm4tl', nas '192.168.1.101'] > Sat Oct 17 02:00:18 2009 : Error: [sql] stop packet with zero session > length. [user 'use...@realm1', nas '192.168.1.101'] > Sat Oct 17 02:00:18 2009 : Error: [sql] stop packet with zero session > length. [user 'use...@realm1', nas '192.168.1.101'] etc. /* * If stop but zero session length AND no previous * session found, drop it as in invalid packet * This is to fix CISCO's aaa from filling our * table with bogus crap */ Your NAS is broken. Fix it so it sends proper accounting packets. Ivan Kalik Kalik Informatika ISP - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes - more information
I've continued to try an investigate the root cause of this, and the last run behaved slightly differently - the parent process seems to have terminated, and there are more messages in the radius log. There were four (4) hung processes left over. I have attached the radius.log file below, as well as gdb sessions for the hung processes showing the results of the gd 'bt' and 'list' commands. It looks like Alan's initial idea that the hung processes are a result of running the acctstop.sh process are correct. I've tried looking at the code to see if anything 'leapt out' at me, but the logic is quite clever, and dissecting it from the middle is quite a challenge. I am hoping that the gdb output might prove helpful to someone already familiar with the logic flow. It seems I can reproduce this issue within 24 hours, so if there is any other information I could gather, please left me know. Thanks, -craig radiusd.log Fri Oct 16 11:15:56 2009 : Info: Exiting normally. Fri Oct 16 11:16:22 2009 : Info: rlm_sql (sql): Driver rlm_sql_mysql (module rlm_sql_mysql) loaded and linked Fri Oct 16 11:16:22 2009 : Info: rlm_sql (sql): Attempting to connect to radi...@localhost:/radius Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #0 Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #1 Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #2 Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #3 Fri Oct 16 11:16:22 2009 : Info: rlm_sql_mysql: Starting connect to MySQL server for #4 Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server inner-tunnel Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server copy-acct-to-home-server Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server copy-acct-to-radius-c Fri Oct 16 11:16:22 2009 : Info: Loaded virtual server Fri Oct 16 11:16:22 2009 : Info: Ready to process requests. Fri Oct 16 17:29:12 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm4tl', nas '192.168.1.101'] Sat Oct 17 02:00:18 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:18 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:18 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm3', nas '192.168.1.101'] Sat Oct 17 02:00:20 2009 : Error: [sql] stop packet with zero session length. [user 'us...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:20 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:20 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:20 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:20 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:20 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'us...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm3', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm4', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'us...@realm3', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'username', nas '192.168.1.101'] Sat Oct 17 02:00:21 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm2', nas '192.168.1.101'] Sat Oct 17 02:00:22 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:22 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:22 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:22 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm3', nas '192.168.1.101'] Sat Oct 17 02:00:22 2009 : Error: [sql] stop packet with zero session length. [user 'use...@realm1', nas '192.168.1.101'] Sat Oct 17 02:00:22 2009 : Error: [sql] stop packet with
Re: over 30 radiusd processes
Craig Campbell wrote: > While the ps command doesn't show the time of these extra processes > (over 24 hours old), in a previous event, I determined they seemed to > coincide with a significant increase in radius traffic (from ~100/min to > over 1000/min) I believe the NAS forces all users to log off (and they > automatically log back in) in the middle of the night, and I suspect > this is related. Well, the server is *supposed* to clean up child processes. > Alan suggested this might be related to shell scripts being run - as > happens when acct STOP records are received. Yes. If you can replace the shell script with a Perl script and use rlm_perl, the issue will go away. > I'm trying to come up with a strategy to narrow down what might be > happening. The server isn't cleaning up child processes. For some reason, it's lost track of them. > Under what circumstances does radiusd fork? Also, I THOUGH I'd heard > somewhere that threads and fork did NOT interact well. It forks when you exec a program. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Nothing in the log except the normal startup. This server is only receiving accounting records currently. While the ps command doesn't show the time of these extra processes (over 24 hours old), in a previous event, I determined they seemed to coincide with a significant increase in radius traffic (from ~100/min to over 1000/min) I believe the NAS forces all users to log off (and they automatically log back in) in the middle of the night, and I suspect this is related. Also, the system provided is a Virtual Machine, which normally seems to have plenty of resources available. The mysql database is local to the same system, and is only used to keep accounting records of currently logged in users (radutmp replacement - a cron job flushed out completed records to prevent database growth). Alan suggested this might be related to shell scripts being run - as happens when acct STOP records are received. I'm trying to come up with a strategy to narrow down what might be happening. Under what circumstances does radiusd fork? Also, I THOUGH I'd heard somewhere that threads and fork did NOT interact well. I am also considering upgrading to 2.1.7 (but I just finished configuring 2.1.6 :( 2.1.7 wasn't released when I started this..) Thoughts? Thanks (everyone), -craig - Original Message - From: "Marinko Tarlac" To: "FreeRadius users mailing list" Sent: Wednesday, October 14, 2009 7:12 AM Subject: Re: over 30 radiusd processes I had a same problem when one of our databases was terrible slow... Is there anything in radius.log ? Alan DeKok wrote: Craig Campbell wrote: Freeradius 2.1.6 Running on Redhat AS5 Update 3 with mysql-devel rpms added to enable mysql support. Compiled with no options specified. (./configure ; make clean ; make ; make install) I don't know.. all I know is it cleans up processes when I run them, and no one else seems to be running into this. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html __ Information from ESET Smart Security, version of virus signature database 4506 (20091014) __ The message was checked by ESET Smart Security. http://www.eset.com __ Information from ESET Smart Security, version of virus signature database 4506 (20091014) __ The message was checked by ESET Smart Security. http://www.eset.com - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
I had a same problem when one of our databases was terrible slow... Is there anything in radius.log ? Alan DeKok wrote: Craig Campbell wrote: Freeradius 2.1.6 Running on Redhat AS5 Update 3 with mysql-devel rpms added to enable mysql support. Compiled with no options specified. (./configure ; make clean ; make ; make install) I don't know.. all I know is it cleans up processes when I run them, and no one else seems to be running into this. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Craig Campbell wrote: > Freeradius 2.1.6 > > Running on Redhat AS5 Update 3 > with mysql-devel rpms added to enable mysql support. > > Compiled with no options specified. (./configure ; make clean ; make ; > make install) I don't know.. all I know is it cleans up processes when I run them, and no one else seems to be running into this. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Freeradius 2.1.6 Running on Redhat AS5 Update 3 with mysql-devel rpms added to enable mysql support. Compiled with no options specified. (./configure ; make clean ; make ; make install) Thanks, -craig - Original Message - From: "Alan DeKok" To: "FreeRadius users mailing list" Sent: Tuesday, October 13, 2009 1:55 AM Subject: Re: over 30 radiusd processes Craig Campbell wrote: Up to 65 processes now Any ideas how to stop this from happening? Which version are you running? Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html __ Information from ESET Smart Security, version of virus signature database 4501 (20091012) __ The message was checked by ESET Smart Security. http://www.eset.com __ Information from ESET Smart Security, version of virus signature database 4502 (20091013) __ The message was checked by ESET Smart Security. http://www.eset.com - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Craig Campbell wrote: > Up to 65 processes now > > Any ideas how to stop this from happening? Which version are you running? Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Up to 65 processes now
Any ideas how to stop this from happening?
Anyone?
Thanks,
-craig
- Original Message -
From: "Alan DeKok"
To: "FreeRadius users mailing list"
Sent: Saturday, October 10, 2009 1:21 AM
Subject: Re: over 30 radiusd processes
Craig Campbell wrote:
Yes, two(2) binaries and one (1) shell script are called via exec as
follows from the file,
Could you NOT CC me on messages to the list? I subscribe, and I read
the messages.
And fix your mailer. I saw a *large* number of duplicates.
- user (an authentication binary program - Exec-Program-Wait
= "/usr/local/sbin/auth -- %{User-Name} %{User-Password}
%{%{Called-Station-Id}:-Missing} %{%{NAS-IP-Address}:-Missing}
%{%{Calling-Station-Id}:-Missing} %{%{NAS-Port-Type}:-Missing}
%{Vendor-Specific}" ,)
-acct_user (shell script - Exec-Program =
"%{exec:/usr/local/sbin/acctstop.sh}", )
and
-attr_rewrite module (a hex translation binary - replacewith =
"%{exec:/usr/local/sbin/hexconvert -lX %{User-Name} }")
Is this bad?
Is there a better alternative?
Thanks so much!
-craig
- Original Message - From: "Alan DeKok"
To: "FreeRadius users mailing list"
Sent: Friday, October 09, 2009 4:17 PM
Subject: Re: over 30 radiusd processes
Craig Campbell wrote:
radius-a seems to be getting the bulk of the radius records. Normally,
it has a single process.
Last night it spawned a bunch of children that seem to be loitering...
Are you forking shell scripts via "exec"?
radius-b and radius-c don't have more than a single radiusd process.
Any idea what is going on? Why all the children? Do I need to be
concerned? Is this normal?
It's not normal. They're likely zombies that need to go away. The
server normally cleans up any zombie children, but...
Alan DeKok.
-
List info/subscribe/unsubscribe? See
http://www.freeradius.org/list/users.html
__ Information from ESET Smart Security, version of virus
signature database 4494 (20091009) __
The message was checked by ESET Smart Security.
http://www.eset.com
__ Information from ESET Smart Security, version of virus
signature database 4494 (20091009) __
The message was checked by ESET Smart Security.
http://www.eset.com
-
List info/subscribe/unsubscribe? See
http://www.freeradius.org/list/users.html
-
List info/subscribe/unsubscribe? See
http://www.freeradius.org/list/users.html
__ Information from ESET Smart Security, version of virus
signature database 4494 (20091009) __
The message was checked by ESET Smart Security.
http://www.eset.com
__ Information from ESET Smart Security, version of virus signature
database 4501 (20091012) __
The message was checked by ESET Smart Security.
http://www.eset.com
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Craig Campbell wrote:
> Yes, two(2) binaries and one (1) shell script are called via exec as
> follows from the file,
Could you NOT CC me on messages to the list? I subscribe, and I read
the messages.
And fix your mailer. I saw a *large* number of duplicates.
>- user (an authentication binary program - Exec-Program-Wait
> = "/usr/local/sbin/auth -- %{User-Name} %{User-Password}
> %{%{Called-Station-Id}:-Missing} %{%{NAS-IP-Address}:-Missing}
> %{%{Calling-Station-Id}:-Missing} %{%{NAS-Port-Type}:-Missing}
> %{Vendor-Specific}" ,)
>
>-acct_user (shell script - Exec-Program =
> "%{exec:/usr/local/sbin/acctstop.sh}", )
> and
>-attr_rewrite module (a hex translation binary - replacewith =
> "%{exec:/usr/local/sbin/hexconvert -lX %{User-Name} }")
>
> Is this bad?
> Is there a better alternative?
>
>
> Thanks so much!
> -craig
>
>
>
> - Original Message - From: "Alan DeKok"
> To: "FreeRadius users mailing list"
> Sent: Friday, October 09, 2009 4:17 PM
> Subject: Re: over 30 radiusd processes
>
>
>> Craig Campbell wrote:
>>> radius-a seems to be getting the bulk of the radius records. Normally,
>>> it has a single process.
>>> Last night it spawned a bunch of children that seem to be loitering...
>>
>> Are you forking shell scripts via "exec"?
>>
>>> radius-b and radius-c don't have more than a single radiusd process.
>>>
>>> Any idea what is going on? Why all the children? Do I need to be
>>> concerned? Is this normal?
>>
>> It's not normal. They're likely zombies that need to go away. The
>> server normally cleans up any zombie children, but...
>>
>> Alan DeKok.
>> -
>> List info/subscribe/unsubscribe? See
>> http://www.freeradius.org/list/users.html
>>
>> __ Information from ESET Smart Security, version of virus
>> signature database 4494 (20091009) __
>>
>> The message was checked by ESET Smart Security.
>>
>> http://www.eset.com
>>
>>
>>
>
>
> __ Information from ESET Smart Security, version of virus
> signature database 4494 (20091009) __
>
> The message was checked by ESET Smart Security.
>
> http://www.eset.com
>
>
>
> -
> List info/subscribe/unsubscribe? See
> http://www.freeradius.org/list/users.html
>
>
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Yes, two(2) binaries and one (1) shell script are called via exec as follows
from the file,
- user (an authentication binary program - Exec-Program-Wait =
"/usr/local/sbin/auth -- %{User-Name} %{User-Password}
%{%{Called-Station-Id}:-Missing} %{%{NAS-IP-Address}:-Missing}
%{%{Calling-Station-Id}:-Missing} %{%{NAS-Port-Type}:-Missing}
%{Vendor-Specific}" ,)
-acct_user (shell script - Exec-Program =
"%{exec:/usr/local/sbin/acctstop.sh}", )
and
-attr_rewrite module (a hex translation binary - replacewith =
"%{exec:/usr/local/sbin/hexconvert -lX %{User-Name} }")
Is this bad?
Is there a better alternative?
Thanks so much!
-craig
- Original Message -
From: "Alan DeKok"
To: "FreeRadius users mailing list"
Sent: Friday, October 09, 2009 4:17 PM
Subject: Re: over 30 radiusd processes
Craig Campbell wrote:
radius-a seems to be getting the bulk of the radius records. Normally,
it has a single process.
Last night it spawned a bunch of children that seem to be loitering...
Are you forking shell scripts via "exec"?
radius-b and radius-c don't have more than a single radiusd process.
Any idea what is going on? Why all the children? Do I need to be
concerned? Is this normal?
It's not normal. They're likely zombies that need to go away. The
server normally cleans up any zombie children, but...
Alan DeKok.
-
List info/subscribe/unsubscribe? See
http://www.freeradius.org/list/users.html
__ Information from ESET Smart Security, version of virus
signature database 4494 (20091009) __
The message was checked by ESET Smart Security.
http://www.eset.com
__ Information from ESET Smart Security, version of virus signature
database 4494 (20091009) __
The message was checked by ESET Smart Security.
http://www.eset.com
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Craig Campbell wrote: > radius-a seems to be getting the bulk of the radius records. Normally, > it has a single process. > Last night it spawned a bunch of children that seem to be loitering... Are you forking shell scripts via "exec"? > radius-b and radius-c don't have more than a single radiusd process. > > Any idea what is going on? Why all the children? Do I need to be > concerned? Is this normal? It's not normal. They're likely zombies that need to go away. The server normally cleans up any zombie children, but... Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: over 30 radiusd processes
Update: strace suggests all the child processes are doing the same thing, [r...@radius-a raddb]# strace -p 30893 Process 30893 attached - interrupt to quit futex(0x3acf752554, FUTEX_WAIT_PRIVATE, 2, NULL Hope this helps, -craig - Original Message - From: Craig Campbell To: FreeRadius users mailing list Sent: Thursday, October 08, 2009 12:07 PM Subject: over 30 radiusd processes I am runnning 2.1.6 on Redhat Linux (Red Hat Enterprise Linux Server release 5.3 (Tikanga)). This server relays all records to an identical server radius-b and radius-c Similarly radius-b relays its records back to radius-a (except those from radius-a) radius-c is just for testing. All relaying appears to be working correctly. These server currently receiving accounting records only. radius-a seems to be getting the bulk of the radius records. Normally, it has a single process. Last night it spawned a bunch of children that seem to be loitering... radius-b and radius-c don't have more than a single radiusd process. Any idea what is going on? Why all the children? Do I need to be concerned? Is this normal? From the detail log, is seems that the message rate increase to about 2000-2400/minute at that time for about 3 minutes, then dropped to <1000/min. [r...@radius-a radius]# ps -aef | grep radiusd radiusd 5426 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 5738 21400 0 03:29 ?00:00:00 /usr/local/sbin/radiusd radiusd 8239 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8240 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8241 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8242 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8243 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8244 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 9029 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 9104 21400 0 03:29 ?00:00:00 /usr/local/sbin/radiusd radiusd 14154 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 14426 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 15039 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 15040 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 16082 21400 0 03:36 ?00:00:00 /usr/local/sbin/radiusd radiusd 17295 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd root 19242 20229 0 11:42 pts/000:00:00 grep radiusd radiusd 19974 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20670 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20673 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20674 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20675 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20679 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20680 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 21207 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 21208 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 21209 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 21300 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 21301 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 21400 1 0 Oct07 ?00:02:40 /usr/local/sbin/radiusd radiusd 26543 21400 0 03:34 ?00:00:00 /usr/local/sbin/radiusd radiusd 26683 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 28411 21400 0 03:34 ?00:00:00 /usr/local/sbin/radiusd radiusd 29065 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 30648 21400 0 03:34 ?00:00:00 /usr/local/sbin/radiusd radiusd 30649 21400 0 03:34 ?00:00:00 /usr/local/sbin/radiusd radiusd 30893 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd Thanks, -craig __ Information from ESET Smart Security, version of virus signature database 4490 (20091008) __ The message was checked by ESET Smart Security. http://www.eset.com __ Information from ESET Smart Security, version of virus signature database 4490 (20091008) __ The message was checked by ESET Smart Security. http://www.eset.com -- - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html __ Information from ESET Smart Security, version of virus signature database 4490 (20091008) __ The message was checked by ESET Smart Security. http://www.eset.com __ Information from ESET Smart Security, version
over 30 radiusd processes
I am runnning 2.1.6 on Redhat Linux (Red Hat Enterprise Linux Server release 5.3 (Tikanga)). This server relays all records to an identical server radius-b and radius-c Similarly radius-b relays its records back to radius-a (except those from radius-a) radius-c is just for testing. All relaying appears to be working correctly. These server currently receiving accounting records only. radius-a seems to be getting the bulk of the radius records. Normally, it has a single process. Last night it spawned a bunch of children that seem to be loitering... radius-b and radius-c don't have more than a single radiusd process. Any idea what is going on? Why all the children? Do I need to be concerned? Is this normal? >From the detail log, is seems that the message rate increase to about >2000-2400/minute at that time for about 3 minutes, then dropped to <1000/min. [r...@radius-a radius]# ps -aef | grep radiusd radiusd 5426 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 5738 21400 0 03:29 ?00:00:00 /usr/local/sbin/radiusd radiusd 8239 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8240 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8241 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8242 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8243 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 8244 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 9029 21400 0 03:32 ?00:00:00 /usr/local/sbin/radiusd radiusd 9104 21400 0 03:29 ?00:00:00 /usr/local/sbin/radiusd radiusd 14154 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 14426 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 15039 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 15040 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 16082 21400 0 03:36 ?00:00:00 /usr/local/sbin/radiusd radiusd 17295 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd root 19242 20229 0 11:42 pts/000:00:00 grep radiusd radiusd 19974 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20670 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20673 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20674 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20675 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20679 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 20680 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 21207 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 21208 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 21209 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 21300 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 21301 21400 0 03:33 ?00:00:00 /usr/local/sbin/radiusd radiusd 21400 1 0 Oct07 ?00:02:40 /usr/local/sbin/radiusd radiusd 26543 21400 0 03:34 ?00:00:00 /usr/local/sbin/radiusd radiusd 26683 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 28411 21400 0 03:34 ?00:00:00 /usr/local/sbin/radiusd radiusd 29065 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd radiusd 30648 21400 0 03:34 ?00:00:00 /usr/local/sbin/radiusd radiusd 30649 21400 0 03:34 ?00:00:00 /usr/local/sbin/radiusd radiusd 30893 21400 0 03:31 ?00:00:00 /usr/local/sbin/radiusd Thanks, -craig __ Information from ESET Smart Security, version of virus signature database 4490 (20091008) __ The message was checked by ESET Smart Security. http://www.eset.com - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
