Re: Segfaulting with rlm_perl
Norman Elton wrote: > Oh, definitely. We'd do that as a failsafe anyway. My main question is > whether this is failing by design? The intention is to *not* crash. But... FreeRADIUS is dependent on the libraries it uses. If they misbehave, then there's little we can do. > If so, is there a best practice to > avoid the crash (or something like monit to recover from a crash)? And, > if not, is it a resolvable bug? Write Perl scripts that work, or track down exactly *why* it's crashing. The server runs as one process with multiple threads. So a crash takes down the entire server. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
On Friday, September 2, 2011, Alan Buxey wrote: > I meant that if you use eg monit then you can have a service recovery rather than a corner case killing off your radius daemon in middle of night > Oh, definitely. We'd do that as a failsafe anyway. My main question is whether this is failing by design? If so, is there a best practice to avoid the crash (or something like monit to recover from a crash)? And, if not, is it a resolvable bug? Thanks, Norman - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
> Use a wrapper around the demon, eg 'monit' ? Around the radiusd daemon? Nope. Running it from bash, or in this case, from within gdb. Norman - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
I've used GDB to generate a stack trace, specifically using the
instructions on http://freeradius.org/radiusd/doc/bugs.
For this particular test case, I configured as I described above, but
instead of a stripped-down example.pl, I just the one provided, but
put "my $i = 1/0;" in the "test_call" subroutine towards the bottom of
the script.
This was reproduced using 2.1.11.
Let me know if there is additional information I can provide, thanks!
Norman
=
Starting program: /usr/local/sbin/radiusd -d /usr/local/etc/raddb -X
[Thread debugging using libthread_db enabled]
Program received signal SIGSEGV, Segmentation fault.
0x75d8a02a in rlmperl_call (instance=,
request=0x8a5b30,
function_name=0x781670 "authorize") at rlm_perl.c:725
725 exitstatus = POPi;
Missing separate debuginfos, use: debuginfo-install
glibc-2.12-1.25.el6_1.3.x86_64 nss-softokn-freebl-3.12.9-3.el6.x86_64
perl-5.10.1-119.el6.x86_64 perl-libs-5.10.1-119.el6.x86_64
* 1 Thread 0x77fef700 (LWP 29993) 0x75d8a02a in
rlmperl_call (instance=,
request=0x8a5b30, function_name=0x781670 "authorize") at rlm_perl.c:725
Thread 1 (Thread 0x77fef700 (LWP 29993)):
#0 0x75d8a02a in rlmperl_call (instance=, request=0x8a5b30,
function_name=0x781670 "authorize") at rlm_perl.c:725
_sv = 0x411
sp = 0x967f50
inst =
vp =
exitstatus = 0
count = 1
n_a = 70
rad_reply_hv = 0x9519d0
rad_check_hv = 0x908a40
rad_config_hv = 0x0
rad_request_hv = 0x914aa0
rad_request_proxy_hv = 0x942640
rad_request_proxy_reply_hv = 0x951c10
interp = 0x967f50
#1 0x0041af53 in call_modsingle (component=1, c=,
request=) at modcall.c:297
myresult =
#2 modcall (component=1, c=, request=) at modcall.c:670
myresult =
stack = {pointer = 1, priority = {0 },
result = {0 },
children = { }, start = {
}}
parent = 0x785c90
child = 0x7b0a80
sp = 0x7b0a80
if_taken = 0
was_if = 0
#3 0x00417b33 in indexed_modcall (comp=1, idx=0,
request=0x8a5b30) at modules.c:737
rcode =
list = 0x785c90
server =
#4 0x00408646 in rad_authenticate (request=0x8a5b30) at auth.c:579
namepair =
check_item =
auth_item = 0x8a5d50
module_msg =
tmp =
result =
password = 0x4349da ""
autz_retry = 0 '\000'
autz_type =
#5 0x0042796e in radius_handle_request (request=0x8a5b30,
fun=0x4083e0 )
at event.c:3780
No locals.
#6 0x0041ed3d in thread_pool_addrequest (request=0x8a5b30,
fun=0x4083e0 )
at threads.c:874
No locals.
#7 0x00428fee in event_socket_handler (xel=, fd=,
ctx=0x7b1380) at event.c:3425
listener = 0x7b1380
fun = 0x4083e0
request = 0x8a5b30
#8 0x77bd343b in fr_event_loop (el=0x7b1e60) at event.c:413
ef =
i =
rcode = 1
maxfd = 12
when = {tv_sec = 1314989256, tv_usec = 664777}
wake =
read_fds = {fds_bits = {1024, 0 }}
master_fds = {fds_bits = {7424, 0 }}
#9 0x0041be24 in main (argc=,
argv=) at radiusd.c:408
rcode =
argval =
spawn_flag = 0
dont_fork = 1
flag = 0
act = {__sigaction_handler = {sa_handler = 0x41c100 ,
sa_sigaction = 0x41c100 }, sa_mask = {__val =
{0 }},
sa_flags = 0, sa_restorer = 0}
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
>> 2.1.11 is out...and 2.1.12 is almost ready for release - does your system
>> behave in the same way with 2.1.11?
>> Are you using a pre-built package for freeradius or one that you have
>> built yourself?
I am using RedHat's pre-built packages, both FreeRADIUS and Perl. I
have not tried newer versions, but it should be pretty straightforward
to test.
>> And FWIW, we've been using rlm_perl extensively with 2.1.10 without any
>> segfaults. But then again, that might just be because we write bug free
>> perl code :-)
Oh I have no doubt that people are using rlm_perl trouble-free. I'm
just a little concerned that a bug has the capability to crash the
entire server. In development, not a big deal. In fact, it encourages
good error recovery. But I'd rather not wake up in the middle of the
night and find my entire RADIUS infrastructure has died due to an
unexpected corner case. Usernames with unicode characters particularly
terrify me.
> Of course a script error shouldn't segfault the server. It would have been
> much more useful if you had explained what the script error was and a stack
> trace from the segfault.
I don't have a stack trace yet, but I've got an easily reproducible
test case. This is on RedHat 6, using FreeRadius 2.1.10-5, perl
5.10.1-119, x86_64 architecture. From a fresh install, I cleared out
sites-enabled and created a single enabled server:
server srv-perl-crash {
authorize {
preprocess
update control {
Auth-Type := Accept
}
perl
}
authenticate {
noop
}
post-auth {
noop
}
preacct {
noop
}
accounting {
noop
}
}
I route localhost to that server:
client 127.0.0.1 {
shortname = localhost
secret = mysecret
virtual_server = srv-perl-crash
}
And I define a very simple example.pl:
use strict;
use constantRLM_MODULE_OK=>2;# /* the module is OK, continue */
sub authorize {
my $i = 1/0;
return RLM_MODULE_OK;
}
Obviously, a division by zero is a bad thing. But one would expect
FreeRadius to stay online. I fire up the server, and test it with:
radtest -x foo bar 127.0.0.1 1812 mysecret
At first, it gives an error, but survives:
rlm_perl: perl_embed:: module = /etc/raddb/example.pl , func =
authorize exit status= Illegal division by zero at
/etc/raddb/example.pl line 58.
I receive an Access-Reject, and things are fine. I send a second
request, and I get:
rlm_perl: perl_embed:: module = /etc/raddb/example.pl , func =
authorize exit status= Illegal division by zero at
/etc/raddb/example.pl line 58.
Segmentation fault
And the server dies. Uh oh!
Like I said, I will work on a stack trace. Just wanted to get this out
on the list while it's fresh in my mind.
Norman
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
On Fri, Sep 02, 2011 at 07:16:26PM +0200, Bjørn Mork wrote: > Alan Buxey writes: > > >> Oh, I've experienced lots of them! So many, in fact, that I figured it > >> was a common and well understood occurrence. Let me come up with an > >> easily reproducible example and I'll post the relevant information. > > > > 2.1.11 is out...and 2.1.12 is almost ready for release - does your system > > behave in the same way with 2.1.11? > Are you using a pre-built package for freeradius or one that you have built yourself? Perl can pull in so many different libraries that version differences can result in segfaults. Cheers, Ken - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
Alan Buxey writes: >> Oh, I've experienced lots of them! So many, in fact, that I figured it >> was a common and well understood occurrence. Let me come up with an >> easily reproducible example and I'll post the relevant information. > > 2.1.11 is out...and 2.1.12 is almost ready for release - does your system > behave in the same way with 2.1.11? Valid question of course, but do note that there aren't *any* changes to rlm_perl in those versions. Only a few new lines of documentation in the example script: bjorn@canardo:/usr/local/src/git/freeradius$ git diff --stat release_2_1_10..v2.1.x src/modules/rlm_perl/ src/modules/rlm_perl/example.pl | 13 + 1 files changed, 13 insertions(+), 0 deletions(-) And FWIW, we've been using rlm_perl extensively with 2.1.10 without any segfaults. But then again, that might just be because we write bug free perl code :-) Bjørn - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
Hi, > Oh, I've experienced lots of them! So many, in fact, that I figured it > was a common and well understood occurrence. Let me come up with an > easily reproducible example and I'll post the relevant information. 2.1.11 is out...and 2.1.12 is almost ready for release - does your system behave in the same way with 2.1.11? alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
> Of course a script error shouldn't segfault the server. It would have been > much more useful if you had explained what the script error was and a stack > trace from the segfault. Oh, I've experienced lots of them! So many, in fact, that I figured it was a common and well understood occurrence. Let me come up with an easily reproducible example and I'll post the relevant information. Thanks Norman - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Segfaulting with rlm_perl
On 09/02/2011 09:43 AM, Norman Elton wrote: I'm beginning the process of replacing a home-grown RADIUS server with freeradius, a good idea on many many fronts. The server will interact with our backend databases in order to determine attributes to inject into the access-accept messages. For initial development, I've begun work on a perl script that is interacting with freeradius 2.1.10. I'm a little alarmed that a malfunctioning perl script can segfault the entire server. I was hoping that freeradius could compartmentalize a failing script, restarting it as necessary. Admittedly, during development, my scripts are not production-ready, and there are lots of safety checks that simply aren't there. But I'm worried that, going into production, we'll discover an unexpected corner-case that crashes our entire radius infrastructure (not a good thing). Is the best practice simply to make sure you're scripts are bullet-proof? Or is there a more stable method of interacting with an external resource? I chosen rlm_perl because the script didn't need to be instantiated every time (as with rlm_exec), but perhaps we'd be better off relying on rlm_exec if it is more tolerant of corner-case failures. Of course a script error shouldn't segfault the server. It would have been much more useful if you had explained what the script error was and a stack trace from the segfault. -- John Dennis Looking to carve out IT costs? www.redhat.com/carveoutcosts/ - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Segfaulting with rlm_perl
I'm beginning the process of replacing a home-grown RADIUS server with freeradius, a good idea on many many fronts. The server will interact with our backend databases in order to determine attributes to inject into the access-accept messages. For initial development, I've begun work on a perl script that is interacting with freeradius 2.1.10. I'm a little alarmed that a malfunctioning perl script can segfault the entire server. I was hoping that freeradius could compartmentalize a failing script, restarting it as necessary. Admittedly, during development, my scripts are not production-ready, and there are lots of safety checks that simply aren't there. But I'm worried that, going into production, we'll discover an unexpected corner-case that crashes our entire radius infrastructure (not a good thing). Is the best practice simply to make sure you're scripts are bullet-proof? Or is there a more stable method of interacting with an external resource? I chosen rlm_perl because the script didn't need to be instantiated every time (as with rlm_exec), but perhaps we'd be better off relying on rlm_exec if it is more tolerant of corner-case failures. Thanks, Norman - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
