Re: Rejected proxy requests not making it to the client
Ok. I'll be firing up gdb and adding more logging. Before I did that I added a post_proxy detail log to see what the proxy server saw in that phase and for Access-Rejected packets they never get to the post_proxy section. Not sure if that sheds any more light on this. Anyway, so I know where to focus my debugging, I want to make sure I understand how a proxied packet makes its way through the system. Is the path: authorize - pre_proxy - post_proxy That's how it looks from the debug logs. Do the authenticate sections ever get hit? Any other sections I should look into? On Jul 2, 2013, at 3:33 PM, Arran Cudbard-Bell a.cudba...@freeradius.org wrote: On 2 Jul 2013, at 19:28, Ti Leggett legg...@mcs.anl.gov wrote: I'm not seeing a spin lock, but I'm running a 2.2.1 branch version that I believe you pointed me at to fix an rlm_krb5 issue I was seeing earlier this year. Is there an update for that branch or should I be moving to some other version/branch? On Jul 2, 2013, at 1:03 PM, Arran Cudbard-Bell a.cudba...@freeradius.org wrote: On 2 Jul 2013, at 18:51, Alan DeKok al...@deployingradius.com wrote: Ti Leggett wrote: I'm not sure how the script could be blocking the server after it's already ran and returned the updated packet so the proxying can take place which does happen: I don't know. All I know is that the default configuration doesn't have child threads blocking when sending Access-Reject. The problem is due to a local change on your system. There was a bug in rlm_perl which caused it to go into an infinite loop processing reply attributes. Check if radiusd is using 100% cpu, if it is, upgrade. Then it's not the same issue. Break out GDB, set relevant breakpoints, and see where it's hanging, that's all I can suggest. Arran Cudbard-Bell a.cudba...@freeradius.org FreeRADIUS Development Team - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Rejected proxy requests not making it to the client
I have a setup where we have three distinct OTP services, one by the organization and 2 specific to our group. Users can choose which service they want to use and this is done by membership in a netgroup (an rlm_perl script looks up the user to see what netgroup they belong to). We have proxy servers that all the clients talk to and their purpose is to determine which upstream OTP service to authenticate a user against and proxy the requests there. I'm seeing strange behavior when a user types their password incorrectly in that the reject never makes it all the way to the client and on the server I see the following on the server: Tue Jul 2 10:39:04 2013 : Error: WARNING: Unresponsive child for request 0, in component core module thread Tue Jul 2 10:39:05 2013 : Info: WARNING: Child is hung for request 0 in component core module thread. Tue Jul 2 10:39:06 2013 : Info: WARNING: Child is hung for request 0 in component core module thread. Tue Jul 2 10:39:09 2013 : Info: WARNING: Child is hung for request 0 in component core module thread. Tue Jul 2 10:39:12 2013 : Info: WARNING: Child is hung for request 0 in component core module thread. Tue Jul 2 10:39:17 2013 : Info: WARNING: Child is hung for request 0 in component core module thread. Tue Jul 2 10:39:25 2013 : Info: WARNING: Child is hung for request 0 in component core module thread. Tue Jul 2 10:39:36 2013 : Info: WARNING: Child is hung for request 0 in component core module thread. ... The upstream server does get the request, send the reject back to the proxy and the proxy receives the reject but doesn't seem to send the reject back to the client. When the user types the password successfully everything works fine - the client gets an OK and none of the hung request errors show up. A debug log of one of these failed sessions is at http://pastebin.com/8n7snaBV. Any ideas what might be going on? - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Rejected proxy requests not making it to the client
I'm not sure how the script could be blocking the server after it's already ran and returned the updated packet so the proxying can take place which does happen: • rlm_perl: Changing User-Name: legg...@yubiauth.mcs.example.com • rlm_perl: Added pair NAS-Port-Type = Virtual • rlm_perl: Added pair Service-Type = Authenticate-Only • rlm_perl: Added pair Auth-Type = System • rlm_perl: Added pair Calling-Station-Id = client.mcs.example.com • rlm_perl: Added pair User-Name = legg...@yubiauth.mcs.example.com • rlm_perl: Added pair User-Password = 654321 • rlm_perl: Added pair NAS-Identifier = sshd • rlm_perl: Added pair Stripped-User-Name = leggett • rlm_perl: Added pair NAS-IP-Address = 192.168.6.203 • rlm_perl: Added pair NAS-Port = 32448 • rlm_perl: Added pair Ldap-UserDn = uid=leggett,ou=people,dc=mcs,dc=example,dc=com • Cached username is legg...@yubiauth.mcs.example.com, list username is legg...@yubiauth.mcs.example.com • ++[get_domain] returns updated • [suffix] Looking up realm yubiauth.mcs.example.com for User-Name = legg...@yubiauth.mcs.example.com • [suffix] Found realm yubiauth.mcs.example.com • [suffix] Adding Stripped-User-Name = leggett • [suffix] Adding Realm = yubiauth.mcs.example.com • [suffix] Proxying request from user leggett to realm yubiauth.mcs.example.com • [suffix] Preparing to proxy authentication request to realm yubiauth.mcs.example.com • Cached username is leggett, list username is legg...@yubiauth.mcs.example.com • ++[suffix] returns updated The request packet then gets proxied off, comes back and this script is never called again. The same script gets called the same way on successful requests and this script is only called in the authorize phase. I've also tested that when one of the failure cases is reached (return RLM_MODULE_FAIL) that a fail packet is sent back to the client and no proxying ever takes place which is what I would expect. The script is at http://pastebin.com/gB91jj8W. On Jul 2, 2013, at 12:20 PM, Alan DeKok al...@deployingradius.com wrote: Ti Leggett wrote: Tue Jul 2 10:39:04 2013 : Error: WARNING: Unresponsive child for request 0, in component core module thread Fix your scripts so that they don't block the server. The upstream server does get the request, send the reject back to the proxy and the proxy receives the reject but doesn't seem to send the reject back to the client. When the user types the password successfully everything works fine - the client gets an OK and none of the hung request errors show up. The default configuration doesn't have this issue. Access-Requests can be proxied. Access-Rejects can be returned through a proxy to a client. A debug log of one of these failed sessions is at http://pastebin.com/8n7snaBV. Any ideas what might be going on? The debug log shows nothing interesting. The most probable issue is that your scripts are blocking the server. Fix that. You can verify this by configuring a test system *without* your scripts. Or a test user, which bypasses the scripts. It will work. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Rejected proxy requests not making it to the client
Well sure. I figured it was in my local configuration, but is there any other debugging I can do to help me pinpoint what may be going on since, as you said, from the debug logs everything looks correct? Disabling the script will give me a working installation but not a usable one in our environment. The default configuration will do the same. Is there anyway to get more specifics about what exactly is hanging? It might not be the script hanging directly (obviously since it's not hanging before the proxy or after a successful proxy) but could be another module hanging because of how the script modifies the packet. On Jul 2, 2013, at 12:51 PM, Alan DeKok al...@deployingradius.com wrote: Ti Leggett wrote: I'm not sure how the script could be blocking the server after it's already ran and returned the updated packet so the proxying can take place which does happen: I don't know. All I know is that the default configuration doesn't have child threads blocking when sending Access-Reject. The problem is due to a local change on your system. Alan DeKok. - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: Rejected proxy requests not making it to the client
I'm not seeing a spin lock, but I'm running a 2.2.1 branch version that I believe you pointed me at to fix an rlm_krb5 issue I was seeing earlier this year. Is there an update for that branch or should I be moving to some other version/branch? On Jul 2, 2013, at 1:03 PM, Arran Cudbard-Bell a.cudba...@freeradius.org wrote: On 2 Jul 2013, at 18:51, Alan DeKok al...@deployingradius.com wrote: Ti Leggett wrote: I'm not sure how the script could be blocking the server after it's already ran and returned the updated packet so the proxying can take place which does happen: I don't know. All I know is that the default configuration doesn't have child threads blocking when sending Access-Reject. The problem is due to a local change on your system. There was a bug in rlm_perl which caused it to go into an infinite loop processing reply attributes. Check if radiusd is using 100% cpu, if it is, upgrade. Arran Cudbard-Bell a.cudba...@freeradius.org FreeRADIUS Development Team - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
rlm_perl changing User-Name and proxy requests
I have an issue with rlm_perl changing the request User-Name attribute but the proxy request not honoring it. First I'll describe what I'm trying to accomplish and why and then what I'm doing. I'm running a branch of 2.2.1 that has some krb5 realm fixes in it. I have multiple realms that users can authenticate against: our division has replayable password (handled by kerberos) and one time passwords (handled by both YubiKeys and Crypto Card), our lab has replayable passwords (handled by AD) and a separate one time password system (handled by Crypto Card). For services that we want to allow replayable passwords (like IMAP access for instance), we want to allow the user to choose which service to use (division or lab). For services requiring OTP we want the user to choose which OTP token they want to use (some people have multiple because of external requirements). We want users to be able to change these auth preferences on their own and not have this require changing the RADIUS configuration (a.k.a., the users file) to do this. Our account information is kept in LDAP. This is all well and good except that usernames between the division and the lab aren't guaranteed to match - User A might have lastname as their division name, but lastnamefirst as their lab username. For the kerberos and AD request the RADIUS server can handle the request directly using rlm_krb5, but for all the OTP requests the server must proxy to the correct OTP server to handle the request. Here's my plan for accomplishing this. During authorization, rlm_ldap is used to make sure if the user is in LDAP. If not the request is rejected outright (this should help with brute force attempts bogging down all the servers for bogus attempts). Next an rlm_perl module is called to get the user's preferred realm and what username to use in that realm from LDAP. An example would be leggett@ yubi.division.example.com (signifying that I want to use YubiKey from my division). The rlm_perl module updates the request User-Name to be this preference and Auth-Type to be System (see http://pastie.org/5670077). Lastly rlm_realm checks the request to determine if it should be proxied or not. The problem is this: Everything works if the username doesn't change. For instance, if I'm legg...@yubi.division.example.com and legg...@crypto.example.com, things work. The User-Name change being done by rlm_perl is being recognized by rlm_realm and rlm_realm is routing to the proper realm; however, rlm_realm seems to want to clobbe! r User-Name in its own way, so that by the time proxying happens User-Name is set to whatever the initial username was - if I started the process as leggett and wanted my username to eventually be ti.legg...@crypto.example.com, by the time it reaches the proxy phase my User-Name attribute has been reset back to leggett even if I rerun the rlm_perl module after rlm_realm (see http://pastie.org/5670076) I notice that rlm_suffix doesn't overwrite the Stripped-User-Name rlm_perl adds, but appends another entry. This is fine for my purposes since the rlm_perl one is added first subsequent calls use that value. I'm afraid the same is happening with User-Name in that rlm_perl is appending another entry instead of overwriting the ones there. So is there any way around this or am I completely insane with this approach? - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: rlm_perl changing User-Name and proxy requests
On Jan 11, 2013, at 2:32 PM, Arran Cudbard-Bell a.cudba...@freeradius.org wrote: [snip] Yeah it'll just bog down your LDAP server instead. You should use rlm_cache to cache the result of the LDAP lookup (once you have all this working)*. Have you added nostrip for all the realms? The only way I can see it clobbering username is if stripping is enabled. So that was my first thought too. However, I have limited visibility into the remote lab crypto server and when I sent a request to with a realm included, it flat out dropped the request. Didn't reply at all. So I need the realm to so the proxy portion can hit the right destination, but I need the User-Name stripped so the remote server can understand it. -Arran PS: You know you want to test the threaded version of the updated rlm_krb5 module :) I do! Once I get this configuration working I'll be happy to try it. One of my todos for this whole config revamp is to stress test the environment against a brute force attack (we get them frequently). Then I'll have some before numbers to compare with the after. * Only use the rlm_cache module from 2.2.1 - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: rlm_perl changing User-Name and proxy requests
On Jan 11, 2013, at 2:35 PM, a.l.m.bu...@lboro.ac.uk wrote: hi, dont play with User-Name, update/modify Stripped-User-Name instead and use that in the authn/authz stages How do I get the remote servers I'm proxying for to understand Stripped-User-Name. As far as I can tell Stripped-User-Name isn't even in the Access-Request to the proxyied server. alan - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html - List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
Re: rlm_perl changing User-Name and proxy requests
On Jan 11, 2013, at 3:21 PM, Arran Cudbard-Bell a.cudba...@freeradius.org
wrote:
@leggett
If you don't mind rebuilding the server, could you change:
https://github.com/FreeRADIUS/freeradius-server/blob/v2.x.x/src/main/modcall.c#L686
And add:
RDEBUG(Cached username is \%s\, list username is \%s\,
request-username-vp_strvalue, pairfind(request-packet-vps,
PW_USER_NAME)-vp_strvalue);
Just after modcall_single()
Run it in debug mode and you'll see exactly where the username isn't being
updated. List username and cached username should always be in sync up until
the call to suffix, at which point the cached username should be stripped of
the realm.
Ok. I'm flumoxed:
+- entering group pre-proxy {...}
[pre_proxy_log] expand:
/var/log/radius/radacct/%{Client-IP-Address}/pre-proxy-detail-%Y%m%d -
/var/log/radius/radacct/192.168.1.1/pre-proxy-detail-20130111
[pre_proxy_log]
/var/log/radius/radacct/%{Client-IP-Address}/pre-proxy-detail-%Y%m%d expands to
/var/log/radius/radacct/192.168.1.1/pre-proxy-detail-20130111
[pre_proxy_log] expand: %t - Fri Jan 11 15:38:05 2013
Cached username is ti.leggett, list username is
ti.legg...@crypto.example.com
++[pre_proxy_log] returns ok
Sending Access-Request of id 217 to 192.168.1.2 port 1812
NAS-Port-Type = Virtual
Service-Type = Authenticate-Only
Calling-Station-Id = host.division.example.com
User-Name = leggett
User-Password = password
NAS-Identifier = sshd
NAS-IP-Address = 192.168.1.1
NAS-Port = 9975
Proxy-State = 0x3831
-
List info/subscribe/unsubscribe? See http://www.freeradius.org/list/users.html
