Re: [Freeipa-users] ipa spamming radius with otp token?
On Thu, 2015-06-04 at 21:48 +, Bahmer, Eric Vaughn wrote: Someone higher up decided that there was no time for me to resolve this and I’ve been forced to implement a different method for now. I can still continue to work on this, I'll just need to find different hardware to troubleshoot with. I have set up a kerberos.xml in /etc/firewalld/services restricting to tcp 88. I have restricted the service to the specific interface via zone and rich rule. …….. zone interface name=“bond0”/ …. rule family=“ipv4” source address=“10.6.0.0/24”/ service name=“kerberos”/ log prefix=“firewall-kerberos: “ level=“info” limit value=“10/m”/ /log accept/ /rule ….. /zone Same for kpasswd on port 464. I’m also made sure that the krb5.conf has a line for udp_preference_limit = 1 I’ve also made sure to turn caching off in sssd.conf and restarted that. I set a 30 second timeout and 0 retries. Attempting to SSH from the firewall/gateway as a user to the idm server itself. I’ve managed to get things down to just 2 copies with maybe 1 second difference: Fri May 15 15:23:05 Packet-Type = Access-Request NAS-Identifier = “idm2.manage.monitor.net” Service-Type = Authenticate-Only User-Name = “bahmer” User-Password = “123-4567 On the Idm server /var/log/secure: May 15 15:23:03 idm2 unix_chkpwd[15103]: check pass; user unknown May 15 15:23:03 idm2 unix_chkpwd[15103]: password check failed for user (bahmer) May 15 15:23:03 idm2 sshd[15101]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=gate1.manage.monitor.net user=bahmer May 15 15:23:07 idm2 sshd[15101]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=gate1.manage.monitor.net user=bahmer May 15 15:23:07 idm2 sshd[15101]: pam_sss(sshd:auth): received for user bahmer: 17 (Failure setting user credentials) May 15 15:23:09 idm2 sshd[15101]: Failed password for bahmer from 10.6.0.41 port 44347 ssh2 I’ve collected some tcpdump information, most of the kerberos traffic is on the loopback interface and nothing stands out. I can see the two requests in the tcpdump on the interface the idm server should be using to talk to radius. I probably need permission in order to send the captures after sanitizing them for security policy reasons. Is it possible that sssd is the culprit trying to do a pre-auth before the real auth? Anything is possible. What you need to do is capture the incoming krb5 traffic and the outgoing RADIUS traffic from the KDC. The question you need to answer from this data is: does the two output RADIUS requests correspond to one or two incoming krb5 requests. If there are two krb5 requests, then there is probably a bug in SSSD. If there is only one krb5 request, then there is probably a bug or configuration issue in the krb5-otp plugin. Nathaniel On 5/13/15, 12:00 PM, Nathaniel McCallum npmccal...@redhat.com wrote: On Wed, 2015-05-13 at 14:44 +, Bahmer, Eric Vaughn wrote: Institutionally we have a hardware token set up, you use a pin to unlock the device and it spits out a passcode. The passcode allows access through kerberos, radius, or ldap binds to linux servers, or with a custom apache module to websites. I have an out-of-band private network set up that attaches to our intranet using a firewall/gateway server which does some port forwarding for various things like SSH, RDP. I¹m attempting to set up RADIUS on this firewall/gateway to be used as a proxy for freeipa to our token system which I¹d like to be able to use behind the firewall. However I seem to be getting nearly a dozen requests into the radius server, about half are dropped as duplicate, but usually 3-6 get through and since it¹s a single use token the first attempt succeeds, but the rest fail and cause the hardware token to be blacklisted. Is there a way to specify that the user radius login is a one -time token or is this something that sssd or pam is causing? Or does the OTP support just not work in the way I need it to? I have this issue with both the inbox 4.1.0 in RHEL7.1 or the upstream 4.1.4 rpms. My only alternative is probably to set up a KDC on the firewall to trust the institutional realm and have the IdM kerberos realm trust that. This is also a mixed linux/windows environment behind the firewall, I¹ve enabled unix attributes in my AD and I¹m using a script to sync uid/gid with the external ldap. I do think a cross realm trust is the right way to set this up. However, let's look more closely at the RADIUS issue. First, I want to ensure that you are using TCP for your kerberos connections. If you are
Re: [Freeipa-users] ipa spamming radius with otp token?
Someone higher up decided that there was no time for me to resolve this and I’ve been forced to implement a different method for now. I can still continue to work on this, I'll just need to find different hardware to troubleshoot with. I have set up a kerberos.xml in /etc/firewalld/services restricting to tcp 88. I have restricted the service to the specific interface via zone and rich rule. …….. zone interface name=“bond0”/ …. rule family=“ipv4” source address=“10.6.0.0/24”/ service name=“kerberos”/ log prefix=“firewall-kerberos: “ level=“info” limit value=“10/m”/ /log accept/ /rule ….. /zone Same for kpasswd on port 464. I’m also made sure that the krb5.conf has a line for udp_preference_limit = 1 I’ve also made sure to turn caching off in sssd.conf and restarted that. I set a 30 second timeout and 0 retries. Attempting to SSH from the firewall/gateway as a user to the idm server itself. I’ve managed to get things down to just 2 copies with maybe 1 second difference: Fri May 15 15:23:05 Packet-Type = Access-Request NAS-Identifier = “idm2.manage.monitor.net” Service-Type = Authenticate-Only User-Name = “bahmer” User-Password = “123-4567 On the Idm server /var/log/secure: May 15 15:23:03 idm2 unix_chkpwd[15103]: check pass; user unknown May 15 15:23:03 idm2 unix_chkpwd[15103]: password check failed for user (bahmer) May 15 15:23:03 idm2 sshd[15101]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=gate1.manage.monitor.net user=bahmer May 15 15:23:07 idm2 sshd[15101]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=gate1.manage.monitor.net user=bahmer May 15 15:23:07 idm2 sshd[15101]: pam_sss(sshd:auth): received for user bahmer: 17 (Failure setting user credentials) May 15 15:23:09 idm2 sshd[15101]: Failed password for bahmer from 10.6.0.41 port 44347 ssh2 I’ve collected some tcpdump information, most of the kerberos traffic is on the loopback interface and nothing stands out. I can see the two requests in the tcpdump on the interface the idm server should be using to talk to radius. I probably need permission in order to send the captures after sanitizing them for security policy reasons. Is it possible that sssd is the culprit trying to do a pre-auth before the real auth? On 5/13/15, 12:00 PM, Nathaniel McCallum npmccal...@redhat.com wrote: On Wed, 2015-05-13 at 14:44 +, Bahmer, Eric Vaughn wrote: Institutionally we have a hardware token set up, you use a pin to unlock the device and it spits out a passcode. The passcode allows access through kerberos, radius, or ldap binds to linux servers, or with a custom apache module to websites. I have an out-of-band private network set up that attaches to our intranet using a firewall/gateway server which does some port forwarding for various things like SSH, RDP. I¹m attempting to set up RADIUS on this firewall/gateway to be used as a proxy for freeipa to our token system which I¹d like to be able to use behind the firewall. However I seem to be getting nearly a dozen requests into the radius server, about half are dropped as duplicate, but usually 3-6 get through and since it¹s a single use token the first attempt succeeds, but the rest fail and cause the hardware token to be blacklisted. Is there a way to specify that the user radius login is a one-time token or is this something that sssd or pam is causing? Or does the OTP support just not work in the way I need it to? I have this issue with both the inbox 4.1.0 in RHEL7.1 or the upstream 4.1.4 rpms. My only alternative is probably to set up a KDC on the firewall to trust the institutional realm and have the IdM kerberos realm trust that. This is also a mixed linux/windows environment behind the firewall, I¹ve enabled unix attributes in my AD and I¹m using a script to sync uid/gid with the external ldap. I do think a cross realm trust is the right way to set this up. However, let's look more closely at the RADIUS issue. First, I want to ensure that you are using TCP for your kerberos connections. If you are using UDP for kerberos, then the kerberos client will send a new packet which will cause the KDC to fire off a new set of RADIUS messages. The use of TCP should be enforced with kerberos when using OTP. How long does it take for the hardware token RADIUS server to respond? Have you tried adjusting the number of retries and timeout for the RADIUS server in FreeIPA? A longer timeout or fewer retries will reduce the number of packets transmitted. If you are able to setup a test user with fake credentials and could perform a packet capture of kerberos and RADIUS traffic it would help me understand what is going on here. Nathaniel PS - If I had to take a guess based on what I know now, I would suspect that the real culprit is kinit sending too many requests. This is
Re: [Freeipa-users] ipa spamming radius with otp token?
On 05/13/2015 03:01 PM, Bahmer, Eric Vaughn wrote: I¹ll have to look into the the ports on the idm server then, I¹m not overly familiar with firewalld, I tried to install iptables and use the rules I had before I rebuilt the box as RHEL7.1 from RHEL6.6 to test. But it wouldn¹t take, so I ended up turning firewalld off during testing since it was behind the other firewall/gateway server. It probably is using UDP for the connections then, I know you can set an option in kerberos to use tcp depending on the packet size, but it still falls back to udp if the attempt fails. The hardware token radius varies on response time depending on which configuration I¹m using on the firewall/gateway machine. If I had them unblock me at the institutional side I can proxy my radius off their radius and the config is identical to another sub-network that works fairly quick, couple seconds at most. The other config uses ldap and takes several seconds since it has to first open the connection, start tls, pass the root certificate, authenticate as the top level for the realm, fetch user information, then rebind as the user with the token. I have not yet tried to use ldap just for the accounting section in my radius with kerberos as the auth mechanism on their side since I¹d probably have to request a radius principal for the firewall/gateway server since I don¹t own the intranet realm, but I expect it to be nearly as slow having to take so many extra steps. I will give setting up firewalld a go then to restrict udp. Since you mentioned that a cross-realm trust is probably better in this case (I could always go back to using ipa-3.x on RHEL6), assuming the following: Some names changed to protect the guilty. Vlans 4-windows, 5-nfs (though I can use for some general traffic as well), 6-management This is to keep certain types of traffic isolated by vlan and restrict user access. Internal networks mostly use 10.vlan.switch.x and are not registered with the institutional kdc. Intranet realm: example.gov (in lowercase unfortunately) My firewall server: mon-gate1.example.gov ‹ external network, internal vlans 5/6 My internal idm: idm2.manage.monitor.net ‹ vlans 4,5,6 full ipa-server realm is MANAGE.MONITOR.NET owns domain manage.monitor.net on vlan 6 and also serves dns for monitor.net on vlan5, I needed a bare-metal dns on the management vlan for vm host machines. My windows ad: mon-ad.windows.monitor.net ‹ vlans 4,5,6, owns domain windows.monitor.net on vlan4 and is a vm. I already have a trust between manage.monitor.net and windows.monitor.net, but not the groups or account mapping since I need consistent uid/gid. I¹m assuming I need to set up a kdc on the firewall/gateway mon-gate1.example.gov, and have idm trust that, or both idm and the ad trust it, but have the kdc on the firewall/gateway trust the example.gov intranet realm. The only place I¹m confused is how to set up the kdc on the gateway as it¹s multi-homed, I¹m assuming it needs to use it¹s own intranet resolvable fqdn but with a different realm name like BRIDGE.MONITOR.NET or something and make sure that krb5.conf indicates that mon-gate1.example.gov is part of the realm. How I get it done isn¹t quite as important is that I can use our hardware token behind the gateway as I¹m required, for security reasons. There is a bit too much complexity to digest in one mail. It would be beneficial to have a diagram and comments around it to try to understand your environment and goals. The only other comment I would make is that trust is not supported in RHEL6.x it is in tech preview and it is done differently in RHEL7 where it is supported. Using 7.1 is recommended. On 5/13/15, 12:00 PM, Nathaniel McCallum npmccal...@redhat.com wrote: On Wed, 2015-05-13 at 14:44 +, Bahmer, Eric Vaughn wrote: Institutionally we have a hardware token set up, you use a pin to unlock the device and it spits out a passcode. The passcode allows access through kerberos, radius, or ldap binds to linux servers, or with a custom apache module to websites. I have an out-of-band private network set up that attaches to our intranet using a firewall/gateway server which does some port forwarding for various things like SSH, RDP. I¹m attempting to set up RADIUS on this firewall/gateway to be used as a proxy for freeipa to our token system which I¹d like to be able to use behind the firewall. However I seem to be getting nearly a dozen requests into the radius server, about half are dropped as duplicate, but usually 3-6 get through and since it¹s a single use token the first attempt succeeds, but the rest fail and cause the hardware token to be blacklisted. Is there a way to specify that the user radius login is a one-time token or is this something that sssd or pam is causing? Or does the OTP support just not work in the way I need it to? I have this issue with both the inbox 4.1.0 in RHEL7.1 or the upstream 4.1.4 rpms. My only alternative is probably to set up a KDC on the
Re: [Freeipa-users] ipa spamming radius with otp token?
I¹ll have to look into the the ports on the idm server then, I¹m not overly familiar with firewalld, I tried to install iptables and use the rules I had before I rebuilt the box as RHEL7.1 from RHEL6.6 to test. But it wouldn¹t take, so I ended up turning firewalld off during testing since it was behind the other firewall/gateway server. It probably is using UDP for the connections then, I know you can set an option in kerberos to use tcp depending on the packet size, but it still falls back to udp if the attempt fails. The hardware token radius varies on response time depending on which configuration I¹m using on the firewall/gateway machine. If I had them unblock me at the institutional side I can proxy my radius off their radius and the config is identical to another sub-network that works fairly quick, couple seconds at most. The other config uses ldap and takes several seconds since it has to first open the connection, start tls, pass the root certificate, authenticate as the top level for the realm, fetch user information, then rebind as the user with the token. I have not yet tried to use ldap just for the accounting section in my radius with kerberos as the auth mechanism on their side since I¹d probably have to request a radius principal for the firewall/gateway server since I don¹t own the intranet realm, but I expect it to be nearly as slow having to take so many extra steps. I will give setting up firewalld a go then to restrict udp. Since you mentioned that a cross-realm trust is probably better in this case (I could always go back to using ipa-3.x on RHEL6), assuming the following: Some names changed to protect the guilty. Vlans 4-windows, 5-nfs (though I can use for some general traffic as well), 6-management This is to keep certain types of traffic isolated by vlan and restrict user access. Internal networks mostly use 10.vlan.switch.x and are not registered with the institutional kdc. Intranet realm: example.gov (in lowercase unfortunately) My firewall server: mon-gate1.example.gov ‹ external network, internal vlans 5/6 My internal idm: idm2.manage.monitor.net ‹ vlans 4,5,6 full ipa-server realm is MANAGE.MONITOR.NET owns domain manage.monitor.net on vlan 6 and also serves dns for monitor.net on vlan5, I needed a bare-metal dns on the management vlan for vm host machines. My windows ad: mon-ad.windows.monitor.net ‹ vlans 4,5,6, owns domain windows.monitor.net on vlan4 and is a vm. I already have a trust between manage.monitor.net and windows.monitor.net, but not the groups or account mapping since I need consistent uid/gid. I¹m assuming I need to set up a kdc on the firewall/gateway mon-gate1.example.gov, and have idm trust that, or both idm and the ad trust it, but have the kdc on the firewall/gateway trust the example.gov intranet realm. The only place I¹m confused is how to set up the kdc on the gateway as it¹s multi-homed, I¹m assuming it needs to use it¹s own intranet resolvable fqdn but with a different realm name like BRIDGE.MONITOR.NET or something and make sure that krb5.conf indicates that mon-gate1.example.gov is part of the realm. How I get it done isn¹t quite as important is that I can use our hardware token behind the gateway as I¹m required, for security reasons. On 5/13/15, 12:00 PM, Nathaniel McCallum npmccal...@redhat.com wrote: On Wed, 2015-05-13 at 14:44 +, Bahmer, Eric Vaughn wrote: Institutionally we have a hardware token set up, you use a pin to unlock the device and it spits out a passcode. The passcode allows access through kerberos, radius, or ldap binds to linux servers, or with a custom apache module to websites. I have an out-of-band private network set up that attaches to our intranet using a firewall/gateway server which does some port forwarding for various things like SSH, RDP. I¹m attempting to set up RADIUS on this firewall/gateway to be used as a proxy for freeipa to our token system which I¹d like to be able to use behind the firewall. However I seem to be getting nearly a dozen requests into the radius server, about half are dropped as duplicate, but usually 3-6 get through and since it¹s a single use token the first attempt succeeds, but the rest fail and cause the hardware token to be blacklisted. Is there a way to specify that the user radius login is a one-time token or is this something that sssd or pam is causing? Or does the OTP support just not work in the way I need it to? I have this issue with both the inbox 4.1.0 in RHEL7.1 or the upstream 4.1.4 rpms. My only alternative is probably to set up a KDC on the firewall to trust the institutional realm and have the IdM kerberos realm trust that. This is also a mixed linux/windows environment behind the firewall, I¹ve enabled unix attributes in my AD and I¹m using a script to sync uid/gid with the external ldap. I do think a cross realm trust is the right way to set this up. However, let's look more closely at the RADIUS issue. First, I want
Re: [Freeipa-users] ipa spamming radius with otp token?
On 05/13/2015 10:44 AM, Bahmer, Eric Vaughn wrote: Institutionally we have a hardware token set up, you use a pin to unlock the device and it spits out a passcode. The passcode allows access through kerberos, radius, or ldap binds to linux servers, or with a custom apache module to websites. I have an out-of-band private network set up that attaches to our intranet using a firewall/gateway server which does some port forwarding for various things like SSH, RDP. I'm attempting to set up RADIUS on this firewall/gateway to be used as a proxy for freeipa to our token system which I'd like to be able to use behind the firewall. However I seem to be getting nearly a dozen requests into the radius server, about half are dropped as duplicate, but usually 3-6 get through and since it's a single use token the first attempt succeeds, but the rest fail and cause the hardware token to be blacklisted. Is there a way to specify that the user radius login is a one-time token or is this something that sssd or pam is causing? Or does the OTP support just not work in the way I need it to? I have this issue with both the inbox 4.1.0 in RHEL7.1 or the upstream 4.1.4 rpms. My only alternative is probably to set up a KDC on the firewall to trust the institutional realm and have the IdM kerberos realm trust that. This is also a mixed linux/windows environment behind the firewall, I've enabled unix attributes in my AD and I'm using a script to sync uid/gid with the external ldap. Let me rephrase the setup to see if I got it. You have an OTP server, it is behind the firewall. IPA is outside the firewall. You configured IPA to use radius to talk to OTP server. The firewall drops some of the packets but some go through. If this is true then: - There can be a problem with our implementation of the RADIUS client retries. If the client starts a new conversation every time rather than retries the same packet then this is a client side bug. Nathaniel, do you have any hints on how to debug, troubleshoot, change configuration of the RADIUS client? Are retries and timeouts configurable? - The problem can be also on the server side. Server should be tolerant to the identical radius packets and not do more than one 2FA authentication sequence. If it starts more than one it is a bug on the server side. Being the former implementer of one of the RADIUS servers for one of the major 2FA vendors I know exactly how that happens. -- Thank you, Dmitri Pal Director of Engineering for IdM portfolio Red Hat, Inc. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] ipa spamming radius with otp token?
On Wed, 2015-05-13 at 14:44 +, Bahmer, Eric Vaughn wrote: Institutionally we have a hardware token set up, you use a pin to unlock the device and it spits out a passcode. The passcode allows access through kerberos, radius, or ldap binds to linux servers, or with a custom apache module to websites. I have an out-of-band private network set up that attaches to our intranet using a firewall/gateway server which does some port forwarding for various things like SSH, RDP. I’m attempting to set up RADIUS on this firewall/gateway to be used as a proxy for freeipa to our token system which I’d like to be able to use behind the firewall. However I seem to be getting nearly a dozen requests into the radius server, about half are dropped as duplicate, but usually 3-6 get through and since it’s a single use token the first attempt succeeds, but the rest fail and cause the hardware token to be blacklisted. Is there a way to specify that the user radius login is a one-time token or is this something that sssd or pam is causing? Or does the OTP support just not work in the way I need it to? I have this issue with both the inbox 4.1.0 in RHEL7.1 or the upstream 4.1.4 rpms. My only alternative is probably to set up a KDC on the firewall to trust the institutional realm and have the IdM kerberos realm trust that. This is also a mixed linux/windows environment behind the firewall, I’ve enabled unix attributes in my AD and I’m using a script to sync uid/gid with the external ldap. I do think a cross realm trust is the right way to set this up. However, let's look more closely at the RADIUS issue. First, I want to ensure that you are using TCP for your kerberos connections. If you are using UDP for kerberos, then the kerberos client will send a new packet which will cause the KDC to fire off a new set of RADIUS messages. The use of TCP should be enforced with kerberos when using OTP. How long does it take for the hardware token RADIUS server to respond? Have you tried adjusting the number of retries and timeout for the RADIUS server in FreeIPA? A longer timeout or fewer retries will reduce the number of packets transmitted. If you are able to setup a test user with fake credentials and could perform a packet capture of kerberos and RADIUS traffic it would help me understand what is going on here. Nathaniel PS - If I had to take a guess based on what I know now, I would suspect that the real culprit is kinit sending too many requests. This is based on your statement that the RADIUS server is dropping *some* duplicates. This means that the other RADIUS packets are *not* duplicates and probably represent a subsequent AS-REQ on the KDC from kinit. -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
