Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly
I don't know why, but half of my hosts refused to talk to IPA over kerberos, even after I have re-enrolled them and put new keytabs. I ended up dropping sssd-ipa over sssd-ldap and it is working like a charm (over LDAPS though). Frankly, debugging and working with Kerberos has been a nightmare... Now I have only ports 22, 443, and 636 open, it gives a bit more confidence in stability of the whole set up. Marat On Sat, Feb 27, 2016 at 6:32 PM, Lukas Slebodnik wrote: > On (24/02/16 14:28), Marat Vyshegorodtsev wrote: >>> Are you just toying with this or did something go horribly wrong and >>you're trying to restore a production environment? >> >>This. :-( >> >>I have actually rebuilt the environment from scratch, then wrote a >>perl script that just recreated all users from the ldif using ipa >>user-add and reset password for everyone. >> >>After the fresh install the following command was used for each user: >>ipa user-add --first='John' --last='Doe' --uid=160361 >>--gid=160361 --email='john@contoso.com' --sshpubkey='ssh-rsa >>' --random john.doe >> >>I had to force uids/gids, so that users don't lose access to their home >>folders. >> >>I have regenerated keytabs on all client hosts, but now there is some >>weird behavior is demonstrated by sssd: users intermittently fail to >>login. This is a log from a client machine (Amazon Linux 2015.09): >> >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [accept_fd_handler] (0x0400): >>Client connected! >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200): >>Received client version [0]. >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200): >>Offered version [0]. >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request] >>(0x0400): Requested domain [] >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request] >>(0x0400): Parsing name [marat.vyshegorodtsev][] >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_parse_name_for_domains] >>(0x0200): name 'marat.vyshegorodtsev' matched without domain, user is >>marat.vyshegorodtsev >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_ssh_cmd_get_user_pubkeys] >>(0x0400): Requesting SSH user public keys for [marat.vyshegorodtsev] >>from [] >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_issue_request] >>(0x0400): Issuing request for >>[0x40b2d0:1:marat.vyshegorodt...@contoso.com] >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_account_msg] >>(0x0400): Creating request for >>[contoso.com][1][1][name=marat.vyshegorodtsev] >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_add_timeout] (0x2000): 0xb99c10 >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_internal_get_send] >>(0x0400): Entering request >>[0x40b2d0:1:marat.vyshegorodt...@contoso.com] >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_remove_timeout] (0x2000): >>0xb99c10 >>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_reply] (0x1000): >>Got reply from Data Provider - DP error code: 1 errno: 11 error >>message: Offline > sssd works in offline mode. > You can find reason/more details would be in different log files > (sssd_$domain.log). > > You instaled server from scratch you it might be acertificate issue > (just a wild guess). > > LS -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly
On (24/02/16 14:28), Marat Vyshegorodtsev wrote: >> Are you just toying with this or did something go horribly wrong and >you're trying to restore a production environment? > >This. :-( > >I have actually rebuilt the environment from scratch, then wrote a >perl script that just recreated all users from the ldif using ipa >user-add and reset password for everyone. > >After the fresh install the following command was used for each user: >ipa user-add --first='John' --last='Doe' --uid=160361 >--gid=160361 --email='john@contoso.com' --sshpubkey='ssh-rsa >' --random john.doe > >I had to force uids/gids, so that users don't lose access to their home >folders. > >I have regenerated keytabs on all client hosts, but now there is some >weird behavior is demonstrated by sssd: users intermittently fail to >login. This is a log from a client machine (Amazon Linux 2015.09): > >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [accept_fd_handler] (0x0400): >Client connected! >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200): >Received client version [0]. >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200): >Offered version [0]. >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request] >(0x0400): Requested domain [] >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request] >(0x0400): Parsing name [marat.vyshegorodtsev][] >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_parse_name_for_domains] >(0x0200): name 'marat.vyshegorodtsev' matched without domain, user is >marat.vyshegorodtsev >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_ssh_cmd_get_user_pubkeys] >(0x0400): Requesting SSH user public keys for [marat.vyshegorodtsev] >from [] >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_issue_request] >(0x0400): Issuing request for >[0x40b2d0:1:marat.vyshegorodt...@contoso.com] >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_account_msg] >(0x0400): Creating request for >[contoso.com][1][1][name=marat.vyshegorodtsev] >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_add_timeout] (0x2000): 0xb99c10 >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_internal_get_send] >(0x0400): Entering request >[0x40b2d0:1:marat.vyshegorodt...@contoso.com] >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_remove_timeout] (0x2000): 0xb99c10 >(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_reply] (0x1000): >Got reply from Data Provider - DP error code: 1 errno: 11 error >message: Offline sssd works in offline mode. You can find reason/more details would be in different log files (sssd_$domain.log). You instaled server from scratch you it might be acertificate issue (just a wild guess). LS -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly
> Are you just toying with this or did something go horribly wrong and you're trying to restore a production environment? This. :-( I have actually rebuilt the environment from scratch, then wrote a perl script that just recreated all users from the ldif using ipa user-add and reset password for everyone. After the fresh install the following command was used for each user: ipa user-add --first='John' --last='Doe' --uid=160361 --gid=160361 --email='john@contoso.com' --sshpubkey='ssh-rsa ' --random john.doe I had to force uids/gids, so that users don't lose access to their home folders. I have regenerated keytabs on all client hosts, but now there is some weird behavior is demonstrated by sssd: users intermittently fail to login. This is a log from a client machine (Amazon Linux 2015.09): (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [accept_fd_handler] (0x0400): Client connected! (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200): Received client version [0]. (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200): Offered version [0]. (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request] (0x0400): Requested domain [] (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request] (0x0400): Parsing name [marat.vyshegorodtsev][] (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_parse_name_for_domains] (0x0200): name 'marat.vyshegorodtsev' matched without domain, user is marat.vyshegorodtsev (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_ssh_cmd_get_user_pubkeys] (0x0400): Requesting SSH user public keys for [marat.vyshegorodtsev] from [] (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_issue_request] (0x0400): Issuing request for [0x40b2d0:1:marat.vyshegorodt...@contoso.com] (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_account_msg] (0x0400): Creating request for [contoso.com][1][1][name=marat.vyshegorodtsev] (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_add_timeout] (0x2000): 0xb99c10 (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_internal_get_send] (0x0400): Entering request [0x40b2d0:1:marat.vyshegorodt...@contoso.com] (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_remove_timeout] (0x2000): 0xb99c10 (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 1 errno: 11 error message: Offline (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_user_pubkeys_search_dp_callback] (0x0040): Unable to get information from Data Provider Error: 1, 11, Offline (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_user_pubkeys_search_next] (0x0400): Requesting SSH user public keys for [marat.vyshegorodt...@contoso.com] (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_req_destructor] (0x0400): Deleting request: [0x40b2d0:1:marat.vyshegorodt...@contoso.com] (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [client_recv] (0x0200): Client disconnected! (Wed Feb 24 22:08:49 2016) [sssd[ssh]] [client_destructor] (0x2000): Terminated client [0xb90dd0][17] This was produced in response to command /usr/bin/sss_ssh_authorizedkeys marat.vyshegorodtsev The whole log file has a lot of garbage in it like this: (Wed Feb 24 22:28:09 2016) [sssd[ssh]] [sbus_get_sender_id_send] (0x2000): Not a sysbus message, quit (Wed Feb 24 22:28:19 2016) [sssd[ssh]] [sbus_message_handler] (0x2000): Received SBUS method org.freedesktop.sssd.service.ping on path /org/freedesktop/sssd/service I can do a ldapsearch on the IPA server successfully using login/password, but sssd just doesn't work. The server runs Fedora 23 with FreeIPA 4.2.3-2.fc23. Best regards, Marat On Wed, Feb 24, 2016 at 6:07 AM, Rob Crittenden wrote: > David Kupka wrote: >> On 23/02/16 20:21, Marat Vyshegorodtsev wrote: >>> Hi! >>> >>> I've been doing backups using the tool like this: >>> ipa-backup --data --online >>> >>> I didn't want any configuration to be backed up, since it is managed >>> from a chef recipe. >>> >>> However, when I tried to recover the backup to a fresh FreeIPA >>> install, Kerberos (GSSAPI) broke — I can't authenticate myself >>> anywhere using Kerberos: CLI, HTTP, etc. >>> >>> LDAP password-based authentication works alright. >>> >>> After some googling and reading through the mailing list, I followed >>> this manual and updated all keytabs for all services — dirsrv, httpd, >>> kadmin: >>> http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server >>> >>> >>> Then it broke in a different way: for a correct session it says that >>> my session is expired or just does nothing, for an incorrect password >>> it responds with "password incorrect" (see screenshot). >>> https://yadi.sk/i/WVe8u1_ZpNh3w >>> >>> For CLI it just says that the credentials are incorrect regardless of >>> what credentials I provide. >>> >>> I suppose that all krbPrincipalKey fields are tied to some other >>> encryption key that is not included in data-only backup. >>> >>> Could you please let me know how to regenerate krbPrincipalKey for all >>> users or how
Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly
David Kupka wrote: > On 23/02/16 20:21, Marat Vyshegorodtsev wrote: >> Hi! >> >> I've been doing backups using the tool like this: >> ipa-backup --data --online >> >> I didn't want any configuration to be backed up, since it is managed >> from a chef recipe. >> >> However, when I tried to recover the backup to a fresh FreeIPA >> install, Kerberos (GSSAPI) broke — I can't authenticate myself >> anywhere using Kerberos: CLI, HTTP, etc. >> >> LDAP password-based authentication works alright. >> >> After some googling and reading through the mailing list, I followed >> this manual and updated all keytabs for all services — dirsrv, httpd, >> kadmin: >> http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server >> >> >> Then it broke in a different way: for a correct session it says that >> my session is expired or just does nothing, for an incorrect password >> it responds with "password incorrect" (see screenshot). >> https://yadi.sk/i/WVe8u1_ZpNh3w >> >> For CLI it just says that the credentials are incorrect regardless of >> what credentials I provide. >> >> I suppose that all krbPrincipalKey fields are tied to some other >> encryption key that is not included in data-only backup. >> >> Could you please let me know how to regenerate krbPrincipalKey for all >> users or how to work around this issue? >> >> Best regards, >> Marat >> > > Hello Marat, > I would say that this is expected. During freeipa-server installation > all service and host kerberos keys are generated randomly, stored in > Directory Server and in keytab accessible to the host/service. > When you reinstall freeipa-server all keys are regenerated and no longer > matches the ones stored in your backup. > > You can use ipa-getkeytab(1) with Directory Manager credentials to > retrieve new keys but think it's not enough to make it work again. > Hopefully, someone, who understand kerberos better will advice. > It sounds like he already re-generated those keytabs. The Kerberos master key is stored in LDAP so you should already have it. Seeing the KDC and/or httpd logs might be useful. Are you just toying with this or did something go horribly wrong and you're trying to restore a production environment? The instructions you used were strictly a brain dump, something I goofed around with as an interesting thought project but didn't entirely nail down. It is quite possible I didn't document some important step in there. rob -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly
On 23/02/16 20:21, Marat Vyshegorodtsev wrote: Hi! I've been doing backups using the tool like this: ipa-backup --data --online I didn't want any configuration to be backed up, since it is managed from a chef recipe. However, when I tried to recover the backup to a fresh FreeIPA install, Kerberos (GSSAPI) broke — I can't authenticate myself anywhere using Kerberos: CLI, HTTP, etc. LDAP password-based authentication works alright. After some googling and reading through the mailing list, I followed this manual and updated all keytabs for all services — dirsrv, httpd, kadmin: http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server Then it broke in a different way: for a correct session it says that my session is expired or just does nothing, for an incorrect password it responds with "password incorrect" (see screenshot). https://yadi.sk/i/WVe8u1_ZpNh3w For CLI it just says that the credentials are incorrect regardless of what credentials I provide. I suppose that all krbPrincipalKey fields are tied to some other encryption key that is not included in data-only backup. Could you please let me know how to regenerate krbPrincipalKey for all users or how to work around this issue? Best regards, Marat Hello Marat, I would say that this is expected. During freeipa-server installation all service and host kerberos keys are generated randomly, stored in Directory Server and in keytab accessible to the host/service. When you reinstall freeipa-server all keys are regenerated and no longer matches the ones stored in your backup. You can use ipa-getkeytab(1) with Directory Manager credentials to retrieve new keys but think it's not enough to make it work again. Hopefully, someone, who understand kerberos better will advice. -- David Kupka -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project
[Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly
Hi! I've been doing backups using the tool like this: ipa-backup --data --online I didn't want any configuration to be backed up, since it is managed from a chef recipe. However, when I tried to recover the backup to a fresh FreeIPA install, Kerberos (GSSAPI) broke — I can't authenticate myself anywhere using Kerberos: CLI, HTTP, etc. LDAP password-based authentication works alright. After some googling and reading through the mailing list, I followed this manual and updated all keytabs for all services — dirsrv, httpd, kadmin: http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server Then it broke in a different way: for a correct session it says that my session is expired or just does nothing, for an incorrect password it responds with "password incorrect" (see screenshot). https://yadi.sk/i/WVe8u1_ZpNh3w For CLI it just says that the credentials are incorrect regardless of what credentials I provide. I suppose that all krbPrincipalKey fields are tied to some other encryption key that is not included in data-only backup. Could you please let me know how to regenerate krbPrincipalKey for all users or how to work around this issue? Best regards, Marat -- Manage your subscription for the Freeipa-users mailing list: https://www.redhat.com/mailman/listinfo/freeipa-users Go to http://freeipa.org for more info on the project