Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly

[Lukas Slebodnik]

On (24/02/16 14:28), Marat Vyshegorodtsev wrote:
>> Are you just toying with this or did something go horribly wrong and
>you're trying to restore a production environment?
>
>This. :-(
>
>I have actually rebuilt the environment from scratch, then wrote a
>perl script that just recreated all users from the ldif using ipa
>user-add and reset password for everyone.
>
>After the fresh install the following command was used for each user:
>ipa user-add --first='John' --last='Doe' --uid=160361
>--gid=160361 --email='john@contoso.com' --sshpubkey='ssh-rsa
>' --random john.doe
>
>I had to force uids/gids, so that users don't lose access to their home 
>folders.
>
>I have regenerated keytabs on all client hosts, but now there is some
>weird behavior is demonstrated by sssd: users intermittently fail to
>login. This is a log from a client machine (Amazon Linux 2015.09):
>
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [accept_fd_handler] (0x0400):
>Client connected!
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200):
>Received client version [0].
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200):
>Offered version [0].
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request]
>(0x0400): Requested domain []
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request]
>(0x0400): Parsing name [marat.vyshegorodtsev][]
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_parse_name_for_domains]
>(0x0200): name 'marat.vyshegorodtsev' matched without domain, user is
>marat.vyshegorodtsev
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_ssh_cmd_get_user_pubkeys]
>(0x0400): Requesting SSH user public keys for [marat.vyshegorodtsev]
>from []
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_issue_request]
>(0x0400): Issuing request for
>[0x40b2d0:1:marat.vyshegorodt...@contoso.com]
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_account_msg]
>(0x0400): Creating request for
>[contoso.com][1][1][name=marat.vyshegorodtsev]
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_add_timeout] (0x2000): 0xb99c10
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_internal_get_send]
>(0x0400): Entering request
>[0x40b2d0:1:marat.vyshegorodt...@contoso.com]
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_remove_timeout] (0x2000): 0xb99c10
>(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_reply] (0x1000):
>Got reply from Data Provider - DP error code: 1 errno: 11 error
>message: Offline
sssd works in offline mode.
You can find reason/more details would be in different log files
(sssd_$domain.log).

You instaled server from scratch you it might be acertificate issue
(just a wild guess).

LS

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly

[Marat Vyshegorodtsev]

> Are you just toying with this or did something go horribly wrong and
you're trying to restore a production environment?

This. :-(

I have actually rebuilt the environment from scratch, then wrote a
perl script that just recreated all users from the ldif using ipa
user-add and reset password for everyone.

After the fresh install the following command was used for each user:
ipa user-add --first='John' --last='Doe' --uid=160361
--gid=160361 --email='john@contoso.com' --sshpubkey='ssh-rsa
' --random john.doe

I had to force uids/gids, so that users don't lose access to their home folders.

I have regenerated keytabs on all client hosts, but now there is some
weird behavior is demonstrated by sssd: users intermittently fail to
login. This is a log from a client machine (Amazon Linux 2015.09):

(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [accept_fd_handler] (0x0400):
Client connected!
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200):
Received client version [0].
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_cmd_get_version] (0x0200):
Offered version [0].
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request]
(0x0400): Requested domain []
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_cmd_parse_request]
(0x0400): Parsing name [marat.vyshegorodtsev][]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_parse_name_for_domains]
(0x0200): name 'marat.vyshegorodtsev' matched without domain, user is
marat.vyshegorodtsev
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_ssh_cmd_get_user_pubkeys]
(0x0400): Requesting SSH user public keys for [marat.vyshegorodtsev]
from []
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_issue_request]
(0x0400): Issuing request for
[0x40b2d0:1:marat.vyshegorodt...@contoso.com]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_account_msg]
(0x0400): Creating request for
[contoso.com][1][1][name=marat.vyshegorodtsev]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_add_timeout] (0x2000): 0xb99c10
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_internal_get_send]
(0x0400): Entering request
[0x40b2d0:1:marat.vyshegorodt...@contoso.com]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sbus_remove_timeout] (0x2000): 0xb99c10
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_get_reply] (0x1000):
Got reply from Data Provider - DP error code: 1 errno: 11 error
message: Offline
(Wed Feb 24 22:08:49 2016) [sssd[ssh]]
[ssh_user_pubkeys_search_dp_callback] (0x0040): Unable to get
information from Data Provider Error: 1, 11, Offline
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [ssh_user_pubkeys_search_next]
(0x0400): Requesting SSH user public keys for
[marat.vyshegorodt...@contoso.com]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [sss_dp_req_destructor]
(0x0400): Deleting request:
[0x40b2d0:1:marat.vyshegorodt...@contoso.com]
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [client_recv] (0x0200): Client
disconnected!
(Wed Feb 24 22:08:49 2016) [sssd[ssh]] [client_destructor] (0x2000):
Terminated client [0xb90dd0][17]


This was produced in response to command
/usr/bin/sss_ssh_authorizedkeys marat.vyshegorodtsev

The whole log file has a lot of garbage in it like this:
(Wed Feb 24 22:28:09 2016) [sssd[ssh]] [sbus_get_sender_id_send]
(0x2000): Not a sysbus message, quit
(Wed Feb 24 22:28:19 2016) [sssd[ssh]] [sbus_message_handler]
(0x2000): Received SBUS method org.freedesktop.sssd.service.ping on
path /org/freedesktop/sssd/service

I can do a ldapsearch on the IPA server successfully using
login/password, but sssd just doesn't work.
The server runs Fedora 23 with FreeIPA 4.2.3-2.fc23.

Best regards, Marat


On Wed, Feb 24, 2016 at 6:07 AM, Rob Crittenden  wrote:
> David Kupka wrote:
>> On 23/02/16 20:21, Marat Vyshegorodtsev wrote:
>>> Hi!
>>>
>>> I've been doing backups using the tool like this:
>>> ipa-backup --data --online
>>>
>>> I didn't want any configuration to be backed up, since it is managed
>>> from a chef recipe.
>>>
>>> However, when I tried to recover the backup to a fresh FreeIPA
>>> install, Kerberos (GSSAPI) broke — I can't authenticate myself
>>> anywhere using Kerberos: CLI, HTTP, etc.
>>>
>>> LDAP password-based authentication works alright.
>>>
>>> After some googling and reading through the mailing list, I followed
>>> this manual and updated all keytabs for all services — dirsrv, httpd,
>>> kadmin:
>>> http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server
>>>
>>>
>>> Then it broke  in a different way: for a correct session it says that
>>> my session is expired or just does nothing, for an incorrect password
>>> it responds with "password incorrect" (see screenshot).
>>> https://yadi.sk/i/WVe8u1_ZpNh3w
>>>
>>> For CLI it just says that the credentials are incorrect regardless of
>>> what credentials I provide.
>>>
>>> I suppose that all krbPrincipalKey fields are tied to some other
>>> encryption key that is not included in data-only backup.
>>>
>>> Could you please let me know how to regenerate krbPrincipalKey 

Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly

[Rob Crittenden]

David Kupka wrote:
> On 23/02/16 20:21, Marat Vyshegorodtsev wrote:
>> Hi!
>>
>> I've been doing backups using the tool like this:
>> ipa-backup --data --online
>>
>> I didn't want any configuration to be backed up, since it is managed
>> from a chef recipe.
>>
>> However, when I tried to recover the backup to a fresh FreeIPA
>> install, Kerberos (GSSAPI) broke — I can't authenticate myself
>> anywhere using Kerberos: CLI, HTTP, etc.
>>
>> LDAP password-based authentication works alright.
>>
>> After some googling and reading through the mailing list, I followed
>> this manual and updated all keytabs for all services — dirsrv, httpd,
>> kadmin:
>> http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server
>>
>>
>> Then it broke  in a different way: for a correct session it says that
>> my session is expired or just does nothing, for an incorrect password
>> it responds with "password incorrect" (see screenshot).
>> https://yadi.sk/i/WVe8u1_ZpNh3w
>>
>> For CLI it just says that the credentials are incorrect regardless of
>> what credentials I provide.
>>
>> I suppose that all krbPrincipalKey fields are tied to some other
>> encryption key that is not included in data-only backup.
>>
>> Could you please let me know how to regenerate krbPrincipalKey for all
>> users or how to work around this issue?
>>
>> Best regards,
>> Marat
>>
> 
> Hello Marat,
> I would say that this is expected. During freeipa-server installation
> all service and host kerberos keys are generated randomly, stored in
> Directory Server and in keytab accessible to the host/service.
> When you reinstall freeipa-server all keys are regenerated and no longer
> matches the ones stored in your backup.
> 
> You can use ipa-getkeytab(1) with Directory Manager credentials to
> retrieve new keys but think it's not enough to make it work again.
> Hopefully, someone, who understand kerberos better will advice.
> 

It sounds like he already re-generated those keytabs.

The Kerberos master key is stored in LDAP so you should already have it.
Seeing the KDC and/or httpd logs might be useful.

Are you just toying with this or did something go horribly wrong and
you're trying to restore a production environment?

The instructions you used were strictly a brain dump, something I goofed
around with as an interesting thought project but didn't entirely nail
down. It is quite possible I didn't document some important step in there.

rob

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Re: [Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly

[David Kupka]

On 23/02/16 20:21, Marat Vyshegorodtsev wrote:

Hi!

I've been doing backups using the tool like this:
ipa-backup --data --online

I didn't want any configuration to be backed up, since it is managed
from a chef recipe.

However, when I tried to recover the backup to a fresh FreeIPA
install, Kerberos (GSSAPI) broke — I can't authenticate myself
anywhere using Kerberos: CLI, HTTP, etc.

LDAP password-based authentication works alright.

After some googling and reading through the mailing list, I followed
this manual and updated all keytabs for all services — dirsrv, httpd,
kadmin: 
http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server

Then it broke  in a different way: for a correct session it says that
my session is expired or just does nothing, for an incorrect password
it responds with "password incorrect" (see screenshot).
https://yadi.sk/i/WVe8u1_ZpNh3w

For CLI it just says that the credentials are incorrect regardless of
what credentials I provide.

I suppose that all krbPrincipalKey fields are tied to some other
encryption key that is not included in data-only backup.

Could you please let me know how to regenerate krbPrincipalKey for all
users or how to work around this issue?

Best regards,
Marat



Hello Marat,
I would say that this is expected. During freeipa-server installation 
all service and host kerberos keys are generated randomly, stored in 
Directory Server and in keytab accessible to the host/service.
When you reinstall freeipa-server all keys are regenerated and no longer 
matches the ones stored in your backup.


You can use ipa-getkeytab(1) with Directory Manager credentials to 
retrieve new keys but think it's not enough to make it work again.

Hopefully, someone, who understand kerberos better will advice.

--
David Kupka

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Freeipa-users] Recovering from data-only backup doesn't recover Kerberos keys properly

[Marat Vyshegorodtsev]

Hi!

I've been doing backups using the tool like this:
ipa-backup --data --online

I didn't want any configuration to be backed up, since it is managed
from a chef recipe.

However, when I tried to recover the backup to a fresh FreeIPA
install, Kerberos (GSSAPI) broke — I can't authenticate myself
anywhere using Kerberos: CLI, HTTP, etc.

LDAP password-based authentication works alright.

After some googling and reading through the mailing list, I followed
this manual and updated all keytabs for all services — dirsrv, httpd,
kadmin: 
http://www.freeipa.org/page/V3/Backup_and_Restore#Backup.2C_uninstall.2C_reinstall.2C_restore_JUST_the_LDAP_server

Then it broke  in a different way: for a correct session it says that
my session is expired or just does nothing, for an incorrect password
it responds with "password incorrect" (see screenshot).
https://yadi.sk/i/WVe8u1_ZpNh3w

For CLI it just says that the credentials are incorrect regardless of
what credentials I provide.

I suppose that all krbPrincipalKey fields are tied to some other
encryption key that is not included in data-only backup.

Could you please let me know how to regenerate krbPrincipalKey for all
users or how to work around this issue?

Best regards,
Marat

-- 
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project