Sorry, I must have misunderstood you. When you said: While going through all the KDC related files, I came across all the files which get created while the kdc database was created for the first time. I was wondering is there any way to restore the master key using either the stash file? or either using the database file which resides in the /var/log/kerberos/krb5kdc location? We have both the stash files and the principal.db file. When I open the file although it's not text readable, I can see the K/M@REALM principal details in this file. I thought you said you found another old copy of the principal database file in /var/log/kerberos/krb5kdc (or was that a typo for where the prod db lives in /var/lib/kerberos/krb5kdc?) from before the deletion of K/M occurred. If you did, that would make this easier as it seems existing tools could remedy the problem (dump the K/M entry from that other DB and then load -update into your prod one).
The trick is to make sure it's the correct K/M with the original key you had, but I have some suggestions on how to test that before you affect prod. If you happened to find the original password you used to create the database, that would also work. Heck, even if you only have some guesses as to what the original password was, that could also help. But still, finding the stash file (which is presumably the correct one) is a big help and avoids needing to hunt it down out of a core file. -- Jeff ________________________________ From: Harshawardhan Kulkarni <harshawardhan...@gmail.com> Sent: Friday, June 19, 2020 1:16 PM To: D'Angelo, Jeff C <j...@psu.edu> Cc: kerberos@mit.edu <kerberos@mit.edu> Subject: Re: MIT Kerberos Master principal deletion Hi Jeff, Many thanks for giving this a try. Really appreciated :) Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted. I was not able to proceed beyond this step. My Kerberos version is (Kerberos 5 version 1.12.5), Below is the error I get, kdb5_util dump -d /var/lib/kerberos/krb5kdc/principal krb5.dump kdb5_util: No such entry in the database while retrieving master entry Did you create the dump after deleting the K/M principal? And were you able to create a dump from the .db file? You have clearly depicted my current situation but, I am not sure if this is due to the Kerberos version that i am not able to create the dump file using the principal.db file. Thanks again, Harsh On Fri, Jun 19, 2020 at 1:18 PM D'Angelo, Jeff C <j...@psu.edu<mailto:j...@psu.edu>> wrote: So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux box seemed to find a possible solution to this. I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try. I deleted the K/M principal on a test database (note there's a speed bump in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state. This is after I already made backup dump of the database using kdb5_util; let's call that file "kdb5.dump". For Harsh, I'd be he'd also need to make a dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump). Then I created a shorter dump file of just the header and K/M entry using grep [1]: sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore' [1] Adding the sudo step here for when you are running a non root shell in a normal environment that has root ownership restrictions over the db and dump files. Make sure it's just those two lines: sudo cat kdb5.dump.km_restore Then do a kdb5_util incremental (-update) load with that file: sudo kdb5_util load -update kdb5.dump.km_restore Surprisingly, it worked. I guess kdb5_util load would use the K/M it finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new database and/or blowing out an exiting one. Harsh, what version kerb are you running? Disclaimer: This presumes you haven't changed (rekeyed) K/M since you created your database (well really since you made that backup copy) and that you are really sure that backup copy was from an earlier date of this existing db. I'm not sure yet what loading a different K/M would do. -- Jeff ________________________________ From: kerberos-boun...@mit.edu<mailto:kerberos-boun...@mit.edu> <kerberos-boun...@mit.edu<mailto:kerberos-boun...@mit.edu>> on behalf of Harshawardhan Kulkarni <harshawardhan...@gmail.com<mailto:harshawardhan...@gmail.com>> Sent: Thursday, June 18, 2020 6:27 PM To: kerberos@mit.edu<mailto:kerberos@mit.edu> <kerberos@mit.edu<mailto:kerberos@mit.edu>> Subject: Re: MIT Kerberos Master principal deletion Hi Team, I am reaching out back again with my existing issue regarding master key deletion. I am trying ways to somehow restore it although I don't have a dump of the KDC. Re-creating is the last option for me as the cluster is live and a lot of people are using it. While going through all the KDC related files, I came across all the files which get created while the kdc database was created for the first time. I was wondering is there any way to restore the master key using either the stash file? or either using the database file which resides in the /var/log/kerberos/krb5kdc location? We have both the stash files and the principal.db file. When I open the file although it's not text readable, I can see the K/M@REALM principal details in this file. So is there any way to restore the master key using this principal.db file or the .k5.... stash file? Thanks, Harsh On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni < harshawardhan...@gmail.com<mailto:harshawardhan...@gmail.com>> wrote: > Hi Team, > > I basically need an advice on an ongoing issue I am currently stuck on. > > We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of > the nodes. We don't have a failover node for KDC server yet. On the KDC > admin server while doing a clean up activity for unwanted kdc principals, I > deleted the master key principal (K/m...@realm.com<mailto:m...@realm.com>) We > never took a kdc dump > of the master key. So we don't have a backup to restore from. > > Is there any way I can restore the master key principal? > > I have tried creating with kdb5_util add_mkey but the error says that KDC > DB is not able to find a master key credential. I assume this would only > work when you want to create another master key without deleting the > primary key. > > Another option for me would be to de-kerberise the cluster and create the > same REALM and kerberise the cluster again. But there could be serious > issues if this doesn't fix as this is a live cluster where people are using > this on a daily basis. > > Can anyone help me here? Looking forward for your reply. > > Thanks, > Harsh Kulkarni > ________________________________________________ Kerberos mailing list Kerberos@mit.edu<mailto:Kerberos@mit.edu> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&reserved=0<https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C24638044c9484278c86008d81474972a%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281838273451942&sdata=nq3dpBcY%2BsC2gMW9bWw%2FnPMQiVkmz7RlYQfzQz48N3s%3D&reserved=0> ________________________________________________ Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos