Re: MIT Kerberos Master principal deletion
A couple points: Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted. I was not able to proceed beyond this step. Same for me in 1.17. It seems that the following actions fail when K/M is missing from the db: (1) A running kadmind cannot ank (create prinicipals) nor cpw (change passwords [1]), but at least other read and metadata modification functions still work (the old mkey is still in memory) (2) A running krb5kdc will continue to issue tickets (the old mkey is still in memory) (3) kadmind cannot be [re]started (4) krb5kdc cannot be [re]started (5) kadmin.local cannot open the database (6) kdb5_util dump will fail the same way you saw [1] I got a misleading error (yes, I had permission in kadm5.acl): change_password: Operation requires ``change-password'' privilege while changing password for "pr...@realm.name". As I mentioned earlier, I was able to re-add K/M from a previous dump with [2]: kdb5_util load -update file_with_just_km.dump [2] And this step doesn't seem to require you to have the correct stash file in place yet, but good to correct as an immediately next or previous step. If it was the correct K/M (and I presume you still have the correct stash file), then life was seemingly back to normal. I also found that when you use an "imposter" K/M (created from a different password/key) but with matching principal name, realm, enctypes and kvno (not sure those all need to match, necessarily, to achieve these results), and replace the stash file to match, I was able to do the following: (1) kdb5_util dump - notably, only the K/M entry was different. The other entries matched what I had before in a previous dump, so presumably still use the same encryption they had before with the old key; in other words, the current K/M does not appear to be used to decrypt or encrypt passwords as this point [3], but the content of the db file are simply dumped in a mostly readable text format. [4] (2) No effect I saw on running krb5kdc [5] (3) kadmind would again permit kadmin clients to ank and cpw again, however, they use the new master key to encrypt not the old, so restoring a database with mixed master keys at the same kvno is likely a recipe for further pain; I might next test using a different kvno (simulating a normal master key rotation) to see how that fares (4) krb5kdc will [re]start again, but fail to issue tickets; kinit attempts are met with an immediate and slightly misleading but arguably technically correct error message "kinit: Password incorrect while getting initial credentials" (before I even get a chance to enter the password) (5) kadmind will start again; I presume some functions may fail at this point such as `kadmin: ktadd -norandkey` due to the master key mismatch [3] Unless you used an option like -mkey_convert [4] If I left the stash file mismatched, it would still dump, but give the warning: kdb5_util: Unable to decrypt latest master key with the provided master key while getting master key list kdb5_util: Warning: proceeding without master key list [5] Not that I tested more than kinits. I'm sure it might be worth testing the effect on updates to "Last successful authentication" and "Last failed authentication", but I presume that will succeed like a `kadmin: modprinc -expire now` did for me. So it would seem a recovery procedure for you would best: (1) first reclaim the correct K/M entry (or your guess at one) into a dump file (2) verify it is the correct entry with a test database, probably best on another system so you don't accidentally break your operational server (3) perhaps [temporarily?] load the correct (or an imposter) K/M onto your operation DB so you can make a dump for emergency backup purpose as well as to use to test on your test db; maybe delete it again so to avoid ank or cpw calls if you can't stop them otherwise (4) See if you can fire up krb5kdc on your test system with a database loaded from prod and kinit against it with a known principal Or something like it. Again, it may be best to test the full scenario on your version of software before assuming it will work for you. As for (1) reclaim the correct K/M entry, I wonder how hard it would be to extend `krb5_util create` to permit using an existing key in a keytab (or stash) file as alternative to the usual password prompt. Otherwise, we may be getting into other esoterica like how to convert a keytab (stash) file to a kdb5_util load_dump format. But if you can find an old copy of the database file, or guesses at the original password, that would also help. -- Jeff From: Harshawardhan Kulkarni Sent: Friday, June 19, 2020 1:16 PM To: D'Angelo, Jeff C Cc: kerberos@mi
Re: MIT Kerberos Master principal deletion
Sorry, I must have misunderstood you. When you said: While going through all the KDC related files, I came across all the files which get created while the kdc database was created for the first time. I was wondering is there any way to restore the master key using either the stash file? or either using the database file which resides in the /var/log/kerberos/krb5kdc location? We have both the stash files and the principal.db file. When I open the file although it's not text readable, I can see the K/M@REALM principal details in this file. I thought you said you found another old copy of the principal database file in /var/log/kerberos/krb5kdc (or was that a typo for where the prod db lives in /var/lib/kerberos/krb5kdc?) from before the deletion of K/M occurred. If you did, that would make this easier as it seems existing tools could remedy the problem (dump the K/M entry from that other DB and then load -update into your prod one). The trick is to make sure it's the correct K/M with the original key you had, but I have some suggestions on how to test that before you affect prod. If you happened to find the original password you used to create the database, that would also work. Heck, even if you only have some guesses as to what the original password was, that could also help. But still, finding the stash file (which is presumably the correct one) is a big help and avoids needing to hunt it down out of a core file. -- Jeff From: Harshawardhan Kulkarni Sent: Friday, June 19, 2020 1:16 PM To: D'Angelo, Jeff C Cc: kerberos@mit.edu Subject: Re: MIT Kerberos Master principal deletion Hi Jeff, Many thanks for giving this a try. Really appreciated :) Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted. I was not able to proceed beyond this step. My Kerberos version is (Kerberos 5 version 1.12.5), Below is the error I get, kdb5_util dump -d /var/lib/kerberos/krb5kdc/principal krb5.dump kdb5_util: No such entry in the database while retrieving master entry Did you create the dump after deleting the K/M principal? And were you able to create a dump from the .db file? You have clearly depicted my current situation but, I am not sure if this is due to the Kerberos version that i am not able to create the dump file using the principal.db file. Thanks again, Harsh On Fri, Jun 19, 2020 at 1:18 PM D'Angelo, Jeff C mailto:j...@psu.edu>> wrote: So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux box seemed to find a possible solution to this. I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try. I deleted the K/M principal on a test database (note there's a speed bump in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state. This is after I already made backup dump of the database using kdb5_util; let's call that file "kdb5.dump". For Harsh, I'd be he'd also need to make a dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump). Then I created a shorter dump file of just the header and K/M entry using grep [1]: sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore' [1] Adding the sudo step here for when you are running a non root shell in a normal environment that has root ownership restrictions over the db and dump files. Make sure it's just those two lines: sudo cat kdb5.dump.km_restore Then do a kdb5_util incremental (-update) load with that file: sudo kdb5_util load -update kdb5.dump.km_restore Surprisingly, it worked. I guess kdb5_util load would use the K/M it finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new database and/or blowing out an exiting one. Harsh, what version kerb are you running? Disclaimer: This presumes you haven't changed (rekeyed) K/M since you created your database (well really since you made that backup copy) and that you are really sure that backup copy was from an earlier date of this existing db. I'm not sure yet what loading a different K/M would do. -- Jeff From: kerberos-boun...@mit.edu<mailto:kerberos-boun...@mit.edu> mailto:kerberos-boun...@mit.edu>> on behalf of Harshawardhan Ku
Re: MIT Kerberos Master principal deletion
Hi Jeff, Many thanks for giving this a try. Really appreciated :) Great that it has worked for you. I never thought of this way. But, I was not able to take the dump from the database (principal) file. Maybe I cannot take the dump as the master principal (K/M) is already deleted. I was not able to proceed beyond this step. My Kerberos version is (Kerberos 5 version 1.12.5), Below is the error I get, kdb5_util dump -d /var/lib/kerberos/krb5kdc/principal krb5.dump kdb5_util: No such entry in the database while retrieving master entry Did you create the dump after deleting the K/M principal? And were you able to create a dump from the .db file? You have clearly depicted my current situation but, I am not sure if this is due to the Kerberos version that i am not able to create the dump file using the principal.db file. Thanks again, Harsh On Fri, Jun 19, 2020 at 1:18 PM D'Angelo, Jeff C wrote: > So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux > box seemed to find a possible solution to this. I'd like to hear from the > veterans if this is a good idea or not as I can guess doing this wrong may > make things worse before I offer it as a suggestion to try. > > I deleted the K/M principal on a test database (note there's a speed bump > in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS > attribute prevents casual deletion over kadmin/kadmind and one would need > kadmin.local to bypass it, so I used kadmin.local to `modprinc > -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind > and krb5kdc running, which is what I expect matches Harsh's state. This is > after I already made backup dump of the database using kdb5_util; let's > call that file "kdb5.dump". For Harsh, I'd be he'd also need to make a > dump of the original db file before continuing (kdb5_util dump -d > /path/to/old/var/krb5kdc/principal krb5.dump). > > Then I created a shorter dump file of just the header and K/M entry using > grep [1]: > sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore' > > [1] Adding the sudo step here for when you are running a non root shell > in a normal environment that has root ownership restrictions over the db > and dump files. > > Make sure it's just those two lines: > sudo cat kdb5.dump.km_restore > > > Then do a kdb5_util incremental (-update) load with that file: > > sudo kdb5_util load -update kdb5.dump.km_restore > > > Surprisingly, it worked. I guess kdb5_util load would use the K/M it > finds in the dump file instead of the living "principal" database file > because it needs to handle the case that it is creating a brand new > database and/or blowing out an exiting one. > > > Harsh, what version kerb are you running? > > > Disclaimer: This presumes you haven't changed (rekeyed) K/M since you > created your database (well really since you made that backup copy) and > that you are really sure that backup copy was from an earlier date of this > existing db. I'm not sure yet what loading a different K/M would do. > > > -- > > Jeff > > -- > *From:* kerberos-boun...@mit.edu on behalf of > Harshawardhan Kulkarni > *Sent:* Thursday, June 18, 2020 6:27 PM > *To:* kerberos@mit.edu > *Subject:* Re: MIT Kerberos Master principal deletion > > Hi Team, > > I am reaching out back again with my existing issue regarding master key > deletion. I am trying ways to somehow restore it although I don't have a > dump of the KDC. > Re-creating is the last option for me as the cluster is live and a lot of > people are using it. > > While going through all the KDC related files, I came across all the files > which get created while the kdc database was created for the first time. > I was wondering is there any way to restore the master key using either the > stash file? or either using the database file which resides in the > /var/log/kerberos/krb5kdc location? > We have both the stash files and the principal.db file. When I open the > file although it's not text readable, I can see the K/M@REALM principal > details in this file. > > So is there any way to restore the master key using this principal.db file > or the .k5 stash file? > > Thanks, > Harsh > > > On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni < > harshawardhan...@gmail.com> wrote: > > > Hi Team, > > > > I basically need an advice on an ongoing issue I am currently stuck on. > > > > We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one > of > > the nodes. We don't have a failover node for KDC server yet. On the
Re: MIT Kerberos Master principal deletion
So doing a simple test on a krb5-1.17 instance I have on a Fedora Linux box seemed to find a possible solution to this. I'd like to hear from the veterans if this is a good idea or not as I can guess doing this wrong may make things worse before I offer it as a suggestion to try. I deleted the K/M principal on a test database (note there's a speed bump in databases created with krb5 versions 1.15+ where the LOCKDOWN_KEYS attribute prevents casual deletion over kadmin/kadmind and one would need kadmin.local to bypass it, so I used kadmin.local to `modprinc -lockdown_keys K/M` first before `delprinc K/M` in kadmin) and left kadmind and krb5kdc running, which is what I expect matches Harsh's state. This is after I already made backup dump of the database using kdb5_util; let's call that file "kdb5.dump". For Harsh, I'd be he'd also need to make a dump of the original db file before continuing (kdb5_util dump -d /path/to/old/var/krb5kdc/principal krb5.dump). Then I created a shorter dump file of just the header and K/M entry using grep [1]: sudo sh -c 'grep -E '(kdb5_util|K/M)' kdb5.dump > kdb5.dump.km_restore' [1] Adding the sudo step here for when you are running a non root shell in a normal environment that has root ownership restrictions over the db and dump files. Make sure it's just those two lines: sudo cat kdb5.dump.km_restore Then do a kdb5_util incremental (-update) load with that file: sudo kdb5_util load -update kdb5.dump.km_restore Surprisingly, it worked. I guess kdb5_util load would use the K/M it finds in the dump file instead of the living "principal" database file because it needs to handle the case that it is creating a brand new database and/or blowing out an exiting one. Harsh, what version kerb are you running? Disclaimer: This presumes you haven't changed (rekeyed) K/M since you created your database (well really since you made that backup copy) and that you are really sure that backup copy was from an earlier date of this existing db. I'm not sure yet what loading a different K/M would do. -- Jeff From: kerberos-boun...@mit.edu on behalf of Harshawardhan Kulkarni Sent: Thursday, June 18, 2020 6:27 PM To: kerberos@mit.edu Subject: Re: MIT Kerberos Master principal deletion Hi Team, I am reaching out back again with my existing issue regarding master key deletion. I am trying ways to somehow restore it although I don't have a dump of the KDC. Re-creating is the last option for me as the cluster is live and a lot of people are using it. While going through all the KDC related files, I came across all the files which get created while the kdc database was created for the first time. I was wondering is there any way to restore the master key using either the stash file? or either using the database file which resides in the /var/log/kerberos/krb5kdc location? We have both the stash files and the principal.db file. When I open the file although it's not text readable, I can see the K/M@REALM principal details in this file. So is there any way to restore the master key using this principal.db file or the .k5 stash file? Thanks, Harsh On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni < harshawardhan...@gmail.com> wrote: > Hi Team, > > I basically need an advice on an ongoing issue I am currently stuck on. > > We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of > the nodes. We don't have a failover node for KDC server yet. On the KDC > admin server while doing a clean up activity for unwanted kdc principals, I > deleted the master key principal (K/m...@realm.com) We never took a kdc dump > of the master key. So we don't have a backup to restore from. > > Is there any way I can restore the master key principal? > > I have tried creating with kdb5_util add_mkey but the error says that KDC > DB is not able to find a master key credential. I assume this would only > work when you want to create another master key without deleting the > primary key. > > Another option for me would be to de-kerberise the cluster and create the > same REALM and kerberise the cluster again. But there could be serious > issues if this doesn't fix as this is a live cluster where people are using > this on a daily basis. > > Can anyone help me here? Looking forward for your reply. > > Thanks, > Harsh Kulkarni > Kerberos mailing list Kerberos@mit.edu https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C0c15f8ef8a3b49a94a8d08d813dc11fc%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637281183207940471&sdata=nXkq2krG5q8Shuw6BQ%2FOKIHxS87a%2FrNinLwV%2BOXEk8g%3D&reserved=0 Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: MIT Kerberos Master principal deletion
Hi Team, I am reaching out back again with my existing issue regarding master key deletion. I am trying ways to somehow restore it although I don't have a dump of the KDC. Re-creating is the last option for me as the cluster is live and a lot of people are using it. While going through all the KDC related files, I came across all the files which get created while the kdc database was created for the first time. I was wondering is there any way to restore the master key using either the stash file? or either using the database file which resides in the /var/log/kerberos/krb5kdc location? We have both the stash files and the principal.db file. When I open the file although it's not text readable, I can see the K/M@REALM principal details in this file. So is there any way to restore the master key using this principal.db file or the .k5 stash file? Thanks, Harsh On Thu, Jun 11, 2020 at 3:32 AM Harshawardhan Kulkarni < harshawardhan...@gmail.com> wrote: > Hi Team, > > I basically need an advice on an ongoing issue I am currently stuck on. > > We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of > the nodes. We don't have a failover node for KDC server yet. On the KDC > admin server while doing a clean up activity for unwanted kdc principals, I > deleted the master key principal (K/m...@realm.com) We never took a kdc dump > of the master key. So we don't have a backup to restore from. > > Is there any way I can restore the master key principal? > > I have tried creating with kdb5_util add_mkey but the error says that KDC > DB is not able to find a master key credential. I assume this would only > work when you want to create another master key without deleting the > primary key. > > Another option for me would be to de-kerberise the cluster and create the > same REALM and kerberise the cluster again. But there could be serious > issues if this doesn't fix as this is a live cluster where people are using > this on a daily basis. > > Can anyone help me here? Looking forward for your reply. > > Thanks, > Harsh Kulkarni > Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: MIT Kerberos Master principal deletion
Hi Jeff, I have found the stash file, can I recover the master key from this file? Do you know any good links to follow? Thanks Harsh Sent from my iPhone > On 16 Jun 2020, at 04:07, D'Angelo, Jeff C wrote: > > > Would the stash file help here (if it exists)? > > -- > Jeff > > From: kerberos-boun...@mit.edu on behalf of Chris > Hecker > Sent: Thursday, June 11, 2020 6:54 PM > To: Nico Williams > Cc: Harshawardhan Kulkarni ; kerberos@mit.edu > > Subject: Re[2]: MIT Kerberos Master principal deletion > > > > I don't think it would make it harder. > > I just mean because you won't be able to set a breakpoint at a function > that uses the key, you'll have to actually chase it around in memory > (assuming you use something like gcore to dump it as fast as possible > without regard to where it is executing when it's dumped). > > If I was doing this live, I'd set a breakpoint on some function that > used the key to decrypt and then inspect there, but with a core file > you'll need to make sure you can find all the structures first. > > Is realm_mkey in the kdc_realm_data struct the one he wants? > > Chris > > -- Original Message -- > From: "Nico Williams" > To: "Chris Hecker" > Cc: "Harshawardhan Kulkarni" ; > "kerberos@mit.edu" > Sent: 2020-06-11 15:31:28 > Subject: Re: MIT Kerberos Master principal deletion > > >On Thu, Jun 11, 2020 at 10:19:39PM +, Chris Hecker wrote: > >> Maybe dump the core of the running process so you don't accidentally crash > >> it while trying to debug it live? But that would make finding it in > >> memory > >> even harder... > > > >I don't think it would make it harder. > > > >BTW, we should make it much harder to delete important principals... > > > > Kerberos mailing list Kerberos@mit.edu > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C5ecb0ae46a0f4206310108d80e5b131f%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637275131630535798&sdata=slErWkRJAvfE0nd%2BMESCEFY5Ucx8c79mIpMN%2BwFBMz8%3D&reserved=0 Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: Re[2]: MIT Kerberos Master principal deletion
Would the stash file help here (if it exists)? -- Jeff From: kerberos-boun...@mit.edu on behalf of Chris Hecker Sent: Thursday, June 11, 2020 6:54 PM To: Nico Williams Cc: Harshawardhan Kulkarni ; kerberos@mit.edu Subject: Re[2]: MIT Kerberos Master principal deletion > I don't think it would make it harder. I just mean because you won't be able to set a breakpoint at a function that uses the key, you'll have to actually chase it around in memory (assuming you use something like gcore to dump it as fast as possible without regard to where it is executing when it's dumped). If I was doing this live, I'd set a breakpoint on some function that used the key to decrypt and then inspect there, but with a core file you'll need to make sure you can find all the structures first. Is realm_mkey in the kdc_realm_data struct the one he wants? Chris -- Original Message -- From: "Nico Williams" To: "Chris Hecker" Cc: "Harshawardhan Kulkarni" ; "kerberos@mit.edu" Sent: 2020-06-11 15:31:28 Subject: Re: MIT Kerberos Master principal deletion >On Thu, Jun 11, 2020 at 10:19:39PM +, Chris Hecker wrote: >> Maybe dump the core of the running process so you don't accidentally crash >> it while trying to debug it live? But that would make finding it in memory >> even harder... > >I don't think it would make it harder. > >BTW, we should make it much harder to delete important principals... Kerberos mailing list Kerberos@mit.edu https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmailman.mit.edu%2Fmailman%2Flistinfo%2Fkerberos&data=02%7C01%7Cjcd%40psu.edu%7C5ecb0ae46a0f4206310108d80e5b131f%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C0%7C637275131630535798&sdata=slErWkRJAvfE0nd%2BMESCEFY5Ucx8c79mIpMN%2BwFBMz8%3D&reserved=0 Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re[3]: MIT Kerberos Master principal deletion
Looks like it might also be in the global master_keyblock in the server_kdb.c file. Chris -- Original Message -- From: "Chris Hecker" To: "Nico Williams" Cc: "Harshawardhan Kulkarni" ; "kerberos@mit.edu" Sent: 2020-06-11 15:54:32 Subject: Re[2]: MIT Kerberos Master principal deletion > > > I don't think it would make it harder. > >I just mean because you won't be able to set a breakpoint at a function that >uses the key, you'll have to actually chase it around in memory (assuming you >use something like gcore to dump it as fast as possible without regard to >where it is executing when it's dumped). > >If I was doing this live, I'd set a breakpoint on some function that used the >key to decrypt and then inspect there, but with a core file you'll need to >make sure you can find all the structures first. > >Is realm_mkey in the kdc_realm_data struct the one he wants? > >Chris > >-- Original Message -- >From: "Nico Williams" >To: "Chris Hecker" >Cc: "Harshawardhan Kulkarni" ; "kerberos@mit.edu" > >Sent: 2020-06-11 15:31:28 >Subject: Re: MIT Kerberos Master principal deletion > >>On Thu, Jun 11, 2020 at 10:19:39PM +, Chris Hecker wrote: >>> Maybe dump the core of the running process so you don't accidentally crash >>> it while trying to debug it live? But that would make finding it in memory >>> even harder... >> >>I don't think it would make it harder. >> >>BTW, we should make it much harder to delete important principals... Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re[2]: MIT Kerberos Master principal deletion
> I don't think it would make it harder. I just mean because you won't be able to set a breakpoint at a function that uses the key, you'll have to actually chase it around in memory (assuming you use something like gcore to dump it as fast as possible without regard to where it is executing when it's dumped). If I was doing this live, I'd set a breakpoint on some function that used the key to decrypt and then inspect there, but with a core file you'll need to make sure you can find all the structures first. Is realm_mkey in the kdc_realm_data struct the one he wants? Chris -- Original Message -- From: "Nico Williams" To: "Chris Hecker" Cc: "Harshawardhan Kulkarni" ; "kerberos@mit.edu" Sent: 2020-06-11 15:31:28 Subject: Re: MIT Kerberos Master principal deletion >On Thu, Jun 11, 2020 at 10:19:39PM +, Chris Hecker wrote: >> Maybe dump the core of the running process so you don't accidentally crash >> it while trying to debug it live? But that would make finding it in memory >> even harder... > >I don't think it would make it harder. > >BTW, we should make it much harder to delete important principals... Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: MIT Kerberos Master principal deletion
On Thu, Jun 11, 2020 at 10:19:39PM +, Chris Hecker wrote: > Maybe dump the core of the running process so you don't accidentally crash > it while trying to debug it live? But that would make finding it in memory > even harder... I don't think it would make it harder. BTW, we should make it much harder to delete important principals... Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re[2]: MIT Kerberos Master principal deletion
Maybe dump the core of the running process so you don't accidentally crash it while trying to debug it live? But that would make finding it in memory even harder... Chris -- Original Message -- From: "Nico Williams" To: "Harshawardhan Kulkarni" Cc: "kerberos@mit.edu" Sent: 2020-06-11 15:05:19 Subject: Re: MIT Kerberos Master principal deletion >On Thu, Jun 11, 2020 at 03:32:35AM +0100, Harshawardhan Kulkarni wrote: >> I basically need an advice on an ongoing issue I am currently stuck on. >> >> We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of >> the nodes. We don't have a failover node for KDC server yet. On the KDC >> admin server while doing a clean up activity for unwanted kdc principals, I >> deleted the master key principal (K/m...@realm.com) We never took a kdc dump >> of the master key. So we don't have a backup to restore from. >> >> Is there any way I can restore the master key principal? > >If you have a running KDC you could use a debugger to recover that key. >It won't be easy. It's not something anyone does on a regular basis, so >I don't have instructions to give you. > >> I have tried creating with kdb5_util add_mkey but the error says that KDC >> DB is not able to find a master key credential. I assume this would only >> work when you want to create another master key without deleting the >> primary key. > >Adding a new key won't help you: the existing records are encrypted in >the old key. > >> Another option for me would be to de-kerberise the cluster and create the >> same REALM and kerberise the cluster again. But there could be serious >> issues if this doesn't fix as this is a live cluster where people are using >> this on a daily basis. > >You could rebuild your realm, yes. That's a flag day. Users in that >realm will need to be re-enrolled, keytabs will need to be re-created >and distributed... > >Nico >-- > >Kerberos mailing list Kerberos@mit.edu >https://mailman.mit.edu/mailman/listinfo/kerberos Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: MIT Kerberos Master principal deletion
On Thu, Jun 11, 2020 at 03:32:35AM +0100, Harshawardhan Kulkarni wrote: > I basically need an advice on an ongoing issue I am currently stuck on. > > We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of > the nodes. We don't have a failover node for KDC server yet. On the KDC > admin server while doing a clean up activity for unwanted kdc principals, I > deleted the master key principal (K/m...@realm.com) We never took a kdc dump > of the master key. So we don't have a backup to restore from. > > Is there any way I can restore the master key principal? If you have a running KDC you could use a debugger to recover that key. It won't be easy. It's not something anyone does on a regular basis, so I don't have instructions to give you. > I have tried creating with kdb5_util add_mkey but the error says that KDC > DB is not able to find a master key credential. I assume this would only > work when you want to create another master key without deleting the > primary key. Adding a new key won't help you: the existing records are encrypted in the old key. > Another option for me would be to de-kerberise the cluster and create the > same REALM and kerberise the cluster again. But there could be serious > issues if this doesn't fix as this is a live cluster where people are using > this on a daily basis. You could rebuild your realm, yes. That's a flag day. Users in that realm will need to be re-enrolled, keytabs will need to be re-created and distributed... Nico -- Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
Re: MIT Kerberos Master principal deletion
On 6/10/20 10:32 PM, Harshawardhan Kulkarni wrote: > We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of > the nodes. We don't have a failover node for KDC server yet. On the KDC > admin server while doing a clean up activity for unwanted kdc principals, I > deleted the master key principal (K/m...@realm.com) We never took a kdc dump > of the master key. So we don't have a backup to restore from. > > Is there any way I can restore the master key principal? Unfortunately, it doesn't look like our tools provide any good recovery options for this case, so I think you're stuck recreating the Kerberos database. I will file a ticket that it shouldn't be possible to delete the K/M principal entry. Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
MIT Kerberos Master principal deletion
Hi Team, I basically need an advice on an ongoing issue I am currently stuck on. We have a Kerberised Hadoop Cloudera Custer. KDC Admin server is on one of the nodes. We don't have a failover node for KDC server yet. On the KDC admin server while doing a clean up activity for unwanted kdc principals, I deleted the master key principal (K/m...@realm.com) We never took a kdc dump of the master key. So we don't have a backup to restore from. Is there any way I can restore the master key principal? I have tried creating with kdb5_util add_mkey but the error says that KDC DB is not able to find a master key credential. I assume this would only work when you want to create another master key without deleting the primary key. Another option for me would be to de-kerberise the cluster and create the same REALM and kerberise the cluster again. But there could be serious issues if this doesn't fix as this is a live cluster where people are using this on a daily basis. Can anyone help me here? Looking forward for your reply. Thanks, Harsh Kulkarni Kerberos mailing list Kerberos@mit.edu https://mailman.mit.edu/mailman/listinfo/kerberos
