Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 cc'ing group list back in for other opinions. On 05/22/2012 03:38 PM, Rich Megginson wrote: On 05/22/2012 08:36 AM, Dmitri Pal wrote: On 05/22/2012 10:10 AM, Rich Megginson wrote: On 05/22/2012 04:38 AM, Dmitri Pal wrote: On 05/22/2012 04:28 AM, Dale Macartney wrote: Dmitri, Rob I thought I might reply to you both directly, just in case others on the list vent frustrations on the ongoing discussion of this topic. I've been reading through the archives of the list for hot backup solutions, and this email thread really stood out. I am seeing a general consensus of backing up everything, and in some cases, even backing up a virtualized guest disk image to maintain a backup. I personally feel this is the wrong message people should be getting into their heads about a DR solution for restoring IPA. I was wondering, and feel free to correct me here if you see fit, if it would be beneficial to have a similar method of backing up IPA (and replicas), in a similar fashion to how Microsoft recommend their Domain Controllers to be backed up. A system state backup of sorts. Where a backup is performed on all Domain Controllers (or in our case, IPA servers). Basically, resulting in an individual restore point for each replica. From here, you have an entire backup, which will only ever bee used for that ONE server it was intended for. Essentially a complete dump and load approach. It is best practice in a Windows environment to perform these backups several times a week in small non-changing environments. So my thinking is, if we have a daily backup solution which could be used to run on each replica or master, then this should suffice in an adequate procedure to give to customers. In short, I'm more than happy to put my hand up on this one to help free up your time. I can easily take this on with a few of the lads here in the UK and get some customer feed back from mates within my former employment who are quite well versed in the realms of IPA. Would this be of any help to you? Do you see this as the right direction to take on this matter? I'd love to hear your thoughts Rhys, Gav, cc'ing you in on this one. I'd like to throw this onto our running list of IPA integration projects. Regards Dale First of all thank you for the offer! It seems that there are two main use cases: 1) Catastrophic failure 2) Data deletion In the case of the catastrophic failure you want to have all data+configuration+keys backed up to be able to effectively start over and re-install/recover from the backup. could we not have the ability of restoring only specific data? Like most backup solutions? for example having a utility where you can run ipa backup all could cover the data+config+keys, however depending on a catastrophic failure or data deletion, maybe have something along the lines of ipa restore data if we simply wanted to restore the data element of the backup. Thoughts? IMO, i think we should look for a KISS method which is specific to the application stack at hand. In this case IMO having a VM approach like the one JR uses is a viable solution. Rob, Simo, Rich do you agree? We would need to test this to make sure VM snapshots don't cause problems with replication and/or kerberos since those are sensitive to time. All of the testing we have ever done for RHDS/389 for backup/restore is based on simple database binary and ldif backups. We've never had to take into account restoring to a filesystem time in the past or a VM state that is in the past. Why in the past? If you take snapshots regularly say every other day when you restart a VM it should act as if the connection to it was lost for couple days. Ok. Maybe it will work just fine. All I'm saying is that we better test this well with a number of replication scenarios. Should we really be considering snapshots as an IPA method of recovery? That puts a prerequisite on the customer using either SAN snapshotting or virtualization technologies. If I use RHEV 2.2 as an example here, RHEV was not adopted by many customers because there was a prerequisite of Windows. If we have a reliance on other technologies for such key recovery principles this will undoubtedly (possibly more in the short than long term) have a knock on effect of adoption. Yes it might work, but should we be sending this message across? In the short term, if it works then great, however I think this should give us insight into a more robust method in the future. I do not see how the Kerberos and time are related here. In the case of data deletion we one needs to keep LDAP data around and not necessarily all the configuration and keys. And not even all the data needs to be backed up. So there should probably be two different procedures. I also think that creating a VM snapshot and recovering from such snapshot can be automated. We should probably provide some kind of script or command to do
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
Dale Macartney wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 cc'ing group list back in for other opinions. On 05/22/2012 03:38 PM, Rich Megginson wrote: On 05/22/2012 08:36 AM, Dmitri Pal wrote: On 05/22/2012 10:10 AM, Rich Megginson wrote: On 05/22/2012 04:38 AM, Dmitri Pal wrote: On 05/22/2012 04:28 AM, Dale Macartney wrote: Dmitri, Rob I thought I might reply to you both directly, just in case others on the list vent frustrations on the ongoing discussion of this topic. I've been reading through the archives of the list for hot backup solutions, and this email thread really stood out. I am seeing a general consensus of backing up everything, and in some cases, even backing up a virtualized guest disk image to maintain a backup. I personally feel this is the wrong message people should be getting into their heads about a DR solution for restoring IPA. I was wondering, and feel free to correct me here if you see fit, if it would be beneficial to have a similar method of backing up IPA (and replicas), in a similar fashion to how Microsoft recommend their Domain Controllers to be backed up. A system state backup of sorts. Where a backup is performed on all Domain Controllers (or in our case, IPA servers). Basically, resulting in an individual restore point for each replica. From here, you have an entire backup, which will only ever bee used for that ONE server it was intended for. Essentially a complete dump and load approach. It is best practice in a Windows environment to perform these backups several times a week in small non-changing environments. So my thinking is, if we have a daily backup solution which could be used to run on each replica or master, then this should suffice in an adequate procedure to give to customers. In short, I'm more than happy to put my hand up on this one to help free up your time. I can easily take this on with a few of the lads here in the UK and get some customer feed back from mates within my former employment who are quite well versed in the realms of IPA. Would this be of any help to you? Do you see this as the right direction to take on this matter? I'd love to hear your thoughts Rhys, Gav, cc'ing you in on this one. I'd like to throw this onto our running list of IPA integration projects. Regards Dale First of all thank you for the offer! It seems that there are two main use cases: 1) Catastrophic failure 2) Data deletion In the case of the catastrophic failure you want to have all data+configuration+keys backed up to be able to effectively start over and re-install/recover from the backup. could we not have the ability of restoring only specific data? Like most backup solutions? for example having a utility where you can run ipa backup all could cover the data+config+keys, however depending on a catastrophic failure or data deletion, maybe have something along the lines of ipa restore data if we simply wanted to restore the data element of the backup. Thoughts? IMO, i think we should look for a KISS method which is specific to the application stack at hand. Yes, this is the approach we're taking, as they are really separate problems. We aren't too keen on trying to back up individual files becuase IPA touches so many things, in so many different configurations, that the possibility that some important file is missed is rather high. The impact could mean a restored server that doesn't work. At this point we're recommending full backups and restores for system-level backups. As for data we're still working on it. Being able to identify and restore certain users/groups, etc is something we want but haven't yet worked out the details. In this case IMO having a VM approach like the one JR uses is a viable solution. Rob, Simo, Rich do you agree? We would need to test this to make sure VM snapshots don't cause problems with replication and/or kerberos since those are sensitive to time. All of the testing we have ever done for RHDS/389 for backup/restore is based on simple database binary and ldif backups. We've never had to take into account restoring to a filesystem time in the past or a VM state that is in the past. Why in the past? If you take snapshots regularly say every other day when you restart a VM it should act as if the connection to it was lost for couple days. Ok. Maybe it will work just fine. All I'm saying is that we better test this well with a number of replication scenarios. Should we really be considering snapshots as an IPA method of recovery? That puts a prerequisite on the customer using either SAN snapshotting or virtualization technologies. If I use RHEV 2.2 as an example here, RHEV was not adopted by many customers because there was a prerequisite of Windows. If we have a reliance on other technologies for such key recovery principles this will undoubtedly (possibly more in the short than long term) have a knock on effect of adoption. Yes it might work, but should we be sending this message
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
help: How to restore IPA Master/Replicas from daily IPA Replica setup??? Robinson Tiemuqinke wrote: Hi Dmitri, Rich and all, I am a newbie to Redhat IPA, It looks like pretty cool compared with other solutions I've tried before. Thanks a lot for this great product! :) But there are still some things I needs your help. My main question is: How to restore the IPA setup with a daily machine-level IPA Replica backup? Please let me explain my IPA setup background and backup/restore goals trying to reach: I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup with Dogtag CA system. It is installed first. Then two IPA replicas are installed -- with '--setup-ca' options -- for load balancing and failover purposes. To describe my problems/objectives, I'll name the IPA Master as machine A, IPA replicas as B and C. and now I've one more extra IPA replica 'D' (virtual machine) setup ONLY for backup purposes. The setup looks like the following, A is the configuration Hub. B,C,D are siblings. A / | \ B C D The following are the steps I backup IPA setups and LDAP backends daily -- it is a whole machine-level backup (through virtual machine D). 1, First, IPA replica D is backed up daily. The backup happens like this: 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h D'. On the Hypervisor which holds virtual machine D, do a daily backup of the whole virtual disk that D is on. 1.2 turn on the IP replica D again. 1.3 after virtual machine D is up, on D optionally run a 'ipa-replica-manage --force-sync --from A' to sync the IPA databases forcibly. Now comes to restore part, which is pretty confusing to me. I've tried several times, and every times it comes this or that kinds of issues and so I am wondering that correct steps/ineraction of IPA Master/replicas are the king :( 2, case #1, A is broken, like disc failure, and then re-imaged after several days. 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the daily backup from IPA replica D? The first thing you'll need to do is to connect your other replias together, either by picking a new hub or adding links to each one. Then you'll need to delete the replication agreement to A. You should be left with a set of servers that continues to replicate. So, for arguments sake, we promote B to be the new hub: On B: # ipa-replica-manage connect C # ipa-replica-manage connect D # ipa-replica-manage del --force A # ipactl restart On C: # ipa-replica-manage del --force A # ipactl restart On D: # ipa-replica-manage del --force A # ipactl restart It is unclear what you mean by re-imaged. Are you restoring from backup or installing it fresh? I'll assume it is a new install. You'll need to prepare a replica file for A and install it as a replica. Then if you want to keep A as the primary you'll need to change the replication agreements back to it is the hub (using ipa-replica-manage connect and disconnect). When you install the new A server it should get all the changes needed, you should be done. You'll want to check the documentation on promoting a master to verify that only one server is the CRL generator (at this point there may be none). 2.2 do I have to check some files on A into subversion immediately after A was initially installed? The only thing you really need to save is the cacert.p12 file. This is your root CA. 2.3 Please describe the steps. I'll follow exactly and report the results. 3, case #2, A is working, but either B, or C is broken. 3.1 It looks that I don't need the daily backup of D to kick in, is that right? No, D is unrelated. 3.2 What are the correct steps on A; and B after it is re-imaged? On A: # ipa-replica-manage del B # ipactl restart # ipa-replica-prepare B On B # ipa-replica-install B You'll probably need/want to clean RUV, http://directory.fedoraproject.org/wiki/Howto:CLEANRUV 3.3 Please describe the steps. I'll follow exactly and report the results. 4, case #3, If some un-expected IPA changes happens on A -- like all users are deleted by human mistakes --, and even worse, all the changes are propagated to B and C in minutes. 4.1 How can I recover the IPA setup from daily backup from D? We have not yet documented how to recover from tombstones or an offline replica. 4.2 which IPA master/replicas I should recover first? IPA master A, or IPA replicas B/C? and then how to recover others left one by one? If the entries are re-added on any of the replicas it will be propogated out. 4.3 Do I have to disconnect replication agreement of B,C,D from A first? Depends on how 4.1 gets answered which we are still investigating. 4.4 Please describe the steps. I'll follow exactly and report the results. I've heard something about tombstone records too, Not sure whether the problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I avoid
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
Hi Rob, Just wonder whether your guys have abandoned IPA 2.1.3 users on Redhat 6.2 or not. :( The IPA replication/restoration procedure/document request has been submitted for more than a week, but I can not see any meaningful work has done for customers although IPA replication and restoration is so vital to users' production IPA reliability! Even when after I've done a lot of investigation work and asking for helps/suggestions, there is still no much attentions paid from you guys. Am I, or any others users here, are just non-paid Q/A IPA team stuff could be ignored for no reasons :) I've mentioned this again and again, and urging IPA team to setup a typical user setup, because only this way you can see what the problems IPA administrators/users are facing and scared of. But unfortunately, we don't have a feeling that you have done so. Thanks. --Gelen From: Gelen James hahaha_...@yahoo.com To: Rob Crittenden rcrit...@redhat.com; Dmitri Pal d...@redhat.com Cc: Freeipa-users@redhat.com Freeipa-users@redhat.com Sent: Sunday, May 20, 2012 12:08 AM Subject: Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup??? Hi Mmitri, Rob and all. Thanks for your instructions. I've performed your steps on case#1: replacing failed IPA master. The results, and my confusion and questions, are all detailed below. In general, please setup your own real test environment, and write down the detailed steps one by one clearly. It took me more than one week and still no clues. Frankly, your steps in the formal email are kind of over-simplified for normal IPA users, and not covering how the CA LDAP backend will be handled. The problem is the CA backend. All the replicas still trying to sync to old failed IPA master, even after reboot. Could be that the 'ipa-replica-manage' only manages the user data replication? and 'ipa-csreplica-manage' only handles CA-end replication? In other words, when build, or tear down, IPA replication between two servers, do we need to deal with both replication types with 'ipa-replica-mange' AND 'ipa-csreplica-manage'? If so, then why who should run first? The error messages in /var/log/dirsrv/slapd-PKI-IPA/errors are attached, same from B,C,D replicas. [19/May/2012:19:40:48 -0700] - 389-Directory/1.2.9.16 B2012.023.214 starting up [19/May/2012:19:40:48 -0700] - slapd started. Listening on All Interfaces port 7389 for LDAP requests [19/May/2012:19:40:48 -0700] - Listening on All Interfaces port 7390 for LDAPS requests [19/May/2012:19:40:50 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:40:50 -0700] NSMMReplicationPlugin - agmt=cn=cloneAgreement1-B.example.com-pki-ca (A:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [19/May/2012:19:40:57 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:41:03 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:41:15 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:41:39 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:42:27 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:44:03 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:47:15 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [root@B ~]# After seeing the above messages, I tried to run similar commands for CA replication, it shows that replication agreement (which replication agreement? User data, or CA data ?? ) exists already. on B, ipa-csreplica-manage connect C ipa-csreplica-manage connect D ipa-csreplica-manage del A --force ipactl restart on C, ipa-csreplica-manage del A --force ipactl restart on D, ipa-csreplica-manage del A --force ipactl restart [root@B ~]# ipa-csreplica-manage --password=xxx connect C.example.com This replication agreement already exists. [root@B ~]# [root@B ~]# ipa-csreplica-manage --password=xxx connect D.example.com This replication agreement already exists. [root@B ~]# [root@B ~]# ipa-csreplica-manage --password=xxx del C.example.com --force Unable to connect to replica A.example.com, forcing removal Failed to get data from 'A.example.com': Can't contact LDAP server Forcing removal on 'B.example.com' [root@B ~]# After restarting IPA services on B, C, D, and now the error messages finally got away from CA errors log file. But we still can not find the CA replication setups. Please see the difference of output from 'ipa-replica
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
On 05/21/2012 01:25 PM, Gelen James wrote: Hi Rob, Just wonder whether your guys have abandoned IPA 2.1.3 users on Redhat 6.2 or not. :( The IPA replication/restoration procedure/document request has been submitted for more than a week, but I can not see any meaningful work has done for customers although IPA replication and restoration is so vital to users' production IPA reliability! Even when after I've done a lot of investigation work and asking for helps/suggestions, there is still no much attentions paid from you guys. Am I, or any others users here, are just non-paid Q/A IPA team stuff could be ignored for no reasons :) I've mentioned this again and again, and urging IPA team to setup a typical user setup, because only this way you can see what the problems IPA administrators/users are facing and scared of. But unfortunately, we don't have a feeling that you have done so. Thanks. --Gelen Hello Glen, We have not done so because we are pretty busy preparing next release and were hoping that our replies were sufficient to help you to figure out the best procedure that works for you. JR has a running environment so his guidance is first hand. We tried to provide as much help as we can. We also have not been going the path of setting the environment because we are not sure what is your typical environment and what are the main concerns. Your input is very valuable but it is one of the first clearly spelled data points. We need to get a bit more info to make sure that we are addressing the right use case and problem. We apologize for the delays but the turn around would not be fast. It will take us at least several weeks to come up with something we are comfortable with upstream and downstream. I hear your frustration and feel the urgency but we can't move faster than we can, sorry. Please do not feel abandoned we are working hard too. Also it seems that setting the environment and crafting the guidelines should also be combined with attempt to automate the process. I already contacted Foreman project developers in attempt to integrate the replica provisioning for scalability and disaster recovery cases. We will have a conversation with them later this week. This might help with doing automatic provisioning of replicas rather than manually performing couple dozen of steps. Would such integration help? Also if you need some immediate help opening a support ticket might be a better avenue to get the situation prioritized accordingly. Sorry for delays, Thanks Dmitri *From:* Gelen James hahaha_...@yahoo.com *To:* Rob Crittenden rcrit...@redhat.com; Dmitri Pal d...@redhat.com *Cc:* Freeipa-users@redhat.com Freeipa-users@redhat.com *Sent:* Sunday, May 20, 2012 12:08 AM *Subject:* Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup??? Hi Mmitri, Rob and all. Thanks for your instructions. I've performed your steps on case#1: replacing failed IPA master. The results, and my confusion and questions, are all detailed below. In general, please setup your own real test environment, and write down the detailed steps one by one clearly. It took me more than one week and still no clues. Frankly, your steps in the formal email are kind of over-simplified for normal IPA users, and not covering how the CA LDAP backend will be handled. The problem is the CA backend. All the replicas still trying to sync to old failed IPA master, even after reboot. Could be that the 'ipa-replica-manage' only manages the user data replication? and 'ipa-csreplica-manage' only handles CA-end replication? In other words, when build, or tear down, IPA replication between two servers, do we need to deal with both replication types with 'ipa-replica-mange' AND 'ipa-csreplica-manage'? If so, then why who should run first? The error messages in /var/log/dirsrv/slapd-PKI-IPA/errors are attached, same from B,C,D replicas. [19/May/2012:19:40:48 -0700] - 389-Directory/1.2.9.16 B2012.023.214 starting up [19/May/2012:19:40:48 -0700] - slapd started. Listening on All Interfaces port 7389 for LDAP requests [19/May/2012:19:40:48 -0700] - Listening on All Interfaces port 7390 for LDAPS requests [19/May/2012:19:40:50 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:40:50 -0700] NSMMReplicationPlugin - agmt=cn=cloneAgreement1-B.example.com-pki-ca (A:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [19/May/2012:19:40:57 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:41:03 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:41:15 -0700] slapi_ldap_bind - Error: could not send startTLS request
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
Hi Mmitri, Rob and all. Thanks for your instructions. I've performed your steps on case#1: replacing failed IPA master. The results, and my confusion and questions, are all detailed below. In general, please setup your own real test environment, and write down the detailed steps one by one clearly. It took me more than one week and still no clues. Frankly, your steps in the formal email are kind of over-simplified for normal IPA users, and not covering how the CA LDAP backend will be handled. The problem is the CA backend. All the replicas still trying to sync to old failed IPA master, even after reboot. Could be that the 'ipa-replica-manage' only manages the user data replication? and 'ipa-csreplica-manage' only handles CA-end replication? In other words, when build, or tear down, IPA replication between two servers, do we need to deal with both replication types with 'ipa-replica-mange' AND 'ipa-csreplica-manage'? If so, then why who should run first? The error messages in /var/log/dirsrv/slapd-PKI-IPA/errors are attached, same from B,C,D replicas. [19/May/2012:19:40:48 -0700] - 389-Directory/1.2.9.16 B2012.023.214 starting up [19/May/2012:19:40:48 -0700] - slapd started. Listening on All Interfaces port 7389 for LDAP requests [19/May/2012:19:40:48 -0700] - Listening on All Interfaces port 7390 for LDAPS requests [19/May/2012:19:40:50 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:40:50 -0700] NSMMReplicationPlugin - agmt=cn=cloneAgreement1-B.example.com-pki-ca (A:7389): Replication bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null)) [19/May/2012:19:40:57 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:41:03 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:41:15 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:41:39 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:42:27 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:44:03 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [19/May/2012:19:47:15 -0700] slapi_ldap_bind - Error: could not send startTLS request: error -1 (Can't contact LDAP server) [root@B ~]# After seeing the above messages, I tried to run similar commands for CA replication, it shows that replication agreement (which replication agreement? User data, or CA data ?? ) exists already. on B, ipa-csreplica-manage connect C ipa-csreplica-manage connect D ipa-csreplica-manage del A --force ipactl restart on C, ipa-csreplica-manage del A --force ipactl restart on D, ipa-csreplica-manage del A --force ipactl restart [root@B ~]# ipa-csreplica-manage --password=xxx connect C.example.com This replication agreement already exists. [root@B ~]# [root@B ~]# ipa-csreplica-manage --password=xxx connect D.example.com This replication agreement already exists. [root@B ~]# [root@B ~]# ipa-csreplica-manage --password=xxx del C.example.com --force Unable to connect to replica A.example.com, forcing removal Failed to get data from 'A.example.com': Can't contact LDAP server Forcing removal on 'B.example.com' [root@B ~]# After restarting IPA services on B, C, D, and now the error messages finally got away from CA errors log file. But we still can not find the CA replication setups. Please see the difference of output from 'ipa-replica-manage' and 'ipa-csreplica-manage': [root@B ~] ipa-replica-manage list B.example.com C.example.com D.example.com [root@B ~] ipa-csreplica-manage list B.example.com C.example.com D.example.com [root@B ~] ipa-replica-manage list B.example.com C.example.com D.example.com [root@B ~] ipa-csreplica-manage list B.example.com ## Nothing at all! Please have a check and give correct command and sequences for us IPA users. It is such a pain to spend so much time and still can not get restoration work as expected. Even worse is, Have no idea how the 'ipa-replica-manage' and 'ipa-csreplica-manage' work together behind the scene. Thanks a lot. --Gelen From: Rob Crittenden rcrit...@redhat.com To: Robinson Tiemuqinke hahaha_...@yahoo.com Cc: Freeipa-users@redhat.com Freeipa-users@redhat.com; Rich Megginson rmegg...@redhat.com; Dmitri Pal d...@redhat.com Sent: Tuesday, May 15, 2012 9:57 AM Subject: Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup??? Robinson Tiemuqinke wrote: Hi Dmitri, Rich and all, I am a newbie to Redhat IPA, It looks like pretty cool compared with other solutions I've tried before. Thanks a lot
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
rebuild the old IPA master A is half success too. The error also happens at CA
replication side.
After replica preparation at replica B, nuke and reinstall old A, and create A
from the replica info file prepared on B, The user LDAP replication works fine.
while the CA replication broken terribly. the error messages on A inside file
/var/log/dirsrv/slapd-PKI-IPA/errors are pasted below:
[20/May/2012:01:17:36 -0700] - 389-Directory/1.2.9.16 B2012.023.214 starting up
[20/May/2012:01:17:36 -0700] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: data for replica o=ipaca does not match
the data in the changelog (replica data (4fb8a7f300040443) changelog
(4fb84ba70056)). Recreating the changelog file. This could affect
replication with replica's consumers in which case the consumers should be
reinitialized.
[20/May/2012:01:17:37 -0700] - slapd started. Listening on All Interfaces port
7389 for LDAP requests
[20/May/2012:01:17:37 -0700] - Listening on All Interfaces port 7390 for LDAPS
requests
[root@A ~]#
check the RUV records shows a number too big: 1091, while all others are
smaller than 100. There are no RUV records to delete/clear.
dn: nsuniqueid=---,o=ipaca
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 4fb8187f0060
nsds50ruv: {replica 97 ldap://B.example.com:7389} 4fb8188600
61 4fb8a7ca00010061
nsds50ruv: {replica 1091 ldap://A.example.com:7389} 4fb8a7c60001044
3 4fb8a8a900010443
nsds50ruv: {replica 91 ldap://C.example.com:7389} 4fb81f5400
5b 4fb84db6005b
nsds50ruv: {replica 86 ldap://D.example.com:7389} 4fb821a600
56 4fb84ba70056
o: ipaca
nsruvReplicaLastModified: {replica 97 ldap://B.example.com:7389}
4fb8a7c7
nsruvReplicaLastModified: {replica 1091 ldap://A.example.com:7389}
4fb8a8a6
nsruvReplicaLastModified: {replica 91 ldap://C.example.com:7389}
nsruvReplicaLastModified: {replica 86 ldap://D.example.com:7389}
Please advise. Thanks.
--Gelen
From: Gelen James hahaha_...@yahoo.com
To: Rob Crittenden rcrit...@redhat.com; Dmitri Pal d...@redhat.com
Cc: Freeipa-users@redhat.com Freeipa-users@redhat.com
Sent: Sunday, May 20, 2012 12:08 AM
Subject: Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas
from daily IPA Replica setup???
Hi Mmitri, Rob and all.
Thanks for your instructions. I've performed your steps on case#1: replacing
failed IPA master. The results, and my confusion and questions, are all
detailed below. In general, please setup your own real test environment, and
write down the detailed steps one by one clearly.
It took me more than one week and still no clues. Frankly, your steps in the
formal email are kind of over-simplified for normal IPA users, and not covering
how the CA LDAP backend will be handled.
The problem is the CA backend. All the replicas still trying to sync to old
failed IPA master, even after reboot.
Could be that the 'ipa-replica-manage' only manages the user data replication?
and 'ipa-csreplica-manage' only handles CA-end replication? In other words,
when build, or tear down, IPA replication between two servers, do we need to
deal with both replication types with 'ipa-replica-mange' AND
'ipa-csreplica-manage'? If so, then why who should run first?
The error messages in /var/log/dirsrv/slapd-PKI-IPA/errors are attached, same
from B,C,D replicas.
[19/May/2012:19:40:48 -0700] - 389-Directory/1.2.9.16 B2012.023.214 starting up
[19/May/2012:19:40:48 -0700] - slapd started. Listening on All Interfaces port
7389 for LDAP requests
[19/May/2012:19:40:48 -0700] - Listening on All Interfaces port 7390 for LDAPS
requests
[19/May/2012:19:40:50 -0700] slapi_ldap_bind - Error: could not send startTLS
request: error -1 (Can't contact LDAP server)
[19/May/2012:19:40:50 -0700] NSMMReplicationPlugin -
agmt=cn=cloneAgreement1-B.example.com-pki-ca (A:7389): Replication bind
with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP server) ((null))
[19/May/2012:19:40:57 -0700] slapi_ldap_bind - Error: could not send startTLS
request: error -1 (Can't contact LDAP server)
[19/May/2012:19:41:03 -0700] slapi_ldap_bind - Error: could not send startTLS
request: error -1 (Can't contact LDAP server)
[19/May/2012:19:41:15 -0700] slapi_ldap_bind - Error: could not send startTLS
request: error -1 (Can't contact LDAP server)
[19/May/2012:19:41:39 -0700] slapi_ldap_bind - Error: could not send startTLS
request: error -1 (Can't contact LDAP server)
[19/May/2012:19:42:27 -0700] slapi_ldap_bind - Error: could not send startTLS
request: error -1 (Can't contact LDAP server)
[19/May/2012:19:44:03 -0700] slapi_ldap_bind - Error: could not send startTLS
request: error -1 (Can't contact LDAP server)
[19/May/2012:19:47:15 -0700] slapi_ldap_bind - Error: could not send startTLS
request: error -1
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
On 05/20/2012 02:28 AM, Gelen James wrote:
rebuild the old IPA master A is half success too. The error also
happens at CA replication side.
After replica preparation at replica B, nuke and reinstall old A, and
create A from the replica info file prepared on B, The user LDAP
replication works fine. while the CA replication broken terribly. the
error messages on A inside file /var/log/dirsrv/slapd-PKI-IPA/errors
are pasted below:
[20/May/2012:01:17:36 -0700] - 389-Directory/1.2.9.16 B2012.023.214
starting up
[20/May/2012:01:17:36 -0700] NSMMReplicationPlugin -
replica_check_for_data_reload: Warning: data for replica o=ipaca does
not match the data in the changelog (replica data
(4fb8a7f300040443) changelog (4fb84ba70056)). Recreating
the changelog file. This could affect replication with replica's
consumers in which case the consumers should be reinitialized.
This error message is normal - you should only see this once, just
after a replica has been initialized.
[20/May/2012:01:17:37 -0700] - slapd started. Listening on All
Interfaces port 7389 for LDAP requests
[20/May/2012:01:17:37 -0700] - Listening on All Interfaces port 7390
for LDAPS requests
[root@A ~]#
check the RUV records shows a number too big: 1091, while all others
are smaller than 100.
It's not too big as far as the protocol is concerned, but it is
strange that it is so much larger than the other values.
There are no RUV records to delete/clear.
dn: nsuniqueid=---,o=ipaca
objectClass: top
objectClass: nsTombstone
objectClass: extensibleobject
nsds50ruv: {replicageneration} 4fb8187f0060
nsds50ruv: {replica 97 ldap://B.example.com:7389} 4fb8188600
61 4fb8a7ca00010061
nsds50ruv: {replica 1091 ldap://A.example.com:7389} 4fb8a7c60001044
3 4fb8a8a900010443
nsds50ruv: {replica 91 ldap://C.example.com:7389} 4fb81f5400
5b 4fb84db6005b
nsds50ruv: {replica 86 ldap://D.example.com:7389} 4fb821a600
56 4fb84ba70056
o: ipaca
nsruvReplicaLastModified: {replica 97 ldap://B.example.com:7389}
4fb8a7c7
nsruvReplicaLastModified: {replica 1091 ldap://A.example.com:7389}
4fb8a8a6
nsruvReplicaLastModified: {replica 91 ldap://C.example.com:7389}
nsruvReplicaLastModified: {replica 86 ldap://D.example.com:7389}
Please advise. Thanks.
--Gelen
*From:* Gelen James hahaha_...@yahoo.com
*To:* Rob Crittenden rcrit...@redhat.com; Dmitri Pal d...@redhat.com
*Cc:* Freeipa-users@redhat.com Freeipa-users@redhat.com
*Sent:* Sunday, May 20, 2012 12:08 AM
*Subject:* Re: [Freeipa-users] Please help: How to restore IPA
Master/Replicas from daily IPA Replica setup???
Hi Mmitri, Rob and all.
Thanks for your instructions. I've performed your steps on case#1:
replacing failed IPA master. The results, and my confusion and
questions, are all detailed below. In general, please setup your own
real test environment, and write down the detailed steps one by one
clearly.
It took me more than one week and still no clues. Frankly, your steps
in the formal email are kind of over-simplified for normal IPA users,
and not covering how the CA LDAP backend will be handled.
The problem is the CA backend. All the replicas still trying to sync
to old failed IPA master, even after reboot.
Could be that the 'ipa-replica-manage' only manages the user data
replication? and 'ipa-csreplica-manage' only handles CA-end
replication? In other words, when build, or tear down, IPA replication
between two servers, do we need to deal with both replication types
with 'ipa-replica-mange' AND 'ipa-csreplica-manage'? If so, then why
who should run first?
The error messages in /var/log/dirsrv/slapd-PKI-IPA/errors are
attached, same from B,C,D replicas.
[19/May/2012:19:40:48 -0700] - 389-Directory/1.2.9.16 B2012.023.214
starting up
[19/May/2012:19:40:48 -0700] - slapd started. Listening on All
Interfaces port 7389 for LDAP requests
[19/May/2012:19:40:48 -0700] - Listening on All Interfaces port 7390
for LDAPS requests
[19/May/2012:19:40:50 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server)
[19/May/2012:19:40:50 -0700] NSMMReplicationPlugin -
agmt=cn=cloneAgreement1-B.example.com-pki-ca (A:7389): Replication
bind with SIMPLE auth failed: LDAP error -1 (Can't contact LDAP
server) ((null))
[19/May/2012:19:40:57 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server)
[19/May/2012:19:41:03 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server)
[19/May/2012:19:41:15 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server)
[19/May/2012:19:41:39 -0700] slapi_ldap_bind - Error: could not send
startTLS request: error -1 (Can't contact LDAP server)
[19/May/2012:19
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
Hello, IMHO it *must* be documented very well. Thank for scenario proposal! There is a new documentation ticket: https://fedorahosted.org/freeipa/ticket/2758 Another ticket exists for CA master recovery procedure: https://fedorahosted.org/freeipa/ticket/2749 Petr^2 Spacek On 05/15/2012 01:19 AM, Gelen James wrote: Hi Dimitri, thanks a lot for your offer. It will be more than appreciated if Rob, or some other talented genius could wiki the steps. The more details, the sooner, and the better. It will help IPA projects and its users dramatically, especially for newbies like me. :) Thanks again for you, Rob and others for the coming documentation work. --Gelen. -- *From:* Dmitri Pal d...@redhat.com *To:* Robinson Tiemuqinke hahaha_...@yahoo.com *Cc:* Freeipa-users@redhat.com Freeipa-users@redhat.com; Rich Megginson rmegg...@redhat.com *Sent:* Monday, May 14, 2012 1:20 PM *Subject:* Re: Please help: How to restore IPA Master/Replicas from daily IPA Replica setup??? On 05/14/2012 03:48 PM, Robinson Tiemuqinke wrote: Hi Dmitri, Rich and all, I am a newbie to Redhat IPA, It looks like pretty cool compared with other solutions I've tried before. Thanks a lot for this great product! :) But there are still some things I needs your help. My main question is: How to restore the IPA setup with a daily machine-level IPA Replica backup? Please let me explain my IPA setup background and backup/restore goals trying to reach: I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup with Dogtag CA system. It is installed first. Then two IPA replicas are installed -- with '--setup-ca' options -- for load balancing and failover purposes. To describe my problems/objectives, I'll name the IPA Master as machine A, IPA replicas as B and C. and now I've one more extra IPA replica 'D' (virtual machine) setup ONLY for backup purposes. The setup looks like the following, A is the configuration Hub. B,C,D are siblings. A / | \ B C D The following are the steps I backup IPA setups and LDAP backends daily -- it is a whole machine-level backup (through virtual machine D). 1, First, IPA replica D is backed up daily. The backup happens like this: 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h D'. On the Hypervisor which holds virtual machine D, do a daily backup of the whole virtual disk that D is on. 1.2 turn on the IP replica D again. 1.3 after virtual machine D is up, on D optionally run a 'ipa-replica-manage --force-sync --from A' to sync the IPA databases forcibly. Now comes to restore part, which is pretty confusing to me. I've tried several times, and every times it comes this or that kinds of issues and so I am wondering that correct steps/ineraction of IPA Master/replicas are the king :( 2, case #1, A is broken, like disc failure, and then re-imaged after several days. 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the daily backup from IPA replica D? 2.2 do I have to check some files on A into subversion immediately after A was initially installed? 2.3 Please describe the steps. I'll follow exactly and report the results. 3, case #2, A is working, but either B, or C is broken. 3.1 It looks that I don't need the daily backup of D to kick in, is that right? 3.2 What are the correct steps on A; and B after it is re-imaged? 3.3 Please describe the steps. I'll follow exactly and report the results. 4, case #3, If some un-expected IPA changes happens on A -- like all users are deleted by human mistakes --, and even worse, all the changes are propagated to B and C in minutes. 4.1 How can I recover the IPA setup from daily backup from D? 4.2 which IPA master/replicas I should recover first? IPA master A, or IPA replicas B/C? and then how to recover others left one by one? 4.3 Do I have to disconnect replication agreement of B,C,D from A first? 4.4 Please describe the steps. I'll follow exactly and report the results. I've heard something about tombstone records too, Not sure whether the problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I avoid it with correct recovery steps/interactions. Thanks a lot. --Gelen. I can explain it conceptually. Rob is probably best to define the exact sequence and commands. If you A is broken you reinstall it, make it connect to D and init (force sync) A from D. Now you have a new A. If B or C dies you just re-install B or C and init from A. If you lost a lot of data I suggest you start a saved D instance and force-sync A from it and then force sync B and C from A. -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ http://www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
I have successfully utilized a similar procedure. The restoration process is the same for both though. I would be willing to accept the tickets and document the various backup and recovery methods. Though, I'd like Dmitri's feedback on whether or not the team approves of making the official method of recovery from catastrophic failure be the use of frozen vm images. Keeping your head in the cloud ~ Jr Aquino | Sr. Information Security Specialist GIAC Certified Incident Handler | GIAC WebApp Penetration Tester Citrix Online | 7408 Hollister Avenue | Goleta, CA 93117 jr.aqu...@citrix.com http://www.citrixonline.com On May 15, 2012, at 2:16 AM, Petr Spacek pspa...@redhat.com wrote: Hello, IMHO it *must* be documented very well. Thank for scenario proposal! There is a new documentation ticket: https://fedorahosted.org/freeipa/ticket/2758 Another ticket exists for CA master recovery procedure: https://fedorahosted.org/freeipa/ticket/2749 Petr^2 Spacek On 05/15/2012 01:19 AM, Gelen James wrote: Hi Dimitri, thanks a lot for your offer. It will be more than appreciated if Rob, or some other talented genius could wiki the steps. The more details, the sooner, and the better. It will help IPA projects and its users dramatically, especially for newbies like me. :) Thanks again for you, Rob and others for the coming documentation work. --Gelen. -- *From:* Dmitri Pal d...@redhat.com *To:* Robinson Tiemuqinke hahaha_...@yahoo.com *Cc:* Freeipa-users@redhat.com Freeipa-users@redhat.com; Rich Megginson rmegg...@redhat.com *Sent:* Monday, May 14, 2012 1:20 PM *Subject:* Re: Please help: How to restore IPA Master/Replicas from daily IPA Replica setup??? On 05/14/2012 03:48 PM, Robinson Tiemuqinke wrote: Hi Dmitri, Rich and all, I am a newbie to Redhat IPA, It looks like pretty cool compared with other solutions I've tried before. Thanks a lot for this great product! :) But there are still some things I needs your help. My main question is: How to restore the IPA setup with a daily machine-level IPA Replica backup? Please let me explain my IPA setup background and backup/restore goals trying to reach: I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup with Dogtag CA system. It is installed first. Then two IPA replicas are installed -- with '--setup-ca' options -- for load balancing and failover purposes. To describe my problems/objectives, I'll name the IPA Master as machine A, IPA replicas as B and C. and now I've one more extra IPA replica 'D' (virtual machine) setup ONLY for backup purposes. The setup looks like the following, A is the configuration Hub. B,C,D are siblings. A / | \ B C D The following are the steps I backup IPA setups and LDAP backends daily -- it is a whole machine-level backup (through virtual machine D). 1, First, IPA replica D is backed up daily. The backup happens like this: 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h D'. On the Hypervisor which holds virtual machine D, do a daily backup of the whole virtual disk that D is on. 1.2 turn on the IP replica D again. 1.3 after virtual machine D is up, on D optionally run a 'ipa-replica-manage --force-sync --from A' to sync the IPA databases forcibly. Now comes to restore part, which is pretty confusing to me. I've tried several times, and every times it comes this or that kinds of issues and so I am wondering that correct steps/ineraction of IPA Master/replicas are the king :( 2, case #1, A is broken, like disc failure, and then re-imaged after several days. 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the daily backup from IPA replica D? 2.2 do I have to check some files on A into subversion immediately after A was initially installed? 2.3 Please describe the steps. I'll follow exactly and report the results. 3, case #2, A is working, but either B, or C is broken. 3.1 It looks that I don't need the daily backup of D to kick in, is that right? 3.2 What are the correct steps on A; and B after it is re-imaged? 3.3 Please describe the steps. I'll follow exactly and report the results. 4, case #3, If some un-expected IPA changes happens on A -- like all users are deleted by human mistakes --, and even worse, all the changes are propagated to B and C in minutes. 4.1 How can I recover the IPA setup from daily backup from D? 4.2 which IPA master/replicas I should recover first? IPA master A, or IPA replicas B/C? and then how to recover others left one by one? 4.3 Do I have to disconnect replication agreement of B,C,D from A first? 4.4 Please describe the steps. I'll follow exactly and report the results. I've heard something about tombstone records too, Not sure whether the problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
Robinson Tiemuqinke wrote: Hi Dmitri, Rich and all, I am a newbie to Redhat IPA, It looks like pretty cool compared with other solutions I've tried before. Thanks a lot for this great product! :) But there are still some things I needs your help. My main question is: How to restore the IPA setup with a daily machine-level IPA Replica backup? Please let me explain my IPA setup background and backup/restore goals trying to reach: I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup with Dogtag CA system. It is installed first. Then two IPA replicas are installed -- with '--setup-ca' options -- for load balancing and failover purposes. To describe my problems/objectives, I'll name the IPA Master as machine A, IPA replicas as B and C. and now I've one more extra IPA replica 'D' (virtual machine) setup ONLY for backup purposes. The setup looks like the following, A is the configuration Hub. B,C,D are siblings. A / | \ B C D The following are the steps I backup IPA setups and LDAP backends daily -- it is a whole machine-level backup (through virtual machine D). 1, First, IPA replica D is backed up daily. The backup happens like this: 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h D'. On the Hypervisor which holds virtual machine D, do a daily backup of the whole virtual disk that D is on. 1.2 turn on the IP replica D again. 1.3 after virtual machine D is up, on D optionally run a 'ipa-replica-manage --force-sync --from A' to sync the IPA databases forcibly. Now comes to restore part, which is pretty confusing to me. I've tried several times, and every times it comes this or that kinds of issues and so I am wondering that correct steps/ineraction of IPA Master/replicas are the king :( 2, case #1, A is broken, like disc failure, and then re-imaged after several days. 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the daily backup from IPA replica D? The first thing you'll need to do is to connect your other replias together, either by picking a new hub or adding links to each one. Then you'll need to delete the replication agreement to A. You should be left with a set of servers that continues to replicate. So, for arguments sake, we promote B to be the new hub: On B: # ipa-replica-manage connect C # ipa-replica-manage connect D # ipa-replica-manage del --force A # ipactl restart On C: # ipa-replica-manage del --force A # ipactl restart On D: # ipa-replica-manage del --force A # ipactl restart It is unclear what you mean by re-imaged. Are you restoring from backup or installing it fresh? I'll assume it is a new install. You'll need to prepare a replica file for A and install it as a replica. Then if you want to keep A as the primary you'll need to change the replication agreements back to it is the hub (using ipa-replica-manage connect and disconnect). When you install the new A server it should get all the changes needed, you should be done. You'll want to check the documentation on promoting a master to verify that only one server is the CRL generator (at this point there may be none). 2.2 do I have to check some files on A into subversion immediately after A was initially installed? The only thing you really need to save is the cacert.p12 file. This is your root CA. 2.3 Please describe the steps. I'll follow exactly and report the results. 3, case #2, A is working, but either B, or C is broken. 3.1 It looks that I don't need the daily backup of D to kick in, is that right? No, D is unrelated. 3.2 What are the correct steps on A; and B after it is re-imaged? On A: # ipa-replica-manage del B # ipactl restart # ipa-replica-prepare B On B # ipa-replica-install B You'll probably need/want to clean RUV, http://directory.fedoraproject.org/wiki/Howto:CLEANRUV 3.3 Please describe the steps. I'll follow exactly and report the results. 4, case #3, If some un-expected IPA changes happens on A -- like all users are deleted by human mistakes --, and even worse, all the changes are propagated to B and C in minutes. 4.1 How can I recover the IPA setup from daily backup from D? We have not yet documented how to recover from tombstones or an offline replica. 4.2 which IPA master/replicas I should recover first? IPA master A, or IPA replicas B/C? and then how to recover others left one by one? If the entries are re-added on any of the replicas it will be propogated out. 4.3 Do I have to disconnect replication agreement of B,C,D from A first? Depends on how 4.1 gets answered which we are still investigating. 4.4 Please describe the steps. I'll follow exactly and report the results. I've heard something about tombstone records too, Not sure whether the problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I avoid it with correct recovery steps/interactions. It is RUV that is the problem. This 389-ds wiki page describes how to clean up:
[Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
Hi Dmitri, Rich and all, I am a newbie to Redhat IPA, It looks like pretty cool compared with other solutions I've tried before. Thanks a lot for this great product! :) But there are still some things I needs your help. My main question is: How to restore the IPA setup with a daily machine-level IPA Replica backup? Please let me explain my IPA setup background and backup/restore goals trying to reach: I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup with Dogtag CA system. It is installed first. Then two IPA replicas are installed -- with '--setup-ca' options -- for load balancing and failover purposes. To describe my problems/objectives, I'll name the IPA Master as machine A, IPA replicas as B and C. and now I've one more extra IPA replica 'D' (virtual machine) setup ONLY for backup purposes. The setup looks like the following, A is the configuration Hub. B,C,D are siblings. A / | \ B C D The following are the steps I backup IPA setups and LDAP backends daily -- it is a whole machine-level backup (through virtual machine D). 1, First, IPA replica D is backed up daily. The backup happens like this: 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h D'. On the Hypervisor which holds virtual machine D, do a daily backup of the whole virtual disk that D is on. 1.2 turn on the IP replica D again. 1.3 after virtual machine D is up, on D optionally run a 'ipa-replica-manage --force-sync --from A' to sync the IPA databases forcibly. Now comes to restore part, which is pretty confusing to me. I've tried several times, and every times it comes this or that kinds of issues and so I am wondering that correct steps/ineraction of IPA Master/replicas are the king :( 2, case #1, A is broken, like disc failure, and then re-imaged after several days. 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the daily backup from IPA replica D? 2.2 do I have to check some files on A into subversion immediately after A was initially installed? 2.3 Please describe the steps. I'll follow exactly and report the results. 3, case #2, A is working, but either B, or C is broken. 3.1 It looks that I don't need the daily backup of D to kick in, is that right? 3.2 What are the correct steps on A; and B after it is re-imaged? 3.3 Please describe the steps. I'll follow exactly and report the results. 4, case #3, If some un-expected IPA changes happens on A -- like all users are deleted by human mistakes --, and even worse, all the changes are propagated to B and C in minutes. 4.1 How can I recover the IPA setup from daily backup from D? 4.2 which IPA master/replicas I should recover first? IPA master A, or IPA replicas B/C? and then how to recover others left one by one? 4.3 Do I have to disconnect replication agreement of B,C,D from A first? 4.4 Please describe the steps. I'll follow exactly and report the results. I've heard something about tombstone records too, Not sure whether the problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I avoid it with correct recovery steps/interactions. Thanks a lot. --Gelen.___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
On 05/14/2012 03:48 PM, Robinson Tiemuqinke wrote: Hi Dmitri, Rich and all, I am a newbie to Redhat IPA, It looks like pretty cool compared with other solutions I've tried before. Thanks a lot for this great product! :) But there are still some things I needs your help. My main question is: How to restore the IPA setup with a daily machine-level IPA Replica backup? Please let me explain my IPA setup background and backup/restore goals trying to reach: I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup with Dogtag CA system. It is installed first. Then two IPA replicas are installed -- with '--setup-ca' options -- for load balancing and failover purposes. To describe my problems/objectives, I'll name the IPA Master as machine A, IPA replicas as B and C. and now I've one more extra IPA replica 'D' (virtual machine) setup ONLY for backup purposes. The setup looks like the following, A is the configuration Hub. B,C,D are siblings. A / | \ B C D The following are the steps I backup IPA setups and LDAP backends daily -- it is a whole machine-level backup (through virtual machine D). 1, First, IPA replica D is backed up daily. The backup happens like this: 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h D'. On the Hypervisor which holds virtual machine D, do a daily backup of the whole virtual disk that D is on. 1.2 turn on the IP replica D again. 1.3 after virtual machine D is up, on D optionally run a 'ipa-replica-manage --force-sync --from A' to sync the IPA databases forcibly. Now comes to restore part, which is pretty confusing to me. I've tried several times, and every times it comes this or that kinds of issues and so I am wondering that correct steps/ineraction of IPA Master/replicas are the king :( 2, case #1, A is broken, like disc failure, and then re-imaged after several days. 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the daily backup from IPA replica D? 2.2 do I have to check some files on A into subversion immediately after A was initially installed? 2.3 Please describe the steps. I'll follow exactly and report the results. 3, case #2, A is working, but either B, or C is broken. 3.1 It looks that I don't need the daily backup of D to kick in, is that right? 3.2 What are the correct steps on A; and B after it is re-imaged? 3.3 Please describe the steps. I'll follow exactly and report the results. 4, case #3, If some un-expected IPA changes happens on A -- like all users are deleted by human mistakes --, and even worse, all the changes are propagated to B and C in minutes. 4.1 How can I recover the IPA setup from daily backup from D? 4.2 which IPA master/replicas I should recover first? IPA master A, or IPA replicas B/C? and then how to recover others left one by one? 4.3 Do I have to disconnect replication agreement of B,C,D from A first? 4.4 Please describe the steps. I'll follow exactly and report the results. I've heard something about tombstone records too, Not sure whether the problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I avoid it with correct recovery steps/interactions. Thanks a lot. --Gelen. I can explain it conceptually. Rob is probably best to define the exact sequence and commands. If you A is broken you reinstall it, make it connect to D and init (force sync) A from D. Now you have a new A. If B or C dies you just re-install B or C and init from A. If you lost a lot of data I suggest you start a saved D instance and force-sync A from it and then force sync B and C from A. -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
Re: [Freeipa-users] Please help: How to restore IPA Master/Replicas from daily IPA Replica setup???
Hi Dimitri, thanks a lot for your offer. It will be more than appreciated if Rob, or some other talented genius could wiki the steps. The more details, the sooner, and the better. It will help IPA projects and its users dramatically, especially for newbies like me. :) Thanks again for you, Rob and others for the coming documentation work. --Gelen. From: Dmitri Pal d...@redhat.com To: Robinson Tiemuqinke hahaha_...@yahoo.com Cc: Freeipa-users@redhat.com Freeipa-users@redhat.com; Rich Megginson rmegg...@redhat.com Sent: Monday, May 14, 2012 1:20 PM Subject: Re: Please help: How to restore IPA Master/Replicas from daily IPA Replica setup??? On 05/14/2012 03:48 PM, Robinson Tiemuqinke wrote: Hi Dmitri, Rich and all, I am a newbie to Redhat IPA, It looks like pretty cool compared with other solutions I've tried before. Thanks a lot for this great product! :) But there are still some things I needs your help. My main question is: How to restore the IPA setup with a daily machine-level IPA Replica backup? Please let me explain my IPA setup background and backup/restore goals trying to reach: I'm running IPA 2.1.3 on Redhat Enterprise 6.2. The IPA master is setup with Dogtag CA system. It is installed first. Then two IPA replicas are installed -- with '--setup-ca' options -- for load balancing and failover purposes. To describe my problems/objectives, I'll name the IPA Master as machine A, IPA replicas as B and C. and now I've one more extra IPA replica 'D' (virtual machine) setup ONLY for backup purposes. The setup looks like the following, A is the configuration Hub. B,C,D are siblings. A / | \ B C D The following are the steps I backup IPA setups and LDAP backends daily -- it is a whole machine-level backup (through virtual machine D). 1, First, IPA replica D is backed up daily. The backup happens like this: 1.1 on IP replica D, run 'service IPA stop'. Then run 'shutdown -h D'. On the Hypervisor which holds virtual machine D, do a daily backup of the whole virtual disk that D is on. 1.2 turn on the IP replica D again. 1.3 after virtual machine D is up, on D optionally run a 'ipa-replica-manage --force-sync --from A' to sync the IPA databases forcibly. Now comes to restore part, which is pretty confusing to me. I've tried several times, and every times it comes this or that kinds of issues and so I am wondering that correct steps/ineraction of IPA Master/replicas are the king :( 2, case #1, A is broken, like disc failure, and then re-imaged after several days. 2.1 How to rebuild the IPA Master/Hub A after A is re-imaged, with the daily backup from IPA replica D? 2.2 do I have to check some files on A into subversion immediately after A was initially installed? 2.3 Please describe the steps. I'll follow exactly and report the results. 3, case #2, A is working, but either B, or C is broken. 3.1 It looks that I don't need the daily backup of D to kick in, is that right? 3.2 What are the correct steps on A; and B after it is re-imaged? 3.3 Please describe the steps. I'll follow exactly and report the results. 4, case #3, If some un-expected IPA changes happens on A -- like all users are deleted by human mistakes --, and even worse, all the changes are propagated to B and C in minutes. 4.1 How can I recover the IPA setup from daily backup from D? 4.2 which IPA master/replicas I should recover first? IPA master A, or IPA replicas B/C? and then how to recover others left one by one? 4.3 Do I have to disconnect replication agreement of B,C,D from A first? 4.4 Please describe the steps. I'll follow exactly and report the results. I've heard something about tombstone records too, Not sure whether the problem still exists in 2.1.3, or 2.2.0(on 6.3Beta)? If so, How can I avoid it with correct recovery steps/interactions. Thanks a lot. --Gelen. I can explain it conceptually. Rob is probably best to define the exact sequence and commands. If you A is broken you reinstall it, make it connect to D and init (force sync) A from D. Now you have a new A. If B or C dies you just re-install B or C and init from A. If you lost a lot of data I suggest you start a saved D instance and force-sync A from it and then force sync B and C from A. -- Thank you, Dmitri Pal Sr. Engineering Manager IPA project, Red Hat Inc. --- Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-users mailing list Freeipa-users@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-users
