Al, I think it's appropriate to explain a little more, to
avoid further confusion, as accidentally deleting + recoverying object and
loosing group memberships are NOT separate problems (especially in
multi-domain forests or even in Windows 2000 single-domain forests).
The issues are indeed very much related to each
other:
-
tracking membership of a group to be able to undo a
change "in case one of it's members gets whacked" is generally a good idea,
no matter if a user has been deleted or if simply an administrator made a
mistake while editing group-memberships. When tracked (e.g. via daily
reports or dumps of the group-memberships - or by having a good
group-concept where all owners "know" the members), the owner of a group
should be able to get a group back to the state it should
be.
-
however, when you delete an object (e.g. a user,
computer, contact or a group itself), these objects naturally replicate as
tombstones to other DCs and GCs in the forest. When this happens, the
memberships of these objects in any group in the forest is
"cleaned" automatically - not only in the same domain where the objects
reside, but also in all of the other domains in the forest. I.e. the objects
are also removed from Universal (UG) and Domain Local Groups (DLG) of any
domain in the forest.
So what's the big deal? Well, if you
restore a DC from a system-state backup (on tape or file) and then
authoritatively restore the objects in their domain or even if you restore
the whole domain authoritatively (which not recommended anyways, unless you
really have to), the objects will never "repopulate" into the UGs and DLGs
of the other domains in the forest.
Good to know: if you restore a GC,
it will at least know of the UGs of the other domains incl. their
memberships (as these are a still stored in the AD database file saved at
the time of taking the system-state backup), which you could leverage to
repopulate the UGs in the respective Domains. However, if you've not
previously dumped your DLGs in the other domains, how will you be able to
recover their memberships? They are not stored on the GC you've recoverd,
and they were "cleaned" when the tombstone was able to replicate to the
other DCs/GCs in forest... And don't forget, that depending on
your group-modell, you could also have various nested groups which are
nothing else but members of other groups - these nestings will also get lost
if a group gets deleted. More about these issues (and others) is
described in the afforementioned whitepaper - incl. details on the
differences between 2000 and 2003 rgd. the
recovery challenge.
Here are some ideas to master the recovery
challenge and be on the safe side (besides relying on the group-owners to
recover memberships themselves):
-
hot-site approach: as mentioned in
another part of this thread, you could use DCs in a hot-site (we call it
LAG-site, as it's replication will be set to "lag" behind the other DCs).
These will hopefully not have replicated the tombstones at the time you
notice the deletion of the objects - you could then first use the
appropriate DCs the hot-site to perform the authoritative restore of the
objects without requiring you to perform a system restore (still need to
boot the DC in to DSRM mode). And secondly, you could analyse the groups on
the other DCs in the hotsite and then re-popluate the groups on a DC
outside of the hotsite. Best to script these restore activities... The
biggest challenge with this approach: you must hat one DC of EVERY domain in
the forest in your hotsite, which is fine if you only have a few, but which
can be nasty if you have many domains in your forest (should really consider
using virtual hw solutions for this approach).
-
database appoach: basically, whith
this approach, you would ensure that you save all the un-recoverable
links (e.g. group-memberships, but also the manager/directReports or
managedBy/managedObjects links which get "cleaned" just like the group
memberships...) into a separate repository. This could be a flat file,
AD/AM, MSDE or "full" SQL or whatever you preferr. When scheduled with
your normal backup cycle of your DCs, the stored data can ensure that you
are able to "fix" the missing object-links (which is what the
group-memberships and the other examples mentioned really are) by leveraging
your "link-repository". Obviously, it would be good if you would include
versioning into your link-storage, to be able to recover group-memberships
an the other links as they were at a specific point in time. Using this
approach will also allow you to easily visualize the memberships of any
object in the forest, as you will have all the objectlinks accessible in the
database (i.e. you can see which DLGs or UGs a user is a member of, no
matter which domain the groups belong to). This is the approach we took and
it's almost getting to a point of being finished (I've been promising the
availability of this solution on the DEC conferences for a while now... -
sorry for the major delay - you know how things go...;-) The challenge
here obviously lies in collecting and storing the links appropriately, so
you have to have some smart mechanism especially for really large AD
environments like our own. But for forests with many domains, this may
be more feasable than the hot-site approach. However, it can even be well
combined with the hot-site approach, as you could leverage the hot-site DCs
for your link-backups, and after a restore use the database to recover the
links (instead of querying each of the hotsite DCs).
-
backup-tool approach:backup tools,
which concentrate on online recovery of objects in AD (now supported in 2003
with the new API for tombstone reanimation) will also be very useful in
performing the full recovery of objects, as they will have stored the object
links (e.g. groups a user is a member of) withing their backup. However,
only few tools really do the job of link-recovery really well, as most
vendors don't care to query the "remote" domains for the memberships of
objects in DLGs - they will be able to recover UGs, if the object's backup
was taken from a GC, but the DLGs will be lost. I only know of one
tool, which is currently able to also fully recover the DLGs in any domain
of the forest, when restoring objects in AD. This isn't a place for
advertising, but you may want to look at the aforementioned whitepaper
;-)
Realize, that whichever approach you take, restoring
objects in an AD forest is not a domain-admin's task => it's the task of an
enterprise admin, as only members of this group have the ability to recover
the links outside of the domain that the restored objects belong to (unless
you grant these permissions to other groups, which would then have similar
permissions as an EA).
Cheers,
Guido
I think there's two approaches here but correct me if I
misunderstood to flow.
One concept is to restore the actual object in case of
accidental deletion, intentional deletion, corruption, etc. The other is
to track the membership in case one of it's members gets whacked. That
about what you're saying?
To me, these are two very important, but separate
scenarios. One solution already in place is a tracking mechanism that
exports group information on a daily basis. That's the custom version of
what I have now, but it's nowhere near as efficient as a SQL/AD/AM solution
would be in a multi-domain environment. It only allows us to put the group
membership back, but has nothing to do with the group object itself. If
we lost that, we lost the sID etc that would make it useful outside of a
restore. In case of administrative error, we can look back at the
reference (keep a week's worth for now) and put it back the way it should be
without having to go to tape.
If you're going to the trouble of creating this homegrown
system wouldn't it make sense to make it part of the lifecycle management
system? There's certainly a market for that in the states with the
current round of laws about data and process.
Just a thought, but having a system that audits (for lack
of a better term) user/group lifecycles and resource allocation would be an
interesting thing to have. Kind of another tier in the management of the
system (a meta-directory type solution or other?)
Al
ha, I knew that would be your answer
;-))
and I can partly understand your strategy => the owner
of the group should know what's in it, so if there is a problem with the
memberships it's his and not yours. But this is really only acceptable
for a small issue, where you loose a couple of memberships - not when you use
a couple of hundred of users incl. their memberships.
Sure losing a DLG membership has the same result in
losing resource access than a GG does - however, DLGs in multi-forest
environments are simply harder to recover the "native" way (i.e. be
authoritatively restoring your accidentally deleted users from one domain), as
that restored DC doesn't know of the DLG memberships outside of it's own
domain, which will then be lost for good (much easier to recover memberships
ins GGs and UGs as the DC/GC will "know" of the memberships after the
recovery). Your DLG memberships won't come back until re-added by
your group-owners, who will be happy to manage re-adding hundreds of users
into various groups via the UI they use... :-(
Obviously your impact will be less than for other
companies, as you have a really cool group-structure however .But no matter
how careful you are, Murphy is watching you. And things will happen. And you
have to be prepared...
The AD/AM idea isn't bad, but I'm just implementing the
same based on SQL and it's almost done - a nice tool that gives you exactly
what you're describing. And will help to recover those lost group-memberhips
and it will allow you to see which group your users or other objects are in
within the forest - in any domain. Stay tuned. However, it will still
require a normal authoritative restore of the actual objects that were deleted
- thus it's not as powerful as some of the online-recovery methods available
out there. So I encurage anyone responsible for back-up of their AD also to
look at these tools.
/Guido
Using the DLG's doesn't kill us any more than if we used
GG's. Same loss of resource access.
As for the accidents, the guys with the big guns don't
use the GUI for most anything, they use very targeted scripts that do very
specific things. We don't, for instance have any mass delete anything scripts.
All one off delete.
The groups are supposed to have well known membership to
the admins running them, they are supposed to be auditing the groups on a very
regular basis as to who should be in them. So loss of a group should simply be
recreate the group, reassign to the proper ACE in the proper file structure
(we don't do one group secures a zillion different things or at least heavily
discourage it), readd the correct people.
I do have some ideas floating in the back of my mind
about pulling all groups, computers, users off into a single AD/AM instance so
we can track things there. Don't sync the deletes other than marking a field
in AD/AM when the delete or occurred. This is more for being able to do quick
checks for things in the directory (everything would be tuple indexed) but
could also help if someone smoked a group that they shouldn't have as we would
have the last known membership for sure. I would also like to get some form of
change log management in there as well but that project is way pie in the sky
at the moment. Trying to get K3 deployed at the moment and the final pieces of
E2K deployed.
-------------
actually, you need to consider this issue more than
others Joe, as you're building all group-memberships on Domain Local Groups
(in a multi-domain environment) which will kill you, if you do accidentally
delete the wrong objects. Obviously you could still restore all
domains - but that's pretty nasty.
And accidents don't only happen to lower privileged
admins - it could be one of you three...
/Guido
Yes, excellent point. We haven't started worrying about
that granularity yet. If something is deleted, we figured the person with the
power to delete it intended it. Have a nice day. There are only three
people who can really do any huge mass deletes across the board and we all sit
within smacking distance of each other so we are careful as we have sensitive
ears and don't want to be cuffed. I do think we need some sort of solution for
this eventually though. But it is more to reduce nuisance factor for silly OU
admins than anything else.
Right now mostly still just worrying about the old South
East Michigan was swallowed by a volcano that came out of nowhere... How do we
make sure we can recover.
-------------
will only be good for restoring the DC hardware, but
depending on your setup won't be sufficient to fully recover accidentally
deleted objects.
I've worked with Aelita on this whitepaper to discuss the
potential issues:
/Guido
1. Multiple DCs in diseparate
locations.
2. Virtual DC for each domain that is shut down nightly
and the disk file for each is copied to some other location.
-------------
What is the best way to backup your domain controller
so you can restore it in a disaster situation.
