Ben, We really need to find out exactly what was defined in the schema when to determine how this occurred. From the information provided it would appear that the groupofURLs class was defined in the schema and objects were instantiated and then its definition was changed. This could explain why the in-site DCs have the objects and out of site ones do not, schema partition changes replicate at a higher priority than domain partition changes so when these got bulked up for out of site replication the objects no longer met the schema definition, i.e. the subclass of group is no longer defined for the object. These objects do not appear to fit the definition of the groupofURLs class as it is now defined and are therefore causing replication to be blocked. This is of course all a hypothesis as I do not have the details on exactly what changes were made when to the schema.
Thanks, -Steve -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of WATSON, BEN Sent: Saturday, September 23, 2006 5:07 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects Hi Steve, Yes, there were some schema modifications one of the other admins was working on to deploy an application that would allow for people to use their Windows accounts to log into our Intrasite (they were formerly using their Unix accounts). He had tested this on our test network, which is a copy of our production network, upgraded to Windows 2003 R2 and also has the Longhorn schema extensions applied as well. From what I understand, it experienced no ill effects, however it is a single site test network. Here is the output from the groupOfURLs extension. AdFind V01.31.00cpp Joe Richards ([EMAIL PROTECTED]) March 2006 Using server: appsig-av.appsig.com:389 Directory: Windows 2000 Base DN: CN=Schema,CN=Configuration,DC=appsig,DC=com dn:CN=groupOfURLs,CN=Schema,CN=Configuration,DC=appsig,DC=com >adminDisplayName: groupOfURLs >cn: groupOfURLs >defaultObjectCategory: CN=groupOfURLs,CN=Schema,CN=Configuration,DC=appsig,DC=com >governsID: 2.16.840.1.113730.3.2.33 >instanceType: 4 >lDAPDisplayName: groupOfURLs >mayContain: memberURL >distinguishedName: CN=groupOfURLs,CN=Schema,CN=Configuration,DC=appsig,DC=com >objectCategory: CN=Class-Schema,CN=Schema,CN=Configuration,DC=appsig,DC=com >objectClass: top >objectClass: classSchema >objectClassCategory: 0 >objectGUID: {2B09EA58-1A00-4170-B419-9ADC0AA0B655} >possSuperiors: container >name: groupOfURLs >rDNAttID: cn >schemaIDGUID: {8B5ACDC4-EAF2-45D9-A596-C196ABD02405} >showInAdvancedViewOnly: TRUE >subClassOf: top >systemOnly: FALSE >uSNChanged: 7985664 >uSNCreated: 7985664 >whenChanged: 20060913180400.0Z >whenCreated: 20060913180359.0Z There are currently 4 other objects that have the groupofURLs listed as an objectClass. dn:CN=InfowebDept12,OU=InfowebGroups,DC=appsig,DC=com >objectClass: top >objectClass: groupOfURLs >objectClass: group dn:CN=InfowebDept24,OU=InfowebGroups,DC=appsig,DC=com >objectClass: top >objectClass: groupOfURLs >objectClass: groupOfNames >objectClass: group dn:CN=InfowebDept25,OU=InfowebGroups,DC=appsig,DC=com >objectClass: top >objectClass: groupOfURLs >objectClass: group dn:CN=InfowebSection581,OU=InfowebGroups,DC=appsig,DC=com >objectClass: top >objectClass: groupOfURLs >objectClass: group Let me know if you need anything else. Thanks, ~Ben -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steve Linehan Sent: Saturday, September 23, 2006 1:13 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects Actually looking at this further you will probably find that the schemas are in sync, i.e. the groupofURLs object class is defined across all of the servers. I say that because the error you would have gotten if it did not exist on the target would have been either schema mismatch or ERROR_DS_OBJ_CLASS_NOT_DEFINED. So what I suspect is that groupofURLs is not defined properly or is being referenced incorrectly. Can you dump the schema entry for this class from one of your servers snd post it? Also if you have the LDIF file that was used to update the schema that includes the definition of this object class that would be great as well. What I do not understand is how you have an object defined this way as I would have expected us to block creation of the object if this class is not defined/referenced properly. Any information on how the schema was modified and how these objects were created would be helpful. The fix will likely be to remove the groupofurls objectclass from the object but you need to determine how you got to this point so that it does not occur again. Thanks, -Steve ________________________________________ From: [EMAIL PROTECTED] On Behalf Of Steve Linehan Sent: Saturday, September 23, 2006 2:54 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects Ben, It would appear that the schema was modified on the source servers but does not match on the destination servers. I am not aware of a default objectclass called groupofURLs. Is this something that you modified recently? Can you dump the definition of this objectclass from a schema on the source and verify that the schema on the target does not match? Can you also send me a repadmin /showreps /v from a source and target. It would appear that you have a schema modification gone bad. Can you also search and see if you have any other objects on the source DC that have that objectclass listed? Thanks, -Steve ________________________________ From: [EMAIL PROTECTED] On Behalf Of WATSON, BEN Sent: Saturday, September 23, 2006 2:28 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects Hi Steve, First off, thanks for all your help, you are always incredibly helpful. Here's the output you requested from the source server. dn:CN=InfowebAccess\0ADEL:e9888888-616b-4944-bbe1-c8265cf4cc89,CN=Delete d Objects,DC=appsig,DC=com >objectClass: top >objectClass: groupOfURLs >objectClass: group I should note though that this object NEVER replicated to other sites. So the only output I can give you is from the source DC. At least on the surface, this object seems to be the source of the replication issues. Thanks again, ~Ben From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steve Linehan Sent: Friday, September 22, 2006 11:49 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects Can you dump the objectclass attribute on the deleted object mentioned in the error on one of the source servers and a destination server? The second error code in the internal error event log seems to indicate that the objectclass is being updated with a value that is not a subclass. C:\tools\err\Err>err 20b4 # for hex 0x20b4 / decimal 8372 : ERROR_DS_OBJ_CLASS_NOT_SUBCLASS winerror.h # The specified class is not a subclass. # 1 matches found for "20b4" Thanks, -Steve From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of WATSON, BEN Sent: Saturday, September 23, 2006 1:03 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects Correction, 10 domain controllers in 9 sites. From: WATSON, BEN Sent: Friday, September 22, 2006 10:58 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects Basic info and troubleshooting I've done to gather symptom information... We are running a single forest, single domain Windows 2000 environment (I know, I know, I'm in the process of getting this ugpraded to Win2k3 R2) with 9 domain controllers and 8 sites. Three of the sites are hub sites, and each hub site has 2 spoke sites. Our main hub site has 2 domain controllers, and all other remote sites have a single domain controller. The replication issues are actually affecting an entire site, unfortunately our main hub site (the one with 2 domain controllers). Oddly enough, it's not Domain Controller specific, the problem is actually site specific, and even more specifically, it's only affecting replication traffic OUTBOUND from the site. Inbound replication traffic works fine as well as replication between the two domain controllers inside the site. At first, I thought the domain controller that was acting as a Bridgehead for our site was having issues, so I forced the other domain controller in the site to be the preferred bridgehead server, deleted all the connection objects, and allowed the KCC to recreate the connection objects. It did this properly. I then attempted to force replication to take place, and the same symptoms still persisted even though it was a completely different domain controller attempting to perform the intersite replication. Here are the results of performing a, "REPADMIN /REPLADMIN /BYSRC /BYDEST /SORT:DELTA" command. Appsig-AV and Appsig-AD are the two domain controllers in the problem site. Appsig-AD was the original DC that began showing problems in the site, and Appsig-AV is the domain controller I switched over to test intersite replication using a different DC. Replication Summary Start Time: 2006-09-22 21:59:43 Beginning data collection for replication summary, this may take awhile: ............. Source DC largest delta fails/total %% error APPSIG-MDOPC 14m:06s 0 / 18 0 APPSIG-LAOPC 10m:09s 0 / 12 0 APPSIG-TXOPC 09m:52s 0 / 3 0 APPSIG-OCOPC 09m:52s 0 / 3 0 APPSIG-OROPC 02m:48s 0 / 6 0 APPSIG-UTOPC 02m:46s 0 / 6 0 APPSIG-DCOPC 02m:08s 0 / 3 0 APPSIG-VAOPC 02m:08s 0 / 3 0 APPSIG-AV (unknown) 4 / 15 26 (8442) The replication system encountered an internal error. APPSIG-AD (unknown) 4 / 15 26 (8442) The replication system encountered an internal error. Destination DC largest delta fails/total %% error APPSIG-VAOPC 14m:12s 0 / 3 0 APPSIG-TXOPC 10m:12s 0 / 3 0 APPSIG-DCOPC 07m:42s 0 / 3 0 APPSIG-OCOPC 07m:07s 0 / 3 0 APPSIG-AD 04m:33s 0 / 3 0 APPSIG-AV 02m:50s 0 / 15 0 APPSIG-LAOPC (unknown) 2 / 15 13 (8442) The replication system encountered an internal error. APPSIG-UTOPC (unknown) 2 / 9 22 (8442) The replication system encountered an internal error. APPSIG-MDOPC (unknown) 2 / 21 9 (8442) The replication system encountered an internal error. APPSIG-OROPC (unknown) 2 / 9 22 (8442) The replication system encountered an internal error. Now on to event log errors and warnings in the Directory Service event log. Oddly enough, the domain controlllers in the problem site show no real errors or warnings to speak of. However, the domain controllers that have direct site connections to this site have plenty of errors when trying to replicate from these sites. I'm showing 4 errors/warnings when replication is attempted. Here are the errors/events after making the registry changes Steve suggested. Event ID: 1173 - Category: Interneal Processing - Type: Warning Internal event: Exception e0010002 has occurred with parameters 8442 and 20b4 (Internal ID 3050bdc). Event ID: 1084 - Category: Replication - Type: Error Replication error: The directory replication agent (DRA) couldn't update object CN="InfowebAccessDEL:e9888888-616b-4944-bbe1-c8265cf4cc89",CN=Deleted Objects,DC=appsig,DC=com (GUID e9888888-616b-4944-bbe1-c8265cf4cc89) on this system with changes which have been received from source server e928ad23-039d-4dbd-b214-f88b4ae54819._msdcs.appsig.com. An error occurred during the application of the changes to the directory database on this system. The error message is: The replication system encountered an internal error. The directory will try to update the object later on the next replication cycle. Synchronization of this server with the source is effectively blocked until the update problem is corrected. If this condition appears to be related to a resource shortage, please stop and restart this Windows Domain Controller. If this condition is an internal error, a database error, or an object relationship or constraint error, manual intervention will be required to correct the database and allow the update to proceed. It is valuable to note that the problem is caused by the fact that the change on the remote system cannot be applied locally. Manually updating the objects on the local system in not recommended. Instead, on the source system (which has the changes already), try to reverse or back out the change. Then, on the next replication cycle, observe whether the change can now be applied locally. The record data is the status code. Event ID: 1085 - Category: Replication - Type: Warning Replication warning: The directory replication agent (DRA) couldn't synchronize partition DC=appsig,DC=com with partition on directory server b04a1a6f-dae6-4795-bb91-9805f458c9d5._msdcs.appsig.com. The error was: The replication system encountered an internal error. Please verify that the address can be resolved with DNS, and that it is reachable via the transport. If this error persists, the KCC will reconfigure the links around this server. The record data is the status code. Event ID: 1061 - Category: Replication - Type: Warning Internal error: The directory replication agent (DRA) call returned error 8442. That's all of it. If you need me to get any further information, let me know and I'll get it immediately. Thank you for your help! ~Ben ________________________________ From: [EMAIL PROTECTED] on behalf of Steve Linehan Sent: Fri 9/22/2006 8:34 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects You could also turn up additional logging which would give more details as to what the internal error is. I would suggest starting with the following: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics 1. Locate the "5 Replication Events" value under the above key. 2. On the Edit menu, click DWORD, type 4, and then click OK. 3. Locate the "9 Internal Processing" value under the same key. 4. On the Edit menu, click DWORD, type 1, and then click OK. After you do this post the full event text for the error and any additional replication or internal processing errors. I would expect to get back an Exception value with parameters and an internal id. These can be used to determine what is causing the problem. To answer your original question the tombstoned object will only be removed once the tombstone lifetime is reached and garbage collection has run. I would not recommend changing the tombstone lifetime to correct this as it is forest wide and can lead to more serious problems than you currently have. We should be able to determine the cause of the internal error and correct it without taking such risky and drastic measures. Thanks, -Steve -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Vinnie Cardona Sent: Friday, September 22, 2006 9:53 PM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] Replication Problems and Tombstoned Objects What event id are you seeing associate with this error? Vinnie Cardona Systems Administrator Ernest Health, Inc Information Technology Dept 505.798.6472 -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of WATSON, BEN Sent: Friday, September 22, 2006 6:18 PM To: ActiveDir@mail.activedir.org Subject: [ActiveDir] Replication Problems and Tombstoned Objects Our forest is currently experiencing some replication issues. The common error we have been receiving has revolved around a single object. To summarize, how do you permanently delete Active Directory objects? More specifically, how do you remove an object that is already tombstoned? Here is why I need to do this, here is the full error... ------- Replication error: The directory replication agent (DRA) couldn't update object CN=InfowebAccess,OU=InfowebGroups,DC=appsig,DC=com (GUID e9888888-616b-4944-bbe1-c8265cf4cc89) on this system with changes which have been received from source server e928ad23-039d-4dbd-b214-f88b4ae54819._msdcs.appsig.com. An error occurred during the application of the changes to the directory database on this system. The error message is: The replication system encountered an internal error. The directory will try to update the object later on the next replication cycle. Synchronization of this server with the source is effectively blocked until the update problem is corrected. If this condition appears to be related to a resource shortage, please stop and restart this Windows Domain Controller. If this condition is an internal error, a database error, or an object relationship or constraint error, manual intervention will be required to correct the database and allow the update to proceed. It is valuable to note that the problem is caused by the fact that the change on the remote system cannot be applied locally. Manually updating the objects on the local system in not recommended. Instead, on the source system (which has the changes already), try to reverse or back out the change. Then, on the next replication cycle, observe whether the change can now be applied locally. The record data is the status code. ------- After I deleted this object, I continue to get the same error, except it now references the deleted (tombstoned) object as a roadblock. ------- Replication error: The directory replication agent (DRA) couldn't update object CN="InfowebAccess DEL:e9888888-616b-4944-bbe1-c8265cf4cc89",CN=Deleted Objects,DC=appsig,DC=com (GUID e9888888-616b-4944-bbe1-c8265cf4cc89) etc... (same as error above) ------- What would be the proper method to permanently remove a tombstoned object? If I'm following the error messages, then removing the object permanently should (hopefully) resolve the issues. Thanks, ~Ben List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.activedir.org/ml/threads.aspx
