I'm posting this as an informative post only as the issue was resolved. Might help someone else in the future because I know I'm not the only one that has done this or will do this.
About 9 months ago, we started a process to replace our domain controllers with new, under warranty, better etc etc hardware. We had 4 DC's that were on old, out of warranty hardware, and even a couple on hardware that was from a defunct company. We went about this process very slowly from the aspect of turning off the old DC's, the 2 new DC's that we built were up and running in a very short time. We did all the moving FSMO roles, GC's, DNS, DHCP from the old ones then waiting before running dcpromo to demote the server to just being a member server rather than a DC. This last Tuesday afternoon, we determined that it was time to demote the last remaining DC which was done just after lunch. In a matter of a few minutes, everyone's Outlook became unresponsive. I could access the console on Exchange, but when I tried to open ESM, I started receiving errors. The event logs led me to believe that Exchange could not communicate with the domain, and my assumption was that the last time Exchange was bounced (in Feb for the MS updates) that it authenticated on that DC and therefore needed a reboot which I did. That's when it got really fun and exciting, Exchange stopped the boot process at "Applying network" screen. It was up enough for me to remotely connect to the event logs & registry. I was primarily focused on these events in the application log: 9144 (which was referring to the just demoted DC), 2007, 2114, 2102 & 2090. Most of these were from MSDSAccess. After trying several things from KB articles found, and still having the same results on reboot, we opened a call with PSS, meanwhile continued to search for something (we were on hold waiting for a tech support person for at least an hour). Finally Event ID 2102 lead me to this KB article: 896703. Basically, the Exchange server had lost the "Manage auditing and security log" permission in the domain, even though when we looked at this on the DC's it appeared that the permission was there, the utility that is referenced (policytest) indicated that it wasn't there. I followed the recommendation of running the domain prep from the Exchange CD, rebooted Exchange again, and all was well. My bottom line conclusion of what happened was that the last DC that was in place when Exchange 2003 was originally installed was the DC that we demoted on Tuesday. From all aspects that we looked at for months, replication was happening, NTFrs was happy, no errors, however, this specific permission was not replicated from the old DC's to the new ones, and it even appeared that the permission had indeed replicated when it was checked from the DC's, but obviously was not. So, if you are replacing, or thinking of replacing old DC's, run the policytest.exe from the Exchange CD (support\exdeploy\policytest.exe) and make sure that the results are good before demoting that last DC, or you could spend an afternoon like I did last Tuesday. Have a great weekend everyone!! -- Sherry Abercrombie "Any sufficiently advanced technology is indistinguishable from magic." Arthur C. Clarke ~ Ninja Email Security with Cloudmark Spam Engine Gets Image Spam ~ ~ http://www.sunbeltsoftware.com/Ninja ~
