Re: Interesting Exchange Event I Had This Week
This information is good to know Sherry. I'm about to do a swing migration of Exchange 2003 this weekend right after I install a couple 2003 Domain Controllers tonight (and soon thereafter demote our 2000 domain controllers so they can be decommissioned). I'll pay close attention while I go through this process. On Fri, Mar 28, 2008 at 12:22 PM, Sherry Abercrombie <[EMAIL PROTECTED]> wrote: > I'm posting this as an informative post only as the issue was resolved. > Might help someone else in the future because I know I'm not the only one > that has done this or will do this. > > About 9 months ago, we started a process to replace our domain controllers > with new, under warranty, better etc etc hardware. We had 4 DC's that were > on old, out of warranty hardware, and even a couple on hardware that was > from a defunct company. We went about this process very slowly from the > aspect of turning off the old DC's, the 2 new DC's that we built were up and > running in a very short time. We did all the moving FSMO roles, GC's, DNS, > DHCP from the old ones then waiting before running dcpromo to demote the > server to just being a member server rather than a DC. This last Tuesday > afternoon, we determined that it was time to demote the last remaining DC > which was done just after lunch. In a matter of a few minutes, everyone's > Outlook became unresponsive. I could access the console on Exchange, but > when I tried to open ESM, I started receiving errors. The event logs led me > to believe that Exchange could not communicate with the domain, and my > assumption was that the last time Exchange was bounced (in Feb for the MS > updates) that it authenticated on that DC and therefore needed a reboot > which I did. That's when it got really fun and exciting, Exchange stopped > the boot process at "Applying network" screen. It was up enough for me to > remotely connect to the event logs & registry. I was primarily focused on > these events in the application log: 9144 (which was referring to the just > demoted DC), 2007, 2114, 2102 & 2090. Most of these were from MSDSAccess. > After trying several things from KB articles found, and still having the > same results on reboot, we opened a call with PSS, meanwhile continued to > search for something (we were on hold waiting for a tech support person for > at least an hour). Finally Event ID 2102 lead me to this KB article: > 896703. Basically, the Exchange server had lost the "Manage auditing and > security log" permission in the domain, even though when we looked at this > on the DC's it appeared that the permission was there, the utility that is > referenced (policytest) indicated that it wasn't there. I followed the > recommendation of running the domain prep from the Exchange CD, rebooted > Exchange again, and all was well. > > My bottom line conclusion of what happened was that the last DC that was > in place when Exchange 2003 was originally installed was the DC that we > demoted on Tuesday. From all aspects that we looked at for months, > replication was happening, NTFrs was happy, no errors, however, this > specific permission was not replicated from the old DC's to the new ones, > and it even appeared that the permission had indeed replicated when it was > checked from the DC's, but obviously was not. > > So, if you are replacing, or thinking of replacing old DC's, run the > policytest.exe from the Exchange CD (support\exdeploy\policytest.exe) and > make sure that the results are good before demoting that last DC, or you > could spend an afternoon like I did last Tuesday. > > Have a great weekend everyone!! > > -- > Sherry Abercrombie > > "Any sufficiently advanced technology is indistinguishable from magic." > Arthur C. Clarke > > > -- Mike Sullivan [EMAIL PROTECTED] ~ Ninja Email Security with Cloudmark Spam Engine Gets Image Spam ~ ~ http://www.sunbeltsoftware.com/Ninja~
Re: Interesting Exchange Event I Had This Week
Definitely going to keep this one in mind for future scenarios. On Fri, Mar 28, 2008 at 12:22 PM, Sherry Abercrombie <[EMAIL PROTECTED]> wrote: > I'm posting this as an informative post only as the issue was resolved. > Might help someone else in the future because I know I'm not the only one > that has done this or will do this. > > About 9 months ago, we started a process to replace our domain controllers > with new, under warranty, better etc etc hardware. We had 4 DC's that were > on old, out of warranty hardware, and even a couple on hardware that was > from a defunct company. We went about this process very slowly from the > aspect of turning off the old DC's, the 2 new DC's that we built were up and > running in a very short time. We did all the moving FSMO roles, GC's, DNS, > DHCP from the old ones then waiting before running dcpromo to demote the > server to just being a member server rather than a DC. This last Tuesday > afternoon, we determined that it was time to demote the last remaining DC > which was done just after lunch. In a matter of a few minutes, everyone's > Outlook became unresponsive. I could access the console on Exchange, but > when I tried to open ESM, I started receiving errors. The event logs led me > to believe that Exchange could not communicate with the domain, and my > assumption was that the last time Exchange was bounced (in Feb for the MS > updates) that it authenticated on that DC and therefore needed a reboot > which I did. That's when it got really fun and exciting, Exchange stopped > the boot process at "Applying network" screen. It was up enough for me to > remotely connect to the event logs & registry. I was primarily focused on > these events in the application log: 9144 (which was referring to the just > demoted DC), 2007, 2114, 2102 & 2090. Most of these were from MSDSAccess. > After trying several things from KB articles found, and still having the > same results on reboot, we opened a call with PSS, meanwhile continued to > search for something (we were on hold waiting for a tech support person for > at least an hour). Finally Event ID 2102 lead me to this KB article: > 896703. Basically, the Exchange server had lost the "Manage auditing and > security log" permission in the domain, even though when we looked at this > on the DC's it appeared that the permission was there, the utility that is > referenced (policytest) indicated that it wasn't there. I followed the > recommendation of running the domain prep from the Exchange CD, rebooted > Exchange again, and all was well. > > My bottom line conclusion of what happened was that the last DC that was in > place when Exchange 2003 was originally installed was the DC that we demoted > on Tuesday. From all aspects that we looked at for months, replication was > happening, NTFrs was happy, no errors, however, this specific permission was > not replicated from the old DC's to the new ones, and it even appeared that > the permission had indeed replicated when it was checked from the DC's, but > obviously was not. > > So, if you are replacing, or thinking of replacing old DC's, run the > policytest.exe from the Exchange CD (support\exdeploy\policytest.exe) and > make sure that the results are good before demoting that last DC, or you > could spend an afternoon like I did last Tuesday. > > Have a great weekend everyone!! > > -- > Sherry Abercrombie > > "Any sufficiently advanced technology is indistinguishable from magic." > Arthur C. Clarke > > ~ Ninja Email Security with Cloudmark Spam Engine Gets Image Spam ~ ~ http://www.sunbeltsoftware.com/Ninja~
RE: Interesting Exchange Event I Had This Week
Very nice work and thanks for sharing. Cheers. From: Sherry Abercrombie [mailto:[EMAIL PROTECTED] Sent: Friday, March 28, 2008 2:23 PM To: MS-Exchange Admin Issues Subject: Interesting Exchange Event I Had This Week ~ Ninja Email Security with Cloudmark Spam Engine Gets Image Spam ~ ~ http://www.sunbeltsoftware.com/Ninja~
RE: Interesting Exchange Event I Had This Week
Same here. Timely information that I'm putting right in my upgrade folder. Thanks Sherry! From: Mark Boersma [mailto:[EMAIL PROTECTED] Sent: Friday, March 28, 2008 2:40 PM To: MS-Exchange Admin Issues Subject: RE: Interesting Exchange Event I Had This Week Thank you. I'll be walking through that process in the next couple of months. Mark - Two rules to success in life: 1. Never tell people everything you know. From: Sherry Abercrombie [mailto:[EMAIL PROTECTED] Sent: Friday, March 28, 2008 3:23 PM To: MS-Exchange Admin Issues Subject: Interesting Exchange Event I Had This Week I'm posting this as an informative post only as the issue was resolved. Might help someone else in the future because I know I'm not the only one that has done this or will do this. About 9 months ago, we started a process to replace our domain controllers with new, under warranty, better etc etc hardware. We had 4 DC's that were on old, out of warranty hardware, and even a couple on hardware that was from a defunct company. We went about this process very slowly from the aspect of turning off the old DC's, the 2 new DC's that we built were up and running in a very short time. We did all the moving FSMO roles, GC's, DNS, DHCP from the old ones then waiting before running dcpromo to demote the server to just being a member server rather than a DC. This last Tuesday afternoon, we determined that it was time to demote the last remaining DC which was done just after lunch. In a matter of a few minutes, everyone's Outlook became unresponsive. I could access the console on Exchange, but when I tried to open ESM, I started receiving errors. The event logs led me to believe that Exchange could not communicate with the domain, and my assumption was that the last time Exchange was bounced (in Feb for the MS updates) that it authenticated on that DC and therefore needed a reboot which I did. That's when it got really fun and exciting, Exchange stopped the boot process at "Applying network" screen. It was up enough for me to remotely connect to the event logs & registry. I was primarily focused on these events in the application log: 9144 (which was referring to the just demoted DC), 2007, 2114, 2102 & 2090. Most of these were from MSDSAccess. After trying several things from KB articles found, and still having the same results on reboot, we opened a call with PSS, meanwhile continued to search for something (we were on hold waiting for a tech support person for at least an hour). Finally Event ID 2102 lead me to this KB article: 896703. Basically, the Exchange server had lost the "Manage auditing and security log" permission in the domain, even though when we looked at this on the DC's it appeared that the permission was there, the utility that is referenced (policytest) indicated that it wasn't there. I followed the recommendation of running the domain prep from the Exchange CD, rebooted Exchange again, and all was well. My bottom line conclusion of what happened was that the last DC that was in place when Exchange 2003 was originally installed was the DC that we demoted on Tuesday. From all aspects that we looked at for months, replication was happening, NTFrs was happy, no errors, however, this specific permission was not replicated from the old DC's to the new ones, and it even appeared that the permission had indeed replicated when it was checked from the DC's, but obviously was not. So, if you are replacing, or thinking of replacing old DC's, run the policytest.exe from the Exchange CD (support\exdeploy\policytest.exe) and make sure that the results are good before demoting that last DC, or you could spend an afternoon like I did last Tuesday. Have a great weekend everyone!! -- Sherry Abercrombie "Any sufficiently advanced technology is indistinguishable from magic." Arthur C. Clarke Please consider the environment before printing this email. CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipients(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ~ Ninja Email Security with Cloudmark Spam Engine Gets Image Spam ~ ~ http://www.sunbeltsoftware.com/Ninja~
RE: Interesting Exchange Event I Had This Week
Thank you. I'll be walking through that process in the next couple of months. Mark - Two rules to success in life: 1. Never tell people everything you know. From: Sherry Abercrombie [mailto:[EMAIL PROTECTED] Sent: Friday, March 28, 2008 3:23 PM To: MS-Exchange Admin Issues Subject: Interesting Exchange Event I Had This Week I'm posting this as an informative post only as the issue was resolved. Might help someone else in the future because I know I'm not the only one that has done this or will do this. About 9 months ago, we started a process to replace our domain controllers with new, under warranty, better etc etc hardware. We had 4 DC's that were on old, out of warranty hardware, and even a couple on hardware that was from a defunct company. We went about this process very slowly from the aspect of turning off the old DC's, the 2 new DC's that we built were up and running in a very short time. We did all the moving FSMO roles, GC's, DNS, DHCP from the old ones then waiting before running dcpromo to demote the server to just being a member server rather than a DC. This last Tuesday afternoon, we determined that it was time to demote the last remaining DC which was done just after lunch. In a matter of a few minutes, everyone's Outlook became unresponsive. I could access the console on Exchange, but when I tried to open ESM, I started receiving errors. The event logs led me to believe that Exchange could not communicate with the domain, and my assumption was that the last time Exchange was bounced (in Feb for the MS updates) that it authenticated on that DC and therefore needed a reboot which I did. That's when it got really fun and exciting, Exchange stopped the boot process at "Applying network" screen. It was up enough for me to remotely connect to the event logs & registry. I was primarily focused on these events in the application log: 9144 (which was referring to the just demoted DC), 2007, 2114, 2102 & 2090. Most of these were from MSDSAccess. After trying several things from KB articles found, and still having the same results on reboot, we opened a call with PSS, meanwhile continued to search for something (we were on hold waiting for a tech support person for at least an hour). Finally Event ID 2102 lead me to this KB article: 896703. Basically, the Exchange server had lost the "Manage auditing and security log" permission in the domain, even though when we looked at this on the DC's it appeared that the permission was there, the utility that is referenced (policytest) indicated that it wasn't there. I followed the recommendation of running the domain prep from the Exchange CD, rebooted Exchange again, and all was well. My bottom line conclusion of what happened was that the last DC that was in place when Exchange 2003 was originally installed was the DC that we demoted on Tuesday. From all aspects that we looked at for months, replication was happening, NTFrs was happy, no errors, however, this specific permission was not replicated from the old DC's to the new ones, and it even appeared that the permission had indeed replicated when it was checked from the DC's, but obviously was not. So, if you are replacing, or thinking of replacing old DC's, run the policytest.exe from the Exchange CD (support\exdeploy\policytest.exe) and make sure that the results are good before demoting that last DC, or you could spend an afternoon like I did last Tuesday. Have a great weekend everyone!! -- Sherry Abercrombie "Any sufficiently advanced technology is indistinguishable from magic." Arthur C. Clarke Please consider the environment before printing this email. CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipients(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. ~ Ninja Email Security with Cloudmark Spam Engine Gets Image Spam ~ ~ http://www.sunbeltsoftware.com/Ninja~
