The more I think about this the more crazy it gets in my mind. Did this turn into a career changing event, or at least come very close to it?
I think I'd rather attempt giving the wolverine a brazillian, the pain will be over faster. On Thu, Sep 29, 2011 at 12:14 PM, Campbell, Rob < rob_campb...@centraltechnology.net> wrote: > I think it was probably right up there with giving a wolverine a bikini > wax.**** > > ** ** > > *From:* Jonathan Link [mailto:jonathan.l...@gmail.com] > *Sent:* Thursday, September 29, 2011 11:06 AM > *To:* MS-Exchange Admin Issues > > *Subject:* Re: Backup of Exchange 2010**** > > ** ** > > Oy! That doesn't sound like a good idea at all.**** > > On Thu, Sep 29, 2011 at 12:02 PM, Campbell, Rob < > rob_campb...@centraltechnology.net> wrote:**** > > From what I’ve been told, they attempted to copy the cluster config files > from one server to all the rest to try and get the cluster back into a > consistent state.**** > > **** > > *From:* Michael B. Smith [mailto:mich...@smithcons.com] > *Sent:* Thursday, September 29, 2011 10:57 AM > **** > > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010**** > > **** > > That’s what I would’ve done.**** > > **** > > Evicted it from the cluster, format, reinstall OS, setup /recoverserver, > add to DAG, start reseed.**** > > **** > > Couple hours work, max. And then the reseed can happen in the background…* > *** > > **** > > Regards,**** > > **** > > Michael B. Smith**** > > Consultant and Exchange MVP**** > > http://TheEssentialExchange.com**** > > **** > > *From:* Campbell, Rob [mailto:rob_campb...@centraltechnology.net] > *Sent:* Thursday, September 29, 2011 11:52 AM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > I don’t know everything that transpired, but I can say that it started when > one server was taken out of production, put into an isolated test > environment and used for a “smoking crater” DR test scenario, and then > brought back into production. That was probably a bad idea. I think the > attempts to recover that one server may have contributed to the downfall of > all the rest. **** > > **** > > It would probably have been better to just rebuild that one server at the > outset.**** > > **** > > *From:* Michael B. Smith [mailto:mich...@smithcons.com] > *Sent:* Thursday, September 29, 2011 10:34 AM > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > I’ve asked a PFE to review Rob’s case and he’s asked for additional data > from the case owner. After that review, I’ll forward the information to Rob > and he can choose whether to share here…**** > > **** > > Now… I’ve never heard of anything like this before and I’ve never run into > anything like this before. I’ve been deploying clustering for almost a > decade with Exchange and while I’ve had some issues, nothing like this. I’ve > even had a user remove the cluster object from AD and been able to recover > without a rebuild. Clustering is finicky – no question about it. But once > you set it up, you leave it alone, and it just WORKS.**** > > **** > > IMHO.**** > > **** > > Regards,**** > > **** > > Michael B. Smith**** > > Consultant and Exchange MVP**** > > http://TheEssentialExchange.com**** > > **** > > *From:* Maglinger, Paul [mailto:pmaglin...@scvl.com] > *Sent:* Thursday, September 29, 2011 10:16 AM > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > Wow… After being reassured about how reliable Exchange 2010 was, this is > making me step back a bit. I’ve even had people ask why we still needed > tape backup if we were using DAGs. I wasn’t ready to let go of my backups > of the databases just yet, but was willing to let go of the system states. > So if the config files hadn’t been copied to the other servers, would this > still have been a, pardon the phrase, “cluster-uh… mess”? I’d be interested > to hear other thoughts on this from the peanut gallery.**** > > **** > > **** > > *From:* Campbell, Rob [mailto:rob_campb...@centraltechnology.net] > *Sent:* Wednesday, September 28, 2011 5:40 PM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > I was on vacation when it happened. I can only relate what I came back > to. **** > > **** > > Apparently something bad happened to the cluster config on one of the > servers, and it was spread to the other servers, either through replication > or by attempts to correct it manually. PSS advised someone to copy the > config files from one server to all the rest, and that totally broke > everything. Then they said to do a system state restore of all the servers, > and it was discovered we didn’t have system state backups.**** > > **** > > I spent a very long night building a new server and copying DBs. After > that is was a “rolling rebuild” process, nuking the clustering on each one, > uninstalling Exchange, then re-installing and adding it back into a new > DAG.**** > > **** > > *From:* Michael B. Smith [mailto:mich...@smithcons.com] > *Sent:* Wednesday, September 28, 2011 5:31 PM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > I’d love to hear more about that. Care to share?**** > > **** > > Regards,**** > > **** > > Michael B. Smith**** > > Consultant and Exchange MVP**** > > http://TheEssentialExchange.com**** > > **** > > *From:* Campbell, Rob [mailto:rob_campb...@centraltechnology.net] > *Sent:* Wednesday, September 28, 2011 6:20 PM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > DAGs are nice, but they need to make the clustering more resilient. **** > > **** > > We just had to go through a process of completely tearing down and > rebuilding all 6 of our mailbox servers because the cluster configuration > got corrupted.**** > > **** > > *From:* Michael B. Smith [mailto:mich...@smithcons.com] > *Sent:* Wednesday, September 28, 2011 5:12 PM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > Yeah, that was the “best practice” preached prior to Exchange 2010 RTM and > at RTM.**** > > **** > > Today, SingleItemRecovery would be recommended instead.**** > > **** > > We’d still recommend a third DAG copy though. Preferably in a remote data > center. Just not lagged.**** > > **** > > Regards,**** > > **** > > Michael B. Smith**** > > Consultant and Exchange MVP**** > > http://TheEssentialExchange.com**** > > **** > > *From:* Maglinger, Paul [mailto:pmaglin...@scvl.com] > *Sent:* Wednesday, September 28, 2011 6:09 PM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > I was told in the Exchange 2010 class I told that you could have a 3 member > DAG, the third being lagged in case your database became corrupted and > replicated to the second DAG, in which case you could roll back. You would > have data loss, but at least you would have a recover point.**** > > **** > > *From:* Michael B. Smith [mailto:mich...@smithcons.com] > *Sent:* Wednesday, September 28, 2011 5:00 PM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > Please elucidate.**** > > **** > > Regards,**** > > **** > > Michael B. Smith**** > > Consultant and Exchange MVP**** > > http://TheEssentialExchange.com**** > > **** > > *From:* Maglinger, Paul [mailto:pmaglin...@scvl.com] > *Sent:* Wednesday, September 28, 2011 5:58 PM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > I was told to recover from a corrupt database.**** > > **** > > *From:* Michael B. Smith [mailto:mich...@smithcons.com] > *Sent:* Wednesday, September 28, 2011 4:41 PM > > *To:* MS-Exchange Admin Issues > *Subject:* RE: Backup of Exchange 2010 > **** > > **** > > Why on earth would you want to lag a DB? Have you ever tried recovering > from a lagged DB?**** > > **** > > Regards,**** > > **** > > Michael B. Smith**** > > Consultant and Exchange MVP**** > > http://TheEssentialExchange.com**** > > **** > > *From:* Jonathan Link [mailto:jonathan.l...@gmail.com] > *Sent:* Wednesday, September 28, 2011 5:13 PM > *To:* MS-Exchange Admin Issues > > *Subject:* Re: Backup of Exchange 2010 > **** > > **** > > Or can you have dbs offsite? And then can you lag those dbs?**** > > **** > > How frequent are those "oops, where'd that message go?" issues? And then, > on average, how far back do you have to go?**** > > On Wed, Sep 28, 2011 at 4:55 PM, Steve Ens <stevey...@gmail.com> wrote:*** > * > > Do you have your databases at different sites? **** > > **** > > On Wed, Sep 28, 2011 at 3:46 PM, Joseph Heaton <jhea...@dfg.ca.gov> wrote: > **** > > If I have my mailbox servers setup in a DAG, is the only reason to backup > to disk/tape to cover the "oops, where'd that message go?" issues? > > If you do backup to disk/tape, what type of retention do you have? And, > yes, I know the retention issue is extremely subjective, depending on legal > requirements, etc. Just curious. > > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > **** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > **** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > ****************************************************************************************************** > > Note: **** > > The information contained in this message may be privileged and confidential > and **** > > protected from disclosure. If the reader of this message is not the intended > **** > > recipient, or an employee or agent responsible for delivering this message to > **** > > the intended recipient, you are hereby notified that any dissemination, **** > > distribution or copying of this communication is strictly prohibited. If you > **** > > have received this communication in error, please notify us immediately by > **** > > replying to the message and deleting it from your computer. **** > > ****************************************************************************************************** > > --- > > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > ****************************************************************************************************** > > Note: **** > > The information contained in this message may be privileged and confidential > and **** > > protected from disclosure. If the reader of this message is not the intended > **** > > recipient, or an employee or agent responsible for delivering this message to > **** > > the intended recipient, you are hereby notified that any dissemination, **** > > distribution or copying of this communication is strictly prohibited. If you > **** > > have received this communication in error, please notify us immediately by > **** > > replying to the message and deleting it from your computer. **** > > ****************************************************************************************************** > > --- > > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > ****************************************************************************************************** > > Note: **** > > The information contained in this message may be privileged and confidential > and **** > > protected from disclosure. If the reader of this message is not the intended > **** > > recipient, or an employee or agent responsible for delivering this message to > **** > > the intended recipient, you are hereby notified that any dissemination, **** > > distribution or copying of this communication is strictly prohibited. If you > **** > > have received this communication in error, please notify us immediately by > **** > > replying to the message and deleting it from your computer. **** > > ****************************************************************************************************** > > --- > > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > ****************************************************************************************************** > > Note: **** > > The information contained in this message may be privileged and confidential > and **** > > protected from disclosure. If the reader of this message is not the intended > **** > > recipient, or an employee or agent responsible for delivering this message to > **** > > the intended recipient, you are hereby notified that any dissemination, **** > > distribution or copying of this communication is strictly prohibited. If you > **** > > have received this communication in error, please notify us immediately by > **** > > replying to the message and deleting it from your computer. **** > > ****************************************************************************************************** > > --- > > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > ** ** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist**** > > ************************************************************************************************** > Note: > The information contained in this message may be privileged and confidential > and > protected from disclosure. If the reader of this message is not the intended > recipient, or an employee or agent responsible for delivering this message to > the intended recipient, you are hereby notified that any dissemination, > distribution or copying of this communication is strictly prohibited. If you > have received this communication in error, please notify us immediately by > replying to the message and deleting it from your computer. > ************************************************************************************************** > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist > --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist
