Re: URGENT - Exchange 2003 database on D: deleted!
Oh my god, you have been so unlucky /lucky with this server. I peronally wouldnt breathe again till management had sprung for a new Server and youd moved every bodys mailboxes to the new server and decommissioned that one. I thin kit would always be at the back of my mind that you were gonna get that call again. I'm with you i thought deleting an array killed the data, its good to know it doesnt but its not something I would want to have to try out. Well done to the HP tech or staying with you. And well one you for surviving an eventful weekend. Graeme On 28 February 2012 14:54, David Mazzaccaro < david.mazzacc...@hudsonmobility.com> wrote: > /Need to be sure that whatever caused your raid 5 to go is fixed./ > > ** ** > > - Yes… this couldn’t be more true. The exact problem resurfaced on Sunday > night. I almost lost it. > > All 4 hard drives that make up my raid5 array (logical D drive) went red. > Logical drive D was gone. Poof. > > There is no way all 4 hard drives were bad.. I just can’t be that unlucky. > > > I got an HP level 2 support person to tell me “with 100% confidence… this > is rare, but does happen if the array itself has bad parity or is corrupt”. > > > He said no hardware is the problem… it is the actual array config. I > asked if he has seen this EXACT problem/solution first hand, and he said > yes. > > He said to simply delete the array and re-create it, and restore data and > you will be fine. Again w/ “100 confidence” – I have never heard HP > support make that kind of commitment. > > ** ** > > Here’s the kicker… I (like most people) assume that deleting an array is a > data-destructive procedure. > > Apparently, it is not. The onsite HP tech told me he has done this > hundreds of times and the data is always there. > > This is a DL380 G4 (7 years old). So I said I want all new hardware (sys > board, array controllers, BBWC, etc). > > The hp tech agrees to be safe, let’s replace hardware. So we do. And > then we open Array Configuration Utility, delete the array, and then > re-create the exact same array (this is the important part). > > You have to re-create the same array config you had (RAID5 w/ spare). > > When we did that, and rebooted, Windows now see a logical drive D and all > the folders are there (w/ Exchange EDB and STM files). > > ** ** > > HOWEVER – this is where it got bad/interesting/a learning experience…. > > Basically HP’s responsibility ends here. And I expected HP tech to leave > my site. But, when Exchange wouldn’t mount the store – he felt bad for me > and stayed until I was on a path to recovery. > > Being an IT department of 1 for 180 users, I really appreciated this. > > So, he stayed with me while I called Microsoft and worked with them to get > my Exchange working. > > Because the logical drive D died while Exchange was running, the Exchange > DB was now corrupt. > > How corrupt? Apparently much more corrupt than MS thought. > > So I explain to MS that the drive failed that contained Exchange DB, and > they say no problem, we can just run Eseutil /p (I believe) to repair it… > ok… go ahead…. > > This turned out to be a huge mistake… I should have said no… let’s just > pull the database from last night’s backup. > > This “repair” ran … and ran.. and ran… After several hours, (and google > telling me that some people have had this run for DAYS… I told them to > cancel it, restore from last night backup. > > There is very little if any data in that database between the time we > thought it was fixed and now. > > MS agreed, and recovered using Recovery Storage Group, Eseutil /CC… , > moved to First Storage Group, then recovered the public folder store to > First Storage Group (you can’t recover pub1.edb to a RSG) Eseutil /CC… etc… > and viola! Email! Calendars! Blackberries! Public folders! I spent all > night send/receiving test messages to myself to ensure it was running. > > ... At what point can I start to breathe again?**** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > ** ** > > *From:* Graeme Carstairs [mailto:loonyto...@gmail.com] > *Sent:* Sunday, February 26, 2012 9:42 AM > > *To:* MS-Exchange Admin Issues > *Subject:* Re: URGENT - Exchange 2003 database on D: deleted! > > ** ** > > Hi David, > > ** ** > > This is a restore up new disks from backup. > > ** ** > > Need to be sure that whatever caused your raid 5 to go is fixed. > > ** ** > > Google exchange restore
RE: URGENT - Exchange 2003 database on D: deleted!
Almost as scary as the horror story I experienced It was a small company with no money to invest in IT and I had just been hired to try and get some order back in support and let the IT manager get back to DB development which was his area of expertise. I hadn't used exchange before (or even AD, but I was an experienced Novell admin and skills were pretty transferable back then...) From what I remember the server was exchange 2000 and kept running out of space. At the time I had no idea that there were no backups on any of the servers... One morning, not long after I joined exchange ran out of space again and stopped working. The director jumped onto the box and told me he was deleting some transaction logs. I didn't think too much of this until about 5 minutes passed and I looked over and he had turned white and looked like he was going to pass out. Turns out he had just deleted everything from the log directory, but also the database and streaming database as they had all been put in the same place. All too big for the recycle bin too Long story short, after a couple of thousand pounds and many days work we managed to recover nothing of use and lost over 5 years' worth of email for the whole company! Somehow we both kept our jobs too, no idea how. I've never worked for anywhere that doesn't put a priority on backups since J From: Sobey, Richard A [mailto:r.so...@imperial.ac.uk] Sent: 27 February 2012 09:09 To: MS-Exchange Admin Issues Subject: RE: URGENT - Exchange 2003 database on D: deleted! It's one of the scariest things that can happen to be honest. Hope you're still drinking that beer :) From: bounce-9491670-8066...@lyris.sunbelt-software.com [mailto:bounce-9491670-8066...@lyris.sunbelt-software.com] On Behalf Of David Mazzaccaro Sent: 26 February 2012 20:40 To: MS-Exchange Admin Issues Subject: RE: URGENT - Exchange 2003 database on D: deleted! Thx! He had to create blank DBs - BackupExec needed to see "something" there to restore over. That and the little problem of D: not being shared (D$) after HP recreated the array. After 12 hours yesterday, and another 9 hours today...I think a single beer would kill me (but I'm willing to find out) LOL I'd like to say that I am very grateful for the help I get on this list (and on a weekend nonetheless). My meager participation has not even come close to all the help I have received over the years. Thank you. From: Graeme Carstairs [mailto:loonyto...@gmail.com]<mailto:[mailto:loonyto...@gmail.com]> Sent: Sunday, February 26, 2012 3:30 PM To: MS-Exchange Admin Issues Subject: Re: URGENT - Exchange 2003 database on D: deleted! no probs, Glad it worked out. Just create new db's and restore into them? Pretty sure thats basically what you do but it's hard to say by email. Just thought I'd try and put your kind at ease that as long as your backups sound you'll be ok. Glad it worked have a beer you deserve it Graeme On Sunday, 26 February 2012, Graeme Carstairs wrote: When exchange is up its a tick box, but look at your log files if you have files dated back to your last backup and over approx 5Mbs worth then it's not enabled and you should have logs Formosa every transaction that's occurred since last backup. If you are that nervous and it's your production system I would work with ms, especially of your company will sign off on the cost. It's really just 2 things is your backup sound, and is circular logging off. If Backup sound your back to where the backup started. If circular logging disable your back to where you crashed. Another tip if you can copy the logs somewhere do so befor you start as with the logs and backup your sorted. Graeme On Sunday, 26 February 2012, David Mazzaccaro wrote: Thank you for the reply. On the phone w/ Microsoft right now. I am really nervous, have no idea what is going on. How do I tell if I have circular logging enabled? --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe exchangelist -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe exchangelist . --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbelt
RE: URGENT - Exchange 2003 database on D: deleted!
It's one of the scariest things that can happen to be honest. Hope you're still drinking that beer :) From: bounce-9491670-8066...@lyris.sunbelt-software.com [mailto:bounce-9491670-8066...@lyris.sunbelt-software.com] On Behalf Of David Mazzaccaro Sent: 26 February 2012 20:40 To: MS-Exchange Admin Issues Subject: RE: URGENT - Exchange 2003 database on D: deleted! Thx! He had to create blank DBs - BackupExec needed to see "something" there to restore over. That and the little problem of D: not being shared (D$) after HP recreated the array. After 12 hours yesterday, and another 9 hours today...I think a single beer would kill me (but I'm willing to find out) LOL I'd like to say that I am very grateful for the help I get on this list (and on a weekend nonetheless). My meager participation has not even come close to all the help I have received over the years. Thank you. From: Graeme Carstairs [mailto:loonyto...@gmail.com]<mailto:[mailto:loonyto...@gmail.com]> Sent: Sunday, February 26, 2012 3:30 PM To: MS-Exchange Admin Issues Subject: Re: URGENT - Exchange 2003 database on D: deleted! no probs, Glad it worked out. Just create new db's and restore into them? Pretty sure thats basically what you do but it's hard to say by email. Just thought I'd try and put your kind at ease that as long as your backups sound you'll be ok. Glad it worked have a beer you deserve it Graeme On Sunday, 26 February 2012, Graeme Carstairs wrote: When exchange is up its a tick box, but look at your log files if you have files dated back to your last backup and over approx 5Mbs worth then it's not enabled and you should have logs Formosa every transaction that's occurred since last backup. If you are that nervous and it's your production system I would work with ms, especially of your company will sign off on the cost. It's really just 2 things is your backup sound, and is circular logging off. If Backup sound your back to where the backup started. If circular logging disable your back to where you crashed. Another tip if you can copy the logs somewhere do so befor you start as with the logs and backup your sorted. Graeme On Sunday, 26 February 2012, David Mazzaccaro wrote: Thank you for the reply. On the phone w/ Microsoft right now. I am really nervous, have no idea what is going on. How do I tell if I have circular logging enabled? --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe exchangelist -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe exchangelist . --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com<mailto:listmana...@lyris.sunbeltsoftware.com> with the body: unsubscribe exchangelist --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist
RE: URGENT - Exchange 2003 database on D: deleted!
Thx! He had to create blank DBs - BackupExec needed to see "something" there to restore over. That and the little problem of D: not being shared (D$) after HP recreated the array. After 12 hours yesterday, and another 9 hours today...I think a single beer would kill me (but I'm willing to find out) LOL I'd like to say that I am very grateful for the help I get on this list (and on a weekend nonetheless). My meager participation has not even come close to all the help I have received over the years. Thank you. From: Graeme Carstairs [mailto:loonyto...@gmail.com] Sent: Sunday, February 26, 2012 3:30 PM To: MS-Exchange Admin Issues Subject: Re: URGENT - Exchange 2003 database on D: deleted! no probs, Glad it worked out. Just create new db's and restore into them? Pretty sure thats basically what you do but it's hard to say by email. Just thought I'd try and put your kind at ease that as long as your backups sound you'll be ok. Glad it worked have a beer you deserve it Graeme On Sunday, 26 February 2012, Graeme Carstairs wrote: When exchange is up its a tick box, but look at your log files if you have files dated back to your last backup and over approx 5Mbs worth then it's not enabled and you should have logs Formosa every transaction that's occurred since last backup. If you are that nervous and it's your production system I would work with ms, especially of your company will sign off on the cost. It's really just 2 things is your backup sound, and is circular logging off. If Backup sound your back to where the backup started. If circular logging disable your back to where you crashed. Another tip if you can copy the logs somewhere do so befor you start as with the logs and backup your sorted. Graeme On Sunday, 26 February 2012, David Mazzaccaro wrote: Thank you for the reply. On the phone w/ Microsoft right now. I am really nervous, have no idea what is going on. How do I tell if I have circular logging enabled? --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist . --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist
Re: URGENT - Exchange 2003 database on D: deleted!
no probs, Glad it worked out. Just create new db's and restore into them? Pretty sure thats basically what you do but it's hard to say by email. Just thought I'd try and put your kind at ease that as long as your backups sound you'll be ok. Glad it worked have a beer you deserve it Graeme On Sunday, 26 February 2012, Graeme Carstairs wrote: > When exchange is up its a tick box, but look at your log files if you have > files dated back to your last backup and over approx 5Mbs worth then it's > not enabled and you should have logs Formosa every transaction that's > occurred since last backup. > If you are that nervous and it's your production system I would work with > ms, especially of your company will sign off on the cost. > It's really just 2 things is your backup sound, and is circular logging > off. > If Backup sound your back to where the backup started. If circular logging > disable your back to where you crashed. > > Another tip if you can copy the logs somewhere do so befor you start as > with the logs and backup your sorted. > > Graeme > > > On Sunday, 26 February 2012, David Mazzaccaro wrote: > >> Thank you for the reply. >> On the phone w/ Microsoft right now. >> I am really nervous, have no idea what is going on. >> How do I tell if I have circular logging enabled? >> >> --- >> To manage subscriptions click here: >> http://lyris.sunbelt-software.com/read/my_forums/ >> or send an email to listmana...@lyris.sunbeltsoftware.com >> with the body: unsubscribe exchangelist >> > > > -- > Good news everyone, you have just received an e-mail from me! > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com 'cvml', 'listmana...@lyris.sunbeltsoftware.com');> > with the body: unsubscribe exchangelist > -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist
RE: URGENT - Exchange 2003 database on D: deleted!
Thanks for the replies today. MS was able to get everything working again. Didn't have to create a Recovery Storage Group or dial-tone method. Long two days, but I seem to be back in business. From: Graeme Carstairs [mailto:loonyto...@gmail.com] Sent: Sunday, February 26, 2012 9:42 AM To: MS-Exchange Admin Issues Subject: Re: URGENT - Exchange 2003 database on D: deleted! Hi David, This is a restore up new disks from backup. Need to be sure that whatever caused your raid 5 to go is fixed. Google exchange restore and follow the steps either Symantec or best microsofts but reference symantecs as the nuances with be As long as your backup is sound and you aren't using circular logging you should be able to get back to as was prior to crash. Also make sure you stop incoming smtp, owa etc at the gateway. This is what your backups for so just take your time work through the ms restore guides along with symantecs and you'll be good Graeme On Sunday, 26 February 2012, David Mazzaccaro wrote: I am really in a bind here. Our Exchange server (2003 SP2) had a raid5 failure (2 drives died) on the array that contained the information store (d: drive) It was totally wiped out and is now empty. Exchange was installed to c: along with the transaction logs being on c: and c is fine. I can load windows and open exchange system manager. But the database (public and private) was on the d: (which is now just an empty drive) I do have a support call into Microsoft to try to help me restore from friday's backup of the information store. Done on BackupExec 10d. Waiting for a call back from MS. Have u guys ever had to deal w/ something like this? I have no idea where to even start. --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist . --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist
Re: URGENT - Exchange 2003 database on D: deleted!
When exchange is up its a tick box, but look at your log files if you have files dated back to your last backup and over approx 5Mbs worth then it's not enabled and you should have logs for every transaction that's occurred since last backup. If you are that nervous and it's your production system I would work with ms, especially of your company will sign off on the cost. It's really just 2 things is your backup sound, and is circular logging off. If Backup sound your back to where the backup started. If circular logging disable your back to where you crashed. Another tip if you can copy the logs somewhere do so befor you start as with the logs and backup your sorted. Graeme On Sunday, 26 February 2012, David Mazzaccaro wrote: > Thank you for the reply. > On the phone w/ Microsoft right now. > I am really nervous, have no idea what is going on. > How do I tell if I have circular logging enabled? > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist > -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist
Re: URGENT - Exchange 2003 database on D: deleted!
Thank you for the reply. On the phone w/ Microsoft right now. I am really nervous, have no idea what is going on. How do I tell if I have circular logging enabled? --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist
Re: URGENT - Exchange 2003 database on D: deleted!
Hi David, This is a restore up new disks from backup. Need to be sure that whatever caused your raid 5 to go is fixed. Google exchange restore and follow the steps either Symantec or best microsofts but reference symantecs as the nuances with be As long as your backup is sound and you aren't using circular logging you should be able to get back to as was prior to crash. Also make sure you stop incoming smtp, owa etc at the gateway. This is what your backups for so just take your time work through the ms restore guides along with symantecs and you'll be good Graeme On Sunday, 26 February 2012, David Mazzaccaro wrote: > I am really in a bind here. > Our Exchange server (2003 SP2) had a raid5 failure (2 drives died) on the > array that contained the information store (d: drive) > It was totally wiped out and is now empty. > > Exchange was installed to c: along with the transaction logs being on c: > and c is fine. > I can load windows and open exchange system manager. > > But the database (public and private) was on the d: (which is now just an > empty drive) > > I do have a support call into Microsoft to try to help me restore from > friday's backup of the information store. Done on BackupExec 10d. Waiting > for a call back from MS. > > Have u guys ever had to deal w/ something like this? > I have no idea where to even start. > > > --- > To manage subscriptions click here: > http://lyris.sunbelt-software.com/read/my_forums/ > or send an email to listmana...@lyris.sunbeltsoftware.com > with the body: unsubscribe exchangelist > -- Good news everyone, you have just received an e-mail from me! --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to listmana...@lyris.sunbeltsoftware.com with the body: unsubscribe exchangelist
