Re: [Veritas-bu] NBU 6.5.1 restores miss files

[Mark Glazerman]

Travis,

Sorry for missing out that detail.

We initiated all of our restores through the GUI.  I just took a look at the 
tech note you mentioned and was certainly interested at what it talked about.  
This would certainly explain our issue.  

However, I followed the steps to see if there was a similar mismatch in our 
production environment.  I chose a directory with 8 files in it and initiated a 
restore of those files to a different "test" directory on the same server.  All 
8 files were restored.

I don't know if the problem mentioned in the tech note happens with EVERY 
restore or just randomly but this very limited test seems to imply that we 
don't have that problem here.

Thanks,

Mark Glazerman
Desk: 314-889-8282
Cell: 618-520-3401
 please don't print this e-mail unless you really need to


-Original Message-
From: Travis Kelley [mailto:travis.kel...@etrade.com] 
Sent: Tuesday, January 20, 2009 8:04 AM
To: Mark Glazerman
Cc: Veritas-bu@mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] NBU 6.5.1 restores miss files

You didn't mention how your performed the restore.  Bob already
suggested using the command line.  In case you are using the java admin
gui I'll point you over to a technote showing a problem we had where
files weren't restored.  I believe this only applies if you are using
the java admin gui.

http://seer.entsupport.symantec.com/docs/313460.htm


Mark Glazerman wrote:
> Len,
> 
>  
> 
> Thanks for your interest in our issue…
> 
>  
> 
> -  When you say that all your backup data resides on 2 Data
> Domain storage devices, I assume you mean just the backup media , tape
> or disk pool, and not the disk unit holding the restored data?
> 
>  
> 
> All backup data created by netbackup is sent straight to one of 2
> DataDomain restorers in our home data center.  That data is then
> replicated to identical DataDomain appliances in our DR hosting site in
> Philadelphia.  All of the images that we restore during either a DR
> exercise or a regular recovery of some kind come from the DataDomain
> appliances.  The DataDomain appliances are not used for primary storage
> or as target for any restored data.  All data was being restored to SAN
> attached disk on each host.
> 
>  
> 
> -  Did Netbackup report a failure to restore files? If so what
> reason was it reporting.
> 
>  
> 
> NetBackup did not report any errors during the restores.  Our signal
> that files / data was missing was that after restoring what we believed
> to be entire mountpoints, we would be short when compared to the amount
> of data in our production environments, sometimes by up to 10GB.
> 
>  
> 
> -  You talk about finding that missing files were found after
> the servers were turned over to your DBA’s. Do you mean missing files
> showed up without doing a restore? If so what file systems were you
> using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?
> 
>  
> 
> I didn’t word this very well.  While it was obvious data was missing if
> we were GB’s short, some files or binaries needed by our DBA’s for RMAN
> recoveries etc.. were not seen to be missing until they were unable to
> do what they needed to.  The files didn’t show up on their own, their
> absence was identified after the data restores had apparently completed
> successfully.  This was almost exclusively an issue with data being
> restored to unix boxes into both VxFS and ZFS file systems on SAN
> attached disk
> 
>  
> 
> -  So are you talking about a failure of Netbackup or
> server/filesystems or both?
> 
>  
> 
> I don’t think we’re talking of a failure of the filesystems and
> NetBackup didn’t fail completely.  Once a file or directory was
> identified as missing after a restore, we would always drill down in
> Netbackup to make sure that the file was actually there to be restored. 
> They always were and were which means that NetBackup is working from a
> backup point of view.  We were then able to successfully restore these
> specific files or directories if we drilled down to the specific file or
> sub directory.  In that respect, NBU didn’t fail from a restore point of
> view either because we were able to restore everything we wanted to,
> just not always the first time around !! 
> 
>  
> 
> It is indeed an interesting problem and thankfully was one that we were
> able to work around.  I haven’t raised this question with veritas yet
> but plan to do so if some more testing in our home data center generates
> similar results.
> 
>  
> 
> *Mark Glazerman*
> 
> Desk: 314-889-8282
> 
> Cell: 618-520-3401
> 
> P please don't print this e-mail unless you really need to
> 
>  
> 
> *F

Re: [Veritas-bu] NBU 6.5.1 restores miss files

[Travis Kelley]

You didn't mention how your performed the restore.  Bob already
suggested using the command line.  In case you are using the java admin
gui I'll point you over to a technote showing a problem we had where
files weren't restored.  I believe this only applies if you are using
the java admin gui.

http://seer.entsupport.symantec.com/docs/313460.htm


Mark Glazerman wrote:
> Len,
> 
>  
> 
> Thanks for your interest in our issue…
> 
>  
> 
> -  When you say that all your backup data resides on 2 Data
> Domain storage devices, I assume you mean just the backup media , tape
> or disk pool, and not the disk unit holding the restored data?
> 
>  
> 
> All backup data created by netbackup is sent straight to one of 2
> DataDomain restorers in our home data center.  That data is then
> replicated to identical DataDomain appliances in our DR hosting site in
> Philadelphia.  All of the images that we restore during either a DR
> exercise or a regular recovery of some kind come from the DataDomain
> appliances.  The DataDomain appliances are not used for primary storage
> or as target for any restored data.  All data was being restored to SAN
> attached disk on each host.
> 
>  
> 
> -  Did Netbackup report a failure to restore files? If so what
> reason was it reporting.
> 
>  
> 
> NetBackup did not report any errors during the restores.  Our signal
> that files / data was missing was that after restoring what we believed
> to be entire mountpoints, we would be short when compared to the amount
> of data in our production environments, sometimes by up to 10GB.
> 
>  
> 
> -  You talk about finding that missing files were found after
> the servers were turned over to your DBA’s. Do you mean missing files
> showed up without doing a restore? If so what file systems were you
> using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?
> 
>  
> 
> I didn’t word this very well.  While it was obvious data was missing if
> we were GB’s short, some files or binaries needed by our DBA’s for RMAN
> recoveries etc.. were not seen to be missing until they were unable to
> do what they needed to.  The files didn’t show up on their own, their
> absence was identified after the data restores had apparently completed
> successfully.  This was almost exclusively an issue with data being
> restored to unix boxes into both VxFS and ZFS file systems on SAN
> attached disk
> 
>  
> 
> -  So are you talking about a failure of Netbackup or
> server/filesystems or both?
> 
>  
> 
> I don’t think we’re talking of a failure of the filesystems and
> NetBackup didn’t fail completely.  Once a file or directory was
> identified as missing after a restore, we would always drill down in
> Netbackup to make sure that the file was actually there to be restored. 
> They always were and were which means that NetBackup is working from a
> backup point of view.  We were then able to successfully restore these
> specific files or directories if we drilled down to the specific file or
> sub directory.  In that respect, NBU didn’t fail from a restore point of
> view either because we were able to restore everything we wanted to,
> just not always the first time around !! 
> 
>  
> 
> It is indeed an interesting problem and thankfully was one that we were
> able to work around.  I haven’t raised this question with veritas yet
> but plan to do so if some more testing in our home data center generates
> similar results.
> 
>  
> 
> *Mark Glazerman*
> 
> Desk: 314-889-8282
> 
> Cell: 618-520-3401
> 
> P please don't print this e-mail unless you really need to
> 
>  
> 
> *From:* Len Boyle [mailto:len.bo...@sas.com]
> *Sent:* Monday, January 19, 2009 2:47 PM
> *To:* Mark Glazerman; Veritas-bu@mailman.eng.auburn.edu
> *Subject:* RE: NBU 6.5.1 restores miss files
> 
>  
> 
> Hello Mark,
> 
>  
> 
> I have a few questions about your report.
> 
>  
> 
> -  When you say that all your backup data resides on 2 Data
> Domain storage devices, I assume you mean just the backup media , tape
> or disk pool, and not the disk unit holding the restored data?
> 
> -  Did Netbackup report a failure to restore files? If so what
> reason was it reporting.
> 
> -  You talk about finding that missing files were found after
> the servers were turned over to your DBA’s. Do you mean missing files
> showed up without doing a restore? If so what file systems were you
> using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?
> 
> -  So are you talking about a failure of Netbackup or
> server/filesystems or both?
> 
>

Re: [Veritas-bu] NBU 6.5.1 restores miss files

[Mark Glazerman]

Len,

 

Thanks for your interest in our issue...

 

-  When you say that all your backup data resides on 2 Data
Domain storage devices, I assume you mean just the backup media , tape
or disk pool, and not the disk unit holding the restored data?

 

All backup data created by netbackup is sent straight to one of 2
DataDomain restorers in our home data center.  That data is then
replicated to identical DataDomain appliances in our DR hosting site in
Philadelphia.  All of the images that we restore during either a DR
exercise or a regular recovery of some kind come from the DataDomain
appliances.  The DataDomain appliances are not used for primary storage
or as target for any restored data.  All data was being restored to SAN
attached disk on each host.

 

-  Did Netbackup report a failure to restore files? If so what
reason was it reporting. 

 

NetBackup did not report any errors during the restores.  Our signal
that files / data was missing was that after restoring what we believed
to be entire mountpoints, we would be short when compared to the amount
of data in our production environments, sometimes by up to 10GB.

 

-  You talk about finding that missing files were found after
the servers were turned over to your DBA's. Do you mean missing files
showed up without doing a restore? If so what file systems were you
using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?

 

I didn't word this very well.  While it was obvious data was missing if
we were GB's short, some files or binaries needed by our DBA's for RMAN
recoveries etc.. were not seen to be missing until they were unable to
do what they needed to.  The files didn't show up on their own, their
absence was identified after the data restores had apparently completed
successfully.  This was almost exclusively an issue with data being
restored to unix boxes into both VxFS and ZFS file systems on SAN
attached disk

 

-  So are you talking about a failure of Netbackup or
server/filesystems or both?

 

I don't think we're talking of a failure of the filesystems and
NetBackup didn't fail completely.  Once a file or directory was
identified as missing after a restore, we would always drill down in
Netbackup to make sure that the file was actually there to be restored.
They always were and were which means that NetBackup is working from a
backup point of view.  We were then able to successfully restore these
specific files or directories if we drilled down to the specific file or
sub directory.  In that respect, NBU didn't fail from a restore point of
view either because we were able to restore everything we wanted to,
just not always the first time around !!  

 

It is indeed an interesting problem and thankfully was one that we were
able to work around.  I haven't raised this question with veritas yet
but plan to do so if some more testing in our home data center generates
similar results.

 

Mark Glazerman

Desk: 314-889-8282

Cell: 618-520-3401

P please don't print this e-mail unless you really need to

 

From: Len Boyle [mailto:len.bo...@sas.com] 
Sent: Monday, January 19, 2009 2:47 PM
To: Mark Glazerman; Veritas-bu@mailman.eng.auburn.edu
Subject: RE: NBU 6.5.1 restores miss files

 

Hello Mark, 

 

I have a few questions about your report. 

 

-  When you say that all your backup data resides on 2 Data
Domain storage devices, I assume you mean just the backup media , tape
or disk pool, and not the disk unit holding the restored data?

-  Did Netbackup report a failure to restore files? If so what
reason was it reporting. 

-  You talk about finding that missing files were found after
the servers were turned over to your DBA's. Do you mean missing files
showed up without doing a restore? If so what file systems were you
using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?

-  So are you talking about a failure of Netbackup or
server/filesystems or both?

 

A very interesting problem.

 

Thanks len

 

 

 

From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Mark
Glazerman
Sent: Monday, January 19, 2009 3:03 PM
To: Veritas-bu@mailman.eng.auburn.edu
Subject: [Veritas-bu] NBU 6.5.1 restores miss files

 

I've just got back to the office after a 24 DR test and wanted to get
this question out there before I forget.

 

We found during our testing that even if an entire directory tree was
selected to be restored, after the restore finished we would be missing
random files and directories.  Re-running the restore with the same
parameters would almost always lay down additional data which should
have been restored the first time around.  

 

Once we turned these servers over to our DBA's to run their RMAN
restores, we spent several hours completing additional requests for
individual files to be restored whi

Re: [Veritas-bu] NBU 6.5.1 restores miss files

[Len Boyle]

Hello Mark,

I have a few questions about your report.


-  When you say that all your backup data resides on 2 Data Domain 
storage devices, I assume you mean just the backup media , tape or disk pool, 
and not the disk unit holding the restored data?

-  Did Netbackup report a failure to restore files? If so what reason 
was it reporting.

-  You talk about finding that missing files were found after the 
servers were turned over to your DBA's. Do you mean missing files showed up 
without doing a restore? If so what file systems were you using? Windows 2003 
ntfs, solaris zfs, Linux ext3? Local disk or remote?

-  So are you talking about a failure of Netbackup or 
server/filesystems or both?

A very interesting problem.

Thanks len



From: veritas-bu-boun...@mailman.eng.auburn.edu 
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Mark Glazerman
Sent: Monday, January 19, 2009 3:03 PM
To: Veritas-bu@mailman.eng.auburn.edu
Subject: [Veritas-bu] NBU 6.5.1 restores miss files

I've just got back to the office after a 24 DR test and wanted to get this 
question out there before I forget.

We found during our testing that even if an entire directory tree was selected 
to be restored, after the restore finished we would be missing random files and 
directories.  Re-running the restore with the same parameters would almost 
always lay down additional data which should have been restored the first time 
around.

Once we turned these servers over to our DBA's to run their RMAN restores, we 
spent several hours completing additional requests for individual files to be 
restored which had been missed earlier on.  When we would drill down to check 
that these files existed we always found that they were there.

For reference, all of our backup data resides on 2 Data Domain storage devices 
so it isn't an issue with missing media.

Has anyone else seen anything like this ?

Thanks,

Mark Glazerman
Enterprise Storage Administrator
Spartech Corporation
Desk: 314-889-8282
Fax: 314-854-8282
Cell: 618-520-3401
mark.glazer...@spartech.com<mailto:mark.glazer...@spartech.com>
http://www.spartech.com<http://www.spartech.com/>
P please don't print this e-mail unless you really need to
This e-mail and any files transmitted with it are confidential, are intended 
solely for the use of the addressee, and may be legally privileged. If you have 
received this email in error please notify the sender immediately, and do not 
copy or forward it.

___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

Re: [Veritas-bu] NBU 6.5.1 restores miss files

[Stump, Bob A]

I suggest you use the command line to do the restores.

If you restore from the GUI then you have to select on the left side.

 

 

Bob Stump
Fidelity National Information Services
EBS - Storage Management - BACKUP
Mobile (918) 894-1782



From: veritas-bu-boun...@mailman.eng.auburn.edu
[mailto:veritas-bu-boun...@mailman.eng.auburn.edu] On Behalf Of Mark
Glazerman
Sent: Monday, January 19, 2009 2:03 PM
To: Veritas-bu@mailman.eng.auburn.edu
Subject: [Veritas-bu] NBU 6.5.1 restores miss files

 

I've just got back to the office after a 24 DR test and wanted to get
this question out there before I forget.

 

We found during our testing that even if an entire directory tree was
selected to be restored, after the restore finished we would be missing
random files and directories.  Re-running the restore with the same
parameters would almost always lay down additional data which should
have been restored the first time around.  

 

Once we turned these servers over to our DBA's to run their RMAN
restores, we spent several hours completing additional requests for
individual files to be restored which had been missed earlier on.  When
we would drill down to check that these files existed we always found
that they were there.

 

For reference, all of our backup data resides on 2 Data Domain storage
devices so it isn't an issue with missing media.

 

Has anyone else seen anything like this ? 

 

Thanks, 

 

Mark Glazerman

Enterprise Storage Administrator

Spartech Corporation

Desk: 314-889-8282

Fax: 314-854-8282

Cell: 618-520-3401

mark.glazer...@spartech.com <mailto:mark.glazer...@spartech.com> 

http://www.spartech.com <http://www.spartech.com/> 

P please don't print this e-mail unless you really need to

This e-mail and any files transmitted with it are confidential, are
intended solely for the use of the addressee, and may be legally
privileged. If you have received this email in error please notify the
sender immediately, and do not copy or forward it.

 

_

The information contained in this message is proprietary and/or confidential. 
If you are not the 
intended recipient, please: (i) delete the message and all copies; (ii) do not 
disclose, 
distribute or use the message in any manner; and (iii) notify the sender 
immediately. In addition, 
please be aware that any message addressed to our domain is subject to 
archiving and review by 
persons other than the intended recipient. Thank you.
_
___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

[Veritas-bu] NBU 6.5.1 restores miss files

[Mark Glazerman]

I've just got back to the office after a 24 DR test and wanted to get
this question out there before I forget.

 

We found during our testing that even if an entire directory tree was
selected to be restored, after the restore finished we would be missing
random files and directories.  Re-running the restore with the same
parameters would almost always lay down additional data which should
have been restored the first time around.  

 

Once we turned these servers over to our DBA's to run their RMAN
restores, we spent several hours completing additional requests for
individual files to be restored which had been missed earlier on.  When
we would drill down to check that these files existed we always found
that they were there.

 

For reference, all of our backup data resides on 2 Data Domain storage
devices so it isn't an issue with missing media.

 

Has anyone else seen anything like this ? 

 

Thanks, 

 

Mark Glazerman

Enterprise Storage Administrator

Spartech Corporation

Desk: 314-889-8282

Fax: 314-854-8282

Cell: 618-520-3401

mark.glazer...@spartech.com  

http://www.spartech.com  

P please don't print this e-mail unless you really need to

This e-mail and any files transmitted with it are confidential, are
intended solely for the use of the addressee, and may be legally
privileged. If you have received this email in error please notify the
sender immediately, and do not copy or forward it.

 

___
Veritas-bu maillist  -  Veritas-bu@mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu