[OpenAFS] recover data from corrupted volume

2013-01-16 Thread Dimitris Z
Hi,

I have been asked to attempt recovery an old soon to be decommissioned
openafs server. It is running openafs-server-1.4.7-68.2.SL4.x86_64 on
Scientific Linux 4.4.

The RAID10 array (3ware Inc 9550SX)  where vipepX partitions lie went
completely missing while replacing a faulty drive (ECC error). We put
the fauled drive back inside in order to be able to access the array
again and rsycned all the vicepX date elsewhere. This was completed
successfully. Errors logged meanwhile in the kernel logs mentioned
sectors being repaired.

The array was then deleted and remade with new disks and the data was
rsynced back. However some volumes have not come online. Salvaging
them causes them to deleted. These volumes have been off-line before
the disk replacement attemp - so it is likely they have been corrupted
for some time.

bash-3.2$ bos salvage -server afs1 -partition vicepe -volume uk  -showlog
Starting salvage.
bos: waiting for salvage to complete.
bos: salvage completed
SalvageLog:
@(#) OpenAFS 1.4.7 built 2009-04-07 (68.2@fnal.gov)
01/16/2013 17:14:45 STARTING AFS SALVAGER 2.4 (/usr/afs/bin/salvager
/vicepe 536874966)
01/16/2013 17:14:45 2 nVolumesInInodeFile 64
01/16/2013 17:14:45 CHECKING CLONED VOLUME 0.
01/16/2013 17:14:45 uk (536874966) updated 01/09/2013 17:13
01/16/2013 17:14:45 No header file for volume 0
01/16/2013 17:14:45 totalInodes 6446
-bash-3.2$ bos salvage -server afs1 -partition vicepe -volume uk  -showlog
Starting salvage.
bos: salvage completed
SalvageLog:
@(#) OpenAFS 1.4.7 built 2009-04-07 (68.2@fnal.gov)
01/16/2013 17:15:22 STARTING AFS SALVAGER 2.4 (/usr/afs/bin/salvager
/vicepe 536874966)
01/16/2013 17:15:22 The volume header file V0536874966.vol is not
associated with any actual data (deleted)
01/16/2013 17:15:22 No applicable vice inodes on vicepe; not salvaged
01/16/2013 17:15:22 0 nVolumesInInodeFile 0
Temporary file /vicepe/salvage.inodes.vicepe.16945 is missing...

/usr/sbin/vos examine uk
Could not fetch the information about volume 536874966 from the server
: No such device
Volume does not exist on server afs1 as indicated by the VLDB

Dump only information from VLDB

uk
RWrite: 536874966
number of sites - 1
   server afs1 partition /vicepe RW Site


My question is, are there any other options to at least partially
recover data from the offline volumes? I do have my last rsync of
vicepe where this volume was. It was probably damaged before it was
backed up but even a partial data recovery would help.

Best regards,
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info


Re: [OpenAFS] recover data from corrupted volume

2013-01-16 Thread Derrick Brashear
On Wed, Jan 16, 2013 at 1:11 PM, Stephan Wiesand
stephan.wies...@desy.de wrote:
 Hi Dmitris,

 On Jan 16, 2013, at 18:25 , Dimitris Z wrote:

 Hi,

 I have been asked to attempt recovery an old soon to be decommissioned
 openafs server. It is running openafs-server-1.4.7-68.2.SL4.x86_64 on

 that was a fine release at its time. For the wizards: on top of the 1.4.7 
 release, it had these patches from CVS applied:

 linux-hlist-unhashed-opencoding-20080520
 butc-xbsa-lwp-protoize-damage-20080501
 uuid-corrected-duplicate-check-20080501
 viced-large-more-threads-20080506
 vos-sync-flag-voltype-properly-20080521

 + backports of the patches to address OPENAFS-SA-2009-001 and -002 (affecting 
 the client only). Thus, not too exotic.

 Scientific Linux 4.4.

 The RAID10 array (3ware Inc 9550SX)  where vipepX partitions lie went
 completely missing while replacing a faulty drive (ECC error). We put
 the fauled drive back inside in order to be able to access the array
 again and rsycned all the vicepX date elsewhere. This was completed
 successfully. Errors logged meanwhile in the kernel logs mentioned
 sectors being repaired.

 The array was then deleted and remade with new disks and the data was
 rsynced back. However some volumes have not come online. Salvaging
 them causes them to deleted. These volumes have been off-line before
 the disk replacement attemp - so it is likely they have been corrupted
 for some time.

 bash-3.2$ bos salvage -server afs1 -partition vicepe -volume uk  -showlog
 Starting salvage.
 bos: waiting for salvage to complete.
 bos: salvage completed
 SalvageLog:
 @(#) OpenAFS 1.4.7 built 2009-04-07 (68.2@fnal.gov)
 01/16/2013 17:14:45 STARTING AFS SALVAGER 2.4 (/usr/afs/bin/salvager
 /vicepe 536874966)
 01/16/2013 17:14:45 2 nVolumesInInodeFile 64
 01/16/2013 17:14:45 CHECKING CLONED VOLUME 0.
 01/16/2013 17:14:45 uk (536874966) updated 01/09/2013 17:13
 01/16/2013 17:14:45 No header file for volume 0
 01/16/2013 17:14:45 totalInodes 6446
 -bash-3.2$ bos salvage -server afs1 -partition vicepe -volume uk  -showlog
 Starting salvage.
 bos: salvage completed
 SalvageLog:
 @(#) OpenAFS 1.4.7 built 2009-04-07 (68.2@fnal.gov)
 01/16/2013 17:15:22 STARTING AFS SALVAGER 2.4 (/usr/afs/bin/salvager
 /vicepe 536874966)
 01/16/2013 17:15:22 The volume header file V0536874966.vol is not
 associated with any actual data (deleted)
 01/16/2013 17:15:22 No applicable vice inodes on vicepe; not salvaged
 01/16/2013 17:15:22 0 nVolumesInInodeFile 0
 Temporary file /vicepe/salvage.inodes.vicepe.16945 is missing...

 /usr/sbin/vos examine uk
 Could not fetch the information about volume 536874966 from the server
 : No such device
 Volume does not exist on server afs1 as indicated by the VLDB

 Dump only information from VLDB

 uk
RWrite: 536874966
number of sites - 1
   server afs1 partition /vicepe RW Site


 My question is, are there any other options to at least partially
 recover data from the offline volumes? I do have my last rsync of
 vicepe where this volume was. It was probably damaged before it was
 backed up but even a partial data recovery would help.

 If there are remnants left, you should find them in AFSIDat/K1/Kz++U . If you 
 can't find that directory, look for the files named 5Mx1++0, 9Mx1++0, 
 DMx1++0, PMx1++0, possibly in lost+found. If you can't find any of 
 those, I'm out of ideas/hope.

The volinfo tool's saveinodes option can also be used for this.


-- 
Derrick
___
OpenAFS-info mailing list
OpenAFS-info@openafs.org
https://lists.openafs.org/mailman/listinfo/openafs-info