Re: [OpenAFS] LiveCD for openafs disaster recovery scenarios ?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Madhusudan Singh wrote: Hello At least one (and possibly two) hard disks on the machine (has RAID 5 - rebuild did not help too much) that runs our openafs cell failed today. This is a Debian machine. Partition /usr is down and attempts to mount it fail with bad superblock messages. As a result, all network and many other system commands (awk, etc.) are inaccessible. Now, is there a live CD that can be used to run the machine so that the users can at least copy over their data ? I think a reinstall is indicated. www.grml.org - it has OpenAFS and krb5 built in. The two partitions housing /vicepa and /vicepb are not affected. Is there a way to retrieve data directly from the hard disks ? yeah, sure, there are a lots of ways... Thanks. MfG, Lars Schimmer - -- - - TU Graz, Institut für ComputerGraphik WissensVisualisierung Tel: +43 316 873-5405 E-Mail: [EMAIL PROTECTED] Fax: +43 316 873-5402 PGP-Key-ID: 0x4A9B1723 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFUaUEmWhuE0qbFyMRAhUrAJwIPJTsThBqLYxX6H2NrFfJy5/UBgCfRX7t 38PE6jxc5TvFpcj5r8dCFvc= =nfpX -END PGP SIGNATURE- ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] LiveCD for openafs disaster recovery scenarios ?
Madhusudan Singh wrote: Hello At least one (and possibly two) hard disks on the machine (has RAID 5 - rebuild did not help too much) that runs our openafs cell failed today. This is a Debian machine. Partition /usr is down and attempts to mount it fail with bad superblock messages. As a result, all network and many other system commands (awk, etc.) are inaccessible. Now, is there a live CD that can be used to run the machine so that the users can at least copy over their data ? I think a reinstall is indicated. The two partitions housing /vicepa and /vicepb are not affected. Is there a way to retrieve data directly from the hard disks ? Thanks. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info Can you reinstall the OS and leave /vicepa and /vicepb alone during the install? Then once you are back in business install openafs , mount /vicpea /vicepb and away you go? We do this often however our partitions are on attached jbods so we swap out servers or rebuild the server from an install server but its basically the same thing. It works as long as the the server hostname remains the same. /sd -- Steve Devine Storage Systems Academic Computing Network Services Michigan State University 506 Computer Center East Lansing, MI 48824-1042 1-517-432-7327 Baseball is ninety percent mental; the other half is physical. - Yogi Berra ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] LiveCD for openafs disaster recovery scenarios ?
I have a suggestion... if you have another machine with the same O/S, export the /usr partition via NFS readonly. Change your fstab on the damaged host to mount /usr from the NFS server host. It should work, similar to the way diskless booting works. Once the machine is up and running, you could just move the volumes off. Make sure the other machine has the same version of openafs installed, or at least make sure your files are consistent between what's on /usr and what's in your / partition. Boot off a rescue CD to update your fstab file. Good luck! Mike Madhusudan Singh wrote: Hello At least one (and possibly two) hard disks on the machine (has RAID 5 - rebuild did not help too much) that runs our openafs cell failed today. This is a Debian machine. Partition /usr is down and attempts to mount it fail with bad superblock messages. As a result, all network and many other system commands (awk, etc.) are inaccessible. Now, is there a live CD that can be used to run the machine so that the users can at least copy over their data ? I think a reinstall is indicated. The two partitions housing /vicepa and /vicepb are not affected. Is there a way to retrieve data directly from the hard disks ? Thanks. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info -- Michael Polek Pictage, Inc. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
[OpenAFS] LiveCD for openafs disaster recovery scenarios ?
Hello At least one (and possibly two) hard disks on the machine (has RAID 5 - rebuild did not help too much) that runs our openafs cell failed today. This is a Debian machine. Partition /usr is down and attempts to mount it fail with bad superblock messages. As a result, all network and many other system commands (awk, etc.) are inaccessible. Now, is there a live CD that can be used to run the machine so that the users can at least copy over their data ? I think a reinstall is indicated. The two partitions housing /vicepa and /vicepb are not affected. Is there a way to retrieve data directly from the hard disks ? Thanks. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery
Dimitris Zilaskos wrote: Thnank you for your tips, So far i was able to either restore from the backup or salvage most of the data. However I would like to recover some of the most recent bits that are currently lying in lost+found. Is there a way? We do have ro copies of all volumes. Unfortunately vos release stopped working about a month ago. Possibly due to the same reasons that resulted to yesterday's disaster. Hello! I had to do this kind of work about two years ago. Take a look at this: http://archiv.tu-chemnitz.de/pub/2003/0130/data/vol/start.htm I found it very helpful. It's in German, but perhaps someone can translate it for you. With the information provided in the document, the tools mentioned in the text, and some cautious looks at the openafs header files, I was able to reconstruct some of the fragments in lost+found. For almost complete volumes structures (small and large vnode index files and large vnodes must be available) I can provide you with a script that restores the volume directory structure and files to another location in 'real' filespace. Filenames are translated to the 'real' filenames. For corrupted or missing files, you will get some informations and the names, so you can try to find them in the remaining fragments. The script also has a 'dry run' mode. With the additional informations from your old ro volumes, you might be able to reconstruct complete volumes. But it's tricky and you will need some time. The script and some of my other tools are a 'first try' and the comments are german. I never had the time to rewrite the source. And I'm not a very experienced programmer Berthold Cogel -- Dipl. Chem. Dr. Berthold Cogel University of Cologne E-Mail: [EMAIL PROTECTED] ZAIK/RRZK Tel.: +49(0)221/470-7873 Robert-Koch-Str. 10 FAX:+49(0)221/478-85845D-50931 Cologne - Germany ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery
http://archiv.tu-chemnitz.de/pub/2003/0130/data/vol/start.htm I found it very helpful. It's in German, but perhaps someone can translate it for you. With the information provided in the document, the tools mentioned in the text, and some cautious looks at the openafs header files, I was able to reconstruct some of the fragments in lost+found. Thnx, I will try to find someone to help with this. For almost complete volumes structures (small and large vnode index files and large vnodes must be available) I can provide you with a script that restores the volume directory structure and files to another location in 'real' filespace. Filenames are translated to the 'real' filenames. For corrupted or missing files, you will get some informations and the names, so you can try to find them in the remaining fragments. The script also has a 'dry run' mode. Please share it, thnx:) With the additional informations from your old ro volumes, you might be able to reconstruct complete volumes. But it's tricky and you will need some time. The script and some of my other tools are a 'first try' and the comments are german. I never had the time to rewrite the source. And I'm not a very experienced programmer It's ok, at this moment I am not missing anything critical. However it should be a good drill for the future. -- Dimitris Zilaskos Department of Physics @ Aristotle University of Thessaloniki , Greece PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc MD5sum : de2bd8f73d545f0e4caf3096894ad83f pgp_public_key.asc ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery
Dimitris Zilaskos wrote: Hi, One 1.3.87 linux fileserver died today. After a reboot, the filesystem check on vicepa spitted out numerous errors, it fixed them filling lost+found with data, and then after salavage I ended up with half the volumes missing or corrupted. I had one backup a few days old which I used to restore the volumes. I also have a copy of the /vicepa contents from yesterday, when the server started to behave strangely. Is there a way to use the /vicepa contents in order to access certain files/directories? Unfortunately I do not have a copy of the db files. The db-files do not matter. If you have a copy of your /vicepa with correct modebits, ownership, and group settings for the files you may use this instead of your old /vicepa. It is possible tar/untar vicep-partitions and to use them after that again. If you do that on another fileserver you should stop the corrupted one, start the new one and do a vos syncvldb newserver in order to update the volume location database. This will overwrite the location of each volume found on the new server. If this doesn't work try vos syncserv newserver (I never understood which one of those does what, but one of them does the job). You need then probably a fs checkvol on the client that he gets the new location. You should also think about having RO-volumes of your RW-volumes on other servers in the future. Then you easily can do a vos convertROtoRW ... to get again working cell. Good luck! Hartmut Cheers, -- Dimitris Zilaskos Department of Physics @ Aristotle University of Thessaloniki , Greece PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc MD5sum : de2bd8f73d545f0e4caf3096894ad83f pgp_public_key.asc ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info -- - Hartmut Reuter e-mail [EMAIL PROTECTED] phone +49-89-3299-1328 RZG (Rechenzentrum Garching) fax +49-89-3299-1301 Computing Center of the Max-Planck-Gesellschaft (MPG) and the Institut fuer Plasmaphysik (IPP) - ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery
Thnank you for your tips, So far i was able to either restore from the backup or salvage most of the data. However I would like to recover some of the most recent bits that are currently lying in lost+found. Is there a way? We do have ro copies of all volumes. Unfortunately vos release stopped working about a month ago. Possibly due to the same reasons that resulted to yesterday's disaster. If you have a copy of your /vicepa with correct modebits, ownership, and group settings for the files you may use this instead of your old /vicepa. It is possible tar/untar vicep-partitions and to use them after that again. If you do that on another fileserver you should stop the corrupted one, start the new one and do a vos syncvldb newserver in order to update the volume location database. This will overwrite the location of each volume found on the new server. If this doesn't work try vos syncserv newserver (I never understood which one of those does what, but one of them does the job). You need then probably a fs checkvol on the client that he gets the new location. You should also think about having RO-volumes of your RW-volumes on other servers in the future. Then you easily can do a vos convertROtoRW ... to get again working cell. Cheers, -- Dimitris Zilaskos Department of Physics @ Aristotle University of Thessaloniki , Greece PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc MD5sum : de2bd8f73d545f0e4caf3096894ad83f pgp_public_key.asc ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
[OpenAFS] disaster recovery
Hi, One 1.3.87 linux fileserver died today. After a reboot, the filesystem check on vicepa spitted out numerous errors, it fixed them filling lost+found with data, and then after salavage I ended up with half the volumes missing or corrupted. I had one backup a few days old which I used to restore the volumes. I also have a copy of the /vicepa contents from yesterday, when the server started to behave strangely. Is there a way to use the /vicepa contents in order to access certain files/directories? Unfortunately I do not have a copy of the db files. Cheers, -- Dimitris Zilaskos Department of Physics @ Aristotle University of Thessaloniki , Greece PGP key : http://tassadar.physics.auth.gr/~dzila/pgp_public_key.asc http://egnatia.ee.auth.gr/~dzila/pgp_public_key.asc MD5sum : de2bd8f73d545f0e4caf3096894ad83f pgp_public_key.asc ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery, help needed.
On Mon, 24 Apr 2006 09:05:11 +0200, Horst Birthelmer [EMAIL PROTECTED] said: On Apr 24, 2006, at 3:42 AM, ph rhole oper wrote: On Sun, 23 Apr 2006 22:20:19 +0200 (CEST), Chris Huebsch [EMAIL PROTECTED] said: Hello, On Sun, 23 Apr 2006, ph rhole oper wrote: The hd holding the root directory of an afs server has crashed, thus the /var/openafs/ files are lost with it, Was the crashed server only a fileserver or both fileserver and databaseserver? If the latter, was it your only databaseserver? It was a db server too, but there is one more left online.I think that if i install a new OS, mount /vicepa , and start afs services, ubik will synchronize and the new OS's afs will just bring the /vicepa volumes up.Right? The DB server cold make some trouble on resynchronization with an empty database (It's really no big deal) Whenever i need to install a new db server on the network, i just start it with an empty database and it synchronizes with other db servers anyway (is it any other way i could do this anyway?) but the fileserver will work, yes. I did that many times in the past. You just have to take care of the KeyFile etc. I'll just copy the KeyFile from another server.. But the /vicepa partition is ok.Is it possible to recover the data (transfer them to another working openafs server), from the /vicepa/VOL* files? I don't see why a new copy of the fileserver wouldn't do the right thing here. A new copy of the fileserver? IIRC it's a linux server. Its FreeBSD actually. If not, you really want to install the same server type (inode or namei). It would be fatal error if you don't. The fileserver of the wrong type can damage your volumes. I'll just use the binary version from the openafs.org site.The previous OS was linux but i don't think i've tampered with the inode/namei option when i built it from source there.. Horst By the way, if the afs files (KeyFile and /vicepa/*) are generated on a 32bit host, would they be compatible if used on a 64bit host? I mean, can you use them on a new 64bit OS installation?My first guess is no..but i don't know. -- http://www.fastmail.fm - And now for something completely different ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery, help needed.
On Apr 24, 2006, at 12:15 PM, ph rhole oper wrote: On Mon, 24 Apr 2006 09:05:11 +0200, Horst Birthelmer [EMAIL PROTECTED] said: On Apr 24, 2006, at 3:42 AM, ph rhole oper wrote: On Sun, 23 Apr 2006 22:20:19 +0200 (CEST), Chris Huebsch [EMAIL PROTECTED] said: Hello, On Sun, 23 Apr 2006, ph rhole oper wrote: The hd holding the root directory of an afs server has crashed, thus the /var/openafs/ files are lost with it, Was the crashed server only a fileserver or both fileserver and databaseserver? If the latter, was it your only databaseserver? It was a db server too, but there is one more left online.I think that if i install a new OS, mount /vicepa , and start afs services, ubik will synchronize and the new OS's afs will just bring the /vicepa volumes up.Right? The DB server cold make some trouble on resynchronization with an empty database (It's really no big deal) Whenever i need to install a new db server on the network, i just start it with an empty database and it synchronizes with other db servers anyway (is it any other way i could do this anyway?) No. My point was, if it doesn't (which I saw sometimes) it's not part of this problem. ;-) but the fileserver will work, yes. I did that many times in the past. You just have to take care of the KeyFile etc. I'll just copy the KeyFile from another server.. Yes, that's the easiest way, of course. I just wanted to remind you... But the /vicepa partition is ok.Is it possible to recover the data (transfer them to another working openafs server), from the /vicepa/VOL* files? I don't see why a new copy of the fileserver wouldn't do the right thing here. A new copy of the fileserver? Yes, another binary ... from wherever. IIRC it's a linux server. Its FreeBSD actually. If not, you really want to install the same server type (inode or namei). It would be fatal error if you don't. The fileserver of the wrong type can damage your volumes. I'll just use the binary version from the openafs.org site.The previous OS was linux but i don't think i've tampered with the inode/namei option when i built it from source there.. On Linux there is no inode fileserver. You don't have a choice there. :-) Did I get that right? You're trying to attach a partition previously used by a Linux fileserver on a fresh installed FreeBSD fileserver? That kinda changes my statement from above. I don't really know if that won't cause you trouble. You better install the same OS on that machine move all the volumes away and install another OS. From my perspective, it's safer. By the way, if the afs files (KeyFile and /vicepa/*) are generated on a 32bit host, would they be compatible if used on a 64bit host? I mean, can you use them on a new 64bit OS installation?My first guess is no..but i don't know. I don't expect any problems but I don't know for sure either. I never switched from 32 to 64 bits on one machine. Horst ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery, help needed.
Hi, On Monday 24 April 2006 03:42, ph rhole oper wrote: It was a db server too, but there is one more left online.I think that if i install a new OS, mount /vicepa , and start afs services, ubik will synchronize and the new OS's afs will just bring the /vicepa volumes up.Right? At first you should make a backup of your administrativ databases of your last running server (kaserver.DB*, prdb.DB*, vldb.DB* (,bdb.DB* - if you have it)) Then you can install a new OS on your broken machine. But _do not start_ AFS-Servers Copy some files manually from your last server left to your reinstalled server, which are: /usr/afs/db/* /usr/afs/etc/* /usr/afs/local/BosConfig (/usr/afs might be at a different location) Then you can start the afs-processes on your reinstalled server. Then you can use udebug server portnumber to watch the databases synchronize. Ports are in range 7002-7004. Good Luck Chris -- Chris Huebschwww.huebsch-gemacht.de | TU Chemnitz, Informatik, RNVS GPG-Encrypted mail welcome! ID:7F2B4DBA | Str. d. Nationen 62, B204 $AChemnitzer Linux-Tage 2007, 3.-4. Maerz | D-09107 Chemnitz http://chemnitzer.linux-tage.de/| +49 371 531-1377, Fax -1803 ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
[OpenAFS] disaster recovery help needed
I've lost some volumes due to a severe hardware crash on one of my afs servers. Now i want to create new empty volumes with the same names, but there are entries on the vldb for the lost volumes. How do i get rid of these bad volumes from the vldb? simon. -- http://www.fastmail.fm - Does exactly what it says on the tin ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery help needed
On Mon, 24 Apr 2006, ph rhole oper wrote: I've lost some volumes due to a severe hardware crash on one of my afs servers. Now i want to create new empty volumes with the same names, but there are entries on the vldb for the lost volumes. How do i get rid of these bad volumes from the vldb? vos delentry Chris -- Chris Huebschwww.huebsch-gemacht.de | TU Chemnitz, Informatik, RNVS GPG-Encrypted mail welcome! ID:7F2B4DBA | Str. d. Nationen 62, B204 Chemnitzer Linux-Tage 2007, 3.-4. Maerz | D-09107 Chemnitz http://chemnitzer.linux-tage.de/| +49 371 531-1377, Fax -1803 ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
[OpenAFS] disaster recovery, help needed.
The hd holding the root directory of an afs server has crashed, thus the /var/openafs/ files are lost with it, But the /vicepa partition is ok.Is it possible to recover the data (transfer them to another working openafs server), from the /vicepa/VOL* files? thanx in advance, simon. -- http://www.fastmail.fm - And now for something completely different ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery, help needed.
how many servers in the cell? which services are still working? what services were running on the one that's down? anne Quoting ph rhole oper [EMAIL PROTECTED]: The hd holding the root directory of an afs server has crashed, thus the /var/openafs/ files are lost with it, But the /vicepa partition is ok.Is it possible to recover the data (transfer them to another working openafs server), from the /vicepa/VOL* files? thanx in advance, simon. -- http://www.fastmail.fm - And now for something completely different ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery, help needed.
Hello, On Sun, 23 Apr 2006, ph rhole oper wrote: The hd holding the root directory of an afs server has crashed, thus the /var/openafs/ files are lost with it, Was the crashed server only a fileserver or both fileserver and databaseserver? If the latter, was it your only databaseserver? But the /vicepa partition is ok.Is it possible to recover the data (transfer them to another working openafs server), from the /vicepa/VOL* files? Try using the voldump utility to dump your voulmes. (http://www.eyrie.org/~eagle/tmp/openafs/8/voldump.html) It should be installed with your openafs-installation. You can later vos resore them. Chris -- Chris Huebschwww.huebsch-gemacht.de | TU Chemnitz, Informatik, RNVS GPG-Encrypted mail welcome! ID:7F2B4DBA | Str. d. Nationen 62, B204 Chemnitzer Linux-Tage 2007, 3.-4. Maerz | D-09107 Chemnitz http://chemnitzer.linux-tage.de/| +49 371 531-1377, Fax -1803 ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
Re: [OpenAFS] disaster recovery, help needed.
On Sun, 23 Apr 2006 22:20:19 +0200 (CEST), Chris Huebsch [EMAIL PROTECTED] said: Hello, On Sun, 23 Apr 2006, ph rhole oper wrote: The hd holding the root directory of an afs server has crashed, thus the /var/openafs/ files are lost with it, Was the crashed server only a fileserver or both fileserver and databaseserver? If the latter, was it your only databaseserver? It was a db server too, but there is one more left online.I think that if i install a new OS, mount /vicepa , and start afs services, ubik will synchronize and the new OS's afs will just bring the /vicepa volumes up.Right? But the /vicepa partition is ok.Is it possible to recover the data (transfer them to another working openafs server), from the /vicepa/VOL* files? Try using the voldump utility to dump your voulmes. Well, if all else fails, i will.thanx (http://www.eyrie.org/~eagle/tmp/openafs/8/voldump.html) It should be installed with your openafs-installation. You can later vos resore them. Chris -- Chris Huebschwww.huebsch-gemacht.de | TU Chemnitz, Informatik, RNVS GPG-Encrypted mail welcome! ID:7F2B4DBA | Str. d. Nationen 62, B204 Chemnitzer Linux-Tage 2007, 3.-4. Maerz | D-09107 Chemnitz http://chemnitzer.linux-tage.de/| +49 371 531-1377, Fax -1803 -- http://www.fastmail.fm - A fast, anti-spam email service. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info
