Hello,
I want to share a success story, and document the steps I did
(in essence, followed Jan's recommendations and read some man pages)
Running a two-server setup with all volumes replicated.
One of the servers got a corrupted volume and refused to start,
complaining about an assertion error when running salvager on one of the
volumes.
[think of an irreparable fsck problem on a central NFS server? ;-]
Well, my other server stayed online so that I had the system running.
The steps to revive the crashing server:
get the info about the volume:
[any of the servers]# grep <volname> /vice/db/VRList
you get
<volname> <groupid> <replica_num> <volid_serv1> <volid_serv2> 0 0 0 0 0 0 <VSG>
now you have to find which <volid> you are interested in:
[any of the servers]# grep <host_with_dead_server> /vice/db/servers
you get
<hostname> <serverid>
the <serverid> is a small number and it matches the beginning of <volid>,
e.g. my dead server has number 2 and the matching volid was 200004d
Let us get more information about the volume (and destroy it, as it is
broken)
[the host with the dead server]# grep rvm_ <where-you-have-it>/server.conf
rvm_log="<LOG>"
rvm_data="<DATA>"
rvm_data_length="<LENGTH>"
[the host with the dead server]# norton -mapprivate <LOG> <DATA> <LENGTH>
(-mapprivate is a lot faster than without it)
norton> show volume <volid>
Id: 0x<volid> Name: <name>.<digit> Parent: 0x200004d
GoupId: 0x<groupid> Partition: <partition>
as I had to remove the broken volume:
norton> delete volume <volid>
norton> Ctrl/D
Now start the server, watch "tail -f /vice/srv/SrvLog" and see the volume
being destroyed, instead of crashing the server.
When the server is up, the moment of truth has come.
Run (substituting the values from the above) :
[the host with now alive server, missing one volume]# \
volutil create_rep <partition> <name>.<digit> 0x<groupid> 0x<volid>
Now go to a well connected client and do "ls -alR" on the corresponding
mountpoint.
You can watch the resolution to happen, looking at the results of
volutil -h <hostname-of-serverN> info <volid_servN> | grep diskused
for the servers concerned (use output of "grep <volname> /vice/db/VRList"
above)
Enjoy Coda!
--
Ivan
