Re: mailboxes.db problem cont.
On Thu, 11 Dec 2003, Craig Ringer wrote: > Naturally this would need to be prefaced with a warning about the loss > of ACL information - a "use as a last resort only" warning. The ACL information is maintained in the cyrus.header file. If you still have that, there is no reason you should suffer a loss of ACL information. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
Re: mailboxes.db problem cont.
Hi
I was wondering ... does anybody mind if I add this script to the Wiki?
I recently created an entry about the non-working 'reconstruct -m'
command
(http://asg.web.cmu.edu/twiki/bin/view/Cyrus/ReconstructMailboxes), and
it'd be nice to offer /some/ recovery option for people who've been
bitten by mailboxes.db issues and don't have a backup.
Ideally, of course, everybody would just do good backups that included a
plaintext mailboxes.db - but it's been shown by the list traffic that
not everybody realises how critical the mailboxes.db is and how hard it
is to reconstruct.
Naturally this would need to be prefaced with a warning about the loss
of ACL information - a "use as a last resort only" warning.
Hank Beatty wrote:
Many thanks to Scott Adkins and Joe Hrbek for all of their help.
Here are a couple of scripts that I wrote to rebuild the mailboxes.db.
The scripts do not take into account default domain, basic or no
directory hashing, multiple partitions, and possibly other things, but I
figured they might give someone a head start if they run into this
problem.
shell script "repairMBoxDB"
#!/bin/sh
#stop Cyrus imapd before running this script
#get the directories
ls -R /var/spool/imap/* | grep "/" > /bin/scripts/mboxRepair/directories
#run perl repair script (this doesn't actually do any repairing)
/bin/scripts/mboxRepair/mboxRepair.pl
#cp the file to Cyrus' home directory and change the ownership
cp /bin/scripts/mboxRepair/newMBoxList /home/cyrus/
chown cyrus:mail /home/cyrus/newMBoxList
#save off the current mailboxes.db
mv /var/imap/mailboxes.db /var/imap/mailboxes.db.backup
#import the new mailbox list
su - cyrus -c "umask 077 ; /usr/cyrus/bin/ctl_mboxlist -u <
/home/cyrus/newMBoxList"
#EOF
perl script mboxRepair.pl
#!/usr/bin/perl
&openDirectoriesFile;
&rewriteFormat;
&scriptExit;
sub openDirectoriesFile {
my ($lSuccess, $lName);
$lName = "directories";
$lSuccess = open (fhDirectories, "/bin/scripts/mboxRepair/$lName");
if (!$lSuccess) {
print "Didn't open " . $lName . ": $!\n";
&scriptExit ();
}
}
sub scriptExit {
close (fhDirectories);
close (fhNewMBoxList);
exit();
}
sub rewriteFormat {
my ($lSuccess, $lNewLine, $lCount, $lArrayCount, $lSortedCount);
my (@lArray, @lUnsortedArray, @lSortedArray);
$lName = "newMBoxList";
$lSuccess = open (fhNewMBoxList, ">/bin/scripts/mboxRepair/$lName");
if (!$lSuccess) {
print "Didn't open " . $lName . ": $!\n";
&scriptExit ();
}
$lArrayCount = 0;
while(){
chomp; # no newline
s/#.*//; # no comments
s/^\s+//; # no leading white
s/\s+$//; # no trailing white
next if ($_ =~ m/\/mail:$/); # no lines ending with
/mail:
s/:$//;# no ending colons
s/^\/var\/spool\/imap\///; # no /var/spool/imap/
next if ($_ !~ m/\//); # get rid of any line that
doesn't
have a /
s/domain\/[A-Z]//; # get rid of domain/A, domain/B,
domain/C, etc.
s/^\///; # no / at the beginning of
a line
next if ($_ !~ m/\//); # get rid of any line that
doesn't
have a /
next if ($_ !~ m/\/user\//); # get rid of any line that
doesn't
have /user
next unless length;# anything left?
@lArray = split ("/", $_);
$lNewLine = sprintf ("%s!%s.%s", $lArray[0], $lArray[2],
$lArray[3]);
$lCount = 4;
while ($lArray[$lCount]) {
$lNewLine = sprintf ("%s.%s", $lNewLine, $lArray[$lCount]);
$lCount++;
}
#now we are going to put it into an array so we can sort it
$lUnsortedArray[$lArrayCount] = $lNewLine;
$lArrayCount++;
}
@lSortedArray = sort { $a cmp $b } @lUnsortedArray;
#now take the sorted array and put it in the proper format and
#write it to a file
while ($lSortedArray[$lSortedCount]) {
@lArray = split (/\./, $lSortedArray[$lSortedCount]);
$lArray[2] =~ s/\^/\./;
my @lDomain = split (/!/, $lSortedArray[$lSortedCount]);
$lNewLine = sprintf ("[EMAIL PROTECTED]",
$lSortedArray[$lSortedCount], $lArray[2], $lDomain[0]);
print fhNewMBoxList "" . $lNewLine . "\n";# write to a new
file
$lSortedCount++;
}
}
Re: mailboxes.db problem cont.
Many thanks to Scott Adkins and Joe Hrbek for all of their help.
Here are a couple of scripts that I wrote to rebuild the mailboxes.db.
The scripts do not take into account default domain, basic or no
directory hashing, multiple partitions, and possibly other things, but I
figured they might give someone a head start if they run into this
problem.
shell script "repairMBoxDB"
#!/bin/sh
#stop Cyrus imapd before running this script
#get the directories
ls -R /var/spool/imap/* | grep "/" > /bin/scripts/mboxRepair/directories
#run perl repair script (this doesn't actually do any repairing)
/bin/scripts/mboxRepair/mboxRepair.pl
#cp the file to Cyrus' home directory and change the ownership
cp /bin/scripts/mboxRepair/newMBoxList /home/cyrus/
chown cyrus:mail /home/cyrus/newMBoxList
#save off the current mailboxes.db
mv /var/imap/mailboxes.db /var/imap/mailboxes.db.backup
#import the new mailbox list
su - cyrus -c "umask 077 ; /usr/cyrus/bin/ctl_mboxlist -u <
/home/cyrus/newMBoxList"
#EOF
perl script mboxRepair.pl
#!/usr/bin/perl
&openDirectoriesFile;
&rewriteFormat;
&scriptExit;
sub openDirectoriesFile {
my ($lSuccess, $lName);
$lName = "directories";
$lSuccess = open (fhDirectories, "/bin/scripts/mboxRepair/$lName");
if (!$lSuccess) {
print "Didn't open " . $lName . ": $!\n";
&scriptExit ();
}
}
sub scriptExit {
close (fhDirectories);
close (fhNewMBoxList);
exit();
}
sub rewriteFormat {
my ($lSuccess, $lNewLine, $lCount, $lArrayCount, $lSortedCount);
my (@lArray, @lUnsortedArray, @lSortedArray);
$lName = "newMBoxList";
$lSuccess = open (fhNewMBoxList, ">/bin/scripts/mboxRepair/$lName");
if (!$lSuccess) {
print "Didn't open " . $lName . ": $!\n";
&scriptExit ();
}
$lArrayCount = 0;
while(){
chomp; # no newline
s/#.*//; # no comments
s/^\s+//; # no leading white
s/\s+$//; # no trailing white
next if ($_ =~ m/\/mail:$/); # no lines ending with
/mail:
s/:$//;# no ending colons
s/^\/var\/spool\/imap\///; # no /var/spool/imap/
next if ($_ !~ m/\//); # get rid of any line that
doesn't
have a /
s/domain\/[A-Z]//; # get rid of domain/A, domain/B,
domain/C, etc.
s/^\///; # no / at the beginning of
a line
next if ($_ !~ m/\//); # get rid of any line that
doesn't
have a /
next if ($_ !~ m/\/user\//); # get rid of any line that
doesn't
have /user
next unless length;# anything left?
@lArray = split ("/", $_);
$lNewLine = sprintf ("%s!%s.%s", $lArray[0], $lArray[2],
$lArray[3]);
$lCount = 4;
while ($lArray[$lCount]) {
$lNewLine = sprintf ("%s.%s", $lNewLine, $lArray[$lCount]);
$lCount++;
}
#now we are going to put it into an array so we can sort it
$lUnsortedArray[$lArrayCount] = $lNewLine;
$lArrayCount++;
}
@lSortedArray = sort { $a cmp $b } @lUnsortedArray;
#now take the sorted array and put it in the proper format and
#write it to a file
while ($lSortedArray[$lSortedCount]) {
@lArray = split (/\./, $lSortedArray[$lSortedCount]);
$lArray[2] =~ s/\^/\./;
my @lDomain = split (/!/, $lSortedArray[$lSortedCount]);
$lNewLine = sprintf ("[EMAIL PROTECTED]",
$lSortedArray[$lSortedCount], $lArray[2], $lDomain[0]);
print fhNewMBoxList "" . $lNewLine . "\n";# write to a new
file
$lSortedCount++;
}
}
On Mon, 2003-09-29 at 08:52, Hank Beatty wrote:
> It was indeed the tab at the end of each line that caused the problem. I
> added the tab to the end of each line and implemented it about 3 PM
> Saturday afternoon and brought the server back on line.
>
> On Sun, 2003-09-28 at 12:14, Rob Siemborski wrote:
> > On Sat, 27 Sep 2003, Hank Beatty wrote:
> >
> > > Took dir.txt and converted it to the format of mboxlist file (except for
> > > the tab on the end of each line. Not sure if this caused a problem)
> >
> > This will likely cause a problem with the ACLs.
> >
> > > The above steps got POP working, but IMAP gives ???Mailbox does not
> > > exist??? when trying to select the ???INBOX??? using squirrelmail.
> >
> > I suspect this can be caused by a munged ACL.
> >
> > > I also tried using the reconstruct command before and after moving the
> > > mailboxes.db to no avail. At this point I???m thinking of writing the
> > > ???m option of reconstruct unless anyon
Re: mailboxes.db problem cont.
It was indeed the tab at the end of each line that caused the problem. I added the tab to the end of each line and implemented it about 3 PM Saturday afternoon and brought the server back on line. On Sun, 2003-09-28 at 12:14, Rob Siemborski wrote: > On Sat, 27 Sep 2003, Hank Beatty wrote: > > > Took dir.txt and converted it to the format of mboxlist file (except for > > the tab on the end of each line. Not sure if this caused a problem) > > This will likely cause a problem with the ACLs. > > > The above steps got POP working, but IMAP gives ???Mailbox does not > > exist??? when trying to select the ???INBOX??? using squirrelmail. > > I suspect this can be caused by a munged ACL. > > > I also tried using the reconstruct command before and after moving the > > mailboxes.db to no avail. At this point I???m thinking of writing the > > ???m option of reconstruct unless anyone has some better ideas or has > > already written something that might help. While writing the ???m option > > I might try to figure out why the ???f and ???r options of reconstruct > > didn???t appear to work in my case. I???m wondering if the ???f and ???r > > options don???t work because I???m using the fulldirhash option. > > -r means "look at the mailbox list and descend" > > -f means "look at subdirectories" which won't always work quite right on > top level mailboxes because of how hashing is done. > > Its unlikely fulldirhash affected this at all. > > -Rob > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 > Research Systems Programmer * /usr/contributed Gatekeeper -- Hank Beatty <[EMAIL PROTECTED]>
Re: mailboxes.db problem cont.
On Sat, 27 Sep 2003, Hank Beatty wrote: > Took dir.txt and converted it to the format of mboxlist file (except for > the tab on the end of each line. Not sure if this caused a problem) This will likely cause a problem with the ACLs. > The above steps got POP working, but IMAP gives ???Mailbox does not > exist??? when trying to select the ???INBOX??? using squirrelmail. I suspect this can be caused by a munged ACL. > I also tried using the reconstruct command before and after moving the > mailboxes.db to no avail. At this point I???m thinking of writing the > ???m option of reconstruct unless anyone has some better ideas or has > already written something that might help. While writing the ???m option > I might try to figure out why the ???f and ???r options of reconstruct > didn???t appear to work in my case. I???m wondering if the ???f and ???r > options don???t work because I???m using the fulldirhash option. -r means "look at the mailbox list and descend" -f means "look at subdirectories" which won't always work quite right on top level mailboxes because of how hashing is done. Its unlikely fulldirhash affected this at all. -Rob -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Rob Siemborski * Andrew Systems Group * Cyert Hall 207 * 412-268-7456 Research Systems Programmer * /usr/contributed Gatekeeper
mailboxes.db problem cont.
Just wanted to post an update and see if anyone had a better solution. I'm running a Red Hat 9 server with Cyrus imapd 2.2.1 BETA. Thursday at about 17:20 we started seeing alarms on our POP monitor for this server. We investigated and found that everyone after the letter 'c' had been removed from the DB. After looking through the logs we found that ctl_cyrusdb ran at 17:15. We're thinking that this somehow caused the problem. UnfortunatJust wanted to post an update and see if anyone had a better solution. I'm running a Red Hat 9 server with Cyrus imapd 2.2.1 BETA. Thursday at about 17:20 we started seeing alarms on our POP monitor for this server. We investigated and found that everyone after the letter 'c' had been removed from the DB. After looking through the logs we found that ctl_cyrusdb ran at 17:15. We're thinking that this somehow caused the problem. Unfortunately, we can???t say for sure. This is what we have done so far: Stopped cyrus Ran ???ctl_mboxlist ???d > mboxlist??? Moved the mailboxes.db to another location Moved the db directory to another location Moved deliver.db to another location Did an ls on the directory structure and sent it to dir.txt Took dir.txt and converted it to the format of mboxlist file (except for the tab on the end of each line. Not sure if this caused a problem) Then did an undump of dir.txt back into mailboxes.db Started cyrus The above steps got POP working, but IMAP gives ???Mailbox does not exist??? when trying to select the ???INBOX??? using squirrelmail. I also tried using the reconstruct command before and after moving the mailboxes.db to no avail. At this point I???m thinking of writing the ???m option of reconstruct unless anyone has some better ideas or has already written something that might help. While writing the ???m option I might try to figure out why the ???f and ???r options of reconstruct didn???t appear to work in my case. I???m wondering if the ???f and ???r options don???t work because I???m using the fulldirhash option. Please help. Thanks, Hank
