RE: Problem recover replica
> On Thu, Feb 02, 2012 at 01:11:24PM +0100, Manel Gimeno Zaragozá wrote: > > What version? I'm running Cyrus 2.4.13. > > > Everything is working fine between Master & Replica until Replica goes down (maintenance or whatever). Once is down, in master there are still connections and modification. If I bring up Replica the changes made during the "blackout" are no fully transmitted to it. Some of the are transmitted, but not all. > > > > Which is the best way to resincronize the missing parts to Replica? > > sync_client -r -f $file for each file in the conf/sync/ directory. I've been trying sync_client -r -f log-PID and it works. All the missing changes are applied without problem, I should do some script that controls this files and apply them. anyway the log-PID is not deleted when its processed. I guess we should control the processed log-PIDs manually and move or delete them, am i right? > > > I was wordering to rsync imap folders (/var/spool/imap & /var/lib/imap), and It works fine in my test environment, but in produccion it could be GB to sincronize and I think this is not optimum. > > No, that's awful. Don't do that. > > > Other option was "sync_client -l -u " but I'm guessing that in production I will have the same problem due to the amount of data to sincronize. > > It's not too bad. We run it occasionally if we suspect things have > got confused for other reasons. > > > On the other hand, I have also doubts about how "sync_client -r" and "sync_server" are initiated. In my test machines I should run manually them each time I start cyrus-imapd, I though that it could be automatic one I start the service cyrus-imapd, but it is not. > > I have attached the script that we run. It has lots of hooks into > our own systems of course. > > I would LIKE to make something more generic that's part of Cyrus > itself, rather than having external tooling. Of course, our external > tooling is a bit special-cased as well. But at least having a basic > "keep trying to replicate all the records" would be good. > > Bron. Thanks for the response. Manel. Manel Gimeno Zaragoza magiz...@hotmail.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: fetching user_deny.db entry for ...
On Thu, February 2, 2012 2:20 pm, Marten Lehmann wrote: > since my upgrade from cyrus-imapd 2.3.x to 2.4.x mail maillog gets cluttered > by entries like this: > > Feb 2 14:18:07 g112 cyrus/imap[16836]: fetching user_deny.db entry for > 'u...@mailbox.com' > > > I have absoletely no use for this user_deny.db stuff. Before I created > an empty user_deny.db I got IOERRORs with "No such file or directory", now > that > it exists, now I get these useless logs. How can I disable this entirely? Or > how can I disable logging of it? Marten, You could reduce the syslog logging level from .debug to .info but this will make you lose some other stuff too. Check whether you really need the latter. Eric Luyten, Computing Centre VUB/ULB, Belgium. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
fetching user_deny.db entry for ...
Hello, since my upgrade from cyrus-imapd 2.3.x to 2.4.x mail maillog gets cluttered by entries like this: Feb 2 14:18:07 g112 cyrus/imap[16836]: fetching user_deny.db entry for 'u...@mailbox.com' I have absoletely no use for this user_deny.db stuff. Before I created an empty user_deny.db I got IOERRORs with "No such file or directory", now that it exists, now I get these useless logs. How can I disable this entirely? Or how can I disable logging of it? Kind regards Marten Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: Problem recover replica
Quoting Manel Gimeno Zaragozá : Hello, I've noticed that the "missing changes" are due to some of the actions done during the "blackout" are processed un "/var/lib/imap/sync" log and named modified to log-PID. It seems that rsync_client have not noticed that Replica is down and it process some of them. Once it notice that replica is down continue login in "log" file. When Replica comes back, sync_client process the changes from "log" without problems, but we have miss some changes. the sync_client in rolling replication locks for the log file and moves it to log-PID. Then processes the actions in log-PID, deletes this file and looks for a new log file. If sync_client can't connect the sync_server it will wait some time and trying again. If there is still the sync_client process with the PID, it should catch up. If the sync_client died you can process the log-PID file with sync_client -r -f log-PID . M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen smime.p7s Description: S/MIME Signatur Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Problem recover replica
On Thu, Feb 02, 2012 at 01:11:24PM +0100, Manel Gimeno Zaragozá wrote:
What version?
> Everything is working fine between Master & Replica until Replica goes down
> (maintenance or whatever). Once is down, in master there are still
> connections and modification. If I bring up Replica the changes made during
> the "blackout" are no fully transmitted to it. Some of the are transmitted,
> but not all.
>
> Which is the best way to resincronize the missing parts to Replica?
sync_client -r -f $file for each file in the conf/sync/ directory.
> I was wordering to rsync imap folders (/var/spool/imap & /var/lib/imap), and
> It works fine in my test environment, but in produccion it could be GB to
> sincronize and I think this is not optimum.
No, that's awful. Don't do that.
> Other option was "sync_client -l -u " but I'm guessing that in
> production I will have the same problem due to the amount of data to
> sincronize.
It's not too bad. We run it occasionally if we suspect things have
got confused for other reasons.
> On the other hand, I have also doubts about how "sync_client -r" and
> "sync_server" are initiated. In my test machines I should run manually them
> each time I start cyrus-imapd, I though that it could be automatic one I
> start the service cyrus-imapd, but it is not.
I have attached the script that we run. It has lots of hooks into
our own systems of course.
I would LIKE to make something more generic that's part of Cyrus
itself, rather than having external tooling. Of course, our external
tooling is a bit special-cased as well. But at least having a basic
"keep trying to replicate all the records" would be good.
Bron.
#!/usr/bin/perl -w
BEGIN { do "/home/mod_perl/hm/ME/FindLibs.pm"; }
use strict;
use warnings;
use ME::FMVars;
use ME::ImapStore;
use ME::ImapSlot;
use ME::Machine;
use ME::Notify qw(NotifyAdmins);
use IO::LockedFile;
use IO::File;
use Getopt::Std;
use Data::Dumper;
my %Opts;
getopts('Cvnl:r:fa', \%Opts);
my $MachineName = ME::Machine->Name();
my $RateLimit = 5000;
my $OneoffLimit = $Opts{l} || $RateLimit;
# easy way to suppress cron runs
unless ($Opts{f}) {
exit 0 if -e "/etc/nomonitorsync";
}
foreach my $Slot (ME::Machine->ImapSlots()) {
my $SlotName = $Slot->Name();
unless ($Opts{a} or $Slot->IsMaster()) {
print "$SlotName: not master\n" if $Opts{v};
next;
}
my @OtherSlots = $Slot->OtherSlots();
unless (@OtherSlots) {
print "$SlotName: not replicated\n" if $Opts{v};
next;
}
my $ConfDir = $Slot->CyrusConfigPath();
my %dirs;
my $ImapdConf = $Slot->ImapdConf();
if ($ImapdConf->{sync_log_channels}) {
foreach my $channel (split / /, $ImapdConf->{sync_log_channels}) {
$dirs{$channel} = "$ConfDir/sync/$channel";
}
my %channels = map { $_ => 1 } split / /, $ImapdConf->{sync_log_channels};
foreach my $osl (map { $_->Name() } @OtherSlots) {
delete $channels{$osl};
}
delete $channels{oslotest};
if (keys %channels) {
warn "$SlotName: extra channels in imapd configuration file - "
.join(" ", keys %channels) . "\n";
}
} elsif ($ImapdConf->{sync_host}) {
$dirs{''} = "$ConfDir/sync";
}
opendir(DH, "$ConfDir/sync/");
while(my $item = readdir(DH)) {
next if $item =~ m/^\./;
next unless -d "$ConfDir/sync/$item";
next if $dirs{$item}; # channel exists
warn "$SlotName: extra channel $item\n";
next unless $Opts{C};
warn "rm -rf $ConfDir/sync/$item\n";
system("rm -rf $ConfDir/sync/$item\n");
}
my $DH;
foreach my $channel (sort keys %dirs) {
print "Doing slot $SlotName/$channel\n" if $Opts{v};
my $dir = $dirs{$channel};
next unless -d $dir; # no logs!
if (-f "$dir/shutdown") {
print "$SlotName/$channel: shutdown file exists, skipping\n";
next;
}
my $OtherSlot;
if ($channel) {
$OtherSlot = eval { ME::ImapSlot->new($channel) };
} else {
# There is only one replica in this case
$OtherSlot = $OtherSlots[0];
}
next unless $OtherSlot;
unless ($OtherSlot->IsRunning()) {
print "$SlotName/$channel: ignoring, replica is down\n" if $Opts{v};
next;
}
if ($Opts{r} and not $OtherSlot->Machine->HasRole($Opts{r})) {
print "$SlotName/$channel: not syncing, doesn't have role $Opts{r}\n";
next;
}
my @rest;
@rest = (-n => $channel) if $channel;
my $Lock = IO::LockedFile->new({block => 0}, ">$dir/monitorsync.lock");
unless ($Lock) {
print "$SlotName/$channel: already locked, skipping\n" if $Opts{v};
next;
}
my @pids = get_pids($SlotName, $channel);
my @ran;
next unless opendir($DH, $dir);
while (my $item = readdir($DH)) {
if ($item eq 'log' and not @pids) {
$item = "slog-$$";
print ("renaming log to $item\n");
rename("$dir/log", "$dir/$item");
}
next unless $item =~ m/^(?:s\d*)?log-(\d+)$/;
my $pid = $1 || '';
# check if pid
Re: Problem recover replica
Hi, Quoting Manel Gimeno Zaragozá : Hello, I'm trying to implement a master-replica cyrus environment and I'm getting some issue that I'm not able to resolve. The scenario is the following: Everything is working fine between Master & Replica until Replica goes down (maintenance or whatever). Once is down, in master there are still connections and modification. If I bring up Replica the changes made during the "blackout" are no fully transmitted to it. Some of the are transmitted, but not all. Which is the best way to resincronize the missing parts to Replica? I was wordering to rsync imap folders (/var/spool/imap & /var/lib/imap), and It works fine in my test environment, but in produccion it could be GB to sincronize and I think this is not optimum. Other option was "sync_client -l -u " but I'm guessing that in production I will have the same problem due to the amount of data to sincronize. If you have configured sync_log: 1 cyrus will keek an actionlog for rolling replication. While "sync_client -r" is not running, or unable to connect to the sync_server this file keeps track of what has to be synced. Once sync_client connects to the server it will catch up. M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen smime.p7s Description: S/MIME Signatur Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: Problem recover replica
Hello, I've noticed that the "missing changes" are due to some of the actions done during the "blackout" are processed un "/var/lib/imap/sync" log and named modified to log-PID. It seems that rsync_client have not noticed that Replica is down and it process some of them. Once it notice that replica is down continue login in "log" file. When Replica comes back, sync_client process the changes from "log" without problems, but we have miss some changes. Is there any configuratio to avoid this situation? Regards. Manel Gimeno Zaragoza magiz...@hotmail.com From: magiz...@hotmail.com To: info-cyrus@lists.andrew.cmu.edu Subject: Problem recover replica Date: Thu, 2 Feb 2012 13:11:24 +0100 Hello, I'm trying to implement a master-replica cyrus environment and I'm getting some issue that I'm not able to resolve. The scenario is the following: Everything is working fine between Master & Replica until Replica goes down (maintenance or whatever). Once is down, in master there are still connections and modification. If I bring up Replica the changes made during the "blackout" are no fully transmitted to it. Some of the are transmitted, but not all. Which is the best way to resincronize the missing parts to Replica? I was wordering to rsync imap folders (/var/spool/imap & /var/lib/imap), and It works fine in my test environment, but in produccion it could be GB to sincronize and I think this is not optimum. Other option was "sync_client -l -u " but I'm guessing that in production I will have the same problem due to the amount of data to sincronize. On the other hand, I have also doubts about how "sync_client -r" and "sync_server" are initiated. In my test machines I should run manually them each time I start cyrus-imapd, I though that it could be automatic one I start the service cyrus-imapd, but it is not. Thanks & Regards. Manel Gimeno Zaragoza magiz...@hotmail.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Problem recover replica
Hello, I'm trying to implement a master-replica cyrus environment and I'm getting some issue that I'm not able to resolve. The scenario is the following: Everything is working fine between Master & Replica until Replica goes down (maintenance or whatever). Once is down, in master there are still connections and modification. If I bring up Replica the changes made during the "blackout" are no fully transmitted to it. Some of the are transmitted, but not all. Which is the best way to resincronize the missing parts to Replica? I was wordering to rsync imap folders (/var/spool/imap & /var/lib/imap), and It works fine in my test environment, but in produccion it could be GB to sincronize and I think this is not optimum. Other option was "sync_client -l -u " but I'm guessing that in production I will have the same problem due to the amount of data to sincronize. On the other hand, I have also doubts about how "sync_client -r" and "sync_server" are initiated. In my test machines I should run manually them each time I start cyrus-imapd, I though that it could be automatic one I start the service cyrus-imapd, but it is not. Thanks & Regards. Manel Gimeno Zaragoza magiz...@hotmail.com Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
