Re: Sync log replica process
Hi, Quoting Manel Gimeno Zaragozá magiz...@hotmail.com: Hello, I've some doubts about how sync logs and the order about how they should be processed. I've done some bunch tests and during the process stop replic to try to simulate a possible crash. In my test I've saw the following: 1.- I send 10 mails to Master and it starts to replicate. 2.- file log is created and change to log-13835 3.- Replica crash and log-13835 never ends and remains in /var/lib/imap/sync/ forever 4.- while replica is crashed the user is making actions in the 10 mails send before (moving aroung folders, deleting, etc...) and all changes are saved in log file 5.- Replica comes up and i run sync_client -r to sincronize the changes. The question is here. Should I first run sync_client -r -f /var/lib/imap/sync/log-13835 before starts sync_client -r? I understand that the log-13835 has the information about the 10 mail received, and log has information about mail that are not yet in replica. 6.- Anyway, I run sync_client -rwithout process log-13835 and, what a surprise!! everything is on their place and sync works ok with out processing the log-13835 7.- I run my integrity_check based on Bron's integrity check and the only problem I see is a mismatch in highestmodseq for the folder where the user moves the mails. Mismatched 'highestmodseq' for user.gdatmgim.temp master=7, replica=9 This error remain until some action is taken in the folder. So, my question is, is it necessary to process the log-PID before sync the log file? Because in my test it looks like it's not necessary. It depends on what actions where done while the syncornisation was down Cyrus does not log every changed mail, but in which mailbox mails where changed. So if log-13835 contains MAILBOX user.gdatmgim.temp and MAILBOX user.gdatmgim and he recieved new mails in the INBOX but did not change mails in the temp folder, the INBOX will be in sync without the log-13835, but not user.gdatmgim.temp. So the replication will at some point catch missed actions from unprocessed logs, but it can take some time (forever in worst case). On the other hand it does not harm to run old logs. Michael M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen smime.p7s Description: S/MIME Signatur Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Sync log replica process
On Tue, Mar 13, 2012 at 11:56:48AM +0100, Michael Menge wrote: So, my question is, is it necessary to process the log-PID before sync the log file? Because in my test it looks like it's not necessary. It depends on what actions where done while the syncornisation was down Cyrus does not log every changed mail, but in which mailbox mails where changed. So if log-13835 contains MAILBOX user.gdatmgim.temp and MAILBOX user.gdatmgim and he recieved new mails in the INBOX but did not change mails in the temp folder, the INBOX will be in sync without the log-13835, but not user.gdatmgim.temp. So the replication will at some point catch missed actions from unprocessed logs, but it can take some time (forever in worst case). On the other hand it does not harm to run old logs. Thanks Michael, that's precisely right. We (FastMail) always run old logs first. The advantage of doing it that way is that you will have all the related folders for a COPY together, so avoid re-uploading the message. Also, making sure the first sync includes both folders from any COPY means de-duplication will happen on the replica. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: Sync log replica process
Hi! So, you to be sure, log just log that have been some change in the mailbox, or it has been created or deleted? So, when some change is made, it check all the mailbox to sync the modifications? is there any casuistry where it should necessary that everything is processed in order? first log-PID and then log? Thanks for the response! Manel Gimeno Zaragoza magiz...@hotmail.com Date: Tue, 13 Mar 2012 11:56:48 +0100 From: michael.me...@zdv.uni-tuebingen.de To: info-cyrus@lists.andrew.cmu.edu Subject: Re: Sync log replica process Hi, Quoting Manel Gimeno Zaragozá magiz...@hotmail.com: Hello, I've some doubts about how sync logs and the order about how they should be processed. I've done some bunch tests and during the process stop replic to try to simulate a possible crash. In my test I've saw the following: 1.- I send 10 mails to Master and it starts to replicate. 2.- file log is created and change to log-13835 3.- Replica crash and log-13835 never ends and remains in /var/lib/imap/sync/ forever 4.- while replica is crashed the user is making actions in the 10 mails send before (moving aroung folders, deleting, etc...) and all changes are saved in log file 5.- Replica comes up and i run sync_client -r to sincronize the changes. The question is here. Should I first run sync_client -r -f /var/lib/imap/sync/log-13835 before starts sync_client -r? I understand that the log-13835 has the information about the 10 mail received, and log has information about mail that are not yet in replica. 6.- Anyway, I run sync_client -rwithout process log-13835 and, what a surprise!! everything is on their place and sync works ok with out processing the log-13835 7.- I run my integrity_check based on Bron's integrity check and the only problem I see is a mismatch in highestmodseq for the folder where the user moves the mails. Mismatched 'highestmodseq' for user.gdatmgim.temp master=7, replica=9 This error remain until some action is taken in the folder. So, my question is, is it necessary to process the log-PID before sync the log file? Because in my test it looks like it's not necessary. It depends on what actions where done while the syncornisation was down Cyrus does not log every changed mail, but in which mailbox mails where changed. So if log-13835 contains MAILBOX user.gdatmgim.temp and MAILBOX user.gdatmgim and he recieved new mails in the INBOX but did not change mails in the temp folder, the INBOX will be in sync without the log-13835, but not user.gdatmgim.temp. So the replication will at some point catch missed actions from unprocessed logs, but it can take some time (forever in worst case). On the other hand it does not harm to run old logs. Michael M.MengeTel.: (49) 7071/29-70316 Universität Tübingen Fax.: (49) 7071/29-5912 Zentrum für Datenverarbeitung mail: michael.me...@zdv.uni-tuebingen.de Wächterstraße 76 72074 Tübingen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Sync log replica process
On 03/13/2012 12:46 PM, Bron Gondwana wrote: We (FastMail) always run old logs first. Bron, but this blocks starting up the master till the replica has catched up? Sometimes the sync is broken, or do you detect it? Thanks Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Sync log replica process
On Tue, Mar 13, 2012 at 03:25:38PM +0100, Rudy Gevaert wrote: On 03/13/2012 12:46 PM, Bron Gondwana wrote: We (FastMail) always run old logs first. Bron, but this blocks starting up the master till the replica has catched up? Sometimes the sync is broken, or do you detect it? We don't start sync_client from cyrus.conf - we start it separately. We run a cron job every 10 minutes which checks for files that don't have an associated pid running, and runs them. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
