Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok update from my end

under 2.3.18 (have not upgraded production to 2.3.19 yet)

replication issues as stated before

however i need to note that i had to manually sync a user that was not 
being listed as a replicator fail


this means i have to force a full sync between servers on all accounts 
regardless of replication status


this was discovered this morning on a customers account that did not 
replicate between the servers properly and thus emails were being 
delivered days later because the client was accessing the other server.


its one thing to be 10 minutes late etc but a day late is not practical

again not complaining

I will load 2.3.19 on the test servers and try that and advise, also 
will test for the folder count replication issue as well and advise


please note NO errors are being thrown in the debug log, it reports the 
replication request, gets qued but does not complete??






Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/11/2022 12:25 AM, Cassidy B. Larson wrote:

Hi Aki,

We just installed 2.3.19, and are seeing a couple of users throwing the 
"INBOX/dovecot.index reset, view is now inconsistent" and their 
replicator status erroring out. Tried force-resync on the full mailbox, 
but to no avail just yet.  Not sure if this bug was supposedly fixed in 
2.3.19?


Thanks,

Cassidy

On Thu, Apr 28, 2022 at 5:02 AM Aki Tuomi > wrote:


2.3.19 is round the corner, so not long. I cannot yet promise an
exact date but hopefully within week or two.

Aki

 > On 28/04/2022 13:57 Paul Kudla (SCOM.CA  Internet
Services Inc.) mailto:p...@scom.ca>> wrote:
 >
 >
 > Thanks for the update.
 >
 > is this for both replication issues (folders +300 etc)
 >
 > Just Asking - Any ETA
 >
 >
 >
 >
 >
 > Happy Thursday !!!
 > Thanks - paul
 >
 > Paul Kudla
 >
 >
 > Scom.ca Internet Services >
 > 004-1009 Byron Street South
 > Whitby, Ontario - Canada
 > L1N 4S3
 >
 > Toronto 416.642.7266
 > Main 1.866.411.7266
 > Fax 1.888.892.7266
 >
 > On 4/27/2022 9:01 AM, Aki Tuomi wrote:
 > >
 > > Hi!
 > >
 > > This is probably going to get fixed in 2.3.19, this looks like
an issue we are already fixing.
 > >
 > > Aki
 > >
 > >> On 26/04/2022 16:38 Paul Kudla (SCOM.CA 
Internet Services Inc.) mailto:p...@scom.ca>> wrote:
 > >>
 > >>
 > >> Agreed there seems to be no way of posting these kinds of
issues to see
 > >> if they are even being addressed or even known about moving
forward on
 > >> new updates
 > >>
 > >> i read somewhere there is a new branch soming out but nothing
as of yet?
 > >>
 > >> 2.4 maybe 
 > >> 5.0 
 > >>
 > >> my previous replication issues (back in feb) went unanswered.
 > >>
 > >> not faulting anyone, but the developers do seem to be
disconnected from
 > >> issues as of late? or concentrating on other issues.
 > >>
 > >> I have no problem with support contracts for day to day maintence
 > >> however as a programmer myself they usually dont work as the
other end
 > >> relies on the latest source code anyways. Thus can not help.
 > >>
 > >> I am trying to take a part the replicator c programming based
on 2.3.18
 > >> as most of it does work to some extent.
 > >>
 > >> tcps just does not work (ie 600 seconds default in the c
programming)
 > >>
 > >> My thoughts are tcp works ok but fails when the replicator through
 > >> dsync-client.c when asked to return the folder list?
 > >>
 > >>
 > >> replicator-brain.c seems to control the overall process and
timing.
 > >>
 > >> replicator-queue.c seems to handle the que file that does seem
to carry
 > >> acurate info.
 > >>
 > >>
 > >> things in the source code are documented enough to figure this
out but i
 > >> am still going through all the related .h files documentation
wise which
 > >> are all over the place.
 > >>
 > >> there is no clear documentation on the .h lib files so i have
to walk
 > >> through the tree one at a time finding relative code.
 > >>
 > >> since the dsync from doveadm does see to work ok i have to
assume the
 > >> dsync-client used to compile the replicator is at fault
somehow or a
 > >> call from it upstream?
 > >>
 > >> Thanks for your input on the other issues noted below, i will
keep that
 > >> in mind when disassembling the source code.
 > >>
 > >> No sense in fixing one thing and leaving something else
behind

Re: no full syncs after upgrading to dovecot 2.3.18

[Cassidy B. Larson]

Hi Aki,

We just installed 2.3.19, and are seeing a couple of users throwing the
"INBOX/dovecot.index reset, view is now inconsistent" and their replicator
status erroring out. Tried force-resync on the full mailbox, but to no
avail just yet.  Not sure if this bug was supposedly fixed in 2.3.19?

Thanks,

Cassidy

On Thu, Apr 28, 2022 at 5:02 AM Aki Tuomi 
wrote:

> 2.3.19 is round the corner, so not long. I cannot yet promise an exact
> date but hopefully within week or two.
>
> Aki
>
> > On 28/04/2022 13:57 Paul Kudla (SCOM.CA Internet Services Inc.) <
> p...@scom.ca> wrote:
> >
> >
> > Thanks for the update.
> >
> > is this for both replication issues (folders +300 etc)
> >
> > Just Asking - Any ETA
> >
> >
> >
> >
> >
> > Happy Thursday !!!
> > Thanks - paul
> >
> > Paul Kudla
> >
> >
> > Scom.ca Internet Services 
> > 004-1009 Byron Street South
> > Whitby, Ontario - Canada
> > L1N 4S3
> >
> > Toronto 416.642.7266
> > Main 1.866.411.7266
> > Fax 1.888.892.7266
> >
> > On 4/27/2022 9:01 AM, Aki Tuomi wrote:
> > >
> > > Hi!
> > >
> > > This is probably going to get fixed in 2.3.19, this looks like an
> issue we are already fixing.
> > >
> > > Aki
> > >
> > >> On 26/04/2022 16:38 Paul Kudla (SCOM.CA Internet Services Inc.) <
> p...@scom.ca> wrote:
> > >>
> > >>
> > >> Agreed there seems to be no way of posting these kinds of issues to
> see
> > >> if they are even being addressed or even known about moving forward on
> > >> new updates
> > >>
> > >> i read somewhere there is a new branch soming out but nothing as of
> yet?
> > >>
> > >> 2.4 maybe 
> > >> 5.0 
> > >>
> > >> my previous replication issues (back in feb) went unanswered.
> > >>
> > >> not faulting anyone, but the developers do seem to be disconnected
> from
> > >> issues as of late? or concentrating on other issues.
> > >>
> > >> I have no problem with support contracts for day to day maintence
> > >> however as a programmer myself they usually dont work as the other end
> > >> relies on the latest source code anyways. Thus can not help.
> > >>
> > >> I am trying to take a part the replicator c programming based on
> 2.3.18
> > >> as most of it does work to some extent.
> > >>
> > >> tcps just does not work (ie 600 seconds default in the c programming)
> > >>
> > >> My thoughts are tcp works ok but fails when the replicator through
> > >> dsync-client.c when asked to return the folder list?
> > >>
> > >>
> > >> replicator-brain.c seems to control the overall process and timing.
> > >>
> > >> replicator-queue.c seems to handle the que file that does seem to
> carry
> > >> acurate info.
> > >>
> > >>
> > >> things in the source code are documented enough to figure this out
> but i
> > >> am still going through all the related .h files documentation wise
> which
> > >> are all over the place.
> > >>
> > >> there is no clear documentation on the .h lib files so i have to walk
> > >> through the tree one at a time finding relative code.
> > >>
> > >> since the dsync from doveadm does see to work ok i have to assume the
> > >> dsync-client used to compile the replicator is at fault somehow or a
> > >> call from it upstream?
> > >>
> > >> Thanks for your input on the other issues noted below, i will keep
> that
> > >> in mind when disassembling the source code.
> > >>
> > >> No sense in fixing one thing and leaving something else behind,
> probably
> > >> all related anyways.
> > >>
> > >> i have two test servers avaliable so i can play with all this offline
> to
> > >> reproduce the issues
> > >>
> > >> Unfortunately I have to make a living first, this will be addressed
> when
> > >> possible as i dont like systems that are live running this way and
> > >> currently only have 5 accounts with this issue (mine included)
> > >>
> > >>
> > >>
> > >>
> > >> Happy Tuesday !!!
> > >> Thanks - paul
> > >>
> > >> Paul Kudla
> > >>
> > >>
> > >> Scom.ca Internet Services 
> > >> 004-1009 Byron Street South
> > >> Whitby, Ontario - Canada
> > >> L1N 4S3
> > >>
> > >> Toronto 416.642.7266
> > >> Main 1.866.411.7266
> > >> Fax 1.888.892.7266
> > >>
> > >> On 4/26/2022 9:03 AM, Reuben Farrelly wrote:
> > >>>
> > >>> I ran into this back in February and documented a reproducible test
> case
> > >>> (and sent it to this list).  In short - I was able to reproduce this
> by
> > >>> having a valid and consistent mailbox on the source/local, creating a
> > >>> very standard empty Maildir/(new|cur|tmp) folder on the remote
> replica,
> > >>> and then initiating the replicate from the source. This consistently
> > >>> caused dsync to fail replication with the error "dovecot.index reset,
> > >>> view is now inconsistent" and sync aborted, leaving the replica
> mailbox
> > >>> in a screwed up inconsistent state. Client connections on the source
> > >>> replica were also dropped when this error occurred.  You can see the
> > >>> error by enabling debug level logging if you initiate dsync manually
> on
> > >>> a test mail

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Thanks for the update

I dont push anyone when asking for updates

I am a programmer by trade as well and nothing ever goes as planned

prefer we all take our time and roll it out correctly then jumping the gun.

Why I am trying to help elsewhere as I have gotten pretty fluid with 
dovecot etc and can help users out with the day to day stuff.


I just can't help with ldap, never got around to that as i use pgsql 
databases that are replicated etc etc etc on all my configs.


Again thanks for the update.



Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/28/2022 7:02 AM, Aki Tuomi wrote:


2.3.19 is round the corner, so not long. I cannot yet promise an exact date but 
hopefully within week or two.

Aki


On 28/04/2022 13:57 Paul Kudla (SCOM.CA Internet Services Inc.)  
wrote:

  
Thanks for the update.


is this for both replication issues (folders +300 etc)

Just Asking - Any ETA





Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/27/2022 9:01 AM, Aki Tuomi wrote:


Hi!

This is probably going to get fixed in 2.3.19, this looks like an issue we are 
already fixing.

Aki


On 26/04/2022 16:38 Paul Kudla (SCOM.CA Internet Services Inc.)  
wrote:

   
Agreed there seems to be no way of posting these kinds of issues to see

if they are even being addressed or even known about moving forward on
new updates

i read somewhere there is a new branch soming out but nothing as of yet?

2.4 maybe 
5.0 

my previous replication issues (back in feb) went unanswered.

not faulting anyone, but the developers do seem to be disconnected from
issues as of late? or concentrating on other issues.

I have no problem with support contracts for day to day maintence
however as a programmer myself they usually dont work as the other end
relies on the latest source code anyways. Thus can not help.

I am trying to take a part the replicator c programming based on 2.3.18
as most of it does work to some extent.

tcps just does not work (ie 600 seconds default in the c programming)

My thoughts are tcp works ok but fails when the replicator through
dsync-client.c when asked to return the folder list?


replicator-brain.c seems to control the overall process and timing.

replicator-queue.c seems to handle the que file that does seem to carry
acurate info.


things in the source code are documented enough to figure this out but i
am still going through all the related .h files documentation wise which
are all over the place.

there is no clear documentation on the .h lib files so i have to walk
through the tree one at a time finding relative code.

since the dsync from doveadm does see to work ok i have to assume the
dsync-client used to compile the replicator is at fault somehow or a
call from it upstream?

Thanks for your input on the other issues noted below, i will keep that
in mind when disassembling the source code.

No sense in fixing one thing and leaving something else behind, probably
all related anyways.

i have two test servers avaliable so i can play with all this offline to
reproduce the issues

Unfortunately I have to make a living first, this will be addressed when
possible as i dont like systems that are live running this way and
currently only have 5 accounts with this issue (mine included)




Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/26/2022 9:03 AM, Reuben Farrelly wrote:


I ran into this back in February and documented a reproducible test case
(and sent it to this list).  In short - I was able to reproduce this by
having a valid and consistent mailbox on the source/local, creating a
very standard empty Maildir/(new|cur|tmp) folder on the remote replica,
and then initiating the replicate from the source. This consistently
caused dsync to fail replication with the error "dovecot.index reset,
view is now inconsistent" and sync aborted, leaving the replica mailbox
in a screwed up inconsistent state. Client connections on the source
replica were also dropped when this error occurred.  You can see the
error by enabling debug level logging if you initiate dsync manually on
a test mailbox.

The only workaround I found was to remove the remote Maildir and let
Dovecot create the whole thing from scratch.  Dovecot did not like any
existing folders on the destination replica even if they were the same
names as the source and completely empty.  I was able to reproduce this
the bare minimum of folders - just an INBOX!

I have no idea if any of the developers saw my post or if the bug has
been fix

Re: no full syncs after upgrading to dovecot 2.3.18

[Aki Tuomi]

2.3.19 is round the corner, so not long. I cannot yet promise an exact date but 
hopefully within week or two.

Aki

> On 28/04/2022 13:57 Paul Kudla (SCOM.CA Internet Services Inc.) 
>  wrote:
> 
>  
> Thanks for the update.
> 
> is this for both replication issues (folders +300 etc)
> 
> Just Asking - Any ETA
> 
> 
> 
> 
> 
> Happy Thursday !!!
> Thanks - paul
> 
> Paul Kudla
> 
> 
> Scom.ca Internet Services 
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
> 
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> 
> On 4/27/2022 9:01 AM, Aki Tuomi wrote:
> > 
> > Hi!
> > 
> > This is probably going to get fixed in 2.3.19, this looks like an issue we 
> > are already fixing.
> > 
> > Aki
> > 
> >> On 26/04/2022 16:38 Paul Kudla (SCOM.CA Internet Services Inc.) 
> >>  wrote:
> >>
> >>   
> >> Agreed there seems to be no way of posting these kinds of issues to see
> >> if they are even being addressed or even known about moving forward on
> >> new updates
> >>
> >> i read somewhere there is a new branch soming out but nothing as of yet?
> >>
> >> 2.4 maybe 
> >> 5.0 
> >>
> >> my previous replication issues (back in feb) went unanswered.
> >>
> >> not faulting anyone, but the developers do seem to be disconnected from
> >> issues as of late? or concentrating on other issues.
> >>
> >> I have no problem with support contracts for day to day maintence
> >> however as a programmer myself they usually dont work as the other end
> >> relies on the latest source code anyways. Thus can not help.
> >>
> >> I am trying to take a part the replicator c programming based on 2.3.18
> >> as most of it does work to some extent.
> >>
> >> tcps just does not work (ie 600 seconds default in the c programming)
> >>
> >> My thoughts are tcp works ok but fails when the replicator through
> >> dsync-client.c when asked to return the folder list?
> >>
> >>
> >> replicator-brain.c seems to control the overall process and timing.
> >>
> >> replicator-queue.c seems to handle the que file that does seem to carry
> >> acurate info.
> >>
> >>
> >> things in the source code are documented enough to figure this out but i
> >> am still going through all the related .h files documentation wise which
> >> are all over the place.
> >>
> >> there is no clear documentation on the .h lib files so i have to walk
> >> through the tree one at a time finding relative code.
> >>
> >> since the dsync from doveadm does see to work ok i have to assume the
> >> dsync-client used to compile the replicator is at fault somehow or a
> >> call from it upstream?
> >>
> >> Thanks for your input on the other issues noted below, i will keep that
> >> in mind when disassembling the source code.
> >>
> >> No sense in fixing one thing and leaving something else behind, probably
> >> all related anyways.
> >>
> >> i have two test servers avaliable so i can play with all this offline to
> >> reproduce the issues
> >>
> >> Unfortunately I have to make a living first, this will be addressed when
> >> possible as i dont like systems that are live running this way and
> >> currently only have 5 accounts with this issue (mine included)
> >>
> >>
> >>
> >>
> >> Happy Tuesday !!!
> >> Thanks - paul
> >>
> >> Paul Kudla
> >>
> >>
> >> Scom.ca Internet Services 
> >> 004-1009 Byron Street South
> >> Whitby, Ontario - Canada
> >> L1N 4S3
> >>
> >> Toronto 416.642.7266
> >> Main 1.866.411.7266
> >> Fax 1.888.892.7266
> >>
> >> On 4/26/2022 9:03 AM, Reuben Farrelly wrote:
> >>>
> >>> I ran into this back in February and documented a reproducible test case
> >>> (and sent it to this list).  In short - I was able to reproduce this by
> >>> having a valid and consistent mailbox on the source/local, creating a
> >>> very standard empty Maildir/(new|cur|tmp) folder on the remote replica,
> >>> and then initiating the replicate from the source. This consistently
> >>> caused dsync to fail replication with the error "dovecot.index reset,
> >>> view is now inconsistent" and sync aborted, leaving the replica mailbox
> >>> in a screwed up inconsistent state. Client connections on the source
> >>> replica were also dropped when this error occurred.  You can see the
> >>> error by enabling debug level logging if you initiate dsync manually on
> >>> a test mailbox.
> >>>
> >>> The only workaround I found was to remove the remote Maildir and let
> >>> Dovecot create the whole thing from scratch.  Dovecot did not like any
> >>> existing folders on the destination replica even if they were the same
> >>> names as the source and completely empty.  I was able to reproduce this
> >>> the bare minimum of folders - just an INBOX!
> >>>
> >>> I have no idea if any of the developers saw my post or if the bug has
> >>> been fixed for the next release.  But it seemed to be quite a common
> >>> problem over time (saw a few posts from people going back a long way
> >>> with the same problem) and it is seriously disruptive

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Thanks for the update.

is this for both replication issues (folders +300 etc)

Just Asking - Any ETA





Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/27/2022 9:01 AM, Aki Tuomi wrote:


Hi!

This is probably going to get fixed in 2.3.19, this looks like an issue we are 
already fixing.

Aki


On 26/04/2022 16:38 Paul Kudla (SCOM.CA Internet Services Inc.)  
wrote:

  
Agreed there seems to be no way of posting these kinds of issues to see

if they are even being addressed or even known about moving forward on
new updates

i read somewhere there is a new branch soming out but nothing as of yet?

2.4 maybe 
5.0 

my previous replication issues (back in feb) went unanswered.

not faulting anyone, but the developers do seem to be disconnected from
issues as of late? or concentrating on other issues.

I have no problem with support contracts for day to day maintence
however as a programmer myself they usually dont work as the other end
relies on the latest source code anyways. Thus can not help.

I am trying to take a part the replicator c programming based on 2.3.18
as most of it does work to some extent.

tcps just does not work (ie 600 seconds default in the c programming)

My thoughts are tcp works ok but fails when the replicator through
dsync-client.c when asked to return the folder list?


replicator-brain.c seems to control the overall process and timing.

replicator-queue.c seems to handle the que file that does seem to carry
acurate info.


things in the source code are documented enough to figure this out but i
am still going through all the related .h files documentation wise which
are all over the place.

there is no clear documentation on the .h lib files so i have to walk
through the tree one at a time finding relative code.

since the dsync from doveadm does see to work ok i have to assume the
dsync-client used to compile the replicator is at fault somehow or a
call from it upstream?

Thanks for your input on the other issues noted below, i will keep that
in mind when disassembling the source code.

No sense in fixing one thing and leaving something else behind, probably
all related anyways.

i have two test servers avaliable so i can play with all this offline to
reproduce the issues

Unfortunately I have to make a living first, this will be addressed when
possible as i dont like systems that are live running this way and
currently only have 5 accounts with this issue (mine included)




Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/26/2022 9:03 AM, Reuben Farrelly wrote:


I ran into this back in February and documented a reproducible test case
(and sent it to this list).  In short - I was able to reproduce this by
having a valid and consistent mailbox on the source/local, creating a
very standard empty Maildir/(new|cur|tmp) folder on the remote replica,
and then initiating the replicate from the source. This consistently
caused dsync to fail replication with the error "dovecot.index reset,
view is now inconsistent" and sync aborted, leaving the replica mailbox
in a screwed up inconsistent state. Client connections on the source
replica were also dropped when this error occurred.  You can see the
error by enabling debug level logging if you initiate dsync manually on
a test mailbox.

The only workaround I found was to remove the remote Maildir and let
Dovecot create the whole thing from scratch.  Dovecot did not like any
existing folders on the destination replica even if they were the same
names as the source and completely empty.  I was able to reproduce this
the bare minimum of folders - just an INBOX!

I have no idea if any of the developers saw my post or if the bug has
been fixed for the next release.  But it seemed to be quite a common
problem over time (saw a few posts from people going back a long way
with the same problem) and it is seriously disruptive to clients.  The
error message is not helpful in tracking down the problem either.

Secondly, I also have had an ongoing and longstanding problem using
tcps: for replication.  For some reason using tcps: (with no other
changes at all to the config) results in a lot of timeout messages
"Error: dsync I/O has stalled, no activity for 600 seconds".  This goes
away if I revert back to tcp: instead of tcps - with tcp: I very rarely
get timeouts.  No idea why, guess this is a bug of some sort also.

It's disappointing that there appears to be no way to have these sorts
or problems addressed like there once was.  I am not using Dovecot for
commercial purposes so paying a fortune for a support contract for a
high end installation just isn't going to happen, and this list seems to
b

Re: Better not post your email password on a public mailing list, was: Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

thanks

i love to share but sometime forget whats noted inside a config file

Been meaning to change this for a while anyways.




Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/27/2022 8:57 AM, Daniel Lange wrote:


Am 26.04.22 um 11:36 schrieb Paul Kudla (SCOM.CA Internet Services Inc.):

#imapc_host = mail.scom.ca
#imapc_password = Pk554669
#imapc_user = p...@scom.ca


I suggest to change that password immediately.

$ openssl s_client -crlf -connect mail.scom.ca:993
CONNECTED(0003)
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
LITERAL+ AUTH=PLAIN AUTH=LOGIN] SCOM.CA Internet Services Inc. - Dovecot 
ready

A login p...@scom.ca Pk554669
A OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT 
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT 
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS 
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES 
WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY 
PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] 
Logged in

A status INBOX (messages)
* STATUS INBOX (MESSAGES 344)
A OK Status completed (0.002 + 0.000 + 0.001 secs).
^C

Kind regards,
Daniel

Re: no full syncs after upgrading to dovecot 2.3.18

[Aki Tuomi]

Hi!

This is probably going to get fixed in 2.3.19, this looks like an issue we are 
already fixing.

Aki

> On 26/04/2022 16:38 Paul Kudla (SCOM.CA Internet Services Inc.) 
>  wrote:
> 
>  
> Agreed there seems to be no way of posting these kinds of issues to see 
> if they are even being addressed or even known about moving forward on 
> new updates
> 
> i read somewhere there is a new branch soming out but nothing as of yet?
> 
> 2.4 maybe 
> 5.0 
> 
> my previous replication issues (back in feb) went unanswered.
> 
> not faulting anyone, but the developers do seem to be disconnected from 
> issues as of late? or concentrating on other issues.
> 
> I have no problem with support contracts for day to day maintence 
> however as a programmer myself they usually dont work as the other end 
> relies on the latest source code anyways. Thus can not help.
> 
> I am trying to take a part the replicator c programming based on 2.3.18 
> as most of it does work to some extent.
> 
> tcps just does not work (ie 600 seconds default in the c programming)
> 
> My thoughts are tcp works ok but fails when the replicator through 
> dsync-client.c when asked to return the folder list?
> 
> 
> replicator-brain.c seems to control the overall process and timing.
> 
> replicator-queue.c seems to handle the que file that does seem to carry 
> acurate info.
> 
> 
> things in the source code are documented enough to figure this out but i 
> am still going through all the related .h files documentation wise which 
> are all over the place.
> 
> there is no clear documentation on the .h lib files so i have to walk 
> through the tree one at a time finding relative code.
> 
> since the dsync from doveadm does see to work ok i have to assume the 
> dsync-client used to compile the replicator is at fault somehow or a 
> call from it upstream?
> 
> Thanks for your input on the other issues noted below, i will keep that 
> in mind when disassembling the source code.
> 
> No sense in fixing one thing and leaving something else behind, probably 
> all related anyways.
> 
> i have two test servers avaliable so i can play with all this offline to 
> reproduce the issues
> 
> Unfortunately I have to make a living first, this will be addressed when 
> possible as i dont like systems that are live running this way and 
> currently only have 5 accounts with this issue (mine included)
> 
> 
> 
> 
> Happy Tuesday !!!
> Thanks - paul
> 
> Paul Kudla
> 
> 
> Scom.ca Internet Services 
> 004-1009 Byron Street South
> Whitby, Ontario - Canada
> L1N 4S3
> 
> Toronto 416.642.7266
> Main 1.866.411.7266
> Fax 1.888.892.7266
> 
> On 4/26/2022 9:03 AM, Reuben Farrelly wrote:
> > 
> > I ran into this back in February and documented a reproducible test case 
> > (and sent it to this list).  In short - I was able to reproduce this by 
> > having a valid and consistent mailbox on the source/local, creating a 
> > very standard empty Maildir/(new|cur|tmp) folder on the remote replica, 
> > and then initiating the replicate from the source. This consistently 
> > caused dsync to fail replication with the error "dovecot.index reset, 
> > view is now inconsistent" and sync aborted, leaving the replica mailbox 
> > in a screwed up inconsistent state. Client connections on the source 
> > replica were also dropped when this error occurred.  You can see the 
> > error by enabling debug level logging if you initiate dsync manually on 
> > a test mailbox.
> > 
> > The only workaround I found was to remove the remote Maildir and let 
> > Dovecot create the whole thing from scratch.  Dovecot did not like any 
> > existing folders on the destination replica even if they were the same 
> > names as the source and completely empty.  I was able to reproduce this 
> > the bare minimum of folders - just an INBOX!
> > 
> > I have no idea if any of the developers saw my post or if the bug has 
> > been fixed for the next release.  But it seemed to be quite a common 
> > problem over time (saw a few posts from people going back a long way 
> > with the same problem) and it is seriously disruptive to clients.  The 
> > error message is not helpful in tracking down the problem either.
> > 
> > Secondly, I also have had an ongoing and longstanding problem using 
> > tcps: for replication.  For some reason using tcps: (with no other 
> > changes at all to the config) results in a lot of timeout messages 
> > "Error: dsync I/O has stalled, no activity for 600 seconds".  This goes 
> > away if I revert back to tcp: instead of tcps - with tcp: I very rarely 
> > get timeouts.  No idea why, guess this is a bug of some sort also.
> > 
> > It's disappointing that there appears to be no way to have these sorts 
> > or problems addressed like there once was.  I am not using Dovecot for 
> > commercial purposes so paying a fortune for a support contract for a 
> > high end installation just isn't going to happen, and this list seems to 
> > be quite ordinary fo

Better not post your email password on a public mailing list, was: Re: no full syncs after upgrading to dovecot 2.3.18

[Daniel Lange]

Am 26.04.22 um 11:36 schrieb Paul Kudla (SCOM.CA Internet Services Inc.):

#imapc_host = mail.scom.ca
#imapc_password = Pk554669
#imapc_user = p...@scom.ca


I suggest to change that password immediately.

$ openssl s_client -crlf -connect mail.scom.ca:993
CONNECTED(0003)
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ 
AUTH=PLAIN AUTH=LOGIN] SCOM.CA Internet Services Inc. - Dovecot ready
A login p...@scom.ca Pk554669
A OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT 
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND 
URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED 
I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH 
LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE 
SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] Logged in
A status INBOX (messages)
* STATUS INBOX (MESSAGES 344)
A OK Status completed (0.002 + 0.000 + 0.001 secs).
^C

Kind regards,
Daniel

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Agreed there seems to be no way of posting these kinds of issues to see 
if they are even being addressed or even known about moving forward on 
new updates


i read somewhere there is a new branch soming out but nothing as of yet?

2.4 maybe 
5.0 

my previous replication issues (back in feb) went unanswered.

not faulting anyone, but the developers do seem to be disconnected from 
issues as of late? or concentrating on other issues.


I have no problem with support contracts for day to day maintence 
however as a programmer myself they usually dont work as the other end 
relies on the latest source code anyways. Thus can not help.


I am trying to take a part the replicator c programming based on 2.3.18 
as most of it does work to some extent.


tcps just does not work (ie 600 seconds default in the c programming)

My thoughts are tcp works ok but fails when the replicator through 
dsync-client.c when asked to return the folder list?



replicator-brain.c seems to control the overall process and timing.

replicator-queue.c seems to handle the que file that does seem to carry 
acurate info.



things in the source code are documented enough to figure this out but i 
am still going through all the related .h files documentation wise which 
are all over the place.


there is no clear documentation on the .h lib files so i have to walk 
through the tree one at a time finding relative code.


since the dsync from doveadm does see to work ok i have to assume the 
dsync-client used to compile the replicator is at fault somehow or a 
call from it upstream?


Thanks for your input on the other issues noted below, i will keep that 
in mind when disassembling the source code.


No sense in fixing one thing and leaving something else behind, probably 
all related anyways.


i have two test servers avaliable so i can play with all this offline to 
reproduce the issues


Unfortunately I have to make a living first, this will be addressed when 
possible as i dont like systems that are live running this way and 
currently only have 5 accounts with this issue (mine included)





Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/26/2022 9:03 AM, Reuben Farrelly wrote:


I ran into this back in February and documented a reproducible test case 
(and sent it to this list).  In short - I was able to reproduce this by 
having a valid and consistent mailbox on the source/local, creating a 
very standard empty Maildir/(new|cur|tmp) folder on the remote replica, 
and then initiating the replicate from the source. This consistently 
caused dsync to fail replication with the error "dovecot.index reset, 
view is now inconsistent" and sync aborted, leaving the replica mailbox 
in a screwed up inconsistent state. Client connections on the source 
replica were also dropped when this error occurred.  You can see the 
error by enabling debug level logging if you initiate dsync manually on 
a test mailbox.


The only workaround I found was to remove the remote Maildir and let 
Dovecot create the whole thing from scratch.  Dovecot did not like any 
existing folders on the destination replica even if they were the same 
names as the source and completely empty.  I was able to reproduce this 
the bare minimum of folders - just an INBOX!


I have no idea if any of the developers saw my post or if the bug has 
been fixed for the next release.  But it seemed to be quite a common 
problem over time (saw a few posts from people going back a long way 
with the same problem) and it is seriously disruptive to clients.  The 
error message is not helpful in tracking down the problem either.


Secondly, I also have had an ongoing and longstanding problem using 
tcps: for replication.  For some reason using tcps: (with no other 
changes at all to the config) results in a lot of timeout messages 
"Error: dsync I/O has stalled, no activity for 600 seconds".  This goes 
away if I revert back to tcp: instead of tcps - with tcp: I very rarely 
get timeouts.  No idea why, guess this is a bug of some sort also.


It's disappointing that there appears to be no way to have these sorts 
or problems addressed like there once was.  I am not using Dovecot for 
commercial purposes so paying a fortune for a support contract for a 
high end installation just isn't going to happen, and this list seems to 
be quite ordinary for getting support and reporting bugs nowadays


Reuben

On 26/04/2022 7:21 pm, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:



side issue

if you are getting inconsistant dsyncs there is no real way to fix 
this in the long run.


i know its a pain (already had to my self)

i needed to do a full sync, take one server offline, delete the user 
dir (with dovecot offline) and then rsync (or somehow duplicate the 
main server's user data) over the the remote a

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok until someone fixes the folder count (or i figure that out)

the errors will stay on the replicator side and i use that error 
(knowing they are not replicating) coming back to trigger a background 
dsync with a python script through doveadm (seems to work)


but this is a hack.

[05:41:37] mail18.scom.ca [root:0] /programs/lib
# sync.users
n...@elirpa.com   high 00:06:41  503:54:00 - 
y
ke...@elirpa.com  high 00:06:20  503:52:42 - 
y
p...@scom.ca  high 00:06:40  503:53:50 - 
y
e...@scom.cahigh 00:06:42  503:54:00 - 
y
ed.ha...@dssmgmt.com  high 00:06:41  503:53:57 - 
y
p...@paulkudla.nethigh 00:06:43  503:54:02 620:42:06 
   y


[05:41:46] mail18.scom.ca [root:0] /programs/lib
# cat /programs/common/sync.users
doveadm replicator status '*' | grep '   y'





crontab :

*/10****root/usr/bin/nohup 
/programs/common/sync.recover > /dev/null



# cat /programs/common/sync.recover
#!/usr/local/bin/python3

#Force sync between servers that are reporting bad?

import os,sys,django,socket
from optparse import OptionParser


from lib import *

#Sample Re-Index MB
#doveadm -D force-resync -u p...@scom.ca -f INBOX*



USAGE_TEXT = '''\
usage: %%prog %s[options]
'''

parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')

parser.add_option("-m", "--send_to", dest="send_to", help="Send Email To")
parser.add_option("-e", "--email", dest="email_box", help="Box to Index")
parser.add_option("-d", "--detail",action='store_true', 
dest="detail",default =False, help="Detailed report")
parser.add_option("-i", "--index",action='store_true', 
dest="index",default =False, help="Index")


options, args = parser.parse_args()

print (options.email_box)
print (options.send_to)
print (options.detail)

#sys.exit()



print ('Getting Current User Sync Status')
command = commands("/usr/local/bin/doveadm replicator status '*'")


#print command

sync_user_status = command.output.split('\n')

#print sync_user_status

synced = []

for n in range(1,len(sync_user_status)) :
user = sync_user_status[n]
print ('Processing User : %s' %user.split(' ')[0])
if user.split(' ')[0] != options.email_box :
if options.email_box != None :
continue

if options.index == True :
command = '/usr/local/bin/doveadm -D force-resync -u %s 
-f INBOX*' %user.split(' ')[0]

command = commands(command)
command = command.output

#print user
for nn in range (len(user)-1,0,-1) :
#print nn
#print user[nn]

if user[nn] == '-' :
#print 'skipping ... %s' %user.split(' ')[0]

break



if user[nn] == 'y': #Found a Bad Mailbox
print ('syncing ... %s' %user.split(' ')[0])


if options.detail == True :
command = '/usr/local/bin/doveadm -D 
sync -u %s -d -N -l 30 -U' %user.split(' ')[0]

print (command)
command = commands(command)
command = command.output.split('\n')
print (command)
print ('Processed Mailbox for ... %s' 
%user.split(' ')[0] )
synced.append('Processed Mailbox for 
... %s' %user.split(' ')[0])

for nnn in range(len(command)):
synced.append(command[nnn] + '\n')
break


if options.detail == False :
#command = '/usr/local/bin/doveadm -D 
sync -u %s -d -N -l 30 -U' %user.split(' ')[0]

#print (command)
#command = os.system(command)
command = subprocess.Popen( 
["/usr/local/bin/doveadm sync -u %s -d -N -l 30 -U" %user.split(' ')[0] ], \
shell = True, stdin=None, stdout=None, 
stderr=None, close_fds=True)


print ( 'Processed Mailbox for ... %s' 
%user.split(' ')[0] )
synced.append('Processed Mailbox for 
... %s' %user.split(' ')[0])

#sys.exit()
break

if len(synced) != 0 :
#send email showing bad synced boxes ?

if options.send_to != None :
send_from = 'moni...@scom.ca'
send_to = ['%s' %options.send_to]
send_subject = 'Dovecot Bad Sync Report for : %s' 
%(socket.gethostname())

send_text = '\n\n'
for n in range (len(synced)) :
send_tex

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

more specific to this issue

i looks like (at this was fun for me to figure out as well)

note replication does not work well on nfs file systems etc

i started with sdram drives on one server and nfs on the other and found 
i simply had to go sdram (or whatever) on the other one


smoothed all of this out a lot.

basically both servers need to be the same at the end of the day.


the replicator can be a bit fun to setup

i found tcpip (no ssl) worked best

i run one config file (not the 10- etc)  so here is one side, other side 
is the same except for the ip replicator address connection just set 
accordingly, and i run a local backbone network hence the 10.221.0./16 
which also smooths things out as there is only replication traffic & 
auth traffic running across this link. No bottlenecks at the end of the day.


i included sni, postgresql & sieve (for duplicates) as well for the 
complete picture.


took three months to get this going, mainly due to outdated documentation

dovecot works way better then cyrus and supports sni but current 
(2.3.18) complete documentation from setting up beginning to end would help!


I program for a living and even with me documentation always seems to 
take a back seat.


note that sni loades from a database and i wrote a python script to do 
that to support auto updating of yearly ssl certs.


/programs/common/getssl.cert


# cat dovecot.conf
# 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 12.1-RELEASE amd64
# Hostname: mail18.scom.ca

auth_debug = no
auth_debug_passwords = no

default_process_limit = 16384

mail_debug = no

#lock_method = dotlock
#mail_max_lock_timeout = 300s

#mbox_read_locks = dotlock
#mbox_write_locks = dotlock

mmap_disable = yes
dotlock_use_excl = no
mail_fsync = always
mail_nfs_storage = no
mail_nfs_index = no

auth_mechanisms = plain login
auth_verbose = yes
base_dir = /data/dovecot/run/
debug_log_path = syslog
disable_plaintext_auth = no
dsync_features = empty-header-workaround

#imapc_features = rfc822.size fetch-headers
#imapc_host = mail.scom.ca
#imapc_password = Pk554669
#imapc_user = p...@scom.ca

info_log_path = syslog
login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c


mail_location = maildir:~/

mail_plugins = " virtual notify replication fts fts_lucene "
mail_prefetch_count = 20

protocols = imap pop3 lmtp sieve


protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = moni...@scom.ca
}

service lmtp {
  process_limit=1000
  vsz_limit = 512m
  client_limit=1
   unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
 group = postfix
 mode = 0600
 user = postfix
  }
}

protocol lda {
  mail_plugins = $mail_plugins sieve
}

service lda {
  process_limit=1000
  vsz_limit = 512m
}

service imap {
  process_limit=4096
  vsz_limit = 2g
  client_limit=1
}

service pop3 {
  process_limit=1000
  vsz_limit = 512m
  client_limit=1
}

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Sent {
auto = subscribe
special_use = \Sent
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix =
  separator = /
}

passdb {
  args = /usr/local/etc/dovecot/dovecot-pgsql.conf
  driver = sql
}

doveadm_port = 12345
doveadm_password = secretxyy

service doveadm {
  process_limit = 0
  process_min_avail = 0
  idle_kill = 0
  client_limit = 1
  user = vmail
  inet_listener {
port = 12345
  }
}

service config {
  unix_listener config {
user = vmail
}
}

dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u
#dsync_remote_cmd = doveadm sync -d -u%u

replication_dsync_parameters = -d -N -l 300 -U

plugin {
  mail_log_events = delete undelete expunge copy mailbox_delete 
mailbox_rename

  mail_log_fields = uid, box, msgid, from, subject, size, vsize, flags
  push_notification_driver = dlog

  sieve = file:~/sieve;active=~/sieve/.dovecot.sieve
  #sieve = ~/.dovecot.sieve
  sieve_duplicate_default_period = 1h
  sieve_duplicate_max_period = 1d
  sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
  sieve_global_dir = /usr/local/etc/dovecot/sieve
  sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve


  mail_replica = tcp:10.221.0.19:12345
  #mail_replica = remote:vmail@10.221.0.19
  #replication_sync_timeout = 2

  fts = lucene
  fts_lucene = whitespace_chars=@.
}

#sieve_extensions = vnd.dovecot.duplicate

#sieve_plugins = vnd.dovecot.duplicate

service anvil {
  process_limit = 1
  client_limit=5000
  vsz_limit = 512m
  unix_listener anvil {
group = vmail
mode = 0666
  }
}


service auth {
   process_limit = 1
   client_limit=5000
   vsz_limit = 1g

   unix_listener auth-userdb {
  mode = 0660
  user = vmail
  group = vmail
   }
   unix_listener /var/spool/postfix/private/auth {
  mode = 0666
   }

}

service stats {
  process_limit = 1000

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

side issue

if you are getting inconsistant dsyncs there is no real way to fix this 
in the long run.


i know its a pain (already had to my self)

i needed to do a full sync, take one server offline, delete the user dir 
(with dovecot offline) and then rsync (or somehow duplicate the main 
server's user data) over the the remote again.


then bring remote back up and it kind or worked worked

best suggestion is to bring the main server down at night so the copy is 
clean?


if using postfix you can enable the soft bounce option and the mail will 
back spool until everything comes back online


(needs to be enable on bother servers)

replication was still an issue on accounts with 300+ folders in them, 
still working on a fix for that.



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/25/2022 10:01 AM, Arnaud Abélard wrote:
Ah, I'm now getting errors in the logs, that would explains the 
increasing number of failed sync requests:


dovecot: imap(x)<2961235>: Error: 
Mailbox INBOX: /vmail/l/i/x/dovecot.index reset, view is now 
inconsistent



And sure enough:

# dovecot replicator status x

x none 00:02:54  07:11:28  -    y


What could explain that error?

Arnaud



On 25/04/2022 15:13, Arnaud Abélard wrote:

Hello,

On my side we are running Linux (Debian Buster).

I'm not sure my problem is actually the same as Paul or you Sebastian 
since I have a lot of boxes but those are actually small (quota of 
110MB) so I doubt any of them have more than a dozen imap folders.


The main symptom is that I have tons of full sync requests awaiting 
but even though no other sync is pending the replicator just waits for 
something to trigger those syncs.


Today, with users back I can see that normal and incremental syncs are 
being done on the 15 connections, with an occasional full sync here or 
there and lots of "Waiting 'failed' requests":


Queued 'sync' requests    0

Queued 'high' requests    0

Queued 'low' requests 0

Queued 'failed' requests  122

Queued 'full resync' requests 28785

Waiting 'failed' requests 4294

Total number of known users   42512



So, why didn't the replicator take advantage of the weekend to 
replicate the mailboxes while no user were using them?


Arnaud




On 25/04/2022 13:54, Sebastian Marske wrote:

Hi there,

thanks for your insights and for diving deeper into this Paul!

For me, the users ending up in 'Waiting for dsync to finish' all have
more than 256 Imap folders as well (ranging from 288 up to >5500; as per
'doveadm mailbox list -u  | wc -l'). For more details on my
setup please see my post from February [1].

@Arnaud: What OS are you running on?


Best
Sebastian


[1] https://dovecot.org/pipermail/dovecot/2022-February/124168.html


On 4/24/22 19:36, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:


Question having similiar replication issues

pls read everything below and advise the folder counts on the
non-replicated users?

i find  the total number of folders / account seems to be a factor and
NOT the size of the mail box

ie i have customers with 40G of emails no problem over 40 or so folders
and it works ok

300+ folders seems to be the issue

i have been going through the replication code

no errors being logged

i am assuming that the replication --> dhclient --> other server is
timing out or not reading the folder lists correctly (ie dies after X
folders read)

thus i am going through the code patching for log entries etc to find
the issues.

see

[13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x  2 root  wheel  uarch    4B Apr 24 11:11 .
drwxr-xr-x  4 root  wheel  uarch    4B Mar  8  2021 ..
-rw-r--r--  1 root  wheel  uarch   73B Apr 24 11:11 instances
-rw-r--r--  1 root  wheel  uarch  160K Apr 24 13:33 replicator.db

[13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
#

replicator.db seems to get updated ok but never processed properly.

# sync.users
n...@elirpa.com   high 00:09:41  463:47:01 -     y
ke...@elirpa.com  high 00:09:23  463:45:43 -     y
p...@scom.ca  high 00:09:41  463:46:51 -     y
e...@scom.ca    high 00:09:43  463:47:01 -     y
ed.ha...@dssmgmt.com  high 00:09:42  463:46:58 -     y
p...@paulkudla.net    high 00:09:44  463:47:03 
580:35:07

    y




so 



two things :

first to get the production stuff to work i had to write a script that
whould find the bad sync's and the force a dsync between the servers

i run this every five minutes or each server.

in crontab

*/10    *    *    *    *    root    /usr/bin/nohup
/programs/common/sync.recover > /dev/null


python script to sort things out

# cat /programs/common/sync.re

Re: no full syncs after upgrading to dovecot 2.3.18

[Arnaud Abélard]

Ah, I'm now getting errors in the logs, that would explains the 
increasing number of failed sync requests:


dovecot: imap(x)<2961235>: Error: 
Mailbox INBOX: /vmail/l/i/x/dovecot.index reset, view is now 
inconsistent



And sure enough:

# dovecot replicator status x

x none 00:02:54  07:11:28  -y


What could explain that error?

Arnaud



On 25/04/2022 15:13, Arnaud Abélard wrote:

Hello,

On my side we are running Linux (Debian Buster).

I'm not sure my problem is actually the same as Paul or you Sebastian 
since I have a lot of boxes but those are actually small (quota of 
110MB) so I doubt any of them have more than a dozen imap folders.


The main symptom is that I have tons of full sync requests awaiting but 
even though no other sync is pending the replicator just waits for 
something to trigger those syncs.


Today, with users back I can see that normal and incremental syncs are 
being done on the 15 connections, with an occasional full sync here or 
there and lots of "Waiting 'failed' requests":


Queued 'sync' requests    0

Queued 'high' requests    0

Queued 'low' requests 0

Queued 'failed' requests  122

Queued 'full resync' requests 28785

Waiting 'failed' requests 4294

Total number of known users   42512



So, why didn't the replicator take advantage of the weekend to replicate 
the mailboxes while no user were using them?


Arnaud




On 25/04/2022 13:54, Sebastian Marske wrote:

Hi there,

thanks for your insights and for diving deeper into this Paul!

For me, the users ending up in 'Waiting for dsync to finish' all have
more than 256 Imap folders as well (ranging from 288 up to >5500; as per
'doveadm mailbox list -u  | wc -l'). For more details on my
setup please see my post from February [1].

@Arnaud: What OS are you running on?


Best
Sebastian


[1] https://dovecot.org/pipermail/dovecot/2022-February/124168.html


On 4/24/22 19:36, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:


Question having similiar replication issues

pls read everything below and advise the folder counts on the
non-replicated users?

i find  the total number of folders / account seems to be a factor and
NOT the size of the mail box

ie i have customers with 40G of emails no problem over 40 or so folders
and it works ok

300+ folders seems to be the issue

i have been going through the replication code

no errors being logged

i am assuming that the replication --> dhclient --> other server is
timing out or not reading the folder lists correctly (ie dies after X
folders read)

thus i am going through the code patching for log entries etc to find
the issues.

see

[13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x  2 root  wheel  uarch    4B Apr 24 11:11 .
drwxr-xr-x  4 root  wheel  uarch    4B Mar  8  2021 ..
-rw-r--r--  1 root  wheel  uarch   73B Apr 24 11:11 instances
-rw-r--r--  1 root  wheel  uarch  160K Apr 24 13:33 replicator.db

[13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
#

replicator.db seems to get updated ok but never processed properly.

# sync.users
n...@elirpa.com   high 00:09:41  463:47:01 -     y
ke...@elirpa.com  high 00:09:23  463:45:43 -     y
p...@scom.ca  high 00:09:41  463:46:51 -     y
e...@scom.ca    high 00:09:43  463:47:01 -     y
ed.ha...@dssmgmt.com  high 00:09:42  463:46:58 -     y
p...@paulkudla.net    high 00:09:44  463:47:03 580:35:07
    y




so 



two things :

first to get the production stuff to work i had to write a script that
whould find the bad sync's and the force a dsync between the servers

i run this every five minutes or each server.

in crontab

*/10    *    *    *    *    root    /usr/bin/nohup
/programs/common/sync.recover > /dev/null


python script to sort things out

# cat /programs/common/sync.recover
#!/usr/local/bin/python3

#Force sync between servers that are reporting bad?

import os,sys,django,socket
from optparse import OptionParser


from lib import *

#Sample Re-Index MB
#doveadm -D force-resync -u p...@scom.ca -f INBOX*



USAGE_TEXT = '''\
usage: %%prog %s[options]
'''

parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')

parser.add_option("-m", "--send_to", dest="send_to", help="Send Email 
To")
parser.add_option("-e", "--email", dest="email_box", help="Box to 
Index")

parser.add_option("-d", "--detail",action='store_true',
dest="detail",default =False, help="Detailed report")
parser.add_option("-i", "--index",action='store_true',
dest="index",default =False, help="Index")

options, args = parser.parse_args()

print (options.email_box)
print (options.send_to)
print (options.detail)

#sys.exit()



print ('Getting Current User Sync Status')
command = commands("/usr/local/bin/doveadm replicator status '*'")


#print command

sync_user_status = command.output.split('\n')

#

Re: no full syncs after upgrading to dovecot 2.3.18

[Arnaud Abélard]

Hello,

On my side we are running Linux (Debian Buster).

I'm not sure my problem is actually the same as Paul or you Sebastian 
since I have a lot of boxes but those are actually small (quota of 
110MB) so I doubt any of them have more than a dozen imap folders.


The main symptom is that I have tons of full sync requests awaiting but 
even though no other sync is pending the replicator just waits for 
something to trigger those syncs.


Today, with users back I can see that normal and incremental syncs are 
being done on the 15 connections, with an occasional full sync here or 
there and lots of "Waiting 'failed' requests":


Queued 'sync' requests0

Queued 'high' requests0

Queued 'low' requests 0

Queued 'failed' requests  122

Queued 'full resync' requests 28785

Waiting 'failed' requests 4294

Total number of known users   42512



So, why didn't the replicator take advantage of the weekend to replicate 
the mailboxes while no user were using them?


Arnaud




On 25/04/2022 13:54, Sebastian Marske wrote:

Hi there,

thanks for your insights and for diving deeper into this Paul!

For me, the users ending up in 'Waiting for dsync to finish' all have
more than 256 Imap folders as well (ranging from 288 up to >5500; as per
'doveadm mailbox list -u  | wc -l'). For more details on my
setup please see my post from February [1].

@Arnaud: What OS are you running on?


Best
Sebastian


[1] https://dovecot.org/pipermail/dovecot/2022-February/124168.html


On 4/24/22 19:36, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:


Question having similiar replication issues

pls read everything below and advise the folder counts on the
non-replicated users?

i find  the total number of folders / account seems to be a factor and
NOT the size of the mail box

ie i have customers with 40G of emails no problem over 40 or so folders
and it works ok

300+ folders seems to be the issue

i have been going through the replication code

no errors being logged

i am assuming that the replication --> dhclient --> other server is
timing out or not reading the folder lists correctly (ie dies after X
folders read)

thus i am going through the code patching for log entries etc to find
the issues.

see

[13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x  2 root  wheel  uarch    4B Apr 24 11:11 .
drwxr-xr-x  4 root  wheel  uarch    4B Mar  8  2021 ..
-rw-r--r--  1 root  wheel  uarch   73B Apr 24 11:11 instances
-rw-r--r--  1 root  wheel  uarch  160K Apr 24 13:33 replicator.db

[13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
#

replicator.db seems to get updated ok but never processed properly.

# sync.users
n...@elirpa.com   high 00:09:41  463:47:01 -     y
ke...@elirpa.com  high 00:09:23  463:45:43 -     y
p...@scom.ca  high 00:09:41  463:46:51 -     y
e...@scom.ca    high 00:09:43  463:47:01 -     y
ed.ha...@dssmgmt.com  high 00:09:42  463:46:58 -     y
p...@paulkudla.net    high 00:09:44  463:47:03 580:35:07
    y




so 



two things :

first to get the production stuff to work i had to write a script that
whould find the bad sync's and the force a dsync between the servers

i run this every five minutes or each server.

in crontab

*/10    *    *    *    *    root    /usr/bin/nohup
/programs/common/sync.recover > /dev/null


python script to sort things out

# cat /programs/common/sync.recover
#!/usr/local/bin/python3

#Force sync between servers that are reporting bad?

import os,sys,django,socket
from optparse import OptionParser


from lib import *

#Sample Re-Index MB
#doveadm -D force-resync -u p...@scom.ca -f INBOX*



USAGE_TEXT = '''\
usage: %%prog %s[options]
'''

parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')

parser.add_option("-m", "--send_to", dest="send_to", help="Send Email To")
parser.add_option("-e", "--email", dest="email_box", help="Box to Index")
parser.add_option("-d", "--detail",action='store_true',
dest="detail",default =False, help="Detailed report")
parser.add_option("-i", "--index",action='store_true',
dest="index",default =False, help="Index")

options, args = parser.parse_args()

print (options.email_box)
print (options.send_to)
print (options.detail)

#sys.exit()



print ('Getting Current User Sync Status')
command = commands("/usr/local/bin/doveadm replicator status '*'")


#print command

sync_user_status = command.output.split('\n')

#print sync_user_status

synced = []

for n in range(1,len(sync_user_status)) :
     user = sync_user_status[n]
     print ('Processing User : %s' %user.split(' ')[0])
     if user.split(' ')[0] != options.email_box :
     if options.email_box != None :
     continue

     if options.index == True :
     command = '/usr/local/bin/doveadm -D force-resync -u %s
-f INBOX*' 

Re: no full syncs after upgrading to dovecot 2.3.18

[Sebastian Marske]

Hi there,

thanks for your insights and for diving deeper into this Paul!

For me, the users ending up in 'Waiting for dsync to finish' all have
more than 256 Imap folders as well (ranging from 288 up to >5500; as per
'doveadm mailbox list -u  | wc -l'). For more details on my
setup please see my post from February [1].

@Arnaud: What OS are you running on?


Best
Sebastian


[1] https://dovecot.org/pipermail/dovecot/2022-February/124168.html


On 4/24/22 19:36, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:
> 
> Question having similiar replication issues
> 
> pls read everything below and advise the folder counts on the
> non-replicated users?
> 
> i find  the total number of folders / account seems to be a factor and
> NOT the size of the mail box
> 
> ie i have customers with 40G of emails no problem over 40 or so folders
> and it works ok
> 
> 300+ folders seems to be the issue
> 
> i have been going through the replication code
> 
> no errors being logged
> 
> i am assuming that the replication --> dhclient --> other server is
> timing out or not reading the folder lists correctly (ie dies after X
> folders read)
> 
> thus i am going through the code patching for log entries etc to find
> the issues.
> 
> see
> 
> [13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
> # ll
> total 86
> drwxr-xr-x  2 root  wheel  uarch    4B Apr 24 11:11 .
> drwxr-xr-x  4 root  wheel  uarch    4B Mar  8  2021 ..
> -rw-r--r--  1 root  wheel  uarch   73B Apr 24 11:11 instances
> -rw-r--r--  1 root  wheel  uarch  160K Apr 24 13:33 replicator.db
> 
> [13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
> #
> 
> replicator.db seems to get updated ok but never processed properly.
> 
> # sync.users
> n...@elirpa.com   high 00:09:41  463:47:01 -     y
> ke...@elirpa.com  high 00:09:23  463:45:43 -     y
> p...@scom.ca  high 00:09:41  463:46:51 -     y
> e...@scom.ca    high 00:09:43  463:47:01 -     y
> ed.ha...@dssmgmt.com  high 00:09:42  463:46:58 -     y
> p...@paulkudla.net    high 00:09:44  463:47:03 580:35:07
>    y
> 
> 
> 
> 
> so 
> 
> 
> 
> two things :
> 
> first to get the production stuff to work i had to write a script that
> whould find the bad sync's and the force a dsync between the servers
> 
> i run this every five minutes or each server.
> 
> in crontab
> 
> */10    *    *    *    *    root    /usr/bin/nohup
> /programs/common/sync.recover > /dev/null
> 
> 
> python script to sort things out
> 
> # cat /programs/common/sync.recover
> #!/usr/local/bin/python3
> 
> #Force sync between servers that are reporting bad?
> 
> import os,sys,django,socket
> from optparse import OptionParser
> 
> 
> from lib import *
> 
> #Sample Re-Index MB
> #doveadm -D force-resync -u p...@scom.ca -f INBOX*
> 
> 
> 
> USAGE_TEXT = '''\
> usage: %%prog %s[options]
> '''
> 
> parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')
> 
> parser.add_option("-m", "--send_to", dest="send_to", help="Send Email To")
> parser.add_option("-e", "--email", dest="email_box", help="Box to Index")
> parser.add_option("-d", "--detail",action='store_true',
> dest="detail",default =False, help="Detailed report")
> parser.add_option("-i", "--index",action='store_true',
> dest="index",default =False, help="Index")
> 
> options, args = parser.parse_args()
> 
> print (options.email_box)
> print (options.send_to)
> print (options.detail)
> 
> #sys.exit()
> 
> 
> 
> print ('Getting Current User Sync Status')
> command = commands("/usr/local/bin/doveadm replicator status '*'")
> 
> 
> #print command
> 
> sync_user_status = command.output.split('\n')
> 
> #print sync_user_status
> 
> synced = []
> 
> for n in range(1,len(sync_user_status)) :
>     user = sync_user_status[n]
>     print ('Processing User : %s' %user.split(' ')[0])
>     if user.split(' ')[0] != options.email_box :
>     if options.email_box != None :
>     continue
> 
>     if options.index == True :
>     command = '/usr/local/bin/doveadm -D force-resync -u %s
> -f INBOX*' %user.split(' ')[0]
>     command = commands(command)
>     command = command.output
> 
>     #print user
>     for nn in range (len(user)-1,0,-1) :
>     #print nn
>     #print user[nn]
> 
>     if user[nn] == '-' :
>     #print 'skipping ... %s' %user.split(' ')[0]
> 
>     break
> 
> 
> 
>     if user[nn] == 'y': #Found a Bad Mailbox
>     print ('syncing ... %s' %user.split(' ')[0])
> 
> 
>     if options.detail == True :
>     command = '/usr/local/bin/doveadm -D
> sync -u %s -d -N -l 30 -U' %user.split(' ')[0]
>     print (command)
>     command = co

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Question having similiar replication issues

pls read everything below and advise the folder counts on the 
non-replicated users?


i find  the total number of folders / account seems to be a factor and 
NOT the size of the mail box


ie i have customers with 40G of emails no problem over 40 or so folders 
and it works ok


300+ folders seems to be the issue

i have been going through the replication code

no errors being logged

i am assuming that the replication --> dhclient --> other server is 
timing out or not reading the folder lists correctly (ie dies after X 
folders read)


thus i am going through the code patching for log entries etc to find 
the issues.


see

[13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x  2 root  wheel  uarch4B Apr 24 11:11 .
drwxr-xr-x  4 root  wheel  uarch4B Mar  8  2021 ..
-rw-r--r--  1 root  wheel  uarch   73B Apr 24 11:11 instances
-rw-r--r--  1 root  wheel  uarch  160K Apr 24 13:33 replicator.db

[13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
#

replicator.db seems to get updated ok but never processed properly.

# sync.users
n...@elirpa.com   high 00:09:41  463:47:01 - 
y
ke...@elirpa.com  high 00:09:23  463:45:43 - 
y
p...@scom.ca  high 00:09:41  463:46:51 - 
y
e...@scom.cahigh 00:09:43  463:47:01 - 
y
ed.ha...@dssmgmt.com  high 00:09:42  463:46:58 - 
y
p...@paulkudla.nethigh 00:09:44  463:47:03 580:35:07 
   y





so 



two things :

first to get the production stuff to work i had to write a script that 
whould find the bad sync's and the force a dsync between the servers


i run this every five minutes or each server.

in crontab

*/10****root/usr/bin/nohup 
/programs/common/sync.recover > /dev/null



python script to sort things out

# cat /programs/common/sync.recover
#!/usr/local/bin/python3

#Force sync between servers that are reporting bad?

import os,sys,django,socket
from optparse import OptionParser


from lib import *

#Sample Re-Index MB
#doveadm -D force-resync -u p...@scom.ca -f INBOX*



USAGE_TEXT = '''\
usage: %%prog %s[options]
'''

parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')

parser.add_option("-m", "--send_to", dest="send_to", help="Send Email To")
parser.add_option("-e", "--email", dest="email_box", help="Box to Index")
parser.add_option("-d", "--detail",action='store_true', 
dest="detail",default =False, help="Detailed report")
parser.add_option("-i", "--index",action='store_true', 
dest="index",default =False, help="Index")


options, args = parser.parse_args()

print (options.email_box)
print (options.send_to)
print (options.detail)

#sys.exit()



print ('Getting Current User Sync Status')
command = commands("/usr/local/bin/doveadm replicator status '*'")


#print command

sync_user_status = command.output.split('\n')

#print sync_user_status

synced = []

for n in range(1,len(sync_user_status)) :
user = sync_user_status[n]
print ('Processing User : %s' %user.split(' ')[0])
if user.split(' ')[0] != options.email_box :
if options.email_box != None :
continue

if options.index == True :
command = '/usr/local/bin/doveadm -D force-resync -u %s 
-f INBOX*' %user.split(' ')[0]

command = commands(command)
command = command.output

#print user
for nn in range (len(user)-1,0,-1) :
#print nn
#print user[nn]

if user[nn] == '-' :
#print 'skipping ... %s' %user.split(' ')[0]

break



if user[nn] == 'y': #Found a Bad Mailbox
print ('syncing ... %s' %user.split(' ')[0])


if options.detail == True :
command = '/usr/local/bin/doveadm -D 
sync -u %s -d -N -l 30 -U' %user.split(' ')[0]

print (command)
command = commands(command)
command = command.output.split('\n')
print (command)
print ('Processed Mailbox for ... %s' 
%user.split(' ')[0] )
synced.append('Processed Mailbox for 
... %s' %user.split(' ')[0])

for nnn in range(len(command)):
synced.append(command[nnn] + '\n')
break


if options.detail == False :
#command = '/usr/local/bin/doveadm -D 
sync -u %s -d -N -l 30 -U' %user.split(' ')[0]

#print (command)
#command = os.system(command)