Re: Sieve scripts not replicated
Hi, I also experience this on dovecot 2.3.2.1 / pigeonhole 0.5.2. JL On 2018-10-12 16:32, Tony wrote: Hi, FWIW this was also reported back in August[1] - experienced with 2.3.2.1 (not sure if earlier 2.3 releases were affected) and currently reproducible with latest 2.3.3 stable release. [1] https://www.dovecot.org/pipermail/dovecot/2018-August/112548.html Cheers, Tony On 2018-10-12 04:26, Pascal wrote: Hello, I use dovecot replication and the sieve scripts are not replicated. Mail replication is working fine. Log when sieve script (with Rainloop webmail) is created: Oct 12 12:57:57 srv1 dovecot: managesieve-login: Login: user=, method=PLAIN, rip=91.67.174.186, lip=195.201.251.57, mpid=5360, TLS, session= Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Loading modules from directory: /usr/local/lib/dovecot Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Module loaded: /usr/local/lib/dovecot/lib01_acl_plugin.so Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Module loaded: /usr/local/lib/dovecot/lib10_mail_crypt_plugin.so Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Module loaded: /usr/local/lib/dovecot/lib10_quota_plugin.so Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Module loaded: /usr/local/lib/dovecot/lib15_notify_plugin.so Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Module loaded: /usr/local/lib/dovecot/lib20_mail_log_plugin.so Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Module loaded: /usr/local/lib/dovecot/lib20_quota_clone_plugin.so Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Module loaded: /usr/local/lib/dovecot/lib20_replication_plugin.so Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Added userdb setting: plugin/mail_crypt_private_password= Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Added userdb setting: plugin/mail_crypt_save_version=0 Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Added userdb setting: plugin/quota_rule=*:bytes=0 Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Effective uid=998, gid=998, home=/srv/mail/example.com/hativ Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Quota root: name= backend=count args= Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Quota rule: root= mailbox=* bytes=0 messages=0 Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Quota grace: root= bytes=0 (10%) Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Namespace INBOX: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes, subscriptions=yes location=sdbox:~/ Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: fs: root=/srv/mail/example.com/hativ, index=, indexpvt=, control=, inbox=, alt= Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: acl: initializing backend with data: vfile Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: acl: acl username = ha...@example.com Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: acl: owner = 1 Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: acl vfile: Global ACLs disabled Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: Namespace Shared: type=shared, prefix=Shared/%u/, sep=, inbox=no, hidden=no, list=children, subscriptions=yes location=sdbox:%h/:INDEXPVT=~/shares/%u Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: shared: root=/usr/local/var/run/dovecot, index=, indexpvt=, control=, inbox=, alt= Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: acl: initializing backend with data: vfile Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: acl: acl username = ha...@example.com Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: acl: owner = 0 Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: acl vfile: Global ACLs disabled Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: sieve: Pigeonhole version 0.5.3 (f018bbab) initializing Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: sieve: include: sieve_global is not set; it is currently not possible to include `:global' scripts. Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: sieve: Sieve Extprograms plugin for Pigeonhole version 0.5.3 (f018bbab) loaded Oct 12 12:57:57 srv1 dovecot: managesieve(ha...@example.com)<5360>: Debug: sieve: Sieve imapsieve plugin for Pigeonhole version
Dsync config help
Hi, I’m having a hard time with the http://wiki2.dovecot.org/Replication page. - for a master-master setup, does the configuration need to be mirrored on both masters? - after aggregating unrelated sections of the wiki page, there seems to be 3 different values for “mail_replica": - "remote”: for SSH - “remoteprefix": for SSH wrapper - “tcp”: for TCP connection using the Doveadm protocol Is this correct? - what’s the purpose of the replicator VS aggregator VS doveadm services? Who talks to who? Where can I find documentation about their configuration (i.e. not examples) - most examples are for a single vmail user setup… what should be done when using system accounts? Thanks, Jean-Luc
Re: Dsync config help
Hi, I'm trying to sync emails between server A and B. A has replicator, aggregator and doveadm. B only has doveadm for now. I'm getting the following error: Error: sync: /var/run/dovecot/auth-userdb: Configured passdbs don't support credentials lookups (to see if user is proxied, because doveadm_port is set) I'm not sure why I'm getting this error because I authenticate using PAM and when I run I do have a "service auth" enabled though... it's used for Postfix SASL to authenticate SMTP. I'm running # dovecot --version 2.2.24 (a82c823) Cheers, Jean-Luc PS: "replication_sync_timeout = 2s" doesn't work, I had to drop the 's' On 2016-08-28 17:43, Jean-Luc Wasmer wrote: Hi, I’m having a hard time with the http://wiki2.dovecot.org/Replication page. - for a master-master setup, does the configuration need to be mirrored on both masters? - after aggregating unrelated sections of the wiki page, there seems to be 3 different values for “mail_replica": - "remote”: for SSH - “remoteprefix": for SSH wrapper - “tcp”: for TCP connection using the Doveadm protocol Is this correct? - what’s the purpose of the replicator VS aggregator VS doveadm services? Who talks to who? Where can I find documentation about their configuration (i.e. not examples) - most examples are for a single vmail user setup… what should be done when using system accounts? Thanks, Jean-Luc
Re: Dsync config help
I was able to workaround the “Error: sync: /var/run/dovecot/auth-userdb:
Configured passdbs don’t support credentials lookups (to see if user is
proxied, because doveadm_port is set)” problem:
=> I simply disabled “doveadm_port” and added the port number at the end of the
remote URL in “mail_replica”
Replication seems to work although I have regular errors (idk if they are
related to each other or not).
I’ve attached excerpts from the log files but here are the different errors:
- out of memory errors:
Aug 31 21:37:10 serverA dovecot: replicator: Panic: data stack: Out of memory
when allocating 268435496 bytes
- timeout errors:
Aug 31 21:36:41 serverA dovecot: dsync-server(user1): Warning:
replication(user1): Sync failure: Timeout in 2 secs
Aug 31 21:37:09 serverA dovecot: dsync-local(user1): Error: Couldn't lock
/home/user1/.dovecot-sync.lock: Timed out after 30 seconds
Aug 31 21:38:34 serverA dovecot: imap(user2): Warning: replication(user2): Sync
failure: Timeout in 2 secs
- permission denied errors:
Aug 31 21:37:14 serverA dovecot: dsync-server(user2): Error:
net_connect_unix(/var/run/dovecot/replicator-doveadm) failed: Permission denied
I also get errors for user “nobody” with uid 65534. To avoid them I added
10-mail.conf:first_valid_uid = 2000
10-mail.conf:last_valid_uid = 3000
but I still get some errors, different ones now:
Sep 1 00:13:42 shaun dovecot: doveadm(a.b.c.d,nobody): Error: Mail access for
users with UID 65534 not permitted (see first_valid_uid in config file, uid
from userdb lookup).
Sep 1 00:13:42 shaun dovecot: doveadm(a.b.c.d,nobody): Error: dsync-server:
User init failed
Sep 1 00:14:00 shaun dovecot: doveadm(nobody): Error: sync: Failed to start
remote dsync-server command: Remote exit_code=75
Jean-Luc
> On Aug 31, 2016, at 12:02 AM, Jean-Luc Wasmer wrote:
>
>
> Hi,
>
>
> I'm trying to sync emails between server A and B.
> A has replicator, aggregator and doveadm.
> B only has doveadm for now.
>
>
> I'm getting the following error:
>
>
> Error: sync: /var/run/dovecot/auth-userdb: Configured passdbs don't
> support credentials lookups (to see if user is proxied, because
> doveadm_port is set)
>
>
> I'm not sure why I'm getting this error because I authenticate using PAM
> and when I run
> I do have a "service auth" enabled though... it's used for Postfix SASL
> to authenticate SMTP.
>
>
> I'm running
>
>
> # dovecot --version
> 2.2.24 (a82c823)
>
>
>
>
>
> Cheers,
> Jean-Luc
>
>
>
>
>
> PS: "replication_sync_timeout = 2s" doesn't work, I had to drop the 's'
>
>
>
>
>
> On 2016-08-28 17:43, Jean-Luc Wasmer wrote:
>>
>>
>> Hi,
>>
>>
>>
>>
>>
>> I’m having a hard time with the http://wiki2.dovecot.org/Replication
>> page.
>>
>>
>>
>>
>>
>> - for a master-master setup, does the configuration need to be
>> mirrored on both masters?
>>
>>
>>
>>
>>
>>
>>
>>
>> - after aggregating unrelated sections of the wiki page, there seems
>> to be 3 different values for “mail_replica":
>> - "remote”: for SSH
>> - “remoteprefix": for SSH wrapper
>> - “tcp”: for TCP connection using the Doveadm protocol
>> Is this correct?
>>
>>
>>
>>
>>
>> - what’s the purpose of the replicator VS aggregator VS doveadm
>> services?
>> Who talks to who? Where can I find documentation about their
>> configuration (i.e. not examples)
>>
>>
>>
>>
>>
>> - most examples are for a single vmail user setup… what should be done
>> when using system accounts?
>>
>>
>>
>>
>>
>> Thanks,
>> Jean-Luc
>>
Aug 31 21:36:41 serverA dovecot: dsync-server(user1): Warning:
replication(user1): Sync failure: Timeout in 2 secs
Aug 31 21:37:09 serverA dovecot: dsync-local(user1): Error: Couldn't lock
/home/user1/.dovecot-sync.lock: Timed out after 30 seconds
Aug 31 21:37:10 serverA dovecot: replicator: Panic: data stack: Out of memory
when allocating 268435496 bytes
Aug 31 21:37:10 serverA dovecot: replicator: Fatal: master:
service(replicator): child 76676 killed with signal 6 (core not dumped - set
service replicator { drop_priv_before_exec=yes })
Aug 31 21:37:14 serverA dovecot: dsync-server(user2): Error:
net_connect_unix(/var/run/dovecot/replicator-doveadm) failed: Permission denied
Aug 31 21:37:18 serverA dovecot: dsync-server(user1): Error:
net_connect_unix(/var/run/dovecot/replicator-doveadm) failed: Permission denied
Aug 31 21:38:34 serverA do
Re: Dsync config help
Looks like I'm having a conversation with myself! lol To avoid the timeouts, I removed the "replication_sync_timeout" setting. It did work and also I don't get the "out of memory errors". I guess they were related. I also figured the "permission denied" errors: there was a typo in one of the "mode" in my config file. So to recap, here are the issues I couldn't fix: - replication_sync_timeout generates a lot of time out error resulting in out of memory errors. - users outside the valid uid range [first_valid_uid;last_valid_uid] won't be synced but will still generate a lot of errors The wiki page is of poor quality... it would need to be rewritten. I'm willing to do it if someone can tell me how to go about doing that. I'n not expecting any reply though... Cheers, Jean-Luc On 2016-08-31 21:01, Jean-Luc Wasmer wrote: I was able to workaround the “Error: sync: /var/run/dovecot/auth-userdb: Configured passdbs don’t support credentials lookups (to see if user is proxied, because doveadm_port is set)” problem: => I simply disabled “doveadm_port” and added the port number at the end of the remote URL in “mail_replica” Replication seems to work although I have regular errors (idk if they are related to each other or not). I’ve attached excerpts from the log files but here are the different errors: - out of memory errors: Aug 31 21:37:10 serverA dovecot: replicator: Panic: data stack: Out of memory when allocating 268435496 bytes - timeout errors: Aug 31 21:36:41 serverA dovecot: dsync-server(user1): Warning: replication(user1): Sync failure: Timeout in 2 secs Aug 31 21:37:09 serverA dovecot: dsync-local(user1): Error: Couldn't lock /home/user1/.dovecot-sync.lock: Timed out after 30 seconds Aug 31 21:38:34 serverA dovecot: imap(user2): Warning: replication(user2): Sync failure: Timeout in 2 secs - permission denied errors: Aug 31 21:37:14 serverA dovecot: dsync-server(user2): Error: net_connect_unix(/var/run/dovecot/replicator-doveadm) failed: Permission denied I also get errors for user “nobody” with uid 65534. To avoid them I added 10-mail.conf:first_valid_uid = 2000 10-mail.conf:last_valid_uid = 3000 but I still get some errors, different ones now: Sep 1 00:13:42 shaun dovecot: doveadm(a.b.c.d,nobody): Error: Mail access for users with UID 65534 not permitted (see first_valid_uid in config file, uid from userdb lookup). Sep 1 00:13:42 shaun dovecot: doveadm(a.b.c.d,nobody): Error: dsync-server: User init failed Sep 1 00:14:00 shaun dovecot: doveadm(nobody): Error: sync: Failed to start remote dsync-server command: Remote exit_code=75 Jean-Luc On Aug 31, 2016, at 12:02 AM, Jean-Luc Wasmer wrote: Hi, I'm trying to sync emails between server A and B. A has replicator, aggregator and doveadm. B only has doveadm for now. I'm getting the following error: Error: sync: /var/run/dovecot/auth-userdb: Configured passdbs don't support credentials lookups (to see if user is proxied, because doveadm_port is set) I'm not sure why I'm getting this error because I authenticate using PAM and when I run I do have a "service auth" enabled though... it's used for Postfix SASL to authenticate SMTP. I'm running # dovecot --version 2.2.24 (a82c823) Cheers, Jean-Luc PS: "replication_sync_timeout = 2s" doesn't work, I had to drop the 's' On 2016-08-28 17:43, Jean-Luc Wasmer wrote: Hi, I’m having a hard time with the http://wiki2.dovecot.org/Replication page. - for a master-master setup, does the configuration need to be mirrored on both masters? - after aggregating unrelated sections of the wiki page, there seems to be 3 different values for “mail_replica": - "remote”: for SSH - “remoteprefix": for SSH wrapper - “tcp”: for TCP connection using the Doveadm protocol Is this correct? - what’s the purpose of the replicator VS aggregator VS doveadm services? Who talks to who? Where can I find documentation about their configuration (i.e. not examples) - most examples are for a single vmail user setup… what should be done when using system accounts? Thanks, Jean-Luc
Re: Sieve Script Replication Gliches (Report #2)
Hey guys, I was gonna report this issue too. New script FILES get replicated right away but changes to an existing file are only replicated with a full sync (looks like this is every 24h by default). My assumption is this happens bc there’s no index file for sieve scripts. Cheers, Jean-Luc > On Sep 7, 2016, at 5:44 AM, Reuben Farrelly wrote: > > > > > > On 24/08/2016 10:58 AM, Stephan Bosch wrote: >> >> >> Op 8/1/2016 om 3:37 AM schreef Reuben Farrelly: >>> >>> >>> In other words, the rules did eventually get propagated across, and >>> based on the file sizes they are complete. >>> >>> >>> But there is obviously something amiss with handling of dates (which >>> in turn may relate to how the system determines that the file on each >>> server is up to date or not, I guess). In this case the two systems >>> are in different timezones - the primary is GMT+10 and the secondary >>> GMT+8. >>> >>> >>> Also the status of active users is not always replicated either. On >>> one host the output of 'doveadm sieve list -A' shows my own account as >>> ACTIVE but the other host shows all users - except for my account - as >>> being active, and the sieve script for my account is not being >>> replicated. >>> >> >> This should fix the file timestamps getting set at unix time_t 0: >> >> >> https://github.com/dovecot/pigeonhole/commit/af91dd3f2d78da752292dce27f9e76d2c936868c >> >> >> I haven't been able to replicate the situation where this occurs though, >> since my current replication setup is very simple. >> >> >> I need to extend my replication setup to test this more thoroughly. >> >> >> So, please test this at your end first. >> >> >> Regards, >> >> >> Stephan. >> > > > > > Thanks Stephan. I have re-tested and the dates are now all look to be > correct on the replicated scripts. We can cross that off as fixed now. > > > There is still a problem with the scripts not being replicated though > between replicated hosts. They do eventually catch up many hours > later. I don't know what the trigger is for them updating but it's not > triggered by delivery attempts (as every time a delivery was attempted > the secondary complained about the missing sieve script). > > > Thanks, > Reuben >
