Re: Filesystem level backups?
Been rsyncing my whole /var/vmail subdir (virtual users) for a while. So far so good. Yassine. On Sunday, April 8, 2018, 11:56:29 PM GMT+1, Steve Atkins wrote: Up until now I've been backing up my IMAP server by taking an atomic snapshot of the entire VM, so I've not really had to worry about it. I'm moving to hosting where I have neither VM snapshots nor a filesystem that can take snapshots, so I need to think about consistent backups a bit more. Will a simple backup of all the files of an multi-dbox give me a consistent state when I recover, or do I need to do something (e.g. lock writes to the mailbox) while I run a backup? (I could use doveadm backup if I were backing up to local storage I controlled, but I'm pushing it to offsite buckets so I'd prefer to avoid backing up the entire thing locally, then copying that offsite). Cheers, Steve
Filesystem level backups?
Up until now I've been backing up my IMAP server by taking an atomic snapshot of the entire VM, so I've not really had to worry about it. I'm moving to hosting where I have neither VM snapshots nor a filesystem that can take snapshots, so I need to think about consistent backups a bit more. Will a simple backup of all the files of an multi-dbox give me a consistent state when I recover, or do I need to do something (e.g. lock writes to the mailbox) while I run a backup? (I could use doveadm backup if I were backing up to local storage I controlled, but I'm pushing it to offsite buckets so I'd prefer to avoid backing up the entire thing locally, then copying that offsite). Cheers, Steve
Re: lda fails in parse_angle_addr if sieve is enabled
Hi all, Reverted back to 2.2.34 and pigeonhole 0.4.22 and (after reverting config changes) it's working again including Sieve. I was notified of commit https://github.com/stephanbosch/dovecot-core/commit/6553f20bb31b5f0eebb73a0db526f00816b47d71 which I'll try. It's in a different option -f (not -d) but will try. Cheers, Bernard. 2018-04-08 20:10 GMT+02:00 Bernard Spil : > Hi, > > Since updating to 2.3.1 on my FreeBSD mailserver mail delivery using > lda is broken if I have sieve enabled. > (Before updating this was 2.2 and pigeonhole 0.4) > > FreeBSD 11.1-p8 amd64 > Dovecot 2.3.1 > Pigeonhole 0.5.1 > > Mailflow is OpenSMTPd as MTA, using mda delivery to rspamc which > utlimately delivers using dovecot-lda. > > smtpd.conf > deliver to mda "rspamc -h scan --mime -e > \"/usr/local/libexec/dovecot/deliver -d %{user.username}\"" > %{user.username} is the local user after virtusers, aliases etc. > verified using a shell wrapper and capturing the username. > > conf.d/15-lda.conf > protocol lda { > mail_plugins = $mail_plugins sieve > } > > maillog: > Apr 8 19:36:54 email smtpd[6390]: smtp-in: Accepted message 9db769b1 > on session 81939f0d30337a47: from=, > to=, size=2871, ndest=1, proto=ESMTP > Apr 8 19:36:54 email smtpd[6390]: smtp-in: Closing session 81939f0d30337a47 > Apr 8 19:36:55 email dovecot: > lda(user2)<21091>: Panic: file > message-address.c: line 147 (parse_angle_addr): assertion failed: > (*ctx->parser.data == '<') > Apr 8 19:36:55 email smtpd[6390]: delivery: TempFail for > 9db769b13edef5a7: from=, to=, > user=bernard, method=mda, delay=1s, stat=Error ("") > Apr 8 19:36:57 email smtpd[6390]: smtp-out: Closing session > 81939f0c09f5140b: 1 message sent. > Apr 8 19:37:04 email dovecot: > lda(user2)<21102>: Panic: file > message-address.c: line 147 (parse_angle_addr): assertion failed: > (*ctx->parser.data == '<') > Apr 8 19:37:04 email smtpd[6390]: delivery: TempFail for > 9db769b13edef5a7: from=, to=, > user=bernard, method=mda, delay=10s, stat=Error ("") > > Removing sieve from mail_plugins configuration and restarting > Apr 8 19:37:32 email dovecot: master: Warning: Killed with signal 15 > (by pid=21106 uid=0 code=kill) > Apr 8 19:37:32 email dovecot: imap(user2)<21085>: > Server shutting down. in=774 out=5373 deleted=0 expunged=0 trashed=0 > hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0 > Apr 8 19:37:32 email dovecot: master: Dovecot v2.3.1 (8e2f634) > starting up for imap, lmtp, sieve > Apr 8 19:37:35 email dovecot: > lda(user2)<21134>: > msgid=<32984450d041e5c2f887bed5f6512...@example.org>: saved mail to > INBOX > Apr 8 19:37:35 email smtpd[6390]: delivery: Ok for 9db769b13edef5a7: > from=, to=, user=user2, > method=mda, delay=41s, stat=Delivered > > Mail gets delivered. > > Don't understand why it is looking for a <>-address if sieve is enabled. > > Cheers, Bernard Spil.
lda fails in parse_angle_addr if sieve is enabled
Hi, Since updating to 2.3.1 on my FreeBSD mailserver mail delivery using lda is broken if I have sieve enabled. (Before updating this was 2.2 and pigeonhole 0.4) FreeBSD 11.1-p8 amd64 Dovecot 2.3.1 Pigeonhole 0.5.1 Mailflow is OpenSMTPd as MTA, using mda delivery to rspamc which utlimately delivers using dovecot-lda. smtpd.conf deliver to mda "rspamc -h scan --mime -e \"/usr/local/libexec/dovecot/deliver -d %{user.username}\"" %{user.username} is the local user after virtusers, aliases etc. verified using a shell wrapper and capturing the username. conf.d/15-lda.conf protocol lda { mail_plugins = $mail_plugins sieve } maillog: Apr 8 19:36:54 email smtpd[6390]: smtp-in: Accepted message 9db769b1 on session 81939f0d30337a47: from=, to=, size=2871, ndest=1, proto=ESMTP Apr 8 19:36:54 email smtpd[6390]: smtp-in: Closing session 81939f0d30337a47 Apr 8 19:36:55 email dovecot: lda(user2)<21091>: Panic: file message-address.c: line 147 (parse_angle_addr): assertion failed: (*ctx->parser.data == '<') Apr 8 19:36:55 email smtpd[6390]: delivery: TempFail for 9db769b13edef5a7: from=, to=, user=bernard, method=mda, delay=1s, stat=Error ("") Apr 8 19:36:57 email smtpd[6390]: smtp-out: Closing session 81939f0c09f5140b: 1 message sent. Apr 8 19:37:04 email dovecot: lda(user2)<21102>: Panic: file message-address.c: line 147 (parse_angle_addr): assertion failed: (*ctx->parser.data == '<') Apr 8 19:37:04 email smtpd[6390]: delivery: TempFail for 9db769b13edef5a7: from=, to=, user=bernard, method=mda, delay=10s, stat=Error ("") Removing sieve from mail_plugins configuration and restarting Apr 8 19:37:32 email dovecot: master: Warning: Killed with signal 15 (by pid=21106 uid=0 code=kill) Apr 8 19:37:32 email dovecot: imap(user2)<21085>: Server shutting down. in=774 out=5373 deleted=0 expunged=0 trashed=0 hdr_count=0 hdr_bytes=0 body_count=0 body_bytes=0 Apr 8 19:37:32 email dovecot: master: Dovecot v2.3.1 (8e2f634) starting up for imap, lmtp, sieve Apr 8 19:37:35 email dovecot: lda(user2)<21134>: msgid=<32984450d041e5c2f887bed5f6512...@example.org>: saved mail to INBOX Apr 8 19:37:35 email smtpd[6390]: delivery: Ok for 9db769b13edef5a7: from=, to=, user=user2, method=mda, delay=41s, stat=Delivered Mail gets delivered. Don't understand why it is looking for a <>-address if sieve is enabled. Cheers, Bernard Spil.
Re: 2.3.1 Replication is throwing scary errors
Hi, [Formatting is a bit rough, replying from a trimmed digest email] Message: 1 Date: Fri, 6 Apr 2018 15:04:35 +0200 From: Michael Grimm To: Dovecot Mailing List Subject: Re: 2.3.1 Replication is throwing scary errors Message-ID: Content-Type: text/plain; charset=utf-8 Reuben Farrelly wrote: From: Michael Grimm [This is Dovecot 2.3.1 at FreeBSD STABLE-11.1 running in two jails at distinct servers.] I did upgrade from 2.2.35 to 2.3.1 today, and I do become pounded by error messages at server1 (and vice versa at server2) as follows: | Apr 2 17:12:18 server1.lan dovecot: doveadm: Error: dsync(server2.lan): I/O has stalled, \ no activity for 600 seconds (last sent=mail_change, last recv=mail_change (EOL)) | Apr 2 17:12:18 server1.lan dovecot: doveadm: Error: Timeout during state=sync_mails \ (send=changes recv=mail_requests) [?] | Apr 2 18:59:03 server1.lan dovecot: doveadm: Error: dsync(server2.lan): I/O has stalled, \ no activity for 600 seconds (last sent=mail, last recv=mail (EOL)) | Apr 2 18:59:03 server1.lan dovecot: doveadm: Error: Timeout during state=sync_mails \ (send=mails recv=recv_last_common) I cannot see in my personal account any missing replications, *but* I haven't tested this thoroughly enough. I do have customers being serviced at these productive servers, *thus* I'm back to 2.2.35 until I do understand or have learned what is going on. In my reply to this statement of mine I mentioned that I have seen those timeouts quite some times during the past year. Thus, I upgraded to 2.3.1 again, and boom: after some hours I ended up in hanging processes [1] like (see Remko's mail in addition) ... doveadm-server: [IP4/6 SOME/MAILBOX import:0/0] (doveadm-server) ? at server2 paired with a file like ? -rw--- 1 vmail dovecot uarch 0 Apr 3 16:52 /home/to/USER1/.dovecot-sync.lock Corresponding logfile entries at server2 are like ? Apr 3 17:10:49 server2.lan dovecot: doveadm: Error: Couldn't lock /home/to/USER1/.dovecot-sync.lock: \ fcntl(/home/to/USER1/.dovecot-sync.lock, write-lock, F_SETLKW) locking failed: Timed out after 30 seconds \ (WRITE lock held by pid 51110) [1] Even stopping dovecot will not end those processes. One has to manually kill those before restarting dovecot. After one day of testing 2.3.1 with a couple of those episodes of locking/timeout, and now missing mails depending with server your MUA will connect to, I went back to 2.2.35. After two days at that version I never had such an episode again. It's not just you. This issue hit me recently, and it was impacting replication noticeably. I am following git master-2.3 . [...] There is also a second issue of a long standing race with replication occurring somewhere whereby if a mail comes in, is written to disk, is replicated and then deleted in short succession, it will reappear again to the MUA. I suspect the mail is being replicated back from the remote. A few people have reported it over the years but it's not reliable or consistent, so it has never been fixed. And lastly there has been an ongoing but seemingly minor issue relating to locking timing out after 30s particularly on the remote host that is being replicated to. I rarely see the problem on my local disk where almost all of the mail comes in, it's almost always occurring on the replicate/remote system. It might be time to describe our setups in order to possibly find common grounds that might trigger this issue you describe and Rimko and myself ran into as well. Servers:Cloud Instances (both identical), around 25ms latency apart. Intel Core Processor (Haswell, no TSX) (3092.91-MHz K8-class CPU) Both servers are connected via IPsec/racoon tunnels OS: FreeBSD 11.1-STABLE (both servers) Filesystem: ZFS MTA:postfix 3.4-20180401 (postfix delivers via dovecot's LMTP) IMAP: dovecot running in FreeBSD jails (issues with 2.3.1, fine with 2.2.35) Replication:unsecured tcp / master-master MUA:mainly iOS or macOS mail.app, rarely roundcube For me: Servers:Main: VM on a VMWare ESXi local system (light load), local SSD disks (no NFS) Redundant: Linode VM in Singapore, around 250ms away Also no NFS. Linode use SSDs for IO. There is native IPv6 connectivity between both VMs. As I am using TCPs I don't have a VPN between them - just raw IPv6 end to end. OS: Gentoo Linux x86_64 kept well up to date Filesystem: EXT4 for both MTA:Postfix 3.4-x (same as you) IMAP: Dovecot running natively on the machine (no jail/chroot) Replication:Secured TCP / master-master (but only one master is actively used for SMTP and MUA access) MUA:
Re: changed behavior for dovecot-lda in 2.3.x
Op 4/7/2018 om 10:16 PM schreef Martin Waschbüsch: > Hi all, > > Hey all, I upgraded to dovecot 2.3.1 (from 2.2.34) and noticed that the > behavior for dovecot-lda changed. Apparently it no longer accepts -f "" or -f > "<>? > With 2.2.34, both were accepted now I get: > > root@mail:~# /usr/local/libexec/dovecot/dovecot-lda -f "<>" > lda(root): Fatal: Invalid -f parameter: Null path not allowed > > or > > root@mail:~# /usr/local/libexec/dovecot/dovecot-lda -f "" > lda(root): Fatal: Invalid -f parameter: Path is empty string > > I guess this must be a bug? I mean, envelope sender *must* be empty for > bounces. That is definitely a bug. Fix is available here: https://github.com/stephanbosch/dovecot-core/commits/lda-fix-allow-empty-sender Fix will be merged after internal review. Regards, Stephan.