Re: [2.3.8] possible replication issue

[Piper Andreas via dovecot]

Hello Timo,

upgrading both replicators did the job! Both replicators now run v2.3.9 
and replication works fine, all sync-jobs which queued up during the 
upgrading have been processed successfully.


Thanks for the reassurement and all your great work with dovecot,

Andreas


Am 05.12.19 um 13:15 schrieb Timo Sirainen via dovecot:
I think there's a good chance that upgrading both will fix it. The bug 
already existed in old versions, it just wasn't normally triggered. 
Since v2.3.8 this situation is triggered on one dsync side, so the 
v2.3.9 fix needs to be on the other side.


On 5. Dec 2019, at 8.34, Piper Andreas via dovecot 
mailto:dovecot@dovecot.org>> wrote:


Hello,

upgrading to 2.3.9 unfortunately does *not* solve this issue:

I upgraded one of my replicators from 2.3.7.2 to 2.3.9 and after some 
seconds replication stopped. The other replicator remained with 
2.3.7.2. After downgrading to 2.3.7.2 replication is again working fine.


I did not try to upgrade both replicators up to now, as this is a live 
production system. Is there a chance, that upgrading both replicators 
will solve the problem?


The machines are running Ubuntu 18.04

Any help is appreciated.

Thanks,
Andreas

Am 18.10.19 um 13:52 schrieb Carsten Rosenberg via dovecot:

Hi,
some of our customers have discovered a replication issue after
upgraded from 2.3.7.2 to 2.3.8.
Running 2.3.8 several replication connections are hanging until defined
timeout. So after some seconds there are $replication_max_conns hanging
connections.
Other replications are running fast and successful.
Also running a doveadm sync tcp:... is working fine for all users.
I can't see exactly, but I haven't seen mailboxes timeouting again and
again. So I would assume it's not related to the mailbox.
From the logs:
server1:
Oct 16 08:29:25 server1 dovecot[5715]:
dsync-local(userna...@domain.com 
<mailto:userna...@domain.com>): Error:

dsync(172.16.0.1): I/O has stalled, no activity for 600 seconds (version
not received)
Oct 16 08:29:25 server1 dovecot[5715]:
dsync-local(userna...@domain.com 
<mailto:userna...@domain.com>): Error:

Timeout during state=master_recv_handshake
server2:
Oct 16 08:29:25 server2 dovecot[8113]: doveadm: Error: read(server1)
failed: EOF (last sent=handshake, last recv=handshake)
There aren't any additional logs regarding the replication.
I have tried increasing vsz_limit or reducing replication_max_conns.
Nothing changed.
--
Both customers have 10k+ users. Currently I couldn't reproduce this on
smaller test systems.
Both installation were downgraded to 2.3.7.2 to fix the issue for now
--
I've attached a tcpdump showing the client showing the client stops
sending any data after the mailbox_guid table headers.
Any idea what could be wrong here or the debug this issue?
Thanks.
Carsten Rosenberg







smime.p7s
Description: S/MIME Cryptographic Signature

Re: [2.3.8] possible replication issue

[Piper Andreas via dovecot]

Hello,

upgrading to 2.3.9 unfortunately does *not* solve this issue:

I upgraded one of my replicators from 2.3.7.2 to 2.3.9 and after some 
seconds replication stopped. The other replicator remained with 2.3.7.2. 
After downgrading to 2.3.7.2 replication is again working fine.


I did not try to upgrade both replicators up to now, as this is a live 
production system. Is there a chance, that upgrading both replicators 
will solve the problem?


The machines are running Ubuntu 18.04

Any help is appreciated.

Thanks,
Andreas

Am 18.10.19 um 13:52 schrieb Carsten Rosenberg via dovecot:

Hi,

some of our customers have discovered a replication issue after
upgraded from 2.3.7.2 to 2.3.8.

Running 2.3.8 several replication connections are hanging until defined
timeout. So after some seconds there are $replication_max_conns hanging
connections.
Other replications are running fast and successful.

Also running a doveadm sync tcp:... is working fine for all users.

I can't see exactly, but I haven't seen mailboxes timeouting again and
again. So I would assume it's not related to the mailbox.

 From the logs:

server1:
Oct 16 08:29:25 server1 dovecot[5715]:
dsync-local(userna...@domain.com): Error:
dsync(172.16.0.1): I/O has stalled, no activity for 600 seconds (version
not received)
Oct 16 08:29:25 server1 dovecot[5715]:
dsync-local(userna...@domain.com): Error:
Timeout during state=master_recv_handshake

server2:

Oct 16 08:29:25 server2 dovecot[8113]: doveadm: Error: read(server1)
failed: EOF (last sent=handshake, last recv=handshake)

There aren't any additional logs regarding the replication.

I have tried increasing vsz_limit or reducing replication_max_conns.
Nothing changed.

--

Both customers have 10k+ users. Currently I couldn't reproduce this on
smaller test systems.

Both installation were downgraded to 2.3.7.2 to fix the issue for now

--

I've attached a tcpdump showing the client showing the client stops
sending any data after the mailbox_guid table headers.



Any idea what could be wrong here or the debug this issue?

Thanks.

Carsten Rosenberg




--

Dr. Andreas Piper, Hochschulrechenzentrum der Philipps-Univ. Marburg
  Hans-Meerwein-Straße 6, 35032 Marburg, Germany
Phone: +49 6421 28-23521  Fax: -26994  E-Mail: pi...@hrz.uni-marburg.de



smime.p7s
Description: S/MIME Cryptographic Signature

Re: sieve scripts not synching for 2.3.5.1 pre-built

[Piper Andreas via dovecot]

Hello,

Am 02.04.19 um 21:51 schrieb Timo Sirainen via dovecot:
> On 2 Apr 2019, at 22.37, Timo Sirainen via dovecot  > wrote:
>>
>> On 2 Apr 2019, at 17.03, Jan-Pieter Cornet via dovecot
>> mailto:dovecot@dovecot.org>> wrote:
>>>
>>> Hi,
>>>
>>> We're synching mailboxes, changing format from maildir to mdbox,
>>> using doveadm backup/doveadm sync.
>>>
>>> When still running 2.2.36, 'doveadm backup' also synched the sieve
>>> scripts, without issues.
>>>
>>> After the upgrade to 2.3.5.1, the sieve sync stopped working. We're
>>> using the pre-built 2.3 packages from
>>> https://repo.dovecot.org/ce-2.3-latest/debian/stretch
>>
>> Looks like this is trivial to reproduce. It used to work still in
>> v2.3.1, but then something broke it. Tracking internally in DOP-1062.
> 
> Reverting 
> https://github.com/dovecot/pigeonhole/commit/479c5e57046dec76078597df844daccbfc0eb75f
>  fixes
> this.
> 

has this been fixed in 2.3.6 ?

In my first tests sieve scripts still seem not to be replicated with
dovecot-2.3.6

Regards,
Andreas



smime.p7s
Description: S/MIME Cryptographic Signature

Re: dovecot-keywords are not preserved any more when moving mails between folders

[Piper Andreas via dovecot]

Hello Timo,

Am 12.03.19 um 22:31 schrieb Timo Sirainen via dovecot:
> On 12 Mar 2019, at 17.55, Dan Christensen via dovecot  
> wrote:
>>
>> On Mar 12, 2019, Aki Tuomi via dovecot  wrote:
>>
>>> On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
>>>
>>>> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
>>>> which in my case are set by thunderbird, are not preserved any more when
>>>> moving a mail between folders.
>>>
>>> We are aware of this bug, and it's being tracked as DOP-842.
>>
>> Could this bug also be causing flags to be lost when using dsync
>> (as I described in some messages to this list Feb 16 to 23)?
>>
>> It seems like it might be a different bug, since in my experience
>> the flags are sometimes synced and then removed later.
> 
> That bug is fixed with attached patch.
> 

I assume, this patch is not included in dovecot-2.3.5?

Because I still observe the same problem (dovecot-keywords set by
thunderbird are lost when moving mail between folders) on my
migration-machines which are running dovecot-2.3.5:

# 2.3.5 (513208660): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.5 (2483b085)
# OS: Linux 4.15.0-46-generic x86_64 Ubuntu 18.04.2 LTS zfs

Regards,
Andreas





smime.p7s
Description: S/MIME Cryptographic Signature

Re: dovecot-keywords are not preserved any more when moving mails between folders

[Piper Andreas via dovecot]

Am 12.03.19 um 12:57 schrieb Aki Tuomi via dovecot:
> 
> On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
>> Hello,
>>
>> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
>> which in my case are set by thunderbird, are not preserved any more when
>> moving a mail between folders.
>>
> 
> Hi!
> 
> We are aware of this bug, and it's being tracked as DOP-842.
> 
> Aki
> 
Hello Aki,

thanks for the immediate answer.

To which version would I have to downgrade to avoid this problem?

Andreas
-- 

Dr. Andreas Piper, Hochschulrechenzentrum der Philipps-Univ. Marburg
  Hans-Meerwein-Straße 6, 35032 Marburg, Germany
Phone: +49 6421 28-23521  Fax: -26994  E-Mail: pi...@hrz.uni-marburg.de



smime.p7s
Description: S/MIME Cryptographic Signature

dovecot-keywords are not preserved any more when moving mails between folders

[Piper Andreas via dovecot]

Hello,

after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
which in my case are set by thunderbird, are not preserved any more when
moving a mail between folders.

Are there any ideas, what may be the reason.

Thanks for any hints on that,
Andreas

'doveconf -n' gives:

# 2.3.4 (0ecbaf23d): /etc/opt/csw/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 (60b0f48d)
# OS: SunOS 5.11 i86pc
# Hostname: x.hrz.uni-marburg.de
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_master_user_separator = *
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_
auth_username_format = %u
auth_worker_max_count = 1024
base_dir = /var/run/dovecot/
default_vsz_limit = 1 G
first_valid_gid = 3
first_valid_uid = 3
imap_max_line_length = 640 k
mail_debug = yes
mail_location = maildir:%h/.maildir
mail_plugins = " mail_log notify"
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace {
  hidden = no
  inbox = yes
  list = yes
  location = maildir:%h/.maildir
  prefix =
  separator = /
  subscriptions = yes
  type = private
}
namespace inbox {
  hidden = yes
  inbox = no
  list = no
  location = maildir:%h/.maildir
  prefix = mail/
  separator = /
  subscriptions = no
  type = private
}
passdb {
  args = /etc/dovecot.deny
  deny = yes
  driver = passwd-file
}
passdb {
  args = /etc/opt/csw/dovecot/private/passwd.masterusers
  driver = passwd-file
  master = yes
}
passdb {
  args = /etc/opt/csw/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_default = /var/lib/dovecot/default.sieve
}
pop3_uidl_format = %08Xv%08Xu
postmaster_address = postmas...@hrz.uni-marburg.de
protocols = imap pop3 lmtp sieve
service auth-worker {
  user = $default_internal_user
}
service auth {
  client_limit = 6000
}
service imap-login {
  process_min_avail = 64
  service_count = 0
}
service imap {
  process_limit = 4096
}
service lmtp {
  inet_listener lmtp {
port = 24
  }
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
}
ssl_cert = 

smime.p7s
Description: S/MIME Cryptographic Signature

Fwd: dovecot/lmtp dumps core when proxying to multiple recipients via protocol=smtp

[Piper Andreas via dovecot]

Hello,

is there already any idea on this topic? Do you need additional information?

I would appreciate any input, as my migration scenario depends on this.

With many thanks and best regards,
Andreas

 Weitergeleitete Nachricht 
Betreff: dovecot/lmtp dumps core when proxying to multiple recipients
via protocol=smtp
Datum: Thu, 14 Feb 2019 13:53:07 +0100
Von: Piper Andreas via dovecot 
Antwort an: Piper Andreas 
An: dovecot@dovecot.org

Hello,

when trying to set up a migration scenario from my old mailbox-servers
to my new dovecot-cluster, I get a coredump from dovecot/lmtp when
proxying mail-delivery to more than one recipient on the old mailbox-server.

My setup:

3 director-hosts (Debian stretch) are proxying to a pair of
replicator-hosts and during migration also to the old mailbox-server.
For migration the not-migrated user-addresses are included via a
passwd-file with entries like:

grte...@hrz.uni-marburg.de:::director_tag=hrz protocol=smtp user=grtest1
grte...@hrz.uni-marburg.de:::director_tag=hrz protocol=smtp user=grtest2
...

The director_tag points to the old mailbox-server, with 'protocol=smtp'
SMTP is used for delivery, as the old machine is not capable of LMTP.

The setup works fine for imap-proxying and for mail-delivery to single
recipients. But if a mail is addressed to more than one recipient, lmtp
produces a coredump after delivering to all recipients.

The error-message in the log is:

Feb 14 12:15:05 vhrzxxx dovecot[63861]: lmtp(64663): Connect from 127.0.0.1
Feb 14 12:15:05 vhrzxxx dovecot[63861]: lmtp(64663):
206UIrlNZVyX/AAAGwthQA: Sent message to  at 137.248.xxx.yyy:25:
250 2.0.0 x1EBF5Ro016092 Message accepted for delivery (1/2 at 79 ms)
Feb 14 12:15:05 vhrzxxx dovecot[63861]: lmtp(64663):
206UIrlNZVyX/AAAGwthQA: Sent message to  at 137.248.xxx.yyy:25:
250 2.0.0 x1EBF5Ro016092 Message accepted for delivery (2/2 at 79 ms)
Feb 14 12:15:05 vhrzxxx dovecot[63861]: lmtp(64663): Fatal: master:
service(lmtp): child 64663 killed with signal 11 (core dumped)

The backtrace from the coredump is:

Reading symbols from /usr/lib/dovecot/lmtp...(no debugging symbols
found)...done.
[New LWP 64663]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `dovecot/lmtp'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f347fc7a2c0 in ?? () from /usr/lib/dovecot/libdovecot.so.0
(gdb) bt full
#0  0x7f347fc7a2c0 in ?? () from /usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#1  0x7f347fc77d73 in smtp_client_command_fail_reply () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#2  0x7f347fc77fbe in smtp_client_commands_list_fail_reply () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#3  0x7f347fc7c8dc in smtp_client_connection_disconnect () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#4  0x7f347fc7cf19 in smtp_client_connection_unref () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#5  0x7f347fc7e35d in ?? () from /usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#6  0x7f347fd1f5d5 in io_loop_call_io () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#7  0x7f347fd20fd9 in io_loop_handler_run_internal () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#8  0x7f347fd1f6e6 in io_loop_handler_run () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#9  0x7f347fd1f8f8 in io_loop_run () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#10 0x7f347fc93d43 in master_service_run () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#11 0x5613fcbc1f70 in main ()
No symbol table info available.
(gdb)

doveconf -n gives:

# 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 (60b0f48d)
# OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.7
# Hostname: vhrzxxx.HRZ.Uni-Marburg.DE
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_master_user_separator = *
auth_worker_max_count = 1024
base_dir = /var/run/dovecot/
default_vsz_limit = 1 G
director_mail_servers = 172.25.1.93 172.25.1.95 137.248.xxx.yyy@hrz
137.248.xxx.zzz@staff 137.248.aaa.bbb@students
director_servers = 172.25.1.42:9090 172.25.1.41:9090 172.25.1.92:9090
director_username_hash = %Ln@%Ld
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
imap_max_line_length = 640 k
lmtp_proxy = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
passdb {
  args = /etc/dovecot.deny
  deny = yes
  driver = passwd-file
}
passdb {
  args = /etc/dovecot/private/passwd.masteruser

dovecot/lmtp dumps core when proxying to multiple recipients via protocol=smtp

[Piper Andreas via dovecot]

Hello,

when trying to set up a migration scenario from my old mailbox-servers
to my new dovecot-cluster, I get a coredump from dovecot/lmtp when
proxying mail-delivery to more than one recipient on the old mailbox-server.

My setup:

3 director-hosts (Debian stretch) are proxying to a pair of
replicator-hosts and during migration also to the old mailbox-server.
For migration the not-migrated user-addresses are included via a
passwd-file with entries like:

grte...@hrz.uni-marburg.de:::director_tag=hrz protocol=smtp user=grtest1
grte...@hrz.uni-marburg.de:::director_tag=hrz protocol=smtp user=grtest2
...

The director_tag points to the old mailbox-server, with 'protocol=smtp'
SMTP is used for delivery, as the old machine is not capable of LMTP.

The setup works fine for imap-proxying and for mail-delivery to single
recipients. But if a mail is addressed to more than one recipient, lmtp
produces a coredump after delivering to all recipients.

The error-message in the log is:

Feb 14 12:15:05 vhrzxxx dovecot[63861]: lmtp(64663): Connect from 127.0.0.1
Feb 14 12:15:05 vhrzxxx dovecot[63861]: lmtp(64663):
206UIrlNZVyX/AAAGwthQA: Sent message to  at 137.248.xxx.yyy:25:
250 2.0.0 x1EBF5Ro016092 Message accepted for delivery (1/2 at 79 ms)
Feb 14 12:15:05 vhrzxxx dovecot[63861]: lmtp(64663):
206UIrlNZVyX/AAAGwthQA: Sent message to  at 137.248.xxx.yyy:25:
250 2.0.0 x1EBF5Ro016092 Message accepted for delivery (2/2 at 79 ms)
Feb 14 12:15:05 vhrzxxx dovecot[63861]: lmtp(64663): Fatal: master:
service(lmtp): child 64663 killed with signal 11 (core dumped)

The backtrace from the coredump is:

Reading symbols from /usr/lib/dovecot/lmtp...(no debugging symbols
found)...done.
[New LWP 64663]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `dovecot/lmtp'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x7f347fc7a2c0 in ?? () from /usr/lib/dovecot/libdovecot.so.0
(gdb) bt full
#0  0x7f347fc7a2c0 in ?? () from /usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#1  0x7f347fc77d73 in smtp_client_command_fail_reply () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#2  0x7f347fc77fbe in smtp_client_commands_list_fail_reply () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#3  0x7f347fc7c8dc in smtp_client_connection_disconnect () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#4  0x7f347fc7cf19 in smtp_client_connection_unref () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#5  0x7f347fc7e35d in ?? () from /usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#6  0x7f347fd1f5d5 in io_loop_call_io () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#7  0x7f347fd20fd9 in io_loop_handler_run_internal () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#8  0x7f347fd1f6e6 in io_loop_handler_run () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#9  0x7f347fd1f8f8 in io_loop_run () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#10 0x7f347fc93d43 in master_service_run () from
/usr/lib/dovecot/libdovecot.so.0
No symbol table info available.
#11 0x5613fcbc1f70 in main ()
No symbol table info available.
(gdb)

doveconf -n gives:

# 2.3.4.1 (3c0b8769e): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 (60b0f48d)
# OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.7
# Hostname: vhrzxxx.HRZ.Uni-Marburg.DE
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_master_user_separator = *
auth_worker_max_count = 1024
base_dir = /var/run/dovecot/
default_vsz_limit = 1 G
director_mail_servers = 172.25.1.93 172.25.1.95 137.248.xxx.yyy@hrz
137.248.xxx.zzz@staff 137.248.aaa.bbb@students
director_servers = 172.25.1.42:9090 172.25.1.41:9090 172.25.1.92:9090
director_username_hash = %Ln@%Ld
doveadm_password = # hidden, use -P to show it
doveadm_port = 12345
imap_max_line_length = 640 k
lmtp_proxy = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
passdb {
  args = /etc/dovecot.deny
  deny = yes
  driver = passwd-file
}
passdb {
  args = /etc/dovecot/private/passwd.masterusers
  driver = passwd-file
  master = yes
}
passdb {
  args = /etc/dovecot/dovecot-ldap.staff.conf.ext
  driver = ldap
  result_success = continue-ok
}
passdb {
  args = /etc/dovecot/migration/migrate-staff
  driver = passwd-file
  name = passdb_staff_query
}
passdb {
  args = /etc/dovecot/dovecot-ldap.students.conf.ext
  driver = ldap
  result_success = continue-ok
}
passdb {
  args = /etc/dovecot/migration/migrate-students
  driver = passwd-file
  name =