Re: dovecot-keywords are not preserved any more when moving mails between folders

2019-03-12 Thread Piper Andreas via dovecot 
Hello Timo,

Am 12.03.19 um 22:31 schrieb Timo Sirainen via dovecot:
> On 12 Mar 2019, at 17.55, Dan Christensen via dovecot  
> wrote:
>>
>> On Mar 12, 2019, Aki Tuomi via dovecot  wrote:
>>
>>> On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
>>>
 after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
 which in my case are set by thunderbird, are not preserved any more when
 moving a mail between folders.
>>>
>>> We are aware of this bug, and it's being tracked as DOP-842.
>>
>> Could this bug also be causing flags to be lost when using dsync
>> (as I described in some messages to this list Feb 16 to 23)?
>>
>> It seems like it might be a different bug, since in my experience
>> the flags are sometimes synced and then removed later.
> 
> That bug is fixed with attached patch.
> 

I assume, this patch is not included in dovecot-2.3.5?

Because I still observe the same problem (dovecot-keywords set by
thunderbird are lost when moving mail between folders) on my
migration-machines which are running dovecot-2.3.5:

# 2.3.5 (513208660): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.5 (2483b085)
# OS: Linux 4.15.0-46-generic x86_64 Ubuntu 18.04.2 LTS zfs

Regards,
Andreas





smime.p7s
Description: S/MIME Cryptographic Signature

recipient bcc and dovecot

2019-03-12 Thread @lbutlr via dovecot 
I have a recipient_bcc_maps which contains a bcc map that is updated everyday:

rbcc.pcre:
if !/backup.*@/
/^([^+_]*).*@([^.]*)/   backup+071.${1}-${2}@adomain.tld 
endif

the 071 portion is changed each day to the current day of the year.

Everything works, but I get an error from dovecot on every message:

lda(bac...@adomain.tld): Error: User initialization failed: Namespace '': 
Ambiguous mail location setting, don't know what to do with it: 
/usr/local/virtual/bac...@adomain.tld (try prefixing it with mbox: or maildir:)

The mail is delivered two the right directory anyway, but after a temporary 
failure.

I'd like to eliminate the error.

the backup user is not a 'real' user (it is not in the lookup map) so should I 
bypass dovecot entirely in delivering these messages? (and if so, how)

If not, where/how would I add the prefix maildir: 

Is there something I can add to dovecot to specifically deal with messages 
coming in to bac...@adomain.tld and add the maildir: specification there?

(Still on dovecot 2.2.35)

-- 
And sometimes there's a short cut. A door or a gate. Some standing
stones. A tree cleft by lightning, a filing cabinet. Maybe just a spot
on some moorland somewhere... A place where THERE is very nearly HERE...
If some people knew where such a spot was, if they had experience of
what happens when here and there become entangled, then they might - if
they knew how - mark such a spot with certain stones. In the hope that
enough daft buggers would take it as a warning and keep away. (Lords and
Ladies)

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Helmut K. C. Tessarek via dovecot 
On 2019-03-12 19:17, Timo Sirainen wrote:
> Well, I suppose it depends on definitions.. But I'm not calling ~/ relative 
> paths, because it expands to an absolute path. The problem is using paths 
> like "mail/" where it depends on the current chdir.

Thank you for clearing this up.

IMO the ~ is still a relative path until it has been expanded to an absolute
path. Depending on the framework/API/code, expansion might not even work.
A path mail/ could also be expanded to an absolute path, so I don't see a
difference there. It's just that the output of getcwd might not be as
predictable, that's all.

But I agree, it's a matter of definition. In that case, I'd kindly ask you to
state in the documentation that ~ is not considered a relative path.

If one wants to get very pedantic, ~ is neither an absolute, nor a relative
path. But I'm not here to be a smartass.

Thanks again for clearing this up for me.

Cheers,
   K. C.

-- 
regards Helmut K. C. Tessarek  KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/



signature.asc
Description: OpenPGP digital signature

Re: Inbox quota usage doubled when mailbox_list_index enabled, under some circumstances

2019-03-12 Thread Mark Moseley via dovecot 
On Tue, Aug 14, 2018 at 12:31 PM Chris Dillon 
wrote:

> I’ve had the opportunity to test the same configuration with a fresh build
> of the git master branch (2.4.devel) and the issue also occurs there.  I
> see that "mailbox_list_index = yes" is now enabled by default.  It can
> still be disabled via "mailbox_list_index = no" which allows the quota to
> be calculated correctly.
>
> ==
> root@ubuntu1804:~# dovecot -n
> # 2.4.devel (44282aeeb): /usr/local/etc/dovecot/dovecot.conf
> # OS: Linux 4.15.0-30-generic x86_64 Ubuntu 18.04.1 LTS
> # Hostname: ubuntu1804
> mail_location = maildir:~/Maildir
> mail_plugins = quota
> namespace inbox {
>   inbox = yes
>   location =
>   prefix = INBOX.
>   separator = .
> }
> passdb {
>   driver = pam
> }
> plugin {
>   quota = maildir:Mailbox
> }
> userdb {
>   driver = passwd
> }
> ==
>
> (To summarize from my previous message -- other than "mailbox_list_index =
> yes", second most important part of replication is that there is at least
> one email in the real inbox and at least one sub-folder named "INBOX" in
> maildir format)
>
> root@ubuntu1804:~# ls -ld
> /home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local\,S\=12712627\,W\=12877782\:2\,S
> /home/myuser/Maildir/.INBOX.Test/
> -rw-rw-r-- 1 myuser myuser 12712627 Aug 14 18:28
> '/home/myuser/Maildir/cur/1532529376.M543965P58007.centos7.local,S=12712627,W=12877782:2,S'
> drwxrwxr-x 5 myuser myuser   87 Aug 14 18:56
> /home/myuser/Maildir/.INBOX.Test/
> =
>
> (In the following example usage is doubled, there is only one email)
>
> root@ubuntu1804:~# doveadm quota recalc -u myuser; doveadm quota get -u
> myuser
> Quota name TypeValue Limit
>   %
> MailboxSTORAGE 24830 -
>   0
> MailboxMESSAGE 2 -
>   0
> ==
>
> (In the following example it works correctly with mailbox_list_index
> disabled)
>
> root@ubuntu1804:~# doveadm -o 'mailbox_list_index=no' quota recalc -u
> myuser; doveadm quota get -u myuser
> Quota name TypeValue Limit
>   %
> MailboxSTORAGE 12415 -
>   0
> MailboxMESSAGE 1 -
>   0
> ==
>
> Best Regards




We recently upgraded from 2.2.32 to 2.2.36.1 and ran into the same issue as
above. I was about to start compiling all the intermediate versions to
pinpoint where it started and happened upon this post first. We are in the
same boat where some users have *sub*folders starting with INBOX, resulting
in directory names like ".INBOX.Event" (and our namespace root is INBOX)
and quota calculation double-counts INBOX's vsize.

Just to be clear too: At least in my case, it's *not* double counting,
e.g., INBOX.Event. But if the above conditions are met, it's
double-counting *INBOX*, because it now sees a folder called INBOX.INBOX
(which does *not* exist on the filesystem).

I hadn't gotten as far as Chris did (this just bubbled up today), but his
solution works here too, i.e. passing -o 'mailbox_list_index=no'  to
doveadm quota recalc.

Also, if I rename the directories to something else (e.g. from above,
rename ".INBOX.Event" to ".notINBOX.Event"), a quota recalc works just
fine. The presence of a directory called .INBOX. is triggering
this.

I'm able to create new subfolders called INBOX. with clients
like Apple Mail, which was a bit surprising (Apple Mail however choked when
I tried to just create a subfolder called 'INBOX'). We've got millions of
mailboxes, so educating users is a non-starter :)

Any fix for this from the dovecot devs?

Re: flags not synced correctly with dovecot sync (dsync)

2019-03-12 Thread Dan Christensen via dovecot 
In another thread, Timo wrote:

On Mar 12, 2019, Timo Sirainen via dovecot  wrote:

> That bug is fixed with attached patch.

Thanks!  I'm attaching the patch here, so it is in this thread as well.

A couple of questions before I test this:

- Do I need the patch on the remote end of the sync, or the local end,
  or both?

- Does it make sense to try it based on the master-2.3 branch from git?
  Following https://wiki.dovecot.org/CompilingSource  ?

I'll report back once I've tested it.

Dan

>From 0ac59ec142bc9adc30f7d1c2c7c4cc0a109cbe15 Mon Sep 17 00:00:00 2001
From: Timo Sirainen 
Date: Fri, 8 Mar 2019 18:39:49 +0200
Subject: [PATCH] dsync: Fix importing keywords with MAIL_TRANSACTION_SYNC flag
 set

Reading transaction logs was handled differently depending on the
MAIL_TRANSACTION_SYNC flag. The flag was set for all transactions written
by dsync.

So for example:
 * doveadm backup mdbox:/tmp/mdbox1 # keywords imported ok
 * doveadm -o mail=mdbox:/tmp/mdbox1 backup mdbox:/tmp/mdbox2 # keywords lost
---
 src/doveadm/dsync/dsync-mailbox-import.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/doveadm/dsync/dsync-mailbox-import.c b/src/doveadm/dsync/dsync-mailbox-import.c
index 39a694a08a..98f54edb35 100644
--- a/src/doveadm/dsync/dsync-mailbox-import.c
+++ b/src/doveadm/dsync/dsync-mailbox-import.c
@@ -2314,6 +2314,7 @@ dsync_mailbox_get_final_keywords(const struct dsync_mail_change *change)
 	t_array_init(, count);
 	for (i = 0; i < count; i++) {
 		if (changes[i][0] == KEYWORD_CHANGE_ADD ||
+		changes[i][0] == KEYWORD_CHANGE_FINAL ||
 		changes[i][0] == KEYWORD_CHANGE_ADD_AND_FINAL) {
 			const char *name = changes[i]+1;
 
-- 
2.18.1

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Helmut K. C. Tessarek via dovecot 
On 2019-03-12 17:23, Timo Sirainen via dovecot wrote
> https://wiki2.dovecot.org/MailLocation

Sorry, this might be off-topic, but while reading up on the link you sent,
I've noticed the following sentence:

Use only absolute paths. Even if relative paths would appear to work, they
might just as well break some day.

Yet, all examples in the documentation use ~ which is a relative path.

Also, using an absolute path is impossible, if your users are located in
different parent home directories. e.g.:
/home2/ /home3/ /home4/ /var/users/

So how am I supposed to use an absolue path in such a case?
(If I wanted to have the maildir in their home directories and not in a
central location?)

Cheers,
  K. C.

-- 
regards Helmut K. C. Tessarek  KeyID 0x172380A011EF4944
Key fingerprint = 8A55 70C1 BD85 D34E ADBC 386C 1723 80A0 11EF 4944

/*
   Thou shalt not follow the NULL pointer for chaos and madness
   await thee at its end.
*/



signature.asc
Description: OpenPGP digital signature

Re: Dovecot 2.3.4 crash

2019-03-12 Thread Stephan Bosch via dovecot 




Op 01/12/2018 om 00:45 schreef azu...@pobox.sk:


Citát azu...@pobox.sk:


Citát azu...@pobox.sk:


Citát Timo Sirainen :


On 29 Nov 2018, at 15.46, azu...@pobox.sk wrote:


Hi,

is this a know problem? Newest Dovecot 2.3.4 package from 
repo.dovecot.org , Debian Stretch (fully upgraded).




Nov 29 14:25:11 server00 dovecot: lmtp(16854): Panic: file 
ostream-dot.c: line 208 (o_stream_dot_sendv): assertion failed: 
((size_t)ret == sent + added)


Is this proxying LMTP traffic?





Yes, it is. The main server is, currently, doing standard 
IMAP/POP3/LMTP for some clients and also proxying IMAP/POP3/LMTP for 
other clients (proxying to other servers). But this one error is 
related to one message which is stucked in postfix queue and cannot 
be delivered - the error appears everytime i do 'postfix flush' 
(error in postfix queue is 'lost connection with 
[private/dovecot-lmtp] while sending 
end of data -- message may be sent more than once').


There are no errors logged on proxy backend side.
Just to clarify this - that one message is to be proxied to other 
server.


Anything new to this? Thank you.


Fix is included in 2.3.5.

Regards,

Stephan.

Re: Crash in director for secure lmtp proxy connections

2019-03-12 Thread Stephan Bosch via dovecot 




Op 21/11/2018 om 14:21 schreef Marcus Fenner:

Hi,

After updating some of our director servers from 2.2.36 to 2.3.3
(dcead646b) we experience crashes in about 1/5000 mails in lmtp delivery:

Nov 13 12:23:36 Fatal: lmtp(113620): master: service(lmtp): child 113620
killed with signal 6 (core dumped)
Nov 13 12:23:36 Panic: lmtp(113623): file ostream-dot.c: line 208
(o_stream_dot_sendv): assertion failed: ((size_t)ret == sent + added)
Nov 13 12:23:36 Error: lmtp(113623): Raw backtrace:
/usr/lib64/dovecot/libdovecot.so.0(+0xe34fb) [0x7f31d89134fb] ->
/usr/lib64/dovecot/libdovecot.so.0(+0xe3597) [0x7f31d8913597] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x51207) [0x7f31d8881207] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x4eeef) [0x7f31d887eeef] ->
/usr/lib64/dovecot/libdovecot.so.0(+0x107cf8) [0x7f31d8937cf8] ->
/usr/lib64/dovecot/libdovecot.so.0(o_stream_sendv+0x32) [0x7f31d89382c2]
-> /usr/lib64/dovecot/libdovecot.so.0(io_stream_copy+0x5a)
[0x7f31d8938b6a] ->
/usr/lib64/dovecot/libdovecot.so.0(o_stream_send_istream+0x53)
[0x7f31d8938873] ->
/usr/lib64/dovecot/libdovecot.so.0(smtp_client_command_send_more+0x145)
[0x7f31d8890805] -> /usr/lib64/dovecot/libdovecot.so.0(+0x660dd)
[0x7f31d88960dd] ->
/usr/lib64/dovecot/libssl_iostream_openssl.so(+0xa7e1) [0x7f31d864a7e1]
-> /usr/lib64/dovecot/libdovecot.so.0(+0x10a558) [0x7f31d893a558] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x73)
[0x7f31d892a203] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x136)
[0x7f31d892b8b6] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x50)
[0x7f31d892a2b0] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x48)
[0x7f31d892a418] ->
/usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x17)
[0x7f31d88a95a7] -> dovecot/lmtp(main+0x245) [0x555b8b59cea5] ->
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f31d867c413] ->
dovecot/lmtp(_start+0x2e) [0x555b8b59cfbe]
Nov 13 12:23:36 Fatal: lmtp(113623): master: service(lmtp): child 113623
killed with signal 6 (core dumped)

On the backend servers the only message is
lmtp(9571): Disconnect from 192.168.1.56: Connection closed (in DATA)

This happens for every server with this version every time postfix tries
to resend the mail.

After removing "starttls=y" from the static passdb entry, the mail is
sent successfully. Starting with version 2.3.2 the lmtp proxy uses
starttls as well, therefore the error seems to be there. I've not been
able to figure out, if this has to do with the content of the mails,
because they are from many different sources.


Fix is included in 2.3.5.

Regards,

Stephan.

Re: lmtp 2.3.2.1 segfault with backtrace

2019-03-12 Thread Stephan Bosch via dovecot 

Hi,

Op 16/08/2018 om 00:53 schreef Stephan Bosch:

Hi,

I have reproduced this problem and I am working on a fix.


Fix is included in 2.3.5.

Regards,

Stephan.


Op 14/08/2018 om 11:44 schreef Tom Sommer:

lmtp on Director crash with 2.3.2.1

# gdb /usr/libexec/dovecot/lmtp /var/core/60174
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show 
copying"

and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/libexec/dovecot/lmtp...Reading symbols from 
/usr/lib/debug/usr/libexec/dovecot/lmtp.debug...done.

done.
[New Thread 60174]
Reading symbols from /usr/lib64/dovecot/libdovecot-lda.so.0...Reading 
symbols from 
/usr/lib/debug/usr/lib64/dovecot/libdovecot-lda.so.0.0.0.debug...done.

done.
Loaded symbols for /usr/lib64/dovecot/libdovecot-lda.so.0
Reading symbols from 
/usr/lib64/dovecot/libdovecot-storage.so.0...Reading symbols from 
/usr/lib/debug/usr/lib64/dovecot/libdovecot-storage.so.0.0.0.debug...done. 


done.
Loaded symbols for /usr/lib64/dovecot/libdovecot-storage.so.0
Reading symbols from /usr/lib64/dovecot/libdovecot.so.0...Reading 
symbols from 
/usr/lib/debug/usr/lib64/dovecot/libdovecot.so.0.0.0.debug...done.

done.
Loaded symbols for /usr/lib64/dovecot/libdovecot.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging 
symbols found)...done.

Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols 
found)...done.

[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from 
/usr/lib64/dovecot/libssl_iostream_openssl.so...Reading symbols from 
/usr/lib/debug/usr/lib64/dovecot/libssl_iostream_openssl.so.debug...done.

done.
Loaded symbols for /usr/lib64/dovecot/libssl_iostream_openssl.so
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols 
found)...done.

Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging 
symbols found)...done.

Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging 
symbols found)...done.

Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging 
symbols found)...done.

Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols 
found)...done.

Loaded symbols for /lib64/libselinux.so.1
Core was generated by `dovecot/lmtp'.
Program terminated with signal 11, Segmentation fault.
#0  smtp_client_command_set_replies (cmd=0x0, replies=1) at 
smtp-client-command.c:401

401 i_assert(cmd->replies_expected == 1 ||
Missing separate debuginfos, use: debuginfo-install 
glibc-2.12-1.212.el6.x86_64 keyutils-libs-1.4-5.el6.x86_64 
krb5-libs-1.10.3-65.el6.x86_64 libcom_err-1.41.12-24.el6.x86_64 
libselinux-2.0.94-7.el6.x86_64 openssl-1.0.1e-57.el6.x86_64 
zlib-1.2.3-29.el6.x86_64

(gdb) bt full
#0  smtp_client_command_set_replies (cmd=0x0, replies=1) at 
smtp-client-command.c:401

    __func__ = "smtp_client_command_set_replies"
#1  0x7f5d14f40f3f in smtp_client_transaction_data_cb 
(reply=0x7ffe5cd19650, trans=0x7f5d176ae2b8) at 
smtp-client-transaction.c:658

    conn = 0x7f5d176ade80
    rcpt = 0x7f5d176ae560
    i = 
    count = 1
#2  0x7f5d14f3e941 in smtp_client_command_fail_reply (_cmd=optimized out>, reply=0x7ffe5cd19650) at smtp-client-command.c:299

    cmd = 0x7f5d17600d18
    tmp_cmd = 
    conn =

Re: 2.3.3: Panic: file ostream-zlib.c: line 37 (o_stream_zlib_close): assertion failed

2019-03-12 Thread Stephan Bosch via dovecot 




Op 02/10/2018 om 11:26 schreef Tom Sommer:

I see this in my logs after 2.3.3:

using zlib plugin, ofc.

Oct 02 10:01:39 imap(u...@example.com)<50643>: 
Panic: file ostream-zlib.c: line 37 (o_stream_zlib_close): assertion 
failed: (zstream->ostream.finished || 
zstream->ostream.ostream.stream_errno != 0 || 
zstream->ostream.error_handling_disabled)
Oct 02 10:01:39 imap(u...@example.com)<50643>: 
Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0xce56a) 
[0x7f442487556a] -> /usr/lib64/dovecot/libdovecot.so.0(+0xce5b1) 
[0x7f44248755b1] -> /usr/lib64/dovecot/libdovecot.so.0(+0x3d941) 
[0x7f44247e4941] -> /usr/lib64/dovecot/lib20_zlib_plugin.so(+0x5cdf) 
[0x7f44233c4cdf] -> /usr/lib64/dovecot/libdovecot.so.0(+0xf4b46) 
[0x7f442489bb46] -> 
/usr/lib64/dovecot/libdovecot.so.0(o_stream_destroy+0x13) 
[0x7f442489be83] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(maildir_save_finish+0x173) 
[0x7f4424b93973] -> 
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_save_cancel+0x36) 
[0x7f4424b68696] -> dovecot/imap [u...@example.com X.X.X.X 
APPEND](+0xede9) [0x55a9d7e86de9] -> dovecot/imap [u...@example.com 
X.X.X.X APPEND](command_exec+0x65) [0x55a9d7e956e5] -> dovecot/imap 
[u...@example.com X.X.X.X APPEND](+0xf9e6) [0x55a9d7e879e6] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x55) 
[0x7f442488c275] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xbf) 
[0x7f442488e13f] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x55) 
[0x7f442488c365] -> 
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f442488c588] 
-> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7f4424808053] -> dovecot/imap [u...@example.com X.X.X.X 
APPEND](main+0x32d) [0x55a9d7ea20dd] -> 
/lib64/libc.so.6(__libc_start_main+0x100) [0x7f4424431d20] -> 
dovecot/imap [u...@example.com X.X.X.X APPEND](+0xe419) [0x55a9d7e86419]


No clue how to reproduce


Fix is included in 2.3.5.

Regards,

Stephan.

Re: submission-login: Fatal: master: service(submission-login):

2019-03-12 Thread Stephan Bosch via dovecot 




Op 11/03/2019 om 12:53 schreef Marcelo Coelho via dovecot:

Hi everyone!

I’m using dovecot 2.3.5. submission-login is crashing many times in a day:

Here is a sample error message:

dovecot: submission-login: Fatal: master: service(submission-login): child 
34247 killed with signal 11 (core not dumped - 
https://dovecot.org/bugreport.html#coredumps - set service submission-login { 
drop_priv_before_exec=yes })

After I added drop_priv_before_exec, I got these error messages:

submission-login: Error: master-service: cannot get filters: 
net_connect_unix(/var/run/dovecot/config) failed: Permission denied
dovecot: master: Error: service(submission-login): command startup failed, 
throttling for 2 secs

How to get submission core dumps properly?

Thank you.


We got a fix for this bug in off-list discussions.

Tracking as DOP-987.

Regards,

Stephan.

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Felipe Gasper via dovecot 


> On Mar 12, 2019, at 5:23 PM, Timo Sirainen via dovecot  
> wrote:
> 
> On 12 Mar 2019, at 21.20, Felipe Gasper via dovecot  
> wrote:
>> 
>> Hello,
>> 
>>  I’ve got a strange misconfiguration where the following command:
>> 
>> doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
>> 'INBOX.*'
>> 
>> … fails with error code 68, saying that it can’t find one of the mailboxes. 
>> (It lists the user’s other mailboxes.) The name of the mailbox in question 
>> is saved to disk in UTF-8 rather than mUTF-7, but strace shows that doveadm 
>> is stat()ing the mUTF-7 path; the failure of that stat() is, assumedly, what 
>> causes doveadm to report the error status.
>> 
>>  I’ve tried to paw through the source code to see what might be causing 
>> this but haven’t made much headway. Can someone here point out where the 
>> misconfiguration might be that is causing doveadm to stat() the mUTF-7 path 
>> rather than UTF-8? Or perhaps offer any tips as to how I might diagnose 
>> what’s going on? What causes doveadm to stat() one path or the other?
> 
> What's your doveconf -n? Using UTF-8 on filesystem requires using "UTF-8" 
> option in mail_location. Do you have it set? 
> https://wiki2.dovecot.org/MailLocation

It turns out we had a missing a cache expiration when a user switches to UTF-8 
filenames. So that’s why doveadm was looking for the mUTF-7 filename.

Now that we tracked that down, this appears to be working as it should.

Thank you, everyone!

-FG

Re: question about %u and %h used in mail_location

2019-03-12 Thread Timo Sirainen via dovecot 
On 12 Mar 2019, at 12.04, Joe Wong via dovecot  wrote:
> 
> Hello,
> 
>  I have defined the following: 
> 
> mail_location = maildir:~:INBOX=~/Maildir:LAYOUT=fs:INDEX=%u
> 
> %u is retrieve via database in that my username contain ":", in which it 
> create some confusion to dovecot:
> 
> doveadm index -u user1:site@domain iNBOX
> doveadm(user1:site@domain): Error: remote(192.168.10.22:24245 
> ): Namespace '': Unknown setting: site@domain
> 
> I cannot change the value in DB so is there a workaround to this problem?

Convert the username to not have ':' when it comes out of auth. Could be 
possible with SQL userdb. With others .. I'm not sure, you might have to use 
Lua to convert it.

Re: Delayed flags changes over IDLE

2019-03-12 Thread Timo Sirainen via dovecot 
On 12 Mar 2019, at 10.21, Kostya Vasilyev via dovecot  
wrote:
> 
> It makes no difference if the IDLE connection does SELECT or SELECT 
> (CONDSTORE) prior to going IDLE.
> 
> But then as far as I know (?) - in Dovecot, once any connection uses 
> CONDSTORE ever, even once, Dovecot creates data structures to track MODSEQ 
> values, and those data structures are forever.

So are you saying that you can reproduce if you do for a completely new user:

doveadm exec imap -u testuser1
a select inbox
b idle

And then run:
echo foo | doveadm save -u testuser1
doveadm flags add -u testuser1 '\Seen' mailbox inbox 1

And the EXISTS shows up immediately after saving, but the flag change won't 
show up? It works fine with me.

Do you see any errors in "doveadm log errors"? Can you reproduce this if you 
try with some other mailbox format than mbox?

Re: dovecot-keywords are not preserved any more when moving mails between folders

2019-03-12 Thread Timo Sirainen via dovecot 
On 12 Mar 2019, at 17.55, Dan Christensen via dovecot  
wrote:
> 
> On Mar 12, 2019, Aki Tuomi via dovecot  wrote:
> 
>> On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
>> 
>>> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
>>> which in my case are set by thunderbird, are not preserved any more when
>>> moving a mail between folders.
>> 
>> We are aware of this bug, and it's being tracked as DOP-842.
> 
> Could this bug also be causing flags to be lost when using dsync
> (as I described in some messages to this list Feb 16 to 23)?
> 
> It seems like it might be a different bug, since in my experience
> the flags are sometimes synced and then removed later.

That bug is fixed with attached patch.



2656.patch
Description: Binary data

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Timo Sirainen via dovecot 
On 12 Mar 2019, at 21.20, Felipe Gasper via dovecot  wrote:
> 
> Hello,
> 
>   I’ve got a strange misconfiguration where the following command:
> 
> doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
> 'INBOX.*'
> 
> … fails with error code 68, saying that it can’t find one of the mailboxes. 
> (It lists the user’s other mailboxes.) The name of the mailbox in question is 
> saved to disk in UTF-8 rather than mUTF-7, but strace shows that doveadm is 
> stat()ing the mUTF-7 path; the failure of that stat() is, assumedly, what 
> causes doveadm to report the error status.
> 
>   I’ve tried to paw through the source code to see what might be causing 
> this but haven’t made much headway. Can someone here point out where the 
> misconfiguration might be that is causing doveadm to stat() the mUTF-7 path 
> rather than UTF-8? Or perhaps offer any tips as to how I might diagnose 
> what’s going on? What causes doveadm to stat() one path or the other?

What's your doveconf -n? Using UTF-8 on filesystem requires using "UTF-8" 
option in mail_location. Do you have it set? 
https://wiki2.dovecot.org/MailLocation

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Bob Gustafson via dovecot 
A tool to determine the encoding of a file is 'file -bi ' This 
command is not perfect though.


On 3/12/19 2:20 PM, Felipe Gasper via dovecot wrote:

Hello,

I’ve got a strange misconfiguration where the following command:

doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
'INBOX.*'

… fails with error code 68, saying that it can’t find one of the mailboxes. (It 
lists the user’s other mailboxes.) The name of the mailbox in question is saved 
to disk in UTF-8 rather than mUTF-7, but strace shows that doveadm is stat()ing 
the mUTF-7 path; the failure of that stat() is, assumedly, what causes doveadm 
to report the error status.

I’ve tried to paw through the source code to see what might be causing 
this but haven’t made much headway. Can someone here point out where the 
misconfiguration might be that is causing doveadm to stat() the mUTF-7 path 
rather than UTF-8? Or perhaps offer any tips as to how I might diagnose what’s 
going on? What causes doveadm to stat() one path or the other?

Thank you!


-Felipe Gasper
Mississauga, ON

Re: Regression ACL & namespace prefix

2019-03-12 Thread Timo Sirainen via dovecot 
On 18 Sep 2018, at 17.10, Michal Hlavinka  wrote:
> 
> Seems that for Global ACL directory, namespace prefix is not part of the 
> path, when looking for acl file.

Is there a reason you're using ACL directory instead of ACL file? I've rather 
been thinking about removing code for ACL directories entirely at some point.

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Aki Tuomi via dovecot 


> On 12 March 2019 21:37 Felipe Gasper via dovecot  wrote:
> 
>  
> > On Mar 12, 2019, at 3:28 PM, Aki Tuomi  wrote:
> > 
> > 
> >> On 12 March 2019 21:20 Felipe Gasper via dovecot  
> >> wrote:
> >> 
> >> 
> >> Hello,
> >> 
> >>I’ve got a strange misconfiguration where the following command:
> >> 
> >> doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
> >> 'INBOX.*'
> >> 
> >> … fails with error code 68, saying that it can’t find one of the 
> >> mailboxes. (It lists the user’s other mailboxes.) The name of the mailbox 
> >> in question is saved to disk in UTF-8 rather than mUTF-7, but strace shows 
> >> that doveadm is stat()ing the mUTF-7 path; the failure of that stat() is, 
> >> assumedly, what causes doveadm to report the error status.
> >> 
> >>I’ve tried to paw through the source code to see what might be causing 
> >> this but haven’t made much headway. Can someone here point out where the 
> >> misconfiguration might be that is causing doveadm to stat() the mUTF-7 
> >> path rather than UTF-8? Or perhaps offer any tips as to how I might 
> >> diagnose what’s going on? What causes doveadm to stat() one path or the 
> >> other?
> >> 
> >>Thank you!
> >> 
> >> 
> >> -Felipe Gasper
> >> Mississauga, ON
> > 
> > Mailbox should be stored on disk using mutf7, not UTF-8.
> > 
> > Aki
> 
> Hm. I have other users where the command above works, and the mailbox is 
> stored on-disk with a UTF-8 name, and the stat() call is indeed referencing 
> the UTF-8 name.
> 
> Is there nothing in Dovecot that allows for a variance?
> 
> -FG

Would you be able to provide examples?

Aki

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Felipe Gasper via dovecot 


> On Mar 12, 2019, at 3:28 PM, Aki Tuomi  wrote:
> 
> 
>> On 12 March 2019 21:20 Felipe Gasper via dovecot  wrote:
>> 
>> 
>> Hello,
>> 
>>  I’ve got a strange misconfiguration where the following command:
>> 
>> doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
>> 'INBOX.*'
>> 
>> … fails with error code 68, saying that it can’t find one of the mailboxes. 
>> (It lists the user’s other mailboxes.) The name of the mailbox in question 
>> is saved to disk in UTF-8 rather than mUTF-7, but strace shows that doveadm 
>> is stat()ing the mUTF-7 path; the failure of that stat() is, assumedly, what 
>> causes doveadm to report the error status.
>> 
>>  I’ve tried to paw through the source code to see what might be causing 
>> this but haven’t made much headway. Can someone here point out where the 
>> misconfiguration might be that is causing doveadm to stat() the mUTF-7 path 
>> rather than UTF-8? Or perhaps offer any tips as to how I might diagnose 
>> what’s going on? What causes doveadm to stat() one path or the other?
>> 
>>  Thank you!
>> 
>> 
>> -Felipe Gasper
>> Mississauga, ON
> 
> Mailbox should be stored on disk using mutf7, not UTF-8.
> 
> Aki

Hm. I have other users where the command above works, and the mailbox is stored 
on-disk with a UTF-8 name, and the stat() call is indeed referencing the UTF-8 
name.

Is there nothing in Dovecot that allows for a variance?

-FG

Re: “doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Aki Tuomi via dovecot 


> On 12 March 2019 21:20 Felipe Gasper via dovecot  wrote:
> 
>  
> Hello,
> 
>   I’ve got a strange misconfiguration where the following command:
> 
> doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
> 'INBOX.*'
> 
> … fails with error code 68, saying that it can’t find one of the mailboxes. 
> (It lists the user’s other mailboxes.) The name of the mailbox in question is 
> saved to disk in UTF-8 rather than mUTF-7, but strace shows that doveadm is 
> stat()ing the mUTF-7 path; the failure of that stat() is, assumedly, what 
> causes doveadm to report the error status.
> 
>   I’ve tried to paw through the source code to see what might be causing 
> this but haven’t made much headway. Can someone here point out where the 
> misconfiguration might be that is causing doveadm to stat() the mUTF-7 path 
> rather than UTF-8? Or perhaps offer any tips as to how I might diagnose 
> what’s going on? What causes doveadm to stat() one path or the other?
> 
>   Thank you!
> 
> 
> -Felipe Gasper
> Mississauga, ON

Mailbox should be stored on disk using mutf7, not UTF-8.

Aki

“doveadm mailbox” command fails with UTF-8 mailboxes

2019-03-12 Thread Felipe Gasper via dovecot 
Hello,

I’ve got a strange misconfiguration where the following command:

doveadm -f pager mailbox status -u spamutf8 'messages vsize guid' INBOX 
'INBOX.*'

… fails with error code 68, saying that it can’t find one of the mailboxes. (It 
lists the user’s other mailboxes.) The name of the mailbox in question is saved 
to disk in UTF-8 rather than mUTF-7, but strace shows that doveadm is stat()ing 
the mUTF-7 path; the failure of that stat() is, assumedly, what causes doveadm 
to report the error status.

I’ve tried to paw through the source code to see what might be causing 
this but haven’t made much headway. Can someone here point out where the 
misconfiguration might be that is causing doveadm to stat() the mUTF-7 path 
rather than UTF-8? Or perhaps offer any tips as to how I might diagnose what’s 
going on? What causes doveadm to stat() one path or the other?

Thank you!


-Felipe Gasper
Mississauga, ON

Re: imap-hibernate not working

2019-03-12 Thread Aki Tuomi via dovecot 
This is reproducible at my end.

Aki

> On 12 March 2019 02:57 Larry Rosenman via dovecot  wrote:
> 
> 
> If I can help (I'm the FreeBSD port maintainer for dovecot), I'm willing.
> 
> I can also provide access to a server to help debug
> 
> 
> On Mon, Mar 11, 2019 at 7:54 PM Timo Sirainen via dovecot 
>  wrote:
> > On 8 Mar 2019, at 20.44, Marcelo Coelho via dovecot  
> > wrote:
> >  > 
> >  > Hi,
> >  > 
> >  > I follow different setup instructions and I can't make imap-hibernate 
> > work. I've tried vmail and dovecot as users, tried to set mode to 0666, 
> > without success. I'm using FreeBSD 11.2.
> >  > 
> >  > Is imap-hibernate compatible with FreeBSD 11.2?
> >  > 
> >  > 
> >  > 
> >  > My operational system:
> >  > 
> >  > # uname -v
> >  > FreeBSD 11.2-RELEASE-p9 #0: Tue Feb 5 15:30:36 UTC 2019 
> > r...@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC 
> >  > 
> >  > Here are my logs:
> >  > 
> >  > Mar 8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> > Error: kevent(-1) for notify remove failed: Bad file descriptor
> >  > Mar 8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> > Error: close(-1) for notify remove failed: Bad file descriptor
> >  > Mar 8 15:30:57 servername dovecot: imap-hibernate: Error: Failed to 
> > parse client input: Invalid peer_dev_minor value: 18446744073709486335
> >  > Mar 8 15:30:57 servername dovecot: imap(u...@domain.com:52.125.128.90): 
> > Error: /opt/dovecot/2.3.5/var/run/dovecot/imap-hibernate returned failure: 
> > Failed to parse client input: Invalid peer_dev_minor value: 
> > 18446744073709486335
> >  
> >  Looks bad. I suppose it's broken with FreeBSD.
> >  
> > 
> 
> 
> 
> -- 
> 
> Larry Rosenman http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c) E-Mail:larry...@gmail.com
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: dovecot-keywords are not preserved any more when moving mails between folders

2019-03-12 Thread Dan Christensen via dovecot 
On Mar 12, 2019, Aki Tuomi via dovecot  wrote:

> On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
>
>> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
>> which in my case are set by thunderbird, are not preserved any more when
>> moving a mail between folders.
>
> We are aware of this bug, and it's being tracked as DOP-842.

Could this bug also be causing flags to be lost when using dsync
(as I described in some messages to this list Feb 16 to 23)?

It seems like it might be a different bug, since in my experience
the flags are sometimes synced and then removed later.

Dan

Re: Regression ACL & namespace prefix

2019-03-12 Thread Michal Hlavinka via dovecot 

Hi,

thanks for the answer. I think your environment was not set up correctly
to reproduce this bug. I've retested with 2.3.5 and I can still
reproduce it. I've attached a script that will configure everything for
testing and if you have a virtual machine available, you can use it
directly (it expects linux with systemd for dovecot restart).

relevant section from config:
namespace {
  hidden = no
  list = yes
  location = maildir:/var/mail/pub
  prefix = pub/
  separator = /
  type = public
}

this expects maildir directly in pub:
/var/mail/pub/cur
/var/mail/pub/new
/var/mail/pub/tmp

as it uses '/' separator and there could be subfolders, it should look
for .DEFAULT file in global acls directory which it does not in your
debug output

doveadm(testuser): Info: Mailbox '' is in namespace 'pub/'
doveadm(testuser): Info: All message flags are shared across users in 
mailbox

doveadm(testuser): Debug: acl vfile: file /etc/dovecot/global-acls//.DEFAULT
not found
doveadm(testuser): Debug: acl vfile: file /var/mail/pub/dovecot-acl not 
found

doveadm(testuser): Info: User testuser has no rights for mailbox
doveadm(testuser): Error: User testuser is missing 'lookup' right
doveadm(testuser): Info: Mailbox pub is NOT visible in LIST

in this output see that it checks this location:
acl vfile: file /etc/dovecot/global-acls//.DEFAULT not found

instead of

/etc/dovecot/global-acls/pub/.DEFAULT

this is caused by line in
src/plugins/acl/acl-backend-vfile.c: acl_backend_vfile_object_init(...)

vname = *name == '\0' ? "" :
 mailbox_list_get_vname(_backend->list, name);

and because name is empty, it will not use the "pub" prefix in the path.
If I'd test acl for "pub/subfolder" that condition would have different
result and bug would not trigger:

doveadm(testuser): Debug: acl vfile: reading file
/etc/dovecot/global-acls/pub/subfolder/.DEFAULT


For testing I use this acl configuration:
cat /etc/dovecot/global-acls/pub/.DEFAULT
user=testuser l

but as this acl file location is not found by dovecot, content should
not matter.


Cheers,
Michal Hlavinka


On 3/7/19 7:00 PM, Aki Tuomi via dovecot wrote:

I tested with release 2.3.5, and

doveadm -Dv acl debug -u testuser pub doveadm(testuser): Debug: acl
vfile: file /etc/dovecot/global-acls/pub/INBOX not found 
doveadm(testuser): Debug: acl vfile: file

/home/vmail/pub/Mail/mailboxes/INBOX/dbox-Mails/dovecot-acl not
found doveadm(testuser): Debug: acl vfile: file
/etc/dovecot/global-acls/ not found doveadm(testuser): Debug: acl
vfile: file /home/vmail/pub/Mail/mailboxes/dovecot-acl not found

so our advice is to upgrade into 2.3.5, as 2.2.36 is no longer in
development.

Aki


On 7 March 2019 19:47 Aki Tuomi via dovecot 
wrote:


Sorry, we have not yet been able to look into this..

It's now in our internal system as DOP-966

Aki


On 7 March 2019 17:31 Michal Hlavinka via dovecot
 wrote:


Hi, any progress with this issue? Do you need more information to
debug and fix this?

Cheers Michal Hlavinka

On 9/18/18 4:10 PM, Michal Hlavinka wrote:

Hi

tl;dr: Seems that for Global ACL directory, namespace prefix is
not part of the path, when looking for acl file.

Long version:

We're planning to update dovecot in next os update to 2.2.36
and while going through regression testing, we found a problem
with ACL configuration combined with namespace.

Test uses "Global ACL directory" configuration.

Relevant configuration part: mail_location = maildir:~/Maildir

namespace inbox { hidden = no inbox = yes list = yes location
= prefix = separator = / } namespace { hidden = no list = yes 
location = maildir:/var/mail/pub prefix = pub/ separator = / 
type = public }


mail_plugins = acl

protocol imap { mail_plugins = $mail_plugins acl imap_acl } 
plugin { acl = vfile:/etc/dovecot/global-acls }


ACL config file is stored at: 
/etc/dovecot/global-acls/pub/.DEFAULT


when trying to examine "pub", it is denied: fetchmail: IMAP>
A0005 EXAMINE "pub" fetchmail: IMAP< A0005 NO Mailbox doesn't
exist: pub (0.001 + 0.000 secs).

# doveadm acl debug -u d2 pub doveadm(d2): Info: Mailbox '' is
in namespace 'pub/' doveadm(d2): Info: Mailbox path:
/var/mail/pub doveadm(d2): Info: All message flags are shared
across users in mailbox doveadm(d2): Info: User d2 has no
rights for mailbox doveadm(d2): Error: User d2 is missing
'lookup' right doveadm(d2): Info: Mailbox pub is NOT visible in
LIST

because it did not find acl file: imap(d2): Debug: Namespace :
type=public, prefix=pub/, sep=/, inbox=no, hidden=no, list=yes,
subscriptions=yes location=maildir:/var/mail/pub imap(d2):
Debug: maildir++: root=/var/mail/pub, index=, indexpvt=, 
control=, inbox=, alt= imap(d2): Debug: acl: initializing

backend with data: vfile:/etc/dovecot/global-acls imap(d2):
Debug: acl: acl username = d2 imap(d2): Debug: acl: owner = 0 
imap(d2): Debug: acl vfile: Global ACL legacy directory: 
/etc/dovecot/global-acls imap(d2): Debug: pub: Mailbox opened

because: EXAMINE imap(d2): Debug: acl vfile: file

Re: dovecot-keywords are not preserved any more when moving mails between folders

2019-03-12 Thread Piper Andreas via dovecot 
Am 12.03.19 um 12:57 schrieb Aki Tuomi via dovecot:
> 
> On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
>> Hello,
>>
>> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
>> which in my case are set by thunderbird, are not preserved any more when
>> moving a mail between folders.
>>
> 
> Hi!
> 
> We are aware of this bug, and it's being tracked as DOP-842.
> 
> Aki
> 
Hello Aki,

thanks for the immediate answer.

To which version would I have to downgrade to avoid this problem?

Andreas
-- 

Dr. Andreas Piper, Hochschulrechenzentrum der Philipps-Univ. Marburg
  Hans-Meerwein-Straße 6, 35032 Marburg, Germany
Phone: +49 6421 28-23521  Fax: -26994  E-Mail: pi...@hrz.uni-marburg.de



smime.p7s
Description: S/MIME Cryptographic Signature

Re: dovecot-keywords are not preserved any more when moving mails between folders

2019-03-12 Thread Aki Tuomi via dovecot 


On 12.3.2019 13.46, Piper Andreas via dovecot wrote:
> Hello,
>
> after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
> which in my case are set by thunderbird, are not preserved any more when
> moving a mail between folders.
>
> Are there any ideas, what may be the reason.
>
> Thanks for any hints on that,
> Andreas
>
> 'doveconf -n' gives:
>
> # 2.3.4 (0ecbaf23d): /etc/opt/csw/dovecot/dovecot.conf
> # Pigeonhole version 0.5.4 (60b0f48d)
> # OS: SunOS 5.11 i86pc
> # Hostname: x.hrz.uni-marburg.de
> auth_cache_negative_ttl = 0
> auth_cache_size = 10 M
> auth_master_user_separator = *
> auth_username_chars =
> abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_
> auth_username_format = %u
> auth_worker_max_count = 1024
> base_dir = /var/run/dovecot/
> default_vsz_limit = 1 G
> first_valid_gid = 3
> first_valid_uid = 3
> imap_max_line_length = 640 k
> mail_debug = yes
> mail_location = maildir:%h/.maildir
> mail_plugins = " mail_log notify"
> mail_privileged_group = mail
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate mime foreverypart extracttext
> namespace {
>   hidden = no
>   inbox = yes
>   list = yes
>   location = maildir:%h/.maildir
>   prefix =
>   separator = /
>   subscriptions = yes
>   type = private
> }
> namespace inbox {
>   hidden = yes
>   inbox = no
>   list = no
>   location = maildir:%h/.maildir
>   prefix = mail/
>   separator = /
>   subscriptions = no
>   type = private
> }
> passdb {
>   args = /etc/dovecot.deny
>   deny = yes
>   driver = passwd-file
> }
> passdb {
>   args = /etc/opt/csw/dovecot/private/passwd.masterusers
>   driver = passwd-file
>   master = yes
> }
> passdb {
>   args = /etc/opt/csw/dovecot/dovecot-ldap.conf.ext
>   driver = ldap
> }
> plugin {
>   sieve = file:~/sieve;active=~/.dovecot.sieve
>   sieve_default = /var/lib/dovecot/default.sieve
> }
> pop3_uidl_format = %08Xv%08Xu
> postmaster_address = postmas...@hrz.uni-marburg.de
> protocols = imap pop3 lmtp sieve
> service auth-worker {
>   user = $default_internal_user
> }
> service auth {
>   client_limit = 6000
> }
> service imap-login {
>   process_min_avail = 64
>   service_count = 0
> }
> service imap {
>   process_limit = 4096
> }
> service lmtp {
>   inet_listener lmtp {
> port = 24
>   }
> }
> service managesieve-login {
>   inet_listener sieve {
> port = 4190
>   }
> }
> ssl_cert =  ssl_dh = # hidden, use -P to show it
> ssl_key = # hidden, use -P to show it
> userdb {
>   args = /etc/opt/csw/dovecot/dovecot-ldap-userdb.conf.ext
>   driver = ldap
> }
> protocol imap {
>   mail_max_userip_connections = 25
>   ssl_cert =ssl_key = # hidden, use -P to show it
> }
> protocol lmtp {
>   mail_plugins = " mail_log notify sieve"
> }
> protocol pop3 {
>   ssl_cert =ssl_key = # hidden, use -P to show it
> }
>

Hi!

We are aware of this bug, and it's being tracked as DOP-842.

Aki

dovecot-keywords are not preserved any more when moving mails between folders

2019-03-12 Thread Piper Andreas via dovecot 
Hello,

after an upgrade of dovecot-2.2.5 to dovecot-2.3.4 the dovecot-keywords,
which in my case are set by thunderbird, are not preserved any more when
moving a mail between folders.

Are there any ideas, what may be the reason.

Thanks for any hints on that,
Andreas

'doveconf -n' gives:

# 2.3.4 (0ecbaf23d): /etc/opt/csw/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 (60b0f48d)
# OS: SunOS 5.11 i86pc
# Hostname: x.hrz.uni-marburg.de
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_master_user_separator = *
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_
auth_username_format = %u
auth_worker_max_count = 1024
base_dir = /var/run/dovecot/
default_vsz_limit = 1 G
first_valid_gid = 3
first_valid_uid = 3
imap_max_line_length = 640 k
mail_debug = yes
mail_location = maildir:%h/.maildir
mail_plugins = " mail_log notify"
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace {
  hidden = no
  inbox = yes
  list = yes
  location = maildir:%h/.maildir
  prefix =
  separator = /
  subscriptions = yes
  type = private
}
namespace inbox {
  hidden = yes
  inbox = no
  list = no
  location = maildir:%h/.maildir
  prefix = mail/
  separator = /
  subscriptions = no
  type = private
}
passdb {
  args = /etc/dovecot.deny
  deny = yes
  driver = passwd-file
}
passdb {
  args = /etc/opt/csw/dovecot/private/passwd.masterusers
  driver = passwd-file
  master = yes
}
passdb {
  args = /etc/opt/csw/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_default = /var/lib/dovecot/default.sieve
}
pop3_uidl_format = %08Xv%08Xu
postmaster_address = postmas...@hrz.uni-marburg.de
protocols = imap pop3 lmtp sieve
service auth-worker {
  user = $default_internal_user
}
service auth {
  client_limit = 6000
}
service imap-login {
  process_min_avail = 64
  service_count = 0
}
service imap {
  process_limit = 4096
}
service lmtp {
  inet_listener lmtp {
port = 24
  }
}
service managesieve-login {
  inet_listener sieve {
port = 4190
  }
}
ssl_cert = 

smime.p7s
Description: S/MIME Cryptographic Signature

question about %u and %h used in mail_location

2019-03-12 Thread Joe Wong via dovecot 
Hello,

 I have defined the following:

mail_location = maildir:~:INBOX=~/Maildir:LAYOUT=fs:INDEX=%u

%u is retrieve via database in that my username contain ":", in which it
create some confusion to dovecot:

doveadm index -u user1:site@domain iNBOX
doveadm(user1:site@domain): Error: remote(192.168.10.22:24245): Namespace
'': Unknown setting: site@domain

I cannot change the value in DB so is there a workaround to this problem?

TIA.

- Joe

Re: Delayed flags changes over IDLE

2019-03-12 Thread Kostya Vasilyev via dovecot 
One more data point Timo:

On Tue, Mar 12, 2019, at 9:58 AM, Kostya Vasilyev via dovecot wrote:
> Timo,
> 
> On Tue, Mar 12, 2019, at 3:56 AM, Timo Sirainen via dovecot wrote:
>> On 10 Mar 2019, at 10.14, Kostya Vasilyev via dovecot  
>> wrote:
>>> 
>>> My mail is stored under ~/mail/.imap (not sure what this format is called), 
>>> I mean not "single file mbox".
>>> 
>>> I have not changed any IDLE related config settings:
>>> 
>>> doveconf | grep -i idle
>>> default_idle_kill = 1 mins
>>> director_ping_idle_timeout = 30 secs
>>> imap_idle_notify_interval = 2 mins
>>> imapc_max_idle_time = 29 mins
>>> mailbox_idle_check_interval = 30 secs
>>> 
>>> What can I do to make Dovecot notify IDLE clients about flags changes - 
>>> more quickly? Preferably near-instant?
>> 
>> It should simply just work, assuming there aren't any weird inotify limits, 
>> but you should get errors logged about reaching those. You could see if it 
>> makes any difference to set mailbox_idle_check_interval=1s
>> 
> 
> mailbox_idle_check_interval = 1 secs
> 
> did not help
> 
> Here is an interesting case
> 
> - Let's say I have my IDLE connection.
> 
> - Connection "A" - an email app where I set or clear \Flagged on two messages.
> 
> The IDLE connection is silent (no unsolicited notifications about these 
> flags).
> 
> - Connection "B" - an email app where I do a refresh - it does a SELECT 
> (CONDSTORE) followed by FETCH UID FLAGS CHANGEDSINCE (because it sees that 
> there are only flags changes).
> 
> ---> the IDLE connection is notified about flags changes immediately right as 
> Connection "B" is pulling its changes.
> 
> I tried a direct network connection (netcat) instead of Connection "B" and 
> the actual trigger is SELECT (CONDSTORE).
> 
> - Start up IDLE connection
> 
> - Use Connection "A" to make flags changes
> 
> ( IDLE connection is silent )
> 
> - Use a netcat connection to SELECT (CONDSTORE)
> 
> Dovecot flushes flags to the IDLE connection immediately
> 
> - Doing SELECT (without CONDSTORE) does not
> 
> Looks like some sort of bug in Dovecot related to CONDSTORE?
> 
> This can probably be reproduced with several direct network connections using 
> netcat / openssl s_client - and CONDSTORE seems to be an important part of 
> the scenario.
> 
> Ideas?
> 
> -- K

It makes no difference if the IDLE connection does SELECT or SELECT (CONDSTORE) 
prior to going IDLE.

But then as far as I know (?) - in Dovecot, once any connection uses CONDSTORE 
ever, even once, Dovecot creates data structures to track MODSEQ values, and 
those data structures are forever.

-- K

Re: Delayed flags changes over IDLE

2019-03-12 Thread Kostya Vasilyev via dovecot 
Timo,

On Tue, Mar 12, 2019, at 3:56 AM, Timo Sirainen via dovecot wrote:
> On 10 Mar 2019, at 10.14, Kostya Vasilyev via dovecot  
> wrote:
>> 
>> My mail is stored under ~/mail/.imap (not sure what this format is called), 
>> I mean not "single file mbox".
>> 
>> I have not changed any IDLE related config settings:
>> 
>> doveconf | grep -i idle
>> default_idle_kill = 1 mins
>> director_ping_idle_timeout = 30 secs
>> imap_idle_notify_interval = 2 mins
>> imapc_max_idle_time = 29 mins
>> mailbox_idle_check_interval = 30 secs
>> 
>> What can I do to make Dovecot notify IDLE clients about flags changes - more 
>> quickly? Preferably near-instant?
> 
> It should simply just work, assuming there aren't any weird inotify limits, 
> but you should get errors logged about reaching those. You could see if it 
> makes any difference to set mailbox_idle_check_interval=1s
> 

mailbox_idle_check_interval = 1 secs

did not help

Here is an interesting case

- Let's say I have my IDLE connection.

- Connection "A" - an email app where I set or clear \Flagged on two messages.

The IDLE connection is silent (no unsolicited notifications about these flags).

- Connection "B" - an email app where I do a refresh - it does a SELECT 
(CONDSTORE) followed by FETCH UID FLAGS CHANGEDSINCE (because it sees that 
there are only flags changes).

---> the IDLE connection is notified about flags changes immediately right as 
Connection "B" is pulling its changes.

I tried a direct network connection (netcat) instead of Connection "B" and the 
actual trigger is SELECT (CONDSTORE).

- Start up IDLE connection

- Use Connection "A" to make flags changes

( IDLE connection is silent )

- Use a netcat connection to SELECT (CONDSTORE)

Dovecot flushes flags to the IDLE connection immediately

- Doing SELECT (without CONDSTORE) does not

Looks like some sort of bug in Dovecot related to CONDSTORE?

This can probably be reproduced with several direct network connections using 
netcat / openssl s_client - and CONDSTORE seems to be an important part of the 
scenario.

Ideas?

-- K