Dynamically set LDAP user namespace using two attributes

[Thomas Leuxner]

Hi,

I’m looking for a way to use two LDAP attributes to create a user-specific 
namespace to be expunged e.g.


mailExpungeTrash=namespace/inbox/mailbox/%{ldap:mailTrashMailbox}/autoexpunge

I can create the namespace fine returning the full string to be expunged:

mailExpungeTrash=namespace/inbox/mailbox/Deleted\_Messages/autoexpunge

However I want to alter the namespace using a second LDAP attribute like:


mailExpungeTrash=namespace/inbox/mailbox/%{ldap:mailTrashMailbox}/autoexpunge

Regards
Thomas





signature.asc
Description: Message signed with OpenPGP

Re: Local access to IMAP mailboxes

[Thomas Leuxner]

* Victor Sudakov  2018.09.26 12:17:

> > >> However, I often read and modify the mailboxes locally with Mutt (e.g.
> > >> append and delete mails).

Why not use Mutt's IMAP capabilities and keep the indexes nice and clean?

Regards
Thomas


signature.asc
Description: PGP signature

Re: imapsieve: script not triggered

[Thomas Leuxner]

* Andreas Krischer  2018.05.07 19:58:

>   sieve_global_extensions = +vnd.dovecot.pipe

Hi,

my working configuration looks like this:

sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute

Regards
Thomas



signature.asc
Description: PGP signature

Dovecot 2.3-rc Logging Format

[Thomas Leuxner]

Hi,

the release candidate defaults to a log format with session IDs.

mail_log_prefix = "%s(%u)<%{pid}><%{session}>: "

As the LMTP service seems to have the session ID hardcoded, the IDs get 
duplicated in the logs:

Dec 21 08:48:03 edi dovecot: lmtp(26573): Connect from local
Dec 21 08:48:03 edi dovecot: lmtp(t...@leuxner.net)[26573]: 
: fCVaBjNnO1rNZwAAIROLbg: sieve: 
msgid=<2323281.OorJHhdMHM@ylum>, time=158ms, status=stored mail into mailbox 
':public/Mailing-Lists/Debian-User'
Dec 21 08:48:03 edi dovecot: lmtp(26573): Disconnect from local: Client has 
quit the connection (state = READY)

Regards
Thomas


signature.asc
Description: PGP signature

Re: v2.3.0 release candidate released

[Thomas Leuxner]

* Timo Sirainen  2017.12.18 16:23:

Hi,

what is the correct way of implementing carbon stats with 2.3?

/etc/dovecot/conf.d/90-stats.conf: 
old_stats_carbon_server=127.0.0.1:2003
old_stats_carbon_name=host_domain_tld
old_stats_carbon_interval=60s

/etc/dovecot/conf.d/20-imap.conf:

mail_plugins =

I changed imap_stats to imap_old_stats, however this yields the following error:

Dec 20 10:20:30 edi dovecot: imap(t...@leuxner.net)<26352><9VA9GMJgns4FkqmS>: 
Error: module /usr/lib/dovecot/modules/lib95_imap_old_stats_plugin.so: 
dlsym(imap_old_stats_plugin_init) failed: 
/usr/lib/dovecot/modules/lib95_imap_old_stats_plugin.so: undefined symbol: 
imap_old_stats_plugin_init
Dec 20 10:20:30 edi dovecot: imap(t...@leuxner.net)<26352><9VA9GMJgns4FkqmS>: 
Error: module /usr/lib/dovecot/modules/lib95_imap_old_stats_plugin.so: 
dlsym(imap_old_stats_plugin_deinit) failed: 
/usr/lib/dovecot/modules/lib95_imap_old_stats_plugin.so: undefined symbol: 
imap_old_stats_plugin_deinit
Dec 20 10:20:30 edi dovecot: imap(t...@leuxner.net): Error: Couldn't load 
required plugin /usr/lib/dovecot/modules/lib95_imap_old_stats_plugin.so: Module 
doesn't have init function

Regards
Thomas


signature.asc
Description: PGP signature

Re: Dovecot lmtp doesn't log

[Thomas Leuxner]

* Tomislav Perisic  2017.12.01 15:30:

> Does anyone have a working configuration regarding this that they don't
> have a problem with LMTP logging? If yes could you please send me your
> config and dovecot version to compare.
2.2.devel (904765b05):

# doveconf deliver_log_format syslog_facility
deliver_log_format = msgid=%m, time=%{delivery_time}ms, status=%$
syslog_facility = local1

rsyslog.conf:
local1.*-/var/log/dovecot/dovecot.log

local1.info   -/var/log/dovecot/dovecot.info
local1.warn   -/var/log/dovecot/dovecot.warn
local1.err/var/log/dovecot/dovecot.err
if ($syslogfacility-text=='local1') and ($programname=='dovecot') and\
($msg contains 'lmtp') and ($msg contains 'stored mail into mailbox')\
 then -/var/log/dovecot/dovecot.lmtp


signature.asc
Description: PGP signature

Re: stats module

[Thomas Leuxner]

* Jeff Abrahamson  2017.11.03 17:45:

> > >>     -rw-r--r-- 1 root root  1856 Nov  3 16:11 91-stats

Please take note of the include scheme:
!include conf.d/*.conf

Regards
Thomas


signature.asc
Description: PGP signature

Re: authenticate as userA, but get authorization to user userB's account

[Thomas Leuxner]

* Heiko Schlittermann  2017.10.25 12:58:

> Question: Is there any way to split the authentication from the
> authorization within common mail clients (as Thunderbird) in combination
> with Dovecot. That is, doing something like logging in to the
> account sa...@example.com, using the credentials of the very own account
> (say h...@example.com)?

Hi,

wouldn't this be a use case for acl_groups, where a user would belong to group 
"Sales" and this "role" would gain specific access?

Regards
Thomas


signature.asc
Description: PGP signature

Re: maildir boxes directory mode upon creation

[Thomas Leuxner]

* vadim  2017.08.23 16:04:

> I am unable to enforce dovecot to create mailboxes with 660 permissions.
> Output of dovecot -n is in the attachment.
> 
> Please tell me what's the right way to control mailbox permissions ?

Hi Vadmin,

inject the mails per LMTP rather than having Postfix save them directly and let 
Dovecot worry about the permissions:

https://wiki2.dovecot.org/HowTo/PostfixDovecotLMTP

Regards
Thomas


signature.asc
Description: PGP signature

Re: Latest Virtual Plugin changes seem to have broken flag changes: 2.2.devel (bf2fa36)

[Thomas Leuxner]

* Thomas Leuxner  2017.07.25 08:27:

https://github.com/dovecot/core/commit/f39bde1e1fb00905b71a4f2a793faea4b915e27b

This finally fixed the issue, I guess along with the other changes that have 
been previously submitted. Thanks!


signature.asc
Description: PGP signature

Re: failed to store into mailbox 'INBOX/Junk': Permission denied

[Thomas Leuxner]

* Davide Marchi  2017.07.26 10:25:

> cat /etc/dovecot/dovecot-acl
> * user=book...@hotelsangiorgioriccione.com lrwsi
> * user=i...@hotelsangiorgioriccione.com lrwsi

Hi Davide,

For LMTP to file the mails you need to add the p flag (POST).

Regards
Thomas


signature.asc
Description: PGP signature

Re: Latest Virtual Plugin changes seem to have broken flag changes: 2.2.devel (bf2fa36)

[Thomas Leuxner]

* Thomas Leuxner  2017.07.24 14:41:

> https://github.com/dovecot/core/commit/88b2ef52690acf66a925e2470331a9cc7c4cd320

It looks like it only partially fixes the problem. If messages get read in a 
virtual mailbox they sync to the "original" mailbox. Changes to the "original" 
mailbox only seem to sync when new mail arrives to this mailbox where the flags 
then get updated in the virtual mailbox.

Thomas


signature.asc
Description: PGP signature

Re: Latest Virtual Plugin changes seem to have broken flag changes: 2.2.devel (bf2fa36)

[Thomas Leuxner]

* Aki Tuomi  2017.07.21 13:45:

Hi Aki and Timo,

I'm still monitoring the flag sync. Appears 
https://github.com/dovecot/core/commit/88b2ef52690acf66a925e2470331a9cc7c4cd320
has done the trick.

Will confirm once I have more evidence.

Regards
Thomas


signature.asc
Description: PGP signature

Re: Latest Virtual Plugin changes seem to have broken flag changes: 2.2.devel (bf2fa36)

[Thomas Leuxner]

* Aki Tuomi  2017.07.21 12:53:

> I cannot repeat this.

Hi Aki,

a bit hard for me to debug speaking protocol, but I can definitely reproduce it 
with multiple MUAs.

Regards
Thomas


signature.asc
Description: PGP signature

Latest Virtual Plugin changes seem to have broken flag changes: 2.2.devel (bf2fa36)

[Thomas Leuxner]

Hi,

I’m using several views like this:

# cat Dovecot/dovecot-virtual
:public/Archive/Mailing-Lists/Dovecot/*
:public/Mailing-Lists/Dovecot
 all

Before the recent changes read mail stayed in sync with the virtual folder. Now 
regardless whether mail is read in the original Folder or the virtual Folder, 
the flags are not synchronized anymore. Read mail is flagged unread in the 
corresponding folder.

Regards
Thomas



signature.asc
Description: Message signed with OpenPGP

Re: 2.2.devel (0bee280) crashdump virtual plugin

[Thomas Leuxner]

* Aki Tuomi  2017.07.20 05:40:

> Should be fixed with
> 
> commit 81e832796cdc6af790ed7be8a6c150889f03171c
> Author: Timo Sirainen 
> Date:   Wed Jul 19 23:19:12 2017 +0300
> 
> virtual: Optimize mailbox_notify_changes() when there's only a single 
> backend mailbox
> 
> commit 2044eb7652b864a05842933e9097c583cb11256c
> Author: Timo Sirainen 
> Date:   Wed Jul 19 23:11:12 2017 +0300
> 
> lib-storage: mailbox_watch_extract_notify_fd() - give better reason if 
> mailbox has no IOs
> 
> This happens currently with virtual mailboxes.
> 
> commit a6280be05b9c90579bb59ff57a3035661706c3d3
> Author: Timo Sirainen 
> Date:   Wed Jul 19 23:09:13 2017 +0300
> 
> lib: io_loop_extract_notify_fd() - Don't crash if no notifys have been 
> added

After some testing with 2.2.devel (bf2fa36) I wasn't able to reproduce the 
segfaults anymore. Thanks!


signature.asc
Description: PGP signature

Re: 2.2.devel (0bee280) crashdump virtual plugin

[Thomas Leuxner]

* Aki Tuomi  2017.07.20 05:40:

> Should be fixed with
> 
> commit 81e832796cdc6af790ed7be8a6c150889f03171c
> Author: Timo Sirainen 
> Date:   Wed Jul 19 23:19:12 2017 +0300
> 
> virtual: Optimize mailbox_notify_changes() when there's only a single 
> backend mailbox
> 
> commit 2044eb7652b864a05842933e9097c583cb11256c
> Author: Timo Sirainen 
> Date:   Wed Jul 19 23:11:12 2017 +0300
> 
> lib-storage: mailbox_watch_extract_notify_fd() - give better reason if 
> mailbox has no IOs
> 
> This happens currently with virtual mailboxes.
> 
> commit a6280be05b9c90579bb59ff57a3035661706c3d3
> Author: Timo Sirainen 
> Date:   Wed Jul 19 23:09:13 2017 +0300
> 
> lib: io_loop_extract_notify_fd() - Don't crash if no notifys have been 
> added

Thanks I'll wait for them to show in 2.2 branch then.

Regards
Thomas


signature.asc
Description: PGP signature

Re: 2.2.devel (0bee280) crashdump virtual plugin

[Thomas Leuxner]

* Thomas Leuxner  2017.07.19 14:29:

It seems to have a new side effect which I haven't noticed upfront as it does 
not produce logging in Dovecot:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  io_loop_extract_notify_fd (ioloop=0x563456973540) at 
ioloop-notify-inotify.c:224
224 ioloop-notify-inotify.c: No such file or directory.
(gdb) bt full
#0  io_loop_extract_notify_fd (ioloop=0x563456973540) at 
ioloop-notify-inotify.c:224
ctx = 0x0
io = 
fd = 
new_inotify_fd = 
#1  0x7f30bc2b740d in mailbox_watch_extract_notify_fd (box=0x563456973fc8, 
reason_r=reason_r@entry=0x7ffea4ebcfe8) at mailbox-watch.c:144
ioloop = 0x563456973540
file = 0x0
io = 0x91e89a45d153b800
iop = 
temp_ios = {arr = {buffer = 0x5634566690e8, element_size = 8}, v = 
0x5634566690e8, v_modifiable = 0x5634566690e8}
ret = 
failed = false
#2  0x563456150418 in imap_client_hibernate 
(_client=_client@entry=0x7ffea4ebd0d0) at imap-client-hibernate.c:221
client = 0x56345668be48
state = 0x563456a4bf20
error = 0x7f30bbfcbb02  "H\211\005\027\353("
ret = 1
fd_notify = -1
fd_hibernate = -1
__FUNCTION__ = "imap_client_hibernate"
#3  0x563456146984 in idle_hibernate_timeout (ctx=0x5634566b6170) at 
cmd-idle.c:191
client = 0x56345668be48
#4  0x7f30bbfe09f2 in io_loop_handle_timeouts_real (ioloop=0x5634566719b0) 
at ioloop.c:567
timeout = 0x56345697e090
item = 0x56345697e090
tv = {tv_sec = 0, tv_usec = 0}
tv_call = {tv_sec = 1500487741, tv_usec = 857399}
t_id = 3
#5  io_loop_handle_timeouts (ioloop=ioloop@entry=0x5634566719b0) at ioloop.c:580
_data_stack_cur_id = 2
#6  0x7f30bbfe2153 in io_loop_handler_run_internal 
(ioloop=ioloop@entry=0x5634566719b0) at ioloop-epoll.c:196
ctx = 0x5634566734d0
io = 
tv = {tv_sec = 0, tv_usec = 32654}
events_count = 5
msecs = 
ret = 0
i = 
j = 
call = 
__FUNCTION__ = "io_loop_handler_run_internal"
#7  0x7f30bbfe0bdc in io_loop_handler_run 
(ioloop=ioloop@entry=0x5634566719b0) at ioloop.c:648
No locals.
#8  0x7f30bbfe0d88 in io_loop_run (ioloop=0x5634566719b0) at ioloop.c:623
__FUNCTION__ = "io_loop_run"
#9  0x7f30bbf666d3 in master_service_run (service=0x563456671850, 
callback=) at master-service.c:666
No locals.
#10 0x5634561421d3 in main (argc=1, argv=0x5634566715e0) at main.c:491
set_roots = {0x56345636d6c0 , 0x7f30bc7b0540 
, 0x0}
login_set = {auth_socket_path = 0x563456669048 "\001", 
postlogin_socket_path = 0x0, postlogin_timeout_secs = 60, callback = 
0x56345615d2b0 , failure_callback = 0x56345615cbf0 
, 
  request_auth_token = 1}
service_flags = 
storage_service_flags = 
username = 0x0
auth_socket_path = 0x563456166074 "auth-master"
c = 


signature.asc
Description: PGP signature

Re: 2.2.devel (0bee280) crashdump virtual plugin

[Thomas Leuxner]

* Aki Tuomi  2017.07.19 11:43:

> Should be fixed with
> 
> commit 2f691d6dd1b98f605043744846534b828a43994d
> Author: Aki Tuomi 
> Date:   Wed Jul 19 12:03:58 2017 +0300
> 
> virtual: Stop watching notification on backend box close
>
> Fixes Panic: file mail-index.c: line 662
> (mail_index_close_nonopened): assertion failed: (index->views == NULL)
> 
> commit 0d318fe07ea8846330b6fe6b6b281ef0ccfbaacd
> Author: Aki Tuomi 
> Date:   Tue Jul 18 21:04:47 2017 +0300
> 
> virtual: Stop notifications only if they were started
>
> Fixes Panic: file mail-storage.c: line 1950: assertion failed:
> (box->opened)

Thanks Aki. Looks good now.


signature.asc
Description: PGP signature

Re: 2.2.devel (0bee280) crashdump virtual plugin

[Thomas Leuxner]

* Aki Tuomi  2017.07.19 08:31:

> oh and any idea which one of these crashes?

This one does:

# cat Flagged/dovecot-virtual
*
:public/*
-Trash
-Trash/*
  flagged


signature.asc
Description: PGP signature

Re: 2.2.devel (0bee280) crashdump virtual plugin

[Thomas Leuxner]

* Aki Tuomi  2017.07.19 08:22:

> Can you provide your virtual config?

Here you go:

# cat Dovecot/dovecot-virtual
:public/Archive/Mailing-Lists/Dovecot/*
:public/Mailing-Lists/Dovecot
  all

# cat Flagged/dovecot-virtual
*
:public/*
-Trash
-Trash/*
  flagged

# cat Postfix-Users/dovecot-virtual
:public/Archive/Mailing-Lists/Postfix-Users/*
:public/Mailing-Lists/Postfix-Users
  all


signature.asc
Description: PGP signature

Re: 2.2.devel (0bee280) crashdump virtual plugin

[Thomas Leuxner]

* Aki Tuomi  2017.07.18 20:01:

> Hi!
> 
> Thank you for your report. Do you have mailbox_list_index=yes?
> 
> Aki

Hi Aki,

yes I do:

doveconf mailbox_list_index
mailbox_list_index = yes

Regards
Thomas


signature.asc
Description: PGP signature

2.2.devel (0bee280) crashdump virtual plugin

[Thomas Leuxner]

#0  0x7f497ecdb067 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 10092
selftid = 10092
#1  0x7f497ecdc448 in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0xfffe, sa_sigaction = 
0xfffe}, sa_mask = {__val = {1024, 14072248576, 94791001010256, 513, 
139953641023651, 14072248368, 94791001010256, 513,
  11573297580199803136, 13995364478, 139953641204307, 
94791001010256, 14072248464, 0, 139953641204665, 94791001010256}}, sa_flags 
= 2131669786, sa_restorer = 0x7ffc829fd301}
sigs = {__val = {32, 0 }}
#2  0x7f497f0f05b6 in default_fatal_finish (type=, 
status=status@entry=0) at failures.c:201
backtrace = 0x56363ff17088 "/usr/lib/dovecot/libdovecot.so.0(+0x9f5be) 
[0x7f497f0f05be] -> /usr/lib/dovecot/libdovecot.so.0(+0x9f6ac) [0x7f497f0f06ac] 
-> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f497f080fde] -> 
/usr/lib/d"...
#3  0x7f497f0f06ac in i_internal_fatal_handler (ctx=0x7ffc829fd350, 
format=, args=) at failures.c:670
status = 0
#4  0x7f497f080fde in i_panic (format=format@entry=0x7f497f47af00 "file %s: 
line %d (%s): assertion failed: (%s)") at failures.c:275
ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, 
timestamp_usecs = 0}
args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 
0x7ffc829fd450, reg_save_area = 0x7ffc829fd390}}
#5  0x7f497f3cb6dc in mailbox_notify_changes_stop (box=) at 
mail-storage.c:1950
__FUNCTION__ = "mailbox_notify_changes_stop"
#6  0x7f497dc5aa6f in virtual_notify_changes (box=0x5636400cdbd8) at 
virtual-storage.c:696
bboxp_end = 0x563640015708
mbox = 0x5636400cdbd8
bboxp = 0x563640015180
#7  0x56363f5e2652 in idle_finish (ctx=ctx@entry=0x56363ff64160, 
done_ok=, free_cmd=free_cmd@entry=true) at cmd-idle.c:46
client = 0x56363ff39e68
#8  0x56363f5e2783 in idle_client_handle_input 
(ctx=ctx@entry=0x56363ff64160, free_cmd=free_cmd@entry=true) at cmd-idle.c:67
line = 
#9  0x56363f5e284d in idle_client_input_more (ctx=0x56363ff64160) at 
cmd-idle.c:100
client = 0x56363ff39e68
#10 idle_client_input (ctx=0x56363ff64160) at cmd-idle.c:107
client = 0x56363ff39e68
#11 0x7f497f105b22 in io_loop_call_io (io=0x56363ffae400) at ioloop.c:599
ioloop = 0x56363ff1f9b0
t_id = 2
__FUNCTION__ = "io_loop_call_io"
#12 0x7f497f10717a in io_loop_handler_run_internal 
(ioloop=ioloop@entry=0x56363ff1f9b0) at ioloop-epoll.c:223
ctx = 0x56363ff214d0
io = 
tv = {tv_sec = 3, tv_usec = 496715}
events_count = 
msecs = 
ret = 1
i = 0
j = 
call = 
__FUNCTION__ = "io_loop_handler_run_internal"
#13 0x7f497f105bbc in io_loop_handler_run 
(ioloop=ioloop@entry=0x56363ff1f9b0) at ioloop.c:648
No locals.
#14 0x7f497f105d68 in io_loop_run (ioloop=0x56363ff1f9b0) at ioloop.c:623
__FUNCTION__ = "io_loop_run"
#15 0x7f497f08b6d3 in master_service_run (service=0x56363ff1f850, 
callback=) at master-service.c:666
No locals.
#16 0x56363f5de1d3 in main (argc=1, argv=0x56363ff1f5e0) at main.c:491
set_roots = {0x56363f8096c0 , 0x7f497f8d5540 
, 0x0}
login_set = {auth_socket_path = 0x56363ff17048 "\001", 
postlogin_socket_path = 0x0, postlogin_timeout_secs = 60, callback = 
0x56363f5f92b0 ,
  failure_callback = 0x56363f5f8bf0 , 
request_auth_token = 1}
service_flags = 
storage_service_flags = 
username = 0x0
auth_socket_path = 0x56363f602074 "auth-master"
c = 



signature.asc
Description: PGP signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Stephan Bosch  2017.02.26 19:49:

> I created the following file in ~/message-lmtp.eml (substitute MAIL
> FROM/RCPT TO with whatever you need):
> 
> LHLO frop
> MAIL FROM:
> RCPT TO:
> DATA
> From: User 
> To: step...@rename-it.nl
> Subject: Frop!
> List-Id: 
> 
> Frop!
> .

I tested a couple of messages via the socket and it worked, _even_ with a 
message that previously has been delivered without flags but should have - 
which is odd to say at least... Still can't find a scheme to narrow it down 
except it only happens to rules on the included script.

Regards
Thomas


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Stephan Bosch  2017.02.24 10:20:

> Could you show me your full configuration (`dovecot -n`)?
> 
> Regards,
> 
> Stephan

Live configuration and scripts sent off-list.

Regards
Thomas


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Thomas Leuxner  2017.02.20 12:56:

> Feb 20 07:00:23 nihlus dovecot: master: Dovecot v2.2.devel (8f42a89) starting 
> up for imap, lmtp
> 
> This one processed the dovecot-news mail for 2.2.28.rc1 fine which uses a 
> similar sieve rule. I will monitor global rules with this build and report 
> back.

The results seem to be arbitrary. Personal rules _always_ work for flag 
actions, global rules included do _sometimes_. This means one day the same rule 
may trigger, the next day it won't. I found no way to reproduce it for global 
rules in order to narrow down the issue.

Regards
Thomas


signature.asc
Description: Digital signature

Re: How to dsync mdbox compressed to maildir uncompressed

[Thomas Leuxner]

* Daniel Betz  2017.02.22 11:07:

Here you go:

# 2.2.devel (d89a40c): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.devel (0d78a30)
# OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.7 
auth_cache_size = 16 k
auth_verbose = yes
deliver_log_format = msgid=%m, time=%{delivery_time}ms, status=%$
hostname = host.domain.tld
imap_hibernate_timeout = 1 mins
imap_id_log = *
imap_logout_format = in=%i out=%o hdr=%{fetch_hdr_count} 
body=%{fetch_body_count} del=%{deleted} exp=%{expunged} trash=%{trashed}
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %k 
session=<%{session}>
mail_location = mdbox:~/mdbox
mail_plugins = acl quota stats zlib virtual
mailbox_list_index = yes
mdbox_rotate_size = 10 M
namespace {
  list = yes
  location = mdbox:/var/vmail/public:INDEXPVT=~/mdbox/public
  prefix = :public/
  separator = /
  subscriptions = no
  type = public
}
namespace {
  list = children
  location = mdbox:/var/vmail/domains/%%d/%%n/mdbox:INDEXPVT=~/mdbox/shared/%%u
  prefix = :shared/%%u/
  separator = /
  subscriptions = no
  type = shared
}
namespace {
  location = virtual:~/mdbox/virtual
  prefix = :virtual/
  separator = /
}
namespace inbox {
  hidden = no
  inbox = yes
  location = 
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix = 
  separator = /
  type = private
}
passdb {
  args = username_format=%u /var/vmail/auth.d/%d/passwd
  driver = passwd-file
}
plugin {
  acl = vfile:/var/vmail/conf.d/%d/global-acl:cache_secs=300
  acl_shared_dict = file:/var/lib/dovecot/db/shared-mailboxes
  imapsieve_mailbox1_before = 
file:/var/vmail/conf.d/domain.tld/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = Spam
  imapsieve_mailbox2_before = 
file:/var/vmail/conf.d/domain.tld/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_name = *
  mail_log_events = expunge mailbox_delete
  quota = count:User quota
  quota_grace = 10%%
  quota_rule = *:storage=1GB
  quota_rule2 = Trash:storage=+10%%
  quota_status_nouser = DUNNO
  quota_status_success = DUNNO
  quota_vsizes = yes
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_global_dir = /var/vmail/conf.d/%d/sieve
  sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.execute
  sieve_pipe_bin_dir = /var/vmail/conf.d/domain.tld/sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
  stats_refresh = 30s
  stats_track_cmds = yes
  zlib_save = gz
  zlib_save_level = 6
}
protocols = " imap lmtp"
quota_full_tempfail = yes
service auth-worker {
  unix_listener auth-worker {
user = doveauth
  }
  user = doveauth
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0660
user = postfix
  }
  user = doveauth
}
service imap-hibernate {
  unix_listener imap-hibernate {
user = vmail
  }
}
service imap-login {
  inet_listener imap {
address = 188.138.4.217 [2001:470:1f0b:bd0::3]
port = 143
reuse_port = yes
  }
  inet_listener imaps {
port = 0
  }
  process_min_avail = 8
}
service imap-postlogin {
  executable = script-login /var/vmail/conf.d/scripts/postlogin.sh
  user = vmail
}
service imap {
  executable = imap
  unix_listener imap-master {
user = dovecot
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0660
user = postfix
  }
}
service quota-status {
  client_limit = 1
  executable = quota-status -p postfix
  unix_listener /var/spool/postfix/private/quota-status {
group = postfix
mode = 0660
user = postfix
  }
}
service stats {
  fifo_listener stats-mail {
mode = 0600
user = vmail
  }
}
ssl_ca = 

signature.asc
Description: Digital signature

Re: How to dsync mdbox compressed to maildir uncompressed

[Thomas Leuxner]

* Daniel Betz  2017.02.22 09:37:

> doveadm -o plugin/quota= -o plugin/zlib_save= backup -u imap at test.de 
> maildir:~/Maildir
> 
> file Maildir/cur/1487752497.M128284P11684.mail,S=7691,W=7887:2,S
> Maildir/cur/1487752497.M128284P11684.mail,S=7691,W=7887:2,S: gzip compressed 
> data, from Unix

Hi Daniel,

I _did_ test this with a mdbox to maildir conversion and could read the maildir 
files as clear text.

Regards
Thomas


signature.asc
Description: Digital signature

Re: How to dsync mdbox compressed to maildir uncompressed

[Thomas Leuxner]

* Daniel Betz  2017.02.21 11:49:

> Have tried this: doveadm -o "maildir_copy_with_hardlinks=no" sync -u 
> i...@test.de maildir:~/Maildir
> From Wiki: If you want to use dsync to convert to a compressed Maildir you 
> may need -o maildir_copy_with_hardlinks=no (this is set to yes by default and 
> will prevent compression).

doveadm -o plugin/quota= -o plugin/zlib_save= backup -u i...@test.de 
maildir:~/Maildir

Regards
Thomas


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Stephan Bosch  2017.02.17 14:34:

> Couldn't reproduce this with v2.3.devel yesterday (i.e. no flags set for the
> Security mailbox and all flags set for the message in INBOX), but I will try
> later with some older version.

I did install a newer build this morning:

Feb 20 07:00:23 nihlus dovecot: master: Dovecot v2.2.devel (8f42a89) starting 
up for imap, lmtp

This one processed the dovecot-news mail for 2.2.28.rc1 fine which uses a 
similar sieve rule. I will monitor global rules with this build and report back.

Regards
Thomas


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Stephan Bosch  2017.01.31 21:31:

> This slipped my attention for the moment. Will look at this soon...

Thanks.


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Stephan Bosch  2017.01.19 10:32:

> Could you provide a more detailed example?

Sure. Personal script v

/var/vmail/domains/leuxner.net/tlx/.dovecot.sieve:

require ["include","copy","fileinto","imap4flags","vacation"];
include :global "global";

--

Global script referenced v

/var/vmail/conf.d/leuxner.net/sieve/global.sieve:

require ["fileinto","imap4flags","duplicate"];

#Newsletters

if header :contains "List-Id" "debian-security-announce.lists.debian.org"
{
removeflag "\\Flagged $MailFlagBit1";
fileinto ":public/Newsletters/Debian/Security";
addflag "\\Flagged $MailFlagBit1";
keep;
}

--
Basically it is reproducible with the same stanza we used before by putting 
this in the included script:

#Test
if address :is "From" "u...@example.com"
   {
   removeflag "\\Flagged $MailFlagBit1";
   fileinto "Trash";
   addflag "\\Flagged $MailFlagBit1";
   keep;
   }

Regards
Thomas


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Stephan Bosch  2017.01.14 12:35:

> Fixed:
> 
> https://github.com/dovecot/pigeonhole/commit/772485538302957ebada484b6eedec57136bc737

Hi Stephan,

I recently noticed that the 'flags' still cause issues when using an _include_ 
script like:

require ["include","copy","fileinto","imap4flags","vacation"];
include :global "global";

In this case the file options work fine, but the flags are not set.

Regards
Thomas


signature.asc
Description: Digital signature

Re: Dynamic autoexpunge for IMAP folders

[Thomas Leuxner]

* Thorsten Hater  2017.01.17 12:04:

> userdb {
>driver = static
>args   = namespace/inbox/mailbox/Bla/autoexpunge=6h
> }

I'm almost sure that 'driver = static' contradicts the concept of userdb 
overrides:

http://wiki.dovecot.org/UserDatabase/ExtraFields

If you can't go with 'passwd-file' style settings, you may able to achieve this 
globaly with:

protocol imap {
  namespace inbox {
location = 
mailbox Blah {
  autoexpunge = 6h
}
prefix = 
  }
}

In any case this is then a global setting rather than one being returned on a 
per-user basis.


signature.asc
Description: Digital signature

Re: Dynamic autoexpunge for IMAP folders

[Thomas Leuxner]

* Thorsten Hater  2017.01.17 10:57:

> Unfortunately, this yields the same problem
> 
> Debug: Unknown userdb
> setting: plugin/userdb_namespace/inbox/mailbox/Bla/autoexpunge=6h

> > userdb_namespace/inbox/mailbox/Bla/autoexpunge=6h

'plugin' is not part of the userdb override. It is literally as above.


signature.asc
Description: Digital signature

Re: Dynamic autoexpunge for IMAP folders

[Thomas Leuxner]

* Thorsten Hater  2017.01.17 09:40:

> namespace/inbox/mailbox/Bla/autoexpunge=6h
> 
> from a userdb, which results in
> 
> Debug: Unknown userdb setting:
> plugin/namespace/inbox/mailbox/Bla/autoexpunge=6h

userdb_namespace/inbox/mailbox/Bla/autoexpunge=6h

Regards
Thomas


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Stephan Bosch  2017.01.14 12:35:

> Fixed:
> 
> https://github.com/dovecot/pigeonhole/commit/772485538302957ebada484b6eedec57136bc737
> 
> Regards,
> 
> Stephan.

Confirmed effective. Thanks!


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Stephan Bosch  2017.01.13 13:50:

> So, that performs as expected. However, when I test this with LDA, Í can
> reproduce your problem:
> 
> $ doveadm fetch flags mailbox "Trash" 1
> flags: \Recent
> $ doveadm fetch flags mailbox "INBOX" 1
> flags: \Recent
> 
> Will investigate more later today...

Thanks for confirming Stephan.


signature.asc
Description: Digital signature

Re: Sieve removeflag Action

[Thomas Leuxner]

* Steffen Kaiser  2017.01.13 10:49:

> from point of "logic" I would turn around both actions:
> 
> removeflag "\\Flagged $MailFlagBit1";
> fileinto "Trash";
> 
> addflag "\\Flagged $MailFlagBit1";
> keep;

Hi Steffen,

that works thanks. Not sure why it worked before.

Regards
Thomas


signature.asc
Description: Digital signature

Sieve removeflag Action

[Thomas Leuxner]

Hi,

I recently noticed that some of my automatically processed mails do no longer 
show flags with current Dovecot builds. This used to work before, not sure what 
broke it or whether I used it in the wrong way. When the removeflag line is 
dropped *both* messages get flagged in the client, one in INBOX and one in 
Trash. With removeflag in place, *both* messages are unflagged, the one in 
INBOX and the one in Trash. This used to work differently before if memory 
doesn't fail me. 

#Test
if address :is "From" "u...@example.com"
{
addflag "\\Flagged $MailFlagBit1";
keep;
removeflag "\\Flagged $MailFlagBit1";
fileinto "Trash";
}

# 2.2.devel (01867a4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.devel (3c071a4)
# OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.6

Regards
Thomas


signature.asc
Description: Digital signature

Latest Dovecot v2.2.devel (9bc8d9b) crashes on login and lmtp

[Thomas Leuxner]

I was unable to produce an IMAP core but have been successful with LMTP. 
Basically it crashes right upon login or delivery:

Dec 20 08:55:23 nihlus dovecot: master: Dovecot v2.2.devel (9bc8d9b) starting 
up for imap, lmtp
Dec 20 08:56:15 nihlus dovecot: imap(t...@leuxner.net): Fatal: master: 
service(imap): child 5710 killed with signal 11 (core not dumped)
Dec 20 08:57:23 nihlus dovecot: lmtp(5967): Connect from local
Dec 20 08:57:23 nihlus dovecot: lmtp(5967): Fatal: master: service(lmtp): child 
5967 killed with signal 11 (core dumped)

$ gdb /usr/lib/dovecot/lmtp core_lmtp.5967.nihlus.leuxner.net
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/dovecot/lmtp...Reading symbols from 
/usr/lib/debug/.build-id/ec/199069dd786feb54e5af43f983ce9c3bb4c29c.debug...done.
done.
[New LWP 5967]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `dovecot/lmtp'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memcpy_sse2_unaligned () at 
../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:33
33  ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or 
directory.
#0  __memcpy_sse2_unaligned () at 
../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:33
No locals.
#1  0x7f9177db4779 in memcpy (__len=, __src=0x0, 
__dest=0x7f917a5c5bf8) at /usr/include/x86_64-linux-gnu/bits/string3.h:51
No locals.
#2  p_strconcat (pool=, str1=) at strfuncs.c:218
args = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 
0x7ffdcadea060, reg_save_area = 0x7ffdcade9f90}}
temp = 0x0
ret = 
len = 49
#3  0x7f91780744fd in mdbox_storage_create (_storage=0x7f917a5c5930, 
ns=0x7f917a5b79d0, error_r=0x7ffdcadea1a0) at mdbox-storage.c:57
storage = 0x7f917a5c5930
dir = 
#4  0x7f917805952b in mail_storage_create_full (ns=ns@entry=0x7f917a5b79d0, 
driver=, driver@entry=0x0, data=, 
flags=flags@entry=(unknown: 0),
storage_r=storage_r@entry=0x7ffdcadea170, 
error_r=error_r@entry=0x7ffdcadea1a0) at mail-storage.c:395
storage = 0x7f917a5c5930
list = 0x7f917a5c3ca0
list_set = {layout = 0x7f91781083c6 "fs", root_dir = 0x7f917a52e100 
"/var/vmail/domains/leuxner.net/tlx/mdbox", index_dir = 0x0, index_pvt_dir = 
0x0, control_dir = 0x0, alt_dir = 0x0,
  inbox_path = 0x0, subscription_fname = 0x7f917810f234 
"subscriptions", list_index_fname = 0x7f917810925e "dovecot.list.index", 
maildir_name = 0x7f917810bc17 "dbox-Mails",
  mailbox_dir_name = 0x7f917810c301 "mailboxes", escape_char = 0 
'\000', broken_char = 0 '\000', utf8 = false, alt_dir_nocheck = false, 
index_control_use_maildir_name = false}
list_flags = 
p = 
__FUNCTION__ = "mail_storage_create_full"
#5  0x7f91780598cc in mail_storage_create (ns=ns@entry=0x7f917a5b79d0, 
driver=driver@entry=0x0, flags=flags@entry=(unknown: 0), 
error_r=error_r@entry=0x7ffdcadea1a0) at mail-storage.c:416
storage = 0x7f917a5bfc90
#6  0x7f9178051b89 in mail_namespaces_init_add 
(user=user@entry=0x7f917a5bfc90, ns_set=, 
unexpanded_ns_set=0x7f917a5c0178, ns_p=ns_p@entry=0x7ffdcadea200,
error_r=error_r@entry=0x7ffdcadea290) at mail-namespace.c:195
mail_set = 
ns = 0x7f917a5b79d0
driver = 0x0
error = 0x0
ret = 
#7  0x7f9178052cd9 in mail_namespaces_init (user=0x7f917a5bfc90, 
error_r=error_r@entry=0x7ffdcadea290) at mail-namespace.c:414
mail_set = 0x7f917a5c0c58
ns_set = 0x7f917a5c0f70
unexpanded_ns_set = 0x7f917a5c0150
namespaces = 0x0
ns_p = 0x7ffdcadea200
i = 
count = 
count2 = 
__FUNCTION__ = "mail_namespaces_init"
#8  0x7f9178060b65 in mail_storage_service_init_post (ctx=, 
error_r=0x7ffdcadea290, mail_user_r=, priv=0x7ffdcadea2a0, 
user=0x7f917a573910) at mail-storage-service.c:728
mail_set = 0x7f917a5c0c58
home = 0x7f917a59e761 "/var/vmail/domains/leuxner.net/tlx"
mail_user = 0x7f917a5bfc90
#9  mail_storage_service_next_real (mail_user_r=, 
user=0x7f917a573910, ctx=) at mail-storage-service.c:1427
len = 
temp_priv_drop = 
priv = {uid = 5000, gid = 5000, uid_source = 0x7f9178107684 "userdb

Re: autoexpunge=30d does not works :(

[Thomas Leuxner]

* Peter Hodur  2016.12.16 11:42:

> mailbox Spam {
> auto= subscribe
> special_use = \Junk
> autoexpunge = 30d

Hi Peter,

may be easier to use per-user settings for testing. Something along the line:

userdb_namespace/inbox/mailbox/Spam/autoexpunge=6h

> As I hope, when user logout (POP3 or IMAP) or after Postfix delivers mail
> via LMTP, Dovecot should check Spam folder for emails, which have ctime
> older than 30 days and deletes this emails.

AFAIK it uses indexes rather than FS attributes, let's see if this query yields 
some candidates (adjust time):

$ doveadm search -u peter  mailbox Spam savedsince 6h

Regards
Thomas


signature.asc
Description: Digital signature

Re: Latest HG Changes (fac92b5) affect Sieve-Plugin/LMTP

[Thomas Leuxner]

> This should fix it:
> 
> https://github.com/dovecot/pigeonhole/commit/63f9b42f20cf0bd26b981be8a08f01b48e23517f

Confirmed fixed. Can you please push to 2.2 so builds pick up there?

Thanks
Thomas

Re: Latest git Changes (fac92b5) affect Sieve-Plugin/LMTP

[Thomas Leuxner]

# doveconf -d | grep discard
# doveconf -a | grep discard
  sieve_discard = ~/.dovecot.sieve

When set the crash disappears.

Re: Latest HG Changes (fac92b5) affect Sieve-Plugin/LMTP

[Thomas Leuxner]

> I hope you mean git since hg is no longer maintained. 
> 
> Aki

Apologies. Latest and greatest in GIT.

Re: Latest HG Changes (fac92b5) affect Sieve-Plugin/LMTP

[Thomas Leuxner]

> Oct 15 09:50:15 nihlus dovecot: lmtp(t...@leuxner.net): Fatal: master: 
> service(lmtp): child 11447 killed with signal 6 (core not dumped)

#0  0x7fdc0b5d7067 in __GI_raise (sig=sig@entry=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 22091
selftid = 22091
#1  0x7fdc0b5d8448 in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x3ed, sa_sigaction = 
0x3ed}, sa_mask = {__val = {520, 140736754611104, 140583104653368, 513, 
140583064431555, 140583029016568, 140583104653368, 513, 140583064421862, 
140736754611344,
  140583064603266, 140583104653368, 140736754611232, 0, 
140583064603369, 140583104653368}}, sa_flags = 194885514, sa_restorer = 
0x7fffd443f401}
sigs = {__val = {32, 0 }}
#2  0x7fdc0b9e08a6 in default_fatal_finish (type=, 
status=status@entry=0) at failures.c:201
backtrace = 0x7fdc0e03a870 "/usr/lib/dovecot/libdovecot.so.0(+0x938ae) 
[0x7fdc0b9e08ae] -> /usr/lib/dovecot/libdovecot.so.0(+0x9399c) [0x7fdc0b9e099c] 
-> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fdc0b97a6de] -> 
/usr/lib/d"...
#3  0x7fdc0b9e099c in i_internal_fatal_handler (ctx=0x7fffd443f470, 
format=, args=) at failures.c:670
status = 0
#4  0x7fdc0b97a6de in i_panic (format=format@entry=0x7fdc098187f8 "file %s: 
line %d (%s): assertion failed: (%s)") at failures.c:275
ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, 
timestamp_usecs = 0}
args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 
0x7fffd443f570, reg_save_area = 0x7fffd443f4b0}}
#5  0x7fdc09817af8 in lda_sieve_execute_scripts (srctx=0x7fffd443f690) at 
lda-sieve-plugin.c:447
sbin = 0x0
script = 0x0
cpflags = (unknown: 0)
exflags = (unknown: 0)
discard_script = 
i = 
ret = 
svinst = 0x7fdc0e133240
action_ehandler = 0x0
more = true
exec_ehandler = 
debug = false
user_script = 
mdctx = 0x7fffd443f8b0
mscript = 0x7fdc0e135d18
last_script = 0x0
compile_error = false
error = SIEVE_ERROR_NONE
#6  lda_sieve_execute (storage_r=0x7fffd443f888, srctx=0x7fffd443f690) at 
lda-sieve-plugin.c:821
msgdata = {mail = 0x7fdc0e0c64d0, return_path = 0x7fdc0e06e038 
"t...@leuxner.net", orig_envelope_to = 0x7fdc0e06e2c8 "t...@leuxner.net", 
final_envelope_to = 0x7fdc0e06e2c8 "t...@leuxner.net",
  auth_user = 0x7fdc0e0cb200 "t...@leuxner.net", id = 0x7fdc0e0c6e60 
""}
estatus = {last_storage = 0x0, message_saved = 0, message_forwarded = 
0, tried_default_save = 0, keep_original = 0, store_failed = 0}
trace_config = {level = SIEVE_TRLVL_NONE, flags = 0}
debug = 
ret = 
mdctx = 0x7fffd443f8b0
svinst = 
scriptenv = {user = 0x7fdc0e0cb100, default_mailbox = 0x7fdc0c3de258 
"INBOX", postmaster_address = 0x7fdc0e06cf38 "postmas...@leuxner.net", 
mailbox_autocreate = false, mailbox_autosubscribe = false,
  script_context = 0x7fffd443f8b0, smtp_start = 0x7fdc098168c0 
, smtp_add_rcpt = 0x7fdc098168b0 
, smtp_send = 0x7fdc098168a0 ,
  smtp_finish = 0x7fdc09816880 , duplicate_check 
= 0x7fdc09816840 , duplicate_mark = 0x7fdc09816860 
,
  duplicate_flush = 0x7fdc09816830 , 
reject_mail = 0x7fdc09816820 , exec_status = 
0x7fffd443f610, trace_log = 0x0, trace_config = {level = SIEVE_TRLVL_NONE, 
flags = 0}}
trace_log = 0x0
#7  lda_sieve_deliver_mail (mdctx=, storage_r=0x7fffd443f888) at 
lda-sieve-plugin.c:883
_data_stack_cur_id = 3
srctx = {svinst = 0x7fdc0e133240, mdctx = 0x7fffd443f8b0, home_dir = 
0x7fdc0e0cceb8 "/var/vmail/domains/leuxner.net/tlx", scripts = 0x7fdc0e03a750, 
script_count = 1, user_script = 0x7fdc0e128630, main_script = 0x7fdc0e128630,
  discard_script = 0x0, msgdata = 0x7fffd443f620, scriptenv = 
0x7fffd443f700, user_ehandler = 0x7fdc0e132d40, master_ehandler = 
0x7fdc0e11b590, action_ehandler = 0x0,
  userlog = 0x7fdc0e03a7d8 
"/var/vmail/domains/leuxner.net/tlx/.dovecot.sieve.log"}
debug = 
svenv = {hostname = 0x7fdc0e0aa700 "spectre.leuxner.net", domainname = 
0x0, base_dir = 0x7fdc0e0cc2b8 "/var/run/dovecot", username = 0x7fdc0e0cb200 
"t...@leuxner.net",
  home_dir = 0x7fdc0e0cceb8 "/var/vmail/domains/leuxner.net/tlx", 
temp_dir = 0x7fdc0e0cd170 "/tmp", flags = SIEVE_FLAG_HOME_RELATIVE, location = 
SIEVE_ENV_LOCATION_MDA, delivery_phase = SIEVE_DELIVERY_PHASE_DURING}
i = 
ret = 0
#8  0x7fdc0bfaab39 in mail_deliver (ctx=ctx@entry=0x7fffd443f8b0, 
storage_r=storage_r@entry=0x7fffd443f888) at mail-deliver.c:478
ret = 
#9  0x7fdc0c3db24e in client_deliver (session=0x7fdc0e0cacf0, 
src_mail=0x7fdc0e0c64d0, rcpt=0x7fdc0e06e288, client=0x7fdc0e06caf0) at 
commands.c:890
set_parser = 
line = 
str = 
mail_error = 235329232
   

Latest HG Changes (fac92b5) affect Sieve-Plugin/LMTP

[Thomas Leuxner]

# 2.2.devel (c73322f): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.devel (fac92b5)
# OS: Linux 3.16.0-4-amd64 x86_64 Debian 8.6 

==> /var/log/dovecot/dovecot.log <==
Oct 15 09:50:15 nihlus dovecot: lmtp(11447): Connect from local
Oct 15 09:50:15 nihlus dovecot: lmtp(t...@leuxner.net): Panic: file 
lda-sieve-plugin.c: line 447 (lda_sieve_execute_scripts): assertion failed: 
(script != NULL)
Oct 15 09:50:15 nihlus dovecot: lmtp(t...@leuxner.net): Error: Raw backtrace: 
/usr/lib/dovecot/libdovecot.so.0(+0x938ae) [0x7fd161fc18ae] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x9399c) [0x7fd161fc199c] -> 
/usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7fd161f5b6de] -> 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x3af8) [0x7fd15fdf8af8] -> 
/usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x49) [0x7fd16258bb39] -> 
dovecot/lmtp [DATA t...@leuxner.net](+0x724e) [0x7fd1629bc24e] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x4c) [0x7fd161fd5e4c] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10a) 
[0x7fd161fd730a] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) 
[0x7fd161fd5ed5] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7fd161fd6078] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7fd161f61be3] -> dovecot/lmtp [DATA t...@leuxner.net](main+0x1a2) 
[0x7fd1629ba382] -> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) 
[0x7fd161ba4b45] -> dovecot/lmtp [DATA t...@leuxner.net](+0x5430) 
[0x7fd1629ba430]
Oct 15 09:50:15 nihlus dovecot: lmtp(t...@leuxner.net): Fatal: master: 
service(lmtp): child 11447 killed with signal 6 (core not dumped)

[SOLVED] Error: istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed

[Thomas Leuxner]

* Thomas Leuxner  2016.03.02 09:54:

> * Timo Sirainen  2016.03.01 23:03:
> 
> > > I'm seeing these recently:
> > > 
> > > Feb 27 09:24:01 nihlus dovecot: imap(t...@leuxner.net): Error: 
> > > istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed: No such file 
> > > or directory
> > 
> > You don't have a /tmp directory or imap is chrooted somewhere where there 
> > is no /tmp?..

Finally got down to the issue. The messages occur if the systemd-*service 
temporary directory gets deleted inside /tmp

$l -d systemd*
drwx-- 3 root root 60 Sep  5 09:29 
systemd-private-37bbda70211940c593ef885fdfc1a1dc-dovecot.service-II7xjd

Case solved.


signature.asc
Description: Digital signature

Re: Save user passwords in clear text

[Thomas Leuxner]

* Lefteris Tsintjelis  2016.08.05 21:01:

> Is it possible to save user passwords as clear text through dovecot?

Have a look at how schemes can be converted using postlogin scripts:

http://wiki2.dovecot.org/HowTo/ConvertPasswordSchemes


signature.asc
Description: Digital signature

Re: Deliver same E-Mail in same Mailbox only once ?

[Thomas Leuxner]

* Heiko Schlittermann  2016.04.20 19:31:

> > * Christoph Adomeit  2016.04.20 10:29:
> > #Deduplicate mails sent to aliases for same mailbox using the msgid
> > if duplicate {
> >   discard;
> > }
> 
> I wouldn't do that. Message with the same message id doesn't need to be
> duplicates. See the difference between this message (I put you on BCC),
> you'll receive it via the mailing list AND directly. Depending on the ML
> configuration the messages are NOT identical.

It really depends on the OPs requirements. The task was to filter cc: mails. 
Anyway if that is your concern, no message was discarded using bcc:

Apr 20 19:31:13 nihlus dovecot: lmtp(27566): Connect from local
Apr 20 19:31:13 nihlus dovecot: lmtp(t...@leuxner.net): NbCKIuG8F1euawAAgUOSbA: 
sieve: msgid=<20160420173100.gr14...@jumper.schlittermann.de>, time=115ms, 
status=stored mail into mailbox 'INBOX'
Apr 20 19:31:13 nihlus dovecot: lmtp(27566): Disconnect from local: Successful 
quit
Apr 20 19:31:24 nihlus dovecot: lmtp(27566): Connect from local
Apr 20 19:31:25 nihlus dovecot: lmtp(t...@leuxner.net): OrCKIuG8F1euawAAgUOSbA: 
sieve: msgid=<20160420173100.gr14...@jumper.schlittermann.de>, time=117ms, 
status=stored mail into mailbox ':public/Mailing-Lists/Dovecot'
Apr 20 19:31:25 nihlus dovecot: lmtp(t...@leuxner.net): OrCKIuG8F1euawAAgUOSbA: 
sieve: msgid=<20160420173100.gr14...@jumper.schlittermann.de>, time=117ms, 
status=marked message to be discarded if not explicitly delivered (discard 
action)
Apr 20 19:31:25 nihlus dovecot: lmtp(27566): Disconnect from local: Successful 
quit

Regards
Thomas


signature.asc
Description: Digital signature

Re: Deliver same E-Mail in same Mailbox only once ?

[Thomas Leuxner]

* Christoph Adomeit  2016.04.20 10:29:

> Is there a way to deliver  the same Mail in the same Mailbox only once ?

Yes this case is supported via an sieve extension:

require ["duplicate"];


#Deduplicate mails sent to aliases for same mailbox using the msgid

if duplicate {
  discard;
}


signature.asc
Description: Digital signature

Re: mdbox and shared \Seen flag

[Thomas Leuxner]

* Dmitry Nezhevenko  2016.04.09 19:38:

> As far as I understand, "shared" namespace will keep \Seen flag private.
> So any suggestions how to make \Seen shared?

This can be done using INDEXPVT:

location = mdbox:/var/vmail/shared:INDEXPVT=~/mdbox/shared

http://wiki2.dovecot.org/SharedMailboxes/Shared


signature.asc
Description: Digital signature

Re: Error: istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed

[Thomas Leuxner]

* Timo Sirainen  2016.03.01 23:03:

> > I'm seeing these recently:
> > 
> > Feb 27 09:24:01 nihlus dovecot: imap(t...@leuxner.net): Error: 
> > istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed: No such file or 
> > directory
> 
> You don't have a /tmp directory or imap is chrooted somewhere where there is 
> no /tmp?..

No and no :) However the error message magically disappeared with newer builds. 
I'm not seeing it since Sunday...


signature.asc
Description: Digital signature

Error: istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed

[Thomas Leuxner]

I'm seeing these recently:

Feb 27 09:24:01 nihlus dovecot: imap(t...@leuxner.net): Error: 
istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed: No such file or 
directory
Feb 27 09:24:01 nihlus dovecot: imap(t...@leuxner.net): Error: 
istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed: No such file or 
directory
Feb 27 09:24:01 nihlus dovecot: imap(t...@leuxner.net): Error: 
istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed: No such file or 
directory
Feb 27 09:24:01 nihlus dovecot: imap(t...@leuxner.net): Error: 
istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed: No such file or 
directory
Feb 27 09:24:01 nihlus dovecot: imap(t...@leuxner.net): Error: 
istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed: No such file or 
directory
Feb 27 09:24:01 nihlus dovecot: imap(t...@leuxner.net): Error: 
istream-seekable: safe_mkstemp(/tmp/dovecot.imap.) failed: No such file or 
directory

$ doveconf -n | head
# 2.2.devel (f1a9b58): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.devel (0de2a19)

Regards
Thomas


signature.asc
Description: Digital signature

Re: systemd changes in Git/Debian Auto-Builds

[Thomas Leuxner]

* Timo Sirainen  2016.02.22 09:28:

> Another fix: 
> https://git.dovecot.net/dovecot/core/commit/2d5fcede6dcce079fdca5f0589c9c2142f016a65

Thanks. Confirmed effective.


signature.asc
Description: Digital signature

Re: systemd changes in Git/Debian Auto-Builds

[Thomas Leuxner]

* aki.tu...@dovecot.fi  2016.02.21 12:43:

> This is fixed in
> https://github.com/dovecot/core/commit/22d75d47097ca83c6e824a5129d81e8f18fb589b

Thanks Aki. As all patches are in the source tree, I guess this is not included 
in Stephan's Auto-Build:

$ dpkg --unpack dovecot-core_2.2.21-1~auto+130_amd64.deb 

$ grep PID /lib/systemd/system/dovecot.service 
PIDFile=/var/run/dovecot/dovecot/master.pid

I manually edited the file afterwards and ran dpkg --configure which brought up 
the server.


signature.asc
Description: Digital signature

systemd changes in Git/Debian Auto-Builds

[Thomas Leuxner]

Not sure if the issue is limited to the Debian Auto-Builds, but after the 
latest commits an update of the packages fails:

https://github.com/dovecot/core/commit/53cc71cae88ee81fd7eae47aed743496f8c884a2

Feb 21 07:31:22 nihlus dovecot: master: Dovecot v2.2.devel (950a6e6) starting 
up for imap, lmtp
Feb 21 07:31:22 nihlus systemd[1]: PID file /var/run/dovecot/dovecot/master.pid 
not readable (yet?) after start.

The PID-File seems to be expected under yet another sub-dir of /var/run/dovecot.

Regards
Thomas


signature.asc
Description: Digital signature

Re: [Dovecot-news] v2.2.20 release candidate released

[Thomas Leuxner]

* Timo Sirainen  2015.12.03 15:27:

> protocol imap {
>   namespace inbox {
> mailbox Trash {
>   autoexpunge = 5 days
> }
>   }
> }

Thanks.


signature.asc
Description: Digital signature

Re: [Dovecot-news] v2.2.20 release candidate released

[Thomas Leuxner]

* Timo Sirainen  2015.12.03 14:51:

>  + Added mailbox { autoexpunge= } setting. See
>http://wiki2.dovecot.org/MailboxSettings for details.

namespace inbox {
  mailbox Trash {
autoexpunge = 5 days
special_use = \Trash
  }
}

I'm using autoexpunge on the Trash mailbox. Looking at the wiki text I'm 
unclear on how to limit it to a specific service:

>So it may be better to explicitly enable this only inside protocol imap, pop3 
>and maybe lmtp.

Regards
Thomas


signature.asc
Description: Digital signature

Re: iterate users with passwd-file passdb?

[Thomas Leuxner]

* Timo Sirainen  2015.10.13 23:20:

> Use userdb passwd-file to get iteration working. passdb isn't used for 
> iteration. userdb static can't be used to iterate.

I guess this works for a single file comprising of all users at the moment:

Oct 15 11:13:58 nihlus dovecot: auth-worker(5545): Error: passwd-file: User 
iteration isn't currently supported with %variable paths

Is variable support planned?


signature.asc
Description: Digital signature

Re: Different behavior of ACLs in MUA and doveadm

[Thomas Leuxner]

* Timo Sirainen  2015.09.08 14:28:

> > Both debug levels raised, it doesn't log about the problem when using 
> > doveadm. I guess the patch is not enough:
> 
> With doveadm you need to give -D parameter for it to log debug output.

Comparing this to the previous imap log it does seem to ignore the global ACL 
pattern:

$ doveadm -D mailbox create -u t...@leuxner.net 
Public/Archive/Mailing-Lists/Dovecot/2015
Debug: Loading modules from directory: /usr/lib/dovecot/modules
Debug: Module loaded: /usr/lib/dovecot/modules/lib01_acl_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib10_quota_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_virtual_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
Debug: Module loaded: /usr/lib/dovecot/modules/lib90_stats_plugin.so
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Module loaded: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_expire_plugin.so: undefined 
symbol: expire_set_deinit (this is usually intentional, so just ignore this 
message)
Debug: Module loaded: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so
Debug: Module loaded: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined 
symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore 
this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: 
fts_backend_rescan (this is usually intentional, so just ignore this message)
doveadm(t...@leuxner.net): Debug: auth input: t...@leuxner.net 
quota_rule=*:storage=5G acl_groups=PublicMailboxAdmins uid=5000 gid=5000 
home=/var/vmail/domains/leuxner.net/tlx
doveadm(t...@leuxner.net): Debug: Added userdb setting: 
plugin/acl_groups=PublicMailboxAdmins
doveadm(t...@leuxner.net): Debug: Added userdb setting: 
plugin/quota_rule=*:storage=5G
doveadm(t...@leuxner.net): Debug: Effective uid=5000, gid=5000, 
home=/var/vmail/domains/leuxner.net/tlx
doveadm(t...@leuxner.net): Debug: acl: No acl_shared_dict setting - shared 
mailbox listing is disabled
doveadm(t...@leuxner.net): Debug: Quota root: name=user backend=dict 
args=:file:/var/vmail/domains/leuxner.net/tlx/mdbox/dovecot-quota
doveadm(t...@leuxner.net): Debug: Quota rule: root=user mailbox=* 
bytes=5368709120 messages=0
doveadm(t...@leuxner.net): Debug: Quota rule: root=user mailbox=Trash 
bytes=+536870912 (10%) messages=0
doveadm(t...@leuxner.net): Debug: Quota grace: root=user bytes=536870912 (10%)
doveadm(t...@leuxner.net): Debug: dict quota: user=t...@leuxner.net, 
uri=file:/var/vmail/domains/leuxner.net/tlx/mdbox/dovecot-quota, noenforcing=0
doveadm(t...@leuxner.net): Debug: Namespace inbox: type=private, prefix=, 
sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox
doveadm(t...@leuxner.net): Debug: fs: 
root=/var/vmail/domains/leuxner.net/tlx/mdbox, index=, indexpvt=, control=, 
inbox=, alt=
doveadm(t...@leuxner.net): Debug: acl: initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
doveadm(t...@leuxner.net): Debug: acl: acl username = t...@leuxner.net
doveadm(t...@leuxner.net): Debug: acl: owner = 1
doveadm(t...@leuxner.net): Debug: acl: group added: PublicMailboxAdmins
doveadm(t...@leuxner.net): Debug: acl vfile: Global ACL file: 
/var/vmail/conf.d/leuxner.net/global-acl
doveadm(t...@leuxner.net): Debug: Namespace : type=public, prefix=Public/, 
sep=/, inbox=no, hidden=no, list=yes, subscriptions=no 
location=mdbox:/var/vmail/public:INDEXPVT=~/mdbox/public
doveadm(t...@leuxner.net): Debug: fs: root=/var/vmail/public, index=, 
indexpvt=/var/vmail/domains/leuxner.net/tlx/mdbox/public, control=, inbox=, alt=
doveadm(t...@leuxner.net): Debug: acl: initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
doveadm(t...@leuxner.net): Debug: acl: acl username = t...@leuxner.net
doveadm(t...@leuxner.net): Debug: acl: owner = 0
doveadm(t...@leuxner.net): Debug: acl: group added: PublicMailboxAdmins
doveadm(t...@leuxner.net): Debug: acl vfile: Global ACL file: 
/var/vmail/conf.d/leuxner.net/global-acl
doveadm(t...@leuxner.net): Debug: Namespace : type=private, prefix=Virtual/, 
sep=/, inbox=no, hidden=no, list=yes, subscriptions=yes 
location=virtual:~/mdbox/virtual
doveadm(t...@leuxner.net): Debug: fs: 
root=/var/vmail/domains/leuxner.net/tlx/mdbox/virtual, index=, indexpvt=, 
control=, inbox=, alt=
doveadm(t...@leuxner.net): Debug: acl: initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
doveadm(t...@leuxner.net): Debug: acl: acl username = t...@leuxner.net
doveadm(t

Re: Different behavior of ACLs in MUA and doveadm

[Thomas Leuxner]

* Timo Sirainen  2015.09.08 12:20:

> How does the PublicMailboxAdmins group get set? Looks to me like the problem 
> is that it's not getting set to doveadm. Here's an easy way to check if 
> that's the problem or something else: 
> http://hg.dovecot.org/dovecot-2.2/rev/500e8dd7a389
> 
> If that doesn't help: Show your full doveconf -n, set auth_debug=yes and 
> mail_debug=yes and show the debug logs for IMAP login and doveadm. There's a 
> difference somewhere in there.

$ doveadm mailbox create -u t...@leuxner.net 
Public/Archive/Mailing-Lists/Dovecot/2015
doveadm(t...@leuxner.net): Error: Can't create mailbox 
Public/Archive/Mailing-Lists/Dovecot/2015: Permission denied

Both debug levels raised, it doesn't log about the problem when using doveadm. 
I guess the patch is not enough:

Sep  8 13:19:07 nihlus dovecot: auth: Debug: master in: 
USER#0111#011...@leuxner.net#011service=doveadm
Sep  8 13:19:07 nihlus dovecot: auth: Debug: passwd-file(t...@leuxner.net): 
userdb cache miss
Sep  8 13:19:07 nihlus dovecot: auth: Debug: passwd-file 
/var/vmail/auth.d/leuxner.net/passwd: Read 1 users in 0 secs
Sep  8 13:19:07 nihlus dovecot: auth: Debug: passwd-file(t...@leuxner.net): 
lookup: user=t...@leuxner.net file=/var/vmail/auth.d/leuxner.net/passwd
Sep  8 13:19:07 nihlus dovecot: auth: Debug: userdb out: 
USER#0111#011...@leuxner.net#011uid=5000#011gid=5000#011home=/var/vmail/domains/leuxner.net/tlx#011quota_rule=*:storage=5G#011acl_groups=PublicMailboxAdmins

With IMAP it is more talkative:

3 create "Public/Archive/Mailing-Lists/Dovecot/2015"

Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Added userdb 
setting: plugin/acl_groups=PublicMailboxAdmins
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Added userdb 
setting: plugin/quota_rule=*:storage=5G
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Effective 
uid=5000, gid=5000, home=/var/vmail/domains/leuxner.net/tlx
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: No 
acl_shared_dict setting - shared mailbox listing is disabled
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Quota root: 
name=user backend=dict 
args=:file:/var/vmail/domains/leuxner.net/tlx/mdbox/dovecot-quota
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Quota rule: 
root=user mailbox=* bytes=5368709120 messages=0
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Quota rule: 
root=user mailbox=Trash bytes=+536870912 (10%) messages=0
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Quota grace: 
root=user bytes=536870912 (10%)
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: dict quota: 
user=t...@leuxner.net, 
uri=file:/var/vmail/domains/leuxner.net/tlx/mdbox/dovecot-quota, noenforcing=0
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Namespace inbox: 
type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes 
location=mdbox:~/mdbox
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: fs: 
root=/var/vmail/domains/leuxner.net/tlx/mdbox, index=, indexpvt=, control=, 
inbox=, alt=
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: 
initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: acl 
username = t...@leuxner.net
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: owner = 1
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: group 
added: PublicMailboxAdmins
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl vfile: 
Global ACL file: /var/vmail/conf.d/leuxner.net/global-acl
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Namespace : 
type=public, prefix=Public/, sep=/, inbox=no, hidden=no, list=yes, 
subscriptions=no location=mdbox:/var/vmail/public:INDEXPVT=~/mdbox
/public
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: fs: 
root=/var/vmail/public, index=, 
indexpvt=/var/vmail/domains/leuxner.net/tlx/mdbox/public, control=, inbox=, alt=
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: 
initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: acl 
username = t...@leuxner.net
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: owner = 0
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl: group 
added: PublicMailboxAdmins
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: acl vfile: 
Global ACL file: /var/vmail/conf.d/leuxner.net/global-acl
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: Namespace : 
type=private, prefix=Virtual/, sep=/, inbox=no, hidden=no, list=yes, 
subscriptions=yes location=virtual:~/mdbox/virtual
Sep  8 13:06:29 nihlus dovecot: imap(t...@leuxner.net): Debug: fs: 
root=/var/vmail/domains/leuxner.net/tlx/mdbox

Re: Different behavior of ACLs in MUA and doveadm

[Thomas Leuxner]

* Thomas Leuxner  2014.12.31 22:10:

namespace {
  list = yes
  location = mdbox:/var/vmail/public:INDEXPVT=~/mdbox/public
  prefix = Public/
  separator = /
  subscriptions = no
  type = public
}

$ cat /var/vmail/conf.d/leuxner.net/global-acl
INBOX owner lrwstiekxap
Public/* group=PublicMailboxAdmins lrwsipk
Public/* anyone lr
Public/* authenticated lrws

$ doveadm mailbox create -u t...@leuxner.net 
Public/Archive/Mailing-Lists/Dovecot/2015
doveadm(t...@leuxner.net): Error: Can't create mailbox 
Public/Archive/Mailing-Lists/Dovecot/2015: Permission denied

$ doveadm acl get -u t...@leuxner.net Public/Archive/Mailing-Lists/Dovecot
doveadm(t...@leuxner.net): Error: Can't open mailbox 
Public/Archive/Mailing-Lists/Dovecot: Mailbox doesn't exist: 
Public/Archive/Mailing-Lists/Dovecot
ID Global Rights

I retested this issue after all the HG commits. Doveadm still treats the 
namespace/ACL differently compared to a MUA. While doveadm refuses to create 
the mailbox, the MUA succeeds. However I'd like to do all this scripted using 
doveadm ideally... 

$ openssl s_client -connect host.domain.tld:143 -starttls imap

. OK Pre-login capabilities listed, post-login capabilities have more.
1 login t...@leuxner.net 
* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT 
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND 
URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED 
I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH 
LIST-STATUS BINARY MOVE NOTIFY SPECIAL-USE QUOTA ACL RIGHTS=texk
1 OK Logged in
2 list "Public/Archive" *
[...]
* LIST (\Noselect \HasChildren) "/" Public/Archive/Mailing-Lists/Dovecot
* LIST (\HasNoChildren \UnMarked) "/" Public/Archive/Mailing-Lists/Dovecot/2014
* LIST (\HasNoChildren \UnMarked) "/" Public/Archive/Mailing-Lists/Dovecot/2013
* LIST (\HasNoChildren \UnMarked) "/" Public/Archive/Mailing-Lists/Dovecot/2012
[...]
2 OK List completed (0.016 secs).
3 create "Public/Archive/Mailing-Lists/Dovecot/2015"
3 OK Create completed (0.006 secs).
4 list "Public/Archive" *
[...]
* LIST (\HasNoChildren) "/" Public/Archive/Mailing-Lists/Dovecot/2015


signature.asc
Description: Digital signature

Re: HG port=0 changes still problematic

[Thomas Leuxner]

* Stephan Bosch  2015.08.31 02:13:

> Yes, yes, yes.. you may smack me now :)
> 
> http://hg.rename-it.nl/dovecot-2.2-patches/file/c51a10802ce6/in_port_t_fix_2.patch

I wouldn't go that far :) Latest HG works. Thanks


signature.asc
Description: Digital signature

HG port=0 changes still problematic

[Thomas Leuxner]

Aug 30 12:35:04 nihlus.leuxner.net systemd[1]: Started Dovecot IMAP/POP3 email 
server.
Aug 30 12:35:04 nihlus.leuxner.net dovecot[60404]: doveconf: Fatal: Error in 
configuration file /etc/dovecot/conf.d/10-master.conf line 24: Invalid port 
number 0
Aug 30 12:35:04 nihlus.leuxner.net systemd[1]: dovecot.service: main process 
exited, code=exit

System with HG e1c7fcd62813 fails on

 inet_listener imaps {
port = 0
#port = 993
#ssl = yes
  }

Regards
Thomas


signature.asc
Description: Digital signature

Re: IMAP hibernate feature committed

[Thomas Leuxner]

* Timo Sirainen  2015.08.29 12:20:

> Set verbose_proctitle=yes and you'll see what the imap processes are doing. 
> It'll show IDLE in the ps output if the client is IDLEing. Although it 
> doesn't say how long the client has been doing it.

dovenull  1318  0.0  0.0  18340  4772 ?S17:50   0:00 
dovecot/imap-login [1 connections (1 TLS)]
dovenull  1333  0.0  0.0  18340  4576 ?S17:50   0:00 
dovecot/imap-login [1 connections (1 TLS)]
dovenull  1335  0.0  0.0  18200  4652 ?S17:50   0:00 
dovecot/imap-login
dovenull  1336  0.0  0.0  18200  4676 ?S17:50   0:00 
dovecot/imap-login
dovenull  1342  0.0  0.0  18340  4772 ?S17:51   0:00 
dovecot/imap-login [1 connections (1 TLS)]
dovenull  1704  0.0  0.0  18200  4572 ?S19:15   0:00 
dovecot/imap-login
dovenull  1709  0.0  0.0  18200  4652 ?S19:15   0:00 
dovecot/imap-login
dovenull  1711  0.0  0.0  18200  4636 ?S19:15   0:00 
dovecot/imap-login
dovenull  1994  0.0  0.0  18200  4740 ?S20:56   0:00 
dovecot/imap-login
dovenull  1997  0.0  0.0  18200  4568 ?S20:56   0:00 
dovecot/imap-login
vmail 1998  0.0  0.0  45448 10048 ?S20:56   0:00 dovecot/imap 
[t...@leuxner.net 1.2.3.4 IDLE]
dovenull  1999  0.0  0.0  18340  4732 ?S20:56   0:00 
dovecot/imap-login [1 connections (1 TLS)]
dovenull  2001  0.0  0.0  18200  4796 ?S20:56   0:00 
dovecot/imap-login
vmail 2002  0.0  0.0  52880 13500 ?S20:56   0:00 dovecot/imap 
[t...@leuxner.net 1.2.3.4 IDLE]
dovecot   2008  0.0  0.0   9380   936 ?S20:57   0:00 
dovecot/imap-hibernate [2 connections]

After disabling "check for new messages" polling in the client it looks a lot 
better, multiple connections have been hibernated. Thanks.


signature.asc
Description: Digital signature

Re: IMAP hibernate feature committed

[Thomas Leuxner]

* Teemu Huovila  2015.08.27 13:58:

> Did you specify a value other than zero for 'imap_hibernate_timeout'?

Yes I did:

$ doveconf imap_hibernate_timeout
imap_hibernate_timeout = 1 mins

I sometimes see one imap-hibernate process (only one), but several imap 
processes active which should be idling... 


signature.asc
Description: Digital signature

Re: IMAP hibernate feature committed

[Thomas Leuxner]

* Timo Sirainen  2015.08.25 22:21:

> There's no good default setting here. It depends on your userdb settings 
> and/or mail_uid setting. So for example if your imap processes are running as 
> vmail user, you should set service imap-hibernate { unix_listener 
> imap-hibernate { user = vmail } }. Then again if you are using system users 
> (or otherwise multiple UIDs) it gets more difficult to implement this 
> securely (mode=0666 works always, but security isn't too good). This same 
> problem exists for various other parts of Dovecot, for example indexer-worker 
> and dict services.

I have it working (I guess) with these user settings (virtual users using 
'vmail'):

service imap-hibernate {
  unix_listener imap-hibernate {
user = vmail
  }
}

I had to assign the imap-master socket the user the imap-hibernate process is 
using to avoid messages like this:

Aug 25 23:16:02 nihlus dovecot: imap-hibernate(t...@leuxner.net): Error: 
net_connect_unix(/var/run/dovecot/imap-master) failed: Permission denied
Aug 25 23:16:02 nihlus dovecot: imap-hibernate(t...@leuxner.net): Failed to 
connect to master socket in=126 out=944 hdr=0 body=0 del=0 exp=0 trash=0

service imap {
  unix_listener imap-master {
user = dovecot
  }
}

With this I see messages like this in the logs:

Aug 26 09:48:06 nihlus dovecot: imap-hibernate(t...@leuxner.net): Connection 
closed in=189 out=4252 hdr=0 body=0 del=0 exp=0 trash=0
Aug 26 12:20:29 nihlus dovecot: imap-hibernate(t...@leuxner.net): Connection 
closed in=109 out=4714 hdr=0 body=0 del=0 exp=0 trash=0

I'm a bit puzzled as to when hibernate actually kicks in because most of the 
time I see normal imap processes running without them being hibernated:

$ ps aux | grep dovecot/imap
dovenull  6791  0.0  0.0  18196  4772 ?S06:39   0:00 
dovecot/imap-login
dovenull  7107  0.0  0.0  18196  4736 ?S08:00   0:00 
dovecot/imap-login
dovenull  7112  0.0  0.0  18332  4492 ?S08:00   0:00 
dovecot/imap-login
dovenull  7333  0.0  0.0  18332  4772 ?S08:45   0:00 
dovecot/imap-login
dovenull  7675  0.0  0.0  18196  4628 ?S10:13   0:00 
dovecot/imap-login
dovenull  7677  0.0  0.0  18332  4532 ?S10:14   0:00 
dovecot/imap-login
dovenull  7821  0.0  0.0  18196  4532 ?S10:44   0:00 
dovecot/imap-login
dovenull  8156  0.0  0.0  18196  4756 ?S12:01   0:00 
dovecot/imap-login
vmail 8157  0.0  0.0  45624  9608 ?S12:01   0:00 dovecot/imap
dovenull  8158  0.0  0.0  18332  4628 ?S12:01   0:00 
dovecot/imap-login
vmail 8159  0.0  0.0  44772  9256 ?S12:01   0:00 dovecot/imap
dovenull  8160  0.0  0.0  18196  4652 ?S12:01   0:00 
dovecot/imap-login
vmail 8161  0.0  0.0  46072  9760 ?S12:01   0:00 dovecot/imap
dovenull  8162  0.0  0.0  18196  4548 ?S12:01   0:00 
dovecot/imap-login
dovenull  8279  0.0  0.0  18332  4736 ?S12:22   0:00 
dovecot/imap-login
vmail 8280  0.0  0.0  40712  5164 ?S12:22   0:00 dovecot/imap
dovenull  8341  0.0  0.0  18196  4740 ?S12:25   0:00 
dovecot/imap-login
vmail 8344  0.0  0.0  46312 10568 ?S12:25   0:00 dovecot/imap


signature.asc
Description: Digital signature

Re: IMAP hibernate feature committed

[Thomas Leuxner]

* Timo Sirainen  2015.08.25 17:28:

> >> ==> /var/log/dovecot/dovecot.log <==
> >> Aug 25 09:42:07 nihlus dovecot: imap(t...@leuxner.net): Error: 
> >> net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied
> >> Aug 25 09:42:07 nihlus dovecot: imap(t...@leuxner.net): Error: Couldn't 
> >> hibernate imap client: Couldn't export state: Virtual mailboxes have no 
> >> GUIDs

Those are completely gone with the latest two commits. I was expecting the 
permission error to fire up. It seems a bit too quite. If it works would it 
spawn a hibernate-process? It looks so from the service section, but I don't 
see any "hibernate" processes active.

> > 'chmod 666' mitigates the permission issue on the socket. However it seems 
> > to have other issues then:
> 
> You can also change the unix_listener { user, group, mode } as needed for 
> different services (imap, imap-hibernate). http://wiki2.dovecot.org/Services 
> has some more info.

$ doveconf -a | grep -A 20 'service imap-hibernate'
service imap-hibernate {
  […]
  unix_listener imap-hibernate {
group = 
mode = 0600
user = 
  }
  user = $default_internal_user

The question is what user it should be - or what user it should match in case 
several users come into play. With the standard setting $default_internal_user 
as above it does not work out of the box (at least with my config). 

Regards
Thomas


signature.asc
Description: Digital signature

Re: IMAP hibernate feature committed

[Thomas Leuxner]

* Thomas Leuxner  2015.08.25 09:45:

> > http://hg.dovecot.org/dovecot-2.2/rev/64c73e6bd397
> 
> ==> /var/log/dovecot/dovecot.log <==
> Aug 25 09:42:07 nihlus dovecot: imap(t...@leuxner.net): Error: 
> net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied
> Aug 25 09:42:07 nihlus dovecot: imap(t...@leuxner.net): Error: Couldn't 
> hibernate imap client: Couldn't export state: Virtual mailboxes have no GUIDs
> Aug 25 09:42:07 nihlus dovecot: imap(t...@leuxner.net): Error: 
> net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied

'chmod 666' mitigates the permission issue on the socket. However it seems to 
have other issues then:

==> /var/log/dovecot/dovecot.log <==
Aug 25 09:56:49 nihlus dovecot: imap-hibernate: Error: Failed to parse client 
input: Invalid peer_ip value:
Aug 25 09:56:49 nihlus dovecot: imap(t...@leuxner.net): Error: 
/var/run/dovecot/imap-hibernate returned failure: Failed to parse client input: 
Invalid peer_ip value:
Aug 25 09:56:49 nihlus dovecot: imap: Error:
Aug 25 09:56:49 nihlus dovecot: imap-hibernate: Error: Failed to parse client 
input: Invalid peer_ip value:
Aug 25 09:56:49 nihlus dovecot: imap(t...@leuxner.net): Error: 
/var/run/dovecot/imap-hibernate returned failure: Failed to parse client input: 
Invalid peer_ip value:
Aug 25 09:56:49 nihlus dovecot: imap: Error:


signature.asc
Description: Digital signature

Re: IMAP hibernate feature committed

[Thomas Leuxner]

> http://hg.dovecot.org/dovecot-2.2/rev/64c73e6bd397

==> /var/log/dovecot/dovecot.log <==
Aug 25 09:42:07 nihlus dovecot: imap(t...@leuxner.net): Error: 
net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied
Aug 25 09:42:07 nihlus dovecot: imap(t...@leuxner.net): Error: Couldn't 
hibernate imap client: Couldn't export state: Virtual mailboxes have no GUIDs
Aug 25 09:42:07 nihlus dovecot: imap(t...@leuxner.net): Error: 
net_connect_unix(/var/run/dovecot/imap-hibernate) failed: Permission denied

$  l /var/run/dovecot/
total 8
srw--- 1 root root   0 Aug 24 18:22 anvil
srw--- 1 root root   0 Aug 24 18:22 anvil-auth-penalty
srw--- 1 dovecot  root   0 Aug 25 09:17 auth-client
srw--- 1 dovecot  root   0 Aug 25 09:17 auth-login
srw--- 1 root root   0 Aug 25 09:17 auth-master
-rw--- 1 root root  32 Aug  7 12:03 auth-token-secret.dat
srw-rw-rw- 1 dovecot  root   0 Aug 25 09:17 auth-userdb
srw--- 1 doveauth root   0 Aug 25 09:17 auth-worker
srw--- 1 root root   0 Aug 25 09:17 config
srw--- 1 root root   0 Aug 25 09:17 dict
srw--- 1 root root   0 Aug 25 09:17 director-admin
srw--- 1 root root   0 Aug 25 09:17 director-userdb
srw-rw-rw- 1 root root   0 Aug 25 09:17 dns-client
srw--- 1 root root   0 Aug 25 09:17 doveadm-server
lrwxrwxrwx 1 root root  25 Aug 24 18:22 dovecot.conf -> 
/etc/dovecot/dovecot.conf
drwxr-xr-x 2 root root  40 Aug  7 12:03 empty
srw--- 1 root root   0 Aug 25 09:17 imap-hibernate
srw--- 1 root root   0 Aug 25 09:17 imap-master
srw-rw-rw- 1 root root   0 Aug 25 09:17 imap-urlauth
srw--- 1 dovecot  root   0 Aug 25 09:17 imap-urlauth-worker
srw-rw-rw- 1 root root   0 Aug 25 09:17 indexer
srw--- 1 dovecot  root   0 Aug 25 09:17 indexer-worker
srw--- 1 root root   0 Aug 25 09:17 ipc
srw-rw-rw- 1 root root   0 Aug 25 09:17 lmtp
srw--- 1 root root   0 Aug 25 09:17 log-errors
drwxr-x--- 2 root dovenull 120 Aug 25 09:17 login
-rw--- 1 root root   6 Aug 24 18:22 master.pid
srw--- 1 root root   0 Aug 25 09:17 replication-notify
prw--- 1 root root   0 Aug 25 09:17 replication-notify-fifo
srw--- 1 dovecot  root   0 Aug 25 09:17 replicator
srw-rw-rw- 1 root root   0 Aug 25 09:17 ssl-params
srw--- 1 root root   0 Aug 25 09:17 stats
prw--- 1 vmailroot   0 Aug 25 09:43 stats-mail
drwxr-x--- 2 root dovenull  80 Aug 25 09:17 token-login

The socket is created as ‚root‘ per default. Does it need to run under another 
user to mitigate the permissions issue?

Regards
Thomas


signature.asc
Description: Message signed with OpenPGP using GPGMail

Typo in systemd commit HG 8dc79a437858

[Thomas Leuxner]

The apostrophes need to be removed from the LimitCORE example, otherwise the 
parameter will not be recognized:

$ cat /etc/systemd/system/dovecot.service.d/service.conf 
[Service]
#Environment='OPTIONS=-p'
#LimitCORE=8192
LimitCORE=infinity


signature.asc
Description: Digital signature

Re: v2.2.17 release candidate released

[Thomas Leuxner]

* Timo Sirainen  2015.05.09 19:41:

>   + lda, lmtp: deliver_log_format can now include %{delivery_time},
> which expands to how many milliseconds it took to deliver the mail.
> With LMTP %{session_time} also expands to how many milliseconds the
> LMTP session took, not including the delivery time.

This could use some nicer defaults similar to Postfix logging e.g.

deliver_log_format = msgid=%m, time=%{delivery_time}ms, status=%$

May 11 18:25:29 nihlus dovecot: lmtp(t...@leuxner.net): CupIHNrXUFXUTAAAgUOSbA: 
sieve: msgid=<5550d7cb.70...@comstyle.com>, time=133ms, status=stored mail into 
mailbox 'Public/Mailing-Lists/Dovecot'


signature.asc
Description: Digital signature

Re: Xi broken

[Thomas Leuxner]

* Stephan Bosch  2015.05.10 17:34:

> Don't hesitate to notify me if there are any problems.

Thanks for fixing this. As this repo stuff is mostly bleeding edge, I noticed 
that the systemd service file needs to be modified to allow coredumps. Not sure 
how it is supposed to interact with the file /etc/default/dovecot and its 
ALLOW_COREDUMPS=1 variable, but one needs to explicitly add LimitCORE=infinity 
in the 'Service' section (until someone with more systemd foo fixes it):

$ cat /lib/systemd/system/dovecot.service 
[Unit]
Description=Dovecot IMAP/POP3 email server
After=local-fs.target network.target

[Service]
Type=simple
LimitCORE=infinity
ExecStart=/usr/sbin/dovecot -F
NonBlocking=yes

[Install]
WantedBy=multi-user.target


signature.asc
Description: Digital signature

Re: Postpone email delivery with LMTP and Postfix

[Thomas Leuxner]

* Miloslav Hůla  2015.04.29 22:47:

> is there any way, based on userdb/passwdb attribute, how to postpone an
> email delivery? The purpose is, I need to freeze an account (Maildir++) for
> a few minutes and new email must not be delivered. But emails must be
> delivered when account is unfrozen.

You can put the messages on hold and then release them again:

http://wiki2.dovecot.org/Migration/Online


signature.asc
Description: Digital signature

Re: ACL Error

[Thomas Leuxner]

* Bobber  2015.02.25 19:01:

> >  acl = vfile:/usr/local/etc/dovecot/dovecot-acl

> And here is my dovecot-acl:
> 
> >user=bobber lrwstipekxa
> >authenticated lr

Global ACLs require a search pattern as described here:

http://wiki2.dovecot.org/ACL

* user=bobber lrwstipekxa
Public/* user=bobber lrwstipekxa

Regards
Thomas


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* Peter Hodur  2015.01.29 05:25:

> > With a current version of Dovecot a search is pretty fast _without_ using
> > external indexes. I have a view defined (virtual plugin) with around 22.000
> > messages in it, and searching the full view only takes 2.5 seconds:
> >
> >
> hmmm, could you please tell me more about your setup? What storage format
> do you use? Maildir or? What is the cumulative size of your messages?


Although the question was not formatted/quoted correctly I guess it is more 
towards myself as it explicitely speaks of a non-FTS setup:

$ dmesg | grep smpboot
[0.00] smpboot: Allowing 64 CPUs, 56 hotplug CPUs
[0.148600] smpboot: CPU0: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz (fam: 
06, model: 3c, stepping: 03)
[0.300030] smpboot: Total of 8 processors activated (52677.72 BogoMIPS)

The volume providing the mail root is using ext4 on an MD mirror (with CRYPT 
and LVM overhead):


$ lsblk
[...]
└─sda38:30   ...  0 part
  └─md2   9:20   ...  0 raid1
└─crypt1 (dm-0) 253:00   ...  0 crypt
  └─vg0-vmail (dm-11)   253:11   0   ...  0 lvm   /var/vmail

$ doveconf -n | head -3
# 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.6
# OS: Linux 3.16.0-0.bpo.4-amd64 x86_64 Debian 7.8

I'm using unspectacular 7,200rpm SATA drives and mdbox.


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* superinterstel...@gmail.com  2015.01.28 12:07:

> Do you think I should try 2.2.15? Or stick to EE?

This is in the repo. My version just has *all* the latest changes applied which 
sometime can be untested.


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* superinterstel...@gmail.com  2015.01.28 11:53:

> Also could you advise which dovecot are you using?

Bleeding edge (latest and hopefully greatest compiled from HG):

# 2.2.15 (7557234ac0f4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.6


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* Kevin Laurie  2015.01.28 03:31:

> . OK Search completed (21.478 secs).

Really hard to say what commit probably fixed that already. Dovecot uses a very 
dynamic development model. There are old versions like 2.0.9 which people still 
try to get help with since they are in some dusted distro repository, but known 
to have partially broken features... Looking at the CVS of Dovecot there are so 
many changes it's really hard to say there was an issue or not.

http://hg.dovecot.org/dovecot-2.2/shortlog

One changelog (2.2.11) however indicates the search was not working as it 
should:

 - IMAP: SEARCH/SORT PARTIAL was handled completely wrong in v2.2.11+

You may want to look at the official Enterprise repository which does support 
Ubuntu. This one does have the current stable releases in it and is not 
undergoing so many changes compared to the CVS auto-compiles also offered:

http://shop.dovecot.fi/home/8-dovecot-ee-repository-access.html

Give that one a try with vanilla search...


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* Kevin Laurie  2015.01.27 11:15:

Hi Kevin,

> # 2.2.9: /etc/dovecot/dovecot.conf
> # OS: Linux 3.10.62-xenU-25-0e6777a-x86_64 x86_64 Ubuntu 14.04.1 LTS

Although this was released back in 2013, let's assume the core search is not 
substantially broken for this release.

> mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
> mail_plugins = quota fts

As the mail storage looks okay, could you try with the parameter 'fts' removed? 
We would then have the search hit a vanilla Dovecot without plugins involved...

Regards
Thomas


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* Kevin Laurie  2015.01.27 03:24:

> . search text dear
> * OK Searched 9% of the mailbox, ETA 1:39
> * OK Searched 20% of the mailbox, ETA 1:18
> * OK Searched 29% of the mailbox, ETA 1:12
> * OK Searched 38% of the mailbox, ETA 1:03
> * OK Searched 50% of the mailbox, ETA 0:49
> * OK Searched 60% of the mailbox, ETA 0:39
> * OK Searched 69% of the mailbox, ETA 0:30
> * OK Searched 79% of the mailbox, ETA 0:20

I guess we need to start looking at your version and the format/mail_location 
configuration, best to post your 'doveconf -n'.

Emulating the same search on my box looks a lot better:

$ /usr/lib/dovecot/imap -u t...@leuxner.net
* PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT 
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS 
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN 
CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NOTIFY QUOTA ACL 
RIGHTS=texk] Logged in as t...@leuxner.net
a select "Virtual/Postfix-Users"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old \*)] Flags 
permitted.
* 22120 EXISTS
* 0 RECENT
* OK [UNSEEN 22109] First unseen.
* OK [UIDVALIDITY 1420123174] UIDs valid
* OK [UIDNEXT 44062] Predicted next UID
* OK [HIGHESTMODSEQ 702] Highest
a OK [READ-WRITE] Select completed (0.031 secs).
b search text dear
* SEARCH 111 187 189 237 266 293 294 339 842 853 882 883 929 946 1346 1348 1349 
1433 1434 1435 1474 1475 1483 1500 1504 1509 1512 1514 1639 1642 1766 2364 2366 
2367 2635 2638 2639 2640 2642 2644 2664 2738 2750 2830 2849 2857 3264 3265 3266 
3267 3268 3271 3274 3301 3302 3303 3305 3486 3489 3494 3499 3500 3752 3899 4013 
4135 4136 4209 4421 4460 4464 4465 4469 4477 4478 4479 4480 4482 4483 4522 4535 
4536 4538 4751 4769 5168 5169 5178 5179 5219 5254 5257 5300 5301 5304 5333 5336 
5363 5364 5384 5406 5476 5479 5482 5493 5494 5499 5507 5516 5521 5768 6241 6243 
7470 7471 7861 7862 7864 7874 7875 7876 7877 7879 8389 8399 8428 8429 8907 8937 
8938 8940 9469 9495 9610 9688 9735 9881 9882 9883 9885 9913 9914 9915 9996 
10009 10091 10094 10218 10229 10230 10332 10335 10340 10341 10517 10610 10612 
10826 10829 10830 11081 11082 11087 11089 11097 11144 11145 11212 11213 11359 
11365 11366 11367 11369 11375 11600 11673 11710 11941 11942 11946 11948 12046 
12205 12208 12209 12407 12409 12411 13104 13551 13589 13806 13815 13818 13881 
13883 13885 13980 14453 14578 14617 14618 14620 14666 14667 14668 14677 14685 
14783 14790 14964 14966 14967 14972 15050 15436 15476 15949 15959 15964 15965 
15971 16106 16110 16283 16284 16293 16310 16318 16331 16372 16373 16374 16785 
16786 16787 16985 16987 17050 17855 17857 17871 17875 17876 17877 17959 17978 
17979 18044 18807 18814 18821 18822 18823 18917 18918 18936 19005 19121 19291 
19292 20058 20372 20467 20468 20469 20475 20489 20510 20516 20521 20522 20523 
20524 20541 20560 20562 20582 20584 20653 20654 20904 21532 21595 21596 21716 
21717 21719 21724 21725
* 7 RECENT
b OK Search completed (4.468 secs).


signature.asc
Description: Digital signature

Dovecot Virtual Plugin Use Cases (was Re: Indexing Mail faster)

[Thomas Leuxner]

* LuKreme  2015.01.25 19:32:

> > Another example, the one I used in the original reply, is 'gluing' together 
> > archives as a single view:
> > 
> > $ cat virtual/Postfix-Users/dovecot-virtual 
> > Public/Archive/Mailing-Lists/Postfix-Users/*
> > Public/Mailing-Lists/Postfix-Users
> >  all
> 
> This all sounds interesting. Is there anything i can read to understand what 
> the purpose of Virtual is and why this would help? It sounds like you are 
> using a shared space for mailing lists (which I can see would be useful if 
> you had many people subscribing to the same lists). If so, doesn’t each user 
> lose their indications of what messages are read or replied to?

http://wiki2.dovecot.org/Plugins/Virtual

It's a bit hard to wrap your head around it in the beginning, but once you 
understand what it can do, I'm sure you'll find a use case. Think of it as 
special view for all messages in Dovecot. You define which set of messages you 
want to display as a mailbox/folder. This view (or multiple views) is being 
updated dynamically going forward.

For instance you can set up a certain filter/selection like you can with a 
modern MUA. The difference however is: your Virtual view(s) will be available 
regardless which client you are using, whether they have integrated filters 
capabilities - or not.


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* A. Schulze  2015.01.25 16:58:

> once setup correctly it works like expected :-)

Good!
 
> how do you separate older postings into the Archive?
> something like "doveadm search + move ?

Yes basically with move. I'd like to have them archived by year e.g. 
Public/Archive/Mailing-Lists/Dovecot/2014. Being lazy when it comes to 
repetitive tasks, I wrote a little script for it. It gives a good idea how to 
do it (hopefully). Please note that this only has minimal error handling as I'm 
setting most parameters in the script rather than the shell - butterfingers... 
No bashism comments please :)

$ cat dovearchive.sh 
#!/bin/bash
# Archive old posts before certain date to 'Public/Archive/Mailbox/Year' 
# Set basic parameters accordingly:
# archive=2011, before_date=$year-mm-dd
# source_mailbox_base=Public/Newsletters
# dest_mailbox_base=Public/Archive/Newsletters
# Actual Mailbox is read from command line
set -e
archive=2014
let year=$archive+1
before_date=$year-01-01
mailbox_owner=t...@leuxner.net
source_mailbox_base='Public/Mailing-Lists'
dest_mailbox_base='Public/Archive/Mailing-Lists'
#acl_admin_group=owner
acl_admin_group='group=PublicMailboxAdmins'
acl_unlock_seq="$acl_admin_group delete expunge insert lookup post read write 
write-seen write-deleted"
acl_lock_seq="$acl_admin_group insert lookup post read write write-seen"
acl_lock_archive="$acl_admin_group insert lookup read write write-seen"
debug_acl_mailbox_path='path: '

msg_formatted() {
  echo "$(date "+%b %d %H:%M:%S") $*"
}

if [ $# -eq 0 ]; then
echo "usage: $0 mailbox" >&2
exit 1
fi

# Mailbox exists?
doveadm acl get -u $mailbox_owner "$source_mailbox_base/$1" || { echo 'Mailbox 
not found.'; exit 1; }

# Create New Archive Mailbox
doveadm mailbox create -u $mailbox_owner "$dest_mailbox_base/$1/$archive"

# Modify ACL, expunge mail and revert ACL
msg_formatted "[>] Archiving \"$dest_mailbox_base/$1/$archive\""

doveadm acl set -u $mailbox_owner "$source_mailbox_base/$1" $acl_unlock_seq
doveadm move -u $mailbox_owner "$dest_mailbox_base/$1/$archive" mailbox 
"$source_mailbox_base/$1" before $before_date
doveadm acl set -u $mailbox_owner "$source_mailbox_base/$1" $acl_lock_seq

# Get actual path of dovecot-acl file and remove it as it is handled by global 
ACL
acl_path=$(doveadm acl debug -u $mailbox_owner "$source_mailbox_base/$1" 2>&1 | 
sed -n  "s/\(.*\)$debug_acl_mailbox_path//p") 

[ -f $acl_path/dovecot-acl ] && rm $acl_path/dovecot-acl

msg_formatted '[ Complete ]'


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* Andreas Schulze  2015.01.25 12:47:

> I assume you drop multiple mailing lists together in a mailbox and separate 
> them using the virtual plugin, right?
> could publish how you configured dovecot virtual plugin?

The namespace "Virtual" will look for its configuration in the user's home 
directory:

namespace {
  location = virtual:~/mdbox/virtual
  prefix = Virtual/
  separator = /
}

Inside the ~/mdbox/virtual directory the actual query is defined in directory 
'Flagged' (that will show up in the MUA as sub-dir). The actual search/query is 
defined in file 'dovecot-virtual' which will create the "magic": 

$ cat virtual/Flagged/dovecot-virtual 
*
Public/*
  flagged

Another example, the one I used in the original reply, is 'gluing' together 
archives as a single view:

$ cat virtual/Postfix-Users/dovecot-virtual 
Public/Archive/Mailing-Lists/Postfix-Users/*
Public/Mailing-Lists/Postfix-Users
  all

HTH


signature.asc
Description: Digital signature

Re: Indexing Mail faster

[Thomas Leuxner]

* Kevin Laurie  2015.01.24 19:41:

> Currently the time it takes to search 25,000mails is 4mins. If indexed how
> much faster are we looking at?

With a current version of Dovecot a search is pretty fast _without_ using 
external indexes. I have a view defined (virtual plugin) with around 22.000 
messages in it, and searching the full view only takes 2.5 seconds:

$ /usr/lib/dovecot/imap -u t...@leuxner.net
* PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT 
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS 
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN 
CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NOTIFY QUOTA ACL 
RIGHTS=texk] Logged in as t...@leuxner.net
a select "Virtual/Postfix-Users"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old \*)] Flags 
permitted.
* 22094 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1420123174] UIDs valid
* OK [UIDNEXT 44036] Predicted next UID
* OK [HIGHESTMODSEQ 679] Highest
a OK [READ-WRITE] Select completed (0.028 secs).
b search charset utf-8 body "SQL"
* SEARCH 51 52 54 83 87 102 108 126 172 173 195 223 234 238 249 250 252 254 260 
263 268 274 293 294 316 317 324 343 412 414 431 432 434 435 436 438 439 440 441 
442 443 450 451 454 457 458 464 478 493 496 497 505 590 617 619 625 628 632 636 
637 638 639 644 645 646 650 655 657 659 671 676 677 699 702 757 759 762 767 769 
770 790 791 792 793 796 799 839 840 841 843 844 845 846 847 848 849 850 864 868 
882 905 907 929 953 955 959 1129 1133 1136 1137 1138 1139 1140 1143 1145 1168 
1174 1175 1176 1177 1178 1179 1182 1205 1208 1244 1247 1248 1251 1316 1320 1334 
1336 1352 1356 1392 1394 1395 1397 1398 1403 1407 1409 1410 1411 1433 1434 1435 
1451 1468 1469 1470 1511 1512 1546 1547 1548 1560 1561 1562 1563 1567 1569 1570 
1574 1578 1587 1588 1590 1601 1606 1648 1661 1669 1670 1671 1726 1748 1749 1750 
1751 1752 1753 1755 1771 1773 1774 1792 1800 1808 1819 1822 1850 1885 1913 1932 
1933 1980 1982 1999 2004 2026 2102 2104 2144 2145 2162 2184 2192 2193 2194 2204 
2205 2217 2237 2272 2336 2337 2339 2341 2342 2343 2344 2346 2348 2368 2369 2380 
2409 2426 2429 2455 2461 2499 2605 2607 2609 2611 2615 2618 2620 2635 2638 2639 
2640 2642 2673 2784 2797 2799 2800 2821 2830 2849 2857 2863 2894 2895 2943 2944 
2953 2954 2955 3049 3050 3052 3195 3244 3418 3419 3420 3421 3425 3465 3470 3474 
3530 3543 3544 3546 3549 3550 3569 3572 3593 3602 3643 3647 3648 3649 3650 3654 
3655 3656 3673 3674 3721 3722 3723 3724 3728 3740 3741 3772 3795 3797 3799 3803 
3804 3806 3830 3832 3837 3838 3839 3840 3841 3859 3865 3866 3867 3875 3896 3898 
3907 3915 3919 3924 3925 3927 3949 3951 4029 4031 4043 4073 4083 4084 4085 4087 
4089 4091 4094 4096 4185 4186 4350 4354 4400 4401 4403 4404 4405 4406 4417 4419 
4422 4423 4424 4425 4427 4431 4463 4501 4554 4556 4583 4615 4640 4642 4650 4683 
4684 4686 4687 4688 4690 4691 4696 4697 4698 4699 4702 4705 4713 4714 4724 4725 
4726 4727 4728 4729 4730 4734 4746 4754 4770 4772 4785 4787 4792 4794 4798 4800 
4912 4922 4955 4956 4958 4966 4972 4973 4974 4975 4978 4979 4988 4990 4991 4992 
4998 5009 5021 5025 5026 5027 5029 5030 5031 5036 5037 5109 5110 5111 5112 5149 
5150 5151 5156 5159 5202 5248 5249 5365 5366 5367 5402 5428 5441 5442 5447 5449 
5483 5484 5486 5487 5489 5493 5494 5495 5496 5497 5498 5499 5501 5502 5503 5504 
5506 5508 5604 5681 5721 5722 5723 5724 5725 5727 5728 5729 5730 5731 5737 5738 
5761 5778 5794 5796 5817 5818 5860 5863 5864 5865 5867 5989 5990 5991 6006 6028 
6075 6078 6080 6081 6082 6083 6090 6092 6093 6103 6104 6115 6122 6193 6194 6197 
6198 6235 6243 6258 6259 6260 6268 6312 6317 6321 6408 6409 6471 6472 6485 6546 
6551 6620 6622 6623 6624 6630 6637 6652 6694 6698 6710 6768 6770 6772 6797 6851 
6853 6868 6899 6900 6949 7092 7093 7095 7156 7219 7226 7227 7265 7278 7280 7281 
7282 7283 7294 7295 7305 7307 7310 7317 7319 7327 7338 7341 7342 7358 7363 7365 
7370 7378 7381 7382 7434 7447 7448 7449 7450 7452 7509 7516 7517 7518 7520 7521 
7522 7523 7524 7525 7526 7544 7585 7614 7616 7617 7637 7644 7670 7671 7794 7795 
7796 7885 7886 7887 7889 7895 7898 7900 7953 8016 8052 8054 8055 8056 8057 8059 
8063 8064 8066 8069 8079 8119 8132 8156 8175 8232 8233 8234 8235 8236 8242 8289 
8354 8420 8506 8507 8508 8511 8546 8622 8666 8667 8709 8873 8874 8875 8876 8877 
8879 8887 8892 8898 8899 8927 8956 8957 8984 8988 8993 9056 9166 9207 9225 9226 
9227 9229 9230 9231 9232 9262 9263 9264 9265 9266 9267 9268 9269 9272 9357 9366 
9369 9371 9378 9381 9390 9392 9393 9402 9403 9409 9416 9417 9426 9430 9431 9432 
9455 9458 9540 9553 9554 9574 9575 9577 9593 9599 9601 9610 9611 9612 9613 9618 
9619 9625 9627 9641 9642 9659 9685 9694 9735 9737 9772 9773 9774 9775 9776 9777 
9781 9795 9802 9803 9806 9827 9829 9844 9866 9867 9875 9914 9949 9954 9957 9958 
9959 9960 9961 9962 9967 9968 9969 9970 9986 9987 998

Re: sieve filter not working

[Thomas Leuxner]

* Michael Williamson  2015.01.22 20:11:

> I have dovecot 2.0.9 running on a CentOS 6.6 email server for a small

This is a notorious version. Not only is it hopelessly outdated, it used to 
contain broken features way back...

> require ["fileinto"];
> # Move spam to spam folder
> if header :contains "X-Spam-Flag" ["YES"] {
>   fileinto "Maildir/.SPAM";
>   stop;
> }

The mail is very awkwardly structured, I had trouble spotting the actual 
question FTR. You may set mail_debug to yes:

$ grep mail_debug *
10-logging.conf:#mail_debug = no

This will then produce verbose logging and most likely will help spotting the 
reason why the mail is not filed. Inject a mail with logging raised and post 
the relevant log excerpt here for further analysis. I have a feeling that it 
should just say fileinto "SPAM", but difficult to say without seeing log output.

Current Dovecot versions also have the benefit of the sieve-test tool which 
lets you apply rules to mailboxes and see what would happen, or refilter mail 
if you're happy with the outcome of the dry-run. 

Regards
Thomas


signature.asc
Description: Digital signature

Re: Changeset c02f29ca104d badly breaks LMTP/mdbox

[Thomas Leuxner]

* Timo Sirainen  2015.01.20 06:42:

> Should be fixed now. I think I'm now done for a while with the new core 
> development work that was needed, so things shouldn't break so easily for a 
> while. Although the body.snippet stuff still needs some more work.

Looks good now after some testing. On a related note: Is there some tool/script 
available which can rebuild indexes in such a scenario? Since a whole 'm.' file 
has been corrupted the only way to mitigate this was restoring from a backup. 
As I don't want to do high frequency snapshots I wonder how to cope with mdbox 
corruptions?

Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Warning: fscking index 
file /var/vmail/public/storage/dovecot.map.index
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Warning: mdbox 
/var/vmail/public/storage: rebuilding indexes
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Error: Corrupted dbox 
file /var/vmail/public/storage/m.132 (around offset=1): Invalid dbox version
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Error: mdbox rebuild: 
Failed to fix file /var/vmail/public/storage/m.132
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Error: Corrupted dbox 
file /var/vmail/public/storage/m.131 (around offset=1): Invalid dbox version
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Error: mdbox rebuild: 
Failed to fix file /var/vmail/public/storage/m.131
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Error: Corrupted dbox 
file /var/vmail/public/storage/m.130 (around offset=1): Invalid dbox version
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Error: mdbox rebuild: 
Failed to fix file /var/vmail/public/storage/m.130
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Error: Corrupted dbox 
file /var/vmail/public/storage/m.127 (around offset=1): Invalid dbox version
Jan 17 10:15:45 nihlus dovecot: lmtp(t...@leuxner.net): Error: mdbox rebuild: 
Failed to fix file /var/vmail/public/storage/m.127
Jan 17 10:15:48 nihlus dovecot: lmtp(48904): Disconnect from local: Successful 
quit


signature.asc
Description: Digital signature

Changeset c02f29ca104d badly breaks LMTP/mdbox

[Thomas Leuxner]

$ gdb /usr/lib/dovecot/lmtp core_lmtp.48883.nihlus.leuxner.net 
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/lib/dovecot/lmtp...Reading symbols from 
/usr/lib/debug/usr/lib/dovecot/lmtp...done.
done.
[New LWP 48883]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `dovecot/lmtp'.
Program terminated with signal 6, Aborted.

#0  0x7f0ca373b165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt full
#0  0x7f0ca373b165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x7f0ca373e3e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2  0x7f0ca3b0ef85 in default_fatal_finish (type=, 
status=status@entry=0) at failures.c:202
backtrace = 0x1385320 "/usr/lib/dovecot/libdovecot.so.0(+0x79f8f) 
[0x7f0ca3b0ef8f] -> /usr/lib/dovecot/libdovecot.so.0(+0x79fee) [0x7f0ca3b0efee] 
-> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f0ca3abebb5] -> 
/usr/lib/d"...
#3  0x7f0ca3b0efee in i_internal_fatal_handler (ctx=0x7fffdfafecb0, 
format=, args=) at failures.c:666
status = 0
#4  0x7f0ca3abebb5 in i_panic (format=format@entry=0x7f0ca3e61438 "file %s: 
line %d (%s): assertion failed: (%s)") at failures.c:276
ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, 
timestamp_usecs = 0}
args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 
0x7fffdfafeda0, reg_save_area = 0x7fffdfafece0}}
#5  0x7f0ca3dc1d7a in mdbox_map_append_finish (ctx=0x142ac90) at 
mdbox-map.c:1153
appends = 0x0
last = 0x142b1d0
count = 
cur_offset = 
__FUNCTION__ = "mdbox_map_append_finish"
#6  0x7f0ca3dc3d08 in mdbox_save_finish_write (_ctx=0x142ab50) at 
mdbox-save.c:205
_data_stack_cur_id = 6
ctx = 0x142ab50
mail = 0x142c4a0
#7  mdbox_save_finish (ctx=0x142ab50) at mdbox-save.c:228
ret = 21152928
#8  0x7f0ca3e070e2 in mailbox_save_finish (_ctx=_ctx@entry=0x7fffdfafee88) 
at mail-storage.c:2145
_data_stack_cur_id = 5
ctx = 0x142ab50
t = 0x1426a80
keywords = 0x0
pvt_flags = 0
copying_via_save = true
ret = 
__FUNCTION__ = "mailbox_save_finish"
#9  0x7f0ca3dfec32 in mail_storage_copy (ctx=0x0, ctx@entry=0x142ab50, 
mail=mail@entry=0x13eaf20) at mail-copy.c:107
No locals.
#10 0x7f0ca3dc42bf in mdbox_copy (_ctx=0x142ab50, mail=0x13eaf20) at 
mdbox-save.c:444
ctx = 0x142ab50
save_mail = 0x13eaf20
src_mbox = 
rec = {map_uid = 21146464, save_date = 0}
guid_data = 0x142ab68
wanted_guid = "\360=B\001\000\000\000\000\340\357E\242\f\177\000"
#11 0x7f0ca245f21b in acl_copy (ctx=0x142ab50, mail=0x13eaf20) at 
acl-mailbox.c:435
t = 
abox = 0x1423df0
#12 0x7f0ca3e0728d in mailbox_copy (_ctx=_ctx@entry=0x7fffdfafefa8, 
mail=mail@entry=0x13eaf20) at mail-storage.c:2215
_data_stack_cur_id = 4
ctx = 0x142ab50
t = 0x1426a80
keywords = 0x0
pvt_flags = 0
backend_mail = 0x13eaf20
ret = 
__FUNCTION__ = "mailbox_copy"
#13 0x7f0ca17ae91e in act_store_execute (action=, 
aenv=0x1421d60, tr_context=0x14220f8) at sieve-actions.c:584
trans = 0x14220f8
mail = 0x13eaf20
save_ctx = 0x0
keywords = 0x0
backends_equal = 
status = 1
#14 0x7f0ca17a5ca5 in sieve_result_transaction_execute (first=, result=) at sieve-result.c:1172
act = 0x1422088
rsef = 
sef = 
rac = 0x1422088
status = 
#15 sieve_result_execute (result=0x1421d30, keep=keep@entry=0x0) at 
sieve-result.c:1387
status = 
result_status = 
first_action = 0x1422088
last_action = 0x0
implicit_keep = true
ret = 
#16 0x7f0ca17b5e45 in sieve_execute (sbin=, 
msgdata=, senv=, 
ehandler=ehandler@entry=0x140cba0, 
flags=flags@entry=SIEVE_RUNTIME_FLAG_NOGLOBAL, keep=keep@entry=0x0) at 
sieve.c:560
result = 0x1421d30
ret = 1
#17 0x7f0ca1a113f5 in lda_sieve_singlescript_execute (srctx=0x7fffdfaff170) 
at lda-sieve-plugin.c:491
sbin = 0x14196a0
rtflags = SIEVE_RUNTIME_FLAG_NOGLOBAL
debug = false
svinst = 0x14092e0
script = 0x140d3c0
user_script = 
cpflags = SIEVE_COMPILE_FLAG_NOGLOBAL
ret = 

Re: HG e3640ccaa76d crashes upon IMAP SEARCH

[Thomas Leuxner]

* Timo Sirainen  2015.01.15 00:24:

> > $ doveconf -n | head
> > # 2.2.15 (e3640ccaa76d): /etc/dovecot/dovecot.conf
> 
> > b search charset utf-8 body "Sleeve"
> > imap(t...@leuxner.net): Panic: file message-decoder.c: line 346 
> > (message_decode_body): assertion failed: (ctx->translation_size <= 
> > sizeof(ctx->translation_buf))
> 
> Hopefully fixed now? Looks like there have been all kinds of small bugs with 
> searching.

Works now with latest commits. Tested with manual approach above and MUAs. 
Thanks!


signature.asc
Description: Digital signature

HG e3640ccaa76d crashes upon IMAP SEARCH

[Thomas Leuxner]

$ doveconf -n | head
# 2.2.15 (e3640ccaa76d): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.6
# OS: Linux 3.16.0-0.bpo.4-amd64 x86_64 Debian 7.8

$ /usr/lib/dovecot/imap -u t...@leuxner.net
* PREAUTH [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT 
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS 
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN 
CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE NOTIFY QUOTA ACL 
RIGHTS=texk] Logged in as t...@leuxner.net
a select "Shops/Amazon"
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $NotJunk NotJunk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Seen \Draft $NotJunk NotJunk \*)] 
Flags permitted.
* 491 EXISTS
* 0 RECENT
* OK [UNSEEN 491] First unseen.
* OK [UIDVALIDITY 1213986801] UIDs valid
* OK [UIDNEXT 568] Predicted next UID
* OK [HIGHESTMODSEQ 1029] Highest
a OK [READ-WRITE] Select completed (0.003 secs).
b search charset utf-8 body "Sleeve"
imap(t...@leuxner.net): Panic: file message-decoder.c: line 346 
(message_decode_body): assertion failed: (ctx->translation_size <= 
sizeof(ctx->translation_buf))
imap(t...@leuxner.net): Error: Raw backtrace: 
/usr/lib/dovecot/libdovecot.so.0(+0x7749f) [0x7ffe68e6d49f] -> 
/usr/lib/dovecot/libdovecot.so.0(default_fatal_handler+0x2a) [0x7ffe68e6d57a] 
-> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7ffe68e1df65] -> 
/usr/lib/dovecot/libdovecot.so.0(message_decoder_decode_next_block+0x688) 
[0x7ffe68e5c858] -> /usr/lib/dovecot/libdovecot.so.0(message_search_more+0xa1) 
[0x7ffe68e60881] -> /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0x5b) 
[0x7ffe68e60a4b] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa9b7b) 
[0x7ffe69187b7b] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x45) 
[0x7ffe6915d675] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa951a) 
[0x7ffe6918751a] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xab76a) 
[0x7ffe6918976a] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0xe2)
 [0x7ffe69189e32] -> /usr/lib/dovecot/modules/lib90_stats_plugin.so(+0x2daa) 
[0x7ffe66dc3daa] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x27) 
[0x7ffe691627a7] -> /usr/lib/dovecot/imap() [0x41da2f] -> 
/usr/lib/dovecot/imap() [0x41e0a2] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0xaa) [0x7ffe68e7f6ca] 
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x69) 
[0x7ffe68e804e9] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x9) 
[0x7ffe68e7f549] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) 
[0x7ffe68e7f5c8] -> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7ffe68e23433] -> /usr/lib/dovecot/imap(main+0x2a1) [0x40c591] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd) [0x7ffe68a88ead] -> 
/usr/lib/dovecot/imap() [0x40c6f5]

I tried to make it dump a core but failed...

Regards
Thomas


signature.asc
Description: Digital signature

Different behavior of ACLs in MUA and doveadm

[Thomas Leuxner]

I have noticed a difference in the behavior of ACLs. When used in a MUA the 
following global ACL works fine and has the desired effect - new mailboxes can 
be created by a user being part of the 'PublicMailboxAdmins' group: 

[ global-acl: ]
INBOX owner lrwstiekxap
Public/* group=PublicMailboxAdmins lrwsipk
Public/* anyone lr
Public/* authenticated lrws

Creating the same mailbox via doveadm however fails with a permission problem:

doveadm(t...@leuxner.net): Debug: acl vfile: Global ACL file: 
/var/vmail/conf.d/leuxner.net/global-acl
doveadm(t...@leuxner.net): Debug: Namespace : type=public, prefix=Public/, 
sep=/, inbox=no, hidden=no, list=yes, subscriptions=no 
location=mdbox:/var/vmail/public:INDEXPVT=~/mdbox/public
doveadm(t...@leuxner.net): Debug: fs: root=/var/vmail/public, index=, 
indexpvt=/var/vmail/domains/leuxner.net/tlx/mdbox/public, control=, inbox=, alt=
doveadm(t...@leuxner.net): Debug: acl: initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
doveadm(t...@leuxner.net): Debug: acl: acl username = t...@leuxner.net
doveadm(t...@leuxner.net): Debug: acl: owner = 0
doveadm(t...@leuxner.net): Debug: acl vfile: Global ACL file: 
/var/vmail/conf.d/leuxner.net/global-acl
doveadm(t...@leuxner.net): Debug: Namespace : type=private, prefix=Virtual/, 
sep=/, inbox=no, hidden=no, list=yes, subscriptions=yes 
location=virtual:~/mdbox/virtual
doveadm(t...@leuxner.net): Debug: fs: 
root=/var/vmail/domains/leuxner.net/tlx/mdbox/virtual, index=, indexpvt=, 
control=, inbox=, alt=
doveadm(t...@leuxner.net): Debug: acl: initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
doveadm(t...@leuxner.net): Debug: acl: acl username = t...@leuxner.net
doveadm(t...@leuxner.net): Debug: acl: owner = 1
doveadm(t...@leuxner.net): Debug: acl vfile: Global ACL file: 
/var/vmail/conf.d/leuxner.net/global-acl
doveadm(t...@leuxner.net): Debug: acl vfile: file 
/var/vmail/public/mailboxes/dovecot-acl not found
doveadm(t...@leuxner.net): Error: Can't create mailbox 
Public/Archive/Newsletters/heise-security/2014: Permission denied

Interestingly, doveadm succeeds when dovecot-acl is present in the namespace 
root - which of course is not desirable in the light of the global ACL:

[ dovecot-acl: ] 
group=PublicMailboxAdmins lrwsipk

doveadm(t...@leuxner.net): Debug: acl vfile: Global ACL file: 
/var/vmail/conf.d/leuxner.net/global-acl
doveadm(t...@leuxner.net): Debug: Namespace : type=public, prefix=Public/, 
sep=/, inbox=no, hidden=no, list=yes, subscriptions=no 
location=mdbox:/var/vmail/public:INDEXPVT=~/mdbox/public
doveadm(t...@leuxner.net): Debug: fs: root=/var/vmail/public, index=, 
indexpvt=/var/vmail/domains/leuxner.net/tlx/mdbox/public, control=, inbox=, alt=
doveadm(t...@leuxner.net): Debug: acl: initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
doveadm(t...@leuxner.net): Debug: acl: acl username = t...@leuxner.net
doveadm(t...@leuxner.net): Debug: acl: owner = 0
doveadm(t...@leuxner.net): Debug: acl vfile: Global ACL file: 
/var/vmail/conf.d/leuxner.net/global-acl
doveadm(t...@leuxner.net): Debug: Namespace : type=private, prefix=Virtual/, 
sep=/, inbox=no, hidden=no, list=yes, subscriptions=yes 
location=virtual:~/mdbox/virtual
doveadm(t...@leuxner.net): Debug: fs: 
root=/var/vmail/domains/leuxner.net/tlx/mdbox/virtual, index=, indexpvt=, 
control=, inbox=, alt=
doveadm(t...@leuxner.net): Debug: acl: initializing backend with data: 
vfile:/var/vmail/conf.d/leuxner.net/global-acl:cache_secs=300
doveadm(t...@leuxner.net): Debug: acl: acl username = t...@leuxner.net
doveadm(t...@leuxner.net): Debug: acl: owner = 1
doveadm(t...@leuxner.net): Debug: acl vfile: Global ACL file: 
/var/vmail/conf.d/leuxner.net/global-acl
doveadm(t...@leuxner.net): Debug: acl vfile: reading file 
/var/vmail/public/mailboxes/dovecot-acl
doveadm(t...@leuxner.net): Debug: Namespace Public/: 
/var/vmail/public/mailboxes/Archive/Newsletters/heise-security/2014 doesn't 
exist yet, using default permissions
doveadm(t...@leuxner.net): Debug: Namespace Public/: Using permissions from 
/var/vmail/public: mode=0700 gid=default
doveadm(t...@leuxner.net): Debug: acl vfile: file 
/var/vmail/public/mailboxes/Archive/Newsletters/heise-security/dbox-Mails/dovecot-acl
 not found
doveadm(t...@leuxner.net): Debug: acl vfile: file 
/var/vmail/public/mailboxes/Archive/Newsletters/heise-security/2014/dbox-Mails/dovecot-acl
 not found
doveadm(t...@leuxner.net): Debug: acl vfile: file 
/var/vmail/public/mailboxes/Archive/Newsletters/heise-security/2014/dbox-Mails/dovecot-acl
 not found
doveadm(t...@leuxner.net): Debug: acl vfile: file 
/var/vmail/public/mailboxes/Archive/Newsletters/heise-security/2014/dbox-Mails/dovecot-acl
 not found

# 2.2.15 (6078354e6238): /etc/dovecot/dovecot.conf

I know there have been some changes in Mercurial as to how global ACLs are 
interpreted. Is doveadm probably behind on them? 

Regards
Thomas


signature.as

Re: Execution of sieve script failed HG 3db5fbb216d3

[Thomas Leuxner]

* Stephan Bosch  2014.11.13 10:19:

> Yes, it does. Fixed by last two commits.

Thanks, Stephan and Timo :)


signature.asc
Description: Digital signature

Re: Execution of sieve script failed HG 3db5fbb216d3

[Thomas Leuxner]

* Timo Sirainen  2014.11.13 09:46:

> Did you (fully) recompile Sieve also? Some structs changed. I can't really 
> think of anything else and it works ok in my tests.

Yes I did. By a hunch I now deleted the binary script and that does seem to do 
the trick...


signature.asc
Description: Digital signature

Execution of sieve script failed HG 3db5fbb216d3

[Thomas Leuxner]

# 2.2.15 (3db5fbb216d3): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.6
[...]

The latest HG commits produce the following sieve execution error. There is no 
error log created though...

Nov 13 08:54:11 nihlus dovecot: lmtp(t...@leuxner.net): kFEQG6JjZFQZBAAAgUOSbA: 
sieve: msgid=: stored 
mail into mailbox 'INBOX'
Nov 13 08:54:11 nihlus dovecot: lmtp(t...@leuxner.net): kFEQG6JjZFQZBAAAgUOSbA: 
sieve: Execution of script 
/var/vmail/domains/leuxner.net/tlx/.dovecot.sieve;name=dovecot failed, but 
implicit keep was successful (user logfile 
/var/vmail/domains/leuxner.net/tlx/.dovecot.sieve.log may reveal additional 
details)

Regards
Thomas


signature.asc
Description: Digital signature

Re: Corrupted SSL parameters file in state_dir with HG 267bca7a62fb

[Thomas Leuxner]

* Timo Sirainen  2014.11.01 02:12:

> Fixed: http://hg.dovecot.org/dovecot-2.2/rev/6efd7ab25b71

Looks good now. Thanks


signature.asc
Description: Digital signature

Corrupted SSL parameters file in state_dir with HG 267bca7a62fb

[Thomas Leuxner]

Hi,

with the latest HG 267bca7a62fb the following error started to appear in the 
logs:

Oct 31 09:39:07 nihlus dovecot: master: Dovecot v2.2.15 (267bca7a62fb) starting 
up for imap, lmtp
[...]
Oct 31 10:10:52 nihlus dovecot: lmtp(20876): Error: Corrupted SSL parameters 
file in state_dir: ssl-parameters.dat - disabling SSL 360
Oct 31 10:10:52 nihlus dovecot: lmtp(20876): Error: Couldn't initialize SSL 
parameters, disabling SSL
Oct 31 10:10:52 nihlus dovecot: lmtp(20876): Connect from local

This most likely has been introduced with a commit after the previous build 
installed (aa5dde56424f). I did not find options to disable SSL for LMTP 
either, as in my setup I'm using a UNIX socket.

Regards
Thomas


signature.asc
Description: Digital signature

Re: virtualfolders under mdbox storage backend

[Thomas Leuxner]

* Jesus Cea  2014.10.01 14:37:

> Reading docs in http://wiki2.dovecot.org/Plugins/Virtual I wonder how is
> that used if I am using MDBOX as my storage backend. The only references
> there are about Maildir.

You basically need to configure the path where it stores its indexes e.g.

namespace {
  location = virtual:~/mdbox/virtual
  prefix = Virtual/
  separator = /
}

Regards
Thomas


signature.asc
Description: Digital signature

Re: virtual plugin in HG unaware of namespaces

[Thomas Leuxner]

* Thomas Leuxner  2014.08.17 12:58:

> While flagged messages inside the mailbox still show up, messages flagged in 
> 'Public' are not picked up anymore.

The 'Public' messages actually show up again if you toggle their flag off/on in 
the mail client...


signature.asc
Description: Digital signature

virtual plugin in HG unaware of namespaces

[Thomas Leuxner]

The latest changes to the virtual plugin seem to have a negative effect on name 
space rules:

$ cat dovecot-virtual 
*
Public/*
  flagged

While flagged messages inside the mailbox still show up, messages flagged in 
'Public' are not picked up anymore.

Regards
Thomas


signature.asc
Description: Digital signature