Re: File system permissions - setgid bit and Netapp NFS volumes
On 17.04.2018 01:42, Shaun Johnson wrote: > On Fri, 23 Mar 2018 09:53:00 -0700 > Shaun Johnson wrote: > >> Greetings Dovecot List, >> >> I have a bit of an edge case I am trying to resolve. I am currently >> using dovecot on Ubuntu 14.04 - Ubuntu package version: >> >> 1:2.2.9-1ubuntu2.3 >> >> I have attached the output of doveconf -n to this email - but to >> describe the configuration in a nutshell: >> >> my server is configured to use Maildir storage >> >> I do not use dovecot delivery service (there is a separate program >> in my system delivering messages) >> >> I am using a 'checkpassword' mechanism for authentication >> >> My mail storage is mounted via NFSv3 from a Netapp Filer FAS3050 >> configured to set anonuid=89, read write access, and root access. >> >> Dovecot has been configured with: >> >> mail_fsync = always >> mail_nfs_index = yes >> mail_nfs_storage = yes >> mmap_disable = yes >> lock_method = fcntl >> >> In order to function with the other components of my system, the >> Mail Storage directories are expected to have permissions: >> >> drwx--S--- uid 89: gid 89 >> >> The oddity I am encountering came up recently when I set up a job to >> monitor and correct directory permissions. I found after observing >> the system in action that overtime, the ~/Maildir directories for all >> active mailboxes would mysteriously lose the setgid bit. Now, while >> the system still functioned, it was something unexpected so I >> investigated further by turning on auditd to monitor what was changing >> the permissions on the path. After tracing through the results, I >> found an entry indicating that the 'imap' process of dovecot was >> performing a 'chown' call on the directory, with an end result of the >> previous mode 2700 being changed to 0700. >> >> Now I understand that on some filesystems, a 'chown' call can end up >> losing any setuid or setgid bits on an inode - and apparently Netapp's >> WAFL file system is one of them. >> >> Looking at the dovecot source code, I *believe* the source of the >> chown call is in src/lib/nfs-workarounds.c function >> nfs_flush_chown_uid - where a 'chown' call is being used to flush NFS >> attribute caches? for what I believe is the purposes of ascertaining >> file lock status? >> >> My question for you folks is if anyone else has encountered this on >> other file systems (or filers) - and if there is a native method in >> dovecot - either the version I am currently running or an updated >> newer version - that allows one to either retain the permissions >> assigned to a directory or specify the permissions I expect on the >> Maildir directory structures. >> >> Any feedback would be welcome, thanks! >> >> > Attached is a quick patch that I wrote up that solves the issue I > encountered. Ideally I would like to have this logic take place > dependent on if a configuration setting were set - something like > mail_nfs_preserve_permissions = yes - but I found that in order to > expose such a setting to the nfs-workarounds.c code base would require > significantly more modifications than I was comfortable with. If > anyone has any thoughts on a simple way to make this configurable, I am > open to suggestions. > > > Thanks! > > - Shaun Hi! Thank you for your patch, we would appreciate if you'd be able to open it on https://github.com/dovecot/core against dovecot master branch. Aki
Corrupted dbox file around offset= EOF reading msg header
Hello Dovecoters, I am using dovecot-2.2.10-8.el7.x86_64 on RHEL 7.4 When I backup mdboxes (using SIS) using this command doveadm -Dv -o plugin/quota= sync -1f -u us...@user.com mdbox:/mailbackup/user1 mails get backedup but all m files are not copied. Last file copied is m.4 If I run the command again I get below error I tried with doveadm backup command as well (many times). Error: Corrupted dbox file / mailbackup/user1/storage/m.4 (around offset=3732536): EOF reading msg header (got 0/30 bytes) Result is all mails are not backedup. Destination is a folder on a NAS with NFS mapped to the server. Thanks/DP --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus
Re: Lmtp issues on dovecot 2.3.x with big messages
Op 16/04/2018 om 19:57 schreef Michael Tratz: Messages are being sent to dovecot LMTP by postfix. If I change this email to another server with dovecot 2.2.x the same message are delivered immediately. Confirmed. Starts to fail here around 30Mb. Tested with Swaks. Working on a fix... Problem found. It is an explicit limit of 40Mb (for the 30Mb I saw in my tests, there was also a base64 encoding I forgot about). Will fix both the unhelpful error and the fact that there should be no limit (currently) for LMTP. Stephan, I’m having the same issue with LMTP and large messages with dovecot 2.3.x. Can you tell me which commit fixes this issue? I looked through the GitHub commits, but it’s not clear to me which commit fixes this issue. I’d like to apply the patch and re-compile dovecot for my machines instead of having to downgrade to 2.2.35. This fix is currently pending: https://github.com/stephanbosch/dovecot-core/commits/lmtp-fix-msg-size-limit I'm seeing some CI issues still and I am waiting on some input from someone else. Regards, Stephan.
Re: File system permissions - setgid bit and Netapp NFS volumes
On Fri, 23 Mar 2018 09:53:00 -0700
Shaun Johnson wrote:
> Greetings Dovecot List,
>
> I have a bit of an edge case I am trying to resolve. I am currently
> using dovecot on Ubuntu 14.04 - Ubuntu package version:
>
> 1:2.2.9-1ubuntu2.3
>
> I have attached the output of doveconf -n to this email - but to
> describe the configuration in a nutshell:
>
> my server is configured to use Maildir storage
>
> I do not use dovecot delivery service (there is a separate program
> in my system delivering messages)
>
> I am using a 'checkpassword' mechanism for authentication
>
> My mail storage is mounted via NFSv3 from a Netapp Filer FAS3050
> configured to set anonuid=89, read write access, and root access.
>
> Dovecot has been configured with:
>
> mail_fsync = always
> mail_nfs_index = yes
> mail_nfs_storage = yes
> mmap_disable = yes
> lock_method = fcntl
>
> In order to function with the other components of my system, the
> Mail Storage directories are expected to have permissions:
>
> drwx--S--- uid 89: gid 89
>
> The oddity I am encountering came up recently when I set up a job to
> monitor and correct directory permissions. I found after observing
> the system in action that overtime, the ~/Maildir directories for all
> active mailboxes would mysteriously lose the setgid bit. Now, while
> the system still functioned, it was something unexpected so I
> investigated further by turning on auditd to monitor what was changing
> the permissions on the path. After tracing through the results, I
> found an entry indicating that the 'imap' process of dovecot was
> performing a 'chown' call on the directory, with an end result of the
> previous mode 2700 being changed to 0700.
>
> Now I understand that on some filesystems, a 'chown' call can end up
> losing any setuid or setgid bits on an inode - and apparently Netapp's
> WAFL file system is one of them.
>
> Looking at the dovecot source code, I *believe* the source of the
> chown call is in src/lib/nfs-workarounds.c function
> nfs_flush_chown_uid - where a 'chown' call is being used to flush NFS
> attribute caches? for what I believe is the purposes of ascertaining
> file lock status?
>
> My question for you folks is if anyone else has encountered this on
> other file systems (or filers) - and if there is a native method in
> dovecot - either the version I am currently running or an updated
> newer version - that allows one to either retain the permissions
> assigned to a directory or specify the permissions I expect on the
> Maildir directory structures.
>
> Any feedback would be welcome, thanks!
>
>
Attached is a quick patch that I wrote up that solves the issue I
encountered. Ideally I would like to have this logic take place
dependent on if a configuration setting were set - something like
mail_nfs_preserve_permissions = yes - but I found that in order to
expose such a setting to the nfs-workarounds.c code base would require
significantly more modifications than I was comfortable with. If
anyone has any thoughts on a simple way to make this configurable, I am
open to suggestions.
Thanks!
- Shaun
diff -urN dovecot-2.2.35.ori/src/lib/nfs-workarounds.c dovecot-2.2.35/src/lib/nfs-workarounds.c
--- dovecot-2.2.35.ori/src/lib/nfs-workarounds.c 2018-03-19 02:27:54.0 -0700
+++ dovecot-2.2.35/src/lib/nfs-workarounds.c 2018-04-16 15:35:42.0 -0700
@@ -179,6 +179,14 @@
}
i_error("nfs_flush_chown_uid: chown(%s) failed: %m", path);
}
+#ifndef ATTRCACHE_FLUSH_CHOWN_UID_1
+if (chmod(path, st.st_mode & 0) < 0) {
+i_error("nfs_flush_chown_uid: chmod(%s, %04o) failed: %m",
+path,
+st.st_mode & 0);
+return;
+}
+#endif
}
#ifdef __FreeBSD__
@@ -210,6 +218,13 @@
i_error("nfs_flush_attr_cache_fd_locked: fchown(%s) failed: %m",
path);
}
+#ifndef ATTRCACHE_FLUSH_CHOWN_UID_1
+if (fchmod(fd, st.st_mode & 0) < 0) {
+i_error("nfs_flush_attr_cache_fd_locked: fchown(%s, %04o) failed: %m",
+path,
+st.st_mode & 0);
+}
+#endif
return TRUE;
}
#endif
Re: LDAP Homedir location: Needs dovecot restart after change it
Hello Andre, try to flush the auth cache: doveadm auth cache flush u...@example.com or: doveadm auth cache flush Best regards, Gerald > Am 16.04.2018 um 20:39 schrieb Andre Luiz Paiz : > > Dear group members. > > I work with Dovecot and Openldap authentication. Sometimes users change > departments and we need to alter their homedir location. Every time this > process is needed, I perform this steps: > > 1 - Change homedir location in openldap > 2 - Move homedir folder to the new location > 3 - Re-apply permissions > 4 - Remove user index folder > > After I do that, users cannot authenticate unless I restart dovecot, process > that I would like to avoid. Can you guys give a tip on what I need to change > to avoid this last problematic step? After the restart, everything works. > > Does the auth_cache feature also store the homedir location?
Is FTS needed for Subject/Recipient/Sender server-side searches?
I've had a number of Dovecot installs happily running for quite a few years now, but I've just stumbled over a new issue. I am trying to get some iPhones to do server-side email searches through ActiveSync/Exchange protocol -> Horde -> Dovecot. The bit I can't figure out is if I need to enable and configure FTS in Dovecot for that to work? I've tested server-side email searches against the same Dovecot server on Thunderbird - through IMAP (there is an option to force the search on the server in Thunderbird) - and it is working really well and fast. I am only really interested in Sender/Recipient/Subject searches. Server-side searching is not working at the moment on the iPhones side - although I can see the ActiveSync commands being sent to Horde. Anyway, to begin with, does anybody know if server-side IMAP searches need FTS in Dovecot (Horde is talking to Dovecot through IMAP). The fact that they work fine in Thunderbird would suggest the answer is 'no' - but could someone confirm that is correct please? Anticipated thanks for any hints. I am on Dovecot 2.2.33 if in case it matters.
LDAP Homedir location: Needs dovecot restart after change it
Dear group members.
I work with Dovecot and Openldap authentication. Sometimes users
change departments and we need to alter their homedir location. Every
time this process is needed, I perform this steps:
1 - Change homedir location in openldap
2 - Move homedir folder to the new location
3 - Re-apply permissions
4 - Remove user index folder
After I do that, users cannot authenticate unless I restart dovecot,
process that I would like to avoid. Can you guys give a tip on what I
need to change to avoid this last problematic step? After the restart,
everything works.
Does the auth_cache feature also store the homedir location?
System: CentOS 7
Versions:
dovecot-pigeonhole-2.2.10-8.el7.x86_64
dovecot-2.2.10-8.el7.x86_64
Doveconf -n:
# 2.2.10: /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-693.5.2.el7.x86_64 x86_64 CentOS Linux release
7.4.1708 (Core)
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_mechanisms = plain login
auth_username_format = %Ln
default_client_limit = 1024
default_process_limit = 1024
default_vsz_limit = 1 G
listen = *
mail_location = maildir:~/Maildir:INDEX=/var/indexes/dovecot/%u
mail_plugins = " mail_log notify quota"
mail_temp_dir = /var/tmp
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave imapflags notify
mbox_write_locks = fcntl
mmap_disable = yes
namespace inbox {
inbox = yes
location =
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Sent {
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
special_use = \Trash
}
prefix =
}
passdb {
driver = pam
}
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
mail_log_fields = uid box msgid size
quota = maildir
quota_exceeded_message = Cota de armazenamento de e-mails
excedida. Quota exceeded for mail store.
quota_warning = storage=95%% quota-warning 95 %u
quota_warning2 = storage=85%% quota-warning 85 %u
quota_warning3 = storage=75%% quota-warning 75 %u
sieve = ~/.dovecot.sieve
sieve_before = /var/lib/dovecot/sieve/default.sieve
sieve_default = /var/lib/dovecot/sieve/default.sieve
sieve_dir = ~/sieve
sieve_extensions = +notify +imapflags
}
protocols = imap pop3 lmtp sieve
service anvil {
client_limit = 14100
process_limit = 1
}
service auth-worker {
client_limit = 1
process_limit = 50
service_count = 1
}
service auth {
client_limit = 15100
process_limit = 1
}
service dict {
client_limit = 1
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_limit = 6000
process_min_avail = 10
service_count = 1
vsz_limit = 256 M
}
service imap {
process_limit = 3000
vsz_limit = 1 G
}
service lmtp {
client_limit = 1
inet_listener lmtp {
port = 24
}
unix_listener lmtp {
mode = 0666
}
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service pop3-login {
inet_listener pop3 {
port = 110
}
inet_listener pop3s {
port = 995
ssl = yes
}
process_limit = 6000
process_min_avail = 10
service_count = 1
}
service pop3 {
process_limit = 3000
}
service quota-warning {
executable = script /usr/local/bin/quota-warning.sh
unix_listener quota-warning {
mode = 0666
user = dovecot
}
user = root
}
ssl = required
ssl_cert = **
ssl_key = ***
userdb {
driver = passwd
}
userdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
protocol lmtp {
mail_plugins = " mail_log notify quota sieve quota"
}
protocol imap {
mail_plugins = " mail_log notify quota imap_quota"
}
Thanks
Andre
Re: [bug] Auth cache, proxy and ITERINDEX
Hi, please post your doveconf -n Sami > On 16 Apr 2018, at 20.17, azu...@pobox.sk wrote: > > Hi, > > today we have exceprienced really major difficulties with our proxy backend > servers. Everything started after enabling auth cache: > auth_cache_size = 100M > auth_cache_verify_password_with_worker = yes > auth_cache_ttl = 1 hour > auth_cache_negative_ttl = 1 hour > > Soon after we start receiving lots of calls from our customers about 'missing > all e-mail messages'. This was in errors logs on master server: > > Apr 16 14:37:10 server00 dovecot: imap(): Error: > autoexpunge: Couldn't create dovecot.autoexpunge.lock lock: > file_create_locked(/var/mail/vhosts///home/dovecot.autoexpunge.lock) > failed: > safe_mkstemp(/var/mail/vhosts///home/dovecot.autoexpunge.lock) > failed: No such file or directory > > Looks like that all users, who are placed on proxy backends, were logged > correctly BUT not proxied to the right server so they saw empty mailboxes > (Dovecot also created some directories on master server). What is worse, > their email software deleted all local emails so they are now downloading > everything from scratch (100Gs of mails so it will take some time). > > After auth cache was disabled, things started to go to normal EXCEPT users > were reporting they don't see all folders - which was true, only few on them > were visible (INBOX, trash, sent etc.). We tried everything to make them > visible again, for example deleting indexes, but nothing helped. Finally, > removing option ITERINDEX from mail_location helped. > > Can anyone explain what happened? Thank you > > azurIt > >
Re: Lmtp issues on dovecot 2.3.x with big messages
> >> Messages are being sent to dovecot LMTP by postfix. If I change this email > >> to another server with dovecot 2.2.x the same message are delivered > >> immediately. > > Confirmed. Starts to fail here around 30Mb. Tested with Swaks. > > > > Working on a fix... > > Problem found. It is an explicit limit of 40Mb (for the 30Mb I saw in my > tests, there was also a base64 encoding I forgot about). > > Will fix both the unhelpful error and the fact that there should be no > limit (currently) for LMTP. > Stephan, I’m having the same issue with LMTP and large messages with dovecot 2.3.x. Can you tell me which commit fixes this issue? I looked through the GitHub commits, but it’s not clear to me which commit fixes this issue. I’d like to apply the patch and re-compile dovecot for my machines instead of having to downgrade to 2.2.35. Thanks, Michael
[bug] Auth cache, proxy and ITERINDEX
Hi, today we have exceprienced really major difficulties with our proxy backend servers. Everything started after enabling auth cache: auth_cache_size = 100M auth_cache_verify_password_with_worker = yes auth_cache_ttl = 1 hour auth_cache_negative_ttl = 1 hour Soon after we start receiving lots of calls from our customers about 'missing all e-mail messages'. This was in errors logs on master server: Apr 16 14:37:10 server00 dovecot: imap(): Error: autoexpunge: Couldn't create dovecot.autoexpunge.lock lock: file_create_locked(/var/mail/vhosts///home/dovecot.autoexpunge.lock) failed: safe_mkstemp(/var/mail/vhosts///home/dovecot.autoexpunge.lock) failed: No such file or directory Looks like that all users, who are placed on proxy backends, were logged correctly BUT not proxied to the right server so they saw empty mailboxes (Dovecot also created some directories on master server). What is worse, their email software deleted all local emails so they are now downloading everything from scratch (100Gs of mails so it will take some time). After auth cache was disabled, things started to go to normal EXCEPT users were reporting they don't see all folders - which was true, only few on them were visible (INBOX, trash, sent etc.). We tried everything to make them visible again, for example deleting indexes, but nothing helped. Finally, removing option ITERINDEX from mail_location helped. Can anyone explain what happened? Thank you azurIt
