Re: Error: mremap_anon(###) failed: Cannot allocate memory
On 12/08/2014 03:07, Teemu Huovila wrote:
A config would always be useful, but I can venture a guess. Perhaps the
affected users have a dovecot.index.cache file
somehwere, e.g. under INBOX, that is larger than the memory limit for
the lmtp process. Try increasing default_vsz_limit or
the service lmtp { vsz_limit }. Removing the overly large index cache
file should also, temporarily, help. In case you do
not get this error from the imap/pop3 processes, perhaps you have
already set a higher vsz_limit for those?
Teemu,
Thanks for your suggestion. I checked the output of doveconf, and by
default it appears the vsz_limit is set to 18446744073709551615B for
each of the services, and 256M for default_vsz_limit.
I checked a user in question, and their index.cache was indeed large,
123M. Seemingly needlessly so, as I deleted the dovecot files and
reindexed, and now it's 6K.
Thanks, I'll keep an eye on the users this affects and try to get their
index.cache in order.
Thanks,
Andy
Error: mremap_anon(###) failed: Cannot allocate memory
We're running dovecot 2.2.15 with pigeonhole 0.4.6, in a clustered environment, nfs with proxy and backend on all servers. I've been seeing some odd errors from lmtp: Error: mremap_anon(127930368) failed: Cannot allocate memory It seems to affect specific users, but it doesn't seem to manifest in any particular way; no user complaints. Just the occasional log message. I would guess this is a bug? I'm open to suggestions and I'd be happy to post config if somebody has an idea. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Possible to adjust username used to determine the proxy destination?
I'm in a fairly standard cluster environment: shared storage, bunch of
servers each acting as both proxies and backends.
We do /bin/checkpassword authentication, allowing a great deal of
flexibility...protection against brute force, billing mechanisms, but
relevant to this issue, I have it set up to allow users to login with
either their username (if they are in one of our default domains) or their
email address.
I'm realizing now that as a consequence of this, joe and j...@xecu.net
are unique as far as dovecot is concerned. Users who login with just their
username (and not the full email address) can get assigned to a different
backend server than when they login with the full email address (which
would also include LMTP deliveries). This has been happening for years, a
few broken indexes here and there that seem to resolve themselves, so it
hasn't been impacting the service, but I'd like to correct it properly.
Is there a way to manipulate this? For example, if I moved the
authentication to the proxy layer (it's currently proxy=y nopassword=y),
and set $ENV{USER} to the full email address, will director use that for
selection instead of the user-supplied username?
I'm open to suggestions on how best to accomplish this.
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Re: v2.2.14 released
2.2.14 is working great in production for us. No more strange errors with indexes[1] (which was a constant thing in 2.2.13), and I'm happy that the clustered LDA issues are resolved. Well done Timo, and congratulations on your relocation to the US! Andy [1] Panic: file mail-index-transaction-export.c: line 203 (log_append_ext_hdr_update): assertion failed: (u32.offset + u32.size = ext_hdr_size) --- - ANDY DILLS - XECUNET, LLC 5744-R Industry Lane Frederick MD 21704 www.xecu.net [1] P: 301-682-9972 P: 1-877-XECUNET F: 240-215-0351 Twitter [2] Facebook [3] - Links: -- [1] http://www.xecu.net/ [2] https://twitter.com/Xecunet [3] http://www.facebook.com/xecunet
Re: LDA randomly failing to write email to disk
On Fri, 3 Oct 2014, Timo Sirainen wrote: This is fixed in hg. I guess I'll just have to make v2.2.14 release soon. Thanks Timo, I had given up hope, and was starting to question if maybe I was having hardware issues. Should I feel hesitant about rolling out a fresh build from hg into production? Would I be better off waiting for an official 2.2.14? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
LDA randomly failing to write email to disk
We're using 2.2.13 with pigeonhole 0.4.3, in a clustered environment (maildir on netapp, dual dovecot instances where each server is both a proxy and a backend). Every now and then (once a month per user, maybe?), users will see a blank email in their inbox. Investigating further, and we will see that the only information recorded in the maildir file for the message is the Return-Path, the Delivered-To, and the first Received line (the one generated by the local LDA via LMTP). For example, here is what I found in one such email today: Return-Path: x...@xecu.net Delivered-To: y...@xecu.net Received: from mail5.xecu.net ([10.0.1.85]) by mail2.xecu.net (Dovecot) with LMTP idX86eBjgaI1RdyQAA3SxDBg for y...@xecu.net; Wed, 24 Sep 2014 15:24:20 -0400 Everything past that is lost, as if the either the LDA on mail2 exited early or was never sent the information from the dovecot instance on mail5. Here is a little more detail from the header of the same email, different recipient (all other recipients received the email properly, only one failed to receive properly): Return-Path: x...@xecu.net Delivered-To: y...@xecu.net Received: from mail5.xecu.net ([10.0.1.85]) by mail1.xecu.net (Dovecot) with LMTP id KPh8ItMXI1StBAAA/c3zFg for y...@xecu.net; Wed, 24 Sep 2014 15:24:20 -0400 Received: from mail5.xecu.net by mail5.xecu.net (Dovecot) with LMTP id QsUBFoQZI1RjhgAAXyr1JQ ; Wed, 24 Sep 2014 15:24:20 -0400 Received: from mail5.xecu.net (localhost [127.0.0.1]) by mail5.xecu.net (Postfix) with ESMTP id 30AAE323BB18; Wed, 24 Sep 2014 15:24:20 -0400 (EDT) ... This is what I see in the logs of mail2, which is where the failure happened: Sep 24 15:24:20 mail2 dovecot: lmtp(51549, y...@xecu.net): X86eBjgaI1RdyQAA3SxDBg: sieve: msgid=unspecified: stored mail into mailbox 'INBOX' When I look at the logs of mail1, which is where the successful delivery happened: Sep 24 15:24:20 mail1 dovecot: lmtp(1197, y...@xecu.net): KPh8ItMXI1StBAAA/c3zFg: sieve: msgid=20140924192412.1435.qm...@xxx.xecu.net: stored mail into mailbox 'INBOX' Nothing of note seems to be logged on mail5 (only a message from postfix, nothing from the proxy side of the lmtp). I do notice, when I check for the PID of 51549 in the logs, all of its other transactions seem to register with proper msgids and were delivered fine. Also, I notice plenty of other messages that have the msgid=unspecified error, but which were delivered with no problems and not truncated, so I'm suspecting what may be happening is that somehow the backend instance is not receiving the actual data portion, and only getting the envelope from proxy instance. How do I approach debugging this? It's very infrequent, but yet quite annoying. Seems to have started since we upgraded to 2.2.13 (from an older 2.1 build) earlier this year. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Outlook 2007 2010 hangs in v2.2?
On 06/09/2014 05:21, Timo Sirainen wrote: On 5.6.2014, at 17.38, Timo Sirainen t...@iki.fi wrote: Has anybody noticed Outlook 2007 2010 (but apparently not 2013) hanging IMAP connections with Dovecot v2.2 (but not v2.1) when they're FETCHing large mails? I can't think of any reasonable explanation for this. Most likely solved by: http://hg.dovecot.org/dovecot-2.2/rev/6a9508d28d34 [1] Strange that it didn't break more commonly or that more people weren't complaining about it.. The bug has been there since v2.2.7. Thank you Timo. This patch did indeed correct the problem, we had no problem doing full syncs with the problematic accounts after implementing this. Is it perhaps something that only happens with directory/proxy environments? I'm surprised it hasn't been mentioned before, I figured it was something particular to our implementation since nobody else was mentioning it, until you did. Fascinating to me that it only affects Outlook as well. I don't understand the code well enough to grasp the impact of the bug, so if somebody has a minute to share an explanation that would be cool. Andy - ANDY DILLS - XECUNET, LLC 5744-R Industry Lane Frederick MD 21704 www.xecu.net [2] P: 301-682-9972 P: 1-877-XECUNET F: 240-215-0351 Twitter [3] Facebook [4] - Links: -- [1] http://hg.dovecot.org/dovecot-2.2/rev/6a9508d28d34 [2] http://www.xecu.net/ [3] https://twitter.com/Xecunet [4] http://www.facebook.com/xecunet
Re: [Dovecot] Outlook 2007 2010 hangs in v2.2?
We just upgraded from 2.1.16 to 2.2.13, and we have been have horrendous troubles with outlook since the upgrade. For users with large mailboxes, they are unable to sync their folders. Perhaps this is specifically because of the problem you are mentioning Timo. The only solution we could come up with for now was to configure outlook to only fetch headers and not bodies, when syncing. For the most part, bringing up individual messages is fine, haven't seen that fail. So, with that tweak to the outlook config, things are working error-free again for our users...but it was certainly an unexpected situation to tackle. Also, we found a marked improvement connecting via SSL on 993 than we do unencrypted on 143. I can definitely confirm 100% there is a regression in dovecot 2.2 that severely impacts the performance of outlook (but works great with everything else). Let me know if I can help you track this down Timo. Andy Sent from my iPhone On Jun 5, 2014, at 1:40 PM, Timo Sirainen t...@iki.fi wrote: On 5.6.2014, at 20.23, Robert Schetterer r...@sys4.de wrote: Am 05.06.2014 17:02, schrieb Timo Sirainen: On 5.6.2014, at 17.41, Martin Rabl martin.r...@rablnet.de wrote: Am 05.06.2014 16:38, schrieb Timo Sirainen: Has anybody noticed Outlook 2007 2010 (but apparently not 2013) hanging IMAP connections with Dovecot v2.2 (but not v2.1) when they're FETCHing large mails? I can't think of any reasonable explanation for this. how large? I don't know yet if the mail size is even relevant. At least one hang was caused by downloading ~55 MB mail where it stopped just before the last 400 kB. I find it strange that v2.2 has been out for a long time and nobody before this complained about any hangs. 55 MB mail may are rare ,i can test it ,but it will need some time ( old outlook install on clean new windows system) perhaps anyone has an existing setup which could do it faster Happens also with smaller mails, for example 3 MB and I think there were also even smaller ones like 1 MB. I see that once Outlook tried to download the same 3 MB mail 3 times and it stopped reading it when it had 400 kB left, but the 4th time succeeded. Dovecot sent exactly the same data with the same TCP packet boundaries all times (at least to Dovecot proxy - would have to look with tcpdump further to see if proxy does something differently..) Anyway, nobody in general has had trouble with Dovecot v2.2 and Outlook 2007/2010? Maybe the problem is something else, although strange if it started happening only immediately after Dovecot upgrade.
[Dovecot] Panic: file mail-index-transaction-export.c: line 203 (log_append_ext_hdr_update): assertion failed: (u32.offset + u32.size = ext_hdr_size)
Hi there,
We recently upgraded to 2.2.12 (the current version in FreeBSD's port
tree), and are seeing these errors in our logs (not super frequently,
but it happens):
May 30 13:20:57 mail1 kernel: pid 15752 (imap), uid 1005: exited on
signal 6
May 30 13:20:57 mail1 dovecot: imap(xxx): Fatal: master: service(imap):
child 15752 killed with signal 6 (core not dumped - set service imap {
drop_priv_before_exec=yes })
I tried manually upgrading to 2.2.13, on the off chance that was fixed,
but I couldn't get the new pigeonhole (0.4.3) to compile once I did
(perhaps why the FreeBSD port maintainer hasn't updated yet?).
Suggestions? Right now we just check every couple of hours for affected
users, and then delete all of the dovecot files for the affected user,
which ends the error.
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Re: [Dovecot] Panic: file mail-index-transaction-export.c: line 203 (log_append_ext_hdr_update): assertion failed: (u32.offset + u32.size = ext_hdr_size)
Thanks to the suggestion by Larry off-list, I snagged an official patch
from the FreeBSD PR and now the ports are compiling cleanly.
I'll report back if I get the errors again.
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
On 05/30/2014 15:34, Andy Dills wrote:
Hi there,
We recently upgraded to 2.2.12 (the current version in FreeBSD's port
tree), and are seeing these errors in our logs (not super frequently,
but it happens):
May 30 13:20:57 mail1 kernel: pid 15752 (imap), uid 1005: exited on
signal 6
May 30 13:20:57 mail1 dovecot: imap(xxx): Fatal: master: service(imap):
child 15752 killed with signal 6 (core not dumped - set service imap {
drop_priv_before_exec=yes })
I tried manually upgrading to 2.2.13, on the off chance that was fixed,
but I couldn't get the new pigeonhole (0.4.3) to compile once I did
(perhaps why the FreeBSD port maintainer hasn't updated yet?).
Suggestions? Right now we just check every couple of hours for affected
users, and then delete all of the dovecot files for the affected user,
which ends the error.
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Re: [Dovecot] Panic: file mail-index-transaction-export.c: line 203 (log_append_ext_hdr_update): assertion failed: (u32.offset + u32.size = ext_hdr_size)
Unfortunately, I'm still getting the same errors post upgrade to 2.2.13.
I'm coming from 2.1.12, so perhaps there is some slight incompatibility
in some circumstances with the index files? I'm continuing to delete
them as this arises, and so far I've no repeat problem accounts.
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
On 05/30/2014 16:02, Larry Rosenman wrote:
I actually submitted the PR's. I'm waiting for the real maintainer to
approve or for the 2 week timeout.
As I said, it's doing great for me :)
On Fri, May 30, 2014 at 3:01 PM, Andy Dills a...@xecu.net wrote:
Thanks to the suggestion by Larry off-list, I snagged an official
patch
from the FreeBSD PR and now the ports are compiling cleanly.
I'll report back if I get the errors again.
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
On 05/30/2014 15:34, Andy Dills wrote:
Hi there,
We recently upgraded to 2.2.12 (the current version in FreeBSD's port
tree), and are seeing these errors in our logs (not super frequently,
but it happens):
May 30 13:20:57 mail1 kernel: pid 15752 (imap), uid 1005: exited on
signal 6
May 30 13:20:57 mail1 dovecot: imap(xxx): Fatal: master:
service(imap):
child 15752 killed with signal 6 (core not dumped - set service imap
{
drop_priv_before_exec=yes })
I tried manually upgrading to 2.2.13, on the off chance that was
fixed,
but I couldn't get the new pigeonhole (0.4.3) to compile once I did
(perhaps why the FreeBSD port maintainer hasn't updated yet?).
Suggestions? Right now we just check every couple of hours for
affected
users, and then delete all of the dovecot files for the affected
user,
which ends the error.
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Re: [Dovecot] Errors with doveadm when using checkpassword
On Tue, 5 Feb 2013, Timo Sirainen wrote: I think you need to remove doveadm_proxy_port from the backend dovecot.conf. Then it doesn't perform the PASS lookup. But you also should run doveadm via the proxy instance so that it gets run in the correct server (doveadm -c /etc/dovecot/proxy.conf or doveadm -i proxy if you've given it a name). On a seperate note I'm sure a lot of people would benefit from -c/-i being mentioned on http://wiki2.dovecot.org/Tools/Doveadm. You are one man with only so much time so I tried registering on the wiki to propose an edit for you, but I'm not allowed. I think all we need is to know that -c and -i exist, and a note about how people in proxy/director configurations need to make sure to tell doveadm to communicate with the instance that is running director. For some reason, my intuition would be that since doveadm is aware of both instances, that it should be aware of which one's config to use for connecting to director for proxy information. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
[Dovecot] Errors with doveadm when using checkpassword
We have a checkpassword authentication with mysql pre-fetch for the userdb lookups. When trying to do: doveadm search -u andyt...@xecu.net mailbox Trash DELETED I get: doveadm(andyt...@xecu.net): Error: user andyt...@xecu.net: Auth PASS lookup failed doveadm(andyt...@xecu.net): Fatal: /var/run/dovecot/auth-userdb: passdb lookup failed (to see if user is proxied, because doveadm_proxy_port is set) And I see this in the logs: Feb 4 12:02:04 mail-out01 dovecot: auth: Error: userdb-checkpassword(andyt...@xecu.net): Child 12591 exited with status 1 I'm not surprised the checkpassword lookup is failing; the password isn't being supplied. Shouldn't it just be doing the prefetch lookup instead, like the lmtp service? Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Errors with doveadm when using checkpassword
On Feb 4, 2013, at 8:56 PM, Timo Sirainen t...@iki.fi wrote: On Mon, 2013-02-04 at 12:08 -0500, Andy Dills wrote: We have a checkpassword authentication with mysql pre-fetch for the userdb lookups. When trying to do: doveadm search -u andyt...@xecu.net mailbox Trash DELETED I get: doveadm(andyt...@xecu.net): Error: user andyt...@xecu.net: Auth PASS lookup failed doveadm(andyt...@xecu.net): Fatal: /var/run/dovecot/auth-userdb: passdb lookup failed (to see if user is proxied, because doveadm_proxy_port is set) Do you have proxying enabled? If not, remove doveadm_proxy_port and this solves itself Yes, I apologize, it appears I edited out the paragraph where I described our setup. Cluster of dovecot dual instance servers, one proxy/director, one backend. The backend config lives in dovecot.conf, the proxy config in proxy.conf (let me know if the specific config would help). Version 2.1.12 (current version in FreeBSD ports). I should mention, I have to specify the director-admin socket manually when using doveadm director, it's as if doveadm isn't quite aware of the config in the proxy instance. However, the authentication happens in the primary dovecot.conf instance...that's where checkpassword gets called, and where the prefetch is configured. The proxy is dumb. I'm not surprised the checkpassword lookup is failing; the password isn't being supplied. Shouldn't it just be doing the prefetch lookup instead, like the lmtp service? doveadm doesn't care about the password. It cares about the proxy destination host so that it can automatically connect to the correct host to run the command. I understand doveadm doesn't care about the password; i was confused why checkpassword was being called when the password is unknown rather than using the mysql prefetch. So, I guess what you're saying is doveadm should be talking to the proxy instance rather than the backend instance? I assume I need to move the backend to a differently named config and make my proxy instances live in dovecot.conf, so that doveadm sees the right config? Thanks, Andy
Re: [Dovecot] Errors with doveadm when using checkpassword
On Tue, 5 Feb 2013, Timo Sirainen wrote:
So, I guess what you're saying is doveadm should be talking to the
proxy instance rather than the backend instance? I assume I need to
move the backend to a differently named config and make my proxy
instances live in dovecot.conf, so that doveadm sees the right
config?
I think you need to remove doveadm_proxy_port from the backend
dovecot.conf.
Thanks Timo.
That gave me:
doveadm -c /usr/local/etc/dovecot/proxy.conf search -u andy...@xecu.net mailbox
Trash DELETED
doveadm(andyt...@xecu.net): Error: doveadm_password not set, can't authenticate
to remote server
doveadm(andyt...@xecu.net): Error: 10.0.0.47:30003: Internal failure for
andyt...@xecu.net
So, on a whim, I moved the doveadm_password out from the local {} sections
into the main level, and then things started to work as expected.
Is that how it should be setup now?
Thanks again for help getting me straightened out! This all started
because I want to start purging the trash with doveadm instead of 'find'
with -delete.
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Re: [Dovecot] Director pop-login and imap-login processes exiting on signal 11
On Mon, 9 Apr 2012, Timo Sirainen wrote:
On 7.4.2012, at 10.13, Andy Dills wrote:
Apr 7 02:18:05 mail-out06 dovecot: pop3-login: Fatal: master:
service(pop3-login): child 75029 killed with signal 11 (core not dumped -
set service pop3-login { drop_priv_before_exec=yes })
v2.1.3 proxying was buggy with SSL connections. Probably crashes because
of that. I was supposed to release v2.1.4 already but..
Thanks Timo. I can confirm this is fixed in 2.1.4.
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
[Dovecot] Director pop-login and imap-login processes exiting on signal 11
We recently upgraded our cluster to 2.1.3, to enable director proxying.
Everything appears to be working fine for the most part; the only odd
thing is that I'm seeing a lot of entries in the logs like this:
Apr 7 02:18:05 mail-out06 dovecot: pop3-login: Fatal: master:
service(pop3-login): child 75029 killed with signal 11 (core not dumped -
set service pop3-login { drop_priv_before_exec=yes })
This is on the proxy side, not that backend side.
When I try to get a dump out of it, and add drop_prive_before_exec and
chroot= to the pop3-login statement on the proxy, I keep running into
permissions errors with the various service sockets.
Any suggestions?
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Re: [Dovecot] Multiple instances
Sorry to respond to an old post, but I've just recently begun implementing multiple instances to facilitate our director proxies running along with our normal dovecot config on the same servers in the cluster. This is a VERY useful feature Timo, it may need just a little refinement. On Mon, 6 Feb 2012, Timo Sirainen wrote: # doveadm instance remove proxy Hmm...maybe I'm doing something wrong or expecting the wrong behavior, but when I do this, while it dissapears from doveadm, it still responds to pop/imap requests, and the process continues to run. Is remove supposed to be different than say, stop? It would be possible to add commands to start/shutdown some/all instances using doveadm, but is it all that useful? I'd guess people would have their own init.d scripts anyway doing that. Eh, in a FreeBSD port-build environment, I have to hack something in place in the rc script that gets installed, and then make sure to duplicate it every time I upgrade dovecot...not ideal. So, if dovecot had some sort of mechanism in the main config file to alert it of the additional instances to start and their config files, that would be nice. Or, if you were to add a instance_enable switch in the config files and then have dovecot scan the /usr/local/etc/dovecot directory for appropriate config files to automatically parse. I dunno, it doesn't feel right to push the startup of the additional instances outside of dovecot. For example, consider postfix's master.cf file. BTW somebody needs to poke the dovecot port maintaner, he still has 2.0.18, I had to manually update the port to get 2.1.3. Anything else that could be useful related to this? Yes...we should probably be able to start instances back up as well. Other then that, looks good. Definitely a great feature. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
[Dovecot] Proxying Authentication on both sides
I've recently set up a director proxy environment on my test servers, with the intention of deploying on our cluster soon. One thing I found confusing in the proxying documentation [1] was the first bit about their being two ways to do the authentication...either you have the proxy forward the auth to the real server for authentication, or you have the proxy authenticate it and then login to the real server with a master password. Well, we use /bin/checkpassword authentication which hooks into a variety of subsytems for various specific customer needs, and sometimes we need to know the username AND password of the user in order to determine their home directory information. So, using a master password (which requires the back-end server not getting the user password) is out. However, when we have the front-end server do a static director proxy, the problem is that authentication failures are logged on the back-end server with a source IP of the proxy, and no authentication failure with the client IP address is logged on the proxy. So, fail2ban (which is a MUST these days, at least for us) will not be able to properly filter out the brute force attackers. My solution was an alternative: I authenticate with our /bin/checkpassword on the proxy, which authenticates the user and only at that point returns the proxy=y nopassword=y switch to proxy the connection and forward the authentication. As a result, we get logs on the proxy for failed attempts, and the full username and password is supplied to the back-end server for proper processing. Food for thought in case anybody else is implementing this. Thanks, Andy [1] http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Proxying Authentication on both sides
On Fri, 30 Mar 2012, Timo Sirainen wrote: On 30.3.2012, at 16.25, Andy Dills wrote: However, when we have the front-end server do a static director proxy, the problem is that authentication failures are logged on the back-end server with a source IP of the proxy, and no authentication failure with the client IP address is logged on the proxy. So, fail2ban (which is a MUST these days, at least for us) will not be able to properly filter out the brute force attackers. This is a simple fix (and something you should do anyway): Add the proxy's IP/netmask to login_trusted_networks setting in the remote server. For this to work with POP3 you need v2.1.2+. Well, the problem isn't that my proxies would be banned; the problem is I have no way of seeing the remote IP of the failed authentication so I can ban the people who should be banned. My solution was an alternative: I authenticate with our /bin/checkpassword on the proxy, which authenticates the user and only at that point returns the proxy=y nopassword=y switch to proxy the connection and forward the authentication. Hm. Doesn't it do that even without nopassword=y? Perhaps...I was going by the docs which seemed to suggest that nopassword=y was how you get the proxy to forward the users authentication credentials to the back-end server. I had been trying a lot of different things, and it was only when I realized I needed to not do a static passdb on the proxy, but instead do a full authentication so that the auth failure would be logged on the proxy for fail2ban, that things began to work the way I needed. It seems obvious in retrospect, but for whatever reason the way the docs were written made me feel like having the full authentication happen on both the proxy and the backend wasn't possible. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Sieve GUI
On Wed, 2 Jun 2010, Gerhard Waldemair wrote: Am 02.06.2010 um 13:23 schrieb Marcio Merlone: Em 01-06-2010 15:45, Frank Cusack escreveu: (...)you are right in that almost no clients support it. Mulberry and thunderbird are the only ones I know of. Are you talking about the extension on wich you have to write the sieve scripts by hand? That is a joke, hope someone creates a decent extension for that some day... -- Marcio Merlone I have found this: http://smartsieve.sourceforge.net/ has someone tried this or knows something similar ? Judging from the screenshots, it's pretty clearly a ripoff of Ingo from Horde, which I'm currently testing and am pretty happy with. http://www.horde.org/ingo/ Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
[Dovecot] strange sieve situation
Hi there, I've been working on rolling out the deliver service so we can
implement sieve. Dovecot 1.2.11, Postfix 2.6.6, FreeBSD 8, installed by
ports. Clustered environment, NFS backend.
Everything is working great for the most part, I'm using it to sort a copy
of my personal email on a test server. However, I noticed two emails were
not properly sorted.
I had added a rule (using Ingo, nice little piece of software) to redirect
the nightly FreeBSD security output emails into a folder servers.
However, nothing had triggered that rule, and the Maildir/.servers
directory had not yet been created.
Then, three of the security outputs came in at one time:
May 29 03:01:48 mg8 dovecot: deliver(andyt...@xecu.net): sieve:
msgid=20100529070108.f2f0b7f...@mail-out04.xecu.net: failed to store
into mailbox 'servers': Mailbox already exists
May 29 03:01:48 mg8 dovecot: deliver(andyt...@xecu.net): sieve:
msgid=20100529070109.d8f5e159...@mail-out01.xecu.net: failed to store
into mailbox 'servers': Mailbox already exists
May 29 03:01:48 mg8 dovecot: deliver(andyt...@xecu.net): sieve:
msgid=20100529070109.d8f5e159...@mail-out01.xecu.net: stored mail into
mailbox 'INBOX'
May 29 03:01:48 mg8 dovecot: deliver(andyt...@xecu.net): sieve: execution
of script /mail/var/mail2/xecunet/andytest.xecu.net//.dovecot.sieve
failed, but implicit keep was successful
May 29 03:01:48 mg8 dovecot: deliver(andyt...@xecu.net): sieve:
msgid=20100529070110.17a781b5...@mail-out02.xecu.net: stored mail into
mailbox 'servers'
May 29 03:01:48 mg8 dovecot: deliver(andyt...@xecu.net): sieve:
msgid=20100529070108.f2f0b7f...@mail-out04.xecu.net: stored mail into
mailbox 'INBOX'
May 29 03:01:48 mg8 dovecot: deliver(andyt...@xecu.net): sieve: execution
of script /mail/var/mail2/xecunet/andytest.xecu.net//.dovecot.sieve
failed, but implicit keep was successful
That's the order they appeared in my logfile. I'm assuming they were all
sent to deliver at the same time, they all thought they needed to make the
directory, the one with msgid
20100529070110.17a781b5...@mail-out02.xecu.net was able to make the
directory and get delivered, whereas the other two freaked out because
they couldn't make the directory they thought needed to be made, and just
sent the mail to the inbox.
That's a reasonable approach, although I wonder if deliver/sieve could be
(or perhaps has been in 2.0) adjusted to more intelligently handle that
edge case.
Also, I'm wondering about dovecot_destination_recipient_limit = 1. I
have that set in my main.cf, but I'm not 100% positive it's taking effect.
# grep dovecot /usr/local/etc/postfix/main.cf
dovecot_destination_recipient_limit = 1
virtual_transport = dovecot
# grep dovecot /usr/local/etc/postfix/master.cf
dovecot unix - n n - - pipe
flags=DRhu user=mailman argv=/usr/local/libexec/dovecot/deliver -f ${sender}
-d ${recipient}
# postconf | grep dovecot
smtpd_sasl_type = dovecot
virtual_transport = dovecot
# postconf | grep destination_recipient_limit
default_destination_recipient_limit = 50
lmtp_destination_recipient_limit = $default_destination_recipient_limit
local_destination_recipient_limit = 1
relay_destination_recipient_limit = $default_destination_recipient_limit
smtp_destination_recipient_limit = $default_destination_recipient_limit
virtual_destination_recipient_limit = $default_destination_recipient_limit
Is dovecot_destination_recipient_limit being converted into local_ ?
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
Re: [Dovecot] Strange checkpassword issue
Figured it out. For whatever reason, I didn't need to set the userdb_mail
previously, but you definitely do now.
Thanks,
Andy
On Fri, 25 Jul 2008, Andy Dills wrote:
I'm helping a friend setup a small mailserver using dovecot, and I'm
finding a strange problem with checkpasswd that I haven't had on my
servers.
How is the following debug output even possible?
Jul 25 12:12:20 company2 dovecot: auth(default): master out: USER 5
joe home=/var/mail/joe.com/joe/Maildir/ uid=1005gid=1005
Jul 25 12:12:20 company2 dovecot: pop3-login: Login: user=joe,
method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured
Jul 25 12:12:20 company2 dovecot: POP3(joe): open(/var/mail/joe) failed:
Permission denied (euid=1005 egid=1005)
I don't see how, when it's clearly getting the correct home directory from
checkpassword, that it would then try the default mail directory.
(mail_location is commented out, version 1.1.1)
Here's the entire auth clause:
auth default {
mechanisms = plain login
passdb checkpassword {
args = /usr/local/sbin/checkpassword
}
userdb prefetch {
}
user = mailman # 1005,1005
socket listen {
master {
mode = 0666
}
client {
path = /var/run/dovecot/auth
mode = 0660
user = postfix
group = postfix
}
}
}
Any suggestions? I'm probably just missing something obvious, not having
messed with setting this up for 6 months.
Thanks,
Andy
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---
[Dovecot] Who's wrong, atmail or dovecot?
We recently switched to atmail, as well as dovecot. I noticed in atmail the size of the mailboxes was always reported as 0kb. So, I did some debugging, and it boiled down to the fact that the regular expression used by dovecot expected UID before SIZE, but Dovecot returned SIZE before UID. No biggy, I changed the regex, but I was curious if there was a standard. Here's the imap query that is sent: UID FETCH 1:* (RFC822.SIZE) Here's the diff I implemented to make it work: http://www.xecu.net/atmail/dovecot_sizes.diff So...is this something that is standard or something atmail needs to handle by making their regex more dynamic? Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] User another userdb to sasl
On Fri, 7 Mar 2008, Nicolas Letellier wrote: Hello, I use Dovecot 1.0.12 with Postfix on FreeBSD. It works perfectly, but I have a question. My users are virtual (stocked in a MySQL database). I use SASL Dovecot authentification, and SASL uses my userdb (for POP/IMAP access). However, I would like my users have an other login/password to send email with SMTP/SASL. Or, I don't want my users use the same login/password for POP/IMAP than SMTP. More, I don't want some users could send emails with their login/password, but I want they can fetch their mails. I'm looking for an option in my dovecot.conf to specifie a table SQL for SASL authentification (and don't use same table SQL than IMAP/POP access). How I could do for this problem? Do you have an idea? Or, simply, is it possible with Dovecot SASL? I read all Dovecot wiki, with no results... One solution, off of the top of my head, would be two dovecot instances, with only one providing the SASL authentication socket to postfix, which would be configured with seperate authentication methods. You might want to look into /bin/checkpassword authentication, I can't imagine using anything else. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Can't load private key file
On Fri, 7 Mar 2008, [EMAIL PROTECTED] wrote: Hi Guys, I have just purchased and installed an authenticated certificate for the mail server, but I am getting errors from dovecot. My mail.err file is showing the following. Mar 7 17:56:04 mailserver dovecot: pop3-login: Can't load private key file /etc/ssl/mailserver/mail.mydomain.tld.key: error:0906A068:PEM routines:PEM_do_header:bad password read My dovecot.conf has the following set. # Uncomment these if using SSL ssl_cert_file = /etc/ssl/mailserver/mail.mydomain.tld.crt ssl_key_file = /etc/ssl/mailserver/mail.mydomain.tld.key #ssl_key_password = #ssl_ca_file = /etc/ssl/mailserver/ca/mydomain.pem #ssl_verify_client_cert = yes ssl_parameters_regenerate = 168 verbose_ssl = no I have been playing about with it all for about 3 hours now and would greatly appreciate any help ;) It sounds like you encrypted the key with a password...you would put that as the ssl_key_password in the config above. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
[Dovecot] Future enhancement of imap: sorting?
After solving a problem with a sweet little application I discovered (more on that below), I wanted to get some feedback about a potential enhancement for IMAP: standardized, MUA independant, server-side sorting. Currently, there are two basic solutions for sorting: handle it during delivery or handle it in the MUA. However, if you handle it in the MUA, you lose some of the client independancy and portability that imap was designed to address, as there is no mechanism to sync sorting rules between clients, and many lightweight clients don't support sorting at all. On the flip side, in distributed mail envrionments sorting at delivery time isn't feasible, as you typically have the entire mailspool owned by a single UID and mounted via NFS. My basic point is that the two paradigms for sorting have fundemental flaws that severaly limit them. It occured to me that it would be ideal to enable sorting in the imap daemon, enabling users to implement sorting rules which are stored in their directory (similar to the other imap metadata), and which get invoked whenever mail is checked. Perhaps that is beyond the reasonable scope of the imap protocol, but in my mind sorting is a critical task that is poorly addressed and needs to be standardized. I'd love to hear opinions on this. See, in our mail cluster environment, it's basically impossible to use procmail without forwarding your mail to our shell server (nor are we able to use the dovecot LDA). So, the question presented to me by one of my business partners was, since Outlook was shortsighted and didn't include the ability to sync sorting rules between clients, nor did they implement sorting functionality whatsoever on the windows mobile version, what options does he have to ensure proper sorting of his mail without having to leave outlook running 24x7? He doesn't want to have to wade through mailing list email on his mobile outlook, nor does he want to have to ensure his filter rules are consistent in both his outlook and our webmail. So, I poked around in the ports tree, and found a gem: imapfilter. http://imapfilter.hellug.gr/ It's configured via lua, supports a very rich array of actions and queries, and has proven to be very robust and lightweight. I helped him get started with some sample rules, added it to run every minute as a cron job, and boom: robust, fully featured, MUA independant filtering in an environment that can't support procmail or the dovecot LDA. As nice as it is, it's still a personal solution. In my mind we need to have standardized, server-side, MUA-independant sorting rules that get invoked when a mailbox is checked. Thoughts? Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Future enhancement of imap: sorting?
On Fri, 7 Mar 2008, Timo Sirainen wrote: On Thu, 2008-03-06 at 20:59 -0500, Andy Dills wrote: As nice as it is, it's still a personal solution. In my mind we need to have standardized, server-side, MUA-independant sorting rules that get invoked when a mailbox is checked. Thoughts? Filtering is a better name for this, took me a few paragraphs before I understood what you meant by sorting. :) Heh, sorry about that. In my mind, sorting implies directing an object to a location based on some arbitrary criteria, whereas filtering (in my mind) implies rejecting or accepting based on some arbitrary criteria. You could already create a plugin that runs a command when it sees new messages in a mailbox. I'm not really interested in writing this feature.. I didn't mean to imply this as a request for functionality...it would be worthless as a feature in dovecot without support in the IMAP protocol and thus the MUAs. I was thinking more along the lines of a future enchanement for the actual IMAP protocol, as this seems like something that should have been addressed but wasn't, leading to an array of flawed solutions. I assume you feel that this is beyond the scope of what IMAP should handle natively? On one hand it feels like it is, but on the other hand it seems like something in desperate need of standardization. The idea for virtual folders based on search criteria is quite interesting, by the way. How would you enable users to create them from their client? Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Future enhancement of imap: sorting?
On Fri, 7 Mar 2008, Timo Sirainen wrote: I think Sieve (+ ManageSieve) could already do all of this. It's just normally meant to be used with MDA, but nothing would prevent from running the script when IMAP server sees it for the first time. I hadn't considered invoking Sieve from imap, I'll look into that as a more scalable solution. The idea for virtual folders based on search criteria is quite interesting, by the way. How would you enable users to create them from their client? I'm just going to make it read the configuration from a file. How it gets there is not my problem, at least not yet. :) So it'll probably get some kind of a web configuration tool. Makes sense. This feature has a large amount of potential and I would love to see it implemented. The possibilities are exciting, such as a adding a button to atmail's addressbook to create a virtual folder for that contact, which would contain mail sent from them, with a sub folder containing mail sent to them. Even something as simple as being able to associate a message with multiple folders without requiring multiple copies on disk is a nice improvement. Do you have any timelines in mind for this feature? I'd be happy to help work out the bugs. No pressure, I'm just excited to offer new and useful functionality to our users. Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
[Dovecot] Effects of migration
So, to follow up to my previous thread, we just successfully migrated our NFS-based mail cluster from qmail pop, courier imap, and bincimap to dovecot 1.1rc1. Overall the transition was very smooth, the only unexpected adjustment was having to implement ntpd on each box, rather than doing an hourly ntpdate against our local ntpd server, to prevent dovecot from crashing itself from too much drift. The impact has been severe! Even with NFS-stored indexes, our netapp is seeing 1/6th of the NFS ops per second, and its CPU utilization is now at 1/3rd previous levels. The only user comment thus far was thanking us for bringing IMAP folders out from under INBOX. Dovecot is truly excellent. In my book, Timo joins Wietse Venema and Marc Martinec to form the backbone of the premiere open source mail solution. For now, praise will have to suffice. I do, however, maintain a little IOU list that I intend to fulfill at some point in the future, and Timo is now high on the list. Thanks again! Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Effects of migration
On Thu, 6 Mar 2008, Timo Sirainen wrote: On Wed, 2008-03-05 at 11:23 -0500, Andy Dills wrote: The impact has been severe! Even with NFS-stored indexes, our netapp is seeing 1/6th of the NFS ops per second, and its CPU utilization is now at 1/3rd previous levels. Have you thought about enabling Squat indexes? I'd like to know how much it would affect I/O and CPU usage in larger installations. CPU grows (maybe a lot) but searches should be faster and use very little I/O as a result. By CPU, do you mean local server (nfs client) CPU or netapp CPU (nfs server)? I'm guessing the former...for what it's worth, for those who have yet to have the pleasure of using a Netapp, the CPU utilization is basically your ultimate barometer of utilization with netapps. As long as you don't start hitting 90% CPU consistently, they will provide better throughput than local SCSI disks. The CPU tops out well before the I/O, a nice change of pace. Local CPU is of little concern typically, as mail serving (filtering is handled elsewhere) is almost purely I/O. However, I'm not sure how much value I would place on optimizing searches at this point...do users really do much of that? It seems to be potentially valuable yet underutilized. Do you have some links so I can educate myself more about squat indexes? Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
Re: [Dovecot] Dovecot NFS Indexes and IMAP Migration
On Wed, 27 Feb 2008, Timo Sirainen wrote: On Feb 27, 2008, at 8:49 AM, Andy Dills wrote: From reading the docs, it appears I would not be able to use the dovecot LDA due to locking issues (bummer), and I should configure dovecot to store index files on a local disk of the primary pop/imap server rather than on the netapp. I don't think locking is an issue. NFS caching is more problematic. But with v1.1's mail_nfs_*=yes settings there should be no problems with storing indexes on NFS. That's good to know. Do you view 1.1 as production ready now, or should I wait for an official release? It sounds production ready in the RC1 release notes, but I was curious how much remains on your few issues you'd like to fix list and how relevant they are to my environment. Thanks, Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
[Dovecot] Dovecot NFS Indexes and IMAP Migration
$fin2; close $fout; chown $owner_uid, $owner_gid, $out_fname; } Thanks in advance for any help or suggestions! Andy --- Andy Dills Xecunet, Inc. www.xecu.net 301-682-9972 ---
