Re: [Dovecot] GETQUOTAROOT under roundcube

2012-05-29 Thread Timo Sirainen
On Sun, 2012-05-20 at 14:07 +0200, Adam Szpakowski wrote:
 Hi,
 
 I'm struggling with the proper quota displaying under roundcube webmail. 
 I've tracked the problem to the different responses on GETQUOTAROOT command.

If the clients are accessing mails via the same username, then the reply
to GETQUOTAROOT command should be the same (assuming of course that
there have been no changes to mailbox). Dovecot doesn't know if it's
Roundcube of whatever asking the quota, the reply is always the same.




Re: [Dovecot] Active Directory : searches in root tree

2012-05-29 Thread Timo Sirainen
On Mon, 2012-05-21 at 01:48 -0700, nicolasfo wrote:
 base  = ou=some_ou,dc=domain,dc=lan
..
 With this configuration file, it works. BUT :
 To allow Dovecot to find users in my AD database, I must specify an OU in
 base. If I only set dc=domain,dc=lan the research doesn't work. It seems
 that Dovecot is not capable to make a research without an OU specified in
 base.
 Am I wrong ?

The base parameter is simply passed to OpenLDAP library. There's no
requirement from Dovecot's side to have OU. But what might be possible
is that you may need to change other settings to make it work. I'm not
expert with LDAP and especially not with AD though.




Re: [Dovecot] Dovecot auth process delays exiting if LDAPS passdb used

2012-05-29 Thread Timo Sirainen
On Thu, 2012-05-24 at 01:22 +0900, SATOH Fumiyasu wrote:
 At Thu, 24 May 2012 01:01:25 +0900,
 SATOH Fumiyasu wrote:
  If Dovecot passdb is configured with LDAP (no TLS/SSL),
  it is no problem. But if Dovecot passdb is configured with
  LDAPS (or LDAP+TLS), Dovecot auth process has a problem
  that Dovecot auth delays exiting about between 20 and
  60 seconds when Dovecot dovecot (master) process is already
  terminated by an administrator.
 
 I can reproduce this problem with LDAP (no TLS/SSL) passdb.

And I suppose you can reproduce it even when not using LDAP?

All of the Dovecot processes are supposed to close all listeners
immediately when the master process dies. If this doesn't happen then
something strange is going on.




Re: [Dovecot] Time stamp or expire_stamp not updating to DATABASE table expires in mysql

2012-05-29 Thread Timo Sirainen
On Thu, 2012-05-24 at 15:50 +, Steve Wells wrote:
 Dovecot upgraded from 1.2 to 2.0.18
..
   expire = Trash 7 Trash/* 7 Junk 30 Sent 30

This setting changed a bit: http://wiki2.dovecot.org/Plugins/Expire




Re: [Dovecot] Bug report - crash on group lookup

2012-05-29 Thread Timo Sirainen
On Fri, 2012-05-25 at 18:52 +0200, Peter Meier wrote:

 I was doing some migration from a 1.2 installation to a 2.1. While
 testing my new installation dovecot crashed at two test-cases constantly
 with with a Panic: Trying to allocate 0 bytes message.
..
 I see two problems:
 
 1. Don't panic while looking up a group
 2. Give a meaningful error messages.

Asserts/Panics can't give very meaningful error messages to users,
because they only happen when there's a bug in the code. The resulting
raw backtrace is helpful to coders though :) Fixed:
http://hg.dovecot.org/dovecot-2.1/rev/fe688ecd7564




Re: [Dovecot] I can't logon to the mail server using an NIS user account

2012-05-29 Thread Timo Sirainen
On Sat, 2012-05-26 at 13:51 +0100, Kayode Odeyemi wrote:
 Below is my configuration
 
 passdb {
   args = scheme=CRYPT username_format=%u /etc/dovecot/users
   driver = passwd-file
 }

Dovecot isn't using NIS. It's using user accounts in /etc/dovecot/users
file.

And in any case set auth_debug_passwords=yes to debug authentication.




Re: [Dovecot] Dovecot 2.1 mbox + maildir

2012-05-29 Thread Timo Sirainen
On Mon, 2012-05-28 at 14:50 +0100, Alan Brown wrote:
 What syntax is needed to make this work?
 
 The 2.0 wiki recomendations don't work - I can see the inboxes or the 
 folders but not both at once and there are lots of error messages about 
 prefix clashes if I simply use the existing 2.0.20 conf file on 2.1.6

Are you saying that it works in v2.0 but not in v2.1? Then something's
wrong. Show your doveconf -n output and what error messages you see.




Re: [Dovecot] Crash on force-resync if / is given as mailbox name

2012-05-29 Thread Timo Sirainen
On Sat, 2012-05-26 at 15:30 +0200, Daniel Parthey wrote:

 when I specify a slash a mailbox name on the command line
 of doveadm force-resync, it throws a panic. I'm not sure
 this is considered a bug.

It's a bug.

 # 2.0.20: /etc/dovecot/dovecot.conf

But it's already been fixed in v2.1 and the fix for v2.0 would be too
difficult.




Re: [Dovecot] Different SSL requirements for connections on different ports?

2012-05-29 Thread Timo Sirainen
On Tue, 2012-05-29 at 15:09 +0100, William Gallafent wrote:
 Hi All,
 
 I'm running dovecot 2.0.19.
 
 I currently have remote users access mail using IMAP over SSL, with
 their client certificates being both required and verified. I do this
 using ssl = required and ssl_verify_client_cert = yes.

And I guess you also have auth_ssl_require_client_cert=yes.

 I would now like to add a webmail front-end (squirrelmail) running on
 the same server. In order to achieve this I would like to have
 squirrelmail connecting locally using IMAP, but without the
 certificate requirement. I'm happy to use the standard IMAP port for
 this, since that port is firewalled so that only localhost has access.
 
 Do I need to run two separate dovecot instances in order to achieve
 this, or can I somehow configure different SSL requirements for the
 two ports? Is there a way to have the ssl directives I mention above
 active only for a certain port (or for certain hosts, i.e. non-local?)

You could work around ssl=required by setting the webmail's IP to
login_trusted_networks, but it won't get around requiring a valid SSL
cert. For that you'd need to put it inside remote IP {} block, but
unfortunately you can't currently change auth settings for specific IPs.
So for now you'd need to run two Dovecot instances.



Re: [Dovecot] Director and backend on the same server

2012-05-29 Thread Timo Sirainen
On 29.5.2012, at 17.23, James Devine wrote:

 I setup the static passdb like:
 
 passdb {
  driver = static
  args = proxy=y port=10024 nopassword=y
 }
 
 and this works fine for the lmtp service, would I have to run a director
 per protocol or can they be combined into one somehow?

I think you can do:

protocol lmtp {
  passdb {
driver = static
args = proxy=y port=10024 nopassword=y
  }
}

And the same for other protocols.



Re: [Dovecot] Different SSL requirements for connections on different ports?

2012-05-29 Thread Timo Sirainen
On 29.5.2012, at 20.17, Ron Leach wrote:

 On 29/05/2012 16:55, Timo Sirainen wrote:
 On Tue, 2012-05-29 at 15:09 +0100, William Gallafent wrote:
 
 I would now like to add a webmail front-end (squirrelmail) running on
 the same server. In order to achieve this I would like to have
 squirrelmail connecting locally using IMAP, but without the
 certificate requirement.
 
 Do I need to run two separate dovecot instances in order to achieve
 this, or can I somehow configure different SSL requirements for the
 two ports?
 
 for now you'd need to run two Dovecot instances.
 
 
 [Keen to do the same thing.]  I guess those two Dovecot instances could run 
 either on (a) the same machine, or (b) different machines - though using a 
 network-visible mail storage location.

Yes.

 Does this dual-Dovecot solution become problematic if the mail storage 
 location is NFS?  I'm aware of the cautions regarding NFS access from two 
 different machines (case (b), here).  

Yes, NFS will cause trouble in that kind of a setup.

 If these two Dovecot instances were on the same machine (case (a)) but which 
 nevertheless used an NFS mail storage location, would we then also hit the 
 NFS dual-access problem from this single machine running the two instances?

If there is a single NFS mountpoint used by both Dovecots, there won't be a 
problem since there is only one kernel accessing and caching it. (I'm not 
entirely sure if there's a problem with more than one mountpoint, might be.) 
Dovecot itself doesn't internally have a problem with multiple Dovecot 
instances accessing the same files, regardless of where they are stored.

Re: [Dovecot] interesting stats pattern

2012-05-29 Thread Timo Sirainen
On 29.5.2012, at 21.03, Cor Bosman wrote:

 es, I am getting a list of sessions/users every 5 minutes through cron. Im 
 already using doveadm stats dump session/user connected

Actually that's not really correct behavior either, since it ignores all the 
connections that happened during the 5 minutes if they don't exist at the time 
when you're asking for them. I'm not sure what the most correct way to do this 
kind of a graph would be :)

 It's not a big deal or anything, just wondering about the weird patterns. If 
 it's really dropping/gaining connections, id like to figure out why.

Are you only counting imap/pop3 sessions or also others? Anything that touches 
mailboxes are counted as sessions (lda, lmtp, doveadm, indexer, ..)



[Dovecot] v2.1.7 released

2012-05-29 Thread Timo Sirainen
http://dovecot.org/releases/2.1/dovecot-2.1.7.tar.gz
http://dovecot.org/releases/2.1/dovecot-2.1.7.tar.gz.sig

* Session ID is now included by default in auth and login process
  log lines. It can be added to mail processes also by adding
  %{session} to mail_log_prefix.

+ Added ssl_require_crl setting, which specifies if CRL check must
  be successful when verifying client certificates.
+ Added mail_shared_explicit_inbox setting to specify if a shared INBOX
  should be accessible as shared/$user or shared/$user/INBOX.
- v2.1.5: Using ~/ as mail_location or elsewhere failed to actually
  expand it to home directory.
- dbox: Fixed potential assert-crash when reading dbox files.
- trash plugin: Fixed behavior when quota is already over limit.
- mail_log plugin: Logging copy event didn't work.
- Proxying to backend server with SSL: Verifying server certificate
  name always failed, because it was compared to an IP address.




Re: [Dovecot] v2.1.7 released

2012-05-29 Thread Timo Sirainen
On Tue, 2012-05-29 at 22:24 +0300, Timo Sirainen wrote:
 http://dovecot.org/releases/2.1/dovecot-2.1.7.tar.gz
 http://dovecot.org/releases/2.1/dovecot-2.1.7.tar.gz.sig

Oops! I copypasted v2.1.6 NEWS somehow. Here's the correct one:

* LDAP: Compatibility fix for v2.0: ldap: If attributes contain
  ldapAttr=key=template%$ and ldapAttr doesn't exist, skip the key
  instead of using template value with empty %$ part for the key.

+ pop3: Added pop3_uidl_duplicates setting for changing the behavior
  for duplicate UIDLs.
+ director: Added doveadm director ring remove command.
- director: Don't crash with quickly disconnecting incoming director
  connections.
- mdbox: If mail was originally saved to non-INBOX, and namespace
  prefix is non-empty, don't assert-crash when rebuilding indexes.
- sdbox: Don't use more fds than necessary when copying mails.
- auth: Fixed crash with DIGEST-MD5 when attempting to do master user
  login without master passdbs. 
- Several fixes to mail_shared_explicit_inbox=no
- imapc: Use imapc_list_prefix also for listing subscriptions.




Re: [Dovecot] v2.1.7 released

2012-05-29 Thread Timo Sirainen
On Tue, 2012-05-29 at 15:31 -0400, Michescu Andrei wrote:

 Do you happen to have any updates on the progress of dsync redesign?

The code is in v2.2 hg tree now:
http://hg.dovecot.org/dovecot-2.2/rev/b2076acc3715

See the commit message for some of the missing things.

I'm hoping to get back to coding it soon, although there are some other
important things as well going on right now. Anyway the idea is that I'd
get it good enough to use my own mails pretty soon, then get v2.2 tree
feature complete and release v2.2.alpha1 within a month or two and
hopefully quickly stabilize it.




Re: [Dovecot] Dovecot 2.1 mbox + maildir

2012-05-30 Thread Timo Sirainen
On 30.5.2012, at 19.15, Alan Brown wrote:

 May 30 17:00:31 msslat dovecot: imap(foobar): Error: user foobar: 
 Initialization failed: namespace configuration error: Duplicate namespace 
 prefix: 

This is because in v2.1 you have two namespaces with prefix=. Most likely 
because upgrading your RPM installed a new conf.d/15-mailboxes.conf file with 
namespace inbox {}. You can either remove it or give inbox name for the other 
prefix= namespace.



Re: [Dovecot] High level of pop3 popping causing server to become unresponsive

2012-05-30 Thread Timo Sirainen
On 30.5.2012, at 20.25, Root Kev wrote:

 mail_location = mbox:/var/empty:INBOX=/var/mail/%u

Note that it's not a good idea for different users to share a single directory 
for indexes, which is what this does. Preferably /var/empty wouldn't be even 
writable to the users so this wouldn't happen accidentally. You could instead 
use something like:

mail_location = mbox:/var/empty:INBOX=/var/mail/%u:INDEX=/var/index/%u

But if the clients always just download + delete everything, the index files 
shouldn't make any (positive) difference.

 Had a chance to test this change this morning, and in my test environment,
 this does drastically improve the ability to ssh and su during heavy pop3
 load (in test environment, change of 10-15sec to 1-2sec login).  While this

If you think the problem is authentication, try with passdb + userdb static 
(http://wiki2.dovecot.org/PasswordDatabase/Static) or passwd-file 
(http://wiki2.dovecot.org/AuthDatabase/PasswdFile) and see if you can reproduce 
the slowdowns with them.



Re: [Dovecot] inet_listener imaps { port = 0 } question

2012-06-01 Thread Timo Sirainen
On 31.5.2012, at 16.58, henrixd wrote:

 Why commenting out inet_listener imaps {} won't stop dovecot to listen port 
 993? I think this would be expected behavior. Just curious, finally got it 
 working with port = 0. :)

When you comment out something, Dovecot uses the default settings for it. By 
default Dovecot listens on port 993.



Re: [Dovecot] dovecot stats: useful data to gather

2012-06-01 Thread Timo Sirainen
On 1.6.2012, at 23.58, Patrick Ben Koetter wrote:

 Besides pulling together all the data we also think it would be useful to have
 an SNMP interface to access the stats.

I had thought about SNMP before also, but for the current kind of stats that 
are exported I couldn't think of any reasonable way to export them.

 Here are the stats we believe to be useful:
 
 Login/Logout
 - total number login success/time
 - total number login failure/time
..

I'll look at these later in more detail, but some important questions / design 
decisions:

Currently stats process only remembers things after Dovecot was started. I 
don't think getting these kind of numbers would really work like that. Perhaps 
all of the statistics should be permanently dumped to disk every ~minute or so 
+ at shutdown and loaded at startup, so the numbers would at least normally 
always just increase since the first time Dovecot was started?

 Mailbox state
 - Inflow rate (number incoming messages/time)
 - Deleted rate (number \Deleted flagged messages/time)

These operations/time type of things I had hoped to be able to externalize :) 
If stats process simply gives the raw stats, the reader could do this kind of 
summing up. Otherwise .. well, I guess it could maybe keep track of the current 
ops/last 60 secs and the reader would then have to read the value about once 
a minute or half or something. It wouldn't give exact results though.

 Performance
 - minimum time to write a message
 - maximum time to write a message
 - average time to write a message

Within last .. day? hour? minute? ..

Re: [Dovecot] best practises for mail systems

2012-06-05 Thread Timo Sirainen
On 5.6.2012, at 6.14, Костырев Александр Алексеевич wrote:

 - not quite sure if glusterfs is production ready solution 'cause I've 
 experienced split-brains during setting it up

Last I've heard glusterfs causes corruption problems with Dovecot. You should 
try stress testing it with imaptest: http://imapwiki.org/ImapTest



Re: [Dovecot] [ Re: best practises for mail systems]

2012-06-05 Thread Timo Sirainen
On 5.6.2012, at 23.33, Michescu Andrei wrote:

 I agree, in practice this is not an issue compared to the unavailability
 of the service, but on longer IMAP sessions (e.g. transferring a big
 file) the connection loss is noticeable.
 
 It is noticeable for somebody that really waits for a large email.

And there is actually some (any!) way this could be avoided?... One server 
dies, another continues sending the mail?

I have had some thoughts about transferring idling Dovecot connections between 
processes / servers so that clients wouldn't notice it, but I haven't even 
thought about moving active (long-running) connections.



Re: [Dovecot] upgrade from 1.0.5 to 2.1.7

2012-06-08 Thread Timo Sirainen
On 8.6.2012, at 14.05, Andreas Meyer wrote:

 I want to upgrade the docevot-installation from v 1.0.5 to 2.1..7
 
 Now I get the following executing
 doveconf -n -c /etc/dovecot/dovecot.conf  /home/mail1/dovecot-2.conf

Didn't this command produce a working dovecot-2.conf file? If not, it's 
probably a bug.

 doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:217: add 
 auth_ prefix to all settings inside auth {} and remove the auth {} section 
 completely
 doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:219: passdb 
 passwd-file {} has been replaced by passdb { driver=passwd-file }
 doveconf: Warning: Obsolete setting in /etc/dovecot/dovecot.conf:222: userdb 
 passwd-file {} has been replaced by userdb { driver=passwd-file }
..
 How do I change it to fullfill the new needs?

doveconf should have done all of those changes for you and placed them to 
dovecot-2.conf



Re: [Dovecot] upgrade from 1.0.5 to 2.1.7

2012-06-08 Thread Timo Sirainen
On 8.6.2012, at 18.36, Andreas Meyer wrote:

 Jun 08 17:20:19 imap: Error: 
 dlopen(/usr/lib/dovecot/modules/imap/lib10_quota_plugin.so) failed: 
 /usr/lib/dovecot/modules/imap/lib10_quota_plugin.so: 
 
 What can I do? Wouldn't it be great to get the new dovecot working with
 my users and the old passwd file?

The quota plugin isn't against the same version of Dovecot.. So you have two 
Dovecot versions now somehow all mixed up. One solution would be to delete all 
files related to Dovecot and install 2.1.7 again.



Re: [Dovecot] difference between client_limit and process_limit

2012-06-08 Thread Timo Sirainen
On 8.6.2012, at 14.12, Angel L. Mateo wrote:

 What is the real difference between client and process limit? According
 to documentation (http://wiki2.dovecot.org/Services#Service_limits):
 
   Sorry, it's friday, my mind is on the weekend :-(
 
   I understand that client_limit is how many connections (imap 
 connections, for example) could be handle by one dovecot process, so if I 
 have client_limit=2 and process_limit=1024, then I could 2048 concurrent 
 connections, right?

Yes, but like the wiki page also says, it's not a good idea increase 
client_limit for imap/pop3 processes.



Re: [Dovecot] Director pop3 real ips v2.1.1

2012-06-08 Thread Timo Sirainen
On 8.6.2012, at 7.39, Костырев Александр Алексеевич wrote:

 yes, I use v.2.1.1 on both director and backends
 and yes, I've added 
 login_trusted_networks = 192.168.5.0/24 on all of them
 but it didn't help.


Missing feature:

v2.1.2 2012-03-15  Timo Sirainen t...@iki.fi

+ Proxying: POP3 now supports sending remote IP+port from proxy to
  backend server via Dovecot-specific XCLIENT extension.



Re: [Dovecot] auth trouble

2012-06-08 Thread Timo Sirainen
On 6.6.2012, at 2.08, Glenn English wrote:

 And these brute force attempts would be logged, each one.
 
 They are, with no rhost. And there are other brute force attempts 
 that *do* have IPs.

I think the answer to this is simply that Dovecot v1.0 didn't tell PAM the 
rhost. Upgrade.



Re: [Dovecot] difference between client_limit and process_limit

2012-06-08 Thread Timo Sirainen
On 8.6.2012, at 19.33, Reindl Harald wrote:

 Yes, but like the wiki page also says, it's not a good idea increase 
 client_limit for imap/pop3 processes.
 
 depends on the usecase / workload
 
 having dovecot as proxy for other imap-backends and 1 process per connection
 will heavily raise up process-count and memory-overhead while memory
 may be needed for the imap-backend (like dbmail) and datanases
 
 process_limit= 15
 client_limit = 300
 
 this way you can have 4500 proxy-connections and use most time
 not more than 4-5 processes

Proxying is done by imap-login process, not imap process. For login processes 
there are different recommendations.



Re: [Dovecot] [ Re: best practises for mail systems]

2012-06-09 Thread Timo Sirainen
On 9.6.2012, at 4.55, Matthias-Christian Ott wrote:

 Yes, there is. You have to replicate the entire state of the IMAP
 session (protocol states, buffers, TLS state etc.) and the TCP state of
 the connection. The state of the IMAP session is (in theory) easily
 replicable (although you probably have to rely on internals of the TLS
 implementation; OpenSSL can serialise TLS sessions from/into ASN.1 via
 i2d_SSL_SESSION, though this is meant to resume session via TLS)

Interesting! I thought OpenSSL didn't have a way to [de]serialize the session 
state. The first time I wanted to do that was 13 years ago. I see there are 
some google hits for i2d_SSL_SESSION, but do you already know a good web page / 
example code I could look at?

 and for
 TCP there is RTCP [1]. RTCP intercepts the TCP session is able to
 recover the TCP state. It works without any modification of the
 operating system (at the moment limited to Linux).

Thanks for this too.

 If this would be implemented in Dovecot it would really set it apart
 from other IMAP servers and software that I've seen so far. Being able
 to transparently handle failover of a TCP connection is unique.

Yes.

Re: [Dovecot] Dovecot 1.x on AIX - Dovecot 2.x on Ubuntu

2012-06-09 Thread Timo Sirainen
On 6.6.2012, at 23.27, r...@yuma.acns.colostate.edu wrote:

 We are working on migrating Dovecot 1.2.17 running on AIX 5.3 (believe it
 or not!) to Dovecot 2.0.13 running on Ubuntu.  We have hundreds of users
 mboxes we will be migrating.  My question is regarding the index files.
 Should we remove those after the migration, but before we open it up to
 users so Dovecot can create new ones?
 
 I did a test migration of a single user, and Dovecot detects the
 architecture change and put out some panic errors, corrupt files and

Yeah, there's still some problem with properly handling index file recreation 
when CPU architecture (endianess) change is detected. Better just delete your 
index files, since they have to be regenerated anyway.



Re: [Dovecot] Deliver quota-warning via director

2012-06-09 Thread Timo Sirainen
On 9.6.2012, at 22.11, Daniel Parthey wrote:

 But it seems that lda delivers the mail directly to
 the local filesystem and is not using our lmtp director,
 which prevents NFS mailboxes from getting corrupted.
 
 Is there a way to tell lda to use LMTP or the director
 and ignore the quota while delivering the notification?

That's a bit tricky problem. Even if LDA used LMTP, it couldn't ignore quota 
since LMTP server is the one enforcing it. Perhaps you need to create two LMTP 
ports, one with a quota ignored configuration. Then you need to somehow get 
the mail delivered there (maybe send it to your MTA and route it from there). 
Or write a script that sends the mail directly to the LMTP port on director.



Re: [Dovecot] Upgrading 1.2.17 - 2.1.x

2012-06-09 Thread Timo Sirainen
On 9.6.2012, at 0.53, Adam G Tilghman wrote:

 
 We're planning to upgrade our site from 1.2.17 to 2.1.x within the
 next few months, but we must ensure our ability to revert to 1.2.17
 if problems arise.
 
 I don't expect our maildir storage would present a problem,
 but am less certain about 2.1.x index/control files remaining
 readable under 1.2.17.
 
 Should I have any reason to worry?

1.2.17 can read v2.0 indexes without problems (it has some forwards 
compatibility code). I don't think I added any incompatible changes to v2.1 
either, at least nothing major..



Re: [Dovecot] Director problems

2012-06-09 Thread Timo Sirainen
On 6.6.2012, at 16.01, Joseba Torre wrote:

 I've just setup a testing enviroment for director, and it's not working as 
 expected. I have just 1 director (called director) and 2 dovecot servers 
 (dovecot1 and dovecot2); these are exact copies.
 
 First problem: when both dovecot servers are up, every imap connection is 
 redirected to the same server as you can see here:
 
 $ sudo doveadm director map
 user  mail server ip expire time 
 unknown 158.227.4.186  2012-06-06 
 13:34:12
 unknown 158.227.4.186  2012-06-06 
 13:34:27
 unknown 158.227.4.186  2012-06-06 
 13:34:34
 
 (I don't know if that unknown is good or not)
 
 I've tried with 3 different users and ips to no change, users are always 
 directed to the same host.

Perhaps you just managed to use such usernames that map to the same director.. 
You can try with doveadm director status user to see where they should go.

 Second problem: if I try to add/remove/modify one of the dovecot servers, the 
 output of doveadm director map/status seems to be ok, but any new user 
 connection fails with this log:
 
 Jun  6 14:51:59 director dovecot: director: Warning: Delaying new user 
 requests until ring is synced

Looks like there's a bug when only one director is used. I'll try and fix it 
later..

Re: [Dovecot] director: backend health monitoring

2012-06-09 Thread Timo Sirainen
On 8.6.2012, at 4.25, Костырев Александр Алексеевич wrote:

 I am wonder if there are plans to include backend health monitoring feature 
 to Dovecot Director ?

Yes, but it's not a very high priority right now.

Re: [Dovecot] difference between client_limit and process_limit

2012-06-11 Thread Timo Sirainen
On 11.6.2012, at 13.19, Angel L. Mateo wrote:

 Proxying is done by imap-login process, not imap process. For login 
 processes there are different recommendations.
 
   What are those recommendations? The ones at 
 http://wiki2.dovecot.org/LoginProcess?

Yes.

   Let's suppose... I have 4 mainly imap backend servers (but they admit 
 also pop3 connections) with a process_limit of 5120 for service imap (and 
 default_client_limit of 1000 applied to pop3). And I have 2 director servers 
 (configured as active-active behind a load balancer), so I need director 
 servers to handle (more or less) 10240 imap connections.
 
   What is it better for the director's? Increasing process_limit for 
 imap-login (so each process should handle less connections) or increasing 
 client_limit (less processes handling more connections each)?

If you increase process_limit to more than the number of CPU cores you have, 
then you increase the number of context switched done by the kernel, which 
decreases your performance. So I'd say increase client_limit.



Re: [Dovecot] Frequently login problem

2012-06-11 Thread Timo Sirainen
On 4.6.2012, at 13.20, Jitendra Bhaskar wrote:

 I am using dovecot 2.1.3 on centos 5.7. It was working fine but last few
 days I need to restart or reload dovecot service because at that time users
 are not able to login.
 
 Each time I am getting information from doveco.log is as :
 Jun 04 11:52:54 auth: Error: BUG: Authentication client gave a PID 17564 of
 existing connection

This happens before restart, not during it? doveconf -n output? Are you using 
Dovecot auth for anything external, like Postfix/Exim?



Re: [Dovecot] Very High Load on Dovecot 2 and Errors in mail.err.

2012-06-11 Thread Timo Sirainen
On 11.6.2012, at 11.09, Jesper Dahl Nyerup wrote:

 Stracing the processes in D state from before they hang has just
 revealed something interesting, however, pointing to an issue with
 inotify rather than epoll.
 
 [snip]
 [...]
 15414 23:27:36 inotify_init()   = 12 0.24
 [...]
 15414 23:27:36 close(12 unfinished ...
 15414 23:28:51 ... close resumed )= 0 74.593917
 15414 23:28:51 close(9 unfinished ...
 15414 23:28:51 ... close resumed )= 0 0.80
 15414 23:28:51 exit_group(0)= ?
 [/snip]
 
 In short, as far as we can tell, all the processes in D state appear to
 be waiting to close the file handle they got from their inotify_init(),
 and eventually all these close()s go through almost simultaneously.

Yeah. Looks like a kernel bug. You could try if it goes away by disabling 
inotify in Dovecot. Either recompile with configure --with-notify=none or 
maybe you can disable inotify globally with:

echo 0  /proc/sys/fs/inotify/max_user_watches
echo 0  /proc/sys/fs/inotify/max_user_instances


Re: [Dovecot] Dovecot over NFS

2012-06-11 Thread Timo Sirainen
On 7.6.2012, at 1.07, James Devine wrote:

 I'm playing with running dovecot over NFS and I am running into some
 issues.  I have followed the guide at  http://wiki2.dovecot.org/NFS and my
 setup includes 1 nfs server and 1 client running postfix/dovecot.

Which NFS server? Which NFS client (Linux)?

  In
 testing I am running postal via the command:
 
 postal -t 10 -c 10 localhost users399
 
 The test file has a list of 399 users to deliver to.  I've provided a
 sample of the errors I'm receiving and my configuration below, I am running
 dovecot 2.0.19.  Any idea what I might be doing wrong and what I might do
 to resolve it?  My ultimate goal is to setup multiple clients with director
 so each user is still handled on a single machine, however with a single
 machine I still seem to be having issues.
..
 Jun  6 15:55:12 test-gluster-client1 dovecot: lmtp(12072, testuser130):
 Error: mdbox /mnt/testuser130/mdbox/mailboxes/INBOX/dbox-Mails: Invalid
 dbox header size: 0

Yeah, something's broken. I'd try:

1. Try Dovecot v2.1.7. I don't think v2.0.19 had these problems anymore but 
wouldn't hurt to try.

2. Try if you can reproduce the same problem with local filesystem.

3. Try another NFS server or client..

Re: [Dovecot] dovecot does not find libpam when compiling with customized prefix

2012-06-11 Thread Timo Sirainen
On 11.6.2012, at 8.20, Roland wrote:

 I try to compile dovecot 2.1.7 with a customized --prefix setting and 
 --with-pam . Although I installed libpam into the same --prefix, dovecot does 
 not find it:
 
 checking for pam_start in -lpam... no
 configure: error: Can't build with PAM support: libpam not found
 
 The same or a similar problem seems to have appeared 4 years ago:
 http://www.dovecot.org/list/dovecot/2008-February/028750.html
 
 Which libpam file does dovecot expect in which directory? And possibly there 
 is something wrong with the pam_start function?

Dovecot doesn't expect anything. gcc/ld expects things. You'll need to use the 
generic options to tell where the PAM files are, something like:

LDFLAGS=-L/where/is/pam/lib CPPFLAGS=-I/where/is/pam/include ./configure



Re: [Dovecot] director: non standart ports at backends

2012-06-11 Thread Timo Sirainen
On 11.6.2012, at 12.27, Костырев Александр Алексеевич wrote:

 hello,
 I'm trying to figure out how to proxy pop3 and pop3s that listens on 
 non-standart ports at backends.
 For example, pop3 is at 1110 and pop3s at 1995 (on backend side).
 is it possible? 
 how should I separate this ports in director's config?
 it's easy for one port:
 for example lmtp - you just use passdb in protocol lmtp {}

The passdb needs to return the port field. You can't use static passdb for 
this, since it has no conditionals and you can't do per-port configuration. 
Maybe use sqlite (simply to use it as a scripting engine - empty database) or 
checkpassword as your passdb.



Re: [Dovecot] Error: doveadm client attempted non-PLAIN authentication

2012-06-11 Thread Timo Sirainen
On 10.6.2012, at 2.56, Daniel Parthey wrote:

 doveadm search -u u...@example.org -S localhost:19000 all
 produces the following error in the logs:
 dovecot: doveadm: Error: doveadm client attempted non-PLAIN authentication
 
 What am I missing?

It's possible that this is just broken in v2.0. Try v2.1.



Re: [Dovecot] auth_krb5_keytab ignored ?

2012-06-11 Thread Timo Sirainen
On Fri, 2012-06-08 at 18:59 +0200, Leon Meßner wrote:
 Hi list,
 
 i noticed that when doing imap gssapi authentication with kerberos,
 dovecot (here 2.1.7) always searches /etc/krb5.keytab although i have
 auth_krb5_keytab = /etc/mail3.krb5.keytab in my etc/dovecot/dovecot.conf
 and doveconf -n also show this setting. If i combine the keytabs in
 krb5.keytab it works. Is there another location where i should put my
 configuration regarding gssapi/kerberos ?

Try if this works:

import_environment = TZ GDB DEBUG_SILENT KRB5_KTNAME 

Then start Dovecot with:

KRB5_KTNAME=/etc/mail3.krb5.keytab dovecot

I'm wondering if the code in mech-gssapi.c that sets KRB5_KTNAME
environment is being called too late.




Re: [Dovecot] how to announce shared folders to clients using non-default mail prefix

2012-06-11 Thread Timo Sirainen
On 8.6.2012, at 3.34, Tom Lieuallen wrote:

 Note that if I change the prefix for that shared namespace to 'iphonemail/', 
 it does present my shared folders as well as anything in a personal 
 iphonemail directory.  However, 'select' didn't work with the personal 
 folders.  My guess is it's mostly due to the difference in mail formats 
 between the two (mbox  maildir).  

You should be able to use prefix=iphonemail/shared/



Re: [Dovecot] Dovecot auth process delays exiting if LDAPS passdb used

2012-06-11 Thread Timo Sirainen
On 7.6.2012, at 6.06, SATOH Fumiyasu wrote:

 Dovecot auth process has a problem
 that Dovecot auth delays exiting about between 20 and
 60 seconds when Dovecot dovecot (master) process is already
 terminated by an administrator.
 
 Yes. I can reproduce with dovecot 1:2.1.7-1 (Debian unstable package)
 with PAM passdb. This PAM environment is configured for
 local UNIX passwd file only (no LDAP).

I can't reproduce this. I installed the 1:2.1.7-1 Debian unstable package. Put 
your dovecot.conf to /etc/dovecot/. Did:

/etc/init.d/dovecot start
telnet localhost 143
x login foo bar
x logout
/etc/init.d/dovecot stop

No dovecot processes left.



Re: [Dovecot] Accessing maildir snapshots through dovecot / namespace

2012-06-11 Thread Timo Sirainen
On 7.6.2012, at 18.26, Karl Oulmi wrote:

 namespace snap {
 prefix = INBOX.snapshot.h0.
 hidden = no
 inbox = no
 list = yes
 location = 
 maildir:/da1/%u/Maildir:INDEX=/da1/dovecot/indexes/%u:CONTROL=/da1/dovecot/control/%u
 type = private
 }
 
 
 The problem is that I don't see the content of the inbox folder contained in 
 the snapshots whereas subfolders are perfectly viewed !

The INBOX should be accessible as the INBOX.snapshot.h0 itself.



Re: [Dovecot] Different but probably related issue

2012-06-11 Thread Timo Sirainen
On 5.6.2012, at 11.09, Johannes Berg wrote:

 Unfortunately, I don't. I can only suggest, as a test, trying with some
 other storage format -- I only use Maildir -- to see if the problem is
 really in the interaction with mdbox. I'm fairly sure that's likely the
 problem, maybe the plugin doesn't pass something through append that is
 needed by mdbox, but I've never even attempted to understand mdbox.
 
 Maybe Timo can comment. Timo, you can find the latest code here:
 http://git.sipsolutions.net/?p=dovecot-antispam.git;a=summary

I don't see anything obviously wrong in there.. Perhaps antispam_save_finish() 
returns failure for some reason and dbox doesn't handle that properly?



Re: [Dovecot] director: non standart ports at backends

2012-06-11 Thread Timo Sirainen
Looking at your old mails, you seem to be using passdb static for director, but 
userdb sql? So you could switch to:

passdb {
 driver = sql
 args = /etc/dovecot/dovecot-sql.conf
}

password_query = select 'y' as proxy, 'y' as nopassword, if('%a'=143, 1430, 
9930) as port

where you'd change the if() to something that handles %s=imap vs %s=pop3 vs 
%s=lmtp and %a=143 vs %a=993 vs %a=110 vs %a=995. Maybe a case statement 
would be less ugly. Or simply make it a real table in sql. Anyway, that's the 
basic idea.

On 11.6.2012, at 15.39, Костырев Александр Алексеевич wrote:

 thanks Timo, for you time
 but I still don't get it)
 should I return port with just port_num1,port_num2 value or how?
 I've tried to google an example but with no success.
 
 -Original Message-
 From: Timo Sirainen [mailto:t...@iki.fi] 
 Sent: Monday, June 11, 2012 11:01 PM
 To: Костырев Александр Алексеевич
 Cc: dovecot@dovecot.org
 Subject: Re: [Dovecot] director: non standart ports at backends
 
 On 11.6.2012, at 12.27, Костырев Александр Алексеевич wrote:
 
 hello,
 I'm trying to figure out how to proxy pop3 and pop3s that listens on 
 non-standart ports at backends.
 For example, pop3 is at 1110 and pop3s at 1995 (on backend side).
 is it possible? 
 how should I separate this ports in director's config?
 it's easy for one port:
 for example lmtp - you just use passdb in protocol lmtp {}
 
 The passdb needs to return the port field. You can't use static passdb for 
 this, since it has no conditionals and you can't do per-port configuration. 
 Maybe use sqlite (simply to use it as a scripting engine - empty database) or 
 checkpassword as your passdb.
 



Re: [Dovecot] fts_lucene crashing

2012-06-11 Thread Timo Sirainen
On 30.5.2012, at 22.13, Joe Beaubien wrote:

 May 22 14:51:51 mba dovecot: imap(formulaire): Panic: file
 lucene-wrapper.cc: line 196: unreached
 
 Thanks for the new release. Unfortunately, it doesn't seem to have fixed my
 specific issue. I got you a gdb trace like you asked in a previous mail. I
 hope that can help. If I didn't get the correct backtrace, or if you need
 some other info from gdb let me know.


Thanks. The problem was pretty far away from where I thought it was. Fixed: 
http://hg.dovecot.org/dovecot-2.1/rev/0fde692cb565



Re: [Dovecot] dsync migration with preserving pop3 uidl

2012-06-11 Thread Timo Sirainen
On 24.5.2012, at 13.17, Tomáš Herceg wrote:

 I'm trying to migrate messages from icewarp (merak) mailserver to dovecot via
 dsync, IMAP migration is looking fine, but I'm unable to migrate pop3 uidls 
 from
 originating server, probably is something wrong with configuration, but I 
 don't
 know what. The only documentation i found is on the wiki:
 http://wiki2.dovecot.org/Migration/Dsync where is bad writen mail_plugins =
 pop3-migration, i corrected it to mail_plugins = pop3_migration, but it still
 didn't work, here is my configuration:
..
 namespace {
  hidden = yes
  list = yes

list=no would be better so clients don't accidentally access this.

  location = pop3c:
  prefix = POP3/
 }
 I'm runnig dsync this way:
 /usr/bin/time -f %E doveadm -vD -o imapc_user=te...@irock.cz -o
 imapc_password=* backup -u te...@irock.cz -f -R imapc:/tmp-ram/imapc-test1

You need to change pop3c_user and pop3c_password also in this command line.

 dsync(te...@irock.cz): Error: stat((null)) failed: Bad address
 dsync(te...@irock.cz): Error: stat((null)) failed: Bad address

I wonder what these are.

Also I wonder why the weren't any messages about missing/wrong user+pass for 
pop3c.

Re: [Dovecot] multi-instance doveadm user -m woes

2012-06-11 Thread Timo Sirainen
On 11.5.2012, at 18.06, David Warden wrote:

 I'm having difficulty with the doveadm who command on a multi-instance setup 
 of dovecot. When I run the who command on the non-standard instance with the 
 -m flag (to see their mail location), this happens:
 
 [root@wardentest3 dovecot]# doveadm -i mailtest user -m warden
 doveadm(root): Error: user warden: Initialization failed: Namespace 'INBOX.': 
 Ambiguous mail location setting, don't know what to do with it: 
 /var/spool/mail/root (try prefixing it with mbox: or maildir:)

Thanks, fixed:

http://hg.dovecot.org/dovecot-2.1/rev/98f2c12eccdb



Re: [Dovecot] Director problems

2012-06-11 Thread Timo Sirainen
On 11.6.2012, at 13.43, Joseba Torre wrote:

 I've tried with 3 different users and ips to no change, users are always 
 directed to the same host.
 
 Perhaps you just managed to use such usernames that map to the same 
 director.. You can try with doveadm director statususer to see where 
 they should go.
 
 I was thinking that users where sent to one server or another in a more or 
 less random way. As always, your guess was right, test[1-4] are all sent to 
 the same server, but for example jorge is sent to the other one.

The randomness is basically md5(username)%2.

 Second problem: if I try to add/remove/modify one of the dovecot servers, 
 the output of doveadm director map/status seems to be ok, but any new user 
 connection fails with this log:
 
 Jun  6 14:51:59 director dovecot: director: Warning: Delaying new user 
 requests until ring is synced
 
 Looks like there's a bug when only one director is used. I'll try and fix it 
 later..
 
 Thanks a lot for your support

Fixed: http://hg.dovecot.org/dovecot-2.1/rev/46d01b728647



Re: [Dovecot] Dovecot auth process delays exiting if LDAPS passdb used

2012-06-11 Thread Timo Sirainen
On 11.6.2012, at 19.39, SATOH Fumiyasu wrote:

 At Mon, 11 Jun 2012 18:32:35 +0300,
 Timo Sirainen wrote:
 If an auth client remains a connection to dovecot/auth,
 dovecot/auth does NOT exit immediately when dovecot master exits.
 
 Ah, now we're getting somewhere :) Yes, this is correct and intentional. But 
 it should still close the listeners, so this shouldn't happen:
 
 May 24 00:42:10 build-aix6 mail:err|error dovecot: master: Error: 
 service(auth): Socket already exists: /opt/osstech/var/run/dovecot/auth-login
 
 (6) Stop dovecot service.
 
 # /etc/init.d/dovecot stop
 
 And (7) /etc/init.d/dovecot start fails?
 
 Yes: AIX 6.1, 7.1
 No: Debian GNU/Linux stable, testing, unstable / Solaris 10

OK, so this is AIX specific. Two problems: 1) I have no access to AIX to test 
and debug this, 2) even if I did, I'm not very motivated in debugging possibly 
hours for a system that is very rarely used in email servers.. (If any AIX user 
wanted to buy one of the Dovecot support services, I could look into this and 
get it fixed in some way.)

It would also be possible to modify the sources a bit to get the pending 
processes killed immediately at shutdown.

Re: [Dovecot] 2.1.7 shared folder index issued

2012-06-11 Thread Timo Sirainen
On 11.6.2012, at 22.05, Alex Crow wrote:

 Sorry to bother the list again so soon after fixing my own problem, but I now 
 have this issue when clients try to view shared folders in Thunderbird (v12). 
 They can see the shared folder, but the first time they click on in nothing 
 happens. The second time they get an authentication failure. The third or 
 fourth time it finally loads the shared mailbox, and I see this a few times 
 in the logs:
 
 Jun 11 19:57:43 alsace dovecot: imap(sharedvie...@integrafin.co.uk): Error: 
 mdbox map 
 /home/indexes/integrafin.co.uk/t/sharedviewer/shared/sharedvie...@integrafin.co.uk/storage/dovecot.map.index
  corrupted: U
 nexpectedly lost shared/vie...@integrafin.co.uk/INBOX uid=73129 map_uid=74192

http://wiki2.dovecot.org/SharedMailboxes/Shared#dbox



[Dovecot] v2.0.21 released

2012-06-11 Thread Timo Sirainen
http://dovecot.org/releases/2.0/dovecot-2.0.21.tar.gz
http://dovecot.org/releases/2.0/dovecot-2.0.21.tar.gz.sig

+ dict: file backend supports now also fcntl/flock locking optionally
- imap-login: Memory leak fixed
- imap: Non-UTF8 input on SEARCH command parameters could have crashed
- auth: Fixed crash with DIGEST-MD5 when attempting to do master user
  login without master passdbs. 
- sdbox: Don't use more fds than necessary when copying mails.
- mdbox kept the user's storage locked a bit longer than it needed to



Re: [Dovecot] 2.1.7 shared folder index issued

2012-06-11 Thread Timo Sirainen
On 11.6.2012, at 23.35, Alex Crow wrote:

 Jun 11 19:57:43 alsace dovecot: imap(sharedvie...@integrafin.co.uk): Error: 
 mdbox map 
 /home/indexes/integrafin.co.uk/t/sharedviewer/shared/sharedvie...@integrafin.co.uk/storage/dovecot.map.index
  corrupted: U
 nexpectedly lost shared/vie...@integrafin.co.uk/INBOX uid=73129 
 map_uid=74192
 http://wiki2.dovecot.org/SharedMailboxes/Shared#dbox
 
 
 
 Thanks Timo,
 
 So should I just remove the INDEX part from the shared namespace? Or should I 
 have the INDEX point to the sharer's indexes rather than the sharee?

That depends on if the regular mail_location has any INDEX or not. In any case 
they must point to the same index.



Re: [Dovecot] Very High Load on Dovecot 2 and Errors in mail.err.

2012-06-11 Thread Timo Sirainen
On 12.6.2012, at 0.37, Jesper Dahl Nyerup wrote:

 Yeah. Looks like a kernel bug. You could try if it goes away by disabling 
 inotify in Dovecot. Either recompile with configure --with-notify=none or 
 maybe you can disable inotify globally with:
 
 echo 0  /proc/sys/fs/inotify/max_user_watches
 echo 0  /proc/sys/fs/inotify/max_user_instances
 
 I can confirm that this removes the symptoms, and that it doesn't affect
 the service. Obviously IDLEing users are now only notified upon polling
 of the file system, but the I/O overhead of doing this seems minimal.

It actually doesn't increase I/O overhead at all. Dovecot always does polling, 
even with inotify, since inotify doesn't necessarily work with shared 
filesystems (e.g. NFS). The main difference is that users don't get immediate 
notifications of new mails now, but have to wait for 
mailbox_idle_check_interval.



Re: [Dovecot] Sieve: Mailbox doesn't exist

2012-06-12 Thread Timo Sirainen
On 12.6.2012, at 20.00, Pascal Volk wrote:

 On 06/12/2012 01:32 AM Stefan Fricke wrote:
 I have just set up a mail server on Ubuntu, using Postfix and Dovecot 
 2.0.19. 
 It works well but I can't get Sieve working. I always get the error that the 
 target mailbox doesn't exist. In fact it doesn't but isn'r Dovecot supposed 
 to 
 create it?
 
 Not with your current configuration. See:
 http://hg.dovecot.org/dovecot-2.0/file/2.0.19/doc/example-config/conf.d/15-lda.conf#l39

Even better (more standard): Use fileinto :create box;



Re: [Dovecot] gnutls support

2012-06-12 Thread Timo Sirainen
On 12.6.2012, at 21.56, Daniel L. Miller wrote:

 On 9/16/2010 5:41 AM, Timo Sirainen wrote:
 On Wed, 2010-09-15 at 20:25 -0700, Daniel L. Miller wrote:
 Other than license issues, is there an advantage to using gnutls vs
 openssl?  Or is openssl superior - at least in the current implementations?
 Dovecot's GNUTLS support was written long time ago and its API has
 changed since. It doesn't work. But a working GNUTLS support would still
 be nice some day. I don't much like OpenSSL.
 With 2.1.7 - is GNUTLS supported?

No, and I have no plans to add it. But I don't mind if someone sends a patch.



Re: [Dovecot] difference between client_limit and process_limit

2012-06-13 Thread Timo Sirainen
On Wed, 2012-06-13 at 13:15 +0200, Angel L. Mateo wrote:
   In my test environment I have configured (this is extracted from 
 doveconf -n output):
 
 service imap-login {
client_limit = 10740
executable = imap-login director
process_limit = 1
process_min_avail = 1
 }
 
   When I made the first connection, there's no problem, but if I try a 
 second while the first is still open, I get:
 
 Jun 13 13:09:12 myotis40 dovecot: master: Warning: service(imap-login): 
 client_limit (1) reached, client connections are being dropped
 
   Why is telling me that client_limit is reached? What client_limit is 
 used?

Dunno. What Dovecot version? Show the whole doveconf -n? You don't have
multiple dovecot.confs, right?




Re: [Dovecot] difference between client_limit and process_limit

2012-06-13 Thread Timo Sirainen
On Wed, 2012-06-13 at 15:28 +0300, Timo Sirainen wrote:
 Oh, right, service_count=1 is the default and that overrides
 client_limit. Set it to 0.

http://hg.dovecot.org/dovecot-2.1/rev/4c31e450a867




Re: [Dovecot] doveadm doesn't subscribe to public folders

2012-06-13 Thread Timo Sirainen
Does it work if you do it via imap?

echo a subscribe public.Conferences | /usr/local/libexec/dovecot/imap -u rago

On Wed, 2012-06-13 at 11:59 +0200, Emiliano Rago wrote:
 Hi,
 
 what it's happening with the doveadm command below is that
 the file modified is /data/MAIL/PUBLIC/subscriptions
 while I'd like to modify the file /data/MAIL/rago/subscriptions
 
 With subscriptions=no every user can subscribe to public folder,
 so perhaps this behaviour is inappropriate; suggestions?
 
 Thanks,
 Emiliano Rago
 
 
 On 06/12/2012 02:08 PM, Emiliano Rago wrote:
  Hi,
 
  I'd like to subscribe folder with doveadm:
 
  doveadm mailbox subscribe -u rago public.Conferences
 
  This command doesn't work, while it works with an ordinary folder.
  However it's possible to subscribe to the folder with an imap connection:
 
  1 login rago mypasswd
  1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
  IDLE SORTSORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT
  CHILDREN NAMESPCE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC
  ESEARCH ESORT SEARCHRES ITHIN CONTEXT=SEARCH LIST-STATUS ACL
  RIGHTS=texk] Logged in
  2 LSUB  *
  * LSUB () . INBOX
  2 OK Lsub completed.
  3 SUBSCRIBE public.Conferences
  3 OK Subscribe completed.
  4 LSUB  *
  * LSUB () . INBOX
  * LSUB () . public.Conferences
  4 OK Lsub completed.
 
  Am I doing anything wrong?
 
  This is my conf, thx for help,
  Emiliano
 
  # 2.0.9: /etc/dovecot/dovecot.conf
  # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 Red Hat Enterprise Linux
  Server release 5 (Tikanga) ext4
  auth_cache_size = 128 M
  auth_master_user_separator = *
  auth_mechanisms = plain cram-md5
  mail_location =
  maildir:/data/MAIL/MAILDIR/%u:INBOX=/data/MAIL/INBOX/%u:INDEX=/data/MAIL/METADATA/%u
 
  maildir_very_dirty_syncs = yes
  managesieve_notify_capability = mailto
  managesieve_sieve_capability = fileinto reject envelope
  encoded-character vacation subaddress comparator-i;ascii-numeric
  relational regex imap4flags copy include variables body enotify
  environment mailbox date
  mbox_write_locks = fcntl
  namespace {
  inbox = yes
  location =
  prefix =
  separator = .
  type = private
  }
  namespace {
  list = children
  location =
  maildir:/data/MAIL/MAILDIR/%%u:INDEX=/data/MAIL/METADATA/SHARED/%u/%%u
  prefix = shared.%%u.
  separator = .
  subscriptions = no
  type = shared
  }
  namespace {
  list = children
  location = maildir:/data/MAIL/PUBLIC:INDEX=/data/MAIL/METADATA/PUBLIC
  prefix = public.
  separator = .
  subscriptions = no
  type = public
  }
  passdb {
  args = scheme=cram-md5 /data/PWDDB/cram_dovecot.txt
  driver = passwd-file
  }
  passdb {
  args = /etc/dovecot/master-shared
  driver = passwd-file
  master = yes
  }
  passdb {
  args = /etc/dovecot/master-shared
  driver = passwd-file
  }
  plugin {
  acl = vfile:/etc/dovecot/global-acls:cache_secs=300
  acl_anyone = allow
  acl_shared_dict = file:/data/MAIL/SHAREDDB/shared-mailboxes.db
  sieve = /data/MAIL/SIEVE/%u/dovecot.sieve
  sieve_dir = /data/MAIL/SIEVE/%u
  }
  postmaster_address = root
  protocols = imap sieve
  service auth {
  unix_listener /var/spool/postfix/private/auth {
  mode = 0666
  }
  unix_listener auth-userdb {
  group = mailreader
  mode = 0600
  user = mailreader
  }
  }
  service imap-login {
  process_min_avail = 8
  service_count = 0
  vsz_limit = 512 M
  }
  service imap-postlogin {
  executable = script-login /etc/dovecot/postlogin.sh
  user = $default_internal_user
  }
  service imap {
  executable = imap imap-postlogin
  }
  ssl_cert = /etc/dovecot/ssl/imaps.lal.in2p3.fr.crt
  ssl_key = /etc/dovecot/ssl/imaps.lal.in2p3.fr.key
  userdb {
  args = /etc/dovecot/master-shared
  driver = passwd-file
  }
  userdb {
  args = uid=mailreader gid=mailreader home=/data/MAIL/SIEVE/%u
  allow_all_users=yes
  driver = static
  }
  protocol lda {
  mail_plugins = acl sieve
  }
  protocol imap {
  mail_max_userip_connections = 128
  mail_plugins = acl imap_acl
  }
 




Re: [Dovecot] Problem with lmtp director proxy

2012-06-13 Thread Timo Sirainen
On Wed, 2012-06-13 at 08:58 +0200, Angel L. Mateo wrote:
   I have checked in almost every error I had that the error is produced 
 whenever happens a timeout of 30 seconds between opening the connection 
 between the director and backend server and the final delivery of the 
 message in the user's mailbox.
 
   When I have mails with just a few of recipients, I have no problem 
 because this 30 seconds timeout is never reached. But when I have mails 
 with more recipients and my storage has workload it is sometimes reached.

Ah, so it's not really a bug. I thought it might be because there had
been such problems before.

   But I haven't found any configuration for this 30 seconds timeout. What 
 could it be this option? Because I have configured proxy_timeout=120 in 
 proxy configuration:
 
 pass_attrs = 
 irisMailbox=userdb_mail,homeDirectory=userdb_home,uidNumber=userdb_uid,gidNumber=userdb_gid,=proxy=y,=proxy_timeout=120,irisMailHost=host

This should work..

 lmtp/commands.c:#define LMTP_PROXY_DEFAULT_TIMEOUT_MSECS (1000*30)

This is the default, but proxy_timeout should override it.

What do you get in logs with auth_debug=yes?



Re: [Dovecot] how to announce shared folders to clients using non-default mail prefix

2012-06-13 Thread Timo Sirainen
On Tue, 2012-06-12 at 11:16 -0700, Tom Lieuallen wrote:
 namespace {
hidden = yes
inbox = no
list = children
location = maildir:/a1/dove-shared:INDEX=/a2/imap-index/dove-shared/%u
prefix = iphonemail/sharedimap/
separator = /
type = shared

type=public and same for the other shared namespace. The type=shared
namespaces are for mailboxes shared between users.




Re: [Dovecot] doveadm doesn't subscribe to public folders

2012-06-13 Thread Timo Sirainen
OK. v2.1 should have fixed this also for doveadm subscribe.

On Wed, 2012-06-13 at 15:24 +0200, Emiliano Rago wrote:
 It works! Thanks!
 
 Emiliano
 
 On 06/13/2012 02:50 PM, Timo Sirainen wrote:
  Does it work if you do it via imap?
 
  echo a subscribe public.Conferences | /usr/local/libexec/dovecot/imap -u 
  rago
 
  On Wed, 2012-06-13 at 11:59 +0200, Emiliano Rago wrote:
  Hi,
 
  what it's happening with the doveadm command below is that
  the file modified is /data/MAIL/PUBLIC/subscriptions
  while I'd like to modify the file /data/MAIL/rago/subscriptions
 
  With subscriptions=no every user can subscribe to public folder,
  so perhaps this behaviour is inappropriate; suggestions?
 
  Thanks,
  Emiliano Rago
 
 
  On 06/12/2012 02:08 PM, Emiliano Rago wrote:
  Hi,
 
  I'd like to subscribe folder with doveadm:
 
  doveadm mailbox subscribe -u rago public.Conferences
 
  This command doesn't work, while it works with an ordinary folder.
  However it's possible to subscribe to the folder with an imap connection:
 
  1 login rago mypasswd
  1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE
  IDLE SORTSORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT
  CHILDREN NAMESPCE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC
  ESEARCH ESORT SEARCHRES ITHIN CONTEXT=SEARCH LIST-STATUS ACL
  RIGHTS=texk] Logged in
  2 LSUB  *
  * LSUB () . INBOX
  2 OK Lsub completed.
  3 SUBSCRIBE public.Conferences
  3 OK Subscribe completed.
  4 LSUB  *
  * LSUB () . INBOX
  * LSUB () . public.Conferences
  4 OK Lsub completed.
 
  Am I doing anything wrong?
 
  This is my conf, thx for help,
  Emiliano
 
  # 2.0.9: /etc/dovecot/dovecot.conf
  # OS: Linux 2.6.32-220.4.1.el6.x86_64 x86_64 Red Hat Enterprise Linux
  Server release 5 (Tikanga) ext4
  auth_cache_size = 128 M
  auth_master_user_separator = *
  auth_mechanisms = plain cram-md5
  mail_location =
  maildir:/data/MAIL/MAILDIR/%u:INBOX=/data/MAIL/INBOX/%u:INDEX=/data/MAIL/METADATA/%u
 
  maildir_very_dirty_syncs = yes
  managesieve_notify_capability = mailto
  managesieve_sieve_capability = fileinto reject envelope
  encoded-character vacation subaddress comparator-i;ascii-numeric
  relational regex imap4flags copy include variables body enotify
  environment mailbox date
  mbox_write_locks = fcntl
  namespace {
  inbox = yes
  location =
  prefix =
  separator = .
  type = private
  }
  namespace {
  list = children
  location =
  maildir:/data/MAIL/MAILDIR/%%u:INDEX=/data/MAIL/METADATA/SHARED/%u/%%u
  prefix = shared.%%u.
  separator = .
  subscriptions = no
  type = shared
  }
  namespace {
  list = children
  location = maildir:/data/MAIL/PUBLIC:INDEX=/data/MAIL/METADATA/PUBLIC
  prefix = public.
  separator = .
  subscriptions = no
  type = public
  }
  passdb {
  args = scheme=cram-md5 /data/PWDDB/cram_dovecot.txt
  driver = passwd-file
  }
  passdb {
  args = /etc/dovecot/master-shared
  driver = passwd-file
  master = yes
  }
  passdb {
  args = /etc/dovecot/master-shared
  driver = passwd-file
  }
  plugin {
  acl = vfile:/etc/dovecot/global-acls:cache_secs=300
  acl_anyone = allow
  acl_shared_dict = file:/data/MAIL/SHAREDDB/shared-mailboxes.db
  sieve = /data/MAIL/SIEVE/%u/dovecot.sieve
  sieve_dir = /data/MAIL/SIEVE/%u
  }
  postmaster_address = root
  protocols = imap sieve
  service auth {
  unix_listener /var/spool/postfix/private/auth {
  mode = 0666
  }
  unix_listener auth-userdb {
  group = mailreader
  mode = 0600
  user = mailreader
  }
  }
  service imap-login {
  process_min_avail = 8
  service_count = 0
  vsz_limit = 512 M
  }
  service imap-postlogin {
  executable = script-login /etc/dovecot/postlogin.sh
  user = $default_internal_user
  }
  service imap {
  executable = imap imap-postlogin
  }
  ssl_cert =/etc/dovecot/ssl/imaps.lal.in2p3.fr.crt
  ssl_key =/etc/dovecot/ssl/imaps.lal.in2p3.fr.key
  userdb {
  args = /etc/dovecot/master-shared
  driver = passwd-file
  }
  userdb {
  args = uid=mailreader gid=mailreader home=/data/MAIL/SIEVE/%u
  allow_all_users=yes
  driver = static
  }
  protocol lda {
  mail_plugins = acl sieve
  }
  protocol imap {
  mail_max_userip_connections = 128
  mail_plugins = acl imap_acl
  }
 
 
 
 




Re: [Dovecot] Problem with lmtp director proxy

2012-06-13 Thread Timo Sirainen
On Wed, 2012-06-13 at 16:57 +0200, Angel L. Mateo wrote:
 Jun 12 11:40:39 myotis41 dovecot: lmtp(6595): Connect from 155.54.212.167
 Jun 12 11:40:39 myotis41 dovecot: lmtp(6595): Debug: auth input: 
 user=user1 proxy host=155.54.211.163 proxy_refresh=450

That says proxy_refresh, not proxy_timeout.

   but I have checked with newer errors, all I see in logs are Connect 
 from and Disconnect from messages. The logs lmtp...Debug: are not 
 produced any more (maybe because director has this information yet?)

Director shouldn't affect it. There should still be auth input lines
logged. doveconf -n?




Re: [Dovecot] how to announce shared folders to clients using non-default mail prefix

2012-06-13 Thread Timo Sirainen
On 13.6.2012, at 19.58, Tom Lieuallen wrote:

 type=public and same for the other shared namespace. The type=shared
 namespaces are for mailboxes shared between users.
 
 Unfortunately, it still isn't working.
..
 It seems to me like the logic for deciding which namespaces to follow is 
 something like this:
 
 * If mail prefix = , inspect and potentially use all namespaces
 
 * else look in default namespace for subdirectories matching prefix listed 
 _AND_ look for namespaces that are exact matches for the prefix passed.
 
 In that 'else' case, it does not appear to look for namespaces where the mail 
 prefix is a subset.

No. I tried with your exact config, except changed namespace types to public, 
and it works fine in my tests.. You're trying with v2.1.7, right?



Re: [Dovecot] doveadm backup panic

2012-06-15 Thread Timo Sirainen
On 15.6.2012, at 21.50, Gedalya wrote:

 #12 imapc_untagged_fetch (reply=0xb184, mbox=0x80fd2c8) at 
 imapc-mailbox.c:349
 old_kws = {arr = {buffer = 0x8093030, element_size = 4}, v = 0x8093030, 
 v_modifiable = 0x8093030}

Fixed: http://hg.dovecot.org/dovecot-2.1/rev/a28c8043842d



Re: [Dovecot] [Dovecot 2.1.7] SegFault on doveadm search through director proxy

2012-06-15 Thread Timo Sirainen
On Fri, 2012-06-15 at 22:11 +0200, Daniel Parthey wrote:
  mail02# doveadm -c /etc/dovecot-director/dovecot-director.conf search -u 
  u...@example.org all
..
 #3  doveadm_print_flow_print (value=0x64697567 Address 0x64697567 out of 
 bounds) at doveadm-print-flow.c:51
 hdr = value optimized out
 #4  0x00415667 in doveadm_print (value=0x1c28970 
 67b3b72453278b4f6a3d51abeb58) at doveadm-print.c:65
 headers = 0x1c37120
 #5  0x0041638d in server_flush_field (conn=0x1c4ab10) at 
 server-connection.c:111
 text = 0x0

Hmm. See if the attached patch fixes it?

diff -r a28c8043842d src/doveadm/doveadm-print.c
--- a/src/doveadm/doveadm-print.c	Sat Jun 16 02:03:53 2012 +0300
+++ b/src/doveadm/doveadm-print.c	Sat Jun 16 02:13:03 2012 +0300
@@ -18,6 +18,7 @@
 	const struct doveadm_print_vfuncs *v;
 
 	unsigned int header_idx;
+	bool print_stream_open;
 };
 
 static struct doveadm_print_context *ctx;
@@ -52,7 +53,7 @@
 	doveadm_print_header(key_title, key_title, 0);
 }
 
-void doveadm_print(const char *value)
+static void doveadm_print_sticky_headers(void)
 {
 	const struct doveadm_print_header_context *headers;
 	unsigned int count;
@@ -68,7 +69,13 @@
 			break;
 		}
 	}
+}
 
+void doveadm_print(const char *value)
+{
+	i_assert(!ctx-print_stream_open);
+
+	doveadm_print_sticky_headers();
 	ctx-v-print(value);
 	ctx-header_idx++;
 }
@@ -82,9 +89,15 @@
 
 void doveadm_print_stream(const void *value, size_t size)
 {
+	if (!ctx-print_stream_open) {
+		doveadm_print_sticky_headers();
+		ctx-print_stream_open = TRUE;
+	}
 	ctx-v-print_stream(value, size);
-	if (size == 0)
+	if (size == 0) {
 		ctx-header_idx++;
+		ctx-print_stream_open = FALSE;
+	}
 }
 
 void doveadm_print_sticky(const char *key, const char *value)


Re: [Dovecot] Problem with lmtp director proxy

2012-06-15 Thread Timo Sirainen
On 13.6.2012, at 20.11, Angel L. Mateo wrote:

   Ok, you were right. I was looking for logs at my log repository, which 
 doesn't receive debug log. Nevertheless, the only auth lines I have found at 
 the ones above, with the proxy_refresh=450. I haven't found any line with a 
 timeout log in the proxies neither the backends

The backend logs don't matter. Director adds the proxy_refresh. You haven't 
shown in your logs what auth process logs as debug messages. This is what is 
supposed to happen:

 Jun 16 02:19:11 auth: Debug: master out: PASS   1   user=director   proxy 
   proxy_timeout=1000

master out must return proxy_timeout=1000. If it doesn't, then the problem is 
with your auth settings.

 Jun 16 02:19:11 lmtp(11845): Debug: auth input: user=director proxy 
 proxy_timeout=1000 host=1.2.3.4 proxy_refresh=450

Director adds proxy_refresh, but preserves proxy_timeout.



Re: [Dovecot] doveadm fetch LARGE attachments and remove message

2012-06-18 Thread Timo Sirainen
On 17.6.2012, at 18.19, Bradley Giesbrecht wrote:

 Looking at the wiki and man pages I am unsure how to fetch email attachments 
 from a unix shell.
 
 I have an doveadm search that returns the messages that have the 
 attachments I am after.
 
 Would I loop through an doveadm fetch and use a commandline imap client to 
 save the attachments and move the message to the Trash?
 
 Or is there a doveadm command for this?

No, there's currently no easy way to do this. doveadm fetch doesn't support 
that. You could possibly do this via IMAP, but it would be difficult to know 
which MIME part to fetch. Actually it's not even obvious if a MIME part is an 
attachment or not..



Re: [Dovecot] Sieve and fileinto encoding change?

2012-06-18 Thread Timo Sirainen
On 18.6.2012, at 12.56, Angel L. Mateo wrote:

   Although I'm planning to patch ingo, my question is why those same 
 scripts, with utf-7 folder's names, are working with dovecot 1.1.16 but not 
 with 2.1.5? Because in my old servers those scripts worked without any 
 problem...

Because v1.1 worked incorrectly and v2.1 works correctly :)



Re: [Dovecot] Dovecot Maildir - How to Seperate mail folders

2012-06-18 Thread Timo Sirainen
On 18.6.2012, at 12.17, Guido Weiler wrote:

 01 OK Logged in.
 02 list  *
 * LIST (\HasNoChildren) / INBOX
 * LIST (\Noselect \HasChildren) / greetings
 * LIST (\HasNoChildren) / greetings/INBOX
 02 OK List completed.
 03 select greetings/INBOX
 03 NO Mailbox doesn't exist: INBOX
 04 select greetings
 04 NO Mailbox doesn't exist: greetings
 
 ---
 
 What is this \Noselect mailbox showing up and why is it saying 
 greetings/INBOX in the third row when in fact there isn't a mailbox with 
 this name?
 
 I am very sorry for having to bother you again, but I don't know what we are 
 doing wrong here.
 (Dovecot version is 1.1.16)

Fixed in newer versions, upgrade.

Re: [Dovecot] migrating sql virtual 1 to 2, namespace configuration error: inbox=yes namespace missing

2012-06-18 Thread Timo Sirainen
On 18.6.2012, at 9.54, Voytek Eymont wrote:

 I'm trying to setup a new server on centos 6, from old dovecot 1.x
 
 I installed  'dovecot --version 2.1.1' from dovecot rpm
 I converted conf file as per migration specs, also, copied sql conf across
..
 dovecot: imap(name@tld): Error: user name@tld: Initialization failed:
 namespace configuration error: inbox=yes namespace missing

Easiest fix: remove 15-mailboxes.conf

Alternative fix: modify this namespace to actually work. Probably adding 
inbox=yes inside it is enough to do that.



Re: [Dovecot] question about fts_squat

2012-06-18 Thread Timo Sirainen
On 16.6.2012, at 13.55, Wojciech Puchar wrote:

 even search all body someword doesn't work.
 
 while
 
 search body someword
 
 always work very well.
 
 what's wrong?

Fixed: http://hg.dovecot.org/dovecot-2.1/rev/4ce1f9649592

Anyway, fts-lucene backend works better than fts-squat.



Re: [Dovecot] Maildir + quota + listescape = wrong dir location

2012-06-18 Thread Timo Sirainen
On 18.6.2012, at 16.45, Mariusz Kruk wrote:

 I've just stumbled across a strange thing which seems to be a bug.
 It happens in 2.0.9 as well as 2.0.11 in which I tested it.

Listescape has some unfixable problems in v2.0. You've most likely hit one of 
them. v2.1 had some larger changes and fixes listescape to work perfectly.



Re: [Dovecot] question about fts_squat

2012-06-18 Thread Timo Sirainen
On 18.6.2012, at 20.23, Wojciech Puchar wrote:

 Anyway, fts-lucene backend works better than fts-squat.
 Better in what respect?
 
 less than a second (when disk I/O was needed) fulltext search over 1 
 mails doesn't look bad :)

Squat index updates are somewhat slow, especially if the index is large.



[Dovecot] message parser: Fixed infinite loop when parsing a specific message.

2012-06-19 Thread Timo Sirainen
I committed this change to all hg branches:
http://hg.dovecot.org/dovecot-2.1/rev/4461b48fcc1f

After that I realized that it doesn't actually matter, because it fixes
only a situation where input buffer's size is less than 84 bytes. This
happened on a test program where I was using a 64 byte buffer, but the
real code in Dovecot always uses much larger buffers.

So, don't worry, there's no way to actually DOS Dovecot with this. No
need for distro people to create any security releases.




Re: [Dovecot] director map and mysql

2012-06-19 Thread Timo Sirainen
On 20.6.2012, at 5.03, Костырев Александр Алексеевич wrote:

 Is doveadm director map command suppose to work when I store host value 
 in mysql table?
 It gives me nothing in output with no errors in log.

If you return a host for a user, then Dovecot does regular proxying and 
director doesn't know anything about the user.



Re: [Dovecot] dovecot 2.1.5 performance

2012-06-20 Thread Timo Sirainen
On Wed, 2012-06-20 at 11:40 +0200, Angel L. Mateo wrote:

 * mmap_disable: both single and multi server configurations have 
 mmap_disable=yes but in index file section says that you need it if you 
 have your index files stored in nfs. I have it stored locally. Do I need 
 mmap_disable=yes? What it's the best?

mmap_disable is used only for index files, so with local indexes use
no. (If indexes were on NFS, no would probably still work but I'm
not sure if the performance would be better or worse. Errors would also
trigger SIGBUS crashes.)

 * dotlock_use_excl: it is set to no in both configurations, but the 
 comment says that it is needed only in nfsv2. Since I have nfs3, I have 
 it set it to yes.

yes is ok.

 * mail_nfs_storage: In single server is set to no, but in multi server 
 it set to yes. Since I have a director in front of my backend server, 
 what is the recommended?

With director you can set this to no.

   With this configuration, when I have a few connections (about 300-400 
 imap connections) everything is working fine, but when I disconnect the 
 old servers and direct all my users' connections to the new servers I 
 have lot of errors. 

Real errors that show up in Dovecot logs? What kind of errors?

 server loads increments to over 300 points, with a 
 very high io wait. With atop, I could see that of my 6 cores, I have one 
 with almost 100% waiting for i/o and the other with almost 100% idle, 
 but load of the server is very, very high.

Does the server's disk IO usage actually go a lot higher, or is it
simply waiting without doing much of anything? I wonder if this is
related to the inotify problems:
http://dovecot.org/list/dovecot/2012-June/066474.html

Another thought: Since indexes are stored locally, is it possible that
the extra load comes simply from building the indexes on the new
servers, while they already exist on the old ones?

 mail_fsync = always

v1.1 did the equivalent of mail_fsync=optimized. You could see if that
makes a difference.

 maildir_stat_dirs = yes

Do you actually need this? It causes unnecessary disk IO and probably
not needed in your case.

 default_process_limit = 1000

Since you haven't enabled high-performance mode for imap-login processes
and haven't otherwise changed the service imap-login settings, this
means that you can have max. 1000 simultaneous IMAP SSL/TLS connections.



Re: [Dovecot] GlusterFS + Dovecot

2012-06-20 Thread Timo Sirainen
On 20.6.2012, at 18.50, Romer Ventura wrote:

 Has anyone used GlusterFS as storage file system for dovecot or any other
 email system..?

I've heard Dovecot complains about index corruption once in a while with 
glusterfs, even when not in multi-master mode. I wouldn't use it without some 
heavy stress testing first (with imaptest tool).



Re: [Dovecot] dovecot 2.1.5 performance

2012-06-21 Thread Timo Sirainen
On 21.6.2012, at 11.44, Angel L. Mateo wrote:

 El 20/06/12 12:05, Timo Sirainen escribió:
 
 default_process_limit = 1000
 
 Since you haven't enabled high-performance mode for imap-login processes
 and haven't otherwise changed the service imap-login settings, this
 means that you can have max. 1000 simultaneous IMAP SSL/TLS connections.
 
   According to http://wiki2.dovecot.org/LoginProcess
 
 Since one login process can handle only one connection, the service's 
 process_limit setting limits the number of users that can be logging in at 
 the same time (defaults to default_process_limit=100).
 
   I understood this as there can only be up to 100 (or 1000 in my case) 
 concurrently trying to log in, but once the user logs, the imap-login process 
 ends (starting corresponding imap processes) and another users could log in. 
 So there could be more than 100 users connected, but up to 100 trying to 
 connect. Am I wrong?
 
   If I am wrong, why in my system there is no imap-login processes (or 
 just a few) but a lot of imap?


Look at the next sentence also: SSL/TLS proxying processes are also counted 
here, so if you're using SSL/TLS you'll need to make sure this count is higher 
than the maximum number of users that can be logged in simultaneously.

I guess you don't have many SSL/TLS connections.

Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-21 Thread Timo Sirainen
On 21.6.2012, at 21.05, email builder wrote:

 We are building a new system that will support a large number of users (high 
 volume, high concurrent usage, etc).  We have played with Dovecot, but in 
 most serious applications we have traditionally used Courier IMAP.  It's my 
 (lay) understanding that with indexing and perhaps other things in Dovecot, 
 it might perform better than Courier in larger environments like this.  Am I 
 correct or is it less clear-cut?

If you disable index index files in Dovecot, its performance should be slightly 
better than Courier. With index files the performance is typically much better 
in Dovecot, especially if you use a (non-caching) webmail.

 Any tips on making the migration (not migrating an existing system, I mean 
 migrating our paradigm - things to consider, things to watch out for)?

If you don't migrate any existing users, I guess this doesn't differ much from 
any other optimized Dovecot installation. Usually large installations (1M 
users) use NetApp NFS + Dovecot director. You might also want to enable full 
text searches. http://wiki2.dovecot.org/PerformanceTuning lists some other 
things.



Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-21 Thread Timo Sirainen
On Thu, 2012-06-21 at 13:05 -0700, email builder wrote:
 Thank you very much for the fast reply.
 
   We are building a new system that will support a large number of users 
 
  (high volume, high concurrent usage, etc).  We have played with Dovecot, 
  but in 
  most serious applications we have traditionally used Courier IMAP.  It's 
  my 
  (lay) understanding that with indexing and perhaps other things in 
  Dovecot, it 
  might perform better than Courier in larger environments like this.  Am I 
  correct or is it less clear-cut?
  
  If you disable index index files in Dovecot, its performance should be 
  slightly 
  better than Courier. With index files the performance is typically much 
  better 
  in Dovecot, especially if you use a (non-caching) webmail.
 
 Interesting.  What would be the motivations for disabling indexing?
 Indexing is by default enabled?

Yes, enabled by default. There aren't many good reasons for disabling
indexing.

 Do you know what webmails are caching vs. non-caching?  

Nearly all of them are non-caching. (I don't know of any caching ones.)

 Am I correct that what you're pointing out is that with non-caching
 webmails you will notice IMAP performance differences more readily
 but that a caching webmail application might be better no matter
 which IMAP server because it reduces the need for webmail to make
 IMAP connections?

It's not about the IMAP connections themselves, but how often they fetch
message (meta)data. http://www.imapwiki.org/Benchmarking should explain
this better. Dovecot's indexing can lower the disk I/O usage perhaps by
10x compared to Courier.

   Any tips on making the migration (not migrating an existing system, I 
  mean 
  migrating our paradigm - things to consider, things to watch out for)?
  
  If you don't migrate any existing users, I guess this doesn't differ 
  much from any other optimized Dovecot installation. Usually large 
  installations 
  (1M users) use NetApp NFS + Dovecot director. You might also want to 
  enable 
  full text searches. http://wiki2.dovecot.org/PerformanceTuning lists some 
  other 
  things.
 
 Ah, I didn't know about Director.  That looks very nice.  I had in mind that
 we would have to use Perdition, but an integrated solution might be good.
 
 Anyone have any thoughts or opinions considering Perdition vs. Director?

Dovecot proxy has several Dovecot-specific features that make it work
better than perdition (forwards client IP address to backend, handle
CAPABILITY stuff better, maybe other things).

 Full text searches don't hurt performance too bad?

They should improve the performance, at least from the user's point of
view when doing a search on webmail. But yes, the indexing itself does
cost CPU cycles, disk I/O and disk usage (perhaps 30% more disk space).



Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-21 Thread Timo Sirainen
On 21.6.2012, at 23.34, Reindl Harald wrote:

 Am 21.06.2012 22:22, schrieb Timo Sirainen:
 Do you know what webmails are caching vs. non-caching?  
 
 Nearly all of them are non-caching. (I don't know of any caching ones.)
 
 roundcube can if configured
 
 additionally you should install imapproxy on the webserver
 wehre your webmail is running and configure the webmail for
 using 127.0.0.1 - so only one connection per user is
 persistent instead make a new one for each ajax-request

Someone benchmarked Dovecot a while ago in this list with and without imapproxy 
and the results showed that imapproxy simply slowed things down by adding extra 
latency. This probably isn't true for all installations, but I don't think 
there's much of a difference either way.



Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-21 Thread Timo Sirainen
On 21.6.2012, at 23.48, Reindl Harald wrote:

 Someone benchmarked Dovecot a while ago in this list with and without 
 imapproxy and the results showed that imapproxy simply slowed things down by 
 adding extra latency. This probably isn't true for all installations, but I 
 don't think there's much of a difference either way.
 
 depends on network-latency, parallel users and last but
 not least count of folders - if you have 30 folders and
 roundcube refreshs every 20 seconds it will make in the
 worst case 180 connections for one user per minute

Really? Doesn't it simply connect once every 20 seconds and send 30 STATUS 
requests in one connection?



Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-21 Thread Timo Sirainen
On 21.6.2012, at 23.48, Reindl Harald wrote:

 Someone benchmarked Dovecot a while ago in this list with and without 
 imapproxy and the results showed that imapproxy simply slowed things down by 
 adding extra latency. This probably isn't true for all installations, but I 
 don't think there's much of a difference either way.
 
 depends on network-latency, parallel users and last but
 not least count of folders - if you have 30 folders and
 roundcube refreshs every 20 seconds it will make in the
 worst case 180 connections for one user per minute
 
 maybe a bechmark with high load shows other values
 
 but felt performance in our setup is much better with
 imapproxy in front - roundcube feels like a desktop client

Oh, and of course it also depends on Dovecot configuration :) Authentication 
cache is needed and login processes must be in high performance mode. There is 
still the extra work of forking a new imap process (could also be avoided with 
yet another config option) and some other extra CPU usage, but those shouldn't 
cause much of a difference.

The extra network latency during login is a good point though.



Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-21 Thread Timo Sirainen
On 22.6.2012, at 0.58, Michael M Slusarz wrote:

 I think the conclusion is that imapproxy is not necessary.  There are some 
 advantages (eg with high network latency between web and imap server, and 
 reducing apparent login count), and some disadvantages (extra complexity, 
 slowdown)
 
 Not entirely true.  See this thread:
 
 http://markmail.org/thread/z7ctwle2go6zafas
 
 Thread in short: imapproxy provides benefits for more MUAs that take 
 advantage of the XIMAPPROXY feature (only IMP, AFAIK), and Timo is/was 
 considering adding a similar state saving feature to Dovecot 2.2.

Well, I had completely forgotten about it :) Reading my old mail:

 There isn't a whole lot of state to be saved really. Mailbox GUID, 
 UIDVALIDITY,
 HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states.
 Clients could restore their earlier state from days ago, as long as Dovecot
 still has the necessary .log records available (similar to how QRESYNC works).

Yeah .. Perhaps something like:

1. if client issues LOGOUT XSTATE

2. And server sees that it can actually save all of the state (some things are 
a bit tricky, and probably not worth the trouble in initial implementation)

3. Then the server server sends
* OK XSTATE string
* BYE 

4. The client can pipeline after LOGIN/AUTHENTICATE:
a XSTATERESTORE string
a OK Yeah!
or
a NO Not gonna work.

Perhaps even a real RFC for this thing? .. If it's worth it.. Would save at 
least a few X bytes from network traffic :)

Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-21 Thread Timo Sirainen
On 22.6.2012, at 5.28, email builder wrote:

 Oh, and of course it also depends on Dovecot configuration :) Authentication 
 cache is needed and login processes must be in high performance mode.
 
 I.e., I think:
 
 http://wiki2.dovecot.org/LoginProcess
 http://wiki2.dovecot.org/Authentication/Caching

Yes.

 There is 
 still the extra work of forking a new imap process (could also be avoided 
 with 
 yet another config option)
 
 Are you referring to client_limit or service_count or something else as yet 
 undeveloped?

service imap { service_count = 0 } (default=1) allows imap processes to be 
reused for more than 1 connection. The downside is that if there are any bugs 
in Dovecot, they might accidentally expose another user's email data to the 
wrong user. That's very unlikely to happen but since this isn't a performance 
problem in most (if any) systems I don't want to enable it by default. Dovecot 
code is written so that write buffer overflows (= arbitrary code execution) is 
minimized to be as zero possibility as I could think of, but read buffer 
overflows (= exposing data within the process) isn't treated nearly as much 
with paranoia.

 Speaking of which, I cannot understand the different between those two.  
 Hints in the
 configuration file (10-master.conf) and the wiki make them sound like they do 
 the same
 thing -- ??

service_count limits the maximum of client_limit. One connection = one service. 
Once a process has serviced service_count number of connections it 
disconnects itself. There can never be more than client_limit number of 
simultaneous connections. The important stuff to understand about these are:

 * service_count=1: The most secure setting for a process. The process serves a 
single connection and kills itself. No possibility of data leaking to 
unintended connection.
 * service_count=0, client_limit=1: The process does blocking operations (e.g. 
blocking disk IO). You don't want one connection's blocking operation to affect 
other connections. But you're not paranoid about security, since in case of 
some bugs some data might leak to unintended connection.
 * service_count0: Restart process ever N connections, just in case it leaks 
some memory.
 * client_limit1: Limit the amount of CPU/memory a single process takes. The 
process should never be blocking on disk I/O or locks or anything else. This 
means it shouldn't be used for imap/pop3/lmtp processes. For CPU bound 
processes it's fine.
 
Maybe these could be copypasted to the wiki2/Services.

Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-22 Thread Timo Sirainen
On 22.6.2012, at 8.27, email builder wrote:

 So really, a new process is created under *two* circumstances?  1. when a
 process reaches client_limit number of *simultaneous* connections or  2. when
 a process has serviced service_count number of connections.  Is this correct?

Yes.

 So for service *-login, is it OK to do something like service_count=5000, 
 client_limit=2000

It would work, but for login processes the service_count can be 0. I haven't 
seen them leaking any memory recently.

One somewhat annoying thing with service_count1 is that the processes have to 
wait until all of the connections have disconnected before shutting down. For 
processes handling long running connections (especially IMAP) this can mean 
that you'll end up with a lot of processes that are ready to shutdown but a 
couple of connections prevent it from doing this.



Re: [Dovecot] dovecot stats error

2012-06-22 Thread Timo Sirainen
On 22.6.2012, at 15.34, Robert Schetterer wrote:

 Hi Timo,
 any idea whats this related too ?
 
 dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats
 shrank: mrbytes 21703727  25193928

Which Dovecot version? I thought I fixed this already..




Re: [Dovecot] cumulative userdb ?

2012-06-22 Thread Timo Sirainen
On 22.6.2012, at 16.59, Benoit Branciard wrote:

 in Dovecot 2.0, is it possible to have kind of cumulative multiple userdb ?
 
 that is, for all users:
 - extract some attributes (let's say: uid, gid, home) from a first userdb 
 (Passwd for example),
 - an extract some other attributes (mail for example, but overwriting those 
 from the first userdb in case of redundancy) from a second userdb (LDAP for 
 example) ?

I've also wanted this a few times. But no, not possible currently.



Re: [Dovecot] Dovecot performance under high load (vs. Courier)

2012-06-23 Thread Timo Sirainen
On 23.6.2012, at 13.21, Ed W wrote:

 But I don't know, whether this is the sort of caching you are referring to.
 
 what's a point of caching imap, except your webmail service is not locally 
 connected (localhost or LAN) to imap server?
 
 Asking for items 600-615 from a threaded list, sorted by something, can be an 
 expensive operation, especially if you just asked for items 585-600 a moment 
 ago?

Can be, but is it? :) Dovecot attempts to cache/index stuff as well. Normally 
there shouldn't be a need for extra caching layer except in cases of higher 
network latency.



Re: [Dovecot] Dovecot list IMAP archives with thunderbird?

2012-06-23 Thread Timo Sirainen
On Sat, 2012-06-23 at 18:00 +0100, Alex Crow wrote:
  I'm trying to access the IMAP archives with Thunderbird but can't seem 
  to get it to work. I have tried an unencrypted connection, SSL and TLS 
  but with no success. Any ideas?
 
  Thanks
 
  Alex
 
 Hi,
 
 Still stuck here - would really like to be able to access the archives 
 in my email client...
 
 Anyone able to see the mailing list archives in Thunderbird or other 
 IMAP clients? Are they currently down?

It works fine as far as I can see, even with Thunderbird. What error do
you get?




Re: [Dovecot] SQLite dovecot query caching

2012-06-23 Thread Timo Sirainen
On Sun, 2012-06-24 at 03:36 +1000, Dominic Malolepszy wrote:
 Hi,
 
 I am wondering if Dovecot caches SQLite queries, and how well it works 
 in high performance setups. I am particularly interested because in the 
 below thread SQLite has been suggested as a means of Dovecot proxying 
 connections to different ports.

You can enable auth cache:
http://wiki2.dovecot.org/Authentication/Caching




Re: [Dovecot] SQLite dovecot query caching

2012-06-23 Thread Timo Sirainen
On Sun, 2012-06-24 at 04:20 +1000, Dominic Malolepszy wrote:
 On 24/06/12 3:39 AM, Timo Sirainen wrote:
  On Sun, 2012-06-24 at 03:36 +1000, Dominic Malolepszy wrote:
  Hi,
 
  I am wondering if Dovecot caches SQLite queries, and how well it works
  in high performance setups. I am particularly interested because in the
  below thread SQLite has been suggested as a means of Dovecot proxying
  connections to different ports.
  You can enable auth cache:
  http://wiki2.dovecot.org/Authentication/Caching
 
 
 
 This is a per user caching though, it will still have to perform a sql 
 look up each time a unique user authenticates to determine what port the 
 proxy should forward each connection. Is that accurate?

It caches the passdb lookup. The cache key consists of the given %
variables in the SQL query. So if your SQL query doesn't contain %n/%u
then the cache doesn't add per-user entries.




Re: [Dovecot] 2.0.19 segfault

2012-06-23 Thread Timo Sirainen
On Sat, 2012-06-23 at 22:39 +0200, Mailing List SVR wrote:

 after the upgrade from dovecot 2.0.13 (ubuntu oneiric) to dovecot 2.0.19 
 (ubuntu precise), in my logs I have a lot of these errors:
 
 Jun 23 00:20:29 server1 dovecot: master: Error: service(imap-login): 
 child 6714 killed with signal 11 (core dumps disabled)
 
 I tested 2.0.21 and the problem is still here. The problem seems to 
 appear only when the client is ms outlook, thunderbird works fine

Looks to me more like OpenSSL library bug. The only reason why it could
be Dovecot bug is if Dovecot is causing memory corruption. Could you run
imap-login via valgrind to see if this is the case?

service imap-login {
  executable = /usr/bin/valgrind -q --vgdb=no 
/usr/local/libexec/dovecot/imap-login
  chroot =
}

Also have you changed any ssl-related settings in dovecot.conf?




Re: [Dovecot] Dovecot Quotas in Version 2

2012-06-23 Thread Timo Sirainen
On 24.6.2012, at 3.10, Brian Spraker wrote:

 plugin {
  quota = maildir
  quota_rule = Trash:storage=100M

This should be quota_rule2. Otherwise you'll just overwrite this here:

 }

 user_query = SELECT uid, gid, home, maildir, concat('maildir:storage=', 
 quota) AS quota FROM horde_users WHERE user_uid = '%u';

You didn't update the quota configuration in here. Should be: 
concat('*:storage=', quota) AS quota_rule.

BTW. This change happened during Dovecot v1.0 - v1.1 change. Years ago for 
most people. :)



Re: [Dovecot] Dovecot Quotas in Version 2

2012-06-23 Thread Timo Sirainen
On 24.6.2012, at 3.29, Brian Spraker wrote:

 This still didn't correct the issue - and Dovecot won't start when I have the 
 mail_plugins line under 'protocol imap' uncommented.

You should also enable quota globally so it will work for doveadm and other 
tools you may end up using.

 Error in the syslog says:
 
 init: dovecot main process (x) terminated with status 89

There should be another error message before this.



Re: [Dovecot] Dovecot Quotas in Version 2

2012-06-23 Thread Timo Sirainen
On 24.6.2012, at 3.45, Brian Spraker wrote:

 I'm not sure I understand how to enable quota to work globally..?

Just put mail_plugins = quota outside protocol {} sections.

  But what would be preventing it from working under 'protocol imap' as it did 
 before?

No idea, the error log should say the reason.

 As for the log, here is the other lines that appeared above it:
 
 Jun 23 19:31:13 server dovecot: master: Warning: Killed with signal 15 (by 
 pid=1 uid=0 code=kill)
 Jun 23 19:31:13 server dovecot: log: Warning: Killed with signal 15 (by pid=1 
 uid=0 code=kill)

These mean that dovecot master process was stopped by init process. Why it's 
doing that I have no idea.

 Jun 23 19:31:13 server kernel: [100996.340925] init: dovecot main process 
 (11580) terminated with status 89

Status 89 means that Dovecot should have logged an error about it. But I see no 
error here. I think Ubuntu is doing something weird. See what happens if you 
start dovecot using dovecot -F instead of any init script or such.



Re: [Dovecot] 2.1.7 TLS issues

2012-06-24 Thread Timo Sirainen
On 24.6.2012, at 12.58, Christian Rößner wrote:

 I have an interesting problem: I am building dovecot packages for Ubuntu 
 since 10.04. Never had bigger trouble with it. Now since 2.1.6 or 2.1.7 (I 
 can not say more precisely), Thunderbird 10ESR and Outlook 2010 can no longer 
 use 143/TLS correctly. Automx delvers 143/TLS and Outlook tells me that it 
 can not create a secure connection. I changed automx to use 993/SSL and 
 everything works. Under Thunderbird 10ESR, I get a box that tells me that I 
 need to change settings. When I sent mail, TB told me that it could not copy 
 the mail to the sent folder. I also changed to 993/SSL and everything is 
 perfect.
 
 At the other and, Apples Mail.app and iOS devices work perfectly over 
 143/TLS. So my guess is that it has to do with OpenSSL. Did something change 
 in dovecot concerning TLS? Can I change options in the built process?

What was the Dovecot version you were using previously which worked?



Re: [Dovecot] 2.1.7 TLS issues

2012-06-24 Thread Timo Sirainen
On 24.6.2012, at 23.20, Christian Rößner wrote:

 I have an interesting problem: I am building dovecot packages for Ubuntu 
 since 10.04. Never had bigger trouble with it. Now since 2.1.6 or 2.1.7 (I 
 can not say more precisely), Thunderbird 10ESR and Outlook 2010 can no 
 longer use 143/TLS correctly. Automx delvers 143/TLS and Outlook tells me 
 that it can not create a secure connection. I changed automx to use 993/SSL 
 and everything works. Under Thunderbird 10ESR, I get a box that tells me 
 that I need to change settings. When I sent mail, TB told me that it could 
 not copy the mail to the sent folder. I also changed to 993/SSL and 
 everything is perfect.
 
 At the other and, Apples Mail.app and iOS devices work perfectly over 
 143/TLS. So my guess is that it has to do with OpenSSL. Did something 
 change in dovecot concerning TLS? Can I change options in the built process?
 
 What was the Dovecot version you were using previously which worked?
 
 I am not sure which version worked. My best guess is 2.1.4 or 2.1.5, because 
 I skipped 2.1.6. But 2.1.6 has the same issues, as my friend Uwe did report 
 the same issues with that one.

Well, there hasn't been many changes in the SSL code. The only thing I can 
think of is this memory leak fix, which temporarily wasn't implemented 
correctly. You could try what happens if you revert it:

changeset:   14418:85ad4baedd43
user:Timo Sirainen t...@iki.fi
date:Thu Apr 12 10:48:55 2012 +0300
summary: login: Another attempt at fixing SSL memory leak.

changeset:   14417:f80f18d0ffa3
user:Timo Sirainen t...@iki.fi
date:Thu Apr 12 10:41:44 2012 +0300
summary: login: Reverted memory leak fix, because it broke some SSL setups?

changeset:   14416:584bd77c38fd
user:Timo Sirainen t...@iki.fi
date:Wed Apr 11 19:06:44 2012 +0300
summary: Memory leak fixes.

<    4   5   6   7   8   9   10   11   12   13   >