Re: [Dovecot] Dovecot - some more optimisations
W dniu 2010-09-01 17:25, Timo Sirainen pisze:
On Wed, 2010-09-01 at 13:23 +0200, Len7hir wrote:
And another problem. Why You use safe_memset instead of memset?
Now this function have the largest impact in Dovecot performance.
Really? I'd think it would be called very rarely since it should be used
only for password-related data. I anyway added that at one point when I
saw some people talking about how some compilers optimize away memset()
when the compiler realizes that the memory isn't used anymore.
It's of course not super-helpful because if attacker can read some
memory it's pretty bad already. But it might help to avoid exposing
users' passwords in some buggy situations.
OK, I check this more in future.
Soon We will have better profiling results and maybe something change.
You are right this is authorisation system.
IMAP is pretty new in our company, but about 800 account use it everyday
in different ways and We are expecting much more authorisations soon
(after advert).
Safe_memset is also called from:
pool_alloconly_clear, pool_alloconly_destroy, pool_system_clean_free and
client_destroy (in login-common/client-common.c)
Another on list is t_push.
I don't know how much there is to be done about that. Or you could see
what happens if you inline the fast path of that code. Or maybe the
whole thing could be redesigned in a way to get the fast code path even
smaller.
Maybe We will focus on t_push in future.
+ /* count lines and missing cr (from begining + 1) */
+ last = (unsigned char *)data + 1;
+ while (TRUE) {
+ curr = memchr(last, '\n', block-size - (last - data));
+ if (curr == NULL)
+ break;
+
+ ctx-part-body_size.lines++;
+
+ if (*(curr - 1) != '\r')
+ missing_cr_count++;
+
+ last = curr + 1;
+ }
I did a couple coding style changes here and to other patches to make it
more consistent with rest of Dovecot (and also to help me understand the
changes with less mental power :)
OK. This is no problem for me :)
+ /* exceptional condition (test begining) */
+ if (*data == '\n' ctx-last_chr != '\r') {
+ ctx-part-body_size.lines++;
+ missing_cr_count++;
}
There's a bug here: lines++ should be done even when last_chr != '\r'.
You are right. Thanks.
Also I moved this before the loop. I'd think that optimizes CPU cache
behavior better (data is accessed linearly, not data[1] .. data[last],
data[0]).
OK. Your points :)
In istream-crlf.c I've changed one thing.
When destination buffer is full after '\r' addition (to dest), It didn't
skip '\n' in source buffer. I think this was buggy in earlier code, and
'\n' was skipped (this piece of code is used very rare).
It looks correct to me .. After buffer is full after '\r', it doesn't
skip '\n' because it's not in the buffer. So it comes back there the
next time and adds '\n' (but not '\r' again).
Original code has:
if (data[i] == '\n') {
(...)
if (data[i-1] != '\r')
stream-w_buffer[dest++] = '\r';
if (dest == stream-buffer_size)
break;
stream-w_buffer[dest++] = data[i] is not executed (no space in dest)
i_stream_skip(stream-parent, i);
i is set on '\n'
I'm not sure I'm right, but this is possible mistake in original code.
Never mind.
+ if (*(s_ptr - 1) != '\r')
+ *(d_ptr++) = '\r';
If data[0] == '\n' and also cstream-last_cr=TRUE, this code accesses
data[-1] which points to random data.
You are right.
(s_ptr s_end *s_ptr == '\n' !cstream-last_cr d_ptr d_end)
It could be when !cstream-last_cr is FALSE.
Always start from data[1] should fix this.
(data[0] is checked at the begining)
Thanks for Your reply
You are the best :)
Len7hir
[Dovecot] [pigeonhole] Libraries not linked correctly
Hello All I am trying to compile and use pigeonhole and I am experiencing the same exact problem that is stated here: http://dovecot.org/pipermail/dovecot/2010-August/051859.htm [r...@mongo dovecot-2.0-pigeonhole-build]# ldd /usr/lib/dovecot/managesieve-login linux-gate.so.1 = (0xb777e000) libdovecot-login.so.0 = not found libdovecot.so.0 = not found libc.so.6 = /lib/libc.so.6 (0xb7622000) /lib/ld-linux.so.2 (0xb777f000) lt does not seem as if -Wl,-rpath -Wl,/usr/local/lib/dovecot is being appended to gcc -std=gnu99 -march=i686 -mtune=generic -O2 -pipe -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wstrict-aliasing=2 -o managesieve-login client.o client-authenticate.o managesieve-login-settings.o managesieve-proxy.o ../../src/lib-managesieve/libmanagesieve.a -ldovecot-login -L/usr/lib/dovecot -ldovecot -lrt ^^See: http://dovecot.org/pipermail/dovecot/2010-August/051907.html However, my lack of programming knowledge and skills prohibit me from trying to fix this myself. I am hoping some kind soul would be nice enough to fix it. Thank you!
[Dovecot] minor director issues
I noticed 2 minor things with the director. 1) if I start 2 directors, but in 1 of them I dont add the other as a director_servers, they both start spamming the logfiles really really fast. It seems there is no backoff mechanism in the protocol between directors. 2) usermail server ip expire time cor 194.109.26.171 2010-09-02 14:32:13 unknown 194.109.26.172 2010-09-02 14:33:13 Why does it say 'unknown'? That session is active, im manually logged in with telnet, issued a login and a select inbox. The first one is a normal mail client login. Regards, Cor
Re: [Dovecot] nfs director
Hi, We use Dovecot only recent, but many I speak use for very many year, if director was really need, why it only come about now and not 5 or more year ago, all many mail network run broken for so many year? I no think so. It might compliment some situation, but not suitable or advantage for all. Imap not likely happen here for user, hardware no so cheap here,such is storage space not big deal, not until we buy hardware for near cheap as U.S., which 50-70% less than we pay. The director is not needed in most cases. If you feel you dont need the director, then just forget about its existence and ignore all emails about it. Just pretend it doesnt exist :) I'll try to explain why the director is needed in some situations. dovecot has an index file. If you look in your configuration file, you'll see something like: mail_location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u The index file holds an index so dovecot can search in your mailbox really fast. (sorry if you know all this). So, if for instance you want to sort your mailbox on subject, dovecot doesnt have to read through all your emails to find all the subjects. Instead, it can read the index file really fast, and return a sorted mailbox basically instantly. A lot of IMAP servers dont even have this feature! Imagine the alternative, where dovecout would have to read through 25.000 emails, just so it can sort your mailbox on subject. A nightmare for performance, but like I said, some imap servers still do that! Dovecot luckily uses an index, and is therefore faster than a lot of other imap servers. So, we have concluded that dovecot has a really cool performance feature in the form of an index file. Now imagine what happens if you have more than 1 imap server. We have 35 imap servers right now. Each server needs access to the index file. One popular way to do this is by using NFS. With NFS you have a central file server, and each imap server uses the remote fileserver for storage. So now, all 35 servers have access to a cool central index file. But, there is a slight snag. NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file. The index file can then end up being corrupted. This causes dovecot to give faulty data back to the client, which in practice ends up looking like the mailbox of the user is empty. Not good. This causes calls to support desks about why a user has just lost all his mail. It's not lost, it's just that the index file is corrupt, and now dovecot thinks the mail is gone. You may wonder rightly so, why this is a problem only now. Well, it hasnt been a problem only now. We've been dealing with this issue for years. We've had discussions with Timo over the years on how to solve this problem. But Timo has only so much time, and this problem was not important enough to fix. He had more important bugs to fix. So, we lived with the problem, or worked around it in ugly ways. Recently Timo has found the time (I believe an ISP paid Timo to solve this problem) to think about ways to solve it. It's really not an easy problem to fix. We've known for years that the best way to fix this would be to have a user always end up on the same server, because thats when that specific NFS problem doesnt occur. If 2 sessions of the same user both end up on Server 1, then both sessions will see the same data, and no problems can occur. But how to get all sessions of a user on the same server? Most loadbalancers dont support this feature. Our Foundries dont, and our previous Alteons didnt either. So Timo thought of a way how dovecot itself could make sure all sessions of a user end up on the same server. Basically by turning dovecot into a proxy, that proxies connections to another server. It maintains an internal state on what user belongs on what server, and redirects connections appropriately. And thats all the director is. A proxy with an internal state of where to proxy a connection to. So why dont you see this problem with your setup? Like I said, this is only a problem with specific setups. - you need to be using NFS for central index storage. If you dont use NFS for your indexes, you wont see this problem, and you wont need the director. carry on as you always have. - even if you use NFS, you may not run into this problem. Because you also need to have situations where multiple servers could be updating the indexes at the same time. There are only 2 situations where this could happen: 1) you're using dovecot-lda, and have index updating turned on. Index updating in the lda is cool, because it means dovecot will always have the latest information about your email in the index files. and/or 2) you allow users to have sessions to multiple servers. For instance, they connect with Outlook and
Re: [Dovecot] nfs director
On 2010-09-02 9:08 AM, Cor Bosman c...@xs4all.nl wrote: NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file. Just curious, but is there no way to just disable the caching on the NFS share where the indexes are stored? -- Best regards, Charles
Re: [Dovecot] nfs director
On Thu, 2010-09-02 at 09:47 -0400, Charles Marcus wrote: On 2010-09-02 9:08 AM, Cor Bosman c...@xs4all.nl wrote: NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file. Just curious, but is there no way to just disable the caching on the NFS share where the indexes are stored? Disabling attribute caching solves most of this, but increases the NFS traffic by 10x or so (and still isn't perfect). Dovecot v1.1 added a setting where it tried to flush the NFS cache only when necessary. It worked better than nothing, but not perfectly. I tried fixing it for a long time, spent a lot of time reading Linux and FreeBSD NFS kernel code and writing test programs, trying to get Linux to fix some things .. then I finally gave up.
Re: [Dovecot] bogus fts percentage
On 01/09/2010 20:35, Timo Sirainen wrote: On Mon, 2010-08-30 at 12:54 -0500, Mike Abbott wrote: Sometimes dovecot-2.0 sends this untagged response to clients: * OK Indexed -2147483648% of the mailbox, ETA 0:00 I imagine there's a bug in there somewhere. Well, I guess this fixes it: http://hg.dovecot.org/dovecot-2.0/rev/1012052e2e85 Inspection of code reveals a bug in this change. This bit: 1.22 -secs = (msecs / (percentage / 100.0) - msecs) / 1000; 1.23 +secs = (msecs*percentage / 100 - msecs) / 1000; changes the calculation from estimated_total_time = elapsed_time ÷ completed_fraction which seems to be correct, to estimated_total_time = elapsed_time × completed_fraction which seems to be incorrect. (I have written a fix and sent the diff to Timo.) Bill
Re: [Dovecot] minor director issues
On Thu, 2010-09-02 at 14:22 +0200, Cor Bosman wrote: I noticed 2 minor things with the director. 1) if I start 2 directors, but in 1 of them I dont add the other as a director_servers, they both start spamming the logfiles really really fast. It seems there is no backoff mechanism in the protocol between directors. I've added some delays to a few places, but I guess not enough. I'll see about fixing this. 2) usermail server ip expire time cor 194.109.26.171 2010-09-02 14:32:13 unknown 194.109.26.172 2010-09-02 14:33:13 Why does it say 'unknown'? That session is active, im manually logged in with telnet, issued a login and a select inbox. The first one is a normal mail client login. Director doesn't know what users are logged in, only CRC32 of their username. So doveadm first gets a list of all users, adds them to hash table, then gets the list of CRC32s of users from director and maps them to usernames via the hash table. unknown means that the user wasn't found from the hash table. So .. my guess is that doveadm didn't get a full list of usernames. And you probably don't even want it to read those 2 million (or whatever) user records from SQL? :)
Re: [Dovecot] minor director issues
Director doesn't know what users are logged in, only CRC32 of their username. So doveadm first gets a list of all users, adds them to hash table, then gets the list of CRC32s of users from director and maps them to usernames via the hash table. unknown means that the user wasn't found from the hash table. So .. my guess is that doveadm didn't get a full list of usernames. And you probably don't even want it to read those 2 million (or whatever) user records from SQL? :) Ah ofcourse! cor is in the local password file, while the other login is in the NIS database. That explains it. And we dont allow processes to ask for the whole NIS table :) Thanks, Cor
Re: [Dovecot] minor director issues
On Thu, 2010-09-02 at 16:11 +0200, Cor Bosman wrote: Director doesn't know what users are logged in, only CRC32 of their username. So doveadm first gets a list of all users, adds them to hash table, then gets the list of CRC32s of users from director and maps them to usernames via the hash table. unknown means that the user wasn't found from the hash table. So .. my guess is that doveadm didn't get a full list of usernames. And you probably don't even want it to read those 2 million (or whatever) user records from SQL? :) Ah ofcourse! cor is in the local password file, while the other login is in the NIS database. That explains it. And we dont allow processes to ask for the whole NIS table :) Note also the -f parameter. You can get a list of usernames and dump them into a file and use that.
Re: [Dovecot] minor director issues
On Thu, 2010-09-02 at 15:06 +0100, Timo Sirainen wrote: Director doesn't know what users are logged in, only CRC32 of their username. So doveadm first gets a list of all users, adds them to hash table, then gets the list of CRC32s of users from director and maps them to usernames via the hash table. unknown means that the user wasn't found from the hash table. Oops, not CRC32, but a 32bit hash based on MD5 of the username. Still, it's 32bit so with lots of users there will be a lot of collisions (which shouldn't be a problem, except doveadm director map will list users who haven't really logged in).
Re: [Dovecot] My %% variables in the namespace settings do not appear to be expanding.
On Tue, 2010-08-31 at 11:33 +0100, Kenneth Cope wrote:
namespace {
inbox = yes
location =
prefix =
separator = .
type = private
}
This is ok.
namespace {
hidden = yes
inbox = no
list = no
location =
prefix = INBOX
separator = .
type = private
}
This doesn't much make sense and I doubt Dovecot actually uses it for
anything. Just remove it.
namespace {
list = children
location = maildir:%%h:LAYOUT=maildir++:INDEX=~/shared.%Dn_%Dd
prefix = shared/%...@%d.
I forgot, prefix here should contain %%u, not %u, because %u gets
expanded to your own username while you want it to be expanding to other
users' username.. I'll add a check against this.
Re: [Dovecot] My %% variables in the namespace settings do not appear to be expanding.
On Thu, 02 Sep 2010 15:44:18 +0100
Timo Sirainen t...@iki.fi wrote:
On Tue, 2010-08-31 at 11:33 +0100, Kenneth Cope wrote:
namespace {
inbox = yes
location =
prefix =
separator = .
type = private
}
This is ok.
namespace {
hidden = yes
inbox = no
list = no
location =
prefix = INBOX
separator = .
type = private
}
This doesn't much make sense and I doubt Dovecot actually uses it for
anything. Just remove it.
namespace {
list = children
location = maildir:%%h:LAYOUT=maildir++:INDEX=~/shared.%Dn_%Dd
prefix = shared/%...@%d.
I forgot, prefix here should contain %%u, not %u, because %u gets
expanded to your own username while you want it to be expanding to
other users' username.. I'll add a check against this.
Unfortunately, I can't use %%u variable as is. The usernames
contain the dot character. That's essentially why I was using the
%Dn_%Dd construct. I can't find a way of doing this type os
escaping/modifying for the %% variables.
%%Dn, for example, gives:
Sep 02 16:26:01 auth: Debug: passwd-file(@23foxgrove.co.uk):
lookup: us...@23foxgrove.co.uk
file=/usr/local/dovecot-2.0.1/auth/passwd.dovecot
%D%n, does not work, understandably, and caused a LOT of error
messages and stops the server being usable.
I was originally hoping for a modifier for the dot character in
variables, in the same way that the quote characters can be escaped
with the %E prefix modifier.
I'm pretty sure that I can't change the separator to a '/' as there
is a large installed user base and we'd prefer this to be as
transparent as possible.
All this help is appreciated.
Kenneth Cope.
--
Kenneth Cope
kenn...@copeonthe.net
**
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom
they are addressed. If you are not the intended recipient you are
not authorised to and must not disclose, copy, distribute, or
retain this message or any part of it.
CopeOnThe.net accepts no responsibility for information,
errors or omissions in this email.
***
Re: [Dovecot] Dovecot - some more optimisations
On Thu, 2010-09-02 at 11:42 +0200, Len7hir wrote:
Safe_memset is also called from:
pool_alloconly_clear, pool_alloconly_destroy,
Only when pool was created by pool_alloconly_create_clean(), which isn't
used anywhere.
pool_system_clean_free
Also this isn't used anywhere. I should probably remove these
clean-pools since my original idea where to use them was a bad idea
after all.
and client_destroy (in login-common/client-common.c)
For clearing the password, yes.
Original code has:
if (data[i] == '\n') {
(...)
if (data[i-1] != '\r')
stream-w_buffer[dest++] = '\r';
if (dest == stream-buffer_size)
break;
stream-w_buffer[dest++] = data[i] is not executed (no space in dest)
i_stream_skip(stream-parent, i);
i is set on '\n'
Yes, but lets say i=1 and data[1] == '\n', then i_stream_skip() skips
one byte (the data[0]) but not the '\n'.
Re: [Dovecot] [pigeonhole] Libraries not linked correctly
On Wed, 2010-09-01 at 21:20 -0400, pyther wrote: Hello All I am trying to compile and use pigeonhole and I am experiencing the same exact problem that is stated here: http://dovecot.org/pipermail/dovecot/2010-August/051859.htm What OS do you use? Maybe a pattern will emerge.. lt does not seem as if -Wl,-rpath -Wl,/usr/local/lib/dovecot is being appended to gcc -std=gnu99 -march=i686 -mtune=generic -O2 -pipe -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -Wstrict-aliasing=2 -o managesieve-login client.o client-authenticate.o managesieve-login-settings.o managesieve-proxy.o ../../src/lib-managesieve/libmanagesieve.a -ldovecot-login -L/usr/lib/dovecot -ldovecot -lrt ^^See: http://dovecot.org/pipermail/dovecot/2010-August/051907.html However, my lack of programming knowledge and skills prohibit me from trying to fix this myself. You can manually copypaste it there to compile it correctly.
Re: [Dovecot] My %% variables in the namespace settings do not appear to be expanding.
On Thu, 2010-09-02 at 16:34 +0100, dove...@copeonthe.net wrote: %D%n, does not work, understandably, and caused a LOT of error messages and stops the server being usable. Use %%Dn and %%Dd I was originally hoping for a modifier for the dot character in variables, in the same way that the quote characters can be escaped with the %E prefix modifier. Hmmh.
Re: [Dovecot] My %% variables in the namespace settings do not appear to be expanding.
On Thu, 2010-09-02 at 16:34 +0100, dove...@copeonthe.net wrote: %%Dn, for example, gives: Sep 02 16:26:01 auth: Debug: passwd-file(@23foxgrove.co.uk): lookup: us...@23foxgrove.co.uk file=/usr/local/dovecot-2.0.1/auth/passwd.dovecot Oops, you mentioned %%Dn. Yeah .. looks like there are some problems using modifiers there. Should be possible to get that fixed..
Re: [Dovecot] My %% variables in the namespace settings do not appear to be expanding.
On Thu, 2010-09-02 at 16:44 +0100, Timo Sirainen wrote: On Thu, 2010-09-02 at 16:34 +0100, dove...@copeonthe.net wrote: %%Dn, for example, gives: Sep 02 16:26:01 auth: Debug: passwd-file(@23foxgrove.co.uk): lookup: us...@23foxgrove.co.uk file=/usr/local/dovecot-2.0.1/auth/passwd.dovecot Oops, you mentioned %%Dn. Yeah .. looks like there are some problems using modifiers there. Should be possible to get that fixed.. Now I remember. It of course needs to be able to reverse the %%Dn_%%Dd when parsing it back to u...@domain, and no such code exists (and I'm not all that interested in adding such code). Maybe one potential way to solve this would be to create user aliases that don't have any dots in the usernames and do the sharing via those names.
Re: [Dovecot] [DC2.x ]pop3-login seg faults when compiled with gcc 4.5.2 and gold
On Thu, 2010-09-02 at 01:13 +0100, interfaSys sàrl wrote: I wonder why it's even trying to use pthreads .. Dovecot doesn't use threads. Maybe that's the problem, it's linking wrong libraries. Probably because I used the CFLAGS for OpenMP. But even if I don't use CFLAGS, I get a segfault. Could it be because some libraries are using it? .. #8 0x0008004e015e in _init () from /usr/local/lib/gcc45/libgomp.so.1 Looks like libgomp, which is OpenMP. So it's still using it even when you haven't enabled it with CFLAGS?
Re: [Dovecot] My %% variables in the namespace settings do not appear to be expanding.
On Thu, 02 Sep 2010 16:52:09 +0100 Timo Sirainen t...@iki.fi wrote: On Thu, 2010-09-02 at 16:44 +0100, Timo Sirainen wrote: On Thu, 2010-09-02 at 16:34 +0100, dove...@copeonthe.net wrote: %%Dn, for example, gives: Sep 02 16:26:01 auth: Debug: passwd-file(@23foxgrove.co.uk): lookup: us...@23foxgrove.co.uk file=/usr/local/dovecot-2.0.1/auth/passwd.dovecot Oops, you mentioned %%Dn. Yeah .. looks like there are some problems using modifiers there. Should be possible to get that fixed.. Now I remember. It of course needs to be able to reverse the %%Dn_%%Dd when parsing it back to u...@domain, and no such code exists (and I'm not all that interested in adding such code). Fair enough, I don't blame you. Maybe one potential way to solve this would be to create user aliases that don't have any dots in the usernames and do the sharing via those names. It may be that I'm leading this down a track because I've already tried to sort this out and gone about it the wrong way. Essentially I have users with a dot in the name, with a namespace separator that is set to be a dot character also, and I'd like a configuration that lets them have shared folders via IMAP ACLs. Is there a configuration that will allow this with the current dovecot? Kenneth -- Kenneth Cope kenn...@copeonthe.net ** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are not authorised to and must not disclose, copy, distribute, or retain this message or any part of it. CopeOnThe.net accepts no responsibility for information, errors or omissions in this email. ***
Re: [Dovecot] Segfault on dovecot 2.0.1 w/ACLs
On Wed, 2010-09-01 at 16:02 -0600, Michael M. Slusarz wrote: prefix = #shared. .. So doveconf wasn't quoting the prefix value correctly in the converted output. Fixed: http://hg.dovecot.org/dovecot-2.0/rev/31387c9d5b88 Not sure if you want to put a check/warning somewhere if namespaces have the same prefixes. Fixed: http://hg.dovecot.org/dovecot-2.0/rev/a3f016b7b914
Re: [Dovecot] [DC2.x ]pop3-login seg faults when compiled with gcc 4.5.2 and gold
On 02/09/2010 16:53, Timo Sirainen wrote: On Thu, 2010-09-02 at 01:13 +0100, interfaSys sàrl wrote: I wonder why it's even trying to use pthreads .. Dovecot doesn't use threads. Maybe that's the problem, it's linking wrong libraries. Probably because I used the CFLAGS for OpenMP. But even if I don't use CFLAGS, I get a segfault. Could it be because some libraries are using it? ... #8 0x0008004e015e in _init () from /usr/local/lib/gcc45/libgomp.so.1 Looks like libgomp, which is OpenMP. So it's still using it even when you haven't enabled it with CFLAGS? Indeed pop3-login: libdovecot-login.so.0 = /usr/lib/dovecot/libdovecot-login.so.0 (0x80043c000) libdovecot.so.0 = /usr/lib/dovecot/libdovecot.so.0 (0x800453000) libssl.so.7 = /usr/local/lib/libssl.so.7 (0x80054a000) libcrypto.so.7 = /usr/local/lib/libcrypto.so.7 (0x8005ae000) librt.so.1 = /usr/lib/librt.so.1 (0x800797000) libc.so.7 = /lib/libc.so.7 (0x80089c000) libgomp.so.1 = /usr/local/lib/gcc45/libgomp.so.1 (0x8004dd000) Maybe because of this? /usr/local/lib/libcrypto.so.7: libgomp.so.1 = /usr/local/lib/gcc45/libgomp.so.1 (0x800542000) OpenSSL was compiled with OpenMP
Re: [Dovecot] My %% variables in the namespace settings do not appear to be expanding.
On Thu, 2010-09-02 at 17:00 +0100, dove...@copeonthe.net wrote: Essentially I have users with a dot in the name, with a namespace separator that is set to be a dot character also, and I'd like a configuration that lets them have shared folders via IMAP ACLs. Is there a configuration that will allow this with the current dovecot? No. That would require changing the dots to something else and back in several places in the code. Maybe this could be done automatically some day..
Re: [Dovecot] [DC2.x ]pop3-login seg faults when compiled with gcc 4.5.2 and gold
On Thu, 2010-09-02 at 17:09 +0100, interfaSys sàrl wrote: #8 0x0008004e015e in _init () from /usr/local/lib/gcc45/libgomp.so.1 Maybe because of this? /usr/local/lib/libcrypto.so.7: libgomp.so.1 = /usr/local/lib/gcc45/libgomp.so.1 (0x800542000) OpenSSL was compiled with OpenMP Yeah, that looks like the problem. I've no idea about OpenMP or pthreads library in general, so I don't know if there's anything Dovecot can do about this.
[Dovecot] Best way to migrate from qmail-ldap's autoreply?
Hi, qmail-ldap uses four LDAP attributes for handling autoreplies in qmail-local: - deliverymode (to see whether the autoreply should be send) - mailreplytext, which is a base64 encoded multiline string* - mailreplystart and mailreplystop, unix dates I'm wondering how this could be handled with Dovecot (2.0) in the most easier way. All I could find out so far: - patching the code - writing a plugin (it it possible to get these LDAP attributes in the LMTPd with a single userdb-lookup and get it from the plugin during a new mail event? Any examples?) - conditionally provide these values to a sieve script and process them accordingly (the base64 string and maybe the dates seem to be hard for the first glimpse) the mad stuff: - using an LD_PRELOAD library, which emulates per user sieve scripts according to LDAP lookups (gives a lot more flexibility) - an LMTP proxy/server, which does all this and forwards the traffic to the Dovecot LMTPd/deliver process * a sample mailreplytext looks like this: '''%HEADER% Content-type: text/plain; charset=utf-8 távol vagyok''' (of course this is in UTF-8).
Re: [Dovecot] My %% variables in the namespace settings do not appear to be expanding.
On Thu, 02 Sep 2010 17:10:23 +0100 Timo Sirainen t...@iki.fi wrote: On Thu, 2010-09-02 at 17:00 +0100, dove...@copeonthe.net wrote: Essentially I have users with a dot in the name, with a namespace separator that is set to be a dot character also, and I'd like a configuration that lets them have shared folders via IMAP ACLs. Is there a configuration that will allow this with the current dovecot? No. That would require changing the dots to something else and back in several places in the code. Maybe this could be done automatically some day.. Okay. Thank you very much for all your help. Kenneth
[Dovecot] LMTP plugin with MySQL query
Hello, I want to create a plugin to dovecot 2.0.1, to call a on the deliver action. My plugin work even if I include other libs like mysql or wathever I tried to use the sql-api.h (lib-sql), but at runtime, I have : undefined symbol: sql_init, Fatal: Couldn't load required plugins Is it possible to use lib-sql api or mysql libs in a plugin ? I've tried with lda or lmtp plugin with the same result... Any idea ? Charly.
Re: [Dovecot] [DC2.x ]pop3-login seg faults when compiled with gcc 4.5.2 and gold
On 02/09/2010 17:12, Timo Sirainen wrote: On Thu, 2010-09-02 at 17:09 +0100, interfaSys sàrl wrote: #8 0x0008004e015e in _init () from /usr/local/lib/gcc45/libgomp.so.1 Maybe because of this? /usr/local/lib/libcrypto.so.7: libgomp.so.1 = /usr/local/lib/gcc45/libgomp.so.1 (0x800542000) OpenSSL was compiled with OpenMP Yeah, that looks like the problem. I've no idea about OpenMP or pthreads library in general, so I don't know if there's anything Dovecot can do about this. The strange thing is that it works fine for all the other components of Dovecot, including plugins, so there must be something done a little differently in pop3-login or gcc still has some bugs with OpenMP
[Dovecot] Maybe wrong quota recalculation behaviour on overquota delivery try
Hi all,
we are experencing some, may be strange, slowness in our dovecot/postfix
mail system.
We have three dovecot 2.0.1 server with LMTP, POP3 and IMAP enabled.
Three shared OCFS2 filesystem are used for mail storage.
We have too a load balancer based on dovecot director in front of these
three server.
Our SMTP cluster receives mail and send by LMTP to load balancer.
Everything work flowless, but sometime, under heavy load, three server
slow down a lot.
Our mail gateway logs a lot of:
status=deferred (host 10.194.1.20[10.194.1.20] said: 451 4.4.0 Remote
server not answering (DATA output timeout)
IP 10.194.1.20 is dovecot director.
Looking at mbox servers log nothing seem to happen, but stracing lmtp
process we can see stat system calls to all Maildir files, as if dovecot
was recalculating quota and regenerating maildirsize file.
It happens always when LMTP tries to deliver a mail to an overquota account.
Is this possible and right? How can we look deeper in system to have
more information?
Best regards
Luca
PS: dovecot configuration for maildir servers is:
# 2.0.1: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.26-2-686-bigmem i686 Debian 5.0.5
auth_master_user_separator = *
auth_verbose = yes
disable_plaintext_auth = no
mail_location = maildir:~/Maildir
mail_plugins = $mail_plugins notify mail_log quota
managesieve_notify_capability = mailto
managesieve_sieve_capability = comparator-i;octet
comparator-i;ascii-casemap fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex
imap4flags copy include variables body enotify environment mailbox date
spamtest spamtestplus virustest
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
mail_log_fields = uid box msgid from size
quota = maildir:User quota
quota_warning = storage=95%% quota-warning.sh 95 %u
quota_warning2 = storage=80%% quota-warning.sh 80 %u
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_global_dir = /etc/dovecot/sieve/global/
sieve_global_path = /etc/dovecot/sieve/default.sieve
}
protocols = imap pop3 lmtp sieve
service auth {
unix_listener auth-userdb {
group = dovecot
mode = 0600
user = dovecot
}
}
service director {
inet_listener {
port = 8100
}
}
service lmtp {
inet_listener lmtp {
address = 10.194.1.31 127.0.0.1 ::1
port = 24
}
}
service managesieve-login {
inet_listener sieve {
address = 10.194.1.31 127.0.0.1 ::1
port = 4190
}
process_min_avail = 0
service_count = 1
}
service quota-warning {
executable = script /virtual/shared/quota-warning.sh
unix_listener quota-warning {
user = vmail
}
user = dovecot
}
ssl_cert = /etc/ssl/certs/dovecot.pem
ssl_key = /etc/ssl/private/dovecot.pem
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
verbose_proctitle = yes
protocol lmtp {
mail_plugins = $mail_plugins quota sieve
}
protocol lda {
mail_plugins = $mail_plugins quota
}
protocol imap {
mail_plugins = $mail_plugins quota imap_quota
}
protocol pop3 {
mail_plugins = $mail_plugins quota
}
Re: [Dovecot] LMTP plugin with MySQL query
On Thu, 2010-09-02 at 18:24 +0200, Charly wrote: Is it possible to use lib-sql api or mysql libs in a plugin ? Most Dovecot binaries aren't linked with SQL libs, so that won't work without you adding those libraries to Makefiles. Do you need complex SQL queries, or just something simple? Maybe you could use lib-dict and dict proxy to talk to your SQL server? For example see http://wiki2.dovecot.org/Quota/Dict
[Dovecot] dovecot-uidlist: Duplicate file entry at line
Hi All,
we are experiencing another strange problem after upgrading to 2.0.1
from 1.2.
Many people's client aree-retrieving mail from server due to some
problem with dovecot-uidlist file.
Looking at log file seem that all file content is sometime duplicated
and appended to the and of same file.
It look as another process was accesing and trying to modify uidlist table.
Our setup has a load balance (dovecot director) and three backend
server. All of them use 2.0.1.
MTA is a cluster of postfix that deliver mail to load balancer by LMTP.
How can I collect some more information? or, better, solve it?
Regards
Luca
PS: Backend configuration is:
# 2.0.1: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.26-2-686-bigmem i686 Debian 5.0.5
auth_master_user_separator = *
auth_verbose = yes
disable_plaintext_auth = no
mail_location = maildir:~/Maildir
mail_plugins = $mail_plugins notify mail_log quota
managesieve_notify_capability = mailto
managesieve_sieve_capability = comparator-i;octet
comparator-i;ascii-casemap fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex
imap4flags copy include variables body enotify environment mailbox date
spamtest spamtestplus virustest
passdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
plugin {
mail_log_events = delete undelete expunge copy mailbox_delete
mailbox_rename
mail_log_fields = uid box msgid from size
quota = maildir:User quota
quota_warning = storage=95%% quota-warning.sh 95 %u
quota_warning2 = storage=80%% quota-warning.sh 80 %u
sieve = ~/.dovecot.sieve
sieve_dir = ~/sieve
sieve_global_dir = /etc/dovecot/sieve/global/
sieve_global_path = /etc/dovecot/sieve/default.sieve
}
protocols = imap pop3 lmtp sieve
service auth {
unix_listener auth-userdb {
group = dovecot
mode = 0600
user = dovecot
}
}
service director {
inet_listener {
port = 8100
}
}
service lmtp {
inet_listener lmtp {
address = 10.194.1.31 127.0.0.1 ::1
port = 24
}
}
service managesieve-login {
inet_listener sieve {
address = 10.194.1.31 127.0.0.1 ::1
port = 4190
}
process_min_avail = 0
service_count = 1
}
service quota-warning {
executable = script /virtual/shared/quota-warning.sh
unix_listener quota-warning {
user = vmail
}
user = dovecot
}
ssl_cert = /etc/ssl/certs/dovecot.pem
ssl_key = /etc/ssl/private/dovecot.pem
userdb {
args = /etc/dovecot/dovecot-sql.conf.ext
driver = sql
}
verbose_proctitle = yes
protocol lmtp {
mail_plugins = $mail_plugins quota sieve
}
protocol lda {
mail_plugins = $mail_plugins quota
}
protocol imap {
mail_plugins = $mail_plugins quota imap_quota
}
protocol pop3 {
mail_plugins = $mail_plugins quota
}
Re: [Dovecot] Dovecot upgrades break Blackberry instant email (BIS)
On 31/08/2010 19:18, Timo Sirainen wrote: Well, if I change this in v2.0.2 then you don't need it anymore. If you keep it anyway, you'll get a duplicate IDLE capability (actually you get that already post-login), but I don't think that breaks anything. On Tue, 2010-08-31 at 18:07 +0100, interfaSys sàrl - Rich Internet Applications wrote: Worked perfectly! Is it something that can be disabled after a few days or is there no harm in keeping it in the config? (And congrats on your work on Dovecot Timo. Great piece of software.) On 31/08/2010 17:53, Timo Sirainen wrote: On Tue, 2010-08-31 at 17:41 +0100, interfaSys sàrl wrote: I don't find this very convenient and was wondering if there was something that could be done on the Dovecot side? imap_capability = +IDLE I'm thinking about making this default.. Assuming Blackberry people don't happen to fix it themselves soon, which would be nice but maybe not that realistic to expect. Anyway, http://dovecot.org/list/dovecot/2010-April/048147.html explains what's happening. There seems to be another thing that bothers BIS servers. If the subscription file is 660, then it seems the connection is dropped (No idling). When it's 770, then things are fine. Is anybody else able to confirm this?
Re: [Dovecot] Dovecot upgrades break Blackberry instant email (BIS)
On Thu, 2010-09-02 at 18:21 +0100, interfaSys sàrl wrote: If the subscription file is 660, then it seems the connection is dropped (No idling). When it's 770, then things are fine. That doesn't really make any sense.. Subscription file doesn't need to be executable. Anything in Dovecot's logs?
Re: [Dovecot] Maybe wrong quota recalculation behaviour on overquota delivery try
On Thu, 2010-09-02 at 19:14 +0200, Luca Palazzo wrote: Looking at mbox servers log nothing seem to happen, but stracing lmtp process we can see stat system calls to all Maildir files, as if dovecot was recalculating quota and regenerating maildirsize file. It happens always when LMTP tries to deliver a mail to an overquota account. Yes, that's pretty much how Maildir++ quota works. If maildirsize file doesn't only contain a single line showing the quota, or if it's older than 15 minutes, it's recalculated. The maildir filenames really should have ,S=size in the filenames, otherwise quota recalculation is horribly slow. Maybe you should switch to dict quota with file backend. http://wiki2.dovecot.org/Quota/Dict
Re: [Dovecot] Dovecot upgrades break Blackberry instant email (BIS)
On 02/09/2010 18:23, Timo Sirainen wrote: On Thu, 2010-09-02 at 18:21 +0100, interfaSys sàrl wrote: If the subscription file is 660, then it seems the connection is dropped (No idling). When it's 770, then things are fine. That doesn't really make any sense.. Subscription file doesn't need to be executable. Anything in Dovecot's logs? Nothing interesting in the logs: Sep 02 19:25:46 imap-login: Info: Login: user=u...@domain.com, method=PLAIN, rip=68.171.236.236, lip=192.168.1.1, mpid=43792, TLS Sep 02 19:25:46 imap(u...@domain.com): Info: Disconnected: Logged out bytes=39/1010 It's only a theory, I've noticed that all my personal accounts with 770 were connected and the ones with 660 were not.
Re: [Dovecot] nfs director
On Thu, Sep 2, 2010 at 11:09 AM, Seth Mattinen se...@rollernet.us wrote: On 8/31/2010 10:22, Ariel Biener wrote: Oh, and Timo, I don't think we are just a couple of NFS users. Maildir and NFS are not as uncommon as you'd think, even in very large installations. NFS with maildir has been the gold standard for a long time, whether the NFS server be insert favorite distro boxes or a huge hardware appliance. It's certainly not uncommon. ~Seth Sad I think most assume single machine use or two machine use, if dovecot not good, we have sub 1 hour rollback time plan in place, since it only daemon and configuration change.
Re: [Dovecot] nfs director
why you reply with 7 page of rubbish? yes we all see you fan of director, many not. why go on tangent about it, it already said it not for us, director not able to help on 24 inbound mail server, else might well turn it all off now, you clear yet? on any other list you be label troll director is not issue, you clear can not grasp this. director not solve the question at hand, so irrelevant go away ranter you waste my time. On Thu, Sep 2, 2010 at 11:08 PM, Cor Bosman c...@xs4all.nl wrote: Hi, We use Dovecot only recent, but many I speak use for very many year, if director was really need, why it only come about now and not 5 or more year ago, all many mail network run broken for so many year? I no think so. It might compliment some situation, but not suitable or advantage for all. Imap not likely happen here for user, hardware no so cheap here,such is storage space not big deal, not until we buy hardware for near cheap as U.S., which 50-70% less than we pay. The director is not needed in most cases. If you feel you dont need the director, then just forget about its existence and ignore all emails about it. Just pretend it doesnt exist :) I'll try to explain why the director is needed in some situations. dovecot has an index file. If you look in your configuration file, you'll see something like: mail_location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u The index file holds an index so dovecot can search in your mailbox really fast. (sorry if you know all this). So, if for instance you want to sort your mailbox on subject, dovecot doesnt have to read through all your emails to find all the subjects. Instead, it can read the index file really fast, and return a sorted mailbox basically instantly. A lot of IMAP servers dont even have this feature! Imagine the alternative, where dovecout would have to read through 25.000 emails, just so it can sort your mailbox on subject. A nightmare for performance, but like I said, some imap servers still do that! Dovecot luckily uses an index, and is therefore faster than a lot of other imap servers. So, we have concluded that dovecot has a really cool performance feature in the form of an index file. Now imagine what happens if you have more than 1 imap server. We have 35 imap servers right now. Each server needs access to the index file. One popular way to do this is by using NFS. With NFS you have a central file server, and each imap server uses the remote fileserver for storage. So now, all 35 servers have access to a cool central index file. But, there is a slight snag. NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file. The index file can then end up being corrupted. This causes dovecot to give faulty data back to the client, which in practice ends up looking like the mailbox of the user is empty. Not good. This causes calls to support desks about why a user has just lost all his mail. It's not lost, it's just that the index file is corrupt, and now dovecot thinks the mail is gone. You may wonder rightly so, why this is a problem only now. Well, it hasnt been a problem only now. We've been dealing with this issue for years. We've had discussions with Timo over the years on how to solve this problem. But Timo has only so much time, and this problem was not important enough to fix. He had more important bugs to fix. So, we lived with the problem, or worked around it in ugly ways. Recently Timo has found the time (I believe an ISP paid Timo to solve this problem) to think about ways to solve it. It's really not an easy problem to fix. We've known for years that the best way to fix this would be to have a user always end up on the same server, because thats when that specific NFS problem doesnt occur. If 2 sessions of the same user both end up on Server 1, then both sessions will see the same data, and no problems can occur. But how to get all sessions of a user on the same server? Most loadbalancers dont support this feature. Our Foundries dont, and our previous Alteons didnt either. So Timo thought of a way how dovecot itself could make sure all sessions of a user end up on the same server. Basically by turning dovecot into a proxy, that proxies connections to another server. It maintains an internal state on what user belongs on what server, and redirects connections appropriately. And thats all the director is. A proxy with an internal state of where to proxy a connection to. So why dont you see this problem with your setup? Like I said, this is only a problem with specific setups. - you need to be using NFS for central index storage. If you dont use NFS for your indexes, you wont see this problem, and you wont need the director. carry on as you always have.
Re: [Dovecot] nfs director
On Fri, 3 Sep 2010 07:22:00 +1000 Edward avanti edward.ava...@gmail.com articulated: why you reply with 7 page of rubbish? yes we all see you fan of director, many not. why go on tangent about it, it already said it not for us, director not able to help on 24 inbound mail server, else might well turn it all off now, you clear yet? on any other list you be label troll director is not issue, you clear can not grasp this. director not solve the question at hand, so irrelevant go away ranter you waste my time. If you are going to go off on a temper tantrum, at least post it in an acceptable fashion; id est, lose the top posting urge. While you are at it, trim the superfluous text from the replied to post. -- Jerry ✌ dovecot.u...@seibercom.net Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __ There is no sadder sight than a young pessimist. signature.asc Description: PGP signature
Re: [Dovecot] nfs director
have you been told where you might go lately and do with some part your anatomy? this Timo list, not you list, best remember this since you nobody this list On Fri, Sep 3, 2010 at 8:19 AM, Jerry dovecot.u...@seibercom.net wrote: On Fri, 3 Sep 2010 07:22:00 +1000 Edward avanti edward.ava...@gmail.com articulated: why you reply with 7 page of rubbish? yes we all see you fan of director, many not. why go on tangent about it, it already said it not for us, director not able to help on 24 inbound mail server, else might well turn it all off now, you clear yet? on any other list you be label troll director is not issue, you clear can not grasp this. director not solve the question at hand, so irrelevant go away ranter you waste my time. If you are going to go off on a temper tantrum, at least post it in an acceptable fashion; id est, lose the top posting urge. While you are at it, trim the superfluous text from the replied to post. -- Jerry ✌ dovecot.u...@seibercom.net Disclaimer: off-list followups get on-list replies or get ignored. Please do not ignore the Reply-To header. __ There is no sadder sight than a young pessimist.
Re: [Dovecot] nfs director
-Original Message- From: Edward avanti have you been told where you might go lately and do with some part your anatomy? this Timo list, not you list, best remember this since you nobody this list Seriously? Grow up and/or take it off-list. -Brad
Re: [Dovecot] [DC2.x ]pop3-login seg faults when compiled with gcc 4.5.2 and gold
On 02/09/2010 17:12, Timo Sirainen wrote: On Thu, 2010-09-02 at 17:09 +0100, interfaSys sàrl wrote: #8 0x0008004e015e in _init () from /usr/local/lib/gcc45/libgomp.so.1 Maybe because of this? /usr/local/lib/libcrypto.so.7: libgomp.so.1 = /usr/local/lib/gcc45/libgomp.so.1 (0x800542000) OpenSSL was compiled with OpenMP Yeah, that looks like the problem. I've no idea about OpenMP or pthreads library in general, so I don't know if there's anything Dovecot can do about this. I recompiled some of the base libs without OpenMP, but I still get a seg fault. I think the Graphite optimisations automatically link GOMP and pthread
