Re: [Dovecot] Managesieve segfault with dovecot 2.1.8
On 9/7/2012 1:56 AM, Daniel Parthey wrote: Hi, I'm getting segfaults and unexpected disconnects from managesieve server, when the Thunderbird SIEVE extension tries to validate SIEVE scripts agains Pidgeonhole in Dovecot 2.1.8. The extension says: "Server terminated unexpectedly the connection, click on reconnect to try again." It can be reproduced by simply calling CHECKSCRIPT followed by a string: This is a known and fixed problem: http://hg.rename-it.nl/dovecot-2.1-pigeonhole/rev/6ceeb6421231 I should make a release soon. Regards, Stephan.
[Dovecot] Managesieve segfault with dovecot 2.1.8
Hi,
I'm getting segfaults and unexpected disconnects from managesieve server,
when the Thunderbird SIEVE extension tries to validate SIEVE scripts
agains Pidgeonhole in Dovecot 2.1.8.
The extension says:
"Server terminated unexpectedly the connection, click on reconnect to try
again."
It can be reproduced by simply calling CHECKSCRIPT followed by a string:
mail01:~# telnet 127.0.0.1 19200
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
"IMPLEMENTATION" "Sieve"
"SIEVE" "fileinto reject envelope encoded-character vacation subaddress
comparator-i;ascii-numeric relational regex imap4flags copy include variables
body enotify environment mailbox date ihave"
"NOTIFY" "mailto"
"SASL" "PLAIN"
"VERSION" "1.0"
OK "Mailbox"
AUTHENTICATE "PLAIN" "AGRwYXJ0aGV5QGV4YW1wbGUub3JnAGRwYXJ0aGV5"
OK "Logged in."
CHECKSCRIPT {6+}
abcdef
Connection closed by foreign host.
The server interrupts the connection and the logs show the following:
# dovecot.log
Sep 7 01:40:46 dovecot: mailbox: mail: managesieve(dpart...@example.org):
Fatal: master: service(managesieve): child 31356 killed with signal 11 (core
dumped)
# kern.log
Sep 7 01:40:46 kernel: [1417105.954609] managesieve[31356]: segfault at 0 ip
7f1c415c4876 sp 7fffb3731f88 error 4 in
libc-2.11.1.so[7f1c41543000+17a000]
Here is the backtrace:
mail01:~# gdb /usr/lib/dovecot/managesieve /var/tmp/core.managesieve.31356
Core was generated by `dovecot-mailbox/managesieve'.
Program terminated with signal 11, Segmentation fault.
#0 0x7f1c415c4876 in ?? () from /lib/libc.so.6
(gdb) bt full
#0 0x7f1c415c4876 in ?? () from /lib/libc.so.6
No symbol table info available.
#1 0x0040b1c3 in sieve_storage_save_will_activate (ctx=0x257f1a0) at
sieve-storage-save.c:328
_data_stack_cur_id = 4
scriptname = 0x25400ae "default.sieve"
ret =
#2 0x004065b0 in cmd_putscript_finish_parsing (cmd=0x2575cb8) at
cmd-putscript.c:206
ehandler =
cpflags =
sbin =
errors =
_data_stack_cur_id = 0
script =
client = 0x2575c30
ctx = 0x2578180
args = 0x254bc68
ret =
#3 0x00406838 in cmd_putscript_continue_script (cmd=0x2575cb8) at
cmd-putscript.c:423
all_written =
client = 0x2575c30
ctx = 0x2578180
size = 39279968
#4 0x00406caf in client_input_putscript (context=0x2575c30) at
cmd-putscript.c:84
cmd = 0x2575cb8
__FUNCTION__ = "client_input_putscript"
#5 0x7f1c41d1d3e6 in io_loop_call_io (io=0x2575fd0) at ioloop.c:379
ioloop = 0x2548680
t_id = 2
#6 0x7f1c41d1e46f in io_loop_handler_run (ioloop=) at
ioloop-epoll.c:213
ctx = 0x25489f0
event = 0x2548a60
list = 0x2576020
io = 0x0
tv = {tv_sec = 18, tv_usec = 988469}
msecs =
ret =
i = 0
call = false
#7 0x7f1c41d1d388 in io_loop_run (ioloop=0x2548680) at ioloop.c:398
No locals.
#8 0x7f1c41d09653 in master_service_run (service=0x2548530,
callback=0x25400ae) at master-service.c:543
No locals.
#9 0x004096ce in main (argc=1, argv=0x2548370) at main.c:308
set_roots = {0x610d60, 0x0}
login_set = {auth_socket_path = 0x2540088
"/var/run/dovecot/auth-master", postlogin_socket_path = 0x0,
postlogin_timeout_secs = 60, callback = 0x409810 ,
failure_callback = 0x409290 }
service_flags =
storage_service_flags = MAIL_STORAGE_SERVICE_FLAG_DISALLOW_ROOT
username = 0x0
c =
Regards
Daniel
--
https://plus.google.com/103021802792276734820
# 2.1.8: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-42-server x86_64 Ubuntu 10.04.4 LTS
auth_cache_negative_ttl = 0
auth_cache_size = 10 M
auth_cache_ttl = 1 mins
auth_verbose = yes
auth_verbose_passwords = sha1
deliver_log_format = mailbox: deliver: msgid=%m from=%f: %$
dict {
quota = mysql:/etc/dovecot/conf.d/dovecot-dict-sql.conf.ext
}
disable_plaintext_auth = no
doveadm_password = xxx
imapc_features = rfc822.size
imapc_host = local-mailbox
imapc_port = 18143
instance_name = dovecot-mailbox
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
login_greeting = Mailbox
login_log_format = mailbox: login: %$: %s
login_trusted_networks = 10.129.3.0/24
mail_debug = yes
mail_fsync = always
mail_gid = vmail
mail_home = /mail/dovecot/%d/%n
mail_location = mdbox:~/mail
mail_log_prefix = "mailbox: mail: %s(%u): "
mail_plugins = quota stats
mail_privileged_group = vmail
mail_uid = vmail
managesieve_implementation_string = Sieve
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy
include variables body enotify environment mailbox date ihave
mdbox_rotate_interval = 1 weeks
mdbox_rotate_size = 50 M
mmap_disable = yes
passdb {
args = /etc/dovecot/conf.d/dovecot-sql.conf.ext
driver =
Re: [Dovecot] thunderbird not connecting
Turn on more debugging with these in your configuration: auth_verbose = yes auth_debug = yes auth_debug_passwords = yes mail_debug = yes verbose_ssl= yes and then check the server logs after trying to login. Also, I'm curious why you masked your PRIVATE ip address and not your public one. Bill On 9/4/2012 12:21 AM, cc young wrote: cannot get TB to recognize either pop3/s or imap/s server can connect just fine with: openssl s_client -connect ms1.myserver.net:993 . login ... but trying with TB /var/log/mail.log gets: dovecot: pop3-login: Aborted login (no auth attempts): rip=223.205.150.234, lip=xxx.xx.xx.xx dovecot: imap-login: Aborted login (no auth attempts): rip=223.205.150.234, lip=xxx.xx.xx.xx -- View this message in context: http://dovecot.2317879.n4.nabble.com/thunderbird-not-connecting-tp37389.html Sent from the Dovecot mailing list archive at Nabble.com.
Re: [Dovecot] dovecotadm error
Hi Michael,
Michael wrote:
> doveadm expunge -A mailbox Junk savedbefore 28d
> doveadm(root): Error: User listing returned failure
>
> The mail log file says:
> dovecot: auth-worker(18549): Error: sql: Iterate query failed: Table
> 'system.users' doesn't exist (using built-in default iterate_query:
> SELECT username, domain FROM users)
>
> dovecot-sql.conf:
> iterate_query = SELECT username AS user FROM accounts
Some examples in the wiki refer to "username", you could try this:
iterate_query = SELECT username FROM accounts
> though I have had to comment the following from dovecot.conf:
> #iteratedb {
> # args = /usr/local/etc/dovecot/dovecot-sql.conf
> # driver = sql
> #}
> because it errors on dovecot start up.
Your primary userdb refers to dovecot-sql.lda.conf, not dovecot-sql.conf.
What does /usr/local/etc/dovecot/dovecot-sql.lda.conf look like
and is there an iterate_query inside this file?
Regards
Daniel
--
https://plus.google.com/103021802792276734820
Re: [Dovecot] Trouble implementing Antispam plug-in for Dovecot
On 9/6/2012 2:19 PM, Ed W wrote: > On 06/09/2012 18:56, Ben Johnson wrote: >> >> On 9/6/2012 6:10 AM, Charles Marcus wrote: >>> On 2012-09-05 6:20 PM, Ben Johnson wrote: >>> My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu >>> 10.04. >>> >>> 1.2.9 is really old... you really need to upgrade to a recent/stable >>> version. >> Thanks, Charles. I do see your point. One of the challenges we face in >> this regard is that we're using a Long-Term-Support version of Ubuntu >> (10.04) and 1.2.9 is the latest package in the OS's repository. >> >> That said, we could upgrade manually, but this is a production server on >> which downtime must be minimized, and we all know how unexpected issues >> arise during installation (even when the procedure is tested in a >> closely equivalent development environment). > > I personally use (lightweight) virtualisation on any new machine, I > really don't see any reason why NOT to. I would typically also setup my > mounts such that the operating system is separate from "the data". This > makes it easy to upgrade the OS/services, but without touching the data > (test before/after on the same data for example) Thanks for your valuable insights, Ed. That seems like a worthwhile approach. > So in my situation I would boot a fairly small (gentoo in my case) > virtual environment that runs only dovecot + postfix, it mounts the mail > spools separately - I say "boot", but because I'm using linux-vservers, > it's really a fancy chroot, and so the instance will start in 2-3 > seconds (restarts are similarly near instant). I would upgrade by > cloning this installation, upgrading it, testing it to bits, and then to > make it live basically you swap this "machine" for the live machine. > There are various ways it could be made near seamless, but in my > situation I can bear a couple of seconds whilst I literally restart the > "machine" > > Similarly I segregate all my services into a dozen or so "virtual > machines", so DNS has it's own "machine" and so does logging, databases, > almost every webservice gets its own virtual environment, etc. You could > use a full blown vmware/kvm/etc if that floats your boat better, but the > point remains it's so trivial to install, makes upgrades to trivial and > massively decreases your downtime risk that it's very hard to find a > reason NOT to do it... While I'm with you here, and I understand the theory (and practice, to some extent), doesn't all of this require a true, physical machine? We can't justify the expense associated with a physical machine in a hosted environment, so we're left with so-called VPSs. My understanding is that OpenVZ cannot be installed on a VPS (for seemingly obvious reasons -- namely, that the VPS is itself an OpenVZ container). > I haven't tried too hard to keep my instances tiny, so each is probably > around 400-600MB in my case. However, if it were important this could > easily be reduced to 10-100s MB each using various hardlink features. > As you can see it's easy to snapshot a whole machine to manage > upgrades/backups, etc > > > This is more about infrastructure, but I honestly can't get over how > many people are sitting on their hands shackled by "I'm on Debian xxx > and I can't install any software newer than 5 years old"... It's so easy > to escape from that trap...!! Perhaps easy, but not necessarily inexpensive. ;-) Thanks again for sharing the details of your strategy; I'll bear all of this in mind moving forward. > Good luck > > Ed W > > -Ben
Re: [Dovecot] Trouble implementing Antispam plug-in for Dovecot
On 06/09/2012 18:56, Ben Johnson wrote: On 9/6/2012 6:10 AM, Charles Marcus wrote: On 2012-09-05 6:20 PM, Ben Johnson wrote: My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu 10.04. 1.2.9 is really old... you really need to upgrade to a recent/stable version. Thanks, Charles. I do see your point. One of the challenges we face in this regard is that we're using a Long-Term-Support version of Ubuntu (10.04) and 1.2.9 is the latest package in the OS's repository. That said, we could upgrade manually, but this is a production server on which downtime must be minimized, and we all know how unexpected issues arise during installation (even when the procedure is tested in a closely equivalent development environment). I personally use (lightweight) virtualisation on any new machine, I really don't see any reason why NOT to. I would typically also setup my mounts such that the operating system is separate from "the data". This makes it easy to upgrade the OS/services, but without touching the data (test before/after on the same data for example) So in my situation I would boot a fairly small (gentoo in my case) virtual environment that runs only dovecot + postfix, it mounts the mail spools separately - I say "boot", but because I'm using linux-vservers, it's really a fancy chroot, and so the instance will start in 2-3 seconds (restarts are similarly near instant). I would upgrade by cloning this installation, upgrading it, testing it to bits, and then to make it live basically you swap this "machine" for the live machine. There are various ways it could be made near seamless, but in my situation I can bear a couple of seconds whilst I literally restart the "machine" Similarly I segregate all my services into a dozen or so "virtual machines", so DNS has it's own "machine" and so does logging, databases, almost every webservice gets its own virtual environment, etc. You could use a full blown vmware/kvm/etc if that floats your boat better, but the point remains it's so trivial to install, makes upgrades to trivial and massively decreases your downtime risk that it's very hard to find a reason NOT to do it... I haven't tried too hard to keep my instances tiny, so each is probably around 400-600MB in my case. However, if it were important this could easily be reduced to 10-100s MB each using various hardlink features. As you can see it's easy to snapshot a whole machine to manage upgrades/backups, etc This is more about infrastructure, but I honestly can't get over how many people are sitting on their hands shackled by "I'm on Debian xxx and I can't install any software newer than 5 years old"... It's so easy to escape from that trap...!! Good luck Ed W
Re: [Dovecot] Trouble implementing Antispam plug-in for Dovecot
On 9/6/2012 6:10 AM, Charles Marcus wrote: > On 2012-09-05 6:20 PM, Ben Johnson wrote: > >> My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu > 10.04. > > 1.2.9 is really old... you really need to upgrade to a recent/stable > version. Thanks, Charles. I do see your point. One of the challenges we face in this regard is that we're using a Long-Term-Support version of Ubuntu (10.04) and 1.2.9 is the latest package in the OS's repository. That said, we could upgrade manually, but this is a production server on which downtime must be minimized, and we all know how unexpected issues arise during installation (even when the procedure is tested in a closely equivalent development environment). >> I followed the Wiki article at http://wiki2.dovecot.org/Plugins/Antispam > > That is for version 2.x (note the wiki2)... Noted; thank you. > I recommend going ahead and upgrading to the latest 2.1.9 and starting > over. > Maybe I'll wait until we upgrade from Ubuntu 10.04 to 12.04 to fiddle with this further. Thanks again, -Ben
Re: [Dovecot] Memory leak by getting the virtual size of a IMAP folder
Hi I think I have fixed this bug. The 'virtual_size_add_new' function in src/lib-storage/index/index-status.c creates a 'search_args' object and forgets to free it at the end. The function does call the 'mailbox_search_deinit' function but 'mailbox_search_deinit' only frees the 'search_ctx' object and reduce the refcount of 'search_args' by one but that doesn't cause 'search_args' to be freed because its refcount was 2. So an extra 'mail_search_args_unref' must be called. Timo, it would be nice if you could write a small comment if the attached patch is the right approach to address this issue. Dafan On 09/05/2012 10:53 AM, Dafan Zhai wrote: Hi everyone, I am writing a dovecot statistic plugin, which calls the 'mailbox_get_metadata' function with MAILBOX_METADATA_VIRTUAL_SIZE as the 2nd parameter. enum mailbox_status_items metadata_items = MAILBOX_METADATA_VIRTUAL_SIZE; struct mailbox_metadata metadata; mailbox_get_metadata(mailbox, metadata_items, &metadata); but Valgrind finds a memory leak when this function is called: ---snip-- ==10304== 12,288 bytes in 3 blocks are definitely lost in loss record 74 of 76 ==10304== at 0x40222A4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==10304== by 0x416EE9F: block_alloc (mempool-alloconly.c:237) ==10304== by 0x416F16C: pool_alloconly_create (mempool-alloconly.c:140) ==10304== by 0x409C07D: mail_search_build_init (mail-search-build.c:187) ==10304== by 0x40C77F1: index_mailbox_get_metadata (index-status.c:200) ==10304== by 0x4067102: maildir_mailbox_get_metadata (maildir-storage.c:486) ==10304== by 0x409F78B: mailbox_get_metadata (mail-storage.c:1298) ==10304== by 0x490492D: mailbox_status_update (statistic-plugin.c:310) ==10304== by 0x4904E2E: statistic_mail_save (statistic-plugin.c:426) ==10304== by 0x41A054D: notify_contexts_mail_save (notify-plugin.c:61) ==10304== by 0x41A104F: notify_save_finish (notify-storage.c:143) ==10304== by 0x409EE94: mailbox_save_finish (mail-storage.c:1673) ---snip-- I have looked into the source code, and found that from the 'mail_search_build_init' function call in lib-storage/index/index-status.c:200 a pool is created, but the pool is not freed in the mailbox_search_deinit function call in lib-storage/index/index-status.c:218. This may be the reason of the memory leak. But I do not know how to free the pool. It seems to me that doveadm is having the same memory leak because it is using 'mail_search_build_init' too. ---snip-- # valgrind --leak-check=full doveadm mailbox status -u testuser1 vsize INBOX ==10457== Memcheck, a memory error detector ==10457== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==10457== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info ==10457== Command: doveadm mailbox status -u testuser1 vsize INBOX ==10457== INBOX vsize=41643319 ==10457== ==10457== HEAP SUMMARY: ==10457== in use at exit: 4,356 bytes in 3 blocks ==10457== total heap usage: 440 allocs, 437 frees, 539,124 bytes allocated ==10457== ==10457== 4,096 bytes in 1 blocks are definitely lost in loss record 3 of 3 ==10457== at 0x40222A4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so) ==10457== by 0x41AEE9F: ??? (in /usr/lib/dovecot/libdovecot.so.0.0.0) ==10457== by 0x41AF16C: pool_alloconly_create (in /usr/lib/dovecot/libdovecot.so.0.0.0) ==10457== by 0x40DC07D: mail_search_build_init (in /usr/lib/dovecot/libdovecot-storage.so.0.0.0) ==10457== by 0x8057EB8: doveadm_mail_mailbox_search_args_build (in /usr/bin/doveadm) ==10457== by 0x805855A: ??? (in /usr/bin/doveadm) ==10457== by 0x8053A7B: doveadm_mail_single_user (in /usr/bin/doveadm) ==10457== by 0x8053D2B: ??? (in /usr/bin/doveadm) ==10457== by 0x80541CF: doveadm_mail_try_run (in /usr/bin/doveadm) ==10457== by 0x805B863: main (in /usr/bin/doveadm) ==10457== ==10457== LEAK SUMMARY: ==10457== definitely lost: 4,096 bytes in 1 blocks ==10457== indirectly lost: 0 bytes in 0 blocks ==10457== possibly lost: 0 bytes in 0 blocks ==10457== still reachable: 260 bytes in 2 blocks ==10457== suppressed: 0 bytes in 0 blocks ==10457== Reachable blocks (those to which a pointer was found) are not shown. ==10457== To see them, rerun with: --leak-check=full --show-reachable=yes ==10457== ==10457== For counts of detected and suppressed errors, rerun with: -v ==10457== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 122 from 35) ---snip-- Dovecot version: 2.1.9 OS: Linux 3.0.30-dist i686 maildir:~/Maildir:LAYOUT=fs:INBOX=~/Maildir/INBOX I have applied the following patches from the dovecot 2.1 bran
Re: [Dovecot] different userdb and/or passdb for lmtp and pop3/imap?
Hi Again,
On Wed, Sep 05, 2012 at 12:20:26PM +0200, Gábor Lénárt wrote:
[...]
> I am also confused, because on receiving a mail (via LMTP) different
> kind of LDAP lookup is needed: then mail must be searched, but it's
> storageMailUid based lookup in case of pop3 or imap login ... Is it
> possible to give different userdb/passdb for lmtp and pop3/imap?
Ok, after some "serious" google usage, I found a message in the mail list
archive, that it helps to put userdb/passdb section inside the protocol
specific part of the configuration. So I have something now like this:
protocol pop3 {
[...]
}
protocol imap {
[...]
}
protocol lmtp {
[...]
passdb {
[...]
}
userdb {
[...]
}
}
passdb {
[...]
}
userdb {
[...]
}
I have the idea, that in this way, lmtp should use dbs specified in the lmtp
specific protocol settings, any other stuffs will use the settings at the
"root level" of the configuration (this also includes iteration specific
filter).
However eg if I try to deliver a mail through lmtp in this way to a
non-existing mail user, I can see in the logs, that lmtp after trying the
specific dbs, it will also try the "global" ones, which is not good for me,
as it can cause mis-deliveries instead of rejecting (as I have "some@thing"
formatted %u for both of uid and mail but often they are not the same for
the same user).
I would be able to put db specifications into pop3 and imap (so not "global"
configuration for them) but I guess in this case other services may (?)
fail, like doveadm stuffs with CLI swtich -A (or am I wrong here?). Also
it's not as nice, since then I must duplicate the same db specifications in
both of imap and pop3 protocol specification parts of the configuration even
they are the very same.
Is there any idea to fix this little problem and/or a suggestion to solve my
problem in a more elegant way than I tried to do?
Thanks a lot in advance.
- Gábor
Re: [Dovecot] thunderbird not connecting
On 2012-09-05 10:36 PM, Stan Hoeppner wrote: He'd already checked the server logs, posted some of them, and found no useful information. Checking the client log is the next logical step, whether it turns out to contain useful information or not. In a client/server application, instructing someone to only check half the logs is bad advice Charles. Useful troubleshooting information can be found in either, or both, depending on the circumstances and operation that's failing. That's what I get for replying in the middle of a thread without reading the whole thing... sorry... Been really busy at $dayjob, and haven't had time to read the lists I'm on for a couple of weeks now. And I didn't advise him to *only* check half the logs, I thought he was *starting out* with the client logs. Of course you're correct that once he'd determined that no auth attempts were happening, checking the client side is the next logical step. But then someone else suggested that it was just because Thunderbird was crapware, which is just plain stupid. If Thunderbird isn't AUTH'ing, it is a config (thus, user) error. -- Best regards, Charles
Re: [Dovecot] Trouble implementing Antispam plug-in for Dovecot
On 2012-09-05 6:20 PM, Ben Johnson wrote: > My configuration is Dovecot (1.2.9) + Sieve + SpamAssassin on Ubuntu 10.04. 1.2.9 is really old... you really need to upgrade to a recent/stable version. > I followed the Wiki article at http://wiki2.dovecot.org/Plugins/Antispam That is for version 2.x (note the wiki2)... I recommend going ahead and upgrading to the latest 2.1.9 and starting over. -- Best regards, Charles
