[Dovecot] When are search indexes updated?
I noticed that occasionally searching in my huge archive mailbox can be really slow, so I tried doveadm index on it and it seemed to do a lot of work, which seemed strange given, for example, that dovecot-lda says it keeps Dovecot index files up-to-date. Then I thought, maybe these are different files than the search indices. If so, that's not entirely clear from the docs and Wiki. So, questions: * When are search indexes updated? * Are they updated incrementally? * If not, why not? * If so, why would a mailbox's index drift out-of-date, as mine had? BTW, I'm using the clucene search backend. -- Dave Abrahams BoostPro Computing Software DevelopmentTraining http://www.boostpro.com Clang/LLVM/EDG Compilers C++ Boost
[Dovecot] Ping: virtual mailbox / INTHREAD use case, issues, questions
Timo, is this a known issue? At Thu, 21 Jan 2010 21:36:41 -0500, David Abrahams wrote: At Thu, 21 Jan 2010 21:17:37 -0500, David Abrahams wrote: But, whatever is wrong with command c, it does seem to be working! Spoke too soon, perhaps. If I delete the last message in a thread from my regular INBOX, the conversation view still shows the thread. By the way, is there any way to get the messages to show up as Seen if they're not found in the INBOX? Thanks, -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
Re: [Dovecot] Ping: virtual mailbox / INTHREAD use case, issues, questions
At Wed, 27 Jan 2010 21:41:58 +0200, Timo Sirainen wrote: [1 text/plain; ISO-8859-15 (quoted-printable)] I'll look into that at the same time when I fix the other problem with virtual mailboxes. Which happens after I release v2.0.beta2 (which in turn requires several bugfixes..). Great, thanks. I'll just sit tight and wait. -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
Re: [Dovecot] virtual mailbox / INTHREAD use case, issues, questions
At Tue, 19 Jan 2010 16:51:04 +0200, Timo Sirainen wrote: On Wed, 2010-01-06 at 11:50 -0500, David Abrahams wrote: At Wed, 6 Jan 2010 14:17:40 +0200, Timo Sirainen wrote: On 4.1.2010, at 20.47, David Abrahams wrote: 1. I had to manually create the virtual folder for all my users or they couldn't access their mail at all. Is this fixed in 2.0? No. OK, is it considered a bug that should be addressed? IMO there should at least be a way to tell dovecot to ignore the non-existence of a particular folder. Actually it looks like v2.0 just creates the directory if it's missing. I'm not really sure if it's a good thing or not, but I guess I moved that part of the code to be common across all storages, so maybe it's good. :) Well, my inclination would be to ignore missing directories until you try to do something with them. If I set up a fancy virtual folder arrangement for advanced users I'd like ordinary users not to ever have to encounter it. lots of mailboxes should cause error to be logged. What does the error look like? I could search my logs for it. Easiest would be to make Dovecot log errors to a different file (log_path to different than info_log_path). Then you wouldn't have to search for errors, simply see if anything shows up in the error log. I'll try that, thanks. If you want to grep, I suppose anything with Error: or Fatal: or Panic: prefix would do. Nothing in the current logs, but they may have been rotated out. 7. I tried to create an IMAP search, rather than a virtual folder, that looked for x-mailbox INBOX header like the virtual folder does. It too came up empty. It doesn't exactly surprise me because I don't see an x-mailbox header in any of these messages. x-mailbox doesn't use a header, it uses the actual mailbox name where the message exists. Oh... what if the message exists in multiple mailboxes? Typically anything in INBOX can also be found in my all mail archive. .. I guess that means it's crucial that, whatever else I do, INBOX should be part of the all mail virtual folder in order for this to work. I didn't quite understand that before, and I think it's important to have a description somewhere of how x-mailbox works that would help me to get to that conclusion. Certainly the existing description at http://wiki.dovecot.org/Plugins/Virtual that it represents the original mailbox isn't adequate for that purpose. Hmm. If you're making a virtual mailbox out of another virtual mailbox, I'm not entirely sure what X-MAILBOX should do. Currently it returns the parent virtual mailbox's name. But should it instead return the mailbox name that physically contains the mails? I'm beginning to think that it probably should. Why? Because fetching from there will be faster? Having got the x-mailbox insight (I think---thanks), I have tried the following combinations: # ~/Maildir/virtual/INBOX/dovecot-virtual INBOX zz_archive* inthread refs x-mailbox inbox shows only messages in INBOX. THe archives are actually in folders like zz_archive.2010.01 Is this fast or slow then? Oh, perfectly fast, but also perfectly ineffective :-) # ~/Maildir/virtual/all/dovecot-virtual * all # ~/Maildir/virtual/INBOX/dovecot-virtual virtual.all inthread refs x-mailbox inbox Appears to hang It's probably going to take a while to build the thread index. After that it should be pretty fast. It would be easier to see how far it gets by talking imap: a login username password b select virtual.all c thread refs all d select virtual.INBOX a login username pw a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH] Logged in b select virtual.all closed (boom!) I guess that explains my problem... after a fashion. Ah, my error file contains lots of these: Jan 21 18:06:01 IMAP(dave): Error: mmap() failed with index cache file /home/dave/Maildir/.zz_archive.2009.06/dovecot.index.cache: Cannot allocate memory Jan 21 18:06:01 IMAP(dave): Fatal: pool_system_realloc(1048576): Out of memory Okay, I bumped up mail_process_size from 256 to 1024 and it seems not to be bailing out quite so soon. b select virtual.all * FLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-3 unknown-5 unknown-6 $Forwarded $NotJunk unknown-0 unknown-2 NonJunk NotJunk gnus-forward $Junk Junk gnus-download gnus-save unknown-1 $Label1 JunkRecorded $MDNSent unknown-4 gnus-expire unknown-8 receipt-handled) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft unknown-3 unknown-5 unknown-6 $Forwarded $NotJunk unknown-0 unknown-2 NonJunk NotJunk gnus-forward $Junk Junk gnus-download gnus-save unknown-1 $Label1
Re: [Dovecot] virtual mailbox / INTHREAD use case, issues, questions
At Wed, 6 Jan 2010 14:17:40 +0200,
Timo Sirainen wrote:
On 4.1.2010, at 20.47, David Abrahams wrote:
1. I had to manually create the virtual folder for all my users or
they couldn't access their mail at all. Is this fixed in 2.0?
No.
OK, is it considered a bug that should be addressed? IMO there should
at least be a way to tell dovecot to ignore the non-existence of a
particular folder.
2. My users were getting errors when accessing via POP3. The possible
meaning of such a virtual folder under POP3 is lost on me, but I
had to enable the virtual plugin for that protocol too. I think
that should at least be in the documentation, or better yet the
requirement should that it be enabled for POP3 should be dropped.
I think you could put the virtual namespace inside protocol imap {} section.
I'm pretty sure I tried that and it didn't work for some reason.
Unfortunately I don't remember the reason, so I could be mistaken.
3. I couldn't get the virtual conversation view inbox to show any
messages. But maybe that's because of a possible issue I describe
below with x-mailbox
5. The all virtual mailbox would only show 89 messages. I switched
it to be a mirror of my archive mailbox and the number started to
look more reasonable. This could be the lots of mailboxes bug
describe in
http://dovecot.org/list/dovecot-news/2009-December/000145.html
lots of mailboxes should cause error to be logged.
What does the error look like? I could search my logs for it.
7. I tried to create an IMAP search, rather than a virtual folder,
that looked for x-mailbox INBOX header like the virtual folder
does. It too came up empty. It doesn't exactly surprise me
because I don't see an x-mailbox header in any of these messages.
x-mailbox doesn't use a header, it uses the actual mailbox name
where the message exists.
Oh... what if the message exists in multiple mailboxes? Typically
anything in INBOX can also be found in my all mail archive.
I guess that means it's crucial that, whatever else I do, INBOX should
be part of the all mail virtual folder in order for this to work. I
didn't quite understand that before, and I think it's important to
have a description somewhere of how x-mailbox works that would help me
to get to that conclusion. Certainly the existing description at
http://wiki.dovecot.org/Plugins/Virtual that it represents the
original mailbox isn't adequate for that purpose.
Lastly, I want to report that for me, the speed of these IMAP searches
doesn't begin to approach what I can do with mairix if I want a
message's entire thread. I'm happy to use mairix, although it would
be nicer not to duplicate that capability if it's already in the
server. But it needs to be fast. Is there something I can do to make
it go faster?
How slow is it? How many messages do you have? I think it should be pretty
fast.
150523 messages in 2.6G
Anyway, dovecot -n output and contents of dovecot-virtual file could help
figuring out what your problem is.
Having got the x-mailbox insight (I think---thanks), I have tried
the following combinations:
# ~/Maildir/virtual/INBOX/dovecot-virtual
INBOX
zz_archive*
inthread refs x-mailbox inbox
shows only messages in INBOX. THe archives are actually in folders
like zz_archive.2010.01
---
# ~/Maildir/virtual/all/dovecot-virtual
INBOX
zz_archive*
all
# ~/Maildir/virtual/INBOX/dovecot-virtual
virtual.all
inthread refs x-mailbox inbox
shows only messages in INBOX
---
# ~/Maildir/virtual/all/dovecot-virtual
*
all
# ~/Maildir/virtual/INBOX/dovecot-virtual
virtual.all
inthread refs x-mailbox inbox
Appears to hang
---
# ~/Maildir/virtual/INBOX/dovecot-virtual
INBOX
zz_archive.*.*
inthread refs x-mailbox inbox
Appears to hang. For example, the little pie chart in Apple Mail that
shows mailbox scanning progress is stopped. Wanderlust gets totally
stuck.
---
# 1.2.8: /usr/local/etc/dovecot.conf
# OS: FreeBSD 6.2-RELEASE amd64
protocols: imap pop3 imaps pop3s
listen: 64.156.193.20
ssl_ca_file: /etc/ssl/private/network_solutions_ca.crt
ssl_cert_file: /etc/ssl/certs/www.boostpro.com.crt
ssl_key_file: /etc/ssl/private/www.boostpro.com.key
login_dir: /var/run/dovecot/login
login_executable(default): /usr/local/libexec/dovecot/imap-login
login_executable(imap): /usr/local/libexec/dovecot/imap-login
login_executable(pop3): /usr/local/libexec/dovecot/pop3-login
verbose_proctitle: yes
first_valid_gid: 0
mail_privileged_group: mail
mail_location: maildir:~/Maildir
mail_executable(default): /usr/local/libexec/dovecot/imap
mail_executable(imap): /usr/local/libexec/dovecot/imap
mail_executable(pop3): /usr/local/libexec/dovecot/pop3
mail_plugins(default): virtual autocreate
mail_plugins(imap): virtual autocreate
mail_plugins(pop3): virtual
mail_plugin_dir(default): /usr/local/lib/dovecot/imap
mail_plugin_dir(imap): /usr/local/lib/dovecot
Re: [Dovecot] virtual mailbox / INTHREAD use case, issues, questions
At Tue, 05 Jan 2010 06:50:23 -0500, Charles Marcus wrote: On 2010-01-04, David Abrahams (d...@boostpro.com) wrote: The problem is that I really need to practice Inbox Zero. I need to not see the things I've decided I'm through with (in every mail client I use), and having the client decide I'm through when I've read a message is highly prone to accidentally losing things I need to deal with. I think that would entail the use of the [Delete] key. Precisely! And then, if I need to get back to older messages in the thread, I need an INTHREAD search. -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
Re: [Dovecot] First time Dovecot user, really impressed so far. What is best IMAP enabled webmail package to go with Dovecot?
At Mon, 4 Jan 2010 22:08:22 +0100, Egbert Jan van den Bussche wrote: Welcome! Have a look at RoundCube webmail. I used to use squirrelmail but had the same issues as you. RoundCube is very nice. I second that emotion -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
[Dovecot] virtual mailbox / INTHREAD use case, issues, questions
For over a year now, I've been trying to acquire the ability to review the thread history of a message in my INBOX while still deleting messages as soon as I believe I'm done with them. Suddenly I'm gratified to see that several approaches have been right under my nose all along! One is the INTHREAD imap extension, which I see is implemented by dovecot. Another (related) is dovecot's virtual folders. When I saw the GMail-style conversation view described at http://wiki.dovecot.org/Plugins/Virtual I was elated, so I tried it out, with exactly that configuration (almost -- my separator is '.'). I ran into several problems (using Dovecot 1.2.8 with http://hg.dovecot.org/dovecot-1.2/rev/9cccb8617820 applied): 1. I had to manually create the virtual folder for all my users or they couldn't access their mail at all. Is this fixed in 2.0? 2. My users were getting errors when accessing via POP3. The possible meaning of such a virtual folder under POP3 is lost on me, but I had to enable the virtual plugin for that protocol too. I think that should at least be in the documentation, or better yet the requirement should that it be enabled for POP3 should be dropped. 3. I couldn't get the virtual conversation view inbox to show any messages. But maybe that's because of a possible issue I describe below with x-mailbox 4. At some point I ended up with an empty mailbox called private.virtual --- I have no idea how. Probably user error. 5. The all virtual mailbox would only show 89 messages. I switched it to be a mirror of my archive mailbox and the number started to look more reasonable. This could be the lots of mailboxes bug describe in http://dovecot.org/list/dovecot-news/2009-December/000145.html 6. I got the impression that looking at the virtual INBOX didn't cause the all mailbox to be updated, which I'd guess that it should be. 7. I tried to create an IMAP search, rather than a virtual folder, that looked for x-mailbox INBOX header like the virtual folder does. It too came up empty. It doesn't exactly surprise me because I don't see an x-mailbox header in any of these messages. In fact, I can't find one in /any/ of my messages. When I look for a description of the x-mailbox header, I don't find much information on where it comes from or what it represents. http://www.dovecot.org/list/dovecot/2009-June/040619.html seems to indicate it's synthesized by the server. If it's really supposed to represent the message's original mailbox as implied by http://wiki.dovecot.org/Plugins/Virtual then I'm not sure what that means for messages that are filed twice by sieve and never moved. Lastly, I want to report that for me, the speed of these IMAP searches doesn't begin to approach what I can do with mairix if I want a message's entire thread. I'm happy to use mairix, although it would be nicer not to duplicate that capability if it's already in the server. But it needs to be fast. Is there something I can do to make it go faster? Thanks in advance for any advice, hints, fixes, workarounds, etc. -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
[Dovecot] Moving to hardlinks?
I just discovered that dovecot supports copying messages via hard-linking. I have a huge mailbox hierarchy of existing messages that I'd prefer to store that way. Is there a simple way to transform non-hard-linked storage to hard-linked? TIA, -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
Re: [Dovecot] virtual mailbox / INTHREAD use case, issues, questions
At Mon, 04 Jan 2010 20:11:50 +, Ed W wrote: On 04/01/2010 18:47, David Abrahams wrote: For over a year now, I've been trying to acquire the ability to review the thread history of a message in my INBOX while still deleting messages as soon as I believe I'm done with them. Suddenly I'm gratified to see that several approaches have been right under my nose all along! I have recently just switched my Thunderbird to have sent items in the Inbox and in the same folder as the message you are replying to. The end result is a fairly workable equivalent of the gmail style thing, although you also see Sent messages which haven't had a reply (which may or may not be what you desire, but it's actually my preference for my needs) No need for virtual folders, but support for this to work well may not be in every mail client... Yeah, I used to do something like that with Gnus. Believe me, I've been through all the simple approaches. The problem is that I really need to practice Inbox Zero. I need to not see the things I've decided I'm through with (in every mail client I use), and having the client decide I'm through when I've read a message is highly prone to accidentally losing things I need to deal with. -- Dave Abrahams Meet me at BoostCon: http://www.boostcon.com BoostPro Computing http://www.boostpro.com
[Dovecot] Problems with ioloop on MacOS
I've been using the macports version of dovecot, which is 1.1.11, on MacOS 10.5.6 $ /opt/local/sbin/dovecot --exec-mail imap Fatal: kevent(EV_ADD, 0) failed: Operation not supported This version is configured --with-ioloop=kqueue The problem seems to be known, if not understood: http://lists.apple.com/archives/Darwin-userlevel/2008/Dec/msg00010.html However, if I use --with-ioloop=poll, I eventually get IMAP(dave): [9079]: invalid I/O fd 0, callback 0x9330 (not from the above command; it's a little later when Gnus tries to interact with dovecot through the shell) I've been unable to build/install any later versions of dovecot on MacOS. autogen.sh fails like this: ... glibtoolize: putting auxiliary files in `.'. glibtoolize: copying file `./ltmain.sh' glibtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.in and glibtoolize: rerunning glibtoolize, to keep the correct libtool macros in-tree. glibtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am. configure.in:281: installing `./config.guess' configure.in:282: required file `./config.rpath' not found configure.in:281: installing `./config.sub' configure.in:5: installing `./install-sh' configure.in:5: installing `./missing' src/auth/Makefile.am: installing `./depcomp' autoreconf: automake failed with exit status: 1 -- Dave Abrahams BoostPro Computing http://www.boostpro.com
