Re: [Dovecot] Thunderbird subscription bug ?
After a quick read, this sounds similar to thunderbird symptoms I've noticed with dovecot but I think it only started with *some* folder trees that were named after 2010 or 10 (abbreviation). It would either show the 2010 or 10 folder and possibly not let me subscribe to it, and if it had children it probably would not even show those for subscription unless I was already subscribed to it using a different client. I haven't put in time to track it down yet. On 04/02/10 12:06, Thomas Hummel wrote: Hello Timo, I'm having a hard time trying to understand why Thunderbird 3.0.3 doesn't allow me to subscribe to a shared mailbox which I can subscribe to via Kmail for instance. I'm running dovecot-1.2.11/Maildir. The user 'doveimap' shares his mailbox folder/subfolder to the 'dovepop' user which should see it in the '#shared' shared namespace at the following location : #shared/doveimap/folder/subfolder ACL (and unix permissions) are ok : # cat /courriel/boites/doveimap/.folder.subfolder/dovecot-acl user=dovepop akxeilprwts rawlogs give : a) kmail case : in : 4 NAMESPACE 5 LIST 6 LSUB % 7 LIST % 8 LSUB #shared/% 9 LIST #shared/% 10 LIST INBOX 11 SELECT INBOX 12 NOOP 13 UID FETCH 1:* (UID FLAGS) 14 UID FETCH 1:2 (UID RFC822.SIZE FLAGS ENVELOPE BODY.PEEK[HEADER.FIELDS (REFERENCES)]) 15 LIST * 16 LIST #shared/* 17 LSUB * 18 LSUB #shared/* 19 LIST #shared/doveimap/folder/subfolder 20 SUBSCRIBE #shared/doveimap/folder/subfolder out : * NAMESPACE (( /)) ((#shared/ /)) NIL 4 OK Namespace completed. * LIST (\Noselect) / 5 OK List completed. 6 OK Lsub completed. * LIST (\HasNoChildren) / Trash * LIST (\HasNoChildren) / INBOX * LIST (\Noselect \HasChildren) / #shared 7 OK List completed. 8 OK Lsub completed. * LIST (\Noselect \HasChildren) / #shared/doveimap 9 OK List completed. * LIST (\HasNoChildren) / INBOX 10 OK List completed. * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. * 2 EXISTS * 0 RECENT * OK [UIDVALIDITY 1270130617] UIDs valid * OK [UIDNEXT 3] Predicted next UID * OK [HIGHESTMODSEQ 1] Highest 11 OK [READ-WRITE] Select completed. 12 OK NOOP completed. * 1 FETCH (UID 1 FLAGS (\Seen)) * 2 FETCH (UID 2 FLAGS (\Seen)) 13 OK Fetch completed. * 1 FETCH (UID 1 RFC822.SIZE 1140 FLAGS (\Seen) ENVELOPE [...] * 2 FETCH (UID 2 RFC822.SIZE 1138 FLAGS (\Seen) ENVELOPE [...] 14 OK Fetch completed. * LIST (\HasNoChildren) / Trash * LIST (\HasNoChildren) / INBOX * LIST (\Noselect \HasChildren) / #shared/doveimap * LIST (\HasNoChildren) / #shared/doveimap/folder/subfolder 15 OK List completed. * LIST (\Noselect \HasChildren) / #shared/doveimap * LIST (\HasNoChildren) / #shared/doveimap/folder/subfolder 16 OK List completed. 17 OK Lsub completed. 18 OK Lsub completed. * LIST (\HasNoChildren) / #shared/doveimap/folder/subfolder 19 OK List completed. 20 OK Subscribe completed. b) Thunderbird case : in : 4 namespace 5 ENABLE CONDSTORE 6 lsub * 7 lsub #shared/* 8 list INBOX 9 list Trash 10 create Trash 11 select INBOX (CONDSTORE) 12 myrights INBOX 13 getacl INBOX 14 UID fetch 1:* (FLAGS) 15 UID fetch 1:2 (UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority X-Priority References Newsgroups In-Reply-To Content-Type)]) 16 UID fetch 1,2 (UID RFC822.SIZE BODY.PEEK[]) 17 IDLE DONE 18 lsub * 19 list % 20 list %/% 21 lsub #shared/* 22 list #shared/% 23 list #shared/%/% 24 IDLE DONE 25 list #shared/#shared/% 26 list #shared/#shared/%/% 27 IDLE DONE 28 noop 29 IDLE DONE out : * NAMESPACE (( /)) ((#shared/ /)) NIL 4 OK Namespace completed. * ENABLED CONDSTORE 5 OK Enabled. 6 OK Lsub completed. 7 OK Lsub completed. * LIST (\HasNoChildren) / INBOX 8 OK List completed. * LIST (\HasNoChildren) / Trash 9 OK List completed. 10 NO [ALREADYEXISTS] Mailbox exists. * FLAGS (\Answered \Flagged \Deleted \Seen \Draft) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted. * 2 EXISTS * 0 RECENT * OK [UIDVALIDITY 1270130617] UIDs valid * OK [UIDNEXT 3] Predicted next UID * OK [HIGHESTMODSEQ 1] Highest 11 OK [READ-WRITE] Select completed. * MYRIGHTS INBOX lrwstipekxacd 12 OK Myrights completed. * ACL INBOX dovepop lrwstipekxacd 13 OK Getacl completed. * 1 FETCH (UID 1 FLAGS (\Seen)) * 2 FETCH (UID 2 FLAGS (\Seen)) 14 OK Fetch completed. * 1 FETCH (UID 1 RFC822.SIZE 1140 FLAGS (\Seen) [...] * 2 FETCH (UID 2 RFC822.SIZE 1138 FLAGS (\Seen) [...] 15 OK Fetch completed. * 1 FETCH (UID 1 RFC822.SIZE 1140 BODY[] [...] * 2 FETCH (UID 2 RFC822.SIZE 1138 BODY[] [...] 16 OK Fetch completed. + idling 17 OK Idle completed. 18 OK Lsub completed. * LIST (\HasNoChildren) / Trash * LIST (\HasNoChildren) / INBOX * LIST (\Noselect \HasChildren) / #shared 19 OK List completed. * LIST (\Noselect \HasChildren) / #shared/doveimap 20 OK List completed. 21 OK Lsub completed. * LIST (\Noselect \HasChildren) / #shared/doveimap 22 OK List completed. * LIST
Re: [Dovecot] NFS issues
On 03/25/10 06:58, Brian Candler wrote: On Tue, Mar 23, 2010 at 03:19:49PM +0200, Timo Sirainen wrote: I have done some small-scale testing and it looks fine. Stress testing by running imaptest for same user's same mailbox in 2+ different servers (i.e. two NFS clients reading/writing same mailbox files) should show up quickly what kind of errors you could get. http://imapwiki.org/ImapTest OK, I've now set this up: ImapTest --- dovecot (same host) - NFS server `--- dovecot (diff host) ' * 172.16.23.104: dovecot 1.2.11 and ImapTest-latest. FreeBSD 7.2. * 172.16.23.101: dovecot 1.2.11 only. FreeBSD 7.2. * 172.16.23.103: NFS server. Ubuntu Karmic. All three hosts are ntpd synced. The following was needed on the FreeBSD boxes to get fcntl locking working: nfs_client_enable=YES rpc_lockd_enable=YES rpc_statd_enable=YES (imapd worked without these, but maillog showed errors about failing to obtain locks, operation not supported) Test results * Pointing a single instance of imaptest at a single host, or two instances of imaptest at the same host (with clients=5 to avoid hitting the 15 client limit) was fine. ImapTest reported no errors, and nothing out of the ordinary in maillog. $ egrep -v Login:|Disconnected:|Aborted login /var/log/maillog * Things went badly wrong with two instances of imaptest pointing at different dovecot hosts. I had seen this sort of thing when I'd previously been using dot locking, and was hoping they'd be fixed by switching to fcntl, but unfortunately not. ImapTest reported errors including: Error: br...@dev.example.com[8]: SELECT failed: 8.3 NO [SERVERBUG] Internal error occurred. Refer to server log for more information. [2010-03-25 10:22:23] - 6 stalled for 16 secs in command: 11 EXPUNGE All sorts of errors reported in maillog, including: Mar 25 10:22:23 freebsd-dev dovecot: IMAP(br...@dev.example.com): fscking index file /mail/0/6/37/30/brian%dev.example.com/dovecot.index Mar 25 10:22:23 freebsd-dev dovecot: IMAP(br...@dev.example.com): Transaction log /mail/0/6/37/30/brian%dev.example.com/dovecot.index.log: duplicate transaction log sequence (10) Mar 25 10:22:23 freebsd-dev dovecot: IMAP(br...@dev.example.com): Our dotlock file /mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.lock was overridden (locked 0 secs ago, touched 0 secs ago) Mar 25 10:22:23 freebsd-dev dovecot: IMAP(br...@dev.example.com): fscking index file /mail/0/6/37/30/brian%dev.example.com/dovecot.index Mar 25 10:22:23 freebsd-dev dovecot: IMAP(br...@dev.example.com): Transaction log /mail/0/6/37/30/brian%dev.example.com/dovecot.index.log: duplicate transaction log sequence (11) Mar 25 10:22:27 freebsd-dev dovecot: IMAP(br...@dev.example.com): /mail/0/6/37/30/brian%dev.example.com/dovecot.index reset, view is now inconsistent Mar 25 10:22:46 freebsd-dev dovecot: IMAP(br...@dev.example.com): Panic: file mail-transaction-log-view.c: line 108 (mail_transaction_log_view_set): assertion failed: (min_file_seq= max_file_seq) Mar 25 10:22:48 freebsd-dev dovecot: IMAP(br...@dev.example.com): rename(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.tmp, /mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist) failed: No such file or directory Mar 25 10:22:48 freebsd-dev dovecot: IMAP(br...@dev.example.com): unlink(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.tmp) failed: No such file or directory Mar 25 10:22:36 wipe-dev dovecot: IMAP(br...@dev.example.com): ftruncate(/mail/0/6/37/30/brian%dev.example.com/dovecot-uidlist.lock) failed: Stale NFS file handle (Logs from a single test run are attached) Interestingly, these messages imply that dovecot is still using dotlocking in some circumstances, even though I've definitely set fcntl locking. $ grep ^lock /usr/local/etc/dovecot.conf lock_method = fcntl $ egrep '^mail_nfs|^mmap' /usr/local/etc/dovecot.conf mmap_disable = yes mail_nfs_storage = yes mail_nfs_index = yes All this suggests I should use some sort of 'sticky' load balancing in front so that all client conns from one IP hit the same frontend box. However, that contradicts the experience Adam McDougall has had with a similar setup: http://dovecot.org/list/dovecot/2010-March/047815.html It's possible that switching the Linux NFS server to a Netapp will help (which is what it will be deployed onto eventually anyway) Adam: did you do any tuning of FreeBSD client NFS settings? And have you tried using ImapTest, or just real IMAP users? I see there are a few tunables: $ grep nfs /etc/defaults/rc.conf netfs_types=nfs:NFS nfs4:NFS4 smbfs:SMB portalfs:PORTAL nwfs:NWFS # Net filesystems. nfs_client_enable=NO# This host is an NFS client (or NO). nfs_access_cache=60 # Client cache timeout in seconds nfs_server_enable=NO# This host is an NFS server (or NO). nfs_server_flags=-u -t -n 4 # Flags to nfsd (if enabled). nfs_reserved_port_only=NO # Provide NFS only on secure port
Re: [Dovecot] NFS issues [was: Dovecot-1.2 + Sieve + Managesieve on Debian]
On 03/23/10 07:43, Brian Candler wrote: On Tue, Mar 23, 2010 at 05:42:02AM -0400, Charles Marcus wrote: On 2010-03-22 9:31 PM, Stan Hoeppner wrote: Dovecot has built in locking support for NFS storage. But it has always been problematic, according to Timo. Have you got any references on this, apart from http://wiki.dovecot.org/NFS ? I'm looking at migrating a courier-imap installation (FreeBSD frontends, Netapp backends, Maildir++) to dovecot. I'd be grateful of any known pitfalls I should be looking out for. I have done some small-scale testing and it looks fine. tcpdumping the NFS traffic, I see that the FreeBSD frontend is sending access requests to check that its local cache is not stale. In tests with mailboxes containing 100 messages and a web IMAP frontend (atmail.org), Dovecot was generating about 1/4 of the total NFS traffic compared to courier-imap, because of how Dovecot creates a cache file containing the message headers. I see a link to http://www.freebsd.org/cgi/query-pr.cgi?pr=123755 in the NFS wiki page. Does this problem affect only mbox over NFS, or maildir too? I've not observed any problem with courier-imap, although courier-imap is much dumber about caching, and also at the moment the majority of the userbase are on POP3 anyway. Regards, Brian. I've used: mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes on FreeBSD 6/7/8 and dovecot with multiple servers accessing the same mailboxes over NFS on a NetApp and it has worked fine since 1.0.16 or so, I think. I use 1.1 now (haven't finished testing 1.2 and haven't done much with 8.x). I have a load balancer in front and it sends IMAPS connections or HTTPS connections (making local IMAP calls) to randomized servers so even one person making 5 connections gets a speed benefit by using more than one server. If the NFS support had problems you could also use mail_location to keep indexes local.
Re: [Dovecot] New need to delete dovecot-acl-list in dovecot 1.2?
Timo Sirainen wrote: On Thu, 2009-08-20 at 23:40 -0400, Adam McDougall wrote: List cache dovecot-acl-list file lists all mailboxes that have l rights assigned. If you manually add/edit dovecot-acl files, you may need to delete the dovecot-acl-list to get the mailboxes visible. It's mainly useful for mailbox sharing between users. http://hg.dovecot.org/dovecot-1.2/rev/aeb3affa0501 makes Dovecot ignore it in public namespaces. Ah thanks, I will try it when I have time. BTW, yesterday I noticed my dovecot 1.1.11 servers acting like the dovecot-acl is recursive in my public folders, much like 1.2 seems to be, but I cannot recall 1.1.x ever acting that way. Am I crazy or did it change sometime before 1.2? Thanks.
Re: [Dovecot] Quick and dirty server optimized for IMAP upload speed?
Adam McDougall wrote: Timo Sirainen wrote: On Aug 28, 2009, at 8:38 PM, Adam McDougall wrote: Early next week I need to upload over 100,000 emails to an IMAP server as quickly as possible from an Outlook client. I am looking for any methods I can use to (temporarily?) speed up the rate at which dovecot can accept and store IMAP uploads, whether it be storing on local disk, ram disk, etc. I can setup a temporary server on a laptop for example and once the upload has finished I can use standard file copying methods to transfer the mail to stable, permanent storage. I haven't been able to see over about 7 msgs/sec upload speed from a local folder in any mail client to dovecot (only NFS or ZFS backend tested so far with Maildir). Is there something horribly wrong with the speed I am seeing or are there just tricks I can try? Any tips? I'll be working on it all weekend until I find something satisfactory. It seems like I can upload mails to an Exchange server quicker. I'll setup just about anything that my experience allows me to, I can be very resourceful with adhoc hardware and software. From Dovecot's side the only thing you can do is fsync_disable=yes. The main problem is probably network latency, because Outlook doesn't support MULTIAPPEND extension (and perhaps not even LITERAL+ extension?) Did you already try running Dovecot on the same computer as Outlook (some virtual thingy or maybe it works in cygwin)? I just tried fsync_disable=yes but with NFS and had to turn off mail_nfs_index = yes as well but the speed was the same. Do you think it would be different with a UFS or ZFS backend with fsync_disable? I have not tried running dovecot on the same computer. When you mention dovecot+cygwin I think of the reported issues in the past on the mailing list and don't know if they were resolved. I could try dovecot in virtualbox I suppose (I put it on my list to try). I did a lot of testing today and found some things. The two biggest real bottlenecks: - Thunderbird is just slow at uploading to IMAP. With a bunch of small msgs it only does a few per second and you can tell the server is waiting for something to do. Outlook is considerably faster. Other clients not tested. - Perdition (IMAP proxy), at least in my current setup, slows down the mail upload speed around 50%. Non-bottlenecks: - fsync (I can't measure the difference at the client, but on the server I can see the behavior change) - filesystem (nfs/ufs/zfs all performed about the same) - server cpu - imap server being over the local network as opposed to running inside virtualbox on the same pc I think I am satisfied with the speeds I am seeing now for the needs I have next week. Depending on the resulting speed across campus, I may run dovecot on a portable laptop for the upload; I'll just go around the perdition proxy (plan to retire that in a few weeks). During my testing I did notice an issue with Outlook 2003 on dovecot 1.2 that I don't have with 1.1, I cannot delete an IMAP folder (maybe after clicking on it first). I get an error about 'folder is open in another session'. It happens on a Maildir store on a local filesystem or NFS and I only have one client accessing it. I might have time to look into it properly tomorrow, but if not, probably not for a few days at least. Unrelated: Outlook 2003 running on Windows 7 seems to abort the upload after just a few hundred messages with an error message. Works on XP. Alternatively I'll take a fast way of converting Exchange email to a tree of local mbox files which I can then run mb2md on. If the mails are in Exchange, can't you connect to it using IMAP? In theory yes, but I don't have access to the actual Exchange server until Monday at the earliest, and the user is using cached exchange mode which in past experience leaves the possibility of local mail which is not actually on the server due to a desync. Unless I am sure it is perfectly in sync, I've seen a second Outlook connect to Exchange using the native protocols and it initiated a massive deletion of mail which we had to toil to recover from obscure cache files on the original client. I don't know if an IMAP connection might trigger the same issue. For performance testing's sake, I'll see if I can upload some mail to our own Exchange server and see how fast an mbox capable mail client can download it. I can do some limited testing in the real environment on Monday but I'm expected to do the real migration on Tuesday unless I have to cancel. Thanks for the ideas.
Re: [Dovecot] Quick and dirty server optimized for IMAP upload speed?
On Sun, Aug 30, 2009 at 12:33:43PM -0400, Charles Marcus wrote: On 8/30/2009, Adam McDougall (mcdou...@egr.msu.edu) wrote: The two biggest real bottlenecks: - Thunderbird is just slow at uploading to IMAP. With a bunch of small msgs it only does a few per second and you can tell the server is waiting for something to do. Did you try TBird 3.0b3? It has many, many IMAP improvements... -- Best regards, Charles Wow it sure is faster at uploading, thanks for mentioning it!
[Dovecot] Quick and dirty server optimized for IMAP upload speed?
Early next week I need to upload over 100,000 emails to an IMAP server as quickly as possible from an Outlook client. I am looking for any methods I can use to (temporarily?) speed up the rate at which dovecot can accept and store IMAP uploads, whether it be storing on local disk, ram disk, etc. I can setup a temporary server on a laptop for example and once the upload has finished I can use standard file copying methods to transfer the mail to stable, permanent storage. I haven't been able to see over about 7 msgs/sec upload speed from a local folder in any mail client to dovecot (only NFS or ZFS backend tested so far with Maildir). Is there something horribly wrong with the speed I am seeing or are there just tricks I can try? Any tips? I'll be working on it all weekend until I find something satisfactory. It seems like I can upload mails to an Exchange server quicker. I'll setup just about anything that my experience allows me to, I can be very resourceful with adhoc hardware and software. Alternatively I'll take a fast way of converting Exchange email to a tree of local mbox files which I can then run mb2md on. I tried using Thunderbird to Import the mails from Outlook and while it was fast, it messed up the formatting of some of the mails so I don't think I can use that. I tried readpst briefly from libpst but it took a long time to run, took alot of cpu, and was spewing lots of errors so I canceled it. Thanks for any input!
Re: [Dovecot] Quick and dirty server optimized for IMAP upload speed?
Timo Sirainen wrote: On Aug 28, 2009, at 8:38 PM, Adam McDougall wrote: Early next week I need to upload over 100,000 emails to an IMAP server as quickly as possible from an Outlook client. I am looking for any methods I can use to (temporarily?) speed up the rate at which dovecot can accept and store IMAP uploads, whether it be storing on local disk, ram disk, etc. I can setup a temporary server on a laptop for example and once the upload has finished I can use standard file copying methods to transfer the mail to stable, permanent storage. I haven't been able to see over about 7 msgs/sec upload speed from a local folder in any mail client to dovecot (only NFS or ZFS backend tested so far with Maildir). Is there something horribly wrong with the speed I am seeing or are there just tricks I can try? Any tips? I'll be working on it all weekend until I find something satisfactory. It seems like I can upload mails to an Exchange server quicker. I'll setup just about anything that my experience allows me to, I can be very resourceful with adhoc hardware and software. From Dovecot's side the only thing you can do is fsync_disable=yes. The main problem is probably network latency, because Outlook doesn't support MULTIAPPEND extension (and perhaps not even LITERAL+ extension?) Did you already try running Dovecot on the same computer as Outlook (some virtual thingy or maybe it works in cygwin)? I just tried fsync_disable=yes but with NFS and had to turn off mail_nfs_index = yes as well but the speed was the same. Do you think it would be different with a UFS or ZFS backend with fsync_disable? I have not tried running dovecot on the same computer. When you mention dovecot+cygwin I think of the reported issues in the past on the mailing list and don't know if they were resolved. I could try dovecot in virtualbox I suppose (I put it on my list to try). Alternatively I'll take a fast way of converting Exchange email to a tree of local mbox files which I can then run mb2md on. If the mails are in Exchange, can't you connect to it using IMAP? In theory yes, but I don't have access to the actual Exchange server until Monday at the earliest, and the user is using cached exchange mode which in past experience leaves the possibility of local mail which is not actually on the server due to a desync. Unless I am sure it is perfectly in sync, I've seen a second Outlook connect to Exchange using the native protocols and it initiated a massive deletion of mail which we had to toil to recover from obscure cache files on the original client. I don't know if an IMAP connection might trigger the same issue. For performance testing's sake, I'll see if I can upload some mail to our own Exchange server and see how fast an mbox capable mail client can download it. I can do some limited testing in the real environment on Monday but I'm expected to do the real migration on Tuesday unless I have to cancel. Thanks for the ideas.
Re: [Dovecot] Per user namespace
Timo Sirainen wrote: On Mon, 2009-08-24 at 11:13 -0600, Ashley M. Kirchner wrote: I'm in the process of upgrading an old server which is running uw-imap to a new one running Dovecot. With the old machine, unfortunately, my users had the opportunity to store their e-mails in various locations. For example, some have their mail boxes stored in ~/mail/[various mbox files], others ALSO have a ~/mail/clients/[various mbox files] and yet others simply stored them in their ~/ path (thanks to the old IMAP). Is there a way to create per-user namespaces? I don't want to create a mail/, mail/clients/, etc., etc. global name space that will end up showing up on everyone's mail client, nor do I want to individually change each user's setup (and rewrite their .subscriptions file). I think the best solution would be to just finally standardize everyone's mailboxes under ~/mail/. It'll probably help you in future.. But yeah, it's possible to create per-user namespaces by returning namespace_* extra fields from your userdb. There isn't existing documentation how exactly to do that, but basically you'll just have to return the same namespace_* fields that exist in NAMESPACE_* environment variables. You can get a dump of those using post-login script, see http://wiki.dovecot.org/PostLoginScripting. Alternatively you could just set up those NAMESPACE_* settings directly in the post-login script by e.g. reading some file from home dir. But don't let users specify anything, the process is still running as root at that point and they'll get root privileges by changing just a few environment settings.. http://wiki.dovecot.org/Plugins/Virtual also gives a simple example how to return a different inbox=yes namespace for different users. It sounds like a similar issue I had to deal with, I ended up making several global name-spaces equivalent to ~, ~/mail, and ~/Mail but I made the last two hidden so legacy client setups will still work but new clients would not automatically find them. The only drawback I found was client apparently could not see my public folder namespaces unless they used the ~ namespace (blank prefix in the client). Since so few of my users need public folders I was fine with that.
[Dovecot] New need to delete dovecot-acl-list in dovecot 1.2?
I just started testing 1.2.3 then 1.2.4 this week, I was too busy earlier in the year to look at 1.2 at all. So far so good except I noticed newly granted permissions via ACL on some newly created public mailboxes were not visible by the client. It seemed to be a version thing because the same files work fine concurrently on a server running 1.1.x. I came across this in the ACLwiki which seems to fix it: List cache dovecot-acl-list file lists all mailboxes that have l rights assigned. If you manually add/edit dovecot-acl files, you may need to delete the dovecot-acl-list to get the mailboxes visible. I don't ever recall needing to do this before, and it sounds similar to this which I have had set to 10 for a long time: # cache_secs parameter # specifies how many seconds to wait between stat()ing dovecot-acl file # to see if it changed. All of my dovecot-acl files are symlinks so hopefully it runs stat() on the target, but just in case I tried recreating the symlink afterwards but it didn't help and its over NFS so who knows if the running dovecot 'saw' it. Do I need to debug this behavior further or will I just have to live with remembering to delete the dovecot-acl-list file when I manually change dovecot-acl contents? Thanks.
Re: [Dovecot] All mail in mbox disappears when using Outlook
Frank Leonhardt wrote: Timo Sirainen wrote on 03 August 2009 20:11: On Mon, 2009-08-03 at 20:03 +0100, Frank Leonhardt wrote: Any suggestions about where I could look for a clue will be followed up immediately. I'm new to Dovecot, but not BSD. My next step will be to put a network analyser on it. Looking at the imap traffic could help, especially if you can reproduce works and doesn't work cases. You could use also http://wiki.dovecot.org/Debugging/Rawlog I saw that a few weeks back and thought I'd give it a go - the analyser looks less scarey ;-) I've had some progress. The problem was the same on 1.1.16. However, if I rename the mbox file to something completely different (rather than adding a suffix as I had done before) it all suddenly starts to work. The old file was called inbox090630. It doesn't like inbox090630-1 but it's okay about Inbox-09-06-30. Moving the file name back makes everything disappear again. Unsubscribing and subscribing alone makes no difference. It's obviously looking like an internal Outlook problem. I'll check it out with other IMAP servers and see if I can confirm it. However, I'll let Microsoft fix it themselves. Thanks for all your help! This sounds like an issue I've noticed with Outlook myself, in that whenever it sees a mail folder starting with the word inbox (whether it is inboX or iNbOX or inbox-dfsjaf), Outlook always looks for Inbox or Inbox-dfsjaf (it uppercases the I and lowercases the nbox before passing the request to the backend). Thus when I was converting my users from UW imap mboxes to Dovecot IMAP, I took care in my conversion script to rename folders on their behalf if they start with inbox so they are Inbox instead. Also, if you have a mailbox called Inbox, Thunderbird will not see it since it is hidden by the real inbox. Workarounds in perl: # If a folder is named Inbox, thunderbird will not see it. Rename. $target =~ s/^inbox$/Inbox-renamed-by-migrate/ig; # If a folder starts with inbox (any case, eg. INbOX) Outlook won't see it properly. # Outlook expects Inbox(something) or bust. $target =~ s/^inbox(.*)$/Inbox$1/ig;
Re: [Dovecot] GUI/WUI for creating (common) sieve scripts?
Roderick A. Anderson wrote: Seth Mattinen wrote: Peter Lindgren wrote: Roderick A. Anderson skrev: Anyone aware of a Web User Interface or GUI to allow users to create simple/common-type sieve scripts? So before I go invent this wheel I'd like to know what others are doing? For Thunderbird, there's a plugin (I haven't tested): http://sieve.mozdev.org/ It doesn't do hand-holding, it's just an interface to managesieve. I was thinking so. I did spot this while searching but it didn't look too promising. Besides I need/want something that can be used from a Webmail interface. Rod http://email.uoa.gr/avelsieve/* Avelsieve* or /Sieve Mail Filters Plugin for Squirrelmail/ is a Squirrelmail http://www.squirrelmail.org plugin http://www.squirrelmail.org/plugins.php for creating Sieve http://sieve.info scripts on a Sieve-compliant mail server.
Re: [Dovecot] v1.1.6 released
Just wanted to mention that 1.1.6 seems fine so far in our testing, and I think the lack of reported problems on the mailing list is probably a very good sign! Timo Sirainen wrote: http://dovecot.org/releases/1.1/dovecot-1.1.6.tar.gz http://dovecot.org/releases/1.1/dovecot-1.1.6.tar.gz.sig The invalid message address parsing bug is pretty important since it allows a remote user to send broken mail headers and prevent the recipient from accessing the mailbox afterwards, because the process will always just crash trying to parse the header. This is assuming that the IMAP client uses FETCH ENVELOPE command, not all do. Note that it doesn't affect versions older than v1.1.4. + dovecot -n and -a now prints some system information at the top. + More error/debug message logging improvements. - pop3-login: Fixed assert-crash if a client sent USER+PASS+USER+PASS commands in the same IP packet. - Parsing an invalid message address like From: ( caused an assert-crash in v1.1.4 and v1.1.5. - Folding whitespace wasn't handled correctly inside quoted-strings, causing some messages to be parsed incorrectly. - mbox: Fixed saving messages that begin with a valid From_-line.
Re: [Dovecot] Best configuration of dovecot for limit Outlook problems with IMAP
Recently I encountered a situation where Outlook Express was giving me a similar error, although it was only on certain emails. It started when I tried to tell Outlook Express to synchronize less than 'All messages' and I must have used 'New headers' or something instead. Switching it back to 'All messages' made it continue to download the entire body + headers of new mail, but they were accessible. For the messages that were inaccessible, to regain access I had to select them, right click the selection, click Move to Folder, and select the Inbox under the same mail server, then Click Send/Recv and the fixed messages would appear. Alessio Cecchi wrote: Hi, in some installation where users using IMAP and Outlook Express I have some problems. Outolook randomly presents problems in messages retrieving, for example: Outlook Express is unable to retrieve the requested message because the server no longer has the message available server response: Fecth Completed or Message could not be displayed error message when I try to view an Outlook Express e-mail message I'm using dovecot 1.1.4 with qmail and vpopmail. This is my dovecot configure options: ./configure --prefix=/usr --sysconfdir=/etc/dovecot --localstatedir=/var --with-ssldir=/etc/ssl and this my dovecot config: # dovecot -n # 1.1.4: /etc/dovecot/dovecot.conf log_path: /var/log/dovecot/dovecot-err.log info_log_path: /var/log/dovecot/dovecot.log ssl_cert_file: /etc/apache2/ssl/comun.crt ssl_key_file: /etc/apache2/ssl/comun.key disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/libexec/dovecot/imap-login login_greeting: Ready login_process_per_connection: no first_valid_uid: 89 last_valid_uid: 89 first_valid_gid: 89 last_valid_gid: 89 mail_uid: 89 mail_gid: 89 mail_location: maildir:~/Maildir mail_debug: yes mail_plugins: quota imap_quota imap_client_workarounds: outlook-idle delay-newmail namespace: type: private separator: . prefix: INBOX. inbox: yes list: yes subscriptions: yes auth default: verbose: yes debug: yes passdb: driver: checkpassword args: /home/vpopmail/bin/vchkpw userdb: driver: prefetch args: uid=89 gid=89 home=/home/vpopmail/domains/%d/%u plugin: quota: maildir What further changes to the configuration of dovecot could I do? Do you have other suggestions? Thanks
Re: [Dovecot] squat fts index build problem
Possibly try increasing mail_process_size. It defaults to 256 and I had to raise it to 1024 (1024 megs) to allow some squat searches to complete. I also have 4G of ram. You can also try watching the process in top to see how large it grows before it dies as a clue to what limit it is hitting. Anton Yuzhaninov wrote: I tried to use squat index, but got error: Sep 18 19:49:45 mailsupport dovecot: Fatal: IMAP(citrin): pool_system_realloc(): Out of memory Sep 18 19:49:45 mailsupport dovecot: child 77606 (imap) returned error 83 (Out of memory) It is strange because this server has 4 Gb or RAM. some lines from dovecot -n mail_plugins(imap): acl mail_log zlib fts fts_squat plugin: acl: vfile fts: squat OS: FreeBSD 7.0-STABLE #0: Sun May 4 13:15:10 MSD 2008, amd64 Resource limits (current): cputime infinity secs filesize infinity kB datasize 33554432 kB stacksize 524288 kB coredumpsize infinity kB memoryuseinfinity kB memorylocked infinity kB maxprocesses 5547 openfiles 11095 sbsize infinity bytes vmemoryuse infinity kB IMHO it will be better to write in log requested memory size: src/lib/mempool-system.c: - pool_system_realloc(): Out of memory); + pool_system_realloc(): Out of memory, when realloc to %PRIuSIZE_T bytes, new_size);
[Dovecot] pam_start() failed: system error with dovecot 1.1.2, cause unknown
I would guess this is unlikely to be dovecot's fault, but I'm wondering if anyone has any ideas of what might have happened based on the evidence. My best guess is some kind of resource limit was reached but I don't see any evidence in the logs, and the condition is now gone. Suddenly this morning, one (and only one) of my dovecot servers decided to start failing all logins since 08:25:04 until we restarted dovecot, at which point they were working fine. The number of imap-login processes was under the limit, but there were some obvious PAM errors at the time. My account could still ssh to the system so I don't think it was a problem general authentication, and NIS on other systems was working fine. No one was logged into that server at the time the problems occurred, and I don't think anything happened to the actual pam libraries to make them missing since dovecot worked after a restart. I should have used other means to prevent people from using that dovecot instance rather than stopping it, and I'll do so if it happens again in hopes of further debugging. /var/log/maillog: Sep 2 08:25:01 hill dovecot: imap-login: Login: user=userA, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Sep 2 08:25:01 hill dovecot: IMAP(userA): Disconnected: Logged out bytes=127/568 Sep 2 08:25:01 hill dovecot: imap-login: Login: user=userA, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Sep 2 08:25:01 hill dovecot: IMAP(userA): Disconnected: Logged out bytes=282/9641 Sep 2 08:25:04 hill dovecot: imap-login: Login: user=userA, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Sep 2 08:25:04 hill dovecot: IMAP(userA): Disconnected: Logged out bytes=46/543 ***problem started here Sep 2 08:25:04 hill dovecot: auth-worker(default): pam(userA,127.0.0.1): pam_start() failed: system error Sep 2 08:25:04 hill dovecot: auth-worker(default): pam(userB,35.9.37.164): pam_start() failed: system error Sep 2 08:25:05 hill dovecot: auth-worker(default): pam(userC,35.9.37.164): pam_start() failed: system error Sep 2 08:25:06 hill dovecot: imap-login: Aborted login (auth failed, 1 attempts): user=userB, method=PLAIN, rip=35.9.37.164, lip=35.9.37.190, TLS Sep 2 08:25:06 hill dovecot: imap-login: Aborted login (auth failed, 1 attempts): user=userA, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Sep 2 08:25:07 hill dovecot: imap-login: Aborted login (auth failed, 1 attempts): user=userC, method=PLAIN, rip=35.9.37.164, lip=35.9.37.190, TLS . /var/log/messages: Sep 2 08:25:04 hill dovecot-auth: in openpam_load_module(): no pam_permit.so found Sep 2 08:25:04 hill dovecot-auth: in openpam_load_module(): no pam_login_access.so found Sep 2 08:25:05 hill dovecot-auth: in openpam_load_module(): no pam_nologin.so found Sep 2 08:25:10 hill dovecot-auth: in openpam_load_module(): no pam_unix.so found Sep 2 08:25:11 hill dovecot-auth: in openpam_load_module(): no pam_unix.so found Sep 2 08:25:20 hill dovecot-auth: in openpam_load_module(): no pam_opieaccess.so found Sep 2 08:25:20 hill dovecot-auth: in openpam_load_module(): no pam_opie.so found Sep 2 08:25:51 hill kernel: Sep 2 08:25:51 hill last message repeated 12 times Sep 2 08:27:52 hill kernel: Sep 2 08:27:52 hill last message repeated 37 times Sep 2 08:38:01 hill kernel: Sep 2 08:38:01 hill last message repeated 144 times Sep 2 08:48:06 hill kernel: Sep 2 08:48:06 hill last message repeated 129 times Sep 2 08:53:36 hill kernel: Sep 2 08:52:51 hill last message repeated 51 times
[Dovecot] Anyone using LG Voyager (vx10000) with dovecot IMAP?
Anyone using the mobile IMAP option on the LG Voyager (mobile phone) with dovecot? I think it costs $5 extra per month. Seems low enough just to try it and find out. Thanks
Re: [Dovecot] Can't purge folders in Trash with listescape loaded in 1.1.2
Timo Sirainen wrote: http://dovecot.org/patches/1.1/listescape-plugin.c should help. I also committed a couple of fixes to hg to fix the error message. Thanks, the new listescape seems to work.
[Dovecot] Can't purge folders in Trash with listescape loaded in 1.1.2
I made sure I didn't have the Trash folder open by restarting dovecot, then going to File - Empty Trash in thunderbird. rawlog in, listescape loaded: 3 select Trash 4 list Trash/* 5 delete Trash/test 6 IDLE rawlog out, listescape loaded: * OK [RAWLOG TIMESTAMP] 2008-08-08 10:54:14 * FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old NonJunk $MDNSent unknown-3 $Forwarded unknown-0 unknown-5 unknown-7 Junk unknown-1 unknown-4 unknown-9 unknown-10 $label1 $label4 $label5) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old NonJunk $MDNSent unknown-3 $Forwarded unknown-0 unknown-5 unknown-7 Junk unknown-1 unknown-4 unknown-9 unknown-10 $label1 $label4 $label5 \*)] Flags permitted. * 0 EXISTS * 0 RECENT * OK [UIDVALIDITY 1194838495] UIDs valid * OK [UIDNEXT 55158] Predicted next UID 3 OK [READ-WRITE] Select completed. * LIST (\HasNoChildren) / Trash/test 4 OK List completed. 5 NO Unknown internal list error + idling rawlog in, listescape loaded: 3 select Trash 4 list Trash/* 5 delete Trash/test 6 unsubscribe Trash/test 7 IDLE rawlog in, listescape loaded: * OK [RAWLOG TIMESTAMP] 2008-08-08 10:56:34 * FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old NonJunk $MDNSent unknown-3 $Forwarded unknown-0 unknown-5 unknown-7 Junk unknown-1 unknown-4 unknown-9 unknown-10 $label1 $label4 $label5) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old NonJunk $MDNSent unknown-3 $Forwarded unknown-0 unknown-5 unknown-7 Junk unknown-1 unknown-4 unknown-9 unknown-10 $label1 $label4 $label5 \*)] Flags permitted. * 0 EXISTS * 0 RECENT * OK [UIDVALIDITY 1194838495] UIDs valid * OK [UIDNEXT 55158] Predicted next UID 3 OK [READ-WRITE] Select completed. * LIST (\HasNoChildren) / Trash/test 4 OK List completed. 5 OK Delete completed. 6 OK Unsubscribe completed. + idling dovecot -n: # 1.1.2: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache22/ssl/mail.pem ssl_key_file: /usr/local/etc/apache22/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_max_processes_count: 512 max_mail_processes: 1024 mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_access_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_process_size: 1024 mail_plugins: acl fts fts_squat listescape quota imap_quota mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: quota: fs:noenforcing:inode_per_mail acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat
Re: [Dovecot] 1.1.1 (maildir_uidlist_sync_update): assertion failed: (uidlist-initial_hdr_read)
Timo Sirainen wrote: On Wed, 2008-07-16 at 16:43 -0400, Adam McDougall wrote: Jun 26 17:23:26 hill dovecot: Panic: IMAP(userA): file maildir-uidlist.c: line 1207 (maildir_uidlist_sync_update): assertion failed: (uidlist-initial_hdr_read) Perhaps this fixes it? http://hg.dovecot.org/dovecot-1.1/rev/1edff123dd8f I am now running 1.1.2 in production, but I got a similar backtrace but different log message. The file mentioned has a later timestamp than the core, so it may have been modified since the Panic. -rw--- 1 userC 3001 980088 Aug 4 19:30 /home/userC/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.search.uids Aug 4 11:15:20 boomhauer dovecot: imap-login: Login: user=userC, method=PLAIN, rip=35.9.37.164, lip=35.9.37.190, TLS Aug 4 11:15:20 boomhauer dovecot: IMAP(userC): fchown(/home/userC/Maildir/temp.boomhauer.34901.6594f6f24f978311, -1, 3000) failed: Operation not permitted Aug 4 11:15:20 boomhauer dovecot: IMAP(userC): dovecot-acl-list creation failed: safe_mkstemp(/home/userC/Maildir/temp.boomhauer.34901.6594f6f24f978311) failed: Operation not permitted Aug 4 11:15:21 boomhauer dovecot: IMAP(userC): Corrupted squat uidlist file /home/userC/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.search.uids: missing/broken uidlist Aug 4 11:15:22 boomhauer dovecot: Panic: IMAP(userC): file squat-uidlist.c: line 153 (uidlist_write_array): assertion failed: ((uid ~UID_LIST_MASK_RANGE) = base_uid) Aug 4 11:15:22 boomhauer dovecot: child 34901 (imap) killed with signal 6 # gdb /usr/local/libexec/dovecot/imap imap.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... Core was generated by `imap'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/libiconv.so.3...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/dovecot/imap/lib01_acl_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib01_acl_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib10_quota_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib10_quota_plugin.so Reading symbols from /usr/lib/librpcsvc.so.4...done. Loaded symbols for /usr/lib/librpcsvc.so.4 Reading symbols from /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_fts_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_fts_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_listescape_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_listescape_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x409de12c in kill () from /lib/libc.so.7 (gdb) bt full #0 0x409de12c in kill () from /lib/libc.so.7 No symbol table info available. #1 0x409dcf9b in abort () from /lib/libc.so.7 No symbol table info available. #2 0x004b5be5 in default_fatal_finish (type=LOG_TYPE_PANIC, status=0) at failures.c:149 backtrace = 0x0 #3 0x004b699e in i_internal_fatal_handler (type=Could not find the frame base for i_internal_fatal_handler. ) at failures.c:423 No locals. #4 0x004b5eaa in i_panic (format=Could not find the frame base for i_panic. ) at failures.c:190 args = Could not find the frame base for i_panic. (gdb) q
Re: [Dovecot] Fix? available for mmap on FreeBSD zfs (corrupted index files)
Just as a heads up since some dovecot users do use ZFS, pjd recently posted a patch to upgrade ZFS in 8-current from v6 to v11 (latest from opensolaris). Its reported to fix a number of problems, and add new features. The patch does not apply to 7, that won't be available for a while apparently, but they are looking for feedback on the fs or current mailing lists to encourage sooner integration (possibly by 7.1). Quick instructions: fetch http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2 bunzip2 zfs_20080727.patch.bz2 cd /usr/src/ patch -p0 /path/to/zfs_20080727.patch do a FULL upgrade of world/kernel. kmem tweaks may still be needed but its a little more resilient than before, also 8-current now lets you configure a larger kmem than the previous ~1.8G limit. It is recommended to remove other tweaks from loader.conf such as zil_disable, etc and see how it does. I've upgraded my laptop and so far so good, but I don't have dovecot on any systems with ZFS so I can't test that aspect. On Fri, Apr 04, 2008 at 02:06:40PM -0400, Dillon Kass wrote: Hi, I've applied this patch to my 7-STABLE box and it doesn't help with the corrupted dovecot index files. Anyone using dovecot with ZFS on FreeBSD should use mmap_disable = yes in their dovecot.conf Cheers, Dillon Adam McDougall wrote: Saw this just come through for 8-current, 7-stable ought to have it in a week or so. I have not encountered the problem so I would not be able to judge if this fixes it. Patch here: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c.diff?r1=1.27;r2=1.28 pjd 2008-03-15 23:23:04 UTC FreeBSD src repository Modified files: sys/contrib/opensolaris/uts/common/fs/zfs zfs_vnops.c Log: Fix mmap(2) on ZFS after some changes in VM subsystem. Submitted by: alc Reported by:kris (originally) and many others Tested with:fsx MFC after: 1 week Revision ChangesPath 1.28 +4 -0 src/sys/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
Re: [Dovecot] Fix? available for mmap on FreeBSD zfs (corrupted index files)
Ugh, I forgot a step you might need. Due to the patching, /usr/src/sys/cddl/compat/opensolaris/sys/acl.h and /usr/src/sys/cddl/compat/opensolaris/sys/callb.h become empty and if they exist, should be deleted before compiling because they will cause the compile to fail. On Mon, Jul 28, 2008 at 10:29:05PM -0400, Adam McDougall wrote: Just as a heads up since some dovecot users do use ZFS, pjd recently posted a patch to upgrade ZFS in 8-current from v6 to v11 (latest from opensolaris). Its reported to fix a number of problems, and add new features. The patch does not apply to 7, that won't be available for a while apparently, but they are looking for feedback on the fs or current mailing lists to encourage sooner integration (possibly by 7.1). Quick instructions: fetch http://people.freebsd.org/~pjd/patches/zfs_20080727.patch.bz2 bunzip2 zfs_20080727.patch.bz2 cd /usr/src/ patch -p0 /path/to/zfs_20080727.patch do a FULL upgrade of world/kernel. kmem tweaks may still be needed but its a little more resilient than before, also 8-current now lets you configure a larger kmem than the previous ~1.8G limit. It is recommended to remove other tweaks from loader.conf such as zil_disable, etc and see how it does. I've upgraded my laptop and so far so good, but I don't have dovecot on any systems with ZFS so I can't test that aspect. On Fri, Apr 04, 2008 at 02:06:40PM -0400, Dillon Kass wrote: Hi, I've applied this patch to my 7-STABLE box and it doesn't help with the corrupted dovecot index files. Anyone using dovecot with ZFS on FreeBSD should use mmap_disable = yes in their dovecot.conf Cheers, Dillon Adam McDougall wrote: Saw this just come through for 8-current, 7-stable ought to have it in a week or so. I have not encountered the problem so I would not be able to judge if this fixes it. Patch here: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c.diff?r1=1.27;r2=1.28 pjd 2008-03-15 23:23:04 UTC FreeBSD src repository Modified files: sys/contrib/opensolaris/uts/common/fs/zfs zfs_vnops.c Log: Fix mmap(2) on ZFS after some changes in VM subsystem. Submitted by: alc Reported by:kris (originally) and many others Tested with:fsx MFC after: 1 week Revision ChangesPath 1.28 +4 -0 src/sys/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
[Dovecot] 1.1.1 (maildir_uidlist_sync_update): assertion failed: (uidlist-initial_hdr_read)
Version: 1.1.1 OS: FreeBSD 7.0-STABLE amd64 I have only seen this happen twice so far. I do not know what the two users were doing to cause it. Jun 26 17:23:26 hill dovecot: IMAP(userA): Invalid transaction log size (32812 vs 32920): /home/userA/Maildir/dovecot/private/indexes/.Deleted Messages/dovecot.index.log Jun 26 17:23:26 hill dovecot: IMAP(userA): Invalid transaction log size (32812 vs 32920): /home/userA/Maildir/dovecot/private/indexes/.Deleted Messages/dovecot.index.log Jun 26 17:23:26 hill dovecot: Panic: IMAP(userA): file maildir-uidlist.c: line 1207 (maildir_uidlist_sync_update): assertion failed: (uidlist-initial_hdr_read) Jun 26 17:23:26 hill dovecot: child 28120 (imap) killed with signal 6 Jul 11 13:41:46 hill dovecot: IMAP(userB): Invalid transaction log size (92984 vs 93076): /home/userB/Maildir/dovecot/private/indexes/.Old Backup.Personal/dovecot.index.log Jul 11 13:41:46 hill dovecot: IMAP(userB): Invalid transaction log size (92984 vs 93076): /home/userB/Maildir/dovecot/private/indexes/.Old Backup.Personal/dovecot.index.log Jul 11 13:41:46 hill dovecot: Panic: IMAP(userB): file maildir-uidlist.c: line 1207 (maildir_uidlist_sync_update): assertion failed: (uidlist-initial_hdr_read) Jul 11 13:41:46 hill kernel: pid 85434 (imap), uid 20128: exited on signal 6 (core dumped) hill# gdb /usr/local/libexec/dovecot/imap imap.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... Core was generated by `imap'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/libiconv.so.3...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/dovecot/imap/lib01_acl_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib01_acl_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib10_quota_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib10_quota_plugin.so Reading symbols from /usr/lib/librpcsvc.so.4...done. Loaded symbols for /usr/lib/librpcsvc.so.4 Reading symbols from /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_fts_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_fts_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_listescape_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_listescape_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x409d7e7c in kill () from /lib/libc.so.7 (gdb) bt full #0 0x409d7e7c in kill () from /lib/libc.so.7 No symbol table info available. #1 0x409d6ceb in abort () from /lib/libc.so.7 No symbol table info available. #2 0x004b5385 in default_fatal_finish (type=LOG_TYPE_PANIC, status=0) at failures.c:152 backtrace = 0x3 Address 0x3 out of bounds #3 0x004b610e in i_internal_fatal_handler (type=Could not find the frame base for i_internal_fatal_handler. ) at failures.c:418 No locals. #4 0x004b564a in i_panic (format=Could not find the frame base for i_panic. ) at failures.c:193 args = Could not find the frame base for i_panic. (gdb) hill# file /usr/local/libexec/dovecot/imap /usr/local/libexec/dovecot/imap: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), for FreeBSD 7.0 (700101), dynamically linked (uses shared libs), FreeBSD-style, not stripped
[Dovecot] Maildir folder renaming disagreement among IMAP clients
A user alerted me to a problem when he renamed an upper level mail folder and lost access to a lower level folder under it. I tried with and without listescape, it didn't seem to make a difference, and I saw no errors in the dovecot log. Symptoms are if a client has a maildir folder tree .a and .a.b and renames a to c in their client, Thunderbird shows c/b but b is inaccessible since it is still actually .a.b on the filesystem, unsubscribed but accessible if you go subscribe to it. Outlook Express seems to behave the same except it cannot access b (.a.b) at all now because a no longer exists. Subscribe window shows it, but you cannot subscribe (as expected for OE). According to the rawlog, Thunderbird just tries to rename the top level folder and change subscriptions, but doesn't rename other maildirs to match the new top level folder. On the other hand, Outlook seems to rename and subscribe the top folder and subfolders properly. Who is right, should the mail client be doing all the work, or should dovecot be doing more, or do I have something setup wrong? I'm pasting what I think are essentials from the rawlogs, and my dovecot -n. Thanks for any input. tbird: in: 23 lsub zoinks2/* 24 close 25 rename zoinks2 zoinks3 26 subscribe zoinks3 27 unsubscribe zoinks2 28 subscribe zoinks3/z2 29 unsubscribe zoinks2/z2 out: * LSUB () / zoinks2/z2 23 OK Lsub completed. 24 OK Close completed. 25 OK Rename completed. 26 OK Subscribe completed. 27 OK Unsubscribe completed. 28 NO [TRYCREATE] Mailbox doesn't exist: zoinks3/z2 29 OK Unsubscribe completed. outlook: in: d7oy LIST INBOX 089m LSUB zoinks3/* uh2o LSUB * buaw SELECT b51w LSUB zoinks3 fivz RENAME zoinks3 zoinks2 6n3p LIST zoinks3/* chbh RENAME zoinks3/z2 zoinks2/z2 3hjt UNSUBSCRIBE zoinks3/z2 xlnc SUBSCRIBE zoinks2/z2 yyep UNSUBSCRIBE zoinks3 6gt2 SUBSCRIBE zoinks2 mvzq IDLE out: * LSUB () / zoinks3/z2 089m OK Lsub completed. uh2o OK Lsub completed. buaw NO Invalid mailbox name * LSUB () / zoinks3 b51w OK Lsub completed. fivz OK Rename completed. * LIST (\HasNoChildren) / zoinks3/z2 6n3p OK List completed. chbh OK Rename completed. 3hjt OK Unsubscribe completed. xlnc OK Subscribe completed. yyep OK Unsubscribe completed. 6gt2 OK Subscribe completed. # 1.1.1: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache22/ssl/mail.pem ssl_key_file: /usr/local/etc/apache22/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_max_processes_count: 512 max_mail_processes: 1024 mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_access_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_executable: /usr/local/libexec/dovecot/rawlog /usr/local/libexec/dovecot/imap mail_process_size: 1024 mail_plugins: acl fts fts_squat listescape quota imap_quota mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail outlook-idle netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: quota: fs:noenforcing:inode_per_mail acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat
Re: [Dovecot] Maildir folder renaming disagreement among IMAP clients
Timo Sirainen wrote: On Tue, 2008-07-15 at 14:57 -0400, Adam McDougall wrote: A user alerted me to a problem when he renamed an upper level mail folder and lost access to a lower level folder under it. I tried with and without listescape, it didn't seem to make a difference, and I saw no errors in the dovecot log. It should have worked properly without listescape (and worked in my tests). Fixed listescape: http://dovecot.org/patches/1.1/listescape-plugin.c The way it should work is: - Dovecot should rename all child mailboxes - Dovecot shouldn't touch subscriptions Yes thanks it works now! I accidentally didn't test without listescape as I claimed, it was still enabled by accident. Come to think of it, this might affect moving folder trees to Trash and expunging it? I think I had one user report problems a few weeks ago, and a co-worker just cleaned it up manually on the back end. I didn't get a chance to inspect closely.
[Dovecot] No log for hitting login_max_processes_count?
I think today just due to increased use, my mail servers hit login_max_processes_count. It wasn't obvious what was happening, logins just weren't acting like they should and sometimes would get disconnected before given a prompt from the server. Something made me think to check if I had too many of certain dovecot processes and I found I had 128 imap-login processes on both servers, but nothing in the logs seemed to point that out for me. Should there be? It would have been alot easier to debug. Also, this seems to be from general use, doesn't seem like a DoS, so is this just a setting I should keep an eye on and increase it by reasonable amounts as needed? Thanks.
Re: [Dovecot] Keeping \Seen flag private
Imobach González Sosa wrote: Hi all, I wanna to set up shared folders for a couple of users and I'd like that everyone could keep the \Seen flag as private. So if user #1 reads some messages and user #2 not, those messages appear as unseen to #2 and seen to #1. I've implemented shared folders using namespaces with every user having their own control and private directories. But all the flags (\Seen included) are shared. Am I on the right path? Some tip or documentation? Oh, I almost forgot: using dovecot 1.0.13 (from Debian Etch backports). Thank you in advance! From Timo, I wanted the same thing so I patch the source each time: src/lib-storage/index/maildir/maildir-storage.c near line 428, change: mbox-ibox.box.private_flags_mask = MAIL_SEEN; to: mbox-ibox.box.private_flags_mask = 0;
[Dovecot] sig11 in 1.1rc5 fts
This happened from one user near noon on the 17th and 19th (today) of
this month. From the backtrace it looks like they were searching, but I
won't know for sure unless I need to ask them. Is this possibly fixed
already? I just haven't upgraded dovecot in a while due to lack of
problems. The sig11 happened a few dozen times, a few seconds apart
each day. I have one coredump from each day, and the size was the
same. This is a trace from only one. The other backtrace looks pretty
much the same.
Version: 1.1rc5
OS: FreeBSD 7.0-STABLE
#0 0x4101bf11 in node_read_children (trie=0x40c5a800,
node=0x40c5a800, level=1) at squat-trie.c:461
data = (const uint8_t *) 0x416b7f9a
;\031\030\023\031\0305\031\030O\031\030\033\031\0309\031\030#\031\030
end = (const uint8_t *) 0x416b7ffe
child_chars = (const unsigned char *) 0x414dbd25 Address
0x414dbd25 out of bounds
child = (struct squat_node *) 0x40c685f8
children = (struct squat_node *) 0x40c68040
node_offset = 1944868
i = 61
child_idx = 61
child_count = 89
base_offset = 1944314
num = 25
__PRETTY_FUNCTION__ = node_read_children
#1 0x4101f17f in squat_trie_map (trie=0x40c5a800,
building=false) at squat-trie.c:1518
file_lock = (struct file_lock *) 0x0
dotlock = (struct dotlock *) 0x0
changed = true
ret = 0
#2 0x4101b4c9 in squat_trie_open (trie=0x40c5a800) at
squat-trie.c:242
No locals.
#3 0x4101f908 in squat_trie_get_last_uid (trie=0x40c5a800,
last_uid_r=0x7fffce5c) at squat-trie.c:1725
No locals.
#4 0x4101a79c in fts_backend_squat_get_last_uid
(_backend=0x40c2b150, last_uid_r=0x7fffce5c) at fts-backend-squat.c:104
backend = (struct squat_fts_backend *) 0x40c2b150
#5 0x40e0f822 in fts_backend_get_last_uid (backend=0x40c2b150,
last_uid_r=0x7fffce5c) at fts-api.c:80
No locals.
#6 0x40e10bb1 in fts_build_init (fctx=0x40cee0f0) at
fts-storage.c:177
t = (struct mailbox_transaction_context *) 0x40c2b1e0
backend = (struct fts_backend *) 0x40c2b150
ctx = (struct fts_storage_build_context *) 0x40c2b150
build = (struct fts_backend_build_context *) 0x40c7e120
seqset = {seq1 = 16, seq2 = 1, next = 0x0}
last_uid = 0
last_uid_locked = 131072
__PRETTY_FUNCTION__ = fts_build_init
#7 0x40e11351 in fts_try_build_init (fctx=0x40cee0f0) at
fts-storage.c:320
No locals.
#8 0x40e114a6 in fts_mailbox_search_init (t=0x40c2b1e0,
charset=0x4ceca0 UTF-8, args=0x40c7e0c0, sort_program=0x0)
at fts-storage.c:355
fbox = (struct fts_mailbox *) 0x40cdc040
ctx = (struct mail_search_context *) 0x40c28bc0
fctx = (struct fts_search_context *) 0x40cee0f0
#9 0x004714e1 in mailbox_search_init (t=0x40c2b1e0,
charset=0x4ceca0 UTF-8, args=0x40c7e0c0, sort_program=0x0)
at mail-storage.c:621
No locals.
#10 0x00419f9c in imap_search_init (cmd=0x40c7e048,
box=0x40c67048, charset=0x4ceca0 UTF-8, sargs=0x40c7e0c0)
at cmd-search.c:36
ctx = (struct imap_search_context *) 0x40c7e190
#11 0x0041a644 in cmd_search (cmd=0x40c7e048) at cmd-search.c:190
ctx = (struct imap_search_context *) 0x40c1e240
sargs = (struct mail_search_arg *) 0x40c7e0c0
args = (const struct imap_arg *) 0x40c84080
args_count = 4
error = 0x0
charset = 0x4ceca0 UTF-8
#12 0x0041bd12 in cmd_uid (cmd=0x40c7e048) at cmd-uid.c:26
command = (struct command *) 0x40c1e2a0
cmd_name = 0x40c84170 SEARCH
#13 0x0041d3a4 in client_command_input (cmd=0x40c7e048) at
client.c:553
client = (struct client *) 0x40c36180
command = (struct command *) 0x2340c7e0b8
__PRETTY_FUNCTION__ = client_command_input
#14 0x0041d5e1 in client_command_input (cmd=0x40c7e048) at
client.c:602
client = (struct client *) 0x40c36180
command = (struct command *) 0x40c1e240
__PRETTY_FUNCTION__ = client_command_input
#15 0x0041d723 in client_handle_next_command (client=0x40c36180,
remove_io_r=0x7fffd0fd) at client.c:643
size = 40
#16 0x0041d769 in client_handle_input (client=0x40c36180) at
client.c:653
---Type return to continue, or q return to quit---
_data_stack_cur_id = 3
ret = false
remove_io = false
handled_commands = false
#17 0x0041d904 in client_input (client=0x40c36180) at client.c:698
cmd = (struct client_command_context *) 0xd16c
output = (struct ostream *) 0x40c7a0a8
bytes = 40
__PRETTY_FUNCTION__ = client_input
#18 0x004be449 in io_loop_handler_run (ioloop=0x40c19140) at
ioloop-kqueue.c:149
ctx = (struct ioloop_handler_context *) 0x40c1be60
events = (struct kevent *) 0x40c35000
event = (const struct kevent *) 0x40c35000
tv = {tv_sec = 1799, tv_usec = 999471}
Re: [Dovecot] Time moved backwards
Charles Marcus wrote: On 5/13/2008, Eugene ([EMAIL PROTECTED]) wrote: I guess terminating all current connections and restarting all processes would be pretty safe, but it's not really a high priority change for me.. Nevertheless, it would be very nice if you could fix it. It's a fairly big availability problem (for us, at least). The problem is not so much how dovecot deals with this issue, the problem is, why is your server having such drastic problems keeping its time sane? Fix that, and your problem disappears. I would just like to mention a circumstance that happened to me this Sunday. We had a total power outage in our building, longer than our UPS's could last and we don't have a generator for servers (nor is it economical or needed). When the power came back on, my local NTP server came on at the same time as my mail servers, as well a majority of my other servers. My servers tried to step their time to be in sync with my local NTP server, which was still busy trying to sync itself with outside sources, which takes a while, so my mail servers did not get an answer. Later, dovecot died because the time finally synced, and I found out why pretty quick (have seen this before) but this was an unusual situation. My point is, we had an unusual circumstance, and even though I've taken steps to have my mail servers sync their time at boot and run ntpd afterwards, there are some circumstances in which this is not enough, and dovecot still died. Its not always because someone was lazy about their time setup. But it doesn't cause me big availability problems since in general, my time is fine.
[Dovecot] dovecot.index.cache: Corrupted virtual size for uid=956: 2182 != 4917
This is from 1.1rc5. I have no idea where the 2182 even came from, all copies of the dovecot-uidlist and all copies of the email in snapshots going back at least 2 weeks show the info below: May 8 22:45:22 hill dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Corrupted virtual size for uid=956: 2182 != 4917 dovecot-uidlist: 956 W4917 :1208807110.M854034P88191.hill:2,RS -rw--- 2 username egrstaff 4831 Mar 26 13:53 1208807110.M854034P88191.hill:2,RS
[Dovecot] Invalid transaction log size (32684 vs 32784)
May 8 10:17:44 boomhauer dovecot: IMAP(username2): Invalid transaction log size (32684 vs 32784): /home/username2/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.log May 8 10:17:44 boomhauer dovecot: IMAP(username2): Invalid transaction log size (32684 vs 32784): /home/username2/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.log May 8 10:17:44 boomhauer dovecot: IMAP(username2): Invalid transaction log size (32684 vs 32784): /home/username2/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.log
[Dovecot] A previously unreported lsub/list discrepancy in 1.1rc5 and earlier
Not sure about 1.0, I don't run it anymore :) But a few users discovered for a small issue that only affects a very small portion of my userbase where instructional correction will suffice. I guess I am reporting it in the interest of getting it out there, and I can live with it if not fixed but it might be an issue for some people. I'm running through my list of issues. Basically, if a user has a valid namespace prefix set such as mail/ (used to be required on old mail server but discouraged on new mail client setups with dovecot), lsub/list do not show public namespaces that the user has permission to via acl. If the user removes the namespace prefix, they can 'see' the public folders and subscribe. Additionally if the user subscribes to a public folder then puts mail/ back in the prefix, the folders are no longer listed. Not sure if public folders should be affected by the requested prefix in this manner, since they are not in the user's account. With prefix set: 3 lsub mail/* 4 list mail/% 5 list mail/%/% * OK [RAWLOG TIMESTAMP] 2008-05-05 21:07:16 * LSUB () / mail/INBOX * LSUB () / mail/Sent Items * LSUB () / mail/Drafts * LSUB () / mail/Junk E-mail * LSUB () / mail/Sent * LSUB () / mail/Trash 3 OK Lsub completed. * LIST (\HasNoChildren) / mail/Drafts * LIST (\HasNoChildren) / mail/Sent Items * LIST (\HasNoChildren) / mail/Junk E-mail * LIST (\HasNoChildren) / mail/Sent * LIST (\HasNoChildren) / mail/Trash 4 OK List completed. 5 OK List completed. Without prefix set: 3 lsub * 4 list % 5 list %/% * OK [RAWLOG TIMESTAMP] 2008-05-05 21:07:49 * LSUB () / INBOX * LSUB () / Sent Items * LSUB () / Drafts * LSUB () / Junk E-mail * LSUB () / Sent * LSUB () / Trash 3 OK Lsub completed. * LIST (\HasNoChildren) / Drafts * LIST (\HasNoChildren) / Sent Items * LIST (\HasNoChildren) / Junk E-mail * LIST (\HasNoChildren) / Sent * LIST (\HasNoChildren) / Trash * LIST (\HasNoChildren) / INBOX 4 OK List completed. * LIST (\Noselect \HasChildren) / #shared/be * LIST (\Noselect \HasChildren) / #shared/me * LIST (\Noselect \HasChildren) / #shared/cee 5 OK List completed.
Re: [Dovecot] 1.1rc1: assertion failed: (output-offset uid_list[0])
Timo Sirainen wrote:
On Fri, 2008-03-07 at 00:17 -0500, Adam McDougall wrote:
$3 = {offset = 40896, stream_errno = 70, last_failed_errno = 70, overflow
= 0, closed = 1,
real_stream = 0x678500}
Ah, 70 = Stale NFS handle in FreeBSD. Linux used something else. Squat
still needs some NFS fixes.
Were these 'NFS fixes' done? I just put rc5 in production at my site an
hour
or two ago so I have just begun gathering log data. I may not see anything
interesting for days but I will review them daily.
(apologies if this gets double posted, I was sending mail and got
disconnected)
Re: [Dovecot] imap fs quota (rpc) won't work?
Bump :) Adam McDougall wrote: In the past I dabbled with the imap quota plugin with the fs backend because I wanted to report usage to my users (not limit them). At the time, the quota plugin would make dovecot crash when trying to write to a folder (I can bring up this report if needed). However, in a later beta of dovecot 1.1 I tried quota again but I cannot get it to report any results. I've been wanting to use fs:noenforcing but also tried just quota = fs to simplify. When I right click a folder stored in my homedir, thunderbird tells me 'the server does not support quotas' in the folder properties. What else can I do? Thanks! IModule loaded: /usr/local/lib/dovecot/imap/lib10_quota_plugin.so IModule loaded: /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota add storage dir = /home/mcdouga9/Maildir Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota block device = cypher:/vol/mail/home Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota mount point = /home gribble# quota -v mcdouga9 | grep home /home 10766672 2048 2048 963778 4294967295 4294967295 # 1.1.beta14: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl fts fts_squat quota imap_quota mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/de list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: quota: fs acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat
[Dovecot] FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919
This happened a while back, I was running 1.1rc3 up until today so I have no idea if it would have an effect. Sorry if its something that has been fixed. The error below was repeated many many times, I deleted the index so the user would stop getting disconnected (although I didn't hear any complaints). I have not seen this happen repeatedly in this manner to any other user yet. Apr 15 11:26:48 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:26:48 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:26:48 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 Apr 15 11:27:08 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:27:08 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:27:08 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 Apr 15 11:28:50 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:28:50 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:28:50 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 Apr 15 11:38:02 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:38:02 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:38:02 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 Apr 15 11:38:52 boomhauer dovecot: IMAP(username): FETCH for mailbox INBOX UID 5003 got too little data: 5369 vs 38919 Apr 15 11:38:52 boomhauer dovecot: IMAP(username): Corrupted index cache file /home/username/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.cache: Broken virtual size for mail UID 5003 Apr 15 11:38:52 boomhauer dovecot: IMAP(username): Disconnected: Disconnected bytes=139/145074 From dovecot-uidlist: 5003 W38919 S38270 :1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 1208208981.7303_0.hill:2,S # ls -l .snapshot/*/1208208981.7303_0.hill* -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.0/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.1/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.10/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.11/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.12/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.13/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.14/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.15/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.16/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.17/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.18/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.19/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.2/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.20/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.21/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.22/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.3/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.4/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.5/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.6/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.7/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.8/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/hourly.9/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14 17:36 .snapshot/nightly.0/1208208981.7303_0.hill:2,S -rw--- 1 username faculty 5236 Apr 14
Re: [Dovecot] Users w/o acl access appear to be subscribed to public folders (1.1b8)
Timo Sirainen wrote: On Tue, 2007-11-20 at 22:20 -0500, Adam McDougall wrote: I noticed this today, I had a user outside of our department test out dovecot. They were using squirrelmail and I noticed that dovecot thinks this user is subscribed to ALL public folders even though a dovecot ACL prevents all access. I'm pretty sure access is still denied. Fixed finally in hg. There were several bugs related to listing mailboxes. Great! I will test this tomorrow. I loaded rc5 on my two test servers and I will review it for any existing issues as you asked in the rc5 announcement. Thanks.
Re: [Dovecot] [RFC] FreeBSD port for dovecot 1.1 series
Xin LI wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Hi,
I have put together a preliminary patchset for dovecot 1.1 at:
http://people.freebsd.org/~delphij/misc/dovecot-1.1-rc4.diff
My intention is to repocopy the current dovecot port to dovecot11 and
make changes on the latter. In this version of patchset, I have
intentionally removed the following chunk of change which by default
allows gid=0 users to log in.
%%%
@@ -333,6 +338,7 @@
~ # belongs to supplementary groups with non-valid GIDs, those groups are
~ # not set.
~ #first_valid_gid = 1
+first_valid_gid = 0
~ #last_valid_gid = 0
~ # Maximum number of running mail processes. When this limit is reached,
%%%
Please let me know if you want more features, have comments, etc., for
the FreeBSD port. I am subscribed to this list but I would appreciate
if you could use 'Reply all' which will give me more opportunity to get
your e-mail from the thousands I receive :)
Cheers,
- --
Xin LI [EMAIL PROTECTED]http://www.delphij.net/
First, I apologize for sitting on my attempt so long, work got busy.
But just now I did a diff between your result and mine, and some notes:
- I added a CONFLICTS?= ${PORTNAME}-1.0.* because using portupgrade
on other ports such as postfix tries to pull in dovecot-1.0 as a runtime
depend whether or not 1.1 is already installed and that makes a mess. I
also added a similar line to the dovecot 1.0 port, which was probably
the more important place.
- Otherwise, you port looks extremely similar in important functionality
to mine. Although one of the reasons I did not submit it earlier was I
had hoped to add an OPTIONS entry and appropriate pkg-plist support for
some of the newer optional support offered by 1.1, but I don't use any
of them myself. Its probably better to have a dovecot 1.1 port even
without that part done.
Re: [Dovecot] FreeBSD port: port's patch for 1.1rc3
Geoffroy Desvernay wrote: For those who wants testing 1.1rc3 on FreeBSD, this is a simple patch against /usr/ports/mail/dovecot. To apply: # cd /usr/ports/mail/dovecot # patch -p1 /tmp/patch_dovecot_1.0.13-1.1.rc3.diff Hope I did not forget something important... This is an adaptation of the existing port. PS: for port's maintainer: Which way will take 1.0 vs 1.1 upgrade ? a new mail/dovecot10 port or ? Hope this helps -- Geoffroy Desvernay I'm not the maintainer but I've also made a dovecot port for 1.1 as a separate port since it is still in development and may require a few config changes, I discussed that with the maintainers. I made it for work purposes and I have been keeping up with necessary changes to keep it working while I use 1.1, and intended to submit it for inclusion in the ports collection but haven't got around to it yet (I'd like to add a few compile options for new things since 1.1). Been really busy at work and haven't had time, but I guess if people show they are interested I could try to speed that up.
[Dovecot] Fix available for mmap on FreeBSD zfs
Saw this just come through for 8-current, 7-stable ought to have it in a week or so. I have not encountered the problem so I would not be able to judge if this fixes it. Patch here: http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c.diff?r1=1.27;r2=1.28 pjd 2008-03-15 23:23:04 UTC FreeBSD src repository Modified files: sys/contrib/opensolaris/uts/common/fs/zfs zfs_vnops.c Log: Fix mmap(2) on ZFS after some changes in VM subsystem. Submitted by: alc Reported by:kris (originally) and many others Tested with:fsx MFC after: 1 week Revision ChangesPath 1.28 +4 -0 src/sys/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c
Re: [Dovecot] Help! OT: Blackberry IMAP client suggestions/experience needed
Stewart Dean wrote: and here are some clients fighting over the lock Mar 6 09:05:48 mercury mail:info imapd[970952]: Killed (lost mailbox lock) user =x host=cpe-24-161-103-11.hvc.res.rr.com [24.161.103.11] Mar 6 09:08:04 mercury mail:info imapd[844000]: Killed (lost mailbox lock) user =x host=[10.40.70.71] Mar 6 09:08:18 mercury mail:info imapd[2547784]: Killed (lost mailbox lock) use r=x host=cpe-24-161-103-11.hvc.res.rr.com [24.161.103.11] All I know is that I had some moderately important people screaming at me (I even heard the word useless) that the mail service was %^$#ed up. That after I made it dianetically clear (took 2 weeks) to them that there Must Only Ever Be One Mail Client Open At A Time, they no longer had problems. We have some users that caused this too on uw-imap with mbox, they would get frustrated that deleted mail would come back as well. I'm happy to say that Maildir with dovecot handles this extremely well, even on NFS with multiple servers in dovecot 1.1 since one of the later betas. You might consider how to direct your needy users to a mail server instance that has Maildir. If you are afraid, or its difficult to wedge this into your current setup, you could have a completely different mail server host their mail, and you could use a MX server and/or a perdition IMAP proxy to redirect them invisibly to this other server once you've moved or copied their data. I wanted to avoid this path and get the migration over with, but I caved in and started going after the power users and the low hanging fruit. I'm very glad I did, because now I can migrate users one at a time and deal with any problems that arise on my schedule, and when I am prepared in the future, I can throw some switches and move the remaining users over that are unrealistic to handhold.
Re: [Dovecot] Allowing tilde at start of mailbox names [listescape-plugin.c]
Timo Sirainen wrote: I think the latest http://dovecot.org/patches/1.1/listescape-plugin.c should work with latest Dovecot from hg. At least I fixed the most obvious problems. I have been using it for a few weeks and it has been working well, but yesterday I noticed it also seems to be escaping the \ (backslash) character in folder names. This wouldn't be a problem except it breaks operation of folders that already had a \ in the name before the plugin was loaded. This isn't a huge problem for us but I want to figure out if I should special case migration of folder names containing a \ to be the new internal representation \5c or if the plugin should just deal with normal \'s in the foldernames properly. Existing folder names with a \ such as \test still show up as \test in the mail client listing, but all known attempts to access it fail, I think internally it is probably looking for a folder named \5ctest and not finding it. Thus the user cannot read, delete, rename, unsubscribe, anything. They are stuck with a useless folder while the plugin is loaded. I doubt many of our users have this situation and they can contact us to fix it if they notice, but I did have one user alert me of it. On the other hand if a folder and its subscription are manually renamed to \5ctest, it works fine, or if they create a new one it uses a \5c on the backend. Thanks for assistance.
Re: [Dovecot] Cingular/ATT killing my IMAP/POP connections with bad TCP FIN packets?
On Mon, Mar 10, 2008 at 08:12:37PM -0400, Peter Tripp wrote: Hello all, I've got an issue I'm almost positive is not related to Dovecot, but was wondering whether anyone else has had similar problems or could duplicate Me: SYN Paradox: SYN, ACK Me: ACK Me: Client Hello Paradox: ACK Paradox: FIN, ACK Me: ACK Paradox: FIN, ACK Me: ACK -dead- http://duckies.org/~peter/damncingular.txt I notice a 6-7 second delay between the first ACK and the first FIN,ACK. Is your dovecot server perhaps unable to do a DNS lookup on the connecting IP? It would be very informative to see what the dovecot server is sending (or not sending) including generated traffic such as DNS queries, if you can isolate that. Is anything in the dovecot logs? Wondering if it is related to something like tcp wrappers.
Re: [Dovecot] question about dovecot imap outlook clients
On Mon, Mar 10, 2008 at 05:04:19PM -0700, Asheesh Laroia wrote: On Mon, 10 Mar 2008, Joseph Norris wrote: Questions: Do I have to get an ssl certificate to make it work? ( cost ouch!) Is there a way around this using my own self-signed certificates? Is there a cheaper ssl certificate service? When I was an admin at acm.jhu.edu, I had us use the free certificates for .edu hosts given out by ipsca.com. They were compatible and well-supported, and signed by the right authorities to have no error messages. (Except in some totally weird interaction with Mozilla, for which we opened a bug and which I *think* is fixed.) You can toy with https://secure.acm.jhu.edu/ and connecting via SSL'd IMAP to secure.acm.jhu.edu (port 993). For my personal servers, I use the RapidSSL certificates sold by Geotrust. I can't seem to find the link for the vendor I use, but they seem to be widely resold for around $10-15 a year. The only serious complaint I can find on the web is that if you use their bulk purchasing option, be sure to read the fine print - your ability to use the bulk-purchased certificates goes away one year after you purchased them. As for how to set them up, I always follow the Apache mod_ssl instructions and then use the certificates everywhere else on my system. As for if any of this is truly necessary, no idea. (-: I did it because I wanted SSL/TLS. -- Asheesh. Godaddy also has cheap SSL certificates. They give you a certificate and another file that contains the chained certificates. You probably need to serve both to avoid browsers giving an unhelpful something is wrong but I'm not telling you what error if the cert chain hierarchy is missing.
Re: [Dovecot] 1.1r1: auth-worker(default): BUG: PASSV had missing parameters, sig11
Timo Sirainen wrote: On Sun, 2008-03-09 at 03:50 -0400, Adam McDougall wrote: Mar 8 17:02:17 boomhauer dovecot: auth-worker(default): BUG: PASSV had missing parameters Thanks, I kept trying to figure out what caused this and then started wondering about password escaping and found the security hole. I still hadn't figured out what caused this though, until I realized that passwords can have linefeeds as well which can cause this. Mar 8 17:05:17 boomhauer dovecot: child 72819 (login) killed with signal 11 This still shouldn't happen though. I didn't try to reproduce this yet. It's anyway quite difficult to get core dumps out of login processes. I'm not sure if FreeBSD lets you do that in some special way, but there are at least two things in the way: 1. Kernel thinks it's a setuid program, and setuid programs don't core dump. 2. It's chrooted to a non-writable directory. 1. I could enable this: # sysctl -d kern.sugid_coredump kern.sugid_coredump: Enable coredumping set user/group ID processes 2. And add an absolute path infront of this that is world writable: # sysctl kern.corefile kern.corefile: %N.%P.boomhauer.core Can you think of a way that I could force the issue to be reproduced so I can get away with making these changes on less servers?
Re: [Dovecot] 1.1r1: auth-worker(default): BUG: PASSV had missing parameters, sig11
Timo Sirainen wrote: On Sun, 2008-03-09 at 11:48 -0400, Adam McDougall wrote: Mar 8 17:05:17 boomhauer dovecot: child 72819 (login) killed with signal 11 1. I could enable this: # sysctl -d kern.sugid_coredump kern.sugid_coredump: Enable coredumping set user/group ID processes 2. And add an absolute path infront of this that is world writable: # sysctl kern.corefile kern.corefile: %N.%P.boomhauer.core Interesting. I added these to: http://dovecot.org/bugreport.html Can you think of a way that I could force the issue to be reproduced so I can get away with making these changes on less servers? I think this fixes it: http://hg.dovecot.org/dovecot-1.1/rev/de4881149c0e Applied to my installation. Do you think the condition was it introduced around rc1, or older?
[Dovecot] 1.1r1: auth-worker(default): BUG: PASSV had missing parameters, sig11
I don't know how this happened. I'm not sure if there is a coredump somewhere because I don't know what user, and I have nothing for 'root' or 'dovecot'. Any advice, or should I make all my coredumps go to a central writable directory so I have better chance of catching it if it happens again? Is it maybe from someone connecting to postfix and causing a SMTP-AUTH to timeout? Although that is unlikely because users should not know how to reach this server's SMTP ports. I cut most of the surrounding log entries below because they seemed unrelated normal activity, but two of the lines below I left because of the timestamp. socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix Mar 8 17:02:17 boomhauer dovecot: auth-worker(default): BUG: PASSV had missing parameters Mar 8 17:03:47 boomhauer dovecot: auth-worker(default): BUG: PASSV had missing parameters Mar 8 17:05:17 boomhauer dovecot: imap-login: Disconnected: Inactivity: method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Mar 8 17:05:17 boomhauer dovecot: child 72819 (login) killed with signal 11 Mar 8 17:06:47 boomhauer dovecot: imap-login: Disconnected: Inactivity: method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Mar 8 17:06:47 boomhauer dovecot: child 72857 (login) killed with signal 11
[Dovecot] dovecot: child 53439 (imap) killed with signal 11
For quite some time I have seen dovecot imap exit with a sig11 without any other error message, but usually with the frequency of between once per week and once per day (per server). A few weeks ago I finally enabled core dumping with debug symbols and it seemed to happen less often. I finally caught a coredump that wasn't overwritten. I have no idea what on the client end may cause it, it has happened to myself too. dovecot 1.11b16 FreeBSD 6.3 amd64 Maildir on NFS # 1.1.beta16: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache22/ssl/mail.pem ssl_key_file: /usr/local/etc/apache22/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_plugins: acl fts fts_squat mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/decs list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat # gdb /usr/local/libexec/dovecot/imap imap.53439.hill.core GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... Core was generated by `imap'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/local/lib/libiconv.so.3...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /usr/local/lib/dovecot/imap/lib01_acl_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib01_acl_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_fts_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_fts_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x00435edd in maildir_uidlist_iter_next_rec (ctx=0x647120, rec_r=0x7fffdeb0) at maildir-uidlist.c:1522 1522maildir-uidlist.c: No such file or directory. in maildir-uidlist.c #0 0x00435edd in maildir_uidlist_iter_next_rec (ctx=0x647120, rec_r=0x7fffdeb0) at maildir-uidlist.c:1522 rec = (struct maildir_uidlist_rec *) 0x0 __PRETTY_FUNCTION__ = maildir_uidlist_iter_next_rec #1 0x0043461a in maildir_uidlist_write_fd (uidlist=0x64a000, fd=9, path=0x612a80 /home/batesst/Maildir/dovecot/private/control/.INBOX/dovecot-uidlist, first_idx=56, file_size_r=0x7fffdf30) at maildir-uidlist.c:936 storage = (struct mail_storage *) 0x60a848 iter = (struct maildir_uidlist_iter_ctx *) 0x647120 output = (struct ostream *) 0x64a268 rec = (struct maildir_uidlist_rec *) 0x0 str = (string_t *) 0x602060 p = (const unsigned char *) 0x7fffdee0 \200*a len = 0 ret = 6295912 __PRETTY_FUNCTION__ = maildir_uidlist_write_fd #2 0x00434ce4 in maildir_uidlist_sync_update (ctx=0x665040) at maildir-uidlist.c:1095 uidlist = (struct maildir_uidlist *) 0x64a000 file_size = 227633266741 __PRETTY_FUNCTION__ = maildir_uidlist_sync_update #3 0x00435b9f in maildir_uidlist_sync_deinit (_ctx=0x601198) at maildir-uidlist.c:1448 _data_stack_cur_id = 5 ctx = (struct maildir_uidlist_sync_ctx *) 0x665040 ret = 0 __PRETTY_FUNCTION__ = maildir_uidlist_sync_deinit #4 0x00430bab in maildir_sync_context (ctx=0x601168, forced=false, find_uid=0x0, lost_files_r=0x7fffdfef) at maildir-sync.c:836 sync_flags = MAILDIR_UIDLIST_SYNC_PARTIAL flags = 4294959056
Re: [Dovecot] 1.1b16(really b15): (buffer_set_used_size): assertion failed: (used_size = buf-alloc)
Adam McDougall wrote: Timo Sirainen wrote: On Tue, 2008-02-19 at 01:04 -0500, Adam McDougall wrote: I haven't seen this before 1.1b16, it happened to two users today while they were searching using fts. Hmm. You sure this is with beta16? Because: #5 0x004a6fd9 in charset_to_utf8_try (t=0x60e2b0, charset-iconv.c:107 In pre-beta16 line 107 contains: buffer_set_used_size(dest, dest-used - destleft); In beta16 it's: if (!dtcase) The change was: http://hg.dovecot.org/dovecot-1.1/rev/fcfe2ea5c3ed And it should have fixed this exact assert-crash. Oops! Sorry, it was b15. I had b16 on my mind because I had just installed it on two other servers. I'll try it. Thanks. The user reports it seems to be fixed.
[Dovecot] imap fs quota (rpc) won't work?
In the past I dabbled with the imap quota plugin with the fs backend because I wanted to report usage to my users (not limit them). At the time, the quota plugin would make dovecot crash when trying to write to a folder (I can bring up this report if needed). However, in a later beta of dovecot 1.1 I tried quota again but I cannot get it to report any results. I've been wanting to use fs:noenforcing but also tried just quota = fs to simplify. When I right click a folder stored in my homedir, thunderbird tells me 'the server does not support quotas' in the folder properties. What else can I do? Thanks! IModule loaded: /usr/local/lib/dovecot/imap/lib10_quota_plugin.so IModule loaded: /usr/local/lib/dovecot/imap/lib11_imap_quota_plugin.so Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota add storage dir = /home/mcdouga9/Maildir Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota block device = cypher:/vol/mail/home Feb 5 21:53:16 gribble dovecot: IMAP(mcdouga9): fs quota mount point = /home gribble# quota -v mcdouga9 | grep home /home 10766672 2048 2048 963778 4294967295 4294967295 # 1.1.beta14: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mail_debug: yes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl fts fts_squat quota imap_quota mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/de list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: quota: fs acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat
[Dovecot] (message_parse_header_next): assertion failed:, +(IS_LWSP(line-value[0])) 1.1beta14
I noticed these happen when one of my users searches his Trash folder which he doesn't empty. He uses thunderbird and it is reproducable. Feb 5 22:47:39 boomhauer dovecot: IMAP(username): file message-header-parser.c: line 350 (message_parse_header_next): assertion failed: +(IS_LWSP(line-value[0])) Feb 5 22:47:41 boomhauer dovecot: child 8022 (imap) killed with signal 6 Feb 5 22:48:21 boomhauer dovecot: IMAP(username): file message-header-parser.c: line 350 (message_parse_header_next): assertion failed: +(IS_LWSP(line-value[0])) Feb 5 22:48:24 boomhauer dovecot: child 8121 (imap) killed with signal 6 Feb 5 22:49:13 boomhauer dovecot: IMAP(username): file message-header-parser.c: line 350 (message_parse_header_next): assertion failed: +(IS_LWSP(line-value[0])) Feb 5 22:49:15 boomhauer dovecot: child 8171 (imap) killed with signal 6 I hope this is enough of a backtrace, let me know if not: GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as amd64-marcel-freebsd... Core was generated by `imap'. Program terminated with signal 6, Aborted. Reading symbols from /usr/local/lib/libiconv.so.3...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /lib/libc.so.6...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /usr/local/lib/dovecot/imap/lib01_acl_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib01_acl_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib20_fts_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib20_fts_plugin.so Reading symbols from /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so...done. Loaded symbols for /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 0x109d54ac in __res_pquery () from /lib/libc.so.6 (gdb) bt #0 0x109d54ac in __res_pquery () from /lib/libc.so.6 #1 0x004a8e1b in i_panic (format=0x4d60e0 Tue) at failures.c:191 #2 0x004a0149 in message_parse_header_next (ctx=0x18c3f00, hdr_r=0x7fffdff0) at message-header-parser.c:341 #3 0x004a11e9 in parse_content_type (ctx=0x600180, hdr=0x0) at message-parser.c:441 #4 0x004a1bb6 in message_parser_init_from_parts (parts=0x450f848, input=0x7fffe0c0, hdr_flags=32767, flags=16769184) at message-parser.c:718 #5 0x10c2355d in fts_mailbox_search_next_nonblock () from /usr/local/lib/dovecot/imap/lib20_fts_plugin.so #6 0x0046b353 in mailbox_search_deinit (_ctx=0xc42200) at mail-storage.c:624 #7 0x00418f2f in imap_search_deinit (cmd=0x60c300, ctx=0x61d048) at cmd-search.c:64 #8 0x0041916e in cmd_search_more (cmd=0x4b2216) at cmd-search.c:119 #9 0x004b25ef in io_loop_handle_timeouts_real (ioloop=0x5ff240) at ioloop.c:257 #10 0x004b263d in io_loop_handle_timeouts_real (ioloop=0x5ff240) at ioloop.c:267 #11 0x004b351e in io_loop_handler_run (ioloop=0x5ff240) at ioloop-kqueue.c:123 #12 0x004b2690 in io_loop_handle_timeouts_real (ioloop=0x5ff240) at ioloop.c:280 #13 0x004265de in main_deinit () at main.c:269 #14 0x0041418e in _start (ap=0x0, cleanup=0x7fffe6ad) at /usr/src/lib/csu/amd64/crt1.c:69 #15 0x7fffe6ca in ?? () #16 0x7fffe6dd in ?? () #17 0x7fffe6f8 in ?? () #18 0x7fffe709 in ?? () #19 0x7fffe71a in ?? () #20 0x7fffe733 in ?? () #21 0x7fffe747 in ?? () # 1.1.beta14: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login mail_max_userip_connections: 25 verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_drop_priv_before_exec: yes mail_plugins: acl fts fts_squat mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/decs list: yes subscriptions:
Re: [Dovecot] (message_parse_header_next): assertion failed:, +(IS_LWSP(line-value[0])) 1.1beta14
Adam McDougall wrote:
I noticed these happen when one of my users searches his Trash folder
which he doesn't empty.
He uses thunderbird and it is reproducable.
Feb 5 22:47:39 boomhauer dovecot: IMAP(username): file
message-header-parser.c: line 350 (message_parse_header_next):
assertion failed:
+(IS_LWSP(line-value[0]))
Feb 5 22:47:41 boomhauer dovecot: child 8022 (imap) killed with signal 6
Feb 5 22:48:21 boomhauer dovecot: IMAP(username): file
message-header-parser.c: line 350 (message_parse_header_next):
assertion failed:
+(IS_LWSP(line-value[0]))
Feb 5 22:48:24 boomhauer dovecot: child 8121 (imap) killed with signal 6
Feb 5 22:49:13 boomhauer dovecot: IMAP(username): file
message-header-parser.c: line 350 (message_parse_header_next):
assertion failed:
+(IS_LWSP(line-value[0]))
Feb 5 22:49:15 boomhauer dovecot: child 8171 (imap) killed with signal 6
I hope this is enough of a backtrace, let me know if not:
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB. Type show warranty for
details.
This GDB was configured as amd64-marcel-freebsd...
Core was generated by `imap'.
Program terminated with signal 6, Aborted.
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from
/usr/local/lib/dovecot/imap/lib01_acl_plugin.so...done.
Loaded symbols for /usr/local/lib/dovecot/imap/lib01_acl_plugin.so
Reading symbols from
/usr/local/lib/dovecot/imap/lib20_fts_plugin.so...done.
Loaded symbols for /usr/local/lib/dovecot/imap/lib20_fts_plugin.so
Reading symbols from
/usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so...done.
Loaded symbols for /usr/local/lib/dovecot/imap/lib21_fts_squat_plugin.so
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0 0x109d54ac in __res_pquery () from /lib/libc.so.6
Sorry, maybe bt full would be more helpful:
#0 0x109d54ac in __res_pquery () from /lib/libc.so.6
No symbol table info available.
#1 0x004a8e1b in i_panic (format=0x4d60e0 Tue) at failures.c:191
args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area =
0x7fffdf30, reg_save_area = 0x7fffde70}}
#2 0x004a0149 in message_parse_header_next (ctx=0x18c3f00,
hdr_r=0x7fffdff0) at message-header-parser.c:341
line = (struct message_header_line *) 0x18c3f00
msg = (const unsigned char *) 0x4511000
i = 46
size = 46
startpos = 0
colon_pos = 0
parse_size = 361
ret = 1
continued = true
continues = false
last_no_newline = true
last_crlf = false
no_newline = false
crlf_newline = false
__PRETTY_FUNCTION__ = message_parse_header_next
#3 0x004a11e9 in parse_content_type (ctx=0x600180, hdr=0x0) at
message-parser.c:441
parser = {data = 0x7fffe000 �\177, end = 0x1004a773d Address
0x1004a773d out of bounds, last_comment = 0x1e010}
key = 0x4a11e9 �H\215u�H\215}��\t,
value = 0x7fffe010 @\177
content_type = (string_t *) 0x1
#4 0x004a1bb6 in message_parser_init_from_parts
(parts=0x450f848, input=0x7fffe0c0, hdr_flags=32767, flags=16769184)
at message-parser.c:718
ctx = (struct message_parser_ctx *) 0xe040
#5 0x10c2355d in fts_mailbox_search_next_nonblock () from
/usr/local/lib/dovecot/imap/lib20_fts_plugin.so
No symbol table info available.
#6 0x0046b353 in mailbox_search_deinit (_ctx=0xc42200) at
mail-storage.c:624
ctx = (struct mail_search_context *) 0xd8f448
#7 0x00418f2f in imap_search_deinit (cmd=0x60c300, ctx=0x61d048)
at cmd-search.c:64
ret = 0
#8 0x0041916e in cmd_search_more (cmd=0x4b2216) at cmd-search.c:119
ctx = (struct imap_search_context *) 0x7fffe1e0
end_time = {tv_sec = 140737488347616, tv_usec = 4297070}
tryagain = false
ret = 3
#9 0x004b25ef in io_loop_handle_timeouts_real (ioloop=0x5ff240)
at ioloop.c:257
diff = 12939328
item = (struct priorityq_item *) 0xc57040
tv = {tv_sec = 0, tv_usec = 0}
tv_call = {tv_sec = 1202269828, tv_usec = 256050}
t_id = 3
#10 0x004b263d in io_loop_handle_timeouts_real (ioloop=0x5ff240)
at ioloop.c:267
item = (struct priorityq_item *) 0x20060a0e8
tv = {tv_sec = 140737488347728, tv_usec = 4924989}
tv_call = {tv_sec = 12939328, tv_usec = 6287936}
t_id = 0
#11 0x004b351e in io_loop_handler_run (ioloop=0x5ff240) at
ioloop-kqueue.c:123
ctx = (struct ioloop_handler_context *) 0x60a0e0
events = (struct kevent *) 0x60e000
event = (const struct kevent *) 0x60e000
tv = {tv_sec = 0, tv_usec = 0}
ts = {tv_sec = 0, tv_nsec = 0}
io = (struct io_file *) 0x5ff940
events_count = 4
t_id = 2
ret = 0
i = 0
__PRETTY_FUNCTION__ = io_loop_handler_run
#12 0x004b2690
Re: [Dovecot] Thunderbird Problem - What causes this?
Marc Perkel wrote: OK - I didn't know that I might be reporting something new. Here's some more details. I leave my computer on at might (Windows XP) and it's worse in the morning when I wake up. Thunderbird's checks the email every 1 minute and it's set up to check several IMAP folders. I'm running the latest Thunderbird release as will as the latest Dovecot (not the beta versions). In the morning it is as if it can't access dovecot at all. I get an hour glass as if it is waiting for something that's never going to respond. But if I shut down Thunderbird and restart it then everything works normal for a while. A summary of below: make sure you are actually still connected to your original session in dovecot before you suspect it: Try running a script to run 'netstat' frequently, say once a minute, and log the port numbers from the connection, example: TCPreinheitsgebot:3482mail.egr.msu.edu:993 ESTABLISHED TCPreinheitsgebot:3485mail.egr.msu.edu:993 ESTABLISHED You probably want to do this on the server side as well as the client side, just to check that if a connection is dropped, it happens on both at the same time (if not, something sounds odd!). I'm not sure if your problem happens if you only have thunderbird check the inbox, but I would expect at least one connection (inbox) to stay connected constantly without disruption. If the client side port number changes (eg. 3482), it got disconnected for some reason. For the most part, Thunderbird tries to make disconnections invisible to the user, which is nice when it works right, but misleading when something goes wrong, such as when thunderbird acts obstinant about loading messages but works when you restart. The reason I suggest this, and even if it might be a simpler case for you but similar symptoms/effect, I had a load balancer situation where connections would get dropped on the client side from a cause outside of the load balancer or server, and thunderbird would not always reconnect nicely when that happened. Literally the fault was not in the load balancer or server, but another server that stuck its nose in and effectively tried to steal the connection, making the client run into a brick wall and disconnect. The same happened to https connections. You might also try a different client such as mutt, which will make it painfully obvious if you get disconnected overnight, because it won't try to reconnect. Now on the other hand, if you can verify a single connection does stay open using the same source/dst port pair, you could start zeroing in on what is actually happening inside that connection, if it takes tcpdump on both side plus nonssl imap, etc.
Re: [Dovecot] v1.1.beta14 released (Compile Error)
Tomi Hakala wrote: Timo Sirainen wrote: On 20.1.2008, at 19.40, Jerry Yeager wrote: Undefined symbols: _posix_fallocate, referenced from: _file_set_size in liblib.a(file-set-size.o) Fixed: http://hg.dovecot.org/dovecot/rev/6c868e7fe7b2 You can also fix it by removing HAVE_POSIX_FALLOCATE from config.h Same problem and fix applies to Solaris 10. Tomi The patch did not work for me on FreeBSD 6.3, I had to edit config.h.in because my build procedure apparently overwrote config.h after I patched it.
Re: [Dovecot] Time just moved backwards error even with ntpd
On Sat, Jan 19, 2008 at 11:07:58AM -0500, Charles Marcus wrote: On 1/19/2008 Luigi Rosa wrote: On one hand we have Dovecot that kill itself until someone kicks it back on, on the other hand we have just some errors on the logfile. Well, IMHO at least there could be a configuration parameter that allows me to choose between a service that kills itself and some errors on a logfile and maybe a temporary service failure. Each SysAdmin could weight the consequences and choose the option he/she thinks is more appropriate to his/her server. Accurate time is CRITICAL for a mail server (and most any other server as well). I think the current behavior is reasonable. Fix the actual problem (server time being incorrect at boot time), and this is not an issue. Even without this problem, it may be wise to add a script to check often if dovecot is running, and start it if it is not. That way your downtime caused by this issue will be minimal. and/or add more servers so users won't notice any downtime except possibly being disconnected once.
[Dovecot] Plans for 1.1beta14 release?
I've been glancing at the hg changelogs, watching a number of basic fixes going in, including squat and uidlist fixes. Are there any plans for a beta14 release soon to increase exposure to these fixes, or is it in flux too much? My time lately to spend on dovecot has been sparse but I can definitely afford the time right now to roll betas into service because it is convenient. I have some tiny issues to track down that I noticed in the logs but if a new snapshot fixes them already, it saves both of us time.
Re: [Dovecot] Allowing tilde at start of mailbox names [listescape-plugin.c]
On Wed, Jul 25, 2007 at 08:28:52PM +0300, Timo Sirainen wrote: On 25.7.2007, at 19.13, pod wrote: TS It's mostly there just to make sure that ~/ or ~user/ can't be TS used to open mailboxes where you weren't supposed to have access TS to. I didn't think anyone would really want to have mailboxes TS beginning with ~. Do you really want them? (I don't think they TS were possible with UW-IMAP either?) Unfortunately yes it would be useful in my case to have them because of historical accident. How about if listescape plugin escaped it? That would make it possible to use them but still safe against accidental problems. For others too, I implemented a plugin for escaping mailbox names with Maildir so that it's possible to use '.' in mailbox names: http://dovecot.org/patches/1.0/listescape-plugin.c I am using the 1.1 version of this plugin. So far it seems to be working fine. Is there a chance of having it be an official plugin distributed with 1.1? That would make it fit into my build environment alot cleaner, as well as a FreeBSD port I intend to submit so others can build 1.1 easily. One way or another I plan to keep using listescape and will need to find the best way to keep it functioning alongside dovecot upgrades, even if that means compiling and installing it manually each time. Thanks.
Re: [Dovecot] namespaces, shared mailboxes
On Sun, Dec 02, 2007 at 09:21:19PM +0100, Alexander 'Leo' Bergolth wrote:
On 12/02/2007 08:59 PM, Benjamin R. Haskell wrote:
On Sun, 2 Dec 2007, Timo Sirainen wrote:
On Sat, 2007-12-01 at 23:59 +0100, Alexander 'Leo' Bergolth wrote:
I'd like to provide shared mailboxes with maildir that should appear
like 'Shared/mailboxname' to the clients.
Using the namespace configuration below, everything works fine, exept
one thing: Mail clients (only tested with Thunderbird) won't allow to
put mails directly in 'Shared/mailboxname' (the folder it is greyed
out). You currently have to create subfolders and put the mails in those
subfolders.
..
namespace shared {
separator = /
# hidden = yes
prefix = Shared/spamrep/
location =
maildir:/var/spool/mail/Shared/spamrep:CONTROL=~/Maildir/control/Shared/spamrep:INDEX=~/Mail\
dir/index/Shared/spamrep
I find it strange that this kind of configuration even appears to
work. :) If you somehow manage to get it working with v1.0, there's a
good chance that it won't work anymore with v1.1.
So if you want to access Shared/spamrep as a mailbox, the prefix must
be Shared/.
If you wanted users to have different views of what the mailboxes look
like, you could just change the location to be different for different
users. Create symlinks to the visible global mailboxes to those
locations.
Having multiple namespaces setup that way seems reasonable to me.
e.g.
/var/spool/mail/Shared/tech/...
/var/spool/mail/Shared/financial/...
where 'Shared/tech' would be visible to many, but 'Shared/financial'
would be visible only to execs/accountants. And both would show up under
a common 'Shared/' prefix to keep them distinct from user folders.
Yes, that's exactly my intention. There should be shared mailboxes for
departments or other groups of people. People having access to those
folders should be able to create subfolders but they should not be able
to add new folders directly under Shared.
I am doing something very similar. I am using a shell script to deliver
mail to various folders under a public namespace tree. I am also using
dovecot's ACLs because they seem to make permissions smoother than depending
on filesystem permissions, and the granularity on the permissions is much
smaller. For example, I don't allow users to create new folders in the
shared tree because the permissions will not be appropriate, and that also
stops mail clients from making thinks like Junk Mail or Trash.
It sounds like he just wants to be able to SELECT the root of a namespace.
I'd like to be able to deliver e.g. new mail to the technical support
directly to Shared/tech/ but if that's not possible, it isn't a big
problem, I'll just use Shared/tech/INCOMING.
I haven't quite tried that. I'm matching a structure from an old mail server
which had #shared/department/interest, and each department may or may not
have more than one interest with different users having access. They used
to have an In folder under that, but on dovecot I decided to deliver right
to the interest folder to avoid a visually larger folder tree.
Re: [Dovecot] Allowing tilde at start of mailbox names [listescape-plugin.c]
On Sun, Dec 02, 2007 at 03:01:51AM -0500, Adam McDougall wrote:
On Wed, Jul 25, 2007 at 08:28:52PM +0300, Timo Sirainen wrote:
On 25.7.2007, at 19.13, pod wrote:
TS It's mostly there just to make sure that ~/ or ~user/ can't be
TS used to open mailboxes where you weren't supposed to have access
TS to. I didn't think anyone would really want to have mailboxes
TS beginning with ~. Do you really want them? (I don't think they
TS were possible with UW-IMAP either?)
Unfortunately yes it would be useful in my case to have them because of
historical accident.
How about if listescape plugin escaped it? That would make it possible to
use them but still safe against accidental problems.
For others too, I implemented a plugin for escaping mailbox names with
Maildir so that it's possible to use '.' in mailbox names:
http://dovecot.org/patches/1.0/listescape-plugin.c
I am using the 1.1 version of this plugin. So far it seems to be working
fine.
Is there a chance of having it be an official plugin distributed with 1.1?
That would make it fit into my build environment alot cleaner, as well as
a FreeBSD port I intend to submit so others can build 1.1 easily. One way
or another I plan to keep using listescape and will need to find the best way
to keep it functioning alongside dovecot upgrades, even if that means
compiling
and installing it manually each time. Thanks.
Actually two issues came up. I think they are both involving folder listing.
- In squirrelmail, if I am subscribed to #shared/decs with the listescape
plugin enabled, the shared folders such as #shared/decs/network are greyed out.
#shared/decs is a public namespace and the only reason I would be subscribed to
it is for Outlook Express's sake or so Squirrelmail will show the shared folder
hierarchy (same reason for subscribing to #shared, but being subscribed to
#shared
doesn't seem to hurt anything). I think squirrelmail sees the subscription to
the folder (I can subscribe to unsubscribed shared folders too) but when it
tries
to LIST the folder, it gets nothing.
- In all programs, it seems like listing folders in my additional private hidden
namespaces comes up with nothing. a LIST mail/% a OK List completed.
If I try to access a folder under mail/ directly by typing the path into mutt,
it works.
with listescape:
---
a list dot.test
* LIST (\HasNoChildren) . dot.test
a OK List completed.
a LIST mail/
* LIST (\Noselect \HasChildren) / mail/
a OK List completed.
a LIST mail/%
a OK List completed.
(a LSUB * shows all the folders I am subscribed to, including
#shared/decs/network)
a LSUB #shared/%
* LSUB () / #shared/decs
a OK Lsub completed.
a LIST #shared/decs
* LIST (\Noselect \HasChildren) / #shared/decs
a OK List completed.
a LSUB #shared/decs/%
a OK Lsub completed.
a LIST #shared/decs/network
a OK List completed.
(for some reason in squirrelmail, if I am unsubscribed from #shared/decs with
the listescape
plugin enabled, the shared folders such as #shared/decs/network are not greyed
out and work,
but I cant seem to easily reproduce that result with imap commands.
Without listescape loaded:
---
a list dot\\2etest
* LIST (\HasNoChildren) . {10}
dot\2etest
a OK List completed.
a LIST mail/
* LIST (\Noselect \HasChildren) / mail/
a OK List completed.
a LIST mail/%
* LIST (\HasNoChildren) / mail/20041223
* LIST (\HasNoChildren) / mail/20050324
* LIST (\HasNoChildren) / mail/APA-MEA-DISCUSS
* LIST (\HasNoChildren) / mail/Drafts
* LIST (\HasNoChildren) / mail/IMAP
* LIST (\HasNoChildren) / mail/Junk E-mail
(etc, all the folders I expect)
a LSUB #shared/%
* LSUB () / #shared/decs
a OK Lsub completed.
a LIST #shared/decs
* LIST (\Noselect \HasChildren) / #shared/decs
a OK List completed.
a LSUB #shared/decs/%
* LSUB () / #shared/decs/unixadmin
* LSUB () / #shared/decs/support
* LSUB () / #shared/decs/security
* LSUB () / #shared/decs/printmaster
* LSUB () / #shared/decs/postmaster
* LSUB () / #shared/decs/pcadmin
* LSUB () / #shared/decs/network
* LSUB () / #shared/decs/linuxadmin
a OK Lsub completed.
a LIST #shared/decs/network
* LIST (\HasChildren) / #shared/decs/network
a OK List completed.
# 1.1.beta9: /usr/local/etc/dovecot.conf
ssl_cert_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem
ssl_key_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem
login_dir: /var/run/dovecot/login
login_executable: /usr/local/libexec/dovecot/imap-login
mail_max_userip_connections: 25
verbose_proctitle: yes
first_valid_uid: 1000
first_valid_gid: 1000
mail_extra_groups: postlocal
mail_location:
maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes
mmap_disable: yes
dotlock_use_excl: yes
mail_nfs_storage: yes
mail_nfs_index: yes
mail_plugins: acl fts fts_squat listescape
mail_log_max_lines_per_sec: 0
Re: [Dovecot] Zimbra benchmarking
On Fri, Nov 30, 2007 at 01:16:25PM +0200, Timo Sirainen wrote: On Fri, 2007-11-30 at 12:00 +0100, DINH Vi?t Ho? wrote: On Nov 30, 2007 11:38 AM, Timo Sirainen [EMAIL PROTECTED] wrote: Zimbra is apparently building full text search indexes while appending, so this test doesn't mean much until I can test Dovecot's performance with Squat indexing. Hello, in fact, I'm not that much convinced by full-text search index server side. We consider response time, server side full text search will include client-server round-trip. So that, for example, on etpanX, I do some local indexing on the imap folders, so that when the user does a search, it's given in a fraction of second even if the server is slow. I think Mail.app on Mac OS X is doing the same. Yes, and so do many other clients. But FTS indexes are useful for webmails, mobile clients and other clients that don't have a local cache. Squat indexes are going to work so that the first time you do a TEXT or BODY search the indexes are built for the mailbox. After that they most likely are updated when saving new mails (although maybe only with deliver, not with APPEND/COPY?). If user hasn't done any TEXT/BODY searches for a month or so, the indexes finally get deleted. Or that's my plan currently, those rules are easy to change. I (and other users here) have some mail folder archives that go back several years and a number of those folders won't change, I what I was thinking of doing was do a full text search over ALL of the mail I can possibly access, so I will be all set for future fast searches, which might be of interest only once a year. I was thinking after we get dovecot into full production for at least half or a whole year, evaluating the space used by indexes of any type and make a site specific purge script if desired. It sounds like the indexes, if not automatically deleted, will be worth the more or less permanent disk space for us. Sounds like it would be useful as a configurable option.
[Dovecot] quick question about fs quota overhead in plugin
Last night I enabled imap_quota so dovecot could report usage reported by disk quota. I don't intend to actually use the quota plugin to place any limits anytime soon though. How much overhead does this add to normal operations that allocate disk space? Ideally I'd like a situation where the only overhead is incurred when the user uses the mail client to specifically check their usage. Is that possible, and/or is there a better way to do this? If it does cause general overhead on the NFS filer, I could just accept it until/unless I feel it becomes a burden I cannot bear. mail_plugins = acl fts fts_squat quota imap_quota ... quota = fs ...
Re: [Dovecot] Users w/o acl access appear to be subscribed to public folders (1.1b8)
On Tue, Nov 20, 2007 at 10:20:49PM -0500, Adam McDougall wrote: I noticed this today, I had a user outside of our department test out dovecot. They were using squirrelmail and I noticed that dovecot thinks this user is subscribed to ALL public folders even though a dovecot ACL prevents all access. I'm pretty sure access is still denied. I was able to reproduce this with a guest account I added: l lsub #shared/decs/% * LSUB (\Noselect) / #shared/decs/linuxadmin * LSUB (\Noselect) / #shared/decs/jbossadmin * LSUB () / #shared/decs/support * LSUB () / #shared/decs/receipts * LSUB (\Noselect) / #shared/decs/pcadmin * LSUB () / #shared/decs/network * LSUB (\Noselect) / #shared/decs/printmaster * LSUB () / #shared/decs/postmaster * LSUB (\Noselect) / #shared/decs/unixadmin * LSUB () / #shared/decs/security * LSUB (\Noselect) / #shared/decs/webmaster l OK Lsub completed. This only seems to happen when the acl plugin is enabled. Without the acl plugin, these are not listed as subscriptions. After deleting /egr/mail/shared/decs/dovecot-acl-list and re-enabling the acl plugin, I get this: l lsub #shared/decs/% * LSUB () / #shared/decs/unixadmin * LSUB () / #shared/decs/support * LSUB () / #shared/decs/security * LSUB () / #shared/decs/printmaster * LSUB () / #shared/decs/postmaster * LSUB () / #shared/decs/pcadmin * LSUB () / #shared/decs/network * LSUB () / #shared/decs/linuxadmin * LSUB () / #shared/decs/webmaster * LSUB () / #shared/decs/jbossadmin l OK Lsub completed. Is it related, or is it different just because a new dovecot-acl-list got created by another user already (but is mode 700?) I found a workaround, if I add authenticated l to the top level acl in each namespace (currently only have one enabled) then users aren't force-subscribed to every public folder. It does however grant them the ability to subscribe to my empty top level fake folder which they have no permissions for anyway. This doesn't seem to reduce the level of access by any valid users.
[Dovecot] Users w/o acl access appear to be subscribed to public folders (1.1b8)
I noticed this today, I had a user outside of our department test out dovecot. They were using squirrelmail and I noticed that dovecot thinks this user is subscribed to ALL public folders even though a dovecot ACL prevents all access. I'm pretty sure access is still denied. I was able to reproduce this with a guest account I added: l lsub #shared/decs/% * LSUB (\Noselect) / #shared/decs/linuxadmin * LSUB (\Noselect) / #shared/decs/jbossadmin * LSUB () / #shared/decs/support * LSUB () / #shared/decs/receipts * LSUB (\Noselect) / #shared/decs/pcadmin * LSUB () / #shared/decs/network * LSUB (\Noselect) / #shared/decs/printmaster * LSUB () / #shared/decs/postmaster * LSUB (\Noselect) / #shared/decs/unixadmin * LSUB () / #shared/decs/security * LSUB (\Noselect) / #shared/decs/webmaster l OK Lsub completed. This only seems to happen when the acl plugin is enabled. Without the acl plugin, these are not listed as subscriptions. After deleting /egr/mail/shared/decs/dovecot-acl-list and re-enabling the acl plugin, I get this: l lsub #shared/decs/% * LSUB () / #shared/decs/unixadmin * LSUB () / #shared/decs/support * LSUB () / #shared/decs/security * LSUB () / #shared/decs/printmaster * LSUB () / #shared/decs/postmaster * LSUB () / #shared/decs/pcadmin * LSUB () / #shared/decs/network * LSUB () / #shared/decs/linuxadmin * LSUB () / #shared/decs/webmaster * LSUB () / #shared/decs/jbossadmin l OK Lsub completed. Is it related, or is it different just because a new dovecot-acl-list got created by another user already (but is mode 700?)
[Dovecot] crash with fts_squat on an identified email
I have some folders with a good amount of spam. While demonstrating full text search to a user, we found a folder that would crash dovecot while doing a fts. While splitting up the mailbox to narrow it down, I found there are a good number of messages in that folder that all made it crash in the same manner. I figured I'd narrow down a single one, get that fixed, then retest the rest. I also added to my todo list to try a full text search on as much email as I can find, to try to weed out any other problems (and see if fts is realistic to use in production). Nov 17 09:03:26 boomhauer dovecot: IMAP(mcdouga9): file message-decoder.c: line 201 (translation_buf_decode): assertion failed: (*size = skip) Nov 17 09:03:26 boomhauer dovecot: child 14668 (imap) killed with signal 6 Let me know if this is not reproducable with the attached tgz containing a maildir with one message, I could do a local gdb if needed. fts-bad-email.tgz Description: GNU Unix tar archive
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Mon, Oct 29, 2007 at 05:10:44PM -0400, Adam McDougall wrote: On Mon, Oct 29, 2007 at 10:49:34PM +0200, Timo Sirainen wrote: On Sun, 2007-10-28 at 15:37 -0400, Adam McDougall wrote: Oct 28 11:01:40 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.97159.dc6633e16f47011d) failed: Operation not permitted From the name, I can't even tell what its for, what dovecot-shared might be causing it, etc. I did some hunting because I was curious (I assume you would know right away) and its from lib/safe-mkstemp.c which dotlock uses. It's used for creating dovecot-acl-list. Hmm. Looks like there are now two places where Dovecot takes permissions from: dovecot-shared file and the directory where it exists. If you set g+s to the dir too this error goes away. I'll have to think a bit more what I'll do about this. Maybe dovecot-shared file just could go away and only the dir permissions would be used. That reminds me, I do think I put dovecot-shared in that directory just to satisfy 1.1. The group owner on /egr/mail/shared/decs/ is the group I am using to restrict entry into the directory (it is mode 770) so all of the users using folders inside are part of the group, so I'm not sure why the fchown would fail? Also just a note (if I remember correctly) the existance of dovecot-shared causes dovecot+acl to treat the folder flags as private unless the code is modified (I still use that hack locally) so its not just the permissions of an object (presently dovecot-shared) that have an affect. Its seems like this is alot of functionality to load onto just the existance and permissions of a file :) I did set g+s on /egr/mail/shared/decs/ and relogged in with thunderbird but got the error right away: Oct 29 17:02:55 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.65681.2a5ad23c6e8cd308) failed: Operation not permitted Lately I have been getting: Nov 17 08:24:36 hill dovecot: IMAP(mcdouga9): open(/egr/mail/shared/decs/temp.hill.80542.6d06d40810d76654) failed: Permission denied Nov 17 08:24:36 hill dovecot: IMAP(mcdouga9): safe_mkstemp(/egr/mail/shared/decs/temp.hill.80542.6d06d40810d76654) failed: Permission denied Nov 17 08:25:18 hill dovecot: IMAP(mcdouga9): open(/egr/mail/shared/decs/temp.hill.80542.1f3d606a7fa4a3cc) failed: Permission denied Nov 17 08:25:18 hill dovecot: IMAP(mcdouga9): safe_mkstemp(/egr/mail/shared/decs/temp.hill.80542.1f3d606a7fa4a3cc) failed: Permission denied Right now I have /egr/mail/shared/decs/ unwritable to users. Some of these errors still happen when the directory is writable, but I am also concerned if it will cause these temp files to be renamed to dovecot-acl-list, which I assume would conflict with other users if created by one user. Should I worry about these errors? Does it impair caching of folder lists or something? Thanks.
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Sat, Nov 17, 2007 at 12:14:37PM -0500, Adam McDougall wrote: On Mon, Oct 29, 2007 at 05:10:44PM -0400, Adam McDougall wrote: On Mon, Oct 29, 2007 at 10:49:34PM +0200, Timo Sirainen wrote: On Sun, 2007-10-28 at 15:37 -0400, Adam McDougall wrote: Oct 28 11:01:40 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.97159.dc6633e16f47011d) failed: Operation not permitted From the name, I can't even tell what its for, what dovecot-shared might be causing it, etc. I did some hunting because I was curious (I assume you would know right away) and its from lib/safe-mkstemp.c which dotlock uses. It's used for creating dovecot-acl-list. Hmm. Looks like there are now two places where Dovecot takes permissions from: dovecot-shared file and the directory where it exists. If you set g+s to the dir too this error goes away. I'll have to think a bit more what I'll do about this. Maybe dovecot-shared file just could go away and only the dir permissions would be used. That reminds me, I do think I put dovecot-shared in that directory just to satisfy 1.1. The group owner on /egr/mail/shared/decs/ is the group I am using to restrict entry into the directory (it is mode 770) so all of the users using folders inside are part of the group, so I'm not sure why the fchown would fail? Also just a note (if I remember correctly) the existance of dovecot-shared causes dovecot+acl to treat the folder flags as private unless the code is modified (I still use that hack locally) so its not just the permissions of an object (presently dovecot-shared) that have an affect. Its seems like this is alot of functionality to load onto just the existance and permissions of a file :) I did set g+s on /egr/mail/shared/decs/ and relogged in with thunderbird but got the error right away: Oct 29 17:02:55 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.65681.2a5ad23c6e8cd308) failed: Operation not permitted Lately I have been getting: Nov 17 08:24:36 hill dovecot: IMAP(mcdouga9): open(/egr/mail/shared/decs/temp.hill.80542.6d06d40810d76654) failed: Permission denied Nov 17 08:24:36 hill dovecot: IMAP(mcdouga9): safe_mkstemp(/egr/mail/shared/decs/temp.hill.80542.6d06d40810d76654) failed: Permission denied Nov 17 08:25:18 hill dovecot: IMAP(mcdouga9): open(/egr/mail/shared/decs/temp.hill.80542.1f3d606a7fa4a3cc) failed: Permission denied Nov 17 08:25:18 hill dovecot: IMAP(mcdouga9): safe_mkstemp(/egr/mail/shared/decs/temp.hill.80542.1f3d606a7fa4a3cc) failed: Permission denied Right now I have /egr/mail/shared/decs/ unwritable to users. Some of these errors still happen when the directory is writable, but I am also concerned if it will cause these temp files to be renamed to dovecot-acl-list, which I assume would conflict with other users if created by one user. Uhh, forget that part about being read only. I forgot it needs write so my shared deliver script can create new mailboxes. Should I worry about these errors? Does it impair caching of folder lists or something? Thanks.
[Dovecot] Path to public folder subscription file not created in 1.1b7?
I use CONTROL=%h/Maildir/dovecot/public/control/[namespace] for each of my public namespaces. A few nights ago, I noticed having a new user subscribe to public folders for the first time will silently fail, becuase the path up to the subscriptions file will not be created. I am pretty sure dovecot used to create it for me, but maybe that changed in a recent version, or maybe I did something to break it? Work around is to mkdir -p /home/[user]/Maildir/dovecot/public/control/[namespace] but I'd like to avoid having to do that. Once I create the directory and make sure it is writable, the client can subscribe to public folders in that namespace. If they do not have permission, it still silently fails. namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/%q/public/indexes/decs list: yes subscriptions: yes
[Dovecot] unlink_directory(....Trash.NewFolder) failed: Directory not empty
I noticed the following this week. No idea how long its been happening because I have not tested deleting folders hardly at all. Steps to reproduce: 1. Create a folder NewFolder, click on it in thunderbird so an index dir is created: ls -ldi Maildir/.NewFolder/ 1504062 drwx-- 5 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/.NewFolder/ ls -ldi Maildir/dovecot/boomhauer/private/indexes/.NewFolder/ 4198832 drwx-- 2 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/dovecot/boomhauer/private/indexes/.NewFolder/ ls -li Maildir/dovecot/boomhauer/private/indexes/.NewFolder/ total 0 4198833 -rw--- 1 mcdouga9 egrstaff 128 Nov 16 19:44 dovecot.index.log 2. Delete NewFolder which makes Thunderbird move both the folder under Trash, and seems to rename the .NewFolder index dir to .Trash.NewFolder (same inode #): ls -ldi Maildir/.Trash.NewFolder/ 1504062 drwx-- 5 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/.Trash.NewFolder/ ls -ldi Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ 4198832 drwx-- 2 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ ls -li Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ total 4 4198833 -rw--- 1 mcdouga9 egrstaff 128 Nov 16 19:44 dovecot.index.log 3. Right click on Trash folder in Thunderbird, choose Empty Trash: Nov 16 19:47:25 boomhauer dovecot: IMAP(mcdouga9): unlink_directory(/home/mcdouga9/Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder) failed: Directory not empty ls -ldi Maildir/.Trash.NewFolder/ 1504062 drwx-- 5 mcdouga9 egrstaff 4096 Nov 16 19:44 Maildir/.Trash.NewFolder/ ls -ldi Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ 4198832 drwx-- 2 mcdouga9 egrstaff 4096 Nov 16 19:47 Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ ls -li Maildir/dovecot/boomhauer/private/indexes/.Trash.NewFolder/ (sometimes has a .nfs.blahblah file, I think I failed to catch it quick enough this time) The folder dissapears from Thunderbird's view but stays on disk, and will reappear in Thunderbird if you collapse and re-expand the folder tree, or close and open thunderbird. Repeatable after more Empty Trash or directly trying to delete the folder in thunderbird from in Trash. # 1.1.beta8: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.egr.msu.edu.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/%q/private/indexes mmap_disable: yes dotlock_use_excl: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl fts fts_squat mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/%q/public/indexes/decs list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 fts: squat
Re: [Dovecot] New full text search indexer
On Thu, Nov 15, 2007 at 12:07:15AM +0900, Asheesh Laroia wrote:
On Wed, 14 Nov 2007, Daniel Watts wrote:
Timo - we were just having a conversation about how we might be able to
provide full body indexed search for our clients and I realised it might
be worth checking the Dovecot list to see if this has been done already...
And then I find this thread!
What is the latest? Searches in the headers work fantastically but any
search in a sizeable folder (1000's messages) often just time out.
FWIW in 1.1beta releases, you can enable the squat plugin and get
full-text indexed search already.
Timo's working on improvements, as you can see, and a replacement, but
don't forget that Squat is real and pretty fast.
To enable it, do:
1. In the protocol imap { section of the config, add this line to the
end:
mail_plugins = fts fts_squat
Do that just before the close curly brace.
2. If you have a plugin { section of the dovecot.conf, do 2(a).
Otherwise, do 2(b).
2a. In the plugin { section, add this line to the end:
fts = squat
2b. At the end of the file, write this:
plugin {
fts = squat
}
That should be all! SEARCH TEXT will be, shall we say, accelerated. (Note
that indexes have to be generated sometime, so the way things are now
they're generated at the first SEARCH TEXT.)
-- Asheesh.
Thanks for this list of steps, I've been intending to test it and was just about
getting ready to ask the same question. Your email would be mice content to
throw on the dovecot wiki under fts (currently empty)
Re: [Dovecot] namespace public folders
On Tue, Nov 13, 2007 at 05:03:37PM +0100, Markus Stumpf wrote: Hoi, On Tue, Nov 13, 2007 at 02:08:46PM +0300, Nikolay Shopik wrote: My question is can I do such setup with dovecot+postfix. Im really appreciate for simple examples how can i accomplish this kind setup if it possible :) AFAIK the Seen Flag is not maintained on a per user basis. Actually it depends. I've found in dovecot 1.0 and 1.1, if you have a dovecot-shared file in the public folder, it forces private flags such as read. The wiki has some information about dovecot-shared. I do the delivery to the public folders via the aliases mechanism and the safecat program http://jeenyus.net/~budney/linux/software/safecat.html which delivers messages from stdin to a Maildir specified on the cmdline. Along with the safecat comes a maildir sh-script that is a wrapper for safecat and makes things shorter. info: |/usr/local/bin/maildir /var/spool/pubfolders/.info contact: |/usr/local/bin/maildir /var/spool/pubfolders/.info sales: |/usr/local/bin/maildir /var/spool/pubfolders/.sales To work around permission problems I have made the safecat programm setuid dovecot user, executable for owner and group and gave it to the group postfix handles aliases deliveries with, e.g.: ---s--x--- 1 dovecotu mail 21024 Oct 19 14:26 /usr/local/bin/safecat HTH, \Maex -- Markus Stumpf
Re: [Dovecot] v1.1.beta8 released
On Sun, Nov 11, 2007 at 10:43:34AM -0800, Jeff Grossman wrote: Timo Sirainen wrote: http://dovecot.org/releases/1.1/beta/dovecot-1.1.beta8.tar.gz http://dovecot.org/releases/1.1/beta/dovecot-1.1.beta8.tar.gz.sig Changes since beta7: - Added a new raw mail storage backend which allows opening files/streams as single mail mailboxes. deliver uses this now instead of opening the incoming mail as a mbox. So deliver should be now faster and it doesn't anymore remove Dovecot's private mbox headers if you're not using mbox. - Cache file tracks message expunges again - All kinds of other fixes - Some optimizations I think we're getting closer to v1.1 RCs. I just installed beta 8 and everything appears to be running smoothly. Of course, my mail server is pretty basic. Jeff I also installed 1.1b8 yesterday and everything has been fine so far. Haven't found anything yet to request a patch for :) I even took a leap and migrated my 6 gigs of personal email to my server running 1.1b8 and am using it as my exclusive working mail set. My next task is to convince a few more heavy users here to do the same (I'm sure I won't need to twist their arm very hard). My email use today has been bliss, while hearing about co-workers complain about the old slow production server! hahahahaha.
Re: [Dovecot] assertion failed: (mailbox_list_is_valid_existing_name(_list, name))
On Sun, Nov 11, 2007 at 12:57:21AM -0500, Adam McDougall wrote: Do you need a gdb backtrace for this one? I'm not sure why all of a sudden this started happening, its probably due to me adding a folder somewhere? not sure. It happened when I went to open my folder subscriptions in thunderbird. It was taking longer than usual (call this phase 1) and I noticed this in the logs: (cant remember if this was phase 1 or 2 but the error is probably the same) Nov 11 00:38:38 boomhauer dovecot: imap-login: Login: user=mcdouga9, method=PLAIN, rip=208.53.102.126, lip=35.9.37.190, TLS Nov 11 00:38:39 boomhauer dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.boomhauer.3050.af3db3ac545170a7) failed: Operation not permitted Nov 11 00:38:44 boomhauer dovecot: IMAP(mcdouga9): file mailbox-list-maildir.c: line 186 (maildir_list_get_path): assertion failed: (mailbox_list_is_valid_existing_name(_list, name)) Nov 11 00:38:44 boomhauer dovecot: child 3050 (imap) killed with signal 6 Nov 11 00:38:45 boomhauer dovecot: imap-login: Login: user=mcdouga9, method=PLAIN, rip=208.53.102.126, lip=35.9.37.190, TLS Nov 11 00:38:45 boomhauer dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.boomhauer.3052.5538be060c2f9c7d) failed: Operation not permitted Nov 11 00:38:46 boomhauer dovecot: IMAP(mcdouga9): file mailbox-list-maildir.c: line 186 (maildir_list_get_path): assertion failed: (mailbox_list_is_valid_existing_name(_list, name)) Nov 11 00:38:46 boomhauer dovecot: child 3052 (imap) killed with signal 6 When I cancelled the subscription window, phase 2 is the same thing but happening more rapidly. I traced what tbird is doing and can reproduce it with: ? OK Logged in. a list #shared/decs/%/% Connection closed by foreign host. Doing a list on #shared/decs/% works though. I found why this happens. I had accidently created a maildir with a dot at the end of its name due to a bug in a script. Can be reproduced with mkdir /egr/mail/shared/decs/.foo. Crashes go away when I deleted the directory.
Re: [Dovecot] some mail accesses in dovecot 1.1 stall for a bit (in progress)
On Fri, Nov 09, 2007 at 10:06:00AM -0500, Adam McDougall wrote: [snip] pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6f3c) = 4 (0x4) pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6f74) = 4 (0x4) pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6fac) = 4 (0x4) pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6fe4) = 4 (0x4) pwrite(0xd,0x7fffcb44,0x4,0x0,0xc701c) = 4 (0x4) I did a little more checking to confirm that the filehandle being written to is dovecot.index.cache. Just to update, right now I am not seeing this issue, but I've made a number of changes this weekend and something may have affected it. I'll bring it up again if I see it, but still writing in chunks of 4 bytes seems suboptimal to me in any case but I won't care if it doesn't happen :)
[Dovecot] Request for variable unique to each server?
I've been looking for a variable I can use in my dovecot.conf within the INDEX= setting so I can have one index dir per imap server on NFS. I've been looking at http://wiki.dovecot.org/Variables but no variable seems to contain something like server hostname, and I've been hunting for the variable expansion code in dovecot but haven't been able to find it. %l (local IP) won't help me because when using our load balancer, %l is the same IP on all of my servers. %r is close to useful but having a local webmail server on each server means I'd end up with 127.0.0.1 being shared (also I would end up with one directory per unique user/clientIP combo). Is there a variable that holds the server hostname or something else unique per server, or could one be added, or could someone point me at the code? Thanks.
Re: [Dovecot] Outlook can't see namespace folders
On Sat, Nov 10, 2007 at 06:10:49PM +0200, Timo Sirainen wrote: On Fri, 2007-11-09 at 14:25 -0500, Adam McDougall wrote: sr6v SUBSCRIBE #shared sr6v NO [TRYCREATE] Mailbox doesn't exist: #shared same for #shared/decs. All of my public folders are under #shared/[namespace]/. Well, I didn't think anyone would want that. :) http://hg.dovecot.org/dovecot/rev/f7aea3542cce Thanks, now I can subscribe to #shared, but it still won't let me for #shared/decs. I verified this was new behavior by removing the patch.
Re: [Dovecot] Outlook can't see namespace folders
On Sat, Nov 10, 2007 at 07:21:21PM +0200, Timo Sirainen wrote: On Sat, 2007-11-10 at 11:46 -0500, Adam McDougall wrote: On Sat, Nov 10, 2007 at 06:10:49PM +0200, Timo Sirainen wrote: On Fri, 2007-11-09 at 14:25 -0500, Adam McDougall wrote: sr6v SUBSCRIBE #shared sr6v NO [TRYCREATE] Mailbox doesn't exist: #shared same for #shared/decs. All of my public folders are under #shared/[namespace]/. Well, I didn't think anyone would want that. :) http://hg.dovecot.org/dovecot/rev/f7aea3542cce Thanks, now I can subscribe to #shared, but it still won't let me for #shared/decs. I verified this was new behavior by removing the patch. Oh? Works with me. What kind of configuration does that namespace have? x namespace * NAMESPACE (( /)(#shared/decs/ /)) NIL NIL x OK Namespace completed. x subscribe #shared x OK Subscribe completed. x subscribe #shared/decs x OK Subscribe completed. My NAMESPACE reply looks a little different (NILs in different places) amd I tried unloading the acl plugin just to rule it out. ? OK Logged in. x namespace * NAMESPACE (( /)) NIL ((#shared/decs/ /)) x OK Namespace completed. x subscribe #shared x OK Subscribe completed. x subscribe #shared/decs x NO [TRYCREATE] Mailbox doesn't exist: #shared/decs x subscribe #shared/decs/unixadmin x OK Subscribe completed. Does this have anything to do with it?: x LIST #shared/ x OK List completed. x LIST #shared/decs/ * LIST (\Noselect \HasChildren) / #shared/decs/ x OK List completed. I tried making a brand new namespace called #shared/empty/ drwxrwx---2 postlocal postlocal 4096 Nov 10 13:36 empty but it behaved the same. # 1.1.beta7: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.pem disable_plaintext_auth: no login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/%q/private/indexes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/%q/public/indexes/decs list: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/empty/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/empty:INDEX=%h/Maildir/dovecot/%q/public/indexes/empty list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10
Re: [Dovecot] Request for variable unique to each server?
On Sat, Nov 10, 2007 at 05:30:08PM +0200, Timo Sirainen wrote: On Sat, 2007-11-10 at 09:12 -0500, Adam McDougall wrote: Is there a variable that holds the server hostname or something else unique per server, or could one be added, or could someone point me at the code? Thanks. Well, the code would in two places: src/auth/auth-request.c auth_request_get_var_expand_table() src/master/mail-process.c get_var_expand_table() You could add a new variable that uses my_hostname variable from hostpid.h. I'm not sure if I should add this to standard variables. These one letter variables are beginning to run out. Maybe v2.0 should have longer $variable names.. Thanks! Got that working as a local patch, which I am satisfied with if you don't add an official one. I just quickly picked a letter that looked unused (%q) so thats what it is when you see it in configs I post ;)
Re: [Dovecot] Outlook can't see namespace folders
On Sat, Nov 10, 2007 at 08:55:48PM +0200, Timo Sirainen wrote: On 10.11.2007, at 20.39, Adam McDougall wrote: x subscribe #shared/decs x NO [TRYCREATE] Mailbox doesn't exist: #shared/decs Maybe this then: http://hg.dovecot.org/dovecot/rev/ba74249c79f0 Yep, fantastic, thanks!
Re: [Dovecot] Various uidlist and index errors with 1.1 on NFS
On Sat, Nov 10, 2007 at 07:54:15PM +0200, Timo Sirainen wrote: On Tue, 2007-11-06 at 15:57 -0500, Adam McDougall wrote: Two nights ago I took a leap and extended my testing of dovecot 1.1 by replacing 1.0 for the approx 15 users I had on 1.0. At that time I also for the first time tried dovecot 1.1 in a load balanced 2 server configuration with indexes on NFS. I hadn't actually tested this myself. Would be nice if someone gave me access to a NFS test system, would be much easier to test and fix these problems. :) I'd be more than happy to give you access to some of my servers. I'll email you privately about that. If you think the corruption is resolvable, I'd be happy if it could be worked out to provide a better service to my users. What OS are you using on NFS clients? FreeBSD 6.2 Nov 4 23:32:23 boomhauer dovecot: IMAP(mcdouga9): Trying to close mailbox support.2005.01-Jan with open transactions Probably some error handling path forgot to free the transaction. Would be nice to know how to reproduce it. I agree it would be nice. If I see it again and find a way to isolate it, I will. Perhaps it is caused when a client gets disconnected for some reason. Could just be a symptom of index problems that could get resolved some other way. Nov 5 01:11:43 boomhauer dovecot: IMAP(mcdouga9): Corrupted transaction log file /home/mcdouga9/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.log: Unexpected garbage at EOF Either writing really left some partially written data there, or attribute cache flushing doesn't work. Nov 6 09:12:26 boomhauer dovecot: IMAP(walbyjon): /home/walbyjon/Maildir/dovecot/public/control/decs/.support/dovecot-uidlist: Duplicate file entry: 1189447198.M910150P10430.hill:2,RS Sources explain this as: /* This can happen if expunged file is moved back and the file was appended to uidlist. */ So that would mean that readdir() skipped some files and later found them again. But this shouldn't happen as long as Dovecot is the only one accessing the maildir, because it locks its syncs.. Nov 5 15:26:53 hill dovecot: IMAP(mcdouga9): file mail-index-sync-update.c: line 599: unreached I replaced this crash with a nice error message. :)
Re: [Dovecot] Various uidlist and index errors with 1.1 on NFS
On Sun, Nov 11, 2007 at 02:34:33AM +0200, Timo Sirainen wrote: I'm beginning to understand the problem. Or at least one of them. :) Dovecot is currently flushing attribute cache for files it wants to stat/open. But that's not enough. It also has to flush attribute cache for the directory, or it might stat/open a file that has already been unlinked (but which still has links somewhere in the filesystem so that it doesn't fail with ESTALE). dotlock_use_excl=yes seems to help somewhat with this problem for now. It also gives better performance. I'm assuming that only helps if I use dotlock locking, right? I've left it at the default locking method so far except for a brief period a few nights ago, when I tried dotlock but it seemed no better (although I neglected to try dotlock_use_excl).
Re: [Dovecot] Various uidlist and index errors with 1.1 on NFS
On Sun, Nov 11, 2007 at 06:19:21AM +0200, Timo Sirainen wrote: On Sat, 2007-11-10 at 19:51 -0500, Adam McDougall wrote: On Sun, Nov 11, 2007 at 02:34:33AM +0200, Timo Sirainen wrote: I'm beginning to understand the problem. Or at least one of them. :) Dovecot is currently flushing attribute cache for files it wants to stat/open. But that's not enough. It also has to flush attribute cache for the directory, or it might stat/open a file that has already been unlinked (but which still has links somewhere in the filesystem so that it doesn't fail with ESTALE). dotlock_use_excl=yes seems to help somewhat with this problem for now. It also gives better performance. I'm assuming that only helps if I use dotlock locking, right? No, maildir always uses dotlocks. This is getting difficult though. After many hours I've finally gotten it to work properly without indexes. But even that required a pretty evil hack. It looks like the only way I can get FreeBSD to flush its filename cache (or whatever it's called that maps filenames to inodes) is to call rmdir() to the directory and hope that it fails with ENOTEMPTY. So this can't be safely done if the directory may be empty, as is easily possible with Maildir/cur and Maildir/new directories.. Wow, scary, but at least finding one awful way to make it work is starting on the road to finding a less awful way :) If I had more time, I'd start looking around the kernel source myself, but I have to get the rest of this project off the ground. With indexes enabled it then starts giving errors immediately about transaction logs. I think I'll leave figuring that out for later. I'm guessing the read cache flushing code doesn't work properly either. No problem, thanks for looking into it. I think I will be satisfied for now with per-server index directories. Do you think it should be safe to have them on NFS as long as only one host accesses that index directory? (with possible, and likely multiple dovecot processes for one user on that host) Also, should I start using dotlock_use_excl anyway as long as it appears to work? I hope Linux supports cache flushes better..
Re: [Dovecot] some mail accesses in dovecot 1.1 stall for a bit (in progress)
On Thu, Nov 08, 2007 at 04:17:57PM -0500, Adam McDougall wrote:
On Thu, Nov 08, 2007 at 09:55:48PM +0200, Timo Sirainen wrote:
On Sun, 2007-11-04 at 11:28 -0500, Adam McDougall wrote:
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0
(0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0
(0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
..
This should fix it: http://hg.dovecot.org/dovecot/rev/de67ceff3199
I applied the patch to beta7, the IMAP client feels the same as before
the patch (several second stalls on each message) but kdump and truss
reveal different behavior from the process, lots of the following.
Going back to a previously selected message in the same folder is fast,
but not if I select another folder then come back to the first.
No errors in log except for a few dovecot: IMAP(mcdouga9):
fchown(/egr/mail/shared/decs/temp.boomhauer.49823.6b66a5e5f1b24d07)
failed: Operation not permitted which I doubt is part of the same
issue. Not every mail folder is slow, it seems like perhaps just some of my
archive folders with around 1000-3000 emails. I haven't established a
strong pattern yet. Let me know if you want me to try to. Thanks.
49823 imap RET pwrite 4
49823 imap CALL pwrite(0xd,0x7fffcb44,0x4,0,0xaf648)
49823 imap GIO fd 13 wrote 4 bytes
0x e89c 0800
||
49823 imap RET pwrite 4
49823 imap CALL pwrite(0xd,0x7fffcb44,0x4,0,0xaf680)
49823 imap GIO fd 13 wrote 4 bytes
0x b89e 0800
||
49823 imap RET pwrite 4
49823 imap CALL pwrite(0xd,0x7fffcb44,0x4,0,0xaf690)
49823 imap GIO fd 13 wrote 4 bytes
0x fc9f 0800
||
49823 imap RET pwrite 4
49823 imap CALL pwrite(0xd,0x7fffcb44,0x4,0,0xaf6a0)
49823 imap GIO fd 13 wrote 4 bytes
0x 5ca1 0800
|\...|
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6f3c) = 4 (0x4)
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6f74) = 4 (0x4)
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6fac) = 4 (0x4)
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6fe4) = 4 (0x4)
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc701c) = 4 (0x4)
I did a little more checking to confirm that the filehandle being written
to is dovecot.index.cache.
Re: [Dovecot] Outlook can't see namespace folders
On Sat, Oct 27, 2007 at 08:30:57PM +0300, Timo Sirainen wrote: On Fri, 2007-10-05 at 12:12 -0700, James wrote: Hi guys i understand that you can't subscribe to the Prefix names of namespace folders right now with dovecot. But for instance if i user outlook express and it can't see the prefix name then it can't see the subfolders of that prefix folder either. So even though it can detect and see the subfolders it won't display them cause it can't see the parent Is there any fix for this? it's also kind of annoying in thunderbird but at least you can still see the subfolders. Changed in v1.1: http://hg.dovecot.org/dovecot/rev/0511e301acbc There's no simple fix for v1.0 though. I tried this, with b6 and b7 which seem to have the patch, it doesn't seem to work? sr6v SUBSCRIBE #shared sr6v NO [TRYCREATE] Mailbox doesn't exist: #shared same for #shared/decs. All of my public folders are under #shared/[namespace]/.
Re: [Dovecot] some mail accesses in dovecot 1.1 stall for a bit (in progress)
On Thu, Nov 08, 2007 at 09:55:48PM +0200, Timo Sirainen wrote:
On Sun, 2007-11-04 at 11:28 -0500, Adam McDougall wrote:
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0
(0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0
(0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
..
This should fix it: http://hg.dovecot.org/dovecot/rev/de67ceff3199
I applied the patch to beta7, the IMAP client feels the same as before
the patch (several second stalls on each message) but kdump and truss
reveal different behavior from the process, lots of the following.
Going back to a previously selected message in the same folder is fast,
but not if I select another folder then come back to the first.
No errors in log except for a few dovecot: IMAP(mcdouga9):
fchown(/egr/mail/shared/decs/temp.boomhauer.49823.6b66a5e5f1b24d07)
failed: Operation not permitted which I doubt is part of the same
issue. Not every mail folder is slow, it seems like perhaps just some of my
archive folders with around 1000-3000 emails. I haven't established a
strong pattern yet. Let me know if you want me to try to. Thanks.
49823 imap RET pwrite 4
49823 imap CALL pwrite(0xd,0x7fffcb44,0x4,0,0xaf648)
49823 imap GIO fd 13 wrote 4 bytes
0x e89c 0800
||
49823 imap RET pwrite 4
49823 imap CALL pwrite(0xd,0x7fffcb44,0x4,0,0xaf680)
49823 imap GIO fd 13 wrote 4 bytes
0x b89e 0800
||
49823 imap RET pwrite 4
49823 imap CALL pwrite(0xd,0x7fffcb44,0x4,0,0xaf690)
49823 imap GIO fd 13 wrote 4 bytes
0x fc9f 0800
||
49823 imap RET pwrite 4
49823 imap CALL pwrite(0xd,0x7fffcb44,0x4,0,0xaf6a0)
49823 imap GIO fd 13 wrote 4 bytes
0x 5ca1 0800
|\...|
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6f3c) = 4 (0x4)
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6f74) = 4 (0x4)
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6fac) = 4 (0x4)
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc6fe4) = 4 (0x4)
pwrite(0xd,0x7fffcb44,0x4,0x0,0xc701c) = 4 (0x4)
Re: [Dovecot] Index corruption ?
On Fri, Nov 02, 2007 at 05:52:49PM +0200, Timo Sirainen wrote: On Fri, 2007-11-02 at 15:05 +0100, Geoffroy Desvernay wrote: We are using dovecot for some times (a little before 1.0), for IMAP and (a little for the moment) deliver. We have ~2 logins/day, Maildirs are over NFS (with indexes too, for the moment...) With more than one server accessing the same maildir at around the same time? See attribute cache problems (and the rest of the page as well) in http://wiki.dovecot.org/NFS I'll bite, I keep hearing mention of each user is assigned a specific computer which is used whenever possible and I'm willing to try it out so my indexes are safe. The part where I get stuck is the whenever possible. I've looked at using perdition and/or dovecot, but both seem only offer a single IP to connect to per user, and the only workarounds I've heard of is to have an additional process monitor the imap servers and modify the mappings when a server is down. Is there a simpler way to have a proxy attempt a list of servers for each user? I would just point it back at my hardware load balancer in essence, but for that to work, the server originating the connection cannot host the VIP of the target otherwise the connection will stay local since the VIP would be local (I don't have my traffic flow through the load balancer physically). I could put a bunch of every server but me VIPs in the LB, one for each server, but thats a messy hack. I've thought of a handful of different ways I could have the connections go through my planned 4 mail servers but none so far give me complete load balancing, automatic failover, and simple enough to make it worth trying to have only one copy of indexes. Anyone have any input? Thanks
[Dovecot] Various uidlist and index errors with 1.1 on NFS
Two nights ago I took a leap and extended my testing of dovecot 1.1 by replacing 1.0 for the approx 15 users I had on 1.0. At that time I also for the first time tried dovecot 1.1 in a load balanced 2 server configuration with indexes on NFS. I was hoping I did this right, using the mail_nfs params and 1.1 so fchown etc would flush the access cache, but I am getting a number of messages and errors that indicate things aren't happy. Attaching error logs. Some of the errors were caused by me running multiple mail clients including mutt, Thunderbird, Outlook Express, other users may be using one or more copies of thunderbird, or possibly other clients. Most commonly thunderbird. Any help or advice would be appreciated. My goal is to have shared indexes between servers since I will eventually have 4. I'd prefer to avoid maintaining some kind of user-server preference mapping in a proxy (haven't looked into it yet). Thanks. # 1.1.beta6: /usr/local/etc/dovecot.conf ssl_cert_file: /usr/local/etc/apache2/ssl/mail.pem ssl_key_file: /usr/local/etc/apache2/ssl/mail.pem login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login verbose_proctitle: yes first_valid_uid: 1000 first_valid_gid: 1000 mail_extra_groups: postlocal mail_location: maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot/private/control:INDEX=%h/Maildir/dovecot/private/indexes mmap_disable: yes mail_nfs_storage: yes mail_nfs_index: yes mail_plugins: acl mail_log_max_lines_per_sec: 0 imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: private separator: / prefix: mail/ hidden: yes subscriptions: yes namespace: type: private separator: / prefix: Mail/ hidden: yes subscriptions: yes namespace: type: public separator: / prefix: #shared/decs/ location: maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot/public/control/decs:INDEX=%h/Maildir/dovecot/public/indexes/decs list: yes subscriptions: yes auth default: passdb: driver: pam userdb: driver: passwd args: system_user= socket: type: listen client: path: /var/spool/postfix/private/auth mode: 384 user: postfix group: postfix plugin: acl: vfile:/usr/local/etc/dovecot-acls:cache_secs=10 List of errors: Trying to close mailbox with open transactions Corrupted transaction log file Unexpected garbage at EOF fscking index file Transaction log got desynced for index start_offset (3872) current sync_offset Fixed index file log_file_head_offset 3872 - 3680 Corrupted transaction log file record size too small Transaction log file marked corrupted file mail-index-sync-update.c: line 599: unreached Duplicate file entry: Corrupted transaction log file header update extends beyond record size Nov 4 22:17:03 boomhauer dovecot: Dovecot v1.1.beta6 starting up Nov 4 23:32:23 boomhauer dovecot: IMAP(mcdouga9): Trying to close mailbox support.2005.01-Jan with open transactions Nov 4 23:32:23 boomhauer dovecot: child 46161 (imap) killed with signal 6 Nov 5 01:11:43 boomhauer dovecot: IMAP(mcdouga9): Corrupted transaction log file /home/mcdouga9/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.log: Unexpected garbage at EOF Nov 5 01:13:59 boomhauer dovecot: IMAP(mcdouga9): Corrupted transaction log file /home/mcdouga9/Maildir/dovecot/private/indexes/.INBOX/dovecot.index.log: Unexpected garbage at EOF Nov 5 01:13:59 boomhauer dovecot: IMAP(mcdouga9): Disconnected: Internal error occurred. Refer to server log for more information. [2007-11-05 01:13:59] bytes=4127/4532482 Nov 5 01:15:43 boomhauer dovecot: IMAP(mcdouga9): fscking index file /home/mcdouga9/Maildir/dovecot/private/indexes/.INBOX/dovecot.index Nov 5 01:16:43 boomhauer dovecot: IMAP(mcdouga9): fscking index file /home/mcdouga9/Maildir/dovecot/private/indexes/.INBOX/dovecot.index Nov 5 01:17:43 boomhauer dovecot: IMAP(mcdouga9): Transaction log got desynced for index /home/mcdouga9/Maildir/dovecot/private/indexes/.INBOX/dovecot.index Nov 5 01:17:43 boomhauer dovecot: IMAP(mcdouga9): Disconnected: Mailbox is in inconsistent state, please relogin. bytes=4299/5007523 Nov 5 02:06:38 boomhauer dovecot: IMAP(mcdouga9): /home/mcdouga9/Maildir/dovecot/public/indexes/decs/.network/dovecot.index.log: start_offset (3872) current sync_offset (3680) Nov 5 02:06:38 boomhauer dovecot: IMAP(mcdouga9): fscking index file /home/mcdouga9/Maildir/dovecot/public/indexes/decs/.network/dovecot.index Nov 5 02:06:38 boomhauer dovecot: IMAP(mcdouga9): Fixed index file /home/mcdouga9/Maildir/dovecot/public/indexes/decs/.network/dovecot.index: log_file_head_offset 3872 - 3680 Nov 5 02:06:38 boomhauer dovecot: IMAP(mcdouga9): Fixed index file /home/mcdouga9/Maildir/dovecot/public/indexes/decs/.network/dovecot.index: log_file_tail_offset 3872 - 3680 Nov 5 02:06:38 boomhauer dovecot: IMAP(mcdouga9):
Re: [Dovecot] some mail accesses in dovecot 1.1 stall for a bit (in progress)
On Fri, Nov 02, 2007 at 12:12:05PM -0400, Adam McDougall wrote:
On Fri, Nov 02, 2007 at 06:04:12PM +0200, Timo Sirainen wrote:
On Mon, 2007-10-29 at 17:15 -0400, Adam McDougall wrote:
Sometimes when I login to dovecot and read a message from an imap maildir
folder,
the message contents do not load for at least 10 seconds, sometimes
closer to a
minute, but after that first message loads, the rest seem fine in that
mailbox.
The time seems to vary some, usually longer rather than shorter.
Getting a strace from what the imap process is doing at the time would
be useful. It sounds a bit like a stale dotlock, although Dovecot waits
for two minutes for that. http://hg.dovecot.org/dovecot/rev/6dd5df1c1ec9
fixes a stale dotlock overriding issue.
Agreed, I've wanted to get a trace since the first time I saw it, but I've
been
using dovecot 1.1 all week and hardly seen it. Its possible a patch improved
it
or there was some other cause, so its just something I'll watch out for and
try
to tackle as soon as it becomes reproducable. After some further preparation
this weekend I plan to expand my testing of 1.1 beyond just myself and one
server,
since I've been very happy with the current state of 1.1 this week versus
1.0.
I noticed some sluggish message body fetches today that I don't think I've seen
before.
Each time I fetch a message from this particular mailbox (network/2007/01-Jan
for
my own record) it takes a few seconds, and mutt makes it look like the fetch
was
slow. So did thunderbird. I didn't waste any time and tried to grab a truss
and kdump of the action asap (attached). Apparently strace is only available
on i386
for FreeBSD, but truss and kdump should be similar, depends what details you
are
looking for.
It looks like the message fetch actually reads in from disk fairly promptly
(you can see the whole spam message in the kdump output) but then it churns
away for a while with something like the below:
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
For now I plan to leave mutt with the current folder selected, incase something
I do makes the behavior go away.
Re: [Dovecot] some mail accesses in dovecot 1.1 stall for a bit (in progress)
On Sun, Nov 04, 2007 at 11:28:36AM -0500, Adam McDougall wrote:
I noticed some sluggish message body fetches today that I don't think I've
seen before.
Each time I fetch a message from this particular mailbox (network/2007/01-Jan
for
my own record) it takes a few seconds, and mutt makes it look like the fetch
was
slow. So did thunderbird. I didn't waste any time and tried to grab a
truss
and kdump of the action asap (attached). Apparently strace is only available
on i386
for FreeBSD, but truss and kdump should be similar, depends what details you
are
looking for.
It looks like the message fetch actually reads in from disk fairly promptly
(you can see the whole spam message in the kdump output) but then it churns
away for a while with something like the below:
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
fstat(13,{mode=-rw--- ,inode=5656735,size=817368,blksize=4096}) = 0 (0x0)
fchown(0xd,0x5321,0x)= 0 (0x0)
madvise(0x10649000,0x82000,0x4) = 0 (0x0)
For now I plan to leave mutt with the current folder selected, incase
something
I do makes the behavior go away.
Oops, forgot to mention this was with dovecot 1.1b6, no local patches.
Re: [Dovecot] Outlook can't see namespace folders
On Sat, Oct 27, 2007 at 08:30:57PM +0300, Timo Sirainen wrote: On Fri, 2007-10-05 at 12:12 -0700, James wrote: Hi guys i understand that you can't subscribe to the Prefix names of namespace folders right now with dovecot. But for instance if i user outlook express and it can't see the prefix name then it can't see the subfolders of that prefix folder either. So even though it can detect and see the subfolders it won't display them cause it can't see the parent Is there any fix for this? it's also kind of annoying in thunderbird but at least you can still see the subfolders. Changed in v1.1: http://hg.dovecot.org/dovecot/rev/0511e301acbc There's no simple fix for v1.0 though. I am glad to see a patch. For 1.0, I was concerned with the same thing, and it turns out that the way our helpdesk was setting up Outlook Express was the same as the workaround I was going to suggest: Setup a seperate Account/profile in OE, set the mail folder prefix to your namespace prefix, then that seperate account will have an Inbox plus all of the folders in that prefix will be available for subscription and stay there. Also, as an alternative (not a very good one) I think you can see the folders in the subscribe window and double click to see them, then OE shows the whole tree for just that session until logout.
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Mon, Oct 29, 2007 at 10:49:34PM +0200, Timo Sirainen wrote: On Sun, 2007-10-28 at 15:37 -0400, Adam McDougall wrote: Oct 28 11:01:40 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.97159.dc6633e16f47011d) failed: Operation not permitted From the name, I can't even tell what its for, what dovecot-shared might be causing it, etc. I did some hunting because I was curious (I assume you would know right away) and its from lib/safe-mkstemp.c which dotlock uses. It's used for creating dovecot-acl-list. Hmm. Looks like there are now two places where Dovecot takes permissions from: dovecot-shared file and the directory where it exists. If you set g+s to the dir too this error goes away. I'll have to think a bit more what I'll do about this. Maybe dovecot-shared file just could go away and only the dir permissions would be used. That reminds me, I do think I put dovecot-shared in that directory just to satisfy 1.1. The group owner on /egr/mail/shared/decs/ is the group I am using to restrict entry into the directory (it is mode 770) so all of the users using folders inside are part of the group, so I'm not sure why the fchown would fail? Also just a note (if I remember correctly) the existance of dovecot-shared causes dovecot+acl to treat the folder flags as private unless the code is modified (I still use that hack locally) so its not just the permissions of an object (presently dovecot-shared) that have an affect. Its seems like this is alot of functionality to load onto just the existance and permissions of a file :) I did set g+s on /egr/mail/shared/decs/ and relogged in with thunderbird but got the error right away: Oct 29 17:02:55 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.65681.2a5ad23c6e8cd308) failed: Operation not permitted
[Dovecot] some mail accesses in dovecot 1.1 stall for a bit (in progress)
I noticed this in an earlier beta and took a note, and I still see it sometimes in 1.1b4, but it is only occasional so I have not tracked down yet what dovecot is doing. Just a FYI for the list incase others see it and to say that its an issue I tend to track down that I did not see in 1.0, and its been low on my list until now. Sometimes when I login to dovecot and read a message from an imap maildir folder, the message contents do not load for at least 10 seconds, sometimes closer to a minute, but after that first message loads, the rest seem fine in that mailbox. The time seems to vary some, usually longer rather than shorter.
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Sun, Oct 28, 2007 at 02:06:49AM +0300, Timo Sirainen wrote:
On Sat, 2007-10-27 at 17:42 -0400, Adam McDougall wrote:
If you don't need the other groups in Dovecot you can get rid of them and
just have the process use the user's primary group and
mail_extra_groups. I
think this should work:
userdb passwd {
args = system_user=
}
Actually, yes I like this alot and put this change into production. I
was
planning on using some secondary groups to prevent filesystem access, but
I can accomplish the same protection easier with this and
mail_extra_groups.
Thanks! I didn't test yet that the secondary groups aren't loaded but I
will
sometime.
According to my logs, it seems it does not prevent the secondary groups.
I'd look at the code to see how it works, but I have to switch tasks and
may not work more with dovecot until tomorrow. I suppose if I cannot get
this to work, it sounds like I may be able to depend on the patch below.
Looks like it overrides the system_user with empty value and Dovecot
ends up calling initgroups(). I'm not sure what that does, if
anything. This fixes it: http://hg.dovecot.org/dovecot/rev/7f2501b3e993
Upon some further testing, this patch doesn't seem to do anything, because
for some reason 1.1 allows me to login when I am in too many groups, but
1.0 fails (this is where I saw the errors), and both versions seem to act the
same
with or without the patch. When I use mail_executable to run a shell script to
report group membership, on both servers I still see the full list when using
system_user= and mail_extra_groups but I don't see the group I set in
mail_extra_groups.
I'm not sure what to think, is there a good place to stick in some debugging?
With some recent permission changes I've
done (affects dovecot 1.0 as well), I get a good amount of these
fchown
errors
and I was thinking of muting them so they do not fill my log, since
they
are harmless
in my setup.
If these errors happen for index files Dovecot currently fallbacks to
using
in-memory indexes.
Oh. Ugh. That might explain why the indexes don't always seem to load.
For some reason I thought dovecot might print a message when it falls back
to in-memory indexes; would that be possible?
Possible yes, but I'm not sure if it's a good idea. Users with
filesystem quotas probably wouldn't want to see them. It's anyway done
silently only when write fails because of ENOSPC/EDQUOT, in other cases
some error is always logged.
Re: [Dovecot] do 1.1b4 assertion failed: (mailbox_list_is_valid_existing_name
On Sun, Oct 28, 2007 at 01:57:52AM +0300, Timo Sirainen wrote: On Sat, 2007-10-27 at 16:31 -0400, Adam McDougall wrote: Oct 27 16:03:27 gribble dovecot: IMAP(mcdouga9): file mailbox-list-maildir.c: line 186 (maildir_list_get_path): assertion failed: (mailbox_list_is_valid_existing_name(_list, name)) OK, this should fix a lot of things: http://hg.dovecot.org/dovecot/rev/e1fde9940f7e No crashes yet, folder access seems fine so far. Thanks! One bug with ACL plugin still is that if you have a foo/bar mailbox in a public namespace but no foo, LIST % doesn't show foo as placeholder mailbox. This happens with v1.0 too I think. I'll try to get that fixed soon. I've probably noticed this back when I moved some courier shared mailboxes to dovecot and created the placeholder mailboxes to satisfy the requirement. If later I don't need these folders it would be nice so I can clean up some directories and scripts, but I'll still need to be able to fake a subscription to them for the sake of some IMAP clients that display folders strangely when the entire path down to a subfolder is not subscribed. For example, squirrelmail can do shared folders, but if I have folders subscribed as below marked with *: * some-private-folder * some-private-folder/subfolder * another-private-folder folderA * folderA/In folderB * folderB/In Squirrelmail shows subscribed folders in tree form: some-private-folder subfolder another-private-folder In In I can definitely understand if this is felt to be something I should continue making folders for, just mentioning a particular need I would need still need to satisfy if these folders are not required by the acl plugin someday.
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Sat, Oct 27, 2007 at 03:52:28PM -0400, Adam McDougall wrote: Right now in my public folder permission scheme, the only thing I need dovecot-shared for (I think) is making client-added emails world-readable at least (currently actually mode 666). As long as the indexes are accessible by the user, I don't care what mode or group they are. How about this: http://hg.dovecot.org/dovecot/rev/0dd9b91fd52c I will roll that in and test alongside the next patch you sent. Thanks. I'm not sure this is working completely, it seems to help but I still notice some errors. I made sure the mount was remounted without nosuid but on second thought I don't think that actually matters since we are not executing dovecot-shared. I set sgid on all of my dovecot-shared files: -rw-rwSrw- 1 postlocal postlocal 0 May 15 17:30 /egr/mail/shared/decs/.receipts.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 May 15 17:30 /egr/mail/shared/decs/.support.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 12 11:03 /egr/mail/shared/decs/.jbossadmin.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Jun 25 15:52 /egr/mail/shared/decs/.network.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 May 15 17:30 /egr/mail/shared/decs/.postmaster.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 May 15 17:30 /egr/mail/shared/decs/.security.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 24 23:26 /egr/mail/shared/decs/.unixadmin.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 24 23:26 /egr/mail/shared/decs/.unixadmin.In-2007-10/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:46 /egr/mail/shared/decs/.unixadmin/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 24 21:48 /egr/mail/shared/decs/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:48 /egr/mail/shared/decs/.linuxadmin.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:48 /egr/mail/shared/decs/.linuxadmin/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:45 /egr/mail/shared/decs/.linuxadmin.In-2007-10/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:51 /egr/mail/shared/decs/.backups.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:51 /egr/mail/shared/decs/.backups/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:51 /egr/mail/shared/decs/.backups.In-2007-10/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:51 /egr/mail/shared/decs/.printmaster.In-2007-10/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:51 /egr/mail/shared/decs/.printmaster/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 25 00:52 /egr/mail/shared/decs/.printmaster.In/dovecot-shared -rw-rwSrw- 1 postlocal postlocal 0 Oct 24 23:26 /egr/mail/shared/decs/.unixadmin.Spam/dovecot-shared Oct 28 09:58:59 gribble dovecot: IMAP(mcdouga9): fchown(/home/mcdouga9/Maildir/dovecot11testing/public/control/decs/.postmaster.In/dovecot-keywords.lock) failed: Operation not permitted Other errors: Oct 28 09:48:12 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.73172.4c26d7ffde1d024c) failed: Operation not permitted Oct 28 09:52:49 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.73230.fba3c6533356d144) failed: Operation not permitted Oct 28 09:53:00 gribble dovecot: IMAP(mcdouga9): fchown(/home/mcdouga9/Maildir/dovecot11testing/public/control/decs/.postmaster.In/dovecot-keywords.lock) failed: Operation not permitted Oct 28 09:53:00 gribble dovecot: IMAP(mcdouga9): fchown(/home/mcdouga9/Maildir/dovecot11testing/public/control/decs/.postmaster.In/dovecot-uidlist.tmp) failed: Operation not permitted Oct 28 09:58:25 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.73350.d3836dd2d12c3731) failed: Operation not permitted
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Sun, Oct 28, 2007 at 03:57:22PM +0200, Timo Sirainen wrote:
On Sun, 2007-10-28 at 09:25 -0400, Adam McDougall wrote:
userdb passwd {
args = system_user=
}
This works only with v1.1. v1.0 just ignores it.
Hmm. I might be able to get by without this.
Looks like it overrides the system_user with empty value and Dovecot
ends up calling initgroups(). I'm not sure what that does, if
anything. This fixes it: http://hg.dovecot.org/dovecot/rev/7f2501b3e993
Upon some further testing, this patch doesn't seem to do anything, because
for some reason 1.1 allows me to login when I am in too many groups, but
1.0 fails (this is where I saw the errors), and both versions seem to act
the same
with or without the patch. When I use mail_executable to run a shell
script to
report group membership, on both servers I still see the full list when
using
system_user= and mail_extra_groups but I don't see the group I set in
mail_extra_groups.
I'm not sure what to think, is there a good place to stick in some
debugging?
Have you set mail_drop_priv_before_exec=yes? If not, it should still be
running as root in your mail_executable. If it's no, I'm not really
sure..
I have not changed it ever, dovecot -n does not report it.
auth_debug=yes at least shows what auth process sends to master. It
should show empty system_user.
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Sun, Oct 28, 2007 at 11:06:01AM -0400, Adam McDougall wrote: On Sun, Oct 28, 2007 at 04:54:12PM +0200, Timo Sirainen wrote: On Sun, 2007-10-28 at 10:08 -0400, Adam McDougall wrote: On Sat, Oct 27, 2007 at 03:52:28PM -0400, Adam McDougall wrote: Right now in my public folder permission scheme, the only thing I need dovecot-shared for (I think) is making client-added emails world-readable at least (currently actually mode 666). As long as the indexes are accessible by the user, I don't care what mode or group they are. How about this: http://hg.dovecot.org/dovecot/rev/0dd9b91fd52c I will roll that in and test alongside the next patch you sent. Thanks. I'm not sure this is working completely, it seems to help but I still notice some errors. I made sure the mount was remounted without nosuid but on second thought I don't think that actually matters since we are not executing dovecot-shared. I didn't notice that changed it only for index files. This should really make it work: http://hg.dovecot.org/dovecot/rev/b2c14c07dcb2 Applied to 1.1, I'll see how it goes. However I just logged in with it and saw one I've been seeing occasionally: Oct 28 11:01:40 gribble dovecot: IMAP(mcdouga9): fchown(/egr/mail/shared/decs/temp.gribble.97159.dc6633e16f47011d) failed: Operation not permitted From the name, I can't even tell what its for, what dovecot-shared might be causing it, etc. I did some hunting because I was curious (I assume you would know right away) and its from lib/safe-mkstemp.c which dotlock uses. In dovecot 1.1, whatever dotlock used doesn't look like it used fchown at all. I was wondering if fchown here in 1.1 has to do with NFS access cache flushing. I don't even understand why it is failing, its hard to inspect when its happening on a randomly named temporary file of course. I also backported http://hg.dovecot.org/dovecot/rev/0dd9b91fd52c and http://hg.dovecot.org/dovecot/rev/b2c14c07dcb2 to my 1.0 semi-production server, and so far I have not seen a single fchown error from it yet. I used to get one on almost every folder access. yay! Thanks for this feature :) I think right now alot of the problems I had with dovecot 1.1 have been patched, and other than the temp file error above I can forsee myself putting 1.1 into testing real soon by other users. Its looking pretty good.
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Sat, Oct 27, 2007 at 08:21:54PM +0300, Timo Sirainen wrote: On Sat, 2007-10-27 at 13:02 -0400, Adam McDougall wrote: I was trying to debug this with gdb, but I'm not sure how to make env MAIL=maildir:~/Maildir MAIL_PLUGINS=acl ACL=vfile gdb /tmp/imap understand the #shared/decs namespace from below so I can SELECT it. Easiest way to figure these kind of things is to set mail_executable=/tmp/dump.sh which contains: #!/bin/sh set /tmp/dovecot.env And run dovecot --exec-mail imap. Then either use that information or just run . /tmp/dovecot.env before gdb imap. Thanks, I'll use it next time If dovecot-shared exists in the folder I try to open, dovecot says: Oct 27 12:57:38 gribble dovecot: IMAP(mcdouga9): fchown() failed with file /home/mcdouga9/Maildir/dovecot11testing/public/indexes/decs/.unixadmin/dovecot.index.log: Operation not permitted Oct 27 12:57:38 gribble dovecot: child 55470 (imap) killed with signal 11 Fixed the crash: http://hg.dovecot.org/dovecot/rev/7373240c3d1d Agreed But the real problem is that dovecot-shared file is owned by a group that your imap process doesn't belong to. You probably want to add it to mail_extra_groups. I want to avoid adding a group to any user that logs in because some of them are in many groups already and it might push them over the limit that FreeBSD allows, then they cannot login at all. With some recent permission changes I've done (affects dovecot 1.0 as well), I get a good amount of these fchown errors and I was thinking of muting them so they do not fill my log, since they are harmless in my setup. Right now in my public folder permission scheme, the only thing I need dovecot-shared for (I think) is making client-added emails world-readable at least (currently actually mode 666). As long as the indexes are accessible by the user, I don't care what mode or group they are.
[Dovecot] dovecot 1.1b4 not listing public folder children
ACL plugin still disabled. The folders listed below definitely do have
Children.
telnet session problem example in 1.1b4:
* OK Dovecot ready.
? login mcdouga9 password
? OK Logged in.
9 LIST #shared/decs/%
* LIST (\HasNoChildren) / #shared/decs/support
* LIST (\HasNoChildren) / #shared/decs/receipts
* LIST (\HasNoChildren) / #shared/decs/network
* LIST (\HasNoChildren) / #shared/decs/postmaster
* LIST (\HasNoChildren) / #shared/decs/security
* LIST (\HasNoChildren) / #shared/decs/jbossadmin
* LIST (\HasNoChildren) / #shared/decs/unixadmin
truss shows:
(null)() = 0 (0x0)
gettimeofday({1193511545.540129},0x5b7f00) = 0 (0x0)
gettimeofday({1193511545.540216},0x0)= 0 (0x0)
kevent(6,{},0,{0x0,EVFILT_READ,0x0,0,0x1c,0x5b83c0},3,{9.999784000}) = 1 (0x1)
gettimeofday({1193511551.475630},0x5b7f00) = 0 (0x0)
break(0x5d) = 0 (0x0)
read(0,9 LIST #shared/decs/%\r\n,4096)= 28 (0x1c)
break(0x5d3000) = 0 (0x0)
break(0x5d4000) = 0 (0x0)
break(0x5d7000) = 0 (0x0)
stat(/home/mcdouga9/Maildir,{mode=drwxr-xr-x
,inode=98,size=16384,blksize=4096}) = 0 (0x0)
open(/home/mcdouga9/Maildir,O_NONBLOCK,00) = 7 (0x7)
fstat(7,{mode=drwxr-xr-x ,inode=98,size=16384,blksize=4096}) = 0 (0x0)
fcntl(7,F_SETFD,FD_CLOEXEC) = 0 (0x0)
__sysctl(0x7fffdf10,0x2,0x10acabf8,0x7fffdf08,0x0,0x0) = 0 (0x0)
fstatfs(0x7,0x7fffdf40) = 0 (0x0)
break(0x5d8000) = 0 (0x0)
getdirentries(0x7,0x5d7000,0x1000,0x5c8428) = 4096 (0x1000)
getdirentries(0x7,0x5d7000,0x1000,0x5c8428) = 4096 (0x1000)
getdirentries(0x7,0x5d7000,0x1000,0x5c8428) = 0 (0x0)
lseek(7,0x0,SEEK_SET)= 0 (0x0)
close(7) = 0 (0x0)
stat(/egr/mail/shared/decs,{mode=drwxrwx---
,inode=6962613,size=8192,blksize=4096}) = 0 (0x0)
open(/egr/mail/shared/decs,O_NONBLOCK,00) = 7 (0x7)
fstat(7,{mode=drwxrwx--- ,inode=6962613,size=8192,blksize=4096}) = 0 (0x0)
fcntl(7,F_SETFD,FD_CLOEXEC) = 0 (0x0)
fstatfs(0x7,0x7fffdf40) = 0 (0x0)
getdirentries(0x7,0x5d7000,0x1000,0x5c8428) = 3072 (0xc00)
getdirentries(0x7,0x5d7000,0x1000,0x5c8428) = 0 (0x0)
lseek(7,0x0,SEEK_SET)= 0 (0x0)
close(7) = 0 (0x0)
write(1,* LIST (\\HasNoChildren) / #s...,559) = 559 (0x22f)
gettimeofday({1193511551.479498},0x0)= 0 (0x0)
kevent(6,{},0,{},3,{4.060502000})= 0 (0x0)
gettimeofday({1193511555.540489},0x5b7f00) = 0 (0x0)
gettimeofday({1193511555.540561},0x0)= 0 (0x0)
Dovecot 1.0 on the same directory:
9 LIST #shared/decs/%
* LIST (\HasChildren) / #shared/decs/receipts
* LIST (\HasChildren) / #shared/decs/support
* LIST (\HasChildren) / #shared/decs/network
* LIST (\HasChildren) / #shared/decs/postmaster
* LIST (\HasChildren) / #shared/decs/security
* LIST (\HasChildren) / #shared/decs/unixadmin
* LIST (\HasChildren) / #shared/decs/linuxadmin
* LIST (\HasChildren) / #shared/decs/backups
* LIST (\HasChildren) / #shared/decs/printmaster
9 OK List completed.
# 1.1.beta4: /usr/local/etc/dovecot.conf
ssl_cert_file: /usr/local/etc/apache2/ssl/cert.pem
ssl_key_file: /usr/local/etc/apache2/ssl/cert.pem
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable: /usr/local/libexec/dovecot/imap-login
verbose_proctitle: yes
first_valid_uid: 1000
first_valid_gid: 0
mail_location:
maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot11testing/private/control:INDEX=%h/Maildir/dovecot11testing/private/indexes
mail_debug: yes
mmap_disable: yes
mail_nfs_storage: yes
mail_nfs_index: yes
mail_log_max_lines_per_sec: 0
imap_client_workarounds: delay-newmail netscape-eoh tb-extra-mailbox-sep
namespace:
type: public
separator: /
prefix: #shared/decs/
location:
maildir:/egr/mail/shared/decs:CONTROL=%h/Maildir/dovecot11testing/public/control/decs:INDEX=%h/Maildir/dovecot11testing/public/indexes/decs
list: yes
subscriptions: yes
namespace:
type: private
separator: /
prefix: mail/
location:
maildir:%h/Maildir:CONTROL=%h/Maildir/dovecot11testing/private/control:INDEX=%h/Maildir/dovecot11testing/private/indexes
hidden: yes
subscriptions: yes
namespace:
type: private
separator: /
inbox: yes
list: yes
subscriptions: yes
auth default:
mechanisms: plain login
passdb:
driver: pam
userdb:
driver: passwd
socket:
type: listen
client:
path: /var/spool/postfix/private/auth
mode: 384
user: postfix
group: postfix
plugin:
acl: vfile:/usr/local/etc/dovecot-acls
Re: [Dovecot] Segfault when opening a public folder, dovecot 1.1 beta4
On Sat, Oct 27, 2007 at 10:13:46PM +0300, Timo Sirainen wrote:
On 27.10.2007, at 21.51, Adam McDougall wrote:
But the real problem is that dovecot-shared file is owned by a group
that your imap process doesn't belong to. You probably want to add it to
mail_extra_groups.
I want to avoid adding a group to any user that logs in because some of
them
are in many groups already and it might push them over the limit that
FreeBSD
allows, then they cannot login at all.
If you don't need the other groups in Dovecot you can get rid of them and
just have the process use the user's primary group and mail_extra_groups. I
think this should work:
userdb passwd {
args = system_user=
}
Actually, yes I like this alot and put this change into production. I was
planning on using some secondary groups to prevent filesystem access, but
I can accomplish the same protection easier with this and mail_extra_groups.
Thanks! I didn't test yet that the secondary groups aren't loaded but I will
sometime.
With some recent permission changes I've
done (affects dovecot 1.0 as well), I get a good amount of these fchown
errors
and I was thinking of muting them so they do not fill my log, since they
are harmless
in my setup.
If these errors happen for index files Dovecot currently fallbacks to using
in-memory indexes.
Oh. Ugh. That might explain why the indexes don't always seem to load.
For some reason I thought dovecot might print a message when it falls back
to in-memory indexes; would that be possible?
Right now in my public folder permission scheme, the only thing I need
dovecot-shared for (I think) is making client-added emails world-readable
at least
(currently actually mode 666). As long as the indexes are accessible by
the user,
I don't care what mode or group they are.
How about this: http://hg.dovecot.org/dovecot/rev/0dd9b91fd52c
I will roll that in and test alongside the next patch you sent. Thanks.
[Dovecot] do 1.1b4 assertion failed: (mailbox_list_is_valid_existing_name
I think this is where I left off last weekend. Instead of this happening at
the base
of one of my shared mail namespaces, it happens when I try to select a top
level folder
inside the namespace, or try to directly select an entire path to a folder.
This only
happens when ACL is enabled. Let me know if I need to provide more. Thanks.
Oct 27 16:03:27 gribble dovecot: IMAP(mcdouga9): file mailbox-list-maildir.c:
line 186
(maildir_list_get_path): assertion failed:
(mailbox_list_is_valid_existing_name(_list, name))
Oct 27 16:03:27 gribble dovecot: child 51261 (imap) killed with signal 6
a0011 LIST #shared/decs/backups/%
IMAP(mcdouga9): Info: acl vfile: file /usr/local/etc/dovecot-acls//.DEFAULT not
found
IMAP(mcdouga9): Info: acl vfile: file /home/mcdouga9/Maildir/dovecot-acl not
found
IMAP(mcdouga9): Info: acl vfile: file /usr/local/etc/dovecot-acls//.DEFAULT not
found
IMAP(mcdouga9): Info: acl vfile: reading file /egr/mail/shared/decs/dovecot-acl
IMAP(mcdouga9): Panic: file mailbox-list-maildir.c: line 186
(maildir_list_get_path): assertion failed:
(mailbox_list_is_valid_existing_name(_list, name))
Program received signal SIGABRT, Aborted.
0x0008009c34bc in kill () from /lib/libc.so.6
(gdb) bt full
#0 0x0008009c34bc in kill () from /lib/libc.so.6
No symbol table info available.
#1 0x0008009c234d in abort () from /lib/libc.so.6
No symbol table info available.
#2 0x004a1414 in default_fatal_handler (type=LOG_TYPE_PANIC, status=0,
format=0x4bc930 file %s: line %d (%s): assertion failed: (%s),
args=0x7fffda50) at failures.c:177
backtrace = 0x5f38c0 \200T`
#3 0x004a167b in i_panic (format=0x4bc930 file %s: line %d (%s):
assertion failed: (%s))
at failures.c:211
args = {{gp_offset = 8, fp_offset = 48, overflow_arg_area =
0x7fffdb40,
reg_save_area = 0x7fffda80}}
#4 0x00429b07 in maildir_list_get_path (_list=0x603048, name=0x602900
#shared/decs/receipts,
type=MAILBOX_LIST_PATH_TYPE_MAILBOX) at mailbox-list-maildir.c:186
list = (struct maildir_mailbox_list *) 0x603048
__PRETTY_FUNCTION__ = maildir_list_get_path
#5 0x0048e4e1 in mailbox_list_get_path (list=0x603048, name=0x602900
#shared/decs/receipts,
type=MAILBOX_LIST_PATH_TYPE_MAILBOX) at mailbox-list.c:265
No locals.
#6 0x0048ce69 in mail_storage_get_mailbox_path (storage=0x5fdc48,
name=0x602900
#shared/decs/receipts,
is_file_r=0x7fffdc07) at mail-storage.c:389
No locals.
#7 0x000800b07720 in acl_backend_vfile_object_init () from
/usr/local/lib/dovecot/imap/lib01_acl_plugin.so
No symbol table info available.
#8 0x000800b08708 in acl_backend_vfile_acllist_rebuild () from
/usr/local/lib/dovecot/imap/lib01_acl_plugin.so
No symbol table info available.
#9 0x000800b08beb in acl_backend_vfile_acllist_refresh () from
/usr/local/lib/dovecot/imap/lib01_acl_plugin.so
No symbol table info available.
#10 0x000800b08ce2 in acl_backend_vfile_acllist_verify () from
/usr/local/lib/dovecot/imap/lib01_acl_plugin.so
No symbol table info available.
#11 0x000800b083a9 in acl_backend_vfile_object_refresh_cache ()
from /usr/local/lib/dovecot/imap/lib01_acl_plugin.so
No symbol table info available.
#12 0x000800b07546 in acl_backend_get_default_rights () from
/usr/local/lib/dovecot/imap/lib01_acl_plugin.so
No symbol table info available.
#13 0x000800b0a9ed in acl_mailbox_list_iter_init () from
/usr/local/lib/dovecot/imap/lib01_acl_plugin.so
No symbol table info available.
#14 0x0048e85f in mailbox_list_iter_init_multiple (list=0x603048,
patterns=0x5f4298,
flags=MAILBOX_LIST_ITER_RETURN_CHILDREN) at mailbox-list.c:357
__PRETTY_FUNCTION__ = mailbox_list_iter_init_multiple
#15 0x0041752c in list_namespace_init (ctx=0x60d0b0) at cmd-list.c:642
ns = (struct mail_namespace *) 0x5fd848
cur_ns_prefix = 0x5fd880 #shared/decs/
cur_ref = 0x5f4318
pat = (const char * const *) 0x5f4298
pattern = 0x611178 #shared/decs/backups/%
inbox_match = IMAP_MATCH_NO
used_patterns = {arr = {buffer = 0x5f4260, element_size = 8}, v =
0x5f4260, v_modifiable = 0x5f4260}
inboxcase = false
#16 0x00417738 in cmd_list_continue (cmd=0x60d048) at cmd-list.c:673
ctx = (struct cmd_list_context *) 0x60d0b0
ret = 1
#17 0x00417e1c in cmd_list_full (cmd=0x60d048, lsub=false) at
cmd-list.c:847
client = (struct client *) 0x602100
args = (const struct imap_arg *) 0x6110c0
arg = (const struct imap_arg *) 0x41bb68
ctx = (struct cmd_list_context *) 0x60d0b0
patterns = {arr = {buffer = 0x60d0f8, element_size = 8}, v = 0x60d0f8,
v_modifiable = 0x60d0f8}
pattern = 0x611178 #shared/decs/backups/%
patterns_strarr = (const char * const *) 0x60d130
#18 0x00417e8a in cmd_list (cmd=0x60d048) at cmd-list.c:862
No locals.
#19 0x0041b0b2 in
Re: [Dovecot] dovecot 1.1b4 not listing public folder children
On Sat, Oct 27, 2007 at 10:43:13PM +0300, Timo Sirainen wrote: On Sat, 2007-10-27 at 15:00 -0400, Adam McDougall wrote: ACL plugin still disabled. The folders listed below definitely do have Children. http://hg.dovecot.org/dovecot/rev/80419a82081f worked
