Re: FW: Re: mailbox-daemon
Hy! You might be right, but fsync() might not be needed. The program deals with the files as follows: - At the start, mailbox.base and mailbox.ext are opened for reading, then the program reads all the data, and closes the files. - 99.9% of the time, the program serves everything from memory. When a new folder is created, an existing is modified, etc, the program opens mailbox.ext for writing (append), appends a line to it, and closes the file. I think the file close should be equal to a sync. Am I right? And so, the "worst" that could happen (with a journalling fs), is that you lose the (new) last line of the file, that's one folder. By the way, since yesterday, the daemon is in live testing now on my system with 300'000 folders, and 4-5000 active users (daily). Cross your fingers ;) Version 0.2 is out, on the page http://opensource.prim.hu/mbdaemon/ (It currently handles only user. folders.) On 20-May-2001 Lawrence Greenfield wrote: I think this is really interesting work. I haven't looked at the implementation but have considered this basic idea several times; I think it's definitely worth the work. A random comment on your message: The mailbox.ext file is always "small", so writing to it should be fast, and you could even so sync()-s on it. (But rather, use a journalling filesystem.) All filesystems _require_ you to use fsync(). Journalling filesystems aren't magical: if you don't fsync() you have no guarantee that the data gets to disk. So please call fsync; otherwise, you're making potentially serious problems for people who use your code. Larry Date: Sun, 20 May 2001 18:17:48 +0200 (CEST) From: Noll Janos [EMAIL PROTECTED] However, mailbox.base is "fixed", it doesn't change at (daemon) runtime, however it could be updated daily or weeky. The mailbox.ext is something like a log file, it also contains actions, keys, values, on separate lines. The daemon writes changes to (the end) of this file at runtime, whenever a folder is created, deleted, or changed. The daemon currently handles only "user.*" folders, this was to save space, and this suits my needs. Any way, if the daemon crashes, no problem, as long as the mailbox.base and mailbox.ext files are ok. The mailbox.base is even safer, as it's read-only for the daemon. About the "write-caching" question: if you use cyrus IMAPd with a text mailboxes.db, inserting a line into it is always slow, because a new file is created (mailboxes.db.NEW), the new line is inserted into it, and then it's moved the "old" mailboxes.db. If you use DB3, that could be better, but even then, every imapd process must read and process some of the data. But back to my daemon, I must emphasise, that this is only an actual implementation of the "socket" mail-folder db protocol. The architecture is there: one could write a daemon, that uses SQL, a daemon, that does more caching. The picture: +---+ ++ | IMAPD + socket_db |--socket--| mailbox-daemon | +---+ ++ I defined a small "protocol" for the communication. An example: IMAPD wants to get the data for the folder "user.test12" The steps 1. IMAPd sends a message through the socket: "GET user.test12" 2. mailbox-daemon replies: "OK user.test12 0 default user.test12 lps" There are a few more functions (iterated GET, SET, REMOVE) that a daemon must implement, but that's all. Caching is easy to do, as every GET, SET, etc. gets to this daemon AND ONLY this daemon. I'll do some real testing over the weekend, and maybe I'll whip up some more documentation.
Re: A problem with sieve-vacation (supplement)
Hello Ken, please excuse my not having well read the vacation draft. The local hostname of lrp.rz.fh-ulm.de is configured as lrp, and a local message to rahlfs@lrp succeeds concerning vacation even without the :addresses options. Therefore: If I only want to use to vacation- reply messages sent to username@`bin/hostname` I do not have to use the :addresses option, but in all other cases. This is what the draft v03 says, I think (for Unix systems with hostname in the /bin directory). Dietmar Rahlfs
Re: FW: Re: mailbox-daemon
On Mon, 21 May 2001 10:45:14 +0200 (CEST), Noll Janos [EMAIL PROTECTED] (nj) writes: nj You might be right, but fsync() might not be needed. why gamble on something so crucial? this is inviting disaster. nj I think the file close should be equal to a sync. Am I right? not necessarily. nj And so, the worst that could happen (with a journalling fs), is that you njlose the (new) last line of the file, that's one folder. 1 or 100, what does it matter? it's still inaccurate data. then you have to rebuild the entire thing? nj By the way, since yesterday, the daemon is in live testing now on my system nj with 300'000 folders, and 4-5000 active users (daily). Cross your fingers ;) perhaps your users are the ones that should cross their fingers. personally, i don't know why berkeley db still couldn't be used for robustness of data. just having this daemon cache this data in memory should in itself be a significant performance boost. afterall, i'm pretty sure that's what netscape (iplanet) directory server does. -amos
IMAPD spool on Global Filesystem
Hello, Is there anybody who has used the spool directory of a Cyrus IMAPD on the Global Filesystem on Linux? I want to use two machines with Cyrus, sharing the same spool. Are there any locking or cache problems? Are there any performance issues? Thank you, Stefanita Vilcu -- Stefanita Valeriu Vilcu, [EMAIL PROTECTED] Dynamic Network Technologies, Romania Tel: +40-1-2106863 Fax: +40-1-3122745
Re: FW: Re: mailbox-daemon
On Mon, 21 May 2001 17:53:34 +0200 (CEST), Noll Janos [EMAIL PROTECTED] (nj) writes: nj So far, so good ;) that's good. nj To think of it, maybe Berkeley DB could be used. It might even be better... if nj you can specify it the amount of cache memory to use. just doing a quick look, the following might be useful: http://www.sleepycat.com/docs/ref/am_conf/cachesize.html#2 http://www.sleepycat.com/faq.html#A13 being able to use the db_stat might be kinda cool. hmmm... looks like a book on berkdb is about to be released. wonder what that'll be like -- Amos
Re: FW: Re: mailbox-daemon
Hy! On 21-May-2001 Amos Gouaux wrote: nj I think the file close should be equal to a sync. Am I right? not necessarily. Ok, I'll fix it. nj with 300'000 folders, and 4-5000 active users (daily). Cross your fingers ;) perhaps your users are the ones that should cross their fingers. So far, so good ;) personally, i don't know why berkeley db still couldn't be used for robustness of data. just having this daemon cache this data in memory should in itself be a significant performance boost. To think of it, maybe Berkeley DB could be used. It might even be better... if you can specify it the amount of cache memory to use. But hey, this is also what mb-daemon is about! The daemon side is like a plug-in, you can have a "text file daemon", a Berkeley DB daemon or an SQL-daemon, with modifications and compilation only needed on the daemon side, not on the IMAPd side. | Noll Janos [EMAIL PROTECTED] | http://www.johnzero.hu | | "Expect the unexpected!"| ICQ# 4547866 | Linux rulez! |
DBERROR?
I have been having some trouble configuring cyrus. Version 2.0.12. Firing up master, I get the following activity in syslog: May 21 15:55:51 animul master[17008]: process started May 21 15:55:51 animul ctl_mboxlist[17009]: running mboxlist recovery May 21 15:56:13 animul ctl_mboxlist[17009]: done running mboxlist recovery May 21 15:56:13 animul master[17008]: process 17010 exited, status 1 May 21 15:56:13 animul master[17008]: ready for work May 21 15:56:13 animul ctl_mboxlist[17011]: checkpointing mboxlist May 21 15:56:13 animul ctl_deliver[17012]: DBERROR: dbenv-open failed: No such file or directory I think the DBERROR is the important line. When I connect via imtest, I get the standard response, so nothing appears to be badly broken: animul:/opt/cyrus-imapd/cyrus-imapd-2.0.12/bin# ./imtest -m login localhost C: C01 CAPABILITY S: * OK animul.cs.uchicago.edu Cyrus IMAP4 v2.0.12 server ready S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT MULTIAPPEND SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES IDLE S: C01 OK Completed Password: Should I worry about the DBERROR? Also, I had to create /var/imap/socket and /var/imap/db and I didn't see instructions to do so in the docs (but I could have missed something, I suppose). Thanks. Brendan [EMAIL PROTECTED]
