switch from mailbox to maildir format
Hi, I want to switch from mailbox to maildir format. In INSTALL.maildir I read the instructions: begin text % maildirmake $HOME/Maildir % echo ./Maildir/ ~/.qmail Make sure you include the trailing slash on Maildir/. The system administrator can setup Maildir as the default for everybody by creating a maildir in the new-user template directory and replacing ./Mailbox with ./Maildir/ in /var/qmail/rc. -end text-- what's the new-user template directory? (where is it?) Thanks in advance Franco Vecchiato
Re: switch from mailbox to maildir format
Franco Vecchiato writes: what's the new-user template directory? (where is it?) Typically /etc/skel/, but every vendor seems to feel that it's crucial to reinvent every wheel, so there's no one definitive answer. man useradd (or perhaps it's adduser) will tell you more. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | Microsoft rivets everything. 521 Pleasant Valley Rd. | +1 315 268 1925 voice | Linux has some loose screws. Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | You own a screwdriver.
Re: switch from mailbox to maildir format
Hi, ---Original Message-- Mailing-List: contact [EMAIL PROTECTED]; run by ezmlm Precedence: bulk Date: Tue, 29 May 2001 12:43:49 +0200 Subject: switch from mailbox to maildir format From: Franco Vecchiato [EMAIL PROTECTED] To: [EMAIL PROTECTED] Content-Transfer-Encoding: 7bit Hi, I want to switch from mailbox to maildir format. In INSTALL.maildir I readthe instructions: begin text % maildirmake $HOME/Maildir % echo ./Maildir/ ~/.qmail Make sure you include the trailing slash on Maildir/. The system administrator can setup Maildir as the default for everybody by creating a maildir in the new-user template directory and replacing ./Mailbox with ./Maildir/ in /var/qmail/rc. -end text-- what's the new-user template directory? (where is it?) Like in RedHat it is /etc/skel directory where you can create Maildir, Maildir/new, Maildir/cur, Maildir/tmp ..and whatever directory and files you want to be created by default in user's home directory when ever you add new user Thanks in advance Franco Vecchiato Santosh Pasi
Re: Can I use qmail-pop3d for both mbox and maildir format?
Michael Cheung [EMAIL PROTECTED] wrote: Can I use qmail-pop3d for both mbox and maildir format? No, but SolidPOP handles both: http://solidpop3d.pld.org.pl/ -Dave
Can I use qmail-pop3d for both mbox and maildir format?
Hi; I move mail system from sendmail to qmail. so I won't to change the former mbox format. But I add a virtualdomain, it seems I have to use maildir format for the virtualdomain. Can I use qmail-pop3d for both mbox and maildir format? Thanks inadvance. Regards; Michael
Re: Conversion to Maildir Format
"Manvendra Bhangui" [EMAIL PROTECTED] wrote: How do I convert from someother mail format to maildir format used by qmail. That depends on the "someother mail format". Basically I am currently having mails being delivered as files with each mail being a single unix file. And these files are in which directory? How are they named? How do users read read their mail? Your mail agents will have to be able to handle maildir format mailboxes. If someone could tell me the logic on how to convert (say from a unix mail) to maildir, it would be helpful I'm not sure how helpful that would be since the details would be wrong for your situation. -Dave
Re: Conversion to Maildir Format
The format I am having currently enables me to convert to unix mbox format by just appending each mail messages into a single file. If you could tell me how to convert from unix mbox format to maildir, that would be helpful. I already have mail agents to handle maildir format Regards Manny - Original Message - From: Dave Sill [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, January 18, 2001 7:47 PM Subject: Re: Conversion to Maildir Format "Manvendra Bhangui" [EMAIL PROTECTED] wrote: How do I convert from someother mail format to maildir format used by qmail. That depends on the "someother mail format". Basically I am currently having mails being delivered as files with each mail being a single unix file. And these files are in which directory? How are they named? How do users read read their mail? Your mail agents will have to be able to handle maildir format mailboxes. If someone could tell me the logic on how to convert (say from a unix mail) to maildir, it would be helpful I'm not sure how helpful that would be since the details would be wrong for your situation. -Dave _ Do You Yahoo!? Get your free @yahoo.com address at http://mail.yahoo.com
Re: Conversion to Maildir Format
Manvendra Bhangui [EMAIL PROTECTED] wrote:
The format I am having currently enables me to convert to unix mbox format
by just appending each mail messages into a single file.
If you could tell me how to convert from unix mbox format to maildir, that
would be helpful. I already have mail agents to handle maildir format
One easy way is to use mutt -- create a Maildir (e.g. ~/Maildir/) using
maildirmake, then open your mbox file with mutt. Tag all messages in it
("T.*"), then save all tagged messages to the Maildir (";s" followed by
specifying the Maildir to save in).
Charles
--
---
Charles Cazabon[EMAIL PROTECTED]
GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/
Any opinions expressed are just that -- my opinions.
---
Re: Conversion to Maildir Format
"Manvendra Bhangui" [EMAIL PROTECTED] wrote: If you could tell me how to convert from unix mbox format to maildir, that would be helpful. I already have mail agents to handle maildir format http://www.qmail.org/mbox2maildir -Dave
Procmail and maildir format
I am in the process of moving from maildrop to procmail. The MTA on my system is Qmail, therfore I chose to use Maildir format for my mail. Procmail has been compiled to point to my spool at $HOME/Maildir The fetchmailrc is invoking procmail fine, but it does not write to the $HOME/Maildir/new directory. Instead it is dropping the mail in the literal $HOME/Maildir/ directory. The LOGFILE too is written to $HOME/Maildir/ directory. (0)subb3@caesar:~ = ll Maildir total 105 drwx-- 5 subb3users1024 Sep 30 17:42 ./ drwx--x--x 36 subb3users5120 Sep 30 17:42 ../ drwx-- 2 subb3users 54272 Sep 30 14:11 cur/ -rw--- 1 subb3users3364 Sep 30 17:42 msg.V9x -rw--- 1 subb3users2966 Sep 30 17:42 msg.W9x -rw--- 1 subb3users3917 Sep 30 17:42 msg.X9x -rw--- 1 subb3users1956 Sep 30 17:42 msg.Y9x drwx-- 2 subb3users 28672 Sep 30 17:40 new/ -rw--- 1 subb3users1842 Sep 30 17:42 procmail.log drwx-- 2 subb3users1024 Sep 30 17:40 tmp/ (0)subb3@caesar:~ = I suppose, I could change the drop location specifically to the "new" directory. Then, the syntax of the mail file is different. Procmail delivered file has - msg.V9x Maildrop delivered file has - 970384606.32149_0.myhost,\=3331 How can I make Procmail deliver mail in maildir format? The version of procmail on my system is v3.15 Procmail variables are as follows, PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:. MAILDIR=$HOME/Maildir DEFAULT=$MAILDIR LOGFILE=procmail.log LOCKFILE=$HOME/.lockmail VERBOSE=yes Thanks for any pointers or info. Subba Rao [EMAIL PROTECTED] http://pws.prserv.net/truemax/
Re: Procmail and maildir format
Subba Rao [EMAIL PROTECTED] wrote: I am in the process of moving from maildrop to procmail. The MTA on my system is Qmail, therfore I chose to use Maildir format for my mail. Procmail has been compiled to point to my spool at $HOME/Maildir [...] The fetchmailrc is invoking procmail fine, but it does not write to the $HOME/Maildir/new directory. Instead it is dropping the mail in the literal $HOME/Maildir/ directory. The LOGFILE too is written to $HOME/Maildir/ directory. [...] Procmail variables are as follows, PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:. MAILDIR=$HOME/Maildir The MAILDIR variable doesn't mean q qmail-style Maildir. Instead, it's more lie a chroot, which is what you're seeing. Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: Procmail and maildir format
On Sat, Sep 30, 2000 at 06:05:56PM +, Subba Rao wrote: I am in the process of moving from maildrop to procmail. The MTA on my system is Qmail, therfore I chose to use Maildir format for my mail. Procmail has been compiled to point to my spool at $HOME/Maildir The fetchmailrc is invoking procmail fine, but it does not write to the $HOME/Maildir/new directory. Instead it is dropping the mail in the literal $HOME/Maildir/ directory. The LOGFILE too is written to $HOME/Maildir/ directory. [snip...] How can I make Procmail deliver mail in maildir format? The version of procmail on my system is v3.15 You must specify a '/' at the end of the name of the maildir to alert procmail that your desired delivery mailbox is, in fact, a maildir. For example, my .procmailrc includes the following recipe to process messages to this list: :0 * ^TO_qmail Qmail/ Qmail is the name of the maildir in MAILDIR ($HOME/Mail, in my case). Procmail automatically delivers to the new/ directory within the specified mailbox. Procmail variables are as follows, PATH=$HOME/bin:/usr/bin:/usr/ucb:/bin:/usr/local/bin:. MAILDIR=$HOME/Maildir DEFAULT=$MAILDIR LOGFILE=procmail.log LOCKFILE=$HOME/.lockmail VERBOSE=yes You probably will need to change DEFAULT to say DEFAULT=$MAILDIR/ if you plan on some mail falling off the end of your processing and getting delivered to the default drop. Thanks for any pointers or info. Hope this helped. Subba Rao [EMAIL PROTECTED] http://pws.prserv.net/truemax/ Tim -- Timothy Legant [EMAIL PROTECTED]
Re: Procmail and maildir format
Quoted from Subba Rao: The MTA on my system is Qmail, therfore I chose to use Maildir format for my mail. I've never heard of an MTA called Qmail. Perhaps you meant qmail? (This distinction is noted in Dave Sill's ``life with qmail'', which every qmail user is advised to read.) The fetchmailrc is invoking procmail fine, but it does not write to the $HOME/Maildir/new directory. Instead it is dropping the mail in the literal $HOME/Maildir/ directory. The LOGFILE too is written to $HOME/Maildir/ directory. [...] MAILDIR=$HOME/Maildir DEFAULT=$MAILDIR As mentioned by someone else, MAILDIR in procmail actually specifies the default directory to use when a relative (not starting with a slash) filename is given as a folder. You must, nonetheless, end the folder name with a slash to tell procmail that you're delivering to a maildir. Try ``DEFAULT=./'', if you have ``MAILDIR=$HOME/Maildir'' as you have. Not tested, but should work. Hope it helps, ---Chris K. -- Chris, the Young One |_ If you can't afford a backup system, you can't Auckland, New Zealand |_ afford to have important data on your computer. http://cloud9.hedgee.com/ |_ ---Tracy R. Reed
Re: Mail clients and Maildir format
On Wed, May 24, 2000 at 10:43:40AM -0300, "Próspero, Esteban" wrote: Does anybody know if mail clients like Netscape Communicator or MS Outlook support the Maildir format? I haven't found out how... Communicator and Outlook communicate with your server via POP3, and don't know or care what kind of storage you use. As long as your POP3 daemon supports Maildir (and qmail-pop3d does), any POP3 client will work. Chris
RE: Mail clients and Maildir format
Thanks!! so please take a look at my second question!
Esteban
-Original Message-
From: Chris Johnson [SMTP:[EMAIL PROTECTED]]
Sent: Wednesday, May 24, 2000 10:45 AM
To: Próspero, Esteban"
Cc: '[EMAIL PROTECTED]'
Subject:Re: Mail clients and Maildir format
On Wed, May 24, 2000 at 10:43:40AM -0300, "Próspero, Esteban" wrote:
Does anybody know if mail clients like Netscape Communicator or MS
Outlook
support the Maildir format? I haven't found out how...
Communicator and Outlook communicate with your server via POP3, and
don't know
or care what kind of storage you use. As long as your POP3 daemon
supports
Maildir (and qmail-pop3d does), any POP3 client will work.
Chris
Re: Mail clients and Maildir format
Hello, i guess there are (at least) two answers with both same result, but one is funnier.. 1.) Netscape Communicator and MS Outlok boes do support Maildir format.. 2.) It (should) does no matter what client (MUA, mail user agent) your´e using, they all are »knocking« on the mail-servers door to ask for its mail if there is any. As far i know, if a mail client accesses this directories directly (like some unix clients do if youre on the same network) so this may cause problems. a.) am i under 10 typos per line (including this one) now? b.) am i right? c.) is there any life before breakfast? Regards from Stuttgart, Germany (not Arkansas nor Kansas) Anton Pirnat Ursprüngliche Nachricht Am 24.05.00, 14:43:40, schrieb "Próspero, Esteban" [EMAIL PROTECTED] zum Thema Mail clients and Maildir format: Does anybody know if mail clients like Netscape Communicator or MS Outlook support the Maildir format? I haven't found out how... Thanks in advance! Esteban Javier Próspero
Re: Mail clients and Maildir format
Hi, you may have a look on my remarks about SUSE Linux and QMAIL: http://www.fehcom.de/qmail_en.html cheers. eh. At 14:46 24.5.2000 GMT, Anton Pirnat wrote: Hello, i guess there are (at least) two answers with both same result, but one is funnier.. 1.) Netscape Communicator and MS Outlok boes do support Maildir format.. 2.) It (should) does no matter what client (MUA, mail user agent) your´e using, they all are »knocking« on the mail-servers door to ask for its mail if there is any. As far i know, if a mail client accesses this directories directly (like some unix clients do if youre on the same network) so this may cause problems. a.) am i under 10 typos per line (including this one) now? b.) am i right? c.) is there any life before breakfast? Regards from Stuttgart, Germany (not Arkansas nor Kansas) Anton Pirnat Ursprüngliche Nachricht Am 24.05.00, 14:43:40, schrieb "Próspero, Esteban" [EMAIL PROTECTED] zum Thema Mail clients and Maildir format: Does anybody know if mail clients like Netscape Communicator or MS Outlook support the Maildir format? I haven't found out how... Thanks in advance! Esteban Javier Próspero +---+ | fffhh http://www.fehcom.deDr. Erwin Hoffmann | | ff hh| | ffeee ccc ooomm mm mm Wiener Weg 8 | | fff ee ee hh hh cc oo oo mmm mm mm 50858 Koeln| | ff ee eee hh hh cc oo oo mm mm mm| | ff eee hh hh cc oo oo mm mm mm Tel 0221 484 4923 | | ff hh hhccc ooomm mm mm Fax 0221 484 4924 | +---+
qpop3d and /var/spool/mail in Maildir format
Hello, I have sendmail and procmail configured to deliver to /var/spool/mail/$LOGNAME in Maildir format. /var/spool/mail is an NFS mounted partition. I am trying to find a pop3 server to retrieve mail from above. qpop3d seems to be the one but it defaults to $HOME/Maildir, which doesn't exist. I've looked at the source, and it's a bit misleading in that that same error message will appear even when the options in inetd.conf are changed :). How can I configure qpop3d to work in this way. I've looked thru the archives and FAQ's and Qmail's site and have yet to find it. TIA -- -- -- -- -- Eric J. Honzay Network Connectivity Specialist Bennett Office Technologies 312 24th Ave. SW PO Box 978 Willmar, MN 56201 ICQ# 28991650 Phone: (320) 235-6425 Fax: (320) 231-1888 http://www.willmar.com
Maildir format
Sorry I have one more question, I am using The Maildir format to make it works with qmail-pop3d but I can't find any client like pine or elm to work with it, do I have to patch something?? THX Mikael
Re: Maildir format
At 9:41 PM +0200 4/17/00, quanta wrote: Sorry I have one more question, I am using The Maildir format to make it works with qmail-pop3d but I can't find any client like pine or elm to work with it, do I have to patch something?? Try mutt. http://www.mutt.org. THX Mikael -- -- Paul J. Schinder NASA Goddard Space Flight Center Code 693 [EMAIL PROTECTED]
Re: Maildir format
Sorry I have one more question, I am using The Maildir format to make it works with qmail-pop3d but I can't find any client like pine or elm to work with it, do I have to patch something?? Try mutt. http://www.mutt.org. Any pop3 mail client should work fine with the qmail-pop3d server, and I use a patched version of IMAP for things like Pine to connect to. Although you don't have all the capabilities that some mailers would give you by them bypassing the server, it does provide a transparen way for any standard pop3 or IMAP client to work. steve
Re: Maildir format
"quanta" [EMAIL PROTECTED] wrote: Sorry I have one more question, I am using The Maildir format to make it works with qmail-pop3d but I can't find any client like pine or elm to work with it, do I have to patch something?? Accessed via POP, the native mailbox format is tranparent and irrelevant. You only need Maildir-compatible MUA's for people *not* using POP. -Dave
Re: Maildir format
quanta wrote: Sorry I have one more question, I am using The Maildir format to make it works with qmail-pop3d but I can't find any client like pine or elm to work with it, do I have to pop3 clients don't have to know anything about maildir. I use netscape pop3 client and it just work fine. patch something?? THX Mikael
Re: Maildir format info
On Tue, Apr 11, 2000 at 08:40:54AM -0700, Duncan Watson wrote: On Tue, Apr 11, 2000 at 08:11:23AM -0600, Charles Cazabon wrote: Duncan Watson [EMAIL PROTECTED] wrote: I just started using maildirs with mutt and procmail. I am planning on writing a utility to allow me to search all of my maildir folders for mail matching certain regexps and then linking them into a result folder also a maildir that I could then browse with mutt. You might find that `find` and `egrep` can do what you want with little extra glue. Maildir format is so simple you don't have to worry about it -- one message per file under /new and /cur, ignore everything under /tmp. Very close to my intent. Find, regexps and python as glue. I may use tkinter as a front end for prettiness. Are you aware that mutt already has the ability to search for regexps? I believe you're restricted to searching a single folder at a time (maildir, mbox, or any other supported type) but aside from that it might be close enough to what you're trying to do. Walt Mankowski
Re: Maildir format info
On Wed, Apr 12, 2000 at 03:18:15PM -0400, Walt Mankowski wrote: On Tue, Apr 11, 2000 at 08:40:54AM -0700, Duncan Watson wrote: Very close to my intent. Find, regexps and python as glue. I may use tkinter as a front end for prettiness. Are you aware that mutt already has the ability to search for regexps? I believe you're restricted to searching a single folder at a time (maildir, mbox, or any other supported type) but aside from that it might be close enough to what you're trying to do. Actually that is part of it. I aggressively sort my mail into folders to keep a handle on it. This causes me some difficulty when I want to find something but can't remember who sent it. If I could remember the folder mutt would find it easily. So I thought "Hey what if I just found the messages I needed in the tree somewhere and linked them into one meta-folder called =search-results. Then I could view the results in mutt and do more specific searching on that folder" So my goal is simple just create an enternal script that searches the various folders I have and then link the result messages into a folder called search-results. I will then browse the folder with mutt. This has the added benefit of maintaining meta-information such as threading. Actually the task doesn't look to hard. I already have scripts to find messages which I use to identify likely folders to search. I just need to bit the bullet and link the results into a maildir. I hope this is more clear. /Duncan -- Duncan Watson nCube
Re: Maildir format info
Duncan Watson [EMAIL PROTECTED] wrote: I just started using maildirs with mutt and procmail. I am planning on writing a utility to allow me to search all of my maildir folders for mail matching certain regexps and then linking them into a result folder also a maildir that I could then browse with mutt. You might find that `find` and `egrep` can do what you want with little extra glue. Maildir format is so simple you don't have to worry about it -- one message per file under /new and /cur, ignore everything under /tmp. Relatively simple but I am looking for details on maildir format so that I dump my results without cheating. Does anyone have any ideas or pointers? djb's page on the Maildir format is here: http://cr.yp.to/proto/maildir.html Charles -- --- Charles Cazabon[EMAIL PROTECTED] GPL'ed software available at: http://www.qcc.sk.ca/~charlesc/software/ Any opinions expressed are just that -- my opinions. ---
Re: Maildir format info
On Tue, Apr 11, 2000 at 08:11:23AM -0600, Charles Cazabon wrote: Duncan Watson [EMAIL PROTECTED] wrote: I just started using maildirs with mutt and procmail. I am planning on writing a utility to allow me to search all of my maildir folders for mail matching certain regexps and then linking them into a result folder also a maildir that I could then browse with mutt. You might find that `find` and `egrep` can do what you want with little extra glue. Maildir format is so simple you don't have to worry about it -- one message per file under /new and /cur, ignore everything under /tmp. Very close to my intent. Find, regexps and python as glue. I may use tkinter as a front end for prettiness. Relatively simple but I am looking for details on maildir format so that I dump my results without cheating. Does anyone have any ideas or pointers? djb's page on the Maildir format is here: http://cr.yp.to/proto/maildir.html Excellent. The astute may note that I currently don't use qmail on my office box but I really love Maildir. /Duncan -- Duncan Watson nCube
Re: Maildir format info
On Tue, Apr 11, 2000 at 08:40:54AM -0700, Duncan Watson wrote: [snip] Excellent. The astute may note that I currently don't use qmail on my office box but I really love Maildir. One dutch ISP (cistron, the people who brought you Cistron radiusd) have implemented their own Maildir MDA, spawned from sendmail. You don't need qmail to do Maildir. I also know people who keep their ~/mail/ folders in Maildir, and have mutt save their incoming mail to a Maildir on quit. Greetz, Peter. -- Peter van Dijk - student/sysadmin/ircoper/madly in love/pretending coder | | 'C makes it easy to shoot yourself in the foot; | C++ makes it harder, but when you do it blows your whole leg off.' | Bjarne Stroustrup, Inventor of C++
Maildir format info
Hi all, I just started using maildirs with mutt and procmail. I am planning on writing a utility to allow me to search all of my maildir folders for mail matching certain regexps and then linking them into a result folder also a maildir that I could then browse with mutt. Relatively simple but I am looking for details on maildir format so that I dump my results without cheating. Does anyone have any ideas or pointers? Thanks, /Duncan -- Duncan Watson nCube
Re: Maildir format info
Duncan Watson [EMAIL PROTECTED] writes: I just started using maildirs with mutt and procmail. I am planning on writing a utility to allow me to search all of my maildir folders for mail matching certain regexps and then linking them into a result folder also a maildir that I could then browse with mutt. Relatively simple but I am looking for details on maildir format so that I dump my results without cheating. Does anyone have any ideas or pointers? man 5 maildir (comes with qmail) There is _very detailed_ step by step information on how maildirs are to be used by programs. -- Manfred Bartz
Re: Maildir format
On 15 Jan 2000, Russ Allbery wrote: RARussell Nelson [EMAIL PROTECTED] writes: RA RA Because every such installation I've ever seen has used NFS. I'm not RA talking about what's good, or what's right. I'm talking about what's RA possible to do tomorrow. Yes, it might be that a specialized mail RA transfer protocol would work better; convince Netapp to support it. RA RAUm, how do you think we're scaling our mail system right now? (And we RAdon't need Netapps to do it.) RA RAIf it's a matter of making it work tomorrow, I can do it faster with a RAfarm of lighter-weight servers than with anything based on NFS. NFS is RAthe complicated and expensive solution. If people are currently all doing RAit that way, it's either because they don't know better and are too used RAto throwing NFS at a problem or it's because they're using the *other* RAfeatures of Netapps (snapshots, reliability, etc.) and therefore are stuck RAhaving to use NFS whether they like it or not. To an extent, but keep in mind that the maildir/nfs solution is _simple_. Now, you can do things to make it more "robust" (read: complex) to add functionality, but if you want "simple and scalable", maildir and nfs is a clear winner. RAGive me a bunch of machines with DiskSuite and a couple of internal disks RAeach any day. Ack. Veritas :-). (nothing wrong with disksuite until you start into 0+1 and the like..., IMHO) -- --Matt Schnierle --mgs at stargate dot net --Stargate Industries, LLC --#include std/disclaimer.h --"It's not that simple."
Re: Maildir format
On Tue, Jan 18, 2000 at 03:26:49PM -, Anthony DeBoer wrote: Bruce Guenter [EMAIL PROTECTED] writes: The needs I am aware of include: - hierarchical multiple mailbox support That should include something that makes sense for a host that's behind a firewall and/or NAT and/or dynamic-IP dialup to authenticate and download mail for multiple users (to basically do what people try to do with fetchmail/multidrop or ETRN or other dodgy solutions nowadays). Would it be acceptable to ensure that each message has an accessable envelope sender address, or are you thinking of something else or more? -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
On Sat, Jan 15, 2000 at 12:25:09PM -0500, Greg Owen wrote: 1) Integrate support for some sort of calendaring. I've run both IMAP and Exchange based environments, and for all its faults, the integrated calendaring that Exchange does is extremely useful. None of the web-based calendaring systems I've seen compare. As much as I would like scheduling, this is an "mailbox" program, which handles email. I believe that proper support for scheduling would add too much to the protocol, given that this is supposed to be "simple". 3) Integrate configuration info onto the server, like ACAP or IMSP tries to do. All you should need to tell your client is you username, password, and server; it'll go do the rest to the best of its abilities. Do you have an URL for a specification of ACAP or IMSP? I've never heard of them, but what you've described is a good idea. -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
On Sun, Jan 16, 2000 at 10:35:09AM -0600, Tim Tsai wrote: What do you guys do for backup's? Do you put two NIC cards in each server and maintain a separate network for that? We actually use a couple Storagetek 9710 tape libraries. 10 DLT 7000 drives and something like 800 slots for tapes. We direct attach a couple drives to each big machine that needs to be backed up. Veritas Netbackup coordinates the actions of the drives and the robot and actually performs the backup. Smaller machines we do back up over the network although usually on the same NIC and network we use for regular operations. This may be overkill for your situation depending on how much data you have to back up. If an autochanging tape library is out of your league direct attached DLT is still the way to go. -- Tracy Reed http://www.ultraviolet.org "Wherever they burn books they will also, in the end, burn human beings." -Heinrich Heine
RE: Maildir format
Warning: opinions, little to do with qmail or maildir. Bruce Guenter wrote: On Sat, Jan 15, 2000 at 12:25:09PM -0500, Greg Owen wrote: 1) Integrate support for some sort of calendaring. I've run both IMAP and Exchange based environments, and for all its faults, the integrated calendaring that Exchange does is extremely useful. None of the web-based calendaring systems I've seen compare. As much as I would like scheduling, this is an "mailbox" program, which handles email. I believe that proper support for scheduling would add too much to the protocol, given that this is supposed to be "simple". True. I was attacking the question not from the "what simple things can we add" POV, but the "if you were designing a replacement, what needs would you try to fulfill?" A good calendaring system requires that users receive requests for meetings and can answer them, and have trouble screwing them up (i.e. putting the metainfo in the subject line is easy to screw up). Email is the ideal medium for this communication, because it meshes well with our existing work patterns. However, as I see it, there are three types of calendar systems: 1) Those that integrate so closely with email that the user can't screw it up 2) Those that can't integrate closely enough with email, and users screw it up 3) Those that don't integrate with email, and are hard to use. I don't expect there is much sympathy for my sort of "big system" thinking on this list. But there are cases where throwing more into the pot results in a win. IMAP and SMTP are an example - IMAP is a mailbox access protocol only. It does not support sending mail. How much of the traffic on this list deals with allowing mail sending without relaying? One could conceivably have looked at SMTP when writing the IMAP spec and said, "Gee, the lack of authentication in SMTP is a real pain in the ass. Let's add a POST command that allows insertion of mail from an authenticated IMAP user, then we don't have to worry about relaying issues in SMTP." Reinventing the wheel? Maybe. But sometimes you should consider how well the wheel works on today's vehicles and roads. 3) Integrate configuration info onto the server, like ACAP or IMSP tries to do. All you should need to tell your client is you username, password, and server; it'll go do the rest to the best of its abilities. Do you have an URL for a specification of ACAP or IMSP? I've never heard of them, but what you've described is a good idea. A good page for ACAP is http://asg.web.cmu.edu/acap/. ACAP is derived from IMSP; there's an RFC for the latter at http://asg.web.cmu.edu/cyrus/rfc/imsp.html. I'm not sure how far ACAP is moving forward. Presumably that'll depend on whether or not people start trying to add this sort of functionality into directory services as they start blooming. -- gowen -- Greg Owen -- [EMAIL PROTECTED]
Re: Maildir format
On Wed, Jan 19, 2000 at 01:22:11PM -0500, Greg Owen wrote: Warning: opinions, little to do with qmail or maildir. Indeed. I should start up a list just to discuss this. As much as I would like scheduling, this is an "mailbox" program, which handles email. I believe that proper support for scheduling would add too much to the protocol, given that this is supposed to be "simple". True. I was attacking the question not from the "what simple things can we add" POV, but the "if you were designing a replacement, what needs would you try to fulfill?" That is also my thinking, but my key phrase is "desigining a *simple* replacement". A good calendaring system requires that users receive requests for meetings and can answer them, and have trouble screwing them up (i.e. putting the metainfo in the subject line is easy to screw up). Email is the ideal medium for this communication, because it meshes well with our existing work patterns. OK, then, how do you see it integrating with email? A good page for ACAP is http://asg.web.cmu.edu/acap/. ACAP is derived from IMSP; there's an RFC for the latter at http://asg.web.cmu.edu/cyrus/rfc/imsp.html. So, then, if there is a defined spec for ACAP, that defines a link protocol and everything, why should it be added to the mailbox protocol? -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
On Wed, Jan 19, 2000 at 12:36:05PM -0600, Bruce Guenter wrote: A good calendaring system requires that users receive requests for meetings and can answer them, and have trouble screwing them up (i.e. putting the metainfo in the subject line is easy to screw up). Email is the ideal medium for this communication, because it meshes well with our existing work patterns. OK, then, how do you see it integrating with email? Let me clarify some of my questions: - Would there be a seperate part of the protocol designed to support calendaring, or should the events be presented as email messages? - Would the calendar be a seperate mailbox of a special type? - How would multiple calendars be dealt with? - Would the events be plain email messages or something else? - How would the events be transmitted to other users? - Would the TA (transfer agent) or the UA (user agent) be responsible for coordinating responses to calendar events (acknowledgements and rejections)? -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
Bruce Guenter [EMAIL PROTECTED] writes: On Tue, Jan 18, 2000 at 03:26:49PM -, Anthony DeBoer wrote: [ protocol wishlist ] That should include something that makes sense for a host that's behind a firewall and/or NAT and/or dynamic-IP dialup to authenticate and download mail for multiple users (to basically do what people try to do with fetchmail/multidrop or ETRN or other dodgy solutions nowadays). Would it be acceptable to ensure that each message has an accessable envelope sender address, or are you thinking of something else or more? You'd want the envelope recipient, actually, not the sender. The protocol should allow for a single piece of mail to have multiple envelope recipients; if you're trying to do an RFC-level protocol and get it used by other MTA communities, you have to allow for the ones that don't split deliveries. You would want to specify that the client mailhost strip other-envelope-recipient information, in case another recipient was Bcc'ed. Considering, in fact, that people sending the same PowerPoint or AVI to all the folks in an office account for a big chunk of bandwidth, there might be motivation for the queue-handling software to identify multiple queue entries that were in fact the same message, and merge them to a single copy with multiple recipients. Whether that would be a win would hinge largely on the popularity of big attachments at a given host and the size of their pipe. Note that the need for this would be pretty much MTA-independent, since the remote host that gave you the large message to multiple recipients might be running qmail. :-) I guess an existing protocol that does most of what's wanted would be UUCP, but nobody wants non-IP solutions anymore. Maybe you could use UUCP's IP transport mode, though. :-) -- Anthony DeBoer [EMAIL PROTECTED]
RE: Maildir format
On Wed, Jan 19, 2000 at 01:22:11PM -0500, Greg Owen wrote: Warning: opinions, little to do with qmail or maildir. Indeed. I should start up a list just to discuss this. Bruce, If you start up such a list, let me know. You are asking good questions, questions which force me to say, "I don't know." I was halfway through a response when I realized that I was doing too much justification of poorly-thought-through opinions, so I'm backing off for now to go reconsider. In specific, the question to ask is really "Where the seperate parts of the whole exist as standards, do products that correctly integrate them exist? If not, why not?" I want to go look at the state of calendaring standards before I shove my foot much deeper into my mouth. -- gowen -- Greg Owen -- [EMAIL PROTECTED]
Re: Maildir format
On Wed, Jan 19, 2000 at 10:53:09AM -0600, Bruce Guenter wrote: Do you have an URL for a specification of ACAP or IMSP? I've never heard of them, but what you've described is a good idea. Actually, the ACAP chapter of O'Reillys "Programming Internet Email" (ISBN 1-56592-479-7) is free! Read it here: http://www.oreilly.com/catalog/progintemail/chapter/ch12.html The protocol: [ACAP] Newman, C. and J. Myers, "ACAP -- Application Configuration Access Protocol", RFC 2244, November 1997. http://rfc2244.x42.com/ -- by far the easiest URL to rfc:s to remember. /magnus -- \ / ASCII Ribbon Campaign - Say NO to HTML in email and news x
Re: Crispin v. Bernstein (was Re: Maildir format)
A few people have responded to my earlier query/rant/whatever about writing high-quality software. A couple of these have asked me to keep them notified about what I learn, if possible. Another identified the starting-point of a resource. I put up a web page on my site on this topic, which I intend to maintain as a repository for resources and other stuff discovered during my (not quite full-time) search for how to write high-quality software: http://world.std.com/~burley/quality.html It's not much, but, for now, it's probably better than nothing. tq vm, (burley)
Re: Maildir format
There is a new GNU project starting up called GLUE that seems to be concerned with at least some of the same things you are (plus other stuff). You can start looking at their goals at: http://www.gnu.org/software/glue/glue.html On Wed, Jan 19, 2000 at 12:46:16PM -0600, Bruce Guenter [EMAIL PROTECTED] wrote: On Wed, Jan 19, 2000 at 12:36:05PM -0600, Bruce Guenter wrote: A good calendaring system requires that users receive requests for meetings and can answer them, and have trouble screwing them up (i.e. putting the metainfo in the subject line is easy to screw up). Email is the ideal medium for this communication, because it meshes well with our existing work patterns. OK, then, how do you see it integrating with email? Let me clarify some of my questions: - Would there be a seperate part of the protocol designed to support calendaring, or should the events be presented as email messages? - Would the calendar be a seperate mailbox of a special type? - How would multiple calendars be dealt with? - Would the events be plain email messages or something else? - How would the events be transmitted to other users? - Would the TA (transfer agent) or the UA (user agent) be responsible for coordinating responses to calendar events (acknowledgements and rejections)? -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
On Wed, Jan 19, 2000 at 03:31:57PM -0500, Greg Owen wrote: Indeed. I should start up a list just to discuss this. If you start up such a list, let me know. I have started up two lists, actually: [EMAIL PROTECTED] [EMAIL PROTECTED] The first is to discuss an authenticated protocol multiplexer. That is, a front-end protocol that handles authentication details before handing off to one of potentially multiple back-ends that require authentication. I've put up a work-in-progress document at http://em.ca/~bruceg/apx/ The second is to discuss issues related to building a simple internet message access protocol. I've made a web page at http://em.ca/~bruceg/simap/ but there's nothing there yet. -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
On Tue, Jan 18, 2000 at 12:11:16AM -0600, Bruce Guenter wrote: On Tue, Jan 18, 2000 at 12:53:37AM -0500, Russell Nelson wrote: Um, am I missing something? I thought the whole point of the "info" portion of the filename of the message in the maildir? Right, and do you want the filename changing all the time? Instead of a simple "open()", you have to do a "opendir(), readdir(), string match, closedir()" set of syscalls. I suppose that you could attempt a simple open() first, and then only if that fails do you go searching. I saw that from another message. Valid point. Perhaps the server would treat the observed filenames as a "cache" mapped by the unchanging portion. Any miss would cause a revalidation of all of them (since readdir typically issues only one syscall per many directory entries). This is basically what you described. I'd say, indeed, a cache based on the unchanging part of the filename, always doing full readdir() [or getdents(), depending on your UNIX], and then gathering info from files that aren't in the cache already. Note that this is from a MUA point of view (not even POP3, just MUA, that wants to work with headers). I don't very much favor the idea of extending the Maildir structure just to add flags like that. On the other hand, such extensions are ideal for storing other persistent client (configuration) data. I don't see the need for that.. On the subject of extensions of Maildir, though, I had a bit of a radical thought: make each message a directory, containing one file for the headers, and one file per attachment. This has the benefit of pre-parsing attachments for processes like IMAP that want to be able to fetch just one of the parts, but at a significant cost. Fetching the entire message would cause quite a bit of conversion and repackaging. Searching now touches even more files. Every message now uses at least 3 inodes now instead of just one, with the side effect of increasing the amount of wasted (slack) space. More disk accesses to examine a mailbox. Hmmm... I don't like this one: - IMAP-stuff is still as complicated, delivery is _more_ complicated now. - wasting inodes and therefore hindering NFS performance which is isn't so good already for Maildir. I see no benefits. Greetz, Peter. -- Peter van Dijk - student/sysadmin/ircoper/madly in love/pretending coder | | 'C makes it easy to shoot yourself in the foot; | C++ makes it harder, but when you do it blows your whole leg off.' | Bjarne Stroustrup, Inventor of C++
Re: Maildir format
On Fri, Jan 14, 2000 at 03:07:09PM -0500, Russell Nelson wrote: Right, and any scalable email system is going to use NFS. Therefore No. We scale with pop-proxies, and do without NFS at all. We rely heavily on LDAP to achieve this. Regards, bert. -- +---+ | http://www.rent-a-nerd.nl | nerd for hire | | +---+ | - U N I X - | | Inspice et cautus eris - D11T'95
Re: Maildir format
Bruce Guenter [EMAIL PROTECTED] writes: The needs I am aware of include: - the basics of POP3 plus... [snip] - hierarchical multiple mailbox support That should include something that makes sense for a host that's behind a firewall and/or NAT and/or dynamic-IP dialup to authenticate and download mail for multiple users (to basically do what people try to do with fetchmail/multidrop or ETRN or other dodgy solutions nowadays). The existing POP3 protocol doesn't have an accepted RFC-level solution for identifying the set of users to whom each message should go, and SMTP requires that the host be reachable at a static IP address. A good modern protocol cannot assume the server can open a link to the client, or that the client is coming from a known address. - message upload (for draft messages and for transmittal) All client/server communications should ideally happen in the new/fixed protocol; I'd just as soon not do any SMTP relaying at all, and instead require that the user offer credentials in order to relay outbound through me. This neatly solves the remote-dialup-relay problem too. A challenge-response authentication system is a debatable need. On one hand, with it you never send your pass phrase in the clear. On the other, all your content is still in the clear, so it would be better to assume a SSL link where necessary. Making the authentication separate from the after-authentication protocol allows you to bolt on whatever you need; simple user-password may be all that's exportable in a vanilla release from a US vendor, but some sites may want something stronger. There may also be sites that want to require internal communications, especially those that have to cross the Internet, go through an encrypted/authenticated tunnel. -- Anthony DeBoer [EMAIL PROTECTED]
Re: Maildir format (indexing)
On Fri, 14 Jan 2000, Russell Nelson wrote: One way to do that would be for Dan to change the Maildir specification so that a Maildir may have multiple "cur" directories. Then, keep a CDB containing a subset of the message headers. Why multiple "cur" directories? I'm guessing that you're trying to avoid rebuilding a large CDB when any cachable item changes. Why not simply use multiple CDB's in a single directory instead? Select a CDB by hashing the file names. I'm also presuming that the CDB will be indexed by something like the message file name. How efficient are things like string searches going to be in that case? My dream states include things like results of previous searches being cached (I have several large folders that I search on the same subset of strings frequently). How would you do that with a CDB? Thanks, -- Jeff Hayward
Re: Maildir format (indexing)
Jeff Hayward writes: On Fri, 14 Jan 2000, Russell Nelson wrote: One way to do that would be for Dan to change the Maildir specification so that a Maildir may have multiple "cur" directories. Then, keep a CDB containing a subset of the message headers. Why multiple "cur" directories? Avoid large subdirectory filesystem lossage. I'm also presuming that the CDB will be indexed by something like the message file name. How efficient are things like string searches going to be in that case? My dream states include things like results of previous searches being cached (I have several large folders that I search on the same subset of strings frequently). How would you do that with a CDB? If you're storing mail on a server, I don't see *any* alternative to server-side searching. Not that I know how best to implement it. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: Maildir format (scaling)
On 14 Jan 2000, Russ Allbery wrote: I'm responding to provide a counterpoint to Russ's views. I certainly don't plan on changing his mind by my argument. It is abundantly clear that "there's more that one way to do it (well)" to borrow a phrase. My experience is quite the contrary, namely that delivering to *any* shared file system, whether it be NFS or AFS, is fundamentally less reliable and harder to maintain than delivering mail to independent mail server machines [...] It is funny how one's experiences can be different. At my site, it is exactly the opposite. The minute we changed from a "user dictates server" correspondence to a separation of the data from the application we saw enormous improvement in reliability and ease of maintenance. We serve about 80K users using layer 4 redirectors, 10 application server boxes and 2 NFS servers. There is virtually no maintenance, no outages, and no performance peaks and valleys. By putting our money in to making the data reliable we don't have to have expensive and complicated schemes to keep application servers up. Load balancing happens automatically, not by adding/moving users to application boxes. Failover is just a special case of load balancing. Scales well for us (about 6.5 million messages stored in maildirs) with no limits on the horizon. That said, maildir indexing would help latency in application response quite a bit. Oh, we've also been down the AFS path. Not recommended based on my experience. Regards, -- Jeff Hayward
Re: Maildir format (scaling)
On Tue, Jan 18, 2000 at 10:32:23AM -0600, Jeff Hayward wrote: On 14 Jan 2000, Russ Allbery wrote: I'm responding to provide a counterpoint to Russ's views. I certainly don't plan on changing his mind by my argument. It is abundantly clear that "there's more that one way to do it (well)" to borrow a phrase. My experience is quite the contrary, namely that delivering to *any* shared file system, whether it be NFS or AFS, is fundamentally less reliable and harder to maintain than delivering mail to independent mail server machines [...] It is funny how one's experiences can be different. At my site, it is exactly the opposite. The minute we changed from a "user dictates server" correspondence to a separation of the data from the application we saw enormous improvement in reliability and ease of maintenance. We serve about 80K users using layer 4 redirectors, 10 application server boxes and 2 NFS servers. There is virtually no maintenance, no outages, and no performance peaks and valleys. By putting our money in to making the data reliable we don't have to have expensive and complicated schemes to keep application servers up. Load balancing happens automatically, not by adding/moving users to application boxes. Failover is just a special case of load balancing. Scales well for us (about 6.5 million messages stored in maildirs) with no limits on the horizon. That said, maildir indexing would help latency in application response quite a bit. Oh, we've also been down the AFS path. Not recommended based on my experience. Regards, -- Jeff Hayward In the near future I will try out to store the users mail on one or several CODA server(s). Have anyone any comment on that? Best regards Michael Boman -- W I Z O F F I C E . C O M P T E L T D - Your Online Wizard 16 Tannery Lane, Crystal Time Building, #06-00, Singapore 347778 Ring : (65) 844 3228 [ext 118] Fax : (65) 842 7228 email : [EMAIL PROTECTED]URL : http://www.wizoffice.com
Re: Maildir format (indexing)
On Tue, Jan 18, 2000 at 10:15:31AM -0600, Jeff Hayward wrote: On Fri, 14 Jan 2000, Russell Nelson wrote: One way to do that would be for Dan to change the Maildir specification so that a Maildir may have multiple "cur" directories. Then, keep a CDB containing a subset of the message headers. Why multiple "cur" directories? I'm guessing that you're trying to avoid rebuilding a large CDB when any cachable item changes. Why not simply use multiple CDB's in a single directory instead? Select a CDB by hashing the file names. CDB is hashed itself. Using multiple CDB's to share one load is useless. The multiple "cur" directory idea helps performance on average filesystems. I'm also presuming that the CDB will be indexed by something like the message file name. How efficient are things like string searches going to be in that case? My dream states include things like results of previous searches being cached (I have several large folders that I search on the same subset of strings frequently). How would you do that with a CDB? Well the CDB (in my idea, at least) will be indexed to the unchanging part of a message filename (without new/ or cur/ in front), and contain the headers that mutt normally reads from the file itself while opening. [Yes, I am targeting mutt specifically, don't flame me ;)] For searches thru headers, the cdb can be used. For body-text-searches my solution won't help much. Greetz, Peter. -- Peter van Dijk - student/sysadmin/ircoper/madly in love/pretending coder | | 'C makes it easy to shoot yourself in the foot; | C++ makes it harder, but when you do it blows your whole leg off.' | Bjarne Stroustrup, Inventor of C++
Re: Maildir format (indexing)
On Tue, Jan 18, 2000 at 06:41:23PM +0100, [EMAIL PROTECTED] wrote: Well the CDB (in my idea, at least) will be indexed to the unchanging part of a message filename (without new/ or cur/ in front), and contain the headers that mutt normally reads from the file itself while opening. [Yes, I am targeting mutt specifically, don't flame me ;)] This is not uncommon in a number of the proprietry message stores. An index file that points directly into the mail/mailbox which identifies such things as MIME boundaries, header boundaries and so on. Many treat the index as a cache of high-use knowledge needed by the client applications. For searches thru headers, the cdb can be used. For body-text-searches my solution won't help much. Your cdb/index *could* contain a cache of recent searches. Mark.
Re: Maildir format
Bruce Guenter writes: On Sat, Jan 15, 2000 at 10:57:00PM -0500, Russell Nelson wrote: I wonder if that couldn't be handled by the Maildir code writing Status: XXX as the very first line in each message? Um, am I missing something? I thought the whole point of the "info" portion of the filename of the message in the maildir? Right, and do you want the filename changing all the time? Instead of a simple "open()", you have to do a "opendir(), readdir(), string match, closedir()" set of syscalls. I suppose that you could attempt a simple open() first, and then only if that fails do you go searching. - explicit message size notification You get this already. In what? In the pop3 protocol's list command: list +OK 1 1998 2 2346 . - message upload (for draft messages ... Couldn't you just send it to $USER-draft, and direct $USER-draft into a draft Maildir? That is an option, but messy, considering that the act of delivery will cause header manipulation and IMHO saving messages should keep them intact. So? When you send the mail, the first header you write out is X-Draft:. That lets you remove the headers which were added in the delivery process. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: Maildir format
On Tue, Jan 18, 2000 at 12:53:37AM -0500, Russell Nelson wrote: Um, am I missing something? I thought the whole point of the "info" portion of the filename of the message in the maildir? Right, and do you want the filename changing all the time? Instead of a simple "open()", you have to do a "opendir(), readdir(), string match, closedir()" set of syscalls. I suppose that you could attempt a simple open() first, and then only if that fails do you go searching. I saw that from another message. Valid point. Perhaps the server would treat the observed filenames as a "cache" mapped by the unchanging portion. Any miss would cause a revalidation of all of them (since readdir typically issues only one syscall per many directory entries). This is basically what you described. I don't very much favor the idea of extending the Maildir structure just to add flags like that. On the other hand, such extensions are ideal for storing other persistent client (configuration) data. On the subject of extensions of Maildir, though, I had a bit of a radical thought: make each message a directory, containing one file for the headers, and one file per attachment. This has the benefit of pre-parsing attachments for processes like IMAP that want to be able to fetch just one of the parts, but at a significant cost. Fetching the entire message would cause quite a bit of conversion and repackaging. Searching now touches even more files. Every message now uses at least 3 inodes now instead of just one, with the side effect of increasing the amount of wasted (slack) space. More disk accesses to examine a mailbox. -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
On Sat, Jan 15, 2000 at 09:41:48AM -0600, Tim Tsai wrote: Russ, what is your definition of a "large" installation? 10k, 100k, 1m users? Just exactly how many lighter-weight servers is practical to manage and upkeep before it's cheaper to buy NetApp's? As someone who has purchased and maintained a lot of NetApp hardware over the last year let me tell you that NetApp is heinously expensive. The head unit alone usually goes for around $50k. Then add disks. We have since ditched the NetApp solution and re-architected things to use clusters of PC's. We are much happier with the cost effectiveness and the reliability. Of course we aren't using them for mail but I can think of ways to distribute a large mail load on cheap PC's. -- Tracy Reed http://www.ultraviolet.org http://www.linux.org - Escape the Gates of Hell "But these are not inherent flaws in the operating system - they don't happen by accident." - Mike Nash, "Director of Microsoft's Infrastructure Systems" explaining why NT has so many patches to fix crashes caused by malicious net users.
Re: Maildir format
Bruce Guenter [EMAIL PROTECTED] schrieb/wrote: First place to start is to figure out what is actually necessary. In a lot of cases, POP3 with a few extensions should be perfectly adequate, but it is necessary to know what the needs actually are. I don't think it's a good idea to overload the POP3, which is a last- step _delivery_ protocol with mailbox access. The needs I am aware of include: - the basics of POP3 plus... No, I'd rather start with IMAP, but leave out: . the requirement that persistent IDs must be numeric and subsequent (just use opaque strings instead), . the very complex syntax, . response fields that are filled in from header fields (instead pass the header fields raw to the client), . the variable hierarchy delimiters (instead, use iURL syntax with %- encoding). Some simplicifations and changes: . Instead of namespaces, have "mailboxes" which then have certain types, . have special folders labelled with out-of-band data for each mailbox. I.e., don't have a folder "inbox", but an unnamed folder which has the function inbox. Same for other commonly used folders such as sent, templates, unsent, drafts etc. The name of that folder is left to the UA programmers. So you would have user folders and "special" folders. Some additional features that would be nice: . regexp search, search for message id. . server-side filering (optional) . sending of email (yes!) . Storage of metadata such as user name etc. -- Claus Andre Faerber http://www.faerber.muc.de PGP: ID=1024/527CADCD FP=12 20 49 F3 E1 04 9E 9E 25 56 69 A5 C6 A0 C9 DC
Re: Maildir format
What do you guys do for backup's? Do you put two NIC cards in each server and maintain a separate network for that? Thanks, from a guy that's about to take that big plunge into a scalable mail design. Tim On Sun, Jan 16, 2000 at 02:56:59AM -0800, Tracy R Reed wrote: On Sat, Jan 15, 2000 at 09:41:48AM -0600, Tim Tsai wrote: Russ, what is your definition of a "large" installation? 10k, 100k, 1m users? Just exactly how many lighter-weight servers is practical to manage and upkeep before it's cheaper to buy NetApp's? As someone who has purchased and maintained a lot of NetApp hardware over the last year let me tell you that NetApp is heinously expensive. The head unit alone usually goes for around $50k. Then add disks. We have since ditched the NetApp solution and re-architected things to use clusters of PC's. We are much happier with the cost effectiveness and the reliability. Of course we aren't using them for mail but I can think of ways to distribute a large mail load on cheap PC's.
Re: Maildir format
What do you guys do for backup's? Do you put two NIC cards in each server and maintain a separate network for that? Do you have *a lot* of pc-hardware around? What failed, last time? And before that? No, that wasn't why I asked. The main reason for two NIC's is to keep the backup traffic separated from the regular traffic. Obviously with NetApp (and other centralized storage) backup is simpler. Tim
Re: Maildir format
On Sat, 15 Jan 2000, Bruce Guenter wrote: On Sat, Jan 15, 2000 at 01:20:05AM -0500, Russell Nelson wrote: What about asynchronous commands and notifications? I'd nuke 'em, myself. Which of course begs the question about what kinds of events are really necessary for a mailbox access protocol. In my admitedly simplistic view, I can only think of a "new mail" event. As far as I'm concerned, asynchronous notifications can be performed using UDP. No reason to tie up a TCP connection. Agreed, except for the unreliability of UDP. in addition, there is no way for the sending application to know if the receiving application is still running, or even that the same user is sat at the same machine. Think of a student lab where one has students rebooting machines instead of closing all applications. By the time one has put the necessary security information into the protocol one might as well have used TCP. OTOH using a separate TCP connection to the machine for events coming from the server makes more sense than embedding them in the command channel. UDP for user-level-applications has to be a carefully made decision with more than just a simplistic approach And even if you didn't mind doing that, then events of interest could be reported using a prompt which conveyed the same information as "You have a pending event". So you'd either be executing a command, or else you'd be running the "wait for event" command. Or polling, which the server is likely to have to do anyways to retrieve the events. polling is bad, if only becuse it leads to: - keeping a TCP connection open (network bandwidth) - keeping a server process alive (memory, OS resources) - delays notificiation by (polling interval)/2 in most cases - waste resources when there is no email. IMHO if one is going to design a new mail retreival protocol then what is requireed is something which runs on a single TCP connection is authenticated securely, and requires the client to register the type of things it wants to know about then then lets the server send them at its own rate. Asynchronous is a total botch. If you want multitasking, then open up another TCP connection. This leads me to question if it would be a good idea to look at the FTP model of opening up a secondary channel (with the option of opening more than just one) that transfers exactly one message before closing, leaving the initial connection available for command data? no, this is bad, one looses the adaptive flow control (slow start, etc) which having only the one connection is worth. What is much better is including the size explicitly of the transfer at the beginning of the message and storing messages in network-byte order. then the receiving application knows the next 4567 bytes (say) are the message, and doesn't mess with it until it is written to disk. At which point it might choose to convert it into a local mailbox format. Sending applications are simplified and there is much less chnagce of the message being damaged in transit. RjL == You know that. I know that. But when || Austin, Texas you talk to a monkey you have to || Email: [EMAIL PROTECTED] grunt and wave your arms -ck ||
Re: Maildir format
We decided against NetApp for the same reasons, and went with Metastor. Performance is great, easy to upgrade, and it fit our needs for a reasonable price vs using seperate file stores for each mail server. I'm sure there are other brands out there of similar price/performance (we spent maybe 15k for 36gig raid 5, 3 hot drives, 1 hot spare, and 6 empty slots for new) For backups I have tape connected to a seperate scsi channel on the NFS server which has the raid box. -- Stephen Comoletti Systems Administrator Delanet, Inc. http://www.delanet.com ph: (302) 326-5800 fax: (302) 326-5802 Tim Tsai wrote: What do you guys do for backup's? Do you put two NIC cards in each server and maintain a separate network for that? Thanks, from a guy that's about to take that big plunge into a scalable mail design. Tim On Sun, Jan 16, 2000 at 02:56:59AM -0800, Tracy R Reed wrote: On Sat, Jan 15, 2000 at 09:41:48AM -0600, Tim Tsai wrote: Russ, what is your definition of a "large" installation? 10k, 100k, 1m users? Just exactly how many lighter-weight servers is practical to manage and upkeep before it's cheaper to buy NetApp's? As someone who has purchased and maintained a lot of NetApp hardware over the last year let me tell you that NetApp is heinously expensive. The head unit alone usually goes for around $50k. Then add disks. We have since ditched the NetApp solution and re-architected things to use clusters of PC's. We are much happier with the cost effectiveness and the reliability. Of course we aren't using them for mail but I can think of ways to distribute a large mail load on cheap PC's.
Re: Maildir format
Ruben van der Leij [EMAIL PROTECTED] writes: On Sun, Jan 16, 2000 at 10:35:09AM -0600, Tim Tsai wrote: What do you guys do for backup's? Do you put two NIC cards in each server and maintain a separate network for that? We just back up over the same network as we do everything else, early in the morning. It's a 100Mb fiber trunk, and normal traffic doesn't come anywhere close to saturating it. Bear in mind that pretty much all of our users are local and on direct Ethernet connections to the rest of the campus network, so bandwidth generally isn't much of a concern. Disk speed is our limiting factor. Do you have *a lot* of pc-hardware around? What failed, last time? And before that? In my experience power-supplies, drives and memory (in that order) are most prone to failure. The only dead NIC's or switches I've seen were after a direct hit by lightning took out a major part of the leased line of a client. Add in CPU fans as more likely to fail than anything else. PC manufacturers don't use decent CPU fans. If you use non-PC hardware, you much more rarely have that problem, but the hardware's a lot more expensive. -- Russ Allbery ([EMAIL PROTECTED]) URL:http://www.eyrie.org/~eagle/
Re: Maildir format
On Sun, Jan 16, 2000 at 02:09:00PM +0100, Claus Färber wrote: Bruce Guenter [EMAIL PROTECTED] schrieb/wrote: First place to start is to figure out what is actually necessary. In a lot of cases, POP3 with a few extensions should be perfectly adequate, but it is necessary to know what the needs actually are. I don't think it's a good idea to overload the POP3, which is a last- step _delivery_ protocol with mailbox access. I'm thinking in a conceptual sense -- ignore the syntax and everything else and consider what it provides: a method of listing the contents of a mailbox (by unique ID), seeing what's new, and transferring the mail from a single mailbox to a client. No frills. However, more is necessary. The biggest being that the system should be designed to keep the mail on the server rather than assuming it should be downloaded. No, I'd rather start with IMAP, but leave out: . the requirement that persistent IDs must be numeric and subsequent (just use opaque strings instead), . the very complex syntax, . response fields that are filled in from header fields (instead pass the header fields raw to the client), Agreed. In otherwords, simplification of the requirements on the server (and in part, the client). . the variable hierarchy delimiters (instead, use iURL syntax with %- encoding). Forgive my ignorance, but how is an "iURL" different from an "URL"? Some simplicifations and changes: . have special folders labelled with out-of-band data for each mailbox. I.e., don't have a folder "inbox", but an unnamed folder which has the function inbox. Same for other commonly used folders such as sent, templates, unsent, drafts etc. The name of that folder is left to the UA programmers. So you would have user folders and "special" folders. Too many special cases. That's not a simplification at all. The simplification is no special cases for mailbox access. The incoming mail (for cooperation with other programs) could be the unnamed mailbox, or a mailbox with a 0-length name. Some additional features that would be nice: . server-side filering (optional) This seems to be a delivery issue rather than a content retrieval issue. Besides, the client could examine the headers, and move the mail into a different folder. . Storage of metadata such as user name etc. Standardized storage of configuration data seems to be a definite requirement. It should be flexable enough to allow client implementations to put what they'd like there as well. -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
On Sun, Jan 16, 2000 at 05:47:50PM +, [EMAIL PROTECTED] wrote: And even if you didn't mind doing that, then events of interest could be reported using a prompt which conveyed the same information as "You have a pending event". So you'd either be executing a command, or else you'd be running the "wait for event" command. Or polling, which the server is likely to have to do anyways to retrieve the events. polling is bad, if only becuse it leads to: - keeping a TCP connection open (network bandwidth) - keeping a server process alive (memory, OS resources) But the methodology described above does exactly this. The question was, poll the server explicitly, or have the server poll and send notifications, both over an open channel. Inactive open connections do not cause significant bandwidth loss, and server processes should be simple enough to reduce memory pressure. -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
Bruce Guenter writes: - message state storage (read, replied to, forwarded, flagged, etc.) seperate from content delivery (a "Status:" header line) I wonder if that couldn't be handled by the Maildir code writing Status: XXX as the very first line in each message? Then, you could change the status by opening the message file, read in the first N bytes, modify one of those characters to set the status, and write out those bytes again. - explicit message size notification You get this already. - message upload (for draft messages ... Couldn't you just send it to $USER-draft, and direct $USER-draft into a draft Maildir? A challenge-response authentication system is a debatable need. On one hand, with it you never send your pass phrase in the clear. On the other, all your content is still in the clear, so it would be better to assume a SSL link where necessary. I'd assume ssh, now that there's http://openssh.org . -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: Maildir format
On Sat, Jan 15, 2000 at 10:57:00PM -0500, Russell Nelson wrote: I wonder if that couldn't be handled by the Maildir code writing Status: XXX as the very first line in each message? Then, you could change the status by opening the message file, read in the first N bytes, modify one of those characters to set the status, and write out those bytes again. Why do you need to store the message state in the message? This could easily be kept in a (lockable) file in Maildir/. All that is required is that "MUA" (e.g. IMAP, Mutt, qmail-pop3d) that use the same file agree on format and possible locking mechanism. Opening the Maildir means reading the directory, reading the state file, and marking the messages not in the state file as new, removing messages that do not exist from the state file. Safe-write the file. Skip locking if it is acceptable that the state file misses modifications if two "MUA" access the dir at the same time (no problem, IMHO). Without locking it is cheap to write the file every x seconds for a client that keeps the connection to the maildir. Delivery notifications can be done via program delivery in .qmail, causing the MUA (that maintains a connection to the maildir) to reread the directory. There is no reason for the user to notice the time this may take. Renaming the files doesn't make much sense since a consequence is that a message may still exist, but not be accessible via its old name. For very large folders over slow links, the client can display info from the state file while scanning the directory. Most likely, the only difference is a new message or two. A cdb keyed on the file name would be nice for the state file. Of course, the name and format for a state file should be defined in the Maildir format. IMHO, the "INFO" extension to the file name should be removed from the Maildir format specification. "new" - "cur" can be used as a delivery notification. Just read "new" every whatever seconds. Move files to "cur" and add them to the cached state info. -- -Sincerely, Fred Fred Lindberg, Inf. Dis., WashU, St. Louis, MO, USA
Re: Maildir format
On Sat, Jan 15, 2000 at 10:57:00PM -0500, Russell Nelson wrote: Bruce Guenter writes: - message state storage (read, replied to, forwarded, flagged, etc.) seperate from content delivery (a "Status:" header line) I wonder if that couldn't be handled by the Maildir code writing Status: XXX as the very first line in each message? Um, am I missing something? I thought the whole point of the "info" portion of the filename of the message in the maildir? - explicit message size notification You get this already. In what? - message upload (for draft messages ... Couldn't you just send it to $USER-draft, and direct $USER-draft into a draft Maildir? That is an option, but messy, considering that the act of delivery will cause header manipulation and IMHO saving messages should keep them intact. Another option is to do it in reverse: no direct email sending, but rather a procedure that uploads a message to a folder, and then sends that email (by some unique identifier). On the other, all your content is still in the clear, so it would be better to assume a SSL link where necessary. I'd assume ssh, now that there's http://openssh.org . Good thought, especially with the tunneling options, but unless things have changed, SSH still requires shell access -- something that should not be required for mailbox access. -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
Actually, using ssh would obviate the need for an ftp-like second connection protocol-mess for async notification. I think you could just forward 2 connections over the already-established link. So all connections from a particular client would be guranteed to land on the same server, despite any intermediary load balancers or firewalls. Does this make sense? -Peter On Sat, Jan 15, 2000 at 10:57:00PM -0500, Russell Nelson wrote: Bruce Guenter writes: - message state storage (read, replied to, forwarded, flagged, etc.) seperate from content delivery (a "Status:" header line) I wonder if that couldn't be handled by the Maildir code writing Status: XXX as the very first line in each message? Then, you could change the status by opening the message file, read in the first N bytes, modify one of those characters to set the status, and write out those bytes again. - explicit message size notification You get this already. - message upload (for draft messages ... Couldn't you just send it to $USER-draft, and direct $USER-draft into a draft Maildir? A challenge-response authentication system is a debatable need. On one hand, with it you never send your pass phrase in the clear. On the other, all your content is still in the clear, so it would be better to assume a SSL link where necessary. I'd assume ssh, now that there's http://openssh.org . -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M. -- The 5 year plan: In five years we'll make up another plan. Or just re-use this one.
Re: Maildir format
On Sun, Jan 16, 2000 at 12:09:13AM -0600, Bruce Guenter wrote: Good thought, especially with the tunneling options, but unless things have changed, SSH still requires shell access -- something that should not be required for mailbox access. Not really. It only spawns a shell because sshd's usual procedure is to invoke /bin/login as it's last action (step 10 in the sshd(8) version 1 man page). If you give the server a unique user, and that user has the server binary as it's shell... then you've saved some effort for yourself. Sshd can be very tightly restricted by an admin. -- The 5 year plan: In five years we'll make up another plan. Or just re-use this one.
Re: Maildir format
On Sat, Jan 15, 2000 at 10:26:22PM -0800, Peter C. Norton wrote: Good thought, especially with the tunneling options, but unless things have changed, SSH still requires shell access -- something that should not be required for mailbox access. Not really. It only spawns a shell because sshd's usual procedure is to invoke /bin/login as it's last action (step 10 in the sshd(8) version 1 man page). If you give the server a unique user, and that user has the server binary as it's shell... then you've saved some effort for yourself. If I read this right, you're saying that to access a service X, the user should (through an inteligent UA that hides the details, of course) connect with SSH and log in to a user (named specifically for the protocol being exchanged) with no password? That user would then have, as a shell, a setuid-root binary (for those protocols requiring setuid capabilities) that executes the required protocol handler? This has definite advantages to it, the biggest I can see being no more magic reserved port numbers. On the downside, I have at least a small distrust for setuid binaries no matter how reliable the source. Also, the initial connection setup cost for SSH is fairly high, both in terms of wall time due to the number of exchanges required, and CPU time, from my experience. Mind you, I always use RSA authentication, which adds a second set of challenge-response exchanges. It would be interesting to compare the connection cost of SSH (as outlined above -- a user with no password running /bin/true for a shell) and SSL, both in terms of wall time (important for perceived responsiveness) and CPU cost (important for scalability). -- Bruce Guenter [EMAIL PROTECTED] http://em.ca/~bruceg/
Re: Maildir format
[EMAIL PROTECTED] wrote: From libc-client4.7 documentation: --strip-- The Maildir format used by qmail has all of the performance disadvantages of mh noted above, with the additional problem that the files are renamed in order to change their status so you end up having to rescan the directory frequently the current names (particularly in a shared mailbox scenario). It doesn't scale, and it represents a support nightmare; it will therefore never be supported in the official distribution. Maildir support code for c-client is available from third parties; but, if you use it, it is entirely at your own risk (read: don't complain about how poorly it performs or bugs). --strip-- Could someone comment this? The author of that comment, Mark Crispin, and the inventor of the maildir format, Dan Bernstein, *really* don't get along. So Mark trumped up some reasons for not supporting maildir since he can't bear to admit it's a good design. -Dave
Re: Maildir format
Ondrej Surý [EMAIL PROTECTED] wrote on Fri, 14 Jan 2000: From libc-client4.7 documentation: --strip-- The Maildir format used by qmail has all of the performance disadvantages of mh noted above, with the additional problem snip Could someone comment this? Well, I can say something from a user's perspective. About scaling... I just tried opening a Maildir folder with close to 8000 messages in it (using Mutt, my preferred client). The folder is in my home dir, which is NFS-mounted over 100Mbit ethernet. It took over a minute, but not quite 2 minutes (although this is just a guess, I didn't time). Considering the number of messages, it didn't seem like too outrageous a waiting time. mbox format would be faster (depending on message size of course), but I'm not sure how much. I'd estimate probably not more than 50%, perhaps less, based on my handling of mbox folders which have over 1 messages in them. I have experienced somewhat lengthy "freezes" while Mutt re-scans the Maildir if there are a lot of messages in there. But, the commentary completely misses the good points and the purpose of Maildirs: that they're ideal for incoming mail delivery, especially when the folder is accesses over NFS (whether "access" delivery or reading or both). Maildir format is not something you should be using for email archival, or for very large mail folders. However, the lack of locking requirement is a big win over NFS. I can see that there might be a problem with shared folder access if the filenames change, however I don't see how the situation is any better with other folder formats that require locking and then possibly re-reading of the entire folder if one message changes. Shared access is tricky, no matter what kind of folder format you have. Also, I'm not sure whether the filenames are supposed to change (are they?), because an email client needs to read in the headers from each message anyway in order to display them, so status information could also be kept there as with the mbox format. In the end, I've been using Maildirs happily for (over?) a year now, and they work very well in practice. I use mbox format for email archival, Maildir for incoming folders. I haven't noticed any of the problems at all that they talk of, with moderate size (1000 messages or less) folders. And Mutt as the MUA seems to handle them just fine. Mikko -- // Mikko Hänninen, aka. Wizzu // [EMAIL PROTECTED] // http://www.iki.fi/wiz/ // The Corrs list maintainer // net.freak // DALnet IRC operator / // Interests: roleplaying, Linux, the Net, fantasy scifi, the Corrs / Beware of low-flying butterflies.
Re: Maildir format
On Wed, 12 Jan 2000, Kristina wrote: [...] Do I need a .qmail file to configure the maildir format. If so, how do I configure it. Please let me know. Yes. Just tell qmail where to put incoming mail's (echo "./Maildir/" .qmail). Anyway, to create Maildir/new, Maildir/cur, Maildir/tmp files you need to run maildirmake program (for example, maildirmake /home/dude/Maildir/). Greetings, Marcin Jaskowiak "It's better to burn out than to fade away..." - Kurt Cobain
Maildir format
From libc-client4.7 documentation: --strip-- The Maildir format used by qmail has all of the performance disadvantages of mh noted above, with the additional problem that the files are renamed in order to change their status so you end up having to rescan the directory frequently the current names (particularly in a shared mailbox scenario). It doesn't scale, and it represents a support nightmare; it will therefore never be supported in the official distribution. Maildir support code for c-client is available from third parties; but, if you use it, it is entirely at your own risk (read: don't complain about how poorly it performs or bugs). --strip-- Could someone comment this? -- Ondrej Sury: [EMAIL PROTECTED], +420602667702 GLOBE Internet s.r.o.: http://www.globe.cz/, [EMAIL PROTECTED], +420233356502 NAJDI.TO; http://najdi.to/, [EMAIL PROTECTED]
Re: Maildir format
=?iso-8859-2?Q?Ond=F8ej=20Sur=FD?= writes: Could someone comment this? Yeah. Mark Crispin doesn't like Dan Bernstein; therefore anything Dan Bernstein does has technical problems which "don't scale" and are "a support nightmare." Mark doesn't want to hear what I have to say about IMAP (which Mark invented). Maybe if I just sent him the piles of hair I wrenched out of my head while reading the IMAP spec, he'd get the point? Na.. He thinks that all the stupid design decisions he made (e.g. having three string representations) were necessary. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: Maildir format
Mikko Hänninen writes: But, the commentary completely misses the good points and the purpose of Maildirs: that they're ideal for incoming mail delivery, especially when the folder is accesses over NFS (whether "access" delivery or reading or both). Maildir format is not something you should be using for email archival, or for very large mail folders. However, the lack of locking requirement is a big win over NFS. Right, and any scalable email system is going to use NFS. Therefore the question in my mind is not "What should be used for large folders instead of Maildirs?" but instead "What must be done to make Maildirs more efficient"? One way to do that would be for Dan to change the Maildir specification so that a Maildir may have multiple "cur" directories. Then, keep a CDB containing a subset of the message headers. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: Maildir format
Russell Nelson [EMAIL PROTECTED] wrote: Right, and any scalable email system is going to use NFS. Therefore the question in my mind is not "What should be used for large folders instead of Maildirs?" but instead "What must be done to make Maildirs more efficient"? One way to do that would be for Dan to change the Maildir specification so that a Maildir may have multiple "cur" directories. Then, keep a CDB containing a subset of the message headers. Doesn't the CDB file then require some trickery to avoid the necessity of locks for multiple writers? Locks for the CDB would defeat the main benefit of Maildirs. Or perhaps I misunderstand. Charles -- Charles Cazabon [EMAIL PROTECTED] Any opinions expressed are just that -- my opinions.
Re: Maildir format
On Fri, Jan 14, 2000 at 02:21:47PM -0600, Charles Cazabon wrote:
Russell Nelson [EMAIL PROTECTED] wrote:
Right, and any scalable email system is going to use NFS. Therefore
the question in my mind is not "What should be used for large folders
instead of Maildirs?" but instead "What must be done to make Maildirs
more efficient"? One way to do that would be for Dan to change the
Maildir specification so that a Maildir may have multiple "cur"
directories. Then, keep a CDB containing a subset of the message
headers.
Doesn't the CDB file then require some trickery to avoid the necessity of
locks for multiple writers? Locks for the CDB would defeat the main benefit
of Maildirs. Or perhaps I misunderstand.
cdb are always updated atomically. One can open the cdb, acquiring a safe
path to the file (even if it is updated in between, it will still be reading
the old copy then), read it in, build a new cdb in a tmpfile and rename()
it over the real one. Only risk here is that in very concurrent updates,
one of the two will just miss. This is from the delivery-agent perspective.
When used from a useragent, the story is much easier: read directory listings,
check if all files are in the CDB already, and if they're not, add them.
I have actually been considering such a feature for mutt, since opening a
2500-message Maildir over NFS does take some time with the linux 2.0 client
NFS-implementation ('request' 'ack' 'request' etc., no paralellism) over a
25km glasfibre ethernet to a NetApp. Since cdb-updates are atomic, and in this
case, the updating process actually checks reality [as opposed to reading the
cdb and applying the known-made changes] when updating, so that the cdb
will be a performance improvement, but no PITA. Only glitch I can see is
someone actually editing files in a Maildir and the cdb not catching up..
doing a check on headers when a message is actually opened should fix most
of this, storing a datestamp in the cdb might help also.
Hmm.. I'm discussing user-agent cdb features now... lemme think about this over
the weekend :)
Note that I don't really see the benefit in multiple cur-directories, apart
from the performance advantages on sub-optimal [most] filesystems, for which
same reason the queue directories are split up.
Greetz, Peter.
--
Peter van Dijk - student/sysadmin/ircoper/madly in love/pretending coder
|
| 'C makes it easy to shoot yourself in the foot;
| C++ makes it harder, but when you do it blows your whole leg off.'
| Bjarne Stroustrup, Inventor of C++
Re: Maildir format
Charles Cazabon writes: Russell Nelson [EMAIL PROTECTED] wrote: Right, and any scalable email system is going to use NFS. Therefore the question in my mind is not "What should be used for large folders instead of Maildirs?" but instead "What must be done to make Maildirs more efficient"? One way to do that would be for Dan to change the Maildir specification so that a Maildir may have multiple "cur" directories. Then, keep a CDB containing a subset of the message headers. Doesn't the CDB file then require some trickery to avoid the necessity of locks for multiple writers? No. What would you need a lock for? A CDB is replaced atomically. If you had two processes each reading the same set of data (the existing CDB and the new messages) and writing a new CDB, would it matter which one won the race to replace the old one? The worst that would happen is that one would start earlier and finish later, thereby missing some new messages, or include a message just before someone else deletes it. But they'll be caught the next time you look for new or deleted messages. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: Maildir format
[EMAIL PROTECTED] writes: Note that I don't really see the benefit in multiple cur-directories, apart from the performance advantages on sub-optimal [most] filesystems, for which same reason the queue directories are split up. Right, me neither. | 'C makes it easy to shoot yourself in the foot; | C++ makes it harder, but when you do it blows your whole leg off.' | Bjarne Stroustrup, Inventor of C++ Hehe. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: Maildir format
Russ Allbery writes: Russell Nelson [EMAIL PROTECTED] writes: Right, and any scalable email system is going to use NFS. Why do you think that? Because every such installation I've ever seen has used NFS. I'm not talking about what's good, or what's right. I'm talking about what's possible to do tomorrow. Yes, it might be that a specialized mail transfer protocol would work better; convince Netapp to support it. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | "Ask not what your country 521 Pleasant Valley Rd. | +1 315 268 1925 voice | can force other people to Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | do for you..." -Perry M.
Re: Maildir format
On Fri, 14 Jan 2000, Ondøej Surý wrote: From libc-client4.7 documentation: --strip-- The Maildir format used by qmail has all of the performance disadvantages of mh noted above, with the additional problem that the files are renamed in order to change their status so you end up having to rescan the directory frequently the current names (particularly in a shared mailbox scenario). It doesn't scale, and it represents a support nightmare; it will therefore never be supported in the official distribution. Maildir support code for c-client is available from third parties; but, if you use it, it is entirely at your own risk (read: don't complain about how poorly it performs or bugs). --strip-- Could someone comment this? Certainly. As the author of a Maildir IMAP server, that kicks the living daylights out of UW-IMAP in terms of performance and resource footprint, I can unequivocally state that there is a very tiny grain of truth in there, but 99% of that dissertation is complete bunk.
Re: Maildir format
On Fri, 14 Jan 2000, Russell Nelson wrote: =?iso-8859-2?Q?Ond=F8ej=20Sur=FD?= writes: Could someone comment this? Yeah. Mark Crispin doesn't like Dan Bernstein; therefore anything Dan Bernstein does has technical problems which "don't scale" and are "a support nightmare." Mark doesn't want to hear what I have to say about IMAP (which Mark invented). Maybe if I just sent him the piles of hair I wrenched out of my head while reading the IMAP spec, he'd get the point? Na.. He thinks that all the stupid design decisions he made (e.g. having three string representations) were necessary. I concur. RFC 2060 is a joke. I had to read it at least half a dozen times before I was able to figure out what it was actually trying to say.
Re: Maildir format
On Fri, 14 Jan 2000, Russell Nelson wrote: Mikko Hänninen writes: But, the commentary completely misses the good points and the purpose of Maildirs: that they're ideal for incoming mail delivery, especially when the folder is accesses over NFS (whether "access" delivery or reading or both). Maildir format is not something you should be using for email archival, or for very large mail folders. However, the lack of locking requirement is a big win over NFS. Right, and any scalable email system is going to use NFS. Therefore the question in my mind is not "What should be used for large folders instead of Maildirs?" but instead "What must be done to make Maildirs more efficient"? One way to do that would be for Dan to change the Maildir specification so that a Maildir may have multiple "cur" directories. Then, keep a CDB containing a subset of the message headers. No need to. Try opening a 1,000 message Maildir with Pine or Netscape Messenger (Outhouse Excuse would probably work too) connected to Courier-IMAP. The initial folder open should take less than 10 seconds, depending upon your hardware. Subsequent folder opens will be almost instantaneous, and, at there won't be any noticeable delays in browsing the maildir. And the only thing that Courier-IMAP caches are the UIDs of each individual message, which is refreshed with a single directory scan. Pine never asks for headers of every message, and Netscape Messenger caches the headers by itself. It seems that the original IMAP implementation by uwimap was so piss-poor performance-wise, that pretty much all IMAP clients either do some form of caching themselves, or are very carefull not to issue any IMAP requests that might bog down the server.
Re: Maildir format
Kristina [EMAIL PROTECTED] wrote: Do I need a .qmail file to configure the maildir format. No. If you make "./Maildir/" the default delivery instruction, then it will work even if nobody has a .qmail file. BUT! qmail can _deliver to_ a maildir. It can't _create_ a maildir. For existing users, you must do the following (as root): su - ~USER /var/qmail/bin/maildirmake Maildir You can arrange for new users to get a maildir automatically. Just create a maildir in /etc/skel (or wherever your system keeps its home-directory template). Len. -- I'm not going to spend code space blindly duplicating sendmail features. -- Dan Bernstein, author of qmail
Maildir format
I thought that configuring the /var/qmail/control/defaultdelivery file and /var/qmail/rc files was all you needed for the maildir format. However, as I was not getting tmp, cur and new directories created I rechecked the Life with Qmail documentation again to find out that I may need the .qmail file! Is this true?? Do I need a .qmail file to configure the maildir format. If so, how do I configure it. Please let me know. Thankyou in advance, Kristina
Maildir format
I have opted to use Maildir format over mbox. The MDA in use is maildrop. Except for the inbox/spool directory, all the folders that maildrop uses are mbox format. The folders maildrop uses are in ~/Mail while the incoming spool is ~/Maildir. I thought one of the benifits of Maildir, is to have mail in seperate files for later processing and portability. Please correct me if I am wrong. Should I have used "makemaildir ~/Mail"? Right now all the mail in each folder in ~/Mail gets appended. I would like maildrop to use Maildir format and the folders to have the Maildir format. Subba Rao [EMAIL PROTECTED] http://pws.prserv.net/truemax/
Message sequence/ordering when using MAILDIR format
What is the mailbox ordering scheme that is returned by applications as qmail-pop3d when using MAILDIR? For example, if when using the 'LIST' or 'UIDL' command to POP, in what sequence will files be listed? In cronological order, or file creation time, or Do files in ~/Maildir/cur come before ~/Maildir/new? I wasn't able to find any mention of this either in the archives or in any of the doc files. Thanks --curtis
Re: Message sequence/ordering when using MAILDIR format
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 11 Nov 99, at 11:03, Curtis Generous wrote: I wasn't able to find any mention of this either in the archives or in any of the doc files. How about - deliberately undocumented? What exactly do you need to know the order for? -BEGIN PGP SIGNATURE- Version: PGP 6.0.2 -- QDPGP 2.60 Comment: http://community.wow.net/grt/qdpgp.html iQA/AwUBOCr3nFMwP8g7qbw/EQJGkACg4PSJgA+p6IjhS6Jf+ZywEJipX1MAoNRv YRRcxiobsrSXbwyH0910eF4y =u7Ys -END PGP SIGNATURE- -- Petr Novotny, ANTEK CS [EMAIL PROTECTED] http://www.antek.cz PGP key ID: 0x3BA9BC3F -- Don't you know there ain't no devil there's just God when he's drunk. [Tom Waits]
Re: Message sequence/ordering when using MAILDIR format
Curtis Generous writes: What is the mailbox ordering scheme that is returned by applications as qmail-pop3d when using MAILDIR? For example, if when using the 'LIST' or 'UIDL' command to POP, in what sequence will files be listed? In cronological order, or file creation time, or Do files in ~/Maildir/cur come before ~/Maildir/new? File modification time. -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | Government schools are so 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool!
Re: Message sequence/ordering when using MAILDIR format
According to Petr Novotny: On 11 Nov 99, at 11:03, Curtis Generous wrote: I wasn't able to find any mention of this either in the archives or in any of the doc files. How about - deliberately undocumented? What exactly do you need to know the order for? I need to make sure that mailbox ordering is preserved for our users after we migrate from old POP client (non MAILDIR based) to the QMAIL-POPPER so that it doesn't confuse their MUA's. The UIDL is being preserved, but some clients are choking on the ordering change. Thanks, --curtis
Re: Message sequence/ordering when using MAILDIR format
According to Russell Nelson: Curtis Generous writes: What is the mailbox ordering scheme that is returned by applications as qmail-pop3d when using MAILDIR? For example, if when using the 'LIST' or 'UIDL' command to POP, in what sequence will files be listed? In cronological order, or file creation time, or Do files in ~/Maildir/cur come before ~/Maildir/new? File modification time. Across both /cur and /new? What if files in /new are older than in /cur? And how does it mitigate between two files with same modification time? Thanks, --curtis
Re: Message sequence/ordering when using MAILDIR format
Curtis Generous writes: According to Russell Nelson: Curtis Generous writes: What is the mailbox ordering scheme that is returned by applications as qmail-pop3d when using MAILDIR? For example, if when using the 'LIST' or 'UIDL' command to POP, in what sequence will files be listed? In cronological order, or file creation time, or Do files in ~/Maildir/cur come before ~/Maildir/new? File modification time. Across both /cur and /new? What if files in /new are older than in /cur? Across both /cur and /new. And how does it mitigate between two files with same modification time? The time is the only key used, so the resultant order depends on the order the files are presented by readdir(). -- -russ nelson [EMAIL PROTECTED] http://russnelson.com Crynwr sells support for free software | PGPok | Government schools are so 521 Pleasant Valley Rd. | +1 315 268 1925 voice | bad that any rank amateur Potsdam, NY 13676-3213 | +1 315 268 9201 FAX | can outdo them. Homeschool!
Re: maildir format delivery problems?
"David Harris" [EMAIL PROTECTED] wrote: He says that Maildir is unsuitable for large servers because the filesytems serialize creation and deletion of files in a single file system, because the inode and free block tables have to be manipulated. Yes, that's what makes maildirs reliable. TANSTAAFL Thus the file creation part of Maildir drivery ends up being serialized and you spend all your time in the filesystem. He states problems with servers processing more than a few hundred messages per second (300+ or so). That's a pretty high rate of delivery. Any server delivery nearly that much mail would surely be delivering to multiple file systems. -Dave
Re: maildir format delivery problems?
Well, from what I've gleaned by reading this list, Crispin has no interest in supporting maildirs because he has a grudge against either qmail or djb. Besides that, it's obvious that he's not interested in improving his product. He doesn't even use autoconf for god's sake.. When I compile PINE on debian I have to manually edit the makefile so that it compiles with ncurses instead of termcap.. --Adam On Thu, Jul 01, 1999 at 09:19:58AM -0400, Dave Sill wrote: "David Harris" [EMAIL PROTECTED] wrote: He says that Maildir is unsuitable for large servers because the filesytems serialize creation and deletion of files in a single file system, because the inode and free block tables have to be manipulated. Yes, that's what makes maildirs reliable. TANSTAAFL Thus the file creation part of Maildir drivery ends up being serialized and you spend all your time in the filesystem. He states problems with servers processing more than a few hundred messages per second (300+ or so). That's a pretty high rate of delivery. Any server delivery nearly that much mail would surely be delivering to multiple file systems. -Dave
RE: maildir format delivery problems?
Is he seriously contending that with such a rapid flurry of mail coming through the pipe, that it would be safer to append mails to open files, possibly concurrently? I'm no genius but it seems to me that if a Mailbox is open and you are writing to it, and another process is also writing to it, that that is when problems could occur. Isn't that the kind of thing that makes Maildirs so great? -Original Message- From: Adam D. McKenna [mailto:[EMAIL PROTECTED]] Sent: Thursday, July 01, 1999 9:45 AM To: Dave Sill Cc: [EMAIL PROTECTED] Subject: Re: maildir format delivery problems? Well, from what I've gleaned by reading this list, Crispin has no interest in supporting maildirs because he has a grudge against either qmail or djb. Besides that, it's obvious that he's not interested in improving his product. He doesn't even use autoconf for god's sake.. When I compile PINE on debian I have to manually edit the makefile so that it compiles with ncurses instead of termcap.. --Adam On Thu, Jul 01, 1999 at 09:19:58AM -0400, Dave Sill wrote: "David Harris" [EMAIL PROTECTED] wrote: He says that Maildir is unsuitable for large servers because the filesytems serialize creation and deletion of files in a single file system, because the inode and free block tables have to be manipulated. Yes, that's what makes maildirs reliable. TANSTAAFL Thus the file creation part of Maildir drivery ends up being serialized and you spend all your time in the filesystem. He states problems with servers processing more than a few hundred messages per second (300+ or so). That's a pretty high rate of delivery. Any server delivery nearly that much mail would surely be delivering to multiple file systems. -Dave
Re: maildir format delivery problems?
Alex Miller [EMAIL PROTECTED] writes: Is he seriously contending that with such a rapid flurry of mail coming through the pipe, that it would be safer to append mails to open files, possibly concurrently? As near as I can tell, he's arguing for some other, different type of file format that he calls mbx. I'm not sure the differences. I'm no genius but it seems to me that if a Mailbox is open and you are writing to it, and another process is also writing to it, that that is when problems could occur. Locking. Isn't that the kind of thing that makes Maildirs so great? Maildirs don't need locking. Historically, this is a really good thing, since locking is quite frequently a zoo. Practically, if you're in complete control of every process that writes to the mailbox and it's stored on local disk (which is true of the typical IMAP server), locking works fine. -- Russ Allbery ([EMAIL PROTECTED]) URL:http://www.eyrie.org/~eagle/
The Maildir format
Warning Could not process message with given Content-Type: multipart/signed; boundary=GV0iVqYguTV4Q9ER; micalg=pgp-md5;protocol="application/pgp-signature"
Re: The Maildir format
On Sat, 19 Jun 1999 [EMAIL PROTECTED] wrote: Hello, where can I read up on the specifics on the Maildir format? That is, an actual specification suitable if you want to programatically manipulate Maildirs. Try 'man maildir' on a qmail-installed system. -- Russ Steffen [EMAIL PROTECTED]
creat a new Maildir format folder
I want to creat a new Maildir format folder through the Netscape Messenger, how to config the qmail-imap server? BTW I'm using "qmail-imap-4.5.beta-2.i386.rpm" package from ftp://ftp.engr.uark.edu/pub/qmail/qmail-imap/ BoLiang [EMAIL PROTECTED]
Fwd: Maildir format mailbox
Hi I'm trying to use the qmail-imap package from ftp://ftp.engr.uark.edu/pub/qmail/qmail-imap/ I encounted some problem, I can't creat a Maildir (cur,new,tmp) format mailbox from the client side, after I creat a new folder from the netscape, I just got a plain text file unde my home directory. Does anyone has some advise or please tell me is there and document about this package beside the README.maildir? BTW, I'm using a RedHat 5.2 box, and a netscape messanger Thanks a lot BoLiang [EMAIL PROTECTED]
