Perhaps I misinterpreted this: http://www.washington.edu/pine/pine-info/2004.04/msg00031.html "If you only access mail through IMAP, file locking is not needed." -- I thought maybe this was in reference to pine, but the wording isn't clear. I am glad to hear that reason doesn't go out the window with IMAP
http://www.washington.edu/imap/documentation/locking.txt.html The c-client file locking documentation says "If the mail file is NFS-mounted then flock() locking is a silent no-op." Is that what would occur in the situation I described? I don't quite understand what "silent no-op" means either. I realize that the jist of this is that it doesn't get a correct response to a lock request, but I thought that this has since been resolved. Is this not the case? are cutsiepop and sendmail two of programs that you refer to as "non c-type applications" The documentation also says that c-client uses flock on SystemV systems. Is this what happens on Linux (Redhat 9)? If what I understand about Lustre is correct, then using Lustre would fix the NFS locking problem because Lustre OST's manage locks for file writes (flocks). It would also fix the I/O bottleneck because you could stripe the Target servers together and each 2MB fragment would be requested from a different network device and a different drive. Would that be sufficient to justify keeping the user space completely whole? The system in question has been implemented with NFS already because 1) of the small number of users we were initially required to support 2) management has a desire to implement this design to make the data servers fully extensible without fragmenting the user space. It is an eyeopener to me to hear that this is fundamentally incorrect. When you say each user gets a separate server do you mean each individual account? and by server do you mean server application or complete server hardware? I'm assuming from what I've read (admittedly from the same source) that there is a way to configure IMAP whether it uses flock or a lock file. I'm not the primary administrator of this system so I'm unable to say how it is configured if that is the case, but I do know that when IMAP is run on a separate server, and accesses the mail box over NFS it is slow, when it is put directly on the server that has the data it runs much faster but seems to block POP and SMTP almost entirely. I wonder if this has any relation to what you said about c-client with non c-client applications. What I am looking for is what would be required to allow all the data to reside in one [system] that would have the capacity to appear as one mount point and expand in space and throughput forever as long as the user demands for space kept increasing, and that on one side as the demands for incoming mail and mail retrieval kept increasing we could drop in a new IMAP/POP/SMTP or IMAP/POP or POP/SMTP or just SMTP server (whatever the best configuration would happen to be) and allow that server to access the same data space. >From what I'm hearing with this architecture the only way to do this would be a complete re-write of all those system services as well as a new file sharing protocol to run it. Or is this architecture simply "Computer Scientifically Incorrect" _ °v° There's no place like root#cd ~/ /(_)\ ^ ^ ----- Original Message ----- From: "Mark Crispin" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Wednesday, July 07, 2004 4:09 PM Subject: Re: File Locking > On Wed, 7 Jul 2004 [EMAIL PROTECTED] wrote: > > I've been doing some research on file locking for mail systems and have > > noticed in several places mention made to IMAP "[Not needing file > > locking]". > > Any place that says such a thing is uninformed (at best). > > Although the IMAP protocol itself has no locking, this says nothing about > what actually goes on within the implementation of an IMAP server. There > is no magic wand in IMAP that repeals all the locking requires of a mail > store which an IMAP server may access. > > Indeed, IMAP has several requirement which, while not necessarily > requiring locking, are difficult to fufill without locking. In > particular, consider the issues discussed in RFCs 2180 and 2683. > > > If I understand correctly c-client is a library that aides in > > development of email systems. > > Basically, yes. > > > Is this library used in general distributions of the UW Imap server? > > The c-client library is the fundamental platform on which all of UW's IMAP > software is built upon. This includes the UW IMAP server, the UW POP3 > server, the tmail MDA, the Pine/PC-Pine/WebPine MUA, the mailutil utility, > and a fair amount of other software. > > c-client is also used in numerous third-party applications. I don't know > if anyone has an authoritative list of them; I don't, and I wrote > c-client. > > > I guess what I'm asking is does the > > information listed regarding c-client have any impact on an installation > > of IMAP as already developed and distributed, or is it a concern for > > those developing additional software? > > The information regarding c-client's locking is only of interest to > non-c-client applications which wish to access local files concurrently > with c-client. It does not apply to IMAP access. > > > Here is why I'm concerned about file locking. Consider a system that has > > one or several IMAP Servers, three or more sendmail servers and at least > > one pop server that must all access the same data server. This data > >[snip...asking if NFS is the solution] > > As you noted, everybody says "do not implement email over NFS." There is > a reason for this. > > The fundamental architecture of your system is wrong. It presumes that > IMAP, SMTP, and POP are CPU-intensive services that require many separate > CPUs but can share a single disk facility via a network. > > The exact opposite of this presumption is the case: these are all > extremely I/O intensive services that require many separate I/O ports. > > An IMAP server system may be slower than cold molasses in January, but > when you look at the load it isn't high at all. Instead, it's doing I/O > just as far as it can. > > To a lesser extent, email server systems are also memory-hungry. The more > memory, the better; this will not only feed individual IMAP server > processes' cache but also feed the host system's I/O buffer cache. A > 300MHz 486 with 4GB of RAM will perform better as an IMAP server than a > 2500MHz Pentium 4 with 512MB. > > What we do -- and many other sites have done -- is split the user space so > that every user has his own assigned IMAP/POP server (IMAP and POP > co-exist on the same server). Each server has its own private disk (this > does not preclude RAID or EMC) with its portion of the user space. > > These systems are in their own DNS zone (at UW, deskmail.washington.edu), > in which every user and his assigned server are listed. For example, > mrc's IMAP server at UW is always mrc.deskmail.washington.edu. We have > tools to move users while they are not online. Today, mrc.deskmail > happens to be a system called mailer52; tomorrow it may be a different > system, and I never care. > > Each of the IMAP/POP servers has an incoming SMTP server that only does > delivery; it could run LMTP just as well. These SMTP servers refuse > connections from outside. The real SMTP servers are on separate systems, > and handle both routing to the appropriate IMAP/POP server and spam > filtering. > > -- Mark -- > > http://staff.washington.edu/mrc > Science does not emerge from voting, party politics, or public debate. > Si vis pacem, para bellum. >
