Mark Crispin wrote:
Comments on your message:
[5] It sounds like you are using traditional UNIX mailbox
format, a.k.a.
"mbox format"; a flat-file containing all the messages, one after
another,
with each message preceded with a line starting with "From ". If this
is
the case: traditional UNIX mailbox format was designed in a time when a
"very large mailbox" was between 100 KB and 800 KB in size. It does
not
scale well to sizes of 100 MB to 800 MB.
I don't know what other formats are available, moreover the "mbox"
format
I thought was actually the canonical format of a file containing
multiple email
messages and is used as well by POP. After reading about the Cyrus IMAP
server I decided not to allow "it" to determine to me what I could and
couldn't
do without it; uw-imap "plays with others".
These developers simply can not grasp the concept of a
stateful protocol
like IMAP [...]
Unfortunately, years of repeated attempts to educate these developers
proved to be utterly futile. When you talk about "state", you get
vacant
stares.
If they can understand the difference between TCP and UDP, they can
understand
state.
Thus, there is no hope of a working version of crapware
such as Outlook or
Thunderbird. The only thing that you can do is see what you can do on
the
server to not unduly stress the limitations of the crapware.
Well, with Thunderbird at least the avenue is (still?) open to muck
with the source.
Perhaps when I've nothing else to do, try to rewrite their IMAP support
to behave
properly and send them the code, or (ha ha) offer my IMAP client as a
"plugin" for
Thunderbird (ha ha.)
One way to do accomplish this is to switch mailbox formats
to mix format.
I can't imagine how anyone can tolerate traditional UNIX mailbox format
with a 100 MB mailbox, much less larger. mix format was designed to
accomodate such large mailboxes; and does in a matter of a second or
two
what can take minutes with traditional UNIX mailbox format.
[6] What you describe is typical of scaling problems. It seems to
"work"
until you reach a critical point. Very likely, the mailboxes got bit
enough that operations now tend to take long enough for Thunderbird to
decide to give up.
I've counseled splitting mail folders up and also patience to allow
operations to
complete, but evidently the volume for the users in question puts the
splitting
behind the congestion. I'm averse to storing user mail in a
noncompatible format
(meaning the POP server can't play) but may have to, for these users at
least
(I'll put them in their own sandbox.) If POP supports the mix format,
then that
objection vanishes. The fact that "mbox" is also easily human-readable
is a big
plus, but by the time the mail folder becomes 800 MB in size, being
tortured by
trying to use an editor on the file is no fun either. Then one plays
games with
"head" and "tail" and "wc". Yuk.
[7] Both Thunderbird and Outlook have a way to record protocol
negotiations. The c-client library also has a mechanism to record
protocol negotiations, at the client (not server) level.
You want to do client-based telemetry, not server-based, due to the
size
of the protocol logs. IMAP protocol logs are HUGE, and on anything
beyond
a small test system you will quickly be buried under the weight of the
logs. On a small test system, you can easily set up a script for
server
logs using "tee", at the cost of giving up SSL capability.
You quickly find out that the server logs are much less useful than you
thought. I went through this exercise on my new IMAP server.
The effects you describe explain what I see, and you've provided an
"out" (mix format),
so I will not worry about the logs. Thanks much. Thanks also to Linda
for the reference
to the other server software. I'll do fixes one way (uw+mix) or the
other (suite to test it.)
Argh, obviously the question becomes how does one convert existing mail
folders to mix
format ... or is there a magic cookie such that the "mix handler" knows
to bow out,
and or is there any automatic (blind) way that upon first use the
program can do a
conversion in the background? All I'd need is a filter to convert
"mbox" to "mix", and
stop the users getting in until that is done.
Thanks again.
-ecsd (Eric Dynamic)
Oakland, Californa
Remember to eat GILROY (California) garlic, not the imported tasteless
Chinese stuff, thanks. :)
==
On Fri, 30 Apr 2010, support wrote:
We have users whose folders are between 100
MB and 800 MB in size.
Most of those users are using Outlook but some are using Thunderbird.
Lately (and seemingly suddenly) the users are encountering trouble in
that
they will move one or more items from the inbox to another folder and
get
inconsistent results:
* sometimes the move is honored, with the item appearing in the target
folder and disappearing from the source folder.
* sometimes the move is partly honored in that the item appears in the
target folder, but is NOT deleted from the source folder.
* and sometimes the move is not honored at all, despite the client
email
program thinking for a moment that it was (before seeing in moments
after that it wasn't.)
The server is FreeBSD 5.4 using imap2006e, I think. I'll upgrade to
imap2007, whatever's current in the FreeBSD ports tree, to see if it
helps.
(1) I insured that /var/mail and /usr/tmp were permission 1777 as
suggested, but just now so I don't know if that will prove to help.
Things used to work just fine without these changes; the only things
that may have changed are (a) the users' mail folders have gotten
larger, (b) the versions of Outlook they are using are new. Thunderbird
(the latest in general) also suffers the failed transfers, though, and
only "suddenly" now for no apparent reason.
(2) I see no reference to debugging on the website. I thought I could
perhaps turn on a flag for imap in inetd to be able to track requests
to
try to see what the server was being told to do, but I see no mention
of
any such facility for either imap or c-client.
-- Mark --
http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
|
