[Imap-uw] SSL and systemd
I have panda-imap running on CentOS 7 with xinetd. We wanted to implement it using a systemd service instead, so I made files /usr/lib/systemd/system/imaps@.service imaps.socket imap.socket imap@.service The server appears to work on port 143 with STARTTLS (or notls) But on port 993 it responds with plaintext. If I enable DEBUG logging in syslog, with xinetd I get messages like Login user=andrew host=blah.example.com [192.168.2.3] imaps SSL service init from 192.168.2.3 but with systemd service I get Login user=andrew host=[NON-IPv4] Line too long before authentication host=[NON-IPv4] I suspect some code in server_init in osdep/unix/env_unix.c which I don't entirely understand Has anyone solved this ? -- Andrew Daviel, TRIUMF, Canada ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman13.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] IDLETIMER and slow/buggy networks
Recently we had some users in one office who were unable to send mail
attachments with Thunderbird - they sent OK with SMTP, but when
Thunderbird tried to save a copy in the user's Sent folder, the operation
failed.
When I looked on the wire, I could see something like this:
20 IDLE
+ Waiting for DONE
DONE
20 OK IDLE completed
21 append Sent (\Seen) {820767+}
Message-ID: blahblah...
.. lots of base-64 text ..
i3X00+ZAg/gDi* BYE Autologout (idle for too long)
The logfile on the server I think logs imapd[n]: Unexpected client
disconnect..
In that office, it looks like we have a real network problem - there are
lots of TCP retransmits on larger packets, so the transaction is running
afoul of IDLETIMER which is hardcoded at one minute in imapd.c.
But that's making mail fail, rather than just running real slow like, for
example, scp.
Since it seems to me that imapd is waiting for DONE, not just network
activity, I was wondering if it ever times out on a non-broken network -
one that's just naturally slow, for instance.
E.g. our sendmail has a byte limit of some 60Mb (some servers have no
limit), so I could conceviably send a 50Mb attachment from home on my
feeble 400kbps uplink, which would take much longer than 1 minute to
write to Sent (I've yet to actually try that).
I do see quite a few Unexpected client disconnect messages in syslog in
general, but have not yet joined all the dots together - I'm not sure if
it's a general imapd issue, or an interaction with Thunderbird.
I was wondering if my interpretation of this is correct, and whether
increasing IDLETIMER might be a good idea.
--
Andrew Daviel
TRIUMF
___
Imap-uw mailing list
Imap-uw@u.washington.edu
http://mailman13.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] MIX and NFS
On Tue, 1 Oct 2013, Oscar del Rio wrote: On 10/ 1/13 08:49 PM, Andrew Daviel wrote: We are using UW imapd 2007 with MIX format mailboxes. Our storage expert would really like to use an NFS-mounted volume on a native ZFS appliance, saying it's much better for snapshots and block-oriented backups than iSCSI. Thanks Oscar for your answers I have been looking at this again, as our storage guy said Oh but Linux supports the local_lock switch in NFS mount. Yes it does - from Linux 2.6.37. But I found a patch to backport that to 2.6.18, which is coincidentally what I am running on CentOS 5. So I've been playing. I ran mailutil to repeatedly append and prune a mailbox on NFS, while dmail keeps delivering mail. Sure enough, after a few minutes, I get mailbox corruptions. Then I built a patched nfs_utils and nfs.ko module, and tried mounting the disk with e.g. mount -t nfs -o rw,nosuid,local_lock=all UW imap defines safe_flock in c-client/flocklnx.c, which makes flock() a no-op, citing https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123415 But that's from 2005, talking about Linux 2.4.21. http://en.wikipedia.org/wiki/File_locking says that Kernel 2.6.12 and above implement flock calls on NFS files using POSIX byte-range locks and the kernel code fs/nfs/file.c says We're simulating flock() locks using posix locks on the server. No BSD flocks over NFS allowed.' Seems to me flock() works now. The flock shell command works fine on an NFS volume (flock lockfile bash). In spite of the fact the manpage for flock() (2) still says it doesn't. So I commented out the no-op in safe_flock, and my test setup works fine - no mailbox corruption - even with local_lock=none. However, it does occasionally hang for a bit - it may be triggering the 5-second sleep in safe_flock() although I don't see any syslog messages. Or it may be the NFS server itself. If I set local_lock=all, again I get no corruption, and I don't get any hangs either. I haven't tried exhaustively yet - running the test for hours - or a proper mailtest suite. If you are planning to change storage, maybe it is also time to consider switching to another server. We recently moved to dovecot with mdbox format, which is similar to Mark's MIX format (but not compatible). I found e.g. http://www.dovecot.org/list/dovecot/2012-June/066291.html http://wiki.dovecot.org/NFS which implies Dovecot may not be 100% robust on NFS. -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] MIX and NFS
Now that Mark sadly is no longer with us, maybe I can raise this without getting my head bitten off. We are using UW imapd 2007 with MIX format mailboxes. Our storage expert would really like to use an NFS-mounted volume on a native ZFS appliance, saying it's much better for snapshots and block-oriented backups than iSCSI. Given that we are only trying to support one flavour of Unix (Linux), and possibly using NFS4 on Centos6/Linux 2.6, is there still a problem with NFS, if the only processes opening mailboxes are using c-client libraries (dmail, imapd, mailutin, alpine) ? I have read some of the copious texts about Sys V and flock() and am not sure what applies to Linux. -- Andrew Daviel, TRIUMF, Canada ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] Intermediate SSL certificates in UW-imapd
We have had UW imapd-2007e running fine for some time. Recently our SSL certificate provider announced that we had to upgrade our 1024-bit certificate to 2048 before it was revoked. The replacement they provided is not signed by a CA in the Mozilla store (and hence OpenSSL which copies it) - it uses an intermediate certificate. I got Apache to work by following instructions and defining SSLCertificateChainFile OpenLdap seems happy with the intermediate certificate added to the bundle defined as TLSCACertificateFile. But imapd does not. It uses a PEM file with both the key and certificate, and the OpenSSL library. OpenSSL (at least, the openssl shell command), uses the bundle in /etc/pki/tls/cert.pem. But Alpine, at least, is unhappy if I just add the intermediate certificate to that. Is there a solution ? Or should we just order a proper directly-signed certificate ? uw-imap-2007e openssl-0.9.8e-20 CentOS 5.7 Linux 2.6.18 x86_64 -- Andrew Daviel, TRIUMF, Canada ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] Concurrent IMAP connections from MacOS
I note that modern email clients maintain multiple concurrent IMAP connections to the server, e.g. Thunderbird has a (configurable) default of (?) 5, which would speed up polling multiple inboxes on one server and allow faster message copying between multiple folders. Excessive connections from one client, on the other hand, may be a dictionary attack. I had set a per_source limit of 10 on POP and 20 on IMAP in Linux xinetd.d/*. That works fine with Thunderbird and Squirrelmail, Alpine etc. (we don't have big NAT'ted client farms). However, we've been getting connection problems from MAC machines and logging xinetd per_source_limit failures. Is this a bug in Mac Mail ? Does it really need that many connections, and if so how many ? I'm reluctant to dedicate so much server resources to one or two users for no good reason, or weaken filters against network attack. -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] imapd automove from spool to inbox
Pardon the weird title, but I have a weird situation. We use procmail for mail delivery via dmail, but historically did not trust users not to mangle procmailrc and lose their mail. So we have some templates, like all mail in INBOX, and file spam in Junk, with user's .procmailrc a symbolic link to the template, which belongs to root. A little while ago I screwed up while changing a user UID, and accidentally changed the ownership of the template. Later I found syslog entries like procmail[3774]: Suspicious rcfile /home/foo/.procmailrc and realized that user's rcfiles that used that template were not being used (except for foo). But there was no outcry of missing mail - only one user noticed that tagged spam was getting in his inbox instead of being filtered to Junk. It seems that procmail reverted to delivering mail to /var/spool/mail/foo. However, when imapd ran in response to a user client, the messages were magically copied across back to /home/foo/INBOX (a mix folder) This is a nice feature to have, as it turns out, but I had no idea that it existed. Is it a design feature ? Can I trust it ? -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] using openssl s_client with uw servers
On Tue, 9 Nov 2010, Andrew Daviel wrote: Has anyone used openssl with UW imapd/pop3d to check certificates ? I use it to check pop3s on port 995, and imaps on port 993. That works OK,e.g. echo 0 LOGOUT|openssl s_client -showcerts -connect $host:993 Any ideas ? Do I need some particular option on the command line ? FYI, fixed it. I just needed to add -tls1, viz. echo 0 logout | openssl s_client -showcerts -starttls imap -connect $host:143 -tls1 -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] running out of disk space
On Thu, 2 Jul 2009, Mark Crispin wrote: Ideally, dmail should fail gracefully. It's supposed to, yes. My bad. I had an old version on my old desktop; dmail has no --version argument but I'm guessing it's from 2006 or earlier. replicating error by deliberately filling up a Linux ext3 partition: old version: Error updating mix index file message delivery failed to /boot/foobar/mail/xxx invalid mailbox name foobar+mail/xxx delivered to /var/spool/mail/foobar - leaves mailbox broken subsequent attempts: Oversize mix status record Can't open append mailbox: Success message delivery failed to /boot/foobar/mail/xxx new (2007) version: Error flushing mix index file Error in mix status sequence record, i=4a53bce4, seq=4a53bce5 Reclaimed 22789 bytes of expunged space message delivery failed to /boot/foobar/mail/xxx invalid mailbox name foobar+mail/xxx delivered to /var/spool/mail/foobar - leaves mailbox OK -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] speeding searches by pre-indexing
On Fri, 12 Jun 2009, Mark Crispin wrote: On Fri, 12 Jun 2009, Andrew Daviel wrote: 129,634 messages in 23 hours Yikes! What is that? Indexing time? Search time? Indexing time. Sorry, stripped off the original detail. I was trying to get a global search for all my mailboxes by pointing htdig at my webmail application. So what I end up with is a web search engine that returns links into webmail. Works, friendlier than grep whatever mail/.mix*, but indexing too slow for real world use. The slowness is almost all in htdig (the indexing robot). It's single-threaded, and when it was written hammering websites at 2 requests/minute was considered rude. -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] running out of disk space
I ran out of disk space this morning, and dmail broke the index files on several of my inboxes. Subsequent access with dmail or Pine gave Oversize mix status record and I had to repair the mailboxes, after deleting some junk and creating a new inbox so I would not lose any more (ought to have got temporary failure and returned to local mail queue, but I'm not sure. Caught it relatively quickly) Ideally, dmail should fail gracefully. I'm not sure quite how. Seeing if there's enough space to write the message, or double-buffering the index file maybe. Fortunately this was all on my desktop, not the main mail server, otherwise I'd still be picking up the pieces tomorrow. Note to self - make quite sure we don't run out of space. -- Andrew Daviel, TRIUMF, Canada ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] speeding searches by pre-indexing
On Wed, 27 May 2009, Andrew Daviel wrote: Right now I'm running it across 100Bt on 1Gb (10%) of my personal mail, and I'll see how many days that takes ... :-7 129,634 messages in 23 hours ... guess I won't be doing that on a daily basis.. on the other hand, 2100 messages in 3min on the local machine (faster hardware, too) is a bit less unreasonable .. still 2 months to do 900 users. yikes! Search interface is general Web one - no search by From or Subject specifically, just by free text (subject appears in HTML title, which gets more weight etc.) -- Just went to a presentation by Zimbra. Seems they index mail as it goes in (or is uploaded with IMAP) and can search across mailboxes with their Web interface. Per-mailbox with IMAP, but using the same fast index. -- Andrew Daviel, TRIUMF, Canada ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] imapd aborts with header size inconsistent
I was just playing with the Zimbra desktop client - at least in Linux, it's a Java app that can do IMAP as well as zimbra, gmail etc. When trying on my test account, it hangs while syncing. It seems that it wants to download everything, not just subscribed mailboxes. I have various unavoidable non-mailbox files like .procmailrc and .vacation.db, but also some other cruft. One of which is a broken mail file of some kind, another is a virus sample, which cause imapd to bomb As zdesktop won't run in plaintext mode I can't check the commands, but I can reproduce the effect, viz. $ imapd 1 list h* * LIST (\NoInferiors \UnMarked) / head1 1 OK LIST completed 2 select head1 * 1 EXISTS * BYE [ALERT] IMAP4rev1 server crashing: header size inconsistent Aborted syslog contains Fatal error user=xx host=yy [a.b.c.d] mbx=/x/y/z/head1: header size inconsistent zdesktop reports this as Failure communicating with remote server. Please try again later. and can't get past this point I was wondering why imapd does not merely return BAD to the SELECT operation, and continue. Or is the thread too screwed up at that point to safely do so ? -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Problem deleting folders with Thunderbird
On Thu, 21 May 2009, Mark Crispin wrote: On Thu, 21 May 2009, Andrew Daviel wrote: If in Thunderbird the option server supports folders which can contain both folders and messages is checked No client should ever have, or need, such a configuration option. This is handled by the IMAP protocol. With the option set, the GUI has a single option create subfolder. With it clear, it asks if you want a folder or a directory. As far as I can see, with MBX or Unix, create test will do open (test, O_RDWR) and create test/ will do mkdir (test). In RFC 3501 you say If the mailbox name is suffixed with the .. hierarchy separator .. the client intends to create mailbox names under this name. Which implies that the client needs to know which to create. Since the server workings are (rightly) hidden from users, and the paradigm is that a folder on a GUI desktop contains both files and folders, users will assume that a folder can contain both subfolders and messages. Maybe I am missing something, but I can't see anything in the protocol for a client to determine whether a server can create subfolders without testing it. Andrew ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] speeding searches by pre-indexing
On Fri, 22 May 2009, Andrew Daviel wrote: I was wondering if it was possible to use a regularly-computed word index to speed up searching, as per Web search engines. Hmm .. I can point htdig at my tinymail webmail interface, which uses SSL/HTTP basic auth, setuid and preauthenticated httpd. A bit of tweaking at both ends, to make tinymail not so tiny and to make the HTML better optimized for searching and replace some POST links with followable GET links, and I might have a passable searchable mailstore. But not searchable via IMAP - at least, not without a lot of hacking. -- Andrew Daviel, TRIUMF, Canada ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] Problem deleting folders with Thunderbird
We have a problem with users trying to delete folders using Thunderbird. Part of it is a TB problem IMO (some discussion with Mark a while back; TB is trying to delete an open folder without deselecting it). However, another issue seems to be that imapd is not unsubscribing from the deleted folder properly from network capture: 3 list Trash/test/* * LIST (\HasNoChildren) / Trash/test/ 3 OK LIST completed 4 delete Trash/test/ 4 OK DELETE completed 5 unsubscribe Trash/test/ 5 NO Not subscribed to mailbox Trash/test/ 6 logout - at this point, Trash/test/ has disappeared from the filesystem, but there is still an entry Trash/test in .mailboxlist. Thunderbird seems to ignore error status and thinks it has successfully unsubscribed - the icon disappears. But after exiting and restarting (logging in again), the icon reappears pointing to a mailbox that cannot be selected. imapd-2007, as I recall https://bugzilla.mozilla.org/show_bug.cgi?id=357911 I submitted 414101 now marked as a duplicate -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] SELinux and dmail
I just upgraded to Fedora 9 at home (disk went bad, FC4 didn't support
new SATA disk on my motherboard, Firefox 3 wouldn't run ... time to
change)
Anyhow, newer Fedora (and RHEL 5 on my new desktop at work) come with
SELinux turned on, and I thought I should get my head around it instead
of turning it off (after all, it is a security feature and I'm supposed
to know about that stuff..)
So, a ton of SELinux errors (10,000 and counting) as soon as I start
importing my stuff onto a virgin system. One relevant to this list is for
dmail
I don't really know what I'm doing here (yet). This is still the standard
sendmail config calling procmail as local delivery; .procmailrc feeds
|dmail. I used chcon to set the attributes for dmail the same as
procmail thinking that might help, but no. The mail is
actually being delivered, though.
.. I just thought .. this is my old home partition using Reiserfs 3 on
sdb10. According to some other error message I'd seen, an ISO volume
(CDROM loopback) also has nfs type as it doesn't support the extended
attributes. So maybe if I move my home directory to the new disk under
ext3, and relabel it, the errors will go away. But it is quite common I
think to have NFS-mounted home directories - certainly we do it at work
on clustered machines - and I presume this would have the same problem.
Anyone done this before ?
Any idea what setroubleshooter is talking about getattr to / ?
I could not find getattr in the source code, and if it's
trying to access my mail folder, that's on /home not /
I also have a script ~/bin/nmail which procmail calls to write message
headers to a pipe that lists incoming mail in a window. That fails
totally. Maybe the same issues around my reiserfs partition
--
SELinux is preventing dmail (procmail_t) getattr to / (nfs_t).
SELinux denied access requested by dmail. It is not expected that this
access is required by dmail and this access may signal an intrusion
attempt.
Source Context:Â Â system_u:system_r:procmail_t:s0Target
Context:Â Â system_u:object_r:nfs_t:s0Target Objects:Â Â / [ filesystem
]Source:Â Â dmailSource Path:Â Â /usr/bin/dmail
denied { getattr } for pid=8337 comm=dmail name=/ dev=sdb10 ino=2
scontext=system_u:system_r:procmail_t:s0
tcontext=system_u:object_r:nfs_t:s0 tclass=filesystem
--
SELinux is preventing sh (procmail_t) execute to ./nmail
(nfs_t).
--
Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376 (Pacific Time)
Network Security Manager___
Imap-uw mailing list
Imap-uw@u.washington.edu
http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] FYI: procmail/dmail rule
On Mon, 7 Jul 2008, Jeroen van Aart wrote: Andrew Daviel wrote: We run procmail rules for server-side spam filtering, using dmail as the delivery agent for MIX format. Generally this works well. Wouldn't spam filtering at the smtp level be more effective and cause less traffic? It's important to be able to reject a message asap, preferably before any data goes through (grey listing, rejecting certain IP-addresses, etc.). So you have the least possible waste of resources. Yes, we do that too - RBL at RCPT time plus spamassassin milter at DATA time. Currently it looks like 80% is rejected at RCPT, 10% at DATA, and 3% filtered on delivery, leaving 7% legitimate mail. The RBL rejected number seems on a linear rise over the last year :-( Mail scored over one threshold is deemed definitely spam and rejected by SMTP DATA response. Mail scored over a lower threshold is deemed possible spam and accepted, allowing users to filter it, adjust whitelists etc. They can also have multiple inboxes for mailing lists etc. if they wish. Mail scored below the low threshold is deemed ham (sic). One problem with filtering up front is that messages must be scored before they are cloned, meaning that a user's personal whitelist cannot be applied if there is more than one recipient. Just in case we have some spam-lovers, or commercial newsletters that trip the threshold... -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Problem deleting MIX folders - locking
On Mon, 28 Jan 2008, Rodolfo Cossalter wrote: This problem has been reported earlier: http://www.mail-archive.com/imap-uw@u.washington.edu/msg00808.html Ah. October 2006. Didn't find those (I never seem to hit the right keywords and end up submitting duplicates). Is it specific to MIX ? Or whether the dual use flag is set in Thunderbird ? I did not try that, just verified that I could not delete an empty folder. RFC 3501 says that a SELECT does an implicit CLOSE, but doesn't mention DELETE or say that a mailbox must be closed before deletion, so I guess it is one of those implementation dependant things (or of course it should be closed first. it's too obvious to mention ..) -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) Network Security Manager ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] repairing Mix ?
I think Mark said there would at some point be a Mix repair tool. We had a power bump last night and I have a user with .mixstatus and .mixindex full of nulls. I may have a go at recovering it, or may have a recent backup, but wondered if this had already been done (writing a tool, that is). -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) [EMAIL PROTECTED] ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] repairing Mix ?
On Fri, 15 Dec 2006, Mark Crispin wrote: Inspect the .mixmeta file in the damaged mailbox. Is it still sane? Compare it with the one in the restored copy. I may have lost the damaged one (I mean, I ran Pine which may have overwritten it, but I think it was sane What I did: - restored .mixstatus and .mixindex from a dirvish backup (easy - it's just an rsync copy) - now I could read all the messages (a sample anyway; there are 3000 messages) But there were about 10 messages arrived since the backup was made, so I cut them off the end of the .mix data file and made them into a Unix mailbox by the expedient of sed 's/^:msg:.*/From some plausible string/' then checked it with Pine, then did $ mailtutil append mailbox extra-messages This pretty much seemed to have fixed things. It is unlikely that the user did a lot of work (deleting, answering) between the backup at 11:45 pm and the power bump at 6am The broken .mixindex and .mixstatus look to have been the original length, but contained 100% zeroes (ASCII NUL) For repairing MBX files, I found that if data went missing in the middle of a message, then the pointer list got lost so I had to regenerate it from SMTP message headers (as marking a new message). I would think that still applies to MIX, though perhaps it's not so likely to break. It would be nice to start from a broken status file to restore message status where it exists, rather than set it all to zero, but you'd still need the option to recreate .mixstatus entirely. Though in that case it may be easier to make a Unix box as I just did and convert it back. Incidentally, in mixfmt.txt for .mixindex it says : uid:date:size:hsiz:file:pos: (6 fields) while in mix.c it defines uid, date, size, spare.data, special.offset, msg.header.offset, msg.header.text.size (7 fields) Hmm, in mixfmt it does actually list 7 fields, so isiz is missing from the list and I'm confused about the order. (in my last problem, with a missing .mix file, I suspect it was probably there all along but in a different save set due to incremental backup.) -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) [EMAIL PROTECTED] ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Corrupted Mix mailbox in 2006c
Apologies for the delay - solved the user's problem and got caught up in other things ... On Thu, 2 Nov 2006, Mark Crispin wrote: The .mix4540cf00 file is over 1MB. Did you set MIXDATAROLL higher than its default of 1MB? Yes, I did - to 4Mb In the .mixindex file, look at the per-message records (anything after the first line). The fourth colon-delimited field is the mix data file number. Is it 4540cf00 in the damaged messages too? No, they are supposed to be in 454083bf What I wonder is whether the 38 lost messages are in a different data file which somehow has gone missing. Quite possible. I'm not 100% confident of amanda, particularly in recovering a single directory (as opposed to the entire filesystem into some multi-gigiabyte scratch space). Could you elaborate on what you mean by crash Pine? Pine should never crash, even if in this circumstance. SIGSEGV Did you get any error messages? If so, what messages? # pine -i -f /home/marcello/30oct06backup/ try to read message #2 (missing) Problem detected: Received abort signal(sig=11). Pine Exiting. Attempting to save debug file to /root/.pine-crash Aborted .pine-crash has nothing especially relevant, IMO (but it's 68kb, so I'm not posting it all here) last few lines are: 14:27:52.575788: Hit: Returning 0x9f15d68 - 19 (2154) 14:27:52.575792:(121), 0x992e7b2b 14:27:52.575813: --update_index done 14:27:52.575825: STATUS cmd:2066, max:-1, min-1 14:27:52.575839: cancel_busy_alarm(-1) 14:27:52.975903: Read char returning: 13 RETURN 14:27:52.975915: Read command returning: 13 RETURN 14:27:52.975920: New_mail_count zeroed 14:27:52.975932: - process_cmd(cmd=505) - 14:27:52.975956: - MAIL VIEW - 14:27:52.975965: busy_alarm(1, Busy, (nil), 0) try again with gdb read #1 OK, #2 OK, #3 crash Program received signal SIGSEGV, Segmentation fault 0x0824fe95 in rfc822_parse_content () (gdb) bt #0 0x0824fe95 in rfc822_parse_content () #1 0x0824ef54 in rfc822_parse_msg_full () #2 0x0823fd55 in mail_fetch_structure () #3 0x081a77c8 in pine_mail_fetchstructure (stream=0x99bd0d0, msgno=3, body=0xbfed3fa4) at pine.c:5745 #4 0x0813c589 in mail_view_screen (ps=0x99ae008) at mailview.c:459 #5 0x0819c0ea in main (argc=4, argv=0xbfed55d4) at pine.c:1342 I built Pine with the new c-client libraries, but I probably built imapd/c-client with no debugging to try and reduce the size of the image (all those imapd and dmail instances sitting in memory..) -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) [EMAIL PROTECTED] ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] Suggestion: set labels/flags in dmail
I'm working on a web interface to procmail - some users wanted it, and it
seemed easier to do than it is - to allow users to build server-side
filters instead of picking one of my canned ones.
Looking at the filters in Thunderbird to get ideas for the layout,
I notice that there is a mark message as flagged/important/work etc.
option. I'm using dmail from procmail, and I note there is the single
option -s mark message as seen.
I made a -F flag option which seems to work OK, so
I should be able to have procmail entries like
:0
# flag as important
*
|dmail -F \$Label1 +INBOX
and maybe I can have fun colouring all my TRIUMF mail as blue (work) etc.
(Pine may have messed up some whitespace..)
--- dmail.c.orig2006-11-27 23:51:21.0 -0800
+++ dmail.c 2006-11-27 23:57:40.0 -0800
@@ -47,7 +47,7 @@
int trycreate = NIL; /* flag saying gotta create before appending */
int critical = NIL;/* flag saying in critical code */
char *sender = NIL;/* message origin */
-
+char *setflag = NIL; /* message flag to set */
/* Function prototypes */
@@ -147,6 +147,10 @@
case 's':/* deliver as seen */
flagseen = T;
break;
+ case 'F':
+if (argc--) setflag = cpystr (*++argv);
+else _exit (fail (missing argument to -F,EX_USAGE));
+break;
case 'f':
case 'r':/* flag giving return path */
if (argc--) sender = cpystr (*++argv);
@@ -376,11 +380,17 @@
((sbuf.st_mode S_IFMT) == S_IFDIR) ? directory : file,path);
mm_dlog (tmp);
/* do the append now! */
- if (!mail_append_full (prt,mailbox,flagseen ? \\Seen : NIL,NIL,st)) {
-sprintf (tmp,message delivery failed to %.80s,path);
-return fail (tmp,EX_CANTCREAT);
- }
- /* note success */
+ if (setflag) {
+if (!mail_append_full (prt,mailbox,setflag,NIL,st)) {
+ sprintf (tmp,message delivery failed to %.80s,path);
+ return fail (tmp,EX_CANTCREAT);
+}
+ } else {
+if (!mail_append_full (prt,mailbox,flagseen ? \\Seen : NIL,NIL,st)) {
+ sprintf (tmp,message delivery failed to %.80s,path);
+ return fail (tmp,EX_CANTCREAT);
+}
+ }/* note success */
sprintf (tmp,delivered to %.80s,path);
mm_log (tmp,NIL);
/* make sure nothing evil this way comes
*/
--
Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376 (Pacific Time)
[EMAIL PROTECTED]
___
Imap-uw mailing list
Imap-uw@u.washington.edu
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] FYI, my conversion scripts
I've now converted all inboxes and most other large boxes that people have been using (to Mix that is) via a script to create 4Mb datafiles. I find people with spaces in folder names (ugh), and even an asterisk, ampersand and brackets (double ugh). The script copes with spaces but had some trouble with the brackets... If anyone's interested I have placed some scripts in http://andrew.triumf.ca/mbx/ They are a bit rough and some make some assumptions, e.g. that user's files are in /home/user, and that certain files (INBOX, junk) are inboxes etc. -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) [EMAIL PROTECTED] ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] mailutil, flags and POP
If I convert a mailbox with mailutil, the old flags get unset, causing all the messages to appear as recent. e.g. $ mailutil check mail/testf1.mbx 1 new message(s) (8 unseen), 20 total in mail/testf1.mbx $ mailutil copy mail/testf1.mbx '#driver.mix'/mail/testf1.mix $ mailutil check mail/testf1.mix 20 new message(s) (8 unseen), 20 total in mail/testf1.mix I tried also mbx - mbx with similar results A unix mailbox always shows no recent messages with mailutil, even if a new message was just delivered with dmail. Hmm, looks like all the user flags are also not being copied into MBX, i.e. Junk, NonJunk, $Label1 etc. (used by Thunderbird for Important, Later etc.) Mailutil says: warning: Unknown flag: $Label1 warning: Unknown flag: NonJunk warning: Unknown flag: Junk even though these labels are in the MBX file prefix and known to both Mix and MBX. mbx to mbx - says Unknown flag, loses old, Junk, $Labeln mbx to mix - loses old mix to mix - loses old mix to mbx - says Unknown flag, loses old, Junk, $Labeln I have not yet converted many users to mix format, but I have had some complaints from MBX users that old mail is being downloaded and deleted mail has reappeared. So far I have: - upgraded from imap2004 (2?) to 2006a - added the UID patch - transferred accounts to new hardware with more recent OS version with rsync - upgraded from 2006a to 2006b One complaint was from just before the 2006b upgrade, but I believe several days after the patch was applied. I presume these are POP users (sometimes I hear from non-technical people in a roundabout way days later). I think there is an 'old' flag in NBX that gets set on non-recent messages. What happens to this flag when UIDs are rebuilt ? Hmm; looking on the wire with Thunderbird using POP3, I see it's using LIST, UIDL and RETR, and there's no reference to recent. So I guess the client has a map of what message IDs it downloaded already, and if the message UIDs are rebuilt it will see old messages as new. This seems an odd way of doing things - the client remembers every single message ID it's ever seen, even deleted messages ?? I was trying to make sure that if I switch everyone to mix, that all the POP users won't see all their old messages as new. It looks from the above that it should be OK; that the damage has already been done by rebuilding illegal UIDs, if any. But I wanted to check. And I'm not sure how many users are using the new(?) user labels; certainly some are using Junk flags. re. splitting mix boxes (so as to get benefits of smaller file size right after conversion, rather than waiting for new mail to arrive and users to purge old mail (ha!), I wrote something to split MBX boxes. Ugly, but may work for one-time mass conversion: - split user.mbx into user.mbx.1, user.mbx.2, user.mbx.3 etc. - mailutil copy user.mbx.1 #driver.mix/user.mix - mailutil append user.mbx.2 user.mix - mailutil append user.mbx.3 user.mix etc. -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) [EMAIL PROTECTED] ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] IMAP funnies migrating from 2004 to 2006
On Tue, 26 Sep 2006, Mark Crispin wrote: If a rebuild happens repeatedly on the same mailbox, I want to know about it. However, make sure that there isn't any software that is adding messages to the mailbox under the old rules. In particular, I believe that some versions of procmail and/or postfix know about mbx mailboxes, and use their own code rather than c-client library code. We are using a .procmailrc file with |dmail. I only use Pine for maintenance. Does the rebuild happen multiple times for the same mailbox, or just once? AFAIK just once. I see this as I say in Pine run directly on the mailbox file. Does imapd do a rebuild automatically when a client connects, or dmail do it when a new message arrives ? Or does one have to run mailutil ? -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) [EMAIL PROTECTED] ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Re: Benchmarking imap, filesystems
On Wed, 28 Jun 2006, Aaron W. LaFramboise wrote: To Andrew: I can confirm Mark's comments that there is something probably wrong with your benchmark. I have done this test on a similar Linux system (although not the same benchmark), and got pretty much the opposite result as you, with or without disk caching effects. Looking at the URLs you mentioned, I see an exceptionally high Error count for mbx (3,873,234,974) compared to mbox (10). Perhaps this is the source of the problem? This mail has been sitting in my drafts folder (OK, Pine postponed) since before I went on vacation. I changed my approach, and scripted a lot of tests with 10-20 clients and running for about a minute. I preloaded accounts with about 5000 messages in a mix of sizes, added a bunch of recent mails and started the Bonnie IMAP client. However, it still doesn't behave like a human. It certainly doesn't behave like me - I read mail in a random order, reading some, deleting others unread etc. while remaining logged on for hours. So if it seems to be saying that mbox on XFS is fastest, I'm not sure I believe it. I also find a discrepancy between messages retrieved (try/sec) and bytes retrieved. .. if I look at the transfer time graphs, the Mix format seems to show more consistent times and be quicker to get an individual message. But that doesn't show in the summary pages. The powers that be want me to get on and upgrade the mailserver before it melts and stop messing around, so I probably won't spend too much more time on this. I'll probably toss a coin and choose, say, XFS and Mix, which gives me back-compatability with existing files and gets away from huge inboxes with their subsequent backup problems. http://andrew.triumf.ca/linfs/mstone.html Back in July, I started to write: What benchmark software did you use ? Have you looked at Maildir format (e.g. Courier) ? It looks similar to mix, though AFAIK some metadata is coded into the filename rather than stored in separate files, and it sounds like mix has more capability. I notice that RHEL4 now includes Dovecot and Cyrus imapd, not UW. As far as I can tell, though Dovecot supports Maildir format, a delivery agent is not included in RHEL (or I didn't find it). I built Courier maildrop which worked for me. Cyrus I have not yet tried. (mail delivery and how to create an inbox seems different for everything, and moreso for Cyrus) I have seen comments that Dovecot imapd outperforms UW imapd through better memory management, though this is not obvious in mailstone. Maybe the comment applied to an earlier version. I've also seen comments, which I do seem to verify, that a filesystem such as XFS or Reiser is better suited to Maildir/mix structure than traditional ext2/3. (I've had in-house comments about stability problems with Reiser on SL4 (~= RHEL4)). I see XFS being somewhat better, cf. ext2, ext3, Reiser 3.4, and JFS. Andrew ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] Mailbox migration/size in MIX
I have been playing with MIX following Mark's comments; it would address some of the system backup concerns besides being faster. I see that MIX starts a new mailbox file after 1Mb, so that a new inbox would tend to have this as a maximum file size. However, if I migrate an existing 100Mb folder, it becomes a single 100Mb MIX file. (though new mail appended to it goes in a new file). Also, if (in Pine, anyway) a large number of messages are selected from a mailbox and moved to a new one, the new one is created as a single large file - even if the old mailbox had many small files. (I tend to batch up the previous year's worth of messages like this when spring cleaning). So even if I wrote a tool to convert existing boxes in chunks, we could still end up with some large mailboxes. Admittedly, the backup problem is with inboxes that must be copied in entirety if a single flag is changed - not so likely with pre-1995 mail. I gather that there is a technical reason for this, but it does seem to detract from the small file principle. -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) [EMAIL PROTECTED] ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] Mail file fragmentation on Linux
I am running UW imapd and dmail with MBX mailbox format. I noticed using debugfs stat file that the mail file can get quite badly fragmented over time. I wondered if there was any way to discourage this in the MBX code. One thing that occurred to me was that maybe when the file is growed to append new messages, that the requested allocation have room for expansion, e.g. rounded up to the next 20 blocks or something. (maybe it already does this) Just copying the mail file to a new location and moving it back seems to get the fragmentation under control. As an example, after this procedure, my mailbox had fragments of about 1k blocks. Before, a heavy user had fragments of about 3 blocks each, not all contiguous or even in order. Seems to me this must slow down disk access a bit -- Andrew Daviel, TRIUMF, Canada Tel. +1 (604) 222-7376 (Pacific Time) [EMAIL PROTECTED] ___ Imap-uw mailing list Imap-uw@u.washington.edu https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
