Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-26 Thread David Champion
* On 24 Mar 2013, Chris Green wrote: 
 Exactly!  So mutt *doesn't* add a blank line before the 'File ' if there
 wasn't one there already, in fact it doesn't do anything except copy
 what it's given by the MDA.
 
 This may be correct in that mutt isn't an MDA but it isn't what everyone
 here has being trying to tell me!   :-)

I don't know why my last four messages haven't made it through the
mailing list, but actually I have been trying without success to tell
you exactly that. :)  You're spot on.

Mutt adds headers to indicate the message length when it saves a
message.  Apart from headers, it copies the message content exactly.
The blank line is considered part of the message content, so if present
it also gets copied.

I'm sending this mainly for Chris's benefit, but since I'm sending
I might as well try the list again.  BCC to Chris since he used a
Mail-Followup-To.

-- 
David Champion • d...@bikeshed.us


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-25 Thread Chris Green
On Mon, Mar 25, 2013 at 12:20:38AM -0500, David Champion wrote:
 * On 24 Mar 2013, Chris Green wrote: 
  Exactly!  So mutt *doesn't* add a blank line before the 'File ' if there
  wasn't one there already, in fact it doesn't do anything except copy
  what it's given by the MDA.
  
  This may be correct in that mutt isn't an MDA but it isn't what everyone
  here has being trying to tell me!   :-)
 
 I don't know why my last four messages haven't made it through the
 mailing list, but actually I have been trying without success to tell
 you exactly that. :)  You're spot on.
 
 Mutt adds headers to indicate the message length when it saves a
 message.  Apart from headers, it copies the message content exactly.
 The blank line is considered part of the message content, so if present
 it also gets copied.
 
 I'm sending this mainly for Chris's benefit, but since I'm sending
 I might as well try the list again.  BCC to Chris since he used a
 Mail-Followup-To.
 
Thanks David, it does seem to have got to the list.

I think I might try and deliver some messages 'direct' using postfix, I
can set up a dummy user here with no filter/procmail and take a look at
what arrives in their inbox.

If postfix *does* append a blank line to all messages I can then,
maybe, go to the python library developers and say I think what
they're doing is wrong.

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread Brian Salter-Duke
I looked at a mbox that was entirely created under Ubuntu and there are
blank lines at the end of each message before the From line.


On Sat, Mar 23, 2013 at 03:28:29PM +, Chris Green wrote:
 On Sun, Mar 24, 2013 at 01:22:56AM +1100, Erik Christiansen wrote:
  On 23.03.13 12:40, Chris Green wrote:
   Well I first actually tried it and saw no blank line.
   
   I've now looked throught my archive (1845 mailboxes) and most of them,
   saved with mutt, using S[ave], seem *not* to have a blank line either.
   There are some with blank lines between but I suspect that's because of
   migrating back and forth between various formats quite a lot over the
   years.
  
  That is weird, because I'm using:
 Mutt 1.5.21+145 (2a1c5d3dd72e) (2012-12-30)
  but have used a whole string of older mutts over the years. And I've
  never made (or heard of) a related config setting which could explain
  the behavioural difference.
  
   Certainly my current mutt 1.5.21 doesn't put a blank line in there and
   it's quite standard from the Ubuntu repositories (though I suppose they
   *might* have done something funny to it).
  
  So your mailboxes, viewed in vim or similar, have the /^From / line
  immediately following the last non-blank line of the previous message!!?
  That's something I've never seen ... in decades, mostly with mutt.
  
 Here, I've just sent myself three test E-Mails and have saved them to a
 mailbox called test, here is the result:-
 
 From MAILER-DAEMON Sat Mar 23 15:19:20 2013
 Return-Path: ch...@zbmc.eu
 X-Original-To: chris
 Delivered-To: ch...@zbmc.eu
 Received: by chris.zbmc.eu (Postfix, from userid 1000)
 id 94F62380255; Sat, 23 Mar 2013 15:19:20 + (GMT)
 Date: Sat, 23 Mar 2013 15:19:20 +
 From: Chris Green ch...@isbd.co.uk
 To: Chris Green ch...@zbmc.eu
 Subject: Test
 Message-ID: 20130323151920.GC10234@chris
 MIME-Version: 1.0
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 User-Agent: Mutt/1.5.21 (2010-09-15)
 Content-Length: 31
 Lines: 3
 
 This is test 3
 
 --
 Chris Green
 From MAILER-DAEMON Sat Mar 23 15:18:53 2013
 Return-Path: ch...@zbmc.eu
 X-Original-To: chris
 Delivered-To: ch...@zbmc.eu
 Received: by chris.zbmc.eu (Postfix, from userid 1000)
 id B4267380255; Sat, 23 Mar 2013 15:18:53 + (GMT)
 Date: Sat, 23 Mar 2013 15:18:53 +
 From: Chris Green ch...@isbd.co.uk
 To: Chris Green ch...@zbmc.eu
 Subject: Test1
 Message-ID: 20130323151853.GA10234@chris
 MIME-Version: 1.0
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 User-Agent: Mutt/1.5.21 (2010-09-15)
 Status: O
 Content-Length: 31
 Lines: 3
 
 This is test 1
 
 --
 Chris Green
 From MAILER-DAEMON Sat Mar 23 15:19:06 2013
 Return-Path: ch...@zbmc.eu
 X-Original-To: chris
 Delivered-To: ch...@zbmc.eu
 Received: by chris.zbmc.eu (Postfix, from userid 1000)
 id 12196380255; Sat, 23 Mar 2013 15:19:06 + (GMT)
 Date: Sat, 23 Mar 2013 15:19:06 +
 From: Chris Green ch...@isbd.co.uk
 To: Chris Green ch...@zbmc.eu
 Subject: Test2
 Message-ID: 20130323151905.GB10234@chris
 MIME-Version: 1.0
 Content-Type: text/plain; charset=us-ascii
 Content-Disposition: inline
 User-Agent: Mutt/1.5.21 (2010-09-15)
 Status: O  
 Content-Length: 31
 Lines: 3
 
 This is test 2
 
 --
 Chris Green
 
 
 I saved them separately (i.e. I didn't do a tag-save), nothing else
 special at all, as you can see I didn't save them in the order I
 sent them.
 
 'mutt -v' returns:-
 
 chris$ mutt -v
 Mutt 1.5.21 (2010-09-15)
 Copyright (C) 1996-2009 Michael R. Elkins and others.
 Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'.
 Mutt is free software, and you are welcome to redistribute it
 under certain conditions; type `mutt -vv' for details.
 
 System: Linux 3.5.0-26-generic (x86_64)
 ncurses: ncurses 5.9.20110404 (compiled with 5.9)
 libidn: 1.25 (compiled with 1.25)
 hcache backend: tokyocabinet 1.4.47
 Compile options:
 -DOMAIN
 +DEBUG
 -HOMESPOOL  +USE_SETGID  +USE_DOTLOCK  +DL_STANDALONE  +USE_FCNTL
 -USE_FLOCK   
 +USE_POP  +USE_IMAP  +USE_SMTP  
 -USE_SSL_OPENSSL  +USE_SSL_GNUTLS  +USE_SASL  +USE_GSS
 +HAVE_GETADDRINFO  
 +HAVE_REGCOMP  -USE_GNU_REGEX  
 +HAVE_COLOR  +HAVE_START_COLOR  +HAVE_TYPEAHEAD  +HAVE_BKGDSET  
 +HAVE_CURS_SET  +HAVE_META  +HAVE_RESIZETERM  
 +CRYPT_BACKEND_CLASSIC_PGP  +CRYPT_BACKEND_CLASSIC_SMIME
 +CRYPT_BACKEND_GPGME  
 -EXACT_ADDRESS  -SUN_ATTACHMENT  
 +ENABLE_NLS  -LOCALES_HACK  +COMPRESSED  +HAVE_WC_FUNCS
 +HAVE_LANGINFO_CODESET  +HAVE_LANGINFO_YESEXPR  
 +HAVE_ICONV  -ICONV_NONTRANS  +HAVE_LIBIDN  +HAVE_GETSID  +USE_HCACHE  
 -ISPELL
 

Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread James Griffin
[- Sun 24.Mar'13 at  7:05:44 +  Brian Salter-Duke :-]

 I looked at a mbox that was entirely created under Ubuntu and there are
 blank lines at the end of each message before the From line.

I can confirm that my sent mbox, created by mutt (current version), also
has a blank line at the end of each message. This is not Ubuntu OS, it's
OpenBSD, but the OS shouldn't be relevent. So I don't think it's a
mutt issue.

The other mboxes, created by procmail, also show the correct format:
with a blank line at the end of each message before the From: line of
the proceeding message.

It looks as though your python delivery program might be doing this?
Have you tried taking the python program out of the process to test, to
see if the messages are being appended to the mbox correctly? I know
you've looked at it again and maybe made some changes to it.

Jamie


-- 
James Griffin:  jmz at kontrol.kode5.net 
jmzgriffin at gmail.com

A4B9 E875 A18C 6E11 F46D  B788 BEE6 1251 1D31 DC38


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread Chris Green
On Sun, Mar 24, 2013 at 11:09:31AM +, James Griffin wrote:
 
 It looks as though your python delivery program might be doing this?
 Have you tried taking the python program out of the process to test, to
 see if the messages are being appended to the mbox correctly? I know
 you've looked at it again and maybe made some changes to it.
 
How could the python delivery program have anything to do with it?

I simply s[aved] three messages using the mutt save command, no other
program is involved, mutt is simply writing three messages to the same
mbox.

Are you all sure that mutt isn't simply saving what it's provided with
by the incoming MDA?

If the *delivered* message ends with a blank line (i.e. what's in the
inbox) then what mutt saves will end with a blank line.

To do exactly what I did:-

Deliver some messages to a mbox file.

Check that there is *no* blank line at the end of these messages.

Use mutt to s[ave] (or c[opy]) the messages to another mbox.

Look at the new mbox file.

I think you'll find there's no added blank line.

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread Will Fiveash
On Sun, Mar 24, 2013 at 06:48:25PM +, Chris Green wrote:
 On Sun, Mar 24, 2013 at 11:09:31AM +, James Griffin wrote:
  
  It looks as though your python delivery program might be doing this?
  Have you tried taking the python program out of the process to test, to
  see if the messages are being appended to the mbox correctly? I know
  you've looked at it again and maybe made some changes to it.
  
 How could the python delivery program have anything to do with it?
 
 I simply s[aved] three messages using the mutt save command, no other
 program is involved, mutt is simply writing three messages to the same
 mbox.
 
 Are you all sure that mutt isn't simply saving what it's provided with
 by the incoming MDA?

 If the *delivered* message ends with a blank line (i.e. what's in the
 inbox) then what mutt saves will end with a blank line.
 
 To do exactly what I did:-
 
 Deliver some messages to a mbox file.
 
 Check that there is *no* blank line at the end of these messages.
 
 Use mutt to s[ave] (or c[opy]) the messages to another mbox.
 
 Look at the new mbox file.
 
 I think you'll find there's no added blank line.

When I use the mutt (on Solaris 11) copy function to copy three messages
in an existing mbox file to an new mbox file I see a blank line before
each new message's From  line except for the first message.  Maybe
it's time to break out the debugging tools on your end?

-- 
Will Fiveash


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread Chris Green
On Sun, Mar 24, 2013 at 02:29:09PM -0500, Will Fiveash wrote:
 On Sun, Mar 24, 2013 at 06:48:25PM +, Chris Green wrote:
  On Sun, Mar 24, 2013 at 11:09:31AM +, James Griffin wrote:
   
   It looks as though your python delivery program might be doing this?
   Have you tried taking the python program out of the process to test, to
   see if the messages are being appended to the mbox correctly? I know
   you've looked at it again and maybe made some changes to it.
   
  How could the python delivery program have anything to do with it?
  
  I simply s[aved] three messages using the mutt save command, no other
  program is involved, mutt is simply writing three messages to the same
  mbox.
  
  Are you all sure that mutt isn't simply saving what it's provided with
  by the incoming MDA?
 
  If the *delivered* message ends with a blank line (i.e. what's in the
  inbox) then what mutt saves will end with a blank line.
  
  To do exactly what I did:-
  
  Deliver some messages to a mbox file.
  
  Check that there is *no* blank line at the end of these messages.
  
  Use mutt to s[ave] (or c[opy]) the messages to another mbox.
  
  Look at the new mbox file.
  
  I think you'll find there's no added blank line.
 
 When I use the mutt (on Solaris 11) copy function to copy three messages
 in an existing mbox file to an new mbox file I see a blank line before
 each new message's From  line except for the first message.  Maybe
 it's time to break out the debugging tools on your end?
 
Yes, but did you look in the existing mbox file to see if there were
blank lines betweeen the messages there?

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread Will Fiveash
On Sun, Mar 24, 2013 at 07:36:13PM +, Chris Green wrote:
 On Sun, Mar 24, 2013 at 02:29:09PM -0500, Will Fiveash wrote:
  On Sun, Mar 24, 2013 at 06:48:25PM +, Chris Green wrote:
   On Sun, Mar 24, 2013 at 11:09:31AM +, James Griffin wrote:

It looks as though your python delivery program might be doing this?
Have you tried taking the python program out of the process to test, to
see if the messages are being appended to the mbox correctly? I know
you've looked at it again and maybe made some changes to it.

   How could the python delivery program have anything to do with it?
   
   I simply s[aved] three messages using the mutt save command, no other
   program is involved, mutt is simply writing three messages to the same
   mbox.
   
   Are you all sure that mutt isn't simply saving what it's provided with
   by the incoming MDA?
  
   If the *delivered* message ends with a blank line (i.e. what's in the
   inbox) then what mutt saves will end with a blank line.
   
   To do exactly what I did:-
   
   Deliver some messages to a mbox file.
   
   Check that there is *no* blank line at the end of these messages.
   
   Use mutt to s[ave] (or c[opy]) the messages to another mbox.
   
   Look at the new mbox file.
   
   I think you'll find there's no added blank line.
  
  When I use the mutt (on Solaris 11) copy function to copy three messages
  in an existing mbox file to an new mbox file I see a blank line before
  each new message's From  line except for the first message.  Maybe
  it's time to break out the debugging tools on your end?
  
 Yes, but did you look in the existing mbox file to see if there were
 blank lines betweeen the messages there?

If I edit an mbox file and remove the blank line preceding each From ,
run mutt on that mbox and have it save all the messages to an mbox it
creates I see no blank lines preceding From  in the new mbox.  So you
are correct in that mutt does not add a blank line.  This mutt behavior
hasn't been a problem for me because all my incoming mail is put into
mbox files via fetchmail-procmail which appears to add the blank line.

-- 
Will Fiveash


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread Nathan Stratton Treadway
On Sun, Mar 24, 2013 at 19:36:13 +, Chris Green wrote:
 Yes, but did you look in the existing mbox file to see if there were
 blank lines betweeen the messages there?

For what it's worth, I took a simple test message, saved it into a new
mailbox, and then used vi to make four copies of the message.  I
removed the Content-Length: and Lines: header lines from all four
copies, and the trailing blank line from two of the copies.

When I opened up this mbox file using mutt -f and saved each message
to a new mailbox (in various orders), I found that the presence-or-not
of a trailing blank line was preserved in the copies.

I got the same results from these tests with both with a current
version of mutt, v1.5.21 from the Ubuntu Precise package, and a very
old one, v1.5.9i from the Debian Sarge package.


So... it does appear that Mutt will preserve whatever trailing-blank-
line situation was originally created by the MDA, even as it saves the
messages into new mailboxes (and that it doesn't actually care what
convention was used by other messages previously found in the mailbox
being appended to).

Nathan


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread Chris Green
On Sun, Mar 24, 2013 at 03:28:28PM -0500, Will Fiveash wrote:
   When I use the mutt (on Solaris 11) copy function to copy three messages
   in an existing mbox file to an new mbox file I see a blank line before
   each new message's From  line except for the first message.  Maybe
   it's time to break out the debugging tools on your end?
   
  Yes, but did you look in the existing mbox file to see if there were
  blank lines betweeen the messages there?
 
 If I edit an mbox file and remove the blank line preceding each From ,
 run mutt on that mbox and have it save all the messages to an mbox it
 creates I see no blank lines preceding From  in the new mbox.  So you
 are correct in that mutt does not add a blank line.  This mutt behavior
 hasn't been a problem for me because all my incoming mail is put into
 mbox files via fetchmail-procmail which appears to add the blank line.
 
Exactly!  So mutt *doesn't* add a blank line before the 'File ' if there
wasn't one there already, in fact it doesn't do anything except copy
what it's given by the MDA.

This may be correct in that mutt isn't an MDA but it isn't what everyone
here has being trying to tell me!   :-)


-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-24 Thread Chris Green
On Sun, Mar 24, 2013 at 05:53:05PM -0400, Nathan Stratton Treadway wrote:
 
 So... it does appear that Mutt will preserve whatever trailing-blank-
 line situation was originally created by the MDA, even as it saves the
 messages into new mailboxes (and that it doesn't actually care what
 convention was used by other messages previously found in the mailbox
 being appended to).
 
Yes, exactly right, mutt just keeps what it's given but it *doesn't*
add the blank line before 'From ' itself.

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-23 Thread Erik Christiansen
On 22.03.13 12:54, Derek Martin wrote:
 On Fri, Mar 22, 2013 at 09:04:21AM +, Chris Green wrote:
  Mutt itself *doesn't* put a blank line there, if you S[ave] or
  C[opy] messages to a new mbox the messages have no blank lines
  before the 'From '.

When I save a message to another mailbox, mutt always leaves a blank
line between messages. (Sometimes there are two blank lines - one is
then presumably part of the message.) I have 1092 mailboxes, many with
thousands of messages. Most of them receive mail only by save or copy.
A quick awk script showed one apparent exception to this mutt behaviour
in the 1596 messages in one of those saved-by-mutt-only mailboxes, but
it was a /^From / occurring within an attachment.


 Yet it works with every MDA I've ever used... which seems to suggest
 that this is the expected behavior, and that the Python library got it
 wrong.  And to be honest it's not really *that* surprising... I've
 seen other cases where Python libraries were doing it wrong (whatever
 it might be).

Having only used fetchmail, these last couple of decades, I can only
confirm that it inserts a blank line, thus mimicking mutt's behaviour.
There are several hundred thousand messages in my archive, and I do
occasionally go in with vim, but have never seen one message butt up
against the previous. Checking a delivery mailbox with the awk script
finds no exceptions in 3180 messages.

Erik

-- 
Life is complex: it has a real part and an imaginary part. - Martin Terma



Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-23 Thread Chris Green
On Sat, Mar 23, 2013 at 08:15:27PM +1100, Erik Christiansen wrote:
 On 22.03.13 12:54, Derek Martin wrote:
  On Fri, Mar 22, 2013 at 09:04:21AM +, Chris Green wrote:
   Mutt itself *doesn't* put a blank line there, if you S[ave] or
   C[opy] messages to a new mbox the messages have no blank lines
   before the 'From '.
 
 When I save a message to another mailbox, mutt always leaves a blank
 line between messages. (Sometimes there are two blank lines - one is
 then presumably part of the message.) I have 1092 mailboxes, many with
 thousands of messages. Most of them receive mail only by save or copy.
 A quick awk script showed one apparent exception to this mutt behaviour
 in the 1596 messages in one of those saved-by-mutt-only mailboxes, but
 it was a /^From / occurring within an attachment.
 
Well I first actually tried it and saw no blank line.

I've now looked throught my archive (1845 mailboxes) and most of them,
saved with mutt, using S[ave], seem *not* to have a blank line either.
There are some with blank lines between but I suspect that's because of
migrating back and forth between various formats quite a lot over the
years.

Certainly my current mutt 1.5.21 doesn't put a blank line in there and
it's quite standard from the Ubuntu repositories (though I suppose they
*might* have done something funny to it).

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-23 Thread Erik Christiansen
On 23.03.13 12:40, Chris Green wrote:
 Well I first actually tried it and saw no blank line.
 
 I've now looked throught my archive (1845 mailboxes) and most of them,
 saved with mutt, using S[ave], seem *not* to have a blank line either.
 There are some with blank lines between but I suspect that's because of
 migrating back and forth between various formats quite a lot over the
 years.

That is weird, because I'm using:
   Mutt 1.5.21+145 (2a1c5d3dd72e) (2012-12-30)
but have used a whole string of older mutts over the years. And I've
never made (or heard of) a related config setting which could explain
the behavioural difference.

 Certainly my current mutt 1.5.21 doesn't put a blank line in there and
 it's quite standard from the Ubuntu repositories (though I suppose they
 *might* have done something funny to it).

So your mailboxes, viewed in vim or similar, have the /^From / line
immediately following the last non-blank line of the previous message!!?
That's something I've never seen ... in decades, mostly with mutt.

I've just run the script on a set of 387 mailboxes exclusively written
by mutt (in the last few years, right up to tonight). Absolutely all of
the 11542 messages are separated by blank lines.

Maybe your mutt has indeed been ubuntued.

Erik

-- 
If society fits you comfortably enough, you call it freedom.
 - Robert Frost



Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-23 Thread Chris Green
On Sun, Mar 24, 2013 at 01:22:56AM +1100, Erik Christiansen wrote:
 On 23.03.13 12:40, Chris Green wrote:
  Well I first actually tried it and saw no blank line.
  
  I've now looked throught my archive (1845 mailboxes) and most of them,
  saved with mutt, using S[ave], seem *not* to have a blank line either.
  There are some with blank lines between but I suspect that's because of
  migrating back and forth between various formats quite a lot over the
  years.
 
 That is weird, because I'm using:
Mutt 1.5.21+145 (2a1c5d3dd72e) (2012-12-30)
 but have used a whole string of older mutts over the years. And I've
 never made (or heard of) a related config setting which could explain
 the behavioural difference.
 
  Certainly my current mutt 1.5.21 doesn't put a blank line in there and
  it's quite standard from the Ubuntu repositories (though I suppose they
  *might* have done something funny to it).
 
 So your mailboxes, viewed in vim or similar, have the /^From / line
 immediately following the last non-blank line of the previous message!!?
 That's something I've never seen ... in decades, mostly with mutt.
 
Here, I've just sent myself three test E-Mails and have saved them to a
mailbox called test, here is the result:-

From MAILER-DAEMON Sat Mar 23 15:19:20 2013
Return-Path: ch...@zbmc.eu
X-Original-To: chris
Delivered-To: ch...@zbmc.eu
Received: by chris.zbmc.eu (Postfix, from userid 1000)
id 94F62380255; Sat, 23 Mar 2013 15:19:20 + (GMT)
Date: Sat, 23 Mar 2013 15:19:20 +
From: Chris Green ch...@isbd.co.uk
To: Chris Green ch...@zbmc.eu
Subject: Test
Message-ID: 20130323151920.GC10234@chris
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
Content-Length: 31
Lines: 3

This is test 3

--
Chris Green
From MAILER-DAEMON Sat Mar 23 15:18:53 2013
Return-Path: ch...@zbmc.eu
X-Original-To: chris
Delivered-To: ch...@zbmc.eu
Received: by chris.zbmc.eu (Postfix, from userid 1000)
id B4267380255; Sat, 23 Mar 2013 15:18:53 + (GMT)
Date: Sat, 23 Mar 2013 15:18:53 +
From: Chris Green ch...@isbd.co.uk
To: Chris Green ch...@zbmc.eu
Subject: Test1
Message-ID: 20130323151853.GA10234@chris
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
Status: O
Content-Length: 31
Lines: 3

This is test 1

--
Chris Green
From MAILER-DAEMON Sat Mar 23 15:19:06 2013
Return-Path: ch...@zbmc.eu
X-Original-To: chris
Delivered-To: ch...@zbmc.eu
Received: by chris.zbmc.eu (Postfix, from userid 1000)
id 12196380255; Sat, 23 Mar 2013 15:19:06 + (GMT)
Date: Sat, 23 Mar 2013 15:19:06 +
From: Chris Green ch...@isbd.co.uk
To: Chris Green ch...@zbmc.eu
Subject: Test2
Message-ID: 20130323151905.GB10234@chris
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.21 (2010-09-15)
Status: O  
Content-Length: 31
Lines: 3

This is test 2

--
Chris Green


I saved them separately (i.e. I didn't do a tag-save), nothing else
special at all, as you can see I didn't save them in the order I
sent them.

'mutt -v' returns:-

chris$ mutt -v
Mutt 1.5.21 (2010-09-15)
Copyright (C) 1996-2009 Michael R. Elkins and others.
Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'.
Mutt is free software, and you are welcome to redistribute it
under certain conditions; type `mutt -vv' for details.

System: Linux 3.5.0-26-generic (x86_64)
ncurses: ncurses 5.9.20110404 (compiled with 5.9)
libidn: 1.25 (compiled with 1.25)
hcache backend: tokyocabinet 1.4.47
Compile options:
-DOMAIN
+DEBUG
-HOMESPOOL  +USE_SETGID  +USE_DOTLOCK  +DL_STANDALONE  +USE_FCNTL
-USE_FLOCK   
+USE_POP  +USE_IMAP  +USE_SMTP  
-USE_SSL_OPENSSL  +USE_SSL_GNUTLS  +USE_SASL  +USE_GSS
+HAVE_GETADDRINFO  
+HAVE_REGCOMP  -USE_GNU_REGEX  
+HAVE_COLOR  +HAVE_START_COLOR  +HAVE_TYPEAHEAD  +HAVE_BKGDSET  
+HAVE_CURS_SET  +HAVE_META  +HAVE_RESIZETERM  
+CRYPT_BACKEND_CLASSIC_PGP  +CRYPT_BACKEND_CLASSIC_SMIME
+CRYPT_BACKEND_GPGME  
-EXACT_ADDRESS  -SUN_ATTACHMENT  
+ENABLE_NLS  -LOCALES_HACK  +COMPRESSED  +HAVE_WC_FUNCS
+HAVE_LANGINFO_CODESET  +HAVE_LANGINFO_YESEXPR  
+HAVE_ICONV  -ICONV_NONTRANS  +HAVE_LIBIDN  +HAVE_GETSID  +USE_HCACHE  
-ISPELL
SENDMAIL=/usr/sbin/sendmail
MAILPATH=/var/mail
PKGDATADIR=/usr/share/mutt
SYSCONFDIR=/etc
EXECSHELL=/bin/sh
MIXMASTER=mixmaster
To contact the developers, please mail to mutt-...@mutt.org.
To report a bug, please visit http://bugs.mutt.org/.

misc/am-maintainer-mode
features/ifdef
features/xtitles

Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-22 Thread Chris Green
On Thu, Mar 21, 2013 at 05:06:17PM -0500, Derek Martin wrote:
 On Wed, Mar 20, 2013 at 06:08:49PM -0500, David Champion wrote:
  There absolutely should be a blank line.  I think though that the order
  is wrong: mutt expects that a message (i.e. a From_ line) appears at
  the old EOF marker, and that the EOF marker is on/after a blank line.
  I think that if you adjust your filter to write the message and then a
  blank line instead of a blank line and then a message, the error will go
  away.
 
 This matches my expectation also.
 
That's all very well but it does have some issues:-

What should an MTA do if there *isn't* a blank line at the end of
the current mbox where it is going to append a new message?  It
seems to me that what the Python libraries do guarantees that there
will always be a blank line before the 'From ' line, if there's one
already then it doesn't matter too much.

Mutt itself *doesn't* put a blank line there, if you S[ave] or
C[opy] messages to a new mbox the messages have no blank lines
before the 'From '.


-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-22 Thread James Griffin
[- Fri 22.Mar'13 at  9:04:21 +  Chris Green :-]

 On Thu, Mar 21, 2013 at 05:06:17PM -0500, Derek Martin wrote:
  On Wed, Mar 20, 2013 at 06:08:49PM -0500, David Champion wrote:
   There absolutely should be a blank line.  I think though that the order
   is wrong: mutt expects that a message (i.e. a From_ line) appears at
   the old EOF marker, and that the EOF marker is on/after a blank line.
   I think that if you adjust your filter to write the message and then a
   blank line instead of a blank line and then a message, the error will go
   away.
  
  This matches my expectation also.
  
 That's all very well but it does have some issues:-
 
 What should an MTA do if there *isn't* a blank line at the end of
 the current mbox where it is going to append a new message?  It
 seems to me that what the Python libraries do guarantees that there
 will always be a blank line before the 'From ' line, if there's one
 already then it doesn't matter too much.
 
 Mutt itself *doesn't* put a blank line there, if you S[ave] or
 C[opy] messages to a new mbox the messages have no blank lines
 before the 'From '.

Sorry Chris, I believe you're confusing MTA with MDA/LDA, although
that's not really relevent. Could you perhaps alter your python script
as David suggested but start over with new mboxes; i.e. move the current
mboxes into an archive folder inside the ~/Mail directory so as to avoid
the confusion with how your messages are currently laid-out in the mbox
file. Do you think that might help?


-- 
James Griffin:  jmz at kontrol.kode5.net 
jmzgriffin at gmail.com

A4B9 E875 A18C 6E11 F46D  B788 BEE6 1251 1D31 DC38


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-22 Thread James Griffin
[- Fri 22.Mar'13 at  7:38:54 -0400  Patrick Shanahan :-]

 * James Griffin j...@kontrol.kode5.net [03-22-13 05:24]:
  [...]
  Sorry Chris, I believe you're confusing MTA with MDA/LDA, although
  that's not really relevent. Could you perhaps alter your python script
  as David suggested but start over with new mboxes; i.e. move the current
  mboxes into an archive folder inside the ~/Mail directory so as to avoid
  the confusion with how your messages are currently laid-out in the mbox
  file. Do you think that might help?
 
 As mbox is actually *one* contiguous file, only the last appended msg will
 be relevant and it's trailing or lack of a trailing blank line.  Archiving
 would provide a definitive separation but is really not necessary.

Ah, well that's ok. I don't know a lot about mbox - although I do use
them - I just thought the files may have been left in a state that could
still be showing the issues described. Having a clean file with the
adjustments to the python script -- starting afresh so to speak - might
have helped.

If that's not the case then I'm glad you pointed that out.

-- 
James Griffin:  jmz at kontrol.kode5.net 
jmzgriffin at gmail.com

A4B9 E875 A18C 6E11 F46D  B788 BEE6 1251 1D31 DC38


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-22 Thread Chris Green
On Fri, Mar 22, 2013 at 09:22:34AM +, James Griffin wrote:
 [- Fri 22.Mar'13 at  9:04:21 +  Chris Green :-]
 
  On Thu, Mar 21, 2013 at 05:06:17PM -0500, Derek Martin wrote:
   On Wed, Mar 20, 2013 at 06:08:49PM -0500, David Champion wrote:
There absolutely should be a blank line.  I think though that the order
is wrong: mutt expects that a message (i.e. a From_ line) appears at
the old EOF marker, and that the EOF marker is on/after a blank line.
I think that if you adjust your filter to write the message and then a
blank line instead of a blank line and then a message, the error will go
away.
   
   This matches my expectation also.
   
  That's all very well but it does have some issues:-
  
  What should an MTA do if there *isn't* a blank line at the end of
  the current mbox where it is going to append a new message?  It
  seems to me that what the Python libraries do guarantees that there
  will always be a blank line before the 'From ' line, if there's one
  already then it doesn't matter too much.
  
  Mutt itself *doesn't* put a blank line there, if you S[ave] or
  C[opy] messages to a new mbox the messages have no blank lines
  before the 'From '.
 
 Sorry Chris, I believe you're confusing MTA with MDA/LDA, although

Yes, I am, though by default both are postfix on my system.


 that's not really relevent. Could you perhaps alter your python script
 as David suggested but start over with new mboxes; i.e. move the current
 mboxes into an archive folder inside the ~/Mail directory so as to avoid
 the confusion with how your messages are currently laid-out in the mbox
 file. Do you think that might help?
 
I've already fixed the problem for myself, as follows:-

Mutt accepts an mbox with no blank lines (not surprising as it's
what it creates itself).

I have created a derived class of my own from the Python library
class mailbox.mbox and I have overriden the _pre_message_hook(self,
f) method so that it doesn't append a blank line before appending a
new message to an mbox.  I use this derived class in my filter
program which delivers mail to my incoming mbox files.

So I have fixed my particular problem.

It does seem to me though that mutt doesn't necessarily do the right
thing to detect whether an mbox has been 'modified' or just appended to.


-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-22 Thread David Champion
* On 22 Mar 2013, Chris Green wrote: 
 What should an MTA do if there *isn't* a blank line at the end of
 the current mbox where it is going to append a new message?  It
 seems to me that what the Python libraries do guarantees that there
 will always be a blank line before the 'From ' line, if there's one
 already then it doesn't matter too much.

I don't want to butt heads with Python library people, but I would argue
that the Python library's approach is backwards.  This has historically
been mildly problematic with Mailman archives, for example, which
begin with a blank line instead of with 'From ', and are unreadable
without modification using some mail applications.  (Mutt was patched to
accomodate this quirk only relatively recently.)


 Mutt itself *doesn't* put a blank line there, if you S[ave] or
 C[opy] messages to a new mbox the messages have no blank lines
 before the 'From '.

Mutt instead writes Lines: and Content-Length: headers, which are a good
alternative to using '\n\nFrom ' as a message delimiter.  If you have
Lines: and/or Content-Length: in your delivery message, your LDA/filter
does not need to place a blank line at all.  But unless you're adding
those yourself, you can't depend on having them.  It is therefore best
to append a blank line to messages during local delivery.

-- 
David Champion • d...@bikeshed.us


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-22 Thread Chris Green
On Fri, Mar 22, 2013 at 11:48:46AM -0500, David Champion wrote:
 * On 22 Mar 2013, Chris Green wrote: 
  What should an MTA do if there *isn't* a blank line at the end of
  the current mbox where it is going to append a new message?  It
  seems to me that what the Python libraries do guarantees that there
  will always be a blank line before the 'From ' line, if there's one
  already then it doesn't matter too much.
 
 I don't want to butt heads with Python library people, but I would argue
 that the Python library's approach is backwards.  This has historically
 been mildly problematic with Mailman archives, for example, which
 begin with a blank line instead of with 'From ', and are unreadable
 without modification using some mail applications.  (Mutt was patched to
 accomodate this quirk only relatively recently.)
 
When I looked through a number of my mbox files it seemed to me that
there must be quite a few 'deliverers' out there appending or prepending
blank lines as there just about always seem to be several between each
message.


 
  Mutt itself *doesn't* put a blank line there, if you S[ave] or
  C[opy] messages to a new mbox the messages have no blank lines
  before the 'From '.
 
 Mutt instead writes Lines: and Content-Length: headers, which are a good
 alternative to using '\n\nFrom ' as a message delimiter.  If you have
 Lines: and/or Content-Length: in your delivery message, your LDA/filter
 does not need to place a blank line at all.  But unless you're adding
 those yourself, you can't depend on having them.  It is therefore best
 to append a blank line to messages during local delivery.
 
However not all software will *use* the Lines: and Content-Length:
headers so the '\n\nFrom ' should be there as well surely.  Apart from
anything else someone/something may have modified the message on the way
who/which doesn't know about Lines: and Content-Length: so they could
well be wrong.

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-22 Thread Derek Martin
On Fri, Mar 22, 2013 at 09:04:21AM +, Chris Green wrote:
 On Thu, Mar 21, 2013 at 05:06:17PM -0500, Derek Martin wrote:
  On Wed, Mar 20, 2013 at 06:08:49PM -0500, David Champion wrote:
   There absolutely should be a blank line.  I think though that the order
   is wrong: mutt expects that a message (i.e. a From_ line) appears at
   the old EOF marker, and that the EOF marker is on/after a blank line.
   I think that if you adjust your filter to write the message and then a
   blank line instead of a blank line and then a message, the error will go
   away.
  
  This matches my expectation also.
  
 That's all very well but it does have some issues:-
 
 What should an MTA do if there *isn't* a blank line at the end of
 the current mbox where it is going to append a new message?  

That's a reasonable question (technically MTA does not come into play
here, this is an MDA or MUA behavior).  To be 100% correct, most
likely the mailer should CHECK what the state of the mailbox is, add a
blank line if required, and NOT add a blank line if one is there.
That's more work than making assumptions, but it's not so horrible.
But it does seem that M*A software in general expects the trailing
blank line, so it should always ensure it's there on messages it
writes itself.  This makes sense: the blank line is often called the
message separator, but what it really is is the end of message
indicator.  Thinking about it as a separator leads to thinking that
it's optional if there's not a subsequent message, which leads to the
problem you're having.

 It seems to me that what the Python libraries do guarantees that
 there will always be a blank line before the 'From ' line, if
 there's one already then it doesn't matter too much.

It does change the content of the last message though, which is
incorrect behavior.  You say it doesn't matter much, but it might, if
you have some automated process that expects a particular mailbox to
be in a particular state.  What does it do if you subsequently delete
the last message, and then add another one?  My suspicion is that it
might add yet another blank line, which clearly is wrong.  Though in
any case, adding even a single extra blank line is wrong.

 Mutt itself *doesn't* put a blank line there, if you S[ave] or
 C[opy] messages to a new mbox the messages have no blank lines
 before the 'From '.

Yet it works with every MDA I've ever used... which seems to suggest
that this is the expected behavior, and that the Python library got it
wrong.  And to be honest it's not really *that* surprising... I've
seen other cases where Python libraries were doing it wrong (whatever
it might be).

-- 
Derek D. Martinhttp://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.  Replying to it will result in
undeliverable mail due to spam prevention.  Sorry for the inconvenience.



pgp__ftuSj8U5.pgp
Description: PGP signature


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-21 Thread James Griffin
[- Wed 20.Mar'13 at 18:08:49 -0500  David Champion :-]

 * On 20 Mar 2013, Chris Green wrote: 
  
  Has the mutt handling of this changed in the last few versions?
 
 Not that I know of, and I doubt it.
 
  The python way of doing this is correct according to the RFC as far as I
  can tell, there *should* be a blank line between the end of a message
  and the 'From ' starting the next message.  Thus the python library I'm
  using does add this blank line.
  
  However mutt detects this as an error and outputs the messsage about the
  mailbox being modified.  
 
 There absolutely should be a blank line.  I think though that the order
 is wrong: mutt expects that a message (i.e. a From_ line) appears at
 the old EOF marker, and that the EOF marker is on/after a blank line.
 I think that if you adjust your filter to write the message and then a
 blank line instead of a blank line and then a message, the error will go
 away.
 
 -- 
 David Champion • d...@bikeshed.us

Chris, do you have a site where this python script is viewable? just so
we can see how it works. It might help clarify the issue and of course
it's interesting to see how people have come with ideas and used
languages/scripts to handle mail.

Jamie


-- 
James Griffin:  jmz at kontrol.kode5.net 
jmzgriffin at gmail.com

A4B9 E875 A18C 6E11 F46D  B788 BEE6 1251 1D31 DC38


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-21 Thread Chris Green
On Thu, Mar 21, 2013 at 09:25:49AM +, James Griffin wrote:
 [- Wed 20.Mar'13 at 18:08:49 -0500  David Champion :-]
 
  * On 20 Mar 2013, Chris Green wrote: 
   
   Has the mutt handling of this changed in the last few versions?
  
  Not that I know of, and I doubt it.
  
   The python way of doing this is correct according to the RFC as far as I
   can tell, there *should* be a blank line between the end of a message
   and the 'From ' starting the next message.  Thus the python library I'm
   using does add this blank line.
   
   However mutt detects this as an error and outputs the messsage about the
   mailbox being modified.  
  
  There absolutely should be a blank line.  I think though that the order
  is wrong: mutt expects that a message (i.e. a From_ line) appears at
  the old EOF marker, and that the EOF marker is on/after a blank line.
  I think that if you adjust your filter to write the message and then a
  blank line instead of a blank line and then a message, the error will go
  away.
  
  -- 
  David Champion • d...@bikeshed.us
 
 Chris, do you have a site where this python script is viewable? just so
 we can see how it works. It might help clarify the issue and of course
 it's interesting to see how people have come with ideas and used
 languages/scripts to handle mail.
 
It uses the standard Python libraries, I have attached the code (there's
nothing sensitive in there).

The relevant bit that appends the message to the mbox is:-


#
#
# Deliver a message to a local mbox
#
def deliverMboxMsg(dest, msg, log):
#
#
# Create the destination mbox instance
#
mbx = mailbox.mbox(dest, factory=None)

log.info(From:  + msg.get(From, unknown))
log.info(Destination is:  + dest)
#
#
# Lock the mbox while we append to it
#

for tries in xrange(3):
try:
mbx.lock()
#
#
# Append the incoming message to the appropriate mbox
#
mbx.add(msg)
#
#
# now set the modified time later than the access time (which is 
'now') so
# that mutt will see new mail in the mbox.
#
os.utime(dest, ((time.time()), (time.time() + 5)))
mbx.unlock()
break

except mailbox.ExternalClashError:
log.info(Destination locked, try  + str(tries))
time.sleep(1)

else: # get here if we ran out of tries
log.warn(Failed to lock destination after 3 attempts, giving up)



As you can see I'm using the 'off the shelf' standard Python library
class mailbox.mbox to deliver the message.  If this is doing it wrong
then I'm very surprised.

-- 
Chris Green
#!/usr/bin/python
#
#
# Mail filtering script, new version October 2010
#
import email
import mailbox
import os
import sys
import time
import mailLib
#
#
# Redirect any exceptions to a file
#
sys.stderr = open(/home/chris/tmp/mail.err, 'a')
#
#
# Some constants (i.e. configuration)
#
home = /home/chris
logfile = home + /tmp/mail.log
filtfile = home + /.mutt/filter
mldir = home + /Mail/
indir = mldir + In/
judir = mldir + Ju/
#
#
# Set to log to mail.log in ~/tmp with name 'filter' and the envelope/from
#
log = mailLib.initLog(filter)
#
#
# Initialise destination mailbox name to empty
#
dest = 
#
#
# Read the message from standard input and make a message object from it
#
# msg = email.message_from_string(sys.stdin.read())
msg = mailbox.mboxMessage(sys.stdin.read())
#
#
# Extract the To:, Cc: and Subject: headers and the envelope/from
#
msgcc = msg.get(Cc, unknown).lower()
msgto = msg.get(To, unknown).lower()
msgsb = msg.get(Subject, unknown)
msgfm = msg.get(From, unknown).lower()
#
#
# See if it's in our filter file
#
f = open(filtfile, 'r')
for ln in f:# for each line in filter
if ln[0] == '#':# ignore comments
continue
#
#
# split the line into fields
#
fld = ln.split()
nm = fld[0] # name/alias
#
#
# deal with possible flags after the destination directory
#
if (':' in fld[1]):
xx = fld[1].split(':')
dd = xx[0] + /
fg = xx[1]
else:
dd = fld[1] + /   # destination directory
fg = 
tocc = fld[2].lower()   # list address
#
#
# see if the filter To/CC column matches the message To: or Cc:
# for DBA lists (flag d) we also look in the From: address
#
if (tocc in msgcc or tocc in msgto or ((fg == d) and (tocc in msgfm))):
if (fg == d):
msg.__setitem__(Cc, msgfm)# so mutt will recognise it as a list
dest = mldir + dd + nm
#
#
# Strip out list name from subject if it's there
#
if len(fld)  3:
# msgText = msgText.replace('[' + fld[3] + ']', '', 1)
msg.replace_header(Subject, msgsb.replace('[' + fld[3] + ']', ''))
#
#
# Handle the 

Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-21 Thread Chris Green
On Thu, Mar 21, 2013 at 11:45:22AM +1100, raf wrote:

 the python code needs to be changed to write the From  header, then
the
 message, then the blank line.

Then it appears to be a bug in the Python mailbox library code.  I have
attached it here.  If you look through you will see, in the mbox class,
the following:-

def _pre_message_hook(self, f):
Called before writing each message to file f.
if f.tell() != 0:
f.write(os.linesep)

I.e. it writes a line separator to the mbox file *before* appending a
new message.  I suppose it could be said that this guarantees the mbox
is correctly formatted even if the *previous* writer to the mbox hasn't
put a blank line there.

By the way I tested mutt itself and it *doesn't* put a blank line
between messages, the 'From x' line is immediately after the last
line of text of the previous message.



I have fixed the problem for myself by creating a class derived from
mailbox.mbox and overriding _pre_message_hook().

-- 
Chris Green


mailbox.py.gz
Description: Binary data


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-21 Thread Derek Martin
On Wed, Mar 20, 2013 at 06:08:49PM -0500, David Champion wrote:
 There absolutely should be a blank line.  I think though that the order
 is wrong: mutt expects that a message (i.e. a From_ line) appears at
 the old EOF marker, and that the EOF marker is on/after a blank line.
 I think that if you adjust your filter to write the message and then a
 blank line instead of a blank line and then a message, the error will go
 away.

This matches my expectation also.

-- 
Derek D. Martinhttp://www.pizzashack.org/   GPG Key ID: 0xDFBEAD02
-=-=-=-=-
This message is posted from an invalid address.  Replying to it will result in
undeliverable mail due to spam prevention.  Sorry for the inconvenience.



pgplHKgc9tfnv.pgp
Description: PGP signature


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread Erik Christiansen
On 20.03.13 13:14, Chris Green wrote:
 What is supposed to happen in the following scenario:-
 
 I'm viewing my incoming mail (inbox), looking at the index view in mutt.
 
 Some new mail arrives, delivered by procmail/python script/whatever
 to the inbox.
 
 Is there supposed to be some mechanism whereby mutt recognises that all
 that has happened is that new mail has been appended to the mbox?

The manual has a brief description, in section: 10. New Mail Detection
(If mutt is well set up, that's on F1, but I figure you know that.)

 At present I *always* get the Mailbox was externally modified.  Flags
 may be wrong. message.

That happens here, if I have two mutts open concurrently. My guess is
that your python isn't satisfying mutt's access/modification time
requirements.

Erik

-- 
Mollison's Bureaucracy Hypothesis:
   If an idea can survive a bureaucratic review and be implemented
   it wasn't worth doing.



Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread Chris Green
On Thu, Mar 21, 2013 at 01:05:27AM +1100, Erik Christiansen wrote:
 On 20.03.13 13:14, Chris Green wrote:
  What is supposed to happen in the following scenario:-
  
  I'm viewing my incoming mail (inbox), looking at the index view in mutt.
  
  Some new mail arrives, delivered by procmail/python script/whatever
  to the inbox.
  
  Is there supposed to be some mechanism whereby mutt recognises that all
  that has happened is that new mail has been appended to the mbox?
 
 The manual has a brief description, in section: 10. New Mail Detection
 (If mutt is well set up, that's on F1, but I figure you know that.)
 
Yes, but it doesn't tell me what provokes the Mailbox was externally
modified message.


  At present I *always* get the Mailbox was externally modified.  Flags
  may be wrong. message.
 
 That happens here, if I have two mutts open concurrently. My guess is
 that your python isn't satisfying mutt's access/modification time
 requirements.
 
It's satisfying them in that mutt recognises new mail arriving in all my
(separate) mailboxes, that's working perfectly.  It's just that if I
happen to be *reading* the mbox where the new mail arrives I get the
error message.

I'm sure this didn't used to happen and I've not changed the delivery
mechanism to my knowledge.

What I want to know is:-

Is it possible for a message to be delivered into an mbox that mutt
is looking at without provoking the Mailbox was externally
modified message?

If the above is possible then what does the delivering MTA have to
do so that the delivered message is 'N' and all others are left
unchanged?

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread Chris Green
 
 What I want to know is:-
 
 Is it possible for a message to be delivered into an mbox that mutt
 is looking at without provoking the Mailbox was externally
 modified message?
 
 If the above is possible then what does the delivering MTA have to
 do so that the delivered message is 'N' and all others are left
 unchanged?
 
I just took a look at the code (desperate measures!) and it gives me the
answer.

If the mbox in question has a message *appended* to it then mutt regards
this as OK and will not output the error.  It detects an append by
looking to see if the 'message separator' before the appended message is
exactly where the end-of-file was previously.

I suspect my MTA doesn't agree exactly with mutt about where the
'message separator' is.

I need to play some more!  :-)

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread Erik Christiansen
On 20.03.13 14:11, Chris Green wrote:
 What I want to know is:-
 
 Is it possible for a message to be delivered into an mbox that mutt
 is looking at without provoking the Mailbox was externally
 modified message?

Yes. Here the message is New mail in this mailbox.
(Confirmed by sending myself an email while in !)

 If the above is possible then what does the delivering MTA have to
 do so that the delivered message is 'N' and all others are left
 unchanged?

I'd grab the source. The message is output by curs_main.c, on detecting
(check == M_REOPENED). There are assignments to check in mx.c and
curs_main.c, and mbox_check_mailbox() in mbox.c tells (in comments) that
our guess at the meaning of M_REOPENED is correct. It's 2 a.m. here,
so I'll leave it to you to figure out how to make the python behave so
that it detects M_NEW_MAIL instead. (Is it respecting the file locking
in mx_lock_file(0 in mx.c?) I see that function does a sleep(). I should
do that too.

Erik

-- 
manual, n.:   
A unit of documentation. There are always three or more on a given item.   
One is on the shelf; someone has the others.  
The information you need is in the others.   - Ray Simard



Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread Chris Green
On Wed, Mar 20, 2013 at 02:58:58PM +, Chris Green wrote:
  
  What I want to know is:-
  
  Is it possible for a message to be delivered into an mbox that mutt
  is looking at without provoking the Mailbox was externally
  modified message?
  
  If the above is possible then what does the delivering MTA have to
  do so that the delivered message is 'N' and all others are left
  unchanged?
  
 I just took a look at the code (desperate measures!) and it gives me the
 answer.
 
 If the mbox in question has a message *appended* to it then mutt regards
 this as OK and will not output the error.  It detects an append by
 looking to see if the 'message separator' before the appended message is
 exactly where the end-of-file was previously.
 
 I suspect my MTA doesn't agree exactly with mutt about where the
 'message separator' is.
 
 I need to play some more!  :-)
 
I have found my problem!

My (python) MTA puts a blank line at the start of each message before
the From MAILER-DAEMON Wed Mar 20 14:27:05 2013 line which is the mbox
start line.  As described above mutt sees the start as being at the
From  so it thinks the mbox has been modified otherwise than by a
simple append.

I guess something must have changed in the python libraries (or maybe in
mutt) to have changed this logic because, as I said, it did used to work
OK.

I'll just have to fiddle something somewhere.  I can probably remove the
empty line in my python code.

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread David Champion
* On 20 Mar 2013, Chris Green wrote: 
 
 I suspect my MTA doesn't agree exactly with mutt about where the
 'message separator' is.

When your script delivers a message, does it append a message and then
a blank line, or does it append a blank line and then a message?

-- 
David Champion • d...@bikeshed.us


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread Chris Green
On Wed, Mar 20, 2013 at 10:11:06AM -0500, David Champion wrote:
 * On 20 Mar 2013, Chris Green wrote: 
  
  I suspect my MTA doesn't agree exactly with mutt about where the
  'message separator' is.
 
 When your script delivers a message, does it append a message and then
 a blank line, or does it append a blank line and then a message?
 
It appears to add a blank line and then the message.

Has the mutt handling of this changed in the last few versions?

The python way of doing this is correct according to the RFC as far as I
can tell, there *should* be a blank line between the end of a message
and the 'From ' starting the next message.  Thus the python library I'm
using does add this blank line.

However mutt detects this as an error and outputs the messsage about the
mailbox being modified.  

If, on the other hand, I append a message to my mbox file by hand and I
*don't* put a blank line in then mutt is quite happy. Thus mutt is quite
happy with the following in an mbox:-

...
...
User-Agent: Mutt/1.5.21 (2010-09-15)
Status: O
Content-Length: 21
Lines: 2

Hello
--
Chris Green
From ch...@zbmc.eu  Wed Mar 20 15:16:33 2013
Return-Path: ch...@zbmc.eu
X-Original-To: ch...@chris.zbmc.eu
Delivered-To: ch...@chris.zbmc.eu
Received: by chris.zbmc.eu (Postfix, from userid 1000)
id 88E8C381B20; Wed, 20 Mar 2013 15:16:33 + (GMT)
Date: Wed, 20 Mar 2013 15:16:33 +
From: Chris Green ch...@isbd.co.uk
...
...

but strictly this is wrong as there should be an empty line there.

-- 
Chris Green


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread David Champion
* On 20 Mar 2013, Chris Green wrote: 
 
 Has the mutt handling of this changed in the last few versions?

Not that I know of, and I doubt it.

 The python way of doing this is correct according to the RFC as far as I
 can tell, there *should* be a blank line between the end of a message
 and the 'From ' starting the next message.  Thus the python library I'm
 using does add this blank line.
 
 However mutt detects this as an error and outputs the messsage about the
 mailbox being modified.  

There absolutely should be a blank line.  I think though that the order
is wrong: mutt expects that a message (i.e. a From_ line) appears at
the old EOF marker, and that the EOF marker is on/after a blank line.
I think that if you adjust your filter to write the message and then a
blank line instead of a blank line and then a message, the error will go
away.

-- 
David Champion • d...@bikeshed.us


Re: Is it supposed to be possible to deliver mail while mutt has the mbox open?

2013-03-20 Thread raf
Chris Green wrote:

 On Wed, Mar 20, 2013 at 10:11:06AM -0500, David Champion wrote:
  * On 20 Mar 2013, Chris Green wrote: 
   
   I suspect my MTA doesn't agree exactly with mutt about where the
   'message separator' is.
  
  When your script delivers a message, does it append a message and then
  a blank line, or does it append a blank line and then a message?
  
 It appears to add a blank line and then the message.
 
 Has the mutt handling of this changed in the last few versions?
 
 The python way of doing this is correct according to the RFC as far as I
 can tell, there *should* be a blank line between the end of a message
 and the 'From ' starting the next message.  Thus the python library I'm
 using does add this blank line.

it is not correct. the blank line should be at the *end* of each message,
not at the *start*. think about it. does an mbox file start with a blank line?
no. it ends with a blank line.

 However mutt detects this as an error and outputs the messsage about the
 mailbox being modified.  

not an error, just a modification.

presumably because there is an extra blank line, mutt sensibly concludes
that the message that was previously the last message now has an extra
blank line in it (i.e. the line that used to be a message separator but
which is now a new last line of the message followed by the new message
separator).

 If, on the other hand, I append a message to my mbox file by hand and I
 *don't* put a blank line in then mutt is quite happy. Thus mutt is quite
 happy with the following in an mbox:-

because that is correct.

the python code needs to be changed to write the From  header, then the
message, then the blank line.

cheers,
raf