Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-07 Thread Paul Vixie
> Date: Thu, 6 May 2010 19:59:04 -0700 (PDT) > From: Mark Crispin > ... > I go to the trouble to teach you how things actually work, and you respond > with a typical nihilistic Gen-X retort. > > The righteous thing is to follow the specifications; and if you think that > the specifications are in

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-07 Thread Paul Vixie
> Date: Thu, 6 May 2010 19:17:19 -0700 (PDT) > From: Mark Crispin > ... > If your IMAP connection "stalls", there is no reason to believe that > disconnecting, creating a new connection, and retrying the operation will > in any way help the situation. > ... i was with you right up until that poin

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-07 Thread Oswald Buddenhagen
On Thu, May 06, 2010 at 05:37:52PM -0700, Mark Crispin wrote: > It is not exceptionally difficult to resume IP connectivity after > disconnect for a dynamic IP. It just requires an engineer with the wit to > recognize that it is desirable to be able to do that, and the imagination > to work out ho

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Brian Hayden
On May 6 2010, Mark Crispin wrote: I go to the trouble to teach you how things actually work, and you respond with a typical nihilistic Gen-X retort. Was that what you said, or was it not? It quite clearly was, so the retort was typically nothing (particularly indicative of a generation to wh

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Mark Crispin
On Thu, 6 May 2010, Brian Hayden wrote: Reliable does not mean "does not fail". Coincidentally, nobody said it did. Interesting! If that was not your meaning in claiming that "TCP is not reliable", then you don't know what you are talking about. TCP is most certainly reliable. This is anoth

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Brian Hayden
On May 6 2010, Mark Crispin wrote: Reliable does not mean "does not fail". Coincidentally, nobody said it did. Interesting! This is another fun historical dissertation, at whose core is: "change the RFCs, and until then maintain some righteous anger." What you think of as being "failure"

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Mark Crispin
On Thu, 6 May 2010, Brian Hayden wrote: TCP connections are more reliable than UDP; that does not mean they are "reliable", full stop. You are using the wrong definition of reliable. Reliable does not mean "does not fail". UDP has no provision for reliability, ordering or data integrity. TCP

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Mark Crispin
On Fri, 7 May 2010, Oswald Buddenhagen wrote: who cares what existed back then? we are talking about *today*. but you are still arguing as if the internet looked the same as 25 years ago. "Who cares about anything from the past? There is nothing to be learned from past experience, and nothing

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Brian Hayden
On May 6 2010, Mark Crispin wrote: On Fri, 7 May 2010, Oswald Buddenhagen wrote: loss of the state we are talking about here. it's all based on mark's postulation that a tcp connection is reliable. TCP connections are reliable. Run, don't walk, to your nearest technical bookstore and read ab

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Mark Crispin
On Fri, 7 May 2010, Oswald Buddenhagen wrote: loss of the state we are talking about here. it's all based on mark's postulation that a tcp connection is reliable. TCP connections are reliable. Run, don't walk, to your nearest technical bookstore and read about network layering. Crappy softwar

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Oswald Buddenhagen
On Thu, May 06, 2010 at 03:20:06PM -0700, Mark Crispin wrote: > On Thu, 6 May 2010, Oswald Buddenhagen wrote: > >ever heard about connection-tracking packet filters and routers? > >ill-tempered transparent proxies? > > Tell us all about the connection-tracking packet filters and routers > that exi

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Mark Crispin
On Thu, 6 May 2010, Dan White wrote: loss of tcp connection != loss of state. Correct. There's a further equation: loss of network connectivity != loss of TCP session != loss of state There is no reason to lose a TCP session because of a short-term loss of network connectivity. Although it

[Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Oswald Buddenhagen
On Thu, May 06, 2010 at 05:34:35PM -0500, Dan White wrote: > loss of tcp connection != loss of state. > loss of the state we are talking about here. it's all based on mark's postulation that a tcp connection is reliable. > In my own review of several IMAP RFCs, it's clear that connection problems

[Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Dan White
On 06/05/10 23:46 +0200, Oswald Buddenhagen wrote: funny, how establishing some reasonable common guidelines for handling loss of state in the standard could alleviate these problems to a significant degree. unfortunately, the creator doesn't even acknowledge the problem. oh, well. tough luck, i

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Oswald Buddenhagen
On Thu, May 06, 2010 at 01:24:00PM -0700, Mark Crispin wrote: > Crapware assumes that it is a network issue that somehow is utterly > irrecoverable in TCP, yet magically goes away if you tear down the TCP > connection and establish a new one. > hmmm ... why might they do such an obviously nonsensi

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Mark Crispin
On Thu, 6 May 2010, Dan White wrote: And what do you do if the server takes longer that you think it should to respond to a query? Do you assume that it's a networking issue or a slow server? Crapware assumes that it is a network issue that somehow is utterly irrecoverable in TCP, yet magically

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Oswald Buddenhagen
On Thu, May 06, 2010 at 11:46:20AM -0700, Mark Crispin wrote: > On Thu, 6 May 2010, Timo Sirainen wrote: > >I think that's a different issue. They're unhappy when an idling device > >gets woken up (constantly). The "Hang in there" messages are sent only > >when client has requested some command tha

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Mark Crispin
On Thu, 6 May 2010, Timo Sirainen wrote: I think that's a different issue. They're unhappy when an idling device gets woken up (constantly). The "Hang in there" messages are sent only when client has requested some command that takes >15 seconds. Most of the users/clients never see those messages

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Timo Sirainen
On Thu, 2010-05-06 at 11:19 -0700, Mark Crispin wrote: > On Thu, 6 May 2010, Timo Sirainen wrote: > > Instead of fighting clients with this, I solved it by having server send > > * OK Hang in there.. > > about every 15 seconds during long running commands. Clients seem to be > > happy with it and n

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Mark Crispin
On Thu, 6 May 2010, Timo Sirainen wrote: Instead of fighting clients with this, I solved it by having server send * OK Hang in there.. about every 15 seconds during long running commands. Clients seem to be happy with it and not disconnect. Unfortunately, the mobile device guys and gals then ge

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Timo Sirainen
On Thu, 2010-05-06 at 03:19 -0500, Dan White wrote: > And what do you do if the server takes longer that you think it should to > respond to a query? Do you assume that it's a networking issue or a slow > server? Instead of fighting clients with this, I solved it by having server send * OK Hang

[Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-06 Thread Dan White
On 06/05/10 09:22 +0200, Oswald Buddenhagen wrote: On Wed, May 05, 2010 at 03:09:20PM -0700, Mark Crispin wrote: On Wed, 5 May 2010, UCTC Sysadmin wrote: >If they can understand the difference between TCP and UDP, they can >understand state. I wouldn't be so certain. If you look at a lot of ap

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Mark Crispin
On Wed, 5 May 2010, Paul Vixie wrote: i must have spoken improperly. if i say "scan" and learn thereby about messages 1,3,5,20 and then one month later after every computer has been power cycled four times i say "show 5" i want the same message. Oh. In that case, what you want are UIDs. ima

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Paul Vixie
> Date: Wed, 5 May 2010 14:49:56 -0700 (PDT) > From: Mark Crispin > > damnably, and as you say, there'd have to be a lot of state to preserve > > the insanity of "message numbers". > > IMAP is a stateful protocol. There is nothing insane about message > numbers in a stateful message access prot

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Paul Vixie
> Date: Thu, 6 May 2010 00:23:14 +0300 > From: Yiorgos Adamopoulos > > That is why people like you and me can donate to Mark (I did last week) > in order for him to continue working on stuff that makes our systems > tick. Enough people can form a "Panda-IMAP empire" :) it would take a lot of us

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Mark Crispin
On Thu, 6 May 2010, Yiorgos Adamopoulos wrote: That is why people like you and me can donate to Mark (I did last week) in order for him to continue working on stuff that makes our systems tick. Enough people can form a "Panda-IMAP empire" :) And thank you! The donations do help keep Panda IMAP

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Mark Crispin
On Wed, 5 May 2010, Paul Vixie wrote: damnably, and as you say, there'd have to be a lot of state to preserve the insanity of "message numbers". IMAP is a stateful protocol. There is nothing insane about message numbers in a stateful message access protocol; this is the entire mechanism upon w

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Yiorgos Adamopoulos
On Wed, May 5, 2010 at 11:50 PM, Paul Vixie wrote: > i regret that i am not part of an empire who can afford to hire you just to > work on open source software.  brian reid of DEC WRL deserves huge thanks > for hiring me and then letting me work on BIND after UCB abandoned it.  we > need more empi

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Paul Vixie
> Date: Wed, 5 May 2010 11:44:26 -0700 (PDT) > From: Mark Crispin > > Have you thought about making mh be an IMAP client? You might need to > have some sort of proxy daemon to keep state, since IIRC mh is actually a > set of programs invoked from the shell. I don't know if your other tools > us

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Andrew Laurence
On May 5, 2010, at 11:44 AM, Mark Crispin wrote: > On Wed, 5 May 2010, Paul Vixie wrote: >> of course. but my primary mail interface is emacs mh-e and i'm not going >> to abandon it, nor the many filters and cronjobs and tools i've based on MH, >> just to support my secondary need to open attach

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Mark Crispin
On Wed, 5 May 2010, Paul Vixie wrote: of course. but my primary mail interface is emacs mh-e and i'm not going to abandon it, nor the many filters and cronjobs and tools i've based on MH, just to support my secondary need to open attachment-containing messages in an IMAP client. i fully underst

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-05 Thread Paul Vixie
> Date: Tue, 4 May 2010 12:03:19 -0700 (PDT) > From: Mark Crispin > ... > A better implementation would use an index file that maps between a UID > and a device/inode number of the file. To open and synchronize, you > stat() all the files and then correlate that with the index to build a > map.

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-04 Thread Timo Sirainen
On Tue, 2010-05-04 at 12:03 -0700, Mark Crispin wrote: > A better implementation would use an index file that maps between a > UID > and a device/inode number of the file. To open and synchronize, you > stat() all the files and then correlate that with the index to build a > map. This would also

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-04 Thread Mark Crispin
On Tue, 4 May 2010, Paul Vixie wrote: nfs file attribute changes are always at least three seconds out of date as witnessed between a process running on a client and process running on the server and possibly much longer between two processes running on two clients, because of the way the caching

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-04 Thread Paul Vixie
> Date: Mon, 3 May 2010 10:17:06 -0700 (PDT) > From: Mark Crispin > > > 1) If mbox file's mtime and size match > > Perhaps that works better today than 20 years ago. Back then, you could > not trust mtime to reflect reality in any reasonable way particularly > when NFS was involved. The most c

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-03 Thread Linda Walsh
Mark Crispin wrote: > On Tue, 4 May 2010, Timo Sirainen wrote: >> So I'm kind of hoping people would stop using mbox. :) > > You and me both! ;) Grr you fraggle-robbin $...@!... My antique perl code's been doing fastidious locking since ...well a long time! plblblbl... I can't help I l

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-03 Thread Mark Crispin
On Tue, 4 May 2010, Timo Sirainen wrote: This mtime flushing actually works pretty easily in all modern OSes: just open and close the file and then stat/fstat. Yup. You just said it: "modern OSes"... I remember an OS which would not update mtime if ANY local agent had the file open. So, no m

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-03 Thread Timo Sirainen
On 3.5.2010, at 20.17, Mark Crispin wrote: > On Mon, 3 May 2010, Timo Sirainen wrote: >> 1) If mbox file's mtime and size match > > Perhaps that works better today than 20 years ago. Back then, you could > not trust mtime to reflect reality in any reasonable way particularly when > NFS was invol

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-03 Thread Mark Crispin
On Mon, 3 May 2010, Timo Sirainen wrote: 1) If mbox file's mtime and size match Perhaps that works better today than 20 years ago. Back then, you could not trust mtime to reflect reality in any reasonable way particularly when NFS was involved. The most common circumstance is that mtime simpl

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-03 Thread Timo Sirainen
On Mon, 2010-05-03 at 08:32 -0700, Mark Crispin wrote: > For what it's worth: Timo is the author of Dovecot. His comments about > its implementation should be considered authoritative. Mine are based > upon memory and second-hand/third-hand information. > > What is important - and what I will/do

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-03 Thread Mark Crispin
For what it's worth: Timo is the author of Dovecot. His comments about its implementation should be considered authoritative. Mine are based upon memory and second-hand/third-hand information. What is important - and what I will/do comment upon - is whether or not another server is compliant wi

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-03 Thread Timo Sirainen
A bit too much misinformation here so I'll have to reply :) On 3.5.2010, at 7.51, Mark Crispin wrote: > The main issue is if any other mail reading program is consuming the mbox. > If Dovecot is the only consumer you will be OK. But if you have other > consumers (including Pine, Alpine, elm, /us

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-02 Thread Mark Crispin
On Sun, 2 May 2010, Linda Walsh wrote: Seems to be fine with me using the 'mbox.lock' locking files to gain exclusive access. I believe was a compatibility setting somewhere. The .lock file is a delivery lock; to prevent more than one agent from writing (= appending) to the mbox at the sam

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-02 Thread Linda Walsh
Mark Crispin wrote: > UW (and Panda) try damn hard to be compatible with even the most ancient > stupid things that people do with mbox files; and take a considerable > performance hit for doing so. Probably from having read your rantings on the topic before, I'd not do any of those things...but

Re: [Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-02 Thread Mark Crispin
Dovecot is a good server. It is one of only two (the other being Panda IMAP) that fully passes IMAP compliance testing: http://imapwiki.org/ImapTest/ServerStatus [UW IMAP flunks two of the tests...it hasn't been updated in 2 years.] The main concern that I have with using Dovecot for tra

[Imap-uw] Re: moving mail between folders is intermittently failing

2010-05-02 Thread Linda Walsh
support wrote: > We have users whose folders are between 100 MB and 800 MB in size. > Most of those users are using Outlook but some are using Thunderbird. > --- Not to negate anything Mark has stated about large email folder. I generally keep my mboxes below 100MB except for archives, ho