Re: [Imap-uw] Unread flag always on after reindexing
fredag 13 mars 2009 04:18:58 skrev Mark Crispin: I am very confused reading this message. Why did you run mixrbld and mixdfix? These are very powerful tools that should only be run in specific circumstances. What were the exact text of any complaint messages which you received? Those message are very specific. Paraphrasing them in a problem report destroys their usefulness. There are several messages in mixrbld, mixdfix, and the mix driver in c-client; but none contain the string out of sequence. I have no idea what kmail and horde/IMP webmail do. Have you tried access with Alpine? If so, what behavior do you see? OK, Compiled and tried with alpine. The same thing happens: If you read a mail of the problematic ones, it is still marked as new when you leave apline and return again. Best Dag ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Unread flag always on after reindexing
On Fri, 13 Mar 2009, Dag Nygren wrote: Why did you run mixrbld and mixdfix? These are very powerful tools that should only be run in specific circumstances. It is too long ago I did this for me to remember the exact reason. I think it had something to do with kmail complaining about an attibute missing for certain mail when doing a search. The only reason to run mixrbld/mixdfix is if the mix driver reports errors and refuses to open the mailbox. You probably ran mixrbld/mixdfix unnecessarily, and that caused your problems. There aren't many imapd messages with the word attribute. If the error message was Bogus attribute list, that means that the IMAP client (kmail) sent faulty IMAP protocol. That would be a bug in kmail, and NOTHING wrong with imapd or the mix mailbox. [Or nothing wrong until the ill-advised usage of mixrbld/mixdfix.] However, since you paraphrased the message, this is only a guess. Please do not paraphrase messages. It wastes time. Sounded very much like a corruption in the mailbox to me, Especially since it happened only in my INBOX. I don't know how you came to that conclusion, but it is almost certainly incorrect. If the mailbox opens, it is not corrupt in any way that mixrbld/mixdfix can remedy. It's like getting heart surgery for a head cold; completely ineffective for the problem, and dangerous in other ways. I saw mixrbld as a means to fix this by rebuilding the mailbox. Ran mixrdfix after this as I saw nothing at all in the mailbox after the rebuild. Now the mails are there, but half of them always marked unread. And cannot be marked as read whatever I do. I don't know what you did with mixrbld/mixdfix, but neither tool does anything with the status file, which is where seen state is kept. I have no idea why seen state would behave that way. Just checked the source code and the actual message was from mixrbld and was the one on row 181, printf (Data file %s UID ran backwards. Isn't that out of sequence. The text out of sequence does not occur in Data file...ran backwards, and thus is a paraphrase. Please don't do that. If you got a Data file...ran backwards message, it is likely that some expunge operation did not complete properly. mixrbld will recover from this. I did check this up and found that you have indicated earlier that it is not a real problem. Also tried mixcvt to copy the mailbox as somewhere suggested, but even the result still had the same problem. mixcvt should have made a clean mailbox. I suggest that you try to find some expert who is local to you to look at it. There is nothing obvious that can be diagnosed from the other side of the world. It is obvious to me that there is more to this story than what you say, but unfortunately I do not have the time to investigate it. Unfortunately, I can't offer the same level of user assistance than I did when I was working at UW, and you seem to need much more assistance than I can offer. I'm sorry. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
[Imap-uw] IMAPD with pam_tally
Hi, I am using IMAPD from pine4.64 on Scientific Linux 5.0. The basic functionality (accessing mails) of IMAPD is working fine. I am using PAM to control access privileges of users. I have configured pam_tally to lock the user account after a fixed number of login failures. The strange behavior which I have observed is that IMAPD does not increment the login failure counter on wrong credentials even it resets the counter to 0. pam_tally is working fine for other services like ssh and login. Have any of you observed this problem earlier. Below is my pam_tally related entries in /etc/pam.d/imap file auth required pam_tally.so per_user deny=10 accountrequired pam_tally.so Please comment. regards Sharma -- Apologizing doesn't mean that you are wrong the other is right. It means that you value the relationship more than your ego...! | ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Unread flag always on after reindexing
fredag 13 mars 2009 10:00:18 skrev Mark Crispin: . The only reason to run mixrbld/mixdfix is if the mix driver reports errors and refuses to open the mailbox. You probably ran mixrbld/mixdfix unnecessarily, and that caused your problems. Ok. Didn't see any warnings on them creating problems. Thought it was more like fsck. Just rebuilding the metadata from the raw messages. However, since you paraphrased the message, this is only a guess. Please do not paraphrase messages. It wastes time. My memory isn't good enough to remember the exact wording of a message 3-4 months ago I don't know how you came to that conclusion, but it is almost certainly incorrect. Hmm... If the mailbox opens, it is not corrupt in any way that mixrbld/mixdfix can remedy. So that is a good enough test? Will remember that in the future. Of course the current problem indicates tha opposite as it very well opens, but still doesn't work as expected... I saw mixrbld as a means to fix this by rebuilding the mailbox. Ran mixrdfix after this as I saw nothing at all in the mailbox after the rebuild. Now the mails are there, but half of them always marked unread. And cannot be marked as read whatever I do. I don't know what you did with mixrbld/mixdfix, but neither tool does anything with the status file, which is where seen state is kept. Is ther any documentation on mix? Just checked the source code and the actual message was from mixrbld and was the one on row 181, printf (Data file %s UID ran backwards. Isn't that out of sequence. The text out of sequence does not occur in Data file...ran backwards, and thus is a paraphrase. Please don't do that. OK. But is does mean the same thing :-) If you got a Data file...ran backwards message, it is likely that some expunge operation did not complete properly. mixrbld will recover from this. Shouldn't a rerun of mixrbld be clean in that case? mixcvt should have made a clean mailbox. Didn't do that. I suggest that you try to find some expert who is local to you to look at it. There is nothing obvious that can be diagnosed from the other side of the world. It is obvious to me that there is more to this story than what you say, but unfortunately I do not have the time to investigate it. I do know my way around C-programming and how to read manuals. Just wanted to know if there is something simple I could do before digging into a protocol analysis. Any other mix analyzing/fixing tool I can use? Unfortunately, I can't offer the same level of user assistance than I did when I was working at UW, and you seem to need much more assistance than I can offer. I'm sorry. Understandable. Just wanted pointers in the right direction (and to mix docus) Best Dag ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Unread flag always on after reindexing
On Thursday 12 March 2009, Dag Nygren wrote: Hi After running mixrbld my mix-format inbox always shows about half of the box is unread, both in kmail and accessed from my horde/imp webmail. Trying to mark the mails as read doesn't do anything. Have also tried using mixdfix. Doesn't change anything though. Had complaints about mails being out of sequence from one of the tools, don't remember which. The mailbox has once been transformed from MH to mix format, but worked fine for months after that. Could this explain the numerous out of sequence messages from the rebuild process? Any hints on how to fix this. Now it is a bit hard to spot new mails... Answering myself here: Did the following: - Copied all files in my INBOX to a new folder recover - Test access showed the same problems - deleted .mixstatus - Access gave me an error on status not available - touch .mixstatus - Reopen mailbox - Mark all read - Now they stay marked In other words: The status file is messed up by my rebuild of the mailbox Comparing the erring .mixstatus with the newly created (by the Mark all read) shows that the last 8-digit hex code is the only difference (except from an occasional flag of course) Now my question is: What does the last number mean? Best Dag ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Best way to refresh the message numbers of a stale folder?
There is no refresh. The concept is meaningless in IMAP, particularly for number of messages and other mailbox state. Mailbox state is always pushed from the server to the client. You need to deal with the underlying cause. If the client state is stalled, then your code is in some way failing to update client state properly. Probably, either your implementation of IDLE is incorrect, or (as suggested in my previous message) you incorrectly expect state to cross from one session to the other without per-session notification. There is nothing that you can do to refresh client state. I understand your desire for a simple palliative. In this case the palliative is neither simple, nor effective, nor existent. I am not hiding some secret technique from you. You can continue searching for such, but will continue to be frustrated until you change course and go about it the right way. Last, but not least, IDLE is not push. In many servers IDLE causes worse battery consumption, and doesn't deliver instantaneous notification for all that (the delay can be up to 1 minute in UW and Panda). If your customers expect push, they will be very disappointed with a product that does IDLE. Apple does not do IDLE on iPhone and iPod Touch at all. RIM does IDLE between BIS and the IMAP server, but does real push (not IDLE or even IMAP at all) to the BlackBerry. Put another way, BIS runs interference and prevents the battery consumption problems caused by IDLE. I am sorry if this free advice is not to your liking. I tell people what they need to know to solve their problem, which is not necessarily they want to hear. On Fri, 13 Mar 2009, Shawn Walker wrote: Mark, c-client as a IMAP client now does support IDLE, multi-threaded (for IDLE and because the Windows application is a multi threaded application) and support asynchronous sessions to be able to handle IDLE. This was done so that the application can use c-client for the IMAP client communication. The code was modified to be able to handle that. I know you are not a fan of IDLE, but our customers wanted the push feature to get e-mails instantaneously instead of the client polling the server X minutes (or seconds if the client want to be aggressive about it). Everything is working great except for one small issue. The client has two IMAP sessions opened in two different threads (in a single process, multi-threaded). The reason is out of my control due to how the windows application was written (but that's another discussion for another day). From the process IDLE thread, the client got the untagged IMAP responses, the client end the IDLE with the DONE command and then the client sent a NOOP, but the message cache is still staled. I'm not here to debate what is wrong in your view with using IDLE, multi-threaded application, using more than one IMAP connections to the IMAP server. I just want a solution to how to get c-client to refresh it's message cache properly without having to disconnect and reconnect to the server. Regards, Shawn Mark Crispin wrote: On Fri, 13 Mar 2009, Shawn Walker wrote: The application has multiple threads with 2 connections to the IMAP server. One of them is for IDLE. This application does not use c-client to do IMAP client. c-client does not support client-end IDLE. Presumably, by thread, you mean threads in a process as opposed to message threads. UW imapd does not run multi-threaded; each IMAP session has its own process. Nor does the c-client library use threads. So, whatever is threading and using IDLE does not seem to have anything to do with c-client or imapd. When something happen on the IDLE thread, the server send a list of untagged IMAP commands to the client of what happened. The server sends untagged IMAP responses, not commands. The IDLE thread see that it need to update a folder, but the IDLE thread has two messages the UID of 100 and 101 (an example). But, UID 101 is doesn't exist anymore, but UID 102 is on the server. So, the IDLE thread request the message cache for UID 102 but c-client doesn't know about 102 in it's message cache due that it only know of UID 100 and 101 and return with a NIL. Hence, the message cache is stale. This makes no sense, so I have to guess what you are talking about. My guess is that client has two IMAP sessions open. One of those sessions did an IDLE command that notified the client of new messages. You expected that the other session would instantaneously know about those new messages, even though that session had not yet been notified. That is not the way IMAP works. Each IMAP session has its own independent state, and is notified of new messages independently. Why do you have two IMAP sessions open on the same mailbox? That, by itself, suggests that you are not using IMAP properly. No well-written application should need more than one IMAP session open to a mailbox at a time. The
Re: [Imap-uw] Unread flag always on after reindexing
On Fri, 13 Mar 2009, Dag Nygren wrote: In other words: The status file is messed up by my rebuild of the mailbox Comparing the erring .mixstatus with the newly created (by the Mark all read) shows that the last 8-digit hex code is the only difference (except from an occasional flag of course) I don't know why that would be, unless the old file had something like . Now my question is: What does the last number mean? Refer to imap-/docs/mixfmt.txt -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Best way to refresh the message numbers of a stale folder?
I never thought you were hiding anything about some secret code to refresh the message cache. Just asking if you have any suggestions to get the message cache updated. I never implied IDLE is push, why I used quotes around push. I understand the logic behind IDLE. What we are doing is giving the users the impression that the messages is being pushed but it's not. IDLE just notify the client something happened, go figure it out. I'll just figure a way to get the message cache update properly. Shawn Mark Crispin wrote: There is no refresh. The concept is meaningless in IMAP, particularly for number of messages and other mailbox state. Mailbox state is always pushed from the server to the client. You need to deal with the underlying cause. If the client state is stalled, then your code is in some way failing to update client state properly. Probably, either your implementation of IDLE is incorrect, or (as suggested in my previous message) you incorrectly expect state to cross from one session to the other without per-session notification. There is nothing that you can do to refresh client state. I understand your desire for a simple palliative. In this case the palliative is neither simple, nor effective, nor existent. I am not hiding some secret technique from you. You can continue searching for such, but will continue to be frustrated until you change course and go about it the right way. Last, but not least, IDLE is not push. In many servers IDLE causes worse battery consumption, and doesn't deliver instantaneous notification for all that (the delay can be up to 1 minute in UW and Panda). If your customers expect push, they will be very disappointed with a product that does IDLE. Apple does not do IDLE on iPhone and iPod Touch at all. RIM does IDLE between BIS and the IMAP server, but does real push (not IDLE or even IMAP at all) to the BlackBerry. Put another way, BIS runs interference and prevents the battery consumption problems caused by IDLE. I am sorry if this free advice is not to your liking. I tell people what they need to know to solve their problem, which is not necessarily they want to hear. On Fri, 13 Mar 2009, Shawn Walker wrote: Mark, c-client as a IMAP client now does support IDLE, multi-threaded (for IDLE and because the Windows application is a multi threaded application) and support asynchronous sessions to be able to handle IDLE. This was done so that the application can use c-client for the IMAP client communication. The code was modified to be able to handle that. I know you are not a fan of IDLE, but our customers wanted the push feature to get e-mails instantaneously instead of the client polling the server X minutes (or seconds if the client want to be aggressive about it). Everything is working great except for one small issue. The client has two IMAP sessions opened in two different threads (in a single process, multi-threaded). The reason is out of my control due to how the windows application was written (but that's another discussion for another day). From the process IDLE thread, the client got the untagged IMAP responses, the client end the IDLE with the DONE command and then the client sent a NOOP, but the message cache is still staled. I'm not here to debate what is wrong in your view with using IDLE, multi-threaded application, using more than one IMAP connections to the IMAP server. I just want a solution to how to get c-client to refresh it's message cache properly without having to disconnect and reconnect to the server. Regards, Shawn Mark Crispin wrote: On Fri, 13 Mar 2009, Shawn Walker wrote: The application has multiple threads with 2 connections to the IMAP server. One of them is for IDLE. This application does not use c-client to do IMAP client. c-client does not support client-end IDLE. Presumably, by thread, you mean threads in a process as opposed to message threads. UW imapd does not run multi-threaded; each IMAP session has its own process. Nor does the c-client library use threads. So, whatever is threading and using IDLE does not seem to have anything to do with c-client or imapd. When something happen on the IDLE thread, the server send a list of untagged IMAP commands to the client of what happened. The server sends untagged IMAP responses, not commands. The IDLE thread see that it need to update a folder, but the IDLE thread has two messages the UID of 100 and 101 (an example). But, UID 101 is doesn't exist anymore, but UID 102 is on the server. So, the IDLE thread request the message cache for UID 102 but c-client doesn't know about 102 in it's message cache due that it only know of UID 100 and 101 and return with a NIL. Hence, the message cache is stale. This makes no sense, so I have to guess what you are talking about. My guess is that client has two IMAP sessions open. One of those sessions did an IDLE command that notified
Re: [Imap-uw] Unread flag always on after reindexing
On Fri, 13 Mar 2009, Dag Nygren wrote: Ok. Didn't see any warnings on them creating problems. Thought it was more like fsck. Just rebuilding the metadata from the raw messages. Good point. Thanks for bringing it up. Yes, these repair tools are not like fsck. Their task is a single-minded effort to create usable .mixindex (mixrbld) and .mix# (mixdfix) files that (by choosing to run these tools) you have said are unusable for some reason. In the course of performing that task they will happily disregard and destroy perfectly valid information. The equivalent to fsck's check functionality is mailutil check. If mailutil check does not snarl, then there is no reason to run mixrbld or mixdfix. An equivalent to the safe fixing functionality of fsck is actually built into the mix code of the c-client library. The mix code is designed to be self-healing; if it sees a problem that it can fix safely, it will do so without even asking you. So, the mixrbld and mixdfix programs are best thought as unsafe fixing, to be deployed when, for some reason, the self-healing safe fixing didn't work. If the mailbox opens, it is not corrupt in any way that mixrbld/mixdfix can remedy. So that is a good enough test? Will remember that in the future. Yes. Of course the current problem indicates tha opposite as it very well opens, but still doesn't work as expected... Whatever anomaly you found is not something that is addressed by mixrbld or mixdfix. I wish that I knew what that anomaly was. However, your solution of deleting the .mixstatus file and recreating is the current way of dealing with .mixstatus issues. Is ther any documentation on mix? docs/mixfmt.txt Just checked the source code and the actual message was from mixrbld and was the one on row 181, printf (Data file %s UID ran backwards. Isn't that out of sequence. The text out of sequence does not occur in Data file...ran backwards, and thus is a paraphrase. Please don't do that. OK. But is does mean the same thing :-) There are numerous things in mix (and IMAP) that are sequenced. There is also a specific datum that is called a sequence. Neither of these are relevant to this warning message. This warning message merely states that it found a message with a lower UID value than one it had seen before. Since mix tends to order data files in ascending order, this is an interesting event worth noting to an expert repairing the mailbox who wants to understand the nature and scope of the damage. But this situation in data files is harmless on its own, and mixrbld is supposed to allow for that. However, the UW version of mixrbld has a bug in handling this particular anomaly. That can cause more serious problems later on. If you got a Data file...ran backwards message, it is likely that some expunge operation did not complete properly. mixrbld will recover from this. Shouldn't a rerun of mixrbld be clean in that case? No. This is a situation in the data files. mixrbld does not change data files. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Best way to refresh the message numbers of a stale folder?
On Fri, 13 Mar 2009, Shawn Walker wrote: I'll just figure a way to get the message cache update properly. The way to figure it out is to fix the bug in your code. I identified the bug. In case you missed it: either your implementation of IDLE is incorrect, or (as suggested in my previous message) you incorrectly expect state to cross from one session to the other without per-session notification. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
Re: [Imap-uw] Unread flag always on after reindexing
On Sat, 14 Mar 2009, Dag Nygren wrote: Wrote a small program that sorts the .mixstatus file and found that things work better. The noticed that the recreated .mixindex also was out of order (on the UID:s) and ran the same sorting on that file. Now everything works perfect. The out of order .mixindex file is the ultimate cause of the problem. It was created by the broken UW version of mixrbld. The mix code itself won't do that. -- Mark -- http://panda.com/mrc Democracy is two wolves and a sheep deciding what to eat for lunch. Liberty is a well-armed sheep contesting the vote. ___ Imap-uw mailing list Imap-uw@u.washington.edu http://mailman2.u.washington.edu/mailman/listinfo/imap-uw
