Finally, after 2 weeks, iPhones have all of their mail.  The problem was that my Lucene index was corrupted. Or to be more precise, only emails from the last two weeks since I first started up James on the new environment were indexed. Even after renaming (effectively removing) the /var/store/lucene folder, the index would not fully rebuild.  Same problem... but worse now... now it showed only mail that came in over the last 10 minutes since I cleaned the lucene index folder and lost the last 2 weeks' mail that actually was indexed.  On a whim, I changed indexer.xml to use lazyIndex instead of luceneIndex... voila.... all of the folders were fully indexed, and iPhone has all of the mail for each folder.  Couldn't find much documentation on lazyIndex vs. luceneIndex.  But at this point, I think I'll take having mail back on my client's iPhones over any downside to using lazyIndex. But.... educate me if there are serious downsides to using lazyIndex.

Thunderbird populates folders using a different imap command which, apparently, goes straight to the JPA database instead of using the Lucene index.  So that's why the problem didn't appear in TBird.

I'm going to start another thread in the dev group specifically to discuss the Lucene issue.

Jerry

On 10/31/2019 12:35 PM, Jerry Malcolm wrote:
(I'm moving this thread to the dev forum since it appears now to not be a user problem)....

More data now... Since iPhone doesn't load the correct email list and TBird does, I traced the precise same scenario with the same email account with TBird as the previous iPhone trace in the post below.  As I suspected, in order to do an initial full sync on a folder, TBird is asking for the list of UIDs in a different way. Note from the two traces:

TBird: UID fetch 1:* (FLAGS)
    Response: 1 FETCH (FLAGS (\\Seen) UID 5801)  (...followed by many other UIDs including the ones in iPhone search below)
---
iPhone: UID SEARCH RETURN (COUNT) 1:* NOT DELETED
     Response: ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033

The two queries above are for the same email account/folder. Again, I'm not an IMAP expert.  But it appears that both should return the same list of UIDs even though they ask for it differently.

Summary... the TBird query gets all UIDs for the folder starting at 5801 as expected.  The iPhone query gets a subset of UIDs starting at 7776.  This continues to support my current theory that the code that processes UID SEARCH RETURN (COUNT) 1:*.... is not correct.

If someone knows what class processes these imap queries, please let me know.  I'm going to dig into the code now.  But a few pointers to the package/classes would be helpful.

Thunderbird trace:

127 Request: 19 select "INBOX"
129,Response: * FLAGS (\\Answered \\Deleted \\Draft \\Flagged \\Seen "MailFlagBit1 "MailFlagBit0 NonJunk "label3 "label1 "Forwarded)
130,Response: 19 OK [READ-WRITE] SELECT completed.
132 Request: 20 getquotaroot "INBOX"
134 Response: * QUOTAROOT "INBOX" #private&je...@malcolms.com
135,Response: 20 OK GETQUOTAROOT completed.
137,Request: 21 UID fetch 1:* (FLAGS)
139,Response: * 1 FETCH (FLAGS (\\Seen) UID 5801)
140,Response: * 2 FETCH (FLAGS (\\Seen) UID 5802)
141,Response: * 4 FETCH (FLAGS (\\Seen) UID 5804)

On 10/29/2019 3:12 PM, Jerry Malcolm wrote:
Ok, I need an IMAP expert.... Below is a very brief trace of the communications between iPhone mail and JAMES (3.4).  I completely deleted an account on my iPhone, then recreated it while in airplane mode to make sure I didn't miss any communications in my trace.  I started the trace, exited airplane mode and let the iPhone do an initial sync with the account.  The inbox folder in this account has over 1000 emails going back to early 2019.

I'm not an expert in IMAP.  But it appears that the iPhone mail app requests all of the emails 1:* (see line 812), but JAMES returns a single id plus two ranges (line 813).  But the total count JAMES reports is nowhere near the full 1000.  Subsequently (line 822), iPhone requests the emails JAMES told it about in line 813.  From what I can tell, the problem is in line 813. JAMES did not report all of the actual mail that is in the INBOX, which is exactly what I'm seeing on the phone.  Shouldn't JAMES return all 1000+ email ids that exist in INBOX?  Or am I reading the IMAP trace incorrectly?

Note that the only emails JAMES reports came in AFTER I migrated to the new server and to the James 3.4 from v3b5.  It appears that there is something different about pre-existing emails in the mailbox folder that is causing JAMES not to recognize them. But this problem ONLY exists on the iPhone mail app. Thunderbird gets all of the mail for the inbox.  So is iPhone must be sending a slightly different request command syntax than other clients do (??)  Just speculating....

 798 IMAP Request: 37 SELECT INBOX
 802 IMAP Response: 37 OK [READ-WRITE] SELECT completed.
 806 IMAP Request: 38 UID SEARCH RETURN (COUNT) 1:* NOT DELETED
 808 IMAP Response: * ESEARCH (TAG "38") UID COUNT 76
 809 IMAP Response: 38 OK SEARCH completed.
 812 IMAP Request: 39 UID SEARCH RETURN (ALL) 1:* NOT DELETED
 813 IMAP Response: * ESEARCH (TAG "39") UID ALL 7776,7955:7970,7975:8033
 814 IMAP Response: 39 OK SEARCH completed.
 817 IMAP Request: 40 UID SEARCH RETURN (ALL) 929:* NOT DELETED
 818 IMAP Response: * ESEARCH (TAG "40") UID ALL 7955:7970,7975:8033
 819 IMAP Response: 40 OK SEARCH completed.
 822 IMAP Request: 41 UID FETCH 7955:7970,7975:8033 (INTERNALDATE UID RFC822.SIZE FLAGS BODY.PEEK[HEADER.FIELDS (date subject from content-type to cc bcc message-id in-reply-to references list-id x-universally-unique-identifier)])

On 10/22/2019 3:07 AM, Matthieu Baechler wrote:
Hi Jerry,

I don't think trying to guess what happen will help you much. I would
rather try to capture the IMAP session.

I see two ways to do that:

1. you write a IMAP session logger in James (it should be really easy,
I don't know why it's not already there)

2. you disable SSL on the phone and do a tcpdump on the server

Once you'll have the IMAP session capture, it will be easier to
diagnose your problem.

Cheers,


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org

Reply via email to