Re: Lucene Not Indexing Existing Mail in Folders
Hi Benoit, do we have a path for getting resolution to this problem? I can do the work with just a little guidance. I have clients getting very upset that they haven't been able to get mail on their iPhone in several months. I just tried the "./bin/james-cli.sh ReindexAll" command on a build from the master branch from a couple of weeks ago, and I still get the same "Not Implemented" message referenced in a post below from several months ago. Again, just a little bit of info as to how to either fix or hack a fix so I can get Lucene to reindex the database, and I can do the fix and testing. I just have no clue where to start. Can you give me a bit of a nudge in the right direction? Thx Jerry On 11/6/2019 1:44 AM, Tellier Benoit wrote: I bet you can overwrite Spring XML bindings and define your own. I have no idea how to do that, and it seems dangerous. Regards, Benoit On 05/11/2019 22:03, Jerry Malcolm wrote: No problem. Is there a quick-fix / hack that perhaps I can hardcode to bypass insertion until the actual fix is available? Just speculating here... we've got an option (lazyIndex) that rebuilds the index every time whether it is necessary or not. And we have an option (luceneIndex) that won't build a full index automatically even when no index exists for the folder. Seems to me a much better option would be a 'build it once if it doesn't exist' option. That would be one less thing for an overwhelmed administrator (me) to remember to do when migrating James. Ideas? Jerry On 11/5/2019 1:04 AM, Tellier Benoit wrote: My bad, there seems to be Spring injection issues: mailbox-tools.xml defines: As the indexer is configurable, we are not able to inject the ListeningMessageSearchIndex for doing that. We may need aditional logic upon the indexer choice for providing the correct reIndexer binding: - keep the throwing one when lucene is not selected - Use ReIndexerImpl with luceneSearchIndex injected in it when lucene is configured. I will write the issue, contributions are welcomed here! See https://issues.apache.org/jira/browse/JAMES-2965 Benoit On 05/11/2019 13:14, Jerry Malcolm wrote: I had already changed back to luceneIndex from lazyIndex and rebooted. I also removed the lazyIndex's version of the lucene folder. But I'm still getting the same exception. On 11/5/2019 12:07 AM, Tellier Benoit wrote: ReIndexing is not implemented on top of lazyIndex. You need to configure lucene in index.xml (meaning for you, downtime in search) On 05/11/2019 12:44, Jerry Malcolm wrote: Benoit, This all makes sense. I tried the CLI to ReindexAll and got an exception. I figured I was just doing something wrong on the CLI with JMX, etc. But every other command I tried worked fine. Just the ReindexAll and ReindexMailbox failed. (BTW the web page shows the command: |{cli} Reindex #private u...@domain.tld INBOX but the actual command is ReindexMailbox for a specific mailbox.)| |Before I start digging into what is happening, does this exception tell you anything? Do I need to configure something in the james-cli.sh file telling it that I'm using JPA instead of file-based mailboxes or anything like that? | |Exception: | [root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while playing command org.apache.james.mailbox.exception.MailboxException: Not implemented at org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43) at org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) On 11/4/2019 8:48 PM, Tellier Benoit wrote: Hi Jerry, Thanks for the updated details, it now makes perfect sens! Long story short: Your Lucene index needs to be migrated as well. I suspect you migrated only your database (full copy), hence issues. To do migration, at Linagora, we prefer using external protocols, and leaving the inner data structures untouched (it's just too easy to make mistakes...). What happens is that then you had the data in SQL database but none in the Lucene index, hence the unaccurate search results harming the IOS experience. From that point: - You can fix that from re-indexing data from the SQL database and rebuild the Lucene index from it. There's webadmin(guice only) [1] and CLI [2] support for that (though the CLI experience, the only available option for Spring is more primitive. lazyIndex means full scan on every search request. Expect the AWS bill to be expansive. Expect your clients to complain about latencies. I would be you, I'd rebuild the Luce
Re: Lucene Not Indexing Existing Mail in Folders
Hi Jerry, Thanks for the updated details, it now makes perfect sens! Long story short: Your Lucene index needs to be migrated as well. I suspect you migrated only your database (full copy), hence issues. To do migration, at Linagora, we prefer using external protocols, and leaving the inner data structures untouched (it's just too easy to make mistakes...). What happens is that then you had the data in SQL database but none in the Lucene index, hence the unaccurate search results harming the IOS experience. >From that point: - You can fix that from re-indexing data from the SQL database and rebuild the Lucene index from it. There's webadmin(guice only) [1] and CLI [2] support for that (though the CLI experience, the only available option for Spring is more primitive. lazyIndex means full scan on every search request. Expect the AWS bill to be expansive. Expect your clients to complain about latencies. I would be you, I'd rebuild the Lucene index at the first occasion. Now, I also believe cascading user flags issues might be caused by some unappropriate table structure, there was some bugs reported regarding this. I'm unable to retrieve corresponding information. [1] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails [2] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-cli.md#re-indexing Cheers, Benoit On 05/11/2019 07:47, Jerry Malcolm wrote: > I'm starting a new thread for this specific issue. One external symptom > that has now been isolated to this problem is documented in my > long-running (and hopefully completed) thread about iPhones losing all > of their mail. > > Here's what I know about this so far: > > 1) I had no clue what 'Lucene' was or what it did. So I've never > touched anything in any config file related to it. Whatever happened > came right out of the box as far as lucene config is concern. (ver > 3.4+... I extracted the master branch a day or so after 3.4 was released). > > 2) I have been in the process of migrating 17+ years of mail in a James > SQL db from v3b5 on a dedicated server to ver 3.4 in Amazon Web Services > environment. > > 3) On 21Oct, I threw the switch and went live on my AWS environment. > > 4) Immediately some of my clients with iPhones complained that all of > their mail disappeared from their phone. > > 5) Being an iPhone user myself, I verified the problem. However, I > noticed that mail arriving to the new James instance was showing up in > the iPhone. Just no mail pre 21Oct. > > 6) Thunderbird was showing all of the old mail in the folders. But I've > since determined that TBird uses a different imap command that goes > straight to the JPA database to populate folders, bypassing the Lucene > index. > > 7) After two weeks of educating myself on the entire imap component and > doing a whole lot of adding log statements and trial & error debug, the > Lucene index became the prime suspect > > 8) Today, I removed the var/store/lucene folder and rebooted. The > iPhones now wiped out the last two weeks of email and either showed "No > Mail" or mail that arrived in the few minutes after wiping the lucene > folder. Note that it STILL didn't reindex old mail, even with an index > 'restart'. > > 9) Finally on a whim, I changed indexer.xml to use "lazyIndex" instead > of "luceneIndex". Finally, the indexes rebuilt successfully, and the > iPhones now show all of the mail. Success... > > Summary: lazyIndex seems to have no problems. Not sure what's going > wrong when luceneIndex is selected. But it isn't right. > > Do I have any mis-assumptions? Is there something else I need to do in > order to get Lucene to index old mail? > > This has been a tough one... I think I'll take a nap now...:-) > > Jerry > > > - > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > - To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
Re: Lucene Not Indexing Existing Mail in Folders
Benoit, This all makes sense. I tried the CLI to ReindexAll and got an exception. I figured I was just doing something wrong on the CLI with JMX, etc. But every other command I tried worked fine. Just the ReindexAll and ReindexMailbox failed. (BTW the web page shows the command: |{cli} Reindex #private u...@domain.tld INBOX but the actual command is ReindexMailbox for a specific mailbox.)| |Before I start digging into what is happening, does this exception tell you anything? Do I need to configure something in the james-cli.sh file telling it that I'm using JPA instead of file-based mailboxes or anything like that? | |Exception: | [root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while playing command org.apache.james.mailbox.exception.MailboxException: Not implemented at org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43) at org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) On 11/4/2019 8:48 PM, Tellier Benoit wrote: Hi Jerry, Thanks for the updated details, it now makes perfect sens! Long story short: Your Lucene index needs to be migrated as well. I suspect you migrated only your database (full copy), hence issues. To do migration, at Linagora, we prefer using external protocols, and leaving the inner data structures untouched (it's just too easy to make mistakes...). What happens is that then you had the data in SQL database but none in the Lucene index, hence the unaccurate search results harming the IOS experience. From that point: - You can fix that from re-indexing data from the SQL database and rebuild the Lucene index from it. There's webadmin(guice only) [1] and CLI [2] support for that (though the CLI experience, the only available option for Spring is more primitive. lazyIndex means full scan on every search request. Expect the AWS bill to be expansive. Expect your clients to complain about latencies. I would be you, I'd rebuild the Lucene index at the first occasion. Now, I also believe cascading user flags issues might be caused by some unappropriate table structure, there was some bugs reported regarding this. I'm unable to retrieve corresponding information. [1] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails [2] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-cli.md#re-indexing Cheers, Benoit On 05/11/2019 07:47, Jerry Malcolm wrote: I'm starting a new thread for this specific issue. One external symptom that has now been isolated to this problem is documented in my long-running (and hopefully completed) thread about iPhones losing all of their mail. Here's what I know about this so far: 1) I had no clue what 'Lucene' was or what it did. So I've never touched anything in any config file related to it. Whatever happened came right out of the box as far as lucene config is concern. (ver 3.4+... I extracted the master branch a day or so after 3.4 was released). 2) I have been in the process of migrating 17+ years of mail in a James SQL db from v3b5 on a dedicated server to ver 3.4 in Amazon Web Services environment. 3) On 21Oct, I threw the switch and went live on my AWS environment. 4) Immediately some of my clients with iPhones complained that all of their mail disappeared from their phone. 5) Being an iPhone user myself, I verified the problem. However, I noticed that mail arriving to the new James instance was showing up in the iPhone. Just no mail pre 21Oct. 6) Thunderbird was showing all of the old mail in the folders. But I've since determined that TBird uses a different imap command that goes straight to the JPA database to populate folders, bypassing the Lucene index. 7) After two weeks of educating myself on the entire imap component and doing a whole lot of adding log statements and trial & error debug, the Lucene index became the prime suspect 8) Today, I removed the var/store/lucene folder and rebooted. The iPhones now wiped out the last two weeks of email and either showed "No Mail" or mail that arrived in the few minutes after wiping the lucene folder. Note that it STILL didn't reindex old mail, even with an index 'restart'. 9) Finally on a whim, I changed indexer.xml to use "lazyIndex" instead of "luceneIndex". Finally, the indexes rebuilt successfully, and the iPhones now show all of the mail. Success... Summary: lazyIndex seems to have no problems. Not sure what's going wrong when luceneIndex is selected. But it isn't right. Do
Re: Lucene Not Indexing Existing Mail in Folders
ReIndexing is not implemented on top of lazyIndex. You need to configure lucene in index.xml (meaning for you, downtime in search) On 05/11/2019 12:44, Jerry Malcolm wrote: > Benoit, > > This all makes sense. I tried the CLI to ReindexAll and got an > exception. I figured I was just doing something wrong on the CLI with > JMX, etc. But every other command I tried worked fine. Just the > ReindexAll and ReindexMailbox failed. (BTW the web page shows the > command: |{cli} Reindex #private u...@domain.tld INBOX but the actual > command is ReindexMailbox for a specific mailbox.)| > > |Before I start digging into what is happening, does this exception tell > you anything? Do I need to configure something in the james-cli.sh file > telling it that I'm using JPA instead of file-based mailboxes or > anything like that? > | > > |Exception: > | > > [root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll > ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while > playing command > org.apache.james.mailbox.exception.MailboxException: Not implemented > at > org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43) > > at > org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > On 11/4/2019 8:48 PM, Tellier Benoit wrote: >> Hi Jerry, >> >> Thanks for the updated details, it now makes perfect sens! >> >> Long story short: Your Lucene index needs to be migrated as well. I >> suspect you migrated only your database (full copy), hence issues. >> >> To do migration, at Linagora, we prefer using external protocols, and >> leaving the inner data structures untouched (it's just too easy to make >> mistakes...). >> >> What happens is that then you had the data in SQL database but none in >> the Lucene index, hence the unaccurate search results harming the IOS >> experience. >> >> From that point: >> - You can fix that from re-indexing data from the SQL database and >> rebuild the Lucene index from it. There's webadmin(guice only) [1] and >> CLI [2] support for that (though the CLI experience, the only available >> option for Spring is more primitive. >> >> lazyIndex means full scan on every search request. Expect the AWS bill >> to be expansive. Expect your clients to complain about latencies. I >> would be you, I'd rebuild the Lucene index at the first occasion. >> >> Now, I also believe cascading user flags issues might be caused by some >> unappropriate table structure, there was some bugs reported regarding >> this. I'm unable to retrieve corresponding information. >> >> [1] >> https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails >> >> [2] >> https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-cli.md#re-indexing >> >> >> Cheers, >> >> Benoit >> >> On 05/11/2019 07:47, Jerry Malcolm wrote: >>> I'm starting a new thread for this specific issue. One external symptom >>> that has now been isolated to this problem is documented in my >>> long-running (and hopefully completed) thread about iPhones losing all >>> of their mail. >>> >>> Here's what I know about this so far: >>> >>> 1) I had no clue what 'Lucene' was or what it did. So I've never >>> touched anything in any config file related to it. Whatever happened >>> came right out of the box as far as lucene config is concern. (ver >>> 3.4+... I extracted the master branch a day or so after 3.4 was >>> released). >>> >>> 2) I have been in the process of migrating 17+ years of mail in a James >>> SQL db from v3b5 on a dedicated server to ver 3.4 in Amazon Web Services >>> environment. >>> >>> 3) On 21Oct, I threw the switch and went live on my AWS environment. >>> >>> 4) Immediately some of my clients with iPhones complained that all of >>> their mail disappeared from their phone. >>> >>> 5) Being an iPhone user myself, I verified the problem. However, I >>> noticed that mail arriving to the new James instance was showing up in >>> the iPhone. Just no mail pre 21Oct. >>> >>> 6) Thunderbird was showing all of the old mail in the folders. But I've >>> since determined that TBird uses a different imap command that goes >>> straight to the JPA database to populate folders, bypassing the Lucene >>> index. >>> >>> 7) After two weeks of educating myself on the entire imap component and >>> doing a whole lot of adding log statements and trial & error debug, the >>> Lucene index became the prime suspect >>> >>> 8) Today, I removed the var/store/lucene folder and rebooted. The >>> iPhones now wiped out the last two weeks of email and eith
Re: Lucene Not Indexing Existing Mail in Folders
I had already changed back to luceneIndex from lazyIndex and rebooted. I also removed the lazyIndex's version of the lucene folder. But I'm still getting the same exception. On 11/5/2019 12:07 AM, Tellier Benoit wrote: ReIndexing is not implemented on top of lazyIndex. You need to configure lucene in index.xml (meaning for you, downtime in search) On 05/11/2019 12:44, Jerry Malcolm wrote: Benoit, This all makes sense. I tried the CLI to ReindexAll and got an exception. I figured I was just doing something wrong on the CLI with JMX, etc. But every other command I tried worked fine. Just the ReindexAll and ReindexMailbox failed. (BTW the web page shows the command: |{cli} Reindex #private u...@domain.tld INBOX but the actual command is ReindexMailbox for a specific mailbox.)| |Before I start digging into what is happening, does this exception tell you anything? Do I need to configure something in the james-cli.sh file telling it that I'm using JPA instead of file-based mailboxes or anything like that? | |Exception: | [root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while playing command org.apache.james.mailbox.exception.MailboxException: Not implemented at org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43) at org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) On 11/4/2019 8:48 PM, Tellier Benoit wrote: Hi Jerry, Thanks for the updated details, it now makes perfect sens! Long story short: Your Lucene index needs to be migrated as well. I suspect you migrated only your database (full copy), hence issues. To do migration, at Linagora, we prefer using external protocols, and leaving the inner data structures untouched (it's just too easy to make mistakes...). What happens is that then you had the data in SQL database but none in the Lucene index, hence the unaccurate search results harming the IOS experience. From that point: - You can fix that from re-indexing data from the SQL database and rebuild the Lucene index from it. There's webadmin(guice only) [1] and CLI [2] support for that (though the CLI experience, the only available option for Spring is more primitive. lazyIndex means full scan on every search request. Expect the AWS bill to be expansive. Expect your clients to complain about latencies. I would be you, I'd rebuild the Lucene index at the first occasion. Now, I also believe cascading user flags issues might be caused by some unappropriate table structure, there was some bugs reported regarding this. I'm unable to retrieve corresponding information. [1] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails [2] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-cli.md#re-indexing Cheers, Benoit On 05/11/2019 07:47, Jerry Malcolm wrote: I'm starting a new thread for this specific issue. One external symptom that has now been isolated to this problem is documented in my long-running (and hopefully completed) thread about iPhones losing all of their mail. Here's what I know about this so far: 1) I had no clue what 'Lucene' was or what it did. So I've never touched anything in any config file related to it. Whatever happened came right out of the box as far as lucene config is concern. (ver 3.4+... I extracted the master branch a day or so after 3.4 was released). 2) I have been in the process of migrating 17+ years of mail in a James SQL db from v3b5 on a dedicated server to ver 3.4 in Amazon Web Services environment. 3) On 21Oct, I threw the switch and went live on my AWS environment. 4) Immediately some of my clients with iPhones complained that all of their mail disappeared from their phone. 5) Being an iPhone user myself, I verified the problem. However, I noticed that mail arriving to the new James instance was showing up in the iPhone. Just no mail pre 21Oct. 6) Thunderbird was showing all of the old mail in the folders. But I've since determined that TBird uses a different imap command that goes straight to the JPA database to populate folders, bypassing the Lucene index. 7) After two weeks of educating myself on the entire imap component and doing a whole lot of adding log statements and trial & error debug, the Lucene index became the prime suspect 8) Today, I removed the var/store/lucene folder and rebooted. The iPhones now wiped out the last two weeks of email and either showed "No Mail" or mail that arrived in the few minutes after wiping the lucene fol
Re: Lucene Not Indexing Existing Mail in Folders
My bad, there seems to be Spring injection issues: mailbox-tools.xml defines: As the indexer is configurable, we are not able to inject the ListeningMessageSearchIndex for doing that. We may need aditional logic upon the indexer choice for providing the correct reIndexer binding: - keep the throwing one when lucene is not selected - Use ReIndexerImpl with luceneSearchIndex injected in it when lucene is configured. I will write the issue, contributions are welcomed here! See https://issues.apache.org/jira/browse/JAMES-2965 Benoit On 05/11/2019 13:14, Jerry Malcolm wrote: > I had already changed back to luceneIndex from lazyIndex and rebooted. > I also removed the lazyIndex's version of the lucene folder. But I'm > still getting the same exception. > > > On 11/5/2019 12:07 AM, Tellier Benoit wrote: >> ReIndexing is not implemented on top of lazyIndex. >> >> You need to configure lucene in index.xml (meaning for you, downtime in >> search) >> >> On 05/11/2019 12:44, Jerry Malcolm wrote: >>> Benoit, >>> >>> This all makes sense. I tried the CLI to ReindexAll and got an >>> exception. I figured I was just doing something wrong on the CLI with >>> JMX, etc. But every other command I tried worked fine. Just the >>> ReindexAll and ReindexMailbox failed. (BTW the web page shows the >>> command: |{cli} Reindex #private u...@domain.tld INBOX but the actual >>> command is ReindexMailbox for a specific mailbox.)| >>> >>> |Before I start digging into what is happening, does this exception tell >>> you anything? Do I need to configure something in the james-cli.sh file >>> telling it that I'm using JPA instead of file-based mailboxes or >>> anything like that? >>> | >>> >>> |Exception: >>> | >>> >>> [root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll >>> ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while >>> playing command >>> org.apache.james.mailbox.exception.MailboxException: Not implemented >>> at >>> org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43) >>> >>> >>> at >>> org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67) >>> >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> >>> On 11/4/2019 8:48 PM, Tellier Benoit wrote: Hi Jerry, Thanks for the updated details, it now makes perfect sens! Long story short: Your Lucene index needs to be migrated as well. I suspect you migrated only your database (full copy), hence issues. To do migration, at Linagora, we prefer using external protocols, and leaving the inner data structures untouched (it's just too easy to make mistakes...). What happens is that then you had the data in SQL database but none in the Lucene index, hence the unaccurate search results harming the IOS experience. From that point: - You can fix that from re-indexing data from the SQL database and rebuild the Lucene index from it. There's webadmin(guice only) [1] and CLI [2] support for that (though the CLI experience, the only available option for Spring is more primitive. lazyIndex means full scan on every search request. Expect the AWS bill to be expansive. Expect your clients to complain about latencies. I would be you, I'd rebuild the Lucene index at the first occasion. Now, I also believe cascading user flags issues might be caused by some unappropriate table structure, there was some bugs reported regarding this. I'm unable to retrieve corresponding information. [1] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails [2] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-cli.md#re-indexing Cheers, Benoit On 05/11/2019 07:47, Jerry Malcolm wrote: > I'm starting a new thread for this specific issue. One external > symptom > that has now been isolated to this problem is documented in my > long-running (and hopefully completed) thread about iPhones losing all > of their mail. > > Here's what I know about this so far: > > 1) I had no clue what 'Lucene' was or what it did. So I've never > touched anything in any config file related to it. Whatever happened > came right out of the box as far as lucene config is concern. (ver > 3.4+... I extracted the master branch a day or so after 3.4 was > released). > > 2) I have been in the process of migrating 17+ years of mail in a
Re: Lucene Not Indexing Existing Mail in Folders
No problem. Is there a quick-fix / hack that perhaps I can hardcode to bypass insertion until the actual fix is available? Just speculating here... we've got an option (lazyIndex) that rebuilds the index every time whether it is necessary or not. And we have an option (luceneIndex) that won't build a full index automatically even when no index exists for the folder. Seems to me a much better option would be a 'build it once if it doesn't exist' option. That would be one less thing for an overwhelmed administrator (me) to remember to do when migrating James. Ideas? Jerry On 11/5/2019 1:04 AM, Tellier Benoit wrote: My bad, there seems to be Spring injection issues: mailbox-tools.xml defines: As the indexer is configurable, we are not able to inject the ListeningMessageSearchIndex for doing that. We may need aditional logic upon the indexer choice for providing the correct reIndexer binding: - keep the throwing one when lucene is not selected - Use ReIndexerImpl with luceneSearchIndex injected in it when lucene is configured. I will write the issue, contributions are welcomed here! See https://issues.apache.org/jira/browse/JAMES-2965 Benoit On 05/11/2019 13:14, Jerry Malcolm wrote: I had already changed back to luceneIndex from lazyIndex and rebooted. I also removed the lazyIndex's version of the lucene folder. But I'm still getting the same exception. On 11/5/2019 12:07 AM, Tellier Benoit wrote: ReIndexing is not implemented on top of lazyIndex. You need to configure lucene in index.xml (meaning for you, downtime in search) On 05/11/2019 12:44, Jerry Malcolm wrote: Benoit, This all makes sense. I tried the CLI to ReindexAll and got an exception. I figured I was just doing something wrong on the CLI with JMX, etc. But every other command I tried worked fine. Just the ReindexAll and ReindexMailbox failed. (BTW the web page shows the command: |{cli} Reindex #private u...@domain.tld INBOX but the actual command is ReindexMailbox for a specific mailbox.)| |Before I start digging into what is happening, does this exception tell you anything? Do I need to configure something in the james-cli.sh file telling it that I'm using JPA instead of file-based mailboxes or anything like that? | |Exception: | [root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while playing command org.apache.james.mailbox.exception.MailboxException: Not implemented at org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43) at org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) On 11/4/2019 8:48 PM, Tellier Benoit wrote: Hi Jerry, Thanks for the updated details, it now makes perfect sens! Long story short: Your Lucene index needs to be migrated as well. I suspect you migrated only your database (full copy), hence issues. To do migration, at Linagora, we prefer using external protocols, and leaving the inner data structures untouched (it's just too easy to make mistakes...). What happens is that then you had the data in SQL database but none in the Lucene index, hence the unaccurate search results harming the IOS experience. From that point: - You can fix that from re-indexing data from the SQL database and rebuild the Lucene index from it. There's webadmin(guice only) [1] and CLI [2] support for that (though the CLI experience, the only available option for Spring is more primitive. lazyIndex means full scan on every search request. Expect the AWS bill to be expansive. Expect your clients to complain about latencies. I would be you, I'd rebuild the Lucene index at the first occasion. Now, I also believe cascading user flags issues might be caused by some unappropriate table structure, there was some bugs reported regarding this. I'm unable to retrieve corresponding information. [1] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails [2] https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-cli.md#re-indexing Cheers, Benoit On 05/11/2019 07:47, Jerry Malcolm wrote: I'm starting a new thread for this specific issue. One external symptom that has now been isolated to this problem is documented in my long-running (and hopefully completed) thread about iPhones losing all of their mail. Here's what I know about this so far: 1) I had no clue what 'Lucene' was or what it did. So I've never touched anything in any config file related to it. Whatever happened came right out of the box as f
Re: Lucene Not Indexing Existing Mail in Folders
I bet you can overwrite Spring XML bindings and define your own. I have no idea how to do that, and it seems dangerous. Regards, Benoit On 05/11/2019 22:03, Jerry Malcolm wrote: > No problem. Is there a quick-fix / hack that perhaps I can hardcode to > bypass insertion until the actual fix is available? > > Just speculating here... we've got an option (lazyIndex) that rebuilds > the index every time whether it is necessary or not. And we have an > option (luceneIndex) that won't build a full index automatically even > when no index exists for the folder. Seems to me a much better option > would be a 'build it once if it doesn't exist' option. That would be > one less thing for an overwhelmed administrator (me) to remember to do > when migrating James. Ideas? > > Jerry > > On 11/5/2019 1:04 AM, Tellier Benoit wrote: >> My bad, there seems to be Spring injection issues: >> >> mailbox-tools.xml defines: >> >> > class="org.apache.mailbox.tools.indexer.ThrowsReIndexer" >> lazy-init="true"/> >> >> As the indexer is configurable, we are not able to inject the >> ListeningMessageSearchIndex for doing that. >> >> We may need aditional logic upon the indexer choice for providing the >> correct reIndexer binding: >> - keep the throwing one when lucene is not selected >> - Use ReIndexerImpl with luceneSearchIndex injected in it when lucene >> is configured. >> >> I will write the issue, contributions are welcomed here! >> >> See https://issues.apache.org/jira/browse/JAMES-2965 >> >> Benoit >> >> On 05/11/2019 13:14, Jerry Malcolm wrote: >>> I had already changed back to luceneIndex from lazyIndex and rebooted. >>> I also removed the lazyIndex's version of the lucene folder. But I'm >>> still getting the same exception. >>> >>> >>> On 11/5/2019 12:07 AM, Tellier Benoit wrote: ReIndexing is not implemented on top of lazyIndex. You need to configure lucene in index.xml (meaning for you, downtime in search) On 05/11/2019 12:44, Jerry Malcolm wrote: > Benoit, > > This all makes sense. I tried the CLI to ReindexAll and got an > exception. I figured I was just doing something wrong on the CLI with > JMX, etc. But every other command I tried worked fine. Just the > ReindexAll and ReindexMailbox failed. (BTW the web page shows > the > command: |{cli} Reindex #private u...@domain.tld INBOX but the actual > command is ReindexMailbox for a specific mailbox.)| > > |Before I start digging into what is happening, does this exception > tell > you anything? Do I need to configure something in the james-cli.sh > file > telling it that I'm using JPA instead of file-based mailboxes or > anything like that? > | > > |Exception: > | > > [root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll > ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while > playing command > org.apache.james.mailbox.exception.MailboxException: Not implemented > at > org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43) > > > > at > org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > > at java.lang.reflect.Method.invoke(Method.java:498) > > On 11/4/2019 8:48 PM, Tellier Benoit wrote: >> Hi Jerry, >> >> Thanks for the updated details, it now makes perfect sens! >> >> Long story short: Your Lucene index needs to be migrated as well. I >> suspect you migrated only your database (full copy), hence issues. >> >> To do migration, at Linagora, we prefer using external protocols, and >> leaving the inner data structures untouched (it's just too easy to >> make >> mistakes...). >> >> What happens is that then you had the data in SQL database but >> none in >> the Lucene index, hence the unaccurate search results harming the IOS >> experience. >> >> From that point: >> - You can fix that from re-indexing data from the SQL database >> and >> rebuild the Lucene index from it. There's webadmin(guice only) [1] >> and >> CLI [2] support for that (though the CLI experience, the only >> available >> option for Spring is more primitive. >> >> lazyIndex means full scan on every search request. Expect the AWS >> bill >> to be expansive. Expect your clients to complain about latencies. I >> would be you, I'd rebuild the Lucene index at the first occasion. >> >> Now, I also believe cascading us