Hi Benoit, do we have a path for getting resolution to this problem? I
can do the work with just a little guidance. I have clients getting
very upset that they haven't been able to get mail on their iPhone in
several months. I just tried the "./bin/james-cli.sh ReindexAll"
command on a build from the master branch from a couple of weeks ago,
and I still get the same "Not Implemented" message referenced in a post
below from several months ago. Again, just a little bit of info as to
how to either fix or hack a fix so I can get Lucene to reindex the
database, and I can do the fix and testing. I just have no clue where
to start. Can you give me a bit of a nudge in the right direction?
Thx
Jerry
On 11/6/2019 1:44 AM, Tellier Benoit wrote:
I bet you can overwrite Spring XML bindings and define your own.
I have no idea how to do that, and it seems dangerous.
Regards,
Benoit
On 05/11/2019 22:03, Jerry Malcolm wrote:
No problem. Is there a quick-fix / hack that perhaps I can hardcode to
bypass insertion until the actual fix is available?
Just speculating here... we've got an option (lazyIndex) that rebuilds
the index every time whether it is necessary or not. And we have an
option (luceneIndex) that won't build a full index automatically even
when no index exists for the folder. Seems to me a much better option
would be a 'build it once if it doesn't exist' option. That would be
one less thing for an overwhelmed administrator (me) to remember to do
when migrating James. Ideas?
Jerry
On 11/5/2019 1:04 AM, Tellier Benoit wrote:
My bad, there seems to be Spring injection issues:
mailbox-tools.xml defines:
<bean id ="fake-reindexer"
class="org.apache.mailbox.tools.indexer.ThrowsReIndexer"
lazy-init="true"/>
As the indexer is configurable, we are not able to inject the
ListeningMessageSearchIndex for doing that.
We may need aditional logic upon the indexer choice for providing the
correct reIndexer binding:
- keep the throwing one when lucene is not selected
- Use ReIndexerImpl with luceneSearchIndex injected in it when lucene
is configured.
I will write the issue, contributions are welcomed here!
See https://issues.apache.org/jira/browse/JAMES-2965
Benoit
On 05/11/2019 13:14, Jerry Malcolm wrote:
I had already changed back to luceneIndex from lazyIndex and rebooted.
I also removed the lazyIndex's version of the lucene folder. But I'm
still getting the same exception.
On 11/5/2019 12:07 AM, Tellier Benoit wrote:
ReIndexing is not implemented on top of lazyIndex.
You need to configure lucene in index.xml (meaning for you, downtime in
search)
On 05/11/2019 12:44, Jerry Malcolm wrote:
Benoit,
This all makes sense. I tried the CLI to ReindexAll and got an
exception. I figured I was just doing something wrong on the CLI with
JMX, etc. But every other command I tried worked fine. Just the
ReindexAll and ReindexMailbox failed. (BTW.... the web page shows
the
command: |{cli} Reindex #private u...@domain.tld INBOX but the actual
command is ReindexMailbox for a specific mailbox.)|
|Before I start digging into what is happening, does this exception
tell
you anything? Do I need to configure something in the james-cli.sh
file
telling it that I'm using JPA instead of file-based mailboxes or
anything like that?
|
|Exception:
|
[root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll
ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while
playing command
org.apache.james.mailbox.exception.MailboxException: Not implemented
at
org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43)
at
org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
On 11/4/2019 8:48 PM, Tellier Benoit wrote:
Hi Jerry,
Thanks for the updated details, it now makes perfect sens!
Long story short: Your Lucene index needs to be migrated as well. I
suspect you migrated only your database (full copy), hence issues.
To do migration, at Linagora, we prefer using external protocols, and
leaving the inner data structures untouched (it's just too easy to
make
mistakes...).
What happens is that then you had the data in SQL database but
none in
the Lucene index, hence the unaccurate search results harming the IOS
experience.
From that point:
- You can fix that from re-indexing data from the SQL database
and
rebuild the Lucene index from it. There's webadmin(guice only) [1]
and
CLI [2] support for that (though the CLI experience, the only
available
option for Spring is more primitive.
lazyIndex means full scan on every search request. Expect the AWS
bill
to be expansive. Expect your clients to complain about latencies. I
would be you, I'd rebuild the Lucene index at the first occasion.
Now, I also believe cascading user flags issues might be caused by
some
unappropriate table structure, there was some bugs reported regarding
this. I'm unable to retrieve corresponding information.
[1]
https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails
[2]
https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-cli.md#re-indexing
Cheers,
Benoit
On 05/11/2019 07:47, Jerry Malcolm wrote:
I'm starting a new thread for this specific issue. One external
symptom
that has now been isolated to this problem is documented in my
long-running (and hopefully completed) thread about iPhones
losing all
of their mail.
Here's what I know about this so far:
1) I had no clue what 'Lucene' was or what it did. So I've never
touched anything in any config file related to it. Whatever
happened
came right out of the box as far as lucene config is concern. (ver
3.4+... I extracted the master branch a day or so after 3.4 was
released).
2) I have been in the process of migrating 17+ years of mail in a
James
SQL db from v3b5 on a dedicated server to ver 3.4 in Amazon Web
Services
environment.
3) On 21Oct, I threw the switch and went live on my AWS environment.
4) Immediately some of my clients with iPhones complained that
all of
their mail disappeared from their phone.
5) Being an iPhone user myself, I verified the problem. However, I
noticed that mail arriving to the new James instance was showing
up in
the iPhone. Just no mail pre 21Oct.
6) Thunderbird was showing all of the old mail in the folders. But
I've
since determined that TBird uses a different imap command that goes
straight to the JPA database to populate folders, bypassing the
Lucene
index.
7) After two weeks of educating myself on the entire imap component
and
doing a whole lot of adding log statements and trial & error debug,
the
Lucene index became the prime suspect
8) Today, I removed the var/store/lucene folder and rebooted. The
iPhones now wiped out the last two weeks of email and either showed
"No
Mail" or mail that arrived in the few minutes after wiping the
lucene
folder. Note that it STILL didn't reindex old mail, even with an
index
'restart'.
9) Finally on a whim, I changed indexer.xml to use "lazyIndex"
instead
of "luceneIndex". Finally, the indexes rebuilt successfully, and
the
iPhones now show all of the mail. Success...
Summary: lazyIndex seems to have no problems. Not sure what's
going
wrong when luceneIndex is selected. But it isn't right.
Do I have any mis-assumptions? Is there something else I need to
do in
order to get Lucene to index old mail?
This has been a tough one... I think I'll take a nap now...:-)
Jerry
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org
For additional commands, e-mail: server-dev-h...@james.apache.org
