Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-26 Thread Aki Tuomi


> On 25/08/2021 23:05 Steve Dondley  wrote:
> 
>  
> > The search time was no better with it on than off.
> > 
> > So I'm thinking I got something misconfigured somewhere. It seems IMAP
> > may not be using solr to fetch results. But this would be odd since I
> > definitely do see a big improvements in times with fts plugins turned
> > on when using roundcube.
> 
> OK, I'm finally getting somewhere. Found this lead here: 
> https://dovecot.org/pipermail/dovecot/2012-February/081514.html
> 
> "doveadm fts optimize" throws an error:
> 
> doveadm(root): Error: Couldn't drop privileges: User is missing UID (see 
> mail_uid setting)
> 
> 
> Checking /var/log/mail.err, I see a ton of these entries:
> 
> 83353 Aug 25 14:53:22 email dovecot: imap(s)<26536>: 
> Error: fts_solr: received invalid uid '
> 83354 Aug 25 14:53:22 email dovecot: imap: Error: '
> 83355 Aug 25 14:58:05 email dovecot: imap(s)<26637>: 
> Error: fts_solr: received invalid uid '
> 83356 Aug 25 14:58:05 email dovecot: imap: Error: '
> 83357 Aug 25 15:26:58 email dovecot: imap(s)<27217>: 
> Error: fts_solr: received invalid uid '
> 83358 Aug 25 15:26:58 email dovecot: imap: Error: '
> 83359 Aug 25 15:27:11 email dovecot: imap(s)<27217>: 
> Error: fts_solr: received invalid uid '
> 83360 Aug 25 15:27:11 email dovecot: imap: Error: '
> 83361 Aug 25 15:27:36 email dovecot: imap(s)<27217>: 
> Error: fts_solr: received invalid uid '
> 83362 Aug 25 15:27:36 email dovecot: imap: Error: '
> 
> I think this will be nailed once I figure out this issue.

doveadm fts optimize -u TARGETUSER

Otherwise, as any other doveadm mail command, it assumes -u $USER

Aki


Re: What kind of search response time are you setting with solr full text search?

2021-08-26 Thread azurit



Citát Steve Dondley :


On 2021-08-24 08:53 PM, Steve Dondley wrote:

MY SETUP: I have apache solr full text search enabled with dovecot. I
have an inbox with about 40 subfolders. I'm using the roundcube
web-based mail client. The find command is showing 15823 email files
and apache solr reports the same number. I'm running a dedicated mail
server with a 1 GB of ram. The solr machine is running on a separate
machine with 4 GB ram.

THE PROBLEM: When I do a full text search through all my inbox and all
subfolders on a single word, search results are returned in about 10
to 15 seconds. This is better than the 40 seconds or so I'm getting
when I turn off the fts and fts_solr plugins but still a little
disappointing.

WHAT I'D LIKE: Now, I don't expect instant search results like gmail,
but getting the search results to display in less than 5 seconds would
be a lot less painful that 10 to 15 seconds.

WHAT I'VE TRIED: I have reindexed the emails on solr to try to speed
things upit up.

ANY ADVICE? Curious to know what response times others are seeing and
if there is anything else I might try to speed things up. Maybe the
number of subfolders is slowing things down?


OK, I figured it out. It was a bug in Debian's version of dovecot:  
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970692


Upgrading to "Bulleyse" fixed things up nicely and I now have super  
fast search.


Now I have to go tackle the plugin errors that are crashing the  
roundcube install. :)



Look also here:
https://repo.dovecot.org/




Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley

On 2021-08-24 08:53 PM, Steve Dondley wrote:

MY SETUP: I have apache solr full text search enabled with dovecot. I
have an inbox with about 40 subfolders. I'm using the roundcube
web-based mail client. The find command is showing 15823 email files
and apache solr reports the same number. I'm running a dedicated mail
server with a 1 GB of ram. The solr machine is running on a separate
machine with 4 GB ram.

THE PROBLEM: When I do a full text search through all my inbox and all
subfolders on a single word, search results are returned in about 10
to 15 seconds. This is better than the 40 seconds or so I'm getting
when I turn off the fts and fts_solr plugins but still a little
disappointing.

WHAT I'D LIKE: Now, I don't expect instant search results like gmail,
but getting the search results to display in less than 5 seconds would
be a lot less painful that 10 to 15 seconds.

WHAT I'VE TRIED: I have reindexed the emails on solr to try to speed
things upit up.

ANY ADVICE? Curious to know what response times others are seeing and
if there is anything else I might try to speed things up. Maybe the
number of subfolders is slowing things down?


OK, I figured it out. It was a bug in Debian's version of dovecot: 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970692


Upgrading to "Bulleyse" fixed things up nicely and I now have super fast 
search.


Now I have to go tackle the plugin errors that are crashing the 
roundcube install. :)


Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley

On 2021-08-25 04:32 PM, Shawn Heisey wrote:

On 8/25/2021 2:10 PM, Steve Dondley wrote:


And it looks like I'm running into a major bug in the slightly dated 
version of dovecot debian uses:


https://www.mail-archive.com/dovecot@dovecot.org/msg78825.html



Recently I did a fairly major upgrade.  I had an older Ubuntu release
with Dovecot 1.x and an older Postfix version, wanted to upgrade it to
v20, which had significantly newer versions of both programs.

Before I did the Ubuntu upgrade, I took advice received here and
installed the dovecot repo, upgrading it first.  I looked at the
following link and only found a couple of things in my config I needed
to change.  I didn't even use the conversion command at the top of the
page.  Everything still worked after I upgraded, which I found a
little surprising.  Postfix also worked after the upgrade, with no
config changes required.

https://doc.dovecot.org/installation_guide/upgrading/from-1.2-to-2.0/

I did the fts_solr integration *after* I upgraded dovecot.

You can find the dovecot repos here, there are options for Debian:

https://repo.dovecot.org/

Thanks,
Shawn


Fixed!

I just upgraded to Debian bullseye and I'm not enjoying lightning fast 
search! Woohoo!


Figuring out what *isn't* wrong is a good strategy to be sure you aren't 
chasing ghosts. So I greatly appreciate your time and patience. I will 
be sure to pay it forward.


Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Shawn Heisey

On 8/25/2021 2:10 PM, Steve Dondley wrote:


And it looks like I'm running into a major bug in the slightly dated 
version of dovecot debian uses:


https://www.mail-archive.com/dovecot@dovecot.org/msg78825.html



Recently I did a fairly major upgrade.  I had an older Ubuntu release 
with Dovecot 1.x and an older Postfix version, wanted to upgrade it to 
v20, which had significantly newer versions of both programs.


Before I did the Ubuntu upgrade, I took advice received here and 
installed the dovecot repo, upgrading it first.  I looked at the 
following link and only found a couple of things in my config I needed 
to change.  I didn't even use the conversion command at the top of the 
page.  Everything still worked after I upgraded, which I found a little 
surprising.  Postfix also worked after the upgrade, with no config 
changes required.


https://doc.dovecot.org/installation_guide/upgrading/from-1.2-to-2.0/

I did the fts_solr integration *after* I upgraded dovecot.

You can find the dovecot repos here, there are options for Debian:

https://repo.dovecot.org/

Thanks,
Shawn



Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley




I think this will be nailed once I figure out this issue.


And it looks like I'm running into a major bug in the slightly dated
version of dovecot debian uses:

https://www.mail-archive.com/dovecot@dovecot.org/msg78825.html


And this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=970692

Though my error is slightly different. Instead of uid '0' I have no uid 
at all.


Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley

On 2021-08-25 04:05 PM, Steve Dondley wrote:

The search time was no better with it on than off.

So I'm thinking I got something misconfigured somewhere. It seems IMAP
may not be using solr to fetch results. But this would be odd since I
definitely do see a big improvements in times with fts plugins turned
on when using roundcube.


OK, I'm finally getting somewhere. Found this lead here:
https://dovecot.org/pipermail/dovecot/2012-February/081514.html

"doveadm fts optimize" throws an error:

doveadm(root): Error: Couldn't drop privileges: User is missing UID
(see mail_uid setting)


Checking /var/log/mail.err, I see a ton of these entries:

83353 Aug 25 14:53:22 email dovecot: imap(s)<26536>:
Error: fts_solr: received invalid uid '
83354 Aug 25 14:53:22 email dovecot: imap: Error: '
83355 Aug 25 14:58:05 email dovecot: imap(s)<26637>:
Error: fts_solr: received invalid uid '
83356 Aug 25 14:58:05 email dovecot: imap: Error: '
83357 Aug 25 15:26:58 email dovecot: imap(s)<27217>:
Error: fts_solr: received invalid uid '
83358 Aug 25 15:26:58 email dovecot: imap: Error: '
83359 Aug 25 15:27:11 email dovecot: imap(s)<27217>:
Error: fts_solr: received invalid uid '
83360 Aug 25 15:27:11 email dovecot: imap: Error: '
83361 Aug 25 15:27:36 email dovecot: imap(s)<27217>:
Error: fts_solr: received invalid uid '
83362 Aug 25 15:27:36 email dovecot: imap: Error: '

I think this will be nailed once I figure out this issue.


And it looks like I'm running into a major bug in the slightly dated 
version of dovecot debian uses:


https://www.mail-archive.com/dovecot@dovecot.org/msg78825.html


Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley




The search time was no better with it on than off.

So I'm thinking I got something misconfigured somewhere. It seems IMAP
may not be using solr to fetch results. But this would be odd since I
definitely do see a big improvements in times with fts plugins turned
on when using roundcube.


OK, I'm finally getting somewhere. Found this lead here: 
https://dovecot.org/pipermail/dovecot/2012-February/081514.html


"doveadm fts optimize" throws an error:

doveadm(root): Error: Couldn't drop privileges: User is missing UID (see 
mail_uid setting)



Checking /var/log/mail.err, I see a ton of these entries:

83353 Aug 25 14:53:22 email dovecot: imap(s)<26536>: 
Error: fts_solr: received invalid uid '

83354 Aug 25 14:53:22 email dovecot: imap: Error: '
83355 Aug 25 14:58:05 email dovecot: imap(s)<26637>: 
Error: fts_solr: received invalid uid '

83356 Aug 25 14:58:05 email dovecot: imap: Error: '
83357 Aug 25 15:26:58 email dovecot: imap(s)<27217>: 
Error: fts_solr: received invalid uid '

83358 Aug 25 15:26:58 email dovecot: imap: Error: '
83359 Aug 25 15:27:11 email dovecot: imap(s)<27217>: 
Error: fts_solr: received invalid uid '

83360 Aug 25 15:27:11 email dovecot: imap: Error: '
83361 Aug 25 15:27:36 email dovecot: imap(s)<27217>: 
Error: fts_solr: received invalid uid '

83362 Aug 25 15:27:36 email dovecot: imap: Error: '

I think this will be nailed once I figure out this issue.









Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley




I'm inclined to believe the problem is not that high up the food
chain. Because when I query IMAP on a single folder over telnet
following the instructions found here:
https://doc.dovecot.org/configuration_manual/fts/solr/, imap reports
that it's taking 3 to 4 seconds to return results:

a search text "maynez"
a OK Search completed (3.386 + 0.001 + 0.250 secs).

This particular search returns a few hundred results.

The speed of the query changed depending on whether a lot of results
are returned. So if I search on a nonsensical word like "zyzzix", imap
reports results nearly instantaneously:

a search text "zyzzix"
a OK Search completed (0.012 + 0.000 + 0.006 secs).


I just did a quick test. I did a search over imap over telnet with fts 
plugins turned off and with them turned on.


The search time was no better with it on than off.

So I'm thinking I got something misconfigured somewhere. It seems IMAP 
may not be using solr to fetch results. But this would be odd since I 
definitely do see a big improvements in times with fts plugins turned on 
when using roundcube.


Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley




Random guess... Buffering?
Whatever is sending to the browser isn't sending enough bytes to flush
the buffer so the data is left in limbo until enough time goes by the
buffer gets flushed anyways. Maybe a apache/nginx thing, php thing or
browser thing. Remember its solr > dovecot > php > web server >
browser. Tried other browsers?


I'm inclined to believe the problem is not that high up the food chain. 
Because when I query IMAP on a single folder over telnet following the 
instructions found here: 
https://doc.dovecot.org/configuration_manual/fts/solr/, imap reports 
that it's taking 3 to 4 seconds to return results:


a search text "maynez"
a OK Search completed (3.386 + 0.001 + 0.250 secs).

This particular search returns a few hundred results.

The speed of the query changed depending on whether a lot of results are 
returned. So if I search on a nonsensical word like "zyzzix", imap 
reports results nearly instantaneously:


a search text "zyzzix"
a OK Search completed (0.012 + 0.000 + 0.006 secs).


Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread dovecot

So this looks really good and fast. So I think we can say with
confidence solr is doing its job. So why is roundcube/dovecot taking
so long to show the results?


Random guess... Buffering?
Whatever is sending to the browser isn't sending enough bytes to flush 
the buffer so the data is left in limbo until enough time goes by the 
buffer gets flushed anyways. Maybe a apache/nginx thing, php thing or 
browser thing. Remember its solr > dovecot > php > web server > browser. 
Tried other browsers?


Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley

On 2021-08-25 02:05 PM, Steve Dondley wrote:

Try this in on the commandline of the Solr server:

time curl
"http://localhost:YYY/solr/dovecot/select?q=maynez=edismax=body+to+subject+cc+from;


OK I had to modify the query path slightly to get it to work with my 
core to:


time curl
http://localhost:8983/solr/dondley/select?q=maynez=edismax=body+to+subjec:t+ccfrom



OK! found the issue. My command line mangled the URL when I edited the 
url. There is a stray ":" in the query string.


So I am now seeing a ton of results and the following query time:

real0m0.118s
user0m0.003s
sys 0m0.011s

So this looks really good and fast. So I think we can say with 
confidence solr is doing its job. So why is roundcube/dovecot taking so 
long to show the results?




Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley




That query should search ALL emails that dovecot has indexed to Solr. 
There is no restriction for mailbox or folder.


OK.

Try replacing "maynez" with something else that you know will be in the 
index.


Did a search on "the". Still nothing. Very, very weird. What would 
explain why my email client is still returning results I get nothing 
from the command line?


Here is the exact command I'm running:

time curl 
http://172.30.0.94:8983/solr/dondley/select?q=the=edismax=body+to+subjec:t+ccfrom


Do you see any typos in there?


Note that you can ask dovecot to completely reindex everybody's email
with this command run as root, and then you can try searching again a
few minutes later:

doveadm index -A -q '*'


Yeah, tried this yesterday. Didn't help. If you think I should try 
again, let me know.




Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Shawn Heisey

On 8/25/2021 12:05 PM, Steve Dondley wrote:


OK I had to modify the query path slightly to get it to work with my 
core to:


time curl 
http://localhost:8983/solr/dondley/select?q=maynez=edismax=body+to+subjec:t+ccfrom


But it didn't return any results:





I only have emails for this person in a subfolder of my main Inbox 
folder so maybe it's only searching the top level folder? 


That query should search ALL emails that dovecot has indexed to Solr.  
There is no restriction for mailbox or folder.


Try replacing "maynez" with something else that you know will be in the 
index.


Note that you can ask dovecot to completely reindex everybody's email 
with this command run as root, and then you can try searching again a 
few minutes later:


doveadm index -A -q '*'

It took me forever to work out the right options for that command.  I 
think it should probably get added to the wiki page about fts_solr.


Question for experts on the dovecot side of this:  What does it take for 
dovecot to index new messages delivered with dovecot's LDA by postfix?  
Is it a matter of waiting a certain amount of time?  I saw in Solr that 
the last update to the index was 3 hours earlier, but I had received 
email only a few minutes before looking at Solr's stats.


Thanks,
Shawn



Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley




Try this in on the commandline of the Solr server:

time curl
"http://localhost:YYY/solr/dovecot/select?q=maynez=edismax=body+to+subject+cc+from;


OK I had to modify the query path slightly to get it to work with my 
core to:


time curl 
http://localhost:8983/solr/dondley/select?q=maynez=edismax=body+to+subjec:t+ccfrom


But it didn't return any results:

Results:


  "responseHeader":{
"status":0,
"QTime":7,
"params":{
  "q":"maynez"}},
  "response":{"numFound":0,"start":0,"numFoundExact":true,"docs":[]
  }}

real0m0.018s
user0m0.004s
sys 0m0.006s

I only have emails for this person in a subfolder of my main Inbox 
folder so maybe it's only searching the top level folder?




You could do the query remotely by changing "localhost" to the
hostname or IP address of the Solr server.



From remote host, I got similar numbers (no results):

real0m0.017s
user0m0.009s
sys 0m0.002s



Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Shawn Heisey

On 8/25/2021 11:33 AM, Steve Dondley wrote:
OK, I take this back. I did an imap search via telnet and solr reports 
the search takes about 3 to 4 seconds. Here's the output:


a search text "maynez" 


Try this in on the commandline of the Solr server:

time curl 
"http://localhost:YYY/solr/dovecot/select?q=maynez=edismax=body+to+subject+cc+from;


Where YYY is the port Solr is listening on.

In the response, QTime is how long it took for Solr to do the search.  
This does not include time for the result set to be gathered or the time 
to transfer the response over the network, which is where the "time" 
part of that comes in.  I see this doing a similar query on my Solr:


real    0m0.026s
user    0m0.011s
sys 0m0.010s

QTime for that request was 1 millisecond -- so it took 25 milliseconds 
for the stuff that is not counted in QTime.


You could do the query remotely by changing "localhost" to the hostname 
or IP address of the Solr server.


Thanks,
Shawn



Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley





This is a search for "a" which I had run several times, so Solr was
serving it from its cache, and this time it only took 6 milliseconds. 
It also shows what a facet can do.  The longest time I got for the "a"
search was 15 milliseconds, before the query was in the cache.


I think they queries themselves are returning very quickly, at least
they were when I did a query on a single inbox. That's why I'm
wondering if doing a search across 40 different inboxes via roundcube
might be the issue. I'm thinking that each time a new mailbox is
selected, it slows things down. But I have no idea how the IMAP search
is performed across multiple inboxes so that's just a wild guess.


OK, I take this back. I did an imap search via telnet and solr reports 
the search takes about 3 to 4 seconds. Here's the output:


a search text "maynez"
* SEARCH 5 6 7 32 61 64 69 70 117 118 119 120 121 122 123 124 126 127 
129 165 197 202 203 204 205 206 207 216 231 259 451 452 453 454 455 456 
482 730 731 810 811 812 813 814 815 816 817 818 819 820 829 830 831 832 
852 853 854 855 867 868 869 870 871 872 873 874 875 886 887 888 889 891 
904 908 909 910 911 912 913 920 924 925 926 927 928 931 936 938 940 941 
944 946 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 
964 965 966 969 970 971 972 973 974 975 976 977 986 987 988 989 993 1012 
1013 1014 1015 1016 1017 1019 1020 1021 1022 1023 1024 1025 1026 1027 
1030 1031 1032 1033 1034 1035 1036 1037 1038 1039 1040 1041 1042 1043 
1044 1047 1048 1069 1091 1112 1113 1114 1121 1126 1127 1128 1129 1130 
1131 1132 1133 1134 1135 1136 1144 1145 1146 1147 1148 1150 1159 1160 
1161 1162 1175 1176 1177 1178 1179 1180 1181 1186 1187 1188 1189 1190 
1192 1193 1197 1198 1202 1204 1205 1208 1214 1215 1216 1217 1218 1228 
1229 1231 1232 1234 1236 1237 1243 1244 1245 1246 1247 1248 1249 1250 
1252 1256 1257 1297 1298 1299 1300 1301 1302 1303 1304 1307 1308 1316 
1317 1319 1320 1327 1328 1331 1332 1335 1336 1348 1349 1352 1355 1356 
1358 1359 1361 1417 1418 1419 1420 1421 1423 1424 1443 1444 1445 1446 
1447 1453 1460 1461 1462 1463 1464 1500 1501 1502 1503 1504 1505 1507 
1508 1509 1510 1513 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 
1532 1533 1534 1535 1539 1541 1542 1543 1544 1594 1596 1597 1598 1599 
1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 
1620 1621 1622 1623 1628 1629 1631 1632 1633 1634 1635 1636 1637 1638 
1639 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1660 1661 1663 
1664 1665 1666 1674 1675 1676 1677 1679 1680 1688 1691 1693 1694 1695 
1696

a OK Search completed (3.029 + 0.001 + 0.228 secs).



Re: [OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Steve Dondley
One other data point from my experimenting that might shed some light on 
the problem:


If I limit a search to a single folder instead of across all folders, it 
still takes 5 or 6 seconds for the results to appear. So that kind of 
destroys my theory that the problem might be caused by having too many 
inbox folders.


[OT] Re: What kind of search response time are you setting with solr full text search?

2021-08-25 Thread Shawn Heisey

On 8/24/2021 7:19 PM, Steve Dondley wrote:

THE PROBLEM: When I do a full text search through all my inbox and all
subfolders on a single word, search results are returned in about 10
to 15 seconds. This is better than the 40 seconds or so I'm getting
when I turn off the fts and fts_solr plugins but still a little
disappointing.


I did some experimenting. I noticed that if the word I'm searching on 
is fairly rare, results will pop up quickly, like in around 3 to 5 
seconds. Words that don't exist at all in any email returns nothing 
almost instantly.


But words that appear in several hundred emails are the ones that are 
take a much longer time. 


This is offtopic for this list, but I will try to help you.  If I am 
unsuccessful, you should raise the issue on the solr-users mailing list.


How much of the total server memory of 4GB did you give to Solr for its 
heap?  Is there other software running on that server besides Solr?


What's the total size of all the Solr indexes on the Solr server?

Can you get the screenshot mentioned at the following URL, put it on a 
file-sharing site, and give me the URL?


https://cwiki.apache.org/confluence/display/SOLR/SolrPerformanceProblems#SolrPerformanceProblems-Askingforhelponamemory/performanceissue

(disclaimer: I wrote that Solr wiki page)

You should read the entire page, the link above is to the section 
describing useful screenshots)


General note:  A Solr search that takes 3 seconds (let alone 15) would 
have me concerned.  If the system is sized appropriately, I would expect 
a search even on a massive index to complete in less than a second.


I happen to be using Solr for dovecot myself.  If I search my index for 
"the" which is very common in English text, the query takes 19 
milliseconds, and that is searching on five fields, as well as doing a 
facet on the user field.  My Solr index has 150048 messages (122K of 
those are in my personal mailbox) and takes up 628 megabytes of disk 
space.  The total size of the email that is indexed is 7 gigabytes.


|+(cc:the | from:the | to:the | 
body:the | subject:the) |


My index is using the stopword filter but the list of stopwords is empty.

The following response may interest you:

https://apaste.info/hm42

This is a search for "a" which I had run several times, so Solr was 
serving it from its cache, and this time it only took 6 milliseconds.  
It also shows what a facet can do.  The longest time I got for the "a" 
search was 15 milliseconds, before the query was in the cache.


Thanks,
Shawn



Re: What kind of search response time are you setting with solr full text search?

2021-08-24 Thread Gedalya
On 8/25/21 9:19 AM, Steve Dondley wrote:
> I did some experimenting. I noticed that if the word I'm searching on is 
> fairly rare, results will pop up quickly, like in around 3 to 5 seconds. 
> Words that don't exist at all in any email returns nothing almost instantly.
>
> But words that appear in several hundred emails are the ones that are take a 
> much longer time. 

Can this be reduced to: getting more search results takes longer?




Re: What kind of search response time are you setting with solr full text search?

2021-08-24 Thread Steve Dondley




THE PROBLEM: When I do a full text search through all my inbox and all
subfolders on a single word, search results are returned in about 10
to 15 seconds. This is better than the 40 seconds or so I'm getting
when I turn off the fts and fts_solr plugins but still a little
disappointing.


I did some experimenting. I noticed that if the word I'm searching on is 
fairly rare, results will pop up quickly, like in around 3 to 5 seconds. 
Words that don't exist at all in any email returns nothing almost 
instantly.


But words that appear in several hundred emails are the ones that are 
take a much longer time.


Not sure if this just might be a slow email client or due to Dovecot 
itself or if maybe 1 GB of ram isn't enough and my machine is 
underpowered.


What kind of search response time are you setting with solr full text search?

2021-08-24 Thread Steve Dondley
MY SETUP: I have apache solr full text search enabled with dovecot. I 
have an inbox with about 40 subfolders. I'm using the roundcube 
web-based mail client. The find command is showing 15823 email files and 
apache solr reports the same number. I'm running a dedicated mail server 
with a 1 GB of ram. The solr machine is running on a separate machine 
with 4 GB ram.


THE PROBLEM: When I do a full text search through all my inbox and all 
subfolders on a single word, search results are returned in about 10 to 
15 seconds. This is better than the 40 seconds or so I'm getting when I 
turn off the fts and fts_solr plugins but still a little disappointing.


WHAT I'D LIKE: Now, I don't expect instant search results like gmail, 
but getting the search results to display in less than 5 seconds would 
be a lot less painful that 10 to 15 seconds.


WHAT I'VE TRIED: I have reindexed the emails on solr to try to speed 
things upit up.


ANY ADVICE? Curious to know what response times others are seeing and if 
there is anything else I might try to speed things up. Maybe the number 
of subfolders is slowing things down?