Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-29 Thread kfx 
Okay I'm progressing here. If I copy an email within my MUA to another 
folder and then make another copy back to my Inbox, the new copied email 
is correctly and immediately indexed ! If I delete an email, the search 
result is also correct and immediate.


Is it possible that my problem come from Dovecot LDA ?

In 10-mail.conf I have:
# cat dovecot/conf.d/10-mail.conf| grep -v -e "^$" -e "^#"  -e "^  #"
mail_location = maildir:/var/vmail/%Ln
namespace {
  type = private
  location = maildir:/var/vmail/%Ln
  inbox = yes
mailbox Trash {
  auto = subscribe
  special_use = \Trash
}
mailbox Drafts {
  auto = subscribe
  special_use = \Drafts
}
mailbox Sent {
  auto = subscribe # autocreate, autosubscribe
  special_use = \Sent
}
mailbox "Sent Messages" {
  special_use = \Sent
}
}
mail_uid=vmail
mail_gid=vmail
auth_socket_path = /var/run/dovecot/auth-userdb
mail_plugins = $mail_plugins quota stats fts fts_solr

And in 15-lda.conf:
# cat dovecot/conf.d/15-lda.conf| grep -v -e "^$" -e "^#"  -e "^  #"
postmaster_address = postmaster@mydomain
hostname = mydomain
recipient_delimiter = +
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
protocol lda {
  mail_plugins = $mail_plugins sieve virtual
}

And the lda portion of "doveconf -n" become:
protocol lda {
  mail_plugins = " quota stats fts fts_solr sieve virtual"
}



 Here is my postfix relevant conf:

## MAIN.CF ##
# postconf -n | grep dovecot
dovecot_destination_recipient_limit = 1
smtpd_sasl_type = dovecot
virtual_transport = dovecot
#

## MASTER.CF ##
dovecot unix-   n   n   -   -  pipe
  flags=DRhu user=vmail:vmail 
argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -a ${recipient} 
-d ${user}@${nexthop}

#



On 28/05/2018 13:30, kfx wrote:

On 28/05/2018 13:23, kfx wrote:

On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx > wrote:



Especially what is in the "fts" header vs. "next uid" header? Does 
the UID in "fts" header keep changing every time you save a new 
mail? I suppose it will.


Diff between 2 emails:
next uid = 30104    |    next uid = 30105
last_indexed_uid = 30103    |    last_indexed_uid = 30104


So Dovecot thinks it has indexed everything.

You could also monitor (e.g. tcpdump/wireshark) the network traffic 
between Dovecot <-> Solr what happens when a new mail arrives. I 
suspect Dovecot sends it to Solr, which for whatever reason just 
ignores the update.


### TCPDUMP 
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml

37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernamename="body">Search Pattern: Kai8oovi

..




And Dovecot sends the mail.


# SOLR'S RESPONSE ###




 0
 0




And Solr receives it. Your tcpdump doesn't show softCommit="true" waitSearcher="true"/> being sent though. Do you see 
it being sent anywhere? 


Yes:

### TCPDUMP ###
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml


### /TCPDUMP ###



Does it make the mails visible if you run it yourself? Or if you run 
hard commit? :


curl http://:8983/solr/update?commit=true



# curl http://127.0.0.1:8983/solr/update?commit=true



 [0/0]


Error 404 Not Found

HTTP ERROR 404
Problem accessing /solr/update. Reason:
    Not Found



# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true




   0
   0



# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4 
results)


In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot

I can see:

lastModified: less than a minute ago
version:1428772
numDocs:6353615
maxDoc:6356213
deletedDocs:2598


So it IS indexing :(
Just below I see the "optimized:" parameter followed by an icon which 
seems saying that is NOT "optimized". Don't know if it's relevant.

This is driving me crazy :(

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-28 Thread kfx 

On 28/05/2018 13:23, kfx wrote:

On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx > wrote:



Especially what is in the "fts" header vs. "next uid" header? Does 
the UID in "fts" header keep changing every time you save a new 
mail? I suppose it will.


Diff between 2 emails:
next uid = 30104    |    next uid = 30105
last_indexed_uid = 30103    |    last_indexed_uid = 30104


So Dovecot thinks it has indexed everything.

You could also monitor (e.g. tcpdump/wireshark) the network traffic 
between Dovecot <-> Solr what happens when a new mail arrives. I 
suspect Dovecot sends it to Solr, which for whatever reason just 
ignores the update.


### TCPDUMP 
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml

37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernamename="body">Search Pattern: Kai8oovi

..




And Dovecot sends the mail.


# SOLR'S RESPONSE ###




 0
 0




And Solr receives it. Your tcpdump doesn't show softCommit="true" waitSearcher="true"/> being sent though. Do you see 
it being sent anywhere? 


Yes:

### TCPDUMP ###
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml


### /TCPDUMP ###



Does it make the mails visible if you run it yourself? Or if you run 
hard commit? :


curl http://:8983/solr/update?commit=true



# curl http://127.0.0.1:8983/solr/update?commit=true



     [0/0]


Error 404 Not Found

HTTP ERROR 404
Problem accessing /solr/update. Reason:
    Not Found



# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true




   0
   0



# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4 
results)


In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot

I can see:

lastModified: less than a minute ago
version:1428772
numDocs:6353615
maxDoc:6356213
deletedDocs:2598


So it IS indexing :(
Just below I see the "optimized:" parameter followed by an icon which 
seems saying that is NOT "optimized". Don't know if it's relevant.

This is driving me crazy :(

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-28 Thread kfx 

On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx > wrote:



Especially what is in the "fts" header vs. "next uid" header? Does 
the UID in "fts" header keep changing every time you save a new mail? 
I suppose it will.


Diff between 2 emails:
next uid = 30104    |    next uid = 30105
last_indexed_uid = 30103    |    last_indexed_uid = 30104


So Dovecot thinks it has indexed everything.

You could also monitor (e.g. tcpdump/wireshark) the network traffic 
between Dovecot <-> Solr what happens when a new mail arrives. I 
suspect Dovecot sends it to Solr, which for whatever reason just 
ignores the update.


### TCPDUMP 
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml

37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernameSearch 
Pattern: Kai8oovi

..




And Dovecot sends the mail.


# SOLR'S RESPONSE ###




 0
 0




And Solr receives it. Your tcpdump doesn't show softCommit="true" waitSearcher="true"/> being sent though. Do you see it 
being sent anywhere? 


Yes:

### TCPDUMP ###
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml


### /TCPDUMP ###



Does it make the mails visible if you run it 
yourself? Or if you run hard commit? :


curl http://:8983/solr/update?commit=true



# curl http://127.0.0.1:8983/solr/update?commit=true
 




[0/0]


Error 404 Not Found

HTTP ERROR 404
Problem accessing /solr/update. Reason:
Not Found



# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true




  0
  0



# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4 
results)

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-28 Thread Timo Sirainen 
On 28 May 2018, at 13.28, kfx  wrote:
> 
> 
>> Especially what is in the "fts" header vs. "next uid" header? Does the UID 
>> in "fts" header keep changing every time you save a new mail? I suppose it 
>> will. 
> 
> Diff between 2 emails:
> next uid = 30104|next uid = 30105
> last_indexed_uid = 30103|last_indexed_uid = 30104

So Dovecot thinks it has indexed everything.

>> You could also monitor (e.g. tcpdump/wireshark) the network traffic between 
>> Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot 
>> sends it to Solr, which for whatever reason just ignores the update.
> 
> ### TCPDUMP 
> POST /solr/dovecot/update HTTP/1.1
> Host: localhost:8983
> Date: Mon, 28 May 2018 10:18:05 GMT
> Transfer-Encoding: chunked
> Connection: Keep-Alive
> Content-Type: text/xml
> 
> 37581 name="box">e0c58a309323515311083ea484a8 name="user">username name="id">37581/e0c58a309323515311083ea484a8/username name="body">Search Pattern: Kai8oovi
..
> 

And Dovecot sends the mail.

> # SOLR'S RESPONSE ###
> 
> 
> 
> 
>  0
>  0
> 
> 

And Solr receives it. Your tcpdump doesn't show  being sent though. Do you see it being sent anywhere? 
Does it make the mails visible if you run it yourself? Or if you run hard 
commit? :

curl http://:8983/solr/update?commit=true

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-28 Thread kfx 

On 24/05/2018 15:33, Timo Sirainen wrote:

On 21 May 2018, at 14.11, kada...@gmail.com wrote:


Le 21/05/2018 à 12:38, Aki Tuomi a écrit :

can you try turning on pluign { fts_enforced = yes } and repeat your test?


Same (wrong) result:
1. Send an email with "too6Ouka" in the body

2. Search against "too6Ouka":
# doveadm search -u username mailbox INBOX body too6Ouka
--> No result

3. Force re-index:
# doveadm fts rescan -u username

4. Search again against "too6Ouka":
# doveadm search -u username mailbox INBOX body too6Ouka
--> e09cce0283e8695ab76002deed92 29055

Don't know if relevant, but on a side note, if I send a second message
with "too6Ouka" in the body, followed by:
# doveadm -v index -u username Inbox
--> doveadm(username): Info: INBOX: Cache is already up to date
And a search against the pattern immediately return only one result
instead of two.


So it looks like after "doveadm fts rescan" you can do initial indexing 
successfully. Afterwards the index isn't updated at all, unless you rebuild it entirely. 
fts_autoindex=yes appears to work as expected, because after fts rescan you did a search 
which automatically updated the index. So this doesn't have anything to do with 
fts_autoindex, just that your index updates don't seem to work at all.

What does it show in the header if you do:
doveadm dump /var/vmail/username/dovecot.index


Detected file type: index
-- INDEX: /var/vmail/username/dovecot.index
version .. = 7.3
base header size . = 120
header size .. = 640
record size .. = 16
compat flags . = 1
index id . = 1527164208 (2018-05-24 14:16:48)
flags  = 0
uid validity . = 1516890243 (2018-01-25 15:24:03)
next uid . = 30104
messages count ... = 17068
seen messages count .. = 17011
deleted messages count ... = 0
first recent uid . = 30104
first unseen uid lowwater  = 393
first deleted uid lowwater = 30087
log file seq . = 14
log file tail offset . = 28144
log file head offset . = 28144
log2 rotate time . = 1527487933 (2018-05-28 08:12:13)
last temp file scan .. = 0 (1970-01-01 01:00:00)
day stamp  = 1527458400 (2018-05-28 00:00:00)
day first uid[0] . = 30037
day first uid[1] . = 29921
day first uid[2] . = 29814
day first uid[3] . = 29669
day first uid[4] . = 2
day first uid[5] . = 0
day first uid[6] . = 0
day first uid[7] . = 0
-- Extension 0 --
name  = maildir
hdr_size  = 36
reset_id  = 0
record_offset = 0
record_size . = 0
record_align  = 0
header
 - new_check_time  = 2018-05-28 11:48:14
 - new_mtime . = 2018-05-28 11:47:59
 - new_mtime_nsecs ... = 129602175
 - cur_check_time  = 2018-05-28 11:48:14
 - cur_mtime . = 2018-05-28 11:48:01
 - cur_mtime_nsecs = 277650283
 - uidlist_mtime . = 2018-05-28 11:47:58
 - uidlist_mtime_nsecs = 521588557
 - uidlist_size .. = 1334026
-- Extension 1 --
name  = keywords
hdr_size  = 292
reset_id  = 0
record_offset = 5
record_size . = 3
record_align  = 1
-- Extension 2 --
name  = hdr-vsize
hdr_size  = 16
reset_id  = 0
record_offset = 0
record_size . = 0
record_align  = 8
header
 - highest uid . = 0
 - message count = 0
 - vsize ... = 0
-- Extension 3 --
name  = cache
hdr_size  = 0
reset_id  = 1527164233
record_offset = 8
record_size . = 4
record_align  = 4
-- Extension 4 --
name  = vsize
hdr_size  = 0
reset_id  = 0
record_offset = 12
record_size . = 4
record_align  = 4
-- Extension 5 --
name  = fts
hdr_size  = 12
reset_id  = 0
record_offset = 0
record_size . = 0
record_align  = 0
header
 - last_indexed_uid . = 30103
 - settings_checksum  = 0
-- Keywords --
  0 = unknown-1
  1 = unknown-2
  2 = unknown-5
  3 = unknown-8
  4 = unknown-6
  5 = unknown-3
  6 = unknown-7
  7 = unknown-0
  8 = NonJunk
  9 = $Forwarded
 10 = $label1
 11 = $label2
 12 = Junk
 13 = $label3
 14 = $label4




Especially what is in the "fts" header vs. "next uid" header? Does the UID in "fts" header keep changing every time you save a new mail? I suppose it will. 


Diff between 2 emails:
next uid = 30104|next uid = 30105
last_indexed_uid = 30103|last_indexed_uid = 30104



You could also monitor (e.g. tcpdump/wireshark) the network traffic between Dovecot 
<-> Solr what happens when a new mail arrives. I suspect Dovecot sends it to 
Solr, which for whatever reason just ignores the update.


### TCPDUMP 
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml

37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernamename="body">Search Pattern:

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-24 Thread Timo Sirainen 
On 21 May 2018, at 14.11, kada...@gmail.com wrote:
> 
> Le 21/05/2018 à 12:38, Aki Tuomi a écrit :
>> can you try turning on pluign { fts_enforced = yes } and repeat your test?
> 
> Same (wrong) result:
> 1. Send an email with "too6Ouka" in the body
> 
> 2. Search against "too6Ouka":
> # doveadm search -u username mailbox INBOX body too6Ouka
> --> No result
> 
> 3. Force re-index:
> # doveadm fts rescan -u username
> 
> 4. Search again against "too6Ouka":
> # doveadm search -u username mailbox INBOX body too6Ouka
> --> e09cce0283e8695ab76002deed92 29055
> 
> Don't know if relevant, but on a side note, if I send a second message
> with "too6Ouka" in the body, followed by:
> # doveadm -v index -u username Inbox
> --> doveadm(username): Info: INBOX: Cache is already up to date
> And a search against the pattern immediately return only one result
> instead of two.

So it looks like after "doveadm fts rescan" you can do initial indexing 
successfully. Afterwards the index isn't updated at all, unless you rebuild it 
entirely. fts_autoindex=yes appears to work as expected, because after fts 
rescan you did a search which automatically updated the index. So this doesn't 
have anything to do with fts_autoindex, just that your index updates don't seem 
to work at all.

What does it show in the header if you do:
doveadm dump /var/vmail/username/dovecot.index

Especially what is in the "fts" header vs. "next uid" header? Does the UID in 
"fts" header keep changing every time you save a new mail? I suppose it will. 
You could also monitor (e.g. tcpdump/wireshark) the network traffic between 
Dovecot <-> Solr what happens when a new mail arrives. I suspect Dovecot sends 
it to Solr, which for whatever reason just ignores the update.

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-24 Thread kfx 
After seeing that dovecot 2.2.36 was out and that:

> - fts: Fixes to indexing mails via virtual mailboxes.
> - fts: If mails contained NUL characters, the text around it wasn't
>indexed.

I've upgraded but the problem still persist :(


On 21/05/2018 13:11, kada...@gmail.com wrote:
> Le 21/05/2018 à 12:38, Aki Tuomi a écrit :
>> can you try turning on pluign { fts_enforced = yes } and repeat your test?
> 
> Same (wrong) result:
> 1. Send an email with "too6Ouka" in the body
> 
> 2. Search against "too6Ouka":
> # doveadm search -u username mailbox INBOX body too6Ouka
> --> No result
> 
> 3. Force re-index:
> # doveadm fts rescan -u username
> 
> 4. Search again against "too6Ouka":
> # doveadm search -u username mailbox INBOX body too6Ouka
> --> e09cce0283e8695ab76002deed92 29055
> 
> Don't know if relevant, but on a side note, if I send a second message
> with "too6Ouka" in the body, followed by:
> # doveadm -v index -u username Inbox
> --> doveadm(username): Info: INBOX: Cache is already up to date
> And a search against the pattern immediately return only one result
> instead of two.
> 
> # doveconf -n
> # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.22 (22940fb7)
> # OS: Linux 3.10.0-862.2.3.el7.x86_64 x86_64 CentOS Linux release
> 7.5.1804 (Core)  ext4
> # Hostname: mailserver.domain
> auth_master_user_separator = *
> auth_socket_path = /var/run/dovecot/auth-userdb
> auth_username_chars =
> abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
> auth_verbose = yes
> base_dir = /var/run/dovecot/
> debug_log_path = /var/log/dovecot-debug.log
> default_client_limit = 5120
> default_process_limit = 1024
> default_vsz_limit = 2 G
> hostname = domain
> info_log_path = /var/log/dovecot.log
> lda_mailbox_autocreate = yes
> lda_mailbox_autosubscribe = yes
> listen = *
> log_path = /var/log/dovecot-error.log
> login_greeting = Welcome to dovecot on mydomain
> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
> mail_gid = vmail
> mail_location = maildir:/var/vmail/%Ln
> mail_plugins = " quota stats fts fts_solr"
> mail_uid = vmail
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate mime foreverypart extracttext
> namespace {
>   inbox = yes
>   location = maildir:/var/vmail/%Ln
>   mailbox Drafts {
>     auto = subscribe
>     special_use = \Drafts
>   }
>   mailbox Sent {
>     auto = subscribe
>     special_use = \Sent
>   }
>   mailbox "Sent Messages" {
>     special_use = \Sent
>   }
>   mailbox Trash {
>     auto = subscribe
>     special_use = \Trash
>   }
>   prefix =
>   type = private
> }
> passdb {
>   args = /usr/local/etc/dovecot/deny-users
>   deny = yes
>   driver = passwd-file
> }
> passdb {
>   args = /usr/local/etc/dovecot/master-users
>   driver = passwd-file
>   master = yes
> }
> passdb {
>   args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
>   driver = ldap
> }
> plugin {
>   fts = solr
>   fts_autoindex = yes
>   fts_enforced = yes
>   fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search
>   quota = maildir:User quota
>   quota_rule = *:storage=20GB
>   quota_rule2 = Trash:storage=+200M
>   quota_warning = storage=95%% quota-warning 95 %u
>   quota_warning2 = storage=80%% quota-warning 80 %u
>   sieve = /var/sieve-scripts/%Ln.sieve
>   sieve_dir = /var/vmail/%Ln/home/sieve
>   stats_command_min_time = 1 mins
>   stats_domain_min_time = 12 hours
>   stats_ip_min_time = 12 hours
>   stats_memory_limit = 16 M
>   stats_refresh = 30 secs
>   stats_session_min_time = 15 mins
>   stats_track_cmds = yes
>   stats_user_min_time = 1 hours
> }
> postmaster_address = postmaster@domain
> protocols = imap pop3 sieve sieve
> service auth-worker {
>   user = root
> }
> service auth {
>   unix_listener /var/spool/postfix/private/auth {
>     mode = 0777
>     user = vmail
>   }
> }
> service imap-login {
>   inet_listener imaps {
>     port = 993
>     ssl = yes
>   }
>   process_min_avail = 10
> }
> service imap {
>   process_limit = 2128
> }
> service managesieve-login {
>   inet_listener sieve {
>     port = 4190
>   }
> }
> service pop3-login {
>   inet_listener pop3s {
>     port = 995
>     ssl = yes
>   }
> }
> service quota-warning {
>   executable = script /usr/local/bin/quota-warning.sh
>   user = nobody
> }
> service stats {
>   fifo_listener stats-mail {
>     mode = 0600
>     user = vmail
>   }
> }
> ssl_cert =  ssl_key =  # hidden, use -P to show it
> userdb {
>   args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext
>   driver = ldap
>   override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home
> }
> verbose_proctitle = yes
> protocol lda {
>   mail_plugins = " quota stats fts fts_solr sieve virtual"
> }
> protocol imap {
>   mail_max_userip_connections =

Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-21 Thread kadafax 
Le 21/05/2018 à 12:38, Aki Tuomi a écrit :
> can you try turning on pluign { fts_enforced = yes } and repeat your test?

Same (wrong) result:
1. Send an email with "too6Ouka" in the body

2. Search against "too6Ouka":
# doveadm search -u username mailbox INBOX body too6Ouka
--> No result

3. Force re-index:
# doveadm fts rescan -u username

4. Search again against "too6Ouka":
# doveadm search -u username mailbox INBOX body too6Ouka
--> e09cce0283e8695ab76002deed92 29055

Don't know if relevant, but on a side note, if I send a second message
with "too6Ouka" in the body, followed by:
# doveadm -v index -u username Inbox
--> doveadm(username): Info: INBOX: Cache is already up to date
And a search against the pattern immediately return only one result
instead of two.

# doveconf -n
# 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.22 (22940fb7)
# OS: Linux 3.10.0-862.2.3.el7.x86_64 x86_64 CentOS Linux release
7.5.1804 (Core)  ext4
# Hostname: mailserver.domain
auth_master_user_separator = *
auth_socket_path = /var/run/dovecot/auth-userdb
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
auth_verbose = yes
base_dir = /var/run/dovecot/
debug_log_path = /var/log/dovecot-debug.log
default_client_limit = 5120
default_process_limit = 1024
default_vsz_limit = 2 G
hostname = domain
info_log_path = /var/log/dovecot.log
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
listen = *
log_path = /var/log/dovecot-error.log
login_greeting = Welcome to dovecot on mydomain
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_gid = vmail
mail_location = maildir:/var/vmail/%Ln
mail_plugins = " quota stats fts fts_solr"
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace {
  inbox = yes
  location = maildir:/var/vmail/%Ln
  mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix =
  type = private
}
passdb {
  args = /usr/local/etc/dovecot/deny-users
  deny = yes
  driver = passwd-file
}
passdb {
  args = /usr/local/etc/dovecot/master-users
  driver = passwd-file
  master = yes
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  fts = solr
  fts_autoindex = yes
  fts_enforced = yes
  fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search
  quota = maildir:User quota
  quota_rule = *:storage=20GB
  quota_rule2 = Trash:storage=+200M
  quota_warning = storage=95%% quota-warning 95 %u
  quota_warning2 = storage=80%% quota-warning 80 %u
  sieve = /var/sieve-scripts/%Ln.sieve
  sieve_dir = /var/vmail/%Ln/home/sieve
  stats_command_min_time = 1 mins
  stats_domain_min_time = 12 hours
  stats_ip_min_time = 12 hours
  stats_memory_limit = 16 M
  stats_refresh = 30 secs
  stats_session_min_time = 15 mins
  stats_track_cmds = yes
  stats_user_min_time = 1 hours
}
postmaster_address = postmaster@domain
protocols = imap pop3 sieve sieve
service auth-worker {
  user = root
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0777
    user = vmail
  }
}
service imap-login {
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 10
}
service imap {
  process_limit = 2128
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
service pop3-login {
  inet_listener pop3s {
    port = 995
    ssl = yes
  }
}
service quota-warning {
  executable = script /usr/local/bin/quota-warning.sh
  user = nobody
}
service stats {
  fifo_listener stats-mail {
    mode = 0600
    user = vmail
  }
}
ssl_cert = 
> Aki
>
>
> On 21.05.2018 13:31, kada...@gmail.com wrote:
>> Le 21/05/2018 à 11:46, Aki Tuomi a écrit :
>>> That message is coming from the temporary raw user, not from the actual
>>> user.
>>>
>>> How did you verify that the FTS indexes are not created or updated
>>> automatically?
>> I send an email containing a specific strings ('thahB8ea' in that case)
>> and search agains it:
>> # doveadm search -u username mailbox INBOX body thahB8ea
>> --> No result
>> (same with integrated search on roundcube)
>>
>> Then I force a rescan, followed by the same search:
>> # doveadm  fts rescan -u username
>> # doveadm search -u username mailbox INBOX body thahB8ea
>> (Long waiting time)
>> --> e09cce0283e8695ab76002deed92 29048
>>
>> So I conclude that autoindexing is not working.
>>
>> Thanks,
>> kfx
>>

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-21 Thread Aki Tuomi 
can you try turning on pluign { fts_enforced = yes } and repeat your test?

Aki


On 21.05.2018 13:31, kada...@gmail.com wrote:
> Le 21/05/2018 à 11:46, Aki Tuomi a écrit :
>> That message is coming from the temporary raw user, not from the actual
>> user.
>>
>> How did you verify that the FTS indexes are not created or updated
>> automatically?
> I send an email containing a specific strings ('thahB8ea' in that case)
> and search agains it:
> # doveadm search -u username mailbox INBOX body thahB8ea
> --> No result
> (same with integrated search on roundcube)
>
> Then I force a rescan, followed by the same search:
> # doveadm  fts rescan -u username
> # doveadm search -u username mailbox INBOX body thahB8ea
> (Long waiting time)
> --> e09cce0283e8695ab76002deed92 29048
>
> So I conclude that autoindexing is not working.
>
> Thanks,
> kfx
>
>> Aki
>>
>>
>> On 04.05.2018 12:47, kada...@gmail.com wrote:
>>>  Bump. Can someone from dovecot dev team please respond something to my
>>> problem ? I've provided all the possible information, and if not please
>>> tell me.
>>> Or do you only provide support for your own FTS solution bundled with
>>> the paid version of dovecot-pro ? If so, you should clearly state it in
>>> the wiki.
>>>
>>> Thanks
>>>
>>> Le 19/04/2018 à 10:30, kada...@gmail.com a écrit :
 I think I'm progressing: if I create a brand new user, send her an
 email, it will not be indexed right away. I need to doveadm rescan /
 index the Inbox before.
 At the end of this email, a dovecot's debug log snipset (grep'ed on this
 user). In it I can read: "Debug: fts: Indexes disabled for namespace ''
 Can this be a problem ?

 Thanks

 doveconf -n output:
 # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf
 # Pigeonhole version 0.4.22 (22940fb7)
 # OS: Linux 3.10.0-693.17.1.el7.x86_64 x86_64 CentOS Linux release
 7.4.1708 (Core)  ext4
 # Hostname: mailServer.myDomain
 auth_master_user_separator = *
 auth_socket_path = /var/run/dovecot/auth-userdb
 auth_username_chars =
 abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
 auth_verbose = yes
 base_dir = /var/run/dovecot/
 debug_log_path = /var/log/dovecot-debug.log
 default_client_limit = 5120
 default_process_limit = 1024
 default_vsz_limit = 2 G
 hostname = myDomain
 info_log_path = /var/log/dovecot.log
 lda_mailbox_autocreate = yes
 lda_mailbox_autosubscribe = yes
 listen = *
 log_path = /var/log/dovecot-error.log
 login_greeting = Welcome
 login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
 mail_gid = vmail
 mail_location = maildir:/var/vmail/%Ln
 mail_plugins = " quota stats fts fts_solr"
 mail_uid = vmail
 managesieve_notify_capability = mailto
 managesieve_sieve_capability = fileinto reject envelope
 encoded-character vacation subaddress comparator-i;ascii-numeric
 relational regex imap4flags copy include variables body enotify
 environment mailbox date index ihave duplicate mime foreverypart 
 extracttext
 namespace {
   inbox = yes
   location = maildir:/var/vmail/%Ln
   mailbox Drafts {
     auto = subscribe
     special_use = \Drafts
   }
   mailbox Sent {
     auto = subscribe
     special_use = \Sent
   }
   mailbox "Sent Messages" {
     special_use = \Sent
   }
   mailbox Trash {
     auto = subscribe
     special_use = \Trash
   }
   prefix =
   type = private
 }
 passdb {
   args = /usr/local/etc/dovecot/deny-users
   deny = yes
   driver = passwd-file
 }
 passdb {
   args = /usr/local/etc/dovecot/master-users
   driver = passwd-file
   master = yes
 }
 passdb {
   args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
   driver = ldap
 }
 plugin {
   fts = solr
   fts_autoindex = yes
   fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search
   quota = maildir:User quota
   quota_rule = *:storage=20GB
   quota_rule2 = Trash:storage=+200M
   quota_warning = storage=95%% quota-warning 95 %u
   quota_warning2 = storage=80%% quota-warning 80 %u
   sieve = /var/sieve-scripts/%Ln.sieve
   sieve_dir = /var/vmail/%Ln/home/sieve
   stats_command_min_time = 1 mins
   stats_domain_min_time = 12 hours
   stats_ip_min_time = 12 hours
   stats_memory_limit = 16 M
   stats_refresh = 30 secs
   stats_session_min_time = 15 mins
   stats_track_cmds = yes
   stats_user_min_time = 1 hours
 }
 postmaster_address = postmaster@myDomain
 protocols = imap pop3 sieve sieve
 service auth-worker {
   user = root
 }
 service auth {
   unix_listener /var/spool/postfix/private/auth {
     mode = 0777
     user = vmail
   }
 }
 service imap-login {
   inet_listener imaps {
  

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-21 Thread kadafax 
Le 21/05/2018 à 11:46, Aki Tuomi a écrit :
> That message is coming from the temporary raw user, not from the actual
> user.
>
> How did you verify that the FTS indexes are not created or updated
> automatically?

I send an email containing a specific strings ('thahB8ea' in that case)
and search agains it:
# doveadm search -u username mailbox INBOX body thahB8ea
--> No result
(same with integrated search on roundcube)

Then I force a rescan, followed by the same search:
# doveadm  fts rescan -u username
# doveadm search -u username mailbox INBOX body thahB8ea
(Long waiting time)
--> e09cce0283e8695ab76002deed92 29048

So I conclude that autoindexing is not working.

Thanks,
kfx

>
> Aki
>
>
> On 04.05.2018 12:47, kada...@gmail.com wrote:
>>  Bump. Can someone from dovecot dev team please respond something to my
>> problem ? I've provided all the possible information, and if not please
>> tell me.
>> Or do you only provide support for your own FTS solution bundled with
>> the paid version of dovecot-pro ? If so, you should clearly state it in
>> the wiki.
>>
>> Thanks
>>
>> Le 19/04/2018 à 10:30, kada...@gmail.com a écrit :
>>> I think I'm progressing: if I create a brand new user, send her an
>>> email, it will not be indexed right away. I need to doveadm rescan /
>>> index the Inbox before.
>>> At the end of this email, a dovecot's debug log snipset (grep'ed on this
>>> user). In it I can read: "Debug: fts: Indexes disabled for namespace ''
>>> Can this be a problem ?
>>>
>>> Thanks
>>>
>>> doveconf -n output:
>>> # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf
>>> # Pigeonhole version 0.4.22 (22940fb7)
>>> # OS: Linux 3.10.0-693.17.1.el7.x86_64 x86_64 CentOS Linux release
>>> 7.4.1708 (Core)  ext4
>>> # Hostname: mailServer.myDomain
>>> auth_master_user_separator = *
>>> auth_socket_path = /var/run/dovecot/auth-userdb
>>> auth_username_chars =
>>> abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
>>> auth_verbose = yes
>>> base_dir = /var/run/dovecot/
>>> debug_log_path = /var/log/dovecot-debug.log
>>> default_client_limit = 5120
>>> default_process_limit = 1024
>>> default_vsz_limit = 2 G
>>> hostname = myDomain
>>> info_log_path = /var/log/dovecot.log
>>> lda_mailbox_autocreate = yes
>>> lda_mailbox_autosubscribe = yes
>>> listen = *
>>> log_path = /var/log/dovecot-error.log
>>> login_greeting = Welcome
>>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
>>> mail_gid = vmail
>>> mail_location = maildir:/var/vmail/%Ln
>>> mail_plugins = " quota stats fts fts_solr"
>>> mail_uid = vmail
>>> managesieve_notify_capability = mailto
>>> managesieve_sieve_capability = fileinto reject envelope
>>> encoded-character vacation subaddress comparator-i;ascii-numeric
>>> relational regex imap4flags copy include variables body enotify
>>> environment mailbox date index ihave duplicate mime foreverypart extracttext
>>> namespace {
>>>   inbox = yes
>>>   location = maildir:/var/vmail/%Ln
>>>   mailbox Drafts {
>>>     auto = subscribe
>>>     special_use = \Drafts
>>>   }
>>>   mailbox Sent {
>>>     auto = subscribe
>>>     special_use = \Sent
>>>   }
>>>   mailbox "Sent Messages" {
>>>     special_use = \Sent
>>>   }
>>>   mailbox Trash {
>>>     auto = subscribe
>>>     special_use = \Trash
>>>   }
>>>   prefix =
>>>   type = private
>>> }
>>> passdb {
>>>   args = /usr/local/etc/dovecot/deny-users
>>>   deny = yes
>>>   driver = passwd-file
>>> }
>>> passdb {
>>>   args = /usr/local/etc/dovecot/master-users
>>>   driver = passwd-file
>>>   master = yes
>>> }
>>> passdb {
>>>   args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
>>>   driver = ldap
>>> }
>>> plugin {
>>>   fts = solr
>>>   fts_autoindex = yes
>>>   fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search
>>>   quota = maildir:User quota
>>>   quota_rule = *:storage=20GB
>>>   quota_rule2 = Trash:storage=+200M
>>>   quota_warning = storage=95%% quota-warning 95 %u
>>>   quota_warning2 = storage=80%% quota-warning 80 %u
>>>   sieve = /var/sieve-scripts/%Ln.sieve
>>>   sieve_dir = /var/vmail/%Ln/home/sieve
>>>   stats_command_min_time = 1 mins
>>>   stats_domain_min_time = 12 hours
>>>   stats_ip_min_time = 12 hours
>>>   stats_memory_limit = 16 M
>>>   stats_refresh = 30 secs
>>>   stats_session_min_time = 15 mins
>>>   stats_track_cmds = yes
>>>   stats_user_min_time = 1 hours
>>> }
>>> postmaster_address = postmaster@myDomain
>>> protocols = imap pop3 sieve sieve
>>> service auth-worker {
>>>   user = root
>>> }
>>> service auth {
>>>   unix_listener /var/spool/postfix/private/auth {
>>>     mode = 0777
>>>     user = vmail
>>>   }
>>> }
>>> service imap-login {
>>>   inet_listener imaps {
>>>     port = 993
>>>     ssl = yes
>>>   }
>>>   process_min_avail = 10
>>> }
>>> service imap {
>>>   process_limit = 2128
>>> }
>>> service managesieve-login {
>>>   inet_listener sieve {
>>>     port = 4190
>>>   }
>>> }
>>> service pop3-login {
>>>   inet_listener pop3s {
>>>     port = 995

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-21 Thread Aki Tuomi 
That message is coming from the temporary raw user, not from the actual
user.

How did you verify that the FTS indexes are not created or updated
automatically?

Aki


On 04.05.2018 12:47, kada...@gmail.com wrote:
>  Bump. Can someone from dovecot dev team please respond something to my
> problem ? I've provided all the possible information, and if not please
> tell me.
> Or do you only provide support for your own FTS solution bundled with
> the paid version of dovecot-pro ? If so, you should clearly state it in
> the wiki.
>
> Thanks
>
> Le 19/04/2018 à 10:30, kada...@gmail.com a écrit :
>> I think I'm progressing: if I create a brand new user, send her an
>> email, it will not be indexed right away. I need to doveadm rescan /
>> index the Inbox before.
>> At the end of this email, a dovecot's debug log snipset (grep'ed on this
>> user). In it I can read: "Debug: fts: Indexes disabled for namespace ''
>> Can this be a problem ?
>>
>> Thanks
>>
>> doveconf -n output:
>> # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf
>> # Pigeonhole version 0.4.22 (22940fb7)
>> # OS: Linux 3.10.0-693.17.1.el7.x86_64 x86_64 CentOS Linux release
>> 7.4.1708 (Core)  ext4
>> # Hostname: mailServer.myDomain
>> auth_master_user_separator = *
>> auth_socket_path = /var/run/dovecot/auth-userdb
>> auth_username_chars =
>> abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
>> auth_verbose = yes
>> base_dir = /var/run/dovecot/
>> debug_log_path = /var/log/dovecot-debug.log
>> default_client_limit = 5120
>> default_process_limit = 1024
>> default_vsz_limit = 2 G
>> hostname = myDomain
>> info_log_path = /var/log/dovecot.log
>> lda_mailbox_autocreate = yes
>> lda_mailbox_autosubscribe = yes
>> listen = *
>> log_path = /var/log/dovecot-error.log
>> login_greeting = Welcome
>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
>> mail_gid = vmail
>> mail_location = maildir:/var/vmail/%Ln
>> mail_plugins = " quota stats fts fts_solr"
>> mail_uid = vmail
>> managesieve_notify_capability = mailto
>> managesieve_sieve_capability = fileinto reject envelope
>> encoded-character vacation subaddress comparator-i;ascii-numeric
>> relational regex imap4flags copy include variables body enotify
>> environment mailbox date index ihave duplicate mime foreverypart extracttext
>> namespace {
>>   inbox = yes
>>   location = maildir:/var/vmail/%Ln
>>   mailbox Drafts {
>>     auto = subscribe
>>     special_use = \Drafts
>>   }
>>   mailbox Sent {
>>     auto = subscribe
>>     special_use = \Sent
>>   }
>>   mailbox "Sent Messages" {
>>     special_use = \Sent
>>   }
>>   mailbox Trash {
>>     auto = subscribe
>>     special_use = \Trash
>>   }
>>   prefix =
>>   type = private
>> }
>> passdb {
>>   args = /usr/local/etc/dovecot/deny-users
>>   deny = yes
>>   driver = passwd-file
>> }
>> passdb {
>>   args = /usr/local/etc/dovecot/master-users
>>   driver = passwd-file
>>   master = yes
>> }
>> passdb {
>>   args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
>>   driver = ldap
>> }
>> plugin {
>>   fts = solr
>>   fts_autoindex = yes
>>   fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search
>>   quota = maildir:User quota
>>   quota_rule = *:storage=20GB
>>   quota_rule2 = Trash:storage=+200M
>>   quota_warning = storage=95%% quota-warning 95 %u
>>   quota_warning2 = storage=80%% quota-warning 80 %u
>>   sieve = /var/sieve-scripts/%Ln.sieve
>>   sieve_dir = /var/vmail/%Ln/home/sieve
>>   stats_command_min_time = 1 mins
>>   stats_domain_min_time = 12 hours
>>   stats_ip_min_time = 12 hours
>>   stats_memory_limit = 16 M
>>   stats_refresh = 30 secs
>>   stats_session_min_time = 15 mins
>>   stats_track_cmds = yes
>>   stats_user_min_time = 1 hours
>> }
>> postmaster_address = postmaster@myDomain
>> protocols = imap pop3 sieve sieve
>> service auth-worker {
>>   user = root
>> }
>> service auth {
>>   unix_listener /var/spool/postfix/private/auth {
>>     mode = 0777
>>     user = vmail
>>   }
>> }
>> service imap-login {
>>   inet_listener imaps {
>>     port = 993
>>     ssl = yes
>>   }
>>   process_min_avail = 10
>> }
>> service imap {
>>   process_limit = 2128
>> }
>> service managesieve-login {
>>   inet_listener sieve {
>>     port = 4190
>>   }
>> }
>> service pop3-login {
>>   inet_listener pop3s {
>>     port = 995
>>     ssl = yes
>>   }
>> }
>> service quota-warning {
>>   executable = script /usr/local/bin/quota-warning.sh
>>   user = nobody
>> }
>> service stats {
>>   fifo_listener stats-mail {
>>     mode = 0600
>>     user = vmail
>>   }
>> }
>> ssl_cert = > ssl_key =  # hidden, use -P to show it
>> userdb {
>>   args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext
>>   driver = ldap
>>   override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home
>> }
>> verbose_proctitle = yes
>> protocol lda {
>>   mail_plugins = " quota stats fts fts_solr sieve virtual"
>> }
>> protocol imap {
>>   mail_max_userip_connections = 20
>>   mail_plugins = " quota stats fts

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-05-04 Thread kadafax 
 Bump. Can someone from dovecot dev team please respond something to my
problem ? I've provided all the possible information, and if not please
tell me.
Or do you only provide support for your own FTS solution bundled with
the paid version of dovecot-pro ? If so, you should clearly state it in
the wiki.

Thanks

Le 19/04/2018 à 10:30, kada...@gmail.com a écrit :
> I think I'm progressing: if I create a brand new user, send her an
> email, it will not be indexed right away. I need to doveadm rescan /
> index the Inbox before.
> At the end of this email, a dovecot's debug log snipset (grep'ed on this
> user). In it I can read: "Debug: fts: Indexes disabled for namespace ''
> Can this be a problem ?
>
> Thanks
>
> doveconf -n output:
> # 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.22 (22940fb7)
> # OS: Linux 3.10.0-693.17.1.el7.x86_64 x86_64 CentOS Linux release
> 7.4.1708 (Core)  ext4
> # Hostname: mailServer.myDomain
> auth_master_user_separator = *
> auth_socket_path = /var/run/dovecot/auth-userdb
> auth_username_chars =
> abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
> auth_verbose = yes
> base_dir = /var/run/dovecot/
> debug_log_path = /var/log/dovecot-debug.log
> default_client_limit = 5120
> default_process_limit = 1024
> default_vsz_limit = 2 G
> hostname = myDomain
> info_log_path = /var/log/dovecot.log
> lda_mailbox_autocreate = yes
> lda_mailbox_autosubscribe = yes
> listen = *
> log_path = /var/log/dovecot-error.log
> login_greeting = Welcome
> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
> mail_gid = vmail
> mail_location = maildir:/var/vmail/%Ln
> mail_plugins = " quota stats fts fts_solr"
> mail_uid = vmail
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate mime foreverypart extracttext
> namespace {
>   inbox = yes
>   location = maildir:/var/vmail/%Ln
>   mailbox Drafts {
>     auto = subscribe
>     special_use = \Drafts
>   }
>   mailbox Sent {
>     auto = subscribe
>     special_use = \Sent
>   }
>   mailbox "Sent Messages" {
>     special_use = \Sent
>   }
>   mailbox Trash {
>     auto = subscribe
>     special_use = \Trash
>   }
>   prefix =
>   type = private
> }
> passdb {
>   args = /usr/local/etc/dovecot/deny-users
>   deny = yes
>   driver = passwd-file
> }
> passdb {
>   args = /usr/local/etc/dovecot/master-users
>   driver = passwd-file
>   master = yes
> }
> passdb {
>   args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
>   driver = ldap
> }
> plugin {
>   fts = solr
>   fts_autoindex = yes
>   fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search
>   quota = maildir:User quota
>   quota_rule = *:storage=20GB
>   quota_rule2 = Trash:storage=+200M
>   quota_warning = storage=95%% quota-warning 95 %u
>   quota_warning2 = storage=80%% quota-warning 80 %u
>   sieve = /var/sieve-scripts/%Ln.sieve
>   sieve_dir = /var/vmail/%Ln/home/sieve
>   stats_command_min_time = 1 mins
>   stats_domain_min_time = 12 hours
>   stats_ip_min_time = 12 hours
>   stats_memory_limit = 16 M
>   stats_refresh = 30 secs
>   stats_session_min_time = 15 mins
>   stats_track_cmds = yes
>   stats_user_min_time = 1 hours
> }
> postmaster_address = postmaster@myDomain
> protocols = imap pop3 sieve sieve
> service auth-worker {
>   user = root
> }
> service auth {
>   unix_listener /var/spool/postfix/private/auth {
>     mode = 0777
>     user = vmail
>   }
> }
> service imap-login {
>   inet_listener imaps {
>     port = 993
>     ssl = yes
>   }
>   process_min_avail = 10
> }
> service imap {
>   process_limit = 2128
> }
> service managesieve-login {
>   inet_listener sieve {
>     port = 4190
>   }
> }
> service pop3-login {
>   inet_listener pop3s {
>     port = 995
>     ssl = yes
>   }
> }
> service quota-warning {
>   executable = script /usr/local/bin/quota-warning.sh
>   user = nobody
> }
> service stats {
>   fifo_listener stats-mail {
>     mode = 0600
>     user = vmail
>   }
> }
> ssl_cert =  ssl_key =  # hidden, use -P to show it
> userdb {
>   args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext
>   driver = ldap
>   override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home
> }
> verbose_proctitle = yes
> protocol lda {
>   mail_plugins = " quota stats fts fts_solr sieve virtual"
> }
> protocol imap {
>   mail_max_userip_connections = 20
>   mail_plugins = " quota stats fts fts_solr imap_quota virtual imap_stats"
> }
> protocol sieve {
>   mail_max_userip_connections = 20
> }
>
> DEBUG LOG:
> Apr 19 10:23:26 lda(NewUser@myDomain): Debug: Loading modules from
> directory: /usr/local/lib/dovecot
> Apr 19 10:23:26 lda(NewUser@myDomain): Debug: Module loaded:
> /usr/local/lib/dovecot/lib10_quota_plugin.so
> Apr 19 10:23:26 lda(NewUser@myDomain): Debug: Module

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-04-19 Thread kadafax 
I think I'm progressing: if I create a brand new user, send her an
email, it will not be indexed right away. I need to doveadm rescan /
index the Inbox before.
At the end of this email, a dovecot's debug log snipset (grep'ed on this
user). In it I can read: "Debug: fts: Indexes disabled for namespace ''
Can this be a problem ?

Thanks

doveconf -n output:
# 2.2.34 (874deae): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.22 (22940fb7)
# OS: Linux 3.10.0-693.17.1.el7.x86_64 x86_64 CentOS Linux release
7.4.1708 (Core)  ext4
# Hostname: mailServer.myDomain
auth_master_user_separator = *
auth_socket_path = /var/run/dovecot/auth-userdb
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
auth_verbose = yes
base_dir = /var/run/dovecot/
debug_log_path = /var/log/dovecot-debug.log
default_client_limit = 5120
default_process_limit = 1024
default_vsz_limit = 2 G
hostname = myDomain
info_log_path = /var/log/dovecot.log
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
listen = *
log_path = /var/log/dovecot-error.log
login_greeting = Welcome
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_gid = vmail
mail_location = maildir:/var/vmail/%Ln
mail_plugins = " quota stats fts fts_solr"
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace {
  inbox = yes
  location = maildir:/var/vmail/%Ln
  mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox "Sent Messages" {
    special_use = \Sent
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix =
  type = private
}
passdb {
  args = /usr/local/etc/dovecot/deny-users
  deny = yes
  driver = passwd-file
}
passdb {
  args = /usr/local/etc/dovecot/master-users
  driver = passwd-file
  master = yes
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
  driver = ldap
}
plugin {
  fts = solr
  fts_autoindex = yes
  fts_solr = url=http://localhost:8983/solr/dovecot/ break-imap-search
  quota = maildir:User quota
  quota_rule = *:storage=20GB
  quota_rule2 = Trash:storage=+200M
  quota_warning = storage=95%% quota-warning 95 %u
  quota_warning2 = storage=80%% quota-warning 80 %u
  sieve = /var/sieve-scripts/%Ln.sieve
  sieve_dir = /var/vmail/%Ln/home/sieve
  stats_command_min_time = 1 mins
  stats_domain_min_time = 12 hours
  stats_ip_min_time = 12 hours
  stats_memory_limit = 16 M
  stats_refresh = 30 secs
  stats_session_min_time = 15 mins
  stats_track_cmds = yes
  stats_user_min_time = 1 hours
}
postmaster_address = postmaster@myDomain
protocols = imap pop3 sieve sieve
service auth-worker {
  user = root
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0777
    user = vmail
  }
}
service imap-login {
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_min_avail = 10
}
service imap {
  process_limit = 2128
}
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
service pop3-login {
  inet_listener pop3s {
    port = 995
    ssl = yes
  }
}
service quota-warning {
  executable = script /usr/local/bin/quota-warning.sh
  user = nobody
}
service stats {
  fifo_listener stats-mail {
    mode = 0600
    user = vmail
  }
}
ssl_cert =

Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-04-17 Thread kadafax 
Le 17/04/2018 à 14:18, kfx a écrit :
> dovecot 2.2.34
> solr 7.2
>
> I only see new messages after typing on the server "doveadm fts rescan
> -u username" though I've followed the wiki and added "fts_autoindex =
> yes" in 90-plugin.conf . Subsequent search for the same pattern always
> gives the same result, ignoring new emails with that particular pattern.
> Solr is working correctly and the correct search results comes
> immediately (whatever the searched word) after a "doveadm fts rescan"
>
> # doveconf | grep fts
> mail_plugins = " quota stats fts fts_solr"
>   fts = solr
>   fts_autoindex = yes
>   fts_solr = url=http://localhost:8983/solr/dovecot/
> [POP]  mail_plugins = " quota stats fts fts_solr sieve virtual"
> [IMAP]  mail_plugins = " quota stats fts fts_solr imap_quota virtual
> imap_stats"
>
>
>
>

Here are some debug logs from a search (responding with incorrect - or
old - results, as new emails have arrived, containing the same
"MySearchPattern" pattern).
This leads me to a problem on the dovecot side. Only a "doveadm fts
rescan -u username" can solve this situation, so should I add a cronjob
for it ? It defeats the purpose of solr as the first search after is
really slow (but correct).

Here are the logs, please tell me if I should provide more info, and
thanks for any help:

Apr 17 21:44:56 imap(MyUsername): Debug: http-client: host localhost:
Host created
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: host localhost:
DNS lookup successful; got 2 IPs
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: peer [::1]:8983:
Peer created
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: queue
http://localhost:8983: Setting up connection to [::1]:8983 (1 requests
pending)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: peer [::1]:8983:
Linked queue http://localhost:8983 (1 queues linked)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: queue
http://localhost:8983: Started new connection to [::1]:8983
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: request [Req1: GET
http://localhost:8983/solr/dovecot/select?fl=uid,score=25495=uid+asc=%7b!lucene+q.op%3dAND%7d(hdr:MySearchPattern+OR+body:MySearchPattern)=%2Bbox:e09cce0283e8695ab76002deed92+%2Buser:MyUsername]:
Submitted
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: Waiting for 1
requests to finish
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: peer [::1]:8983:
Creating 1 new connections to handle requests (already 0 usable,
connecting to 0, closing 0)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: peer [::1]:8983:
Making new connection 1 of 1
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: conn [::1]:8983
[0]: HTTP connection created (1 parallel connections exist)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: conn [::1]:8983
[0]: Connected
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: conn [::1]:8983
[0]: Ready for requests
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: peer [::1]:8983:
Successfully connected (connections=1)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: peer [::1]:8983:
Using 1 idle connections to handle 1 requests (1 total connections ready)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: queue
http://localhost:8983: Connection to peer [::1]:8983 claimed request
[Req1: GET
http://localhost:8983/solr/dovecot/select?fl=uid,score=25495=uid+asc=%7b!lucene+q.op%3dAND%7d(hdr:MySearchPattern+OR+body:MySearchPattern)=%2Bbox:e09cce0283e8695ab76002deed92+%2Buser:MyUsername]
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: conn [::1]:8983
[0]: Claimed request [Req1: GET
http://localhost:8983/solr/dovecot/select?fl=uid,score=25495=uid+asc=%7b!lucene+q.op%3dAND%7d(hdr:MySearchPattern+OR+body:MySearchPattern)=%2Bbox:e09cce0283e8695ab76002deed92+%2Buser:MyUsername]
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: request [Req1: GET
http://localhost:8983/solr/dovecot/select?fl=uid,score=25495=uid+asc=%7b!lucene+q.op%3dAND%7d(hdr:MySearchPattern+OR+body:MySearchPattern)=%2Bbox:e09cce0283e8695ab76002deed92+%2Buser:MyUsername]:
Sent header
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: peer [::1]:8983:
No more requests to service for this peer (1 connections exist)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: conn [::1]:8983
[0]: Got 200 response for request [Req1: GET
http://localhost:8983/solr/dovecot/select?fl=uid,score=25495=uid+asc=%7b!lucene+q.op%3dAND%7d(hdr:MySearchPattern+OR+body:MySearchPattern)=%2Bbox:e09cce0283e8695ab76002deed92+%2Buser:MyUsername]
(took 109 ms + 23 ms in queue)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: conn [::1]:8983
[0]: Response payload stream destroyed (0 ms after initial response)
Apr 17 21:44:56 imap(MyUsername): Debug: http-client: request [Req1: GET
http://localhost:8983/solr/dovecot/select?fl=uid,score=25495=uid+asc=%7b!lucene+q.op%3dAND%7d(hdr:MySearchPattern+OR+body:MySearchPattern)=%2Bbox:e09cce0283e8695ab76002deed92+%2Buser:MyUsername]:
Finished
Apr 17

Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"

2018-04-17 Thread kfx 
dovecot 2.2.34
solr 7.2

I only see new messages after typing on the server "doveadm fts rescan
-u username" though I've followed the wiki and added "fts_autoindex =
yes" in 90-plugin.conf . Subsequent search for the same pattern always
gives the same result, ignoring new emails with that particular pattern.
Solr is working correctly and the correct search results comes
immediately (whatever the searched word) after a "doveadm fts rescan"

# doveconf | grep fts
mail_plugins = " quota stats fts fts_solr"
  fts = solr
  fts_autoindex = yes
  fts_solr = url=http://localhost:8983/solr/dovecot/
[POP]  mail_plugins = " quota stats fts fts_solr sieve virtual"
[IMAP]  mail_plugins = " quota stats fts fts_solr imap_quota virtual
imap_stats"