Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
Okay I'm progressing here. If I copy an email within my MUA to another
folder and then make another copy back to my Inbox, the new copied email
is correctly and immediately indexed ! If I delete an email, the search
result is also correct and immediate.
Is it possible that my problem come from Dovecot LDA ?
In 10-mail.conf I have:
# cat dovecot/conf.d/10-mail.conf| grep -v -e "^$" -e "^#" -e "^ #"
mail_location = maildir:/var/vmail/%Ln
namespace {
type = private
location = maildir:/var/vmail/%Ln
inbox = yes
mailbox Trash {
auto = subscribe
special_use = \Trash
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe # autocreate, autosubscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
}
mail_uid=vmail
mail_gid=vmail
auth_socket_path = /var/run/dovecot/auth-userdb
mail_plugins = $mail_plugins quota stats fts fts_solr
And in 15-lda.conf:
# cat dovecot/conf.d/15-lda.conf| grep -v -e "^$" -e "^#" -e "^ #"
postmaster_address = postmaster@mydomain
hostname = mydomain
recipient_delimiter = +
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
protocol lda {
mail_plugins = $mail_plugins sieve virtual
}
And the lda portion of "doveconf -n" become:
protocol lda {
mail_plugins = " quota stats fts fts_solr sieve virtual"
}
Here is my postfix relevant conf:
## MAIN.CF ##
# postconf -n | grep dovecot
dovecot_destination_recipient_limit = 1
smtpd_sasl_type = dovecot
virtual_transport = dovecot
#
## MASTER.CF ##
dovecot unix- n n - - pipe
flags=DRhu user=vmail:vmail
argv=/usr/local/libexec/dovecot/dovecot-lda -f ${sender} -a ${recipient}
-d ${user}@${nexthop}
#####
On 28/05/2018 13:30, kfx wrote:
On 28/05/2018 13:23, kfx wrote:
On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx <mailto:kada...@gmail.com>> wrote:
Especially what is in the "fts" header vs. "next uid" header? Does
the UID in "fts" header keep changing every time you save a new
mail? I suppose it will.
Diff between 2 emails:
next uid = 30104 | next uid = 30105
last_indexed_uid = 30103 | last_indexed_uid = 30104
So Dovecot thinks it has indexed everything.
You could also monitor (e.g. tcpdump/wireshark) the network traffic
between Dovecot <-> Solr what happens when a new mail arrives. I
suspect Dovecot sends it to Solr, which for whatever reason just
ignores the update.
### TCPDUMP
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml
37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernamename="body">Search Pattern: Kai8oovi
..
And Dovecot sends the mail.
# SOLR'S RESPONSE ###
0
0
And Solr receives it. Your tcpdump doesn't show softCommit="true" waitSearcher="true"/> being sent though. Do you see
it being sent anywhere?
Yes:
### TCPDUMP ###
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml
### /TCPDUMP ###
Does it make the mails visible if you run it yourself? Or if you run
hard commit? :
curl http://:8983/solr/update?commit=true
# curl http://127.0.0.1:8983/solr/update?commit=true
[0/0]
Error 404 Not Found
HTTP ERROR 404
Problem accessing /solr/update. Reason:
Not Found
# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true
0
0
# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4
results)
In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot
I can see:
lastModified: less than a minute ago
version:1428772
numDocs:6353615
maxDoc:6356213
deletedDocs:2598
So it IS indexing :(
Just below I see the "optimized:" parameter followed by an icon which
seems saying that is NOT "optimized". Don't know if it's relevant.
This is driving me crazy :(
Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
On 28/05/2018 13:23, kfx wrote:
On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx <kada...@gmail.com
<mailto:kada...@gmail.com>> wrote:
Especially what is in the "fts" header vs. "next uid" header? Does
the UID in "fts" header keep changing every time you save a new
mail? I suppose it will.
Diff between 2 emails:
next uid = 30104 | next uid = 30105
last_indexed_uid = 30103 | last_indexed_uid = 30104
So Dovecot thinks it has indexed everything.
You could also monitor (e.g. tcpdump/wireshark) the network traffic
between Dovecot <-> Solr what happens when a new mail arrives. I
suspect Dovecot sends it to Solr, which for whatever reason just
ignores the update.
### TCPDUMP
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml
37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernamename="body">Search Pattern: Kai8oovi
..
And Dovecot sends the mail.
# SOLR'S RESPONSE ###
0
0
And Solr receives it. Your tcpdump doesn't show softCommit="true" waitSearcher="true"/> being sent though. Do you see
it being sent anywhere?
Yes:
### TCPDUMP ###
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml
### /TCPDUMP ###
Does it make the mails visible if you run it yourself? Or if you run
hard commit? :
curl http://:8983/solr/update?commit=true
# curl http://127.0.0.1:8983/solr/update?commit=true
[0/0]
Error 404 Not Found
HTTP ERROR 404
Problem accessing /solr/update. Reason:
Not Found
# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true
0
0
# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4
results)
In the web interface of solr at http://127.0.0.1:8983/solr/#/~cores/dovecot
I can see:
lastModified: less than a minute ago
version:1428772
numDocs:6353615
maxDoc:6356213
deletedDocs:2598
So it IS indexing :(
Just below I see the "optimized:" parameter followed by an icon which
seems saying that is NOT "optimized". Don't know if it's relevant.
This is driving me crazy :(
Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
On 28/05/2018 13:04, Timo Sirainen wrote:
On 28 May 2018, at 13.28, kfx <kada...@gmail.com
<mailto:kada...@gmail.com>> wrote:
Especially what is in the "fts" header vs. "next uid" header? Does
the UID in "fts" header keep changing every time you save a new mail?
I suppose it will.
Diff between 2 emails:
next uid = 30104 | next uid = 30105
last_indexed_uid = 30103 | last_indexed_uid = 30104
So Dovecot thinks it has indexed everything.
You could also monitor (e.g. tcpdump/wireshark) the network traffic
between Dovecot <-> Solr what happens when a new mail arrives. I
suspect Dovecot sends it to Solr, which for whatever reason just
ignores the update.
### TCPDUMP
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml
37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernameSearch
Pattern: Kai8oovi
..
And Dovecot sends the mail.
# SOLR'S RESPONSE ###
0
0
And Solr receives it. Your tcpdump doesn't show softCommit="true" waitSearcher="true"/> being sent though. Do you see it
being sent anywhere?
Yes:
### TCPDUMP ###
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Expect: 100-continue
Content-Length: 47
Connection: Keep-Alive
Content-Type: text/xml
### /TCPDUMP ###
Does it make the mails visible if you run it
yourself? Or if you run hard commit? :
curl http://:8983/solr/update?commit=true
# curl http://127.0.0.1:8983/solr/update?commit=true
[0/0]
Error 404 Not Found
HTTP ERROR 404
Problem accessing /solr/update. Reason:
Not Found
# curl http://127.0.0.1:8983/solr/dovecot/update?commit=true
0
0
# doveadm search -u username mailbox INBOX body Kai8oovi
==> No result ('Kai8oovi' is the search pattern, it should returns 4
results)
Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
On 24/05/2018 15:33, Timo Sirainen wrote:
On 21 May 2018, at 14.11, kada...@gmail.com wrote:
Le 21/05/2018 à 12:38, Aki Tuomi a écrit :
can you try turning on pluign { fts_enforced = yes } and repeat your test?
Same (wrong) result:
1. Send an email with "too6Ouka" in the body
2. Search against "too6Ouka":
# doveadm search -u username mailbox INBOX body too6Ouka
--> No result
3. Force re-index:
# doveadm fts rescan -u username
4. Search again against "too6Ouka":
# doveadm search -u username mailbox INBOX body too6Ouka
--> e09cce0283e8695ab76002deed92 29055
Don't know if relevant, but on a side note, if I send a second message
with "too6Ouka" in the body, followed by:
# doveadm -v index -u username Inbox
--> doveadm(username): Info: INBOX: Cache is already up to date
And a search against the pattern immediately return only one result
instead of two.
So it looks like after "doveadm fts rescan" you can do initial indexing
successfully. Afterwards the index isn't updated at all, unless you rebuild it entirely.
fts_autoindex=yes appears to work as expected, because after fts rescan you did a search
which automatically updated the index. So this doesn't have anything to do with
fts_autoindex, just that your index updates don't seem to work at all.
What does it show in the header if you do:
doveadm dump /var/vmail/username/dovecot.index
Detected file type: index
-- INDEX: /var/vmail/username/dovecot.index
version .. = 7.3
base header size . = 120
header size .. = 640
record size .. = 16
compat flags . = 1
index id . = 1527164208 (2018-05-24 14:16:48)
flags = 0
uid validity . = 1516890243 (2018-01-25 15:24:03)
next uid . = 30104
messages count ... = 17068
seen messages count .. = 17011
deleted messages count ... = 0
first recent uid . = 30104
first unseen uid lowwater = 393
first deleted uid lowwater = 30087
log file seq . = 14
log file tail offset . = 28144
log file head offset . = 28144
log2 rotate time . = 1527487933 (2018-05-28 08:12:13)
last temp file scan .. = 0 (1970-01-01 01:00:00)
day stamp = 1527458400 (2018-05-28 00:00:00)
day first uid[0] . = 30037
day first uid[1] . = 29921
day first uid[2] . = 29814
day first uid[3] . = 29669
day first uid[4] . = 2
day first uid[5] . = 0
day first uid[6] . = 0
day first uid[7] . = 0
-- Extension 0 --
name = maildir
hdr_size = 36
reset_id = 0
record_offset = 0
record_size . = 0
record_align = 0
header
- new_check_time = 2018-05-28 11:48:14
- new_mtime . = 2018-05-28 11:47:59
- new_mtime_nsecs ... = 129602175
- cur_check_time = 2018-05-28 11:48:14
- cur_mtime . = 2018-05-28 11:48:01
- cur_mtime_nsecs = 277650283
- uidlist_mtime . = 2018-05-28 11:47:58
- uidlist_mtime_nsecs = 521588557
- uidlist_size .. = 1334026
-- Extension 1 --
name = keywords
hdr_size = 292
reset_id = 0
record_offset = 5
record_size . = 3
record_align = 1
-- Extension 2 --
name = hdr-vsize
hdr_size = 16
reset_id = 0
record_offset = 0
record_size . = 0
record_align = 8
header
- highest uid . = 0
- message count = 0
- vsize ... = 0
-- Extension 3 --
name = cache
hdr_size = 0
reset_id = 1527164233
record_offset = 8
record_size . = 4
record_align = 4
-- Extension 4 --
name = vsize
hdr_size = 0
reset_id = 0
record_offset = 12
record_size . = 4
record_align = 4
-- Extension 5 --
name = fts
hdr_size = 12
reset_id = 0
record_offset = 0
record_size . = 0
record_align = 0
header
- last_indexed_uid . = 30103
- settings_checksum = 0
-- Keywords --
0 = unknown-1
1 = unknown-2
2 = unknown-5
3 = unknown-8
4 = unknown-6
5 = unknown-3
6 = unknown-7
7 = unknown-0
8 = NonJunk
9 = $Forwarded
10 = $label1
11 = $label2
12 = Junk
13 = $label3
14 = $label4
Especially what is in the "fts" header vs. "next uid" header? Does the UID in "fts" header keep changing every time you save a new mail? I suppose it will.
Diff between 2 emails:
next uid = 30104|next uid = 30105
last_indexed_uid = 30103|last_indexed_uid = 30104
You could also monitor (e.g. tcpdump/wireshark) the network traffic between Dovecot
<-> Solr what happens when a new mail arrives. I suspect Dovecot sends it to
Solr, which for whatever reason just ignores the update.
### TCPDUMP
POST /solr/dovecot/update HTTP/1.1
Host: localhost:8983
Date: Mon, 28 May 2018 10:18:05 GMT
Transfer-Encoding: chunked
Connection: Keep-Alive
Content-Type: text/xml
37581name="box">e0c58a309323515311083ea484a8name="user">usernamename="id">37581/e0c58a309323515311083ea484a8/usernamename="body">Search Pattern:
Re: Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
ript /usr/local/bin/quota-warning.sh
> user = nobody
> }
> service stats {
> fifo_listener stats-mail {
> mode = 0600
> user = vmail
> }
> }
> ssl_cert = ssl_key = # hidden, use -P to show it
> userdb {
> args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext
> driver = ldap
> override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home
> }
> verbose_proctitle = yes
> protocol lda {
> mail_plugins = " quota stats fts fts_solr sieve virtual"
> }
> protocol imap {
> mail_max_userip_connections = 20
> mail_plugins = " quota stats fts fts_solr imap_quota virtual imap_stats"
> }
> protocol sieve {
> mail_max_userip_connections = 20
> }
>
>
>
>>
>> Aki
>>
>>
>> On 21.05.2018 13:31, kada...@gmail.com wrote:
>>> Le 21/05/2018 à 11:46, Aki Tuomi a écrit :
>>>> That message is coming from the temporary raw user, not from the actual
>>>> user.
>>>>
>>>> How did you verify that the FTS indexes are not created or updated
>>>> automatically?
>>> I send an email containing a specific strings ('thahB8ea' in that case)
>>> and search agains it:
>>> # doveadm search -u username mailbox INBOX body thahB8ea
>>> --> No result
>>> (same with integrated search on roundcube)
>>>
>>> Then I force a rescan, followed by the same search:
>>> # doveadm fts rescan -u username
>>> # doveadm search -u username mailbox INBOX body thahB8ea
>>> (Long waiting time)
>>> --> e09cce0283e8695ab76002deed92 29048
>>>
>>> So I conclude that autoindexing is not working.
>>>
>>> Thanks,
>>> kfx
>>>
Bug: Dovecot index loosing sync with FTS despite "fts_autoindex = yes"
dovecot 2.2.34 solr 7.2 I only see new messages after typing on the server "doveadm fts rescan -u username" though I've followed the wiki and added "fts_autoindex = yes" in 90-plugin.conf . Subsequent search for the same pattern always gives the same result, ignoring new emails with that particular pattern. Solr is working correctly and the correct search results comes immediately (whatever the searched word) after a "doveadm fts rescan" # doveconf | grep fts mail_plugins = " quota stats fts fts_solr" fts = solr fts_autoindex = yes fts_solr = url=http://localhost:8983/solr/dovecot/ [POP] mail_plugins = " quota stats fts fts_solr sieve virtual" [IMAP] mail_plugins = " quota stats fts fts_solr imap_quota virtual imap_stats"
Dovecot 2.2.34 FTS-SOLR = inconsistent results
Hello,
I'm trying to figure why I have inconsistent results with my fts-solr
powered search:
My test is to find the string "am5oocobey9hait5Wiex8yohgoob9thu" in the
3 emails containing it that I sent to myself:
- Directly on the server:
doveadm search -u username text am5oocobey9hait5Wiex8yohgoob9thu
e2c58a309323515311083ea484a8 7937
e2c58a309323515311083ea484a8 7938
e0c58a309323515311083ea484a8 36479
e0c58a309323515311083ea484a8 36480
e0c58a309323515311083ea484a8 36481
- RoundCube webmail:
0 Result
- On the server with "telnet localhost imap" (2 identical searches):
[snip...]
a23 SEARCH text am5oocobey9hait5Wiex8yohgoob9thu
* SEARCH
* 14488 FETCH (FLAGS (\Seen NonJunk))
* 14495 EXISTS
a23 OK Search completed (0.015 + 0.000 + 0.004 secs).
a25 SEARCH text am5oocobey9hait5Wiex8yohgoob9thu
* SEARCH
a25 OK Search completed (0.006 + 0.000 + 0.003 secs).
Here is my setup:
root@myMailServer ~ # uname -a
Linux myMailServer.mydomain 3.10.0-693.17.1.el7.x86_64 #1 SMP Thu Jan 25
20:13:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@myMailServer ~ # cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
root@plainte ~ # doveconf -n
# 2.2.34 (874deae):
/usr/local/etc/dovecot/dovecot.conf
[50/1259]
# Pigeonhole version 0.4.22 (22940fb7)
# OS: Linux 3.10.0-693.17.1.el7.x86_64 x86_64 CentOS Linux release
7.4.1708 (Core) ext4
# Hostname: myMailServer.mydomain
auth_cache_size = 10 M
auth_master_user_separator = *
auth_socket_path = /var/run/dovecot/auth-userdb
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
auth_verbose = yes
base_dir = /var/run/dovecot/
debug_log_path = /var/log/dovecot-debug.log
default_client_limit = 5120
default_process_limit = 1024
default_vsz_limit = 2 G
hostname = mydomain
info_log_path = /var/log/dovecot.log
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
listen = *
log_path = /var/log/dovecot-error.log
login_greeting = Welcome
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c
mail_gid = vmail
mail_location = maildir:/var/vmail/%Ln
mail_plugins = fts fts_solr quota stats
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart extracttext
namespace {
inbox = yes
location = maildir:/var/vmail/%Ln
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox "Sent Messages" {
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
type = private
}
passdb {
args = /usr/local/etc/dovecot/deny-users
deny = yes
driver = passwd-file
}
passdb {
args = /usr/local/etc/dovecot/master-users
driver = passwd-file
master = yes
}
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
fts = solr
fts_autoindex = yes
fts_autoindex_max_recent = 1000
fts_solr = url=http://127.0.0.1:8983/solr/dovecot/
quota = maildir:User quota
quota_rule = *:storage=20GB
quota_rule2 = Trash:storage=+200M
quota_warning = storage=95%% quota-warning 95 %u
quota_warning2 = storage=80%% quota-warning 80 %u
sieve = /var/sieve-scripts/%Ln.sieve
sieve_dir = /var/vmail/%Ln/home/sieve
stats_command_min_time = 1 mins
stats_domain_min_time = 12 hours
stats_ip_min_time = 12 hours
stats_memory_limit = 16 M
stats_refresh = 30 secs
stats_session_min_time = 15 mins
stats_track_cmds = yes
stats_user_min_time = 1 hours
}
postmaster_address = postmaster@mydomain
protocols = imap pop3 sieve sieve
service auth-worker {
user = root
}
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0777
user = vmail
}
}
service imap-login {
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 10
}
service imap {
process_limit = 2128
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service pop3-login {
inet_listener pop3s {
port = 995
ssl = yes
}
}
service quota-warning {
executable = script /usr/local/bin/quota-warning.sh
user = nobody
}
service stats {
fifo_listener stats-mail {
mode = 0600
user = vmail
}
}
ssl_cert = http://127.0.0.1:8983/solr/dovecot/update?optimize=true > /dev/null 2>&1
*/5 * * * * /usr/bin/curl
http://127.0.0.1:8983/solr/dovecot/update?commit=true > /dev/null 2>&1
I would be very grateful if someone can point me where my error is, as
I've been on this problem for weeks :(
Thanks,
kfx
[Dovecot] FTS Lucene: Fatal: master: service(indexer-worker): child * killed with signal 6
Hi List,
Since we migrated from dovecot 1.2.4 to 2.2.12 with lucene as FTS
engine, I have this in my error log:
Apr 22 14:01:14 indexer-worker(username): Error: lucene index
/var/vmail/username/lucene-indexes: IndexWriter() failed (#1): Lock
obtain timed out
Apr 22 14:01:14 indexer-worker: Error: indexer-worker
[usern...@mydomain.com MailBox-1 - 20800/22707]:
/home/anotherusername/newnew/clucene/src/core/CLucene/index/DocumentsWriter.cpp:210:
std::string lucene::index::DocumentsWriter::closeDocStore(): Assertion
`numDocsInStore*8 == directory-fileLength( (docStoreSegment + . +
IndexFileNames::FIELDS_INDEX_EXTENSION).c_str() )' failed.
Apr 22 14:01:14 indexer: Error: Indexer worker disconnected, discarding
1 requests for usern...@mydomain.com
Apr 22 14:01:14 indexer-worker(usern...@mydomain.com): Fatal: master:
service(indexer-worker): child 31158 killed with signal 6 (core not dumped)
Apr 22 14:01:16 imap(username): Error: indexer failed to index mailbox
MailBox-2
Does that sound familiar to someone ?
Thanks.
# 2.2.12: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-431.11.2.el6.x86_64 x86_64 CentOS release 6.5 (Final)
ext4
auth_cache_size = 10 M
auth_debug = yes
auth_master_user_separator = *
auth_socket_path = /var/run/dovecot/auth-userdb
auth_username_chars =
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@*
auth_verbose = yes
auth_verbose_passwords = sha1
base_dir = /var/run/dovecot/
debug_log_path = /var/log/dovecot-debug.log
default_client_limit = 2128
default_process_limit = 512
default_vsz_limit = 2 G
hostname = mydomain.com
info_log_path = /var/log/dovecot.log
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
listen = *
log_path = /var/log/dovecot-error.log
login_greeting = Welcome
login_log_format_elements = user=%u method=%m rip=%r lip=%l mpid=%e %c
mail_debug = yes
mail_gid = vmail
mail_location = maildir:/var/vmail/%Ln
mail_plugins = fts fts_lucene quota stats
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date ihave
namespace {
inbox = yes
location = maildir:/var/vmail/%Ln
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
prefix =
type = private
}
passdb {
args = /usr/local/etc/dovecot/deny-users
deny = yes
driver = passwd-file
}
passdb {
args = /usr/local/etc/dovecot/master-users
driver = passwd-file
master = yes
}
passdb {
args = /usr/local/etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
fts = lucene
fts_autoindex = yes
fts_autoindex_max_recent = 1000
fts_lucene = whitespace_chars=@
quota = maildir:User quota
quota_rule = *:storage=20GB
quota_rule2 = Trash:storage=+200M
quota_warning = storage=95%% quota-warning 95 %u
quota_warning2 = storage=80%% quota-warning 80 %u
sieve = /var/sieve-scripts/%Ln.sieve
sieve_dir = /var/vmail/%Ln/home/sieve
stats_command_min_time = 1 mins
stats_domain_min_time = 12 hours
stats_ip_min_time = 12 hours
stats_memory_limit = 16 M
stats_refresh = 30 secs
stats_session_min_time = 15 mins
stats_track_cmds = yes
stats_user_min_time = 1 hours
}
postmaster_address = postmas...@mydomain.com
protocols = imap pop3 sieve sieve
service auth-worker {
user = root
}
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0777
user = vmail
}
}
service decode2text {
executable = script /usr/local/libexec/dovecot/decode2text.sh
unix_listener decode2text {
mode = 0666
}
user = dovecot
}
service imap-login {
inet_listener imaps {
port = 993
ssl = yes
}
process_min_avail = 5
}
service imap {
process_limit = 1024
}
service managesieve-login {
inet_listener sieve {
port = 4190
}
}
service pop3-login {
inet_listener pop3s {
port = 995
ssl = yes
}
}
service quota-warning {
executable = script /usr/local/bin/quota-warning.sh
user = nobody
}
service stats {
fifo_listener stats-mail {
mode = 0600
user = vmail
}
}
ssl_cert = /etc/pki/tls/certs/mail.mydomain.com.pem
ssl_key = /etc/pki/tls/private/mail.mydomain.com.key
userdb {
args = /usr/local/etc/dovecot/dovecot-ldap-userdb.conf.ext
driver = ldap
override_fields = uid=vmail gid=vmail home=/var/vmail/%Ln/home
}
verbose_proctitle = yes
protocol lda {
mail_plugins = fts fts_lucene quota stats sieve virtual
}
protocol imap {
mail_plugins = fts fts_lucene quota stats imap_quota virtual imap_stats
}
Re: [Dovecot] testing fts-solr?
On 02/28/2012 03:04 AM, Timo Sirainen wrote: On 28.2.2012, at 2.14, Jean-Daniel Beaubien wrote: Ok, I had to fix a few things. First of all, I had to declare mail_plugins = fts fts_solr in global file (dovecot.conf). After doing that, running doveadm fts optimize stopped giving me errors. I ran doveadm index -user INBOX (and other folders). I'm assuming this actually did something because java/tomcat6 were really busy for a few mins and I could see the solr logs going nuts (/var/log/tomcat6/catalina.out). OK, so it's indexing. Dovecot 2.1.1 and latest solr. Indexing seems ok because the size of the solr's data directory is growing and log shows things like this: Feb 28 14:30:45 indexer-worker(username): Info: Indexed 12239 messages in SubFolder but thunderbird returns nothing (it says not match found for hello for example). I dont know what more can I do to debug this... Solr is working as the nobody user, is it OK for dovecot ? Does dovecot need to be have write access to the solr's data directory ? Thanks
Re: [Dovecot] testing fts-solr?
On 02/28/2012 03:23 PM, Charles Marcus wrote:
On 2012-02-28 8:41 AM, kfx kada...@gmail.com wrote:
Dovecot 2.1.1 and latest solr. Indexing seems ok because the size of the
solr's data directory is growing and log shows things like this:
Feb 28 14:30:45 indexer-worker(username): Info: Indexed 12239 messages
in SubFolder
but thunderbird returns nothing (it says not match found for hello
for example).
*How* are you testing with Thunderbird?
Did you enable the 'Run search on server' option in the Advanced Search
window? Doing this *should* result in Thunderbird using dovecots indexes
server side.
Yes I did.
Some more info:
by telnet'ing directly and issuing:
c search text pattern
* SEARCH 1208
c OK Search completed (0.003 secs).
But the same search in thunderbird return No matches found :(
Here are the solr logs, first search is the c search text pattern
second is from thunderbird:
Feb 28, 2012 3:26:21 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={fl=uid,scoresort=uid+ascfq=%2Bbox:d8f5c7250ed14c4f1a5d3ea484a8+%2Buser:usernameq=(hdr:pattern+OR+body:pattern)rows=108227}
hits=1 status=0 QTime=1
Feb 28, 2012 3:27:49 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={fl=uid,scoresort=uid+ascfq=%2Bbox:d8f5c7250ed14c4f1a5d3ea484a8+%2Buser:usernameq=body:patternrows=108227}
hits=0 status=0 QTime=1
It seems thunderbird is only searching in the body and not the header
unlike the c search text method and that the pattern was found in the
header of some random email. So none of the two method is working when
searching in the body of the emails but work for the headers (???).
fts_solr module seems to be correctly launched:
Feb 28 15:07:53 imap: Debug: Module loaded:
/usr/local/lib/dovecot/lib20_fts_plugin.so
Feb 28 15:07:53 imap: Debug: Module loaded:
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so
[root@server log]# grep -R mail_plugins /usr/local/etc/dovecot/ | grep -v \#
/usr/local/etc/dovecot/conf.d/10-mail.conf:mail_plugins = autocreate fts
fts_solr quota
/usr/local/etc/dovecot/conf.d/15-lda.conf: mail_plugins = $mail_plugins
sieve
[root@server log]# grep fts /usr/local/etc/dovecot/conf.d/90-plugin.conf
fts = solr
fts_solr = url=http://127.0.0.1:8983/solr/ break-imap-search
Don't know where to search :(
Re: [Dovecot] testing fts-solr?
Ok I feel ashame... it was a third party init scrip who was the problem :( Sorry for the noise and thank you for dovecot
[Dovecot] about Update Road and no IDLE in CAPABILITY
Hi list, I'm in the path of moving from courier-imap to dovecot and testing it since 1.2-rc3. So far it is very promising. Thank you for your amazing work. So my first question is: what is the recommended update road ? For now, I first managesieve patch/compile the source, go on to the old source folder and do a make uninstall, switch back to the new source folder and do the make install. As I am using dovecot-sieve and managesieve, the update is a little bit tricky. My second question is how to have the IDLE mode to appear in the server's CAPABILITY announce (without touching imap_capability in the conf ?) I think the IDLE mode is working because when I send me a mail from the server directly in command line like: echo test | mail -s test m...@address.com The mail immediately pop-up in my thunderbird from my workstation. # !tel telnet 127.0.0.1 143 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN] My Message --- No IDLE. Details: - Centos 5.3 x86_64 (Selinux in permissive mode) - /proc/sys/fs/inotify/ exist - # rpm -qa | grep gamin gamin-0.1.7-8.el5 gamin-0.1.7-8.el5 gamin-devel-0.1.7-8.el5 gamin-devel-0.1.7-8.el5 - Dovecote 1.2.0 Compilation: $ ./configure --with-ldap=yes --with-notify=inotify Install prefix . : /usr/local File offsets ... : 64bit I/O polling : epoll I/O notifys : inotify SSL : yes (OpenSSL) GSSAPI . : no passdbs : passwd passwd-file shadow checkpassword ldap : -pam -bsdauth -sia -sql -vpopmail userdbs : static prefetch passwd passwd-file checkpassword ldap nss : -sql -vpopmail SQL drivers : : -pgsql -mysql -sqlite - # dovecot --version 1.2.0 - # dovecot -n # 1.2.0: /usr/local/etc/dovecot.conf # OS: Linux 2.6.18-128.1.10.el5 x86_64 CentOS release 5.3 (Final) ext3 log_path: /var/log/dovecot-error.log info_log_path: /var/log/dovecot-debug.log protocols: imap imaps pop3 pop3s managesieve ssl_ca_file: /etc/pki/CA/chaine.crt ssl_cert_file: /etc/pki/dovecot/certs/server.pem ssl_key_file: /etc/pki/dovecot/private/server.key login_dir: /usr/local/var/run/dovecot/login login_executable(default): /usr/local/libexec/dovecot/imap-login login_executable(imap): /usr/local/libexec/dovecot/imap-login login_executable(pop3): /usr/local/libexec/dovecot/pop3-login login_executable(managesieve): /usr/local/libexec/dovecot/managesieve-login login_greeting: My Message verbose_proctitle: yes mail_uid: vmail mail_gid: vmail mail_location: maildir:/var/vmail/%n mail_debug: yes mail_executable(default): /usr/local/libexec/dovecot/imap mail_executable(imap): /usr/local/libexec/dovecot/imap mail_executable(pop3): /usr/local/libexec/dovecot/pop3 mail_executable(managesieve): /usr/local/libexec/dovecot/managesieve mail_plugins(default): quota imap_quota expire mail_plugins(imap): quota imap_quota expire mail_plugins(pop3): quota expire mail_plugins(managesieve): mail_plugin_dir(default): /usr/local/lib/dovecot/imap mail_plugin_dir(imap): /usr/local/lib/dovecot/imap mail_plugin_dir(pop3): /usr/local/lib/dovecot/pop3 mail_plugin_dir(managesieve): /usr/local/lib/dovecot/managesieve imap_client_workarounds(default): outlook-idle imap_client_workarounds(imap): outlook-idle imap_client_workarounds(pop3): imap_client_workarounds(managesieve): auth default: username_format: %Ln verbose: yes debug: yes passdb: driver: ldap args: /etc/dovecot-ldap.conf userdb: driver: ldap args: /etc/dovecot-ldap-userdb.conf socket: type: listen master: path: /var/run/dovecot/auth-master mode: 384 user: vmail group: vmail plugin: quota: maildir:User quota quota_rule: *:storage=10GB quota_rule2: Trash:storage=10%% expire: Trash 1 Trash/* 1 expire_dict: proxy::expire sieve_dir: /var/sieve-upload/ sieve: /var/sieve-scripts/%u.sieve dict: expire: db:/var/lib/dovecot/expire.db Thank you, kfx
Re: [Dovecot] about Update Road and no IDLE in CAPABILITY
Timo Sirainen wrote: On Jul 2, 2009, at 1:18 PM, kfx wrote: My second question is how to have the IDLE mode to appear in the server's CAPABILITY announce (without touching imap_capability in the conf ?) .. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN] My Message Are you sure it's not working? The capabilities listed above are the pre-login capabilities. If client issues CAPABILITY command or after client logs in, the full capability list is sent which contains IDLE. Oh ok it is indeed working ! # telnet 127.0.0.1 143 Trying 127.0.0.1... Connected to 127.0.0.1. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE STARTTLS AUTH=PLAIN] MyMessage WILLITBLEND? CAPABILITY * CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH QUOTA STARTTLS AUTH=PLAIN WILLITBLEND? OK Capability completed. So if now some sysadmin can tell how do they manage to update their dovecot (from source) installation, it would be great ! Thanks again, kfx
