date:20190421

Re: mbsync updating new emails but dovecot not reading them ? [2.3.3]

2019-04-21 Thread Kunal A. via dovecot

Hi Everyone,
I think i found the problem. It has to do with mbsync.
many apologies.
regards
kunal

On Sun, Apr 21, 2019 at 8:57 AM Kunal A.  wrote:

> Hi,
> Once again sincere apologies for emailing here.
> My dovecot isn't reading the newly updated e-mails from the mbsync.
> I have tried running doveadm -D -v force-resync -u ema...@example.com
> "Inbox/Public/Other Users/us...@example.com"
>
> Plese help if possible. Highly appreciate if someone could help here...
>
> Below is the debug log :-
>
> Debug: Loading modules from directory: /usr/lib64/dovecot
> Debug: Module loaded: /usr/lib64/dovecot/lib01_acl_plugin.so
> Debug: Module loaded: /usr/lib64/dovecot/lib10_quota_plugin.so
> Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm
> Debug: Module loaded:
> /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so
> Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
> /usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined
> symbol: expire_set_deinit (this is usually intentional, so just ignore this
> message)
> Debug: Module loaded:
> /usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so
> Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed:
> /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined
> symbol: lucene_index_iter_deinit (this is usually intentional, so just
> ignore this message)
> Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
> /usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol:
> fts_user_get_language_list (this is usually intentional, so just ignore
> this message)
> Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed:
> /usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined
> symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so
> just ignore this message)
> doveadm(ema...@example.com)<20402><>: Debug: auth USER input:
> ema...@example.com uid=5000 gid=5000 home=/var/mail/vhosts/
> example.com/email1
> doveadm(ema...@example.com): Debug: Effective uid=5000, gid=5000,
> home=/var/mail/vhosts/example.com/email1
> doveadm(ema...@example.com): Debug: quota: No quota setting - plugin
> disabled
> doveadm(ema...@example.com): Debug: Namespace inbox: type=private,
> prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
> location=maildir:~/Maildir
> doveadm(ema...@example.com): Debug: maildir++: root=/var/mail/vhosts/
> example.com/email1/Maildir, index=, indexpvt=, control=,
> inbox=/var/mail/vhosts/example.com/email1/Maildir, alt=
> doveadm(ema...@example.com): Debug: acl: initializing backend with data:
> vfile
> doveadm(ema...@example.com): Debug: acl: acl username = ema...@example.com
> doveadm(ema...@example.com): Debug: acl: owner = 1
> doveadm(ema...@example.com): Debug: acl vfile: Global ACLs disabled
> doveadm(ema...@example.com): Debug: Namespace : type=public,
> prefix=Public/, sep=/, inbox=no, hidden=no, list=children,
> subscriptions=yes
> location=maildir:/mnt/5ea86261-cfae-4e17-b21f-3a6c1a03d795/Email/fastmail:LAYOUT=fs
> doveadm(ema...@example.com): Debug: fs:
> root=/mnt/5ea86261-cfae-4e17-b21f-3a6c1a03d795/Email/fastmail, index=,
> indexpvt=, control=, inbox=, alt=
> doveadm(ema...@example.com): Debug: acl: initializing backend with data:
> vfile
> doveadm(ema...@example.com): Debug: acl: acl username = ema...@example.com
> doveadm(ema...@example.com): Debug: acl: owner = 0
> doveadm(ema...@example.com): Debug: acl vfile: Global ACLs disabled
> doveadm(ema...@example.com): Debug: Namespace : type=private, prefix=,
> sep=, inbox=no, hidden=yes, list=no, subscriptions=no
> location=fail::LAYOUT=none
> doveadm(ema...@example.com): Debug: none: root=, index=, indexpvt=,
> control=, inbox=, alt=
> doveadm(ema...@example.com): Debug: acl vfile: file /var/mail/vhosts/
> example.com/email1/Maildir/dovecot-acl not found
>
>
> Doveconf output :-
>
>
> # 2.3.5 (513208660): /etc/dovecot/dovecot.conf
> # OS: Linux 5.0.7-200.fc29.x86_64 x86_64 Fedora release 29 (Twenty Nine)
> # Hostname: machine
> auth_mechanisms = plain login
> mail_debug = yes
> mail_gid = vmail
> mail_location = maildir:~/Maildir
> mail_plugins = acl quota
> mail_privileged_group = mail
> mail_uid = vmail
> mbox_write_locks = fcntl
> namespace {
>   list = children
>   location =
> maildir:/mnt/5ea86261-cfae-4e17-b21f-3a6c1a03d795/Email/fastmail:LAYOUT=fs
>   prefix = Public/
>   separator = /
>   subscriptions = yes
>   type = public
> }
> namespace inbox {
>   inbox = yes
>   list = yes
>   location =
>   mailbox Drafts {
> special_use = \Drafts
>   }
>   mailbox Junk {
> special_use = \Junk
>   }
>   mailbox Sent {
> special_use = \Sent
>   }
>   mailbox "Sent Messages" {
> special_use = \Sent
>   }
>   mailbox Trash {
> special_use = \Trash
>   }
>   prefix = INBOX/
>   separator = /
> }
> passdb {
>   driver = pam
> }
> passdb {
>   args = /etc/dovecot/dovecot-sql.conf.ext
>   driver = sql
> }

mbsync updating new emails but dovecot not reading them ? [2.3.3]

2019-04-21 Thread Kunal A. via dovecot

Hi,
Once again sincere apologies for emailing here.
My dovecot isn't reading the newly updated e-mails from the mbsync.
I have tried running doveadm -D -v force-resync -u ema...@example.com
"Inbox/Public/Other Users/us...@example.com"

Plese help if possible. Highly appreciate if someone could help here...

Below is the debug log :-

Debug: Loading modules from directory: /usr/lib64/dovecot
Debug: Module loaded: /usr/lib64/dovecot/lib01_acl_plugin.so
Debug: Module loaded: /usr/lib64/dovecot/lib10_quota_plugin.so
Debug: Loading modules from directory: /usr/lib64/dovecot/doveadm
Debug: Module loaded: /usr/lib64/dovecot/doveadm/lib10_doveadm_acl_plugin.so
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/lib10_doveadm_expire_plugin.so: undefined
symbol: expire_set_deinit (this is usually intentional, so just ignore this
message)
Debug: Module loaded:
/usr/lib64/dovecot/doveadm/lib10_doveadm_quota_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined
symbol: lucene_index_iter_deinit (this is usually intentional, so just
ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol:
fts_user_get_language_list (this is usually intentional, so just ignore
this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed:
/usr/lib64/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: undefined
symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so
just ignore this message)
doveadm(ema...@example.com)<20402><>: Debug: auth USER input:
ema...@example.com uid=5000 gid=5000 home=/var/mail/vhosts/
example.com/email1
doveadm(ema...@example.com): Debug: Effective uid=5000, gid=5000,
home=/var/mail/vhosts/example.com/email1
doveadm(ema...@example.com): Debug: quota: No quota setting - plugin
disabled
doveadm(ema...@example.com): Debug: Namespace inbox: type=private,
prefix=INBOX/, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes
location=maildir:~/Maildir
doveadm(ema...@example.com): Debug: maildir++: root=/var/mail/vhosts/
example.com/email1/Maildir, index=, indexpvt=, control=,
inbox=/var/mail/vhosts/example.com/email1/Maildir, alt=
doveadm(ema...@example.com): Debug: acl: initializing backend with data:
vfile
doveadm(ema...@example.com): Debug: acl: acl username = ema...@example.com
doveadm(ema...@example.com): Debug: acl: owner = 1
doveadm(ema...@example.com): Debug: acl vfile: Global ACLs disabled
doveadm(ema...@example.com): Debug: Namespace : type=public,
prefix=Public/, sep=/, inbox=no, hidden=no, list=children,
subscriptions=yes
location=maildir:/mnt/5ea86261-cfae-4e17-b21f-3a6c1a03d795/Email/fastmail:LAYOUT=fs
doveadm(ema...@example.com): Debug: fs:
root=/mnt/5ea86261-cfae-4e17-b21f-3a6c1a03d795/Email/fastmail, index=,
indexpvt=, control=, inbox=, alt=
doveadm(ema...@example.com): Debug: acl: initializing backend with data:
vfile
doveadm(ema...@example.com): Debug: acl: acl username = ema...@example.com
doveadm(ema...@example.com): Debug: acl: owner = 0
doveadm(ema...@example.com): Debug: acl vfile: Global ACLs disabled
doveadm(ema...@example.com): Debug: Namespace : type=private, prefix=,
sep=, inbox=no, hidden=yes, list=no, subscriptions=no
location=fail::LAYOUT=none
doveadm(ema...@example.com): Debug: none: root=, index=, indexpvt=,
control=, inbox=, alt=
doveadm(ema...@example.com): Debug: acl vfile: file /var/mail/vhosts/
example.com/email1/Maildir/dovecot-acl not found


Doveconf output :-


# 2.3.5 (513208660): /etc/dovecot/dovecot.conf
# OS: Linux 5.0.7-200.fc29.x86_64 x86_64 Fedora release 29 (Twenty Nine)
# Hostname: machine
auth_mechanisms = plain login
mail_debug = yes
mail_gid = vmail
mail_location = maildir:~/Maildir
mail_plugins = acl quota
mail_privileged_group = mail
mail_uid = vmail
mbox_write_locks = fcntl
namespace {
  list = children
  location =
maildir:/mnt/5ea86261-cfae-4e17-b21f-3a6c1a03d795/Email/fastmail:LAYOUT=fs
  prefix = Public/
  separator = /
  subscriptions = yes
  type = public
}
namespace inbox {
  inbox = yes
  list = yes
  location =
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix = INBOX/
  separator = /
}
passdb {
  driver = pam
}
passdb {
  args = /etc/dovecot/dovecot-sql.conf.ext
  driver = sql
}
plugin {
  acl = vfile
  acl_shared_dict = file:/var/mail/vhosts/example.com/Sharedbox
}
postmaster_address = postmaster at example.com
protocols = imap pop3
service auth-worker {
  user = vmail
}
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0666
user = postfix
  }
  unix_listener auth-userdb {
mode = 0600
user = vmail
  }
  us

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot


for instance, if I do a search from roundcube, the inbo name is NOT
passed to the backend (which is normal) 


the same search from the command line add the mailbox name ADDITIONALLY
to the mailbox * pointer 


However, passing a search from roudcube ask TWICE the backend  (first
with AND flag, second with OR flag) 


THis is obviously a clear bug form the part calling the backend (even if
the backend may need improvements ! this is really not the point here) 


Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Get last UID of Sent =
61714
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Get last UID of Sent =
61714
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: FLAG=AND
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(1/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(2/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(3/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(4/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(5/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_OR
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Testing if wildcard
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: set GLOBAL (no
specified header)
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query : ( bcc:milao OR
body:milao OR cc:milao OR from:milao OR message-id:milao OR
subject:milao OR to:milao )
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: 0 results in 0 ms
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: FLAG=OR
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(1): add
term(SUBJECT) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(2): add term(TO) :
milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(3): add term(FROM)
: milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(4): add term(CC) :
milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(5): add term(BCC) :
milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Testing if wildcard
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query : ( bcc:milao ) OR
( cc:milao ) OR ( from:milao ) OR ( subject:milao ) OR ( to:milao )
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: 0 results in 0 ms

On 2019-04-21 11:56, Joan Moreau via dovecot wrote:

Timo, 

A little of logic here : 

1 - the mailbox is passed by dovecot to the backend as a mailbox * pointer  , NOT as a search parameter. 

-> It works properly when entering a search from roundcube or evolution for instance. 

-> therefore this is a clear bug of the command line 

2 - the loop : Actually, the timeout occurs because the dovecot core is DISCARDING the results of the backend and do its own search (ie. in my example , it search fo "milan" in my inbox , which is huge , without even considering the backend results 


-> This is a enormous error.

On 2019-04-21 11:29, Timo Sirainen wrote: It's because you're misunderstanding how the lookup() function works. It gets ALL the search parameters, including the "mailbox inbox". This is intentional, and not a bug. Two reasons being: 

1) The FTS plugin in theory could support indexing/searching any kinds of searches, not just regular word searches. So I didn't want to limit it unnecessarily. 

2) Especially with "mailbox inbox" this is important when searching from virtual mailboxes. If you configure "All mails in all folders" virtual mailbox, you can do a search in there that restricts which physical mailboxes are matc

Re: maildirlock fails

2019-04-21 Thread Martynas Bendorius via dovecot

Any news on this? :) Maildirlock listed in official documentation, compression 
section at https://wiki.dovecot.org/Plugins/Zlib.

--
Best regards,
Martynas Bendorius

> On 15 Apr 2019, at 17:37, Martynas Bendorius  wrote:
> 
> It seems something like this has been reported already, however, I was unable 
> to find any changelog entries or patches: 
> http://dovecot.2317879.n4.nabble.com/maildirlock-time-unit-td65122.html
> 
> --
> Best regards,
> Martynas Bendorius
> 
> 
>> On 12 Apr 2019, at 01:08, Martynas Bendorius  wrote:
>> 
>> Hello,
>> 
>> Maildirlock seems to panic on locking:
>> [root@centos7 home]# /usr/libexec/dovecot/maildirlock 
>> "/home/user/imap/domain.com/email/Maildir" 10
>> Panic: BUG: No IOs or timeouts set. Not waiting for infinity.
>> Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xd90ee) 
>> [0x7f5bf02f10ee] -> /usr/lib/dovecot/libdovecot.so.0(+0xd9131) 
>> [0x7f5bf02f1131] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) 
>> [0x7f5bf0256efd] -> /usr/lib/dovecot/libdovecot.so.0(+0xf078c) 
>> [0x7f5bf030878c] -> 
>> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x36) 
>> [0x7f5bf030b1f6] -> 
>> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x56) [0x7f5bf03099c6] 
>> -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f5bf0309be8] -> 
>> /usr/libexec/dovecot/maildirlock(main+0x24a) [0x558ae0e032ca] -> 
>> /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5befe6d3d5] -> 
>> /usr/libexec/dovecot/maildirlock(+0x13a2) [0x558ae0e033a2]
>> 
>> Is this a known issue?
>> 
>> Thank you!
>> 
>> --
>> Best regards,
>> Martynas Bendorius
>> 
>> 
>

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot

Timo,

A little of logic here :

1 - the mailbox is passed by dovecot to the backend as a mailbox *
pointer , NOT as a search parameter.

-> It works properly when entering a search from roundcube or evolution
for instance.

-> therefore this is a clear bug of the command line

2 - the loop : Actually, the timeout occurs because the dovecot core is
DISCARDING the results of the backend and do its own search (ie. in my
example , it search fo "milan" in my inbox , which is huge , without
even considering the backend results

-> This is a enormous error.

On 2019-04-21 11:29, Timo Sirainen wrote:

It's because you're misunderstanding how the lookup() function works. It gets ALL the search parameters, including the "mailbox inbox". This is intentional, and not a bug. Two reasons being:

1) The FTS plugin in theory could support indexing/searching any kinds of searches, not just regular word searches. So I didn't want to limit it unnecessarily.

2) Especially with "mailbox inbox" this is important when searching from virtual mailboxes. If you configure "All mails in all folders" virtual mailbox, you can do a search in there that restricts which physical mailboxes are matched. In this case the FTS backend can optimize this lookup so it can filter only the physical mailboxes that have matches, leaving the others out. And it can do this in a single query if all the mailboxes are in the same FTS index.

So again: Your lookup() function needs to be changed to only use those search args that it really wants to search, and ignore the others. Use solr_add_definite_query_args() as the template.

Also I see now the reason for the timeout problem. It's because you're not setting search_arg->match_always=TRUE. These need to be set for the search args that you're actually using to generate the Xapian query. If it's not set, then Dovecot core doesn't think that the arg was part of the FTS search and it processes it itself. Meaning that it opens all the emails and does the search the slow way, practically making the FTS lookup ignored.

On 21 Apr 2019, at 19.50, Joan Moreau wrote:

No, the parsing is made by dovecot core, that is nothing the backend can do about it. The backend shall *never* reveive this. (would it be buggy or no)

PLease, have a look deeper

And the loop is a very big problem as it times out all the time (and once
again, this is not in any of the backend functions)

On 2019-04-21 10:42, Timo Sirainen via dovecot wrote:
Inbox appears in the list of arguments, because fts_backend_xapian_lookup() is parsing the search args wrong. Not sure about the other issue.

On 21 Apr 2019, at 19.31, Joan Moreau wrote:

For this first point, the problem is that dovecot core sends TWICE the request and "Inbox" appears in the list of arguments ! (inbox shall serve to select teh right mailbox, never sent to the backend)

And even if this would be solved, the dovecot core loops *after* the backend hs returneds the results

# doveadm search -u j...@grosjo.net mailbox inbox text milan
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND (
bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR
subject:milan OR to:milan )
DOVEADM(j...@grosjo.net): INFO: QUERY: 2 RESULTS IN 1 MS // THIS IS WHEN
BACKEND HAS FOUND RESULTS AND STOPPED
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
D82B4B0F550D3859364495331209 3121 -> LOOPING FOREVER

On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:
On 3 Apr 2019, at 20.30, Joan Moreau via dovecot wrote: doveadm search -u j...@grosjo.net mailbox inbox text milan

output

doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox )
AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR
subject:milan OR to:milan OR uid:milan )

1 - The query is wrong
That's because fts_backend_xapian_lookup() isn't anywhere close to being correct. Try to copy the logic based on solr_add_definite_query_args().

Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot

It's because you're misunderstanding how the lookup() function works. It gets 
ALL the search parameters, including the "mailbox inbox". This is intentional, 
and not a bug. Two reasons being:

1) The FTS plugin in theory could support indexing/searching any kinds of 
searches, not just regular word searches. So I didn't want to limit it 
unnecessarily.

2) Especially with "mailbox inbox" this is important when searching from 
virtual mailboxes. If you configure "All mails in all folders" virtual mailbox, 
you can do a search in there that restricts which physical mailboxes are 
matched. In this case the FTS backend can optimize this lookup so it can filter 
only the physical mailboxes that have matches, leaving the others out. And it 
can do this in a single query if all the mailboxes are in the same FTS index.

So again: Your lookup() function needs to be changed to only use those search 
args that it really wants to search, and ignore the others. Use 
solr_add_definite_query_args() as the template.

Also I see now the reason for the timeout problem. It's because you're not 
setting search_arg->match_always=TRUE. These need to be set for the search args 
that you're actually using to generate the Xapian query. If it's not set, then 
Dovecot core doesn't think that the arg was part of the FTS search and it 
processes it itself. Meaning that it opens all the emails and does the search 
the slow way, practically making the FTS lookup ignored.

> On 21 Apr 2019, at 19.50, Joan Moreau  wrote:
> 
> No, the parsing is made by dovecot core, that is nothing the backend can do 
> about it. The backend shall *never*  reveive this. (would it be buggy or no)
> 
> 
> 
> PLease, have a look deeper
> 
> And the loop is a very big problem as it times out all the time (and once 
> again, this is not in any of the backend  functions)
> 
>  
> 
> 
> On 2019-04-21 10:42, Timo Sirainen via dovecot wrote:
> 
>> Inbox appears in the list of arguments, because fts_backend_xapian_lookup() 
>> is parsing the search args wrong. Not sure about the other issue.
>> 
>>> On 21 Apr 2019, at 19.31, Joan Moreau >> > wrote:
>>> 
>>> For this first point, the problem is that dovecot core sends TWICE the 
>>> request and "Inbox" appears in the list of arguments ! (inbox shall serve 
>>> to select teh right mailbox, never sent to the backend)
>>> 
>>> And even if this would be solved, the dovecot core loops *after* the 
>>> backend hs returneds the results
>>> 
>>> 
>>> 
>>> # doveadm search -u j...@grosjo.net  mailbox inbox 
>>> text milan
>>> doveadm(j...@grosjo.net ): Info: Get last UID of 
>>> INBOX = 315526
>>> doveadm(j...@grosjo.net ): Info: Get last UID of 
>>> INBOX = 315526
>>> doveadm(j...@grosjo.net ): Info: Query: FLAG=AND
>>> doveadm(j...@grosjo.net ): Info: Query(1): add 
>>> term(wilcard) : inbox
>>> doveadm(j...@grosjo.net ): Info: Query(2): add 
>>> term(wilcard) : milan
>>> doveadm(j...@grosjo.net ): Info: Testing if wildcard
>>> doveadm(j...@grosjo.net ): Info: Query: set GLOBAL 
>>> (no specified header)
>>> doveadm(j...@grosjo.net ): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox ) AND ( bcc:milan OR body:milan OR cc:milan OR 
>>> from:milan OR message-id:milan OR subject:milan OR to:milan )
>>> doveadm(j...@grosjo.net ): Info: Query: 2 results 
>>> in 1 ms // THIS IS WHEN BACKEND HAS FOUND RESULTS AND STOPPED
>>> d82b4b0f550d3859364495331209 847
>>> d82b4b0f550d3859364495331209 1569
>>> d82b4b0f550d3859364495331209 2260
>>> d82b4b0f550d3859364495331209 2575
>>> d82b4b0f550d3859364495331209 2811
>>> d82b4b0f550d3859364495331209 2885
>>> d82b4b0f550d3859364495331209 3038
>>> d82b4b0f550d3859364495331209 3121 -> LOOPING FOREVER
>>> 
>>> 
>>> 
>>>  
>>> 
>>> 
>>> On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:
>>> 
>>> On 3 Apr 2019, at 20.30, Joan Moreau via dovecot >> > wrote:
>>> doveadm search -u j...@grosjo.net  mailbox inbox 
>>> text milan
>>> output
>>> 
>>> doveadm(j...@grosjo.net ): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox OR uid:inbox ) AND ( bcc:milan OR body:milan OR 
>>> cc:milan OR from:milan OR message-id:milan OR subject:milan OR to:milan OR 
>>> uid:milan )
>>> 
>>> 1 - The query is wrong
>>> 
>>> That's because fts_backend_xapian_lookup() isn't anywhere close to being 
>>> correct. Try to copy the logic based on solr_add_definite_query_args().
>>> 
>>>

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot


No, the parsing is made by dovecot core, that is nothing the backend can
do about it. The backend shall *never*  reveive this. (would it be buggy
or no) 

PLease, have a look deeper 


And the loop is a very big problem as it times out all the time (and
once again, this is not in any of the backend  functions)

On 2019-04-21 10:42, Timo Sirainen via dovecot wrote:

Inbox appears in the list of arguments, because fts_backend_xapian_lookup() is parsing the search args wrong. Not sure about the other issue. 

On 21 Apr 2019, at 19.31, Joan Moreau  wrote: 

For this first point, the problem is that dovecot core sends TWICE the request and "Inbox" appears in the list of arguments ! (inbox shall serve to select teh right mailbox, never sent to the backend) 

And even if this would be solved, the dovecot core loops *after* the backend hs returneds the results 


# doveadm search -u j...@grosjo.net mailbox inbox text milan
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND ( 
bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan )
DOVEADM(j...@grosjo.net): INFO: QUERY: 2 RESULTS IN 1 MS // THIS IS WHEN 
BACKEND HAS FOUND RESULTS AND STOPPED
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
D82B4B0F550D3859364495331209 3121 -> LOOPING FOREVER 

On 2019-04-21 09:57, Timo Sirainen via dovecot wrote: 
On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote: doveadm search -u j...@grosjo.net mailbox inbox text milan

output

doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan OR uid:milan )

1 - The query is wrong 
That's because fts_backend_xapian_lookup() isn't anywhere close to being correct. Try to copy the logic based on solr_add_definite_query_args().

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot


Antoher example so you understand how may understand the bug in dovecote
core : 

# doveadm search -u j...@grosjo.net mailbox SENT text milan 


doveadm(j...@grosjo.net): Info: Get last UID of Sent = 61707 -> CORRECTLY
ASSIGNED THE PROPER MAILBOX TO THE BACK END
doveadm(j...@grosjo.net): Info: Get last UID of Sent = 61707
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : Sent -> WHY
IS "SENT" AMONG THE SERACH PARAMETERS ???
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:milan OR body:milan OR
cc:milan OR from:milan OR message-id:milan OR subject:milan OR to:milan
) AND ( bcc:sent OR body:sent OR cc:sent OR from:sent OR message-id:sent
OR subject:sent OR to:sent )
doveadm(j...@grosjo.net): Info: Query: 7 results in 71 ms 

(AND SAME LOOP) 


In this example, the "Sent" shall *never*  be passed as argument to the
backend (xapian, solr or any other), only the mailbox reference.
However, it appears in the search parameters 


On 2019-04-21 10:31, Joan Moreau via dovecot wrote:

For this first point, the problem is that dovecot core sends TWICE the request and "Inbox" appears in the list of arguments ! (inbox shall serve to select teh right mailbox, never sent to the backend) 

And even if this would be solved, the dovecot core loops *after* the backend hs returneds the results 


# doveadm search -u j...@grosjo.net mailbox inbox text milan
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND ( 
bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan )
DOVEADM(j...@grosjo.net): INFO: QUERY: 2 RESULTS IN 1 MS // THIS IS WHEN 
BACKEND HAS FOUND RESULTS AND STOPPED
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
D82B4B0F550D3859364495331209 3121 -> LOOPING FOREVER 

On 2019-04-21 09:57, Timo Sirainen via dovecot wrote: 
On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote: doveadm search -u j...@grosjo.net mailbox inbox text milan

output

doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan OR uid:milan )

1 - The query is wrong 
That's because fts_backend_xapian_lookup() isn't anywhere close to being correct. Try to copy the logic based on solr_add_definite_query_args().

Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot

Inbox appears in the list of arguments, because fts_backend_xapian_lookup() is 
parsing the search args wrong. Not sure about the other issue.

> On 21 Apr 2019, at 19.31, Joan Moreau  wrote:
> 
> For this first point, the problem is that dovecot core sends TWICE the 
> request and "Inbox" appears in the list of arguments ! (inbox shall serve to 
> select teh right mailbox, never sent to the backend)
> 
> And even if this would be solved, the dovecot core loops *after* the backend 
> hs returneds the results
> 
> 
> 
> # doveadm search -u j...@grosjo.net mailbox inbox text milan
> doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
> doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
> doveadm(j...@grosjo.net): Info: Query: FLAG=AND
> doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
> doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
> doveadm(j...@grosjo.net): Info: Testing if wildcard
> doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
> doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
> OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND ( 
> bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
> subject:milan OR to:milan )
> doveadm(j...@grosjo.net): Info: Query: 2 results in 1 ms // THIS IS WHEN 
> BACKEND HAS FOUND RESULTS AND STOPPED
> d82b4b0f550d3859364495331209 847
> d82b4b0f550d3859364495331209 1569
> d82b4b0f550d3859364495331209 2260
> d82b4b0f550d3859364495331209 2575
> d82b4b0f550d3859364495331209 2811
> d82b4b0f550d3859364495331209 2885
> d82b4b0f550d3859364495331209 3038
> d82b4b0f550d3859364495331209 3121 -> LOOPING FOREVER
> 
> 
> 
>  
> 
> 
> On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:
> 
>> On 3 Apr 2019, at 20.30, Joan Moreau via dovecot > > wrote:
>>> 
>>> doveadm search -u j...@grosjo.net  mailbox inbox 
>>> text milan
>>> output
>>> 
>>> doveadm(j...@grosjo.net ): Info: Query : ( 
>>> bcc:inbox OR body:inbox OR cc:inbox OR from:inbox OR message-id:inbox OR 
>>> subject:inbox OR to:inbox OR uid:inbox ) AND ( bcc:milan OR body:milan OR 
>>> cc:milan OR from:milan OR message-id:milan OR subject:milan OR to:milan OR 
>>> uid:milan )
>>> 
>>> 1 - The query is wrong
>> 
>> That's because fts_backend_xapian_lookup() isn't anywhere close to being 
>> correct. Try to copy the logic based on solr_add_definite_query_args().
>> 
>>

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot


For this first point, the problem is that dovecot core sends TWICE the
request and "Inbox" appears in the list of arguments ! (inbox shall
serve to select teh right mailbox, never sent to the backend) 


And even if this would be solved, the dovecot core loops *after* the
backend hs returneds the results 


# doveadm search -u j...@grosjo.net mailbox inbox text milan
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR
cc:inbox OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox
) AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR
message-id:milan OR subject:milan OR to:milan )
DOVEADM(j...@grosjo.net): INFO: QUERY: 2 RESULTS IN 1 MS // THIS IS WHEN
BACKEND HAS FOUND RESULTS AND STOPPED
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
D82B4B0F550D3859364495331209 3121 -> LOOPING FOREVER 


On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:

On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote: 


doveadm search -u j...@grosjo.net mailbox inbox text milan
output

doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan OR uid:milan )

1 - The query is wrong


That's because fts_backend_xapian_lookup() isn't anywhere close to being 
correct. Try to copy the logic based on solr_add_definite_query_args().

Re: FTS delays

2019-04-21 Thread Timo Sirainen via dovecot

On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote:
> doveadm search -u j...@grosjo.net mailbox inbox text milan
> output
> 
> doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
> OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
> AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan 
> OR subject:milan OR to:milan OR uid:milan )
> 
> 1 - The query is wrong

That's because fts_backend_xapian_lookup() isn't anywhere close to being 
correct. Try to copy the logic based on solr_add_definite_query_args().

Re: mbsync updating new emails but dovecot not reading them ? [2.3.3]

mbsync updating new emails but dovecot not reading them ? [2.3.3]

Re: FTS delays

Re: maildirlock fails

Re: FTS delays

Re: FTS delays

Re: FTS delays

Re: FTS delays

Re: FTS delays

Re: FTS delays

Re: FTS delays

11 matches

Site Navigation

Mail list logo

Footer information