from:"Joan Moreau via dovecot"

Re: Sieve not getting recompiled

2024-04-20 Thread Joan Moreau via dovecot


I changed it to the following to stick to the doc

   sieve = file:/mails/%d/%n/sieve/
sieve_after = file:/mails/sieve/after.sieve
sieve_default = file:/mails/sieve/before.sieve
sieve_before = file:/mails/sieve/before.sieve

Still no scripts are compiled/executed (and it was working fine before 
!)


On 2024-04-21 09:21, Joan Moreau wrote:


Hi

I have

sieve = /mails/%d/%n/sieve/roundcube.sieve
sieve_after = /mails/sieve/after.sieve
sieve_before = /mails/sieve/before.sieve
sieve_dir = /mails/%d/%n/sieve/
sieve_global_dir = /mails/sieve/

But sieve scripts are not compiled and not executed

It was working until I removed the setting "sieve_global_path"

Something I dont understand ?

Thank you

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Sieve not getting recompiled

2024-04-20 Thread Joan Moreau via dovecot


Hi

I have

sieve = /mails/%d/%n/sieve/roundcube.sieve
sieve_after = /mails/sieve/after.sieve
sieve_before = /mails/sieve/before.sieve
sieve_dir = /mails/%d/%n/sieve/
sieve_global_dir = /mails/sieve/

But sieve scripts are not compiled and not executed

It was working until I removed the setting "sieve_global_path"

Something I dont understand ?

Thank you
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: exfat not supported ?

2024-04-20 Thread Joan Moreau via dovecot


That resolve the fisrt bug

but I get now :

Error: link(/xxx/dovecot.list.index.log, /xxx/dovecot.list.index.log.2) 
failed: Operation not permitted


On 2024-04-21 02:02, Aki Tuomi via dovecot wrote:


Try setting lock_method = dotlock

Aki
On 20/04/2024 15:32 EEST Joan Moreau via dovecot
 wrote:

I tried and get the following:

Error: Couldn't create mailbox list lock /xxx/mailboxes.lock:
file_create_locked(/xxx/mailboxes.lock) failed:
link(/xxx/mailboxes.locka94f3757318b0b90, /xxx/mailboxes.lock)
failed:
Operation not permitted

On 2024-04-20 17:39, Aki Tuomi via dovecot wrote:

On 20/04/2024 12:27 EEST Joan Moreau via dovecot
 wrote:

Hi

Would placing my storage on a exfat partition work ? If no,
why ?

Thank you
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

I can't see any reason why not. As long as it behaves like
POSIX
filesystem.

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: exfat not supported ?

2024-04-20 Thread Joan Moreau via dovecot


I tried and get the following:

Error: Couldn't create mailbox list lock /xxx/mailboxes.lock: 
file_create_locked(/xxx/mailboxes.lock) failed: 
link(/xxx/mailboxes.locka94f3757318b0b90, /xxx/mailboxes.lock) failed: 
Operation not permitted


On 2024-04-20 17:39, Aki Tuomi via dovecot wrote:


On 20/04/2024 12:27 EEST Joan Moreau via dovecot
 wrote:

Hi

Would placing my storage on a exfat partition work ? If no, why ?

Thank you
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

I can't see any reason why not. As long as it behaves like POSIX 
filesystem.


Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

thread->detach() creates confusion of dovecot

2024-04-20 Thread Joan Moreau via dovecot


Hi

When I try to "detach" 
(https://en.cppreference.com/w/cpp/thread/thread/detach) a thread 
running inside a plugin, it seems the core dovecot has some influence on 
that , tries to close this for some unknown reason and usually ends up 
crashing


What is the cause of this ?

Thank you
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

exfat not supported ?

2024-04-20 Thread Joan Moreau via dovecot


Hi

Would placing my storage on a exfat partition work ? If no, why ?

Thank you
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [EXT] Re: How to get a memory pointer in the core process

2024-03-14 Thread Joan Moreau via dovecot

Thanks Eduardo

I am trying to avoid closing/ reopening a file pointer to the exact same file
between each call to the plugin



On 14 March 2024 20:08:37 Eduardo M KALINOWSKI via dovecot
 wrote:

 On 14/03/2024 02:49, Joan Moreau via dovecot wrote:
  No, you don´t understand
  There is a core process (/usr/bin/dovecot) running all the
  time. So I want to
  allocate a memory block, the core process keep it and it is
  retrievable by the
  pluging when laded again
  At exit of /usr/bin/dovecot, it just does a "delete()" of
  the said allocation

 While I cannot help you with plugin writing or dovecot internals,
 this 
 does seem like an example of the XY problem[0]. Perhaps if you
 provide a 
 high level description of what you're attempting to do someone might 
 come up with a way to achieve that.

 [0] https://en.wikipedia.org/wiki/XY_problem

 -- 
 Eduardo M KALINOWSKI
 edua...@kalinowski.com.br

 ___
 dovecot mailing list -- dovecot@dovecot.org
 To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [EXT] Re: How to get a memory pointer in the core process

2024-03-13 Thread Joan Moreau via dovecot

No, you don´t understand
There is a core process (/usr/bin/dovecot) running all the time. So I want to
allocate a memory block, the core process keep it and it is retrievable by the
pluging when laded again
At exit of /usr/bin/dovecot, it just does a "delete()" of the said allocation


On 2024-03-14 13:25, Aki Tuomi via dovecot wrote:
 Hi!

 Sorry but that's just not possible, ther is no "core" where to create
 such object. There is no "dovecot" where to store things.

 When user logs in, dovecot executes /usr/libexec/dovecot/imap and
 transfers the connection fd there. then plugins and stuff are loaded,
 and the user does what he does, and then plugins and stuff are
 unloaded and process exists and no longer exists in memory.

 You are clearly asking about memory persistence between sessions, and
 this can be done with

 a) services (internal or external), such as redis, sql, or something
 else
 b) storing things to disk

 Aki

  On 13/03/2024 18:45 EET Joan Moreau via dovecot
   wrote:

   
  No, I am not referring to that

  I want to create an object at first call in memory

  that object would be retrievable at second and furthers
  calls of the
  plugin, as long as dovecot is running

  On 2024-03-13 16:29, Aki Tuomi via dovecot wrote:

   Not really no. You should use e.g. dict inteface
   for storing this kind
   of stateful data. When deinit is called the
   calling core process will
   likely die too.

   Aki

   On 13/03/2024 10:19 EET Joan Moreau
wrote:

   Keep a pointer in memory retrievable each time a
   plugin is called

   So the plugin keep the memory, not has to restart
   everything at each
   call

   On 12 March 2024 08:53:38 Aki Tuomi via dovecot
   
   wrote:

   On 11/03/2024 10:42 EET Joan Moreau
wrote:

   Hi
   Is it possible, from a plugin perspective, to
   create and recover a
   pointer in the core process (i.e. memory not lost
   between 2 calls to
   the plugin, even after the "deinit" of the
   plugin" ) ?

   Thanks
   Hi Joan!

   May I ask what you are attempting to achieve in
   more detail?

   Aki
   ___
   dovecot mailing list -- dovecot@dovecot.org
   To unsubscribe send an email to dovecot-
   le...@dovecot.org
    ___
  dovecot mailing list -- dovecot@dovecot.org
  To unsubscribe send an email to dovecot-
  leave@dovecot.orgNo, I am not referring to that
  I want to create an object at first call in memory
  that object would be retrievable at second and furthers
  calls of the plugin, as
  long as dovecot is running




  On 2024-03-13 16:29, Aki Tuomi via dovecot wrote:
       Not really no. You should use e.g. dict inteface for
  storing this
       kind of stateful data. When deinit is called the
  calling core process
       will likely die too.

       Aki

            On 13/03/2024 10:19 EET Joan Moreau
   wrote:


            Keep a pointer in memory retrievable each time a
  plugin is
            called

            So the plugin keep the memory, not has to restart
            everything at each call



            On 12 March 2024 08:53:38 Aki Tuomi via dovecot
             wrote:

                      On 11/03/2024 10:42 EET Joan Moreau
                       wrote:


                      Hi
                      Is it possible, from a plugin
                      perspective, to create and recover a
                      pointer in the core process (i.e.
                      memory not lost between 2 calls to the
                      plugin, even after the "deinit" of the
                      plugin" ) ?

                      Thanks

                 Hi Joan!

                 May I ask what you are attempting to achieve
  in
                 more detail?

                 Aki
               
   ___
                 dovecot mailing list -- dovecot@dovecot.org
                 To unsubscribe send an email to dovecot-
                 le...@dovecot.org
       __

Re: [EXT] Re: How to get a memory pointer in the core process

2024-03-13 Thread Joan Moreau via dovecot

No, I am not referring to that
I want to create an object at first call in memory
that object would be retrievable at second and furthers calls of the plugin, as
long as dovecot is running




On 2024-03-13 16:29, Aki Tuomi via dovecot wrote:
 Not really no. You should use e.g. dict inteface for storing this
 kind of stateful data. When deinit is called the calling core process
 will likely die too.

 Aki

  On 13/03/2024 10:19 EET Joan Moreau  wrote:


  Keep a pointer in memory retrievable each time a plugin is
  called

  So the plugin keep the memory, not has to restart
  everything at each call



  On 12 March 2024 08:53:38 Aki Tuomi via dovecot
   wrote:

On 11/03/2024 10:42 EET Joan Moreau
 wrote:


Hi
Is it possible, from a plugin
perspective, to create and recover a
pointer in the core process (i.e.
memory not lost between 2 calls to the
plugin, even after the "deinit" of the
plugin" ) ?

Thanks

   Hi Joan!

   May I ask what you are attempting to achieve in
   more detail?

   Aki
   ___
   dovecot mailing list -- dovecot@dovecot.org
   To unsubscribe send an email to dovecot-
   le...@dovecot.org
 ___
 dovecot mailing list -- dovecot@dovecot.org
 To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: How to get a memory pointer in the core process

2024-03-13 Thread Joan Moreau via dovecot

Keep a pointer in memory retrievable each time a plugin is called

So the plugin keep the memory, not has to restart everything at each call



On 12 March 2024 08:53:38 Aki Tuomi via dovecot  wrote:

  On 11/03/2024 10:42 EET Joan Moreau  wrote:


  Hi
  Is it possible, from a plugin perspective, to create and
  recover a pointer in the core process (i.e. memory not lost
  between 2 calls to the plugin, even after the "deinit" of
  the plugin" ) ?

  Thanks

 Hi Joan!

 May I ask what you are attempting to achieve in more detail?

 Aki
 ___
 dovecot mailing list -- dovecot@dovecot.org
 To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

How to get a memory pointer in the core process

2024-03-11 Thread Joan Moreau via dovecot

Hi
Is it possible, from a plugin perspective, to create and recover a pointer in
the core process (i.e. memory not lost between 2 calls to the plugin, even
after the "deinit" of the plugin" ) ?

Thanks
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: FTS Xapian -> FTS core issue

2019-06-09 Thread Joan Moreau via dovecot

The issue is not in plug-in then. 


Maybe Aki or Timo knows where does this bug come from ?

On 2019-06-07 14:09, Daniel Miller wrote:

Yes, latest git version.  

The logs show (as I read them) returned results - yet nothing shows in the client. The logs look the same (with different numbers) when querying "regular" folders - but results are shown in clients.  


--
Daniel 

On June 6, 2019 12:16:08 AM Joan Moreau  wrote: 

Hi 

Are you using the latest git version ? 

WHich part exactly of your logs relates to "virtual folders do not work" ? 

On 2019-06-05 13:08, Daniel Miller via dovecot wrote: 
Logs:


Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2857830c70c844e2f1d3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 1 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_78544714f3f1ae5b9b0d3bda95b5
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 53 results in 40 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_bdcb8e2172fadf4db50b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 12 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_be25c00241fedf4de00b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 3 results in 32 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_a7e75820d9fadf4dd90b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 11 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6fa78f2738cbdf4d007b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 21 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6ea78f2738cbdf4d007b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 1 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2c3522c5d9b9d4f8847e130c744
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot:

Re: FTS Xapian

2019-06-09 Thread Joan Moreau via dovecot

Hi 

Are you using the latest git version ? 


WHich part exactly of your logs relates to "virtual folders do not work"
? 


On 2019-06-05 13:08, Daniel Miller via dovecot wrote:


Logs:

Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2857830c70c844e2f1d3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 1 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_78544714f3f1ae5b9b0d3bda95b5
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 53 results in 40 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_bdcb8e2172fadf4db50b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 12 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_be25c00241fedf4de00b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 3 results in 32 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_a7e75820d9fadf4dd90b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 11 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6fa78f2738cbdf4d007b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 21 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6ea78f2738cbdf4d007b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 0 results in 1 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2c3522c5d9b9d4f8847e130c744
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR 
body:"dovecot")
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: 43 results in 51 ms
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
Opening DB (RO) 
/var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_58c26f3b9085134fe04b3bc41c5f
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: 
FTS Xapian: FLAG=AND
Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR 
from:"dovecot" OR

Re: FTS Xapian

2019-06-04 Thread Joan Moreau via dovecot


Hi

Can you post your dovecot conf file and the subset of the log files related 
to the issue ?


thanks


On June 5, 2019 9:29:13 AM Daniel Miller via dovecot  
wrote:



For my primary namespace this is working fine - thanks to the developers!


It also appears to work great for shared folders as well.


But my virtual folders aren't returning results - at least not to the
client. The logs show FTS Xapian opening several DB files and getting
results - but nothing is being returned to client. Is this a config
issue on my side or is this a current limitation of the plugin?
--
Daniel

Further issues on FTS engine

2019-05-20 Thread Joan Moreau via dovecot

Hi, 


Additionally to the long list of problem on the FTS previously
discussed, here a new: 


WHen I reset the indexes, the indexer-worker seems paralelleizing the
indexing (which is good), however, the number available in "ps aux |
grep dove" shows that it does not move: 


dovecot 28549 0.0 0.0 8620 3920 ? S 06:20 0:00 dovecot/indexer [0
clients, 3 requests]
mailuse+ 28550 98.6 0.1 167412 86916 ? R 06:20 5:28
dovecot/indexer-worker [j...@grosjo.net  - 800/37755] 


Looking further, if I put a tracer in teh backend, it treats the *same*
message several times in parallel, and therefore does not move very fast
on the global indexing of the box 

ANy clue ? 


THanks

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot


for instance, if I do a search from roundcube, the inbo name is NOT
passed to the backend (which is normal) 


the same search from the command line add the mailbox name ADDITIONALLY
to the mailbox * pointer 


However, passing a search from roudcube ask TWICE the backend  (first
with AND flag, second with OR flag) 


THis is obviously a clear bug form the part calling the backend (even if
the backend may need improvements ! this is really not the point here) 


Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Get last UID of Sent =
61714
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Get last UID of Sent =
61714
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: FLAG=AND
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(1/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(2/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(3/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(4/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(5/1): add
term(wilcard) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_OR
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Testing if wildcard
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: set GLOBAL (no
specified header)
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query : ( bcc:milao OR
body:milao OR cc:milao OR from:milao OR message-id:milao OR
subject:milao OR to:milao )
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: 0 results in 0 ms
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: FLAG=OR
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(1): add
term(SUBJECT) : milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(2): add term(TO) :
milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(3): add term(FROM)
: milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(4): add term(CC) :
milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query(5): add term(BCC) :
milao
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: SEARCH_HEADER
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: MATCH NOT : 0
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Testing if wildcard
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query : ( bcc:milao ) OR
( cc:milao ) OR ( from:milao ) OR ( subject:milao ) OR ( to:milao )
Apr 21 11:08:39 gjserver dovecot[14251]:
imap(j...@grosjo.net)<15709>: Query: 0 results in 0 ms

On 2019-04-21 11:56, Joan Moreau via dovecot wrote:

Timo, 

A little of logic here : 

1 - the mailbox is passed by dovecot to the backend as a mailbox * pointer  , NOT as a search parameter. 

-> It works properly when entering a search from roundcube or evolution for instance. 

-> therefore this is a clear bug of the command line 

2 - the loop : Actually, the timeout occurs because the dovecot core is DISCARDING the results of the backend and do its own search (ie. in my example , it search fo "milan" in my inbox , which is huge , without even considering the backend results 


-> This is a enormous error.

On 2019-04-21 11:29, Timo Sirainen wrote: It's because you're misunderstanding how the lookup() function works. It gets ALL the search parameters, including the "mailbox inbox". This is intentional, and not a bug. Two reasons being: 

1) The FTS plugin in theory could support indexing/searching any kinds of searches, not just regular word searches. So I didn't want to limit it unnecessari

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot

Timo,

A little of logic here :

1 - the mailbox is passed by dovecot to the backend as a mailbox *
pointer , NOT as a search parameter.

-> It works properly when entering a search from roundcube or evolution
for instance.

-> therefore this is a clear bug of the command line

2 - the loop : Actually, the timeout occurs because the dovecot core is
DISCARDING the results of the backend and do its own search (ie. in my
example , it search fo "milan" in my inbox , which is huge , without
even considering the backend results

-> This is a enormous error.

On 2019-04-21 11:29, Timo Sirainen wrote:

It's because you're misunderstanding how the lookup() function works. It gets ALL the search parameters, including the "mailbox inbox". This is intentional, and not a bug. Two reasons being:

1) The FTS plugin in theory could support indexing/searching any kinds of searches, not just regular word searches. So I didn't want to limit it unnecessarily.

2) Especially with "mailbox inbox" this is important when searching from virtual mailboxes. If you configure "All mails in all folders" virtual mailbox, you can do a search in there that restricts which physical mailboxes are matched. In this case the FTS backend can optimize this lookup so it can filter only the physical mailboxes that have matches, leaving the others out. And it can do this in a single query if all the mailboxes are in the same FTS index.

So again: Your lookup() function needs to be changed to only use those search args that it really wants to search, and ignore the others. Use solr_add_definite_query_args() as the template.

Also I see now the reason for the timeout problem. It's because you're not setting search_arg->match_always=TRUE. These need to be set for the search args that you're actually using to generate the Xapian query. If it's not set, then Dovecot core doesn't think that the arg was part of the FTS search and it processes it itself. Meaning that it opens all the emails and does the search the slow way, practically making the FTS lookup ignored.

On 21 Apr 2019, at 19.50, Joan Moreau wrote:

No, the parsing is made by dovecot core, that is nothing the backend can do about it. The backend shall *never* reveive this. (would it be buggy or no)

PLease, have a look deeper

And the loop is a very big problem as it times out all the time (and once
again, this is not in any of the backend functions)

On 2019-04-21 10:42, Timo Sirainen via dovecot wrote:
Inbox appears in the list of arguments, because fts_backend_xapian_lookup() is parsing the search args wrong. Not sure about the other issue.

On 21 Apr 2019, at 19.31, Joan Moreau wrote:

For this first point, the problem is that dovecot core sends TWICE the request and "Inbox" appears in the list of arguments ! (inbox shall serve to select teh right mailbox, never sent to the backend)

And even if this would be solved, the dovecot core loops *after* the backend hs returneds the results

# doveadm search -u j...@grosjo.net mailbox inbox text milan
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND (
bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR
subject:milan OR to:milan )
DOVEADM(j...@grosjo.net): INFO: QUERY: 2 RESULTS IN 1 MS // THIS IS WHEN
BACKEND HAS FOUND RESULTS AND STOPPED
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
D82B4B0F550D3859364495331209 3121 -> LOOPING FOREVER

On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:
On 3 Apr 2019, at 20.30, Joan Moreau via dovecot wrote: doveadm search -u j...@grosjo.net mailbox inbox text milan

output

doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox )
AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR
subject:milan OR to:milan OR uid:milan )

1 - The query is wrong
That's because fts_backend_xapian_lookup() isn't anywhere close to being correct. Try to copy the logic based on solr_add_definite_query_args().

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot


No, the parsing is made by dovecot core, that is nothing the backend can
do about it. The backend shall *never*  reveive this. (would it be buggy
or no) 

PLease, have a look deeper 


And the loop is a very big problem as it times out all the time (and
once again, this is not in any of the backend  functions)

On 2019-04-21 10:42, Timo Sirainen via dovecot wrote:

Inbox appears in the list of arguments, because fts_backend_xapian_lookup() is parsing the search args wrong. Not sure about the other issue. 

On 21 Apr 2019, at 19.31, Joan Moreau  wrote: 

For this first point, the problem is that dovecot core sends TWICE the request and "Inbox" appears in the list of arguments ! (inbox shall serve to select teh right mailbox, never sent to the backend) 

And even if this would be solved, the dovecot core loops *after* the backend hs returneds the results 


# doveadm search -u j...@grosjo.net mailbox inbox text milan
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND ( 
bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan )
DOVEADM(j...@grosjo.net): INFO: QUERY: 2 RESULTS IN 1 MS // THIS IS WHEN 
BACKEND HAS FOUND RESULTS AND STOPPED
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
D82B4B0F550D3859364495331209 3121 -> LOOPING FOREVER 

On 2019-04-21 09:57, Timo Sirainen via dovecot wrote: 
On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote: doveadm search -u j...@grosjo.net mailbox inbox text milan

output

doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan OR uid:milan )

1 - The query is wrong 
That's because fts_backend_xapian_lookup() isn't anywhere close to being correct. Try to copy the logic based on solr_add_definite_query_args().

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot


Antoher example so you understand how may understand the bug in dovecote
core : 

# doveadm search -u j...@grosjo.net mailbox SENT text milan 


doveadm(j...@grosjo.net): Info: Get last UID of Sent = 61707 -> CORRECTLY
ASSIGNED THE PROPER MAILBOX TO THE BACK END
doveadm(j...@grosjo.net): Info: Get last UID of Sent = 61707
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : Sent -> WHY
IS "SENT" AMONG THE SERACH PARAMETERS ???
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:milan OR body:milan OR
cc:milan OR from:milan OR message-id:milan OR subject:milan OR to:milan
) AND ( bcc:sent OR body:sent OR cc:sent OR from:sent OR message-id:sent
OR subject:sent OR to:sent )
doveadm(j...@grosjo.net): Info: Query: 7 results in 71 ms 

(AND SAME LOOP) 


In this example, the "Sent" shall *never*  be passed as argument to the
backend (xapian, solr or any other), only the mailbox reference.
However, it appears in the search parameters 


On 2019-04-21 10:31, Joan Moreau via dovecot wrote:

For this first point, the problem is that dovecot core sends TWICE the request and "Inbox" appears in the list of arguments ! (inbox shall serve to select teh right mailbox, never sent to the backend) 

And even if this would be solved, the dovecot core loops *after* the backend hs returneds the results 


# doveadm search -u j...@grosjo.net mailbox inbox text milan
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox ) AND ( 
bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan )
DOVEADM(j...@grosjo.net): INFO: QUERY: 2 RESULTS IN 1 MS // THIS IS WHEN 
BACKEND HAS FOUND RESULTS AND STOPPED
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
D82B4B0F550D3859364495331209 3121 -> LOOPING FOREVER 

On 2019-04-21 09:57, Timo Sirainen via dovecot wrote: 
On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote: doveadm search -u j...@grosjo.net mailbox inbox text milan

output

doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan OR uid:milan )

1 - The query is wrong 
That's because fts_backend_xapian_lookup() isn't anywhere close to being correct. Try to copy the logic based on solr_add_definite_query_args().

Re: FTS delays

2019-04-21 Thread Joan Moreau via dovecot


For this first point, the problem is that dovecot core sends TWICE the
request and "Inbox" appears in the list of arguments ! (inbox shall
serve to select teh right mailbox, never sent to the backend) 


And even if this would be solved, the dovecot core loops *after* the
backend hs returneds the results 


# doveadm search -u j...@grosjo.net mailbox inbox text milan
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Get last UID of INBOX = 315526
doveadm(j...@grosjo.net): Info: Query: FLAG=AND
doveadm(j...@grosjo.net): Info: Query(1): add term(wilcard) : inbox
doveadm(j...@grosjo.net): Info: Query(2): add term(wilcard) : milan
doveadm(j...@grosjo.net): Info: Testing if wildcard
doveadm(j...@grosjo.net): Info: Query: set GLOBAL (no specified header)
doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR
cc:inbox OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox
) AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR
message-id:milan OR subject:milan OR to:milan )
DOVEADM(j...@grosjo.net): INFO: QUERY: 2 RESULTS IN 1 MS // THIS IS WHEN
BACKEND HAS FOUND RESULTS AND STOPPED
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
D82B4B0F550D3859364495331209 3121 -> LOOPING FOREVER 


On 2019-04-21 09:57, Timo Sirainen via dovecot wrote:

On 3 Apr 2019, at 20.30, Joan Moreau via dovecot  wrote: 


doveadm search -u j...@grosjo.net mailbox inbox text milan
output

doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR cc:inbox 
OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox OR uid:inbox ) 
AND ( bcc:milan OR body:milan OR cc:milan OR from:milan OR message-id:milan OR 
subject:milan OR to:milan OR uid:milan )

1 - The query is wrong


That's because fts_backend_xapian_lookup() isn't anywhere close to being 
correct. Try to copy the logic based on solr_add_definite_query_args().

Re: FTS delays

2019-04-20 Thread Joan Moreau via dovecot

I have no idea how to use git-bitsec

On 2019-04-15 15:31, Josef 'Jeff' Sipek wrote:

On Sun, Apr 14, 2019 at 21:09:54 +0800, Joan Moreau wrote:
...

THe "loop" part seems the most urgent : It breaks everything (search
timeout 100% of the time)

Any luck with git-bisect?

Jeff.

On 2019-04-06 09:56, Joan Moreau via dovecot wrote:

For the point 1, this is not "suboptimal", it is plain wrong (results are damn
wrong ! and this is not related to the backend, but the FTS logic in Dovecot core)

For the point 2 , this has been discussed already numerous times but without action. The dovecot core shall be the one re-submitting the emails to scan, not the backend to try to figure out where and which are the emails to be re-scaned

For the point 3, I will do a bit of research in the existing code and will get back to you

For the point 4, this is random. FTS backend (xapian, lucene, solr, whatever..) returns X, then dovecot core choose to select only Y emails. THis is a clear bug.

On 2019-04-05 20:08, Josef 'Jeff' Sipek via dovecot wrote:
On Fri, Apr 05, 2019 at 19:33:57 +0800, Joan Moreau via dovecot wrote: Hi

If you plan to fix the FTS part of Dovecot, I will be very gratefull.
I'm trying to figure out what is causing the 3rd issue you listed, so we can

decide how severe it is and therefore how quickly it needs to be fixed. At
the moment we are unable to reproduce it, and therefore we cannot fix it.

Not sure this is related to any specific commit but rahter the overall
design
Ok.

The list of bugs so far

1 - Double call to fts plugins with inconsistent parameter (first call
diferent from second call for the same request)
Understood. It is my understanding that this is simply suboptimal rather

than causing crashes/etc.

2 - "Rescan" features for now consists of deleting indexes. SHall be
resending emails to rescan to the fts plugin instead
I'm not sure I follow. The rescan operation is invoked on the fts backend

and it is up to the implementation to somehow ensure that after it is done
the fts index is up to date. The easiest way to implement it is to simply
delete the fts index and re-index all the mails. That is what currently
happens in the solr backend.

The lucene fts backend does a more complicated matching of the fts index
with the emails. Finally, the deprecated squat backend seem to ignore the
rescan requests (its rescan vfunc is NULL).

3 - the loop when body search (just do a "doveadm search -u user@domain
mailbox inbox text whatevertexte")

Refer to my email to Timo on 2019-04-03 18:30 on the same thread for bug
details

(especially the loop)
This seems to be the most important of the 4 issues you listed, so I'd like

to focus on this one for now.

As I mentioned, we cannot reproduce this ourselves. So, we need your help
to narrow things down. Therefore, can you give us the commit hashes of
revisions that you know are good and which are bad? You can use git-bisect
to narrow the range down.

4 - Most notably, I notice that header search usually does not care
about fts plugin (even with fts_enforced) and rely on some internal
search , which si total non-sense
You're right, that doesn't seem to make sense. Can you provide a test case?

Jeff.

Let me know how can I help on thos 4 points

On 2019-04-05 18:37, Josef 'Jeff' Sipek wrote:

On Fri, Apr 05, 2019 at 17:45:36 +0800, Joan Moreau wrote:

I am on master (very latest)

No clue exactly when this problem appears, but

1 - the "request twice the fts plugin instead of once" issue has always
been there (since my first RC release of fts-xapian)
Ok, good to know.

2 - the body/text loop has appeared recently (maybe during the month of
March)
Our testing doesn't seem to be able to reproduce this. Can you try to

git-bisect this to find which commit broke it?

Thanks,

Jeff.

On 2019-04-05 16:36, Josef 'Jeff' Sipek via dovecot wrote:

On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote:

issue seems in the Git version :
Which git revision?

Before you updated to the broken revision, which revision/version were you
running?

Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit
just before the fts_enforced=body introduction)? That's the only recent fts
change.

Thanks,

Jeff.

On 2019-04-03 18:58, @lbutlr via dovecot wrote:

On 3 Apr 2019, at 04:30, Joan Moreau via dovecot wrote:

doveadm search -u j...@grosjo.net mailbox inbox text milan
Did that search over my list mail and got 83 results, not able to duplicate your issue.

What version of dovecot and have you tried to reindex?

dovecot-2.3.5.1 here.

Re: FTS delays

2019-04-14 Thread Joan Moreau via dovecot

I have tried to spend some time of understanding the logic (if any !) of
the fts part

Honestly, the one who created this mess shall be the one to fix it, or
one shall refactor it totally.

Basically, the fts "core" should be able to do

- select the backend according to conf file

- send new emails/maiblox to backend

- send teh ID of the emails to be removed

- resend an entire mailbox ('rescan')

- send the search parameters (from client) to backend and return the
email to front end based on backend results (and NOTHING more)

Today, the fts part is plain wong and must be totally reviewed.

I do not have the time but I can participate in testing if someone is
ready to roll up its sleeves on teh mater

THe "loop" part seems the most urgent : It breaks everything (search
timeout 100% of the time)

On 2019-04-06 09:56, Joan Moreau via dovecot wrote:

For the point 1, this is not "suboptimal", it is plain wrong (results are damn
wrong ! and this is not related to the backend, but the FTS logic in Dovecot core)

For the point 3, I will do a bit of research in the existing code and will get back to you

For the point 4, this is random. FTS backend (xapian, lucene, solr, whatever..) returns X, then dovecot core choose to select only Y emails. THis is a clear bug.

On 2019-04-05 20:08, Josef 'Jeff' Sipek via dovecot wrote:
On Fri, Apr 05, 2019 at 19:33:57 +0800, Joan Moreau via dovecot wrote: Hi

If you plan to fix the FTS part of Dovecot, I will be very gratefull.
I'm trying to figure out what is causing the 3rd issue you listed, so we can

decide how severe it is and therefore how quickly it needs to be fixed. At
the moment we are unable to reproduce it, and therefore we cannot fix it.

Not sure this is related to any specific commit but rahter the overall
design
Ok.

The list of bugs so far

1 - Double call to fts plugins with inconsistent parameter (first call
diferent from second call for the same request)
Understood. It is my understanding that this is simply suboptimal rather

than causing crashes/etc.

2 - "Rescan" features for now consists of deleting indexes. SHall be
resending emails to rescan to the fts plugin instead
I'm not sure I follow. The rescan operation is invoked on the fts backend

The lucene fts backend does a more complicated matching of the fts index
with the emails. Finally, the deprecated squat backend seem to ignore the
rescan requests (its rescan vfunc is NULL).

3 - the loop when body search (just do a "doveadm search -u user@domain
mailbox inbox text whatevertexte")

Refer to my email to Timo on 2019-04-03 18:30 on the same thread for bug
details

(especially the loop)
This seems to be the most important of the 4 issues you listed, so I'd like

to focus on this one for now.

Jeff.

Let me know how can I help on thos 4 points

On 2019-04-05 18:37, Josef 'Jeff' Sipek wrote:

On Fri, Apr 05, 2019 at 17:45:36 +0800, Joan Moreau wrote:

I am on master (very latest)

No clue exactly when this problem appears, but

1 - the "request twice the fts plugin instead of once" issue has always
been there (since my first RC release of fts-xapian)
Ok, good to know.

2 - the body/text loop has appeared recently (maybe during the month of
March)
Our testing doesn't seem to be able to reproduce this. Can you try to

git-bisect this to find which commit broke it?

Thanks,

Jeff.

On 2019-04-05 16:36, Josef 'Jeff' Sipek via dovecot wrote:

On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote:

issue seems in the Git version :
Which git revision?

Before you updated to the broken revision, which revision/version were you
running?

Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit
just before the fts_enforced=body introduction)? That's the only recent fts
change.

Thanks,

Jeff.

On 2019-04-03 18:58, @lbutlr via dovecot wrote:

On 3 Apr 2019, at 04:30, Joan Moreau via dovecot w

Re: FTS delays

2019-04-05 Thread Joan Moreau via dovecot


For the point 1, this is not "suboptimal", it is plain wrong (results
are damn wrong ! and this is not related to the backend, but the FTS
logic in Dovecot core)

For the point 2 , this has been discussed already numerous times but
without action. The dovecot core shall be the one re-submitting the
emails to scan, not the backend to try to figure out where and which are
the emails to be re-scaned 


For the point 3, I will do a bit of research in the existing code and
will get back to you 


For the point 4, this is random. FTS backend (xapian, lucene, solr,
whatever..) returns X, then dovecot core choose to select only Y emails.
THis is a clear bug. 


On 2019-04-05 20:08, Josef 'Jeff' Sipek via dovecot wrote:

On Fri, Apr 05, 2019 at 19:33:57 +0800, Joan Moreau via dovecot wrote: 

Hi 


If you plan to fix the FTS part of Dovecot, I will be very gratefull.


I'm trying to figure out what is causing the 3rd issue you listed, so we can
decide how severe it is and therefore how quickly it needs to be fixed.  At
the moment we are unable to reproduce it, and therefore we cannot fix it.


Not sure this is related to any specific commit but rahter the overall
design


Ok.

The list of bugs so far 


1 - Double call to fts plugins with inconsistent parameter (first call
diferent from second call for the same request)


Understood.  It is my understanding that this is simply suboptimal rather
than causing crashes/etc.


2 - "Rescan" features for now consists of deleting indexes. SHall be
resending emails to rescan to the fts plugin instead


I'm not sure I follow.  The rescan operation is invoked on the fts backend
and it is up to the implementation to somehow ensure that after it is done
the fts index is up to date.  The easiest way to implement it is to simply
delete the fts index and re-index all the mails.  That is what currently
happens in the solr backend.

The lucene fts backend does a more complicated matching of the fts index
with the emails.  Finally, the deprecated squat backend seem to ignore the
rescan requests (its rescan vfunc is NULL).


3 - the loop when body search (just do a "doveadm search -u user@domain
mailbox inbox text whatevertexte") 


Refer to my email to Timo on 2019-04-03 18:30 on the same thread for bug
details 


(especially the loop)


This seems to be the most important of the 4 issues you listed, so I'd like
to focus on this one for now.

As I mentioned, we cannot reproduce this ourselves.  So, we need your help
to narrow things down.  Therefore, can you give us the commit hashes of
revisions that you know are good and which are bad?  You can use git-bisect
to narrow the range down.


4 - Most notably, I notice that header search usually does not care
about fts plugin (even with fts_enforced) and rely on some internal
search , which si total non-sense


You're right, that doesn't seem to make sense.  Can you provide a test case?

Jeff.

Let me know how can I help on thos 4 points 


On 2019-04-05 18:37, Josef 'Jeff' Sipek wrote:

On Fri, Apr 05, 2019 at 17:45:36 +0800, Joan Moreau wrote: 

I am on master (very latest) 

No clue exactly when this problem appears, but 


1 - the "request twice the fts plugin instead of once" issue has always
been there (since my first RC release of fts-xapian) 
Ok, good to know.


2 - the body/text loop has appeared recently (maybe during the month of
March) 
Our testing doesn't seem to be able to reproduce this.  Can you try to

git-bisect this to find which commit broke it?

Thanks,

Jeff.

On 2019-04-05 16:36, Josef 'Jeff' Sipek via dovecot wrote:

On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote: 

issue seems in the Git version : 
Which git revision?


Before you updated to the broken revision, which revision/version were you
running?

Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit
just before the fts_enforced=body introduction)?  That's the only recent fts
change.

Thanks,

Jeff.

On 2019-04-03 18:58, @lbutlr via dovecot wrote:

On 3 Apr 2019, at 04:30, Joan Moreau via dovecot  wrote: 

doveadm search -u j...@grosjo.net mailbox inbox text milan 
Did that search over my list mail and got 83 results, not able to duplicate your issue.


What version of dovecot and have you tried to reindex?

dovecot-2.3.5.1 here.

Re: FTS delays

2019-04-05 Thread Joan Moreau via dovecot

Hi 


If you plan to fix the FTS part of Dovecot, I will be very gratefull.
Not sure this is related to any specific commit but rahter the overall
design 

The list of bugs so far 


1 - Double call to fts plugins with inconsistent parameter (first call
diferent from second call for the same request) 


2 - "Rescan" features for now consists of deleting indexes. SHall be
resending emails to rescan to the fts plugin instead 


3 - the loop when body search (just do a "doveadm search -u user@domain
mailbox inbox text whatevertexte") 


Refer to my email to Timo on 2019-04-03 18:30 on the same thread for bug
details 

(especially the loop) 


4 - Most notably, I notice that header search usually does not care
about fts plugin (even with fts_enforced) and rely on some internal
search , which si total non-sense 

Let me know how can I help on thos 4 points 


On 2019-04-05 18:37, Josef 'Jeff' Sipek wrote:

On Fri, Apr 05, 2019 at 17:45:36 +0800, Joan Moreau wrote: 

I am on master (very latest) 

No clue exactly when this problem appears, but 


1 - the "request twice the fts plugin instead of once" issue has always
been there (since my first RC release of fts-xapian)


Ok, good to know.


2 - the body/text loop has appeared recently (maybe during the month of
March)


Our testing doesn't seem to be able to reproduce this.  Can you try to
git-bisect this to find which commit broke it?

Thanks,

Jeff.

On 2019-04-05 16:36, Josef 'Jeff' Sipek via dovecot wrote:

On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote: 

issue seems in the Git version : 
Which git revision?


Before you updated to the broken revision, which revision/version were you
running?

Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit
just before the fts_enforced=body introduction)?  That's the only recent fts
change.

Thanks,

Jeff.

On 2019-04-03 18:58, @lbutlr via dovecot wrote:

On 3 Apr 2019, at 04:30, Joan Moreau via dovecot  wrote: 

doveadm search -u j...@grosjo.net mailbox inbox text milan 
Did that search over my list mail and got 83 results, not able to duplicate your issue.


What version of dovecot and have you tried to reindex?

dovecot-2.3.5.1 here.

Re: FTS delays

2019-04-05 Thread Joan Moreau via dovecot

I am on master (very latest) 

No clue exactly when this problem appears, but 


1 - the "request twice the fts plugin instead of once" issue has always
been there (since my first RC release of fts-xapian) 


2 - the body/text loop has appeared recently (maybe during the month of
March) 


On 2019-04-05 16:36, Josef 'Jeff' Sipek via dovecot wrote:

On Wed, Apr 03, 2019 at 19:02:52 +0800, Joan Moreau via dovecot wrote: 


issue seems in the Git version :


Which git revision?

Before you updated to the broken revision, which revision/version were you
running?

Can you try it with 5f6e39c50ec79ba8847b2fdb571a9152c71cd1b6 (the commit
just before the fts_enforced=body introduction)?  That's the only recent fts
change.

Thanks,

Jeff.

On 2019-04-03 18:58, @lbutlr via dovecot wrote:

On 3 Apr 2019, at 04:30, Joan Moreau via dovecot  wrote: 

doveadm search -u j...@grosjo.net mailbox inbox text milan 
Did that search over my list mail and got 83 results, not able to duplicate your issue.


What version of dovecot and have you tried to reindex?

dovecot-2.3.5.1 here.

Re: FTS delays

2019-04-03 Thread Joan Moreau via dovecot

issue seems in the Git version : 

FTS search in teh body ends up with looping 

Other search call twice the FTS plugin (for no reason) 


On 2019-04-03 18:58, @lbutlr via dovecot wrote:

On 3 Apr 2019, at 04:30, Joan Moreau via dovecot  wrote: 


doveadm search -u j...@grosjo.net mailbox inbox text milan


Did that search over my list mail and got 83 results, not able to duplicate 
your issue.

What version of dovecot and have you tried to reindex?

dovecot-2.3.5.1 here.

Re: FTS delays

2019-04-03 Thread Joan Moreau via dovecot

Example from real life 

From Roubdcube, I serach "milan" in full message (body & headers) 


Logs : 


Apr 3 10:24:01 gjserver dovecot[29778]:
imap(j...@grosjo.net)<30311><4pACp52FfCF/AAAB>: Query : ( bcc:milan OR
body:milan OR cc:milan OR from:milan OR message-id:milan OR
subject:milan OR to:milan OR uid:milan )
Apr 3 10:24:01 gjserver dovecot[29778]:
imap(j...@grosjo.net)<30311><4pACp52FfCF/AAAB>: Query: 81 results in 2 ms


81 results is correct 

but Roundcube times out 

from command line, I do : 

doveadm search -u j...@grosjo.net mailbox inbox text milan 

output 


doveadm(j...@grosjo.net): Info: Query : ( bcc:inbox OR body:inbox OR
cc:inbox OR from:inbox OR message-id:inbox OR subject:inbox OR to:inbox
OR uid:inbox ) AND ( bcc:milan OR body:milan OR cc:milan OR from:milan
OR message-id:milan OR subject:milan OR to:milan OR uid:milan )
doveadm(j...@grosjo.net): Info: Query: 1 results in 1 ms
d82b4b0f550d3859364495331209 847
d82b4b0f550d3859364495331209 1569
d82b4b0f550d3859364495331209 2260
d82b4b0f550d3859364495331209 2575
d82b4b0f550d3859364495331209 2811
d82b4b0f550d3859364495331209 2885
d82b4b0f550d3859364495331209 3038
d82b4b0f550d3859364495331209 3121
d82b4b0f550d3859364495331209 3170 

1 - The query is wrong 

2 - teh last line "d8...209 3170" gets repeated for ages 


On 2019-04-02 16:30, Timo Sirainen wrote:

On 2 Apr 2019, at 6.38, Joan Moreau via dovecot  wrote: 


Further on this topic:

When choosing any headers in the search box, dovecot core calls the plugin 
TWICE (and returns the results quickly, but not immediatly after getting the 
IDs from the plugins)

When choosing the BODY search, dovecot core calls the plugin ONCE (and never 
returns) (whereas the plugins returns properly the IDs)


If we simplify this, do you mean this calls it once and is fast:

doveadm search -u user@domain mailbox inbox body helloworld

But this calls twice and is slow:

doveadm search -u user@domain mailbox inbox text helloworld

And what about searching e.g. subject? :

doveadm search -u user@domain mailbox inbox subject helloworld

And does the slowness depend on whether there were any matches or not?


This is based on GIT version. (previous versions were working properly)


Previous versions were fast? Do you mean v2.3.5?

Re: FTS delays

2019-04-01 Thread Joan Moreau via dovecot

Further on this topic: 


When choosing any headers in the search box, dovecot core calls the
plugin TWICE (and returns the results quickly, but not immediatly after
getting the IDs from the plugins) 


When choosing the BODY search, dovecot core calls the plugin ONCE (and
never returns) (whereas the plugins returns properly the IDs) 

This is based on GIT version. (previous versions were working properly) 

Looking for feedback 


Thank you

On 2019-03-30 21:48, Joan Moreau wrote:

it is already on 

On March 31, 2019 03:47:52 Aki Tuomi via dovecot  wrote: 

On 30 March 2019 21:37 Joan Moreau via dovecot  wrote: 

Hi 

When I do a FTS search (using Xapian plugin) in the BODY part, the plugins returns the matching IDs within few milliseconds (as seen in the log). 

However, roundcube (connected on dovecot) takes ages to show (headers only vie IMAP) the few results (I tested with a matching requests of 9 emails) 

What could be the root cause ? 

Thank you 

does it help if you set 

plugin { 
fts_enforced=yes 
} 


---
Aki Tuomi

Re: FTS delays

2019-03-30 Thread Joan Moreau via dovecot


it is already on

On March 31, 2019 03:47:52 Aki Tuomi via dovecot  wrote:



On 30 March 2019 21:37 Joan Moreau via dovecot  wrote:





Hi

When I do a FTS search (using Xapian plugin) in the BODY part, the plugins 
returns the matching IDs within few milliseconds (as seen in the log).


However, roundcube (connected on dovecot) takes ages to show (headers only 
vie IMAP) the few results (I tested with a matching requests of 9 emails)


What could be the root cause ?

Thank you


does it help if you set

plugin {
  fts_enforced=yes
}
---
Aki Tuomi

FTS delays

2019-03-30 Thread Joan Moreau via dovecot

Hi 


When I do a FTS search (using Xapian plugin) in the BODY part, the
plugins returns the matching IDs within few milliseconds (as seen in the
log). 


However, roundcube (connected on dovecot) takes ages to show (headers
only vie IMAP) the few results (I tested with a matching requests of 9
emails) 

What could be the root cause ? 


Thank you

Re: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

2019-02-18 Thread Joan Moreau via dovecot

Can you clarify the piece of code or give an example on how to "Get
list of UIDs for all mails in each folder " and how to get the "list of
all folder/mailbox" from a *backend input ?

On 2019-02-17 14:52, Aki Tuomi wrote:

Not really, as the steps outlined by Timo would not get done.

Aki

On 17 February 2019 at 10:56 Joan Moreau via dovecot
wrote:

In such case, as long as the API is not upgraded, should

doveadm index -A -q \*

be considered a replacement of

doveadm fts rescan

On 2019-02-14 16:24, Timo Sirainen via dovecot wrote:

Hi,

The rescan() function is a bit badly designed. Currently what you could do what fts-lucene does and:
- Get list of UIDs for all mails in each folder
- If Xapian has UID that doesn't exist -> delete it from Xapian
- If UID is missing from Xapian -> expunge the rest of the UIDs in that folder, so the next indexing will cause them to be indexed

The expunging of rest of the mails is rather ugly, yes.. A better API would be if backend simply had a way to iterate all mails in the index, preferrably sorted by folder. Then a more generic code could go through them and expunge the necessary mails and index the missing mails. Although not all FTS backends support indexing in the middle. Anyway, we don't really have time to implement this new API soon.

I'm not sure if this is a big problem though. I don't think most people running FTS have ever run rescan.

On 8 Feb 2019, at 9.54, Joan Moreau via dovecot wrote:

Hi,

THis is a core problem in Dovecot in my understanding.

In my opinion, the rescan in dovecot should send to the FTS plugin the list of "supposedly" indexed emails (UID), and the plugin shall purge the redundant UID (i..e UID present in the index but not in the list sent by dovecot) and send back the list of UID not in its indexes to dovecot, so Dovect can send one by one the missing emails

WHat do you think ?

Original Message

SUBJECT:
[grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

DATE:
2019-02-08 08:28

FROM:
Leonard Lausen

TO:
grosjo/fts-xapian

CC:
Subscribed

REPLY-TO:
grosjo/fts-xapian

doveadm fts rescan -A deletes all indices, ie. all folders and files in the xapian-indexes are deleted. However, according to man doveadm fts, the rescan command should only

Scan what mails exist in the full text search index and compare those to what
actually exist in mailboxes. This removes mails from the index that have already
been expunged and makes sure that the next doveadm index will index all the
missing mails (if any).

Deleting all indices does not seem to be the intended action, especially as constructing the index anew may take very long on large mailboxes.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub [1 [1]], or mute the thread [2].

Links:
--
[1] https://github.com/grosjo/fts-xapian/issues/15
[2]
https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp

Links:
--
[1] https://github.com/grosjo/fts-xapian/issues/15

Re: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

2019-02-17 Thread Joan Moreau via dovecot

In such case, as long as the API is not upgraded, should

doveadm index -A -q \*

be considered a replacement of

doveadm fts rescan

On 2019-02-14 16:24, Timo Sirainen via dovecot wrote:

Hi,

I'm not sure if this is a big problem though. I don't think most people running FTS have ever run rescan.

On 8 Feb 2019, at 9.54, Joan Moreau via dovecot wrote:

Hi,

THis is a core problem in Dovecot in my understanding.

WHat do you think ?

Original Message

SUBJECT:
[grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

DATE:
2019-02-08 08:28

FROM:
Leonard Lausen

TO:
grosjo/fts-xapian

CC:
Subscribed

REPLY-TO:
grosjo/fts-xapian

doveadm fts rescan -A deletes all indices, ie. all folders and files in the xapian-indexes are deleted. However, according to man doveadm fts, the rescan command should only

Deleting all indices does not seem to be the intended action, especially as constructing the index anew may take very long on large mailboxes.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:
--
[1] https://github.com/grosjo/fts-xapian/issues/15
[2]
https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp

Re: Fwd: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

2019-02-13 Thread Joan Moreau via dovecot

Anyone ?

On 2019-02-08 08:54, Joan Moreau via dovecot wrote:

Hi,

THis is a core problem in Dovecot in my understanding.

WHat do you think ?

Original Message

SUBJECT:
[grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

DATE:
2019-02-08 08:28

FROM:
Leonard Lausen

TO:
grosjo/fts-xapian

CC:
Subscribed

REPLY-TO:
grosjo/fts-xapian

doveadm fts rescan -A deletes all indices, ie. all folders and files in the
xapian-indexes are deleted. However, according to man doveadm fts, the rescan
command should only

Deleting all indices does not seem to be the intended action, especially as constructing the index anew may take very long on large mailboxes.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:
--
[1] https://github.com/grosjo/fts-xapian/issues/15
[2]
https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp

Fwd: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)

2019-02-07 Thread Joan Moreau via dovecot

Hi,

THis is a core problem in Dovecot in my understanding.

In my opinion, the rescan in dovecot should send to the FTS plugin the
list of "supposedly" indexed emails (UID), and the plugin shall purge
the redundant UID (i..e UID present in the index but not in the list
sent by dovecot) and send back the list of UID not in its indexes to
dovecot, so Dovect can send one by one the missing emails

WHat do you think ?

Original Message

SUBJECT:
[grosjo/fts-xapian] `doveadm fts rescan` removes all indices
(#15)

DATE:
2019-02-08 08:28

FROM:
Leonard Lausen

TO:
grosjo/fts-xapian

CC:
Subscribed

REPLY-TO:
grosjo/fts-xapian

doveadm fts rescan -A deletes all indices, ie. all folders and files in
the xapian-indexes are deleted. However, according to man doveadm fts,
the rescan command should only

Deleting all indices does not seem to be the intended action, especially
as constructing the index anew may take very long on large mailboxes.

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub [1], or mute the thread
[2].

Links:
--
[1] https://github.com/grosjo/fts-xapian/issues/15
[2]
https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp

Re: Solr - complete setup (update)

2019-01-29 Thread Joan Moreau via dovecot

On 2019-01-30 07:33, Stephan Bosch wrote:

(forgot to CC mailing list)

Op 26/01/2019 om 20:07 schreef Joan Moreau via dovecot:

*- Bugs so far*

-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated
("huge header" warning for a simple email, which kilss the index of that
considered email, so basically MOST emails as the calculation is wrong) *You can check that
regularly in dovecot log file. My guess is the mix of Unicode which is not properly
addressed here.*

Does this happen with specific messages? Do you have a sample message
for me? I don't see how Unicode could cause this.

MY ONLY GUESS IS THAT IT REFERS TO SOME 'STRLEN', WHICH IS WRONG OF
COURSE IN CASE OF UNICODE EMAILS. THIS IS JUST A GUESS.

BUT DO A GREP FOR "HUGE" IN THE DOVECOT LOG OF A BUSY SERVER TO FIND
EXAMPLES.

(SORRY, I SWITCHED TO XAPIAN, AS SOLR IS CREATING TOO MUCH TROUBLES FOR
MY SERVER, SO NO MORE CONCRETE EXAMPLE)

-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of
problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)

*This is just highly visible in Solr schema.xml. Swithcing it to "long" in
schema.xml returns plenty of errors.*

I cannot reproduce this so far (see modified schema below). In a simple
test I just get the desired results and no errors logged.

I got this with large mailboxes (where UID seems not acceptable for Solr
). The fault is not on Dovecot side but Solr, and the returned UID(s)
for a search is garbage instead of a proper value -> Putting it as
string solves this

-> Java errors : A lot of non sense for me, I am not expert in Java. But, with
increased memory, it seems not crashing, even if complaining quite a lot in the
logs

Can you elaborate on the errors you have seen so far? When do these happen? How
can I reproduce them?

*Honestly, I have no clue what the problems are. I just increased the memory of
the JVM and the systems stopped crashing. Log files are huge anyway.*

What errors do you see? I see only INFO entries in my
/var/solr/logs/solr.log. Looks like Solr is pretty verbose by default
(lots of INFO output), but there must be a way to reduce that.

I DELETED SOLR. NO MORE LOGS. MAYBE SOMEONE ELSE CAN TELL.

Re: Solr - complete setup (update)

2019-01-26 Thread Joan Moreau via dovecot


*- Installation:*

-> Create a clean install using the default, (at least in the Archlinux package), and do 
a "sudo -u solr solr create -c dovecot ". The config files are then in 
/opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data


On my system (Debian) these directories are wildly different (e.g. data
is under /var), but other than that, this information is OK.

Used this as a side-reference for Debian installation:
https://tecadmin.net/install-apache-solr-on-debian/

Accessed http://solr-host.tld:8983/solr/ to check whether all is OK. 


MAKE SURE YOU HAVE A DOVECOT INSTANCE (NOT THE DEFAULT INSTANCE) , WITH
THE FUNCTION BELOW: 

SOLR CREATE -C DOVECOT (OR WHATEVER NAME) 


Weirdly, rescan returns immediately here. When I perform `doveadm index INBOX` 
for my test user, I do see a lot of fts and HTTP activity.


THE SOLR PLUGIN IS NOT CODED ENTIRELY, REFRESH AND RESCAN FUNCTIONS ARE
MISSING : 


https://github.com/dovecot/core/blob/master/src/plugins/fts-solr/fts-backend-solr.c


static int fts_backend_solr_refresh(struct fts_backend *backend
ATTR_UNUSED)
{
return 0;
} 


static int fts_backend_solr_rescan(struct fts_backend *backend)
{
/* FIXME: proper rescan needed. for now we'll just reset the
last-uids */
return fts_backend_reset_last_uids(backend);
} 


*- Bugs so far*

-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated 
("huge header" warning for a simple email, which kilss the index of that 
considered email, so basically MOST emails as the calculation is wrong)


YOU CAN CHECK THAT REGULARLY IN DOVECOT LOG FILE. MY GUESS IS THE MIX OF
UNICODE WHICH IS NOT PROPERLY ADDRESSED HERE. 


-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of 
problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)


THIS IS JUST HIGHLY VISIBLE IN SOLR SCHEMA.XML. SWITHCING IT TO "LONG"
IN SCHEMA.XML RETURNS PLENTY OF ERRORS. 

-> Java errors : A lot of non sense for me, I am not expert in Java. But, with increased memory, it seems not crashing, even if complaining quite a lot in the logs 


Can you elaborate on the errors you have seen so far? When do these happen? How 
can I reproduce them?


HONESTLY, I HAVE NO CLUE WHAT THE PROBLEMS ARE. I JUST INCREASED THE
MEMORY OF THE JVM AND THE SYSTEMS STOPPED CRASHING. LOG FILES ARE HUGE
ANYWAY.

[FTS Xapian] RC release

2019-01-24 Thread Joan Moreau via dovecot

Hi, 


FTS Xapian matches my targets for the plugins (replacing deprecated
fts-squat in a production environment) 

https://github.com/grosjo/fts-xapian 

Please do not hesitate to add "issues" on github, if the case happen 

Hope it helps 


JM

Encoding input

2019-01-22 Thread Joan Moreau via dovecot

Hi 


Are data inputs of fts_backend_update_build_more(struct
fts_backend_update_context *_ctx, const unsigned char *data, size_t
size) already converted to UTF8 ? 


Thanks

1 2 >

1 - 100 of 120 matches

Mail list logo