date:20190112

Re: [solved] managesieve configuration

2019-01-12 Thread Dominik Menke

For reference: if you put ssl=yes there, the TLS layer is established 
immediately. However, the standard ManageSieve protocol does not support 
that (not currently anyway): only the establishment of the TLS layer 
using the STARTTLS command is part of the standard. That is why your 
clients fail to connect: they're speaking plaintext while the server is 
speaking TLS. Still, Dovecot supports configuring it that way, which is 
what you did.


Regards,

Stephan.





I'm just surprised that ssl=yes leads to STARTTLS being disabled, as per 
the wiki [1]:



> ssl=yes and disable_plaintext_auth=no: SSL/TLS is offered to the
> client, but the client isn't required to use it. [...]
>
> ssl=yes and disable_plaintext_auth=yes: SSL/TLS is offered to the
> client, but the client isn't required to use it. [...]
>
> ssl=required: SSL/TLS is always required [...]. Any attempt to
> authenticate before SSL/TLS is enabled will cause an authentication
> failure.


Maybe this bit needs to be clarified a bit? I think I've read that page 
a few times and it still didn't occur to me that this could be a problem.


Best regards,
--Dominik


[1]: https://wiki.dovecot.org/SSL/DovecotConfiguration

Re: Issue with LMTP proxying and port number

2019-01-12 Thread Stephan Bosch





Op 12/01/2019 om 23:08 schreef Stephan Bosch:



Op 06/01/2019 om 17:05 schreef Stephan Bosch:


Op 31/12/2018 om 06:32 schreef Laz C. Peterson:

Hello Sami, yes, see below.

We run Dovecot at a different versions, mainly 2.2.10 (CentOS), 
2.2.22 (Ubuntu) and now 2.2.36 (CentOS).  The issue is weird, 
because it only happened after the update from 2.2.10->36.  Just to 
understand it would be great.


I'm actually checking out the configs now ... Our SQL userdb does 
not specify port.  So I'm guessing this may be to blame?


(This was by design, though -- we don't want to specify one port for 
different client protocols.  Though, I do recall seeing some hack 
online using CASE in SQL query ...)


These servers run LMTP as a unix socket as well as a TCP port 24 
serving all IP sources.  The internal servers are running LMTP on 
TCP port 24 (as well as unix socket, but that's irrelevant), but no 
LMTP comm happens between directors and backend mail servers after 
the 2.2.10->36 update on the directors with our config.  I do 
apologize that I can't get more specific than those versions ...


The backend mail servers function the same in our environment on 
both versions 2.2.10 and 2.2.36.


We are good now, as we changed the config to go to the TCP port 
instead of unix socket.  But we had a good jolt of fun this morning. 
:-)


Would love to understand what we have done wrong, or how we 
misunderstood the configuration directives -- in either version.


I can reproduce it here, even with master.

We'll get back to you.

BTW, similar thread here:

https://www.dovecot.org/pipermail/dovecot/2019-January/114071.html


Hmm, did you try returning a protocol=lmtp field from passdb? This is 
ignored by services other than lmtp and the code tells me it will then 
default to port 24. That should be a workaround.


Oh, right, this is v2.2. There, this apparently doesn't apply :/

Regards,

Stephan.

Re: Issue with LMTP proxying and port number

2019-01-12 Thread Stephan Bosch





Op 06/01/2019 om 17:05 schreef Stephan Bosch:


Op 31/12/2018 om 06:32 schreef Laz C. Peterson:

Hello Sami, yes, see below.

We run Dovecot at a different versions, mainly 2.2.10 (CentOS), 
2.2.22 (Ubuntu) and now 2.2.36 (CentOS).  The issue is weird, because 
it only happened after the update from 2.2.10->36.  Just to 
understand it would be great.


I'm actually checking out the configs now ... Our SQL userdb does not 
specify port.  So I'm guessing this may be to blame?


(This was by design, though -- we don't want to specify one port for 
different client protocols.  Though, I do recall seeing some hack 
online using CASE in SQL query ...)


These servers run LMTP as a unix socket as well as a TCP port 24 
serving all IP sources.  The internal servers are running LMTP on TCP 
port 24 (as well as unix socket, but that's irrelevant), but no LMTP 
comm happens between directors and backend mail servers after the 
2.2.10->36 update on the directors with our config.  I do apologize 
that I can't get more specific than those versions ...


The backend mail servers function the same in our environment on both 
versions 2.2.10 and 2.2.36.


We are good now, as we changed the config to go to the TCP port 
instead of unix socket.  But we had a good jolt of fun this morning. :-)


Would love to understand what we have done wrong, or how we 
misunderstood the configuration directives -- in either version.


I can reproduce it here, even with master.

We'll get back to you.

BTW, similar thread here:

https://www.dovecot.org/pipermail/dovecot/2019-January/114071.html


Hmm, did you try returning a protocol=lmtp field from passdb? This is 
ignored by services other than lmtp and the code tells me it will then 
default to port 24. That should be a workaround.


Regards,

Stephan.

Re: Dovecot Submission Proxy Auth

2019-01-12 Thread Stephan Bosch





Op 11/01/2019 om 02:52 schreef Jacky:


Hi,

Just found out that Postfix does not implement/support the AUTH=sender 
parameter.


So, back to Dovecot, can we use variables in the

submission_relay_user =
submission_relay_password =



No, that is not supported. :/

then Dovecot will forward the username and password information of the 
current user to the Postfix submission service for authentication?




Would Postfix do something with the XCLIENT LOGIN field in that regard?

(Note that 2.3.4 messes up XCLIENT in several ways, so --- if Postfix 
can do this --- you'll have to wait for the next release).


Regards,

Stephan.


Best regards,

Jacky



On 10/1/2019 10:46 AM, Jacky wrote:


Hi Gerald and Odhiambo Washington,

Thank you for your suggestions and will try them out.

Best regards,

Jacky

On 9/1/2019 6:38 PM, Odhiambo Washington wrote:



On Wed, 9 Jan 2019 at 13:09, Jacky > wrote:


Hi Gerald,

in my postfix/main.cf 

smtpd_sasl_authenticated_header = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = $myhostname
smtpd_sasl_type = dovecot
smtpd_sasl_path = /var/run/dovecot/auth-client
broken_sasl_auth_clients = yes

I am already using dovecot for SASL

The dovecot submission service authenticates users and already
added the
AUTH= parameter in the MAIL FROM

MAIL FROM:mailto:ja...@xxx.com>>
AUTH=ja...@xxx.com  SIZE=1430

But, it seems that postfix does not accept the AUTH= parameter and
reject the sender as no logged in.


Best regards,

Jacky


Hi Jacky,

Your question belongs to postfix mailinng list.

Anyway, the last time I was playing with postfix (I am an Exim user 
normally), I had to check that:

smtpd_sasl_path = /var/run/dovecot/auth-client

..the socket is readable by the postfix user:

So, check 10-master.conf for the socket. Something like:

# Postfix smtp-auth
  unix_listener  var/run/dovecot/auth-client  {
    mode = 0666
  }

Restart dovecot and see...

You can read the https://wiki.dovecot.org/HowTo/PostfixAndDovecotSASL



--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
"Oh, the cruft.", grep ^[^#] :-)

Re: Solr -> Xapian ?

2019-01-12 Thread Joan Moreau via dovecot


I somehow fixed the folder issue. (seems some unix rights after too many
tests) 

Getting back on the "fts_results" structure: 

I am trying: 


I_ARRAY_INIT(&(RESULT->DEFINITE_UIDS),R->SIZE);
I_ARRAY_INIT(&(RESULT->MAYBE_UIDS),0); 


uint32_t uid;
for(i=0;isize;i++)
{
  try
  {

uid=atol(backend->dbr->get_document(r->data[i]).get_value(1).c_str());

 i_warning("Rresult UID=%d",uid);
 ARRAY_IDX_SET(&(RESULT->DEFINITE_UIDS),I,&UID);
  }
  catch(Xapian::Error e)
  {
 i_warning(e.get_msg().c_str());
  }
} 


I can see in hte log that UID are properly found on Xapian database, but
no results are transmitted to dovecot and to the imap client (roundcube
in my case) 

Help please :) 


On 2019-01-12 18:15, Joan Moreau wrote:

additionally, my logic is that the backend stores one databalse per mailox in /xapian-indexes (in the "root" dir of the user), the name od the database is the GUID of the mailbox 

For INBOX, that works perfectly, and database is properly createdm and backed starts indexing all emails 

For other folder, somehow, the process can not access that (root) folder. 

Am I missing something ? 

On 2019-01-12 17:37, Joan Moreau wrote: 

THank you 

Now, for the results 

I see the member of fts_result is : 


ARRAY_TYPE(seq_range) definite_uids;

I have the UID as a aray of uint32_t * 

How to put my UIDs into this "definite_uids" ? Obviously this is not a simple array/pointer. How to say someting similar to result->definite_uids[1]=my_uid ? 

On 2019-01-12 10:25, Timo Sirainen wrote: 
On 11 Jan 2019, at 21.23, Joan Moreau via dovecot  wrote: 
The below patch resolves the compilation error


$ diff -p compat.h compat.h.joan 
*** compat.h 2019-01-11 20:21:00.726625427 +0100

--- compat.h.joan 2019-01-11 20:14:41.729109919 +0100
*** struct iovec;
*** 202,207 
--- 202,211 
ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len);
#endif

+ #ifdef __cplusplus
+ extern "C" {
+ #endif

You should put this extern "C" into the C++ file you're creating. See for 
example how fts-lucene/lucene-wrapper.cc does this.

1 - WHat does represent "subargs" in mail_search_args 
It's set only for SEARCH_OR and SEARCH_SUB. So for example:


SEARCH TEXT foo TEXT bar TEXT baz

results in:

type=SEARCH_SUB
value.subargs = (
{ type=SEARCH, value.str="foo" },
{ type=SEARCH, value.str="bar" },
{ type=SEARCH, value.str="baz" },
)

Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other 
combination of OR/ANDs.
2 - for rescan : who is responsible for passing again the new email ? Is
the Dovecot core sending again all the emails to index ? or the fts
shall somehow access the mailbox and read all emails ? Wouldn't just be
saying "delete all index and get_last_uid is now 0" the easy way ? or
the fts must process all emails (and block the current thread as a
mailbx maybe quite large) 
The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.


3 - for get_last_uid : this uncertainity is very unclear. "If there is a
gap, then indexer first indexes all the missing" -> this mean at a
certain point, indexer maybe rebuilding a previous email, so *last* uid
is something different than max. And how indexer does know whther there
is a gap wihtout callong the fts backend (whch it does not as there are
no function for that) ? 
I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.


You can implement get_last_uid() simply by keeping track of it in 
dovecot.index* files, similar to how Lucene and Solr already do it with 
fts_index_get_header() / fts_index_set_header(). They also have a fallback that 
if the index doesn't have the last_uid value, they do a slower search from the 
Lucene/Solr index to find the last UID.

Re: degfault imaptest

2019-01-12 Thread Сергей via dovecot

Please, unsubscribe me.
I really don't want to read this.

12.01.2019 22:47, Stephan Bosch пишет:
>
> Op 11/01/2019 om 15:13 schreef Marc Roos:
>> imaptest host=mail04.local port=143 user=xxx pass=xxx mbox=test.mbox
>>
>> [Fri Jan 11 15:09:14 2019] imaptest[7634]: segfault at 1354ff0 ip
>> 01354ff0 sp 7ffed5d8f558 error 15
>> [Fri Jan 11 15:09:22 2019] imaptest[7635]: segfault at 2267ff0 ip
>> 02267ff0 sp 7ffee5890308 error 15
>> [Fri Jan 11 15:10:47 2019] imaptest[7648]: segfault at 10e6ff0 ip
>> 010e6ff0 sp 7ffedbb19f98 error 15
>>
>> CentOS Linux release 7.6.1810 (Core)
>> imaptest-20151005-1.el7.x86_64
>> Linux test2 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC
>> 2018 x86_64 x86_64 x86_64 GNU/Linux
>
> That imaptest is from several years ago. I'd say this problem is
> likely fixed. You can check with a newer version (compile one). And
> for bug reports like this we need to have a gdb backtrace (e.g. run it
> in gdb --args  and issue 'bt full' once it fails).
>
> Regards,
>
> Stephan.
>
>
> .
>

Re: degfault imaptest

2019-01-12 Thread Stephan Bosch




Op 11/01/2019 om 15:13 schreef Marc Roos:

imaptest host=mail04.local port=143 user=xxx pass=xxx mbox=test.mbox

[Fri Jan 11 15:09:14 2019] imaptest[7634]: segfault at 1354ff0 ip
01354ff0 sp 7ffed5d8f558 error 15
[Fri Jan 11 15:09:22 2019] imaptest[7635]: segfault at 2267ff0 ip
02267ff0 sp 7ffee5890308 error 15
[Fri Jan 11 15:10:47 2019] imaptest[7648]: segfault at 10e6ff0 ip
010e6ff0 sp 7ffedbb19f98 error 15

CentOS Linux release 7.6.1810 (Core)
imaptest-20151005-1.el7.x86_64
Linux test2 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux


That imaptest is from several years ago. I'd say this problem is likely 
fixed. You can check with a newer version (compile one). And for bug 
reports like this we need to have a gdb backtrace (e.g. run it in gdb 
--args  and issue 'bt full' once it fails).


Regards,

Stephan.

Re: Error: User b...@aaa.bbb doesn't have home dir set, disabling duplicate database

2019-01-12 Thread Stephan Bosch




Op 11/01/2019 om 15:27 schreef subscription1:
I made a mistake when I moved dovecot to a new server and only 
specified mail_location instead of mail_home


All I have in 10-mail.conf is

-
mail_location = maildir:/home/vmail/mailboxes/%d/%n


All emails for the few accounts I have are in these mailboxes and I 
can get/see them via my mail client


I do, however, get the following error

imap(b...@aaa.bbb): Error: User b...@aaa.bbb doesn't have home dir set, 
disabling duplicate database

---

After looking at https://wiki2.dovecot.org/VirtualUsers/Home I tried 
the following


mail_home = /home/vmail/mailboxes/%d/%n

mail_location = maildir:~/mail
--

Problem with this is that I now can't see any of my emails in the client.

Appreciate any help on how to fix this.


Well, you changed the effective maildir directory to 
/home/vmail/mailboxes/%d/%n/mail. So, if you want to use it like that, 
you'll have to move the maildir for each user to that new path template.


Regards,

Stephan.

Re: [solved] managesieve configuration

2019-01-12 Thread Stephan Bosch




Op 11/01/2019 om 16:05 schreef Dominik Menke:

Hello Gerald,

that did the trick, thank you very much!

--Dominik


On 1/11/19 10:54 AM, Gerald Galster wrote:

Hi Dominik,

I have set ssl = required in 10-ssl.conf globally but no ssl here:

service managesieve-login {
   inet_listener sieve {
 port = 4190
   }
   ...
}

For reference: if you put ssl=yes there, the TLS layer is established 
immediately. However, the standard ManageSieve protocol does not support 
that (not currently anyway): only the establishment of the TLS layer 
using the STARTTLS command is part of the standard. That is why your 
clients fail to connect: they're speaking plaintext while the server is 
speaking TLS. Still, Dovecot supports configuring it that way, which is 
what you did.


Regards,

Stephan.




Nevertheless, STARTTLS is offered

"IMPLEMENTATION" "Dovecot Pigeonhole"
"SIEVE" "fileinto reject envelope encoded-character vacation 
subaddress comparator-i;ascii-numeric relational regex imap4flags 
copy include variables body enotify environment mailbox date index 
ihave duplicate mime foreverypart extracttext"

"NOTIFY" "mailto"
"SASL" ""
"STARTTLS"
"VERSION" "1.0"
OK "service active"


and the connection will be encrypted (tested with roudcube webmail)



STARTTLS

< OK "Begin TLS negotiation now."

...


You can check if it works with tcpdump:

tcpdump -nn -l -A -i eth0 port 4190


Best regards
Gerald



Am 11.01.2019 um 09:59 schrieb Dominik Menke :

Sure, here you go (I've masked a few unimportant fields, though):


    # 2.2.33.2 (d6601f4ec): /etc/dovecot/dovecot.conf
    # Pigeonhole version 0.4.21 (92477967)
    # OS: Linux 4.15.0-42-generic x86_64 Ubuntu 18.04.1 LTS
    auth_default_realm = masked
    auth_master_user_separator = *
    auth_mechanisms = plain login scram-sha-1
    default_vsz_limit = 4 G
    doveadm_worker_count = 8
    log_path = /dev/stderr
    mail_attachment_dir = /var/mail/sis
    mail_attachment_hash = %{sha256}
    mail_location = mdbox:~/mdbox
    managesieve_notify_capability = mailto
    managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart 
extracttext vacation-seconds imapsieve vnd.dovecot.imapsieve

    mdbox_rotate_size = 128 M
    namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
    auto = subscribe
    special_use = \Drafts
  }
  mailbox Junk {
    auto = subscribe
    special_use = \Junk
  }
  mailbox Sent {
    auto = subscribe
    special_use = \Sent
  }
  mailbox Trash {
    auto = subscribe
    special_use = \Trash
  }
  prefix =
    }
    passdb {
  args = username_format=%n /etc/dovecot/passwd.masterusers
  driver = passwd-file
  master = yes
  pass = yes
    }
    passdb {
  args = username_format=%n /etc/dovecot/passwd
  driver = passwd-file
    }
    plugin {
  imapsieve_mailbox1_before = 
file:/etc/dovecot/sieve/learn-spam.sieve

  imapsieve_mailbox1_cause = COPY FLAG
  imapsieve_mailbox1_name = Junk
  imapsieve_mailbox2_before = 
file:/etc/dovecot/sieve/learn-ham.sieve

  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = Junk
  imapsieve_mailbox2_name = *
  sieve = ~/dovecot.sieve
  sieve_after = /etc/dovecot/sieve/after
  sieve_dir = ~/sieve
  sieve_extensions = +vacation-seconds
  sieve_global_extensions = +vnd.dovecot.pipe
  sieve_pipe_bin_dir = /etc/dovecot/sieve
  sieve_plugins = sieve_imapsieve sieve_extprograms
  sieve_vacation_default_period = 1d
  sieve_vacation_max_period = 30d
  sieve_vacation_min_period = 1d
    }
    protocols = imap lmtp sieve
    service auth {
  unix_listener /var/spool/postfix/private/dovecot-auth {
    group = postfix
    mode = 0600
    user = postfix
  }
    }
    service imap-login {
  inet_listener imap {
    port = 143
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }
  process_limit = 128
    }
    service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0600
    user = postfix
  }
    }
    service managesieve-login {
  inet_listener sieve {
    port = 4190
    ssl = yes
  }
  service_count = 1
    }
    service managesieve {
  process_limit = 256
    }
    ssl_cert = 
On 10.1.2019 18.28, Dominik Menke wrote:

I've missed a part at the end:


This leads me to my question: How do I force Dovecot to print at
least a STARTTLS line after a client connects to port 4190? Looking


... at the default configuration files in /etc/dovecot/conf.d/ I 
don't

see an obvious difference.


--Dominik

Can you provide output of `doveconf -n`
Aki

Re: Fatal: master: service(indexer-worker): child 493 killed with signal 11 (core dumped)

2019-01-12 Thread Larry Rosenman

Ok, thanks for the info.

Hadn't read the whole thread.

On Sat, Jan 12, 2019 at 1:30 PM Stephan Bosch  wrote:

>
> Op 11/01/2019 om 21:07 schreef Larry Rosenman:
> > do you use both fts-solr and tika by chance?
>
> I doubt it. The original post is about Lucene and it fails in the Lucene
> code. The FTS Solr + Tika problem is caused in Dovecot lib-http.
>
> Regards,
>
> Stephan.
>
>
> >
> > On Fri, Jan 11, 2019 at 11:19 AM Fírvida  > > wrote:
> >
> > Hello have you manage to solve your issue. I have a Mailu/Dovecot
> > server in
> > production which is giving me the same error. Any help would be
> > appreciated.
> >
> >
> >
> > --
> > Sent from: http://dovecot.2317879.n4.nabble.com/
> >
> >
> >
> > --
> > Larry Rosenman http://www.lerctr.org/~ler
> > Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
> > 
> > US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>


-- 
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: panic when using dovecot master account

2019-01-12 Thread André Rodier via dovecot

On Sat, 2019-01-12 at 19:11 +, André Rodier via dovecot wrote:
> On 2018-11-09 07:40, André Rodier wrote:
> > On 2018-11-09 05:25, Aki Tuomi wrote:
> > > This seems to have nothing to do with master account or not. Does this
> > > happen if you try to open the virtual mailbox again?
> > > 
> > > Aki
> > > 
> > > > On 09 November 2018 at 00:13 André Rodier  wrote:
> > > > 
> > > > 
> > > > Hello,
> > > > 
> > > > I am running dovecot 2.2.34 (874deae), on Debian stable, from 
> > > > backports.
> > > > 
> > > > I just tried the master account, and although everything worked in 
> > > > the
> > > > email client, I had logs in the error logs:
> > > > 
> > > > 
> > > > > imap(mirina): Panic: file mail-index-sync.c: line 413
> > > > > (mail_index_sync_begin_to2): assertion failed: (!index->syncing)
> > > > > Nov 08 22:06:24 osaka dovecot[1450]: imap(mirina): Error: Raw
> > > > > backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x9e412) [0x7f20ae813412]
> > > > > -> /usr/lib/dovecot/libdovecot.so.0(+0x9e50d) [0x7f20ae81350d] ->
> > > > > /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f20ae7a2c51] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(+0xe0fe4) [0x7f20aeb88fe4] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_begin_to+0x4f)
> > > > > [0x7f20aeb890bf] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_begin+0x1c)
> > > > > [0x7f20aeb8915c] ->
> > > > > /usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x218)
> > > > > [0x7f20ad51f308] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
> > > > > [0x7f20aeaf02d4] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
> > > > > [0x7f20aeaf0387] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_get_status+0x31)
> > > > > [0x7f20aeb6bbf1] ->
> > > > > /usr/lib/dovecot/modules/lib20_virtual_plugin.so(+0x936d)
> > > > > [0x7f20ad51c36d] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa8771)
> > > > > [0x7f20aeb50771] ->
> > > > > /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xfd46) [0x7f20adb8bd46]
> > > > > -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_status+0x31)
> > > > > [0x7f20aeaf0781] ->
> > > > > /usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x10b8)
> > > > > [0x7f20ad5201a8] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
> > > > > [0x7f20aeaf02d4] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
> > > > > [0x7f20aeaf0387] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_get_status+0x31)
> > > > > [0x7f20aeb6bbf1] ->
> > > > > /usr/lib/dovecot/modules/lib20_virtual_plugin.so(+0x936d)
> > > > > [0x7f20ad51c36d] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa8771)
> > > > > [0x7f20aeb50771] ->
> > > > > /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xfd46) [0x7f20adb8bd46]
> > > > > -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_status+0x31)
> > > > > [0x7f20aeaf0781] ->
> > > > > /usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x10b8)
> > > > > [0x7f20ad5201a8] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
> > > > > [0x7f20aeaf02d4] ->
> > > > > /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
> > > > > [0x7f20aeaf0387] -> dovecot/imap(cmd_select_full+0x16b)
> > > > > [0x557de970cc3b] -> dovecot/imap(command_exec+0x5c) [0x557de971444c] 
> > > > > ->
> > > > > dovecot/imap(+0x1a912) [0x557de9712912]
> > > > > Nov 08 22:06:24 osaka dovecot[1450]: imap(mirina): Fatal: master:
> > > > > service(imap): child 4289 killed with signal 6 (core dumps disabled)
> > > > 
> > > > I attach my dovecot configuration too.
> > > > 
> > > > Thanks for your help.
> > > > 
> > 
> > Hello Aki,
> > 
> > You are right, the same error happens, even if I do not use the master 
> > password.
> > 
> > The weird thing is that it happens only with one specific account. For
> > this account, the virtual folders I have don't work, but they work for
> > the other accounts.
> > 
> > I do not have too much time to investigate now, but I will continue
> > this weekend.
> > 
> > Kind regards,
> > André
> 
> Happy new year, everyone!
> 
> Aki, I have been able to reproduce the problem, and this time, with the 
> packages from Debian stable. So you were right, this had nothing to do 
> with master user, but virtual folders.
> 
> I can now send the full stack trace, and doveconf.
> 
> Package versions:
> 
> ii  dovecot-core   1:2.2.27-3+deb9u2 amd64secure 
> POP3/IMAP server - core files
> ii  dovecot-dbg1:2.2.27-3+deb9u2 amd64secure 
> POP3/IMAP server - debug symbols
> ii  dovecot-imapd  1:2.2.27-3+deb9u2 amd64secure 
> POP3/IMAP server - IMAP daemon
> ii  dovecot-ldap   1:2.2.27-3+deb9u2 amd64secure 
> POP3/IMAP server - LDAP support
> ii  dovecot-lmtpd  1:2.2.27-3+deb9u2 amd64secure 
>

Re: Fatal: master: service(indexer-worker): child 493 killed with signal 11 (core dumped)

2019-01-12 Thread Stephan Bosch




Op 11/01/2019 om 21:07 schreef Larry Rosenman:

do you use both fts-solr and tika by chance?


I doubt it. The original post is about Lucene and it fails in the Lucene 
code. The FTS Solr + Tika problem is caused in Dovecot lib-http.


Regards,

Stephan.




On Fri, Jan 11, 2019 at 11:19 AM Fírvida > wrote:


Hello have you manage to solve your issue. I have a Mailu/Dovecot
server in
production which is giving me the same error. Any help would be
appreciated.



--
Sent from: http://dovecot.2317879.n4.nabble.com/



--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com 


US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106

Re: panic when using dovecot master account

2019-01-12 Thread André Rodier via dovecot


On 2018-11-09 07:40, André Rodier wrote:

On 2018-11-09 05:25, Aki Tuomi wrote:

This seems to have nothing to do with master account or not. Does this
happen if you try to open the virtual mailbox again?

Aki


On 09 November 2018 at 00:13 André Rodier  wrote:


Hello,

I am running dovecot 2.2.34 (874deae), on Debian stable, from 
backports.


I just tried the master account, and although everything worked in 
the

email client, I had logs in the error logs:


> imap(mirina): Panic: file mail-index-sync.c: line 413
> (mail_index_sync_begin_to2): assertion failed: (!index->syncing)
> Nov 08 22:06:24 osaka dovecot[1450]: imap(mirina): Error: Raw
> backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x9e412) [0x7f20ae813412]
> -> /usr/lib/dovecot/libdovecot.so.0(+0x9e50d) [0x7f20ae81350d] ->
> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f20ae7a2c51] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(+0xe0fe4) [0x7f20aeb88fe4] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_begin_to+0x4f)
> [0x7f20aeb890bf] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mail_index_sync_begin+0x1c)
> [0x7f20aeb8915c] ->
> 
/usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x218)
> [0x7f20ad51f308] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
> [0x7f20aeaf02d4] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
> [0x7f20aeaf0387] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_get_status+0x31)
> [0x7f20aeb6bbf1] ->
> /usr/lib/dovecot/modules/lib20_virtual_plugin.so(+0x936d)
> [0x7f20ad51c36d] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa8771)
> [0x7f20aeb50771] ->
> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xfd46) [0x7f20adb8bd46]
> -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_status+0x31)
> [0x7f20aeaf0781] ->
> 
/usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x10b8)
> [0x7f20ad5201a8] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
> [0x7f20aeaf02d4] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
> [0x7f20aeaf0387] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_get_status+0x31)
> [0x7f20aeb6bbf1] ->
> /usr/lib/dovecot/modules/lib20_virtual_plugin.so(+0x936d)
> [0x7f20ad51c36d] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xa8771)
> [0x7f20aeb50771] ->
> /usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xfd46) [0x7f20adb8bd46]
> -> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_get_status+0x31)
> [0x7f20aeaf0781] ->
> 
/usr/lib/dovecot/modules/lib20_virtual_plugin.so(virtual_storage_sync_init+0x10b8)
> [0x7f20ad5201a8] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
> [0x7f20aeaf02d4] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
> [0x7f20aeaf0387] -> dovecot/imap(cmd_select_full+0x16b)
> [0x557de970cc3b] -> dovecot/imap(command_exec+0x5c) [0x557de971444c] ->
> dovecot/imap(+0x1a912) [0x557de9712912]
> Nov 08 22:06:24 osaka dovecot[1450]: imap(mirina): Fatal: master:
> service(imap): child 4289 killed with signal 6 (core dumps disabled)

I attach my dovecot configuration too.

Thanks for your help.



Hello Aki,

You are right, the same error happens, even if I do not use the master 
password.


The weird thing is that it happens only with one specific account. For
this account, the virtual folders I have don't work, but they work for
the other accounts.

I do not have too much time to investigate now, but I will continue
this weekend.

Kind regards,
André


Happy new year, everyone!

Aki, I have been able to reproduce the problem, and this time, with the 
packages from Debian stable. So you were right, this had nothing to do 
with master user, but virtual folders.


I can now send the full stack trace, and doveconf.

Package versions:

ii  dovecot-core   1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - core files
ii  dovecot-dbg1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - debug symbols
ii  dovecot-imapd  1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - IMAP daemon
ii  dovecot-ldap   1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - LDAP support
ii  dovecot-lmtpd  1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - LMTP server
ii  dovecot-managesieved   1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - ManageSieve server
ii  dovecot-pop3d  1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - POP3 daemon
ii  dovecot-sieve  1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - Sieve filters support
ii  dovecot-solr   1:2.2.27-3+deb9u2 amd64secure 
POP3/IMAP server - Solr support



Kind regards,
André
























# 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.16 (fed8554)
# OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.6 
auth_debug = yes
auth_master_user_separator = /
auth_username_chars = 
abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01

Re: Solr -> Xapian ?

2019-01-12 Thread Joan Moreau via dovecot


additionally, my logic is that the backend stores one databalse per
mailox in /xapian-indexes (in the "root" dir of the user), the name od
the database is the GUID of the mailbox 


For INBOX, that works perfectly, and database is properly createdm and
backed starts indexing all emails 


For other folder, somehow, the process can not access that (root)
folder. 

Am I missing something ? 


On 2019-01-12 17:37, Joan Moreau wrote:

THank you 

Now, for the results 

I see the member of fts_result is : 


ARRAY_TYPE(seq_range) definite_uids;

I have the UID as a aray of uint32_t * 

How to put my UIDs into this "definite_uids" ? Obviously this is not a simple array/pointer. How to say someting similar to result->definite_uids[1]=my_uid ? 

On 2019-01-12 10:25, Timo Sirainen wrote: 
On 11 Jan 2019, at 21.23, Joan Moreau via dovecot  wrote: 
The below patch resolves the compilation error


$ diff -p compat.h compat.h.joan 
*** compat.h 2019-01-11 20:21:00.726625427 +0100

--- compat.h.joan 2019-01-11 20:14:41.729109919 +0100
*** struct iovec;
*** 202,207 
--- 202,211 
ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len);
#endif

+ #ifdef __cplusplus
+ extern "C" {
+ #endif

You should put this extern "C" into the C++ file you're creating. See for 
example how fts-lucene/lucene-wrapper.cc does this.

1 - WHat does represent "subargs" in mail_search_args 
It's set only for SEARCH_OR and SEARCH_SUB. So for example:


SEARCH TEXT foo TEXT bar TEXT baz

results in:

type=SEARCH_SUB
value.subargs = (
{ type=SEARCH, value.str="foo" },
{ type=SEARCH, value.str="bar" },
{ type=SEARCH, value.str="baz" },
)

Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other 
combination of OR/ANDs.
2 - for rescan : who is responsible for passing again the new email ? Is
the Dovecot core sending again all the emails to index ? or the fts
shall somehow access the mailbox and read all emails ? Wouldn't just be
saying "delete all index and get_last_uid is now 0" the easy way ? or
the fts must process all emails (and block the current thread as a
mailbx maybe quite large) 
The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.


3 - for get_last_uid : this uncertainity is very unclear. "If there is a
gap, then indexer first indexes all the missing" -> this mean at a
certain point, indexer maybe rebuilding a previous email, so *last* uid
is something different than max. And how indexer does know whther there
is a gap wihtout callong the fts backend (whch it does not as there are
no function for that) ? 
I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.


You can implement get_last_uid() simply by keeping track of it in 
dovecot.index* files, similar to how Lucene and Solr already do it with 
fts_index_get_header() / fts_index_set_header(). They also have a fallback that 
if the index doesn't have the last_uid value, they do a slower search from the 
Lucene/Solr index to find the last UID.

Re: Solr -> Xapian ?

2019-01-12 Thread Joan Moreau via dovecot

THank you 

Now, for the results 

I see the member of fts_result is : 


ARRAY_TYPE(seq_range) definite_uids;

I have the UID as a aray of uint32_t * 


How to put my UIDs into this "definite_uids" ? Obviously this is not a
simple array/pointer. How to say someting similar to
result->definite_uids[1]=my_uid ? 


On 2019-01-12 10:25, Timo Sirainen wrote:

On 11 Jan 2019, at 21.23, Joan Moreau via dovecot  wrote: 


The below patch resolves the compilation error

$ diff -p compat.h compat.h.joan 
*** compat.h 2019-01-11 20:21:00.726625427 +0100

--- compat.h.joan 2019-01-11 20:14:41.729109919 +0100
*** struct iovec;
*** 202,207 
--- 202,211 
ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len);
#endif

+ #ifdef __cplusplus
+ extern "C" {
+ #endif


You should put this extern "C" into the C++ file you're creating. See for 
example how fts-lucene/lucene-wrapper.cc does this.


1 - WHat does represent "subargs" in mail_search_args


It's set only for SEARCH_OR and SEARCH_SUB. So for example:

SEARCH TEXT foo TEXT bar TEXT baz

results in:

type=SEARCH_SUB
value.subargs = (
{ type=SEARCH, value.str="foo" },
{ type=SEARCH, value.str="bar" },
{ type=SEARCH, value.str="baz" },
)

Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other 
combination of OR/ANDs.


2 - for rescan : who is responsible for passing again the new email ? Is
the Dovecot core sending again all the emails to index ? or the fts
shall somehow access the mailbox and read all emails ? Wouldn't just be
saying "delete all index and get_last_uid is now 0" the easy way ? or
the fts must process all emails (and block the current thread as a
mailbx maybe quite large)


The next indexing run is responsible for it. If you return get_last_uid=0, then 
indexer starts feeding you all mails. So fts backend doesn't have to know about 
it.


3 - for get_last_uid : this uncertainity is very unclear. "If there is a
gap, then indexer first indexes all the missing" -> this mean at a
certain point, indexer maybe rebuilding a previous email, so *last* uid
is something different than max. And how indexer does know whther there
is a gap wihtout callong the fts backend (whch it does not as there are
no function for that) ?


I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 
have been indexed by the FTS backend. It's possible that at this point there 
are already mails with UIDs 101..200 in the folder. So when UID=201 is 
delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so 
far, and starts feeding it UIDs 101..201 in that order.

You can implement get_last_uid() simply by keeping track of it in 
dovecot.index* files, similar to how Lucene and Solr already do it with 
fts_index_get_header() / fts_index_set_header(). They also have a fallback that 
if the index doesn't have the last_uid value, they do a slower search from the 
Lucene/Solr index to find the last UID.

Re: Migration

2019-01-12 Thread Tanstaafl

On 12/31/2018, 5:22:48 AM, Ignacio García  wrote:
> A totally different approach (that is imap-server agnostic), providing 
> that you're setting up those new accounts with temporary passwords 
> (which you know), before users change their passwords to their liking: 
> you could also use imapsync ( https://github.com/imapsync/imapsync) . We 
> here use it with a batch file and a text file containing all accounts to 
> do mass-migrations, usually at night, when there's little to none user 
> interaction with their mail accounts. I like this approach because mail 
> service never gets interrupted and we do programmed syncs all night in 
> case DNS propagation takes more than expected and mail still arrives to 
> the old server.

Or, you can use Master Passwords on both sides, and just do the
migration at your leisure. I did this when migrating from our dovecot
server to Office 365.

Next time I'll look at using a dovecot method (DSync? doveadm?) but
still using Master Passwords.

Re: [FTS Xapian] Status & Questions

2019-01-12 Thread Joan Moreau via dovecot


Sorted this out. sorry for noise

On 2019-01-12 11:39, Joan Moreau wrote:

The change of "Extern C" suggested by Timo actually solved the situation 

Now, further question : 

I put a "i_warning" at each of my functions, and I see in the log : 


Jan 12 10:33:27 
indexer-worker(j...@grosjo.net)<30970>:
 Warning: fts_backend_xapian_alloc
Jan 12 10:33:27 
indexer-worker(j...@grosjo.net)<30970>:
 Warning: fts_backend_xapian_init
Jan 12 10:33:27 
indexer-worker(j...@grosjo.net)<30970>:
 Warning: fts_xapian init (3,25)
Jan 12 10:33:27 
indexer-worker(j...@grosjo.net)<30970>:
 Warning: fts_backend_xapian_get_last_uid
Jan 12 10:33:27 
indexer-worker(j...@grosjo.net)<30970>:
 Error: Mailbox XXX: Status lookup failed: Internal error occurred. Refer to server log for 
more information. [2019-01-12 10:33:27]
Jan 12 10:33:27 
indexer-worker(j...@grosjo.net)<30970>:
 Warning: fts_backend_xapian_deinit
Jan 12 10:33:27 
indexer-worker(j...@grosjo.net)<30970>:
 Warning: fts_backend_xapian_unset_box
Jan 12 10:33:27 lda(j...@grosjo.net)<31367>: Warning: fts_backend_xapian_deinit 

The error comes from the fact taht the "get_last_uid" is called without selecting a mailbox first, and therefore there is of course no uid. 

How come ? 

On 2019-01-12 10:50, Aki Tuomi wrote: 
Did you remember to load fts first? 

mail_plugins =$mail_plugins fts fts_xapian 

Aki 
On 12 January 2019 at 10:37 Joan Moreau via dovecot < dovecot@dovecot.org> wrote: 

STATUS 

- Alpha code is written and compiling now. (attached) 

- I would like to start testing. However, there is an error when 
starting dovecot (git) : 

Error: Couldn't load required plugin 
/usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: 
/usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: 
_Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg 

ldd shows that fts lib is properly linked: 
# ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so 
(...) 
lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so 
(0x7f25f75e) 
(...) 
libxapian.so.30 => /usr/lib/libxapian.so.30 (0x7fe3a51e2000) 

Your help very welcome 

PENDING QUESTIONS 

1 - WHat does represent "subargs" in mail_search_args 

2 - for rescan : who is responsible for passing again the new email ? Is 

the Dovecot core sending again all the emails to index ? or the fts 
shall somehow access the mailbox and read all emails ? Wouldn't just be 
saying "delete all index and get_last_uid is now 0" the easy way ? or 
the fts must process all emails (and block the current thread as a 
mailbx maybe quite large) 

3 - for get_last_uid : this uncertainity is very unclear. "If there is a 

gap, then indexer first indexes all the missing" -> this mean at a 
certain point, indexer maybe rebuilding a previous email, so *last* uid 
is something different than max. And how indexer does know whther there 
is a gap wihtout callong the fts backend (whch it does not as there are 
no function for that) ? 

Thank you 

--- 
Aki Tuomi

Re: [FTS Xapian] Status & Questions

2019-01-12 Thread Joan Moreau via dovecot


The change of "Extern C" suggested by Timo actually solved the situation


Now, further question : 

I put a "i_warning" at each of my functions, and I see in the log : 


Jan 12 10:33:27
indexer-worker(j...@grosjo.net)<30970>:
Warning: fts_backend_xapian_alloc
Jan 12 10:33:27
indexer-worker(j...@grosjo.net)<30970>:
Warning: fts_backend_xapian_init
Jan 12 10:33:27
indexer-worker(j...@grosjo.net)<30970>:
Warning: fts_xapian init (3,25)
Jan 12 10:33:27
indexer-worker(j...@grosjo.net)<30970>:
Warning: fts_backend_xapian_get_last_uid
Jan 12 10:33:27
indexer-worker(j...@grosjo.net)<30970>:
Error: Mailbox XXX: Status lookup failed: Internal error occurred. Refer
to server log for more information. [2019-01-12 10:33:27]
Jan 12 10:33:27
indexer-worker(j...@grosjo.net)<30970>:
Warning: fts_backend_xapian_deinit
Jan 12 10:33:27
indexer-worker(j...@grosjo.net)<30970>:
Warning: fts_backend_xapian_unset_box
Jan 12 10:33:27 lda(j...@grosjo.net)<31367>:
Warning: fts_backend_xapian_deinit 


The error comes from the fact taht the "get_last_uid" is called without
selecting a mailbox first, and therefore there is of course no uid. 

How come ? 


On 2019-01-12 10:50, Aki Tuomi wrote:

Did you remember to load fts first? 

mail_plugins =$mail_plugins fts fts_xapian 

Aki 

On 12 January 2019 at 10:37 Joan Moreau via dovecot < dovecot@dovecot.org> wrote: 

STATUS 

- Alpha code is written and compiling now. (attached) 

- I would like to start testing. However, there is an error when 
starting dovecot (git) : 

Error: Couldn't load required plugin 
/usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: 
/usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: 
_Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg 

ldd shows that fts lib is properly linked: 
# ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so 
(...) 
lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so 
(0x7f25f75e) 
(...) 
libxapian.so.30 => /usr/lib/libxapian.so.30 (0x7fe3a51e2000) 

Your help very welcome 

PENDING QUESTIONS 

1 - WHat does represent "subargs" in mail_search_args 

2 - for rescan : who is responsible for passing again the new email ? Is 

the Dovecot core sending again all the emails to index ? or the fts 
shall somehow access the mailbox and read all emails ? Wouldn't just be 
saying "delete all index and get_last_uid is now 0" the easy way ? or 
the fts must process all emails (and block the current thread as a 
mailbx maybe quite large) 

3 - for get_last_uid : this uncertainity is very unclear. "If there is a 

gap, then indexer first indexes all the missing" -> this mean at a 
certain point, indexer maybe rebuilding a previous email, so *last* uid 
is something different than max. And how indexer does know whther there 
is a gap wihtout callong the fts backend (whch it does not as there are 
no function for that) ? 


Thank you


--- 
Aki Tuomi

Re: [FTS Xapian] Status & Questions

2019-01-12 Thread Aki Tuomi



 
 
  
   Did you remember to load fts first?
  
  
   
  
  
   mail_plugins =$mail_plugins fts fts_xapian
  
  
   
  
  
   Aki
  
  
   
On 12 January 2019 at 10:37 Joan Moreau via dovecot <
dovecot@dovecot.org> wrote:
   
   

   
   

   
   
STATUS
   
   

   
   
- Alpha code is written and compiling now. (attached)
   
   

   
   
- I would like to start testing. However, there is an error when
   
   
starting dovecot (git) :
   
   

   
   
Error: Couldn't load required plugin
   
   
/usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed:
   
   
/usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol:
   
   
_Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg
   
   

   
   
ldd shows that fts lib is properly linked:
   
   
# ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so
   
   
(...)
   
   
lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so
   
   
(0x7f25f75e)
   
   
(...)
   
   
libxapian.so.30 => /usr/lib/libxapian.so.30 (0x7fe3a51e2000)
   
   

   
   
Your help very welcome
   
   

   
   
PENDING QUESTIONS
   
   

   
   
1 - WHat does represent "subargs" in mail_search_args
   
   

   
   
2 - for rescan : who is responsible for passing again the new email ? Is
   
   

   
   
the Dovecot core sending again all the emails to index ? or the fts
   
   
shall somehow access the mailbox and read all emails ? Wouldn't just be
   
   
saying "delete all index and get_last_uid is now 0" the easy way ? or
   
   
the fts must process all emails (and block the current thread as a
   
   
mailbx maybe quite large)
   
   

   
   
3 - for get_last_uid : this uncertainity is very unclear. "If there is a
   
   

   
   
gap, then indexer first indexes all the missing" -> this mean at a
   
   
certain point, indexer maybe rebuilding a previous email, so *last* uid
   
   
is something different than max. And how indexer does know whther there
   
   
is a gap wihtout callong the fts backend (whch it does not as there are
   
   
no function for that) ?
   
   

   
   
Thank you
   
  
  
   
  
  
   ---
   Aki Tuomi

Re: Solr -> Xapian ?

2019-01-12 Thread Timo Sirainen

On 11 Jan 2019, at 21.23, Joan Moreau via dovecot  wrote:
> 
> The below patch resolves the compilation error
> 
> $ diff -p compat.h compat.h.joan 
> *** compat.h 2019-01-11 20:21:00.726625427 +0100
> --- compat.h.joan 2019-01-11 20:14:41.729109919 +0100
> *** struct iovec;
> *** 202,207 
> --- 202,211 
> ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len);
> #endif
> 
> + #ifdef __cplusplus
> + extern "C" {
> + #endif
> 

You should put this extern "C" into the C++ file you're creating. See for 
example how fts-lucene/lucene-wrapper.cc does this.

> 1 - WHat does represent "subargs" in mail_search_args

It's set only for SEARCH_OR and SEARCH_SUB. So for example:

SEARCH TEXT foo TEXT bar TEXT baz

results in:

type=SEARCH_SUB
value.subargs = (
  { type=SEARCH, value.str="foo" },
  { type=SEARCH, value.str="bar" },
  { type=SEARCH, value.str="baz" },
)

Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other 
combination of OR/ANDs.

> 2 - for rescan : who is responsible for passing again the new email ? Is
> the Dovecot core sending again all the emails to index ? or the fts
> shall somehow access the mailbox and read all emails ? Wouldn't just be
> saying "delete all index and get_last_uid is now 0" the easy way ? or
> the fts must process all emails (and block the current thread as a
> mailbx maybe quite large)

The next indexing run is responsible for it. If you return get_last_uid=0, then 
indexer starts feeding you all mails. So fts backend doesn't have to know about 
it.

> 3 - for get_last_uid : this uncertainity is very unclear. "If there is a
> gap, then indexer first indexes all the missing" -> this mean at a
> certain point, indexer maybe rebuilding a previous email, so *last* uid
> is something different than max. And how indexer does know whther there
> is a gap wihtout callong the fts backend (whch it does not as there are
> no function for that) ?

I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 
have been indexed by the FTS backend. It's possible that at this point there 
are already mails with UIDs 101..200 in the folder. So when UID=201 is 
delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so 
far, and starts feeding it UIDs 101..201 in that order.

You can implement get_last_uid() simply by keeping track of it in 
dovecot.index* files, similar to how Lucene and Solr already do it with 
fts_index_get_header() / fts_index_set_header(). They also have a fallback that 
if the index doesn't have the last_uid value, they do a slower search from the 
Lucene/Solr index to find the last UID.

[FTS Xapian] Status & Questions

2019-01-12 Thread Joan Moreau via dovecot

STATUS 

- Alpha code is written and compiling now. (attached) 


- I would like to start testing. However, there is an error when
starting dovecot (git) : 


Error: Couldn't load required plugin
/usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed:
/usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol:
_Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg 

ldd shows that  fts lib is properly linked: 
# ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so 
(...) 
lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so

(0x7f25f75e)
(...) 
libxapian.so.30 => /usr/lib/libxapian.so.30 (0x7fe3a51e2000) 

Your help very welcome 

PENDING QUESTIONS 

1 - WHat does represent "subargs" in mail_search_args 


2 - for rescan : who is responsible for passing again the new email ? Is

the Dovecot core sending again all the emails to index ? or the fts 
shall somehow access the mailbox and read all emails ? Wouldn't just be 
saying "delete all index and get_last_uid is now 0" the easy way ? or 
the fts must process all emails (and block the current thread as a 
mailbx maybe quite large) 


3 - for get_last_uid : this uncertainity is very unclear. "If there is a

gap, then indexer first indexes all the missing" -> this mean at a 
certain point, indexer maybe rebuilding a previous email, so *last* uid 
is something different than max. And how indexer does know whther there 
is a gap wihtout callong the fts backend (whch it does not as there are 
no function for that) ? 


Thank you

dovecot-xapian.tar.gz
Description: GNU Zip compressed data

Re: [solved] managesieve configuration

Re: Issue with LMTP proxying and port number

Re: Issue with LMTP proxying and port number

Re: Dovecot Submission Proxy Auth

Re: Solr -> Xapian ?

Re: degfault imaptest

Re: degfault imaptest

Re: Error: User b...@aaa.bbb doesn't have home dir set, disabling duplicate database

Re: [solved] managesieve configuration

Re: Fatal: master: service(indexer-worker): child 493 killed with signal 11 (core dumped)

Re: panic when using dovecot master account

Re: Fatal: master: service(indexer-worker): child 493 killed with signal 11 (core dumped)

Re: panic when using dovecot master account

Re: Solr -> Xapian ?

Re: Solr -> Xapian ?

Re: Migration

Re: [FTS Xapian] Status & Questions

Re: [FTS Xapian] Status & Questions

Re: [FTS Xapian] Status & Questions

Re: Solr -> Xapian ?

[FTS Xapian] Status & Questions

21 matches

Site Navigation

Mail list logo

Footer information