Re: [FTS Xapian] Beta release

[Odhiambo Washington]

In your README.md, perhaps "This project intends to provide a
straightforward and simple *procedure *to configure FTS plugin for Dovecot,
leveraging the efforts by the Xapian.org team." is better??
Also in the part after cloning from git:

./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This
/path/to/dovecot is not obvious. Is it the dovecot binary or what??]

On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot 
wrote:

> Thank you Stephan.
>
> The version here shall be up and running :
> https://github.com/grosjo/fts-xapian
>
>
>
>
>
> On 2019-01-14 00:07, Stephan Bosch wrote:
>
>
>
> Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:
>
>
> I tried to combined it, the "autoreconf" errors are solved
>
> Now, when I type "make install", the lib is not pushed into dovecot
> folder, but somewhere in /usr/local/...
>
> How to adjust this to have it arriving in the proper folder ?
>
>
> Depends on your system. It mostly a matter of setting a proper --prefix
> directory for configure, but other paths are configurable as well. I
> usually check what the official distribution package for Dovecot is doing
> and use that as a basis.
>
> For Debian I use the following configure command:
>
> ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin
> --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \
> --with-gssapi=plugin --with-solr --with-ioloop=best
> --enable-maintainer-mode \
> --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
> --localstatedir=/var --mandir=/usr/share/man \
> --infodir=/usr/share/info
> --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static
>
> Regards,
>
> Stephan
>
>
> On 2019-01-13 21:01, Tuomi, Aki wrote:
>
> You copied your Makefile.am there. Stephan made you a working version, can
> you try that?
> (sorry for dup)
> Aki
>  Original message 
> From: Joan Moreau 
> Date: 13/01/2019 21:39 (GMT+02:00)
> To: Stephan Bosch 
> Cc: Aki Tuomi 
> Subject: Re: [FTS Xapian] Beta release
>
> I used the skeleton from Aki : https://github.com/grosjo/fts-xapian
>
> However, when I try to act as a visitor, I reach teh follwoing error:
>
> # autoreconf -vi
> autoreconf: Entering directory `.'
> autoreconf: configure.ac: not using Gettext
> autoreconf: running: aclocal -I m4
> autoreconf: configure.ac: tracing
> autoreconf: running: libtoolize --copy
> libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
> libtoolize: copying file './ltmain.sh'
> libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
> libtoolize: copying file 'm4/libtool.m4'
> libtoolize: copying file 'm4/ltoptions.m4'
> libtoolize: copying file 'm4/ltsugar.m4'
> libtoolize: copying file 'm4/ltversion.m4'
> libtoolize: copying file 'm4/lt~obsolete.m4'
> autoreconf: running: /usr/bin/autoconf
> autoreconf: running: /usr/bin/autoheader
> autoreconf: running: automake --add-missing --copy --no-force
> configure.ac:9: installing './compile'
> configure.ac:11: installing './config.guess'
> configure.ac:11: installing './config.sub'
> configure.ac:7: installing './install-sh'
> configure.ac:7: installing './missing'
> src/Makefile.am: installing './depcomp'
> /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not
> appear in AM_CONDITIONAL
> /usr/share/automake-1.16/am/depend2.am: The usual way to define
> 'am__fastdepCXX' is to add 'AC_PROG_CXX'
> /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run
> 'aclocal' and 'autoconf' again
> src/Makefile.am: error: C++ source seen but 'CXX' is undefined
> src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
> src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
> src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no
> program or
> src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible
> typo)
> autoreconf: automake failed with exit status: 1
>
>
>
> On 2019-01-13 20:24, Stephan Bosch wrote:
>
> Oh, right, a distribution tarball doesn't include some of the
> necessary files for your repository like autogen.sh and
> .gitignore. The attached tarball includes all those and is ready
> for `git init`. The previous tarball was made with `make
> distcheck` from this one.
>
> Regards,
>
> Stephan.
>
> Op 13/01/2019 om 20:14 schreef Stephan Bosch:
>
> Hi Joan,
>
> Op 13/01/2019 om 19:03 schreef Aki Tuomi:
>
> Yes, from compiling point of view it is done.
>
> Unfortunately what is not done is all the other work
> involved, such as fixing all the inevitable bugs it has
> and maintaining it. We do not want, at this moment, take
> up maintaining and developing yet another FTS plugin as
> we have plenty of things to do already.
>
> I invite you to setup your own repository and provide
> this plugin from there, being the maintainer of this
> plugin. We can add 

Re: Solr - complete setup (update)

[Joan Moreau via dovecot]

Hi Stephan, 

What's up with that ? 

Thank you so much 


On 2019-01-05 02:04, Stephan Bosch wrote:


Hi,

Op 04/01/2019 om 05:36 schreef Joan Moreau via dovecot: 


Hi

This is the summary of my work with SOLR-Dovecot, in my *quest to reproduce the 
previoulsy excellent work of fts_squat*

@Aki : Based on the time I have spent on this, I would love to see you updating 
the Wiki with those improvements, and adding my name somewhere

@All : Hope it helps

I'll be going through the description below soon. I've recently independently 
installed fts-solr from scratch. Although this wasn't a flawless effort, I 
managed to get some basic indexing going. From this mail thread I understand 
that there are quite a few more problems than I've seen myself so far. Then 
again, I didn't perform extensive tests with actual searches.

Maybe we can turn all this into a test suite that we can run internally here at 
Dovecot. At the very least, the described Dovecot bugs need to be addressed and 
the wiki needs to be updated.

I'll get back to you.

Regards,

Stephan.


*- Installation:*

-> Create a clean install using the default, (at least in the Archlinux package), and do 
a "sudo -u solr solr create -c dovecot ". The config files are then in 
/opt/solr/server/solr/dovecot/conf and datafiles in /opt/solr/server/solr/dovecot/data

-> In /opt/solr/server/solr/dovecot/conf/solrconfig.xml:

* around line 313, change false to 
true

* around line 147, set 2000 (or above)

* around line 696 : uncomment hdr

* around line 1127, before , 
add 

* around line 1161, delete the whole 

* around line 1192, remove the whole 

-> Remove /opt/solr/server/solr/dovecot/conf/managed-schema

-> Change "schema.xml" by the one below to reproduce fts_squat behavior  (equivalent to 
" fts_squat = partial=3 full=25" in dovecot.conf) (note : such a huge trouble to replace a 
single line setup, anyway...)

-> Move /opt/solr/server/solr (or the subfolder data) to a partition with 
*space*, ideally ext4 or faster file system (it looks like Solr is not considering 
using a simple mysql database, which would make sense to avoid all the fuzz and 
let it transit to a non-java state, but that is another story)

-> Config of dovecot.conf is as below

-> The systemd unit shall specify high ulimit for files and proc (see below)

-> Increase the memory available for the JavaVM (I put 12Gb as I have quite a space on my 
server, but you may adapt it as per your specs) : in /opt/solr/bin/solr.in.sh, set 
SOLR_HEAP="12288m"

-> As Solr is complaining a lot, you may consider a filter for it in your 
syslog-ng or journald as it pollutes greatly your audit files

-> (re)Start solr (first) and dovecot by systemctl

-> Launch redindex ( doveadm fts rescan -u  )

-> wait for a big while to let the system re-index all your mail boxes

*- Bugs so far*

-> Line 620 of fts_solr dovecot plugin : the size oof header is improperly calculated 
("huge header" warning for a simple email, which kilss the index of that 
considered email, so basically MOST emails as the calculation is wrong)

-> The UID returned by SOlr is to be considered as a STRING (and that is maybe the source of 
problem of the "out of bound" errors in fts_solr dovecot, as "long" is not enough)

-> Java errors : A lot of non sense for me, I am not expert in Java. But, with 
increased memory, it seems not crashing, even if complaining quite a lot in the 
logs

*---SCHEMA.XML in /opt/solr/server/solr/dovecot/conf*



id















































*-- DOVECOT.CONF*

mail_plugins = fts fts_solr

plugin {
plugin = fts fts_solr managesieve sieve

fts = solr
fts_autoindex = yes
fts_enforced = yes
fts_solr = url=http://127.0.0.1:8983/solr/dovecot/

(replace 127.0.0.1 by your solr server if you want to use an external server)
(...)

}

*-- /etc/systemd/system/multi-user.target.wants/solr.service*

[Unit]
Description=Solr full text search engine
After=network.target

[Service]
Type=simple
User=solr
Group=solr
PrivateTmp=yes
WorkingDirectory=/opt/solr
*LimitNOFILE=65000*
*LimitNPROC=65000*
ExecStart=/opt/solr/bin/solr start -f

[Install]
WantedBy=multi-user.target

Re: [FTS Xapian] Beta release

[Joan Moreau via dovecot]

Thank you Stephan. 


The version here shall be up and running :
https://github.com/grosjo/fts-xapian 


On 2019-01-14 00:07, Stephan Bosch wrote:

Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: 


I tried to combined it, the "autoreconf" errors are solved

Now, when I type "make install", the lib is not pushed into dovecot folder, but 
somewhere in /usr/local/...

How to adjust this to have it arriving in the proper folder ?


Depends on your system. It mostly a matter of setting a proper --prefix 
directory for configure, but other paths are configurable as well. I usually 
check what the official distribution package for Dovecot is doing and use that 
as a basis.

For Debian I use the following configure command:

./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin 
--with-lua=plugin --with-pgsql --with-mysql --with-sqlite \
--with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \
--prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var 
--mandir=/usr/share/man \
--infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules 
--disable-rpath --disable-static

Regards,

Stephan

On 2019-01-13 21:01, Tuomi, Aki wrote:

You copied your Makefile.am there. Stephan made you a working version, can you 
try that?
(sorry for dup)
Aki
 Original message 
From: Joan Moreau 
Date: 13/01/2019 21:39 (GMT+02:00)
To: Stephan Bosch 
Cc: Aki Tuomi 
Subject: Re: [FTS Xapian] Beta release

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian

However, when I try to act as a visitor, I reach teh follwoing error:

# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear 
in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define 
'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 
'autoconf' again
src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no 
program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
autoreconf: automake failed with exit status: 1

On 2019-01-13 20:24, Stephan Bosch wrote:

Oh, right, a distribution tarball doesn't include some of the
necessary files for your repository like autogen.sh and
.gitignore. The attached tarball includes all those and is ready
for `git init`. The previous tarball was made with `make
distcheck` from this one.

Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch:

Hi Joan,

Op 13/01/2019 om 19:03 schreef Aki Tuomi:

Yes, from compiling point of view it is done.

Unfortunately what is not done is all the other work
involved, such as fixing all the inevitable bugs it has
and maintaining it. We do not want, at this moment, take
up maintaining and developing yet another FTS plugin as
we have plenty of things to do already.

I invite you to setup your own repository and provide
this plugin from there, being the maintainer of this
plugin. We can add a link to your plugin on our FTS page
so people can also find it.

There are other plugins like this, e.g.
https://github.com/st3fan/dovecot-xaps-plugin

I turned the code you provided into a separate plugin
package. The distribution tarball is attached.

Notable changes:

- Added example copyright headers and COPYING and AUTHORS
files. You should modify those to your preference.
- Added README and INSTALL files (in markdown using Pandoc).
Those need to be amended with details.
- Amended the plugin code to display a debug message with the
plugin name and version upon plugin load.

I advise you to turn this into a git repository and continue from there.

I do not recommend releasing this plugin with the
-fpermissive flag and the resulting warning as it is now. But
I'm assuming this is still a work in progress, so that is OK.

Re: dovecot/config processes one more time - which are safe to kill?

[Timo Sirainen]

On 13 Dec 2018, at 11.18, Arkadiusz Miśkiewicz  wrote:
> 
> 
> Hello.
> 
> The problem with dovecot/config processes never ending and spawning new
> one on each reload
> (https://www.dovecot.org/list/dovecot/2016-November/106058.html) is
> becoming a problem here:
> 
> # ps aux|grep dovecot/config|wc -l
> 206

I think you also have 206 other Dovecot processes that are keeping the config 
process open? Maybe 206 imap-login processes or something? Anyway I'd expect 
that this would happen only if some other process is keeping a UNIX socket 
connection open to the config process. Unless of course there's some bug that 
just isn't shutting them down even though they don't have any connections.. But 
at least I couldn't easily reproduce that.

I suppose there isn't much of a reason for existing processes to keep the 
config socket open after reload, so a patch like below would likely help. 
Although it probably should be delayed so that existing imap/pop3-login 
connections doing STARTTLS wouldn't fail if that causes a new config lookup.

diff --git a/src/lib-master/master-service.c b/src/lib-master/master-service.c
index 3de11fa1b..41005cb5e 100644
--- a/src/lib-master/master-service.c
+++ b/src/lib-master/master-service.c
@@ -815,6 +815,7 @@ void master_service_stop_new_connections(struct 
master_service *service)
}
if (service->login != NULL)
master_login_stop(service->login);
+   master_service_close_config_fd(service);
 }

 bool master_service_is_killed(struct master_service *service)

> That's a lot of wasted memory - dovecot/config processes ate over 30GB
> of ram on 64GB box.

Are you saying your config processes take 149 MB each? That doesn't sound 
right, unless you have a huge number of SSL certificates?

Re: Indexing paralelism

[Timo Sirainen]

On 13 Jan 2019, at 19.23, Joan Moreau via dovecot  wrote:
> 
> Hi
> 
> Observing the processes of FTS, I observe the following:
> 
> 
> 
> 1 - For one user, indexer-wroker does not start several threads for each 
> request. On teh contrary, it waits for the first request to finish before 
> starting the second. How to make sure all requests (or a limited number of 
> it, for instance linked to the CPU number on the machine) are started asap ?

I guess two answers:

1) Dovecot doesn't use threads anywhere. It uses only processes. So a single 
indexer-worker couldn't start multiple parallel threads.

2) It's intentional that there aren't more than one indexer-worker per user. 
This is especially because fts-lucene would break if there were more. Also 
generally in larger installations it's better if all the workers weren't stuck 
processing a single user, blocking other users' indexing.

> 2 - If a IMAP query is received, the dovecot checks the last UID from FTS and 
> launch a request of indexing to finish the index *before* running the search 
> query . THis creates timeouts (and can take a while if many request are 
> pending - see point 1) How to prevent that (i.e. the serach request is 
> launched (read only) no matter what ? THe completeion of missing UIDs is 
> launched in a separate thread ?

This would be violating IMAP protocol if it didn't include latest mails in the 
SEARCH response.. Generally I haven't noticed this being a big practical 
problem. The initial indexing can be done before FTS searches are enabled for 
the user, and afterwards the indexing shouldn't have especially long queues. 
Note that when doing indexing due to a SEARCH, that indexing's priority is 
higher than the indexing triggered by new mail deliveries. So unless all the 
indexer-workers are busy indexing mails from large folders without any indexes, 
this shouldn't be a huge problem normally.

Re: Solr -> Xapian ?

[Timo Sirainen]

On 13 Jan 2019, at 10.45, Joan Moreau via dovecot  wrote:
> 
> Now, I can see in the logs that several times, the dovecot calls the 
> fts_backend_xapian_update_set_mailbox with box == NULL. WHy so ?
> 
fts-api.h says:

/* Switch to updating the specified mailbox. box may also be set to NULL to
   make sure the previous mailbox won't tried to be accessed anymore. */
void fts_backend_update_set_mailbox(struct fts_backend_update_context *ctx,
struct mailbox *box);

So it's just telling you that you can close/free any stuff related to that 
mailbox.
>> additionally, my logic is that the backend stores one databalse per mailox 
>> in /xapian-indexes (in the "root" dir of the user), the name od the database 
>> is the GUID of the mailbox
>> 
>> For INBOX, that works perfectly, and database is properly createdm and 
>> backed starts indexing all emails
>> 
>> For other folder, somehow, the process can not access that (root) folder.
>> 
>> Am I missing something ?
>> 

This is a bit ambiguous, because some people mean mailbox=folder and others 
mean mailbox=user account, and GUID can also be the internal Dovecot folder 
GUID, or a GUID of the user.

I'd recommend using a single database per user anyway.

Re: [FTS Xapian] Beta release

[Stephan Bosch]

Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot:


I tried to combined it, the "autoreconf" errors are solved

Now, when I type "make install", the lib is not pushed into dovecot 
folder, but somewhere in /usr/local/...


How to adjust this to have it arriving in the proper folder ?



Depends on your system. It mostly a matter of setting a proper --prefix 
directory for configure, but other paths are configurable as well. I 
usually check what the official distribution package for Dovecot is 
doing and use that as a basis.


For Debian I use the following configure command:

./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin 
--with-lua=plugin --with-pgsql --with-mysql --with-sqlite \
    --with-gssapi=plugin --with-solr --with-ioloop=best 
--enable-maintainer-mode \
    --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib 
--localstatedir=/var --mandir=/usr/share/man \
    --infodir=/usr/share/info 
--with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static


Regards,

Stephan



On 2019-01-13 21:01, Tuomi, Aki wrote:

You copied your Makefile.am there. Stephan made you a working 
version, can you try that?

(sorry for dup)
Aki
 Original message 
From: Joan Moreau 
Date: 13/01/2019 21:39 (GMT+02:00)
To: Stephan Bosch 
Cc: Aki Tuomi 
Subject: Re: [FTS Xapian] Beta release

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian

However, when I try to act as a visitor, I reach teh follwoing error:

# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does 
not appear in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define 
'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 
'aclocal' and 'autoconf' again

src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined 
but no program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name 
(possible typo)

autoreconf: automake failed with exit status: 1



On 2019-01-13 20:24, Stephan Bosch wrote:

Oh, right, a distribution tarball doesn't include some of the
necessary files for your repository like autogen.sh and
.gitignore. The attached tarball includes all those and is ready
for `git init`. The previous tarball was made with `make
distcheck` from this one.

Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch:

Hi Joan,

Op 13/01/2019 om 19:03 schreef Aki Tuomi:

Yes, from compiling point of view it is done.

Unfortunately what is not done is all the other work
involved, such as fixing all the inevitable bugs it has
and maintaining it. We do not want, at this moment, take
up maintaining and developing yet another FTS plugin as
we have plenty of things to do already.

I invite you to setup your own repository and provide
this plugin from there, being the maintainer of this
plugin. We can add a link to your plugin on our FTS page
so people can also find it.

There are other plugins like this, e.g.
https://github.com/st3fan/dovecot-xaps-plugin


I turned the code you provided into a separate plugin
package. The distribution tarball is attached.

Notable changes:

- Added example copyright headers and COPYING and AUTHORS
files. You should modify those to your preference.
- Added README and INSTALL files (in markdown using Pandoc).
Those need to be amended with details.
- Amended the plugin code to display a debug message with the
plugin name and version upon plugin load.

I advise you to turn this into a git repository and continue from there.

Re: [FTS Xapian] Beta release

[Joan Moreau via dovecot]

(or wherever dovecot installed its libraries)

On 2019-01-13 21:25, Joan Moreau via dovecot wrote:

I tried to combined it, the "autoreconf" errors are solved 

Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... 


How to adjust this to have it arriving in the proper folder ?

On 2019-01-13 21:01, Tuomi, Aki wrote: 
You copied your Makefile.am there. Stephan made you a working version, can you try that? 

(sorry for dup) 

Aki 

 Original message  
From: Joan Moreau  
Date: 13/01/2019 21:39 (GMT+02:00) 
To: Stephan Bosch  
Cc: Aki Tuomi  
Subject: Re: [FTS Xapian] Beta release 

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian 

However, when I try to act as a visitor, I reach teh follwoing error: 


# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear 
in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define 
'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 
'autoconf' again
src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no 
program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
autoreconf: automake failed with exit status: 1 

On 2019-01-13 20:24, Stephan Bosch wrote: 
Oh, right, a distribution tarball doesn't include some of the necessary files for your repository like autogen.sh and .gitignore. The attached tarball includes all those and is ready for `git init`. The previous tarball was made with `make distcheck` from this one.


Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch: Hi Joan,

Op 13/01/2019 om 19:03 schreef Aki Tuomi: Yes, from compiling point of view it 
is done.

Unfortunately what is not done is all the other work involved, such as fixing 
all the inevitable bugs it has and maintaining it. We do not want, at this 
moment, take up maintaining and developing yet another FTS plugin as we have 
plenty of things to do already.

I invite you to setup your own repository and provide this plugin from there, 
being the maintainer of this plugin. We can add a link to your plugin on our 
FTS page so people can also find it.

There are other plugins like this, e.g. https://github.com/st3fan/dovecot-xaps-plugin 
I turned the code you provided into a separate plugin package. The distribution tarball is attached.


Notable changes:

- Added example copyright headers and COPYING and AUTHORS files. You should 
modify those to your preference.
- Added README and INSTALL files (in markdown using Pandoc). Those need to be 
amended with details.
- Amended the plugin code to display a debug message with the plugin name and 
version upon plugin load.

I advise you to turn this into a git repository and continue from there.

I do not recommend releasing this plugin with the -fpermissive flag and the 
resulting warning as it is now. But I'm assuming this is still a work in 
progress, so that is OK.

Regards,

Stephan.

On 13 January 2019 at 19:52 Joan Moreau  wrote:

The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).

I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly

The only remaining point is to push it in hte git (yes, everything is
already done)

On 2019-01-13 18:45, Aki Tuomi wrote:

On 13 January 2019 at 17:05 Joan Moreau via dovecot  wrote:

Hi

Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.

Configuration is exactly the same as for fts_squat:

plugin {

plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes

Re: [FTS Xapian] Beta release

[Joan Moreau via dovecot]

I tried to combined it, the "autoreconf" errors are solved 


Now, when I type "make install", the lib is not pushed into dovecot
folder, but somewhere in /usr/local/... 


How to adjust this to have it arriving in the proper folder ?

On 2019-01-13 21:01, Tuomi, Aki wrote:

You copied your Makefile.am there. Stephan made you a working version, can you try that? 

(sorry for dup) 

Aki 

 Original message  
From: Joan Moreau  
Date: 13/01/2019 21:39 (GMT+02:00) 
To: Stephan Bosch  
Cc: Aki Tuomi  
Subject: Re: [FTS Xapian] Beta release 

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian 

However, when I try to act as a visitor, I reach teh follwoing error: 


# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear 
in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define 
'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 
'autoconf' again
src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no 
program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
autoreconf: automake failed with exit status: 1 

On 2019-01-13 20:24, Stephan Bosch wrote: 
Oh, right, a distribution tarball doesn't include some of the necessary files for your repository like autogen.sh and .gitignore. The attached tarball includes all those and is ready for `git init`. The previous tarball was made with `make distcheck` from this one.


Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch: Hi Joan,

Op 13/01/2019 om 19:03 schreef Aki Tuomi: Yes, from compiling point of view it 
is done.

Unfortunately what is not done is all the other work involved, such as fixing 
all the inevitable bugs it has and maintaining it. We do not want, at this 
moment, take up maintaining and developing yet another FTS plugin as we have 
plenty of things to do already.

I invite you to setup your own repository and provide this plugin from there, 
being the maintainer of this plugin. We can add a link to your plugin on our 
FTS page so people can also find it.

There are other plugins like this, e.g. https://github.com/st3fan/dovecot-xaps-plugin 
I turned the code you provided into a separate plugin package. The distribution tarball is attached.


Notable changes:

- Added example copyright headers and COPYING and AUTHORS files. You should 
modify those to your preference.
- Added README and INSTALL files (in markdown using Pandoc). Those need to be 
amended with details.
- Amended the plugin code to display a debug message with the plugin name and 
version upon plugin load.

I advise you to turn this into a git repository and continue from there.

I do not recommend releasing this plugin with the -fpermissive flag and the 
resulting warning as it is now. But I'm assuming this is still a work in 
progress, so that is OK.

Regards,

Stephan.

On 13 January 2019 at 19:52 Joan Moreau  wrote:

The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).

I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly

The only remaining point is to push it in hte git (yes, everything is
already done)

On 2019-01-13 18:45, Aki Tuomi wrote:

On 13 January 2019 at 17:05 Joan Moreau via dovecot  wrote:

Hi

Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.

Configuration is exactly the same as for fts_squat:

plugin {

plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20

This is installed on my production server (>120Gb of mailboxes), and I

Re: [FTS Xapian] Beta release

[Tuomi, Aki]

You copied your Makefile.am there. Stephan made you a working version, can you 
try that?
(sorry for dup)
Aki
 Original message From: Joan Moreau  Date: 
13/01/2019  21:39  (GMT+02:00) To: Stephan Bosch  Cc: Aki 
Tuomi  Subject: Re: [FTS Xapian] Beta release 

I used the skeleton from Aki : https://github.com/grosjo/fts-xapian
However, when I try to act as a visitor, I reach teh follwoing error:
# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear 
in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define 
'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 
'autoconf' again
src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no 
program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
autoreconf: automake failed with exit status: 1


 


On 2019-01-13 20:24, Stephan Bosch wrote:

Oh, right, a distribution tarball doesn't include some of the necessary files 
for your repository like autogen.sh and .gitignore. The attached tarball 
includes all those and is ready for `git init`. The previous tarball was made 
with `make distcheck` from this one.

Regards,

Stephan.

Op 13/01/2019 om 20:14 schreef Stephan Bosch:
Hi Joan,

Op 13/01/2019 om 19:03 schreef Aki Tuomi:
Yes, from compiling point of view it is done.

Unfortunately what is not done is all the other work involved, such as fixing 
all the inevitable bugs it has and maintaining it. We do not want, at this 
moment, take up maintaining and developing yet another FTS plugin as we have 
plenty of things to do already.

I invite you to setup your own repository and provide this plugin from there, 
being the maintainer of this plugin. We can add a link to your plugin on our 
FTS page so people can also find it.

There are other plugins like this, e.g. 
https://github.com/st3fan/dovecot-xaps-plugin

I turned the code you provided into a separate plugin package. The distribution 
tarball is attached.

Notable changes:

- Added example copyright headers and COPYING and AUTHORS files. You should 
modify those to your preference.
- Added README and INSTALL files (in markdown using Pandoc). Those need to be 
amended with details.
- Amended the plugin code to display a debug message with the plugin name and 
version upon plugin load.

I advise you to turn this into a git repository and continue from there.

I do not recommend releasing this plugin with the -fpermissive flag and the 
resulting warning as it is now. But I'm assuming this is still a work in 
progress, so that is OK.

Regards,

Stephan.




On 13 January 2019 at 19:52 Joan Moreau  wrote:


The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).

I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly

The only remaining point is to push it in hte git (yes, everything is
already done)

On 2019-01-13 18:45, Aki Tuomi wrote:



On 13 January 2019 at 17:05 Joan Moreau via dovecot  wrote:

Hi

Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.

Configuration is exactly the same as for fts_squat:

plugin {

plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20

This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.

I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.

Thanks

JM
Hi!

I still 

Re: [FTS Xapian] Beta release

[Joan Moreau via dovecot]

I get the following error: 


# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/USR/SHARE/AUTOMAKE-1.16/AM/DEPEND2.AM: ERROR: AM__FASTDEPCXX DOES NOT
APPEAR IN AM_CONDITIONAL
/USR/SHARE/AUTOMAKE-1.16/AM/DEPEND2.AM: THE USUAL WAY TO DEFINE
'AM__FASTDEPCXX' IS TO ADD 'AC_PROG_CXX'
/USR/SHARE/AUTOMAKE-1.16/AM/DEPEND2.AM: TO 'CONFIGURE.AC' AND RUN
'ACLOCAL' AND 'AUTOCONF' AGAIN
SRC/MAKEFILE.AM: ERROR: C++ SOURCE SEEN BUT 'CXX' IS UNDEFINED
SRC/MAKEFILE.AM: THE USUAL WAY TO DEFINE 'CXX' IS TO ADD 'AC_PROG_CXX'
SRC/MAKEFILE.AM: TO 'CONFIGURE.AC' AND RUN 'AUTOCONF' AGAIN.
SRC/MAKEFILE.AM:11: WARNING: VARIABLE 'NOPLUGIN_LDFLAGS' IS DEFINED BUT
NO PROGRAM OR
SRC/MAKEFILE.AM:11: LIBRARY HAS 'NOPLUGIN' AS CANONICAL NAME (POSSIBLE
TYPO)
AUTORECONF: AUTOMAKE FAILED WITH EXIT STATUS: 1 


On 2019-01-13 20:32, Joan Moreau via dovecot wrote:

Please kindly check https://github.com/grosjo/fts-xapian 

On 2019-01-13 20:11, Aki Tuomi wrote: 
If you had looked at what I sent, you'd seen it's quite different from what you sent.


Anyways, put the contents of skeleton.tar.gz and 


./src/plugins/fts-xapian/fts-xapian-plugin.h
./src/plugins/fts-xapian/fts-backend-xapian.cpp
./src/plugins/fts-xapian/Makefile.am
./src/plugins/fts-xapian/fts-backend-xapian-functions.cpp
./src/plugins/fts-xapian/fts-xapian-plugin.c

into src/ directory (included in skeleton.tar.gz), then put those into git.

You can compile it with 


autoreconf -vi
./configure --with-dovecot=/path/to/dovecot
make

Aki

On 13 January 2019 at 21:03 Joan Moreau  wrote:

THis is already what I send earlier (see : dovecot-xapian-1.0b2.tar.gz
[1 [1]]  ) 


What I would need is the files so one can download (git) it, and type
some command (make ?) to compile it and place it in the right forlder
(/usr/lib/dovecot/ or whatever is configured in the installed dovecot,
which may differ from distribution to distribution) 


On 2019-01-13 19:47, Aki Tuomi wrote:

You need the fts-xapian.c mostly. All the rest comes from automake (unless you 
decide to use cmake or something else).

I have attached you a skeleton plugin, which should work against dovecot master.

If you experience problems with the skeleton, let us know.

Aki

On 13 January 2019 at 20:23 Joan Moreau  wrote:

Ok for having a link on the FTS page.

PLease clarify what files are necessary to package it as a separate
package

On 2019-01-13 19:03, Aki Tuomi wrote:

Yes, from compiling point of view it is done.

Unfortunately what is not done is all the other work involved, such as fixing 
all the inevitable bugs it has and maintaining it. We do not want, at this 
moment, take up maintaining and developing yet another FTS plugin as we have 
plenty of things to do already.

I invite you to setup your own repository and provide this plugin from there, 
being the maintainer of this plugin. We can add a link to your plugin on our 
FTS page so people can also find it.

There are other plugins like this, e.g. 
https://github.com/st3fan/dovecot-xaps-plugin

Aki

On 13 January 2019 at 19:52 Joan Moreau  wrote:

The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).

I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly

The only remaining point is to push it in hte git (yes, everything is
already done)

On 2019-01-13 18:45, Aki Tuomi wrote:

On 13 January 2019 at 17:05 Joan Moreau via dovecot  wrote:

Hi

Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.

Configuration is exactly the same as for fts_squat:

plugin {

plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20

This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.

I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to 

Re: [FTS Xapian] Beta release

[Joan Moreau via dovecot]

Please kindly check https://github.com/grosjo/fts-xapian 


On 2019-01-13 20:11, Aki Tuomi wrote:


If you had looked at what I sent, you'd seen it's quite different from what you 
sent.

Anyways, put the contents of skeleton.tar.gz and 


./src/plugins/fts-xapian/fts-xapian-plugin.h
./src/plugins/fts-xapian/fts-backend-xapian.cpp
./src/plugins/fts-xapian/Makefile.am
./src/plugins/fts-xapian/fts-backend-xapian-functions.cpp
./src/plugins/fts-xapian/fts-xapian-plugin.c

into src/ directory (included in skeleton.tar.gz), then put those into git.

You can compile it with 


autoreconf -vi
./configure --with-dovecot=/path/to/dovecot
make

Aki

On 13 January 2019 at 21:03 Joan Moreau  wrote:

THis is already what I send earlier (see : dovecot-xapian-1.0b2.tar.gz
[1 [1]]  ) 


What I would need is the files so one can download (git) it, and type
some command (make ?) to compile it and place it in the right forlder
(/usr/lib/dovecot/ or whatever is configured in the installed dovecot,
which may differ from distribution to distribution) 


On 2019-01-13 19:47, Aki Tuomi wrote:

You need the fts-xapian.c mostly. All the rest comes from automake (unless you 
decide to use cmake or something else).

I have attached you a skeleton plugin, which should work against dovecot master.

If you experience problems with the skeleton, let us know.

Aki

On 13 January 2019 at 20:23 Joan Moreau  wrote:

Ok for having a link on the FTS page.

PLease clarify what files are necessary to package it as a separate
package

On 2019-01-13 19:03, Aki Tuomi wrote:

Yes, from compiling point of view it is done.

Unfortunately what is not done is all the other work involved, such as fixing 
all the inevitable bugs it has and maintaining it. We do not want, at this 
moment, take up maintaining and developing yet another FTS plugin as we have 
plenty of things to do already.

I invite you to setup your own repository and provide this plugin from there, 
being the maintainer of this plugin. We can add a link to your plugin on our 
FTS page so people can also find it.

There are other plugins like this, e.g. 
https://github.com/st3fan/dovecot-xaps-plugin

Aki

On 13 January 2019 at 19:52 Joan Moreau  wrote:

The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).

I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly

The only remaining point is to push it in hte git (yes, everything is
already done)

On 2019-01-13 18:45, Aki Tuomi wrote:

On 13 January 2019 at 17:05 Joan Moreau via dovecot  wrote:

Hi

Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.

Configuration is exactly the same as for fts_squat:

plugin {

plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20

This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.

I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.

Thanks

JM
Hi!

I still recommend you setup a, say, github repository for your plugin. We are 
not able to currently include your work in dovecot core as it is more work than 
just pushing the code into the repo. Maybe it can be included in the future.

If you want, I can help you in setting up the required configuration scripts 
and such to make it possible to compile it as plugin.

Then anyone can download it and install it for their dovecot, even if dovecot 
itself has been installed from packages, and also makes it possible for package 
maintainers to consider including it in distributions.

Aki  


Links:
--
[1] https://grosjo.net/dovecot-xapian-1.0b2.tar.gz



Links:
--
[1] https://grosjo.net/dovecot-xapian-1.0b2.tar.gz

Re: Freebsd: Fatal error - Support not compiled in for passdb driver 'sql'

[Larry Rosenman]

I've decided not to change the current behavior.

Get Outlook for Android


From: dovecot  on behalf of gsjarvis 

Sent: Sunday, January 13, 2019 1:14:49 PM
To: dovecot@dovecot.org
Subject: Re: Freebsd: Fatal error - Support not compiled in for passdb driver 
'sql'

Hello,

I was wondering if there was any progress on this.

I just upgraded a FreeBSD box and had the same issue again.

I keep (reasonably) good support notes so I found the one that said I had to
install from ports - so all is well. I was just wondering if there was any
news.

I look forward to hearing from you.

Thanks,

-Graham-



--
Sent from: http://dovecot.2317879.n4.nabble.com/

Re: Freebsd: Fatal error - Support not compiled in for passdb driver 'sql'

[gsjarvis]

Hello,

I was wondering if there was any progress on this.

I just upgraded a FreeBSD box and had the same issue again.

I keep (reasonably) good support notes so I found the one that said I had to
install from ports - so all is well. I was just wondering if there was any
news.

I look forward to hearing from you.

Thanks,

-Graham-



--
Sent from: http://dovecot.2317879.n4.nabble.com/

Re: Solr -> Xapian ?

[Joan Moreau via dovecot]

because fts_squat is set to be deleted 

Xapian and similar libraries offers a very easy interface for FTS 

(and basically, I have done it already) 


On 2019-01-07 18:31, Michael Slusarz wrote:

Maybe a dumb question (I admit I haven't followed this thread very closely)... 

But why are you writing a new FTS driver?  If squat allegedly does everything you need it to do, why don't you just take that plugin and fix it up to do what you need?  That seems way easier than trying to create a FTS driver from scratch. 

michael 

On January 7, 2019 at 7:05 AM Joan Moreau via dovecot  wrote: 

Hi 

ANyone to answer specifically ? 

Q1 : get_last_uid -> Is this the last UID indexed (which may be not the greatest value), or the gratest value (which may not be the latest) (the code of existing plugins is unclear about this, Solr looks for the greatest for insance) 

Q2 : WHen Indexing an email, the data is not passed by "build_key". Why so ? What is the link with "build_more" ? 

Q3 : Searching/Lookup : THe fheader in which to llok for (must be a least among "cc, to, from, subject, body") is not appearing in the 'struct' data. WHere to find it ? 

Q4 : Refresh : this is very unclear. How come there would not be the "latest" view on index. What is the real meaning of this function ? 

Q5 : Rescan : is it just a bout remonving all indexes for a specific mailbox ? 


Q6 : lokkup_multi : isn't the function the same for all plugnins (see below) ?

Re: [FTS Xapian] Beta release

[Aki Tuomi]

If you had looked at what I sent, you'd seen it's quite different from what you 
sent.

Anyways, put the contents of skeleton.tar.gz and 

./src/plugins/fts-xapian/fts-xapian-plugin.h
./src/plugins/fts-xapian/fts-backend-xapian.cpp
./src/plugins/fts-xapian/Makefile.am
./src/plugins/fts-xapian/fts-backend-xapian-functions.cpp
./src/plugins/fts-xapian/fts-xapian-plugin.c

into src/ directory (included in skeleton.tar.gz), then put those into git.

You can compile it with 

autoreconf -vi
./configure --with-dovecot=/path/to/dovecot
make

Aki

> On 13 January 2019 at 21:03 Joan Moreau  wrote:
> 
> 
> THis is already what I send earlier (see : dovecot-xapian-1.0b2.tar.gz
> [1]  ) 
> 
> What I would need is the files so one can download (git) it, and type
> some command (make ?) to compile it and place it in the right forlder
> (/usr/lib/dovecot/ or whatever is configured in the installed dovecot,
> which may differ from distribution to distribution) 
> 
> On 2019-01-13 19:47, Aki Tuomi wrote:
> 
> > You need the fts-xapian.c mostly. All the rest comes from automake (unless 
> > you decide to use cmake or something else).
> > 
> > I have attached you a skeleton plugin, which should work against dovecot 
> > master.
> > 
> > If you experience problems with the skeleton, let us know.
> > 
> > Aki
> > 
> > On 13 January 2019 at 20:23 Joan Moreau  wrote:
> > 
> > Ok for having a link on the FTS page.
> > 
> > PLease clarify what files are necessary to package it as a separate
> > package
> > 
> > On 2019-01-13 19:03, Aki Tuomi wrote:
> > 
> > Yes, from compiling point of view it is done.
> > 
> > Unfortunately what is not done is all the other work involved, such as 
> > fixing all the inevitable bugs it has and maintaining it. We do not want, 
> > at this moment, take up maintaining and developing yet another FTS plugin 
> > as we have plenty of things to do already.
> > 
> > I invite you to setup your own repository and provide this plugin from 
> > there, being the maintainer of this plugin. We can add a link to your 
> > plugin on our FTS page so people can also find it.
> > 
> > There are other plugins like this, e.g. 
> > https://github.com/st3fan/dovecot-xaps-plugin
> > 
> > Aki
> > 
> > On 13 January 2019 at 19:52 Joan Moreau  wrote:
> > 
> > The only point here of this fts-xapian is to get rid of solr (because it
> > is just a nightmare to setup) and squat (because it is considere
> > obsolete).
> > 
> > I already sent the changed in configure.ac, makefile.am, etc.. in order
> > to include it in the dovecot, and it compiles properly
> > 
> > The only remaining point is to push it in hte git (yes, everything is
> > already done)
> > 
> > On 2019-01-13 18:45, Aki Tuomi wrote:
> > 
> > On 13 January 2019 at 17:05 Joan Moreau via dovecot  
> > wrote:
> > 
> > Hi
> > 
> > Please find attached the beta release of FTS Xapian, with the objective
> > to replace fts_squat that is being deprecated.
> > 
> > Configuration is exactly the same as for fts_squat:
> > 
> > plugin {
> > 
> > plugin = fts fts_xapian (...)
> > fts = xapian
> > fts_autoindex = yes
> > fts_enforced = yes
> > fts_xapian = partial=2 full=20
> > 
> > This is installed on my production server (>120Gb of mailboxes), and I
> > will observe it during the coming days.
> > 
> > I will definitely appreciate that this is added in the core git of
> > docevot, in order to have a versionning of it, to remove squat and let
> > basic users able to avoid Solr alternative as much as possible.
> > 
> > Thanks
> > 
> > JM
> > Hi!
> > 
> > I still recommend you setup a, say, github repository for your plugin. We 
> > are not able to currently include your work in dovecot core as it is more 
> > work than just pushing the code into the repo. Maybe it can be included in 
> > the future.
> > 
> > If you want, I can help you in setting up the required configuration 
> > scripts and such to make it possible to compile it as plugin.
> > 
> > Then anyone can download it and install it for their dovecot, even if 
> > dovecot itself has been installed from packages, and also makes it possible 
> > for package maintainers to consider including it in distributions.
> > 
> > Aki
>  
> 
> Links:
> --
> [1] https://grosjo.net/dovecot-xapian-1.0b2.tar.gz

Re: [FTS Xapian] Beta release

[Joan Moreau via dovecot]

THis is already what I send earlier (see : dovecot-xapian-1.0b2.tar.gz
[1]  ) 


What I would need is the files so one can download (git) it, and type
some command (make ?) to compile it and place it in the right forlder
(/usr/lib/dovecot/ or whatever is configured in the installed dovecot,
which may differ from distribution to distribution) 


On 2019-01-13 19:47, Aki Tuomi wrote:


You need the fts-xapian.c mostly. All the rest comes from automake (unless you 
decide to use cmake or something else).

I have attached you a skeleton plugin, which should work against dovecot master.

If you experience problems with the skeleton, let us know.

Aki

On 13 January 2019 at 20:23 Joan Moreau  wrote:

Ok for having a link on the FTS page.

PLease clarify what files are necessary to package it as a separate
package

On 2019-01-13 19:03, Aki Tuomi wrote:

Yes, from compiling point of view it is done.

Unfortunately what is not done is all the other work involved, such as fixing 
all the inevitable bugs it has and maintaining it. We do not want, at this 
moment, take up maintaining and developing yet another FTS plugin as we have 
plenty of things to do already.

I invite you to setup your own repository and provide this plugin from there, 
being the maintainer of this plugin. We can add a link to your plugin on our 
FTS page so people can also find it.

There are other plugins like this, e.g. 
https://github.com/st3fan/dovecot-xaps-plugin

Aki

On 13 January 2019 at 19:52 Joan Moreau  wrote:

The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).

I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly

The only remaining point is to push it in hte git (yes, everything is
already done)

On 2019-01-13 18:45, Aki Tuomi wrote:

On 13 January 2019 at 17:05 Joan Moreau via dovecot  wrote:

Hi

Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.

Configuration is exactly the same as for fts_squat:

plugin {

plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20

This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.

I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.

Thanks

JM
Hi!

I still recommend you setup a, say, github repository for your plugin. We are 
not able to currently include your work in dovecot core as it is more work than 
just pushing the code into the repo. Maybe it can be included in the future.

If you want, I can help you in setting up the required configuration scripts 
and such to make it possible to compile it as plugin.

Then anyone can download it and install it for their dovecot, even if dovecot 
itself has been installed from packages, and also makes it possible for package 
maintainers to consider including it in distributions.

Aki



Links:
--
[1] https://grosjo.net/dovecot-xapian-1.0b2.tar.gz

Re: [FTS Xapian] Beta release

[Aki Tuomi]

You need the fts-xapian.c mostly. All the rest comes from automake (unless you 
decide to use cmake or something else).

I have attached you a skeleton plugin, which should work against dovecot master.

If you experience problems with the skeleton, let us know.

Aki


> On 13 January 2019 at 20:23 Joan Moreau  wrote:
>
>
> Ok for having a link on the FTS page.
>
> PLease clarify what files are necessary to package it as a separate
> package
>
> On 2019-01-13 19:03, Aki Tuomi wrote:
>
> > Yes, from compiling point of view it is done.
> >
> > Unfortunately what is not done is all the other work involved, such as 
> > fixing all the inevitable bugs it has and maintaining it. We do not want, 
> > at this moment, take up maintaining and developing yet another FTS plugin 
> > as we have plenty of things to do already.
> >
> > I invite you to setup your own repository and provide this plugin from 
> > there, being the maintainer of this plugin. We can add a link to your 
> > plugin on our FTS page so people can also find it.
> >
> > There are other plugins like this, e.g. 
> > https://github.com/st3fan/dovecot-xaps-plugin
> >
> > Aki
> >
> > On 13 January 2019 at 19:52 Joan Moreau  wrote:
> >
> > The only point here of this fts-xapian is to get rid of solr (because it
> > is just a nightmare to setup) and squat (because it is considere
> > obsolete).
> >
> > I already sent the changed in configure.ac, makefile.am, etc.. in order
> > to include it in the dovecot, and it compiles properly
> >
> > The only remaining point is to push it in hte git (yes, everything is
> > already done)
> >
> > On 2019-01-13 18:45, Aki Tuomi wrote:
> >
> > On 13 January 2019 at 17:05 Joan Moreau via dovecot  
> > wrote:
> >
> > Hi
> >
> > Please find attached the beta release of FTS Xapian, with the objective
> > to replace fts_squat that is being deprecated.
> >
> > Configuration is exactly the same as for fts_squat:
> >
> > plugin {
> >
> > plugin = fts fts_xapian (...)
> > fts = xapian
> > fts_autoindex = yes
> > fts_enforced = yes
> > fts_xapian = partial=2 full=20
> >
> > This is installed on my production server (>120Gb of mailboxes), and I
> > will observe it during the coming days.
> >
> > I will definitely appreciate that this is added in the core git of
> > docevot, in order to have a versionning of it, to remove squat and let
> > basic users able to avoid Solr alternative as much as possible.
> >
> > Thanks
> >
> > JM
> > Hi!
> >
> > I still recommend you setup a, say, github repository for your plugin. We 
> > are not able to currently include your work in dovecot core as it is more 
> > work than just pushing the code into the repo. Maybe it can be included in 
> > the future.
> >
> > If you want, I can help you in setting up the required configuration 
> > scripts and such to make it possible to compile it as plugin.
> >
> > Then anyone can download it and install it for their dovecot, even if 
> > dovecot itself has been installed from packages, and also makes it possible 
> > for package maintainers to consider including it in distributions.
> >
> > Aki

skeleton.tar.gz
Description: application/gzip


signature.asc
Description: PGP signature

Re: [FTS Xapian] Beta release

[Joan Moreau via dovecot]

Ok for having a link on the FTS page. 


PLease clarify what files are necessary to package it as a separate
package

On 2019-01-13 19:03, Aki Tuomi wrote:


Yes, from compiling point of view it is done.

Unfortunately what is not done is all the other work involved, such as fixing 
all the inevitable bugs it has and maintaining it. We do not want, at this 
moment, take up maintaining and developing yet another FTS plugin as we have 
plenty of things to do already.

I invite you to setup your own repository and provide this plugin from there, 
being the maintainer of this plugin. We can add a link to your plugin on our 
FTS page so people can also find it.

There are other plugins like this, e.g. 
https://github.com/st3fan/dovecot-xaps-plugin

Aki

On 13 January 2019 at 19:52 Joan Moreau  wrote:

The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete). 


I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly 


The only remaining point is to push it in hte git (yes, everything is
already done) 


On 2019-01-13 18:45, Aki Tuomi wrote:

On 13 January 2019 at 17:05 Joan Moreau via dovecot  wrote:

Hi 


Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated. 

Configuration is exactly the same as for fts_squat: 

plugin { 


plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20 


This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days. 


I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible. 

Thanks 

JM 
Hi!


I still recommend you setup a, say, github repository for your plugin. We are 
not able to currently include your work in dovecot core as it is more work than 
just pushing the code into the repo. Maybe it can be included in the future.

If you want, I can help you in setting up the required configuration scripts 
and such to make it possible to compile it as plugin.

Then anyone can download it and install it for their dovecot, even if dovecot 
itself has been installed from packages, and also makes it possible for package 
maintainers to consider including it in distributions.

Aki

Re: [FTS Xapian] Beta release

[Aki Tuomi]

Yes, from compiling point of view it is done.

Unfortunately what is not done is all the other work involved, such as fixing 
all the inevitable bugs it has and maintaining it. We do not want, at this 
moment, take up maintaining and developing yet another FTS plugin as we have 
plenty of things to do already.

I invite you to setup your own repository and provide this plugin from there, 
being the maintainer of this plugin. We can add a link to your plugin on our 
FTS page so people can also find it.

There are other plugins like this, e.g. 
https://github.com/st3fan/dovecot-xaps-plugin

Aki

> On 13 January 2019 at 19:52 Joan Moreau  wrote:
> 
> 
> The only point here of this fts-xapian is to get rid of solr (because it
> is just a nightmare to setup) and squat (because it is considere
> obsolete). 
> 
> I already sent the changed in configure.ac, makefile.am, etc.. in order
> to include it in the dovecot, and it compiles properly 
> 
> The only remaining point is to push it in hte git (yes, everything is
> already done) 
> 
> On 2019-01-13 18:45, Aki Tuomi wrote:
> 
> >> On 13 January 2019 at 17:05 Joan Moreau via dovecot  
> >> wrote:
> >> 
> >> Hi 
> >> 
> >> Please find attached the beta release of FTS Xapian, with the objective
> >> to replace fts_squat that is being deprecated. 
> >> 
> >> Configuration is exactly the same as for fts_squat: 
> >> 
> >> plugin { 
> >> 
> >> plugin = fts fts_xapian (...)
> >> fts = xapian
> >> fts_autoindex = yes
> >> fts_enforced = yes
> >> fts_xapian = partial=2 full=20 
> >> 
> >> This is installed on my production server (>120Gb of mailboxes), and I
> >> will observe it during the coming days. 
> >> 
> >> I will definitely appreciate that this is added in the core git of
> >> docevot, in order to have a versionning of it, to remove squat and let
> >> basic users able to avoid Solr alternative as much as possible. 
> >> 
> >> Thanks 
> >> 
> >> JM
> > 
> > Hi!
> > 
> > I still recommend you setup a, say, github repository for your plugin. We 
> > are not able to currently include your work in dovecot core as it is more 
> > work than just pushing the code into the repo. Maybe it can be included in 
> > the future.
> > 
> > If you want, I can help you in setting up the required configuration 
> > scripts and such to make it possible to compile it as plugin.
> > 
> > Then anyone can download it and install it for their dovecot, even if 
> > dovecot itself has been installed from packages, and also makes it possible 
> > for package maintainers to consider including it in distributions.
> > 
> > Aki

Re: [FTS Xapian] Beta release

[Joan Moreau via dovecot]

The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete). 


I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly 


The only remaining point is to push it in hte git (yes, everything is
already done) 


On 2019-01-13 18:45, Aki Tuomi wrote:


On 13 January 2019 at 17:05 Joan Moreau via dovecot  wrote:

Hi 


Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated. 

Configuration is exactly the same as for fts_squat: 

plugin { 


plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20 


This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days. 


I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible. 

Thanks 


JM


Hi!

I still recommend you setup a, say, github repository for your plugin. We are 
not able to currently include your work in dovecot core as it is more work than 
just pushing the code into the repo. Maybe it can be included in the future.

If you want, I can help you in setting up the required configuration scripts 
and such to make it possible to compile it as plugin.

Then anyone can download it and install it for their dovecot, even if dovecot 
itself has been installed from packages, and also makes it possible for package 
maintainers to consider including it in distributions.

Aki

Re: [FTS Xapian] Beta release

[Aki Tuomi]

> On 13 January 2019 at 17:05 Joan Moreau via dovecot  
> wrote:
> 
> 
> Hi 
> 
> Please find attached the beta release of FTS Xapian, with the objective
> to replace fts_squat that is being deprecated. 
> 
> Configuration is exactly the same as for fts_squat: 
> 
> plugin { 
> 
> plugin = fts fts_xapian (...)
> fts = xapian
> fts_autoindex = yes
> fts_enforced = yes
> fts_xapian = partial=2 full=20 
> 
> This is installed on my production server (>120Gb of mailboxes), and I
> will observe it during the coming days. 
> 
> I will definitely appreciate that this is added in the core git of
> docevot, in order to have a versionning of it, to remove squat and let
> basic users able to avoid Solr alternative as much as possible. 
> 
> Thanks 
> 
> JM

Hi!

I still recommend you setup a, say, github repository for your plugin. We are 
not able to currently include your work in dovecot core as it is more work than 
just pushing the code into the repo. Maybe it can be included in the future.

If you want, I can help you in setting up the required configuration scripts 
and such to make it possible to compile it as plugin.

Then anyone can download it and install it for their dovecot, even if dovecot 
itself has been installed from packages, and also makes it possible for package 
maintainers to consider including it in distributions.

Aki

Indexing paralelism

[Joan Moreau via dovecot]

Hi 

Observing the processes of FTS, I observe the following: 


1 - For one user, indexer-wroker does not start several threads for each
request. On teh contrary, it waits for the first request to finish
before starting the second. How to make sure all requests (or a limited
number of it, for instance linked to the CPU number on the machine) are
started asap ? 


2 - If a IMAP query is received, the dovecot checks the last UID from
FTS and launch a request of indexing to finish the index *before*
running the search query . THis creates timeouts (and can take a while
if many request are pending - see point 1) How to prevent that (i.e. the
serach request is launched (read only) no matter what ? THe completeion
of missing UIDs is launched in a separate thread ? 


Thank you

[FTS Xapian] Beta release

[Joan Moreau via dovecot]

Hi 


Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated. 

Configuration is exactly the same as for fts_squat: 

plugin { 


plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20 


This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days. 


I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible. 

Thanks 


JM

dovecot-xapian.tar.gz
Description: GNU Zip compressed data

Re: Re: Feature request SCRAM-SHA-256

[Tributh via dovecot]

Hi,
sorry for my late reply. Was too busy during the week.
Thank you for your patches. I hope I will be able with them to get now
some client support for SCRAM-SHA-256. Will report how I succeed in the
future.

Regards,

Torsten


On 07.01.19 20:31, Stephan Bosch wrote:
> 
> Op 16/12/2018 om 10:06 schreef Tributh via dovecot:
>>
>> Am 16.12.18 um 09:42 schrieb Aki Tuomi:
 On 16 December 2018 at 10:27 Tributh via dovecot
  wrote:


 Hi,
 is that here the right place to make feature requests?

 dovecot supports as authentication mechanism
 SCRAM-SHA-1 from RFC 5802
 which was updated to
 SCRAM-SHA-256 in RFC 7677

 Can SCRAM-SHA-256 be added to the authentication mechanisms?

 I would not like to request, that SCRAM-SHA-1 will be exchanged by
 SCRAM-SHA-256, since several applications only support SCRAM-SHA-1

 Regards

 Torsten
>>> Hi!
>>>
>>> Adding this is possible, it can even be done as a separate plugin.
>>> But I have to ask, why? Do you actually have clients that support this?
>>>
>>> Aki
>>>
>> Hi Aki,
>> let me first answer the second question.
>> Sadly I have no client which supports it, yet.
>> Here we have a chicken or the egg causality dilemma.
>> There was some communication with mail-client developers which stated
>> that they would start developing it, when they have a publicly usable
>> server to test against.
>> Now I hope that the most common IMAP server could be the one, which
>> gives this possibility.
>> Sadly, most communication is not publicly available.
>>
>> In the past CRAM-MD5 was very popular. When the insecurity came out,
>> everything just shifted to TLS, but that prevented not from sending a
>> plain password now. If a malicious actor is able to change DNS/TLS
>> endpoints, he will receive the plain passwords immediately.
>> I am not the expert in explaining how such an actor could do this. I
>> just wanted to have possibilities for everybody to prevent this possible
>> exposure of a plain password, which could than easily used abusively.
>>
>> I just hope for better security in the future.
> 
> 
> I looked a this a bit and since it is basically only a matter of
> replacing the hash algorithm, I created a quick implementation (after
> some refactoring):
> https://github.com/stephanbosch/dovecot-core/commits/auth-scram-sha-256
> 
> However, since there is no client that actually supports this, I cannot
> test this myself. I've briefly tested that the old SHA-1 still works
> (using mpop) and that the server properly announces the new mechanism
> when enabled, but that is it. It is based on the master branch.
> Configuration is identical to SCRAM-SHA-1, apart from the mechanism (and
> password scheme) name of course.
> 
> Don't expect this to be released or even merged to the master branch any
> time soon: this is likely currently very low on our priority list. But,
> at least you can run your own server with SCRAM-SHA-256 support (and so
> can client developers).  Maybe if this gets endorsed and supported by
> clients and gets some testing in the field, we can speed it along a bit,
> but that is not something I can promise.
> 
> So, I hatched a chick for you. I hope you can make it lay a few eggs in
> the future...
> 
> Regards,
> 
> Stephan.
> 
> 

RE: degfault imaptest

[Marc Roos]

 
I already got an rpm from the fc repo. But this one is still in epel, 
would be nice if packages are at least working/updated in these repos, 
lots of sysadmin would like to have single/verified source. Don't know 
who is responsible for addressing this, just wanted to make sure you 
know.

-Original Message-
From: Stephan Bosch [mailto:step...@rename-it.nl] 
Sent: zaterdag 12 januari 2019 20:47
To: Marc Roos; dovecot
Subject: Re: degfault imaptest


Op 11/01/2019 om 15:13 schreef Marc Roos:
> imaptest host=mail04.local port=143 user=xxx pass=xxx mbox=test.mbox
>
> [Fri Jan 11 15:09:14 2019] imaptest[7634]: segfault at 1354ff0 ip 
> 01354ff0 sp 7ffed5d8f558 error 15 [Fri Jan 11 15:09:22 
> 2019] imaptest[7635]: segfault at 2267ff0 ip 02267ff0 sp 
> 7ffee5890308 error 15 [Fri Jan 11 15:10:47 2019] imaptest[7648]: 
> segfault at 10e6ff0 ip 010e6ff0 sp 7ffedbb19f98 error 15
>
> CentOS Linux release 7.6.1810 (Core)
> imaptest-20151005-1.el7.x86_64
> Linux test2 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux

That imaptest is from several years ago. I'd say this problem is likely 
fixed. You can check with a newer version (compile one). And for bug 
reports like this we need to have a gdb backtrace (e.g. run it in gdb 
--args  and issue 'bt full' once it fails).

Regards,

Stephan.

Re: [solved] managesieve configuration

[Stephan Bosch]

Op 13/01/2019 om 00:22 schreef Dominik Menke:
For reference: if you put ssl=yes there, the TLS layer is established 
immediately. However, the standard ManageSieve protocol does not 
support that (not currently anyway): only the establishment of the 
TLS layer using the STARTTLS command is part of the standard. That is 
why your clients fail to connect: they're speaking plaintext while 
the server is speaking TLS. Still, Dovecot supports configuring it 
that way, which is what you did.


Regards,

Stephan.





I'm just surprised that ssl=yes leads to STARTTLS being disabled, as 
per the wiki [1]:


With ssl=yes, the TLS layer is enabled immediately on the connection. 
So, there is no need to perform STARTTLS. But worse, a client that 
doesn't work this way will try to send "STARTTLS" in plaintext to a 
service talking TLS already. This will obviously not work.


Regards,

Stephan.




> ssl=yes and disable_plaintext_auth=no: SSL/TLS is offered to the
> client, but the client isn't required to use it. [...]
>
> ssl=yes and disable_plaintext_auth=yes: SSL/TLS is offered to the
> client, but the client isn't required to use it. [...]
>
> ssl=required: SSL/TLS is always required [...]. Any attempt to
> authenticate before SSL/TLS is enabled will cause an authentication
> failure.


Maybe this bit needs to be clarified a bit? I think I've read that 
page a few times and it still didn't occur to me that this could be a 
problem.


Best regards,
--Dominik


[1]: https://wiki.dovecot.org/SSL/DovecotConfiguration

Re: Error: User b...@aaa.bbb doesn't have home dir set, disabling duplicate database

[Stephan Bosch]

Op 13/01/2019 om 11:56 schreef subscription1:
Yes, that is what I did and it all works apart from the error message 
on the server about the missing home dir


The doc seems to indicate that a home dir is needed.



If you enable mail_debug=yes and perhaps auth_debug=yes, you should see 
what the imap service is using as the home directory and why.


Regards,

Stephan.


Thanks,

Leo

On 12/1/19 8:42 pm, Stephan Bosch wrote:


Op 11/01/2019 om 15:27 schreef subscription1:
I made a mistake when I moved dovecot to a new server and only 
specified mail_location instead of mail_home


All I have in 10-mail.conf is

-
mail_location = maildir:/home/vmail/mailboxes/%d/%n


All emails for the few accounts I have are in these mailboxes and I 
can get/see them via my mail client


I do, however, get the following error

imap(b...@aaa.bbb): Error: User b...@aaa.bbb doesn't have home dir 
set, disabling duplicate database

---

After looking at https://wiki2.dovecot.org/VirtualUsers/Home I tried 
the following


mail_home = /home/vmail/mailboxes/%d/%n

mail_location = maildir:~/mail
--

Problem with this is that I now can't see any of my emails in the 
client.


Appreciate any help on how to fix this.


Well, you changed the effective maildir directory to 
/home/vmail/mailboxes/%d/%n/mail. So, if you want to use it like 
that, you'll have to move the maildir for each user to that new path 
template.


Regards,

Stephan.

Re: Error: User b...@aaa.bbb doesn't have home dir set, disabling duplicate database

[subscription1]

Yes, that is what I did and it all works apart from the error message on 
the server about the missing home dir


The doc seems to indicate that a home dir is needed.

Thanks,

Leo

On 12/1/19 8:42 pm, Stephan Bosch wrote:


Op 11/01/2019 om 15:27 schreef subscription1:
I made a mistake when I moved dovecot to a new server and only 
specified mail_location instead of mail_home


All I have in 10-mail.conf is

-
mail_location = maildir:/home/vmail/mailboxes/%d/%n


All emails for the few accounts I have are in these mailboxes and I 
can get/see them via my mail client


I do, however, get the following error

imap(b...@aaa.bbb): Error: User b...@aaa.bbb doesn't have home dir set, 
disabling duplicate database

---

After looking at https://wiki2.dovecot.org/VirtualUsers/Home I tried 
the following


mail_home = /home/vmail/mailboxes/%d/%n

mail_location = maildir:~/mail
--

Problem with this is that I now can't see any of my emails in the 
client.


Appreciate any help on how to fix this.


Well, you changed the effective maildir directory to 
/home/vmail/mailboxes/%d/%n/mail. So, if you want to use it like that, 
you'll have to move the maildir for each user to that new path template.


Regards,

Stephan.

Re: Solr -> Xapian ?

[Joan Moreau via dovecot]

I found the solution o this using
SEQ_RANGE_ARRAY_ADD(>DEFINITE_UIDS, UID); 


Now, I can see in the logs that several times, the dovecot calls the
fts_backend_xapian_update_set_mailbox with box == NULL. WHy so ? 

THank you 


On 2019-01-12 21:40, Joan Moreau via dovecot wrote:

I somehow fixed the folder issue. (seems some unix rights after too many tests) 

Getting back on the "fts_results" structure: 

I am trying: 


I_ARRAY_INIT(&(RESULT->DEFINITE_UIDS),R->SIZE);
I_ARRAY_INIT(&(RESULT->MAYBE_UIDS),0); 


uint32_t uid;
for(i=0;isize;i++)
{
try
{
uid=atol(backend->dbr->get_document(r->data[i]).get_value(1).c_str());
i_warning("Rresult UID=%d",uid);
ARRAY_IDX_SET(&(RESULT->DEFINITE_UIDS),I,);
}
catch(Xapian::Error e)
{
i_warning(e.get_msg().c_str());
}
} 

I can see in hte log that UID are properly found on Xapian database, but no results are transmitted to dovecot and to the imap client (roundcube in my case) 

Help please :) 

On 2019-01-12 18:15, Joan Moreau wrote: 

additionally, my logic is that the backend stores one databalse per mailox in /xapian-indexes (in the "root" dir of the user), the name od the database is the GUID of the mailbox 

For INBOX, that works perfectly, and database is properly createdm and backed starts indexing all emails 

For other folder, somehow, the process can not access that (root) folder. 

Am I missing something ? 

On 2019-01-12 17:37, Joan Moreau wrote: 

THank you 

Now, for the results 

I see the member of fts_result is : 


ARRAY_TYPE(seq_range) definite_uids;

I have the UID as a aray of uint32_t * 

How to put my UIDs into this "definite_uids" ? Obviously this is not a simple array/pointer. How to say someting similar to result->definite_uids[1]=my_uid ? 

On 2019-01-12 10:25, Timo Sirainen wrote: 
On 11 Jan 2019, at 21.23, Joan Moreau via dovecot  wrote: 
The below patch resolves the compilation error


$ diff -p compat.h compat.h.joan 
*** compat.h 2019-01-11 20:21:00.726625427 +0100

--- compat.h.joan 2019-01-11 20:14:41.729109919 +0100
*** struct iovec;
*** 202,207 
--- 202,211 
ssize_t i_my_writev(int fd, const struct iovec *iov, int iov_len);
#endif

+ #ifdef __cplusplus
+ extern "C" {
+ #endif

You should put this extern "C" into the C++ file you're creating. See for 
example how fts-lucene/lucene-wrapper.cc does this.

1 - WHat does represent "subargs" in mail_search_args 
It's set only for SEARCH_OR and SEARCH_SUB. So for example:


SEARCH TEXT foo TEXT bar TEXT baz

results in:

type=SEARCH_SUB
value.subargs = (
{ type=SEARCH, value.str="foo" },
{ type=SEARCH, value.str="bar" },
{ type=SEARCH, value.str="baz" },
)

Or similarly if there's SEARCH OR foo OR TEXT bar TEXT baz or some other 
combination of OR/ANDs.
2 - for rescan : who is responsible for passing again the new email ? Is
the Dovecot core sending again all the emails to index ? or the fts
shall somehow access the mailbox and read all emails ? Wouldn't just be
saying "delete all index and get_last_uid is now 0" the easy way ? or
the fts must process all emails (and block the current thread as a
mailbx maybe quite large) 
The next indexing run is responsible for it. If you return get_last_uid=0, then indexer starts feeding you all mails. So fts backend doesn't have to know about it.


3 - for get_last_uid : this uncertainity is very unclear. "If there is a
gap, then indexer first indexes all the missing" -> this mean at a
certain point, indexer maybe rebuilding a previous email, so *last* uid
is something different than max. And how indexer does know whther there
is a gap wihtout callong the fts backend (whch it does not as there are
no function for that) ? 
I mean if get_last_uid() returns for example 100, it means that UIDs 1..100 have been indexed by the FTS backend. It's possible that at this point there are already mails with UIDs 101..200 in the folder. So when UID=201 is delivered, indexer notices that FTS backend has only UIDs 1..100 indexed so far, and starts feeding it UIDs 101..201 in that order.


You can implement get_last_uid() simply by keeping track of it in 
dovecot.index* files, similar to how Lucene and Solr already do it with 
fts_index_get_header() / fts_index_set_header(). They also have a fallback that 
if the index doesn't have the last_uid value, they do a slower search from the 
Lucene/Solr index to find the last UID.