Exclude folders from fts xapian on a per-user basis
Is there any way for a user to specify folders they want excluded from fts xapian indexing? Or can this only be done globally in /etc? ___ dovecot mailing list -- dovecot@dovecot.org To unsubscribe send an email to dovecot-le...@dovecot.org
Re: 2.3.17 - breaks with fts-xapian.
> On 31/10/2021 00:01 Juan Felipe Arjona wrote: > > > Hello: > > > I just upgraded my Debian server with dovecot 2.3.17, and it breaks when > using the plugin fts-xapian. > > Apparently is a bug in Dovecot, please see: > > > https://github.com/grosjo/fts-xapian/issues/99 > > > Thanks, > > > Juan Hi! This issue is due to a bug in fts-xapian. The plugin does not call fts_mail_user_init, causing the crash. Aki
2.3.17 - breaks with fts-xapian.
Hello: I just upgraded my Debian server with dovecot 2.3.17, and it breaks when using the plugin fts-xapian. Apparently is a bug in Dovecot, please see: https://github.com/grosjo/fts-xapian/issues/99 Thanks, Juan
Re: Problem with the dovecot-fts-xapian package.
@Bob : The package has been recompiled against the new version of Dovecot. Hope it works now @Aki : It would be nice to have all plugins included in the source code for major releases (with a simple rule that non-maintained packages are removed), including Pigeonhole, FTS plugins, and many other existing plugins from all over the world On 2021-09-12 13:54, Aki Tuomi wrote: On 12/09/2021 15:12 Bob Marcan wrote: On Sun, 12 Sep 2021 11:36:46 +0100 Joan Moreau wrote: This is where I am for now : https://koji.fedoraproject.org/koji/packageinfo?packageID=34417 Probably, I should wait for Fedora batch programs to push that into main rep On 2021-09-12 11:18, Joan Moreau wrote: Hi Bob, I am trying to achieve that. But do you know the process of pushing an update as a maintainer, in > fedore repositories ? Thank you On 2021-09-12 11:02, Bob Marcan wrote: On Sun, 12 Sep 2021 09:45:35 +0100 Joan Moreau wrote: Thank you for notice. What is the process to rebuild the package with recent dovecot, as > (instead of existing 1.4.12-1) ? There are no (yet) 1.4.12-2 in updates-testing or > updates-testing-modular repository. Should i'll wait for update? BR, Bob Got the new version and there is no more API mismatch. It's not so important for me, since i'm retired and i'm running this on my home computer. But i think it needs more support from te dovecot group. There a lot file protection issues nad lack of documentation on dovecot side. BR, Bob Hi Bob, Dovecot does not maintain either the packages for fedora. These are maintained by Fedora Project. Also we do not maintain or document the dovecot-fts-xapian plugin, since it's 3rd party plugin. It's maintained by Joan Moreau. Kind regards, Aki
Re: Problem with the dovecot-fts-xapian package.
> On 12/09/2021 15:12 Bob Marcan wrote: > > > On Sun, 12 Sep 2021 11:36:46 +0100 > Joan Moreau wrote: > > > This is where I am for now : > > > > https://koji.fedoraproject.org/koji/packageinfo?packageID=34417 > > > > Probably, I should wait for Fedora batch programs to push that into main rep > > > > On 2021-09-12 11:18, Joan Moreau wrote: > > > > > Hi Bob, > > > > > > I am trying to achieve that. > > > > > > But do you know the process of pushing an update as a maintainer, in > > > > fedore repositories ? > > > > > > Thank you > > > > > > On 2021-09-12 11:02, Bob Marcan wrote: > > > On Sun, 12 Sep 2021 09:45:35 +0100 > > > Joan Moreau wrote: > > > > > > Thank you for notice. > > > > > > What is the process to rebuild the package with recent dovecot, as > > > > (instead of existing 1.4.12-1) ? > > > There are no (yet) 1.4.12-2 in updates-testing or > > > > updates-testing-modular repository. > > > Should i'll wait for update? > > > BR, Bob > > Got the new version and there is no more API mismatch. > It's not so important for me, since i'm retired and i'm running this on my > home computer. > But i think it needs more support from te dovecot group. > There a lot file protection issues nad lack of documentation on dovecot side. > > BR, Bob Hi Bob, Dovecot does not maintain either the packages for fedora. These are maintained by Fedora Project. Also we do not maintain or document the dovecot-fts-xapian plugin, since it's 3rd party plugin. It's maintained by Joan Moreau. Kind regards, Aki
Re: Problem with the dovecot-fts-xapian package.
On Sun, 12 Sep 2021 11:36:46 +0100 Joan Moreau wrote: > This is where I am for now : > > https://koji.fedoraproject.org/koji/packageinfo?packageID=34417 > > Probably, I should wait for Fedora batch programs to push that into main rep > > On 2021-09-12 11:18, Joan Moreau wrote: > > > Hi Bob, > > > > I am trying to achieve that. > > > > But do you know the process of pushing an update as a maintainer, in > > > fedore repositories ? > > > > Thank you > > > > On 2021-09-12 11:02, Bob Marcan wrote: > > On Sun, 12 Sep 2021 09:45:35 +0100 > > Joan Moreau wrote: > > > > Thank you for notice. > > > > What is the process to rebuild the package with recent dovecot, as > > > (instead of existing 1.4.12-1) ? > > There are no (yet) 1.4.12-2 in updates-testing or > > > updates-testing-modular repository. > > Should i'll wait for update? > > BR, Bob Got the new version and there is no more API mismatch. It's not so important for me, since i'm retired and i'm running this on my home computer. But i think it needs more support from te dovecot group. There a lot file protection issues nad lack of documentation on dovecot side. BR, Bob
Re: Problem with the dovecot-fts-xapian package.
Thank you for notice. What is the process to rebuild the package with recent dovecot, as 1.4.12-2 (instead of existing 1.4.12-1) ? On 2021-09-12 07:21, Bob Marcan wrote: Problem with the dovecot-fts-xapian package. Fedora 34 with latest updates. dovecot-2.3.16-1.fc34.x86_64 dovecot-fts-xapian-1.4.12-1.fc34.x86_64 [root@smicro conf.d]# systemctl restart dovecot [root@smicro conf.d]# doveadm index -A \* Fatal: Couldn't load required plugin /usr/lib64/dovecot/lib21_fts_xapian_plugin.so: Module is for different ABI version 2.3.ABIv15(2.3.15) (we have 2.3.ABIv16(2.3.16)) BR, Bob
Problem with the dovecot-fts-xapian package.
Problem with the dovecot-fts-xapian package. Fedora 34 with latest updates. dovecot-2.3.16-1.fc34.x86_64 dovecot-fts-xapian-1.4.12-1.fc34.x86_64 [root@smicro conf.d]# systemctl restart dovecot [root@smicro conf.d]# doveadm index -A \* Fatal: Couldn't load required plugin /usr/lib64/dovecot/lib21_fts_xapian_plugin.so: Module is for different ABI version 2.3.ABIv15(2.3.15) (we have 2.3.ABIv16(2.3.16)) BR, Bob
Re: Duplicate plugins - FTS Xapian
Just for clarity, Open-Xchange has not written any xapian plugin whatsoever. Yes but the doc says that Open Xchaneg "supports" one over the other. Honestly, I am doing this over my free time, begin very reactive to user requests, and have this confirmed by Debian, Archlinux and now Fedora in their core packages This is not very encouraging despite all the efforts achieved.
Re: Duplicate plugins - FTS Xapian
> On Sep 1, 2021, at 2:21 PM, Joan Moreau wrote: > > > > Just for clarity, Open-Xchange has not written any xapian plugin > > whatsoever. > > > Yes but the doc says that Open Xchaneg "supports" one over the other. > > Honestly, I am doing this over my free time, begin very reactive to user > requests, and have this confirmed by Debian, Archlinux and now Fedora in > their core packages > > This is not very encouraging despite all the efforts achieved. For my own part, thank you VERY much for developing and maintaining this plugin. It’s wonderful to have a viable alternative to Solr, which oftentimes exceeds available resources. And thank-you also to Michael for making an alternative implementation (of which, until this thread, I was actually unaware) available. -FG
Re: Duplicate plugins - FTS Xapian
On 09-01-2021 2:21 pm, Joan Moreau wrote: Just for clarity, Open-Xchange has not written any xapian plugin whatsoever. Yes but the doc says that Open Xchaneg "supports" one over the other. Honestly, I am doing this over my free time, begin very reactive to user requests, and have this confirmed by Debian, Archlinux and now Fedora in their core packages This is not very encouraging despite all the efforts achieved. What is your goal for making software? To put out something good in the world that others can use if they choose to? Or a pat on the back? Can't users choose which ever plugin they prefer to use? Isn't that mission accomplished? Sounds like you are upset no one gave you a trophy. You will be satisfied with a thank you for creating open source software just like the thousands of others before you? Thank you.
Re: Duplicate plugins - FTS Xapian
See below. > On 08/31/2021 5:35 AM Aki Tuomi wrote: > > > On 31/08/2021 13:59 Tom Hendrikx wrote: > > > > On 31-08-2021 12:01, Aki Tuomi wrote: > > > > > >> On 31/08/2021 10:56 Felix Zielcke wrote: > > >> > > >> Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: > > >>> > > >>>> On 31/08/2021 00:11 Joan Moreau wrote: > > >>>> > > >>>> Hi > > >>>> There seems to be 2 plugins doing the same thins > > >>>> - https://github.com/slusarz/dovecot-fts-flatcurve/ > > >>>> - https://github.com/grosjo/fts-xapian/ (mine) ... > > >>>> Isn't there double work here ? FYI, Joan also reached out to me on the Github project page, so you can see my response to this question there: https://github.com/slusarz/dovecot-fts-flatcurve/issues/5 Short answer: I see the plugins as "different". Yes, they both use the same underlying indexing technology (Xapian), but the design goals, features, and implementations are unique for each project. Thus, I don't see them as duplicate work. > > >> Is there somewhere a direct comparison of them? > > >> I currenty use fts-xapian from Joan without problems. > > >> But what would be the advantages of fts-flatcurve over fts-xapian? I was asked a similar question previously. See https://github.com/slusarz/dovecot-fts-flatcurve/issues/4#issuecomment-902425597 > > > fts_flatcurve does only full word searching, although you can use > > > fts_filters and fts_tokenizers settings to affect stemming and other > > > matching to make it work with plurals and such. Aki's knowledge is a bit dated - fts_flatcurve now supports both prefix matching (this is what Xapian provides by default) AND full substring matching (this is what is technically required by the IMAP RFC). Although the current fts_flatcurve default is for full substring matching, it is debatable whether that is the correct choice. Users tend to think of search in terms of "prefix" matching - that's how Google works - and substring matching requires significant additional storage since Xapian doesn't support it natively. > > I still think it's weird to see that Open-Xchange starts a FTS Xapian > > plugin with mostly the same basic functionality that is already > > available in an existing plugin maintained by someone in the community As Aki noted, Open-Xchange didn't "start" anything. fts_flatcurve is entirely a project I have developed on my personal time. As a Dovecot CE user, I had the same complaint that most people have - in current versions of Dovecot, there is no easy-to-use FTS option that works out-of-the-box. Solr is the only search driver still actively supported, and it requires significant additional resources and configuration to setup. Unfortunately, we don't have the resources within Open-Xchange to create a CE-only search option as our customers generally use our proprietary, object storage aware FTS system that is part of OX Dovecot Pro. Without commercial demand, it is difficult to justify paid work on features that do not produce revenue in return. (fts_flatcurve does use Dovecot core features that are maintained for the Pro FTS system, like stemming, so we can take advantage of trickle down benefits from the commercial code maintenance.) My goal was to give back to the community by providing an easy-to-use, resource sensitive, no-maintenance option to address this current product limitation. Hopefully, fts_flatcurve will fit the requirements for most smaller installations that run Dovecot that don't require the carrier/enterprise grade FTS solutions large customers demand. > > Especially if that happens without any (apparent) communication with the > > existing plugin developer to find out whether fixing the issues that > > slusarz/Open-Xchange seem to have with the existing plugin, can be fixed. There were fundamental disagreements about design goals, and Joan (or anybody, really) would likely not have appreciated someone coming in and demanding that the entire architecture and design of the project needed to be changed based on my personal opinions. This is open source. Everybody should be free to participate as they wish, as long as you are not negatively affecting someone else. I don't think that has happened here. > > Combining forces just seems a better way to spend scarce development > > resources than building something similar (but different) without any > > communication. I disagree, but that's ok. People can do that. michael
Re: Duplicate plugins - FTS Xapian
> On 31/08/2021 13:59 Tom Hendrikx wrote: > > > On 31-08-2021 12:01, Aki Tuomi wrote: > > > >> On 31/08/2021 10:56 Felix Zielcke wrote: > >> > >> > >> Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: > >>> > >>>> On 31/08/2021 00:11 Joan Moreau wrote: > >>>> > >>>> > >>>> Hi > >>>> There seems to be 2 plugins doing the same thins > >>>> - https://github.com/slusarz/dovecot-fts-flatcurve/ > >>>> - https://github.com/grosjo/fts-xapian/ (mine) > >>>> Both are in the doc of dovecot > >>>> https://doc.dovecot.org/configuration_manual/fts/ > >>>> > >>>> I am currently working hard to push it to RPM package, and plugin > >>>> is already approved by ArchLinux and Debian > >>>> > >>>> Isn't there double work here ? > >>>> Thanks > >>>> JM > >>> > >>> If you look closer, you can see they are not exactly duplicates. > >>> Flatcurve works differently than your plugin. > >>> > >>> Aki > >> > >> Is there somewhere a direct comparison of them? > >> I currenty use fts-xapian from Joan without problems. > >> But what would be the advantages of fts-flatcurve over fts-xapian? > > > > fts_flatcurve does only full word searching, although you can use > > fts_filters and fts_tokenizers settings to affect stemming and other > > matching to make it work with plurals and such. > > > > Both plugins have their merits. > > I still think it's weird to see that Open-Xchange starts a FTS Xapian > plugin with mostly the same basic functionality that is already > available in an existing plugin maintained by someone in the community > Especially if that happens without any (apparent) communication with the > existing plugin developer to find out whether fixing the issues that > slusarz/Open-Xchange seem to have with the existing plugin, can be fixed. > > Combining forces just seems a better way to spend scarce development > resources than building something similar (but different) without any > communication. > > (Note: I don't use any of these plugins). > > My 2 cents, > > Tom Just for clarity, Open-Xchange has not written any xapian plugin whatsoever. Aki
Re: Duplicate plugins - FTS Xapian
On 31-08-2021 12:01, Aki Tuomi wrote: On 31/08/2021 10:56 Felix Zielcke wrote: Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: On 31/08/2021 00:11 Joan Moreau wrote: Hi There seems to be 2 plugins doing the same thins - https://github.com/slusarz/dovecot-fts-flatcurve/ - https://github.com/grosjo/fts-xapian/ (mine) Both are in the doc of dovecot https://doc.dovecot.org/configuration_manual/fts/ I am currently working hard to push it to RPM package, and plugin is already approved by ArchLinux and Debian Isn't there double work here ? Thanks JM If you look closer, you can see they are not exactly duplicates. Flatcurve works differently than your plugin. Aki Is there somewhere a direct comparison of them? I currenty use fts-xapian from Joan without problems. But what would be the advantages of fts-flatcurve over fts-xapian? fts_flatcurve does only full word searching, although you can use fts_filters and fts_tokenizers settings to affect stemming and other matching to make it work with plurals and such. Both plugins have their merits. I still think it's weird to see that Open-Xchange starts a FTS Xapian plugin with mostly the same basic functionality that is already available in an existing plugin maintained by someone in the community Especially if that happens without any (apparent) communication with the existing plugin developer to find out whether fixing the issues that slusarz/Open-Xchange seem to have with the existing plugin, can be fixed. Combining forces just seems a better way to spend scarce development resources than building something similar (but different) without any communication. (Note: I don't use any of these plugins). My 2 cents, Tom
Re: Duplicate plugins - FTS Xapian
> On 31/08/2021 10:56 Felix Zielcke wrote: > > > Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: > > > > > On 31/08/2021 00:11 Joan Moreau wrote: > > > > > > > > > Hi > > > There seems to be 2 plugins doing the same thins > > > - https://github.com/slusarz/dovecot-fts-flatcurve/ > > > - https://github.com/grosjo/fts-xapian/ (mine) > > > Both are in the doc of dovecot > > > https://doc.dovecot.org/configuration_manual/fts/ > > > > > > I am currently working hard to push it to RPM package, and plugin > > > is already approved by ArchLinux and Debian > > > > > > Isn't there double work here ? > > > Thanks > > > JM > > > > If you look closer, you can see they are not exactly duplicates. > > Flatcurve works differently than your plugin. > > > > Aki > > Is there somewhere a direct comparison of them? > I currenty use fts-xapian from Joan without problems. > But what would be the advantages of fts-flatcurve over fts-xapian? fts_flatcurve does only full word searching, although you can use fts_filters and fts_tokenizers settings to affect stemming and other matching to make it work with plurals and such. Both plugins have their merits. Aki
Re: Duplicate plugins - FTS Xapian
Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: > > > On 31/08/2021 00:11 Joan Moreau wrote: > > > > > > Hi > > There seems to be 2 plugins doing the same thins > > - https://github.com/slusarz/dovecot-fts-flatcurve/ > > - https://github.com/grosjo/fts-xapian/ (mine) > > Both are in the doc of dovecot > > https://doc.dovecot.org/configuration_manual/fts/ > > > > I am currently working hard to push it to RPM package, and plugin > > is already approved by ArchLinux and Debian > > > > Isn't there double work here ? > > Thanks > > JM > > If you look closer, you can see they are not exactly duplicates. > Flatcurve works differently than your plugin. > > Aki Is there somewhere a direct comparison of them? I currenty use fts-xapian from Joan without problems. But what would be the advantages of fts-flatcurve over fts-xapian?
Re: Duplicate plugins - FTS Xapian
> On 31/08/2021 00:11 Joan Moreau wrote: > > > Hi > There seems to be 2 plugins doing the same thins > - https://github.com/slusarz/dovecot-fts-flatcurve/ > - https://github.com/grosjo/fts-xapian/ (mine) > Both are in the doc of dovecot > https://doc.dovecot.org/configuration_manual/fts/ > > I am currently working hard to push it to RPM package, and plugin is already > approved by ArchLinux and Debian > > Isn't there double work here ? > Thanks > JM If you look closer, you can see they are not exactly duplicates. Flatcurve works differently than your plugin. Aki
Duplicate plugins - FTS Xapian
Hi There seems to be 2 plugins doing the same thins - https://github.com/slusarz/dovecot-fts-flatcurve/ - https://github.com/grosjo/fts-xapian/ (mine) Both are in the doc of dovecot https://doc.dovecot.org/configuration_manual/fts/ I am currently working hard to push it to RPM package, and plugin is already approved by ArchLinux and Debian Isn't there double work here ? Thanks JM
Re: FTS Xapian -> FTS core issue
The issue is not in plug-in then. Maybe Aki or Timo knows where does this bug come from ? On 2019-06-07 14:09, Daniel Miller wrote: Yes, latest git version. The logs show (as I read them) returned results - yet nothing shows in the client. The logs look the same (with different numbers) when querying "regular" folders - but results are shown in clients. -- Daniel On June 6, 2019 12:16:08 AM Joan Moreau wrote: Hi Are you using the latest git version ? WHich part exactly of your logs relates to "virtual folders do not work" ? On 2019-06-05 13:08, Daniel Miller via dovecot wrote: Logs: Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2857830c70c844e2f1d3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 1 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_78544714f3f1ae5b9b0d3bda95b5 Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 53 results in 40 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_bdcb8e2172fadf4db50b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 12 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_be25c00241fedf4de00b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 3 results in 32 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_a7e75820d9fadf4dd90b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 11 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6fa78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 21 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6ea78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:
Re: FTS Xapian
Yes, latest git version. The logs show (as I read them) returned results - yet nothing shows in the client. The logs look the same (with different numbers) when querying "regular" folders - but results are shown in clients. -- Daniel On June 6, 2019 12:16:08 AM Joan Moreau wrote: Hi Are you using the latest git version ? WHich part exactly of your logs relates to "virtual folders do not work" ? On 2019-06-05 13:08, Daniel Miller via dovecot wrote: Logs: Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2857830c70c844e2f1d3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 1 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_78544714f3f1ae5b9b0d3bda95b5 Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 53 results in 40 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_bdcb8e2172fadf4db50b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 12 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_be25c00241fedf4de00b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 3 results in 32 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_a7e75820d9fadf4dd90b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 11 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6fa78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 21 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6ea78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes
Re: FTS Xapian
Hi Are you using the latest git version ? WHich part exactly of your logs relates to "virtual folders do not work" ? On 2019-06-05 13:08, Daniel Miller via dovecot wrote: Logs: Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2857830c70c844e2f1d3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 1 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_78544714f3f1ae5b9b0d3bda95b5 Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 53 results in 40 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_bdcb8e2172fadf4db50b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 12 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_be25c00241fedf4de00b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 3 results in 32 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_a7e75820d9fadf4dd90b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 11 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6fa78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 21 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6ea78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 1 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2c3522c5d9b9d4f8847e130c744 Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FL
Re: FTS Xapian
Logs: Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2857830c70c844e2f1d3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 1 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_78544714f3f1ae5b9b0d3bda95b5 Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 53 results in 40 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_bdcb8e2172fadf4db50b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 12 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_be25c00241fedf4de00b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 3 results in 32 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_a7e75820d9fadf4dd90b00003bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 11 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6fa78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 21 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_6ea78f2738cbdf4d007b3bc41c5f Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR cc:"dovecot" OR bcc:"dovecot" OR message-id:"dovecot" OR body:"dovecot") Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: 0 results in 1 ms Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: Opening DB (RO) /var/mail/amfes.com/dmiller/sdbox/xapian-indexes/db_f2c3522c5d9b9d4f8847e130c744 Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: FLAG=AND Jun 5 06:02:25 bubba dovecot: imap(dmil...@amfes.com)<25877>: FTS Xapian: Query= (subject:"dovecot" OR from:"dovecot" OR to:"dovecot" OR
Re: FTS Xapian
Hi Can you post your dovecot conf file and the subset of the log files related to the issue ? thanks On June 5, 2019 9:29:13 AM Daniel Miller via dovecot wrote: For my primary namespace this is working fine - thanks to the developers! It also appears to work great for shared folders as well. But my virtual folders aren't returning results - at least not to the client. The logs show FTS Xapian opening several DB files and getting results - but nothing is being returned to client. Is this a config issue on my side or is this a current limitation of the plugin? -- Daniel
FTS Xapian
For my primary namespace this is working fine - thanks to the developers! It also appears to work great for shared folders as well. But my virtual folders aren't returning results - at least not to the client. The logs show FTS Xapian opening several DB files and getting results - but nothing is being returned to client. Is this a config issue on my side or is this a current limitation of the plugin? -- Daniel
Re: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)
Can you clarify the piece of code or give an example on how to "Get list of UIDs for all mails in each folder " and how to get the "list of all folder/mailbox" from a *backend input ? On 2019-02-17 14:52, Aki Tuomi wrote: Not really, as the steps outlined by Timo would not get done. Aki On 17 February 2019 at 10:56 Joan Moreau via dovecot wrote: In such case, as long as the API is not upgraded, should doveadm index -A -q \* be considered a replacement of doveadm fts rescan On 2019-02-14 16:24, Timo Sirainen via dovecot wrote: Hi, The rescan() function is a bit badly designed. Currently what you could do what fts-lucene does and: - Get list of UIDs for all mails in each folder - If Xapian has UID that doesn't exist -> delete it from Xapian - If UID is missing from Xapian -> expunge the rest of the UIDs in that folder, so the next indexing will cause them to be indexed The expunging of rest of the mails is rather ugly, yes.. A better API would be if backend simply had a way to iterate all mails in the index, preferrably sorted by folder. Then a more generic code could go through them and expunge the necessary mails and index the missing mails. Although not all FTS backends support indexing in the middle. Anyway, we don't really have time to implement this new API soon. I'm not sure if this is a big problem though. I don't think most people running FTS have ever run rescan. On 8 Feb 2019, at 9.54, Joan Moreau via dovecot wrote: Hi, THis is a core problem in Dovecot in my understanding. In my opinion, the rescan in dovecot should send to the FTS plugin the list of "supposedly" indexed emails (UID), and the plugin shall purge the redundant UID (i..e UID present in the index but not in the list sent by dovecot) and send back the list of UID not in its indexes to dovecot, so Dovect can send one by one the missing emails WHat do you think ? Original Message ---- SUBJECT: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15) DATE: 2019-02-08 08:28 FROM: Leonard Lausen TO: grosjo/fts-xapian CC: Subscribed REPLY-TO: grosjo/fts-xapian doveadm fts rescan -A deletes all indices, ie. all folders and files in the xapian-indexes are deleted. However, according to man doveadm fts, the rescan command should only Scan what mails exist in the full text search index and compare those to what actually exist in mailboxes. This removes mails from the index that have already been expunged and makes sure that the next doveadm index will index all the missing mails (if any). Deleting all indices does not seem to be the intended action, especially as constructing the index anew may take very long on large mailboxes. -- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub [1 [1]], or mute the thread [2]. Links: ------ [1] https://github.com/grosjo/fts-xapian/issues/15 [2] https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp Links: ------ [1] https://github.com/grosjo/fts-xapian/issues/15
Re: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)
Not really, as the steps outlined by Timo would not get done. Aki > On 17 February 2019 at 10:56 Joan Moreau via dovecot > wrote: > > > In such case, as long as the API is not upgraded, should > > doveadm index -A -q \* > > be considered a replacement of > > doveadm fts rescan > > On 2019-02-14 16:24, Timo Sirainen via dovecot wrote: > > > Hi, > > > > The rescan() function is a bit badly designed. Currently what you could do > > what fts-lucene does and: > > - Get list of UIDs for all mails in each folder > > - If Xapian has UID that doesn't exist -> delete it from Xapian > > - If UID is missing from Xapian -> expunge the rest of the UIDs in that > > folder, so the next indexing will cause them to be indexed > > > > The expunging of rest of the mails is rather ugly, yes.. A better API would > > be if backend simply had a way to iterate all mails in the index, > > preferrably sorted by folder. Then a more generic code could go through > > them and expunge the necessary mails and index the missing mails. Although > > not all FTS backends support indexing in the middle. Anyway, we don't > > really have time to implement this new API soon. > > > > I'm not sure if this is a big problem though. I don't think most people > > running FTS have ever run rescan. > > > > On 8 Feb 2019, at 9.54, Joan Moreau via dovecot > > wrote: > > > > Hi, > > > > THis is a core problem in Dovecot in my understanding. > > > > In my opinion, the rescan in dovecot should send to the FTS plugin the list > > of "supposedly" indexed emails (UID), and the plugin shall purge the > > redundant UID (i..e UID present in the index but not in the list sent by > > dovecot) and send back the list of UID not in its indexes to dovecot, so > > Dovect can send one by one the missing emails > > > > WHat do you think ? > > > > Original Message ---- > > > > SUBJECT: > > [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15) > > > > DATE: > > 2019-02-08 08:28 > > > > FROM: > > Leonard Lausen > > > > TO: > > grosjo/fts-xapian > > > > CC: > > Subscribed > > > > REPLY-TO: > > grosjo/fts-xapian > > > > > > doveadm fts rescan -A deletes all indices, ie. all folders and files in the > > xapian-indexes are deleted. However, according to man doveadm fts, the > > rescan command should only > > > > Scan what mails exist in the full text search index and compare those to > > what > > actually exist in mailboxes. This removes mails from the index that have > > already > > been expunged and makes sure that the next doveadm index will index all the > > missing mails (if any). > > > > Deleting all indices does not seem to be the intended action, especially as > > constructing the index anew may take very long on large mailboxes. > > > > -- > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub [1], or mute the thread [2]. > > > Links: > -- > [1] https://github.com/grosjo/fts-xapian/issues/15 > [2] > https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp
Re: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)
In such case, as long as the API is not upgraded, should doveadm index -A -q \* be considered a replacement of doveadm fts rescan On 2019-02-14 16:24, Timo Sirainen via dovecot wrote: Hi, The rescan() function is a bit badly designed. Currently what you could do what fts-lucene does and: - Get list of UIDs for all mails in each folder - If Xapian has UID that doesn't exist -> delete it from Xapian - If UID is missing from Xapian -> expunge the rest of the UIDs in that folder, so the next indexing will cause them to be indexed The expunging of rest of the mails is rather ugly, yes.. A better API would be if backend simply had a way to iterate all mails in the index, preferrably sorted by folder. Then a more generic code could go through them and expunge the necessary mails and index the missing mails. Although not all FTS backends support indexing in the middle. Anyway, we don't really have time to implement this new API soon. I'm not sure if this is a big problem though. I don't think most people running FTS have ever run rescan. On 8 Feb 2019, at 9.54, Joan Moreau via dovecot wrote: Hi, THis is a core problem in Dovecot in my understanding. In my opinion, the rescan in dovecot should send to the FTS plugin the list of "supposedly" indexed emails (UID), and the plugin shall purge the redundant UID (i..e UID present in the index but not in the list sent by dovecot) and send back the list of UID not in its indexes to dovecot, so Dovect can send one by one the missing emails WHat do you think ? Original Message SUBJECT: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15) DATE: 2019-02-08 08:28 FROM: Leonard Lausen TO: grosjo/fts-xapian CC: Subscribed REPLY-TO: grosjo/fts-xapian doveadm fts rescan -A deletes all indices, ie. all folders and files in the xapian-indexes are deleted. However, according to man doveadm fts, the rescan command should only Scan what mails exist in the full text search index and compare those to what actually exist in mailboxes. This removes mails from the index that have already been expunged and makes sure that the next doveadm index will index all the missing mails (if any). Deleting all indices does not seem to be the intended action, especially as constructing the index anew may take very long on large mailboxes. -- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2]. Links: -- [1] https://github.com/grosjo/fts-xapian/issues/15 [2] https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp
Re: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)
Hi, The rescan() function is a bit badly designed. Currently what you could do what fts-lucene does and: - Get list of UIDs for all mails in each folder - If Xapian has UID that doesn't exist -> delete it from Xapian - If UID is missing from Xapian -> expunge the rest of the UIDs in that folder, so the next indexing will cause them to be indexed The expunging of rest of the mails is rather ugly, yes.. A better API would be if backend simply had a way to iterate all mails in the index, preferrably sorted by folder. Then a more generic code could go through them and expunge the necessary mails and index the missing mails. Although not all FTS backends support indexing in the middle. Anyway, we don't really have time to implement this new API soon. I'm not sure if this is a big problem though. I don't think most people running FTS have ever run rescan. > On 8 Feb 2019, at 9.54, Joan Moreau via dovecot wrote: > > > > > Hi, > > THis is a core problem in Dovecot in my understanding. > > In my opinion, the rescan in dovecot should send to the FTS plugin the list > of "supposedly" indexed emails (UID), and the plugin shall purge the > redundant UID (i..e UID present in the index but not in the list sent by > dovecot) and send back the list of UID not in its indexes to dovecot, so > Dovect can send one by one the missing emails > > > > WHat do you think ? > > > > Original Message > > Subject: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices > (#15) > Date: 2019-02-08 08:28 > From: Leonard Lausen > To: grosjo/fts-xapian > Cc: Subscribed > Reply-To: grosjo/fts-xapian > > > > doveadm fts rescan -A deletes all indices, ie. all folders and files in the > xapian-indexes are deleted. However, according to man doveadm fts, the rescan > command should only > >> Scan what mails exist in the full text search index and compare those to what >> actually exist in mailboxes. This removes mails from the index that have >> already >> been expunged and makes sure that the next doveadm index will index all the >> missing mails (if any). >> > Deleting all indices does not seem to be the intended action, especially as > constructing the index anew may take very long on large mailboxes. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <https://github.com/grosjo/fts-xapian/issues/15>, or mute the thread > <https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp>. > > >
Re: Fwd: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)
Hi Anyone ? On 2019-02-08 08:54, Joan Moreau via dovecot wrote: Hi, THis is a core problem in Dovecot in my understanding. In my opinion, the rescan in dovecot should send to the FTS plugin the list of "supposedly" indexed emails (UID), and the plugin shall purge the redundant UID (i..e UID present in the index but not in the list sent by dovecot) and send back the list of UID not in its indexes to dovecot, so Dovect can send one by one the missing emails WHat do you think ? Original Message SUBJECT: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15) DATE: 2019-02-08 08:28 FROM: Leonard Lausen TO: grosjo/fts-xapian CC: Subscribed REPLY-TO: grosjo/fts-xapian doveadm fts rescan -A deletes all indices, ie. all folders and files in the xapian-indexes are deleted. However, according to man doveadm fts, the rescan command should only Scan what mails exist in the full text search index and compare those to what actually exist in mailboxes. This removes mails from the index that have already been expunged and makes sure that the next doveadm index will index all the missing mails (if any). Deleting all indices does not seem to be the intended action, especially as constructing the index anew may take very long on large mailboxes. -- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2]. Links: -- [1] https://github.com/grosjo/fts-xapian/issues/15 [2] https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp
Fwd: [grosjo/fts-xapian] `doveadm fts rescan` removes all indices (#15)
Hi,
THis is a core problem in Dovecot in my understanding.
In my opinion, the rescan in dovecot should send to the FTS plugin the
list of "supposedly" indexed emails (UID), and the plugin shall purge
the redundant UID (i..e UID present in the index but not in the list
sent by dovecot) and send back the list of UID not in its indexes to
dovecot, so Dovect can send one by one the missing emails
WHat do you think ?
Original Message
SUBJECT:
[grosjo/fts-xapian] `doveadm fts rescan` removes all indices
(#15)
DATE:
2019-02-08 08:28
FROM:
Leonard Lausen
TO:
grosjo/fts-xapian
CC:
Subscribed
REPLY-TO:
grosjo/fts-xapian
doveadm fts rescan -A deletes all indices, ie. all folders and files in
the xapian-indexes are deleted. However, according to man doveadm fts,
the rescan command should only
Scan what mails exist in the full text search index and compare those to what
actually exist in mailboxes. This removes mails from the index that have already
been expunged and makes sure that the next doveadm index will index all the
missing mails (if any).
Deleting all indices does not seem to be the intended action, especially
as constructing the index anew may take very long on large mailboxes.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub [1], or mute the thread
[2].
Links:
--
[1] https://github.com/grosjo/fts-xapian/issues/15
[2]
https://github.com/notifications/unsubscribe-auth/ACLmB9OB-7GaKIvhNc8sCgi7KQTrjNnoks5vLScugaJpZM4auCWp
Re: [FTS Xapian] RC release
On Thu, 24 Jan 2019 14:35:13 +0100 Joan Moreau via dovecot wrote: > Hi, > > FTS Xapian matches my targets for the plugins (replacing deprecated > fts-squat in a production environment) > > https://github.com/grosjo/fts-xapian > > Please do not hesitate to add "issues" on github, if the case happen created a ticket with all the issues I found while trying to package it for openSUSE. -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
Re: [FTS Xapian] RC release
On 24 Jan 2019, at 14:35, Joan Moreau via dovecot wrote: > Hi, Hi Joan, > FTS Xapian matches my targets for the plugins (replacing deprecated > fts-squat in a production environment) > https://github.com/grosjo/fts-xapian > Please do not hesitate to add "issues" on github, if the case happen > Hope it helps > JM I've been following the recent huge thread about Solr/Xapian as I planned to move from old fts-squat that still works ok on 2.3.4 but marked as deprecated and really appreciate your effort along with others contributors who helped you to make Xapian available as a dovecot fas plugin, thanks! -- matt [at] lv223.org GPG key ID: 7D91A8CA signature.asc Description: OpenPGP digital signature
[FTS Xapian] RC release
Hi, FTS Xapian matches my targets for the plugins (replacing deprecated fts-squat in a production environment) https://github.com/grosjo/fts-xapian Please do not hesitate to add "issues" on github, if the case happen Hope it helps JM
Re: [FTS Xapian] Beta release
It is indeed better is you use the issue tracker of github: https://github.com/grosjo/fts-xapian/issues I updated the Readme accordingly On 2019-01-14 14:24, Stephan Bosch wrote: Op 14-1-2019 om 13:40 schreef Aki Tuomi: Just to remind that now that there is a github repo for fts-xapian, you could maybe open these issues there instead? Although README.md currently says: "Please feel free to send your questions, together with the dovecot log file, to j...@grosjo.net <mailto:j...@grosjo.net> or to the dovecot ML dovecot@dovecot.org <mailto:dovecot@dovecot.org>." Regards, Stephan. Aki On 14.1.2019 14.29, Odhiambo Washington wrote: Testing a compile on FreeBSD. gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/fts-xapian/src' /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:3:10: fatal error: 'xapian.h' file not found #include ^~ Well, I installed xapian-core and the xapian.h is in /usr/local/include/ I can overcome the fatal error by doing: env CPPFLAGS=-I/usr/local/include PANDOC=false ./configure --prefix=/opt --with-dovecot=/opt/dovecot2.3/lib/dovecot/ Is that something that you can address within the code or we (*BSD) have to live with it? During `make`, the following warning is generated: /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:486:14: warning: format string is not a string literal (potentially insecure) [-Wformat-security] i_warning(e.get_msg().c_str()); ^~~ fts-backend-xapian.cpp:486:14: note: treat the string as an argument to avoid this i_warning(e.get_msg().c_str()); ^ "%s", 1 warning generated. Is that something you can look into as well? On Mon, 14 Jan 2019 at 11:43, Joan Moreau mailto:j...@grosjo.net>> wrote: THank you Odhiambo. I updated accordingly On 2019-01-14 08:07, Odhiambo Washington wrote: In your README.md, perhaps "This project intends to provide a straightforward and simple *procedure *to configure FTS plugin for Dovecot, leveraging the efforts by the Xapian.org team." is better?? Also in the part after cloning from git: ./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This /path/to/dovecot is not obvious. Is it the dovecot binary or what??] On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot mailto:dovecot@dovecot.org>> wrote: Thank you Stephan. The version here shall be up and running : https://github.com/grosjo/fts-xapian On 2019-01-14 00:07, Stephan Bosch wrote: Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: I tried to combined it, the "autoreconf" errors are solved Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... How to adjust this to have it arriving in the proper folder ? Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. For Debian I use the following configure command: ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static Regards, Stephan On 2019-01-13 21:01, Tuomi, Aki wrote: You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki Original message From: Joan Moreau mailto:j...@grosjo.net>> Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch mailto:step...@rename-it.nl>> Cc: Aki Tuomi mailto:aki.tu...@open-xchange.com>> Subject: Re: [FTS Xapian] Beta relea
Re: [FTS Xapian] Beta release
It is indeed better is you use the issue tracker of github: https://github.com/grosjo/fts-xapian/issues I updated the Readme accordingly On 2019-01-14 14:24, Stephan Bosch wrote: Op 14-1-2019 om 13:40 schreef Aki Tuomi: Just to remind that now that there is a github repo for fts-xapian, you could maybe open these issues there instead? Although README.md currently says: "Please feel free to send your questions, together with the dovecot log file, to j...@grosjo.net <mailto:j...@grosjo.net> or to the dovecot ML dovecot@dovecot.org <mailto:dovecot@dovecot.org>." Regards, Stephan.
Re: [FTS Xapian] Beta release
Op 14-1-2019 om 13:40 schreef Aki Tuomi: Just to remind that now that there is a github repo for fts-xapian, you could maybe open these issues there instead? Although README.md currently says: "Please feel free to send your questions, together with the dovecot log file, to j...@grosjo.net <mailto:j...@grosjo.net> or to the dovecot ML dovecot@dovecot.org <mailto:dovecot@dovecot.org>." Regards, Stephan. Aki On 14.1.2019 14.29, Odhiambo Washington wrote: Testing a compile on FreeBSD. gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/fts-xapian/src' /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:3:10: fatal error: 'xapian.h' file not found #include ^~ Well, I installed xapian-core and the xapian.h is in /usr/local/include/ I can overcome the fatal error by doing: env CPPFLAGS=-I/usr/local/include PANDOC=false ./configure --prefix=/opt --with-dovecot=/opt/dovecot2.3/lib/dovecot/ Is that something that you can address within the code or we (*BSD) have to live with it? During `make`, the following warning is generated: /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:486:14: warning: format string is not a string literal (potentially insecure) [-Wformat-security] i_warning(e.get_msg().c_str()); ^~~ fts-backend-xapian.cpp:486:14: note: treat the string as an argument to avoid this i_warning(e.get_msg().c_str()); ^ "%s", 1 warning generated. Is that something you can look into as well? On Mon, 14 Jan 2019 at 11:43, Joan Moreau <mailto:j...@grosjo.net>> wrote: THank you Odhiambo. I updated accordingly On 2019-01-14 08:07, Odhiambo Washington wrote: In your README.md, perhaps "This project intends to provide a straightforward and simple *procedure *to configure FTS plugin for Dovecot, leveraging the efforts by the Xapian.org team." is better?? Also in the part after cloning from git: ./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This /path/to/dovecot is not obvious. Is it the dovecot binary or what??] On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot mailto:dovecot@dovecot.org>> wrote: Thank you Stephan. The version here shall be up and running : https://github.com/grosjo/fts-xapian On 2019-01-14 00:07, Stephan Bosch wrote: Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: I tried to combined it, the "autoreconf" errors are solved Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... How to adjust this to have it arriving in the proper folder ? Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. For Debian I use the following configure command: ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static Regards, Stephan On 2019-01-13 21:01, Tuomi, Aki wrote:
Re: [FTS Xapian] Beta release
Op 14-1-2019 om 13:29 schreef Odhiambo Washington:
Testing a compile on FreeBSD.
[...]
During `make`, the following warning is generated:
/bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I.
-I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include
-I/opt/dovecot2.3/include/dovecot -g -O2 -MT
fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o
fts-backend-xapian.lo fts-backend-xapian.cpp
libtool: compile: c++ -DHAVE_CONFIG_H -I. -I..
-I/opt/dovecot2.3/include/dovecot -I/usr/local/include
-I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD
-MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC
-DPIC -o .libs/fts-backend-xapian.o
fts-backend-xapian.cpp:486:14: warning: format string is not a string
literal (potentially insecure) [-Wformat-security]
i_warning(e.get_msg().c_str());
^~~
fts-backend-xapian.cpp:486:14: note: treat the string as an argument
to avoid this
i_warning(e.get_msg().c_str());
^
"%s",
1 warning generated.
-> i_warning("%s", e.get_msg().c_str());
Regards,
Stephan.
Re: [FTS Xapian] Beta release
Hi Joan, opened an issue here: <https://github.com/grosjo/fts-xapian/issues/2 <https://github.com/grosjo/fts-xapian/issues/2>> But no, this does not fix the crash, sorry. Thanks, Paul > On 14. Jan 2019, at 13:58, Joan Moreau via dovecot > wrote: > > THanks Paul > > Can you try indexing the same emails with "full=10" for instance in the > dovecot.conf ? > smime.p7s Description: S/MIME cryptographic signature
Re: [FTS Xapian] Beta release
THanks Paul Can you try indexing the same emails with "full=10" for instance in the dovecot.conf ? On 2019-01-14 12:19, Paul Hecker via dovecot wrote: Thank you. Here is the stack trace with all debug symbols: On 14. Jan 2019, at 11:07, Stephan Bosch wrote: Op 14-1-2019 om 10:55 schreef Paul Hecker via dovecot: OK, got it (my fault, as always, put the LimitCORE in the wrong line). Here is the stack trace: If you want to get rid of those "??" stack trace elements, you'll need to install debug symbols for the xapian library. It depends on your system how to do that (usually some separate package). Regards, Stephan. On 14. Jan 2019, at 10:33, Joan Moreau via dovecot mailto:dovecot@dovecot.org>> wrote: Difficult to figure out without a coredump + gdb I have also battled quite a lot to make sure dovecot can core dump on my Archlinux servers. I remember that the key point was putting*fs.suid_dumpable=2* in /etc/sysctl.d/ conf files, *LimitCORE=infinity* in /etc/systemd/system/multi-user.target.wants/dovecot.service, and rebooting the server. My own coredumps are on /var/lib/systemd/coredump/
Re: [FTS Xapian] Beta release
Just to remind that now that there is a github repo for fts-xapian, you could maybe open these issues there instead? Aki On 14.1.2019 14.29, Odhiambo Washington wrote: > Testing a compile on FreeBSD. > > gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/fts-xapian/src' > /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. > -I.. -I/opt/dovecot2.3/include/dovecot > -I/opt/dovecot2.3/include/dovecot -g -O2 -MT > fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o > fts-backend-xapian.lo fts-backend-xapian.cpp > libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. > -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g > -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo > -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o > fts-backend-xapian.cpp:3:10: fatal error: 'xapian.h' file not found > #include > ^~ > > Well, I installed xapian-core and the xapian.h is in /usr/local/include/ > > I can overcome the fatal error by doing: > > env CPPFLAGS=-I/usr/local/include PANDOC=false ./configure > --prefix=/opt --with-dovecot=/opt/dovecot2.3/lib/dovecot/ > > Is that something that you can address within the code or we (*BSD) > have to live with it? > > During `make`, the following warning is generated: > > /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. > -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include > -I/opt/dovecot2.3/include/dovecot -g -O2 -MT > fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o > fts-backend-xapian.lo fts-backend-xapian.cpp > libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. > -I/opt/dovecot2.3/include/dovecot -I/usr/local/include > -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD > -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC > -DPIC -o .libs/fts-backend-xapian.o > fts-backend-xapian.cpp:486:14: warning: format string is not a string > literal (potentially insecure) [-Wformat-security] > i_warning(e.get_msg().c_str()); > ^~~ > fts-backend-xapian.cpp:486:14: note: treat the string as an argument > to avoid this > i_warning(e.get_msg().c_str()); > ^ > "%s", > 1 warning generated. > > > Is that something you can look into as well? > > > > On Mon, 14 Jan 2019 at 11:43, Joan Moreau <mailto:j...@grosjo.net>> wrote: > > THank you Odhiambo. I updated accordingly > > > > > > On 2019-01-14 08:07, Odhiambo Washington wrote: > >> In your README.md, perhaps "This project intends to provide a >> straightforward and simple *procedure *to configure FTS plugin >> for Dovecot, leveraging the efforts by the Xapian.org team." is >> better?? >> Also in the part after cloning from git: >> >> ./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This >> /path/to/dovecot is not obvious. Is it the dovecot binary or what??] >> >> On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot >> mailto:dovecot@dovecot.org>> wrote: >> >> Thank you Stephan. >> >> The version here shall be up and running : >> https://github.com/grosjo/fts-xapian >> >> >> >> >> >> On 2019-01-14 00:07, Stephan Bosch wrote: >> >> >> >> Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: >> >> >> I tried to combined it, the "autoreconf" errors are solved >> >> Now, when I type "make install", the lib is not >> pushed into dovecot folder, but somewhere in >> /usr/local/... >> >> How to adjust this to have it arriving in the proper folder ? >> >> >> Depends on your system. It mostly a matter of setting a >> proper --prefix directory for configure, but other paths >> are configurable as well. I usually check what the >> official distribution package for Dovecot is doing and >> use that as a basis. >> >> For Debian I use the following configure command: >> >> ./configure --with-ldap=plugin --with-ssl=openssl >> --with-sql=plugin --with-lua=plugin --with-pgsql >> --with-mysql --with-sqlite \ >> --with-gssapi=
Re: [FTS Xapian] Beta release
Testing a compile on FreeBSD. gmake[2]: Entering directory '/usr/home/wash/Tools/Dovecot/fts-xapian/src' /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:3:10: fatal error: 'xapian.h' file not found #include ^~ Well, I installed xapian-core and the xapian.h is in /usr/local/include/ I can overcome the fatal error by doing: env CPPFLAGS=-I/usr/local/include PANDOC=false ./configure --prefix=/opt --with-dovecot=/opt/dovecot2.3/lib/dovecot/ Is that something that you can address within the code or we (*BSD) have to live with it? During `make`, the following warning is generated: /bin/sh ../libtool --tag=CXX --mode=compile c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c -o fts-backend-xapian.lo fts-backend-xapian.cpp libtool: compile: c++ -DHAVE_CONFIG_H -I. -I.. -I/opt/dovecot2.3/include/dovecot -I/usr/local/include -I/opt/dovecot2.3/include/dovecot -g -O2 -MT fts-backend-xapian.lo -MD -MP -MF .deps/fts-backend-xapian.Tpo -c fts-backend-xapian.cpp -fPIC -DPIC -o .libs/fts-backend-xapian.o fts-backend-xapian.cpp:486:14: warning: format string is not a string literal (potentially insecure) [-Wformat-security] i_warning(e.get_msg().c_str()); ^~~ fts-backend-xapian.cpp:486:14: note: treat the string as an argument to avoid this i_warning(e.get_msg().c_str()); ^ "%s", 1 warning generated. Is that something you can look into as well? On Mon, 14 Jan 2019 at 11:43, Joan Moreau wrote: > THank you Odhiambo. I updated accordingly > > > > > > On 2019-01-14 08:07, Odhiambo Washington wrote: > > In your README.md, perhaps "This project intends to provide a > straightforward and simple *procedure *to configure FTS plugin for > Dovecot, leveraging the efforts by the Xapian.org team." is better?? > Also in the part after cloning from git: > > ./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This > /path/to/dovecot is not obvious. Is it the dovecot binary or what??] > > On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot > wrote: > >> Thank you Stephan. >> >> The version here shall be up and running : >> https://github.com/grosjo/fts-xapian >> >> >> >> >> >> On 2019-01-14 00:07, Stephan Bosch wrote: >> >> >> >> Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: >> >> >> I tried to combined it, the "autoreconf" errors are solved >> >> Now, when I type "make install", the lib is not pushed into dovecot >> folder, but somewhere in /usr/local/... >> >> How to adjust this to have it arriving in the proper folder ? >> >> >> Depends on your system. It mostly a matter of setting a proper --prefix >> directory for configure, but other paths are configurable as well. I >> usually check what the official distribution package for Dovecot is doing >> and use that as a basis. >> >> For Debian I use the following configure command: >> >> ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin >> --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ >> --with-gssapi=plugin --with-solr --with-ioloop=best >> --enable-maintainer-mode \ >> --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib >> --localstatedir=/var --mandir=/usr/share/man \ >> --infodir=/usr/share/info >> --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static >> >> Regards, >> >> Stephan >> >> >> On 2019-01-13 21:01, Tuomi, Aki wrote: >> >> You copied your Makefile.am there. Stephan made you a working version, >> can you try that? >> (sorry for dup) >> Aki >> Original message >> From: Joan Moreau >> Date: 13/01/2019 21:39 (GMT+02:00) >> To: Stephan Bosch >> Cc: Aki Tuomi >> Subject: Re: [FTS Xapian] Beta release >> >> I used the skeleton from Aki : https://github.com/grosj
Re: [FTS Xapian] Beta release
Thank you. Here is the stack trace with all debug symbols: (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x7f88b6cc742a in __GI_abort () at abort.c:89 #2 0x7f88b5b240ad in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x7f88b5b22066 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x7f88b5b220b1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x7f88b5b222c9 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x7f88b5b227ec in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #7 0x7f88b5e681b6 in __gnu_cxx::new_allocator, std::allocator > const, OmDocumentTerm> > >::allocate (this=, __n=1) at /usr/include/c++/6/ext/new_allocator.h:104 #8 std::allocator_traits, std::allocator > const, OmDocumentTerm> > > >::allocate (__a=..., __n=1) at /usr/include/c++/6/bits/alloc_traits.h:436 #9 std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, OmDocumentTerm>, std::_Select1st, std::allocator > const, OmDocumentTerm> >, std::less, std::allocator > >, std::allocator, std::allocator > const, OmDocumentTerm> > >::_M_get_node (this=) at /usr/include/c++/6/bits/stl_tree.h:505 #10 std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, OmDocumentTerm>, std::_Select1st, std::allocator > const, OmDocumentTerm> >, std::less, std::allocator > >, std::allocator, std::allocator > const, OmDocumentTerm> > >::_M_create_node, std::allocator >, OmDocumentTerm> >(std::pair, std::allocator >, OmDocumentTerm>&&) (this=) at /usr/include/c++/6/bits/stl_tree.h:559 #11 std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, OmDocumentTerm>, std::_Select1st, std::allocator > const, OmDocumentTerm> >, std::less, std::allocator > >, std::allocator, std::allocator > const, OmDocumentTerm> > >::_Alloc_node::operator(), std::allocator >, OmDocumentTerm> >(std::pair, std::allocator >, OmDocumentTerm>&&) const ( this=, __arg=) at /usr/include/c++/6/bits/stl_tree.h:473 #12 std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, OmDocumentTerm>, std::_Select1st, std::allocator > const, OmDocumentTerm> >, std::less, std::allocator > >, std::allocator, std::allocator > const, OmDocumentTerm> > >::_M_insert_, std::allocator >, OmDocumentTerm>, std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, OmDocumentTerm>, std::_Select1st, std::allocator > const, OmDocumentTerm> >, std::less, std::allocator > >, std::allocator, std::allocator > const, OmDocumentTerm> > >::_Alloc_node>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::pair, std::allocator >, OmDocumentTerm>&&, std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, OmDocumentTerm>, std::_Select1st, std::allocator > const, OmDocumentTerm> >, std::less, std::allocator > >, std::allocator, std::allocator > const, OmDocumentTerm> > >::_Alloc_node&) (__node_gen=..., __v=, __p=0x7f88b78c7010, __x=0x0, this=0x55d404f29c60) at /usr/include/c++/6/bits/stl_tree.h:1535 #13 std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, OmDocumentTerm>, std::_Select1st, std::allocator > const, OmDocumentTerm> >, std::less, std::allocator > >, std::allocator, std::allocator > const, OmDocumentTerm> > >::_M_insert_unique, std::allocator >, OmDocumentTerm> >(std::pair, std::allocator >, OmDocumentTerm>&&) ( this=this@entry=0x55d404f29c60, __v=__v@entry=) at /usr/include/c++/6/bits/stl_tree.h:1894 #14 0x7f88b5e67282 in std::map, std::allocator >, OmDocumentTerm, std::less, std::allocator > >, std::allocator, std::allocator > const, OmDocumentTerm> > >::insert, std::allocator >, OmDocumentTerm>, void>(std::pair, std::allocator >, OmDocumentTerm>&&) (__x=, this=) at /usr/include/c++/6/bits/stl_map.h:740 #15 Xapian::Document::Internal::add_term (this=, tname="XMIDascoding.c", wdfinc=wdfinc@entry=1) at ../api/omdocument.cc:371 #16 0x7f88b5e672f9 in Xapian::Document::add_term (this=this@entry=0x7fff87c611a0, tname="XMIDascoding.c", wdfinc=wdfinc@entry=1) at ../api/omdocument.cc:139 #17 0x7f88b622e9e6 in fts_backend_xapian_index_hdr (dbx=0x55d3f840f9f0, uid=, field=, data=data@entry=0x55d40433af50 "<4f31eb5f-7fcb-478d-91d0-12c0ae75a...@iwascoding.com>MESSAGE-ID", p=, f=) at fts-backend-xapian-functions.cpp:661 #18 0x7f88b62325d6 in fts_backend_xapian_update_build_more (_ctx=0x55d3f827ef80, data=0x55d40433af50 "<4f31eb5f-7fcb-478d-91d0-12c0ae75a...@iwascoding.com>MESSAGE-ID", size=) at fts-backend-xapian.cpp:330 #19 0x7f88b66439a8 in fts_build_data (ctx=ctx@entry=0x7fff87c61820, data=0x55d40433af50 "<4f31eb5f-7fcb-478d-91d0-12c0ae75a...@iwascoding.com>MESSAGE-ID", size=53, last=last@entry=true) at fts-build-mail.c:425 #20 0x7f88b66441f7 in fts_build_unstructured_header (hdr=, hdr=, ctx=0x7fff87c61820) at
Re: [FTS Xapian] Beta release
Op 14-1-2019 om 10:55 schreef Paul Hecker via dovecot: OK, got it (my fault, as always, put the LimitCORE in the wrong line). Here is the stack trace: If you want to get rid of those "??" stack trace elements, you'll need to install debug symbols for the xapian library. It depends on your system how to do that (usually some separate package). Regards, Stephan. On 14. Jan 2019, at 10:33, Joan Moreau via dovecot mailto:dovecot@dovecot.org>> wrote: Difficult to figure out without a coredump + gdb I have also battled quite a lot to make sure dovecot can core dump on my Archlinux servers. I remember that the key point was putting*fs.suid_dumpable=2* in /etc/sysctl.d/ conf files, *LimitCORE=infinity* in /etc/systemd/system/multi-user.target.wants/dovecot.service, and rebooting the server. My own coredumps are on /var/lib/systemd/coredump/
Re: [FTS Xapian] Beta release
OK, got it (my fault, as always, put the LimitCORE in the wrong line). Here is the stack trace:(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x7fe342b0942a in __GI_abort () at abort.c:89
#2 0x7fe3419660ad in __gnu_cxx::__verbose_terminate_handler() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x7fe341964066 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x7fe3419640b1 in std::terminate() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x7fe3419642c9 in __cxa_throw () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x7fe3419647ec in operator new(unsigned long) () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x7fe341ca748d in ?? () from /usr/lib/x86_64-linux-gnu/libxapian.so.30
#8 0x7fe341ca9258 in
Xapian::Document::Internal::add_term(std::__cxx11::basic_string, std::allocator > const&, unsigned int) () from
/usr/lib/x86_64-linux-gnu/libxapian.so.30
#9 0x7fe341ca92f9 in
Xapian::Document::add_term(std::__cxx11::basic_string, std::allocator > const&, unsigned int) () from
/usr/lib/x86_64-linux-gnu/libxapian.so.30
#10 0x7fe3420709e6 in fts_backend_xapian_index_hdr (dbx=0x56307b00,
uid=, field=, data=data@entry=0x56301cf420e0
"MESSAGE-ID", p=, f=)
at fts-backend-xapian-functions.cpp:661
#11 0x7fe3420745d6 in fts_backend_xapian_update_build_more
(_ctx=0x56301117e0a0, data=0x56301cf420e0
"MESSAGE-ID",
size=) at fts-backend-xapian.cpp:330
#12 0x7fe3424859a8 in fts_build_data (ctx=ctx@entry=0x7ffeb9c7d3a0,
data=0x56301cf420e0
"MESSAGE-ID", size=53,
last=last@entry=true) at fts-build-mail.c:425
#13 0x7fe3424861f7 in fts_build_unstructured_header (hdr=,
hdr=, ctx=0x7ffeb9c7d3a0) at fts-build-mail.c:103
#14 fts_build_mail_header (block=0x7ffeb9c7d330, block=0x7ffeb9c7d330,
ctx=0x7ffeb9c7d3a0) at fts-build-mail.c:178
#15 fts_build_mail_real (may_need_retry_r=0x7ffeb9c7d2e3,
retriable_err_msg_r=0x7ffeb9c7d2f0, mail=0x5630111b9a08,
update_ctx=0x56301117e0a0) at fts-build-mail.c:568
#16 fts_build_mail (update_ctx=0x56301117e0a0, mail=mail@entry=0x5630111b9a08)
at fts-build-mail.c:617
#17 0x7fe34248d6c3 in fts_mail_index (_mail=0x5630111b9a08) at
fts-storage.c:538
#18 fts_mail_precache (_mail=0x5630111b9a08) at fts-storage.c:559
#19 0x7fe343237d6a in mail_precache (mail=0x5630111b9a08) at mail.c:424
#20 0x56300ff5a603 in index_mailbox_precache (conn=0x5630110e99e0,
box=0x56303fc8) at master-connection.c:102
#21 index_mailbox (user=, user=, what=, max_recent_msgs=, mailbox=,
conn=0x5630110e99e0) at master-connection.c:205
#22 master_connection_input_line (line=, conn=0x5630110e99e0) at
master-connection.c:247
#23 master_connection_input (conn=0x5630110e99e0) at master-connection.c:287
#24 0x7fe342f5f5d5 in io_loop_call_io (io=0x5630110e9a20) at ioloop.c:698
#25 0x7fe342f60fd9 in io_loop_handler_run_internal
(ioloop=ioloop@entry=0x5630110dec90) at ioloop-epoll.c:221
#26 0x7fe342f5f6e6 in io_loop_handler_run (ioloop=) at
ioloop.c:750
#27 0x7fe342f5f8f8 in io_loop_run (ioloop=0x5630110dec90) at ioloop.c:723
#28 0x7fe342ed3d43 in master_service_run (service=0x5630110deb20,
callback=) at master-service.c:781
#29 0x56300ff59fea in main (argc=, argv=) at
indexer-worker.c:77
(gdb) bt full
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
set = {__val = {0, 51539607553, 0, 140732015298048, 3, 0,
7233170478761014387, 109330211561823, 140732015303136, 140732015302656,
140732015303144, 0, 140732015303160, 0, 0, 0}}
pid =
tid =
#1 0x7fe342b0942a in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x7fe342e6f520
<_IO_2_1_stderr_>, sa_sigaction = 0x7fe342e6f520 <_IO_2_1_stderr_>}, sa_mask =
{__val = {3432, 140614056716544, 140614056719424, 140614056736163,
140614053423970, 14, 1, 10, 0, 94764659026720, 50,
140732015300080, 140614053431513, 140614056736032,
140614053432547, 140614056736032}}, sa_flags = 1100362432, sa_restorer = 0x0}
sigs = {__val = {32, 0 }}
#2 0x7fe3419660ad in __gnu_cxx::__verbose_terminate_handler() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#3 0x7fe341964066 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#4 0x7fe3419640b1 in std::terminate() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#5 0x7fe3419642c9 in __cxa_throw () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#6 0x7fe3419647ec in operator new(unsigned long) () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
No symbol table info available.
#7 0x7fe341ca748d in ?? () from /usr/lib/x86_64-linux-gnu/libxapian.so.30
No symbol table info available.
#8 0x7fe341ca9258 in
Xapian::Document::Internal::add_term(std::__cxx11::basic_string,
Re: [FTS Xapian] Beta release
Difficult to figure out without a coredump + gdb I have also battled quite a lot to make sure dovecot can core dump on my Archlinux servers. I remember that the key point was putting FS.SUID_DUMPABLE=2 in /etc/sysctl.d/ conf files, LIMITCORE=INFINITY in /etc/systemd/system/multi-user.target.wants/dovecot.service, and rebooting the server. My own coredumps are on /var/lib/systemd/coredump/ On 2019-01-14 10:20, Paul Hecker via dovecot wrote: Here it is: Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=6 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Quota grace: root=User quota bytes=214748364 (10%) Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: dict quota: user=p...@iwascoding.com, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, alt= Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: Mailbox opened because: indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c12654d370e Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing Thank you! On 14. Jan 2019, at 10:11, Joan Moreau via dovecot wrote: Can you send the log part that includes the "init" of the plugins (something similar as below) ? WHich version of Xapian are you on ? Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net)<14725>: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes [1] Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net)<14725>: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net)<14725>: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net)<14725>: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e55d6a3092d [2] On 2019-01-14 10:08, Paul Hecker via dovecot wrote: Hi, I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed: doveadm index -u p...@iwascoding.com -q \* Jan 14 09:26:15 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for p...@iwascoding.com Jan 14 09:26:15 mail dovecot: indexer-worker(p...@iwascodi
Re: [FTS Xapian] Beta release
Here it is: Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=6 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Quota grace: root=User quota bytes=214748364 (10%) Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: dict quota: user=p...@iwascoding.com, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, alt= Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: Mailbox opened because: indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c12654d370e Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing Thank you! > On 14. Jan 2019, at 10:11, Joan Moreau via dovecot > wrote: > > Can you send the log part that includes the "init" of the plugins (something > similar as below) ? > > WHich version of Xapian are you on ? > > Jan 14 09:10:04 gjserver dovecot[31082]: > indexer-worker(ad...@grosjo.net)<14725>: > FTS Xapian: Partial=2, Full=20 > DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes > Jan 14 09:10:04 gjserver dovecot[31082]: > indexer-worker(ad...@grosjo.net)<14725>: > FTS Xapian : Mailbox Mail : Last UID=815055 > Jan 14 09:10:04 gjserver dovecot[31082]: > indexer-worker(ad...@grosjo.net)<14725>: > FTS Xapian : Mailbox Mail : Last UID=815055 > Jan 14 09:10:04 gjserver dovecot[31082]: > indexer-worker(ad...@grosjo.net)<14725>: > Opening RW > /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e55d6a3092d > > > > > > > On 2019-01-14 10:08, Paul Hecker via dovecot wrote: > >> Hi, >> >> I installed and tested your version, but the indexer process crashes >> reproducible with the following command after about 2000 messages were >> indexed: >> >> doveadm index -u p...@iwascoding.com <mailto:p...@iwascoding.com> -q \* >> >> Jan 14 09:26:15 mail dovecot: indexer-worker(p...@iwascoding.com >> <mailto:p...@iwascoding.com>)<16777>: Debug: Mailbox >> sent: UID 2038: Opened mail because: fts indexing >> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after >> throwing an instance of 'std::bad_alloc' >> Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): >> std::bad_alloc >> Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, >> discarding 48 requests for p...@iwascoding.com <mailto:p...@iwascoding.com> >> Jan 14 09:26:15 mail dovecot: indexer-worker(p...@iwascoding.com >> <mailto:p...@iwascoding.c
Re: [FTS Xapian] Beta release
Here it is: Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Effective uid=8, gid=8, home=/var/spool/mail/iwascoding/paul Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Quota root: name=User quota backend=dict args=:file:/var/spool/mail/iwascoding/paul/dovecot-quota Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Quota rule: root=User quota mailbox=* bytes=2147483648 messages=6 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Quota grace: root=User quota bytes=214748364 (10%) Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: dict quota: user=p...@iwascoding.com <mailto:user=p...@iwascoding.com>, uri=file:/var/spool/mail/iwascoding/paul/dovecot-quota, noenforcing=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Namespace inbox: type=private, prefix=, sep=/, inbox=yes, hidden=no, list=yes, subscriptions=yes location=mdbox:~/mdbox Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: fs: root=/var/spool/mail/iwascoding/paul/mdbox, index=, indexpvt=, control=, inbox=, alt= Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: FTS Xapian: Partial=2, Full=20 DB_PATH=/var/spool/mail/iwascoding/paul/mdbox/xapian-indexes Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: quota: quota_over_flag check: quota_over_script unset - skipping Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Mailbox sent: Mailbox opened because: indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: FTS Xapian : Mailbox sent : Last UID=0 Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Namespace : Using permissions from /var/spool/mail/iwascoding/paul/mdbox: mode=0700 gid=default Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Mailbox sent: UID 1: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Opening RW /var/spool/mail/iwascoding/paul/mdbox/xapian-indexes/db_9ddfe10d8a8a8a568c12654d370e Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Mailbox sent: UID 2: Opened mail because: fts indexing Jan 14 09:26:08 mail dovecot: indexer-worker(p...@iwascoding.com <mailto:p...@iwascoding.com>)<16777>: Debug: Mailbox sent: UID 3: Opened mail because: fts indexing Thank you! > On 14. Jan 2019, at 10:11, Joan Moreau via dovecot <mailto:dovecot@dovecot.org>> wrote: > > Can you send the log part that includes the "init" of the plugins (something > similar as below) ? > > WHich version of Xapian are you on ? > > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net > <mailto:ad...@grosjo.net>)<14725>: > FTS Xapian: Partial=2, Full=20 > DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes > <http://grosjo.net/admin/xapian-indexes> > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net > <mailto:ad...@grosjo.net>)<14725>: > FTS Xapian : Mailbox Mail : Last UID=815055 > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net > <mailto:ad...@grosjo.net>)<14725>: > FTS Xapian : Mailbox Mail : Last UID=815055 > Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net > <mailto:ad...@grosjo.net>)<14725>: > Opening RW > /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e55d6a3092d > <http://grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e55d6a3092d> > > > > > On 2019-01-14 10:08, Paul Hecker via dovecot wrote: > >> Hi, >> >> I installed and tested your version, but the
Re: [FTS Xapian] Beta release
Can you send the log part that includes the "init" of the plugins (something similar as below) ? WHich version of Xapian are you on ? Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net)<14725>: FTS Xapian: Partial=2, Full=20 DB_PATH=/data/mail/grosjo.net/admin/xapian-indexes Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net)<14725>: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net)<14725>: FTS Xapian : Mailbox Mail : Last UID=815055 Jan 14 09:10:04 gjserver dovecot[31082]: indexer-worker(ad...@grosjo.net)<14725>: Opening RW /data/mail/grosjo.net/admin/xapian-indexes/db_5c935034609bc14c0e55d6a3092d On 2019-01-14 10:08, Paul Hecker via dovecot wrote: Hi, I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed: doveadm index -u p...@iwascoding.com -q \* Jan 14 09:26:15 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for p...@iwascoding.com Jan 14 09:26:15 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before. Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch. Thanks, Paul On 14. Jan 2019, at 07:42, Joan Moreau via dovecot wrote: Thank you Stephan. The version here shall be up and running : https://github.com/grosjo/fts-xapian On 2019-01-14 00:07, Stephan Bosch wrote: Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: I tried to combined it, the "autoreconf" errors are solved Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... How to adjust this to have it arriving in the proper folder ? Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. For Debian I use the following configure command: ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static Regards, Stephan On 2019-01-13 21:01, Tuomi, Aki wrote: You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki Original message From: Joan Moreau Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch Cc: Aki Tuomi Subject: Re: [FTS Xapian] Beta release I used the skeleton from Aki : https://github.com/grosjo/fts-xapian However, when I try to act as a visitor, I reach teh follwoing error: # autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastd
Re: [FTS Xapian] Beta release
Hi, I installed and tested your version, but the indexer process crashes reproducible with the following command after about 2000 messages were indexed: doveadm index -u p...@iwascoding.com -q \* Jan 14 09:26:15 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Debug: Mailbox sent: UID 2038: Opened mail because: fts indexing Jan 14 09:26:15 mail dovecot: indexer-worker: Error: terminate called after throwing an instance of 'std::bad_alloc' Jan 14 09:26:15 mail dovecot: indexer-worker: Error: what(): std::bad_alloc Jan 14 09:26:15 mail dovecot: indexer: Error: Indexer worker disconnected, discarding 48 requests for p...@iwascoding.com Jan 14 09:26:15 mail dovecot: indexer-worker(p...@iwascoding.com)<16777>: Fatal: master: service(indexer-worker): child 16777 killed with signal 6 (core dumps disabled - https://dovecot.org/bugreport.html#coredumps) I tried to delete the message, but this does not help (crashes e.g. after message 2029 or 2044). Other folders with fewer messages were successfully indexed before. Sorry, could not convince dovecot to create core dumps (read the docs, changed /proc/sys/kernel/core_pattern, added LimitCORE=unlimited/infinity, even created /etc/systemd/system/dovecot.service.d/coredump.conf to no avail). Custom Dovecot 2.3.4 on Debian Stretch. Thanks, Paul > On 14. Jan 2019, at 07:42, Joan Moreau via dovecot > wrote: > > Thank you Stephan. > > The version here shall be up and running : > https://github.com/grosjo/fts-xapian > > > > > > > On 2019-01-14 00:07, Stephan Bosch wrote: > >> >> >> Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: >>> >>> I tried to combined it, the "autoreconf" errors are solved >>> >>> Now, when I type "make install", the lib is not pushed into dovecot folder, >>> but somewhere in /usr/local/... >>> >>> How to adjust this to have it arriving in the proper folder ? >>> >> >> Depends on your system. It mostly a matter of setting a proper --prefix >> directory for configure, but other paths are configurable as well. I usually >> check what the official distribution package for Dovecot is doing and use >> that as a basis. >> >> For Debian I use the following configure command: >> >> ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin >> --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ >> --with-gssapi=plugin --with-solr --with-ioloop=best >> --enable-maintainer-mode \ >> --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib >> --localstatedir=/var --mandir=/usr/share/man \ >> --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules >> --disable-rpath --disable-static >> >> Regards, >> >> Stephan >> >>> >>> On 2019-01-13 21:01, Tuomi, Aki wrote: >>> >>>> You copied your Makefile.am there. Stephan made you a working version, can >>>> you try that? >>>> (sorry for dup) >>>> Aki >>>> Original message >>>> From: Joan Moreau >>>> Date: 13/01/2019 21:39 (GMT+02:00) >>>> To: Stephan Bosch >>>> Cc: Aki Tuomi >>>> Subject: Re: [FTS Xapian] Beta release >>>> >>>> I used the skeleton from Aki : https://github.com/grosjo/fts-xapian >>>> >>>> However, when I try to act as a visitor, I reach teh follwoing error: >>>> >>>> # autoreconf -vi >>>> autoreconf: Entering directory `.' >>>> autoreconf: configure.ac: not using Gettext >>>> autoreconf: running: aclocal -I m4 >>>> autoreconf: configure.ac: tracing >>>> autoreconf: running: libtoolize --copy >>>> libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. >>>> libtoolize: copying file './ltmain.sh' >>>> libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. >>>> libtoolize: copying file 'm4/libtool.m4' >>>> libtoolize: copying file 'm4/ltoptions.m4' >>>> libtoolize: copying file 'm4/ltsugar.m4' >>>> libtoolize: copying file 'm4/ltversion.m4' >>>> libtoolize: copying file 'm4/lt~obsolete.m4' >>>> autoreconf: running: /usr/bin/autoconf >>>> autoreconf: running: /usr/bin/autoheader >>>> autoreconf: running: automake --add-missing --copy --no-force >>>> configure.ac:9: installing './compile' >>>> configure.ac:11: installing './config.guess' >>>> configure.ac:11: installing './config.sub' >>>> configure.ac:7: installin
Re: [FTS Xapian] Beta release
THank you Odhiambo. I updated accordingly On 2019-01-14 08:07, Odhiambo Washington wrote: In your README.md, perhaps "This project intends to provide a straightforward and simple PROCEDURE to configure FTS plugin for Dovecot, leveraging the efforts by the Xapian.org team." is better?? Also in the part after cloning from git: ./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This /path/to/dovecot is not obvious. Is it the dovecot binary or what??] On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot wrote: Thank you Stephan. The version here shall be up and running : https://github.com/grosjo/fts-xapian On 2019-01-14 00:07, Stephan Bosch wrote: Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: I tried to combined it, the "autoreconf" errors are solved Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... How to adjust this to have it arriving in the proper folder ? Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. For Debian I use the following configure command: ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static Regards, Stephan On 2019-01-13 21:01, Tuomi, Aki wrote: You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki Original message From: Joan Moreau Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch Cc: Aki Tuomi Subject: Re: [FTS Xapian] Beta release I used the skeleton from Aki : https://github.com/grosjo/fts-xapian However, when I try to act as a visitor, I reach teh follwoing error: # autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac [1]: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac [1]: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9 [2]: installing './compile' configure.ac:11 [3]: installing './config.guess' configure.ac:11 [3]: installing './config.sub' configure.ac:7 [4]: installing './install-sh' configure.ac:7 [4]: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am [5]: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am [5]: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am [5]: to 'configure.ac [1]' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac [1]' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1 On 2019-01-13 20:24, Stephan Bosch wrote: Oh, right, a distribution tarball doesn't include some of the necessary files for your repository like autogen.sh and .gitignore. The attached tarball includes all those and is ready for `git init`. The previous tarball was made with `make distcheck` from this one. Regards, Stephan. Op 13/01/2019 om 20:14 schreef Stephan Bosch: Hi Joan, Op 13/01/2019 om 19:03 schreef Aki Tuomi: Yes, from compiling point of view it is done. Unfortunately what is not done is all the other work involved, such as fixing all the inevitable bugs it has and maintaining it. We do not want, at this moment, take up maintaining and developing yet another FTS plugin as we have plenty of things to do already. I invite you to setup your own repository and provide this plugin from there, being the maintainer of this plugin. We can add a link to your plugin on our FTS page so people can also find it. There are other plugins like this, e.g. https://github.com/st3fan/dovecot-xaps-plugin I turned the code you provided into a separate plugin package. The di
Re: [FTS Xapian] Beta release
In your README.md, perhaps "This project intends to provide a straightforward and simple *procedure *to configure FTS plugin for Dovecot, leveraging the efforts by the Xapian.org team." is better?? Also in the part after cloning from git: ./configure --prefix=/usr --with-dovecot=/path/to/dovecot [ This /path/to/dovecot is not obvious. Is it the dovecot binary or what??] On Mon, 14 Jan 2019 at 09:42, Joan Moreau via dovecot wrote: > Thank you Stephan. > > The version here shall be up and running : > https://github.com/grosjo/fts-xapian > > > > > > On 2019-01-14 00:07, Stephan Bosch wrote: > > > > Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: > > > I tried to combined it, the "autoreconf" errors are solved > > Now, when I type "make install", the lib is not pushed into dovecot > folder, but somewhere in /usr/local/... > > How to adjust this to have it arriving in the proper folder ? > > > Depends on your system. It mostly a matter of setting a proper --prefix > directory for configure, but other paths are configurable as well. I > usually check what the official distribution package for Dovecot is doing > and use that as a basis. > > For Debian I use the following configure command: > > ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin > --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ > --with-gssapi=plugin --with-solr --with-ioloop=best > --enable-maintainer-mode \ > --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib > --localstatedir=/var --mandir=/usr/share/man \ > --infodir=/usr/share/info > --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static > > Regards, > > Stephan > > > On 2019-01-13 21:01, Tuomi, Aki wrote: > > You copied your Makefile.am there. Stephan made you a working version, can > you try that? > (sorry for dup) > Aki > Original message > From: Joan Moreau > Date: 13/01/2019 21:39 (GMT+02:00) > To: Stephan Bosch > Cc: Aki Tuomi > Subject: Re: [FTS Xapian] Beta release > > I used the skeleton from Aki : https://github.com/grosjo/fts-xapian > > However, when I try to act as a visitor, I reach teh follwoing error: > > # autoreconf -vi > autoreconf: Entering directory `.' > autoreconf: configure.ac: not using Gettext > autoreconf: running: aclocal -I m4 > autoreconf: configure.ac: tracing > autoreconf: running: libtoolize --copy > libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. > libtoolize: copying file './ltmain.sh' > libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. > libtoolize: copying file 'm4/libtool.m4' > libtoolize: copying file 'm4/ltoptions.m4' > libtoolize: copying file 'm4/ltsugar.m4' > libtoolize: copying file 'm4/ltversion.m4' > libtoolize: copying file 'm4/lt~obsolete.m4' > autoreconf: running: /usr/bin/autoconf > autoreconf: running: /usr/bin/autoheader > autoreconf: running: automake --add-missing --copy --no-force > configure.ac:9: installing './compile' > configure.ac:11: installing './config.guess' > configure.ac:11: installing './config.sub' > configure.ac:7: installing './install-sh' > configure.ac:7: installing './missing' > src/Makefile.am: installing './depcomp' > /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not > appear in AM_CONDITIONAL > /usr/share/automake-1.16/am/depend2.am: The usual way to define > 'am__fastdepCXX' is to add 'AC_PROG_CXX' > /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run > 'aclocal' and 'autoconf' again > src/Makefile.am: error: C++ source seen but 'CXX' is undefined > src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' > src/Makefile.am: to 'configure.ac' and run 'autoconf' again. > src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no > program or > src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible > typo) > autoreconf: automake failed with exit status: 1 > > > > On 2019-01-13 20:24, Stephan Bosch wrote: > > Oh, right, a distribution tarball doesn't include some of the > necessary files for your repository like autogen.sh and > .gitignore. The attached tarball includes all those and is ready > for `git init`. The previous tarball was made with `make > distcheck` from this one. > > Regards, > > Stephan. > > Op 13/01/2019 om 20:14 schreef Stephan Bosch: > > Hi Joan, > > Op 13/01/2019 om 19:03 schreef Aki Tuomi: > > Yes, from compiling point of view it is done. > > Unfortunately what is not done is all the other work > involved, such as
Re: [FTS Xapian] Beta release
Thank you Stephan. The version here shall be up and running : https://github.com/grosjo/fts-xapian On 2019-01-14 00:07, Stephan Bosch wrote: Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: I tried to combined it, the "autoreconf" errors are solved Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... How to adjust this to have it arriving in the proper folder ? Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. For Debian I use the following configure command: ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static Regards, Stephan On 2019-01-13 21:01, Tuomi, Aki wrote: You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki Original message From: Joan Moreau Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch Cc: Aki Tuomi Subject: Re: [FTS Xapian] Beta release I used the skeleton from Aki : https://github.com/grosjo/fts-xapian However, when I try to act as a visitor, I reach teh follwoing error: # autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1 On 2019-01-13 20:24, Stephan Bosch wrote: Oh, right, a distribution tarball doesn't include some of the necessary files for your repository like autogen.sh and .gitignore. The attached tarball includes all those and is ready for `git init`. The previous tarball was made with `make distcheck` from this one. Regards, Stephan. Op 13/01/2019 om 20:14 schreef Stephan Bosch: Hi Joan, Op 13/01/2019 om 19:03 schreef Aki Tuomi: Yes, from compiling point of view it is done. Unfortunately what is not done is all the other work involved, such as fixing all the inevitable bugs it has and maintaining it. We do not want, at this moment, take up maintaining and developing yet another FTS plugin as we have plenty of things to do already. I invite you to setup your own repository and provide this plugin from there, being the maintainer of this plugin. We can add a link to your plugin on our FTS page so people can also find it. There are other plugins like this, e.g. https://github.com/st3fan/dovecot-xaps-plugin I turned the code you provided into a separate plugin package. The distribution tarball is attached. Notable changes: - Added example copyright headers and COPYING and AUTHORS files. You should modify those to your preference. - Added README and INSTALL files (in markdown using Pandoc). Those need to be amended with details. - Amended the plugin code to display a debug message with the plugin name and version upon plugin load. I advise you to turn this into a git repository and continue from there. I do not recommend releasing this plugin with the -fpermissive flag and the resulting warning as it is now. But I'm assuming this is still a work in pro
Re: [FTS Xapian] Beta release
Op 13/01/2019 om 21:25 schreef Joan Moreau via dovecot: I tried to combined it, the "autoreconf" errors are solved Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/... How to adjust this to have it arriving in the proper folder ? Depends on your system. It mostly a matter of setting a proper --prefix directory for configure, but other paths are configurable as well. I usually check what the official distribution package for Dovecot is doing and use that as a basis. For Debian I use the following configure command: ./configure --with-ldap=plugin --with-ssl=openssl --with-sql=plugin --with-lua=plugin --with-pgsql --with-mysql --with-sqlite \ --with-gssapi=plugin --with-solr --with-ioloop=best --enable-maintainer-mode \ --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --localstatedir=/var --mandir=/usr/share/man \ --infodir=/usr/share/info --with-moduledir=/usr/lib/dovecot/modules --disable-rpath --disable-static Regards, Stephan On 2019-01-13 21:01, Tuomi, Aki wrote: You copied your Makefile.am there. Stephan made you a working version, can you try that? (sorry for dup) Aki Original message From: Joan Moreau Date: 13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch Cc: Aki Tuomi Subject: Re: [FTS Xapian] Beta release I used the skeleton from Aki : https://github.com/grosjo/fts-xapian However, when I try to act as a visitor, I reach teh follwoing error: # autoreconf -vi autoreconf: Entering directory `.' autoreconf: configure.ac: not using Gettext autoreconf: running: aclocal -I m4 autoreconf: configure.ac: tracing autoreconf: running: libtoolize --copy libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'. libtoolize: copying file './ltmain.sh' libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. libtoolize: copying file 'm4/libtool.m4' libtoolize: copying file 'm4/ltoptions.m4' libtoolize: copying file 'm4/ltsugar.m4' libtoolize: copying file 'm4/ltversion.m4' libtoolize: copying file 'm4/lt~obsolete.m4' autoreconf: running: /usr/bin/autoconf autoreconf: running: /usr/bin/autoheader autoreconf: running: automake --add-missing --copy --no-force configure.ac:9: installing './compile' configure.ac:11: installing './config.guess' configure.ac:11: installing './config.sub' configure.ac:7: installing './install-sh' configure.ac:7: installing './missing' src/Makefile.am: installing './depcomp' /usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear in AM_CONDITIONAL /usr/share/automake-1.16/am/depend2.am: The usual way to define 'am__fastdepCXX' is to add 'AC_PROG_CXX' /usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and 'autoconf' again src/Makefile.am: error: C++ source seen but 'CXX' is undefined src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX' src/Makefile.am: to 'configure.ac' and run 'autoconf' again. src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no program or src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo) autoreconf: automake failed with exit status: 1 On 2019-01-13 20:24, Stephan Bosch wrote: Oh, right, a distribution tarball doesn't include some of the necessary files for your repository like autogen.sh and .gitignore. The attached tarball includes all those and is ready for `git init`. The previous tarball was made with `make distcheck` from this one. Regards, Stephan. Op 13/01/2019 om 20:14 schreef Stephan Bosch: Hi Joan, Op 13/01/2019 om 19:03 schreef Aki Tuomi: Yes, from compiling point of view it is done. Unfortunately what is not done is all the other work involved, such as fixing all the inevitable bugs it has and maintaining it. We do not want, at this moment, take up maintaining and developing yet another FTS plugin as we have plenty of things to do already. I invite you to setup your own repository and provide this plugin from there, being the maintainer of this plugin. We can add a link to your plugin on our FTS page so people can also find it. There are other plugins like this, e.g. https://github.com/st3fan/dovecot-xaps-plugin I turned the code you provided into a separate plugin package. The distribution tarball is attached. Notable changes: - Added example copyright headers and COPYING and AUTHORS files. You should modify those to your preference. - Added README and INSTALL files (in markdown using Pandoc). Those need to be amended with details. - Amended the plugin code to display a debug message with the plugin name and version upon plugin load. I advise you to turn this into a git reposito
Re: [FTS Xapian] Beta release
(or wherever dovecot installed its libraries)
On 2019-01-13 21:25, Joan Moreau via dovecot wrote:
I tried to combined it, the "autoreconf" errors are solved
Now, when I type "make install", the lib is not pushed into dovecot folder, but somewhere in /usr/local/...
How to adjust this to have it arriving in the proper folder ?
On 2019-01-13 21:01, Tuomi, Aki wrote:
You copied your Makefile.am there. Stephan made you a working version, can you try that?
(sorry for dup)
Aki
Original message
From: Joan Moreau
Date: 13/01/2019 21:39 (GMT+02:00)
To: Stephan Bosch
Cc: Aki Tuomi
Subject: Re: [FTS Xapian] Beta release
I used the skeleton from Aki : https://github.com/grosjo/fts-xapian
However, when I try to act as a visitor, I reach teh follwoing error:
# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear
in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define
'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and
'autoconf' again
src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no
program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
autoreconf: automake failed with exit status: 1
On 2019-01-13 20:24, Stephan Bosch wrote:
Oh, right, a distribution tarball doesn't include some of the necessary files for your repository like autogen.sh and .gitignore. The attached tarball includes all those and is ready for `git init`. The previous tarball was made with `make distcheck` from this one.
Regards,
Stephan.
Op 13/01/2019 om 20:14 schreef Stephan Bosch: Hi Joan,
Op 13/01/2019 om 19:03 schreef Aki Tuomi: Yes, from compiling point of view it
is done.
Unfortunately what is not done is all the other work involved, such as fixing
all the inevitable bugs it has and maintaining it. We do not want, at this
moment, take up maintaining and developing yet another FTS plugin as we have
plenty of things to do already.
I invite you to setup your own repository and provide this plugin from there,
being the maintainer of this plugin. We can add a link to your plugin on our
FTS page so people can also find it.
There are other plugins like this, e.g. https://github.com/st3fan/dovecot-xaps-plugin
I turned the code you provided into a separate plugin package. The distribution tarball is attached.
Notable changes:
- Added example copyright headers and COPYING and AUTHORS files. You should
modify those to your preference.
- Added README and INSTALL files (in markdown using Pandoc). Those need to be
amended with details.
- Amended the plugin code to display a debug message with the plugin name and
version upon plugin load.
I advise you to turn this into a git repository and continue from there.
I do not recommend releasing this plugin with the -fpermissive flag and the
resulting warning as it is now. But I'm assuming this is still a work in
progress, so that is OK.
Regards,
Stephan.
On 13 January 2019 at 19:52 Joan Moreau wrote:
The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).
I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly
The only remaining point is to push it in hte git (yes, everything is
already done)
On 2019-01-13 18:45, Aki Tuomi wrote:
On 13 January 2019 at 17:05 Joan Moreau via dovecot wrote:
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = ye
Re: [FTS Xapian] Beta release
I tried to combined it, the "autoreconf" errors are solved
Now, when I type "make install", the lib is not pushed into dovecot
folder, but somewhere in /usr/local/...
How to adjust this to have it arriving in the proper folder ?
On 2019-01-13 21:01, Tuomi, Aki wrote:
You copied your Makefile.am there. Stephan made you a working version, can you try that?
(sorry for dup)
Aki
Original message
From: Joan Moreau
Date: 13/01/2019 21:39 (GMT+02:00)
To: Stephan Bosch
Cc: Aki Tuomi
Subject: Re: [FTS Xapian] Beta release
I used the skeleton from Aki : https://github.com/grosjo/fts-xapian
However, when I try to act as a visitor, I reach teh follwoing error:
# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear
in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define
'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and
'autoconf' again
src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no
program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
autoreconf: automake failed with exit status: 1
On 2019-01-13 20:24, Stephan Bosch wrote:
Oh, right, a distribution tarball doesn't include some of the necessary files for your repository like autogen.sh and .gitignore. The attached tarball includes all those and is ready for `git init`. The previous tarball was made with `make distcheck` from this one.
Regards,
Stephan.
Op 13/01/2019 om 20:14 schreef Stephan Bosch: Hi Joan,
Op 13/01/2019 om 19:03 schreef Aki Tuomi: Yes, from compiling point of view it
is done.
Unfortunately what is not done is all the other work involved, such as fixing
all the inevitable bugs it has and maintaining it. We do not want, at this
moment, take up maintaining and developing yet another FTS plugin as we have
plenty of things to do already.
I invite you to setup your own repository and provide this plugin from there,
being the maintainer of this plugin. We can add a link to your plugin on our
FTS page so people can also find it.
There are other plugins like this, e.g. https://github.com/st3fan/dovecot-xaps-plugin
I turned the code you provided into a separate plugin package. The distribution tarball is attached.
Notable changes:
- Added example copyright headers and COPYING and AUTHORS files. You should
modify those to your preference.
- Added README and INSTALL files (in markdown using Pandoc). Those need to be
amended with details.
- Amended the plugin code to display a debug message with the plugin name and
version upon plugin load.
I advise you to turn this into a git repository and continue from there.
I do not recommend releasing this plugin with the -fpermissive flag and the
resulting warning as it is now. But I'm assuming this is still a work in
progress, so that is OK.
Regards,
Stephan.
On 13 January 2019 at 19:52 Joan Moreau wrote:
The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).
I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly
The only remaining point is to push it in hte git (yes, everything is
already done)
On 2019-01-13 18:45, Aki Tuomi wrote:
On 13 January 2019 at 17:05 Joan Moreau via dovecot wrote:
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20
This is installed on my production server (>120
Re: [FTS Xapian] Beta release
You copied your Makefile.am there. Stephan made you a working version, can you
try that?
(sorry for dup)
Aki
Original message From: Joan Moreau Date:
13/01/2019 21:39 (GMT+02:00) To: Stephan Bosch Cc: Aki
Tuomi Subject: Re: [FTS Xapian] Beta release
I used the skeleton from Aki : https://github.com/grosjo/fts-xapian
However, when I try to act as a visitor, I reach teh follwoing error:
# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/usr/share/automake-1.16/am/depend2.am: error: am__fastdepCXX does not appear
in AM_CONDITIONAL
/usr/share/automake-1.16/am/depend2.am: The usual way to define
'am__fastdepCXX' is to add 'AC_PROG_CXX'
/usr/share/automake-1.16/am/depend2.am: to 'configure.ac' and run 'aclocal' and
'autoconf' again
src/Makefile.am: error: C++ source seen but 'CXX' is undefined
src/Makefile.am: The usual way to define 'CXX' is to add 'AC_PROG_CXX'
src/Makefile.am: to 'configure.ac' and run 'autoconf' again.
src/Makefile.am:11: warning: variable 'NOPLUGIN_LDFLAGS' is defined but no
program or
src/Makefile.am:11: library has 'NOPLUGIN' as canonical name (possible typo)
autoreconf: automake failed with exit status: 1
On 2019-01-13 20:24, Stephan Bosch wrote:
Oh, right, a distribution tarball doesn't include some of the necessary files
for your repository like autogen.sh and .gitignore. The attached tarball
includes all those and is ready for `git init`. The previous tarball was made
with `make distcheck` from this one.
Regards,
Stephan.
Op 13/01/2019 om 20:14 schreef Stephan Bosch:
Hi Joan,
Op 13/01/2019 om 19:03 schreef Aki Tuomi:
Yes, from compiling point of view it is done.
Unfortunately what is not done is all the other work involved, such as fixing
all the inevitable bugs it has and maintaining it. We do not want, at this
moment, take up maintaining and developing yet another FTS plugin as we have
plenty of things to do already.
I invite you to setup your own repository and provide this plugin from there,
being the maintainer of this plugin. We can add a link to your plugin on our
FTS page so people can also find it.
There are other plugins like this, e.g.
https://github.com/st3fan/dovecot-xaps-plugin
I turned the code you provided into a separate plugin package. The distribution
tarball is attached.
Notable changes:
- Added example copyright headers and COPYING and AUTHORS files. You should
modify those to your preference.
- Added README and INSTALL files (in markdown using Pandoc). Those need to be
amended with details.
- Amended the plugin code to display a debug message with the plugin name and
version upon plugin load.
I advise you to turn this into a git repository and continue from there.
I do not recommend releasing this plugin with the -fpermissive flag and the
resulting warning as it is now. But I'm assuming this is still a work in
progress, so that is OK.
Regards,
Stephan.
On 13 January 2019 at 19:52 Joan Moreau wrote:
The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).
I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly
The only remaining point is to push it in hte git (yes, everything is
already done)
On 2019-01-13 18:45, Aki Tuomi wrote:
On 13 January 2019 at 17:05 Joan Moreau via dovecot wrote:
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20
This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.
I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.
Thanks
JM
Hi!
I st
Re: [FTS Xapian] Beta release
I get the following error:
# autoreconf -vi
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal -I m4
autoreconf: configure.ac: tracing
autoreconf: running: libtoolize --copy
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
autoreconf: running: /usr/bin/autoconf
autoreconf: running: /usr/bin/autoheader
autoreconf: running: automake --add-missing --copy --no-force
configure.ac:9: installing './compile'
configure.ac:11: installing './config.guess'
configure.ac:11: installing './config.sub'
configure.ac:7: installing './install-sh'
configure.ac:7: installing './missing'
src/Makefile.am: installing './depcomp'
/USR/SHARE/AUTOMAKE-1.16/AM/DEPEND2.AM: ERROR: AM__FASTDEPCXX DOES NOT
APPEAR IN AM_CONDITIONAL
/USR/SHARE/AUTOMAKE-1.16/AM/DEPEND2.AM: THE USUAL WAY TO DEFINE
'AM__FASTDEPCXX' IS TO ADD 'AC_PROG_CXX'
/USR/SHARE/AUTOMAKE-1.16/AM/DEPEND2.AM: TO 'CONFIGURE.AC' AND RUN
'ACLOCAL' AND 'AUTOCONF' AGAIN
SRC/MAKEFILE.AM: ERROR: C++ SOURCE SEEN BUT 'CXX' IS UNDEFINED
SRC/MAKEFILE.AM: THE USUAL WAY TO DEFINE 'CXX' IS TO ADD 'AC_PROG_CXX'
SRC/MAKEFILE.AM: TO 'CONFIGURE.AC' AND RUN 'AUTOCONF' AGAIN.
SRC/MAKEFILE.AM:11: WARNING: VARIABLE 'NOPLUGIN_LDFLAGS' IS DEFINED BUT
NO PROGRAM OR
SRC/MAKEFILE.AM:11: LIBRARY HAS 'NOPLUGIN' AS CANONICAL NAME (POSSIBLE
TYPO)
AUTORECONF: AUTOMAKE FAILED WITH EXIT STATUS: 1
On 2019-01-13 20:32, Joan Moreau via dovecot wrote:
Please kindly check https://github.com/grosjo/fts-xapian
On 2019-01-13 20:11, Aki Tuomi wrote:
If you had looked at what I sent, you'd seen it's quite different from what you sent.
Anyways, put the contents of skeleton.tar.gz and
./src/plugins/fts-xapian/fts-xapian-plugin.h
./src/plugins/fts-xapian/fts-backend-xapian.cpp
./src/plugins/fts-xapian/Makefile.am
./src/plugins/fts-xapian/fts-backend-xapian-functions.cpp
./src/plugins/fts-xapian/fts-xapian-plugin.c
into src/ directory (included in skeleton.tar.gz), then put those into git.
You can compile it with
autoreconf -vi
./configure --with-dovecot=/path/to/dovecot
make
Aki
On 13 January 2019 at 21:03 Joan Moreau wrote:
THis is already what I send earlier (see : dovecot-xapian-1.0b2.tar.gz
[1 [1]] )
What I would need is the files so one can download (git) it, and type
some command (make ?) to compile it and place it in the right forlder
(/usr/lib/dovecot/ or whatever is configured in the installed dovecot,
which may differ from distribution to distribution)
On 2019-01-13 19:47, Aki Tuomi wrote:
You need the fts-xapian.c mostly. All the rest comes from automake (unless you
decide to use cmake or something else).
I have attached you a skeleton plugin, which should work against dovecot master.
If you experience problems with the skeleton, let us know.
Aki
On 13 January 2019 at 20:23 Joan Moreau wrote:
Ok for having a link on the FTS page.
PLease clarify what files are necessary to package it as a separate
package
On 2019-01-13 19:03, Aki Tuomi wrote:
Yes, from compiling point of view it is done.
Unfortunately what is not done is all the other work involved, such as fixing
all the inevitable bugs it has and maintaining it. We do not want, at this
moment, take up maintaining and developing yet another FTS plugin as we have
plenty of things to do already.
I invite you to setup your own repository and provide this plugin from there,
being the maintainer of this plugin. We can add a link to your plugin on our
FTS page so people can also find it.
There are other plugins like this, e.g.
https://github.com/st3fan/dovecot-xaps-plugin
Aki
On 13 January 2019 at 19:52 Joan Moreau wrote:
The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).
I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly
The only remaining point is to push it in hte git (yes, everything is
already done)
On 2019-01-13 18:45, Aki Tuomi wrote:
On 13 January 2019 at 17:05 Joan Moreau via dovecot wrote:
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20
This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.
I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users a
Re: [FTS Xapian] Beta release
Please kindly check https://github.com/grosjo/fts-xapian
On 2019-01-13 20:11, Aki Tuomi wrote:
If you had looked at what I sent, you'd seen it's quite different from what you
sent.
Anyways, put the contents of skeleton.tar.gz and
./src/plugins/fts-xapian/fts-xapian-plugin.h
./src/plugins/fts-xapian/fts-backend-xapian.cpp
./src/plugins/fts-xapian/Makefile.am
./src/plugins/fts-xapian/fts-backend-xapian-functions.cpp
./src/plugins/fts-xapian/fts-xapian-plugin.c
into src/ directory (included in skeleton.tar.gz), then put those into git.
You can compile it with
autoreconf -vi
./configure --with-dovecot=/path/to/dovecot
make
Aki
On 13 January 2019 at 21:03 Joan Moreau wrote:
THis is already what I send earlier (see : dovecot-xapian-1.0b2.tar.gz
[1 [1]] )
What I would need is the files so one can download (git) it, and type
some command (make ?) to compile it and place it in the right forlder
(/usr/lib/dovecot/ or whatever is configured in the installed dovecot,
which may differ from distribution to distribution)
On 2019-01-13 19:47, Aki Tuomi wrote:
You need the fts-xapian.c mostly. All the rest comes from automake (unless you
decide to use cmake or something else).
I have attached you a skeleton plugin, which should work against dovecot master.
If you experience problems with the skeleton, let us know.
Aki
On 13 January 2019 at 20:23 Joan Moreau wrote:
Ok for having a link on the FTS page.
PLease clarify what files are necessary to package it as a separate
package
On 2019-01-13 19:03, Aki Tuomi wrote:
Yes, from compiling point of view it is done.
Unfortunately what is not done is all the other work involved, such as fixing
all the inevitable bugs it has and maintaining it. We do not want, at this
moment, take up maintaining and developing yet another FTS plugin as we have
plenty of things to do already.
I invite you to setup your own repository and provide this plugin from there,
being the maintainer of this plugin. We can add a link to your plugin on our
FTS page so people can also find it.
There are other plugins like this, e.g.
https://github.com/st3fan/dovecot-xaps-plugin
Aki
On 13 January 2019 at 19:52 Joan Moreau wrote:
The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).
I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly
The only remaining point is to push it in hte git (yes, everything is
already done)
On 2019-01-13 18:45, Aki Tuomi wrote:
On 13 January 2019 at 17:05 Joan Moreau via dovecot wrote:
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20
This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.
I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.
Thanks
JM
Hi!
I still recommend you setup a, say, github repository for your plugin. We are
not able to currently include your work in dovecot core as it is more work than
just pushing the code into the repo. Maybe it can be included in the future.
If you want, I can help you in setting up the required configuration scripts
and such to make it possible to compile it as plugin.
Then anyone can download it and install it for their dovecot, even if dovecot
itself has been installed from packages, and also makes it possible for package
maintainers to consider including it in distributions.
Aki
Links:
--
[1] https://grosjo.net/dovecot-xapian-1.0b2.tar.gz
Links:
--
[1] https://grosjo.net/dovecot-xapian-1.0b2.tar.gz
Re: [FTS Xapian] Beta release
If you had looked at what I sent, you'd seen it's quite different from what you
sent.
Anyways, put the contents of skeleton.tar.gz and
./src/plugins/fts-xapian/fts-xapian-plugin.h
./src/plugins/fts-xapian/fts-backend-xapian.cpp
./src/plugins/fts-xapian/Makefile.am
./src/plugins/fts-xapian/fts-backend-xapian-functions.cpp
./src/plugins/fts-xapian/fts-xapian-plugin.c
into src/ directory (included in skeleton.tar.gz), then put those into git.
You can compile it with
autoreconf -vi
./configure --with-dovecot=/path/to/dovecot
make
Aki
> On 13 January 2019 at 21:03 Joan Moreau wrote:
>
>
> THis is already what I send earlier (see : dovecot-xapian-1.0b2.tar.gz
> [1] )
>
> What I would need is the files so one can download (git) it, and type
> some command (make ?) to compile it and place it in the right forlder
> (/usr/lib/dovecot/ or whatever is configured in the installed dovecot,
> which may differ from distribution to distribution)
>
> On 2019-01-13 19:47, Aki Tuomi wrote:
>
> > You need the fts-xapian.c mostly. All the rest comes from automake (unless
> > you decide to use cmake or something else).
> >
> > I have attached you a skeleton plugin, which should work against dovecot
> > master.
> >
> > If you experience problems with the skeleton, let us know.
> >
> > Aki
> >
> > On 13 January 2019 at 20:23 Joan Moreau wrote:
> >
> > Ok for having a link on the FTS page.
> >
> > PLease clarify what files are necessary to package it as a separate
> > package
> >
> > On 2019-01-13 19:03, Aki Tuomi wrote:
> >
> > Yes, from compiling point of view it is done.
> >
> > Unfortunately what is not done is all the other work involved, such as
> > fixing all the inevitable bugs it has and maintaining it. We do not want,
> > at this moment, take up maintaining and developing yet another FTS plugin
> > as we have plenty of things to do already.
> >
> > I invite you to setup your own repository and provide this plugin from
> > there, being the maintainer of this plugin. We can add a link to your
> > plugin on our FTS page so people can also find it.
> >
> > There are other plugins like this, e.g.
> > https://github.com/st3fan/dovecot-xaps-plugin
> >
> > Aki
> >
> > On 13 January 2019 at 19:52 Joan Moreau wrote:
> >
> > The only point here of this fts-xapian is to get rid of solr (because it
> > is just a nightmare to setup) and squat (because it is considere
> > obsolete).
> >
> > I already sent the changed in configure.ac, makefile.am, etc.. in order
> > to include it in the dovecot, and it compiles properly
> >
> > The only remaining point is to push it in hte git (yes, everything is
> > already done)
> >
> > On 2019-01-13 18:45, Aki Tuomi wrote:
> >
> > On 13 January 2019 at 17:05 Joan Moreau via dovecot
> > wrote:
> >
> > Hi
> >
> > Please find attached the beta release of FTS Xapian, with the objective
> > to replace fts_squat that is being deprecated.
> >
> > Configuration is exactly the same as for fts_squat:
> >
> > plugin {
> >
> > plugin = fts fts_xapian (...)
> > fts = xapian
> > fts_autoindex = yes
> > fts_enforced = yes
> > fts_xapian = partial=2 full=20
> >
> > This is installed on my production server (>120Gb of mailboxes), and I
> > will observe it during the coming days.
> >
> > I will definitely appreciate that this is added in the core git of
> > docevot, in order to have a versionning of it, to remove squat and let
> > basic users able to avoid Solr alternative as much as possible.
> >
> > Thanks
> >
> > JM
> > Hi!
> >
> > I still recommend you setup a, say, github repository for your plugin. We
> > are not able to currently include your work in dovecot core as it is more
> > work than just pushing the code into the repo. Maybe it can be included in
> > the future.
> >
> > If you want, I can help you in setting up the required configuration
> > scripts and such to make it possible to compile it as plugin.
> >
> > Then anyone can download it and install it for their dovecot, even if
> > dovecot itself has been installed from packages, and also makes it possible
> > for package maintainers to consider including it in distributions.
> >
> > Aki
>
>
> Links:
> --
> [1] https://grosjo.net/dovecot-xapian-1.0b2.tar.gz
Re: [FTS Xapian] Beta release
THis is already what I send earlier (see : dovecot-xapian-1.0b2.tar.gz
[1] )
What I would need is the files so one can download (git) it, and type
some command (make ?) to compile it and place it in the right forlder
(/usr/lib/dovecot/ or whatever is configured in the installed dovecot,
which may differ from distribution to distribution)
On 2019-01-13 19:47, Aki Tuomi wrote:
You need the fts-xapian.c mostly. All the rest comes from automake (unless you
decide to use cmake or something else).
I have attached you a skeleton plugin, which should work against dovecot master.
If you experience problems with the skeleton, let us know.
Aki
On 13 January 2019 at 20:23 Joan Moreau wrote:
Ok for having a link on the FTS page.
PLease clarify what files are necessary to package it as a separate
package
On 2019-01-13 19:03, Aki Tuomi wrote:
Yes, from compiling point of view it is done.
Unfortunately what is not done is all the other work involved, such as fixing
all the inevitable bugs it has and maintaining it. We do not want, at this
moment, take up maintaining and developing yet another FTS plugin as we have
plenty of things to do already.
I invite you to setup your own repository and provide this plugin from there,
being the maintainer of this plugin. We can add a link to your plugin on our
FTS page so people can also find it.
There are other plugins like this, e.g.
https://github.com/st3fan/dovecot-xaps-plugin
Aki
On 13 January 2019 at 19:52 Joan Moreau wrote:
The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).
I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly
The only remaining point is to push it in hte git (yes, everything is
already done)
On 2019-01-13 18:45, Aki Tuomi wrote:
On 13 January 2019 at 17:05 Joan Moreau via dovecot wrote:
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20
This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.
I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.
Thanks
JM
Hi!
I still recommend you setup a, say, github repository for your plugin. We are
not able to currently include your work in dovecot core as it is more work than
just pushing the code into the repo. Maybe it can be included in the future.
If you want, I can help you in setting up the required configuration scripts
and such to make it possible to compile it as plugin.
Then anyone can download it and install it for their dovecot, even if dovecot
itself has been installed from packages, and also makes it possible for package
maintainers to consider including it in distributions.
Aki
Links:
--
[1] https://grosjo.net/dovecot-xapian-1.0b2.tar.gz
Re: [FTS Xapian] Beta release
You need the fts-xapian.c mostly. All the rest comes from automake (unless you
decide to use cmake or something else).
I have attached you a skeleton plugin, which should work against dovecot master.
If you experience problems with the skeleton, let us know.
Aki
> On 13 January 2019 at 20:23 Joan Moreau wrote:
>
>
> Ok for having a link on the FTS page.
>
> PLease clarify what files are necessary to package it as a separate
> package
>
> On 2019-01-13 19:03, Aki Tuomi wrote:
>
> > Yes, from compiling point of view it is done.
> >
> > Unfortunately what is not done is all the other work involved, such as
> > fixing all the inevitable bugs it has and maintaining it. We do not want,
> > at this moment, take up maintaining and developing yet another FTS plugin
> > as we have plenty of things to do already.
> >
> > I invite you to setup your own repository and provide this plugin from
> > there, being the maintainer of this plugin. We can add a link to your
> > plugin on our FTS page so people can also find it.
> >
> > There are other plugins like this, e.g.
> > https://github.com/st3fan/dovecot-xaps-plugin
> >
> > Aki
> >
> > On 13 January 2019 at 19:52 Joan Moreau wrote:
> >
> > The only point here of this fts-xapian is to get rid of solr (because it
> > is just a nightmare to setup) and squat (because it is considere
> > obsolete).
> >
> > I already sent the changed in configure.ac, makefile.am, etc.. in order
> > to include it in the dovecot, and it compiles properly
> >
> > The only remaining point is to push it in hte git (yes, everything is
> > already done)
> >
> > On 2019-01-13 18:45, Aki Tuomi wrote:
> >
> > On 13 January 2019 at 17:05 Joan Moreau via dovecot
> > wrote:
> >
> > Hi
> >
> > Please find attached the beta release of FTS Xapian, with the objective
> > to replace fts_squat that is being deprecated.
> >
> > Configuration is exactly the same as for fts_squat:
> >
> > plugin {
> >
> > plugin = fts fts_xapian (...)
> > fts = xapian
> > fts_autoindex = yes
> > fts_enforced = yes
> > fts_xapian = partial=2 full=20
> >
> > This is installed on my production server (>120Gb of mailboxes), and I
> > will observe it during the coming days.
> >
> > I will definitely appreciate that this is added in the core git of
> > docevot, in order to have a versionning of it, to remove squat and let
> > basic users able to avoid Solr alternative as much as possible.
> >
> > Thanks
> >
> > JM
> > Hi!
> >
> > I still recommend you setup a, say, github repository for your plugin. We
> > are not able to currently include your work in dovecot core as it is more
> > work than just pushing the code into the repo. Maybe it can be included in
> > the future.
> >
> > If you want, I can help you in setting up the required configuration
> > scripts and such to make it possible to compile it as plugin.
> >
> > Then anyone can download it and install it for their dovecot, even if
> > dovecot itself has been installed from packages, and also makes it possible
> > for package maintainers to consider including it in distributions.
> >
> > Aki
skeleton.tar.gz
Description: application/gzip
signature.asc
Description: PGP signature
Re: [FTS Xapian] Beta release
Ok for having a link on the FTS page.
PLease clarify what files are necessary to package it as a separate
package
On 2019-01-13 19:03, Aki Tuomi wrote:
Yes, from compiling point of view it is done.
Unfortunately what is not done is all the other work involved, such as fixing
all the inevitable bugs it has and maintaining it. We do not want, at this
moment, take up maintaining and developing yet another FTS plugin as we have
plenty of things to do already.
I invite you to setup your own repository and provide this plugin from there,
being the maintainer of this plugin. We can add a link to your plugin on our
FTS page so people can also find it.
There are other plugins like this, e.g.
https://github.com/st3fan/dovecot-xaps-plugin
Aki
On 13 January 2019 at 19:52 Joan Moreau wrote:
The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).
I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly
The only remaining point is to push it in hte git (yes, everything is
already done)
On 2019-01-13 18:45, Aki Tuomi wrote:
On 13 January 2019 at 17:05 Joan Moreau via dovecot wrote:
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20
This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.
I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.
Thanks
JM
Hi!
I still recommend you setup a, say, github repository for your plugin. We are
not able to currently include your work in dovecot core as it is more work than
just pushing the code into the repo. Maybe it can be included in the future.
If you want, I can help you in setting up the required configuration scripts
and such to make it possible to compile it as plugin.
Then anyone can download it and install it for their dovecot, even if dovecot
itself has been installed from packages, and also makes it possible for package
maintainers to consider including it in distributions.
Aki
Re: [FTS Xapian] Beta release
Yes, from compiling point of view it is done.
Unfortunately what is not done is all the other work involved, such as fixing
all the inevitable bugs it has and maintaining it. We do not want, at this
moment, take up maintaining and developing yet another FTS plugin as we have
plenty of things to do already.
I invite you to setup your own repository and provide this plugin from there,
being the maintainer of this plugin. We can add a link to your plugin on our
FTS page so people can also find it.
There are other plugins like this, e.g.
https://github.com/st3fan/dovecot-xaps-plugin
Aki
> On 13 January 2019 at 19:52 Joan Moreau wrote:
>
>
> The only point here of this fts-xapian is to get rid of solr (because it
> is just a nightmare to setup) and squat (because it is considere
> obsolete).
>
> I already sent the changed in configure.ac, makefile.am, etc.. in order
> to include it in the dovecot, and it compiles properly
>
> The only remaining point is to push it in hte git (yes, everything is
> already done)
>
> On 2019-01-13 18:45, Aki Tuomi wrote:
>
> >> On 13 January 2019 at 17:05 Joan Moreau via dovecot
> >> wrote:
> >>
> >> Hi
> >>
> >> Please find attached the beta release of FTS Xapian, with the objective
> >> to replace fts_squat that is being deprecated.
> >>
> >> Configuration is exactly the same as for fts_squat:
> >>
> >> plugin {
> >>
> >> plugin = fts fts_xapian (...)
> >> fts = xapian
> >> fts_autoindex = yes
> >> fts_enforced = yes
> >> fts_xapian = partial=2 full=20
> >>
> >> This is installed on my production server (>120Gb of mailboxes), and I
> >> will observe it during the coming days.
> >>
> >> I will definitely appreciate that this is added in the core git of
> >> docevot, in order to have a versionning of it, to remove squat and let
> >> basic users able to avoid Solr alternative as much as possible.
> >>
> >> Thanks
> >>
> >> JM
> >
> > Hi!
> >
> > I still recommend you setup a, say, github repository for your plugin. We
> > are not able to currently include your work in dovecot core as it is more
> > work than just pushing the code into the repo. Maybe it can be included in
> > the future.
> >
> > If you want, I can help you in setting up the required configuration
> > scripts and such to make it possible to compile it as plugin.
> >
> > Then anyone can download it and install it for their dovecot, even if
> > dovecot itself has been installed from packages, and also makes it possible
> > for package maintainers to consider including it in distributions.
> >
> > Aki
Re: [FTS Xapian] Beta release
The only point here of this fts-xapian is to get rid of solr (because it
is just a nightmare to setup) and squat (because it is considere
obsolete).
I already sent the changed in configure.ac, makefile.am, etc.. in order
to include it in the dovecot, and it compiles properly
The only remaining point is to push it in hte git (yes, everything is
already done)
On 2019-01-13 18:45, Aki Tuomi wrote:
On 13 January 2019 at 17:05 Joan Moreau via dovecot wrote:
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20
This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.
I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.
Thanks
JM
Hi!
I still recommend you setup a, say, github repository for your plugin. We are
not able to currently include your work in dovecot core as it is more work than
just pushing the code into the repo. Maybe it can be included in the future.
If you want, I can help you in setting up the required configuration scripts
and such to make it possible to compile it as plugin.
Then anyone can download it and install it for their dovecot, even if dovecot
itself has been installed from packages, and also makes it possible for package
maintainers to consider including it in distributions.
Aki
Re: [FTS Xapian] Beta release
> On 13 January 2019 at 17:05 Joan Moreau via dovecot
> wrote:
>
>
> Hi
>
> Please find attached the beta release of FTS Xapian, with the objective
> to replace fts_squat that is being deprecated.
>
> Configuration is exactly the same as for fts_squat:
>
> plugin {
>
> plugin = fts fts_xapian (...)
> fts = xapian
> fts_autoindex = yes
> fts_enforced = yes
> fts_xapian = partial=2 full=20
>
> This is installed on my production server (>120Gb of mailboxes), and I
> will observe it during the coming days.
>
> I will definitely appreciate that this is added in the core git of
> docevot, in order to have a versionning of it, to remove squat and let
> basic users able to avoid Solr alternative as much as possible.
>
> Thanks
>
> JM
Hi!
I still recommend you setup a, say, github repository for your plugin. We are
not able to currently include your work in dovecot core as it is more work than
just pushing the code into the repo. Maybe it can be included in the future.
If you want, I can help you in setting up the required configuration scripts
and such to make it possible to compile it as plugin.
Then anyone can download it and install it for their dovecot, even if dovecot
itself has been installed from packages, and also makes it possible for package
maintainers to consider including it in distributions.
Aki
[FTS Xapian] Beta release
Hi
Please find attached the beta release of FTS Xapian, with the objective
to replace fts_squat that is being deprecated.
Configuration is exactly the same as for fts_squat:
plugin {
plugin = fts fts_xapian (...)
fts = xapian
fts_autoindex = yes
fts_enforced = yes
fts_xapian = partial=2 full=20
This is installed on my production server (>120Gb of mailboxes), and I
will observe it during the coming days.
I will definitely appreciate that this is added in the core git of
docevot, in order to have a versionning of it, to remove squat and let
basic users able to avoid Solr alternative as much as possible.
Thanks
JM
dovecot-xapian.tar.gz
Description: GNU Zip compressed data
Re: [FTS Xapian] Status & Questions
Sorted this out. sorry for noise On 2019-01-12 11:39, Joan Moreau wrote: The change of "Extern C" suggested by Timo actually solved the situation Now, further question : I put a "i_warning" at each of my functions, and I see in the log : Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_alloc Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_init Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_xapian init (3,25) Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_get_last_uid Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Error: Mailbox XXX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2019-01-12 10:33:27] Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_deinit Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_unset_box Jan 12 10:33:27 lda(j...@grosjo.net)<31367>: Warning: fts_backend_xapian_deinit The error comes from the fact taht the "get_last_uid" is called without selecting a mailbox first, and therefore there is of course no uid. How come ? On 2019-01-12 10:50, Aki Tuomi wrote: Did you remember to load fts first? mail_plugins =$mail_plugins fts fts_xapian Aki On 12 January 2019 at 10:37 Joan Moreau via dovecot < dovecot@dovecot.org> wrote: STATUS - Alpha code is written and compiling now. (attached) - I would like to start testing. However, there is an error when starting dovecot (git) : Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x7f25f75e) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x7fe3a51e2000) Your help very welcome PENDING QUESTIONS 1 - WHat does represent "subargs" in mail_search_args 2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large) 3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ? Thank you --- Aki Tuomi
Re: [FTS Xapian] Status & Questions
The change of "Extern C" suggested by Timo actually solved the situation Now, further question : I put a "i_warning" at each of my functions, and I see in the log : Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_alloc Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_init Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_xapian init (3,25) Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_get_last_uid Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Error: Mailbox XXX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2019-01-12 10:33:27] Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_deinit Jan 12 10:33:27 indexer-worker(j...@grosjo.net)<30970>: Warning: fts_backend_xapian_unset_box Jan 12 10:33:27 lda(j...@grosjo.net)<31367>: Warning: fts_backend_xapian_deinit The error comes from the fact taht the "get_last_uid" is called without selecting a mailbox first, and therefore there is of course no uid. How come ? On 2019-01-12 10:50, Aki Tuomi wrote: Did you remember to load fts first? mail_plugins =$mail_plugins fts fts_xapian Aki On 12 January 2019 at 10:37 Joan Moreau via dovecot < dovecot@dovecot.org> wrote: STATUS - Alpha code is written and compiling now. (attached) - I would like to start testing. However, there is an error when starting dovecot (git) : Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x7f25f75e) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x7fe3a51e2000) Your help very welcome PENDING QUESTIONS 1 - WHat does represent "subargs" in mail_search_args 2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large) 3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ? Thank you --- Aki Tuomi
Re: [FTS Xapian] Status & Questions
Did you remember to load fts first? mail_plugins =$mail_plugins fts fts_xapian Aki On 12 January 2019 at 10:37 Joan Moreau via dovecot < dovecot@dovecot.org> wrote: STATUS - Alpha code is written and compiling now. (attached) - I would like to start testing. However, there is an error when starting dovecot (git) : Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x7f25f75e) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x7fe3a51e2000) Your help very welcome PENDING QUESTIONS 1 - WHat does represent "subargs" in mail_search_args 2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large) 3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ? Thank you --- Aki Tuomi
[FTS Xapian] Status & Questions
STATUS - Alpha code is written and compiling now. (attached) - I would like to start testing. However, there is an error when starting dovecot (git) : Error: Couldn't load required plugin /usr/lib/dovecot/lib21_fts_xapian_plugin.so: dlopen() failed: /usr/lib/dovecot/lib21_fts_xapian_plugin.so: undefined symbol: _Z30fts_backend_default_can_lookupP11fts_backendPK15mail_search_arg ldd shows that fts lib is properly linked: # ldd /usr/lib/dovecot/lib21_fts_xapian_plugin.so (...) lib20_fts_plugin.so => /usr/lib/dovecot/lib20_fts_plugin.so (0x7f25f75e) (...) libxapian.so.30 => /usr/lib/libxapian.so.30 (0x7fe3a51e2000) Your help very welcome PENDING QUESTIONS 1 - WHat does represent "subargs" in mail_search_args 2 - for rescan : who is responsible for passing again the new email ? Is the Dovecot core sending again all the emails to index ? or the fts shall somehow access the mailbox and read all emails ? Wouldn't just be saying "delete all index and get_last_uid is now 0" the easy way ? or the fts must process all emails (and block the current thread as a mailbx maybe quite large) 3 - for get_last_uid : this uncertainity is very unclear. "If there is a gap, then indexer first indexes all the missing" -> this mean at a certain point, indexer maybe rebuilding a previous email, so *last* uid is something different than max. And how indexer does know whther there is a gap wihtout callong the fts backend (whch it does not as there are no function for that) ? Thank you dovecot-xapian.tar.gz Description: GNU Zip compressed data
