Re: Duplicate plugins - FTS Xapian
See below. > On 08/31/2021 5:35 AM Aki Tuomi wrote: > > > On 31/08/2021 13:59 Tom Hendrikx wrote: > > > > On 31-08-2021 12:01, Aki Tuomi wrote: > > > > > >> On 31/08/2021 10:56 Felix Zielcke wrote: > > >> > > >> Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: > > >>> > > On 31/08/2021 00:11 Joan Moreau wrote: > > > > Hi > > There seems to be 2 plugins doing the same thins > > - https://github.com/slusarz/dovecot-fts-flatcurve/ > > - https://github.com/grosjo/fts-xapian/ (mine) ... > > Isn't there double work here ? FYI, Joan also reached out to me on the Github project page, so you can see my response to this question there: https://github.com/slusarz/dovecot-fts-flatcurve/issues/5 Short answer: I see the plugins as "different". Yes, they both use the same underlying indexing technology (Xapian), but the design goals, features, and implementations are unique for each project. Thus, I don't see them as duplicate work. > > >> Is there somewhere a direct comparison of them? > > >> I currenty use fts-xapian from Joan without problems. > > >> But what would be the advantages of fts-flatcurve over fts-xapian? I was asked a similar question previously. See https://github.com/slusarz/dovecot-fts-flatcurve/issues/4#issuecomment-902425597 > > > fts_flatcurve does only full word searching, although you can use > > > fts_filters and fts_tokenizers settings to affect stemming and other > > > matching to make it work with plurals and such. Aki's knowledge is a bit dated - fts_flatcurve now supports both prefix matching (this is what Xapian provides by default) AND full substring matching (this is what is technically required by the IMAP RFC). Although the current fts_flatcurve default is for full substring matching, it is debatable whether that is the correct choice. Users tend to think of search in terms of "prefix" matching - that's how Google works - and substring matching requires significant additional storage since Xapian doesn't support it natively. > > I still think it's weird to see that Open-Xchange starts a FTS Xapian > > plugin with mostly the same basic functionality that is already > > available in an existing plugin maintained by someone in the community As Aki noted, Open-Xchange didn't "start" anything. fts_flatcurve is entirely a project I have developed on my personal time. As a Dovecot CE user, I had the same complaint that most people have - in current versions of Dovecot, there is no easy-to-use FTS option that works out-of-the-box. Solr is the only search driver still actively supported, and it requires significant additional resources and configuration to setup. Unfortunately, we don't have the resources within Open-Xchange to create a CE-only search option as our customers generally use our proprietary, object storage aware FTS system that is part of OX Dovecot Pro. Without commercial demand, it is difficult to justify paid work on features that do not produce revenue in return. (fts_flatcurve does use Dovecot core features that are maintained for the Pro FTS system, like stemming, so we can take advantage of trickle down benefits from the commercial code maintenance.) My goal was to give back to the community by providing an easy-to-use, resource sensitive, no-maintenance option to address this current product limitation. Hopefully, fts_flatcurve will fit the requirements for most smaller installations that run Dovecot that don't require the carrier/enterprise grade FTS solutions large customers demand. > > Especially if that happens without any (apparent) communication with the > > existing plugin developer to find out whether fixing the issues that > > slusarz/Open-Xchange seem to have with the existing plugin, can be fixed. There were fundamental disagreements about design goals, and Joan (or anybody, really) would likely not have appreciated someone coming in and demanding that the entire architecture and design of the project needed to be changed based on my personal opinions. This is open source. Everybody should be free to participate as they wish, as long as you are not negatively affecting someone else. I don't think that has happened here. > > Combining forces just seems a better way to spend scarce development > > resources than building something similar (but different) without any > > communication. I disagree, but that's ok. People can do that. michael
Re: Duplicate plugins - FTS Xapian
> On 31/08/2021 13:59 Tom Hendrikx wrote: > > > On 31-08-2021 12:01, Aki Tuomi wrote: > > > >> On 31/08/2021 10:56 Felix Zielcke wrote: > >> > >> > >> Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: > >>> > On 31/08/2021 00:11 Joan Moreau wrote: > > > Hi > There seems to be 2 plugins doing the same thins > - https://github.com/slusarz/dovecot-fts-flatcurve/ > - https://github.com/grosjo/fts-xapian/ (mine) > Both are in the doc of dovecot > https://doc.dovecot.org/configuration_manual/fts/ > > I am currently working hard to push it to RPM package, and plugin > is already approved by ArchLinux and Debian > > Isn't there double work here ? > Thanks > JM > >>> > >>> If you look closer, you can see they are not exactly duplicates. > >>> Flatcurve works differently than your plugin. > >>> > >>> Aki > >> > >> Is there somewhere a direct comparison of them? > >> I currenty use fts-xapian from Joan without problems. > >> But what would be the advantages of fts-flatcurve over fts-xapian? > > > > fts_flatcurve does only full word searching, although you can use > > fts_filters and fts_tokenizers settings to affect stemming and other > > matching to make it work with plurals and such. > > > > Both plugins have their merits. > > I still think it's weird to see that Open-Xchange starts a FTS Xapian > plugin with mostly the same basic functionality that is already > available in an existing plugin maintained by someone in the community > Especially if that happens without any (apparent) communication with the > existing plugin developer to find out whether fixing the issues that > slusarz/Open-Xchange seem to have with the existing plugin, can be fixed. > > Combining forces just seems a better way to spend scarce development > resources than building something similar (but different) without any > communication. > > (Note: I don't use any of these plugins). > > My 2 cents, > > Tom Just for clarity, Open-Xchange has not written any xapian plugin whatsoever. Aki
Re: Duplicate plugins - FTS Xapian
On 31-08-2021 12:01, Aki Tuomi wrote: On 31/08/2021 10:56 Felix Zielcke wrote: Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: On 31/08/2021 00:11 Joan Moreau wrote: Hi There seems to be 2 plugins doing the same thins - https://github.com/slusarz/dovecot-fts-flatcurve/ - https://github.com/grosjo/fts-xapian/ (mine) Both are in the doc of dovecot https://doc.dovecot.org/configuration_manual/fts/ I am currently working hard to push it to RPM package, and plugin is already approved by ArchLinux and Debian Isn't there double work here ? Thanks JM If you look closer, you can see they are not exactly duplicates. Flatcurve works differently than your plugin. Aki Is there somewhere a direct comparison of them? I currenty use fts-xapian from Joan without problems. But what would be the advantages of fts-flatcurve over fts-xapian? fts_flatcurve does only full word searching, although you can use fts_filters and fts_tokenizers settings to affect stemming and other matching to make it work with plurals and such. Both plugins have their merits. I still think it's weird to see that Open-Xchange starts a FTS Xapian plugin with mostly the same basic functionality that is already available in an existing plugin maintained by someone in the community Especially if that happens without any (apparent) communication with the existing plugin developer to find out whether fixing the issues that slusarz/Open-Xchange seem to have with the existing plugin, can be fixed. Combining forces just seems a better way to spend scarce development resources than building something similar (but different) without any communication. (Note: I don't use any of these plugins). My 2 cents, Tom
Re: indexer crash
Op 31-08-2021 om 11:17 schreef Timo Sirainen: The attached patch should fix it? It does, thanks! Onno OpenPGP_signature Description: OpenPGP digital signature
Re: Duplicate plugins - FTS Xapian
> On 31/08/2021 10:56 Felix Zielcke wrote: > > > Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: > > > > > On 31/08/2021 00:11 Joan Moreau wrote: > > > > > > > > > Hi > > > There seems to be 2 plugins doing the same thins > > > - https://github.com/slusarz/dovecot-fts-flatcurve/ > > > - https://github.com/grosjo/fts-xapian/ (mine) > > > Both are in the doc of dovecot > > > https://doc.dovecot.org/configuration_manual/fts/ > > > > > > I am currently working hard to push it to RPM package, and plugin > > > is already approved by ArchLinux and Debian > > > > > > Isn't there double work here ? > > > Thanks > > > JM > > > > If you look closer, you can see they are not exactly duplicates. > > Flatcurve works differently than your plugin. > > > > Aki > > Is there somewhere a direct comparison of them? > I currenty use fts-xapian from Joan without problems. > But what would be the advantages of fts-flatcurve over fts-xapian? fts_flatcurve does only full word searching, although you can use fts_filters and fts_tokenizers settings to affect stemming and other matching to make it work with plurals and such. Both plugins have their merits. Aki
Re: Dovecot Director: Preferred backend server
>From dovecot director point of view it does not make any difference if your >imap clusters have only one server each. mail_hosts takes IP addresses, DNS >names whatever, and you can assign tag per host if you want. And yes, if you have only one server for a tag, then your clients are denied access. Aki > On 31/08/2021 12:28 Steven Varco wrote: > > > Hi Aki > > Thanks for pointing out the tag feature which sound really interesting in > the first place. > > However, if I understand the documentation correctly: > > With tags you can use a single director ring to serve multiple backend > > clusters. Each backend cluster is assigned a tag name, which can be > > anything > > you want. By default everything has an empty tag. A passdb lookup can > > return "director_tag" field containing the wanted tag name. If there aren't > > any > > backend servers with the wanted tag, it's treated the same as if there > > aren't any backend servers available (= wait for 30 secs for a backend and > > then return temporary failure). > > As of my understanding, this only helps if there are multiple IMAP _clusters_ > in the doveadm ring. > In my case I have only one cluster (with two IMAP _servers_) and would want > to go to a specific server, failing over to another if that is unavailable. > Now if I have the following scenario: > > # Director Server > (DEV) root@lb01 [~] # doveadm director status > mail server ip tag vhosts state state changed users > mx01.example.com mx01 100up- 0 > mx02.example.com mx02 100up- 1 > > # IMAP Server > (DEV) root@mx01 [~] # doveadm user 't...@example.com' > field value > uid 1025 > gid 12 > home /srv/mail/example.com/test > mail maildir:~/Maildir > maildir example.com/test/ > mail_home /srv/mail/example.com/test > quota_rule*:storage=20480 > sieve_dir /srv/mail/example.com/test/sieve > director_tag mx01 > > Than user 't...@example.com‘ would go to the backend host mx01.example.com, > BUT, if mx01.example.com goes down, it would probably fail, because user > 't...@example.com‘ wants tag „mx01“, which is now down and the only server > with that tag? > > > By the way, I did a quick live test and it does not even seem to work, when > both hosts are up, failing with the log entry on the dovecot server: > Aug 31 11:11:11 lb01 dovecot: director: Error: director: User > t...@example.com host lookup failed: Timeout because no hosts - queued for 30 > secs (Ring synced for 385 secs, hash=1561836376) > > Do see what I’m missing out here? > Using dovecot 2.2.36 (1f10bfa63) on both the directror and IMAP backend. > > thanks, > Steven > > -- > https://steven.varco.ch/ > > > Am 30.08.2021 um 19:20 schrieb Aki Tuomi : > > > > > >> On 30/08/2021 19:09 Steven Varco wrote: > >> > >> > >> Hi All > >> > >> I have a dovecot cluster with directror and two IMAP Servers behind. > >> > >> Since they are in geographical different locations I would like to have > >> users to go to a specific IMAP backend server (if both are up) and only > >> switch to the other if one goes down (failover). > >> > >> As to my current knowledge the PassDB extra field „host=„ is not suitable > >> in this case as it would never route the client to a different backend, > >> even if the „user specific backend“ would be down. > >> > >> Is their a way in dovecot to achive this? As far as I searched the > >> documentation I could not find any information on this so far. > >> > >> If not, it may also help if I could get certain users to „initially" go to > >> a specific backend (since director usually routes a client/user to the > >> same backend server it initially connects) and therefore it would be > >> interesting to know how dovecot director chooses wether a user goes to > >> server1 or server2? > >> And if a client already gets to server2, how to bring it „back“ to server1? > >> > >> thanks in advance, > >> Steven > >> > >> -- > >> https://steven.varco.ch/ > >> https://www.tech-island.com/ > > > > > > Hi! > > > > Use dovecot director tag feature. You can match users with tag= to a > > specific backend@tag. > > > > Aki
Re: Dovecot Director: Preferred backend server
Hi Aki Thanks for pointing out the tag feature which sound really interesting in the first place. However, if I understand the documentation correctly: > With tags you can use a single director ring to serve multiple backend > clusters. Each backend cluster is assigned a tag name, which can be anything > you want. By default everything has an empty tag. A passdb lookup can return > "director_tag" field containing the wanted tag name. If there aren't any > backend servers with the wanted tag, it's treated the same as if there aren't > any backend servers available (= wait for 30 secs for a backend and > then return temporary failure). As of my understanding, this only helps if there are multiple IMAP _clusters_ in the doveadm ring. In my case I have only one cluster (with two IMAP _servers_) and would want to go to a specific server, failing over to another if that is unavailable. Now if I have the following scenario: # Director Server (DEV) root@lb01 [~] # doveadm director status mail server ip tag vhosts state state changed users mx01.example.com mx01 100up- 0 mx02.example.com mx02 100up- 1 # IMAP Server (DEV) root@mx01 [~] # doveadm user 't...@example.com' field value uid 1025 gid 12 home/srv/mail/example.com/test mailmaildir:~/Maildir maildir example.com/test/ mail_home /srv/mail/example.com/test quota_rule *:storage=20480 sieve_dir /srv/mail/example.com/test/sieve director_tagmx01 Than user 't...@example.com‘ would go to the backend host mx01.example.com, BUT, if mx01.example.com goes down, it would probably fail, because user 't...@example.com‘ wants tag „mx01“, which is now down and the only server with that tag? By the way, I did a quick live test and it does not even seem to work, when both hosts are up, failing with the log entry on the dovecot server: Aug 31 11:11:11 lb01 dovecot: director: Error: director: User t...@example.com host lookup failed: Timeout because no hosts - queued for 30 secs (Ring synced for 385 secs, hash=1561836376) Do see what I’m missing out here? Using dovecot 2.2.36 (1f10bfa63) on both the directror and IMAP backend. thanks, Steven -- https://steven.varco.ch/ > Am 30.08.2021 um 19:20 schrieb Aki Tuomi : > > >> On 30/08/2021 19:09 Steven Varco wrote: >> >> >> Hi All >> >> I have a dovecot cluster with directror and two IMAP Servers behind. >> >> Since they are in geographical different locations I would like to have >> users to go to a specific IMAP backend server (if both are up) and only >> switch to the other if one goes down (failover). >> >> As to my current knowledge the PassDB extra field „host=„ is not suitable in >> this case as it would never route the client to a different backend, even if >> the „user specific backend“ would be down. >> >> Is their a way in dovecot to achive this? As far as I searched the >> documentation I could not find any information on this so far. >> >> If not, it may also help if I could get certain users to „initially" go to a >> specific backend (since director usually routes a client/user to the same >> backend server it initially connects) and therefore it would be interesting >> to know how dovecot director chooses wether a user goes to server1 or >> server2? >> And if a client already gets to server2, how to bring it „back“ to server1? >> >> thanks in advance, >> Steven >> >> -- >> https://steven.varco.ch/ >> https://www.tech-island.com/ > > > Hi! > > Use dovecot director tag feature. You can match users with tag= to a specific > backend@tag. > > Aki
Re: indexer crash
On 29. Aug 2021, at 0.42, Onno Molenkamp wrote: > > Hi, > > Because I was getting segfaults in libc in calloc calls by the indexer > process of Dovecot 2.3.16, I ran indexer with valgrind to look for possible > memory corruption. It did indeed find that indexer was accessing memory that > had already been freed, I've attached the log file. > > I've then tried running Dovecot with the fix in commit > 97367ef61784a364ccf615cb816be6492329050d, but that didn't make any > difference. Reverting (just the code in src/indexer) to the last revision > before the recent rewrite to use connection.c > (d76761698a06bab60a741c6794252fddec4292ef) did: valgrind doesn't log any > errors anymore, and indexer has stopped crashing. The attached patch should fix it? 3879.patch Description: Binary data
Re: Duplicate plugins - FTS Xapian
Am Dienstag, dem 31.08.2021 um 10:33 +0300 schrieb Aki Tuomi: > > > On 31/08/2021 00:11 Joan Moreau wrote: > > > > > > Hi > > There seems to be 2 plugins doing the same thins > > - https://github.com/slusarz/dovecot-fts-flatcurve/ > > - https://github.com/grosjo/fts-xapian/ (mine) > > Both are in the doc of dovecot > > https://doc.dovecot.org/configuration_manual/fts/ > > > > I am currently working hard to push it to RPM package, and plugin > > is already approved by ArchLinux and Debian > > > > Isn't there double work here ? > > Thanks > > JM > > If you look closer, you can see they are not exactly duplicates. > Flatcurve works differently than your plugin. > > Aki Is there somewhere a direct comparison of them? I currenty use fts-xapian from Joan without problems. But what would be the advantages of fts-flatcurve over fts-xapian?
Re: Duplicate plugins - FTS Xapian
> On 31/08/2021 00:11 Joan Moreau wrote: > > > Hi > There seems to be 2 plugins doing the same thins > - https://github.com/slusarz/dovecot-fts-flatcurve/ > - https://github.com/grosjo/fts-xapian/ (mine) > Both are in the doc of dovecot > https://doc.dovecot.org/configuration_manual/fts/ > > I am currently working hard to push it to RPM package, and plugin is already > approved by ArchLinux and Debian > > Isn't there double work here ? > Thanks > JM If you look closer, you can see they are not exactly duplicates. Flatcurve works differently than your plugin. Aki
