Re: Cyrus IMAP 3.2.0 released
On Thu, May 7, 2020, at 11:51 AM, Marco wrote: > 1) The Rename.rename_inbox fails for this syslog error: Probably ellie knows more for this one? > 2) The SearchFuzzy.striphtml_alternative fails for the following reason: > 3) The SearchFuzzy.striphtml_rfc822 fails for the following reason: These fail because they test a feature on the development branch. I have fixed that on the cassandane master branch, so they should not fail for you anymore. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: squatter segfaulting
On Wed, Feb 5, 2020, at 9:36 PM, Heiler Bemerguy wrote: > Is this normal ? Absolutely not. Which Cyrus version are you running? Are you using the Xapian search backend? Would you be able to run this with a debug-enabled build and look at the core dump? Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Database upgrade and Xapian version dependency
On Mon, Jan 27, 2020, at 9:51 AM, Egoitz Aurrekoetxea via Info-cyrus wrote: > Just for having it slightly clearer… When you upgrade the Cyrus version and > the version you are upgrading to is a too close one… for instance from 3.0.8 > to 3.0.13 and you see the Cyrus version is the same for users mail folders, > 13 in both… is it needed to launch (or recommended for some reason) the final > upgrade commands : > > reconstruct -V max > ctl_conversationsdb -b -r > quota -f I'm not sure. Perhaps @ellie could answer that? > By the way, does exist any kind of Xapian needed version for the last 3.0.13 > version?. I’m running Xapian 1.4.9, it’s pretty new… One feature that's missing in Xapian 1.4 is improved support for Chinese and Japanese snippet generation. If you don't need that, you should be fine with 1.4. Otherwise I suggest to use either Xapian upstream master, or our cyruslibs copy at https://github.com/cyrusimap/cyruslibs Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Create ics in another calendar (internal server error)
On Tue, Jan 28, 2020, at 10:22 AM, Zorg wrote: > put every time i this to add event (thunderbird, evolution, curl ) i > have this in the log "HTTP/1.1 500 Internal Server Error" (error=The > server encountered an internal error.) Do you see any indicative error message in the log file (e.g. syslog)? Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 3.0.8 running :) and feedback of the upgrade
On Wed, Feb 13, 2019, at 2:24 PM, Egoitz Aurrekoetxea wrote: > Or.. if using in imapd.conf "search_fuzzy_always: 1" isn't it?. Yes, that will instruct imapd to always use FUZZY search for IMAP SEARCH commands [1]. If you use JMAP, it always use fuzzy search (and hence the Xapian backend). [1]https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/imapd.c#L6059 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 3.0.8 running :) and feedback of the upgrade
Hi Egoitz, On Wed, Feb 13, 2019, at 1:10 PM, Egoitz Aurrekoetxea wrote: > Xapian version is xapian-core-1.4.9_1 (FreeBSD port). Could I see by > some debbuging log Xapian is working in the searchs in order to know, > that can't be faster and that all is ok?. At the moment I have tree > tiers. But I'm just with the default one in almost all mailboxes If you use FUZZY search in your search expression then it must go through Xapian. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian searches of the body of an email
Hi, On Mon, Jan 7, 2019, at 7:32 PM, Egoitz Aurrekoetxea wrote: > So, if I run Squatter in the master in rolling mode... then I assume > there's no need to launch manually squatter command in the master... > isn't it?. If you indexed all messages in the mailbox before, then there shouldn't be a need to manually run squatter afterwards. The rolling squatter should pick up and index all newly created messages. > I'm planning to upgrade some 2.3 running machines to either 2.4 or > 3.0 and you know... is a big responsability... am doing tons of > sandbox testing... is it known (as Michael stated) not to be working > traditional squatter in 3.0 if you don't want to use now the Xapian > engine?. Sorry, I have no experience with the upgrades from version 2. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian searches of the body of an email
Hi Egon, Yes, the slave should index in conversations.db automatically AFAIK. You should run squatter in rolling mode on the master, too. BTW: in 2014, Bron wrote a blog post about the search setup at FastMail: https://fastmail.blog/2014/12/01/email-search-system/It’s quite technical, but should give you a good idea at how it’s set up for fast indexing and search Cheers, Robert On Mon, Jan 7, 2019, at 5:54 PM, Egoitz Aurrekoetxea wrote: > Hi Robert! > > Thank you so much for helping us (mainly which is the one boring the > list with questions :) although I promise I've checked the doc before > asking :) :) ).> > When you have a master/slave config... in the slave one, when running > Squatter in rolling mode... does it update the conversations db too?. > By the way, Squatter in rolling mode only makes sense in slave > machines isn't it?.> > Many thanks! > > --- > > sarenet > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnológico. Edificio 103 > 48170 Zamudio (Bizkaia) > ego...@sarenet.es > www.sarenet.es > > Antes de imprimir este correo electrónico piense si es necesario > hacerlo.> > El 07-01-2019 16:42, Robert Stepanek escribió: >> Hi, >> >> Sebastian is right: >> >> On Mon, Jan 7, 2019, at 3:57 PM, Sebastian Hagedorn wrote: >>> >>> squatter is nowadays a bit of a misnomer, because it uses >>> whatever index>>> you have configured. In cyrus 2.4, squatter would always >>> create >>> a SQUAT>>> index. When you run squatter with Xapian, it will build the >>> index, >>> but for>>> the index to actually work, you also need the conversationsdb. >> >> conversations.db is indeed a misnomer now. The database was only used >> to keep track of mail threads (hence the name), but its role >> expanded. One of the indexes it stores is the SHA1 hashes of every >> message, and separate hashes for each of that message MIME parts. >> Such a hash is named the GUID, and for each GUID we store a list of >> all mailbox:UID[bodypart] pairs where this content occurs in.>> >> For search, we keep track of the indexed messages by GUID, so we can >> avoid reindexing duplicate mails. To return a search result, we can >> now map that GUID back to its mailbox:message pairs. That's why we >> need conversations.db for search.>> >> I can't help with upgrading from 2.4, unfortunately, but if you re- >> index your mailboxes once in conversations.db, you should be all set.>> >> Cheers, >> Robert >> >> >> Cyrus Home Page: http://www.cyrusimap.org/ >> List Archives/Info: >> http://lists.andrew.cmu.edu/pipermail/info-cyrus/>> To Unsubscribe: >> https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian searches of the body of an email
On Mon, Jan 7, 2019, at 4:13 PM, Elías Halldór Ágústsson wrote: > Regarding indexing and searching in body of emails; what if the body > text is encoded in base64 or quoted-printable? It won't yield any > unencoded search strings, or what? If the MIME body part is of type text, then base64 and QP-encoded bodies will get decoded for the search index. And they will get decoded before presenting the search result in snippets. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian searches of the body of an email
Hi, Sebastian is right: On Mon, Jan 7, 2019, at 3:57 PM, Sebastian Hagedorn wrote: > > squatter is nowadays a bit of a misnomer, because it uses > whatever index> you have configured. In cyrus 2.4, squatter would always > create a > SQUAT> index. When you run squatter with Xapian, it will build the > index, but for> the index to actually work, you also need the > conversationsdb. conversations.db is indeed a misnomer now. The database was only used to keep track of mail threads (hence the name), but its role expanded. One of the indexes it stores is the SHA1 hashes of every message, and separate hashes for each of that message MIME parts. Such a hash is named the GUID, and for each GUID we store a list of all mailbox:UID[bodypart] pairs where this content occurs in. For search, we keep track of the indexed messages by GUID, so we can avoid reindexing duplicate mails. To return a search result, we can now map that GUID back to its mailbox:message pairs. That's why we need conversations.db for search. I can't help with upgrading from 2.4, unfortunately, but if you re-index your mailboxes once in conversations.db, you should be all set. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian searches in Cyrus 3.0
Hello, On Thu, Jan 3, 2019, at 3:55 PM, Egoitz Aurrekoetxea wrote: > I was planning to perform mailboxes squattering in rolling mode. Have > you had some kind of expience on searching with Xapian and IMAP > protocol?. Perhaps is more designed for JMAP?. Or perhaps, using it > with IMAP is not so advantageous?. Both IMAP and JMAP search share the same core search API in Cyrus, so all search features available in JMAP can also be used using IMAP. The key is to use FUZZY search for text search, and this can be an advantage especially for non-English search over the other search backends. The complete list of attributes to search is in this C file: https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/search_expr.c#L2235 Any attribute that is marked SEA_FUZZABLE will get stored in Xapian and is available for stem-based search. I'm not sure if there's better documentation on the non-standard fields, but if you have any questions feel free to ask! Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Debugging fuzzy searches
Does any of the message returned by a regular SEARCH contain the verbatim word "Jahren"? The issue you are describing might be related to stemming. Cheers, Robert On Fri, May 25, 2018, at 15:59, Sebastian Hagedorn wrote: > Hi, > > now that I know that in current releases of Cyrus 3 is only used for fuzzy > searches, I wanted to experiment with them, but something isn't right: > > C SEARCH BODY "Jahren" > * SEARCH 20 77 211 224 255 264 331 351 368 383 404 436 438 439 484 551 568 > 596 738 754 757 760 761 773 780 810 840 850 892 918 939 946 973 991 1000 > 1005 1042 1094 1105 1118 1119 1151 1159 1223 1272 1280 1299 1311 1318 1337 > 1349 1352 1366 1375 1378 1385 1417 1422 1423 1431 1436 1445 1446 1459 1460 > 1465 1469 1475 1476 1481 1485 1489 1490 1493 1494 1501 1503 1510 1514 1524 > 1551 1557 1559 1561 1567 1573 1575 1577 1578 1607 1611 1614 1621 1622 1624 > 1629 1633 1637 1638 1643 1647 1648 1649 1653 1673 1676 1677 1682 1683 1686 > 1687 1688 1706 1708 1711 1712 1715 1721 1722 1724 1725 1726 1731 1735 1748 > 1749 1753 1756 1759 1760 1761 1762 1766 1779 1780 1803 1805 1806 1807 1809 > 1810 1812 1903 1904 > C OK Completed (144 msgs in 0.650 secs) > D SEARCH FUZZY BODY "Jahren" > * SEARCH > D OK Completed (0 msgs in 0.000 secs) > > The xapian index for the user exists, and I can see in strace that the > imapd process accesses it. What is going on? This is 3.0.6 ... > -- > .:.Sebastian Hagedorn - Weyertal 121 (Gebäude 133), Zimmer 2.02.:. > .:.Regionales Rechenzentrum (RRZK).:. >.:.Universität zu Köln / Cologne University - ✆ +49-221-470-89578.:. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > Email had 1 attachment: > + Attachment1.2 > 1k (application/pgp-signature) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian/Cyrus/Thunderbird
Hi Albert, On Fri, May 18, 2018, at 22:39, Albert Shih wrote: > After turn on telemetry it's seem Thunderbird just do a « classic » search. > > So that's mean when I do search from Thunderbird it's very slow, and I > build the xapian index for almost nothing because almost 90% of my users > use Thunderbird as MUA. > > Do you want I create a issue on github ? Yes, please create an issue for this on Github so we can track this. I don't know why it's implemented like it currently is, and I haven't assessed what's involved to change this. If you find time, please feel free to join our weekly Cyrus developer hangout, next meeting is on Monday, May 28 at 11am UTC where I will bring this up: https://plus.google.com/hangouts/_/g4xnqjjb5zvomzeb4kqvja3fz4a The hangout is open for anyone interested in Cyrus development, and we always look to forward to meet new people :) Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian/Cyrus/Thunderbird
On Fri, May 11, 2018, at 14:25, Sebastian Hagedorn wrote: > --On 11. Mai 2018 um 13:32:29 +0200 Robert Stepanek <r...@fastmailteam.com> > wrote: > > For non-FUZZY text SEARCH, Cyrus attempts to match the string on its own > > [1]. > > That sounds strange to me, because Cyrus 2.4 and earlier don't support > FUZZY, and there the SQUAT index was used, if present. Only messages that > were added after the last squatter run were searched directly. Why would > that have changed? Right, it hasn't. SQUAT is still the backend for non-FUZZY text search. > I think if Xapian only does fuzzy, some searches may be *slower* than using > a SQUAT index. That seems counterintuitive, at the least. Do you have > internal search benchmarks? Without metrics it's hard to say if any of this > actually matters ... I don't know of any benchmarks for this. It hasn't popped up as a performance issue. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian/Cyrus/Thunderbird
On Fri, May 11, 2018, at 13:04, Sebastian Hagedorn wrote: > For my understanding: does that mean the Xapian index is only used for > clients that support RFC 6203? If that is the case, how are "traditional" > IMAP searches handled? For non-FUZZY text SEARCH, Cyrus attempts to match the string on its own [1]. Off the top of my head, I don't think Cyrus falls back to the search-engine index for large corpus text matches. Using FUZZY always is the better choice, but if there's a real-world issue, please let me know. It might be possible to switch to Xapian also for non-FUZZY search. Cheers, Robert [1] https://github.com/cyrusimap/cyrus-imapd/blob/master/imap/search_expr.c#L2077 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian/Cyrus/Thunderbird
Hi,
On Wed, May 9, 2018, at 09:45, Albert Shih wrote:
> So I get
>
> "search": {
> "squat": true,
> "sphinx": false,
> "xapian": true,
> "xapian_flavor": "vanilla"
> },
>
> I don't know if vanilla are a correct value of xapian_flavor, I would say
> yes because vanilla are ... a flavor...and a flavor of xapian.
That's OK. It means that you are using the upstream version of Xapian, rather
than the Xapian fork of the Cyrus project. You are missing out on a few
features that didn't get pulled into the upstream release, yet: namely improved
Chinese/Japanese search, and improved snippet generation. Both are optional.
The cyruslibs fork is at https://github.com/cyrusimap/xapian
The complete set of forked libraries, including a build script for convenience,
is located at https://github.com/cyrusimap/cyruslibs
> search_index_headers: no
The rest of your config looks fine, but you might want to change
search_index_headers to yes, in case you are doing a lot of searches on the
From:, To:, etc headers. If disabled, searching these headers still will work,
but will be slower.
> Just one question, are the xapian handle all search ? Including body search ?
Actually, Xapian mainly is useful for the body search, e.g. searching text
within a somewhat large corpus.
Cheers,
Robert
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Xapian/Cyrus/Thunderbird
Hi,
On Fri, May 4, 2018, at 14:48, Albert Shih wrote:
> Is they are any way to make a test to know if xapian work or not ?
Cyrus comes with two binaries that will tell you its build and runtime
configuration:
E.g. running
$ /usr/cyrus/sbin/cyr_buildinfo
on my development server yields a JSON-formatted output about Cyrus build
config. Look for something like
"search": {
"squat": true,
"xapian": true,
"xapian_flavor": "cyruslibs"
},
to check if Xapian is enabled in the build (presumably, yes, since you have
*.glass databases in your directories).
Next, cyr_info will tell you, if xapian got enabled indeed at runtime:
$ /usr/cyrus/sbin/cyr_info -C /imapd.conf conf-all | grep
search
should yield something like
[...]
search_engine: xapian
Next, make sure that the IMAP commands submitted by your clients are using
SEARCH FUZZY. You might want to inspect the IMAP telemetry to check this.
If all these checks passed, there shouldn't be any reason why Cyrus should not
use Xapian during search. One might want to enable verbose logging for search
then, but unfortunately, that's currently not a runtime-option.
Cheers,
Robert
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
To Unsubscribe:
https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Size of partition.
On Fri, Apr 27, 2018, at 15:48, Albert Shih wrote: > I activated squatter but put the index not on ssd. The index created by > squatter are (in my case) about 5-8% of the size of the mailbox. > > But I got lot of « conversation » index. What's the purpose of those « > conversations » ? conversations.db initially was used to keep an index of email conversations (e.g. the threaded views typically found in most email readers nowadays). But today it also helps for most use-cases where we need to keep track of messages across mailboxes (e.g. if you have two copies of an email in two mailboxes, you'll see that reflected in conversations.db). It confusingly is still named conversations.db for historic reasons. But for any recent (v3) Cyrus IMAP installation, including Xapian and JMAP, it's a very critical piece in the puzzle. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus + Xapian
Hi Albert, we just had a chat on the issue you are reporting in the regular Cyrus call: there have been a few fixes for squatter since 3.0.5. Would you be able to compile from the current development snapshot [1]? If not, 3.0.6 will be released soon, which will also contain the fixes. This probably will not resolve the issue, but it would give us one more data point if squatter turns out to run cleanly on the development release. Cheers, Robert [1] https://github.com/cyrusimap/cyrus-imapd On Mon, Apr 23, 2018, at 09:48, Robert Stepanek wrote: > Hi Albert, > > On Sat, Apr 21, 2018, at 08:05, Albert Shih wrote: > > Le 20/04/2018 à 16:18:23+0200, Robert Stepanek a écrit > > > > Would you be able to inspect a [..] > > > core dump of squatter? > > > > No dump (don't know why). > > For Linux, one has to set a kernel flag to allow core dumps for > processes that changed ownership: > https://github.com/cyrusimap/cyrus-imapd/blob/master/docsrc/imap/reference/faqs/o-coredump.rst > > There's probably a similar thing for FreeBSD, but I haven't worked with > FreeBSD since quite some time. > > > The « funny » thing is, know it's absolutly sure, the first time I get a > > Seg Fault, but the second time it's working perfectly > > I'd be tempted to start guessing here, but it's really too early to > tell. A core trace that tells us exactly where it's crashing is really > the best next step. > > > One question about the way squatter/xapian/cyrus interact, but is it > > possible to loose email if squatter/xapian goes to segmentation during a > > incomming mail ? > > The squatter process is running separately from the IMAP-related > processes. There's shouldn't be any issue losing mail. At worst, you > might end up with corrupted indices, but they all can be rebuilt from > the mail folders. > > Cheers, > Robert > > > > > Thanks for the help. > > > > Regards. > > > > > > > > > > > > > > > > I'm trying to configure xapian and cyrus-imapd, so I put this in > > > > my config > > > > > > > > sync_log: on sync_log_channels: squatter search_engine: > > > > xapian search_index_headers: no search_batchsize: 16384 > > > > defaultsearchtier: t1 t1searchpartition-default: /var/imap/search > > > > conversations: on conversations_db: twoskip > > > > > > > > but when I launch squatter they crash randomly, meaning I can > > > > launch squatter, it going to run during 10sec, sometime 30 sec, > > > > after that they give a me > > > > > > > > squatter[34140]: indexing mailbox user.... > > > > Indexing mailbox user.... Segmentation fault > > > > > > > > It seems (human watch) this happen only when beginning indexing > > > > a new user, so I tough it's because he need to create the > > > > > > > > /var/imap/search/*/user/*/xapian > > > > > > > > so I create all this directorybut don't seem give any > > > > difference. > > > > > > > > But I pretty certain squatter give a segmentation fault every > > > > *FIRST* time he run on a user. > > > > > > > > So I guessing I do something wrong dans squatter need some « > > > > initialize »... > > > > > > > > Regards. > > > > > > > > > > > > > > > > > > > > -- Albert SHIH DIO bâtiment 15 xmpp: j...@obspm.fr Heure > > > > local/Local time: Fri Apr 20 15:30:07 CEST 2018 Cyrus > > > > Home Page: http://www.cyrusimap.org/ List Archives/Info: > > > > http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: > > > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: > > > http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: > > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > -- > > Albert SHIH > > DIO bâtiment 15 > > Observatoire de Paris > > xmpp: j...@obspm.fr > > Heure local/Local time: > > Sat Apr 21 07:57:04 CEST 2018 > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus + Xapian
Hi Albert, On Sat, Apr 21, 2018, at 08:05, Albert Shih wrote: > Le 20/04/2018 à 16:18:23+0200, Robert Stepanek a écrit > > Would you be able to inspect a [..] > > core dump of squatter? > > No dump (don't know why). For Linux, one has to set a kernel flag to allow core dumps for processes that changed ownership: https://github.com/cyrusimap/cyrus-imapd/blob/master/docsrc/imap/reference/faqs/o-coredump.rst There's probably a similar thing for FreeBSD, but I haven't worked with FreeBSD since quite some time. > The « funny » thing is, know it's absolutly sure, the first time I get a > Seg Fault, but the second time it's working perfectly I'd be tempted to start guessing here, but it's really too early to tell. A core trace that tells us exactly where it's crashing is really the best next step. > One question about the way squatter/xapian/cyrus interact, but is it > possible to loose email if squatter/xapian goes to segmentation during a > incomming mail ? The squatter process is running separately from the IMAP-related processes. There's shouldn't be any issue losing mail. At worst, you might end up with corrupted indices, but they all can be rebuilt from the mail folders. Cheers, Robert > > Thanks for the help. > > Regards. > > > > > > > > > > > I'm trying to configure xapian and cyrus-imapd, so I put this in > > > my config > > > > > > sync_log: on sync_log_channels: squatter search_engine: > > > xapian search_index_headers: no search_batchsize: 16384 > > > defaultsearchtier: t1 t1searchpartition-default: /var/imap/search > > > conversations: on conversations_db: twoskip > > > > > > but when I launch squatter they crash randomly, meaning I can > > > launch squatter, it going to run during 10sec, sometime 30 sec, > > > after that they give a me > > > > > > squatter[34140]: indexing mailbox user.... > > > Indexing mailbox user.... Segmentation fault > > > > > > It seems (human watch) this happen only when beginning indexing > > > a new user, so I tough it's because he need to create the > > > > > > /var/imap/search/*/user/*/xapian > > > > > > so I create all this directorybut don't seem give any > > > difference. > > > > > > But I pretty certain squatter give a segmentation fault every > > > *FIRST* time he run on a user. > > > > > > So I guessing I do something wrong dans squatter need some « > > > initialize »... > > > > > > Regards. > > > > > > > > > > > > > > > -- Albert SHIH DIO bâtiment 15 xmpp: j...@obspm.fr Heure > > > local/Local time: Fri Apr 20 15:30:07 CEST 2018 Cyrus > > > Home Page: http://www.cyrusimap.org/ List Archives/Info: > > > http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: > > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: > > http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: > > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > -- > Albert SHIH > DIO bâtiment 15 > Observatoire de Paris > xmpp: j...@obspm.fr > Heure local/Local time: > Sat Apr 21 07:57:04 CEST 2018 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus + Xapian
Hi Albert, Which Cyrus version are you using? Is it a packaged distribution or are you building from source? Would you be able to inspect a core dump of squatter? Do you see squatter log anything before the segfault? I haven't seen the issue you are describing, and it certainly shouldn't happen. From first glance I would suspect an issue in the build setup or library environment, but it's hard to narrow down the root cause with the current information. Cheers, Robert On Fri, Apr 20, 2018, at 15:39, Albert Shih wrote: > Hi, > > > I'm trying to configure xapian and cyrus-imapd, so I put this in my config > > sync_log: on > sync_log_channels: squatter > search_engine: xapian > search_index_headers: no > search_batchsize: 16384 > defaultsearchtier: t1 > t1searchpartition-default: /var/imap/search > conversations: on > conversations_db: twoskip > > but when I launch squatter they crash randomly, meaning I can launch > squatter, it going to run during 10sec, sometime 30 sec, after that they > give a me > > squatter[34140]: indexing mailbox user.... > Indexing mailbox user.... Segmentation fault > > It seems (human watch) this happen only when beginning indexing a new user, > so I tough it's because he need to create the > > /var/imap/search/*/user/*/xapian > > so I create all this directorybut don't seem give any difference. > > But I pretty certain squatter give a segmentation fault every *FIRST* time > he run on a user. > > So I guessing I do something wrong dans squatter need some « initialize > »... > > Regards. > > > > > -- > Albert SHIH > DIO bâtiment 15 > xmpp: j...@obspm.fr > Heure local/Local time: > Fri Apr 20 15:30:07 CEST 2018 > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: sieve and utf-8 MIME/base64 content
On Mon, Feb 19, 2018, at 14:33, Eugene M. Zheganin wrote: > But my users kind of demand more and more :), now they want not only the > header/body search, but also sorting out the letters to the folders that > contain the localized symbols. So, it's working fine when a folder is > named 'Junk e-mail', but when it's named 'Спам', this stops working. Of > course, the workaround is to determine on-disk name of the folder, this > can be done by ls'ing the folder name on the disk, but only few users > can ssh to server, so the rest have a choice of either using latin names > for folders or to ask their system engineer. So I wanted to ask - is > there any more elegant solution ? The folder name either needs to be in UTF-8 or modified UTF-7 encoding (aka IMAP UTF-7). If you want to use UTF-8, you'll need to enable the sieve_utf8fileinto option in imapd.conf (see https://www.cyrusimap.org/imap/reference/manpages/configs/imapd.conf.html). Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: sieve and utf-8 MIME/base64 content
Thanks for making this more clear to me :) On Fri, Feb 16, 2018, at 11:07, Vladislav Kurz wrote: > The problem is that headers with non-ascii chars are encoded in form > like this: > Subject: =?UTF-8?B?UmU6I.gVG9v?= > And also the body is sometimes completely in base64, even though it is > just plaintext or HTML in UTF-8. Depends on sender's mail client. AFAIK the Sieve implementation in Cyrus 2.5 fully implements RFC 5228, including the string comparison requirements of section 2.7 [1]. That is, it implements the ascii-casemap and octet collations. It decodes MIME headers to UTF-8 before matching (e.g. see [2] and [3]). The RFC 5173 Sieve body extension is also supported [4]. Eugene, does that work for you? Cheers, Robert [1] https://tools.ietf.org/html/rfc5228#section-2.7 [2] https://github.com/cyrusimap/cyrus-imapd/blob/cyrus-imapd-2.5/sieve/script.c#L261 [3] https://github.com/cyrusimap/cyrus-imapd/blob/cyrus-imapd-2.5/sieve/bc_eval.c#L809 [4] https://cyrusimap.org/imap/rfc-support.html Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: sieve and utf-8 MIME/base64 content
On Fri, Feb 16, 2018, at 10:17, Eugene M. Zheganin wrote: > I'm using sieve with cyrus to sort incoming mail, and it works perfectly > with latin symbols. But what if I need to sort out the mail that has all > sorts utf-8 sumbols in it ? Like MIME-encoded headers and base64 > -encoded body ? The main developer for Sieve support (and 2.5) in Cyrus IMAP might not be able to respond the next days. That being said, I'm not sure I understand what use case you are trying to accomplish? Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Truncated text during Xapian indexing
On Thu, Feb 15, 2018, at 13:08, Sebastian Hagedorn wrote: > Is the setting "search_skipdiacrit" in imapd.conf honored during the > indexing or is that only relevant while searching? Given your comment > regarding search normalization above I take it Umlaut characters are not > considered diacriticals? It's not a huge issue, but as a German university > it would be nice for our users if a search could distinguish between > "hatte" and "hätte", as an example. Cyrus considers Umlaut characters as diacriticals (I was just handwaving that away in my previous comment due to the default settings). The skip_diacrit setting applies to both indexing and search. As an example, let's append two emails to a mailbox. The body of message 1 contains the German verb "gären". Message 2 contains the verb "garen" (for the non-German speakers: these verbs mean two different things). With skip_diacrit set to true (the default), this is what lands in the Xapian database: [...] Zgaren garen and searches for "garen" and "gären" will both match both messages. With skip_diacrit set to false, however, we get [...] Zgaren Zgären garen gären and searches for "garen" and "gären" will only match the respective messages. I uploaded a new test to Cassandane that demonstrates this [1] (the subject_isutf8 test case might also be of interest). I'd just deactivate search_skipdiacrit if you are sure that your users will benefit from it. If in doubt, I would rather err on the safe side and return false positives by skipping diacritics (the default). There's more to say about the Z prefixes: Cyrus currently uses the English stemmer for all text, resulting in stem terms that typically match their non-stemmed original input for non-English text. While this might seem odd, it's the best we can do without proper language detection for both indexing and search. I implemented multi-language stem support in an experimental feature branch, but didn't resolve the issues around fingerprinting search queries, yet. There's an open issue to track this [2]. [1] https://github.com/cyrusimap/cassandane/blob/master/Cassandane/Cyrus/SearchFuzzy.pm#L403 [2] https://github.com/cyrusimap/cyrus-imapd/issues/72 > Just out of curiosity, how is the mapping between a Xapian docid and a > message file on disk achieved? I played around with xapian-delve and the > Perl example simplesearch.pl. When I search a term, I get a list of > docid's, but how do I know which message that is? In 3.x, Cyrus search stores an internal unique message id, called guid, as docid in Xapian. The guid currently is a SHA-1 hash of the raw message, allowing for deduplication and to avoid re-indexing already seen messages. The conversations.db of a user maps this guid to a list of mailbox:UID pairs. Off the top of my head, there currently isn't an "official" way in Cyrus to retrieve the mailbox:UID list for a given guid outside the Cyrus process. Depending on your use case, you could either: 1.) build your custom mapper on imap/conversations.h, 2.) use cvt_cyrusdb to dump the contents of a conversations.db into plain text. Or 3.) use the JMAP layer to fetch JMAP-formatted message or the raw message blob by id. For JMAP email, use the guid and prefix it with 'M' in an Email/get method. For blobs, use 'G' as prefix. Both are "unofficial": we might change the JMAP id scheme in future releases. But I guess this isn't going to happen any time soon, if ever. Hope it helps, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Truncated text during Xapian indexing
On Thu, Feb 15, 2018, at 10:44, Sebastian Hagedorn wrote: > ^Simon^: Is that the first 4Mb of the text/html and/or text/plain parts, or > first 4Mb of the entire message body, ignoring any mime parts? This limit defines the maximum byte length per MIME body-part of type "text". The byte length is calculated after decoding (e.g. quoted-printable), conversion to UTF-8 and search text normalisation (e.g. stripping HTML tags, replacing Umlaut characters with their ASCII counterparts, etc.). Actually, it also applies to any other search-indexed fields, such as subjects, headers, etc. but in practice only is relevant for mail bodies. > nicola_fm: For a faster response, drop some queries about cyrus and xapian > on the mailing list. I am a poor proxy for sending messages to Robert S! > As suggested by Nicola, I am taking it to the list :-) Good idea :) Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: squatter lower limits
Hi, I haven't worked with the squat backend, but I highly recommend to switch to the latest stable 3.0 branch and the Xapian backend, if you can. The Xapian backend does not discriminate between short and long strings, provides stemming and multi-tiered search databases, among other improvements. Note that the squat backend and the squatter tool share the same name, but really become two separate things, e.g. you'd still use the squatter executable to index Xapian-backed search databases. Bron wrote up a description of search setup in 2014: https://blog.fastmail.com/2014/12/01/email-search-system/ Cheers, Robert On Fri, Sep 15, 2017, at 11:35, Paolo Cravero wrote: > Hello. > > While looking to do low-level disk usage optimization, some simple > performance tests relied on full-text searches (2.4 branch). Metadata > always resides on local disks, while messages are on slower hardware. > > I noticed that full-text searches with short strings take much longer > than longer text. For example, a FT search on 3 letters takes >60" while > a 9-letter long string on the same corpus lasts ~20". These tests have > been repeated over and over again to exclude disk caching being the > culprit: reversing the search order - longer first - has no impact. > > So I opened up the cyrus source code and looked for search-related code. > As I understand it, squatter is not used if the search string is shorter > than 4 symbols. From squat.h it's quite clear: > > /* > Don't change this unless you're SURE you know what you're doing. > Its only effect on the API is that searches for strings that are > shorter than SQUAT_WORD_SIZE are not allowed. > In SQUAT, a 'word' simply refers to a string of SQUAT_WORD_SIZE > arbitrary bytes. > */ > > #define SQUAT_WORD_SIZE 4 > > So, question to who knows the squatter implementation in cyrus: is this > lower limit applied to all searches? Body, subject, addresse(s)? > > And, does this lower bound still apply to 3.0 branch and the new indexing > engine Xapian? > > Let alone low level disk compression or optimization, a client might not > handle well long search times without receiving data on the IMAP channel > and dismiss the connection (or a network device could do it). So, if > searching for short strings means reading all raw message files, I should > warn users through the client interface of possible failures since the > mail corpus keeps growing and growing and growing. That's until we > upgrade to 3.0, it that helps. > > Thanks, > > Paolo > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: About upgrade and building without caldav/carddav server
Hi, On Fri, Jun 23, 2017, at 14:57, Egoitz Aurrekoetxea wrote: > Is it possible to build Cyrus IMAP 3.0 without Caldav/Carddav server?. > I say this because we have been using Davical from some time now, have > all development and all done for it. Can it be build without it?. Caldav/Carddav is only built into Cyrus if you pass the --enable-http flag to the configure script. Even if HTTP support is built in, Cyrus won't claim any HTTP/HTTPS standard port, as long as the 'http' service is not enabled in the SERVICES section of cyrus.conf (note that '#' character at the start of the line): # httpcmd="httpd" listen="http" prefork=0 If you want to use some of Cyrus HTTP services on a non-standard port, change the 'listen' parameter to whatever port you prefer. Make sure the HTTP services you want from Cyrus are enabled in the 'httpmodules' parameter in imapd.conf. > By the way, when upgrading from a 2.3.1X server... can you directly > install a 3.0 in the server and should it work?. Is it recommended to > perhaps go thought the 2.5 version, reconvert> databases to suit it's needs > (the 2.5 needs) and later pass to 3.0?. I have no experience with upgrading from 2.3 to 3.0, so can't help you with that. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Cyrus 2.5.10 IMAP search
Hi, On Thu, Jun 1, 2017, at 12:41, Anton Shilov via Info-cyrus wrote: > I've upgraded CyrusIMAP from 2.4.17 to 2.5.10 and after that has been > broken IMAP search in email body russian language. Search for english > works fine. Is upgrading to 3.0 an option for you? 3.0 includes both Xapian and a new character set implementation for international language search. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Error: same message appears twice
On Wed, Apr 19, 2017, at 10:51, Frederik Himpe wrote: > Sometimes I see in the logs messages like this: "same message appears > twice 2394 2395". Looking at the source code, it does not appear to be > a big problem, but I am not sure. Does this indicate a real problem? > Should I run cyrreconstruct on the affected mailbox, and which options > do you recommend? When a new message is appended to a mailbox, Cyrus checks if the hash (SHA1) of the message content matches the latest stored message in this mailbox. This log message is printed when the hashes match. Since you see this log message only sometimes, I wouldn't think it's a problem. It indicates that some client might call APPEND for the same message twice or more often, though. I can't imagine that cyrreconstruct will make a difference, except if your mailbox index is broken. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
Re: Can't find icu-uc to build IMAP 3.0.0
On Wed, Apr 12, 2017, at 18:29, Rosenbaum, Larry M. wrote: > I am trying to build Cyrus-IMAP 3.0.0 on RHEL6. I am getting the > following build error: > > checking for ICU... configure: error: Package requirements (icu-uc) > were not met: > > No package 'icu-uc' found > > Consider adjusting the PKG_CONFIG_PATH environment variable if you > installed software in a non-standard prefix. > > Alternatively, you may set the environment variables ICU_CFLAGS > and ICU_LIBS to avoid the need to call pkg-config. > See the pkg-config man page for more details. > > I have already installed libicu-devel. What other package do I need to > install to get past this check? Your pkg-config executable seems to be unable to find the icu-uc library. You can check if pkg-config finds icu-uc with $ pkg-config --cflags --libs icu-uc which should print something like -I/usr/local/cyruslibs/include -L/usr/local/cyruslibs/lib - licuuc -licudata If it doesn't, does pkg-config find any of the other ICU packages, e.g. icu-io? If yes, you might have an incomplete ICU installation. If no, you might want to check where the RPM did install libicu-devel to and update your PKG_CONFIG_PATH environment variable accordingly. Cheers, Robert Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
