Re: Solr -> Xapian ?
also, for fts_backend_solr_update_set_build_key -> where is the data (of
the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those
functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */
int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box,
uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this
returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to
index this key, it can return FALSE and caller will skip to next key. */
bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx,
const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key.
Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters,
but it doesn't need to be NUL-terminated. size contains the data size in
bytes, not characters. This function may be called many times and the data
block sizes may be small. Backend returns 0 if ok, -1 if build should be
aborted. */
int fts_backend_update_build_more(struct fts_backend_update_context *ctx,
const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a
feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups.
Returns 0 if ok, -1 if error. */
int fts_backend_refresh(struct fts_backend *backend);
/* Go through the entire index and make sure all mails are indexed,
and delete any extra mails in the index. */
int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the
fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are
documented. The remaining questions can be answered by looking at examples
found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a
small effort though. As soon as you have concrete questions, we can help you
(don't expect rapid responses though).
Regards,
Stephan.
Re: Solr -> Xapian ?
for the "last uid"-> this is not the last added, but the maximum of the
UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those
functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */
int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box,
uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this
returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to
index this key, it can return FALSE and caller will skip to next key. */
bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx,
const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key.
Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters,
but it doesn't need to be NUL-terminated. size contains the data size in
bytes, not characters. This function may be called many times and the data
block sizes may be small. Backend returns 0 if ok, -1 if build should be
aborted. */
int fts_backend_update_build_more(struct fts_backend_update_context *ctx,
const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a
feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups.
Returns 0 if ok, -1 if error. */
int fts_backend_refresh(struct fts_backend *backend);
/* Go through the entire index and make sure all mails are indexed,
and delete any extra mails in the index. */
int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the
fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are
documented. The remaining questions can be answered by looking at examples
found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a
small effort though. As soon as you have concrete questions, we can help you
(don't expect rapid responses though).
Regards,
Stephan.
Re: Solr -> Xapian ?
Thank you
I still don't get the "build_key" function. The email (body, hearders,
.. and the uid) is the one (and only) to index . What "key" is that
function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau:
Anyone willing to explain those functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */
int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box,
uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this
returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to
index this key, it can return FALSE and caller will skip to next key. */
bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx,
const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key.
Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters,
but it doesn't need to be NUL-terminated. size contains the data size in
bytes, not characters. This function may be called many times and the data
block sizes may be small. Backend returns 0 if ok, -1 if build should be
aborted. */
int fts_backend_update_build_more(struct fts_backend_update_context *ctx,
const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a
feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups.
Returns 0 if ok, -1 if error. */
int fts_backend_refresh(struct fts_backend *backend);
/* Go through the entire index and make sure all mails are indexed,
and delete any extra mails in the index. */
int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the
fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are
documented. The remaining questions can be answered by looking at examples
found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a
small effort though. As soon as you have concrete questions, we can help you
(don't expect rapid responses though).
Regards,
Stephan.
Re: IMAP preauth and stats-writer
On 06/01/2019 02:26, John Fawcett wrote: > On 05/01/2019 15:49, Mark Hills wrote: >> I use IMAP preauth; I connect with Alpine over SSH which is very useful. >> >> The last few upgrades this has become more difficult to to. Last time >> (moving 2.2 -> 2.3, I think) I had to put in a workaround: >> >> stats_writer_socket_path = >> >> It prevented /usr/local/libexec/dovecot/imap attempting to >> connect to a central stats service. >> >> As of an upgrade today (2.3.2.1_1 -> 2.3.4_3 on FreeBSD) it looks like >> that 'fix' stopped working, and I get: >> >> imap(mark,)Error: net_connect_unix() failed: Permission denied >> >> It goes to stderr, which breaks Alpine. >> >> "()" is actually the filename. It seems that the empty string is no longer >> an indication to disable it. Here it is with the default configuration: >> >> imap(mark,)Error: net_connect_unix(/var/run/dovecot/stats-writer) failed: >> Permission denied >> >> I do also have a dovecot running as a system daemon, and, interestingly, >> disable this and it's 'fixed'; no attempt to connect. However, disabling >> the service is not an option (needed for smartphone) >> >> The best I've come up with so far is when using preauth to hack it to send >> stderr to /dev/null. And yes, probably Alpine is at fault for interpreting >> stderr content (separate issue) >> >> * Is there a way to cleanly disable reporting to the stats service? >> Previously, running as preauth was all very clean. >> >> * Just wanted to highlight that IMAP preauth is really useful. Even >> though it might not be mainstream, it seems healthy to be able to easily >> install dovecot as an unprivilidged user in a "unixy" way. >> >> I'm on FreeBSD 11.2, with dovecot from ports. dovecot.conf below. >> > Can't see anything in the Dovecot 2.3.4 code that would give this > problem, setting > > stats_writer_socket_path = > > will overwrite the default value and dovecot does not attempt to open a > socket in that case. > > Using your configuration (though not FreeBSD) I don't get the > net_connect_unix error whether I use a blank setting or leave the default. In > the case of leaving the default I do get an additional process > (dovecot/stats). No errors on connecting to the imap service or by running > preauth (with the dovecot daemon already running). > > The net_connect_unix() error with a zero length socket name is inexplicable > to me, unless it's got a non printing character in it or there is something > different happening on FreeBSD. > > One suggestion is to run with the default setting, but look at resolving the > permission problem for the default socket creation at > /var/run/dovecot/stats-writer rather than working round it. > > John > Just following up, I don't get the error when I run preauth as root with -u parameter. I do get something similar when I run as the user (this wil the socket path set to blank) Centos 7: Error: net_connect_unix() failed: Connection refused FreeBSD 11.2: Error: net_connect_unix() failed: No such file or directory So it's close. If I get time I'll see if I can track it down. John
Re: IMAP preauth and stats-writer
On 05/01/2019 15:49, Mark Hills wrote: > I use IMAP preauth; I connect with Alpine over SSH which is very useful. > > The last few upgrades this has become more difficult to to. Last time > (moving 2.2 -> 2.3, I think) I had to put in a workaround: > > stats_writer_socket_path = > > It prevented /usr/local/libexec/dovecot/imap attempting to > connect to a central stats service. > > As of an upgrade today (2.3.2.1_1 -> 2.3.4_3 on FreeBSD) it looks like > that 'fix' stopped working, and I get: > > imap(mark,)Error: net_connect_unix() failed: Permission denied > > It goes to stderr, which breaks Alpine. > > "()" is actually the filename. It seems that the empty string is no longer > an indication to disable it. Here it is with the default configuration: > > imap(mark,)Error: net_connect_unix(/var/run/dovecot/stats-writer) failed: > Permission denied > > I do also have a dovecot running as a system daemon, and, interestingly, > disable this and it's 'fixed'; no attempt to connect. However, disabling > the service is not an option (needed for smartphone) > > The best I've come up with so far is when using preauth to hack it to send > stderr to /dev/null. And yes, probably Alpine is at fault for interpreting > stderr content (separate issue) > > * Is there a way to cleanly disable reporting to the stats service? > Previously, running as preauth was all very clean. > > * Just wanted to highlight that IMAP preauth is really useful. Even > though it might not be mainstream, it seems healthy to be able to easily > install dovecot as an unprivilidged user in a "unixy" way. > > I'm on FreeBSD 11.2, with dovecot from ports. dovecot.conf below. > Can't see anything in the Dovecot 2.3.4 code that would give this problem, setting stats_writer_socket_path = will overwrite the default value and dovecot does not attempt to open a socket in that case. Using your configuration (though not FreeBSD) I don't get the net_connect_unix error whether I use a blank setting or leave the default. In the case of leaving the default I do get an additional process (dovecot/stats). No errors on connecting to the imap service or by running preauth (with the dovecot daemon already running). The net_connect_unix() error with a zero length socket name is inexplicable to me, unless it's got a non printing character in it or there is something different happening on FreeBSD. One suggestion is to run with the default setting, but look at resolving the permission problem for the default socket creation at /var/run/dovecot/stats-writer rather than working round it. John
Re: Solr -> Xapian ?
Op 06/01/2019 om 01:00 schreef Joan Moreau:
Anyone willing to explain those functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */
int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox
*box,
uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me
this returns the last UID added to the index for the given mailbox and
FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to
index this key, it can return FALSE and caller will skip to next key. */
bool fts_backend_update_set_build_key(struct fts_backend_update_context
*ctx,
const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key.
Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters,
but it doesn't need to be NUL-terminated. size contains the data size in
bytes, not characters. This function may be called many times and
the data
block sizes may be small. Backend returns 0 if ok, -1 if build should be
aborted. */
int fts_backend_update_build_more(struct fts_backend_update_context *ctx,
const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to
get a feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups.
Returns 0 if ok, -1 if error. */
int fts_backend_refresh(struct fts_backend *backend);
/* Go through the entire index and make sure all mails are indexed,
and delete any extra mails in the index. */
int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot
wrote:
Thank Stephan
I basically need to know the role/description of each of the
functions of the fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory
funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the
basics are documented. The remaining questions can be answered by
looking at examples found in similar plugins or the relevant API
sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will
not be a small effort though. As soon as you have concrete
questions, we can help you (don't expect rapid responses though).
Regards,
Stephan.
Re: Solr
On 1/5/2019 9:58 AM, Tanstaafl wrote: Thanks Daniel... So, as one who has no experience of the benefit of either... How does this compare with Squat? Meaning, Is it exponentially faster? Twice as fast? It's been many years since I last had a Squat setup - but that's my memory. -- Daniel
Re: Solr -> Xapian ?
Anyone willing to explain those functions ?
Most notably " get_last_uid" "set_build_key" "build_more" , what is refresh
versus rescan ?
On January 5, 2019 14:23:10 Joan Moreau via dovecot
wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of
the fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> what other flags ?
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons,
etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics
are documented. The remaining questions can be answered by looking at
examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be
a small effort though. As soon as you have concrete questions, we can help
you (don't expect rapid responses though).
Regards,
Stephan.
Re: Solr -> Xapian ?
Anyone willing to explain those functions ?
On January 5, 2019 14:23:10 Joan Moreau via dovecot
wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of
the fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT, -> what other flags ?
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons,
etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics
are documented. The remaining questions can be answered by looking at
examples found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be
a small effort though. As soon as you have concrete questions, we can help
you (don't expect rapid responses though).
Regards,
Stephan.
Re: Solr
On 1/3/2019, 3:07:18 PM, Daniel Miller via dovecot wrote: > On 1/3/2019 10:56 AM, Tanstaafl wrote: >> On 12/21/2018, 11:19:42 AM, Daniel Miller via dovecot >> wrote: >>> There is a *huge* difference between a functional Solr setup & squat >> Interesting. Care to elaborate? > > This is one of those things that has to be experienced to be > understood. When you can perform an FTS search across (pause while I > check current stats...): > > du -c -h /var/mail 136G > > Solr numDocs: 520102 > > and using any IMAP client that supports server-side searches (like > Thunderbird & AquaMail) the results are basically instantaneous...it's > worth the effort. And that's searching a Dovecot virtual folder defined > as "* all", including all my archives, all my list subscriptions, and > all the shared Inbox/Sent folders from my other users. > > But I certainly wish it was easier to setup. Thanks Daniel... So, as one who has no experience of the benefit of either... How does this compare with Squat? Meaning, Is it exponentially faster? Twice as fast?
Re: Sieve "OOO" configuration
On Sat, 05 Jan 2019 11:40:52 -0500, James Cassell stated: >On Sat, Jan 5, 2019, at 11:26 AM, Jerry wrote: >> I am able to get sieve issuing an "out of office"message correctly. >> However, I want to configure it to send an "OOO" message only during >> certain dates, say on weekends, or only between certain hours. I can >> do that manually; however, was wondering if there is any automatic >> method available that could handle this chore. >Look at the currentdate test >https://tools.ietf.org/html/rfc5260#section-5 > > >V/r, >James Cassell Wow, that looks to be just what I wanted. I will experiment with it a little bit and see if I can get it working the way I want. Thanks! -- Jerry
Re: Sieve "OOO" configuration
Look at the currentdate test https://tools.ietf.org/html/rfc5260#section-5 V/r, James Cassell On Sat, Jan 5, 2019, at 11:26 AM, Jerry wrote: > I am able to get sieve issuing an "out of office"message correctly. > However, I want to configure it to send an "OOO" message only during > certain dates, say on weekends, or only between certain hours. I can do > that manually; however, was wondering if there is any automatic method > available that could handle this chore. > > Thanks! > > -- > Jerry
Sieve "OOO" configuration
I am able to get sieve issuing an "out of office"message correctly. However, I want to configure it to send an "OOO" message only during certain dates, say on weekends, or only between certain hours. I can do that manually; however, was wondering if there is any automatic method available that could handle this chore. Thanks! -- Jerry
zlib - some emails maybe double or partial gzipped
Hi, I found these in my logs: Jan 5 15:58:56 mail dovecot: imap-login: Login: user=, method=PLAIN, rip=1.2.3.4, lip=2.3.4.5, mpid=5852, TLS, session= Jan 5 15:58:56 mail dovecot: imap-login: Login: user=, method=PLAIN, rip=1.2.3.4, lip=2.3.4.5, mpid=5854, TLS, session= Jan 5 15:58:56 mail dovecot: imap(): Error: Maildir filename has wrong S value, renamed the file from /var/vmail/mailboxes//mail/Sent/cur/1537260554.M80670P8164.mail,S=17247,W=17549:2,Sab,Z to /var/vmail/mailboxes//mail/Sent/cur/1537260554.M80670P8164.mail,S=4172:2,Sab,Z Jan 5 15:58:56 mail dovecot: imap(): Error: Corrupted index cache file /var/vmail/mailboxes//mail/Sent/dovecot.index.cache: Broken physical size for mail UID 99 in mailbox Sent: read(zlib(/var/vmail/mailboxes//mail/Sent/cur/1537260554.M80670P8164.mail,S=17247,W=17549:2,Sab,Z)) failed: Cached message size larger than expected (17247 > 4268, box=Sent, UID=99) Jan 5 15:58:56 mail dovecot: imap(): Panic: file istream.c: line 175 (i_stream_read): assertion failed: (old_size <= _stream->pos - _stream->skip) Jan 5 15:58:56 mail dovecot: imap(): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7f5068d3ee92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7f5068d3ef8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f5068cd4a91] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x290) [0x7f5068d49f20] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5068d4a72d] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header_next+0x72) [0x7f5068d25d12] -> /usr/lib/dovecot/libdovecot.so.0(+0x76fd1) [0x7f5068d1ffd1] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x53) [0x7f5068d49ce3] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5068d4a72d] -> /usr/lib/dovecot/libdovecot.so.0(message_parse_header_next+0x72) [0x7f5068d25d12] -> /usr/lib/dovecot/libdovecot.so.0(+0x76fd1) [0x7f5068d1ffd1] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x53) [0x7f5068d49ce3] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f5068d4a72d] -> /usr/lib/dovecot/libdovecot.so.0(message_get_header_size+0x74) [0x7f5068d29904] -> /usr/lib/dovecot/libdovecot-storage.so.0(imap_msgpart_open+0x32f) [0x7f50690b52ef] -> dovecot/imap(+0x1ebbc) [0x556dd2cd7bbc] -> dovecot/imap(+0x1cfb6) [0x556dd2cd5fb6] -> dovecot/imap(imap_fetch_more+0x39) [0x556dd2cd70e9] -> dovecot/imap(cmd_fetch+0x33b) [0x556dd2cc8d9b] -> dovecot/imap(command_exec+0xa5) [0x556dd2cd4735] -> dovecot/imap(+0x199c2) [0x556dd2cd29c2] -> dovecot/imap(+0x19a4c) [0x556dd2cd2a4c] -> dovecot/imap(client_handle_input+0x1b5) [0x556dd2cd2e55] -> dovecot/imap(client_input+0x86) [0x556dd2cd33c6] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7f5068d539f2] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x109) [0x7f5068d55029] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7f5068d53a8c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f5068d53c38] Jan 5 15:58:56 mail dovecot: imap(): Fatal: master: service(imap): child 5852 killed with signal 6 (core dumps disabled) # and again: Jan 5 15:58:56 mail dovecot: imap(): Error: Maildir filename has wrong S value, renamed the file from /var/vmail/mailboxes//mail/Sent/cur/1537260554.M80670P8164.mail,S=4172:2,Sab,Z to /var/vmail/mailboxes//mail/Sent/cur/1537260554.M80670P8164.mail,S=4268:2,Sab,Z Jan 5 15:58:56 mail dovecot: imap(): Error: Corrupted index cache file /var/vmail/mailboxes//mail/Sent/dovecot.index.cache: Broken physical size for mail UID 254 in mailbox Sent: read(zlib(/var/vmail/mailboxes//mail/Sent/cur/1537260554.M80670P8164.mail,S=4172:2,Sab,Z)) failed: Cached message size smaller than expected (4172 < 4268, box=Sent, UID=254) Jan 5 15:58:56 mail dovecot: imap(): Panic: file istream.c: line 212 (i_stream_read): assertion failed: (!i_stream_is_buffer_invalid(_stream)) Jan 5 15:58:56 mail dovecot: imap(): Error: Raw backtrace: /usr/lib/dovecot/libdovecot.so.0(+0x95e92) [0x7f1dd9f6be92] -> /usr/lib/dovecot/libdovecot.so.0(+0x95f8d) [0x7f1dd9f6bf8d] -> /usr/lib/dovecot/libdovecot.so.0(i_fatal+0) [0x7f1dd9f01a91] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read+0x190) [0x7f1dd9f76e20] -> /usr/lib/dovecot/libdovecot.so.0(i_stream_read_data+0x3d) [0x7f1dd9f7772d] -> /usr/lib/dovecot/libdovecot.so.0(message_get_header_size+0x74) [0x7f1dd9f56904] -> /usr/lib/dovecot/libdovecot-storage.so.0(index_mail_init_stream+0x388) [0x7f1dda2a7538] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66fc0) [0x7f1dda25efc0] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_stream_because+0x5d) [0x7f1dda22ee1d] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0x66d87) [0x7f1dda25ed87] -> /usr/lib/dovecot/libdovecot-storage.so.0(mail_get_virtual_size+0x33) [0x7f1dda22ea83] -> dovecot/imap(+0x1eae3) [0x5595eb2dfae3] -> dovecot/imap(+0x1cfb6) [0x5595eb2ddfb6] -> dovecot/imap(imap_fetch_more+0x39)
IMAP preauth and stats-writer
I use IMAP preauth; I connect with Alpine over SSH which is very useful.
The last few upgrades this has become more difficult to to. Last time
(moving 2.2 -> 2.3, I think) I had to put in a workaround:
stats_writer_socket_path =
It prevented /usr/local/libexec/dovecot/imap attempting to
connect to a central stats service.
As of an upgrade today (2.3.2.1_1 -> 2.3.4_3 on FreeBSD) it looks like
that 'fix' stopped working, and I get:
imap(mark,)Error: net_connect_unix() failed: Permission denied
It goes to stderr, which breaks Alpine.
"()" is actually the filename. It seems that the empty string is no longer
an indication to disable it. Here it is with the default configuration:
imap(mark,)Error: net_connect_unix(/var/run/dovecot/stats-writer) failed:
Permission denied
I do also have a dovecot running as a system daemon, and, interestingly,
disable this and it's 'fixed'; no attempt to connect. However, disabling
the service is not an option (needed for smartphone)
The best I've come up with so far is when using preauth to hack it to send
stderr to /dev/null. And yes, probably Alpine is at fault for interpreting
stderr content (separate issue)
* Is there a way to cleanly disable reporting to the stats service?
Previously, running as preauth was all very clean.
* Just wanted to highlight that IMAP preauth is really useful. Even
though it might not be mainstream, it seems healthy to be able to easily
install dovecot as an unprivilidged user in a "unixy" way.
I'm on FreeBSD 11.2, with dovecot from ports. dovecot.conf below.
--
Mark
#
# Dovecot configuration
#
mail_location = maildir:~/Maildir
postmaster_address = postmaster
namespace {
inbox = yes
}
ssl = required
ssl_cert =
Searching small(ish) sites on ubuntu
I run a small dovecot site with ~5 users, some of whom have many large mailboxes. I used to use lucene until the packagers decided that it broke some licencing rules and stopped including it. I have dismally failed to configure Solr, plus it is huge and potentially exposes too many services as well. Squash is deprecated (but still used?) in 2.2+. I am currently on 2.2.33. So what can I use? Or do I have to bite the bullet and install from source, thus losing the advantages of packaging and automatic updates? Dirk
Re: Fwd: Re: gcc -> clang
Same problem with 2.3.4. configure lacks debug flags. No joy with gdb breakpoint in mallock_error_break on make test. On Thu, Jan 3, 2019 at 13:58, Aki Tuomi wrote: > The arithmetic stuff has already been fixed in master with > > https://github.com/dovecot/core/commit > /5cccb4af850bb3ba81e73a8fb4f6881c3e1d4046.patch > https://github.com/dovecot/core/commit > /ac7aa955db4c77bbb169baa5d104a4c128674646.patch > > I have not seen the second error ever, it would need more information, can > you gdb to it and see what the actual value is? > > Aki > >> On 03 January 2019 at 13:38 Rupert Gallagher via dovecot >> wrote: >> >> >> The compiler returns many warnings, and the test returns two IPv6-related >> errors. I am attaching both logs as reference. >> >> > ‐‐‐ Original Message ‐‐‐ >> > On Thursday, January 3, 2019 9:53 AM, Aki Tuomi >> > wrote: >> > >> >> We compile all core code with both gcc and clang. What sort of >> >> interesting things did you find? >> >> >> >> Aki >> >> >> >>> On 03 January 2019 at 11:50 Rupert Gallagher via dovecot < >> >>> dovecot@dovecot.org> wrote: >> >>> >> >>> Please, use clang instead of gcc. Code quality can only profit from it. >> >>> I just compiled 2.3.4 and compiler stderr is full of interesting >> >>> problems. >> >> >> >> --- >> >> Aki Tuomi
