Re: special_use folders still created
Hi, I also experienced that thunderbird behavior, I guess > [...] > > When I move a message to Archive in Thunderbird. On the server the > files What exactly means move? Like drag and drop or via context menu "archive/archivieren"? > .Archives and .Archives.2018 are created (Thunderbird creates a > subfolder in Archives) > > How is this possible? On the server there is already an .Archive > folder > so it should create .Archive.2018 not? I think this is intended. At least all my clients (thunderbird) do this. I think mozilla hoped to improve performance on big mailboxes by simply sorting archived mails by the mail's year. > > [...] > -M i...@awib.it
Re: special_use folders still created
Op 15/12/2018 om 19:10 schreef Marc Roos:
dovecot-2.2.36-3.el7.x86_64
The below configuration seems to work for Thunderbird and emclient, when
moving a message to Junk. Emclient uses "Junk E-mail" and Thunderbird
uses "Junk". On the server is only the file .Junk created as expected.
When I move a message to Archive in Thunderbird. On the server the files
.Archives and .Archives.2018 are created (Thunderbird creates a
subfolder in Archives)
How is this possible? On the server there is already an .Archive folder
so it should create .Archive.2018 not?
I can't say I have much experience with such client issues. Your
configuration doesn't look wrong at first glance. Maybe assigning the
special use flag to the mailboxes you don't want is a bad idea, but I am
not sure.
First, you could connect to your IMAP server directly (e.g. using
telnet) to verify that:
1) The SPECIAL-USE capability is actually announced.
2) The mailbox listed below are actually being returned by LIST command
with the appropriate arguments.
Then (or alternatively), you can check the communication between your
clients and Dovecot using the rawlog facility to find out what those are
doing. That may give you some clues.
Regards,
Stephan.
imap_capability = +SPECIAL-USE
# drafts folders merging
mailbox Drafts {
special_use = \Drafts
auto = create
}
# archive folders merging
mailbox Archive {
special_use = \Archive
auto = create
}
mailbox Archives {
special_use = \Archive
auto = no
}
# spam folders merging
mailbox Junk {
special_use = \Junk
auto = create
}
mailbox Spam {
special_use = \Junk
auto = no
}
mailbox "Junk E-mail" {
special_use = \Junk
auto = no
}
Re: Authentication/Penalty disabled (socket mode=0) introduces constant 5 sec delays (2.27 on debian 9)
Op 20/12/2018 om 18:09 schreef Ludovic Pouzenc:
Hi,
I hit a bizare problem with dovecot 2.2.7 on debian 9 with LMTP enabled and
auth/penalty disabled as documented here :
https://wiki.dovecot.org/Authentication/Penalty
Use case : I run a swaks command to send an email to an exim4 that tries to
make a callout to dovecot-lmtp.
At RCPT TO: swaks hangs 5.0 seconds then process normally
(exim is waiting for callout completion).
with strace, I see 5 second delay with many tries to socket connection.
The sources confirm this behavior, but only for the first connection
attempt (i.e. the first 5 seconds of reconnections), which happens at
auth startup (which then takes 5 seconds longer). After that, it will
mark auth-penalty as disabled and it will not try connecting to it
again. Is that what you're referring to?
Regards,
Stephan.
(PID 9652 was an auth process freshly forked)
[pid 9652] socket(AF_UNIX, SOCK_STREAM, 0) = 14
[pid 9652] fcntl(14, F_GETFL) = 0x2 (flags O_RDWR)
[pid 9652] fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 9652] connect(14, {sa_family=AF_UNIX, sun_path="anvil-auth-penalty"},
110) = -1 ECONNREFUSED (Connection refused)
[pid 9652] close(14) = 0
[pid 9652] nanosleep({tv_sec=0, tv_nsec=2000}, NULL) = 0
[pid 9652] socket(AF_UNIX, SOCK_STREAM, 0) = 14
[pid 9652] fcntl(14, F_GETFL) = 0x2 (flags O_RDWR)
[pid 9652] fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 9652] connect(14, {sa_family=AF_UNIX, sun_path="anvil-auth-penalty"},
110) = -1 ECONNREFUSED (Connection refused)
[pid 9652] close(14) = 0
[pid 9652] nanosleep({tv_sec=0, tv_nsec=5000}, NULL) = 0
[pid 9652] socket(AF_UNIX, SOCK_STREAM, 0) = 14
[pid 9652] fcntl(14, F_GETFL) = 0x2 (flags O_RDWR)
[pid 9652] fcntl(14, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 9652] connect(14, {sa_family=AF_UNIX, sun_path="anvil-auth-penalty"},
110) = -1 ECONNREFUSED (Connection refused)
[pid 9652] close(14) = 0
[pid 9652] nanosleep({tv_sec=0, tv_nsec=9000}, NULL) = 0
with ddd I get to:
src/lib-master/anvil-client.c: int anvil_client_connect(struct anvil_client
*client, bool retry) {
//[...]
fd = retry ? net_connect_unix_with_retries(client->path, 5000) :
net_connect_unix(client->path);
//[...]
and retry is forced to TRUE by the caller and net_connect_unix_with_retries
retries in case of ECONNREFUSED.
How I get into it :
ssh -X root@telegraphe-test
apt install dovecot-dbg
cd /dev/shm
apt source dovecot
cd dovecot-2.2.27
pidof dovecot
ddd /usr/sbin/dovecot
set follow-fork-mode child
attach 11833
cont
# Send an e-mail from swaks to exim that use transport=smtp in LMTP mode to
dovecot-lmtp, press interrupt while exim is blocking after RCPT TO:
(gdb) bt
#0 0x7f81528af270 in __nanosleep_nocancel () at
../sysdeps/unix/syscall-template.S:84
#1 0x7f81528d8b84 in usleep (useconds=) at
../sysdeps/posix/usleep.c:32
#2 0x7f815329368e in net_connect_unix_with_retries (path=0x562dc200ce10
,
msecs=msecs@entry=5000) at net.c:347
#3 0x7f815320ce0d in anvil_client_connect (client=0x562dc200cda0,
retry=retry@entry=true) at anvil-client.c:136
#4 0x562dc1429f5d in auth_penalty_init (path=0x562dc1451846 ) at auth-penalty.c:37
#5 0x562dc1424166 in main_preinit () at main.c:202
#6 main (argc=, argv=) at main.c:396
(gdb) bt full
#1 0x7f81528d8b84 in usleep (useconds=) at
../sysdeps/posix/usleep.c:32
ts = {tv_sec = 0, tv_nsec = 8000}
#2 0x7f815329368e in net_connect_unix_with_retries (path=0x562dc200ce10
,
msecs=msecs@entry=5000) at net.c:347
start = {tv_sec = 1545319486, tv_usec = 241864}
now = {tv_sec = 1545319490, tv_usec = 198061}
fd = -1
#3 0x7f815320ce0d in anvil_client_connect (client=0x562dc200cda0,
retry=retry@entry=true) at anvil-client.c:136
fd =
__FUNCTION__ =
#4 0x562dc1429f5d in auth_penalty_init (path=0x562dc1451846 ) at auth-penalty.c:37
No locals.
#5 0x562dc1424166 in main_preinit () at main.c:202
mod_set =
#6 main (argc=, argv=) at main.c:396
c =
Backtrace stopped: Cannot access memory at address 0x7ffda885dbf8
(gdb)
Code I've read :
src/auth/main.c: static void main_preinit(void)
if (!worker)
auth_penalty = auth_penalty_init(AUTH_PENALTY_ANVIL_PATH);
auth_request_stats_init();
src/auth/auth-penalty.c: struct auth_penalty *auth_penalty_init(const char
*path)
if (anvil_client_connect(penalty->client, TRUE) < 0)
penalty->disabled = TRUE;
else {
src/lib-master/anvil-client.c: int anvil_client_connect(struct anvil_client
*client, bool retry)
fd = retry ? net_connect_unix_with_retries(client->path, 5000) :
net_connect_unix(client->path);
src/lib/net.c : int net_connect_unix_with_retries(const char *path, unsigned
int msecs) {
struct timeval start, now;
int fd;
Re: Issue with LMTP proxying and port number
Op 06/01/2019 om 17:02 schreef Steven Craig: Thanks, its a weird one. On 1/6/2019 8:00 AM, Stephan Bosch wrote: Op 06/01/2019 om 11:50 schreef Stephan Bosch: Op 31/12/2018 om 23:59 schreef Steven Craig: Hello there, Everything was working fine on Dovecot 2.2.10 (on CentOS 7), but after updating to version 2.2.36, our director servers are not able to proxy LMTP. We are sending mail from exim to the local Dovecot LMTP socket, which then used to send it to our internal mail servers on port 24. Besides the update, nothing has changed in the config files. The configuration has been working great for months … Then we had a great idea of doing holiday maintenance. (Currently, we are working around this issue by having exim send mail to the IP and port, instead of the socket.) Doing a tcpdump capture shows no TCP traffic either. The logs seem to say that it is trying to go to port 0, so my guess is that it’s not getting the port directive from either static or sql — since the docs state that if these settings aren't specified, it will always use the same connection type for the proxy that it received the connection on. Any thoughts? I can share config if necessary. Yes, output from `dovecot -n` could be helpful. Also a literal dump of those errors in the log could help. Never mind, I can reproduce it here, even with master. We'll get back to you. Actually, this is equally problematic for Dovecot v2.2.10. I could reproduce this there too by my method (after some considerable difficulty to get that compiled on a modern system). So, first of all, I am wondering why this worked at your end in the first place and why it finally failed upon upgrade. The problem is that services like imap, pop3, managesieve, and submission define a literal default port for proxying, while lmtp really doesn't. Instead, LMTP uses the local port of the server as the default for the outgoing proxy connection. With unix services, there is no local port and hence no default, which instead results in using an (erroneous) port 0. Arguably, it would be useful to define a lmtp_proxy_port setting to have a default for situations when none other can be obtained. LMTP has no properly registered standard port, so hard-coding the default to 24 could be a bad idea. Regards, Stephan.
Re: IMAP preauth and stats-writer
and stupid me missed a character, the full fix is in SVN r489516.
On Sun, Jan 6, 2019 at 11:47 AM Larry Rosenman wrote:
> I'm the FreeBSD port maintainer for dovecot and I just added this patch to
> the port in SVN. 489515.
>
> On Sun, Jan 6, 2019 at 11:39 AM Stephan Bosch
> wrote:
>
>>
>> Op 06/01/2019 om 18:12 schreef John Fawcett:
>> > On 06/01/2019 11:37, Stephan Bosch wrote:
>> >> Op 06/01/2019 om 03:35 schreef John Fawcett:
>> >>> On 06/01/2019 02:26, John Fawcett wrote:
>> >>>
>> Can't see anything in the Dovecot 2.3.4 code that would give this
>> problem, setting
>>
>> stats_writer_socket_path =
>>
>> will overwrite the default value and dovecot does not attempt to
>> open a socket in that case.
>>
>> Using your configuration (though not FreeBSD) I don't get the
>> net_connect_unix error whether I use a blank setting or leave the
>> default. In the case of leaving the default I do get an additional
>> process (dovecot/stats). No errors on connecting to the imap service
>> or by running preauth (with the dovecot daemon already running).
>>
>> The net_connect_unix() error with a zero length socket name is
>> inexplicable to me, unless it's got a non printing character in it
>> or there is something different happening on FreeBSD.
>>
>> One suggestion is to run with the default setting, but look at
>> resolving the permission problem for the default socket creation at
>> /var/run/dovecot/stats-writer rather than working round it.
>>
>> John
>>
>> >>> Just following up, I don't get the error when I run preauth as root
>> with
>> >>> -u parameter. I do get something similar when I run as the user (this
>> >>> wil the socket path set to blank)
>> >>>
>> >>> Centos 7:
>> >>>
>> >>> Error: net_connect_unix() failed: Connection refused
>> >>>
>> >>> FreeBSD 11.2:
>> >>>
>> >>> Error: net_connect_unix() failed: No such file or directory
>> >>>
>> >>> So it's close. If I get time I'll see if I can track it down.
>> >> Does this fix it?
>> >>
>> >> diff --git a/src/lib-master/master-service.c
>> >> b/src/lib-master/master-service.c
>> >> index 3de11fa1b..3c60a7a39 100644
>> >> --- a/src/lib-master/master-service.c
>> >> +++ b/src/lib-master/master-service.c
>> >> @@ -341,7 +341,7 @@ master_service_init(const char *name, enum
>> >> master_service_flags flags,
>> >> if ((flags & MASTER_SERVICE_FLAG_DONT_SEND_STATS) == 0) {
>> >> /* Initialize stats-client early so it can see all
>> >> events. */
>> >> value = getenv(DOVECOT_STATS_WRITER_SOCKET_PATH);
>> >> - if (value != NULL)
>> >> + if (value != NULL && *value != '\0')
>> >> service->stats_client =
>> >> stats_client_init(value, FALSE);
>> >> }
>> >>
>> >>
>> >> Regards,
>> >>
>> >> Stephan.
>> >>
>> > Hi Stephan
>> >
>> > that fixes the issue that I reproduced. For the OP he will probably need
>> > to wait for this to be picked up for FreeBSD ports.
>> >
>> > This code seems safer than the original but it is still a mystery as to
>> > why DOVECOT_STATS_WRITER_SOCKET_PATH is being put into the environment
>> > as an empty string (changed behaviour reported by OP compared to
>> 2.3.2.1_1).
>>
>> Behavior changed because the code I patched didn't exist in 2.3.2.1.
>>
>> Regards,
>>
>> Stephan.
>>
>>
>
> --
> Larry Rosenman http://www.lerctr.org/~ler
> Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
> US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
>
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Re: Dovecot no logs by sieve user script
Op 06/01/2019 om 18:58 schreef Christian Wolf:
Dear mailinglist,
I have the problem that I had to migrate a setup and since then my
user's sieve script does no longer log anything.
I have sieve_user_log unset and sieve points to a folder (ready for
ManageSievev service). In the old config I had sieve point to the
symlink and sieve_dir point to the folder of all scripts. As sieve_dir
is deprecated I use the sieve directive only (see below).
According to my understanding of
https://wiki2.dovecot.org/Pigeonhole/Sieve/Configuration this should
result in user logging in ~/.dovecot.sieve.log. This file is never
generated by dovecot and if I create it manually, it is not filled.
Can you tell me, what I am missing here in order to get the per-user
logs running?
Can you enable mail_debug=yes and look at your debug log. Sieve will
tell you what directories files are being used.
Regards,
Stephan.
Thank you very much
Christian
# dovecot --version 2.2.27 (c0f36b0)
# dovecot -n # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf #
Pigeonhole version 0.4.16 (fed8554) # OS: Linux 4.9.0-8-amd64 x86_64
Debian 9.6 auth_mechanisms = plain login disable_plaintext_auth = no
mail_location = maildir:~/Mailbox managesieve_notify_capability =
mailto managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body en otify
environment mailbox date index ihave duplicate mime foreverypart
extracttext namespace inbox { inbox = yes location = mailbox Drafts
{ auto = create special_use = \Drafts } mailbox Junk { auto
= subscribe special_use = \Junk } mailbox Sent { auto =
subscribe special_use = \Sent } mailbox Trash { special_use =
\Trash } prefix = } passdb { driver = pam } passdb { args =
scheme=CRYPT username_format=%u /etc/dovecot/users driver =
passwd-file } plugin { sieve = file:~/sieve;active=~/.dovecot.sieve }
postmaster_address = postmas...@wolf-stuttgart.net protocols = " imap
sieve sieve" service auth { unix_listener
/var/spool/postfix/private/auth { mode = 0666 } } service
managesieve-login { inet_listener sieve { port = 4190 } } ssl =
required ssl_cert = = # hidden, use -P to show it userdb { driver = passwd } userdb {
args = username_format=%u /etc/dovecot/users driver = passwd-file }
protocol lda { mail_plugins = " sieve" }
Dovecot no logs by sieve user script
Dear mailinglist,
I have the problem that I had to migrate a setup and since then my user's sieve
script does
no longer log anything.
I have sieve_user_log unset and sieve points to a folder (ready for
ManageSievev service).
In the old config I had sieve point to the symlink and sieve_dir point to the
folder of all
scripts. As sieve_dir is deprecated I use the sieve directive only (see below).
According to my understanding of https://wiki2.dovecot.org/Pigeonhole/Sieve/
Configuration this should result in user logging in ~/.dovecot.sieve.log. This
file is never
generated by dovecot and if I create it manually, it is not filled.
Can you tell me, what I am missing here in order to get the per-user logs
running?
Thank you very much
Christian
# dovecot --version
# dovecot -n # 2.2.27 (c0f36b0): /etc/dovecot/dovecot.conf # Pigeonhole version
0.4.16
(fed8554) # OS: Linux 4.9.0-8-amd64 x86_64 Debian 9.6 auth_mechanisms = plain
login
disable_plaintext_auth = no mail_location = maildir:~/Mailbox
managesieve_notify_capability = mailto managesieve_sieve_capability = fileinto
reject
envelope encoded-character vacation subaddress comparator-i;ascii-numeric
relational
regex imap4flags copy include variables body enotify environment mailbox date
index
ihave duplicate mime foreverypart extracttext namespace inbox { inbox = yes
location =
mailbox Drafts {auto = createspecial_use = \Drafts } mailbox Junk {
auto =
subscribespecial_use = \Junk } mailbox Sent {auto = subscribe
special_use = \Sent }
mailbox Trash {special_use = \Trash } prefix = } passdb { driver = pam
} passdb { args
= scheme=CRYPT username_format=%u /etc/dovecot/users driver = passwd-file }
plugin {
sieve = file:~/sieve;active=~/.dovecot.sieve } postmaster_address =
postmaster@wolf-
stuttgart.net protocols = " imap sieve sieve" service auth { unix_listener
/var/spool/postfix/
private/auth {mode = 0666 } } service managesieve-login { inet_listener
sieve {port =
4190 } } ssl = required ssl_cert =
Re: IMAP preauth and stats-writer
I'm the FreeBSD port maintainer for dovecot and I just added this patch to
the port in SVN. 489515.
On Sun, Jan 6, 2019 at 11:39 AM Stephan Bosch wrote:
>
> Op 06/01/2019 om 18:12 schreef John Fawcett:
> > On 06/01/2019 11:37, Stephan Bosch wrote:
> >> Op 06/01/2019 om 03:35 schreef John Fawcett:
> >>> On 06/01/2019 02:26, John Fawcett wrote:
> >>>
> Can't see anything in the Dovecot 2.3.4 code that would give this
> problem, setting
>
> stats_writer_socket_path =
>
> will overwrite the default value and dovecot does not attempt to
> open a socket in that case.
>
> Using your configuration (though not FreeBSD) I don't get the
> net_connect_unix error whether I use a blank setting or leave the
> default. In the case of leaving the default I do get an additional
> process (dovecot/stats). No errors on connecting to the imap service
> or by running preauth (with the dovecot daemon already running).
>
> The net_connect_unix() error with a zero length socket name is
> inexplicable to me, unless it's got a non printing character in it
> or there is something different happening on FreeBSD.
>
> One suggestion is to run with the default setting, but look at
> resolving the permission problem for the default socket creation at
> /var/run/dovecot/stats-writer rather than working round it.
>
> John
>
> >>> Just following up, I don't get the error when I run preauth as root
> with
> >>> -u parameter. I do get something similar when I run as the user (this
> >>> wil the socket path set to blank)
> >>>
> >>> Centos 7:
> >>>
> >>> Error: net_connect_unix() failed: Connection refused
> >>>
> >>> FreeBSD 11.2:
> >>>
> >>> Error: net_connect_unix() failed: No such file or directory
> >>>
> >>> So it's close. If I get time I'll see if I can track it down.
> >> Does this fix it?
> >>
> >> diff --git a/src/lib-master/master-service.c
> >> b/src/lib-master/master-service.c
> >> index 3de11fa1b..3c60a7a39 100644
> >> --- a/src/lib-master/master-service.c
> >> +++ b/src/lib-master/master-service.c
> >> @@ -341,7 +341,7 @@ master_service_init(const char *name, enum
> >> master_service_flags flags,
> >> if ((flags & MASTER_SERVICE_FLAG_DONT_SEND_STATS) == 0) {
> >> /* Initialize stats-client early so it can see all
> >> events. */
> >> value = getenv(DOVECOT_STATS_WRITER_SOCKET_PATH);
> >> - if (value != NULL)
> >> + if (value != NULL && *value != '\0')
> >> service->stats_client =
> >> stats_client_init(value, FALSE);
> >> }
> >>
> >>
> >> Regards,
> >>
> >> Stephan.
> >>
> > Hi Stephan
> >
> > that fixes the issue that I reproduced. For the OP he will probably need
> > to wait for this to be picked up for FreeBSD ports.
> >
> > This code seems safer than the original but it is still a mystery as to
> > why DOVECOT_STATS_WRITER_SOCKET_PATH is being put into the environment
> > as an empty string (changed behaviour reported by OP compared to
> 2.3.2.1_1).
>
> Behavior changed because the code I patched didn't exist in 2.3.2.1.
>
> Regards,
>
> Stephan.
>
>
--
Larry Rosenman http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c) E-Mail: larry...@gmail.com
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
Re: IMAP preauth and stats-writer
Op 06/01/2019 om 18:12 schreef John Fawcett:
On 06/01/2019 11:37, Stephan Bosch wrote:
Op 06/01/2019 om 03:35 schreef John Fawcett:
On 06/01/2019 02:26, John Fawcett wrote:
Can't see anything in the Dovecot 2.3.4 code that would give this
problem, setting
stats_writer_socket_path =
will overwrite the default value and dovecot does not attempt to
open a socket in that case.
Using your configuration (though not FreeBSD) I don't get the
net_connect_unix error whether I use a blank setting or leave the
default. In the case of leaving the default I do get an additional
process (dovecot/stats). No errors on connecting to the imap service
or by running preauth (with the dovecot daemon already running).
The net_connect_unix() error with a zero length socket name is
inexplicable to me, unless it's got a non printing character in it
or there is something different happening on FreeBSD.
One suggestion is to run with the default setting, but look at
resolving the permission problem for the default socket creation at
/var/run/dovecot/stats-writer rather than working round it.
John
Just following up, I don't get the error when I run preauth as root with
-u parameter. I do get something similar when I run as the user (this
wil the socket path set to blank)
Centos 7:
Error: net_connect_unix() failed: Connection refused
FreeBSD 11.2:
Error: net_connect_unix() failed: No such file or directory
So it's close. If I get time I'll see if I can track it down.
Does this fix it?
diff --git a/src/lib-master/master-service.c
b/src/lib-master/master-service.c
index 3de11fa1b..3c60a7a39 100644
--- a/src/lib-master/master-service.c
+++ b/src/lib-master/master-service.c
@@ -341,7 +341,7 @@ master_service_init(const char *name, enum
master_service_flags flags,
if ((flags & MASTER_SERVICE_FLAG_DONT_SEND_STATS) == 0) {
/* Initialize stats-client early so it can see all
events. */
value = getenv(DOVECOT_STATS_WRITER_SOCKET_PATH);
- if (value != NULL)
+ if (value != NULL && *value != '\0')
service->stats_client =
stats_client_init(value, FALSE);
}
Regards,
Stephan.
Hi Stephan
that fixes the issue that I reproduced. For the OP he will probably need
to wait for this to be picked up for FreeBSD ports.
This code seems safer than the original but it is still a mystery as to
why DOVECOT_STATS_WRITER_SOCKET_PATH is being put into the environment
as an empty string (changed behaviour reported by OP compared to 2.3.2.1_1).
Behavior changed because the code I patched didn't exist in 2.3.2.1.
Regards,
Stephan.
Re: IMAP preauth and stats-writer
On 06/01/2019 11:37, Stephan Bosch wrote:
>
> Op 06/01/2019 om 03:35 schreef John Fawcett:
>> On 06/01/2019 02:26, John Fawcett wrote:
>>
>>> Can't see anything in the Dovecot 2.3.4 code that would give this
>>> problem, setting
>>>
>>> stats_writer_socket_path =
>>>
>>> will overwrite the default value and dovecot does not attempt to
>>> open a socket in that case.
>>>
>>> Using your configuration (though not FreeBSD) I don't get the
>>> net_connect_unix error whether I use a blank setting or leave the
>>> default. In the case of leaving the default I do get an additional
>>> process (dovecot/stats). No errors on connecting to the imap service
>>> or by running preauth (with the dovecot daemon already running).
>>>
>>> The net_connect_unix() error with a zero length socket name is
>>> inexplicable to me, unless it's got a non printing character in it
>>> or there is something different happening on FreeBSD.
>>>
>>> One suggestion is to run with the default setting, but look at
>>> resolving the permission problem for the default socket creation at
>>> /var/run/dovecot/stats-writer rather than working round it.
>>>
>>> John
>>>
>> Just following up, I don't get the error when I run preauth as root with
>> -u parameter. I do get something similar when I run as the user (this
>> wil the socket path set to blank)
>>
>> Centos 7:
>>
>> Error: net_connect_unix() failed: Connection refused
>>
>> FreeBSD 11.2:
>>
>> Error: net_connect_unix() failed: No such file or directory
>>
>> So it's close. If I get time I'll see if I can track it down.
>
> Does this fix it?
>
> diff --git a/src/lib-master/master-service.c
> b/src/lib-master/master-service.c
> index 3de11fa1b..3c60a7a39 100644
> --- a/src/lib-master/master-service.c
> +++ b/src/lib-master/master-service.c
> @@ -341,7 +341,7 @@ master_service_init(const char *name, enum
> master_service_flags flags,
> if ((flags & MASTER_SERVICE_FLAG_DONT_SEND_STATS) == 0) {
> /* Initialize stats-client early so it can see all
> events. */
> value = getenv(DOVECOT_STATS_WRITER_SOCKET_PATH);
> - if (value != NULL)
> + if (value != NULL && *value != '\0')
> service->stats_client =
> stats_client_init(value, FALSE);
> }
>
>
> Regards,
>
> Stephan.
>
Hi Stephan
that fixes the issue that I reproduced. For the OP he will probably need
to wait for this to be picked up for FreeBSD ports.
This code seems safer than the original but it is still a mystery as to
why DOVECOT_STATS_WRITER_SOCKET_PATH is being put into the environment
as an empty string (changed behaviour reported by OP compared to 2.3.2.1_1).
The function that is doing the env_put call with the empty string is
config_request_putenv from src/config/doveconf.c.
John
Re: Issue with LMTP proxying and port number
Op 31/12/2018 om 06:32 schreef Laz C. Peterson: Hello Sami, yes, see below. We run Dovecot at a different versions, mainly 2.2.10 (CentOS), 2.2.22 (Ubuntu) and now 2.2.36 (CentOS). The issue is weird, because it only happened after the update from 2.2.10->36. Just to understand it would be great. I'm actually checking out the configs now ... Our SQL userdb does not specify port. So I'm guessing this may be to blame? (This was by design, though -- we don't want to specify one port for different client protocols. Though, I do recall seeing some hack online using CASE in SQL query ...) These servers run LMTP as a unix socket as well as a TCP port 24 serving all IP sources. The internal servers are running LMTP on TCP port 24 (as well as unix socket, but that's irrelevant), but no LMTP comm happens between directors and backend mail servers after the 2.2.10->36 update on the directors with our config. I do apologize that I can't get more specific than those versions ... The backend mail servers function the same in our environment on both versions 2.2.10 and 2.2.36. We are good now, as we changed the config to go to the TCP port instead of unix socket. But we had a good jolt of fun this morning. :-) Would love to understand what we have done wrong, or how we misunderstood the configuration directives -- in either version. I can reproduce it here, even with master. We'll get back to you. BTW, similar thread here: https://www.dovecot.org/pipermail/dovecot/2019-January/114071.html Regards, Stephan.
Re: Issue with LMTP proxying and port number
Thanks, its a weird one. On 1/6/2019 8:00 AM, Stephan Bosch wrote: Op 06/01/2019 om 11:50 schreef Stephan Bosch: Op 31/12/2018 om 23:59 schreef Steven Craig: Hello there, Everything was working fine on Dovecot 2.2.10 (on CentOS 7), but after updating to version 2.2.36, our director servers are not able to proxy LMTP. We are sending mail from exim to the local Dovecot LMTP socket, which then used to send it to our internal mail servers on port 24. Besides the update, nothing has changed in the config files. The configuration has been working great for months … Then we had a great idea of doing holiday maintenance. (Currently, we are working around this issue by having exim send mail to the IP and port, instead of the socket.) Doing a tcpdump capture shows no TCP traffic either. The logs seem to say that it is trying to go to port 0, so my guess is that it’s not getting the port directive from either static or sql — since the docs state that if these settings aren't specified, it will always use the same connection type for the proxy that it received the connection on. Any thoughts? I can share config if necessary. Yes, output from `dovecot -n` could be helpful. Also a literal dump of those errors in the log could help. Never mind, I can reproduce it here, even with master. We'll get back to you. Regards, Stephan.
Re: Issue with LMTP proxying and port number
Op 06/01/2019 om 11:50 schreef Stephan Bosch: Op 31/12/2018 om 23:59 schreef Steven Craig: Hello there, Everything was working fine on Dovecot 2.2.10 (on CentOS 7), but after updating to version 2.2.36, our director servers are not able to proxy LMTP. We are sending mail from exim to the local Dovecot LMTP socket, which then used to send it to our internal mail servers on port 24. Besides the update, nothing has changed in the config files. The configuration has been working great for months … Then we had a great idea of doing holiday maintenance. (Currently, we are working around this issue by having exim send mail to the IP and port, instead of the socket.) Doing a tcpdump capture shows no TCP traffic either. The logs seem to say that it is trying to go to port 0, so my guess is that it’s not getting the port directive from either static or sql — since the docs state that if these settings aren't specified, it will always use the same connection type for the proxy that it received the connection on. Any thoughts? I can share config if necessary. Yes, output from `dovecot -n` could be helpful. Also a literal dump of those errors in the log could help. Never mind, I can reproduce it here, even with master. We'll get back to you. Regards, Stephan.
Re: Localhost imap server fails after upgrade from 2.2.33.2-1 to 1:2.3.2.1
Hello Stephan,
I'm interested to see this get fixed (and ideally one day for it to be
really easy for people to set up Alpine so that it competes fairly with
heavier GUIs!), but I do not know that I am qualified to help (beyond
light testing). Is there someone from Alpine on this thread? The second
(askubuntu.com) link you give is my own question (and answer, based on Aki
Tuomi's earlier responses).
Thanks and best wishes,
Chris
On Sun, 6 Jan 2019, Stephan Bosch wrote:
>
> Op 31/12/2018 om 16:53 schreef Chris Barrington-Leigh:
> > I then followed the instructions in the Warning lines above and now I get:
> >
> > dovecot -n -c /home/meuser/dotfiles/auto-stanford-dovecot.conf
> > # 2.3.2.1 (0719df592): /home/meuser/dotfiles/auto-stanford-dovecot.conf
> > # Pigeonhole version 0.5.2 ()
> > # OS: Linux 4.18.0-13-generic x86_64 Ubuntu 18.10 ext4
> > # Hostname: cpbl-t450s
> > mail_location =
> > maildir:/home/meuser/imap/gmailStanford:LAYOUT=fs:INBOX=/home/meuser/imap/gmailStanford/INBOX
> > service stats {
> > unix_listener stats-writer {
> > mode = 0666
> > }
> > }
> > ssl_dh = # hidden, use -P to show it
> > verbose_proctitle = yes
> >
> >
> > However, this does not help my primary symptom:
> >
> >
> > $ /usr/lib/dovecot/imap -c /home/meuser/dotfiles/auto-stanford-dovecot.conf
> > imap(meuser,)Error: net_connect_unix(/var/run/dovecot/stats-writer) failed:
> > Permission denied
> > * PREAUTH [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT
> > SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND
> > URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED
> > I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH
> > LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+ NOTIFY] Logged in as meuser
> >
>
> I notice that you use an explicit configuration file for your imap invocation.
> Which likely means that this is something different from the config used the
> main dovecot service running on your system. That service block needs to be in
> the configuration of the dovecot instance running the stats service (which is
> likely just the main /etc/dovecot/dovecot.conf). Given the location of your
> own config file, I'm assuming you cannot manage the main dovecot instance?
>
> BTW, there is a related thread with a different solution to this issue:
>
> https://www.dovecot.org/pipermail/dovecot/2019-January/114151.html
>
> We're still working on getting that bug fixed, so that will not help you
> immediately, unless you can patch dovecot.
>
> BTW, I find it a bit weird that Alpine would fail on this. It is not a fatal
> error, so the imap service will just work. The error is sent to /dev/stderr,
> so I'd normally expect a service like Alpine to be able to ignore output sent
> there.
>
> The question below is about the same issue and their config suggests that
> stderr is sent to a log file:
>
> https://askubuntu.com/questions/1104056/how-to-use-local-dovecot-imap-with-alpine-18-10-changes
>
> So, I really don't understand what the core of the problem is. Why does Alpine
> fail on this at all?
>
> Regards,
>
> Stephan.
>
>
>
>
Re: Localhost imap server fails after upgrade from 2.2.33.2-1 to 1:2.3.2.1
Op 31/12/2018 om 16:53 schreef Chris Barrington-Leigh:
I then followed the instructions in the Warning lines above and now I
get:
dovecot -n -c /home/meuser/dotfiles/auto-stanford-dovecot.conf
# 2.3.2.1 (0719df592): /home/meuser/dotfiles/auto-stanford-dovecot.conf
# Pigeonhole version 0.5.2 ()
# OS: Linux 4.18.0-13-generic x86_64 Ubuntu 18.10 ext4
# Hostname: cpbl-t450s
mail_location =
maildir:/home/meuser/imap/gmailStanford:LAYOUT=fs:INBOX=/home/meuser/imap/gmailStanford/INBOX
service stats {
unix_listener stats-writer {
mode = 0666
}
}
ssl_dh = # hidden, use -P to show it
verbose_proctitle = yes
However, this does not help my primary symptom:
$ /usr/lib/dovecot/imap -c
/home/meuser/dotfiles/auto-stanford-dovecot.conf
imap(meuser,)Error: net_connect_unix(/var/run/dovecot/stats-writer)
failed: Permission denied
* PREAUTH [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES
WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+
NOTIFY] Logged in as meuser
I notice that you use an explicit configuration file for your imap
invocation. Which likely means that this is something different from the
config used the main dovecot service running on your system. That
service block needs to be in the configuration of the dovecot instance
running the stats service (which is likely just the main
/etc/dovecot/dovecot.conf). Given the location of your own config file,
I'm assuming you cannot manage the main dovecot instance?
BTW, there is a related thread with a different solution to this issue:
https://www.dovecot.org/pipermail/dovecot/2019-January/114151.html
We're still working on getting that bug fixed, so that will not help you
immediately, unless you can patch dovecot.
BTW, I find it a bit weird that Alpine would fail on this. It is not a
fatal error, so the imap service will just work. The error is sent to
/dev/stderr, so I'd normally expect a service like Alpine to be able to
ignore output sent there.
The question below is about the same issue and their config suggests
that stderr is sent to a log file:
https://askubuntu.com/questions/1104056/how-to-use-local-dovecot-imap-with-alpine-18-10-changes
So, I really don't understand what the core of the problem is. Why does
Alpine fail on this at all?
Regards,
Stephan.
Re: Issue with LMTP proxying and port number
Op 31/12/2018 om 23:59 schreef Steven Craig: Hello there, Everything was working fine on Dovecot 2.2.10 (on CentOS 7), but after updating to version 2.2.36, our director servers are not able to proxy LMTP. We are sending mail from exim to the local Dovecot LMTP socket, which then used to send it to our internal mail servers on port 24. Besides the update, nothing has changed in the config files. The configuration has been working great for months … Then we had a great idea of doing holiday maintenance. (Currently, we are working around this issue by having exim send mail to the IP and port, instead of the socket.) Doing a tcpdump capture shows no TCP traffic either. The logs seem to say that it is trying to go to port 0, so my guess is that it’s not getting the port directive from either static or sql — since the docs state that if these settings aren't specified, it will always use the same connection type for the proxy that it received the connection on. Any thoughts? I can share config if necessary. Yes, output from `dovecot -n` could be helpful. Also a literal dump of those errors in the log could help. Regards, Stephan.
Re: IMAP preauth and stats-writer
Op 06/01/2019 om 03:35 schreef John Fawcett:
On 06/01/2019 02:26, John Fawcett wrote:
Can't see anything in the Dovecot 2.3.4 code that would give this
problem, setting
stats_writer_socket_path =
will overwrite the default value and dovecot does not attempt to open a socket
in that case.
Using your configuration (though not FreeBSD) I don't get the net_connect_unix
error whether I use a blank setting or leave the default. In the case of
leaving the default I do get an additional process (dovecot/stats). No errors
on connecting to the imap service or by running preauth (with the dovecot
daemon already running).
The net_connect_unix() error with a zero length socket name is inexplicable to
me, unless it's got a non printing character in it or there is something
different happening on FreeBSD.
One suggestion is to run with the default setting, but look at resolving the
permission problem for the default socket creation at
/var/run/dovecot/stats-writer rather than working round it.
John
Just following up, I don't get the error when I run preauth as root with
-u parameter. I do get something similar when I run as the user (this
wil the socket path set to blank)
Centos 7:
Error: net_connect_unix() failed: Connection refused
FreeBSD 11.2:
Error: net_connect_unix() failed: No such file or directory
So it's close. If I get time I'll see if I can track it down.
Does this fix it?
diff --git a/src/lib-master/master-service.c
b/src/lib-master/master-service.c
index 3de11fa1b..3c60a7a39 100644
--- a/src/lib-master/master-service.c
+++ b/src/lib-master/master-service.c
@@ -341,7 +341,7 @@ master_service_init(const char *name, enum
master_service_flags flags,
if ((flags & MASTER_SERVICE_FLAG_DONT_SEND_STATS) == 0) {
/* Initialize stats-client early so it can see all
events. */
value = getenv(DOVECOT_STATS_WRITER_SOCKET_PATH);
- if (value != NULL)
+ if (value != NULL && *value != '\0')
service->stats_client =
stats_client_init(value, FALSE);
}
Regards,
Stephan.
Re: Solr -> Xapian ?
and finally , for fts_backend__lookup_multi, why is that backend
dependent ?
Would- nt the below function below be the same for any backend ?
Waiting fro your feedback on all those questions
Thank you
JM
-
static int fts_backend_xapian_lookup_multi(struct fts_backend *_backend,
struct mailbox *const boxes[], struct mail_search_arg *args, enum
fts_lookup_flags flags, struct fts_multi_result *result)
{
struct xapian_fts_backend_update_context *ctx =
(struct xapian_fts_backend_update_context *)_ctx;
int i=0;
while(boxes[i]!=NULL)
{
if(fts_backend_xapian_lookup(backend,box[i],args,flags,result->box_results[i])<0)
return -1;
i++;
}
return 0;
}
On 2019-01-06 16:31, Joan Moreau via dovecot wrote:
for fts_backend_xxx_lookup, where is specidifed in which field (to, cc,
subject, body, from, all) to lookup ?
On 2019-01-06 16:03, Joan Moreau wrote:
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management
of the expunged messages is *NOT* in the backend, right ?
On 2019-01-06 15:41, Joan Moreau wrote:
also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those
functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */
int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box,
uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this
returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to
index this key, it can return FALSE and caller will skip to next key. */
bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx,
const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key.
Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters,
but it doesn't need to be NUL-terminated. size contains the data size in
bytes, not characters. This function may be called many times and the data
block sizes may be small. Backend returns 0 if ok, -1 if build should be
aborted. */
int fts_backend_update_build_more(struct fts_backend_update_context *ctx,
const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a
feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups.
Returns 0 if ok, -1 if error. */
int fts_backend_refresh(struct fts_backend *backend);
/* Go through the entire index and make sure all mails are indexed,
and delete any extra mails in the index. */
int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the
fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are
documented. The remaining questions can be answered by looking at examples
found in
Re: Solr -> Xapian ?
for fts_backend_xxx_lookup, where is specidifed in which field (to, cc,
subject, body, from, all) to lookup ?
On 2019-01-06 16:03, Joan Moreau wrote:
For "rescan " and "optimize", wouldn't it be the dovecot core who indicate which are to be dismissed (expunged), or re-ask for indexing a particular (or all) uid ? WHy would the backend be aware of the transactions on the mailbox ???
There is alredy "fts_backend_xxx_update_expunge", so I beleive the management
of the expunged messages is *NOT* in the backend, right ?
On 2019-01-06 15:41, Joan Moreau wrote:
also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those
functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */
int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box,
uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this
returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to
index this key, it can return FALSE and caller will skip to next key. */
bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx,
const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key.
Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters,
but it doesn't need to be NUL-terminated. size contains the data size in
bytes, not characters. This function may be called many times and the data
block sizes may be small. Backend returns 0 if ok, -1 if build should be
aborted. */
int fts_backend_update_build_more(struct fts_backend_update_context *ctx,
const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a
feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups.
Returns 0 if ok, -1 if error. */
int fts_backend_refresh(struct fts_backend *backend);
/* Go through the entire index and make sure all mails are indexed,
and delete any extra mails in the index. */
int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the
fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are
documented. The remaining questions can be answered by looking at examples
found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a
small effort though. As soon as you have concrete questions, we can help you
(don't expect rapid responses though).
Regards,
Stephan.
reprocessing mail
Hey guys, I had some corruptions whereby dovecot would bomb out on mail files in my maildir. I found that if piped the files to dovecot-lda the all email was clean I even retrieved email I thought might have been lost for good. I have some better ideas now, but I still processed a few thousand emails like that now & would like to do something like this: * stop dovecot * iterate over each file ** get date ** touch date (linux - to set the date/time) ** rename the file with the correct epoch ** update dovecot-uidlist with correct epoch * start dovecot Will that work? Or is there something I've missed or should do? - Marc
Re: Solr -> Xapian ?
For "rescan " and "optimize", wouldn't it be the dovecot core who
indicate which are to be dismissed (expunged), or re-ask for indexing a
particular (or all) uid ? WHy would the backend be aware of the
transactions on the mailbox ???
There is alredy "fts_backend_xxx_update_expunge", so I beleive the
management of the expunged messages is *NOT* in the backend, right ?
On 2019-01-06 15:41, Joan Moreau wrote:
also, for fts_backend_solr_update_set_build_key -> where is the data (of the hdr_name or the body) ?
On 2019-01-06 14:10, Joan Moreau wrote:
for the "last uid"-> this is not the last added, but the maximum of the UID in the indexed emails, right ?
On 2019-01-06 11:53, Joan Moreau via dovecot wrote:
Thank you
I still don't get the "build_key" function. The email (body, hearders, .. and the uid) is the one (and only) to index . What "key" is that function referring to ? Or is the "key" here the actual email ?
On 2019-01-06 08:43, Stephan Bosch wrote:
Op 06/01/2019 om 01:00 schreef Joan Moreau: Anyone willing to explain those
functions ?
Most notably " get_last_uid"
From src/plugins/fts/fts-api.h:
/* Get the last_uid for the mailbox. */
int fts_backend_get_last_uid(struct fts_backend *backend, struct mailbox *box,
uint32_t *last_uid_r);
The solr sources ( src/plugins/fts-solr/fts-backend-solr.c:213) tell me this
returns the last UID added to the index for the given mailbox and FTS index.
"set_build_key"
From src/plugins/fts/fts-api.h:
/* Switch to building index for specified key. If backend doesn't want to
index this key, it can return FALSE and caller will skip to next key. */
bool fts_backend_update_set_build_key(struct fts_backend_update_context *ctx,
const struct fts_backend_build_key *key);
Same file provides outline of what a build_key is.
"build_more" ,
/* Add more content to the index for the currently specified build key.
Non-BODY_PART_BINARY data must contain only full valid UTF-8 characters,
but it doesn't need to be NUL-terminated. size contains the data size in
bytes, not characters. This function may be called many times and the data
block sizes may be small. Backend returns 0 if ok, -1 if build should be
aborted. */
int fts_backend_update_build_more(struct fts_backend_update_context *ctx,
const unsigned char *data, size_t size);
You should look at the sources of a few backends like squat and solr to get a
feel of what exactly this is doing.
what is refresh versus rescan ?
From fts-api.h:
/* Refresh index to make sure we see latest changes from lookups.
Returns 0 if ok, -1 if error. */
int fts_backend_refresh(struct fts_backend *backend);
/* Go through the entire index and make sure all mails are indexed,
and delete any extra mails in the index. */
int fts_backend_rescan(struct fts_backend *backend);
Regards,
Stepham
On January 5, 2019 14:23:10 Joan Moreau via dovecot wrote:
Thank Stephan
I basically need to know the role/description of each of the functions of the
fts_backend:
struct fts_backend fts_backend_xapian = {
.name = "xapian",
.flags = FTS_BACKEND_FLAG_NORMALIZE_INPUT,*-> what other flags ?*
{
fts_backend_xapian_alloc,
fts_backend_xapian_init,
fts_backend_xapian_deinit,
fts_backend_xapian_get_last_uid,
fts_backend_xapian_update_init,
fts_backend_xapian_update_deinit,
fts_backend_xapian_update_set_mailbox,
fts_backend_xapian_update_expunge,
fts_backend_xapian_update_set_build_key,
fts_backend_xapian_update_unset_build_key,
fts_backend_xapian_update_build_more,
fts_backend_xapian_refresh,
fts_backend_xapian_rescan,
fts_backend_xapian_optimize,
fts_backend_default_can_lookup,
fts_backend_xapian_lookup,
fts_backend_xapian_lookup_multi,
fts_backend_xapian_lookup_done
}
};
THank you
On 2019-01-05 08:49, Stephan Bosch wrote:
Op 04/01/2019 om 11:17 schreef Joan Moreau via dovecot:
Why not, but please guide me about the core structure (mandatory funcitons, etc..) of a typical Dovecot FTS plugin
The Dovecot API documentation is not exhaustive everywhere, but the basics are
documented. The remaining questions can be answered by looking at examples
found in similar plugins or the relevant API sources.
I know of one FTS plugin not written by Dovecot developers:
https://github.com/atkinsj/fts-elasticsearch
If you really wish to do something like this, just go ahead. It will not be a
small effort though. As soon as you have concrete questions, we can help you
(don't expect rapid responses though).
Regards,
Stephan.
