Re: sieve mail date change

[John Fawcett via dovecot]

On 22/07/2024 21:03, Marc via dovecot wrote:

I am a bit rookie with sieve. Some time ago I created a sieve pipe that would 
sometimes alter alter the subject and move a message to a different mailbox. 
Currently this results in that a date changes of this message while I don't see 
anything in the source altering a date.

What could be causing this and how to work around this?
___


Hi Marc

can you show info that might be useful to understand the issue you are 
facing? For example, what are the contents of the sieve script? When is 
the script being executed (for example on mail delivery by lmtp/lda)?  
When you say the date changes of the message can you give an example of 
what you are seeing?


John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Sieve duplicate detection not working as expected

[John Fawcett via dovecot]

On 18/07/2024 15:37, Rosario Esposito via dovecot wrote:


Hi,

On a dovecot server (2.3.21) I have a simple sieve script:

require ["duplicate"];
# rule:[duplicate]
if duplicate {
    discard;
    stop;
}


I use virtual users with mail aliases on a LDAP backend.

I have user "resposit" having 2 mail aliases:

- respo...@na.infn.it
- rosario.espos...@na.infn.it

If I send a single message to both aliases I still receive 2 copies as 
you can see from the attached log.


Is the "duplicate" sieve extension supposed to work this way ?

Any help would be much appreciated !



Hi Rosario

I guess the duplicate db lookup key includes both message id and 
username so the same message id for different users will not be flagged 
as a duplicate.


If you define the aliases in the mta rather than via the dovecot ldap 
lookup, so that the username that comes into dovecot is the same one for 
both aliases I suspect it will work.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: doveadm: SSL handshake failed - wrong version number

[John Fawcett via dovecot]

On 18/07/2024 14:41, Rachel Roch via dovecot wrote:

When attempting a doveadm backup, e.g. :

doveadm -D backup -Ru user tcp:foo.example.com:12345


Hi Rachel

maybe that should be with tcps

doveadm -D backup -Ru user tcps:foo.example.com:12345

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Invalid/Unmapped Key with Last Login Plugin

[John Fawcett via dovecot]

On 17/07/2024 23:35, Adam Miller via dovecot wrote:
Yes, I did see that, thank you. If I did want to get this working 
globally, do you know what the issue is here and how to resolve it? I 
think it has something to do with my database table but I am not 
positive what the issue is. Here’s my database table (MySQL):


CREATE TABLE `mailboxLastLogin` (
  `email` varchar(192) NOT NULL DEFAULT ‘’,
  `service` varchar(16) NOT NULL DEFAULT ‘’,
  `timestamp` bigint(20) NOT NULL,
  `ipaddress` varchar(40) NOT NULL,
  `createdAt` datetime NOT NULL DEFAULT current_timestamp(),
  PRIMARY KEY (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

I also have two triggers on this table so when a new row is inserted 
or an existing row is updated, a new row is inserted into another 
tables named `mailboxHistory`. This is working fine and should not be 
causing any issues with LMTP trying to query itself.


BEGIN
    DECLARE mailboxUuid VARCHAR(36);

    SELECT uuid INTO mailboxUuid FROM mailbox WHERE email = NEW.email;

    INSERT INTO mailboxHistory (`mailboxUuid`, `service`, `command`, 
`ipaddress`, `uuid`)

    VALUES (mailboxUuid, NEW.service, 'login', NEW.ipaddress, uuidv4());
END

BEGIN
    DECLARE mailboxUuid VARCHAR(36);

    SELECT uuid INTO mailboxUuid FROM mailbox WHERE email = NEW.email;

    INSERT INTO mailboxHistory (`mailboxUuid`, `service`, `command`, 
`ipaddress`, `uuid`)

    VALUES (mailboxUuid, NEW.service, 'login', NEW.ipaddress, uuidv4());
END

CREATE TABLE `mailboxHistory` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `uuid` varchar(36) DEFAULT ‘’,
  `mailboxUuid` varchar(36) NOT NULL DEFAULT ‘’,
  `service` varchar(16) NOT NULL,
  `command` varchar(36) DEFAULT NULL,
  `extra` longtext DEFAULT NULL,
  `ipaddress` varchar(40) NOT NULL,
  `createdAt` datetime NOT NULL DEFAULT current_timestamp(),
  PRIMARY KEY (`id`),
  UNIQUE KEY `UUID` (`uuid`)
) ENGINE=InnoDB AUTO_INCREMENT=2047 DEFAULT CHARSET=utf8mb4 
COLLATE=utf8mb4_general_ci;


Thank you,

Adam



Hi Adam

the solution is to change

mail_plugins = " quota mailbox_alias last_login"

to

mail_plugins = quota mailbox_alias

(without the last_login)

The last_login should be configured inside the protocols where you need 
it, like imap. I don't think it's useful to have the plugin setup for 
services different to imap, pop3 and smtp where you have users logging 
in. If you want to have last_login for lmtp without that error, I'd need 
a bit more info


What are the contents of /etc/dovecot/inc.d/inc.lastlogin.sql.conf 
(masking any passwords) and what is the output of doveconf 
plugin/last_login_key


John



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Invalid/Unmapped Key with Last Login Plugin

[John Fawcett via dovecot]

On 17/07/2024 22:59, Adam Miller via dovecot wrote:
Yikes, this is my bad! I AM using the LMTP service for delivery. What 
I meant was that I do not necessarily need to track the logins to the 
LMTP service. I suppose it wouldn’t hurt to track them though. Any 
idea why it would be throwing this error? Which key is it expecting to 
be mapped? Am I missing a configuration specifically for this? Should 
the last login for LMTP use the same dict as the IMAP configuration?



Hi Adam

not sure if you saw it but in my earlier response I gave the info about 
excluding last_login plugin for services where it is not needed. You 
have it configured globally at the moment.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Invalid/Unmapped Key with Last Login Plugin

[John Fawcett via dovecot]

On 17/07/2024 16:03, dovecot--- via dovecot wrote:
Hello all! I have the last login plugin working with the IMAP service 
however with LMTP, it errors.
I do not necessarily need / want the LMTP service and did not 
explicitly enable it but it is throwing an error in my logs.


Jul 17 10:22:14 lmtp(184051):Error: last_login_dict: Failed to write 
value: dict-server returned failure: sql dict set: Invalid/unmapped 
key:shared/last-login/lmtp/usern...@domain.com/ (reply took 0.234 
secs (0.001 in dict wait, 0.033 in other ioloops, 0.030 in locks, as 
ync-id reply 0.000 secs ago, started on dict-server 0.220 secs ago, 
took 0.000 secs))


It would be nice to get rid of this error one or another: either 
disable last login for LMTP or get it working.


Current Configuration from doveconf -n

...

protocols = imap lmtp

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    group = postfix
    mode = 0666
    user = postfix
  }
}



But LMTP is being explicitly enabled in your config. Remove it from 
the protocols= line. And optionally remove the service {...} block for 
it. However just taking it out of protocols should be good enough.



Hi Adam

My recommendation, before disabling lmtp, would be to investigate and 
maybe revise this point "I do not necessarily need / want the LMTP 
service and did not explicitly enable it". That error is being generated 
by a login to lmtp. So what is logging in despite you not needing lmtp?


If for example you are using lmtp to deliver email from postfix to 
dovecot, disabling lmtp will stop the flow of new email.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Invalid/Unmapped Key with Last Login Plugin

[John Fawcett via dovecot]

On 17/07/2024 12:28, Adam Miller via dovecot wrote:

OS: Ubuntu 22.04.1
Dovecot: 2.3.16 (7e2e900c1a)

Hello all! I have the last login plugin working with the IMAP service 
however with LMTP, it errors. I do not necessarily need / want the 
LMTP service and did not explicitly enable it but it is throwing an 
error in my logs.


Jul 17 10:22:14 lmtp(184051):Error: last_login_dict: Failed to write 
value: dict-server returned failure: sql dict set: Invalid/unmapped 
key:shared/last-login/lmtp/usern...@domain.com/ (reply took 0.234 secs 
(0.001 in dict wait, 0.033 in other ioloops, 0.030 in locks, as


ync-id reply 0.000 secs ago, started on dict-server 0.220 secs ago, 
took 0.000 secs))




It would be nice to get rid of this error one or another: either 
disable last login for LMTP or get it working.


Current Configuration from doveconf -n

# 2.3.16 (7e2e900c1a): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.16 (09c29328)
# OS: Linux 6.5.0-1020-aws x86_64 Ubuntu 22.04.4 LTS nfs4
# Hostname: ec2-us-east-1a-arda-mail-001.ardaemail.com

mail_plugins = " quota mailbox_alias last_login"

protocol imap {
  mail_plugins = " quota mailbox_alias last_login imap_sieve 
imap_quota last_login"

}
protocol lmtp {
  mail_plugins = " quota mailbox_alias last_login sieve"
}

In the configuration above, I do not have "last_login" in my actual 
config for "protocol lmtp", something is adding it.




Hi Adam

maybe you do have last_login enabled for lmtp without realizing it. What 
is your actual setting for mail_plugins in the protocol lmtp section?


If it something like:

    mail_plugins = $mail_plugins sieve

then you are adding the following definition too from the general definition

mail_plugins = " quota mailbox_alias last_login"

So it would make sense to remove last_login from the general 
mail_plugins setting and add it only to the protocol specific 
mail_plugins section for the protocols where you want it be used.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dictionary with %d variable in sql stops insert and select domain part - dovecot 2.3.17+

[John Fawcett via dovecot]

On 16/07/2024 08:53, Mariusz Piasecki via dovecot wrote:

Hi,

unfortunately no :(

In sql query instead of variable %d I get user

UPDATE used_domain_quota SET bytes=bytes+4359,messages=messages+1 
WHERE domain

= 'h.u...@example.com';

and domain can receive emails when over quota.

On 15.07.2024 20:15, yoh2010--- via dovecot wrote:

Mariusz Piasecki, hello.

I have a similar problem after migrating from Debian 10 (Dovecot 
2.3.4) to Debian 12 (Dovecot 2.3.19). Have you found a way to solve 
the problem?


Best regards, Evgeniy Rusachenko
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi looks like a bug introduced as an unintended regression between 
2.3.16 and 2.3.17. In 2.3.17 the username field is always taken as the 
default i.e. username. The %d override, though it is read when parsing 
the config is not used for the lookup.


The quota dict is going away in 2.4 which is a strong indication that no 
patch would be accepted, since the code is no longer present in the main 
branch. You'll probably want to start looking into quota count plugin 
and quota clone plugin if you still want to publish the quota to mysql 
tables. I don't know if it's possible to use a domain quota with quota 
count plugin. I didn't see an explicitly documented way.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: doveadm auth lookup fails for system user

[John Fawcett via dovecot]

veadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() 
failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: 
undefined symbol: lucene_index_iter_deinit (this is usually 
intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: 
undefined symbol: fts_user_get_language_list (this is usually 
intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() 
failed: 
/usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: 
undefined symbol: mail_crypt_box_get_pvt_digests (this is usually 
intentional, so just ignore this message)
Jul 15 19:45:44 Debug: Loading modules from directory: 
/usr/lib/dovecot/modules/doveadm
Jul 15 19:45:44 Debug: Skipping module doveadm_acl_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: 
undefined symbol: acl_user_module (this is usually intentional, so 
just ignore this message)
Jul 15 19:45:44 Debug: Skipping module doveadm_quota_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: 
undefined symbol: quota_user_module (this is usually intentional, so 
just ignore this message)
Jul 15 19:45:44 Debug: Skipping module doveadm_fts_lucene_plugin, 
because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: 
undefined symbol: lucene_index_iter_deinit (this is usually 
intentional, so just ignore this message)
Jul 15 19:45:44 Debug: Skipping module doveadm_fts_plugin, because 
dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: 
undefined symbol: fts_user_get_language_list (this is usually 
intentional, so just ignore this message)
Jul 15 19:45:44 Debug: Skipping module doveadm_mail_crypt_plugin, 
because dlopen() failed: 
/usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: 
undefined symbol: mail_crypt_box_get_pvt_digests (this is usually 
intentional, so just ignore this message)
Jul 15 19:45:44 Debug: auth-master: passdb lookup(qno): Started passdb 
lookup
Jul 15 19:45:44 Debug: auth-master: conn 
unix:/run/dovecot/auth-userdb: Connecting
Jul 15 19:45:44 Debug: auth-master: conn unix:/run/dovecot/auth-userdb 
(pid=2542,uid=0): Client connected (fd=9)
Jul 15 19:45:44 Debug: auth-master: passdb lookup(qno): auth PASS 
input: user=qno
Jul 15 19:45:44 Debug: auth-master: passdb lookup(qno): Finished 
passdb lookup (user=qno )

passdb: qno
  user  : qno

Jul 15 19:45:44 Debug: auth-master: conn unix:/run/dovecot/auth-userdb 
(pid=2542,uid=0): Disconnected: Connection closed (fd=9)



Am 13.07.2024 um 13:01 schrieb John Fawcett via dovecot:


On 12/07/2024 21:47, Christian H. Kuhn via dovecot wrote:

Hi all,

next step with my auth problem with dovecot.

I want to authenticate a system user. The user exists, can log in, 
can sudo -i etc.pp. SASL with sql passdb and userdb works fine.


root@bywater /etc/dovecot/conf.d # doveadm user qno
field   value
uid 1001
gid 1001
home    /home/qno
mail    maildir:~/Maildir
system_groups_user  qno

But:
root@bywater /etc/dovecot/conf.d # doveadm auth lookup qno
passdb lookup: user qno doesn't exist

And no surprise:
root@bywater /etc/dovecot/conf.d # doveadm auth test qno
Password:
passdb: qno auth failed
extra fields:
  user=qno


Hi QNo

I can't see why this happening. Only suggestion I have is to run 
those commands with -D to get more detailed info.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: doveadm auth lookup fails for system user

[John Fawcett via dovecot]

On 12/07/2024 21:47, Christian H. Kuhn via dovecot wrote:

Hi all,

next step with my auth problem with dovecot.

I want to authenticate a system user. The user exists, can log in, can 
sudo -i etc.pp. SASL with sql passdb and userdb works fine.


root@bywater /etc/dovecot/conf.d # doveadm user qno
field   value
uid 1001
gid 1001
home    /home/qno
mail    maildir:~/Maildir
system_groups_user  qno

But:
root@bywater /etc/dovecot/conf.d # doveadm auth lookup qno
passdb lookup: user qno doesn't exist

And no surprise:
root@bywater /etc/dovecot/conf.d # doveadm auth test qno
Password:
passdb: qno auth failed
extra fields:
  user=qno


Hi QNo

I can't see why this happening. Only suggestion I have is to run those 
commands with -D to get more detailed info.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot replication

[John Fawcett via dovecot]

On 12/07/2024 21:14, James Cook wrote:

On Fri, Jul 12, 2024 at 06:28:13PM GMT, John Fawcett via dovecot wrote:

Hi James

I want to avoid the -1 parameter because it doesn't do deletes in the 
target.


-l, not -1.


Thanks I missed that - so locking can be done within Dovecot

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot replication

[John Fawcett via dovecot]

On 12/07/2024 17:38, James Cook via dovecot wrote:

Replication is in the current dovecot version but will go away in 2.4.

The doveadm sync feature is staying. So with some work you can set it 
up what you are requesting.


I used to use replication and now I'm thinking about using sync but 
have not implemented it. The following are thoughs on it.


There are some points to be addressed that are outside dovecot. I 
think you'd have to make sure that your sync happened frequently 
enough that you could live with losing the emails that arrives 
bewteen syncs for example.


I have been thinking of writing a hacky delivery script that passes 
the email on to dovecot-lda, then runs doveadm sync, and only returns 
success after the sync is done (or after a timeout). No idea what 
problems I will run into, but the idea is to never accept an email 
until it's guaranteed replicated.


That would tend to lead to a requirement to sync more frequently and 
reduce risk of email loss. But then you'd need to avoid more than one 
sync being active simultaneously (that is my assumption that this 
would not work, but I don't know if it is a real problem).


Doesn't the -l option protect against simultaneous syncs? Just based 
on reading the doveadm-sync man page. (I guess it could cause a 
problem if you start sync processes more quickly than they can finish.)


NB I'm just running a one-person email server so don't take my ideas 
too seriously :)



Hi James

I want to avoid the -1 parameter because it doesn't do deletes in the 
target. It might not be an issue, but I'd like to keep the target the 
same as the source. I don't know if it would protect against 
simultaneous jobs, but I dont' know even if that is an issue. I was 
making the assumption it could be so I plan to avoid it somehow.


As for the lda to doveadm sync script, I have been wondering too about 
how to close the gap for emails arriving between syncs, even though the 
risk might not be so significant.


With delivering to two dovecot servers before accepting the email, 
either one going down will stop email delivery. That's an inconvenience 
but without necessarily introducing risks for losing the emails. 
Ultimately it depends on where those undelivered emails are being kept 
in the meantime  (presumably MTA queue) and whether they are safer there 
than being delivered to a single instance. Though that is related only 
to the downtime. When both are up the risk would be mitigated by the 
solution.


Other idea I was thinking of is a replicated file system. That could 
make emails available in real time to both dovecot instances assuming a 
functionality to sync to disk on both/multiple nodes before replying 
back to dovecot to accept the email. I believe there would still need to 
be only one dovecot instance active at a time. However, I have heard of 
but don't know personally of horror stories about email outages on 
clustered file systems, which leads me to believe that there would still 
be sufficient residual risk to require a backup copy of the mailboxes 
with doveadm sync or other tools.


John

John





___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot replication

[John Fawcett via dovecot]

On 12/07/2024 13:05, Jeff Pang via dovecot wrote:

Hello,

Does the community version of dovecot have the replication feature?
When one dovecot was down, another one could take over the tasks.

Thanks.

_


Jeff

Replication is in the current dovecot version but will go away in 2.4.

The doveadm sync feature is staying. So with some work you can set it up 
what you are requesting.


I used to use replication and now I'm thinking about using sync but have 
not implemented it. The following are thoughs on it.


There are some points to be addressed that are outside dovecot. I think 
you'd have to make sure that your sync happened frequently enough that 
you could live with losing the emails that arrives bewteen syncs for 
example. That would tend to lead to a requirement to sync more 
frequently and reduce risk of email loss. But then you'd need to avoid 
more than one sync being active simultaneously (that is my assumption 
that this would not work, but I don't know if it is a real problem).


The failover would require you to stop people accessing the old server, 
stop syncing and start the backup instance. When the main instance is 
available to come back on line, you'd need to stop the backup instance, 
sync in the opposite direction and then start dovecot and re-enable the 
sync mechansim towards the backup.


You need to ensure people don't connect simultaneously to both 
instances. So some thought would be needed about those cases where the 
main node goes and then comes backup to ensure that your sync is not 
still active at that point and replicates the old state onto the backup 
server and to ensure then people don't start connecting to it again 
without being able to control it.


It's a disaster recover rather than high availability solution, but I 
think it can work. Others may already have got working implementations 
and thought through some of the implications.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: New Member Introduction

[John Fawcett via dovecot]

On 10/07/2024 07:37, Selena Thomas via dovecot wrote:

Hi everyone,

I am new to this forum and excited to be here.
I'm interested in learning more about Dovecot and its features, and I'm eager 
to participate in the discussions here.
Could someone please guide me on how to ask questions here ?
Where should I post if I have a query ?

Looking forward to your advice and connecting with you all.

Thanks,
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi Selena

here is some info  about posting to the list:

https://www.dovecot.org/mailing-lists/

The mailing list archives and the dovecot documentation are good 
starting points for finding info and resolving issues.


https://dovecot.org/mailman3/hyperkitty/list/dovecot@dovecot.org/

https://doc.dovecot.org/

Otherwise if you've got some specific issue or question just post it to 
the list. For configuration issues it helps if you post your 
configuration "dovecot -n" output and relevant logging. Hope that helps


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot-lda: timeout for external scripts called by sieve

[John Fawcett via dovecot]

Corrected a typo in the previous message, should have read 
sieve_filter_exec_timeout and not sieve_pipe_exec_timeout


On 10/07/2024 10:41, John Fawcett wrote:



On 10/07/2024 09:40, John Fawcett via dovecot wrote:


On 09/07/2024 09:02, Meinhard Schneider via dovecot wrote:

Hello,


I have set both the values for "sieve_execute_exec_timeout" and 
"sieve_extension_exec_timeout" to 60 sec, but according to the log, 
this does not seem to affect the 1 msec mentioned therein.


The SpamAssassin daemon has enough child processes to handle the 
requests (most of them were idle at the time of the error).


How can I configure that the external script gets more time for 
execution? And what can I do so that - should the script terminate 
contrary to expectations or be killed in the timeout - the mail is 
not lost but simply processed further?


Many thanks & best regards
Meinhard



Hi Mainhard

I can see the error in the log lines you posted above, but I don't 
see anything in your configuration that invokes 
/usr/lib/dovecot/sieve-filter/spamc-filter.sh script.


John

___

Just one thing I noticed which could well be relevant to the issue. 
Seems there is no setting called "|sieve_extension_exec_timeout|".


|For sieve_extension_socket_dir|and |sieve_extension_bin_dir| you 
correctly substituted extension with filter, but you did not do the 
same with sieve_extension_exec_timeout which should be 
sieve_filter_exec_timeout


https://doc.dovecot.org/configuration_manual/sieve/plugins/extprograms/

The three extensions introduced by this plugin
-|vnd.dovecot.pipe|,|vnd.dovecot.filter|and|vnd.dovecot.execute|-
each have separate but similar configuration. The following
configuration settings are used, for which “” in the
setting name is replaced by
either|pipe|,|filter|or|execute|depending on which extension is
being configured:

|sieve_extension_socket_dir|=

Points to a directory relative to the Dovecot base_dir where
the plugin looks for script service sockets.

|sieve_extension_bin_dir|=

Points to a directory where the plugin looks for programs
(shell scripts) to execute directly and pipe messages to.

|sieve_extension_exec_timeout|= 10s

Configures the maximum execution time after which the program
is forcibly terminated.

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot-lda: timeout for external scripts called by sieve

[John Fawcett via dovecot]

On 10/07/2024 09:40, John Fawcett via dovecot wrote:


On 09/07/2024 09:02, Meinhard Schneider via dovecot wrote:

Hello,


I have set both the values for "sieve_execute_exec_timeout" and 
"sieve_extension_exec_timeout" to 60 sec, but according to the log, 
this does not seem to affect the 1 msec mentioned therein.


The SpamAssassin daemon has enough child processes to handle the 
requests (most of them were idle at the time of the error).


How can I configure that the external script gets more time for 
execution? And what can I do so that - should the script terminate 
contrary to expectations or be killed in the timeout - the mail is 
not lost but simply processed further?


Many thanks & best regards
Meinhard



Hi Mainhard

I can see the error in the log lines you posted above, but I don't see 
anything in your configuration that invokes 
/usr/lib/dovecot/sieve-filter/spamc-filter.sh script.


John

___

Just one thing I noticed which could well be relevant to the issue. 
Seems there is no setting called "|sieve_extension_exec_timeout|".


|For sieve_extension_socket_dir|and |sieve_extension_bin_dir| you 
correctly substituted extension with filter, but you did not do the same 
with sieve_extension_exec_timeout which should be sieve_pipe_exec_timeout


https://doc.dovecot.org/configuration_manual/sieve/plugins/extprograms/

   The three extensions introduced by this plugin
   -|vnd.dovecot.pipe|,|vnd.dovecot.filter|and|vnd.dovecot.execute|-
   each have separate but similar configuration. The following
   configuration settings are used, for which “” in the
   setting name is replaced by
   either|pipe|,|filter|or|execute|depending on which extension is
   being configured:

   |sieve_extension_socket_dir|=

   Points to a directory relative to the Dovecot base_dir where the
   plugin looks for script service sockets.

   |sieve_extension_bin_dir|=

   Points to a directory where the plugin looks for programs (shell
   scripts) to execute directly and pipe messages to.

   |sieve_extension_exec_timeout|= 10s

   Configures the maximum execution time after which the program is
   forcibly terminated.

John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dovecot-lda: timeout for external scripts called by sieve

[John Fawcett via dovecot]

On 09/07/2024 09:02, Meinhard Schneider via dovecot wrote:

Hello,

I have a small problem with dovecot-lda. I am using Debian 12 Bookworm:

root@lnxmail01:~# dovecot --version
2.3.13 (89f716dc2)


Historically grown (probably still from sendmail times, long time ago) 
I always used procmail for local e-mail delivery. This also included 
the call for spamc for SpamAssassin.


I have now changed this and set up a route via sieve. There is a small 
script for this:


root@lnxmail01:~# cat /usr/lib/dovecot/sieve-filter/spamc-filter.sh
#!/bin/bash

/usr/bin/spamc


My sieve configuration says (in extracts):

root@lnxmail01:~# head -12 ~meini/.dovecot.sieve
## Generated by Roundcube Webmail SieveRules Plugin ##

require 
["body","comparator-i;ascii-numeric","copy","envelope","fileinto","imap4flags","include","regex","relational","spamtestplus","vacation","variables","spamtest"];


include :global "spamc";

# rule:[Spam]
if spamtest :value "ge" :comparator "i;ascii-numeric" "5"
{
    fileinto "Spam";
    stop;
}


In general, this has been working for many months without any 
problems. But sometimes the whole thing fails due to a timeout:


Jul  7 22:00:13 lnxmail01 dovecot: 
lda(meini)<766014>: Error: program 
exec:/usr/lib/dovecot/sieve-filter/spamc-filter.sh (766016): Execution 
timed out (> 1 msecs)
Jul  7 22:00:13 lnxmail01 dovecot: 
lda(meini)<766014>: Error: program 
exec:/usr/lib/dovecot/sieve-filter/spamc-filter.sh (766016): Forcibly 
terminated with signal 15
Jul  7 22:00:13 lnxmail01 dovecot: 
lda(meini)<766014>: Panic: output stream (temp 
iostream in /tmp/dovecot.lda. for (program client seekable output)) is 
missing error handling
Jul  7 22:00:13 lnxmail01 dovecot: 
lda(meini)<766014>: Error: Raw backtrace: 
/usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) 
[0x7ff00aee04e2] -> 
/usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7ff00aee05fe] 
-> /usr/lib/dovecot/libdovecot.so.0(+0xfc49b) [0x7ff00aeec49b] -> 
/usr/lib/dovecot/libdovecot.so.0(+0xfc501) [0x7ff00aeec501] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x53aee) [0x7ff00ae43aee] -> 
/usr/lib/dovecot/libdovecot.so.0(o_stream_unref+0x5e) [0x7ff00af1003e] 
-> /usr/lib/dovecot/libdovecot.so.0(program_client_destroy+0x8e) 
[0x7ff00ae6ee6e] -> 
/usr/lib/dovecot/modules/sieve/lib90_sieve_extprograms_plugin.so(sieve_extprogram_destroy+0x18) 
[0x7ff00aacd8f8] -> 
/usr/lib/dovecot/modules/sieve/lib90_sieve_extprograms_plugin.so(+0x556d) 
[0x7ff00aacb56d] -> 
/usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0xea) 
[0x7ff00ab4c9ba] -> 
/usr/lib/dovecot/libdovecot-sieve.so.0(ext_include_execute_include+0x2c7) 
[0x7ff00ab72e17] -> 
/usr/lib/dovecot/libdovecot-sieve.so.0(sieve_interpreter_continue+0xea) 
[0x7ff00ab4c9ba] -> 
/usr/lib/dovecot/libdovecot-sieve.so.0(sieve_multiscript_run+0x82) 
[0x7ff00ab62b32] -> 
/usr/lib/dovecot/modules/lib90_sieve_plugin.so(+0x37a3) 
[0x7ff00abd47a3] -> 
/usr/lib/dovecot/libdovecot-lda.so.0(mail_deliver+0x176) 
[0x7ff00b11f906] -> /usr/lib/dovecot/dovecot-lda(main+0x879) 
[0x55ef085e4fb9] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) 
[0x7ff00ac3ad0a] -> /usr/lib/dovecot/dovecot-lda(_start+0x2a) 
[0x55ef085e56ea]




The worst thing is that the entire mail is lost (/dev/null)!

Then I tried to set the timeout higher (than the current 10 seconds):

root@lnxmail01:~# doveconf | grep -A 12 ^plugin
plugin {
  sieve = file:~/sieve;active=~/.dovecot.sieve
  sieve_execute_exec_timeout = 60s
  sieve_extension_exec_timeout = 60s
  sieve_extensions = +vnd.dovecot.filter +spamtest +spamtestplus
  sieve_filter_bin_dir = /usr/lib/dovecot/sieve-filter
  sieve_filter_socket_dir = sieve-filter
  sieve_global = /usr/lib/dovecot/sieve-global
  sieve_plugins = sieve_extprograms
  sieve_spamtest_max_value = 5
  sieve_spamtest_status_header = X-Spam-Level
  sieve_spamtest_status_type = strlen
}


I have set both the values for "sieve_execute_exec_timeout" and 
"sieve_extension_exec_timeout" to 60 sec, but according to the log, 
this does not seem to affect the 1 msec mentioned therein.


The SpamAssassin daemon has enough child processes to handle the 
requests (most of them were idle at the time of the error).


How can I configure that the external script gets more time for 
execution? And what can I do so that - should the script terminate 
contrary to expectations or be killed in the timeout - the mail is not 
lost but simply processed further?


Many thanks & best regards
Meinhard



Hi Mainhard

I can see the error in the log lines you posted above, but I don't see 
anything in your configuration that invokes 
/usr/lib/dovecot/sieve-filter/spamc-filter.sh script.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Pread error over smb3

[John Fawcett via dovecot]

Hi Joan

not sure what OS you're using, so just a guess: but maybe this is 
selinux related or something similar. When it's the OS providing the 
error code to dovecot, it's very unlikely to be anything in dovecot itself.


On 06/07/2024 12:51, Joan Moreau via dovecot wrote:

No error on the error side

the error occurs only with dovecot. all other soft do nto complains
about the smb3 protocol

I get also
Jul 6 10:49:45 gjserver dovecot[4220]:
lmtp(ad...@grosjo.net)<4355>: Error:
rename(/net/mails/grosjo.net/admin/storage/dovecot.map.index.tmp,
/net/mails/grosjo.net/admin/storage/dovecot.map.index) failed:
Permission denied



On Tue, 2024-07-02 at 08:00 +0300, Aki Tuomi via dovecot wrote:

Ok. But the error is coming from kernel, so not much Dovecot can do
about it. Maybe try turning on some debugging in your server to see
what is going on?

Aki


On 02/07/2024 07:54 EEST Joan Moreau via dovecot
 wrote:

  
Permissions on the server are very fine


The problem occurs ONLY with dovecot


On Tue, 2024-07-02 at 07:49 +0300, Aki Tuomi wrote:

On 02/07/2024 02:05 EEST Joan Moreau via dovecot
 wrote:

  
Hi


I am trying to move my storage of email on a smb3 mounted
volume.

I am getting the following error :
Error: pread(/net/.../storage/dovecot.map.index.log) failed:
Permission
denied (euid=1004(mailusers) egid=12(mail) UNIX perms appear ok
(ACL/MAC wrong?))

How to resolve that ?

Thank you


This seems to be some kind of smb3 ACL problem, check permissions
on
the server?

Aki

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: dsync will be removed in 2.4 - why?

[John Fawcett via dovecot]

On 05/07/2024 11:51, Aki Tuomi via dovecot wrote:

On 05/07/2024 12:34 EEST Oliver Krone via dovecot  wrote:

  
Hi there,


I recently learned that dsync (mail replication) functionality will be
removed in the upcoming releases of dovecot. What is the reason and will
there be a replacement? How can I synchronize my e-mails  in the future?

Thanks a lot.

Regards
      Oliver


You have learned slightly wrong. Replicator is going to be removed, not doveadm 
sync (dsync).

You can still synchronize your emails by running doveadm sync periodically.

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


That's good to know (also thanks to the person that wrote me directly to 
say so too), I will look into this. I had completely disabled 
replication in order to get used to the future.


Just a doubt, when scheduling doveadm sync at short intervals. Will 
doveadm sync work ok if more than one job is overlapping (i.e. new cron 
job starts but old one is still running) or should I define some locking 
mechanism to ensure that there is no more than one sync job running?


Thanks

John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

d=3: Finish implicit keep action

imap(info@demo.example): Debug: sieve: uid=3: Finishing actions

imap(info@demo.example): Debug: sieve: uid=3: Finish pipe action

imap(info@demo.example): Debug: sieve: uid=3: Finish pipe action

imap(info@demo.example): Debug: sieve: uid=3: Finished executing result (final, 
status=ok, keep=yes)

imap(info@demo.example): Debug: sieve: multi-script: Sequence finished 
(status=ok, keep=yes)

imap(info@demo.example): Debug: sieve: multi-script: Destroy

imap(info@demo.example): Debug: Mailbox INBOX/Junk: Mailbox opened

imap(info@demo.example)<2107>: Debug: imapsieve: mailbox 
INBOX: FLAG event (changed flags: \Deleted)

imap(info@demo.example)<2107>: Debug: Mailbox INBOX: UID 3: 
Expunge requested

imap(info@demo.example)<2107>: Debug: Mailbox INBOX: UID 3: 
Mail expunged

  

  


#dovecot -n
# 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf

# Pigeonhole version 0.5.19 (4eae2f79)

# OS: Linux 6.1.0-21-amd64 x86_64 Debian 12.5

# Hostname: ServerIV-home.demo.example

auth_mechanisms = plain login

mail_debug = yes

mail_location = maildir:~/Maildir

mail_privileged_group = mail

managesieve_notify_capability = mailto

managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve

namespace inbox {

   hidden = no

   ignore_on_failure = no

   inbox = yes

   list = yes

   location =

   mailbox Drafts {

 auto = subscribe

 special_use = \Drafts

   }

   mailbox Junk {

 auto = subscribe

 special_use = \Junk

   }

   mailbox Sent {

 auto = subscribe

 special_use = \Sent

   }

   mailbox "Sent Messages" {

 auto = subscribe

 special_use = \Sent

   }

   mailbox Trash {

 auto = subscribe

 special_use = \Trash

   }

   prefix = INBOX/

   separator = /

   subscriptions = yes

   type = private

}

passdb {

   driver = pam

}

passdb {

   args = scheme=CRYPT username_format=%u /etc/dovecot/users

   driver = passwd-file

}

plugin {

   imapsieve_mailbox1_before =file:/usr/lib/dovecot/sieve/report-spam.sieve

   imapsieve_mailbox1_causes = COPY APPEND

   imapsieve_mailbox1_name = INBOX/Junk

   imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve

   imapsieve_mailbox2_causes = COPY APPEND

   imapsieve_mailbox2_from = INBOX/Junk

   imapsieve_mailbox2_name = *

   sieve =file:~/sieve;active=~/.dovecot.sieve

   sieve_after = /etc/dovecot/conf.d/custom-sieve/global_after.sieve

   sieve_before = /etc/dovecot/conf.d/custom-sieve/global_before.sieve

   sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment 
+vnd.dovecot.debug

   sieve_pipe_bin_dir = /usr/lib/dovecot/sieve

   sieve_plugins = sieve_imapsieve sieve_extprograms

}

protocols = imap lmtp sieve

service auth {

   unix_listener /var/spool/postfix/private/auth {

 mode = 0666

   }

}

service lmtp {

   inet_listener lmtp {

 address = 127.0.0.1 ::1

 port = 24

   }

}

ssl_cert =   

  

  


-Ursprüngliche Nachricht-
Von: John Fawcett via dovecot  
Gesendet: Dienstag, 2. Juli 2024 12:34

An:dovecot@dovecot.org
Betreff: Re: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

  


Hi Jens

  


just one update and some more insight looking at the code

  


Your causes should be COPY APPEND (whether applying proposed solution

(1) or (2). I updated inline below for solution (2).

  


Logic for that: I was testing from Thunderbird which is generating a MOVE 
event. In the imapsieve plugin code MOVE and COPY are treated almost the same, 
so my rule with COPY fire on MOVE too. In your case the event you're getting is 
APPEND, but none of your rules have that as a cause so won't match even if the 
mailbox matches.

  


John

___
dovecot mailing list --dovecot@dovecot.org
To unsubscribe send an email todovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: One-way synchronization with doveadm/TCP

[John Fawcett via dovecot]

On 04/07/2024 09:35, Дилян Палаузов via dovecot wrote:

Hello,

the replication procedure described at 
https://doc.dovecot.org/configuration_manual/replication/ suggests to open a 
server port 12345 on two systems, and then point each system to the other one.  
This creates bi-directional synchronization.

I want to synchronize only from server A to server B, but not in the opposite 
direction.  Is it sufficient on server A to set

   service doveadm {
 inet_listener {
   port = 12345
 }
   }
   doveadm_password = secret

and server B to set

   doveadm_port = 12345
   doveadm_password = secret

?  In other words, when a doveadm client (system B) connects to a server 
(system A) for replication purposes, does it only read data? Or does it 
instruct the server also to delete data?

To start the replication I had to executed «doveadm replicator replicate '*'».  
Is in necessary to run this periodically?

Thanks for your answer,
   Дилян


Hi Дилян

in case you had not seen it, from my understanding of previous posts on 
the list, the replication feature will be removed in 2.4 release.  I 
know that doesn't help with the detailed questions, but thought I'd 
mention it before you decide to spend time on setting it up.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: AW: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

On 03/07/2024 20:53, postfix_dovecot--- via dovecot wrote:

Hi John,

again you're hitting the point!

I put a "report-spam.sieve fired!" and "report-ham.sieve fired!" debug message in the 
scripts and monitored with "journalctl -u dovecot -f | grep fired".
The APPEND event (move to junk) triggers BOTH scripts while the APPEND (move 
from junk to elsewhere) do nothing.

With your second listing it runs as expected. Now I will crawling some RFCs 
about the commands and a possible solution.

But I wonder am I the only one with this problem? I expected Outlook, even the 
older 2016, to be very popular?!

Jens


Hi Jens

after posting I was thinking over the rule for APPENDing to Inbox and I 
did have my doubts on that. I think it will fire even if people move 
email from some other folder to Inbox or even potentially on receiving 
new email, so probably not an ideal solution. I think the only real 
solution is to have negative matching rules (which at the moment there 
isn't)


I don't have access to Outlook 2016. I have tried it with Outlook 
1.2024.701.200 (which seems a very strange version number, but that's 
what it says) and I got a MOVE event just like Thunderbird. I think that 
a mail client should generate a MOVE event when moving a message between 
folders that are on the same server. I would only expect APPEND events 
if the move was happening for example between two different servers or 
between server and local folders. If you want to investigate further 
maybe the rawlog_dir feature would help to capture more info.


best of luck!

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

login_access_sockets

[John Fawcett via dovecot]

Hi

I've used Dovecot for a long time but I never stop learning about the 
depth of features I've never used I just discovered the 
login_access_sockets setting when reading this page:


https://doc.dovecot.org/admin_manual/login_processes/

The compilation using  --with-libwrap configure setting was not 
immediate, since on Fedora which I am using there is no 
tcp_wrappers-devel package so tcpd.h wasn't installed in /usr/include 
and the library name libwrap.so that was expected by Dovecot needed a 
symlink to the library installed by tcp_wrappers package.


So differently to what I previously understood, Dovecot does have a way 
of controlling access at connect time via tcp_wrappers hosts.allow and 
hosts.deny files. But now I also realize that this discovery will be 
short lived, since I understand this is going away in 3.0 and also 2.4 
though I can't find a page on that:


https://doc.dovecotpro.com/3.0/installation/upgrading/2.3.x-3.x.html

I can understand the logic about slimming down the code to leave out 
less used features, but this part of the code looks well written and 
hasn't required much maintenance over the years. I know there's zero 
chance of reversing the removal of login_access_sockets, but I just 
wondered whether there could be the chance to introduce a LUA hook at 
connect time rather than waiting for authentication to be done. Also I 
have no idea what the overhead of LUA script is, does it spawn 
additional processes per connection?


Thanks

John




___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

(Resending because of size limit exceeded on previous post)

Hi Jens

that looks much better, though not yet completely solved. What you're 
now getting is both rules firing when you move a message to Junk, 
whereas you only want the first one firing. I guess you will have no 
rules firing when you move from Junk to elsewhere. My suspicion is that 
for APPEND event the target and destination mailboxes are set to the 
same value.  i.e. APPEND event for moving to Junk will have 
mailbox=INBOX/Junk and from=INBOX/Junk which fires both rules, whereas 
when moving from Junk to INBOX mailbox=INBOX and from=INBOX, leading to 
KO for rule one on "mailbox" and KO on rule 2 for "from".


Static mailbox rule [1]: mailbox=`INBOX/Junk' from=`*' causes=(COPY 
APPEND) => before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' 
after=(none)


Static mailbox rule [2]: mailbox=`*' from=`INBOX/Junk' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)

Whenever I do the move of the message I get a MOVE (which is treated 
like COPY). I don't see these APPEND events. Not sure why your email 
client generates APPEND instead of MOVE. That could be a point to 
investigate. If you're going to have to manage APPEND events you'll 
probably need different rules. If you'll going to manage MOVE/COPY and 
APPEND events then you can probably combine sets of rules (adjusting the 
sequential numbering).


The following work for MOVE/COPY events

  imapsieve_mailbox1_before =file:/usr/lib/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_name = INBOX/Junk
  imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_from = INBOX/Junk
  imapsieve_mailbox2_name = *

The following should work for APPEND events, but it won't support 
wildcards fo rule 2 else it will fire when moving to Junk as well as 
from Junk. I'm pretty sure you can't specify negation e.g. 
imapsieve_mailbox2_name = !INBOX/Junk, which is what would be needed to 
make it work for moving from Junk to any mailbox. As written rule 2 
fires for moving from Junk to INBOX


  imapsieve_mailbox1_before =file:/usr/lib/dovecot/sieve/report-spam.sieve
  imapsieve_mailbox1_causes = APPEND
  imapsieve_mailbox1_name = INBOX/Junk
  imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = APPEND
  imapsieve_mailbox2_name = INBOX

John

On 02/07/2024 14:51, postfix_dovecot--- via dovecot wrote:

Gotcha

It looks like you nailed it John! (Log and config attached)

  


The log looks now very different and my log entry is also shown.

Now I can continue working at this point. Adjusting the events i.e.

  


The reason was actually simple - but I spent a few evenings searching and was 
blind. I could have figured it out on my own! Thank you so much John for took 
such care of this issue!

Jens

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

Hi Jens

just one update and some more insight looking at the code

Your causes should be COPY APPEND (whether applying proposed solution 
(1) or (2). I updated inline below for solution (2).


Logic for that: I was testing from Thunderbird which is generating a 
MOVE event. In the imapsieve plugin code MOVE and COPY are treated 
almost the same, so my rule with COPY fire on MOVE too. In your case the 
event you're getting is APPEND, but none of your rules have that as a 
cause so won't match even if the mailbox matches.


John

On 02/07/2024 12:07, John Fawcett via dovecot wrote:

Hi Jens

that log looks to me like it is for moving a message from Inbox to 
Junk rather than Junk to Inbox.


The issue I see is that how you defined the rules does not match your 
namespace naming scheme. Event being received is the following. In 
that event you can see the name of the mailbox is "INBOX/Junk".


imap(info@demo.example)<1827>: Debug: imapsieve: 
mailbox INBOX/Junk: APPEND event


However the rule you defined has the mailbox as "Junk".

Static mailbox rule [1]: mailbox=`Junk' from=`*' causes=(COPY) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)


The logging you show confirms that the rule is not matching, else you 
would get a log line like the one I showed with the text "Debug: 
imapsieve: Matched static mailbox rule". So I have two possible 
alternative solutions, that in my tests I found both working:


(1) you define your inbox namespace with blank prefix instead of 
prefix=INBOX/


OR

(2) you adjust the rules as follows

  imapsieve_mailbox1_before 
=file:/usr/lib/dovecot/sieve/report-spam.sieve

  imapsieve_mailbox1_causes = COPY APPEND
  imapsieve_mailbox1_name = INBOX/Junk
  imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve
  imapsieve_mailbox2_causes = COPY APPEND
  imapsieve_mailbox2_from = INBOX/Junk
  imapsieve_mailbox2_name = *

If you apply one of those solutions and repeat the test 
unsuccessfully, it would be good to see the logging from that test.

John

On 02/07/2024 11:28, postfix_dovecot--- via dovecot wrote:

Hi John!

You are of course absolutely right - wild experimentation is rarely a 
good approach. But you are surely familiar with the situation when 
you have done everything logical and then, in your desperation, you 
start to change things that you would not otherwise change ;)


I've actually gotten to the point where I'm going to give up on the 
idea and run a cron job at night. Then the spam won't be detected in 
real time - but what the heck?


As I wrote yesterday, I implemented a very popular tutorial for 
Debian 10 in Germany 1:1. The result was the same. Nevertheless, I 
reset my VM, where I do everything with the current releases, and 
attaching the log here. I would be very happy if it still works.


To be sure, I deleted the whole inbox and recreated the account on 
Outlook to create a fresh, new folder structure given by the Dovecot 
configuration. I also added the plugin vnd.dovecot.debug to the 
sieve_global_extensions line and added >debug_log "report-spam.sieve 
was running!";< to the report-spam.sieve file. So this string should 
appear in the log if the script were running at all. But it doesn't. 
So I still guess, it's not getting triggered.


That's the log if I move a mail from Junk (Spam) to Inbox:

# journalctl -u dovecot -f
imap-login: Login: user=, method=PLAIN, 
rip=192.168.30.26, lip=192.168.30.34, mpid=1827, TLS, 
session=
imap(info@demo.example)<1827>: Debug: Loading 
modules from directory: /usr/lib/dovecot/modules
imap(info@demo.example)<1827>: Debug: Module 
loaded: /usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so
imap(info@demo.example)<1827>: Debug: Effective 
uid=1, gid=1, home=/var/mail/vhosts/demo.example/info
imap(info@demo.example)<1827>: Debug: 
open(/proc/self/io) failed: Permission denied
imap(info@demo.example)<1827>: Debug: Namespace 
inbox: type=private, prefix=INBOX/, sep=/, inbox=yes, hidden=no, 
list=yes, subscriptions=yes location=maildir:~/Maildir
imap(info@demo.example)<1827>: Debug: maildir++: 
root=/var/mail/vhosts/demo.example/info/Maildir, index=, indexpvt=, 
control=, inbox=/var/mail/vhosts/demo.example/info/Maildir, alt=
imap(info@demo.example)<1827>: Debug: Namespace : 
type=private, prefix=, sep=, inbox=no, hidden=yes, list=no, 
subscriptions=no location=fail::LAYOUT=none
imap(info@demo.example)<1827>: Debug: none: root=, 
index=, indexpvt=, control=, inbox=, alt=
imap(info@demo.example)<1827>: Debug: Mailbox 
INBOX/Junk: Mailbox opened
imap(info@demo.example)<1827>: Debug: imapsieve: 
mailbox INBOX/Junk: APPEND event
imap(info@demo.example)<1827>: Debug: duplicate db: 
Initialize
imap(info@demo.example)<1827>: Debug: sieve: 
Pigeonhole version 0.5.19 (4eae2f79) initializing
imap(info@demo.example)<1827>: Debug: sieve: 
incl

Re: AW: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

 [2]: mailbox=`*' from=`Junk' causes=(COPY) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)
imap(info@demo.example)<1827>: Debug: Mailbox INBOX/Junk: 
Mailbox opened
imap(info@demo.example)<1825>: Debug: imapsieve: mailbox 
INBOX: FLAG event (changed flags: \Deleted)
imap(info@demo.example)<1825>: Debug: Mailbox INBOX: UID 1: 
Expunge requested
imap(info@demo.example)<1825>: Debug: Mailbox INBOX: UID 1: 
Mail expunged

# doveconf -n
# 2.3.19.1 (9b53102964): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.19 (4eae2f79)
# OS: Linux 6.1.0-21-amd64 x86_64 Debian 12.5
# Hostname: ServerIV-home.demo.example
auth_mechanisms = plain login
mail_debug = yes
mail_location = maildir:~/Maildir
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date index ihave duplicate 
mime foreverypart extracttext imapsieve vnd.dovecot.imapsieve
namespace inbox {
   hidden = no
   ignore_on_failure = no
   inbox = yes
   list = yes
   location =
   mailbox Drafts {
 auto = subscribe
 special_use = \Drafts
   }
   mailbox Junk {
 auto = subscribe
 special_use = \Junk
   }
   mailbox Sent {
 auto = subscribe
 special_use = \Sent
   }
   mailbox "Sent Messages" {
 auto = subscribe
 special_use = \Sent
   }
   mailbox Trash {
 auto = subscribe
 special_use = \Trash
   }
   prefix = INBOX/
   separator = /
   subscriptions = yes
   type = private
}
passdb {
   driver = pam
}
passdb {
   args = scheme=CRYPT username_format=%u /etc/dovecot/users
   driver = passwd-file
}
plugin {
   imapsieve_mailbox1_before =file:/usr/lib/dovecot/sieve/report-spam.sieve
   imapsieve_mailbox1_causes = COPY
   imapsieve_mailbox1_name = Junk
   imapsieve_mailbox2_before =file:/usr/lib/dovecot/sieve/report-ham.sieve
   imapsieve_mailbox2_causes = COPY
   imapsieve_mailbox2_from = Junk
   imapsieve_mailbox2_name = *
   sieve =file:~/sieve;active=~/.dovecot.sieve
   sieve_after = /etc/dovecot/conf.d/custom-sieve/global_after.sieve
   sieve_before = /etc/dovecot/conf.d/custom-sieve/global_before.sieve
   sieve_global_extensions = +vnd.dovecot.pipe +vnd.dovecot.environment 
+vnd.dovecot.debug
   sieve_pipe_bin_dir = /usr/lib/dovecot/sieve
   sieve_plugins = sieve_imapsieve sieve_extprograms
}
protocols = imap lmtp sieve
service auth {
   unix_listener /var/spool/postfix/private/auth {
 mode = 0666
   }
}
service lmtp {
   inet_listener lmtp {
 address = 127.0.0.1 ::1
 port = 24
   }
}
ssl_cert = Von: John Fawcett via dovecot  
Gesendet: Montag, 1. Juli 2024 23:16

An:dovecot@dovecot.org
Betreff: Re: AW: AW: AW: IMAPSieve plugin will not run rspamd script

Hi Jens

changing random things is rarely a good way to solve these kinds of issues. My 
advice, if you don't need the inbox namespace prefix set for a specific reason, 
would be to go with the default prefix (ie. blank) and then set up the 
imapsieve rules as per your original post and repeat the test. If that does not 
work then post the debug logging and configuration from that test.

There could be multiple places where this is failing but the very first point is to have 
your sieve rules match. They will show something like the following in the logging. 
Notice the "Matched static mailbox rule"
message. If that is not happening it's pointless to look further down the line 
into issues in the scripts themselves.

Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: mailbox 
INBOX: MOVE event
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: Mailbox Spam: UID 2: 
Expunge requested
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: duplicate db: Initialize
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Pigeonhole 
version 0.5.21 (f6cd4b8e) initializing
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: include: 
sieve_global is not set; it is currently not possible to include `:global' scripts.
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Sieve imapsieve 
plugin for Pigeonhole version 0.5.21 (f6cd4b8e) loaded
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Sieve 
Extprograms plugin for Pigeonhole version 0.5.21 (f6cd4b8e) loaded
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: Static mailbox 
rule [1]: mailbox=`Spam' from=`*' causes=(COPY) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 

Re: AW: AW: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

On 01/07/2024 22:09, postfix_dovecot--- via dovecot wrote:

Hi John,

  


the prefix is ​​just a sign of my desperation - I tried all sorts of variations 
yesterday and now forgot to undo it.

  


There’s a very detailed tutorial available (German language) with Debian 10. 
Just the sieve scripts are little different. I spend the time to raise a Debian 
10 with this tutorial – and with exact the same result as with my up to date 
configuration.


So it would be interesting for me to know if there is anyone here who has 
managed to get this to work on Debian?

  


Jens



Hi Jens

changing random things is rarely a good way to solve these kinds of 
issues. My advice, if you don't need the inbox namespace prefix set for 
a specific reason, would be to go with the default prefix (ie. blank) 
and then set up the imapsieve rules as per your original post and repeat 
the test. If that does not work then post the debug logging and 
configuration from that test.


There could be multiple places where this is failing but the very first 
point is to have your sieve rules match. They will show something like 
the following in the logging. Notice the "Matched static mailbox rule" 
message. If that is not happening it's pointless to look further down 
the line into issues in the scripts themselves.


Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: 
mailbox INBOX: MOVE event
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: Mailbox Spam: 
UID 2: Expunge requested
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: duplicate db: 
Initialize
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: 
Pigeonhole version 0.5.21 (f6cd4b8e) initializing
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: include: 
sieve_global is not set; it is currently not possible to include 
`:global' scripts.
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Sieve 
imapsieve plugin for Pigeonhole version 0.5.21 (f6cd4b8e) loaded
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: sieve: Sieve 
Extprograms plugin for Pigeonhole version 0.5.21 (f6cd4b8e) loaded
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: 
Static mailbox rule [1]: mailbox=`Spam' from=`*' causes=(COPY) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: 
Static mailbox rule [2]: mailbox=`*' from=`Spam' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)
Jul 01 22:57:55 imola.site24.it dovecot[1145722]: 
imap(t...@site24.it)<1145777>: Debug: imapsieve: 
Matched static mailbox rule [2]


As for Debian, sorry I can't help on that, I'm using Fedora :-)

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Sieve Symlink Error

[John Fawcett via dovecot]

On 01/07/2024 03:32, Benjamin Rose via dovecot wrote:


On 6/30/24 16:48, John Fawcett via dovecot wrote:


On 30/06/2024 07:17, Benjamin Rose via dovecot wrote:

Hello,

I'm in the process of moving our mail server from RHEL 6 to RHEL 9. 
We will be moving to:


# dovecot --version
2.3.16 (7e2e900c1a)

My issue is that sieve does not appear to work on the new setup, 
where it does work on the old one. I made a simple filter rule:


# cat /u/mail0test/.sieve/ingo.sieve
# Sieve Filter
# Generated by Ingo (http://www.horde.org/apps/ingo/) (06/28/2024, 
11:14:52 PM)

require "fileinto";
# Test
if header :comparator "i;ascii-casemap" :contains "Subject" 
"filtertest"  {

    fileinto "Fun";
    stop;
}

Upon sending an email to this test account, the following appears in 
/var/log/maillog:


Jun 29 23:19:56 mail5 dovecot[3066980]: 
lda(mail0test)<3066980>: Warning: sieve: 
file storage: Active sieve script symlink 
/u/mail0test/.dovecot.sieve is broken: Invalid/unknown path to 
storage (points to /u/mail0test/.sieve).
Jun 29 23:19:56 mail5 dovecot[2987026]: 
doveadm(mail0test)<3066983>: Warning: sieve: 
file storage: Active sieve script symlink 
/u/mail0test/.dovecot.sieve is broken: Invalid/unknown path to 
storage (points to /u/mail0test/.sieve).
Jun 29 23:19:56 mail5 dovecot[2987026]: 
doveadm(mail0test)<3067016>: Warning: sieve: 
file storage: Active sieve script symlink 
/u/mail0test/.dovecot.sieve is broken: Invalid/unknown path to 
storage (points to /u/mail0test/.sieve).


Yet:

# ll /u/mail0test/.dovecot.sieve
lrwxrwxrwx. 1 mail0test sysguest 17 Jun 28 23:26 
/u/mail0test/.dovecot.sieve -> .sieve/ingo.sieve

# file /u/mail0test/.sieve/ingo.sieve
/u/mail0test/.sieve/ingo.sieve: ASCII text

That is the filter file I've pasted above.

I've set the following directives in 
/etc/dovecot/conf.d/90-sieve.conf via puppet:


augeas {
  "dovecot_sieve_settings":
    context => "/files/etc/dovecot/conf.d/90-sieve.conf",
    changes => [
      "set plugin/sieve_dir ~/.sieve",
  "set plugin/sieve_user_log ~/.sieve/log"
    ],
    require => Package["dovecot"],
    notify => Service["dovecot"];
}

The full configuration dump is attached.

/u in our environment is the path for user homedirs, which is an NFS 
mount to a NetApp. The OS is Springdale Linux 9.2, a clone of RedHat 
from before the IBM license change. It will soon be RHEL 9.4 as we 
have obtained a license, but for all intents and purposes, 
Springdale 9.2 and RHEL 9.2 should be considered bug-for-bug 
compatible. The arch is x86_64 with both machines mail5 and mail6 
(replicated) having Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz and 
768gb of memory. I have the same issue with SELinux in both 
enforcing and permissive modes, so this is not a permissions error 
due to SELinux.


Am I doing something wrong, or is this a bug? I've seen that there 
have been some previous issues similar to this that ended up being 
bugs in pigeonhole, so here I am.


Thanks,
Ben


Hi Ben

what version of Pigeonhole are you using?

I read here that sieve_dir is deprecated since v0.3.1

https://doc.dovecot.org/settings/pigeonhole/#pigeonhole_setting-sieve_dir 



In any case these settings look as though they don't really match up. 
Is the correct directory .sieve or sieve?


sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_dir = ~/.sieve

Also, I was curious if your inboxes are really under 
/var/spool/mail/%u and your indexes under /home/%u/indexes?


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org



Hello,

Thank you! Adding the line to puppet to enforce that this exists in 
/etc/dovecot/conf.d/90-sieve.conf:


sieve=file:~/.sieve;active=~/.dovecot.sieve

has solved the problem. Filters now work as expected!

To answer your questions, I am using 
dovecot-pigeonhole-2.3.16-8.el9.x86_64, and yes, user mail spools live 
under /var/spool/mail (NFS-mounted mbox files) and indexes live under 
/home (local disk - soon to be SSD). That's only for users who are 
using mbox format / pine / mutt. Most users are using only modern 
clients and in this case their storage is mdbox and entirely kept 
inside of /home. This is configured on a per-user basis inside of an 
LDAP value named mailMessageStore. Either it exists such as 
"mdbox:/home//mail", or it does not exist at all, in which 
case, delivery falls back to old-style mbox format. If they are on 
mbox format, only INBOX is kept in /var/spool/mail, all other folders 
are kept in ~/mail (/u//mail/).


Ben


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi Ben

as far as I can see, dovecot-pigeonhole-2.3.16-8.el9.x86_64 uses 
pigeonhole version 0.5.16. In that case I

Re: Sieve Symlink Error

[John Fawcett via dovecot]

On 30/06/2024 07:17, Benjamin Rose via dovecot wrote:

Hello,

I'm in the process of moving our mail server from RHEL 6 to RHEL 9. We 
will be moving to:


# dovecot --version
2.3.16 (7e2e900c1a)

My issue is that sieve does not appear to work on the new setup, where 
it does work on the old one. I made a simple filter rule:


# cat /u/mail0test/.sieve/ingo.sieve
# Sieve Filter
# Generated by Ingo (http://www.horde.org/apps/ingo/) (06/28/2024, 
11:14:52 PM)

require "fileinto";
# Test
if header :comparator "i;ascii-casemap" :contains "Subject" 
"filtertest"  {

    fileinto "Fun";
    stop;
}

Upon sending an email to this test account, the following appears in 
/var/log/maillog:


Jun 29 23:19:56 mail5 dovecot[3066980]: 
lda(mail0test)<3066980>: Warning: sieve: file 
storage: Active sieve script symlink /u/mail0test/.dovecot.sieve is 
broken: Invalid/unknown path to storage (points to /u/mail0test/.sieve).
Jun 29 23:19:56 mail5 dovecot[2987026]: 
doveadm(mail0test)<3066983>: Warning: sieve: 
file storage: Active sieve script symlink /u/mail0test/.dovecot.sieve 
is broken: Invalid/unknown path to storage (points to 
/u/mail0test/.sieve).
Jun 29 23:19:56 mail5 dovecot[2987026]: 
doveadm(mail0test)<3067016>: Warning: sieve: 
file storage: Active sieve script symlink /u/mail0test/.dovecot.sieve 
is broken: Invalid/unknown path to storage (points to 
/u/mail0test/.sieve).


Yet:

# ll /u/mail0test/.dovecot.sieve
lrwxrwxrwx. 1 mail0test sysguest 17 Jun 28 23:26 
/u/mail0test/.dovecot.sieve -> .sieve/ingo.sieve

# file /u/mail0test/.sieve/ingo.sieve
/u/mail0test/.sieve/ingo.sieve: ASCII text

That is the filter file I've pasted above.

I've set the following directives in /etc/dovecot/conf.d/90-sieve.conf 
via puppet:


augeas {
  "dovecot_sieve_settings":
    context => "/files/etc/dovecot/conf.d/90-sieve.conf",
    changes => [
      "set plugin/sieve_dir ~/.sieve",
  "set plugin/sieve_user_log ~/.sieve/log"
    ],
    require => Package["dovecot"],
    notify => Service["dovecot"];
}

The full configuration dump is attached.

/u in our environment is the path for user homedirs, which is an NFS 
mount to a NetApp. The OS is Springdale Linux 9.2, a clone of RedHat 
from before the IBM license change. It will soon be RHEL 9.4 as we 
have obtained a license, but for all intents and purposes, Springdale 
9.2 and RHEL 9.2 should be considered bug-for-bug compatible. The arch 
is x86_64 with both machines mail5 and mail6 (replicated) having 
Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz and 768gb of memory. I have 
the same issue with SELinux in both enforcing and permissive modes, so 
this is not a permissions error due to SELinux.


Am I doing something wrong, or is this a bug? I've seen that there 
have been some previous issues similar to this that ended up being 
bugs in pigeonhole, so here I am.


Thanks,
Ben


Hi Ben

what version of Pigeonhole are you using?

I read here that sieve_dir is deprecated since v0.3.1

https://doc.dovecot.org/settings/pigeonhole/#pigeonhole_setting-sieve_dir

In any case these settings look as though they don't really match up. Is 
the correct directory .sieve or sieve?


sieve = file:~/sieve;active=~/.dovecot.sieve
sieve_dir = ~/.sieve

Also, I was curious if your inboxes are really under /var/spool/mail/%u 
and your indexes under /home/%u/indexes?


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: AW: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

On 30/06/2024 19:57, postfix_dovecot--- via dovecot wrote:

Tried it now in every known combination. Nothing changes. It's my 7th evening 
about this and I'm starting to despair :(


Hi Jens

any chance of posting the debugging from that  test in case it gives any 
clues.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: IMAPSieve plugin will not run rspamd script

[John Fawcett via dovecot]

On 29/06/2024 10:08, postfix_dovecot--- via dovecot wrote:

Host: VMware Workstation 14.1.8

OS: Debian 12 (Bookworm)

Dovecot: 2.3.19.1

Postfix: 3.7.10
Mailclient: Outlook 2016

  


Hello,

I am currently working on a new mail server to replace my Debian 10 mail 
server. For preparation, I use VMware Workstation to learn and test the 
installation steps. When I'm eventually done, I'll rebuild my root server from 
scratch.

This time, my problem is getting the IMAPSieve plugin working to trigger rspamd 
if mail gets moved to the junk folder. Sieve runs well – if rspam recognizes a 
spam mail, it will be transferred to the junk folder.



For me it looks like the IMAPSieve plugin recognizes the move but will not 
running the script behind. I can see some actions in the log but nothing 
happens on the rspamd side where the log is also open to view.
Moving a mail with Outlook produces the following log entry:

imap(mail@test.example)<1797>: Debug: Module loaded: 
/usr/lib/dovecot/modules/lib95_imap_sieve_plugin.so

imap(mail@test.example)<1797>: Debug: imapsieve: mailbox 
INBOX/Spam: APPEND event

imap(mail@test.example)<1797>: Debug: sieve: Pigeonhole 
version 0.5.19 (4eae2f79) initializing

imap(mail@test.example)<1797>: Debug: sieve: include: 
sieve_global is not set; it is currently not possible to include `:global' scripts.

imap(mail@test.example)<1797>: Debug: sieve: Sieve imapsieve 
plugin for Pigeonhole version 0.5.19 (4eae2f79) loaded

imap(mail@test.example)<1797>: Debug: sieve: Sieve 
Extprograms plugin for Pigeonhole version 0.5.19 (4eae2f79) loaded

imap(mail@test.example)<1797>: Debug: imapsieve: Static mailbox 
rule [1]: mailbox=`Spam' from=`*' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)

imap(mail@test.example)<1797>: Debug: imapsieve: Static mailbox 
rule [2]: mailbox=`*' from=`Spam' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)

imap(mail@test.example)<1795>: Debug: imapsieve: mailbox 
INBOX: FLAG event (changed flags: \Deleted)

imap(mail@test.example)<1795>: Debug: sieve: Pigeonhole 
version 0.5.19 (4eae2f79) initializing

imap(mail@test.example)<1795>: Debug: sieve: include: 
sieve_global is not set; it is currently not possible to include `:global' scripts.

imap(mail@test.example)<1795>: Debug: sieve: Sieve imapsieve 
plugin for Pigeonhole version 0.5.19 (4eae2f79) loaded

imap(mail@test.example)<1795>: Debug: sieve: Sieve 
Extprograms plugin for Pigeonhole version 0.5.19 (4eae2f79) loaded

imap(mail@test.example)<1795>: Debug: imapsieve: Static mailbox 
rule [1]: mailbox=`Spam' from=`*' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-spam.sieve' after=(none)

imap(mail@test.example)<1795>: Debug: imapsieve: Static mailbox 
rule [2]: mailbox=`*' from=`Spam' causes=(COPY APPEND) => 
before=`file:/usr/lib/dovecot/sieve/report-ham.sieve' after=(none)

imap(mail@test.example)<1795>: Debug: imapsieve: mailbox 
INBOX: FLAG event (changed flags: \Seen)

  ..


Hi Jens

I can't give you any expert advise about that plugin as I haven't used 
it. However, looking at the debug info you posted I don't see where the 
rules are being matched.


That debugging looks like it is reading in the configuration, so you can 
see that dovecot recognises the rules configuration, but nothing in the 
debugging suggests that those rules are being matched or that you sieve 
scripts are being executed.


So it seems that the event that is being recognized "mailbox INBOX/Spam: 
APPEND event" doesn't match your defined rules. Maybe it's to do with 
name of the mailbox in the event not matching the one in your rule 1. 
It's a wild guess on my part, but maybe changing the rules to use 
"INBOX/Spam" might just solve it.


When the rules match you should see a message like "Matched static 
mailbox rule n" where n=1,2 etc in the debugging output.


John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Debian Bookworm packages, please !

[John Fawcett via dovecot]

On 26/06/2024 20:48, pgnd via dovecot wrote:

for anyone interested, for dovecot v2.3.14+ @ Fedora,

https://src.fedoraproject.org/rpms/dovecot/blob/rawhide/f/dovecot-2.3.14-opensslv3.patch 




Until this discussion started I didn't realize that I've been using the 
unsupported version of openssl 3 for quite some time with dovecot 2.3.21 
on Fedora 39 and probably previous versions of Fedora too, without any 
issues. As others have mentioned it may depend on which features are in use.


I actually also compiled a vanilla 2.3.21 (i.e. without the fedora 
patches) for development work and didn't see any issues, though I 
wouldn't use it for a live system without the patches.


I can understand that with 2.4 getting quite close (and after having 
originally an earlier plan for it) that Ox wasn't planning to invest in 
backporting stuff to 2.3 branches. There is nothing stopping the 
community from doing that where needed. But given the issues mentioned, 
can anyone point to reproducable issue reports with the current packaged 
versions in Debian? (apart from the non working mail crypt plugin 
mentioned by Aki).


Thanks

John



___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [Patch] blocklist access control for dovecot pop3/imap and pigeonhole managesieve

[John Fawcett via dovecot]

On 09/06/2024 18:39, Aki Tuomi via dovecot wrote:

On 09/06/2024 16:50 EEST John Fawcett via dovecot  wrote:

  
The files themselves didn't make it through the list server. I'll try

with .txt


On 09/06/2024 15:42, John Fawcett via dovecot wrote:

Hi

Just in case this is useful more generally, I'm posting it to the list.

While Dovecot has an access control via allow_nets, it is a user
database field that applies only at the authentication stage to deny
access for the specific user when there is a connection attempt from
an unauthorized ip for that user.

https://doc.dovecot.org/configuration_manual/authentication/allow_nets/

I don't believe there is anything that checks access at connect time
to deny unwanted traffic prior to authentication, for example from
compromised machines, botnets etc. Though failed connection attempts
do not appear to be a significant issue, maybe better to add some
safety net for the future.

The attached patch is proof of concept code that introduces the
parameters rbl_check and rbl_check_timeout (msecs) to the protocol
section. Tested for imap, pop3 and sieve. The following is an example
for sieve.

protocol sieve {
     rbl_check = zen.spamhaus.net=127.0.0.4

     rbl_check_timeout = 5000

}

If the lookup results in a hit the client is disconnected with a BYE
"Disconnected for policy." message and the logs report:

     Jun 09 12:00:56 server.example.com dovecot[977650]:
managesieve-login: Disconnected: Policy (disconnected before auth was
ready, waited 1 secs): user=<>, service=sieve, rip=n.n.n.n, lip=n.n.n.n

The patch also makes the number of pre-login errors and post-login
errors configurable (max_login_command_errors and max_command_errors
respectively) for pop3, imap and sieve protocols .

protocol sieve {

   max_command_errors = 1
   max_login_command_errors = 1

}

A potential extension to the logic would be "allow_nets" and
"disallow_nets" parameters or maybe something more sophisticated to
allow ips/networks that would otherwise be blocked or deny additional
ips/networks.

John


How is this different than using weakforced which already supports rbl? Or 
doing this in auth lua?


Hi Aki

I'm not that familiar with weakforced or auth lua, but I guess those 
intervene at the authentication stage, rather than the connection stage.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: [Patch] blocklist access control for dovecot pop3/imap and pigeonhole managesieve

[John Fawcett via dovecot]

The files themselves didn't make it through the list server. I'll try 
with .txt



On 09/06/2024 15:42, John Fawcett via dovecot wrote:

Hi

Just in case this is useful more generally, I'm posting it to the list.

While Dovecot has an access control via allow_nets, it is a user 
database field that applies only at the authentication stage to deny 
access for the specific user when there is a connection attempt from 
an unauthorized ip for that user.


https://doc.dovecot.org/configuration_manual/authentication/allow_nets/

I don't believe there is anything that checks access at connect time 
to deny unwanted traffic prior to authentication, for example from 
compromised machines, botnets etc. Though failed connection attempts 
do not appear to be a significant issue, maybe better to add some 
safety net for the future.


The attached patch is proof of concept code that introduces the 
parameters rbl_check and rbl_check_timeout (msecs) to the protocol 
section. Tested for imap, pop3 and sieve. The following is an example 
for sieve.


protocol sieve {
    rbl_check = zen.spamhaus.net=127.0.0.4

    rbl_check_timeout = 5000

}

If the lookup results in a hit the client is disconnected with a BYE 
"Disconnected for policy." message and the logs report:


    Jun 09 12:00:56 server.example.com dovecot[977650]: 
managesieve-login: Disconnected: Policy (disconnected before auth was 
ready, waited 1 secs): user=<>, service=sieve, rip=n.n.n.n, lip=n.n.n.n


The patch also makes the number of pre-login errors and post-login 
errors configurable (max_login_command_errors and max_command_errors 
respectively) for pop3, imap and sieve protocols .


protocol sieve {

  max_command_errors = 1
  max_login_command_errors = 1

}

A potential extension to the logic would be "allow_nets" and 
"disallow_nets" parameters or maybe something more sophisticated to 
allow ips/networks that would otherwise be blocked or deny additional 
ips/networks.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.orgdiff -x '*.log' -x '*.status' -x '*.deps' -x '*.a' -x '*.o' -ur 
dovecot-2.3.21-orig/src/config/all-settings.c 
dovecot-2.3.21/src/config/all-settings.c
--- dovecot-2.3.21-orig/src/config/all-settings.c   2023-09-14 
15:18:20.0 +0200
+++ dovecot-2.3.21/src/config/all-settings.c2024-06-09 11:31:45.820319492 
+0200
@@ -1750,6 +1750,7 @@
 struct pop3_settings {
bool verbose_proctitle;
const char *rawlog_dir;
+   unsigned int max_bad_commands;
 
/* pop3: */
bool pop3_no_flag_updates;
@@ -1849,6 +1850,10 @@
 
unsigned int mail_max_userip_connections;
 
+   const char *rbl_check;
+   unsigned int rbl_check_timeout;
+   unsigned int max_login_command_errors;
+
/* generated: */
char *const *log_format_elements_split;
 };
@@ -1907,6 +1912,7 @@
 struct imap_settings {
bool verbose_proctitle;
const char *rawlog_dir;
+   unsigned int max_command_errors;
 
/* imap: */
uoff_t imap_max_line_length;
@@ -3277,6 +3283,7 @@
 static const struct setting_define pop3_setting_defines[] = {
DEF(BOOL, verbose_proctitle),
DEF(STR_VARS, rawlog_dir),
+   DEF(UINT, max_bad_commands),
 
DEF(BOOL, pop3_no_flag_updates),
DEF(BOOL, pop3_enable_last),
@@ -3295,6 +3302,7 @@
 static const struct pop3_settings pop3_default_settings = {
.verbose_proctitle = FALSE,
.rawlog_dir = "",
+   .max_bad_commands = 20,
 
.pop3_no_flag_updates = FALSE,
.pop3_enable_last = FALSE,
@@ -4183,6 +4191,10 @@
 
DEF(UINT, mail_max_userip_connections),
 
+   DEF(STR, rbl_check),
+   DEF(TIME_MSECS, rbl_check_timeout),
+   DEF(UINT, max_login_command_errors),
+
SETTING_DEFINE_LIST_END
 };
 static const struct login_settings login_default_settings = {
@@ -4209,7 +4221,11 @@
.auth_debug = FALSE,
.verbose_proctitle = FALSE,
 
-   .mail_max_userip_connections = 10
+   .mail_max_userip_connections = 10,
+
+   .rbl_check = "",
+   .rbl_check_timeout = 10*1000,
+   .max_login_command_errors = 3
 };
 const struct setting_parser_info login_setting_parser_info = {
.module_name = "login",
@@ -4656,6 +4672,7 @@
 static const struct setting_define imap_setting_defines[] = {
DEF(BOOL, verbose_proctitle),
DEF(STR_VARS, rawlog_dir),
+   DEF(UINT, max_command_errors),
 
DEF(SIZE, imap_max_line_length),
DEF(TIME, imap_idle_notify_interval),
@@ -4677,6 +4694,7 @@
 static const struct imap_settings imap_default_settings = {
.verbose_proctitle = FALSE,
.rawlog_dir = "",
+   .max_command_errors = 20,
 
/* RFC-2683 recommends at least 8000 bytes. Some clients however don't
   break large message sets to

[Patch] blocklist access control for dovecot pop3/imap and pigeonhole managesieve

[John Fawcett via dovecot]

Hi

Just in case this is useful more generally, I'm posting it to the list.

While Dovecot has an access control via allow_nets, it is a user 
database field that applies only at the authentication stage to deny 
access for the specific user when there is a connection attempt from an 
unauthorized ip for that user.


https://doc.dovecot.org/configuration_manual/authentication/allow_nets/

I don't believe there is anything that checks access at connect time to 
deny unwanted traffic prior to authentication, for example from 
compromised machines, botnets etc. Though failed connection attempts do 
not appear to be a significant issue, maybe better to add some safety 
net for the future.


The attached patch is proof of concept code that introduces the 
parameters rbl_check and rbl_check_timeout (msecs) to the protocol 
section. Tested for imap, pop3 and sieve. The following is an example 
for sieve.


protocol sieve {
    rbl_check = zen.spamhaus.net=127.0.0.4

    rbl_check_timeout = 5000

}

If the lookup results in a hit the client is disconnected with a BYE 
"Disconnected for policy." message and the logs report:


    Jun 09 12:00:56 server.example.com dovecot[977650]: 
managesieve-login: Disconnected: Policy (disconnected before auth was 
ready, waited 1 secs): user=<>, service=sieve, rip=n.n.n.n, lip=n.n.n.n


The patch also makes the number of pre-login errors and post-login 
errors configurable (max_login_command_errors and max_command_errors 
respectively) for pop3, imap and sieve protocols .


protocol sieve {

  max_command_errors = 1
  max_login_command_errors = 1

}

A potential extension to the logic would be "allow_nets" and 
"disallow_nets" parameters or maybe something more sophisticated to 
allow ips/networks that would otherwise be blocked or deny additional 
ips/networks.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: sasl database

[John Fawcett via dovecot]

On 01/06/2024 13:22, Jeff P via dovecot wrote:

Hello,

Is there a guide for using a customized SASL database for dovecot?
for example, SQLite.

Thanks in advance.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Hi Jeff

I've no experience with using sqlite with dovecot but there is 
documentation here:


https://doc.dovecot.org/configuration_manual/authentication/sql/#sqlite

The same page also contains configurations for other db servers.

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Pigenohole bad commands limits and proposed minor patch

[John Fawcett via dovecot]

Hi

I was looking into Pigeonhole behaviour in the case of managesieve 
receiving invalid input either before or after login. I can see that 
there are hard coded limits after 3 sequential bad commands prior to 
authentication and 20 after authentication.


I was wondering if there is a reason for those values. Sieve is a 
protocol that is used with software and except for the limited use case 
of testing it is not used by people typing manually.


I don't know of a lot of sieve clients. Unfortunately the one I used in 
Thunderbird is no longer maintained, so I only have roundcube. But do 
legitimate sieve clients in general make a lot of mistakes?


An additional doubt about errors that I am seeing is that differently to 
imap and pop3 there doen't appear to be  a dedicated ssl port. I only 
have starttls configured. I do see what look like people trying to 
connect with ssl directly on port 4190, which with my configuration is 
never going to be valid.


I am attaching a very simple proposed patch to make the error limits 
configurable via the following settings in conf.d/20-managesieve.conf 
with defaults as per the existing hard coded ones:


  managesieve_max_command_errors = 20
  managesieve_login_max_command_errors = 3

I have similar doubts about imap error limits but I'll start with sieve 
ones.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Pigeonhole Sieve only runs scripts if in users' home directories, or if Sieve is purposely broken

[John Fawcett via dovecot]

On 15/05/2024 18:27, Douglas Morse via dovecot wrote:

My apologies, a small inaccuracy:

Below where I mention .svbin files being created, I incorrectly described what 
is occurring. If the `sieve` is set to the default of users’ home directories 
etc., the file `.dovecot.svbin` is created in the user’s home directory. If 
`sieve` is set to the centralized location, then there is no `.dovecot.svbin` 
but rather a `sieve.svbin` created or updated there. Also a bit confusing.


maybe you have a sieve_default setting that is affecting this.


Am 15.05.2024 um 18:18 schrieb Douglas Morse:

Hi,


Dovecot version is 2.3.19.1 (9b53102964) running on fully up-to-date Debian 12 
(Bookworm) operating system.


For the life of me I cannot figure out how to get sieve scripts working 
correctly in a central directory location. Calling them from users’ home 
directories works fine (as described below).

(Note: This system has recently been migrated from Cyrus IMAP, so some of the 
directories for a central location are very similar to what would find in a 
standard Cyrus IMAP installation.)


If I set `sieve` to its default value of `file:~/sieve;active=~/.dovecot.sieve` 
in `conf.d/90-sieve.conf`, sieve works as expected. I have a file 
`~/sieve/default.sieve` and, as per documentation, I make a symbolic link 
`~/.dovecot.sieve` that points to that file. The sieve script runs, and I see a 
`default.svbin` either created or updated in the ~/sieve directories as emails 
arrive.

Setting `sieve` to `file:~/lib/sieve;active=~/.dovecot.sieve` also works as 
expected. So this verifies I can change the directory where sieve scripts are 
stored.

However, if I then set `sieve` to 
`file:/var/spool/dovecot/home/%1n/user/%n/sieve;active=file:/var/spool/dovecot/home/%1n/user/%n/.dovecot.sieve`,
 again a `default.sieve` in each of these user’s centralized `sieve` 
directories, sieve scripts silently fail to run. No messages of any kind found 
in any log files.


I don't believe that the active option can take the "file:" prefix.

John



So, with the same exact same configuration as one that is the default and as 
proven to working, with the single exception of changing the top-level 
directory (TLD) for the scripts, results in silent failure. Why?

What I find particularly odd is that if I keep `sieve` set to this central 
directory location (i.e., no config changes), and `mv sieve/default.sieve 
sieve.tmp; rmdir sieve; mv sieve.tmp`, that is, have a *file* called `sieve` 
where the setting `sieve` points to (which, per docs and experience, should be 
a *directory*), the users’ sieve scripts *do* run, but with a warning along the 
lines of `Warning: sieve: file storage: Explicitly specified active script path 
`file:/var/spool/dovecot/home/m/user/morse/.dovecot.sieve' is ignored; storage 
path `/var/spool/dovecot/home/m/user/morse/sieve' is not a directory`.

In the docs I see that `sieve` used to refer to a file (PidgeonHole Sieve 
v.0.31 or prior, I believe), and there is mention of a a `sieve_dir` setting 
(which I have *never* set or used). So my guess is that sieve working in this 
centralized TLD with `sieve` as a file rather than directory is due to support 
for some deprecated config / functionality.

I have fiddle around with this for some hours, including setting and not 
setting `mail_home` to the central directory location (i.e., `mail_home` in 
`conf.d/10-mail.conf` set to the same value as `sieve` in 
`conf.d/90-sieve.conf`),to no avail. No matter what I do, I cannot get sieve 
scripts to run in a centralized directory location as they do under `/home`. 
The only exception is as just described in the preceding paragraph. But the 
solution only allows a user to have a single sieve script, and I imagine 
`managesieve` will not work properly is sieve is set up as described in the 
preceding paragraph.

What am I missing or doing incorrectly?

Any and all assistance most appreciated.

Doug


___
dovecot mailing list --dovecot@dovecot.org
To unsubscribe send an email todovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: post-login scripting

[John Fawcett]

On 18/01/2024 02:31, Alex wrote:
 Hi,
 I'm trying to determine the last login time for my IMAP users. Some
 systems use
 mbox while others are Maildir, but not sure that matters.

 I'd just like to touch a file similar to how it's done in this
 article, but it
 doesn't work for me.

 https://doc.dovecot.org/admin_manual/post_login_scripting/

 I've followed the instructions explicitly - made the required changes
 to 10-
 master.conf and created a shell script in /usr/local/bin that touches
 a file in
 the user's home directory. However, the file doesn't get created/
 touched.

 I've enabled debugging and don't see any errors that might lead to
 what I'm
 doing wrong. It doesn't say explicitly in the article that the last-
 login
 script referenced in the article is actually the contents of the
 postlogin.sh
 script, but I'm assuming it is?

 When I put the following in my dovecot.conf instead of conf.d/10-
 auth.conf, it
 core dumps. I thought local changes should be made in dovecot.conf?

 service imap-login {
   executable = imap imap-postlogin
   inet_listener imap {
     port = 0
   }
   inet_listener imaps {
     port = 993
   }
   process_min_avail = 6
   service_count = 0
   vsz_limit = 512 M
 }
 service imap-postlogin {
   executable = script-login /usr/local/bin/postlogin.sh
   user = $default_internal_user
 }
 service pop3-login {
   inet_listener pop3 {
     port = 0
   }
   inet_listener pop3s {
     port = 995
   }
 }
Hi Alex
I notice you didn't define the listner socket. You're missing this inside your
imap-postlogin service definition:
unix_listener imap-postlogin {
  }
John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: testing sieve/pigeonhole, auth errors

[John Fawcett]

On 15/01/2024 21:13, Joe Acquisto wrote:

A small SOHO system, of installed perhaps a year.  Seems to be working and
wished to add pigeonhole/sieve.
Per Pigeonhole/sieve troubleshooting in the docs, set on mail_debug = yes and
this is what I see, upon launching Thunderbird as the client.  Thought that
"unknown user" and "permission denied" deserved attention before any other
steps.
Just the dialog for one user is pasted below.
pid 475 is dovecot,vmail for dovecot imapd
This is what I see for /proc/self/io
blah # ll /proc/self/io
-r 1 root root 0 Jan 15 15:10 /proc/self/io
Jan 15 14:53:35 auth-worker(22903): Info: conn unix:auth-worker
(pid=22897,uid=475): auth-worker<1>: pam
(m...@here.com,192.168.11.166,): unknown user
Jan 15 14:53:35 auth-worker(22901): Info: conn unix:auth-worker
(pid=22897,uid=475): auth-worker<4>: passwd
(m...@here.com,192.168.11.166,): unknown user
Jan 15 14:53:35 imap-login: Info: Login: user=, method=PLAIN,
rip=192.168.11.166, lip=192.168.0.225, mpid=22907, session=
Jan 15 14:53:35 imap(m...@here.com)<22907>: Debug: Effective
uid=100, gid=1000, home=/home/vmail/m...@here.com
Jan 15 14:53:35 imap(m...@here.com)<22907>: Debug: open(/proc/
self/io) failed: Permission denied
Jan 15 14:53:35 imap(m...@here.com)<22907>: Debug: Namespace
inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes,
subscriptions=yes location=maildir:~/
Maildir
Jan 15 14:53:35 imap(m...@here.com)<22907>: Debug: maildir++:
root=/home/vmail/m...@here.com/Maildir, index=, indexpvt=, control=, 
inbox=/home/
vmail/joea@j4com
puters.com/Maildir, alt=
Jan 15 14:53:35 auth-worker(22904): Info: conn unix:auth-worker
(pid=22897,uid=475): auth-worker<1>: pam
(m...@here.com,192.168.11.166,): unknown user
Jan 15 14:53:35 auth-worker(22901): Info: conn unix:auth-worker
(pid=22897,uid=475): auth-worker<5>: passwd
(m...@here.com,192.168.11.166,): unknown user
Jan 15 14:53:35 imap-login: Info: Login: user=, method=PLAIN,
rip=192.168.11.166, lip=192.168.0.225, mpid=22908, session=
Jan 15 14:53:35 imap(m...@here.com)<22908>: Debug: Effective
uid=100, gid=1000, home=/home/vmail/m...@here.com
Jan 15 14:53:35 imap(m...@here.com)<22908>: Debug: open(/proc/
self/io) failed: Permission denied
Jan 15 14:53:35 imap(m...@here.com)<22908>: Debug: Namespace
inbox: type=private, prefix=, sep=, inbox=yes, hidden=no, list=yes,
subscriptions=yes location=maildir:~/
Maildir
Jan 15 14:53:35 imap(m...@here.com)<22908>: Debug: maildir++:
root=/home/vmail/m...@here.com/Maildir, index=, indexpvt=, control=, 
inbox=/home/
vmail/joea@j4com
puters.com/Maildir, alt=
Jan 15 14:53:36 imap(j...@acquisto-us.com)<22906>: Debug:
Mailbox INBOX: Mailbox opened
Jan 15 14:53:36 imap(m...@here.com)<22907>: Debug: Mailbox
INBOX: Mailbox opened
Jan 15 14:53:37 imap(j...@acquisto-us.com)<22906>: Debug:
Mailbox INBOX: UID 13710: Opened mail because: prefetch



Hi Joe

it looks like two different issues. The unknown user issue doesn't seem 
related to you having activated debugging, as far as I can see. Also you 
should check if that username is expected to exist or not. If the 
username does not exist, then the message is correct and you should 
investigate your client sending that request.


The permission denied is a consequence of imap running as dovecot user 
without access to that file which requires root user permission. I would 
not change the user though. Ideally a fix should be done to avoid this 
message, but I believe it is harmless, particuarly if you don't have any 
other issues.


I don't normally run debug mode, but when I have done so, I have seen 
that too.


best regards

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Virtual folder with custom flags

[John Fawcett]

On 15/01/2024 03:49, Kirill Korinsky wrote:

Greetings,

I can't figure out how doesn it possible to create a virtual folder which
contains mails which has falgs: \Flagged or any customers flag.

For example I do have a test mail:

mx2$ doas doveadm fetch -u kir...@korins.ky "mailbox flags" mailbox-guid 
fb9e821c7b778b65889679e31c01 uid 67096
mailbox: Archive
flags: \Flagged \Seen $NotJunk NotJunk $MailFlagBit0 $MailFlagBit2 Invoice
mx2$

and virtual folder:

mx2$ cat /etc/dovecot/virtual/Flagged/dovecot-virtual
*
-INBOX/Trash
-INBOX/Trash/*
-INBOX/Junk
-INBOX/Junk/*
   flagged
mx2$

If I unset flag \Flagged from that mail, it dissapear from the folder.

So, how should I change definition to keep it here if it has user's specified
flags like Invoice or ASAP or whatsoever?


Hi Kirill

the possible criteria that can be used in the search criteria for 
virtual folders are the ones in the SEARCH part of the stanard.


There is a KEYWORD criteria for searching for a flag but it's only 
useful if you know the names of the flag.


There doesn't seem to be a wild card capability for the flag name or a 
search criteria that says "if there's any user flags" or for that matter 
"if there's any flags". The flagged criteria only checks for the 
\Flagged flag, so the behaviour where it disappears from the folder when 
removing \Flagged looks correct.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Quota clone plugin - initial population of data

[John Fawcett]

Hi

while responding on a previous thread, I noticed in the documentation:

https://doc.dovecot.org/configuration_manual/quota_clone_plugin/

"For example you might want to access everybody’s quota via Redis ... 
but you don’t store the Redis database permanently so it could become 
empty once in a while. "


It aslo seems that the quota clone plugin propagates only quota changes. 
Putting both of these points together, you'll going to have missing 
quotas in redis if it is restarted. It will gradually get populated but 
only on basis of mailbox changes.


Maybe there should be an initial load of all the quotas when dovecot 
starts, or could that be a performance issue? Maybe even doveadm quota 
recalc should propagate or contain an option to propage quotas so that 
even if it can't be done on dovecot startup, at least it can be done on 
demand by the administrator.


John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Custom map for quota

[John Fawcett]

On 15/01/2024 01:10, m87562...@gmail.com wrote:
 John Fawcett wrote:
  On 13/01/2024 14:01, Matt wrote:
   Hi,
   I'm trying to setup quota with the following
   layout
   CREATE TABLE "domain" (
   "id" INTEGER,
   "name" TEXT,
   "active" INTEGER DEFAULT 0,
   PRIMARY KEY("id"),
   UNIQUE("name")
   )
   CREATE TABLE "address" (
   "id" INTEGER,
   "localpart" TEXT NOT NULL,
   "domain_id" INTEGER NOT NULL,
   PRIMARY KEY("id"),
   UNIQUE("localpart","domain_id")
   )
   CREATE TABLE "mailbox" (
        "id" INTEGER,
   "address_id" INTEGER,
   "active" INTEGER DEFAULT 0,
   "password" TEXT,
   "quota_bytes" INTEGER DEFAULT 0,
   PRIMARY KEY("id")
   )
   As I like to update the quota and I like to use a
   dict/map I'm not sure if I
   can update this following:
   map {
  pattern = priv/quota/storage
  table = maibox
  value_field = dummy
  fields {
    value_field = quota_bytes
    id = mailbox_id
  }
   mailbox_id will be queried using password_query
   using: "mailbox.id as
   mailbox_id"
   Using such map I hope to be able to use:
   SELECT quota_bytes FROM mailbox WHERE id =
   '$mailbox_id'
   INSERT INTO mailbox (quota_bytes) VALUES
   ('$value') ON DUPLICATE KEY UPDATE
   quota_bytes='$value'
   Can someone help me with this ?
   Hi Matt
 Hi John, thank you very much for reaching out to me with your clear
 explanation, I will respond below your quoted text.
  as far as I know the quota plugin with the dict backend
  requires you to
  configure the name of your username field in the map. The
  usual way to
  configure it is to have a quota table like in the
  documentation.
  https://doc.dovecot.org/configuration_manual/quota/
  quota_dict/
  In your case where you don't have a username field in the
  table that
  stores the quotas, then depending on your database, you may
  be able to
  create a view that does include the username and configure
  that in Dovecot.
 Yes, the idea was a view from my side as well but as I try to keep
 things as clean as possible I didn't mention it yet, also because -
 you say later on in your reply - quota_dict is deprecated so I
 already implemented "count" but was not sure about quota_clone as I
 assumed I needed Redis because the manual said as "more complex
 example" dict in the MySQL part so I was confused and thought to
 implement Redis because of it; it seems I can put in almost any
 backend that uses dict it seems.

 https://doc.dovecot.org/configuration_manual/quota_clone_plugin
  The quota plugin updates the values of two fields holding
  the number of
  messages and the number of bytes used. I noticed you don't
  have the
  messages field. I suggest to add it.
 I know, I copied that part from the documentation when trying to find
 a sane config; messages will be added but thank for you notice when
 people read this thread later on. (I lots of them confused me all the
 time as this was not a major one for me actually - point taken!)

 https://doc.dovecot.org/configuration_manual/dict/#dict-sql
  You shouldn't need to write any queries to update the
  quota: Dovecot
  will generate the queries.
 True and that is what I like so the question remains what to do here,
 a dict for Mysql with a view should do the trick I believe to
 understand ?
  On a final note, before you invest time in this, it may be
  worthwhile
  keeping an eye on the future plans
  https://doc.dovecot.org/3.0/installation_guide/upgrading/
  from-2.3-to-3.0/
 Thanks for that! As I wasn't upgrading (yet) I could not find any
 roadmap; this is what I needed!
  You could use the count backend with quota plugin (which is
  the
  recommended backend for new installations) and then use the
  quota_clone
  plugin to keep your database in sync.
 Indeed, but this will be a map in combination with a (SQL) dict and
 the bespoken view

Re: Custom map for quota

[John Fawcett]

On 13/01/2024 14:01, Matt wrote:

Hi,

I'm trying to setup quota with the following layout

CREATE TABLE "domain" (
"id" INTEGER,
"name" TEXT,
"active" INTEGER DEFAULT 0,
PRIMARY KEY("id"),
UNIQUE("name")
)

CREATE TABLE "address" (
"id" INTEGER,
"localpart" TEXT NOT NULL,
"domain_id" INTEGER NOT NULL,
PRIMARY KEY("id"),
UNIQUE("localpart","domain_id")
)

CREATE TABLE "mailbox" (
         "id" INTEGER,
"address_id" INTEGER,
"active" INTEGER DEFAULT 0,
"password" TEXT,
"quota_bytes" INTEGER DEFAULT 0,
PRIMARY KEY("id")
)

As I like to update the quota and I like to use a dict/map I'm not sure if I
can update this following:

map {
   pattern = priv/quota/storage
   table = maibox
   value_field = dummy
   fields {
     value_field = quota_bytes
     id = mailbox_id
   }


mailbox_id will be queried using password_query using: "mailbox.id as
mailbox_id"

Using such map I hope to be able to use:

SELECT quota_bytes FROM mailbox WHERE id = '$mailbox_id'
INSERT INTO mailbox (quota_bytes) VALUES ('$value') ON DUPLICATE KEY UPDATE
quota_bytes='$value'

Can someone help me with this ?


Hi Matt

as far as I know the quota plugin with the dict backend requires you to 
configure the name of your username field in the map. The usual way to 
configure it is to have a quota table like in the documentation.


https://doc.dovecot.org/configuration_manual/quota/quota_dict/

In your case where you don't have a username field in the table that 
stores the quotas, then depending on your database, you may be able to 
create a view that does include the username and configure that in Dovecot.


The quota plugin updates the values of two fields holding the number of 
messages and the number of bytes used. I noticed you don't have the 
messages field. I suggest to add it.


You shouldn't need to write any queries to update the quota: Dovecot 
will generate the queries.


On a final note, before you invest time in this, it may be worthwhile 
keeping an eye on the future plans


https://doc.dovecot.org/3.0/installation_guide/upgrading/from-2.3-to-3.0/

You could use the count backend with quota plugin (which is the 
recommended backend for new installations) and then use the quota_clone 
plugin to keep your database in sync.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Panic: file index-search-result.c: line 132 (index_search_result_update_flags): assertion failed: (result->search_args->args == _arg)

[John Fawcett]

On 09/01/2024 19:00, Markus Doits via dovecot wrote:


Am 03.01.24 um 10:14 schrieb Aki Tuomi:

namespace {
    prefix = _/
    separator = /
    location =
virtual:~/Maildir/virtual:LAYOUT=maildir++:INDEX=~/Maildir/virtual
}



You should not store virtual folders inside Maildir (assuming this is 
your mail location). Please use
~/virtual/ instead. The INDEX directive is not needed in this case as 
it points to same place anyways.


Thanks for your suggestion! I changed it and moved the virtual folders 
out of the Maildir manually:


```
location = virtual:~/virtual:LAYOUT=maildir++
```

Unfortunately the same panic ist still there.

I just noticed there is another panic with the same assertion and 
backtrace but a different file:


```
Panic: file index-search-result.c: line 174 
(index_search_result_update_appends): assertion failed: 
(result->search_args->args == _arg)

```

By the way there is nothing that seems to be broken (fts searching 
seems to work like it should), but the logs are filled with this 
panics and backtraces. 路‍♂️


Some more info: The panics happen after logging in (tested with 
Thunderbird, Roundcube, K-9 Mail on Android). I cannot trigger the 
panic when doing a fts search after login (e.g. on Thunderbird or K-9 
which holds the connection after login).


Hi Markus

my guess is that it has nothing to do with fts or flatcurve, but when 
index_search_result_update_flags is invoked from the virtual plugin.


From the comment in index_search_result_update_flags a search parameter 
is added in order to search for only changed messages. That parameter is 
added at the front of a linked list of search parameters and after a 
call to search_result_update_search, it is still expected to be there at 
the front of the list, but isn't. Presumably either another search 
parameter was added on the front of the linked list or the existing one 
was removed.


My suggestion would be to try and capture some debug logging if it is 
possible. If this is an error that other people haven't found, I am 
wondering if it's linked to soemthing specific in your dovecot-virtual 
file(s).


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Autocreate virtual users' home directory?

[John Fawcett]

On 04/01/2024 08:21, Aki Tuomi via dovecot wrote:

On 03/01/2024 23:38 EET John Fawcett  wrote:

  


Hi
I have recently rectified an old configuration error whereby I had virtual
users' home and mail directories set to the same value.
I used option 3 from "ways to set up home directory"
https://doc.dovecot.org/configuration_manual/
home_directories_for_virtual_users/
so I have:
home= /var/vmail/domain/user/home/
mail= /var/vmail/domain/user/
where domain and user are substituted for the real domain and user.
For existing users where I created the home directory when migrating the
settings, there is no issue. But I am seeing errors like this for new users:
lmtp(t...@example.com)<125035>: Error: autoexpunge:
Couldn't create dovecot.autoexpunge.lock lock: file_create_locked(/var/vmail/
example.com/t...@example.com/home//dovecot.autoexpunge.lock) failed:
safe_mkstemp(/var/vmail/example.com/t...@example.com/home//
dovecot.autoexpunge.lock) failed: No such file or directory
Seems that differently to mail directory which is automatically created, the
home directory is not created. I couldn't see any config setting that would
autocreate home directories (where home directories are subdirectories of mail
directories).
Would it be worthwhile introducing an autocreate option for home directories?
John




I don't really understand why you want home to be under mail directory? How 
about swapping them around like

home= /var/vmail/domain/user/
mail= /var/vmail/domain/user/mail

because this makes more sense in many ways?

Aki


Hi Aki

thanks, it's clear that swapping them would solve it since mail 
directories are being auto created. I can do that.


I was following one of the documented ways of doing it, albeit not the 
most preferred way (see point 3 of ways to set up home directory half 
way down the page). I guess that both option 2 and 3 will not work 
properly unless the home directory is created outside of dovecot (or a 
sieve script is published via mangesieve which does autocreate home 
directory). Option 4 is what I was using but just discovered it causes 
issues (with file that begin with . being interpreted as directories). 
It could be useful to add those notes to the documentation.


https://doc.dovecot.org/configuration_manual/home_directories_for_virtual_users/

best regards
John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Autocreate virtual users' home directory?

[John Fawcett]

Hi
I have recently rectified an old configuration error whereby I had virtual
users' home and mail directories set to the same value.
I used option 3 from "ways to set up home directory"
https://doc.dovecot.org/configuration_manual/
home_directories_for_virtual_users/
so I have:
home= /var/vmail/domain/user/home/ 
mail= /var/vmail/domain/user/
where domain and user are substituted for the real domain and user.
For existing users where I created the home directory when migrating the
settings, there is no issue. But I am seeing errors like this for new users:
lmtp(t...@example.com)<125035>: Error: autoexpunge:
Couldn't create dovecot.autoexpunge.lock lock: file_create_locked(/var/vmail/
example.com/t...@example.com/home//dovecot.autoexpunge.lock) failed:
safe_mkstemp(/var/vmail/example.com/t...@example.com/home//
dovecot.autoexpunge.lock) failed: No such file or directory
Seems that differently to mail directory which is automatically created, the
home directory is not created. I couldn't see any config setting that would
autocreate home directories (where home directories are subdirectories of mail
directories).
Would it be worthwhile introducing an autocreate option for home directories?
John





___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Solr/Tika patches for dovecot 2.3.21

[John Fawcett]

On 03/01/2024 10:16, Aki Tuomi via dovecot wrote:

On 09/12/2023 15:02 EET John Fawcett  wrote:

  
Hi


I've just made updated versions of 3 patches against the latest release
2.3.21 in case they are useful to someone or might get considered for
official inclusion.

John
___

Could you please post these into https://github.com/dovecot/core against main 
please?

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Thanks Aki, please see

https://github.com/dovecot/core/pull/215

All three patches are in a single pull request, but each patch has a 
separate commit. Also just noticed that pull request 213 contains a fix 
about solr "rows" query parameter with a different approach. If you 
intend to merge #213, let me know and I can either remove my patch or 
adjust it, whichever is preferred, since both approaches could co-exist.


John


John
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Certificate of the documentation site (https://doc.dovecot.org/)

[John Fawcett]

On 02/01/2024 10:55, gileh13242--- via dovecot wrote:

Hey !

Your certificate expired today, I can't access it because of my business proxy 
:).


[No Error] (TLS code: X509_V_ERR_CERT_HAS_EXPIRED)
SSL Certificate expired on: Jan 2 06:31:02 2024 GMT

Maybe you have the old certificate cached somewhere. I see the 
doc.dovecot.org site returns a certificate with valid dates that seems 
to have been renewed a month ago.


openssl s_client -connect doc.dovecot.org:443
...
Certificate chain
 0 s:CN = doc.dovecot.org
   i:C = US, O = Let's Encrypt, CN = R3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec  3 06:29:22 2023 GMT; NotAfter: Mar  2 06:29:21 
2024 GMT

 ...

John

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Solr/Tika patches for dovecot 2.3.21

[John Fawcett]

Hi
I've just made updated versions of 3 patches against the latest release 2.3.21
in case they are useful to someone or might get considered for official
inclusion.
John
dovecot-2.3.21-tika-http-auth.patch
Allows specification of username and password in the fts_tika setting for basic
auth against tika server. For example
fts_tika = https://user:password@tika_server:443/tika
dovecot-2.3.21-solr-max-size.patch
This is a simplified version of my previous patch. Sets a size limit
(configuration fts_max_size) on message bodies that are to be indexed. Message
bodies for messages larger than fts_max_size are not sent to solr. Defaults to
zero which means no limit. For example
fts_max_size = 10M
dovecot-2.3.21-solr-max-rows.patch
When dovecot sends a search to solr it uses the rows parameter. For multiple
mailbox search the value used is SOLR_MAX_MULTI_ROWS , hardcoded to 10. For
single mailbox search the value is uidnext. This patch introduces an upper
limit for single mailbox search using the same value as SOLR_MAX_MULTI_ROWS,
while leaving the existing functionality of sending the uidnext value if it is
smaller. This is just to place a more reasonable upper bound since uidnext can
get much larger.

--- dovecot-2.3.21/src/plugins/fts/fts-parser-tika.c	2023-09-14 15:17:47.0 +0200
+++ dovecot-2.3.21-new/src/plugins/fts/fts-parser-tika.c	2023-12-09 11:07:46.436259394 +0100
@@ -57,7 +57,7 @@
 	tuser = p_new(user->pool, struct fts_parser_tika_user, 1);
 	MODULE_CONTEXT_SET(user, fts_parser_tika_user_module, tuser);
 
-	if (http_url_parse(url, NULL, 0, user->pool,
+	if (http_url_parse(url, NULL, HTTP_URL_ALLOW_USERINFO_PART, user->pool,
 			   >http_url, ) < 0) {
 		i_error("fts_tika: Failed to parse HTTP url %s: %s", url, error);
 		return -1;
@@ -159,6 +159,11 @@
 			http_url->host.name,
 			t_strconcat(http_url->path, http_url->enc_query, NULL),
 			fts_tika_parser_response, parser);
+	if (http_url->user != NULL) {
+		http_client_request_set_auth_simple(
+			http_req, http_url->user, http_url->password);
+	}
+
 	http_client_request_set_port(http_req, http_url->port);
 	http_client_request_set_ssl(http_req, http_url->have_ssl);
 	if (parser_context->content_type != NULL)
--- dovecot-2.3.21/src/plugins/fts/fts-build-mail.c	2023-09-14 15:17:47.0 +0200
+++ dovecot-2.3.21-new/src/plugins/fts/fts-build-mail.c	2023-12-09 11:04:02.205207091 +0100
@@ -17,6 +17,7 @@
 #include "fts-filter.h"
 #include "fts-api-private.h"
 #include "fts-build-mail.h"
+#include "settings-parser.h"
 
 /* there are other characters as well, but this doesn't have to be exact */
 #define IS_WORD_WHITESPACE(c) \
@@ -573,6 +574,18 @@
 	bool binary_body;
 	const char *error;
 	int ret;
+	uoff_t msg_size;
+	uoff_t fts_max_size = 0;
+	const char * fts_max_size_setting;
+	bool oversized_msg;
+
+	fts_max_size_setting = mail_user_plugin_getenv(update_ctx->backend->ns->user, "fts_max_size");
+	if (fts_max_size_setting != NULL) {
+		if (settings_get_size(fts_max_size_setting, _max_size, ) < 0) {
+		i_error("%s",error);
+			fts_max_size = 0;
+		}
+	}
 
 	*may_need_retry_r = FALSE;
 	if (mail_get_stream_because(mail, NULL, NULL, "fts indexing", ) < 0) {
@@ -583,6 +596,14 @@
 		return -1;
 	}
 
+	oversized_msg = FALSE;
+	i_stream_get_size(input,TRUE,_size);
+	if (fts_max_size > 0 && msg_size > fts_max_size) {
+		i_info("Skipping message body indexing because size %"PRIuUOFF_T" exceeds setting fts_max_size %s",msg_size,fts_max_size_setting);
+		oversized_msg = TRUE;
+	}
+
+
 	i_zero();
 	ctx.update_ctx = update_ctx;
 	ctx.mail = mail;
@@ -640,7 +661,7 @@
 message_decoder_set_return_binary(decoder, TRUE);
 			body_part = TRUE;
 		} else {
-			if (skip_body)
+			if (skip_body||oversized_msg)
 continue;
 		}
 
@@ -675,7 +696,7 @@
 		else
 			(void)fts_parser_deinit(_parser, NULL);
 	}
-	if (ret == 0 && body_part && !skip_body && !body_added) {
+	if (ret == 0 && body_part && !skip_body && !oversized_msg && !body_added) {
 		/* make sure body is added even when it doesn't exist */
 		block.data = NULL; block.size = 0;
 		ret = fts_build_body_block(, , TRUE);
--- dovecot-2.3.21/src/plugins/fts-solr/fts-backend-solr.c	2023-09-14 15:17:47.0 +0200
+++ dovecot-2.3.21-new/src/plugins/fts-solr/fts-backend-solr.c	2023-12-09 10:46:52.976808250 +0100
@@ -837,7 +837,7 @@
 
 	str = t_str_new(256);
 	str_printfa(str, "wt=xml=uid,score=%u=uid+asc=%%7b!lucene+q.op%%3dAND%%7d",
-		status.uidnext);
+		   I_MIN(status.uidnext,SOLR_MAX_MULTI_ROWS)); 
 	prefix_len = str_len(str);
 
 	if (solr_add_definite_query_args(str, args, and_args)) {
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Username used by dovecot to access database certificate and key files

[John Fawcett]

On 08/12/2023 14:18, Aki Tuomi via dovecot wrote:


The group depends on process, usually dovecot uses four different identities:

root, dovecot:dovecot, dovenull:dovenull and vmail:vmail (or per-user identity).

Any extra groups are removed during execution, but if you want, you can use

https://doc.dovecot.org/configuration_manual/service_configuration/#extra-groups

like

service imap {
   ...
   extra_groups = users
}

to grant the process users membership during execution.

Aki
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Thanks Aki, with this info I solved it. I didn't get to the bottom of 
which process needed the extra_groups attribute, but changing the files 
group ownership to dovecot was just as good.


John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Username used by dovecot to access database certificate and key files

[John Fawcett]

Hi

does anyone happen to know the username under which dovecot accesses the 
certificate and key files when connecting to a mariadb database with 
ssl? I can't seem to get dovecot to read the files without a "other" 
readable bit set.


I have dovecot users in mysql with the following connect string in 
dovecot-sql.conf.ext


connect = host=x.x.x host=y.y.y host=z.z.z dbname=maildb user=mailuser 
password=xx \

    ssl_ca=/etc/ssl/mysql/ca-cert.pem \
    ssl_cert=/etc/ssl/mysql/client-cert.pem \
    ssl_key=/etc/ssl/mysql/client-key.pem \
    ssl_verify_server_cert=yes

The certificate and key files in /etc/ssl/mysql/ are owned by root with 
group users:


ls -l /etc/ssl/mysql

total 12
-r--r--r--. 1 root root  2041 Jun 22 15:32 ca-cert.pem
-r--r--r--. 1 root users 1907 Jun 22 15:32 client-cert.pem
-r--r--r--. 1 root users 3272 Jun 22 15:32 client-key.pem

For my test dovecot, dovenull and vmail are all members of group users:

grep users /etc/group
users:x:100:postfix,dovecot,dovenull,vmail

If I remove the "other" readable attribute on the certificate and key 
files, dovecot complains it can't connect to the db server:


auth: Error: mysql(x.x.x): Connect failed to database (maildb): TLS/SSL 
error: Permission denied (13) - waiting for 1 seconds before retry


I realize it only adds marginal security to not have these files world 
readable, but still it is puzzling why dovecot cannot read them in that 
case. Can anyone see what I am missing?


Thanks

John


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: source code doesn't compile

[John Fawcett]

On 12/01/2022 04:19, Ruben Safir wrote:

I downloaded the dovecot source and it doesn't compile.  The install
script fails in auth because it makes a file and then tries to make a
directory over it

make[2]: Entering directory '/home/ruben/src/dovecot-2.3.17.1/src/auth'
make[3]: Entering directory
'/home/ruben/src/dovecot-2.3.17.1/src/auth'
   /usr/bin/mkdir -p '/usr/local/lib/dovecot'
/bin/sh ../../libtool   --mode=install /usr/bin/install -c
auth checkpassword-reply '/usr/local/lib/dovecot'
  libtool: warning: 'libstats_auth.la' has not been installed
in '/usr/local/lib/dovecot/old-stats'
  libtool: install: /usr/bin/install -c .libs/auth
/usr/local/lib/dovecot/auth
  libtool: install: /usr/bin/install -c
.libs/checkpassword-reply /usr/local/lib/dovecot/checkpassword-reply
   /usr/bin/mkdir -p '/usr/local/lib/dovecot/auth'

I am running an older version of dovecott and would really like to
upgrade it and there is no package for this linux version


I guess you want these to be installed in /usr/local/libexec/dovecot 
directory not /usr/local/lib.


libtool: install: /usr/bin/install -c .libs/auth 
/usr/local/libexec/dovecot/auth
libtool: install: /usr/bin/install -c .libs/checkpassword-reply 
/usr/local/libexec/dovecot/checkpassword-reply


According to ./configure --help this is the default:

  --libexecdir=DIR    program executables [EPREFIX/libexec]

John

Patch: enhancements for solr/tika integration

[John Fawcett]

Hi

here's a patch with some enhancements that I am applying locally for 
solr/tika integration. Hopefully this can be considered for inclusion. 
I've tested up to 2.3.16 and this patch applies against latest version 
2.3.17.1. The contents are:


1. Allow username and password in tika configuration (as is already 
supported for solr). For example:


fts_tika = https://user:password@tika_host:tika_port/tika/tika

This means that tika can be more easily located on a different server to 
dovecot. Locating Solr and Tika on separate servers can isolate the 
Dovecot server from issues that may happen on Solr or Tika (like out of 
memory).


2. Introduce an optional configurable limit on the size of emails whose 
message bodies will be indexed with solr via a new setting fts_max_size, 
to be put in the plugin section. For example


plugin {

...

fts_max_size = 2M

...

}

Message body indexing is skipped for messages that are larger than this 
setting.


3. Introduce an optional configurable limit on the size of emails whose 
attachments will be sent to tika for keyword extraction, via a new 
setting fts_max_size_tika, to be put in the plugin section. For example:


plugin {

...

fts_max_size_tika = 1M

}
I noticed that some very large attachments are being sent to tika and 
these often do not have any useful content to index. For simplicity the 
check is done on message size not individual or total attachment size.


Here's the patch:

diff -ur dovecot-2.3.17.1-orig/src/plugins/fts/fts-build-mail.c 
dovecot-2.3.17.1-new/src/plugins/fts/fts-build-mail.c
--- dovecot-2.3.17.1-orig/src/plugins/fts/fts-build-mail.c 2021-12-03 
12:48:47.0 +0100
+++ dovecot-2.3.17.1-new/src/plugins/fts/fts-build-mail.c 2022-01-09 
01:33:23.398341998 +0100

@@ -17,6 +17,7 @@
 #include "fts-filter.h"
 #include "fts-api-private.h"
 #include "fts-build-mail.h"
+#include "settings-parser.h"

 /* there are other characters as well, but this doesn't have to be 
exact */

 #define IS_WORD_WHITESPACE(c) \
@@ -25,6 +26,8 @@
    wherever */
 #define MAX_WORD_SIZE 1024

+static bool debug = FALSE;
+
 struct fts_mail_build_context {
    struct mail *mail;
    struct fts_backend_update_context *update_ctx;
@@ -34,6 +37,7 @@

    buffer_t *word_buf, *pending_input;
    struct fts_user_language *cur_user_lang;
+   bool oversized_tika;
 };

 static int fts_build_data(struct fts_mail_build_context *ctx,
@@ -223,7 +227,7 @@
    parser_context.user = mail_storage_get_user(storage);
    parser_context.content_disposition = ctx->content_disposition;

-
+   parser_context.oversized_tika = ctx->oversized_tika;
    if (fts_parser_init(_context, >body_parser)) {
    /* extract text using the the returned parser */
    *binary_body_r = TRUE;
@@ -496,7 +500,32 @@
    bool binary_body;
    const char *error;
    int ret;
-
+   uoff_t msg_size;
+   uoff_t fts_max_size = 0;
+   uoff_t fts_max_size_tika = 0;
+   const char * fts_max_size_setting;
+   const char * fts_max_size_tika_setting;
+   bool oversized_msg;
+   bool oversized_tika;
+
+   fts_max_size_setting = 
mail_user_plugin_getenv(update_ctx->backend->ns->user, "fts_max_size");

+   if (fts_max_size_setting != NULL) {
+   if (debug) i_debug("fts_max_size %s",fts_max_size_setting);
+   if (settings_get_size(fts_max_size_setting, 
_max_size, ) < 0) {

+   i_error("%s",error);
+   fts_max_size = 0;
+   }
+   if (debug) i_debug("fts_max_size (value) 
%"PRIuUOFF_T,fts_max_size);

+   }
+   fts_max_size_tika_setting = 
mail_user_plugin_getenv(update_ctx->backend->ns->user, 
"fts_max_size_tika");

+   if (fts_max_size_tika_setting != NULL) {
+   if (debug) i_debug("fts_max_size_tika 
%s",fts_max_size_tika_setting);
+   if (settings_get_size(fts_max_size_tika_setting, 
_max_size_tika, ) < 0) {

+   i_error("%s",error);
+   fts_max_size_tika = 0;
+   }
+   if (debug) i_debug("fts_max_size_tika (value) 
%"PRIuUOFF_T,fts_max_size_tika);

+   }
    *may_need_retry_r = FALSE;
    if (mail_get_stream_because(mail, NULL, NULL, "fts indexing", 
) < 0) {

    if (mail->expunged)
@@ -505,10 +534,21 @@
mailbox_get_last_internal_error(mail->box, NULL));
    return -1;
    }
-
+   oversized_msg = FALSE;
+   oversized_tika = FALSE;
+   i_stream_get_size(input,TRUE,_size);
+   if (fts_max_size > 0 && msg_size > fts_max_size) {
+   i_info("Skipping message body indexing because size 
%"PRIuUOFF_T" exceeds setting fts_max_size 
%s",msg_size,fts_max_size_setting);

+   oversized_msg = TRUE;
+   }
+   if (fts_max_size_tika > 0 && msg_size > fts_max_size_tika) {
+   i_info("Skipping message attachment indexing because 
size 

Patch: safeguard against too large value for uid being sent to solr for single mailbox searches

[John Fawcett]

Hi

I'm reposting a patch for solr integration which I have been applying 
locally. It applies against 2.3.17.1.


Dovecot already has a mechanism when doing solr fts searches on multiple 
mailboxes that prevents a too large value for maximum rows being sent to 
solr.


#define SOLR_MAX_MULTI_ROWS 10

(defined and used in plugins/fts-solr/fts-backend-solr.c)

However on single mailbox searches Dovecot uses the last uid of the 
mailbox, which can be too large for solr. My suggested patch is to also 
limit maximum rows to return on single mailbox searches.


Here is the patch:

diff -ur dovecot-2.3.17.1-orig/src/plugins/fts-solr/fts-backend-solr.c 
dovecot-2.3.17.1-new/src/plugins/fts-solr/fts-backend-solr.c
--- dovecot-2.3.17.1-orig/src/plugins/fts-solr/fts-backend-solr.c 
2021-12-03 12:48:47.0 +0100
+++ dovecot-2.3.17.1-new/src/plugins/fts-solr/fts-backend-solr.c 
2022-01-09 01:33:23.401341959 +0100

@@ -837,7 +837,7 @@

    str = t_str_new(256);
    str_printfa(str, 
"wt=xml=uid,score=%u=uid+asc=%%7b!lucene+q.op%%3dAND%%7d",

-   status.uidnext);
+   I_MIN(status.uidnext,SOLR_MAX_MULTI_ROWS));
    prefix_len = str_len(str);

    if (solr_add_definite_query_args(str, args, and_args)) {

John

Patch: assertion failed in doveadm fts lookup

[John Fawcett]

Hi

I'm reposting this patch, which I have been applying locally since I 
originally posted it. Hopefully it can be considered for inclusion in 
the official release. It was tested on dovecot 2.3.16 but applies 
against the latest version 2.3.17.1.


This is the "assertion failed" that it addresses

doveadm fts lookup -u j...@voipsupport.it body "text to look for"

doveadm(j...@voipsupport.it): Panic: file mail-storage.c: line 2108 
(mailbox_get_open_status): assertion failed: (box->opened)
doveadm(j...@voipsupport.it): Error: Raw backtrace: 
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) 
[0x7ff624fa7072] -> 
/usr/local/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) 
[0x7ff624fa718e] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xffc3b) 
[0x7ff624fb3c3b] -> /usr/local/lib/dovecot/libdovecot.so.0(+0xffc71) 
[0x7ff624fb3c71] -> /usr/local/lib/dovecot/libdovecot.so.0(+0x5536f) 
[0x7ff624f0936f] -> 
/usr/local/lib/dovecot/libdovecot-storage.so.0(+0x3fc31) 
[0x7ff6250bcc31] -> 
/usr/local/lib/dovecot/lib21_fts_solr_plugin.so(+0x540d) 
[0x7ff62244f40d] -> 
/usr/local/lib/dovecot/lib20_fts_plugin.so(fts_backend_lookup+0x4d) 
[0x7ff6246b624d] -> 
/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so(+0x2e8c) 
[0x7ff62240ce8c] -> doveadm(+0x31ded) [0x55f0c7f3aded] -> 
doveadm(+0x32862) [0x55f0c7f3b862] -> 
doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x55f0c7f3c6fd] -> 
doveadm(doveadm_cmd_run_ver2+0x4d8) [0x55f0c7f4d158] -> 
doveadm(doveadm_cmd_try_run_ver2+0x3a) [0x55f0c7f4d1aa] -> 
doveadm(main+0x1f6) [0x55f0c7f2b606] -> 
/lib64/libc.so.6(__libc_start_main+0xd5) [0x7ff624bc8b75] -> 
doveadm(_start+0x2e) [0x55f0c7f2ba0e]

Aborted (core dumped)

Here's the patch:

diff -ur dovecot-2.3.17.1-orig/src/plugins/fts/doveadm-fts.c 
dovecot-2.3.17.1-new/src/plugins/fts/doveadm-fts.c
--- dovecot-2.3.17.1-orig/src/plugins/fts/doveadm-fts.c 2021-12-03 
12:48:47.0 +0100
+++ dovecot-2.3.17.1-new/src/plugins/fts/doveadm-fts.c  2022-01-09 
01:33:23.396342025 +0100

@@ -47,6 +47,14 @@
    i_array_init(, 16);

    box = mailbox_alloc(info->ns->list, info->vname, 0);
+   mailbox_set_reason(box,"fts search");
+   if (mailbox_open(box) < 0) {
+   i_error("Couldn't open mailbox: %s",
+   mailbox_get_last_internal_error(box, NULL));
+   doveadm_mail_failed_error(ctx, MAIL_ERROR_TEMP);
+   return -1;
+   }
+
    if (fts_backend_lookup(backend, box, ctx->search_args->args,
  FTS_LOOKUP_FLAG_AND_ARGS, 
) < 0) {

    i_error("fts lookup failed");

John

Re: banning, was Re: Non-user logins?

[John Fawcett]

On 08/01/2022 17:22, Dave McGuire wrote:

On 1/8/22 8:57 AM, John Fawcett wrote:
yes, blocking on the first wrong password sounds like overkill. But 
it does depend on user base. For a small mail server with few known 
users it could be workable.


  It may be overkill for your network, but it's not overkill for mine.

But even on small servers I'd recommend blocking for a small time 
(like up to an hour) after a small number of failures (example 3). 
Then if this pattern repeats (for example 3 times) within a longer 
period (for example up to a day), blocking for a longer period 
(example 1 week) using the recidive jail.


  My mail server is a small one with about 135 users, a corporate 
network.  NONE of them actually type their password when they're 
checking their mail.  They type it exactly once when they set up the 
account in their mail client(s), like everyone else in the world for 
the past decade or more.


Mileage will vary depending on user base and number of support 
requests generated.


The point about fail2ban is that it slows down attackers stopping 
infinite and fast repeating attacks from the same ip. That should be 
in combination with a good password policy which reduces the 
probability of any single attack guessing the password. It doesn't 
necessarily have to zero out attacks. As Dave has experimented, to 
bypass fail2ban all the attacker has to do is use a different ip. 
10-15K blocks in place at any time seem very high compared to the few 
attacks I see.


  Sigh.

  I don't "experiment" with production networks.  I set up a banning 
policy that works for the attack patterns that I see in my logs, and 
that work with my user base.  As I explained to the other guy who 
decided that how I run my network is wrong, I'm not new at this.


  Any attack mitigation strategy has to begin with observation and 
rational thought.  Network security is no place for guesswork or 
assumptions.


I'd hazard a guess that the restrictive fail2ban policy is causing 
the attacker(s) to try immediately from a new ip and isn't generating 
a great deal more security than a slightly less restrictive policy 
which lures the attacker into trying a few times more from the same 
ip with longer intervals between the attempts.


  I wasn't asking for a critique of my configuration; I explained my 
approach to a new user who came here looking for help.


  Which is the last time I'll do THAT on this list, by the way.

  But since you brought it up, the attack pattern that I see most 
frequently is a single IP address trying different, obviously 
algorithmically-generated usernames at long intervals, many seconds to 
many hours, and in some cases a day or more.  This started around 2014 
or so, and has persisted and grown.  The approach that you describe 
above seems to make sense on the surface, but looking at actual logs 
doesn't support the idea.


  The "attackers" in this case are almost exclusively little programs 
thrown together by kids and run from zombie Windows boxes on clueless 
users' home networks organized as botnets.  This pattern is visible 
when the same sequences of generated usernames start appearing from 
different IP addresses within hours of each other.


  Some of the more advanced stuff has the adaptive behaviors that you 
describe, but not many.  These script k1ddiez are trolling for 
low-hanging fruit to get bank account info etc out of peoples' mail 
accounts.  These are almost never focused attacks from one motivated, 
knowledgeable person trying hard to get into one user's mail.


  So, hazard all the guesses you like, I will continue to develop 
banning strategies for my servers which fit the attacks that I observe 
through direct analysis of my logs.


  Now, please don't misunderstand, I actually do appreciate the 
thought and intention behind your unsolicited advice.  But, for future 
reference, don't assume that someone doesn't know what they're doing 
just because their approach differs from either your approach for your 
own network, or your guesses about theirs.


   -Dave, pre-coffee


Dave

no one is telling you how to run your server or offering you unsolicited 
advice but on a public mailing list there can be multiple points of view 
about a topic and there may be not be a single right way to do 
something. There is no reason then to take this as someone telling you 
that how you run your network is wrong or to avoid participating in future.


John

Re: Non-user logins?

[John Fawcett]

On 08/01/2022 14:26, dc...@dvl.werbittewas.de wrote:


Am 08.01.22 um 05:27 schrieb Dave McGuire:


trying to mess with other peoples' stuff.  I run fail2ban to catch those
log entries and block the source IP address for a month on the first
failed login.  At any one time I have between 12,000 and 15,000

well, I don't know how _your_ users are connected to the internet, but
in germany most people has at least daily changing IPs out of larger
pools (when connected via xDSL) or even sometimes shares ip-addresses
with others (when connected via tv-cable or mobile - having a private
network-address, which is natted), so it's possible to get/use an IP,
which was used before by some script-kiddies...

so everyone, who's blocking such requests for more than some
minutes/hours should be aware of maybe blocking legitimate user-logins...

btw.: setting up a new mail-client and making any mistake by reading it
from old install or writing it into new install also leads to a
months-blocking with above restrictive handling...
(any may drive this user mad)

so anyone, who has no experience with blocking should be really careful
with it.

d.


yes, blocking on the first wrong password sounds like overkill. But it 
does depend on user base. For a small mail server with few known users 
it could be workable.


But even on small servers I'd recommend blocking for a small time (like 
up to an hour) after a small number of failures (example 3). Then if 
this pattern repeats (for example 3 times) within a longer period (for 
example up to a day), blocking for a longer period (example 1 week) 
using the recidive jail.


Mileage will vary depending on user base and number of support requests 
generated.


The point about fail2ban is that it slows down attackers stopping 
infinite and fast repeating attacks from the same ip. That should be in 
combination with a good password policy which reduces the probability of 
any single attack guessing the password. It doesn't necessarily have to 
zero out attacks. As Dave has experimented, to bypass fail2ban all the 
attacker has to do is use a different ip. 10-15K blocks in place at any 
time seem very high compared to the few attacks I see.


I'd hazard a guess that the restrictive fail2ban policy is causing the 
attacker(s) to try immediately from a new ip and isn't generating a 
great deal more security than a slightly less restrictive policy which 
lures the attacker into trying a few times more from the same ip with 
longer intervals between the attempts.


John

Re: TLS connection closed unexpectedly

[John Fawcett]

On 07/01/2022 21:03, Ken Wright wrote:

On Fri, 2022-01-07 at 18:50 +0100, John Fawcett wrote:

it may or may not be related to the tls issue, but I think you will
want to investigate that message about the SQL query syntax error.
You are not going to be able to login if the query is giving errors.
Check the log doesn't reveal the cause.

Know anything about SQL queries, John?  Here's the user query in
question:

user_query = SELECT maildir, 2000 AS uid, 2000 AS gid FROM mailbox
WHERE username = '%u' AND active='1'

I copied this directly from the tutorial I've been following and this
is the first time I've seen this error.


Hi Ken

looks fine to me. However, mariadb is not accepting it. I suggest you 
run with auth_debug = yes and check the logs.


John

Re: TLS connection closed unexpectedly

[John Fawcett]

On 07/01/2022 18:31, Ken Wright wrote:

On Fri, 2022-01-07 at 10:02 +0100, Christian Kivalo wrote:


On January 7, 2022 9:51:20 AM GMT+01:00, Ken Wright
 wrote:

I reinstalled Dovecot this evening, because I haven't been able to
receive emails recently.  After configuring and adjusting for the use
of PostfixAdmin, I get the following error message in Evolution:;

Failed to open folder.
The reported error was "Failed to authenticate: TLS connection closed
unexpectedly".

Does anyone recognize this error?

Do you have a connection attempt logged by dovecot?
Is there anything in the logs on the server?

Here's the output of tail /var/log/mail.log:


Jan  7 12:23:02 grace dovecot: auth-worker(665770): Error:
sql(www-d...@mydomain.com): User query failed: You have an error in
your SQL syntax; check the manual that corresponds to your MariaDB
server version for the right syntax to use near 'FROM mailbox WHERE
username = 'www-d...@mydomain.com' AND active='1'' at line 1
Jan  7 12:23:02 grace dovecot:
lmtp(www-d...@mydomain.com)<666838>: Error:
auth-master: userdb lookup(www-d...@mydomain.com): Auth USER lookup
failed
Jan  7 12:23:02 grace dovecot: lmtp(666838): Error: lmtp-server: conn
unix:pid=666840,uid=117 [6]: rcpt www-d...@mydomain.com: Failed to
lookup user www-d...@mydomain.com: Internal error occurred. Refer to
server log for more information.

Hope this helps!

Ken


Ken

it may or may not be related to the tls issue, but I think you will want 
to investigate that message about the SQL query syntax error. You are 
not going to be able to login if the query is giving errors. Check the 
user_query in the dovecot configuration, since the SQL snippet in the 
log doesn't reveal the cause.


John

Re: GDPR/sender-ip (was: make received-header on submission optional or at least drop the ip in it)

[John Fawcett]

On 07/01/2022 14:01, Sam Kuper wrote:


You say you cannot see it, but I gave an example below, in my previous
email:



Secondly, a person sending an email to a mailing list might very well
consent for the mailing list's recipients to receive the content,
subject, and reply address of that email - but *not* the IP address
from which it was sent.


Your example was clear. I was replying about "assumed consent". In the 
case that I send an email to a public mailing list I don't think you 
would need to get explicit consent (for processing the email contents). 
That you're not necessarily consenting to the diffusion of the ip 
address by sending the email is clear.

The IP address is a different kind of datum to the content, subject, and
reply address.

For instance:

-   The IP address is likely to allow the user's location (city or
 region) to be inferred, in a manner typically outside the user's
 control.  As such, disseminating the IP address unnecessarily  would
 reduce the user's privacy.

-   The sender of an email is likely to be aware of the content,
 subject, and sender address of an email that they send, because MUA
 UIs typically make this clear.  But many (most?) email users don't
 know what IP addresses are or what can be inferred from them, and so
 *cannot* (without being provided with a clear explanation) give
 informed consent about divulging their IP addresses unnecessarily.


So, unless a service provider obtains user consents meeting the four
tests above, in respect of *each kind* of datum they intend to process,
then the service provider would on the face of it be in breach of the
GDPR in respect of that kind of datum.

In particular, the "freely given" consent means that provision of a
service, etc, should not be contingent upon consent.  I.e. if it is not
*necessary* (which it isn't, by definition) for some kind of datum (e.g.
users' IP addresses) to be disseminated more widely than necessary, then
a service provider cannot validly under the GDPR require a user to
consent to such dissemination in order to receive the service.  Such
contingency would render the consent not freely given.

Sam

Yes, I stand corrected. Consent would not be a solution. You'd still 
need a way of NOT sending the ip if consent was not given and if that 
way did not exist, consent would not be freely given, even for those 
that give it.


So only lawful processing category that potentially could remain 
feasible I think is legitimate interest (i.e. email headers can 
generally be expected to contain ip info, potentially useful for spam 
prevention) but given that the info is available in log files, it would 
be hard to argue that the inclusion in the email header is legitimate 
when compared to rights of data subject.


So indeed the safest thing is to remove originating ips from headers, so 
as not to be on wrong side of GDPR legislation.


John

Re: Doveadm auth test fails

[John Fawcett]

On 06/01/2022 04:20, Ken Wright wrote:

On Thu, 2022-01-06 at 03:44 +0100, John Fawcett wrote:

On 06/01/2022 01:16, Ken Wright wrote:

I've been having trouble logging into my email server (postfix
3.4.13, dovecot 2.3.7.2, postfixadmin 3.3.8).  I decided to try the
doveadm auth test, and got the following result:

kwright@grace:~$ sudo doveadm auth test m...@mydomain.com
Password:
passdb: m...@mydomain.com auth failed
extra fields:
    user=m...@mydomain.com

I logged in to PostfixAdmin and made sure I was using the correct
password, but got exactly the same result afterward.  Should I have
restarted Dovecot after changing the password?  I'm totally confused
by this problem; any and all suggestions will be gratefully received!

Ken




Ken

Dovecot does have credential caching, so potentially the info could be
coming from the cache though dovecot uses some logic to understand
when it should do a new query so normally its not necessary to flush
the cache or restart dovecot after changing a password. If you're doing
testing on a non live server in the process of being set up then you
may want to take the cautious approach of restarting dovecot.

about why the command is failing. You may be able to find other
information in the log.

You may want to investigate turning on authentication and password
debugging to progress this problem.

auth_debug = yes

auth_debug_passwords = yes

(and restart dovecot)

Okay, I've done this.


Then try an authentication test again or even a full login test

doveadm auth login username

Those settings will give you information in the log about what dovecot
is doing internally in relation to lookup up the user info and password
including information about password mismatches.

Here's the latest output of tail /var/log/mail.log:

Jan  5 22:09:28 grace dovecot: auth-worker(218040): Debug: conn
unix:auth-worker (pid=171742,uid=118): auth-worker<1590>: Handling
PASSV request
Jan  5 22:09:28 grace dovecot: auth-worker(218040): Debug:
sql(m...@mydomain.com,192.168.1.1,<3VfPMuHUrpvAqAEB>): Performing passdb
lookup
Jan  5 22:09:28 grace dovecot: auth-worker(218040): Debug:
sql(m...@mydomain.com,192.168.1.1,<3VfPMuHUrpvAqAEB>): query: SELECT
username AS user,password FROM mailbox WHERE username =
'm...@mydomain.com' AND active='1'
Jan  5 22:09:28 grace dovecot: auth-worker(218040):
sql(m...@mydomain.com,192.168.1.1,<3VfPMuHUrpvAqAEB>): Password mismatch
Jan  5 22:09:28 grace dovecot: auth-worker(218040): Debug:
sql(m...@mydomain.com,192.168.1.1,<3VfPMuHUrpvAqAEB>): ARGON2I(password)
!= '$1$c9809462$ecGdXzPm2xqMK0TKngGkc.', try DES-CRYPT scheme instead
Jan  5 22:09:28 grace dovecot: auth-worker(218040): Debug:
sql(m...@mydomain.com,192.168.1.1,<3VfPMuHUrpvAqAEB>): Finished passdb
lookup
Jan  5 22:09:28 grace dovecot: auth-worker(218040): Debug: conn
unix:auth-worker (pid=171742,uid=118): auth-worker<1590>: Finished
Jan  5 22:09:28 grace dovecot: auth: Debug:
sql(m...@mydomain.com,192.168.1.1,<3VfPMuHUrpvAqAEB>): Finished passdb
lookup
Jan  5 22:09:28 grace dovecot: auth: Debug:
auth(m...@mydomain.com,192.168.1.1,<3VfPMuHUrpvAqAEB>): Auth request
finished
Jan  5 22:09:30 grace dovecot: auth: Debug: client passdb out:
FAIL#0111#011user=m...@mydomain.com

I know the password is correct, but it still fails.  I had some
problems getting this mailbox set up in PostfixAdmin; could it be the
database is faulty?

Ken


Ken

It looks like a mismatch between your dovecot and postfixadmin password 
encryption schemes. If I'm reading this correctly you have configured 
ARGON2I in dovecot and are using a MD5-crypt scheme in postfixadmin. 
Therefore when you set the password in postfixadmin it is saving the 
password with a different encryption scheme to the one that dovecot is 
using when it verifies the password. I suggest to align them. If you 
change the postfixadmin setting, remember you'll have to change the 
existing passwords that have been stored while using a different setting 
to the dovecot one.


Also one other point (not sure if it's related to the multiple issues 
you've been posting about), but ARGON2 apparently requires a lot of 
virtual memory. Were you using this previously or did you change to it 
during the server installation you did recently? Here's some more info 
in case you haven't seen it already:


https://doc.dovecot.org/configuration_manual/authentication/password_schemes/

John

Re: Doveadm auth test fails

[John Fawcett]

On 06/01/2022 01:16, Ken Wright wrote:

I've been having trouble logging into my email server (postfix 3.4.13,
dovecot 2.3.7.2, postfixadmin 3.3.8).  I decided to try the doveadm
auth test, and got the following result:

kwright@grace:~$ sudo doveadm auth test m...@mydomain.com
Password:
passdb: m...@mydomain.com auth failed
extra fields:
   user=m...@mydomain.com

I logged in to PostfixAdmin and made sure I was using the correct
password, but got exactly the same result afterward.  Should I have
restarted Dovecot after changing the password?  I'm totally confused by
this problem; any and all suggestions will be gratefully received!

Ken




Ken

Dovecot does have credential caching, so potentially the info could be 
coming from the cache though dovecot uses some logic to understand when 
it should do a new query so normally its not necessary to flush the 
cache or restart dovecot after changing a password. If you're doing 
testing on a a non live server in the process of being set up then you 
may want to take the cautious approach of restarting dovecot.


The command output you gave above does not provide useful information 
about why the command is failing. You may be able to find other 
information in the log.


You may want to investigate turning on authentication and password 
debugging to progress this problem.


auth_debug = yes

auth_debug_passwords = yes

(and restart dovecot)

Then try an authentication test again or even a full login test

doveadm auth login username

Those settings will give you information in the log about what dovecot 
is doing internally in relation to lookup up the user info and password 
including information about password mismatches.


John

Re: GDPR/sender-ip (was: make received-header on submission optional or at least drop the ip in it)

[John Fawcett]

On 05/01/2022 18:36, Sam Kuper wrote:

On Wed, Jan 05, 2022 at 06:00:31PM +0100, John Fawcett wrote:

my understanding of the GDPR legislation is that it defines what is
considered lawful processing. One of those items that makes the
processing lawful is consent.

Not necessarily.

An action that would not be lawful without consent is not automatically
made lawful with consent, including under GDPR.

Correct there could be other reasons that make processing unlawful. 
However, GDPR will allow processing if the data subject consents and I 
think taht is what we are talking about in this thread.



If I send an email to a public mailing list I think it's fair to say
that I am providing consent.

Again, not necessarily.

First of all, consent cannot necessarily be assumed.
Correct that it cannot necessarily be assumed. But in this case I think 
it would be fair to assume it when someone sends an email to a public 
mailing list that consent has been given. I cannot see how having sent 
an email to a public mailing list I can then object to people processing 
it. Although it's not a question about GDPR, if I DID then change my 
mind, I cannot see a technical way to enforce it.


Secondly, a person sending an email to a mailing list might very well
consent for the mailing list's recipients to receive the content,
subject, and reply address of that email - but *not* the IP address from
which it was sent.
Correct. That is why I mentioned as an alternative "request that your 
users consent to the processing of the data".



Sam

Re: GDPR/sender-ip

[John Fawcett]

On 05/01/2022 18:27, dc...@dvl.werbittewas.de wrote:


Am 05.01.22 um 18:00 schrieb John Fawcett:


my understanding of the GDPR legislation is that it defines what is
considered lawful processing. One of those items that makes the
processing lawful is consent. If I send an email to a public mailing
list I think it's fair to say that I am providing consent. I was not

sorry, you're wrong.

have a look at the given link to the EUGH-judgement.

it's irrelevant if someone acts in private or public area.

peoples privacy rights always exists and may only violated, if there's
really no other possibility to fulfil higher rights
I did look at it and my answer was based partly on what I read there. 
This is more of a technical mailing list so I'll avoid commenting here 
further, other than to say that I think you're reading things into the 
judgement and the GDPR legislation that go beyond what the legislation 
requires.


and:

think about whistleblowers or dissidents in autotcratic regimes: maybe
they send mails from the same place, where they do their important work
for society. Even if they change their IP between, their location may be
traced in detail.
I think this is getting way off topic for this list. I would sincerely 
hope that people in these circumstances would trust to more robust 
privacy mechanisms than the one in discussion.


and don't forget: no one here will force you, to change your handling
with your users privacy-data. but some of us want to have the
possibility to change it. So why you're aguing here, what's your aim?
I have nothing against the proposed patch. I was just suggesting other 
ways that the GDPR legislation requirements may be met.


if there are technical issues with the patch - ok
if it brings more complexity in the further development (which I don't
think due its simplicity) - ok

but I think that interpretation of the GDPR is neither on-topic here,
nor may lead to any kind of consensus,


I can only agree!


since even in Europe there're
still enough people, who think, that's not useful.


If this phrase is related to my posts, you're reading things I didn't 
say, mean, mean to imply or even think.




d.


John

Re: GDPR/sender-ip (was: make received-header on submission optional or at least drop the ip in it)

[John Fawcett]

On 05/01/2022 17:44, Marc wrote:

Where people are sending email to public
mailing lists, there is no presumption of privacy.


I think the legislation is like this: the privacy is legislation is always 
applicable, there are no conditions under which there is 'no presumption of 
privacy'


Marc

my understanding of the GDPR legislation is that it defines what is 
considered lawful processing. One of those items that makes the 
processing lawful is consent. If I send an email to a public mailing 
list I think it's fair to say that I am providing consent. I was not 
suggesting that the legislation does not apply.


John

Re: GDPR/sender-ip (was: make received-header on submission optional or at least drop the ip in it)

[John Fawcett]

On 05/01/2022 14:41, dc...@dvl.werbittewas.de wrote:


Am 04.01.22 um 08:39 schrieb Aki Tuomi:


We'll take a look at your patch. Can you please point out to some legal 
information about the Received header's GDPR incompliance, I would be 
interested to see it.

thanks for doing so.


the GDPR says about personal data:
- that only really needed data has to be stored
- that this data has to be used only for that declared needs
- that any other usage has to be prevented, especially by third-parties

the EuGH has judged in 2016 (Patrick Breyer vs. Germany, C-582/14), that
an IP-addresses can be personal data, because the person may be
identified via this IP, so they have to be handled as such.

http://curia.europa.eu/juris/documents.jsf?num=C-582/14

therefore the possibility, that others may for example see when a person
was at a place (connected to an IP) has to be prevented at least in europe.

if such information is published for people with high email-activity,
then it would be possible for everyone, who has access to this email
(which might be really everyone on earth for example in archived
mailing-lists) to track these people over the whole time.


for security-reasons we're logging any submission-request together with
the origin-IP in our logs for at least seven days. so any mis-use of our
service may be prosecuted even without storing this information in every
email. In germany some courts judged, that if the police asks us for the
IP, we've to store the log-entry at least as long, as a court needs to
judge, that we have to give it to the police.
(I think this is a reasonable balance between protection of personal
data and legitimate public interest)

if there are further questions to this topic I'll try to reply, but you
should know, that my english isn't that good, especially to explain
juridicial things...


regards


d.


Hi

the safest non technical approach is to request that your users consent 
to the processing of the data. The dynamic ip address is however 
personal data in a very limited sense ( i.e. where the service provider 
can link the dynamic ip address to other information that then 
identifies the individual). Where people are sending email to public 
mailing lists, there is no presumption of privacy.


John

Re: Error: fts_solr: received invalid uid (John Fawcett)

[John Fawcett]

On 03/01/2022 01:18, oliver.kr...@snoog.ch wrote:

Hi John,

You are right this was not a dovecot solr query. It is really strange 
everything seems to work with Dovecot 2.3.4. E.g. search and  
rebuilding index from scratch using doveadm. As soon as I use version  
2.3.13 things are getting wild: dovecot crashed when I re-scan the 
index using doveadm and search does not work anymore, after re-scan:


Panic: file http-client-request.c: line 1240 
(http_client_request_send_more): assertion failed: (req->payload_input 
!= NULL)
doveadm(kr...@invectrix.ch): Error: Raw backtrace: 
/usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) 
[0x7fc54cace4e2] -> 
/usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7fc54cace5fe] 
-> /usr/lib/dovecot/libdovecot.so.0(+0xfc49b) [0x7fc54cada49b] -> 
/usr/lib/dovecot/libdovecot.so.0(+0xfc4d1) [0x7fc54cada4d1] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x53aee) [0x7fc54ca31aee] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x4aec2) [0x7fc54ca28ec2] -> 
/usr/lib/dovecot/libdovecot.so.0(http_client_connection_output+0xee) 
[0x7fc54ca7ebde] -> /usr/lib/dovecot/libdovecot.so.0(+0x122171) 
[0x7fc54cb00171] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) 
[0x7fc54caeff59] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x132) 
[0x7fc54caf1592] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x50) 
[0x7fc54caf] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) 
[0x7fc54caf01c0] -> /usr/lib/dovecot/libdovecot.so.0(+0x9c4cd) 
[0x7fc54ca7a4cd] -> 
/usr/lib/dovecot/libdovecot.so.0(http_client_request_finish_payload+0x2c) 
[0x7fc54ca7a6dc] -> 
/usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xf243) [0x7fc54c0a6243] 
-> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_parser_more+0x25) 
[0x7fc54c0a5345] -> 
/usr/lib/dovecot/modules/lib20_fts_plugin.so(+0xbfcf) [0x7fc54c0a2fcf] 
-> /usr/lib/dovecot/modules/lib20_fts_plugin.so(fts_build_mail+0xa27) 
[0x7fc54c0a3a87] -> 
/usr/lib/dovecot/modules/lib20_fts_plugin.so(+0x11f0b) 
[0x7fc54c0a8f0b] -> 
/usr/lib/dovecot/libdovecot-storage.so.0(mail_precache+0x2e) 
[0x7fc54cbf14ae] -> doveadm(+0x3761f) [0x55d18bdbf61f] -> 
doveadm(+0x31bad) [0x55d18bdb9bad] -> doveadm(+0x32860) 
[0x55d18bdba860] -> 
doveadm(doveadm_cmd_ver2_to_mail_cmd_wrapper+0x22d) [0x55d18bdbb4ad] 
-> doveadm(doveadm_cmd_run_ver2+0x4c8) [0x55d18bdcbb88] -> 
doveadm(doveadm_cmd_try_run_ver2+0x3a) [0x55d18bdcbbda] -> 
doveadm(main+0x1d0) [0x55d18bdaa450] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) 
[0x7fc54c69fd0a] -> doveadm(_start+0x2a) [0x55d18bdaa92a]


and solr:

2022-01-03 00:13:20.829 ERROR (qtp1991278377-21) [   x:dovecot] 
o.a.s.s.HttpSolrCall null:[com.ctc.wstx.exc.WstxLazyException] 
com.ctc.wstx.exc.WstxIOException: Early EOF




Caused by: org.eclipse.jetty.io.EofException: Early EOF
at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1143)
at 
org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1131)

at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:318)

Seems like dovecot is sending an empty query?

Oliver 


Hi Oliver

I remember there being some patches to dovecot for using solr that 
addressed errors like this and also an additional patch for using solr 
and tika. You could try looking back in the archives and applying them. 
But a quicker way if it's open is to use a later version. I am sure that 
by 2.3.16 all the fixes were already incorporated by the development team.


However, I'm not sure if this is related to your original issue. The 
response from solr looks strange to me. But you'll need to get a working 
version before progressing that issue (and maybe in doing so you'll find 
it resolved by itself).


John

Re: Error: fts_solr: received invalid uid

[John Fawcett]

On 02/01/2022 12:13, oliver.kr...@snoog.ch wrote:
Here is what solr sends to dovecot, the offending uid in this exmaple 
is '21'

{
  "responseHeader":{
    "status":0,
    "QTime":1,
    "params":{
  "q":"body:Zeitserver"}},
  "response":{"numFound":1,"start":0,"numFoundExact":true,"docs":[
  {
    "uid":[21],
    "box":["2e60f3054059c95fbf580600eb1947e0"],
    "user":["oliver.kr...@snoog.ch"],
"id":"21/2e60f3054059c95fbf580600eb1947e0/oliver.kr...@snoog.ch",
    "body":[" ... "],
    "hdr":[" ... "],
    "from":["..."],
    "subject":["..."],
    "to":["..."],
    "_version_":1720682332306800640}]
  }}

Oliver 


Hi Oliver

unless I'm missing something that doesn't look like the type of query 
dovecot launches - dovecot sends various parameters including the 
request for xml results. The thing that looks odd to me is that the 
results fields should be single values not arrays enclosed in [], ie I 
would have expected


"uid":21,

instead of

"uid":[21],

I'd be interested to see the equivalent xml output produced by running a 
query that dovecot sends.


John

Re: Error: fts_solr: received invalid uid

[John Fawcett]

On 30/12/2021 19:14, oliver.kr...@snoog.ch wrote:


I'm using dovecot 2.3.13,  solr 8.x. and roundcube 1.5.1. However when 
I do a search I get the the Error: fts_solr: received invalid uid, 
search results are ok.


Thanks

Presumably one of the uid returned from solr could not be decoded (i.e. 
converted to an int). The offending uid should have been printed as part 
of the message. If you need to get more info you could investigate what 
is being returned by solr running the exact same query from a browser if 
you're able to retrieve it from the solr log.


Alternatively you could set rawlog_dir in the fts_solr dovecot settings 
and then look at what is being sent back from solr in the .in file that 
is logged.


https://doc.dovecot.org/settings/plugin/fts-solr-plugin/?highlight=fts%20dovecot%20plugin

John

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

[John Fawcett]

On 15/12/2021 08:52, Aki Tuomi wrote:

The suggested configuration is good, and although we did some checking to 
ensure that dovecot escapes the search queries and usernames sent to solr, so 
it is not trivial to send the JNDI expansion strings to be logged by solr, it 
is still good idea to set this.

Aki

Agreed, it is worthwhile taking the advised mitigation steps regardless 
of the escaping done in Dovecot. Reasoning is


1) escaping may not be 100% foolproof - there are people out there 
working on bypassing such things


2) the search string method is not the only attack vector for SOLR. If 
people have SOLR exposed on an internet host, even if password 
protected, it doesn't mean to say that SOLR is not logging failed access 
attempts that can easily contain the attack string.


John

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

[John Fawcett]

On 14/12/2021 03:23, Scott wrote:

Is this assuming you log at some verbose level ? What if you log at WARN or 
higher ?

For production it seems kind of silly to log search queries anyways.

Scott


It's a pretty much standard install where most things are at INFO level. 
Probably could turn it down, but has not been an issue.


I think the logic behind it is much like web server logs where every 
single request gets a log entry.


John

Re: Can dovecot be leveraged to exploit Solr/Log4shell?

[John Fawcett]

On 13/12/2021 23:43, Joseph Tam wrote:


I'm surprised I haven't seen this mentioned yet.

An internet red alert went out Friday on a new zero-day exploit. It is an
input validation problem where Java's Log4j module can be instructed via
a specially crafted string to fetch and execute code from a remote LDAP
server.  It has been designated the Log4shell exploit (CVE-2021-44228).

Although I don't use it, I immediately thought of Solr, which provides
some dovecot installations with search indexing.  Can dovecot be made
to pass on arbitrary loggable strings to affected versions of Solr 
(7.4.0-7.7.3,

8.0.0-8.11.0)?

Those running Solr to implement Dovecot FTS should look at

https://solr.apache.org/security.html#apache-solr-affected-by-apache-log4j-cve-2021-44228 



Joseph Tam 


Solr logs the search strings passed, so potentially authenticated users 
could log malicious strings by searching for them. I do see escaping of 
some special characters in the log, but not sure if that would be a 
sufficient mitigation. In my web server logs I see all kinds of patterns 
that are trying to circumvent WAF rules, so maybe someone will come up 
with a way of getting the malicious string into the solr log.


As Apache Solr is mentioned as one of the software that is impacted, the 
mitigations are to upgrade to a non vulnerable version asap and in the 
meantime turn off JNDI lookups.


John

Re: Incorrect value sent to solr

[John Fawcett]

On 08/04/2021 08:02, Aki Tuomi wrote:
> Hi!
> Thanks for reminding us, I'll make a ticket about this to avoid forgetting it 
> again.
>
> Aki

Thanks to you and the team for taking it into consideration!

PS is this list the best way to post patch proposals or is it easier for
you to get code from pull requests?

John

Re: Incorrect value sent to solr

[John Fawcett]

On 07/04/2021 13:36, Łukasz Szczepański wrote:
> I'm not as familiar with C, but I don't see in solr backed in dovecot
> any  clue of subsequent queries for single mailbox lookup (which most
> mail client uses). There is a hard limit of 10 rows for multiple
> mailbox lookup.


This was reported back in December 2020. I submitted this fix to the
list on 31/12/2020 to give an upper bound for single mailbox queries as
there is in place for multiple mailbox queries.

diff -ur dovecot-2.3.11.3-orig/src/plugins/fts-solr/fts-backend-solr.c
dovecot-2.3.11.3/src/plugins/fts-solr/fts-backend-solr.c

--- dovecot-2.3.11.3-orig/src/plugins/fts-solr/fts-backend-solr.c  
2020-08-12 14:20:41.0 +0200
+++ dovecot-2.3.11.3/src/plugins/fts-solr/fts-backend-solr.c   
2020-12-31 09:05:07.681897716 +0100
@@ -838,7 +838,7 @@

    str = t_str_new(256);
    str_printfa(str,
"wt=xml=uid,score=%u=uid+asc=%%7b!lucene+q.op%%3dAND%%7d",
-   status.uidnext);
+   I_MIN(status.uidnext,SOLR_MAX_MULTI_ROWS));
    prefix_len = str_len(str);

    if (solr_add_definite_query_args(str, args, and_args)) {

Re: fts_encoder

[John Fawcett]

On 11/02/2021 14:25, Joan Moreau wrote:
>
> Hello
>
> Checking further, and putting logs a bit every where in the dovecot
> code, the core is sending FIRST the initial document (not decoded)
> then SECOND the decoded version
>
> Thisi is really weird, and the indexer then indexes a lot of binary crap
>
> I am struggling to find where in the code this double call is made.
>
> Anyone knows ?
>
>
Joan

I didn't get round to working out where it happens. But your observation
is in line with what I see for solr plugin. Only difference is that as
far as I can see, the raw data does not make it to solr. That the rawlog
does not contain the data is a good indication, but the proof is that
searching for PDF string on solr does not get a hit on the messages.

John

Re: fts_encoder

[John Fawcett]

On 09/02/2021 15:33, Joan Moreau wrote:
>
> If I place the following code in the plugin
> fts_backend_xxx_update_build_more function (lucene, squat and xapian,
> as solr refuses to work properly on my setup)
>
>         {
>                 char * s = i_strdup("EMPTY");
>                 if(data != NULL) { i_free(s); s = i_strndup(data,20); }
>                 i_info("fts_backend_update_build_more: data like '%s'",s);
>                 i_free(s);
>         }
>
> and if I send a PDF by email, the data shown in the log is "%PDF-1.7 "
>
> so it does mean the decoder data is not properly transmitted to the plugin
>
> Something is wrong in the data transmission
>
Joan

I too see something similar with fts_solr. I do see the raw %PDF string
and PDF binary data being passed through to
fts_backend_xxx_update_build_more function but I disagree with the
conclusion you draw from it.

After the raw data I also see the decoded data, so at least in my case
it is possible to see both the raw and decoded data in
fts_backend_xxx_update_build_more function. In the rawlog I no longer
see the binary data (but some blank lines), so something is filtering
it. I do see the decoded data in the rawlog. I do get hits on the solr
search for the decoded text.

John

Re: fts_encoder

[John Fawcett]

On 08/02/2021 23:05, Stuart Henderson wrote:
> On 2021/02/08 21:33, Joan Moreau wrote:
>> Yes , once again : output of the decoder is fine, I also put log inide the 
>> dovecot core to
>> check whether data is properly transmitted, and result is that it is (i.e. 
>> dovecot core
>> receives the proper output of pdftotext via the decoder
>>
>> Now, that data is the /not/ the one sent from dovecot core to the fts plugin 
>> (and this is the
>> same issue for solr and all other plugins)
> Seems that something is different with your setup than John's and mine
> then, as fts_solr rawlog (which is just the http request split into
> .in and .out files) has the decoded file for us.
>
> Did you try with the actual fts_solr plugin so it's a direct comparison
> with what we see? There is no need for a real solr server, just point it
> at any http server (or I guess netcat listening on a port will also do)
> with
>
> mail_plugins = fts fts_solr
>
> plugin {
>   fts_autoindex = yes
>   fts = solr
>   fts_solr = url=http://127.0.0.1:80/ rawlog_dir=/tmp/solr
> }
>
> If that is not showing decoded for you then I suppose there's some
> problem on the way into/through fts. And if it does show as decoded
> then perhaps fts_solr is doing something slightly different than the
> places you're examining in fts and your plugin, and that might give
> a point to work backwards from.
>
I'd also recommend Joan to look into some of the potential configuration
issues I mentioned in my first reply and if the problem persists, post
some clear evidence.

John

Re: fts_encoder

[John Fawcett]

On 08/02/2021 21:35, Joan Moreau wrote:
>
> Well, in the function xxx_build_more of FTS plugin, the data received
> in the original PDF, not the output of pdftotext
>
> Can you clarify where do you put your log in the solr plugin , so I
> can check the situation in the xapian plugin ?
>
I used the following setting in fts_solr parameter

rawlog_dir=

and made sure the directory was writeable by dovecot (777 just to be sure)

John

Re: Getting panic in http-client-request.c: line 1240 during indexing on Ubuntu 20.04

[John Fawcett]

On 08/02/2021 18:40, deano-dove...@areyes.com wrote:
>
> On 2021-02-07 7:32 pm, John Fawcett wrote:
>
>> On 07/02/2021 20:15, @lbutlr wrote:
>>> On 07 Feb 2021, at 02:07, @lbutlr >> <mailto:krem...@kreme.com>> wrote:
>>>> On 06 Feb 2021, at 11:06, John Fawcett >>> <mailto:j...@voipsupport.it>> wrote:
>>>>> 19.08.20
>>>> Is that a malformed ISO date 2019-08-20 or a truncated europeans
>>>> style 19-08-2020?
>>> Either way, I cannot find the message in my dovecot folder. Closest
>>> I can find is Message-ID:
>>> <20200820141341.ga1...@meili.valhalla.31bits.net
>>> <mailto:20200820141341.ga1...@meili.valhalla.31bits.net>> from the
>>> 2020-08-20 at 16:13:52 but there is no patch in that mail.
>> here's the post in the list archives
>>
>> https://dovecot.org/pipermail/dovecot/2020-August/119703.html 
>> <https://dovecot.org/pipermail/dovecot/2020-August/119703.html>
>>
> Do we have when (or even if) that patch will make it into the main ? 
> I would really rather prefer pulling from repo ...
>
+1 from me.

I'd like to see this patch (or something equivalent go in). Without this
Tika is unusable for me.

Tika is also unusable in my opinion without the (basic) limits on
processing unbounded amounts of text. There is also another patch
introducing basic auth which I find useful since my Tika server is
password protected.

If all three could make it into the repo then I could avoid manual
patching that I do at each release.

John

Re: fts_encoder

[John Fawcett]

On 08/02/2021 15:22, Joan Moreau wrote:
>
> Well, thank you for the answer, but the actual issue is that data sent
> by the decoder (stipulated in the conf file) is properly collected by
> dovecot core, but /not/ sent to the plugin : the plugin receives the
> original data.
>
> This is not linked to a particular plugin (xapian, solr, squat, etc..)
> but seems to be a general issue of dovecot core
>
>

Hi Joan

as far as I can see there's not a general issue in the dovecot core with
using the decoder. It works for me. I see the text extracted from PDF
sent to solr (I enable raw_log feature to see the actual data going over
) Also when I query solr I get a search hit for attachment text.

John

Re: fts_encoder

[John Fawcett]

On 07/02/2021 18:51, Joan Moreau wrote:
>
> more info : the function fts_parser_script_more in
> plugins/fts/fts-parser.c properly read the output of the script
>
> still, the data is not sent to the FTS pligins (xapian or any other)
>
>
>
> On 2021-02-07 17:37, Joan Moreau wrote:
>
>> more info : I am running dovecot git version
>>
>>
>> On 2021-02-07 17:15, Joan Moreau wrote:
>>
>> a bit more on this, adding log in the decode2text.sh, I can see
>> that pdftotext output the right data, but that data is /not/
>> transmitted to the fts plugin for indexing (only the original pdf
>> code is)
>>
>>
>>
>> On 2021-02-07 17:00, Joan Moreau wrote:
>>
>> Hello,
>>
>> I am trying to deal properly with email attachements in
>> fts-xapian plugins.
>>
>> I tried the default script with a PDF file.
>>
>> The data I receive in the fts plugin part ("xxx_build_more")
>> is the original document, no the output of the pdftotext
>>
>> Is there anything I am missing ?
>>
>> Here my config:
>>
>>
>> plugin {
>>         plugin = fts_xapian managesieve sieve
>>
>>         fts = xapian
>>         fts_xapian = partial=2 full=20 verbose=1 attachments=1
>>
>>         fts_autoindex = yes
>>         fts_enforced = yes
>>         fts_autoindex_exclude = \Trash
>>         fts_autoindex_exclude2 = \Drafts
>>
>>         fts_decoder = decode2text
>>
>>         sieve = /data/mail/%d/%n/local.sieve
>>         sieve_after = /data/mail/after.sieve
>>         sieve_before = /data/mail/before.sieve
>>         sieve_dir = /data/mail/%d/%n/sieve
>>         sieve_global_dir = /data/mail
>>         sieve_global_path = /data/mail/global.sieve
>> }
>>
>> ...
>>
>> service decode2text {
>>    executable = script /usr/libexec/dovecot/decode2text.sh
>>    user = dovecot
>>    unix_listener decode2text {
>>      mode = 0666
>>    }
>> }
>>
>>
>> Thank you
>>
>>
Joan

I'm not sure I can be much use for xapian, but looking at your
configuration I did notice some differences with the documentation. I
don't know if they are relevant to the issue you're seeing.

First of all I don't see

|mail_plugins = fts|

plugin = fts

settings which are both mentioned in the xapian documentation.

Also the documentation states that attachments=1 can only index text
attachments. Maybe you should be using attachments=0 and let fts_decode
handle the attachments.

Failing that, I can only advise to turn on some debugging and see what
that brings.

best regards

John

Re: Getting panic in http-client-request.c: line 1240 during indexing on Ubuntu 20.04

[John Fawcett]

On 07/02/2021 20:15, @lbutlr wrote:
> On 07 Feb 2021, at 02:07, @lbutlr  wrote:
>> On 06 Feb 2021, at 11:06, John Fawcett  wrote:
>>> 19.08.20 
>> Is that a malformed ISO date 2019-08-20 or a truncated europeans style 
>> 19-08-2020?
> Either way, I cannot find the message in my dovecot folder.
>
> Closest I can find is Message-ID: 
> <20200820141341.ga1...@meili.valhalla.31bits.net> from the 2020-08-20 at 
> 16:13:52 but there is no patch in that mail.
>
here's the post in the list archives

https://dovecot.org/pipermail/dovecot/2020-August/119703.html

Re: Getting panic in http-client-request.c: line 1240 during indexing on Ubuntu 20.04

[John Fawcett]

On 05/02/2021 15:04, Dean Carpenter wrote:
>
> Is there anything I can do here ?  This makes Tika unusable :(  That
> really sucks because we have a *lot* of attachments.
>
> Thanks -
>
> On 2021-02-03 2:07 pm, Dean Carpenter wrote:
>
>> Just noticed this in the logs
>>
>> doveadm(har...@example.com): Debug: http-client[2]: request
>> [Req2: PUT http://localhost/tika/]: Submitted (requests left=1)
>> doveadm(har...@example.com): Debug: http-client[2]: request
>> [Req2: PUT http://localhost/tika/]: Waiting for request to finish
>>
>> It doesn't have the port on the URL ...
>>
>>  
>>
>> On 2021-02-03 1:59 pm, Dean Carpenter wrote:
>>
>> Getting panic in http-client-request.c: line 1240 during indexing
>>
>> I'm testing the install/setup of dovecot on an Ubuntu Focal 20.04
>> system, using the dovecot repo. The packages that are installed are
>>
>> harryb@dove1:~$ dpkg -l dovecot\* solr\* jetty\* tesser\* | fgrep ii
>> ii dovecot-core 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP
>> server - core files
>> ii dovecot-imapd 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP
>> server - IMAP daemon
>> ii dovecot-lmtpd 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP
>> server - LMTP server
>> ii dovecot-managesieved 2:2.3.13-2+ubuntu20.04 amd64 secure
>> POP3/IMAP server - ManageSieve server
>> ii dovecot-mysql 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP
>> server - MySQL support
>> ii dovecot-sieve 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP
>> server - Sieve filters support
>> ii dovecot-solr 2:2.3.13-2+ubuntu20.04 amd64 secure POP3/IMAP
>> server - Solr support
>> ii jetty9 9.4.26-1 all Java servlet engine and webserver
>> ii solr-common 3.6.2+dfsg-22 all Enterprise search server based
>> on Lucene3 - common files
>> ii solr-jetty 3.6.2+dfsg-22 all Enterprise search server based on
>> Lucene3 - Jetty integration
>> ii tesseract-ocr 4.1.1-2build2 amd64 Tesseract command line OCR tool
>> ii tesseract-ocr-eng 1:4.00~git30-7274cfa-1 all tesseract-ocr
>> language files for English
>> ii tesseract-ocr-osd 1:4.00~git30-7274cfa-1 all tesseract-ocr
>> language files for script and orientation
>>
>> Apache Tika v1.25 is also installed
>>
>> harryb@dove1:~$ l /usr/local/bin/tika*
>> -rw-r--r-- 1 root root 79337717 Feb 3 12:40
>> /usr/local/bin/tika-server-1.25.jar
>>
>> and running pretty standard-ish
>>
>> /usr/bin/java -jar /usr/local/bin/tika-server-1.25.jar --port
>> 9998 --host localhost -enableUnsecureFeatures -enableFileUrl
>> --log info
>>
>> I use mbsync to pull in sample mailboxes for testing
>>
>> The solr conf in /etc/dovecot/conf.d/ is as follows
>> plugin {
>> fts = solr
>> fts_solr = url=http://localhost:8693/solr/
>> fts_autoindex = yes
>> fts_filters = lowercase snowball stopwords
>> fts_tokenizers = generic email-address
>> fts_tokenizer_generic = algorithm=simple
>> fts_autoindex_exclude = \Junk
>> fts_autoindex_exclude2 = \Trash
>> fts_autoindex_exclude3 = \Spam
>> }
>> plugin {
>> fts_tika = http://localhost:9998/tika/
>> }
>>
>> Unfortunately indexing is panic'ing. Whether I index a test
>> folder manually, or just looking in /var/log/dovecot.log
>>
>> I'll see the following very consistently :
>>
>> harryb@dove1:~$ sudo doveadm -D index -u har...@example.com Receipts
>>
>> Debug: Loading modules from directory: /usr/lib/dovecot/modules
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/lib10_mail_crypt_plugin.so
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib15_notify_plugin.so
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_fts_plugin.so
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/lib20_replication_plugin.so
>> Debug: Module loaded: /usr/lib/dovecot/modules/lib20_zlib_plugin.so
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/lib21_fts_solr_plugin.so
>> Debug: Loading modules from directory:
>> /usr/lib/dovecot/modules/doveadm
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so
>> Debug: Module loaded:
>> /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so
>> doveadm(har...@example.com): Debug: auth-master: passdb
>> lookup(har...@example.com): Started passdb lookup
>> doveadm(har...@example.com): Debug: auth-master: conn
>> unix:/var/run/dovecot/auth-userdb: Connecting
>> doveadm(har...@example.com): Debug: auth-master: conn
>> unix:/var/run/dovecot/auth-userdb (pid=165148,uid=0): Client
>> connected (fd=9)
>> doveadm(har...@example.com): Debug: auth-master: passdb
>> lookup(har...@example.com): auth PASS input: user=har...@example.com
>> doveadm(har...@example.com): Debug: auth-master: passdb
>> lookup(har...@example.com): Finished passdb lookup
>> (user=har...@example.com 

Re: [BUG REPORT] In some cases dovecot sends (huge) binary data to solr for indexing

[John Fawcett]

On 21/01/2021 15:10, Alexey Panov wrote:
> In some cases (exact condition still unknown) dovecot sends binary
> data (attachments) to SOLR for indexing. This reduces index and
> overall FTS efficiency dramatically. 
>
> In extreme condition (below an example of 20MB) dovecot’s hardwired
> timeout of 60s gets triggered during HTTP exchange with SOLR on just a
> single file. This results in an unfinished index which, by initial
> indexing, gets restarted over and over. With multiple affected
> mailboxes even on moderate usage this can cause an IO overload of the
> whole system.
>
> Message example (doveadm fetch
> text): https://filebin.ca/5oy5Wc1QrBK3/fetch-text.obfuscated.txt
> <https://filebin.ca/5oy5Wc1QrBK3/fetch-text.obfuscated.txt>
> Corresponding raw log
> data: https://filebin.ca/5oy6yqLSCr3H/rawlog.obfuscated.txt
> <https://filebin.ca/5oy6yqLSCr3H/rawlog.obfuscated.txt>
>
> (Both files were processed with perl doveadm-obfuscate.pl
> <https://www.dovecot.org/tools/doveadm-obfuscate.pl>; the script
> doesn’t replace non-latin characters so they were replaced with ‘R’
> manually)
>
> Workaround: there is a useful patch by John Fawcett 
> <https://www.mail-archive.com/dovecot@dovecot.org/msg82296.html> that
> allows to set the FTS indexing message body maximum size. It works
> perfectly, but affected messages are getting completely ignored by FTS.
>
> This bug report is a summarised result of this discussion
> <https://www.mail-archive.com/dovecot@dovecot.org/msg82599.html>. 

Alexey

just a couple of questions. I am expecting that the messages with sizes
exceeding the configurable limit introduced by my patch submission are
not completely ignored, but that headers are getting indexed. I don't
have time to check it now, but I'm pretty sure about it. Do you have
evidence that the messages are not being indexed at all. The desired
behaviour of my patch fts_max_size configuration was to bypass only
message body indexing not bypass indexing completely.

Are you requesting a different behaviour to the one provided by the
patch? I imagine that people would find it useful to still parse the
message body up to the limit. That would be a little more trickly, but
potentially a good idea for a further enhancement.

Thanks

John

Re: fts_solr: Indexing failed (fails on particular messages/mailboxes by timeout)

[John Fawcett]

On 15/01/2021 11:39, Alexey Panov wrote:
> I guess I was indeed correct. As message body is the part of mail
> index dovecot indeed sends all those binary messages as a part of
> message body:
>
>> doveadm fetch -u test.u...@interpont.com
>>  "body" mailbox KS2 uid 2 | wc --bytes
>> 21549696
>
> This affects FTS indexing dramatically I guess… 
>
>> On 15. Jan 2021, at 17:10, Alexey Panov > > wrote:
>>
>> I could find messages causing problems as solr log indeed shows the
>> IMAP UID. 
>>
>> Reducing batch_size to a single message I could identify one of those
>> messages causing the problem. 
>>
>> Now I can confirm the following behaviour: there are some SINGLE
>> messages where 60 seconds is NOT ENOUGH to index. The message I am
>> looking at now is 20MB big (compressed 17MB) and has a lot of inline
>> pasted screenshots. I guess that dovecot might send those pasted
>> images as a part of the message body to solr?
>>
>> Here are some logs for better understanding:
>>
>> Folder contents (single message):
>>
>>> doveadm search -u u@d mailbox KS2
>>> e836ff1d20640160340b532cee39 2
>>
>> Force indexing:
>>
>>> doveadm index -u u@d KS2
>>> doveadm(u@d): Error: fts_solr: Indexing failed: Request timed out
>>> (Request queued 61.105 secs ago, 1 send attempts in 61.089 secs,
>>> 60.071 in http ioloop, 0.000 in other ioloops, connected 61.089 secs
>>> ago)
>>> doveadm(u@d): Error: Mailbox KS2: Transaction commit failed: FTS
>>> transaction commit failed: backend deinit
>>
>> Corresponding solr logs:
>>
>>> solr-mailcow_1       | 2021-01-15 10:02:34.901 INFO
>>>  (qtp524223214-2127) [   x:dovecot-fts]
>>> o.a.s.u.p.LogUpdateProcessorFactory [dovecot-fts]  webapp=/solr
>>> path=/update params={}{add=[2/e836ff1d20640160340b532cee39/u@d
>>> (1688946475272241152)]} 0 80683
>>
>> I would greatly appreciate any help. My users using pasted images
>> heavily (designers exchanging mockups etc) and if this truly is a
>> problem then I am screwed :) 
>>
>>
>>> On 14. Jan 2021, at 23:46, Alexey Panov >> > wrote:
>>>
>>> Hello, 
>>>
>>> I am running recent dovecot 2.3.13 (89f716dc2) within dockerized
>>> mailcow setup. 
>>>
>>> During just another update routine I decided to reindex all
>>> mailboxes and some of them fail (timeout) on exactly SAME messages
>>> in SAME mailboxes:
>>>
 doveadm(u@d): Error: fts_solr: Indexing failed: Request timed out
 (Request queued 73.213 secs ago, 1 send attempts in 73.098 secs,
 60.597 in http ioloop, 0.000 in other ioloops, connected 127.875
 secs ago)
 doveadm(u@d): Error: Mailbox Sent: Mail search failed: Internal
 error occurred. Refer to server log for more information.
 [2021-01-14 11:23:46]
 doveadm(u@d): Error: Mailbox Sent: Transaction commit failed: FTS
 transaction commit failed: backend deinit
>>>
>>> This error is persistent and affects only some, but always same
>>> folders of same mailboxes (i mean solr can't finish indexing them at
>>> all). 
>>>
>>> I/O is in perfect condition and has no corruptions. I also tried to
>>> do fts rescan which made no change. RAM is fine, I haven't seen OOM
>>> conditions.
>>>
>>> During my research I found this discussion
>>>  which
>>> lead me idea of reducing the batch_size to as low as 10 (for testing
>>> purposes). Observing SOLR logs during reindexing on the
>>> mailbox/folder causing the error I noticed the following strange
>>> behaviour. 
>>>
>>> Firstly, it goes very fast but then always kinda freezes on these
>>> messages:
>>>
 solr-mailcow_1       | *2021-01-14 16:05:09.615* INFO
  (qtp524223214-1151) [   x:dovecot-fts]
 o.a.s.u.p.LogUpdateProcessorFactory [dovecot-fts]  webapp=/solr
 path=/update
 params={}{add=[6883/78a0ec27b091da5e405413681708/u@d
 (1688878773188952064), 6884/78a0ec27b091da5e405413681708/u@d
 (1688878773203632128), 6885/78a0ec27b091da5e405413681708/u@d
 (1688878773218312192), 6886/78a0ec27b091da5e405413681708/u@d
 (1688878773227749376), 6887/78a0ec27b091da5e405413681708/u@d
 (1688878773237186560), 6888/78a0ec27b091da5e405413681708/u@d
 (1688878773244526592), 6889/78a0ec27b091da5e405413681708/u@d
 (1688878773250818048), 6890/78a0ec27b091da5e405413681708/u@d
 (1688878773259206656), 6891/78a0ec27b091da5e405413681708/u@d
 (1688878773272838144), 6892/78a0ec27b091da5e405413681708/u@d
 (1688878773288566784)]} 0 101
 solr-mailcow_1       | *2021-01-14 16:06:42.540* INFO
  (qtp524223214-1143) [   x:dovecot-fts]
 o.a.s.u.p.LogUpdateProcessorFactory [dovecot-fts]  webapp=/solr
 path=/update
 params={}{add=[6893/78a0ec27b091da5e405413681708/u@d
 (1688878773329461248), 6894/78a0ec27b091da5e405413681708/u@d
 (1688878773332606976), 6895/78a0ec27b091da5e405413681708/u@d
 

Re: Solr and FTS - assertion failure [proposed patch for upper bound on rows in solr search]

[John Fawcett]

On 30/12/2020 16:04, Antonino Esposito wrote:
> Hi,
>
> in the latest weeks i'm working on the Solr integration and
> immediately i've faced the assertion failure errors, on 2.0.19, 2.2.9
> and 2.3.11.3 servers in our network.
> Reading the thread on debian ML, I realize this issue is related to
> nested MIME and it affects large mailboxes
>
> In my case, the error in dovecot.log pairs with the following on
> solr.log and it seems the rows value has the same value of the last
> UID recorded in the mailbox. 
>
> For your reference, here is the Solr logs, where *2276996170* is the
> value passed by Dovecot as rows number and it clearly don't fit with
> the rows data type.
>
> Have you had experienced the same behaviour? Is there a workaround?
> Thanks
> Antonino

Whatever the reason for this happening, it would make sense not to
supply unbounded values to solr.

The "rows" value that is passed for a lookup on a single mailbox is the
value of the uidnext for the searched mailbox. For lookups on multiple
mailboxes there is a hard coded value:

#define SOLR_MAX_MULTI_ROWS 10

If 10 is a good maximum for lookups on multiple mailboxes it could
also be a good upper bound for lookups on single mailboxes too.

My proposed patch, which stops too large "rows" values going to solr is
as follows. This doesn't solve the issue of why uidnext is so large in
the first place for the specific mailbox. Nevertheless I think it makes
sense both as a potential workaround to the original issue and to
incorporate it as a safeguard. If the hard-coded value is too limiting,
it could be made configurable.

diff -ur dovecot-2.3.11.3-orig/src/plugins/fts-solr/fts-backend-solr.c
dovecot-2.3.11.3/src/plugins/fts-solr/fts-backend-solr.c
--- dovecot-2.3.11.3-orig/src/plugins/fts-solr/fts-backend-solr.c  
2020-08-12 14:20:41.0 +0200
+++ dovecot-2.3.11.3/src/plugins/fts-solr/fts-backend-solr.c   
2020-12-31 09:05:07.681897716 +0100
@@ -838,7 +838,7 @@

    str = t_str_new(256);
    str_printfa(str,
"wt=xml=uid,score=%u=uid+asc=%%7b!lucene+q.op%%3dAND%%7d",
-   status.uidnext);
+   I_MIN(status.uidnext,SOLR_MAX_MULTI_ROWS));
    prefix_len = str_len(str);

    if (solr_add_definite_query_args(str, args, and_args)) {

John

Re: Solr and FTS - assertion failure

[John Fawcett]

On 30/12/2020 16:04, Antonino Esposito wrote:
> Hi,
>
> in the latest weeks i'm working on the Solr integration and
> immediately i've faced the assertion failure errors, on 2.0.19, 2.2.9
> and 2.3.11.3 servers in our network.
> Reading the thread on debian ML, I realize this issue is related to
> nested MIME and it affects large mailboxes
>
> In my case, the error in dovecot.log pairs with the following on
> solr.log and it seems the rows value has the same value of the last
> UID recorded in the mailbox. 
>
> For your reference, here is the Solr logs, where *2276996170* is the
> value passed by Dovecot as rows number and it clearly don't fit with
> the rows data type.
>
> Have you had experienced the same behaviour? Is there a workaround?
> Thanks
> Antonino
>
Hi Antonio

out of curiosity, what does the dovecot log show for this issue?

John

Re: AUTH command behaviour when authentication process failed or aborted by user is changed

[John Fawcett]

On 24/12/2020 08:42, Михаил Сандаков wrote:
> Hi all,
>
> .
>
> Also seems like new behaviour is not fully accorded to rfc1734
> (https://tools.ietf.org/html/rfc1734
> ). Based on this part of rfc:
> "If an AUTH
>  command fails with a negative response, the session remains
>  in the AUTHORIZATION state and client may try another
>  authentication mechanism by issuing another AUTH command,
>  or may attempt to authenticate by using the USER/PASS or
>  APOP commands."
> I think "AUTH command" means full auth command started with "AUTH",
> not only authentication method. But I may be wrong in the
> interpretation of this part.
>
Your reading of the RFC looks correct. AUTH commands start with the
string "AUTH". This is no different after a negative response.

John
> I suppose this changes was made in commit
> https://github.com/dovecot/core/commit/2c42881c056e5ab2e2e14b2f800d6dc72026399b
> ,
> but not sure about this. Seems like if authentication is aborted or
> failed client->current_cmd is not going to be cleaned.
>
> So I suggest to revert old behaviour in this cases. If I am right with
> commit, it seems like we just need to add clearing client->current_cmd
> if the authentication process has failed somehow.

Re: Unexpected behavior of doveadm mailbox delete -e -s

[John Fawcett]

On 13/12/2020 05:59, Mar Alegre wrote:
> Hello,
>
> I have encountered what I believe to be a bug, or at least inaccurate
> documentation. I am running version 2.3.4.1 of Dovecot (installed via
> the default repo for Debian 10), so my apologies if this has been
> fixed in a more recent version. I tried to look for changelogs but was
> not able to find any.
>
> The `doveadm mailbox delete` command accepts flags `-e` and `-s`
> (among others). The flags are described by both the man pages and the
> wiki as follows:
>
> - `-e`: Require mailboxes to be empty before deleting.
> - `-s`: Unsubscribe deleted mailboxes.
>
> Based on these descriptions, I would expect that when both of these
> flags are combined, a directory that is not empty would be neither
> deleted nor unsubscribed. However, I have discovered that when I run
> the command `doveadm mailbox delete -u $myusername -e -s
> $non_empty_dir`, the result is that the directory is not deleted, but
> it *is* unsubscribed.
>
> I assume this is a bug, but if it is intended, the documentation
> should probably be updated to more clearly communicate that the `-s`
> flag will cause a directory to be unsubscribed even if it is not deleted.
>
> Best,
> Mar Alegre

Hopefully this will fix the issue, allowing the unsubscribe to go ahead
only if the delete did not return an error (<0). There are some cases,
namely  mailbox not open, in which 0 is returned without deleting the
mailbox (autocreated mailboxes) or the delete is done without checking
for an empty mailbox, but I guess those are not relevant when calling
the delete from doveadm, though I may be wrong on that.

--- doveadm/doveadm-mail-mailbox.c.orig    2020-12-13 16:58:59.589077259
+0100
+++ doveadm/doveadm-mail-mailbox.c    2020-12-13 17:00:28.326194491 +0100
@@ -383,7 +383,7 @@
         doveadm_mail_failed_mailbox(_ctx, box);
         ret = -1;
     }
-        if (ctx->ctx.subscriptions) {
+        if (ret2>=0 && ctx->ctx.subscriptions) {
         if (mailbox_set_subscribed(box, FALSE) < 0) {
             i_error("Can't unsubscribe mailbox %s: %s", name,
                 mail_storage_get_last_internal_error(storage, NULL));

Potential issue with lmtp_rawlog_dir when using TLS

[John Fawcett]

Hi

not sure why this happens, but looks like lmtp_rawlog_dir blocks the
successful negotiation of an lmtp starttls connection.

Reproducible with openssl s_client.  Two runs with only difference the
presence of lmtp_rawlog_dir setting in dovecot config.

Failed connection with lmtp_rawlog_dir

Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Connect from ::1
Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Debug: SSL:
where=0x10, ret=1: before SSL initialization
Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Debug: SSL:
where=0x2001, ret=1: before SSL initialization
Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Debug: SSL:
where=0x2002, ret=-1: before SSL initialization
Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Debug: SSL:
where=0x2002, ret=-1: error
Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Debug: SSL error:
SSL_accept() failed: error:1408F10B:SSL routines:ssl3_get_record:wrong
version number
Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Error:
lmtp-server: conn [::1]:53876 [2]: SSL handshake failed: SSL_accept()
failed: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Debug: SSL error:
SSL_accept() syscall failed: Invalid argument
Dec 10 21:01:31 server03 dovecot[41002]: lmtp(41021): Disconnect from
::1: SSL Initialization failed (state=READY)


Successful connection without lmtp_rawlog_dir

Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Connect from ::1
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x10, ret=1: before SSL initialization
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: before SSL initialization
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: before SSL initialization
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: before SSL initialization
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: before SSL initialization
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: before SSL initialization
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: before SSL initialization
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: before SSL initialization
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug:
SSL_get_servername() failed
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: SSLv3/TLS read client hello
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: SSLv3/TLS write server hello
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: SSLv3/TLS write change cipher spec
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: TLSv1.3 write encrypted extensions
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: SSLv3/TLS write certificate
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: TLSv1.3 write server certificate verify
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: SSLv3/TLS write finished
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=-1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: TLSv1.3 early data
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: SSLv3/TLS read finished
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x20, ret=1: SSLv3/TLS write session ticket
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: SSLv3/TLS write session ticket
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2001, ret=1: SSLv3/TLS write session ticket
Dec 10 20:59:07 server03 dovecot[40970]: lmtp(40988): Debug: SSL:
where=0x2002, ret=1: SSL negotiation finished successfully

John

Re: Dovecot 2.3.11.3 LMTP dropping connection after first part of multipart message received

[John Fawcett]

On 10/12/2020 14:58, Steve Akerman wrote:
> Good afternoon
>
> I have been using Dovecot with great success for several months now
> with one problem:
>
> I have received several messages (3 to date out of a much larger
> number) which cause Dovecot LMTP to drop the connection with sendmail
> after only receiving the first part of the message.
>
> Dovecot saves the first part to INBOX, and then drops the link with
> sendmail as it sees the rest of the data from sendmail as new (and
> invalid) commands
>
> Sendmail sees the delivery as unsuccessful and requeues the message,
> so the cycle continues until i kill the queue
>
> These messages are all Multipart MIME
>
> I have attached an extract from the log which shows the above sequence
>
> Finally, I have kept one such message from the sendmail queue (df and
> qf) should this be useful, but prefer not to post here as not sanitised 
>
> Any help in correcting my configuration would be appreciated.
>
> Thanks
>
> Steve
>
> 
>
Hi Steve

strange as it sounds, looks like your mta is sending the lmtp end of
message sequence (.)in the middle of an email.

You might be able to debug it in dovecot with lmtp_rawlog_dir setting
which will show the input and output dialog of the lmtp session.

John

Re: Question about login_log_format_elements in a proxy environment

[John Fawcett]

On 08/12/2020 01:01, Adi Pircalabu wrote:
> On 08-12-2020 10:33, Adi Pircalabu wrote:
>> On 08-12-2020 9:41, John Fawcett wrote:
>>> On 07/12/2020 23:22, John Fawcett wrote:
>>>> On 07/12/2020 23:09, Adi Pircalabu wrote:
>>>>> On 08-12-2020 3:13, John Fawcett wrote:
>>>>>> On 07/12/2020 06:02, Adi Pircalabu wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have a Dovecot proxy setup with several proxy machines (currently
>>>>>>> running 2.3.11.3) in front of the real Dovecot servers (2.3.10.1)
>>>>>>> storing the mailboxes. "doveconf -a | egrep lip" returns:
>>>>>>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l
>>>>>>> mpid=%e
>>>>>>> %c session=<%{session}>
>>>>>>>
>>>>>>> In the real server maillog I'm expecting to have "lip" replaced
>>>>>>> with
>>>>>>> the IP address of the proxy. It works as expected for imap-login
>>>>>>> processes, however for pop3-login processes I still see the real
>>>>>>> server IP instead of the proxy IP. Ideas?
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>> Hi Adi
>>>>>>
>>>>>> in general people want to get the original ip not the proxied ip.
>>>>>> The
>>>>>> proxying of the original ip is done by a different method for imap
>>>>>> and pop3
>>>>>>
>>>>>> https://wiki.dovecot.org/Design/ParameterForwarding
>>>>>>
>>>>>> However, unless I'm reading this wrongly, both methods are
>>>>>> affected by
>>>>>> trusted_networks settings. I guess for people to help further, you'd
>>>>>> need to give more info your configuration settings.
>>>>> Thanks John. login_trusted_networks, if this is the setting you're
>>>>> referring to, lists the proxy IPs. I'd have thought, by having this
>>>>> setting on the real servers, the proxy IP will be logged by both IMAP
>>>>> and POP3 login processes, but it appears it isn't the case. It works
>>>>> for IMAP, not for POP3.
>>>>> The reason I need the proxy IP in the "lip" instead of the local
>>>>> IP in
>>>>> the real server mail log is that I need to filter certain
>>>>> connections,
>>>>> both IMAP and POP3, that are coming directly into the real server IP.
>>>>> By capturing the IMAP & POP3 traffic on the real servers and matching
>>>>> the results to the mail log entries I *should* be able to tell what
>>>>> mail accounts from which remote IP addresses are coming in via the
>>>>> proxies and which ones are coming into the real servers directly.
>>>>> Hope
>>>>> that makes sense.
>>>>> Cheers,
>>>>>
>>>> The way I read it is that by specifing login_trusted_networks the
>>>> proxy
>>>> ip can be overwritten by the real ip. I think that's the opposite of
>>>> what you need.
>>>>
>>>> I can't throw any light on why that is not working for imap but is
>>>> working for pop3. But as you don't want the overwriting, maybe you
>>>> should try without login_trusted_networks.
>>>>
>>>> John
>>>>
>>> You're probably not getting the real ip logged for imap despite having
>>> login_trusted_networks due to the default for imap_id_retain on the
>>> proxies.
>>>
>>> John
>>
>> (Aki cc-ed)
>> Thanks. I actually need login_trusted_networks on the real servers so
>> that the real server has access to the client IP address, aka "rip" in
>> the log entry. What I need is consistent values for "lip" field for
>> both IMAP and POP3 login processes. Looking at
>> https://doc.dovecot.org/configuration_manual/proxy_settings/ there's
>> no mention the setting is working for IMAP only, not for POP3. What I
>> need for my use case is to get consistent logging for both protocols.
>> More precisely, considering:
>> - REALSERVER.IP as the real server IP address
>> - CLIENT.IP as the client IP address
>> - "login_trusted_networks = PROXY.IP" set in the real server config
>> I'm expecting to see the following information in the mail log

Re: Question about login_log_format_elements in a proxy environment

[John Fawcett]

On 07/12/2020 23:22, John Fawcett wrote:
> On 07/12/2020 23:09, Adi Pircalabu wrote:
>> On 08-12-2020 3:13, John Fawcett wrote:
>>> On 07/12/2020 06:02, Adi Pircalabu wrote:
>>>> Hi,
>>>>
>>>> I have a Dovecot proxy setup with several proxy machines (currently
>>>> running 2.3.11.3) in front of the real Dovecot servers (2.3.10.1)
>>>> storing the mailboxes. "doveconf -a | egrep lip" returns:
>>>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e
>>>> %c session=<%{session}>
>>>>
>>>> In the real server maillog I'm expecting to have "lip" replaced with
>>>> the IP address of the proxy. It works as expected for imap-login
>>>> processes, however for pop3-login processes I still see the real
>>>> server IP instead of the proxy IP. Ideas?
>>>>
>>>> Regards,
>>>>
>>> Hi Adi
>>>
>>> in general people want to get the original ip not the proxied ip. The
>>> proxying of the original ip is done by a different method for imap
>>> and pop3
>>>
>>> https://wiki.dovecot.org/Design/ParameterForwarding
>>>
>>> However, unless I'm reading this wrongly, both methods are affected by
>>> trusted_networks settings. I guess for people to help further, you'd
>>> need to give more info your configuration settings.
>> Thanks John. login_trusted_networks, if this is the setting you're
>> referring to, lists the proxy IPs. I'd have thought, by having this
>> setting on the real servers, the proxy IP will be logged by both IMAP
>> and POP3 login processes, but it appears it isn't the case. It works
>> for IMAP, not for POP3.
>> The reason I need the proxy IP in the "lip" instead of the local IP in
>> the real server mail log is that I need to filter certain connections,
>> both IMAP and POP3, that are coming directly into the real server IP.
>> By capturing the IMAP & POP3 traffic on the real servers and matching
>> the results to the mail log entries I *should* be able to tell what
>> mail accounts from which remote IP addresses are coming in via the
>> proxies and which ones are coming into the real servers directly. Hope
>> that makes sense.
>> Cheers,
>>
> The way I read it is that by specifing login_trusted_networks the proxy
> ip can be overwritten by the real ip. I think that's the opposite of
> what you need.
>
> I can't throw any light on why that is not working for imap but is
> working for pop3. But as you don't want the overwriting, maybe you
> should try without login_trusted_networks.
>
> John
>
You're probably not getting the real ip logged for imap despite having
login_trusted_networks due to the default for imap_id_retain on the proxies.

John

Re: Question about login_log_format_elements in a proxy environment

[John Fawcett]

On 07/12/2020 23:09, Adi Pircalabu wrote:
> On 08-12-2020 3:13, John Fawcett wrote:
>> On 07/12/2020 06:02, Adi Pircalabu wrote:
>>> Hi,
>>>
>>> I have a Dovecot proxy setup with several proxy machines (currently
>>> running 2.3.11.3) in front of the real Dovecot servers (2.3.10.1)
>>> storing the mailboxes. "doveconf -a | egrep lip" returns:
>>> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e
>>> %c session=<%{session}>
>>>
>>> In the real server maillog I'm expecting to have "lip" replaced with
>>> the IP address of the proxy. It works as expected for imap-login
>>> processes, however for pop3-login processes I still see the real
>>> server IP instead of the proxy IP. Ideas?
>>>
>>> Regards,
>>>
>> Hi Adi
>>
>> in general people want to get the original ip not the proxied ip. The
>> proxying of the original ip is done by a different method for imap
>> and pop3
>>
>> https://wiki.dovecot.org/Design/ParameterForwarding
>>
>> However, unless I'm reading this wrongly, both methods are affected by
>> trusted_networks settings. I guess for people to help further, you'd
>> need to give more info your configuration settings.
>
> Thanks John. login_trusted_networks, if this is the setting you're
> referring to, lists the proxy IPs. I'd have thought, by having this
> setting on the real servers, the proxy IP will be logged by both IMAP
> and POP3 login processes, but it appears it isn't the case. It works
> for IMAP, not for POP3.
> The reason I need the proxy IP in the "lip" instead of the local IP in
> the real server mail log is that I need to filter certain connections,
> both IMAP and POP3, that are coming directly into the real server IP.
> By capturing the IMAP & POP3 traffic on the real servers and matching
> the results to the mail log entries I *should* be able to tell what
> mail accounts from which remote IP addresses are coming in via the
> proxies and which ones are coming into the real servers directly. Hope
> that makes sense.
> Cheers,
>
The way I read it is that by specifing login_trusted_networks the proxy
ip can be overwritten by the real ip. I think that's the opposite of
what you need.

I can't throw any light on why that is not working for imap but is
working for pop3. But as you don't want the overwriting, maybe you
should try without login_trusted_networks.

John

Re: Question about login_log_format_elements in a proxy environment

[John Fawcett]

On 07/12/2020 06:02, Adi Pircalabu wrote:
> Hi,
>
> I have a Dovecot proxy setup with several proxy machines (currently
> running 2.3.11.3) in front of the real Dovecot servers (2.3.10.1)
> storing the mailboxes. "doveconf -a | egrep lip" returns:
> login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e
> %c session=<%{session}>
>
> In the real server maillog I'm expecting to have "lip" replaced with
> the IP address of the proxy. It works as expected for imap-login
> processes, however for pop3-login processes I still see the real
> server IP instead of the proxy IP. Ideas?
>
> Regards,
>
Hi Adi

in general people want to get the original ip not the proxied ip. The
proxying of the original ip is done by a different method for imap and pop3

https://wiki.dovecot.org/Design/ParameterForwarding

However, unless I'm reading this wrongly, both methods are affected by
trusted_networks settings. I guess for people to help further, you'd
need to give more info your configuration settings.

John

Re: dovecot/imap killed with signal 6 after SEARCH in specific mailbox

[John Fawcett]

On 07/12/2020 16:58, Jiri wrote:
> Hi All,
> on my older production server (Deb9 dovecot version 2.2.27-3+deb9u6) a
> client reported that searching text in one mailbox failed while other
> mailboxes do search ok.
> He is using Rouddcube webmail and is searching the whole messages
> including the body.
> Errorlog shows panic and assertion failed logs.
> So I'v moved the 'problematic' mailbox to my latest production server
> (Deb10 2.3.4.1-5+deb10u4 = 2.3.4.1 (f79e8e7e4)) where I have more RAM
> to eliminate that and the same error appeared.
> There is nothing special about the mailbox, roughly 600MiB and 2200
> messges.
> Other much larger mailboxes on the same server do search fine.
> After turning on imap logging in Roundcube mail, the critical command
> was:
> A0004 UID SEARCH RETURN (ALL) TEXT whatever
> I increased default_process_limit and default_client_limit, I run:
> doveadm index -u t...@mail2.swpraha.cz -q INBOX
> but didn't help.
>
>
> I used the gdb method to reproduce it and to get the whole info
> including backtrace:
>
> vmail@s6mail:~$ gdb --args /usr/lib/dovecot/imap -u t...@mail2.swpraha.cz
> (gdb) r
> Starting program: /usr/lib/dovecot/imap -u t...@mail2.swpraha.cz
> process 4458 is executing new program: /usr/bin/doveconf
> process 4458 is executing new program: /usr/lib/dovecot/imap
> * PREAUTH [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE
> SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT
> MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS
> LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES
> WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY LITERAL+
> NOTIFY SPECIAL-USE QUOTA] Logged in as t...@mail2.swpraha.cz
> a select INBOX
> * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $MDNSent Junk
> $Forwarded NonJunk)
> * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft
> $MDNSent Junk $Forwarded NonJunk \*)] Flags permitted.
> * 682 EXISTS
> * 0 RECENT
> * OK [UNSEEN 639] First unseen.
> * OK [UIDVALIDITY 1147771637] UIDs valid
> * OK [UIDNEXT 104487] Predicted next UID
> * OK [HIGHESTMODSEQ 53108] Highest
> a OK [READ-WRITE] Select completed (0.002 + 0.000 + 0.001 secs).
> A0004 UID SEARCH RETURN (ALL) TEXT whatewer
> imap(t...@mail2.swpraha.cz)<4458>: Panic: file
> message-parser.c: line 174 (message_part_finish): assertion failed:
> (ctx->nested_parts_count > 0)
> imap(t...@mail2.swpraha.cz)<4458>: Error: Raw
> backtrace: /usr/lib/dovecot/libdovecot.so.0(+0xdb62b) [0x77dbb62b]
> -> /usr/lib/dovecot/libdovecot.so.0(+0xdb661) [0x77dbb661] ->
> /usr/lib/dovecot/libdovecot.so.0(+0x4a149) [0x77d2a149] ->
> /usr/lib/dovecot/libdovecot.so.0(+0x474ac) [0x77d274ac] ->
> /usr/lib/dovecot/libdovecot.so.0(message_parser_parse_next_block+0x104)
> [0x77da3844] ->
> /usr/lib/dovecot/libdovecot.so.0(message_search_msg+0xa8)
> [0x77da5df8] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xcf89e)
> [0x77f3e89e] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mail_search_args_foreach+0x45)
> [0x77ec0445] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd0774)
> [0x77f3f774] -> /usr/lib/dovecot/libdovecot-storage.so.0(+0xd1a68)
> [0x77f40a68] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(index_storage_search_next_nonblock+0x10d)
> [0x77f4118d] ->
> /usr/lib/dovecot/libdovecot-storage.so.0(mailbox_search_next_nonblock+0x28)
> [0x77ec9e58] -> /usr/lib/dovecot/imap [t...@mail2.swpraha.cz  UID
> SEARCH](+0x2691f) [0x5557a91f] -> /usr/lib/dovecot/imap
> [t...@mail2.swpraha.cz  UID SEARCH](command_exec+0x70) [0x55573d80]
> -> /usr/lib/dovecot/imap [t...@mail2.swpraha.cz  UID SEARCH](+0x25ed2)
> [0x55579ed2] ->
> /usr/lib/dovecot/libdovecot.so.0(io_loop_handle_timeouts+0x111)
> [0x77dd18f1] ->
> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xd0)
> [0x77dd3070] ->
> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x4c)
> [0x77dd1b7c] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40)
> [0x77dd1ce0] ->
> /usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13)
> [0x77d520d3] -> /usr/lib/dovecot/imap [t...@mail2.swpraha.cz  UID
> SEARCH](main+0x325) [0x55564bf5] ->
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)
> [0x77b3c09b] -> /usr/lib/dovecot/imap [t...@mail2.swpraha.cz  UID
> SEARCH](_start+0x2a) [0x55564d8a]
>
> Program received signal SIGABRT, Aborted.
> __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
> 50  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt full
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
>     set = {__val = {0, 9355454118400172544, 1928, 1862, 1863,
> 93824992506008, 65, 140737352109185, 140737351192526, 140737351951989,
> 257, 145, 206158430224, 140737488344944, 140737488344736,
> 140737351756174}}
>     pid = 
>     tid = 
>     ret = 
> #1  0x77b3a535 in __GI_abort () 

Re: logging in to postfixadmin failed probably because of a dovecot error

[John Fawcett]

On 02/12/2020 02:25, peacecop kalmer: wrote:
>
> I can’t log in to postfixadmin anymore. It says that the username or
> password is incorrect. It can’t be incorrect as I’ve stored them in
> Google and “Chrome”. I have two admin accounts and none of them can’t
> be used for logging in. So I changed the main password with
> “setup.php” and copied that hash into the configuration file. Then, I
> deleted one of the admin accounts directly in the database from the
> table “admin”. Finally, I tried to recreate that deleted admin account
> using “setup.php”. I get the following error message: “can’t encrypt
> password with dovecotpw, see error log for details”.
>
> I had a look into the error log file of postfixadmin and found these
> complaints:
>
> [Wed Dec 02 02:33:37.551048 2020] [authz_core:debug] [pid 30540]
> mod_authz_core.c(809): [client 192.168.1.65:54948] AH01626:
> authorization result of Require all granted: granted,
> referer: https://postfixadmin.tennis24.ee/setup.php
> 
> [Wed Dec 02 02:33:37.551070 2020] [authz_core:debug] [pid 30540]
> mod_authz_core.c(809): [client 192.168.1.65:54948] AH01626:
> authorization result of : granted,
> referer: https://postfixadmin.tennis24.ee/setup.php
> 
> [Wed Dec 02 02:33:37.551280 2020] [php7:warn] [pid 30540] [client
> 192.168.1.65:54948] PHP Warning: Cannot assign an empty string to
> a string offset in /usr/share/postfixadmin/variables.inc.php on
> line 31, referer: https://postfixadmin.tennis24.ee/setup.php
> 
> [Wed Dec 02 02:33:37.568128 2020] [php7:notice] [pid 30540]
> [client 192.168.1.65:54948] dovecotpw password encryption failed.,
> referer: https://postfixadmin.tennis24.ee/setup.php
> 
> [Wed Dec 02 02:33:37.568155 2020] [php7:notice] [pid 30540]
> [client 192.168.1.65:54948] STDERR output: doveconf: Fatal: Error
> in configuration file /etc/dovecot/conf.d/10-ssl.conf line 13:
> ssl_cert: Can't open file
> /etc/letsencrypt/live/test.tennis24.ee/fullchain.pem: Permission
> denied\n, referer: https://postfixadmin.tennis24.ee/setup.php
> 
> [Wed Dec 02 02:33:37.568897 2020] [deflate:debug] [pid 30540]
> mod_deflate.c(854): [client 192.168.1.65:54948] AH01384: Zlib:
> Compressed 1320 to 732 : URL /setup.php,
> referer: https://postfixadmin.tennis24.ee/setup.php
> 
>
> Is the issue connected to the permission problem? If so why is the
> permission denied?
>
> I can't even receive any mails to the address kal...@test.tennis24.ee
> anymore. If I send out it works fine but no mail arrives.
>
> -- 
>
> *peacecop kalmer:*
>
> *Instituudi tee 3-27 *
>
> *76902 * *Harku *
>
> *Harku*
>
> *Harjumaa*
> Estonia
>
> *+372  652 4228 *
>
> *+372  5620 4556 *
>
> *__ *
>
The following may help

https://sourceforge.net/p/postfixadmin/discussion/676076/thread/0a7aaf6305/

https://dovecot.org/pipermail/dovecot/2020-August/119642.html

John

[patch] enhancement for adding limits to solr and tika parsing

[John Fawcett]

Hi

As mentioned in a previous thread, the solr + tika combination has
caused me some issues due to attachment size. While tika seems to be
able to parse large attachments, the resulting volume of text can
overwhelm the solr server.

One solution would be to throw resources at the problem, but in my case
such large attachments don't contain anything worthwile indexing.
Additionally I don't want people to be able to randomly crash my solr
server by sending large compressed attachments that expand into huge
volumes for solr. It's also a safety feature to have sane limits on what
can be indexed.

Attached is a first attempt to address the problem. I did not find a way
to easily get actual attachment sizes, so I used an already available
information - the overall message size. It may not be ideal but at least
introduces limits where none existed.

I have introduced two new parameters for the plugin section, for example:

plugin {

    fts_max_size = 2M
    fts_max_size_tika = 1M

}
They can be used separately or together. Both sizes refer to the overall
message size. The meaning is:

fts_max_size - do not parse message bodies if the message size exceeds
this value. A value of 0 indicates no limit.  If the message body is not
parsed, attachments are also not parsed.

fts_max_size_tika - do not parse message attachments with tika if the
message size exceeds this value. A value of 0 indicates no limit.

If using both settings it makes sense to have fts_max_size >
fts_max_size_tika, since  with a smaller fts_max_size bodies are not
indexed including attachments and the fts_max_size_tika will have no
effect.

The difference (ft_max_size - fts_max_size_tika) places an upper bound
on the size of the non attachment body text that will be indexed.
However, any attachments over the fts_size will automatically consume
this limit and no body text will be indexed for those messages. I've
only updated the tika parser not the script parser though the script
parser potentially could benefit from this approach.

The attached patch also includes the rolled up patch for using basic
auth with the tika server and the previous posted patch (not mine) which
solves an assert when using solr and tika together.

John






diff -ur dovecot-2.3.11.3-orig/src/plugins/fts/fts-build-mail.c 
dovecot-2.3.11.3/src/plugins/fts/fts-build-mail.c
--- dovecot-2.3.11.3-orig/src/plugins/fts/fts-build-mail.c  2020-08-12 
14:20:41.0 +0200
+++ dovecot-2.3.11.3/src/plugins/fts/fts-build-mail.c   2020-12-07 
14:05:23.654217555 +0100
@@ -17,6 +17,7 @@
 #include "fts-filter.h"
 #include "fts-api-private.h"
 #include "fts-build-mail.h"
+#include "settings-parser.h"
 
 /* there are other characters as well, but this doesn't have to be exact */
 #define IS_WORD_WHITESPACE(c) \
@@ -34,6 +35,7 @@
 
buffer_t *word_buf, *pending_input;
struct fts_user_language *cur_user_lang;
+   bool oversized_tika;
 };
 
 static int fts_build_data(struct fts_mail_build_context *ctx,
@@ -236,7 +238,7 @@
parser_context.user = mail_storage_get_user(storage);
parser_context.content_disposition = ctx->content_disposition;
 
-   
+   parser_context.oversized_tika = ctx->oversized_tika;
if (fts_parser_init(_context, >body_parser)) {
/* extract text using the the returned parser */
*binary_body_r = TRUE;
@@ -488,7 +490,32 @@
bool binary_body;
const char *error;
int ret;
-
+   uoff_t msg_size;
+   uoff_t fts_max_size = 0;
+   uoff_t fts_max_size_tika = 0;
+   const char * fts_max_size_setting;
+   const char * fts_max_size_tika_setting;
+   bool oversized_msg;
+   bool oversized_tika;
+
+   fts_max_size_setting = 
mail_user_plugin_getenv(update_ctx->backend->ns->user, "fts_max_size");   
+   if (fts_max_size_setting != NULL) {
+   i_debug("fts_max_size %s",fts_max_size_setting);
+   if (settings_get_size(fts_max_size_setting, _max_size, 
) < 0) {
+   i_error("%s",error);
+   fts_max_size = 0;   
+   }
+   i_debug("fts_max_size (value) %"PRIuUOFF_T,fts_max_size);
+   }
+   fts_max_size_tika_setting = 
mail_user_plugin_getenv(update_ctx->backend->ns->user, "fts_max_size_tika");   
+   if (fts_max_size_tika_setting != NULL) {
+   i_debug("fts_max_size_tika %s",fts_max_size_tika_setting);
+   if (settings_get_size(fts_max_size_tika_setting, 
_max_size_tika, ) < 0) {
+   i_error("%s",error);
+   fts_max_size_tika = 0;  
+   }
+   i_debug("fts_max_size_tika (value) 
%"PRIuUOFF_T,fts_max_size_tika);
+   }
*may_need_retry_r = FALSE;
if (mail_get_stream_because(mail, NULL, NULL, "fts indexing", ) < 
0) {
if (mail->expunged)
@@ -498,10 +525,21 @@

Re: Fts solr add fields message-id, date, in-reply-to

[John Fawcett]

On 07/12/2020 02:21, peacecop kalmer: wrote:
>
> I remember to have written a message to the list but maybe, it wasn't
> to the list but just the sender. Anyway, here it is:
>
> "Can't you start using an environment where we can separate different
> topics and lessen mail traffic? I subscribed to the list because there
> was a problem with reading a certificate. I could solve it by manually
> changing the file permissions although that solution wasn't elegant.
> Nobody of creators of /dovecot/ did help. Now, I get lots of emails on
> topics that are irrelevant to me. Why do we put all that stuff into
> one bucket where we can't sort things?"
>
> And yes, I did write it to Peter Cooper Jr. only hoping to get to the
> superior manager of this overwhelming list. He didn't respond. It
> didn't happen, so please respond to me and stop this spamming machine.
> I don't need to know how to set up an inconvenient environment or how
> to add fields or any segmentation faults but only the solution to my
> particular problem which is solved for now without your help guys. Or
> actually, it isn't solved, that was just a workaround. A not smart
> one. So please solve the problem because after all some of you have
> created this software. For the ones of you that can't find my initial
> message to the list, it was about /dovecot/ lurking for some
> certificates after an update making logging in into postfix
> impossible. I made fire by performing that workaround, can you guys
> make fire by making this all right by fixing your software or at least
> reasoning the anomaly.
>
> *peacecop kalmer:*
>
> *Instituudi tee 3-27 *
>
> *76902 * *Harku *
>
> *Harku*
>
> *Harjumaa*
> Estonia
>
> *+372  652 4228 *
>
> *+372  5620 4556 *
>
> *__ *
>
You can avoid being overwhelmed by the list traffic by defining a
subfolder and filter email into automatically, for example using sieve
or your email client filtering. If you have a modern email client, it
will be able to organize email by threads and so you can easily ignore
any messages which are not in reply to your thread or on topics that are
not of interest.

Alternatively if none of the list traffic is of interest, you can easily
stop the messages from this list by using the unsubscribe link which you
can find in the header of all the list email or by using this link

https://dovecot.org/mailman/listinfo/dovecot

John