Re: Corrupted sizes in cache once again

[Tim Evers]

Hi again,

so this is the actual bug report :)

I installed 2.3.20 from repo yet the errors persist.

I made the following observations:

For a mailbox producing the "broken physical size" messages the culprit 
seems not to be the index.cache file but the dovecot-uidlist file. 
Removing the cache file does nothing regarding the bug while removing 
the dovecot-uidlist file immediately cleared the errors and all 
subsequent access went through without error.


An example of an uidlist entry *with* errors:

70755 G1674471014.M447254P8688.node2 S937 
:1674471014.M447254P8688.node2,S=1693,W=1730


Note: the S937 entry. This is definitely wrong.

Same mail *without* the errors (dovecot-uidlist auto-recreated after 
deletion):


76040 :1674471014.M447254P8688.node2,S=1693,W=1730

Errors in log (domain anonymized):

Feb 15 21:02:57 node2 dovecot: 
imap(local_u...@domain.com)<36929>: Error: Mailbox 
INBOX.Trash: Deleting corrupted cache record uid=70755: UID 70755: 
Broken physical size in mailbox INBOX.Trash: 
read(compress(/home/mail/domains/v/a/domain.com/local_user/Maildir/.Trash/cur/1674471014.M447254P8688.node2,S=1693,W=1730:2,)) 
failed: Cached message size smaller than expected (937 < 1693, 
box=INBOX.Trash, UID=70755)
Feb 15 21:02:57 node2 dovecot: 
imap(local_u...@domain.com)<36929>: Error: Mailbox 
INBOX.Trash: UID=70755: 
read(compress(/home/mail/domains/v/a/domain.com/local_user/Maildir/.Trash/cur/1674471014.M447254P8688.node2,S=1693,W=1730:2,)) 
failed: Cached message size smaller than expected (937 < 1693, 
box=INBOX.Trash, UID=70755) (read reason=)


Note: reported physical size from "cache", 937 is the compressed size on 
disk. Decompressed size is 1693.


Excerpt from strace (full strace  attached):

36929 openat(AT_FDCWD, 
"/home/mail/domains/v/a/domain.com/local_user/Maildir/.Trash/dovecot.index.cache", 
O_RDWR) = -1 ENOENT (No such file or directory)
36929 openat(AT_FDCWD, 
"/home/mail/domains/v/a/domain.com/local_user/Maildir/.Trash/cur/1674471014.M447254P8688.node2,S=1693,W=1730:2,", 
O_RDONLY) = 20

36929 fstat(20, {st_mode=S_IFREG|0600, st_size=937, ...}) = 0
36929 pread64(20, 
"\37\213\10\0\0\0\0\0\0\3\255\224]O\3438\30\205\257\311\257\260\346\252hI\326\316g\223\231\2I\232\360!\272Ci\27\255\26\241\310M\334\326C\342tl\247|\374\372y\303\262\2\26\0207\233H\215\353\274\366y\217\237\243\\\260\222\361-\253\"\264\224m\203\32\312\353\266\323\266E\245^\362\232)\253bhpeS\214#\262\f\206\21]b;\212\354\360z\327\330Y\334?\326saclU\345F1\271e\262_q\313\365\0321\325\350\215B\203\371\331\214XN\224\245\343\343\254\230e\351\271\355\371\27\244(.fqq>\233\365O\230?\216a\272(\342lV\364\203\243t\22\301\23T\6\331\35o\220k\205\256e\303_^!\262"..., 
8192, 0) = 937
36929 openat(AT_FDCWD, 
"/home/mail/domains/v/a/domain.com/local_user/Maildir/.Trash/dovecot.index.cache", 
O_RDWR) = -1 ENOENT (No such file or directory)
36929 openat(AT_FDCWD, 
"/home/mail/domains/v/a/domain.com/local_user/Maildir/.Trash/dovecot.index.cache", 
O_RDWR) = -1 ENOENT (No such file or directory)

36929 alarm(180)    = 0
36929 fcntl(16, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, 
l_len=0}) = 0

36929 alarm(0)  = 180
36929 
stat("/home/mail/domains/v/a/domain.com/local_user/Maildir/.Trash/dovecot.index.log", 
{st_mode=S_IFREG|0600, st_size=144, ...}) = 0

36929 fstat(16, {st_mode=S_IFREG|0600, st_size=144, ...}) = 0
36929 write(16, 
"\200\200\200\203\0\0\10\0208\0\0\0\200\200\200\207@\0\0\20\2\0\0\0002\315\321\\\0\0\0\0\4\0\4\0\1\0\0\0\200\200\200\204\0\2\0\20c\24\1\0\0\0\0\0", 
56) = 56
36929 fcntl(16, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, 
l_len=0}) = 0
36929 
stat("/home/mail/domains/v/a/domain.com/local_user/Maildir/.Trash/dovecot.index.log", 
{st_mode=S_IFREG|0600, st_size=200, ...}) = 0

36929 fstat(16, {st_mode=S_IFREG|0600, st_size=200, ...}) = 0
36929 write(2, "\1\30436929 47 
imap(local_u...@domain.com)<36929>: Mailbox 
INBOX.Trash: Deleting corrupted cache record uid=70755: UID 70755: 
Broken physical size in mailbox INBOX.Trash: read(compress(/home/ma"..., 
389) = 389


No crash, no core, no backtrace.

That's what I found so far. I have no idea how the S937 made it into the 
uidlist file. The email itself seems to be uploaded (not imap copied) 
through Outlook, which mangled the received headers in the process. 
Don't know if that is of any significance, but they are no longer in 
chronologically descending order.


Please not that in the config zlib is off for lmtp to reduce the impact 
of this issue. It was on when this mail was delivered.


I will be more than happy to provide further information.

Thanks

Tim

Am 02.02.23 um 16:23 schrieb Aki Tuomi:

On 02/02/2023 17:19 EET Stuart Henderson  wrote:

  
On 2023-02-01, Tim Evers  wrote:

I run a fairly large Dovecot Installation (around 100k mailboxes) on
several servers.

gzip compression is on.

Every once in a while I get the dreaded "cache corruption" messages in
the log:

Error: Corrupted record 

Can I encrypt already existant unencrypted mail before I start using the mail-crypt plugin?

[mailinglist-subscriptions]

Hi,

I am using dovecot 2.3.16, along with postfix and a PostgreSQL database for 
managing virtual accounts.

I'd like to start using the mail-crypt plugin. However, I'm having a bit some 
difficulty understanding the documentation at 

https://doc.dovecot.org/configuration_manual/mail_crypt_plugin 

to reach my goal. I plan to ask questions about those issues by starting new 
threads in this mailing list. But before I even come to that, I'd like to 
investigate the following:

The above documentation only addresses a clean install and doesn't seem to 
mention encrypting already existent unencrypted mails, like my server has. Is 
it possible to encrypt those before I start using the mail-crypt plugin, such 
that it will be able to decrypt those messages as well?

If it is, I am assuming that how I would go about achieving that will be very 
dependent on the ultimate configuration I have in mind (pub/priv keys, etc.). 
So I don't expect a full-fledged guide. However, if you could perhaps give a 
general overview of what would be needed to achieve this, I would very much 
appreciate that.

Thank you.

Re: Error: Corrupted index cache file

[Alessio Cecchi]

Hi Sohin,

I don't remember how was solved the problem with Ubuntu, but in my 
current setup (some hundreds of thousands of mailboxes), and in general 
with Maildir and small size files I prefer to use NFSv3 that have less 
"problem" with lock since it is stateless. Try also to remove all "ac" 
options from fstab, my fstab for NFS Maildir is very simple:


rw,nfsvers=3,noatime,nodiratime,_netdev,nordirplus

But I'm not using Linux as NFS for server. Let me know if you have any 
improvements with my mount options.


Ciao

Il 13/02/23 10:39, Sohin Vyacheslav ha scritto:



12.02.2023 21:36, Alessio Cecchi пишет:

I've run into this error in the past in some situations:

- one was during the migration from Centos 6 to 7, probably the NFS 
client in the kernel had some differences in cache management


- one was with Ubuntu and NFS server in Google Cloud, I don't 
remember exactly how I solved it in that case but the problem was the 
NFS server (maybe because the NFS server only supported version 4.1 
and there were locking issues)


- one was where I used Director but local delivery via LDA, I solved 
it by switching to delivery via LMTP


What NFS server/storage are you using? And with what NFS version your 
Maildir are mounted?

We use NFS on Ubuntu
nfs-kernel-server 1:1.3.4-2.1ubuntu5.5
nfs-common 1:1.3.4-2.1ubuntu5.5

$ nfsstat | grep nfs
Server nfs v4:

Now Maildir mounted with these options:
type nfs4 
(rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=1800,acregmax=1800,acdirmin=1800,acdirmax=1800,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=IP-address,local_lock=none,addr=ip-address)



Are you using LDA or LMTP for delivery?

Uses LMTP for delivery.


After what change did the problem start?

It seems that problem exists a long time.

Re: Error: Broken file dovecot-uidlist

[Sohin Vyacheslav]

15.02.2023 16:58, Maciej Milaszewski пишет:


Can you send me info about mounted options (fstab) and what is your 
kernel and nfs version in client and your storage


Hi Maciej,

Client=>
fstab:
IP-address:/data  /data   nfs 
auto,nofail,noatime,intr,tcp,nordirplus,actimeo=18000   0


# uname -r
4.15.0-204-generic

nfs-common-1:1.3.4-2.1ubuntu5.5

currently mounted as NFSv4.2
# mount | grep nfs
IP-address-2:/data on /data type nfs4 
(rw,noatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=1800,acregmax=1800,acdirmin=1800,acdirmax=1800,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=IP-address-1,local_lock=none,addr=IP-address-2)



NFS storage =>

# uname -r
4.15.0-158-generic

nfs-common-1:1.3.4-2.1ubuntu5.5
nfs-kernel-server-1:1.3.4-2.1ubuntu5.5

--
Best wishes
Sohin Vyacheslav

Re: Error: Broken file dovecot-uidlist

[Maciej Milaszewski]

Hi
Can you send me info about mounted options (fstab) and what is your 
kernel and nfs version in client and your storage


W dniu 15.02.2023 o 12:32, Sohin Vyacheslav pisze:


Hi All,

In mail.log are exists error messages "Error: Broken file 
/data/mail/vhosts/domain.com/u...@domain.com/Maildir/dovecot-uidlist 
line XX: Invalid data: for some email accounts.


For example,
dovecot-uidlist line 21197: Invalid data:

When I open line 21197 in editor:
# vim +21197 dovecot-uidlist
1750207 :1676454069.M698819P4439.mail-b,S=963716,W=976359

and then check this file size
# ls -l 
/data/mail/vhosts/domain.com/u...@domain.com/Maildir/cur/1676454069.M698819P4439.mail-b,S=963716,W=976359:2,S

-rw--- 1 vmail vmail 963716 Feb 15 10:41


I see that size is the same: 963176 bytes. So what exactly mentioned 
in error message "dovecot-uidlist line 21197: Invalid data:"?






OpenPGP_signature
Description: OpenPGP digital signature

RE: NFS and performances

[Marc]

> 
> I apologize in advance as you probably don't want to here this.

:) apology accepted!

> I have a replicated system and tried to use NFS to a file share server
> with dedicated gigabit links etc and my second replicated system.
> 
> I have 300+ accounts and many have 20+ gig of data over 600+ folders,
> your setup seems larger.
> 
> I spent 2 months trying to make this work reliably with nothing working
> out.
> 
> that being said (and this IS NOT a dovecot thing) NFS simply will not
> work reliably especially in the environment that you seem to be using
> 
> I went to local SDRAM drives on the second server and have had zero
> issues since.

Even a bit bigger setup would run ok on ceph. I have more or less default setup 
but have split up mailboxes on ssd and hdd, indexes on ssd. Although it is much 
slower than native performance. You have complete redundancy.

Re: NFS and performances

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Good morning

I apologize in advance as you probably don't want to here this.

I have a replicated system and tried to use NFS to a file share server 
with dedicated gigabit links etc and my second replicated system.


I have 300+ accounts and many have 20+ gig of data over 600+ folders, 
your setup seems larger.


I spent 2 months trying to make this work reliably with nothing working out.

that being said (and this IS NOT a dovecot thing) NFS simply will not 
work reliably especially in the environment that you seem to be using


I went to local SDRAM drives on the second server and have had zero 
issues since.


NFS tweaks can be done and dovecot does try to support this but Linux 
flavors (i use FreeBSD) all seem to handle NFS slightly differently thus 
leading to the issues of timeouts, data not so much being dropped but 
delayed between the NFS mount points.


NFS inherently on most systems runs a 30 second cache and file locking 
for the mailboxes can usually is an issue.


Just easier to use hdd's on any local server.

NFS is good for tar backups etc though.

Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2023-02-15 9:25 a.m., tomate aceite wrote:

Hello,  i have some question about NFS, dovecot director, and imap settings.

I was reading all dovecot documentation ad mail lists, but some aspect  
are not clear to me.


I am looking for performance / tunning my infra to work in a more 
efficient way because we experiences some issues some days ago.


This is my infra:

I got an infra with 2 dovecot-directors and 3 imap backend.
I got all the emails stored in a common NFS share filer to all the imap 
nodes. ( Index are locally stored in each imap node.)



My NFS mount options:

(0)#: nfsstat -m

/data/mail from myipaddress:/export/mail/maildirs
  Flags:

rw,nosuid,noexec,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.205,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.205



*Questions*:  ( https://wiki.dovecot.org/PerformanceTuning 
 , 
https://doc.dovecot.org/configuration_manual/nfs/ 
  >> i am following 
these steps )


1) Is my NFS correct setup with the mount options well optimized ?  Not 
sure if someone is using the same flags that me or got a better 
recomendation to used.


2) Set *mmap_disable = yes ??? * >>  This must be set to yes if you 
store indexes to shared filesystems. In my case i got them locally in 
each imap node not in NFS share folder.


I got setup  mmap_disable = no , is this correct?  I think no is the 
correct option here with indexes locally.


because i can read here:

https://wiki1.dovecot.org/NFS 


 >> High performance NFS setup with indexes on local disk (see below
for benefits):

mmap_disable = no



3) Set*mail_fsync = always  ???*

Documentation: https://wiki.dovecot.org/PerformanceTuning 



always

     Use fsync after all disk writes.

     Recommended for NFS to make sure there aren’t any delayed write()s.


3.a) where i can setup this option *mail_fsync = always , *because i 
run  doveconf -n in director,  and imap nodes, and they are not showing 
nothing.


3.b) *In which node ? *Do i need to add  the setting in dovecot.conf in 
*director node or in imap node or in both ?*  Not sure if this is the 
correct way:


This is an attemp of setup, not sure if is correct?

0)#: doveconf -n
# 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-20-amd64 x86_64 Debian 11.6

mail_debug = yes
mail_fsync = always
mail_max_userip_connections = 20
mail_nfs_storage = yes
mail_plugins = " notify mail_log"
mail_privileged_group = mail



protocol lmtp {
   mail_fsync = always
   mail_plugins = " notify mail_log sieve mail_lua push_notification
push_notification_lua"
   plugin {
    ...
   }



4) Do not set *mail_nfs_index *or *mail_nfs_storage* (i.e. keep them as 
no)   ?


First option make sense but the second one not.

https://doc.dovecot.org/settings/core/#core_setting-mail_nfs_storage 



mail_nfs_storage

         Default: no

         Values: Boolean

     Flush NFS caches whenever it is necessary to do so.


     This setting should only be enabled if you are using multiple
servers on NFS.


So should be possible to enable this option *mail_nfs_storage = yes ?*




4) I got this setting in *dovecot-sql.conf  ( director ) *

NFS and performances

[tomate aceite]

Hello,  i have some question about NFS, dovecot director, and imap settings.

I was reading all dovecot documentation ad mail lists, but some aspect  are
not clear to me.

I am looking for performance / tunning my infra to work in a more efficient
way because we experiences some issues some days ago.

This is my infra:

I got an infra with 2 dovecot-directors and 3 imap backend.
I got all the emails stored in a common NFS share filer to all the imap
nodes. ( Index are locally stored in each imap node.)


My NFS mount options:

(0)#: nfsstat -m

/data/mail from myipaddress:/export/mail/maildirs
>  Flags:
> rw,nosuid,noexec,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.205,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.205
>


*Questions*:  ( https://wiki.dovecot.org/PerformanceTuning ,
https://doc.dovecot.org/configuration_manual/nfs/  >> i am following these
steps )

1) Is my NFS correct setup with the mount options well optimized ?  Not
sure if someone is using the same flags that me or got a better
recomendation to used.

2) Set *mmap_disable = yes ???  * >>  This must be set to yes if you store
indexes to shared filesystems. In my case i got them locally in each imap
node not in NFS share folder.

I got setup  mmap_disable = no , is this correct?  I think no is the
correct option here with indexes locally.

because i can read here:

  https://wiki1.dovecot.org/NFS

>
> >> High performance NFS setup with indexes on local disk (see below for
> benefits):
>
> mmap_disable = no
>


3) Set* mail_fsync = always  ???*

Documentation: https://wiki.dovecot.org/PerformanceTuning

always
>
> Use fsync after all disk writes.
>
> Recommended for NFS to make sure there aren’t any delayed write()s.
>

3.a) where i can setup this option *mail_fsync = always , *because i run
doveconf -n in director,  and imap nodes, and they are not showing nothing.

3.b) *In which node ? *Do i need to add  the setting in dovecot.conf
in *director
node or in imap node or in both ?*  Not sure if this is the correct way:

This is an attemp of setup, not sure if is correct?

0)#: doveconf -n
> # 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.13 (cdd19fe3)
> # OS: Linux 5.10.0-20-amd64 x86_64 Debian 11.6
>
> mail_debug = yes
> mail_fsync = always
> mail_max_userip_connections = 20
> mail_nfs_storage = yes
> mail_plugins = " notify mail_log"
> mail_privileged_group = mail
>
>
>
> protocol lmtp {
>   mail_fsync = always
>   mail_plugins = " notify mail_log sieve mail_lua push_notification
> push_notification_lua"
>   plugin {
>...
>   }
>


4) Do not set *mail_nfs_index *or *mail_nfs_storage* (i.e. keep them as no)
  ?

First option make sense but the second one not.

https://doc.dovecot.org/settings/core/#core_setting-mail_nfs_storage

mail_nfs_storage
>>>
>>> Default: no
>>>
>>> Values: Boolean
>>>
>>> Flush NFS caches whenever it is necessary to do so.
>>>
>>
> This setting should only be enabled if you are using multiple servers
> on NFS.
>

So should be possible to enable this option   *mail_nfs_storage = yes ?*




4) I got this setting in *dovecot-sql.conf  ( director )  *

driver = mysql
> connect = host=myserver.X dbname=maildb user=dovecot
> password=
> default_pass_scheme = SHA
> password_query = select login as user, crypt_sha1 as password, home as
> userdb_home, uid AS userdb_uid, gid AS userdb_gid,
> concat(maildir,':INDEX=/data/indexes/',login) as userdb_mail from mailbox
> left join aliases on aliases.systemid = aliasid where login = '%u' and
> inactive = 0;
> user_query = select home, maildir as mail, uid, gid from mailbox left join
> aliases on aliases.systemid = aliasid where login = '%u' and inactive = 0;
> iterate_query = select distinct login as user from mailbox;
>


I would like to implement * ITERINDEX*  and probably *VOLATILEDIR *but not
sure if this is the correct change or if i need to change something into
the database.

from:

password_query = select login as user, crypt_sha1 as password, home as
> userdb_home, uid AS userdb_uid, gid AS userdb_gid,
> concat(maildir,':INDEX=/data/indexes/',login) as userdb_mail from mailbox
> left join aliases on aliases.systemid = aliasid where login = '%u' and
> inactive = 0;
>

to:

password_query = select login as user, crypt_sha1 as password, home as
> userdb_home, uid AS userdb_uid, gid AS userdb_gid,
> concat(maildir,':INDEX=/data/indexes/',login,':ITERINDEX',':VOLATILEDIR=/tmp/%2.256Nu/%u',login)
> as userdb_mail from mailbox left join aliases on aliases.systemid = aliasid
> where login = '%u' and inactive = 0;
>


But not sure if is working correctly,  * i checked debuging* in my imap
node in this way:

auth_verbose = yes
> auth_verbose_passwords = no
> auth_debug = yes
> auth_debug_passwords = yes
> mail_debug = yes
> verbose_ssl = yes
>

This is the output log:


Feb 15 09:32:53 

Re: Hide local IP from non delivery notifications

[Claudio Corvino]

Right! I did the setup and it works correctly!

Thanks to all

On 15/02/23 14:34, Daryl Richards wrote:

On 2023-02-15 6:41 a.m., Claudio Corvino wrote:
Ok thanks for your answer but I have another question: if I reject 
the e-mail instead of sending the non delivery notification, how can 
a "good" user be notified of the fact that his e-mail was not delivered?


Claudio


That should be the job of the sender's MTA, not you.



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Hide local IP from non delivery notifications

[Daryl Richards]

On 2023-02-15 6:41 a.m., Claudio Corvino wrote:
Ok thanks for your answer but I have another question: if I reject the 
e-mail instead of sending the non delivery notification, how can a 
"good" user be notified of the fact that his e-mail was not delivered?


Claudio


That should be the job of the sender's MTA, not you.

Re: crash when trying to upgrade from 2.0.x to 2.3.x

[Aki Tuomi]

squat has been obsoleted since 2.3.0. You should use either solr or lucene (if 
it works), there is also xapian based flatcurve plugin available for 2.3 from 
community (will be in core for 2.4).

Aki

> On 15/02/2023 14:07 EET Eugene M. Zheganin  wrote:
> 
> 
> Hello,
> 
> 
> 
> On 15.02.2023 12:48, Aki Tuomi wrote:
> 
> > Thank you for reporting this issue, which fts driver are you using? Can you 
> > provide `doveconf -n` output?
> > 
> > 
> Seems like I'm using ... squat, and this, in turn, seems to be the old legacy 
> piece from who knows how long ago installed instance. From your question - am 
> I right supposing that if I will get rid of squat things will improve ? Also 
> seems like it's worth mentioning that downgrading to dovecot 2.3.15 improved 
> things radically - no crash at this time (with the same set of configuration 
> files).
> dovecot -n output:
> ===Cut===
> # doveconf -n
>  # 2.3.15 (0503334ab1): /usr/local/etc/dovecot/dovecot.conf 
>  # Pigeonhole version 0.5.15 (e6a84e31) 
>  # OS: FreeBSD 13.1-RELEASE amd64 
>  # Hostname: it-r-support 
>  auth_debug = yes 
>  auth_debug_passwords = yes 
>  auth_mechanisms = plain login cram-md5 digest-md5 
>  auth_verbose = yes 
>  auth_verbose_passwords = yes 
>  default_client_limit = 40963 
>  default_process_limit = 10240 
>  disable_plaintext_auth = no 
>  first_valid_gid = 1143 
>  first_valid_uid = 1143 
>  hostname = alamics.ru 
>  mail_home = /var/imap/%d/%n 
>  mail_location = maildir:~/Maildir 
>  managesieve_notify_capability = mailto 
>  managesieve_sieve_capability = fileinto reject envelope encoded-character 
> vacation subaddress comparator-i;ascii-numeric
>  relational regex imap4flags copy include variables body enotify environment 
> mailbox date index ihave duplicate mime for
>  everypart extracttext 
>  mbox_write_locks = fcntl 
>  namespace { 
>  inbox = yes 
>  list = yes 
>  location = 
>  prefix = 
>  separator = / 
>  subscriptions = yes 
>  type = private 
>  } 
>  namespace { 
>  location = maildir:/var/imap/%d/public 
>  prefix = Public/ 
>  separator = / 
>  subscriptions = no 
>  type = public 
>  } 
>  namespace { 
>  list = children 
>  location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u 
>  prefix = shared/%%u/ 
>  separator = / 
>  subscriptions = no 
>  type = shared 
>  } 
>  passdb { 
>  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext 
>  driver = sql 
>  } 
>  plugin { 
>  acl = vfile 
>  acl_shared_dict = file:/var/db/dovecot/shared-mailboxes 
>  fts = squat 
>  fts_autoindex = yes 
>  fts_squat = partial=4 full=10 
>  quota = maildir:User quota 
>  quota_rule = *:storage=50G 
>  quota_rule2 = Trash:storage=+10%% 
>  quota_rule3 = Spam:storage=+20%% 
>  quota_warning = storage=95%% quota-warning 95 %u 
>  quota_warning2 = storage=80%% quota-warning 80 %u 
>  sieve = ~/.dovecot.sieve 
>  sieve_after = /var/imap/sieve/sieve-after 
>  sieve_before = /var/imap/sieve/sieve-before 
>  sieve_default = /var/imap/sieve/default.sieve 
>  sieve_dir = /var/imap/%d/%n/sieve 
>  sieve_global_dir = /var/imap/sieve 
>  sieve_max_script_size = 1M 
>  } 
>  postmaster_address = postmas...@alamics.ru 
>  protocols = imap pop3 lmtp sieve 
>  service auth { 
>  client_limit = 62464 
>  unix_listener /var/spool/postfix/private/auth { 
>  group = postfix 
>  mode = 0666 
>  user = postfix 
>  } 
>  unix_listener auth-userdb { 
>  group = dovecot 
>  mode = 0660 
>  user = dovecot 
>  } 
>  } 
>  service imap-login { 
>  client_limit = 1 
>  inet_listener imap { 
>  port = 143 
>  } 
>  inet_listener imaps { 
>  port = 993 
>  ssl = yes 
>  } 
>  process_limit = 10240 
>  process_min_avail = 1 
>  service_count = 0 
>  vsz_limit = 256 M 
>  } 
>  service imap { 
>  drop_priv_before_exec = yes 
>  process_limit = 10240 
>  } 
>  service managesieve-login { 
>  inet_listener sieve { 
>  port = 4190 
>  } 
>  } 
>  service pop3-login { 
>  inet_listener pop3 { 
>  port = 110 
>  } 
>  inet_listener pop3s { 
>  port = 995 
>  ssl = yes 
>  } 
>  } 
>  service quota-warning { 
>  executable = script /usr/local/bin/quota-warning.sh 
>  unix_listener quota-warning { 
>  user = dovecot 
>  } 
>  user = dovecot 
>  } 
>  ssl_cert =   ssl_key = # hidden, use -P to show it 
>  syslog_facility = local6 
>  userdb { 
>  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext 
>  driver = sql 
>  } 
>  protocol lmtp { 
>  mail_plugins = " quota acl sieve" 
>  } 
>  protocol lda { 
>  mail_plugins = " quota acl sieve" 
>  } 
>  protocol imap { 
>  imap_client_workarounds = tb-extra-mailbox-sep 
>  mail_max_userip_connections = 512 
>  mail_plugins = quota imap_quota fts fts_squat 
>  }
> 
> ===Cut===
> 
> 
> Thanks.
> Eugene.
>

Re: Error: Broken file dovecot-uidlist

[Aki Tuomi]

> On 15/02/2023 13:58 EET Sohin Vyacheslav  wrote:
> 
>  
> 15.02.2023 13:42, Aki Tuomi пишет:
> > Every time that happens I've seen it's been zerofilled (due to NFS), and 
> > not yet filled with actual data. Size might be right, but it just contains 
> > NULs.
> 
> Thank you, Aki for clarifying! you mean lines like this:
> 
> ICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
> ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
> 
> ?
> 
> -- 
> Best wishes
> Sohin Vyacheslav

Possibly. That's base64 for repeated 0x20.

Aki

Re: crash when trying to upgrade from 2.0.x to 2.3.x

[Eugene M. Zheganin]

Hello,

On 15.02.2023 12:48, Aki Tuomi wrote:

Thank you for reporting this issue, which fts driver are you using? Can you 
provide `doveconf -n` output?

Seems like I'm using ... squat, and this, in turn, seems to be the old 
legacy piece from who knows how long ago installed instance. From your 
question - am I right supposing that if I will get rid of squat things 
will improve ? Also seems like it's worth mentioning that downgrading to 
dovecot 2.3.15 improved things radically - no crash at this time (with 
the same set of configuration files).


dovecot -n output:

===Cut===

# doveconf -n
# 2.3.15 (0503334ab1): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.15 (e6a84e31)
# OS: FreeBSD 13.1-RELEASE amd64
# Hostname: it-r-support
auth_debug = yes
auth_debug_passwords = yes
auth_mechanisms = plain login cram-md5 digest-md5
auth_verbose = yes
auth_verbose_passwords = yes
default_client_limit = 40963
default_process_limit = 10240
disable_plaintext_auth = no
first_valid_gid = 1143
first_valid_uid = 1143
hostname = alamics.ru
mail_home = /var/imap/%d/%n
mail_location = maildir:~/Maildir
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime for

everypart extracttext
mbox_write_locks = fcntl
namespace {
 inbox = yes
 list = yes
 location =
 prefix =
 separator = /
 subscriptions = yes
 type = private
}
namespace {
 location = maildir:/var/imap/%d/public
 prefix = Public/
 separator = /
 subscriptions = no
 type = public
}
namespace {
 list = children
 location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u
 prefix = shared/%%u/
 separator = /
 subscriptions = no
 type = shared
}
passdb {
 args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
 driver = sql
}
plugin {
 acl = vfile
 acl_shared_dict = file:/var/db/dovecot/shared-mailboxes
 fts = squat
 fts_autoindex = yes
 fts_squat = partial=4 full=10
 quota = maildir:User quota
 quota_rule = *:storage=50G
 quota_rule2 = Trash:storage=+10%%
 quota_rule3 = Spam:storage=+20%%
 quota_warning = storage=95%% quota-warning 95 %u
 quota_warning2 = storage=80%% quota-warning 80 %u
 sieve = ~/.dovecot.sieve
 sieve_after = /var/imap/sieve/sieve-after
 sieve_before = /var/imap/sieve/sieve-before
 sieve_default = /var/imap/sieve/default.sieve
 sieve_dir = /var/imap/%d/%n/sieve
 sieve_global_dir = /var/imap/sieve
 sieve_max_script_size = 1M
}
postmaster_address = postmas...@alamics.ru
protocols = imap pop3 lmtp sieve
service auth {
 client_limit = 62464
 unix_listener /var/spool/postfix/private/auth {
   group = postfix
   mode = 0666
   user = postfix
 }
 unix_listener auth-userdb {
   group = dovecot
   mode = 0660
   user = dovecot
 }
}
service imap-login {
 client_limit = 1
 inet_listener imap {
   port = 143
 }
 inet_listener imaps {
   port = 993
   ssl = yes
 }
 process_limit = 10240
 process_min_avail = 1
 service_count = 0
 vsz_limit = 256 M
}
service imap {
 drop_priv_before_exec = yes
 process_limit = 10240
}
service managesieve-login {
 inet_listener sieve {
   port = 4190
 }
}
service pop3-login {
 inet_listener pop3 {
   port = 110
 }
 inet_listener pop3s {
   port = 995
   ssl = yes
 }
}
service quota-warning {
 executable = script /usr/local/bin/quota-warning.sh
 unix_listener quota-warning {
   user = dovecot
 }
 user = dovecot
}
ssl_cert = 

Re: Error: Broken file dovecot-uidlist

[Sohin Vyacheslav]

15.02.2023 13:42, Aki Tuomi пишет:

Every time that happens I've seen it's been zerofilled (due to NFS), and not 
yet filled with actual data. Size might be right, but it just contains NULs.


Thank you, Aki for clarifying! you mean lines like this:

ICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg

?

--
Best wishes
Sohin Vyacheslav

Re: Error: Broken file dovecot-uidlist

[Aki Tuomi]

> On 15/02/2023 13:32 EET Sohin Vyacheslav  wrote:
> 
>  
> Hi All,
> 
> In mail.log are exists error messages "Error: Broken file 
> /data/mail/vhosts/domain.com/u...@domain.com/Maildir/dovecot-uidlist 
> line XX: Invalid data: for some email accounts.
> 
> For example,
> dovecot-uidlist line 21197: Invalid data:
> 
> When I open line 21197 in editor:
> # vim +21197 dovecot-uidlist
> 1750207 :1676454069.M698819P4439.mail-b,S=963716,W=976359
> 
> and then check this file size
> # ls -l 
> /data/mail/vhosts/domain.com/u...@domain.com/Maildir/cur/1676454069.M698819P4439.mail-b,S=963716,W=976359:2,S
> -rw--- 1 vmail vmail 963716 Feb 15 10:41
> 
> 
> I see that size is the same: 963176 bytes. So what exactly mentioned in 
> error message "dovecot-uidlist line 21197: Invalid data:"?
> 
> -- 
> Best wishes
> Sohin Vyacheslav

Every time that happens I've seen it's been zerofilled (due to NFS), and not 
yet filled with actual data. Size might be right, but it just contains NULs.

Aki

Re: Hide local IP from non delivery notifications

[Claudio Corvino]

Ok thanks for your answer but I have another question: if I reject the 
e-mail instead of sending the non delivery notification, how can a 
"good" user be notified of the fact that his e-mail was not delivered?


Claudio

On 14/02/23 19:48, dove...@ptld.com wrote:
I have an external MTA configured with Postfix that delivers email 
to an internal IMAP/LMTP

Dovecot server configured to bind an LDAP to check if users exist.


You should have postfix do the checking for whether or not users exist
and then have postfix reject and deny the message.  Then you don't
care because the IP of the postfix server is almost certainly your MX 
server.



Plus backscatter. Sending a fail notification back to the sender (vs 
rejecting) puts your mail server at risk of sending spam to email 
accounts that had their address forged. And thus maybe yoru server 
being added to a spam blacklist.


smime.p7s
Description: S/MIME Cryptographic Signature

Error: Broken file dovecot-uidlist

[Sohin Vyacheslav]

Hi All,

In mail.log are exists error messages "Error: Broken file 
/data/mail/vhosts/domain.com/u...@domain.com/Maildir/dovecot-uidlist 
line XX: Invalid data: for some email accounts.


For example,
dovecot-uidlist line 21197: Invalid data:

When I open line 21197 in editor:
# vim +21197 dovecot-uidlist
1750207 :1676454069.M698819P4439.mail-b,S=963716,W=976359

and then check this file size
# ls -l 
/data/mail/vhosts/domain.com/u...@domain.com/Maildir/cur/1676454069.M698819P4439.mail-b,S=963716,W=976359:2,S

-rw--- 1 vmail vmail 963716 Feb 15 10:41


I see that size is the same: 963176 bytes. So what exactly mentioned in 
error message "dovecot-uidlist line 21197: Invalid data:"?


--
Best wishes
Sohin Vyacheslav

Feature Request: login_trusted_networks to take FQDN

[Sean Gallagher]

In a previous post to this list I described a problem I was having 
validating client certificates on inet_listener lmtp connections.


Subject: "Please Help: Dovecot ssl_ca selection based on remote IP 
address filtering not working."


The problem there was that Dovecot does not "inspect" the subject name 
on the client certificate on LMTP connections. As such Any valid 
certificate will pass. In this context "valid" means the same as OpenSSL 
SSL_set_verify( ,SSL_VERIFY_PEER, ). I.e. the certificate chain is well 
formed and can be traced back to a trusted root. It does not say 
anything about the peer's identity.


I propose here, that the "login_trusted_networks" setting be allowed to 
take a domain name - possibly with wildcards. Then the name on the 
client certificate could be checked against login_trusted_networks in 
much the same way that web browsers work.


If you tell your web browser that you want to connect to 
www.example.com, the browser will check that the server's certificate 
matches "www.example.com".


In the present case, if you tell Dovecot (through the 
login_trusted_networks setting) to allow connections from 
"smtp.example.com", then Dovecot could check the name on the client's 
certificate matches "smtp.example.com".


More generally, example.com could issue client certificates with names 
matching "*.mua.example.com". Then you could tell Dovecot to allow 
connections from "*.mua.example.com" through the login_trusted_networks 
setting.


These usages could largely replace the IP host and CIDR subnet usages 
currently allowed in the login_trusted_networks setting but both could 
exist side by side.


Of course, more elaborate schemes could be devised involving database 
lookups, but the outlined proposal would be relatively easy to implement 
and cover a good majority of use cases.


The alternative is to force the use of application-specific certificate 
authorities, or just ignore it and hope that no one knows how to spoof 
network traffic.


  That's My two cents...

    Sean.


--
This email has been checked for viruses by AVG antivirus software.
www.avg.com