Re: Glean all from addresses from a users mailbox?

[Paul Kudla (SCOM.CA Internet Services Inc.) via dovecot]

please find python imap script below i use to make csv files etc

it will search all folders, sub folders etc and make a file based on 
display name and email address, i had started on a dav cal format but 
csv is all that is supported at this time. it was also designed to 
import hotmail address books for conversion?


was origionally intended to import to thunderbird.

it will give you a starting point.

#!/usr/local/bin/python2
# encoding: utf-8


import sys, os, socket,traceback
import getopt,base64
import time,datetime
from datetime import date
import select
from urllib import unquote_plus

import psycopg2,imaplib
from email.header import decode_header
import imaplib2
from threading import *

from lib import *

from optparse import OptionParser

USAGE_TEXT = '''\
usage: %%prog %s[options]
'''


parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')
parser.add_option("-e", "--email", dest="email_address", help="Email 
Adress to Process")
parser.add_option("-o", "--outlook", dest="hotmail_import_from", 
help="Filespec for Hotmail Address Book")
parser.add_option("-p", "--password", dest="password", help="Email 
Account Password")
parser.add_option("-f", "--filespec", dest="file_out", default = 
'addressbook.csv', help="Output File")
parser.add_option("-v", "--vcard", dest="vcard", default = False, action 
= 'store_true', help="Output File type vCard")

options, args = parser.parse_args()


if options.email_address == None :
print 'No Email Specified ..., Exiting'
sys.exit()


if options.password == None : #Go get the password
print 'Email Account Not Found, Exiting '
sys.exit()


print 'Processing Email   : %s' %options.email_address
print 'Sending To : %s' %options.file_out
print 'Processing Hotmail Addressbook : %s' %options.hotmail_import_from


print 'Logging in to Account : %s with Password : %s\n\n' 
%(options.email_address,options.password)


# Set the following two lines to your creds and server
M = imaplib2.IMAP4("mail.local.scom.ca")
status = M.login(options.email_address,options.password)
# We need to get out of the AUTH state, so we just select
# the INBOX.
if 'OK' not in status :
print 'Bad Username or Password : %s / %s' 
%(options.uemail_address,options.password)

sys.exit()

address_book = []

#Ok import hotmail address book into system if avaliable

if options.hotmail_import_from != None : #go get the file in csv format
f = open(options.hotmail_import_from,'r')
data = f.read()
f.close

#Now start importing the data

data = data.split('\r\n')

#print data
#sys.exit()

for n in range ( 1,len(data) ) :
entry = data[n]
entry = entry.split(',')
#print
#print n,entry[0]


try :
email_address = entry[8].lower()
display_name = entry[0] + ' ' + entry[2]

print 'Importing Email  : %s' 
%email_address

print 'Importing Display Adress : %s' %display_name

#sys.exit()

b = []
b.append( email_address )
b.append( display_name )
b.append( display_name.split(' ')[0] )

try :
b.append( display_name.split(' ')[1] )
except:
b.append( display_name )

except :
pass

#sys.exit()

#Go get the folder list


for i in M.list()[1]:
mbox = (i.split('"/" ')[1])
print '\nProcessing Mbox : %s' %mbox
select_mbox = M.select(mbox)
if 'OK' not in select_mbox :
print 'Error on MBOX : %s, skipping ...' %mbox
continue
#sys.exit()

result, data = M.search(None, "ALL")
ids = data[0] # data is a list.
id_list = ids.split() # ids is a space separated string
#print id_list
#print
#print
#Ok process each email one at a time trying to look up a match


for ii in range( 0,len(id_list) ):
id = id_list[ii]
print
print 'Processing Message ID Number : %s' %str(id)
result, message_header = M.fetch(id, '(BODY.PEEK[HEADER])')
#print message_header
print
#print
message_header = str(message_header[0]).split('\\r\\n')
print 'Message Header for ID : %s:\n' %id
for n in range (0,len(message_header)) :
a = message_header[n]

if a[0:4] == 'To: ' or a[0:6] == 'From: ' :

if 'To: ' in a :
a = a.split('To: ')[1]
  

Re: Any way to make a shared mailbox(not a shared folder)

[Paul Kudla (SCOM.CA Internet Services Inc.)]

In general how i do it is just make an info@ account and login as an 
imap user


this allows multiple info@ users especially in thunderbird to be 
connected as a seconday account accross any device.


aka just treat any shared mailboxes as actual mail accounts, way 
simplier and easier to manage?


any user can then move/delete etc an email that they will be looking 
after thus updating the info@ at the same time.


its very common today to have info@ and accounting@ with multiple users 
attached to these as different people will be responsible for different 
emails


thunderbird allows drag and drop accross email accounts (only one email 
at a time), this allows easy management.






Thanks - Paul Kudla (Manager SCOM.CA Internet Services Inc.)

Have A Happy Tuesday !!!

Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2024-04-02 7:25 a.m., Maksim Rodin wrote:


Hello
I wonder if there is a right way to make a shared mailbox?
I do not mean "shared folder" but a whole mailbox.
E.g. I have a mailbox i...@company.com and I have a user mailbox 
us...@company.com.
I would like that a user which can already authenticate as us...@company.com
can setup another account in his Thunderbird as, say, 
us...@company.com\i...@company.com
or something like user1*info, enters his own password and can use the mailbox
i...@company.com as his second mailbox.
It might be something similar to master user feature but I do not want
the us...@company.com to have access to all the mailboxes on the dovecot imap 
server.


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Geofencing

[Paul Kudla (SCOM.CA Internet Services Inc.)]

thanks for the insite, being an ISP I like this kind of info even if it 
is off topic a bit on the dovecot mail lists, security today is up there 
with opertional stuff.



Have A Happy Thursday !!!

Thanks - Paul Kudla (Manager SCOM.CA Internet Services Inc.)


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2023-11-16 5:31 p.m., Jochen Bern wrote:

On 16.11.23 16:56, Paul Kudla wrote:
the ip that triggered all this says it is allocated from NL 
(Neatherlands) but physicaly exists in Hawii ?


As someone working for a LIR, let me clarify a couple things:

IPs get assigned to organizations. The registered contacts may well be 
that organization's main offices on one continent while the hardware 
actually using those addresses is located someplace different - and the 
users whose traffic gets its public IP from that hardware could well be 
in a third.


If we were also an upstream provider operating in several nations, we 
would not be obliged to use separate IP ranges for (the customers in) 
different nations, or to register such information with the RIR, much 
less making it public.


One of our customers uses the services of ZScaler to access the 
Internet, and thus a service where we maintain a whitelist of client IPs 
that may connect. Every now and then, "their" IPs will change from, 
e.g., a range assigned to "ZScaler Düsseldorf", to one designated 
"ZScaler Zürich", to "ZScaler Frankfurt", etc., while our actual 
customer doesn't move more than whatever amount the keycaps on his 
keyboard need to travel.


Having that said, there are people trying to *second guess* the actual 
location behind an IP address, from Google (ever wondered why, when you 
open Google Maps, it usually *happens* to show the place you're in?) to 
https://www.maxmind.com/en/solutions/ip-geolocation-databases-api-services to hobbyists, and there are software frameworks to make services geofenced or location aware (e.g., there are packages "GeoIP" and "plasma-workspace-geolocation" installed on my laptop apparently right off the bat). And yes, there might easily be no info for an IP you look up, or some that's plain wrong.


And *then* there are things like Anycast or BGP hijacking or VPN 
services to obscure one's origin or ...


Kind regards,

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: imap syncing issues

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok just a suggestion

Dovecot supports idle to inform clients of message updates

this works well with most clients

however

microsoft outlook & other microsoft email clients do not seem to support 
idle and have not since outlook2010


Microsoft want everyone using outlook / exchange 365 ??

I have not tried windows mail (sorry no need for using that)

I do however find that thunderbird on windows is the best client out there.

If you need to use microsoft email clients then you will need to set the 
fetch email intervals to like 5 minutes or whatever is good for you, 
most microsoft is set for 30 minute sync's


also note this is also an issue with iphone as email servers need to 
send push notifications to the apple in order to notify the end user of 
a new/changed email box. this has been the case since ios 10. again you 
need to set fetch intervals for stuff to be updated.


fyi



Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2023-06-14 12:58 p.m., dovecot-boun...@dovecot.org wrote:

The issue seems to be with the clients (an update maybe?), not the server.

do you have console/root access to dovecot and files?

On 6/14/23 11:17, Michael Grant via dovecot wrote:


I’m having synchronization issues in imap. I am accessing my mail from 
several different imap clients: K9 on Android, Windows 11 and 10 mail 
client, and Android Gmail app.  Both desktop and laptop, tablet, 
phone.  I know I have more than the usual number of imap connections...


Often when I delete a message in one place, it doesn’t get deleted in 
another.  For example, if I delete a message on K9 then open my 
laptop, it’s still there in W11 Mail.  But just now, I deleted some 
messages on my laptop and swiped down on K9 and the message 
disappeared on K9.  But K9 shows other messages which have been 
deleted in Windows 11 Mail.  I’ve not yet been able to figure out a 
pattern.  It’s annoying me.  I have to delete messages in several 
different places.  Messages not coming back, they’re just not being 
deleted in one place and that delete operation is not syncing to the 
others.


Messages seem to be being marked as read properly across devices.  
This seems to be an issue with delete only, so far as I’ve noticed.


I’ve long been using multiple imap clients, this syncing issue started 
maybe 6 to 8 months ago.


Is there a good way to get debugging info out of dovecot as to what 
the clients are doing?


Or does anyone have any advise which might help resolve this without 
resorting to me digging into the imap protocol?


Michael Grant


___
dovecot mailing list --dovecot@dovecot.org
To unsubscribe send an email todovecot-le...@dovecot.org


--
This message has been scanned for viruses and
dangerous content by *MailScanner* , and is
believed to be clean.

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Migration dovecot server with local users to dovecot with virtual mailboxes

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok i went through this and the best solution although it is a bit of work

is imapsync


imapsync --host1  --user1 --password1 \

--host2 --user2 --password2 \

--regextrans2 "s,\.,_,g"

see : https://github.com/imapsync/imapsync

note the reg parameter above deal with dot in mailboxes etc

virtual mail requires a database of user data, i made a python script to 
step through all the account to sync to the new servers.



It basically deals with the compatibility issues between servers.




Happy Friday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2023-06-09 6:51 a.m., t Seeger wrote:
Thank you for the link, unfortunately I am missing the crucial clue. I 
get the directories backed up, but I don't get them imported 
properly all mails are displayed as unseen. i tried with simple 
pasting and the doveadm import command. Is there a trick to preserve the 
flags? Maybe it is because the old dovecot server uses short username 
without the @ and the domain name and the new one uses the mail address 
to identify the user.


On Thu, Jun 8, 2023 at 3:26 PM aki.tuomi via dovecot 
mailto:dovecot@dovecot.org>> wrote:


Please see https://doc.dovecot.org/admin_manual/migrating_mailboxes/


Aki

 Original message 
From: tseeger...@gmail.com 
Date: 6/8/23 16:10 (GMT+02:00)
To: dovecot@dovecot.org 
Subject: Migration dovecot server with local users to dovecot with
virtual mailboxes

Hello,
we are using a dovecot server with NIS, we want to transfer it to a
new setup. As backend a mysql server will be used in the future.
My problem is, if I just copy the maildir or use "doveadm import"
all mails are flagged as "unseen". How can i preserve the flags?

This is how I have tried it so far.
root@mail / $ doveadm import -s -u testuser@testdomain.local
maildir:/tmp/testuser-maildir-230608/Maildir/:INDEX=MEMORY "" "All"

As a "workaround" I tried to change the flags.
root@mail / $ doveadm -v flags add -u testuser@testdomain.local
"Seen" "ALL"
This led partially to the desired success, the mails are shown as
"seen", but the folders (e.g. Roundcube) still show unread mails.

dovecot -n

# 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-23-cloud-amd64 x86_64 Debian 11.7
# Hostname: mail.testdomain.local
auth_mechanisms = plain login
mail_fsync = always
mail_gid = vmail
mail_home = /var/vmail/mailboxes/%d/%n
mail_location = maildir:~/mail:LAYOUT=fs:INDEX=MEMORY
mail_privileged_group = vmail
mail_uid = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope
encoded-character vacation subaddress comparator-i;ascii-numeric
relational regex imap4flags copy include variables body enotify
environment mailbox date index ihave duplicate mime foreverypart
extracttext imapsieve vnd.dovecot.imapsieve
mmap_disable = yes
namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     auto = subscribe
     special_use = \Drafts
   }
   mailbox Sent {
     auto = subscribe
     special_use = \Sent
   }
   mailbox Spam {
     auto = subscribe
     special_use = \Junk
   }
   mailbox Trash {
     auto = subscribe
     special_use = \Trash
   }
   prefix =
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
plugin {
   imapsieve_mailbox1_before =
file:/var/vmail/sieve/global/learn-spam.sieve
   imapsieve_mailbox1_causes = COPY
   imapsieve_mailbox1_name = Spam
   imapsieve_mailbox2_before =
file:/var/vmail/sieve/global/learn-ham.sieve
   imapsieve_mailbox2_causes = COPY
   imapsieve_mailbox2_from = Spam
   imapsieve_mailbox2_name = *
   quota = maildir:User quota
   quota_exceeded_message = User %u has exhausted allowed storage space.
   sieve =

file:/var/vmail/sieve/%d/%n/scripts;active=/var/vmail/sieve/%d/%n/active-script.sieve
   sieve_before = /var/vmail/sieve/global/spam-global.sieve
   sieve_global_extensions = +vnd.dovecot.pipe
   sieve_pipe_bin_dir = /usr/bin
   sieve_plugins = sieve_imapsieve sieve_extprograms
}
protocols = imap lmtp sieve
service auth {
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   unix_listener auth-userdb {
     group = vmail
     mode = 0660
     user = vmail
   }
}
service imap-log

Re: replicator service

[Paul Kudla (SCOM.CA Internet Services Inc.)]

replication needs to run on both servers

you need to assume that a mailbox can change on either server

replication keeps them synced in real time




Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2023-05-14 5:19 p.m., Przemysław Kwiatkowski via dovecot wrote:


I'm confused.

Replicator service is bidirectional. Does it mean I should start it only 
on *one* end of a replication pair?


Or on both, working simultaneously?


___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: creating a mailbox via imap

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok basically (please read the entire post - its techy),

You need the username/password in the database before doing this

Then make sure dovecot config carries

___

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Sent {
auto = subscribe
special_use = \Sent
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix =
  separator = /
}
___

note auto = subscribe above.

When you use (in python as per this example) the imap library calls to 
create a mailbox dovecot (like cyrus) should create the account etc


In my case I do a

CM (create mailbox) for

/INBOX
/Sent
/Trash
/Drafts

I make all of them by default, some mail clients will make these my 
default and some do not.


It's overkill but should work.

Also note this needs to be carried out on the actual mail server to
get around os system rights etc.

Please note i run a django for my admin system and ended up writing a 
"Listener" to communicate with the server over tcpip


the listener does use the create mailbox function with dovecot

I dont remember the specifics but feel free to ask if the code below has 
issues.


Listener sits in the background on a dovecot server waiting to do something.


[15:36:24] mail18.scom.ca [root:0] ~
# psx list
Displaying One Conditional ... list


 6005  -  Is   0:00.19 /usr/local/bin/python2 
/sbin/scripts/dovecot.listen (python2.7)


run in unix with the & (background command)

dovecot.listen as follows :

___
# cat /sbin/scripts/dovecot.listen
#!/usr/local/bin/python2

import os,sys
import socket
import commands
import time

from lib import *

a = onlyone ('dovecot.listen')
if a.status == 'BAD' :
print 'Another Process Is running '
sys.exit()

TCP_IP = '10.220.0.18'
TCP_PORT = 8444
BUFFER_SIZE = 1024  # Normally 1024, but we want fast response

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)

while 1 : #Process received data?
conn, addr = s.accept() #Wait for data
print 'Connection address:', addr
data = conn.recv(BUFFER_SIZE)
if not data: break
print "received data:", data
data = data.split (' ')
command = data[0]
print command

#If CM (Create Mailbox)
if command == 'CM' :
username = data[1]
print 'Creating Email Account : %s' % (username)
#Now create the mail box
#Now go make the email account
message = ''
for n in range (0,100) :
command1 = 
commands.getoutput('/usr/local/bin/doveadm mailbox create -s -u %s 
INBOX' %str(username))

print 'Command1 : %s' %command1
if 'Mailbox already exists' in command1 :
message = 'BAD'
conn.send( str(message) )
break

else :
if message == '' :
message = 'OK'
conn.send(message)
message = 'SENT'

if "User doesn't exist" in command1 :
time.sleep(2)
continue
else :
print 'Command1 : %s' %command1
message = 'SENT'
break

if message == 'SENT' : #Create the rest
command2 = 
commands.getoutput('/usr/local/bin/doveadm mailbox create -s -u %s Sent' 
%str(username))

print 'Command2 : %s' %command2
command3 = 
commands.getoutput('/usr/local/bin/doveadm mailbox create -s -u %s 
Trash' %str(username))

print 'Command3 : %s' %command3
command4 = 
commands.getoutput('/usr/local/bin/doveadm mailbox create -s -u %s 
Drafts' %str(username))

print 'Command4 : %s' %command4




if command == 'INFO' :
username = data[1]
print 'Getting Email Account Info : %s' % ( username )
command1 = commands.getoutput("/usr/local/bin/doveadm 
mailbox status -t all -u %s '*' " %str(username))

if 'Error' in command1 :
message = 'BAD'
else :
message = 'OK : ' + command1

print message
conn.send( str(message) )  # echo


if command == 'DM' :
data = data[1]
data = data.split('@')

Re: Redundant Database, Pgsql ?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

yes that seems to be the approach

i setup a dns entry and pointed to 3 servers

it does work round robin (ie from main, secondary etc) but that is ok

at least it is working when i take the main server offline for maintenance !




Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2023-02-19 12:56 p.m., Benny Pedersen wrote:


Paul Kudla skrev den 2023-02-19 16:01:

May I please get some guidance on what to add to talk to another
postgresql server (i have 6 replicated servers so would probably want
a couple worst case issue)


change host=localhost to host=some-other-hostname-with-multiple-ips :)

then dovecot with timeout and test next server ip

there might be more to it, but i think this is how to do it

Re: NFS and performances

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Good morning

I apologize in advance as you probably don't want to here this.

I have a replicated system and tried to use NFS to a file share server 
with dedicated gigabit links etc and my second replicated system.


I have 300+ accounts and many have 20+ gig of data over 600+ folders, 
your setup seems larger.


I spent 2 months trying to make this work reliably with nothing working out.

that being said (and this IS NOT a dovecot thing) NFS simply will not 
work reliably especially in the environment that you seem to be using


I went to local SDRAM drives on the second server and have had zero 
issues since.


NFS tweaks can be done and dovecot does try to support this but Linux 
flavors (i use FreeBSD) all seem to handle NFS slightly differently thus 
leading to the issues of timeouts, data not so much being dropped but 
delayed between the NFS mount points.


NFS inherently on most systems runs a 30 second cache and file locking 
for the mailboxes can usually is an issue.


Just easier to use hdd's on any local server.

NFS is good for tar backups etc though.

Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2023-02-15 9:25 a.m., tomate aceite wrote:

Hello,  i have some question about NFS, dovecot director, and imap settings.

I was reading all dovecot documentation ad mail lists, but some aspect  
are not clear to me.


I am looking for performance / tunning my infra to work in a more 
efficient way because we experiences some issues some days ago.


This is my infra:

I got an infra with 2 dovecot-directors and 3 imap backend.
I got all the emails stored in a common NFS share filer to all the imap 
nodes. ( Index are locally stored in each imap node.)



My NFS mount options:

(0)#: nfsstat -m

/data/mail from myipaddress:/export/mail/maildirs
  Flags:

rw,nosuid,noexec,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nordirplus,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.205,mountvers=3,mountport=20048,mountproto=tcp,local_lock=none,addr=10.0.0.205



*Questions*:  ( https://wiki.dovecot.org/PerformanceTuning 
 , 
https://doc.dovecot.org/configuration_manual/nfs/ 
  >> i am following 
these steps )


1) Is my NFS correct setup with the mount options well optimized ?  Not 
sure if someone is using the same flags that me or got a better 
recomendation to used.


2) Set *mmap_disable = yes ??? * >>  This must be set to yes if you 
store indexes to shared filesystems. In my case i got them locally in 
each imap node not in NFS share folder.


I got setup  mmap_disable = no , is this correct?  I think no is the 
correct option here with indexes locally.


because i can read here:

https://wiki1.dovecot.org/NFS 


 >> High performance NFS setup with indexes on local disk (see below
for benefits):

mmap_disable = no



3) Set*mail_fsync = always  ???*

Documentation: https://wiki.dovecot.org/PerformanceTuning 



always

     Use fsync after all disk writes.

     Recommended for NFS to make sure there aren’t any delayed write()s.


3.a) where i can setup this option *mail_fsync = always , *because i 
run  doveconf -n in director,  and imap nodes, and they are not showing 
nothing.


3.b) *In which node ? *Do i need to add  the setting in dovecot.conf in 
*director node or in imap node or in both ?*  Not sure if this is the 
correct way:


This is an attemp of setup, not sure if is correct?

0)#: doveconf -n
# 2.3.13 (89f716dc2): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-20-amd64 x86_64 Debian 11.6

mail_debug = yes
mail_fsync = always
mail_max_userip_connections = 20
mail_nfs_storage = yes
mail_plugins = " notify mail_log"
mail_privileged_group = mail



protocol lmtp {
   mail_fsync = always
   mail_plugins = " notify mail_log sieve mail_lua push_notification
push_notification_lua"
   plugin {
    ...
   }



4) Do not set *mail_nfs_index *or *mail_nfs_storage* (i.e. keep them as 
no)   ?


First option make sense but the second one not.

https://doc.dovecot.org/settings/core/#core_setting-mail_nfs_storage 



mail_nfs_storage

         Default: no

         Values: Boolean

     Flush NFS caches whenever it is necessary to do so.


     This setting should only be enabled if you are using multiple
servers on NFS.


So should be possible to enable this option *mail_nfs_storage = yes ?*




4) I got this setting in *dovecot-sql.conf  ( director ) *

Re: OT: Two simultaneous POP3 connections from Outlook

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Good morning
this is probably not want you are looking for

Couple of issues

even if you get around the outlook locking issue (unless you are leaving 
on the server but would be flagged that way anyways) pop3 is first come 
first serve.


Outlook when it comes to imap & pop3 its been my experience that 
microsoft just does not support these properly anymore as in favor for 
the exchange protocols.


Outlook 2010 was the last version that kinda worked.

Also Outlook does not support idle for imap making outlook for outlook 
and other clients (like thunderbird) more practical.


for example in the case of imap outlook on large email boxes will just 
cycle forever on a folder sync.


Completly useless.

Just some experience to share with you.


Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 2022-12-20 9:53 p.m., dovecot-boun...@dovecot.org wrote:


Hello,

I recognize this may be somewhat off-topic, but my search-fu has failed to 
provide useful results, so I ask for pointers to further information.

Today, while examining Dovecot logs, I found a case where the Outlook mail 
client (probably 2016 or later) on a user's computer opened two simultaneous, 
or at least overlapping, POP3 connections to Dovecot.  This did not go well, 
naturally.

Warning: Transaction log file .../mail/.imap/INBOX/dovecot.index.log was locked 
for 156 seconds (rotating while syncing)

If anyone else has found useful information about this peculiar behavior for an 
email client, I would be pleased to receive a pointer in that direction.

Thank you.

Ken

Re: SNI Config

[Paul Kudla (SCOM.CA Internet Services Inc.)]

much appreciated for the response

maybe a feature down the road??





Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 10/12/2022 8:12 AM, Aki Tuomi wrote:


Hi!

The pipe syntax has never worked, no idea why you think it would have. 
Unfortunately at the moment, files are your best option. I do understand the 
annoyance.

Aki


On 12/10/2022 13:54 EEST Paul Kudla (SCOM.CA Internet Services Inc.) 
 wrote:

  
ok thanks for your input


I finally tracked down the issue

It was how i was loading the certificates in the first place

that being said (and i must have missed this) 2.3.18 seems to allow
importing a cert from a program

thus sni config

local_name mail.paulkudla.net {
ssl_key =/programs/common/getssl.cert -k mail.paulkudla.net -q yes
ssl_cert =/programs/common/getssl.cert -r mail.paulkudla.net -q yes
ssl_ca =/programs/common/getssl.cert -i mail.paulkudla.net -q yes
}

would work instead of file pipes from individual text files.


#local_name mail.paulkudla.net {
#  ssl_key =http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 10/11/2022 12:46 PM, Jochen Bern wrote:


On 11.10.22 17:46, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:

ok according to
https://www.openssl.org/docs/man1.0.2/man5/x509v3_config.html
SAN is not a valid option along with CN


... I don't see that being said in the page you refer to?

Anyhow, "stop giving a CN, use SANs instead" is a rather recent
development coming from the CA/Browser Forum - and IIUC still not a
*requirement*, not even for web browsers/servers. I would be surprised
if OpenSSL (already) were trying to enforce that policy.

Hmmm, what's our company's "IMAPS server" throwing at my TB again ... ?


$ openssl s_client -connect outlook.office365.com:993 -showcerts |
openssl x509 -noout -text

[...]

     Subject: C = US, ST = Washington, L = Redmond, O = Microsoft
Corporation, CN = outlook.com

[...]

     X509v3 Subject Alternative Name:
DNS:*.clo.footprintdns.com, DNS:*.hotmail.com,
DNS:*.internal.outlook.com, [...]


... yeah, no, nothing that Thunderbird (from 69-ish to 102) should get
indigestion over.


Upoin further testing thunderbird seems to be locking onto the primary
domain (*.scom.ca) of the server skipp any sni setup ??


You might want to get a network trace of your Thunderbird talking to the
server to see what cert actually is presented by the server, and
ideally, what domain is requested by SNI (if at all). That all happens
before the connection starts to be encrypted, so you should be able to
read it (say, with Wireshark) without having to crack any crypto ...

Kind regards,

Re: SNI Config

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok thanks for your input

I finally tracked down the issue

It was how i was loading the certificates in the first place

that being said (and i must have missed this) 2.3.18 seems to allow 
importing a cert from a program


thus sni config

local_name mail.paulkudla.net {
  ssl_key =/programs/common/getssl.cert -k mail.paulkudla.net -q yes
  ssl_cert =/programs/common/getssl.cert -r mail.paulkudla.net -q yes
  ssl_ca =/programs/common/getssl.cert -i mail.paulkudla.net -q yes
}

would work instead of file pipes from individual text files.


#local_name mail.paulkudla.net {
#  ssl_key =I am sure you can appreciate generating files for 1000+ ssl certs can 
become a nightmare management wise


either that or a pgsql select ?

I have gone back to text files in the mean time ?



Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 10/11/2022 12:46 PM, Jochen Bern wrote:


On 11.10.22 17:46, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:

ok according to
https://www.openssl.org/docs/man1.0.2/man5/x509v3_config.html
SAN is not a valid option along with CN


... I don't see that being said in the page you refer to?

Anyhow, "stop giving a CN, use SANs instead" is a rather recent 
development coming from the CA/Browser Forum - and IIUC still not a 
*requirement*, not even for web browsers/servers. I would be surprised 
if OpenSSL (already) were trying to enforce that policy.


Hmmm, what's our company's "IMAPS server" throwing at my TB again ... ?

$ openssl s_client -connect outlook.office365.com:993 -showcerts | 
openssl x509 -noout -text

[...]
    Subject: C = US, ST = Washington, L = Redmond, O = Microsoft 
Corporation, CN = outlook.com

[...]
    X509v3 Subject Alternative Name: 
DNS:*.clo.footprintdns.com, DNS:*.hotmail.com, 
DNS:*.internal.outlook.com, [...]


... yeah, no, nothing that Thunderbird (from 69-ish to 102) should get 
indigestion over.


Upoin further testing thunderbird seems to be locking onto the primary 
domain (*.scom.ca) of the server skipp any sni setup ??


You might want to get a network trace of your Thunderbird talking to the 
server to see what cert actually is presented by the server, and 
ideally, what domain is requested by SNI (if at all). That all happens 
before the connection starts to be encrypted, so you should be able to 
read it (say, with Wireshark) without having to crack any crypto ...


Kind regards,

Re: Thunderbird can't connect to Dovecot (bad certificate: SSL alert number 42) - sni

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok according to

https://www.openssl.org/docs/man1.0.2/man5/x509v3_config.html

SAN is not a valid option along with CN

CN is part of the subject ??

Upoin further testing thunderbird seems to be locking onto the primary 
domain (*.scom.ca) of the server skipp any sni setup ??


again thoughts 




Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 10/11/2022 9:17 AM, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:



ok it appears that all this revolves around openssl

does anyone have explicit instructions on how to generate a proper ssl

key, csr etc file

with the proper SAN & CN etc

i tried

# openssl req -new -nodes -newkey rsa:2048 -config ./openssl.cnf 
-reqexts req_ext -keyout mail.paulkudla.net.key -out mail.paulkudla.net.csr

Error Loading request extension section req_ext

34371092480:error:22075075:X509 V3 
routines:v2i_GENERAL_NAME_ex:unsupported 
option:/usr/src/crypto/openssl/crypto/x509v3/v3_alt.c:534:name=SAN.1


34371092480:error:22098080:X509 V3 routines:X509V3_EXT_nconf:error in 
extension:/usr/src/crypto/openssl/crypto/x509v3/v3_conf.c:47:name=subjectAltName, value=@alt_names


and got the errors above

there not seem to be much on the web about how to generate these certs??



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 10/11/2022 7:47 AM, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:



Good morning to all

i guess things have changed yet again

to keep this simple :

i buy a certificate (example) : mail.paulkudla.net

i generated the key / csr as per normal using

data = '/usr/local/bin/openssl req -new -key /tmp/temp.key -out 
/tmp/temp.csr -subj "/C=%s/ST=%s/L=%s/O=%s/CN=%s"' 
%(country,state,location,organization,self.domain)


please note the above is done in django

(yes i am running thunderbird v102)

i go buy the certificate

i database the CRT & CA

CSR is :

-BEGIN CERTIFICATE REQUEST-
MIICpzCCAY8CAQAwYjELMAkGA1UEBhMCQ0ExEDAOBgNVBAgMB09udGFyaW8xDzAN
BgNVBAcMBldoaXRieTETMBEGA1UECgwKUGF1bCBLdWRsYTEbMBkGA1UEAwwSbWFp
bC5wYXVsa3VkbGEubmV0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
mSWAdwbxwjkjALQa4UdgOBHcFJDA5XkGI/8SswotYMnzjRAAE4S88vUTO3ltMasY
rprEvWEiEzUrRon3hh1ZZguV775fNCbyKUGKwGLKPDpmKxYCsE4gi2z7LKY13wSv
lLE8++Hqvt3cmZZ+wxWP/hy6LcS/6PvUPgN7S+cEC5TNLQ6VRZdpSGolRCrN9hsN
15GWYEQ/zcLW2PeCWav9DOr6NHBRE+fruDy3jFT0TkHWf3H+GKB0/RZ0agMJcEGc
ZLdJ1LkvNAn6gslppm3otZyu7XTvY9qZXcYOlMN0KL3a3488OwXTwWJHEN58eCMc
juax1f7ad8Z/+Pi+OFwfWQIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBAFgL24yi
WPat73tg1fANvutWXa2WEXeegqOawqvsV74lcyqMes8yhxiz/niOAt3oOLmViRF4
VlorgUwL0eAxtNeY4lgURW6XM5oz8TBINnPPohSAuDL9azLV1U1+M/vAvLs+LRd9
7wfVCN5bov7y735u2w38GAjmXJCBdoc+glUa+eGd5WH2+r/QQW/lRqVTDq+arqNk
9DTZc73gDCDmV45vTtbrlLnOxtmpqaQKsoFCCJW8OWaaDXfc8I+TdClVsThsbrWu
iz1/QClBPbKvfufNb+asTQSCDeJFc2EynDSE1yeYzliMLo+77ZoMqJPvI9IJCuj5
yq88NESoIYaO6Do=
-END CERTIFICATE REQUEST-

CRT is :

-BEGIN CERTIFICATE-
MIIGRTCCBS2gAwIBAgIRAKTmHoDG9LF3heBvAT8gZkYwDQYJKoZIhvcNAQELBQAw
gY8xCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE3MDUGA1UE
AxMuU2VjdGlnbyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBD
QTAeFw0yMjA2MTYwMDAwMDBaFw0yMzA2MTYyMzU5NTlaMB0xGzAZBgNVBAMTEm1h
aWwucGF1bGt1ZGxhLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJklgHcG8cI5IwC0GuFHYDgR3BSQwOV5BiP/ErMKLWDJ840QABOEvPL1Ezt5bTGr
GK6axL1hIhM1K0aJ94YdWWYLle++XzQm8ilBisBiyjw6ZisWArBOIIts+yymNd8E
r5SxPPvh6r7d3JmWfsMVj/4cui3Ev+j71D4De0vnBAuUzS0OlUWXaUhqJUQqzfYb
DdeRlmBEP83C1tj3glmr/Qzq+jRwURPn67g8t4xU9E5B1n9x/higdP0WdGoDCXBB
nGS3SdS5LzQJ+oLJaaZt6LWcru1072PamV3GDpTDdCi92t+PPDsF08FiRxDefHgj
HI7msdX+2nfGf/j4vjhcH1kCAwEAAaOCAwswggMHMB8GA1UdIwQYMBaAFI2MXsRU
rYrhd+mb+ZsF4bgBjWHhMB0GA1UdDgQWBBROA5NFqfrlHGbkp9v1JBxZe0fZsDAO
BgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwSQYDVR0gBEIwQDA0BgsrBgEEAbIxAQICBzAlMCMGCCsGAQUF
BwIBFhdodHRwczovL3NlY3RpZ28uY29tL0NQUzAIBgZngQwBAgEwgYQGCCsGAQUF
BwEBBHgwdjBPBggrBgEFBQcwAoZDaHR0cDovL2NydC5zZWN0aWdvLmNvbS9TZWN0
aWdvUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNlcnZlckNBLmNydDAjBggrBgEF
BQcwAYYXaHR0cDovL29jc3Auc2VjdGlnby5jb20wNQYDVR0RBC4wLIISbWFpbC5w
YXVsa3VkbGEubmV0ghZ3d3cubWFpbC5wYXVsa3VkbGEubmV0MIIBfQYKKwYBBAHW
eQIEAgSCAW0EggFpAWcAdgCt9776fP8QyIudPZwePhhqtGcpXc+xDCTKhYY069yC
igAAAYFsxJHxAAAEAwBHMEUCIQDxa9L+JaMJJImKuYPmfCAwJOiGXwECgtruOegv
vPqGpwIgWW8B0SWqVNPEFBveoBlIZF3jjj4nQIzYi2LnLizoVDMAdQB6MoxU2Lct
tiDqOOBSHumEFnAyE4VNO9IrwTpXo1LrUgAAAYFsxJHJAAAEAwBGMEQCIDIgNptW
Qum0KFyemHNTTfonlq4FvWTgzR1AGUnOgotPAiAAiwyN9MjZNiP76P3fel6BqEqj
jwnSVleJR1DgLIoyPQB2AOg+0No+9QY1MudXKLyJa8kD08vREWvs62nhd31

Re: Thunderbird can't connect to Dovecot (bad certificate: SSL alert number 42) - sni

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok it appears that all this revolves around openssl

does anyone have explicit instructions on how to generate a proper ssl

key, csr etc file

with the proper SAN & CN etc

i tried

# openssl req -new -nodes -newkey rsa:2048 -config ./openssl.cnf 
-reqexts req_ext -keyout mail.paulkudla.net.key -out mail.paulkudla.net.csr

Error Loading request extension section req_ext

34371092480:error:22075075:X509 V3 
routines:v2i_GENERAL_NAME_ex:unsupported 
option:/usr/src/crypto/openssl/crypto/x509v3/v3_alt.c:534:name=SAN.1


34371092480:error:22098080:X509 V3 routines:X509V3_EXT_nconf:error in 
extension:/usr/src/crypto/openssl/crypto/x509v3/v3_conf.c:47:name=subjectAltName, 
value=@alt_names


and got the errors above

there not seem to be much on the web about how to generate these certs??



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 10/11/2022 7:47 AM, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:



Good morning to all

i guess things have changed yet again

to keep this simple :

i buy a certificate (example) : mail.paulkudla.net

i generated the key / csr as per normal using

data = '/usr/local/bin/openssl req -new -key /tmp/temp.key -out 
/tmp/temp.csr -subj "/C=%s/ST=%s/L=%s/O=%s/CN=%s"' 
%(country,state,location,organization,self.domain)


please note the above is done in django

(yes i am running thunderbird v102)

i go buy the certificate

i database the CRT & CA

CSR is :

-BEGIN CERTIFICATE REQUEST-
MIICpzCCAY8CAQAwYjELMAkGA1UEBhMCQ0ExEDAOBgNVBAgMB09udGFyaW8xDzAN
BgNVBAcMBldoaXRieTETMBEGA1UECgwKUGF1bCBLdWRsYTEbMBkGA1UEAwwSbWFp
bC5wYXVsa3VkbGEubmV0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
mSWAdwbxwjkjALQa4UdgOBHcFJDA5XkGI/8SswotYMnzjRAAE4S88vUTO3ltMasY
rprEvWEiEzUrRon3hh1ZZguV775fNCbyKUGKwGLKPDpmKxYCsE4gi2z7LKY13wSv
lLE8++Hqvt3cmZZ+wxWP/hy6LcS/6PvUPgN7S+cEC5TNLQ6VRZdpSGolRCrN9hsN
15GWYEQ/zcLW2PeCWav9DOr6NHBRE+fruDy3jFT0TkHWf3H+GKB0/RZ0agMJcEGc
ZLdJ1LkvNAn6gslppm3otZyu7XTvY9qZXcYOlMN0KL3a3488OwXTwWJHEN58eCMc
juax1f7ad8Z/+Pi+OFwfWQIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBAFgL24yi
WPat73tg1fANvutWXa2WEXeegqOawqvsV74lcyqMes8yhxiz/niOAt3oOLmViRF4
VlorgUwL0eAxtNeY4lgURW6XM5oz8TBINnPPohSAuDL9azLV1U1+M/vAvLs+LRd9
7wfVCN5bov7y735u2w38GAjmXJCBdoc+glUa+eGd5WH2+r/QQW/lRqVTDq+arqNk
9DTZc73gDCDmV45vTtbrlLnOxtmpqaQKsoFCCJW8OWaaDXfc8I+TdClVsThsbrWu
iz1/QClBPbKvfufNb+asTQSCDeJFc2EynDSE1yeYzliMLo+77ZoMqJPvI9IJCuj5
yq88NESoIYaO6Do=
-END CERTIFICATE REQUEST-

CRT is :

-BEGIN CERTIFICATE-
MIIGRTCCBS2gAwIBAgIRAKTmHoDG9LF3heBvAT8gZkYwDQYJKoZIhvcNAQELBQAw
gY8xCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE3MDUGA1UE
AxMuU2VjdGlnbyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBD
QTAeFw0yMjA2MTYwMDAwMDBaFw0yMzA2MTYyMzU5NTlaMB0xGzAZBgNVBAMTEm1h
aWwucGF1bGt1ZGxhLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJklgHcG8cI5IwC0GuFHYDgR3BSQwOV5BiP/ErMKLWDJ840QABOEvPL1Ezt5bTGr
GK6axL1hIhM1K0aJ94YdWWYLle++XzQm8ilBisBiyjw6ZisWArBOIIts+yymNd8E
r5SxPPvh6r7d3JmWfsMVj/4cui3Ev+j71D4De0vnBAuUzS0OlUWXaUhqJUQqzfYb
DdeRlmBEP83C1tj3glmr/Qzq+jRwURPn67g8t4xU9E5B1n9x/higdP0WdGoDCXBB
nGS3SdS5LzQJ+oLJaaZt6LWcru1072PamV3GDpTDdCi92t+PPDsF08FiRxDefHgj
HI7msdX+2nfGf/j4vjhcH1kCAwEAAaOCAwswggMHMB8GA1UdIwQYMBaAFI2MXsRU
rYrhd+mb+ZsF4bgBjWHhMB0GA1UdDgQWBBROA5NFqfrlHGbkp9v1JBxZe0fZsDAO
BgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwSQYDVR0gBEIwQDA0BgsrBgEEAbIxAQICBzAlMCMGCCsGAQUF
BwIBFhdodHRwczovL3NlY3RpZ28uY29tL0NQUzAIBgZngQwBAgEwgYQGCCsGAQUF
BwEBBHgwdjBPBggrBgEFBQcwAoZDaHR0cDovL2NydC5zZWN0aWdvLmNvbS9TZWN0
aWdvUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNlcnZlckNBLmNydDAjBggrBgEF
BQcwAYYXaHR0cDovL29jc3Auc2VjdGlnby5jb20wNQYDVR0RBC4wLIISbWFpbC5w
YXVsa3VkbGEubmV0ghZ3d3cubWFpbC5wYXVsa3VkbGEubmV0MIIBfQYKKwYBBAHW
eQIEAgSCAW0EggFpAWcAdgCt9776fP8QyIudPZwePhhqtGcpXc+xDCTKhYY069yC
igAAAYFsxJHxAAAEAwBHMEUCIQDxa9L+JaMJJImKuYPmfCAwJOiGXwECgtruOegv
vPqGpwIgWW8B0SWqVNPEFBveoBlIZF3jjj4nQIzYi2LnLizoVDMAdQB6MoxU2Lct
tiDqOOBSHumEFnAyE4VNO9IrwTpXo1LrUgAAAYFsxJHJAAAEAwBGMEQCIDIgNptW
Qum0KFyemHNTTfonlq4FvWTgzR1AGUnOgotPAiAAiwyN9MjZNiP76P3fel6BqEqj
jwnSVleJR1DgLIoyPQB2AOg+0No+9QY1MudXKLyJa8kD08vREWvs62nhd31tBr1u
AAABgWzEkYoAAAQDAEcwRQIgOYjevKp5RI+c0JhIi6JflaxiNokRTSeXN6LrdIVt
Cf8CIQCG+aLreYVV8xCPV0skr0ats5zMf5PLPN2y8EIxGPPNVTANBgkqhkiG9w0B
AQsFAAOCAQEAJX544qDTgkGGLUOher7tH7yUgEhQFYkBDAirO37MXrhtuzH6pGSp
XfYVNB9e2ydprfmLDh8O8oTaXpaQfp/jwK3U0GfvG57MfdQTLOunpWnCjaMUPUcv
jPU90/mXc5oWlO5iJ6jPDkS/x47K03P6vftSr7AMwnLq4kYwuG9fHLslMHhoojen
9S2G1QjKVp5jkFecmQib+JOZV9Ub9r6iumHICfdcSO+tyBL2IDqWDQhuAVUXgyOV
11O9ZgikoeRhgsMhwiQA1z/Fs6Xqx/XCs6nUciebRiQuuHYm/PUG2H+tg0sLhJ6L
ntIEhjjkumL0oJEfDidP/8wmrsPuwfSDCQ==
-END CERTIFICATE-

CA (INTER) :

-BEGIN CERTIFICATE-
MIIGEzCCA/ugAwIBAgIQfVtRJrR2uh

Thunderbird can't connect to Dovecot (bad certificate: SSL alert number 42) - sni

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Good morning to all

i guess things have changed yet again

to keep this simple :

i buy a certificate (example) : mail.paulkudla.net

i generated the key / csr as per normal using

data = '/usr/local/bin/openssl req -new -key /tmp/temp.key -out 
/tmp/temp.csr -subj "/C=%s/ST=%s/L=%s/O=%s/CN=%s"' 
%(country,state,location,organization,self.domain)


please note the above is done in django

(yes i am running thunderbird v102)

i go buy the certificate

i database the CRT & CA

CSR is :

-BEGIN CERTIFICATE REQUEST-
MIICpzCCAY8CAQAwYjELMAkGA1UEBhMCQ0ExEDAOBgNVBAgMB09udGFyaW8xDzAN
BgNVBAcMBldoaXRieTETMBEGA1UECgwKUGF1bCBLdWRsYTEbMBkGA1UEAwwSbWFp
bC5wYXVsa3VkbGEubmV0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
mSWAdwbxwjkjALQa4UdgOBHcFJDA5XkGI/8SswotYMnzjRAAE4S88vUTO3ltMasY
rprEvWEiEzUrRon3hh1ZZguV775fNCbyKUGKwGLKPDpmKxYCsE4gi2z7LKY13wSv
lLE8++Hqvt3cmZZ+wxWP/hy6LcS/6PvUPgN7S+cEC5TNLQ6VRZdpSGolRCrN9hsN
15GWYEQ/zcLW2PeCWav9DOr6NHBRE+fruDy3jFT0TkHWf3H+GKB0/RZ0agMJcEGc
ZLdJ1LkvNAn6gslppm3otZyu7XTvY9qZXcYOlMN0KL3a3488OwXTwWJHEN58eCMc
juax1f7ad8Z/+Pi+OFwfWQIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBAFgL24yi
WPat73tg1fANvutWXa2WEXeegqOawqvsV74lcyqMes8yhxiz/niOAt3oOLmViRF4
VlorgUwL0eAxtNeY4lgURW6XM5oz8TBINnPPohSAuDL9azLV1U1+M/vAvLs+LRd9
7wfVCN5bov7y735u2w38GAjmXJCBdoc+glUa+eGd5WH2+r/QQW/lRqVTDq+arqNk
9DTZc73gDCDmV45vTtbrlLnOxtmpqaQKsoFCCJW8OWaaDXfc8I+TdClVsThsbrWu
iz1/QClBPbKvfufNb+asTQSCDeJFc2EynDSE1yeYzliMLo+77ZoMqJPvI9IJCuj5
yq88NESoIYaO6Do=
-END CERTIFICATE REQUEST-

CRT is :

-BEGIN CERTIFICATE-
MIIGRTCCBS2gAwIBAgIRAKTmHoDG9LF3heBvAT8gZkYwDQYJKoZIhvcNAQELBQAw
gY8xCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0ZXIxEDAO
BgNVBAcTB1NhbGZvcmQxGDAWBgNVBAoTD1NlY3RpZ28gTGltaXRlZDE3MDUGA1UE
AxMuU2VjdGlnbyBSU0EgRG9tYWluIFZhbGlkYXRpb24gU2VjdXJlIFNlcnZlciBD
QTAeFw0yMjA2MTYwMDAwMDBaFw0yMzA2MTYyMzU5NTlaMB0xGzAZBgNVBAMTEm1h
aWwucGF1bGt1ZGxhLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AJklgHcG8cI5IwC0GuFHYDgR3BSQwOV5BiP/ErMKLWDJ840QABOEvPL1Ezt5bTGr
GK6axL1hIhM1K0aJ94YdWWYLle++XzQm8ilBisBiyjw6ZisWArBOIIts+yymNd8E
r5SxPPvh6r7d3JmWfsMVj/4cui3Ev+j71D4De0vnBAuUzS0OlUWXaUhqJUQqzfYb
DdeRlmBEP83C1tj3glmr/Qzq+jRwURPn67g8t4xU9E5B1n9x/higdP0WdGoDCXBB
nGS3SdS5LzQJ+oLJaaZt6LWcru1072PamV3GDpTDdCi92t+PPDsF08FiRxDefHgj
HI7msdX+2nfGf/j4vjhcH1kCAwEAAaOCAwswggMHMB8GA1UdIwQYMBaAFI2MXsRU
rYrhd+mb+ZsF4bgBjWHhMB0GA1UdDgQWBBROA5NFqfrlHGbkp9v1JBxZe0fZsDAO
BgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwSQYDVR0gBEIwQDA0BgsrBgEEAbIxAQICBzAlMCMGCCsGAQUF
BwIBFhdodHRwczovL3NlY3RpZ28uY29tL0NQUzAIBgZngQwBAgEwgYQGCCsGAQUF
BwEBBHgwdjBPBggrBgEFBQcwAoZDaHR0cDovL2NydC5zZWN0aWdvLmNvbS9TZWN0
aWdvUlNBRG9tYWluVmFsaWRhdGlvblNlY3VyZVNlcnZlckNBLmNydDAjBggrBgEF
BQcwAYYXaHR0cDovL29jc3Auc2VjdGlnby5jb20wNQYDVR0RBC4wLIISbWFpbC5w
YXVsa3VkbGEubmV0ghZ3d3cubWFpbC5wYXVsa3VkbGEubmV0MIIBfQYKKwYBBAHW
eQIEAgSCAW0EggFpAWcAdgCt9776fP8QyIudPZwePhhqtGcpXc+xDCTKhYY069yC
igAAAYFsxJHxAAAEAwBHMEUCIQDxa9L+JaMJJImKuYPmfCAwJOiGXwECgtruOegv
vPqGpwIgWW8B0SWqVNPEFBveoBlIZF3jjj4nQIzYi2LnLizoVDMAdQB6MoxU2Lct
tiDqOOBSHumEFnAyE4VNO9IrwTpXo1LrUgAAAYFsxJHJAAAEAwBGMEQCIDIgNptW
Qum0KFyemHNTTfonlq4FvWTgzR1AGUnOgotPAiAAiwyN9MjZNiP76P3fel6BqEqj
jwnSVleJR1DgLIoyPQB2AOg+0No+9QY1MudXKLyJa8kD08vREWvs62nhd31tBr1u
AAABgWzEkYoAAAQDAEcwRQIgOYjevKp5RI+c0JhIi6JflaxiNokRTSeXN6LrdIVt
Cf8CIQCG+aLreYVV8xCPV0skr0ats5zMf5PLPN2y8EIxGPPNVTANBgkqhkiG9w0B
AQsFAAOCAQEAJX544qDTgkGGLUOher7tH7yUgEhQFYkBDAirO37MXrhtuzH6pGSp
XfYVNB9e2ydprfmLDh8O8oTaXpaQfp/jwK3U0GfvG57MfdQTLOunpWnCjaMUPUcv
jPU90/mXc5oWlO5iJ6jPDkS/x47K03P6vftSr7AMwnLq4kYwuG9fHLslMHhoojen
9S2G1QjKVp5jkFecmQib+JOZV9Ub9r6iumHICfdcSO+tyBL2IDqWDQhuAVUXgyOV
11O9ZgikoeRhgsMhwiQA1z/Fs6Xqx/XCs6nUciebRiQuuHYm/PUG2H+tg0sLhJ6L
ntIEhjjkumL0oJEfDidP/8wmrsPuwfSDCQ==
-END CERTIFICATE-

CA (INTER) :

-BEGIN CERTIFICATE-
MIIGEzCCA/ugAwIBAgIQfVtRJrR2uhHbdBYLvFMNpzANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTgx
MTAyMDAwMDAwWhcNMzAxMjMxMjM1OTU5WjCBjzELMAkGA1UEBhMCR0IxGzAZBgNV
BAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEYMBYGA1UE
ChMPU2VjdGlnbyBMaW1pdGVkMTcwNQYDVQQDEy5TZWN0aWdvIFJTQSBEb21haW4g
VmFsaWRhdGlvbiBTZWN1cmUgU2VydmVyIENBMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEA1nMz1tc8INAA0hdFuNY+B6I/x0HuMjDJsGz99J/LEpgPLT+N
TQEMgg8Xf2Iu6bhIefsWg06t1zIlk7cHv7lQP6lMw0Aq6Tn/2YHKHxYyQdqAJrkj
eocgHuP/IJo8lURvh3UGkEC0MpMWCRAIIz7S3YcPb11RFGoKacVPAXJpz9OTTG0E
oKMbgn6xmrntxZ7FN3ifmgg0+1YuWMQJDgZkW7w33PGfKGioVrCSo1yfu4iYCBsk
Haswha6vsC6eep3BwEIc4gLw6uBK0u+QDrTBQBbwb4VCSmT3pDCg/r8uoydajotY
uK3DGReEY+1vVv2Dy2A0xHS+5p3b4eTlygxfFQIDAQABo4IBbjCCAWowHwYDVR0j
BBgwFoAUU3m/WqorSs9UgOHYm8Cd8rIDZsswHQYDVR0OBBYEFI2MXsRUrYrhd+mb
+ZsF4bgBjWHhMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAbBgNVHSAEFDASMAYGBFUdIAAw
CAYGZ4EMAQIBMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9jcmwudXNlcnRydXN0
LmNvbS9VU0VSVHJ1c3RSU0FDZXJ0aWZpY2F0aW9uQXV0aG9y

Re: new feature: sieve forward plugin

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok this in itself is a issue however forwards should be fully received 
by the server and then resent to get around this issue.


I use the mapping feature & the database to handle forwards in postfix 
which appears to work without any issues bypassing sieve.


#Postgres Stuff
mydestination = pgsql:/usr/home/postfix/config/pgsql-mydestination.cf
#virtual_maps=pgsql:/usr/home/postfix/config/pgsql-virtual.cf
alias_maps=pgsql:/usr/home/postfix/config/pgsql-aliases.cf
#sender_canonical_maps = pgsql:/usr/home/postfix/config/pgsql-canonical.cf

with

# cat pgsql-mydestination.cf
# /etc/postfix/pgsql-mydestination.cf
#
# pgsql config file for local domain (like sendmail's sendmail.cw)
# lookups on postfix
# comments are ok.
#
# the user name and password to log into the pgsql server
hosts = pg.scom.ca:5433
user =
password =
# the database name on the servers
dbname = scom_billing
# the table name
table = email_users
#
select_field = domain
where_field = domain

and


# cat /usr/home/postfix/config/pgsql-aliases.cf
#  pgsql-virtual.cf   ***
#
# pgsql config file for alias lookups on postfix
# comments are ok.
#
# the user name and password to log into the pgsql server
hosts = pg.scom.ca:5433
user =
password =
# the database name on the servers
dbname = scom_billing
# the table name
table = email_users
#
#Select source email address alias (ie sales@ etc aliases )
where_field = source

#Select destination email account address (final delivery)
select_field = destination

#Account Status (1=good)
additional_conditions = and status = '1'


So when using the above postfix receives it, remaps it and resends the 
email as its own thus fixing any spf issues along the way as it is sent 
by the local server.


I understand that forwarding in a sieve script might over ride this and 
cause an spf failure, in that case (and i have not tried) then the sieve 
script should somehow deliver local and then resend?


remapping the address through postfix would be the better approach.

this would mark the email as coming from the local sending server and 
the spf record sent down the line would reflect that.


spf verification would have already been verified by incoming postfix so 
you are not passing along something that got rejected in the first place?



(The comments below are a little off topic but not really - spf issues 
are starting to become more of an issue as of late, it seems upstream 
providers are more interested in collecting customer data then just 
simply processing it and moving it along.)


I recently went through this with microsoft outlook 365

please refer to microsoft [Case #:32804718] - Ticket 32800372 / 32795526 
spf records :


they refered me to a few links

https://docs.microsoft.com/en-us/microsoft-365/troubleshoot/antispam/sender-rewriting-scheme

sender-rewriting-scheme (which can not be turned off by microsoft) 
basically takes the from address of the origional email and resets it to 
the account email address thus when forwarded onwards the email only 
see's the microsft email address, the origional is dropped and does not 
even get passed in the headers.


&

https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/high-risk-delivery-pool-for-outbound-messages?view=o365-worldwide

the above link was microsoft's explanation on why they refused to fix 
their spf record.


when forwarding by a rule in microsoft (the only way to preserve the 
senders address email), microsoft receives it and then reque's the email 
through another server (in principal this is correct) but the forwarding 
server is considered insecure because microsoft fails their own spf 
record on purpose.


Apparently google is now also using unverified (or insecure) servers 
setup the same way.


why i have no idea?

Microsoft replied with the ticket after 3 months of messing around that 
they would not fix their spf record.


so basically microsoft was by design blocking their own outgoing server 
with an spf fail.


Caption from microsoft ticket.

We know nothing will make this better  but to fix things on our side as 
you have said, but based on what we understood and what you have 
submitted to us, we understand the IP addresses you have mentioned are 
not listed Exchange Online's SPF record, however this is by design as we 
do not publish addresses for all traffic coming from our servers. In 
particular, traffic being relayed through Exchange Online may be treated 
this way. In order to ensure that mail flow is secure and being 
processed correctly, we recommend you use connectors when routing mail 
between Exchange Online and any on premise or partner organizations.


With your permission, I am archiving the ticket # 32804718.

that being said when i did a dig on outlook

dig txt spf.protection.outlook.com

i got back

spf.protection.outlook.com. 600 IN  TXT "v=spf1 ip4:40.92.0.0/15 
ip4:40.107.0.0/16 ip4:52.100.0.0/14 ip4:104.47.0.0/17 
ip6:2a01:111:f400::/4

Re: Get a list of currently active IMAP connections?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

NO 

it is showing active open imap connections

fyi



Happy Friday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 8/19/2022 3:40 AM, Narcis Garcia wrote:


Do you mean ps is reading dovecot.conf ?!



Narcis Garcia

__
I'm using this dedicated address because personal addresses aren't 
masked enough at this mail public archive. Public archive administrator 
should fix this against automated addresses collectors.

El 19/8/22 a les 9:40, Aki Tuomi ha escrit:

dovecot.conf, not ps config.

Aki


On 19/08/2022 10:38 EEST Narcis Garcia  wrote:

What config?

I see no configuration file documented on ps manpage.



Narcis Garcia

__
I'm using this dedicated address because personal addresses aren't
masked enough at this mail public archive. Public archive administrator
should fix this against automated addresses collectors.
El 19/8/22 a les 9:33, 202107-dove...@planhack.com ha escrit:
Add `verbose_proctitle = yes` to your config to get usernames and 
IPs in the ps listing.

Re: Get a list of currently active IMAP connections?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

I use ps : (greping by imap & idle)

# ps -axww | grep imap | grep IDLE

thant and split() in python



 8606  -  S 0:08.78 imap: [ke...@elirpa.com 54.242.98.60 IDLE] 
(imap)
12234  -  I 0:01.00 imap: [recept...@clancyca.com 72.143.119.178 
IDLE] (imap)

20668  -  S 0:02.01 imap: [p...@scom.ca 216.58.25.131 IDLE] (imap)
23219  -  I 0:00.33 imap: [cla...@clancyca.com 72.143.119.178 
IDLE] (imap)
26761  -  S 0:00.52 imap: [ed.ha...@ekst.ca 204.237.91.165 IDLE] 
(imap)

26785  -  I 0:00.87 imap: [e...@scom.ca 204.237.91.165 IDLE] (imap)
26787  -  I 0:00.80 imap: [ed.ha...@dssmgmt.com 204.237.91.165 
IDLE] (imap)

27378  -  S 0:00.42 imap: [e...@scom.ca 204.237.91.165 IDLE] (imap)
31404  -  S 0:03.90 imap: [p...@scom.ca 216.58.25.131 IDLE] (imap)
32494  -  S 0:00.13 imap: [install...@tomkudla.ca 167.94.196.10 
IDLE] (imap)
32497  -  S 0:00.13 imap: [install...@tomkudla.ca 167.94.196.10 
IDLE] (imap)
33809  -  I 0:00.28 imap: [cla...@clancyca.com 72.143.119.178 
IDLE] (imap)
36321  -  I 0:00.21 imap: [cla...@clancyca.com 72.143.119.178 
IDLE] (imap)
39188  -  I 0:00.39 imap: [cla...@clancyca.com 72.143.119.178 
IDLE] (imap)

42706  -  S 0:00.45 imap: [e...@scom.ca 204.237.91.165 IDLE] (imap)
46356  -  S 0:02.98 imap: [rco...@tnky.ca 198.91.141.141 IDLE] 
(imap)
46422  -  S 0:01.32 imap: [rco...@tnky.ca 198.91.141.141 IDLE] 
(imap)
46424  -  S 0:01.27 imap: [rco...@tnky.ca 198.91.141.141 IDLE] 
(imap)
50756  -  S 0:01.36 imap: [rco...@tnky.ca 198.91.141.141 IDLE] 
(imap)
58656  -  I 0:00.07 imap: [ditchb...@clancyca.com 216.58.50.30 
IDLE] (imap)
63886  -  S 0:00.70 imap: [rco...@tnky.ca 198.91.141.141 IDLE] 
(imap)
68246  -  I 0:00.08 imap: [l...@clancyca.com 72.143.119.178 IDLE] 
(imap)

74719  -  I 0:00.03 imap: [d...@elirpa.com 142.183.30.44 IDLE] (imap)
76580  -  I 0:00.02 imap: [i...@willsagriquipandfencing.ca 
173.32.244.194 IDLE] (imap)
76584  -  I 0:00.02 imap: [how...@willsagriquipandfencing.ca 
173.32.244.194 IDLE] (imap)
77567  -  S 0:00.04 imap: [rco...@tnky.ca 198.91.141.141 IDLE] 
(imap)
77569  -  I 0:00.03 imap: [rco...@tnky.ca 198.91.141.141 IDLE] 
(imap)







Happy Friday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 8/18/2022 6:28 PM, J Doe wrote:


On 2022-08-16 16:46, Antonio Leding wrote:
At the risk of being pedestrian, I just use something like |sudo 
netstat -an | grep ‘:[ IMAP_PORT ]’|


I’m pretty sure you thought of this but still, thought I would toss it 
out…




Hi Antonio and Jaroslaw,

I don't think the second solution is pedestrian; I think it's cool that 
people have come up with different solutions for the same problem!


I am thinking that this may not be the solution that Jaroslaw is looking 
for, as this also requires spawning a process to run netstat and then 
capturing the results.  The socket approach avoids an additional process.


- J

Re: dovecot/config processes open, and consuming all memory

[Paul Kudla (SCOM.CA Internet Services Inc.)]

for what's it is worth

i am not running any vsize on the config

service config {
  unix_listener config {
user = vmail
}
}

i'm just running defaults

i do use vsz_limit elsewhere main to curve the replication processes ??

never had an issue 2.3.17 / 18 / 19

never used 16 (fyi)

maybe 2048M is insufficent system wide, try increasing 10 1g ?




Happy Friday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 8/12/2022 6:06 PM, dovecot-boun...@dovecot.org wrote:


I'm having strange behavior in dovecot 2.3.16.
It's opening dozens of dovecot/config process and consuming all server 
memory. Normally each process consumes between 700Mb and 1Gb of ram.


Would anyone have an idea about this?

service config {
   vsz_limit = 2048M
   idle_kill = 60s
   service_count = 1024
}

pstree
systemd─┬─ModemManager───2*[{ModemManager}]
     ├─agetty
     ├─cron
     ├─dbus-daemon
     ├─dovecot─┬─anvil
     │ ├─6*[auth]
     │ ├─46*[config]
     │ ├─1212*[imap]
     │ ├─155*[imap-login]
     │ ├─12*[lmtp]
     │ ├─38*[log]
     │ ├─10*[managesieve]
     │ ├─19*[pop3]
     │ ├─3*[pop3-login]
     │ └─18*[stats]


root   45831  0.0  1.1 774688 752732 ?   S    09:31 0:31 
dovecot/config
root  388792  0.0  1.1 775060 753276 ?   S    14:00   0:15 
dovecot/config
root  510685  0.0  1.1 775384 753604 ?   S    15:06   0:20 
dovecot/config
root  675638  0.0  1.1 775348 753620 ?   S    16:56   0:15 
dovecot/config
root  795375  0.0  1.1 775460 753516 ?   S    18:03   0:07 
dovecot/config
root  798754  0.2  1.1 775592 753712 ?   S    18:05   0:30 
dovecot/config
root 1082696  0.2  1.1 774892 753216 ?   S    21:10   0:07 
dovecot/config
root 1098433  0.4  1.1 774924 753244 ?   S    21:33   0:07 
dovecot/config
root 1109255  0.9  1.1 774924 753344 ?   S    21:50   0:07 
dovecot/config
root 1112976  2.0  1.1 774956 753528 ?   S    21:57   0:07 
dovecot/config
root 1114137  3.0  1.1 775028 753308 ?   S    21:59   0:07 
dovecot/config
root 1115382  5.4  1.1 774924 753496 ?   S    22:01   0:06 
dovecot/config
root 1883627  0.0  1.1 759120 728832 ?   S    Aug11   0:07 
dovecot/config
root 1889705  0.0  1.8 1251460 1221872 ? S    Aug11   0:11 
dovecot/config
root 1895022  0.0  1.8 1253280 1224284 ? S    Aug11   0:11 
dovecot/config
root 1900690  0.0  1.8 1255684 1227528 ? S    Aug11   0:12 
dovecot/config
root 1905648  0.0  1.8 1257880 1229912 ? S    Aug11   0:12 
dovecot/config
root 1910857  0.0  1.8 1259156 1231552 ? S    Aug11   0:12 
dovecot/config
root 1914332  0.0  1.1 764328 736552 ?   S    Aug11   0:20 
dovecot/config
root 2343896  0.0  1.8 1259472 1231516 ? S    Aug11   0:12 
dovecot/config
root 2346351  0.0  1.8 1259472 1231836 ? S    Aug11   0:13 
dovecot/config
root 2348559  0.0  1.1 764704 736440 ?   S    Aug11   0:14 
dovecot/config
root 2445701  0.0  1.1 764276 736540 ?   S    Aug11   0:19 
dovecot/config
root 2572525  0.0  1.1 764640 736880 ?   S    Aug11   0:18 
dovecot/config
root 2734251  0.0  1.1 764776 737696 ?   S    Aug11   0:08 
dovecot/config
root 2740980  0.0  1.1 764768 737244 ?   S    Aug11   0:17 
dovecot/config
root 2899925  0.0  1.1 764624 737760 ?   S    Aug11   0:43 
dovecot/config
root 3517063  0.0  1.1 764984 738004 ?   S    Aug11   0:09 
dovecot/config
root 3541465  0.0  1.1 765224 738756 ?   S    Aug11   0:07 
dovecot/config
root 3545589  0.0  1.1 766452 740408 ?   S    Aug11   0:06 
dovecot/config
root 3549259  0.0  1.1 766796 741048 ?   S    Aug11   0:07 
dovecot/config
root 3553902  0.0  1.1 767812 742284 ?   S    Aug11   0:07 
dovecot/config
root 3558080  0.0  1.1 768440 743524 ?   S    Aug11   0:08 
dovecot/config
root 3562091  0.0  1.1 769224 744424 ?   S    Aug11   0:07 
dovecot/config
root 3568721  0.0  1.1 769388 744888 ?   S    Aug11   0:07 
dovecot/config
root 3573024  0.0  1.1 770048 745904 ?   S    Aug11   0:08 
dovecot/config
root 3578416  0.0  1.1 770836 746736 ?   S    Aug11   0:06 
dovecot/config
root 3581765  0.0  1.1 771948 748492 ?   S    Aug11   0:06 
dovecot/config
root 3585837  0.0  1.1 772828 749860 ?   S    Aug11   0:07 
dovecot/config
root 3590276  0.0  1.1 773560 750520 ?   S    Aug11   0:07 
dovecot/config
root 3594640  0.0  1.1 774612 752552 ?   S    Aug11   0:08 
dovecot/config
root 3597417  0.0  1.1 774968 753076 ?   S    Aug11   0:35 
dovecot/config
root 4116314  0.0  1.9 1280496 1258580 ? S    05:31   0:13 
dovecot/co

Re: Replication not working - GUIDs conflict - will be merged later

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok i went through this as well a bit

there is a replication full sync variable (i am having trouble finding it)

24h is the default but i might have rebuilt dovecot modifying this default

after i got things working i put everything back to default code.


yep i did

from dovecot-2.3.19/src/replication

see :

aggregator/replicator-connection.c:#define MAX_INBUF_SIZE 1024
aggregator/replicator-connection.c:#define REPLICATOR_MEMBUF_MAX_SIZE 
1024*1024
aggregator/replicator-connection.c: conn->queue[i] = 
buffer_create_dynamic(default_pool, 1024);

Binary file replicator/replicator-brain.o matches
replicator/replicator-settings.c:   .replication_full_sync_interval 
= 60*60*24,

replicator/notify-connection.c:#define MAX_INBUF_SIZE (1024*64)
Binary file replicator/doveadm-connection.o matches
Binary file replicator/.libs/replicator matches
replicator/replicator-brain.c:  pool = 
pool_alloconly_create("replication brain", 1024);
replicator/replicator-queue.c:  queue->user_queue = 
priorityq_init(user_priority_cmp, 1024);
replicator/replicator-queue.c:  hash_table_create(&queue->user_hash, 
default_pool, 1024,

Binary file replicator/notify-connection.o matches
Binary file replicator/dsync-client.o matches


I do not believe there is a settable variable in dovecot.conf ?

I could be wrong.

the actual code containing the variable is below, change and recompile 
all and that should/might help.


replicator/replicator-settings.c:   .replication_full_sync_interval 
= 60*60*24,


change to 24 so something more practical ?

note 60*60*24 is math (ie how many seconds in between full syncs)  - ie 
do not change 24 to 24h for example.


do this on both servers.

note that a full sync interval stress wise on the server is dependant on 
how much physical mail you have in the mbox.


note that the full resync interval syncs both accounts from scratch.

also note 6hrs is not a bad place to start?

the replicator service will deal with this in the background

there are also other variables hard set (like i believe 15m for the 
retry bad sync interval ?)


you will need to dig through the replicator code to find these.



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 8/2/2022 9:30 AM, Sebastian Marske wrote:




On 8/1/22 11:15, Patrick Westenberg wrote:

Very interesting new insights:

When I use imapsync and let it synchronize mails from INBOX to
INBOX/testfolder, the automatic replication works fine.
All mails are synchronized between my two backends.


When I move the mails to the INBOX (doveadm move -u m...@example.com
INBOX mailbox INBOX/testfolder all), these mails are lost on the
replica! They are neither in INBOX, nor in INBOX/testfolder

Regards
Patrick


Hi,

every now and then I have the same problem on our servers. Currently,
I'm running Dovecot 2.3.19.1 as well, but I upgraded directly from
2.3.16 due to other issues with the versions in between.

Last time I observed a de-sync due to a GUID change, it appeared like
the user had moved a folder around in their mailbox. And indeed, the
output of 'doveadm mailbox status -u someuser guid '*' listed different
GUIDs. Dovecot actually logged some errors for this case:

Dovecot log from replica1:
Jul 27 12:06:08 replica1 dovecot[3431]:
doveadm(someuser)<10206>: Error: Duplicate
mailbox GUID 78c9dc2c0c0ee162c1080ca22142 for mailboxes
path/to/folder and path/to/folder-temp-1 - giving a new GUID
b0053e390f0ee162de27c9042436 to path/to/folder
Jul 27 12:06:08 replica1 dovecot[3431]:
doveadm(someuser)<10208>: Error: Duplicate
mailbox GUID 78c9dc2c0c0ee162c1080ca22142 for mailboxes
path/to/folder and path/to/folder-temp-1 - giving a new GUID
5823fe0d100ee162e027c9042436 to path/to/folder

Dovecot log from replica2:
Jul 27 12:06:04 replica2 dovecot[47018]:
doveadm(someuser)<2239>: Warning: Failed to do
incremental sync for mailbox path/to/folder, retry with a full sync
(uidnext 1 < 13)
Jul 27 12:06:04 replica2 dovecot[47018]:
doveadm(someuser)<2241>: Error: Duplicate
mailbox GUID 0ccaab01079031620e1e0ca22142 for mailboxes
path/to/folder and some/folder - giving a new GUID
78c9dc2c0c0ee162c1080ca22142 to path/to/folder

At that time, only replica2 was accepting imap connections.
In this particular case, Dovecot eventually managed to get things back
in sync after way over 24h, but I also had users out of sync for
multiple days.
Running 'doveadm -Dv sync -u someuser -d' manually gave me the same
error message, but didn't change anything.

Other things I've observed:
* it's not limited to a fixed set of users (unlike the
too-many-folders-thing with Dovecot 2.3.1[78])
* it's not limited to newly created users, but also affects users, that
have been in sync for months/years
* it's not limited to mailboxes with lots of imap operations going on
* i

Re: Doveadm Move Query

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok u...@domain.com needs to exist before any operations can be done on it.

I discovered that dovecot does not consider a virtual mailbox active 
until it is returned in the user database


see : doveadm user '*'

both accounts MUST be returned in the list (user@.net & user@.com)

from there it should work as expected.

i went through this with my django email user interface as the user was 
not being saved in the database until the django model had completing 
saving a new entry, thus when creating the new account i had to put a 
delay check in my create email account that continued to loop until 
django had finished it's processing, very anoying (not dovecot's issue) 
but i think you are facing something similiar?



it seems you might be renaming the mbox ?

again both user@.net & user@.com must exist along the way before the 
account(s) can be accessed.


if renaming the mbox is your intention than add the user@.com account

move should now work

then delete the user@.net account.





Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 8/2/2022 6:49 AM, Simon B wrote:
I have a production Dovecot problem and although I searched the mailing 
lists, I could not find an answer and I hope you can give me a quick 
answer/pointer in the right direction.



I have mails for a user (u...@domain.net ) 
under /var/spool/mail/virtual/domain.net/user   
and I want to move ALL the mails to 
/var/spool/mail/virtual/domain.com/user 


If I use

#doveadm -Dv move -u u...@domain.net 
  Maildir:/var/spool/mail/virtual/domain.net/user   Maildir:/var/spool/mail/virtual/domain.com/user  ALL


I get
doveadm(root): Fatal: Unknown argument 
MAILDIR:/var/spool/mail/virtual/domain.com/user 


if I use
#doveadm -Dv move -uu...@domain.net 
 Maildir:/var/spool/mail/virtual/domain.net/user 
 /var/spool/mail/virtual/domain.com/user 
ALL


doveadm(root): Fatal: Unknown argument 
/var/spool/mail/virtual/domain.com/user 


What the hell am I doing wrong!? :)

Thanks.

Simon



--
This message has been scanned for viruses and
dangerous content by *MailScanner* , and is
believed to be clean.

Re: variable %w recursive expanding

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok not what you probably want to hear

c, python etc uses % as a variable indicator along with the next 
character using a format variable (like %s being a string variable)


this is what is causing the issue

Not saying for sure but experience wise i am not familiar with any 
system allowing a % sign in the password now a days ?


could be wrong, i guess it depends on the backbone password fetcher process

even if you could get dovecot (or any c based programming) to allow for 
this clients would probably have similiar issues which there would be no 
real control over.


another charater would also be '\' as it is used to being a delimiter 
for the next character being absolute


along with \x00 (non ascii characters in a string)

again just an experience fyi



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 8/1/2022 4:47 AM, Franz Beslmeisl wrote:


In order to change the password scheme I wrote a script named
updateproxy that needs the plain text password from the user.
To get that I use the line

     password_query = SELECT username as user, password, \
   '%w' as userdb_plain_pass FROM auth_user WHERE username='%n'

This works nicely with almost all passwords but not with this one

     1234567%&/abcd

the error message being

     dovecot: Failed to expand plugin setting plain_pass =
   '1234567%&/abcd': Unknown variable '%&'

It seems to me that dovecot tries to do another level of variable
evaluation upon the **value** of the already evaluated variable.

So I searched for ways to escape problematic characters like %
and changed my line to

     password_query = SELECT username as user, password, \
   '%E{w}' as userdb_plain_pass FROM auth_user WHERE username='%n'

but this produces problems with password values containing quotes.

So how can I get a plain text password containing any ascii char
(or even better any utf-8 char) safely to my script?

Thanks for your suggestions



-- here the nasty details, if you want -
$ dovecot -n
# 2.3.7.2 (3c910f64b): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.7.2 ()
# OS: Linux 5.4.0-122-generic x86_64 Ubuntu 20.04.4 LTS
# Hostname: mx-10-2.bildung.hessen.de
auth_mechanisms = plain login
auth_username_chars = 
abcdefghijklmnopqrstuvwxyz_0123456789.ABCDEFGHIJKLMNOPQRSTUVWXYZ-@

lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
mail_location = maildir:~/Maildir
mail_privileged_group = mail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart 
extracttext

namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     auto = subscribe
     special_use = \Drafts
   }
   mailbox Junk {
     auto = subscribe
     special_use = \Junk
   }
   mailbox Sent {
     auto = subscribe
     special_use = \Sent
   }
   mailbox "Sent Messages" {
     special_use = \Sent
   }
   mailbox Trash {
     auto = subscribe
     special_use = \Trash
   }
   prefix =
}
passdb {
   # the following file contains a '%w'-line
   args = /etc/dovecot/db1.conf
   driver = sql
}
passdb {
   # the following file contains a '%w'-line
   args = /etc/dovecot/db2.conf
   driver = sql
}
passdb {
   # the following file contains no '%w'-line (just for detail)
   args = /etc/dovecot/db3.conf
   driver = sql
}
plugin {
   sieve = ~/.dovecot.sieve
   sieve_dir = ~/sieve
   sieve_max_actions = 64
   sieve_max_redirects = 16
   sieve_max_script_size = 10M
   sieve_trace_debug = yes
   sieve_user_log = ~/sievelog
   sieve_vacation_dont_check_recipient = yes
   sieve_vacation_use_original_recipient = yes
}
protocols = imap sieve lmtp
service auth {
   unix_listener /var/spool/postfix/private/dovecot-auth {
     group = postfix
     mode = 0660
     user = postfix
   }
}
service imap {
   executable = imap after-login
}
service lmtp {
   unix_listener /var/spool/postfix/private/dovecot-lmtp {
     group = postfix
     mode = 0660
     user = postfix
   }
}
service after-login {
   executable = script-login /etc/dovecot/updateproxy
   user = vmail
}
service stats {
   unix_listener stats-reader {
     group = mail
     mode = 0666
   }
   unix_listener stats-writer {
     group = mail
     mode = 0666
   }
}
ssl_cert =  was automatically rejected:%n%r
}
protocol imap {
   imap_client_workarounds = delay-newmail
   mail_max_userip_connections = 300
}

Re: Replication not working - GUIDs conflict - will be merged later

[Paul Kudla (SCOM.CA Internet Services Inc.)]

OK thanks for the updates

Long story short i went through a bunch of replication issues when i was 
first setting up dovecot.


Most of which were fixed in 2.3.19 and i have not seen any issues since.

in general i had to turn on the debugging mode (mail_debug = yes)

and filter syslog by "replication"

It was discovered through some work that any account over 300 physical 
folders (give or take) would not replicate (physical size of mailbox had 
nothing to do with the issue just folder count) and would fail without 
error (why i asked the version this was an issue in 2.3.18 and before?)


when a replication sync failed it did say in the logs replication 
requested for  but it would fail and not log the error 
why, replication timeouts were however recorded?


I my self ended up patching the c code in the replicator to get more 
detail how far a replication sync would go (ie i added a bunch of 
logging code to track the issues better)


maybe look at folder counts? - It was only effecting 5 of my customers 
but was a pain to find the issue.


that being said i had to make some scripts to show user replication and 
what was outstanding between the servers (i had to run them on both 
servers seperately to acurately get the replication status going in both 
directions.


I now run these scripts in the background every 5 minutes to make sure 
replication on both sides are in sync.


also i found tcpip replication (without ssl) worked the best? tcpip over 
SSL had timing errors, ssl in my case was not required as the two 
servers were hardwired together (ie no security leaks)


if your two servers are at two different sites consider a small vpn setup

i like using gre for this as it is hardwired by static ip addresses on 
both sides and you can set a 10.x.x.x (or whatever) to communicate 
between the servers ??


also on using the doveadm ssh scripts introduced rights issues between 
the two servers file systems (even though they were identical)


Basically i tried everything !

tcpip - non ssl just seem to work the best.

sync.status :

doveadm replicator status
echo ' '
doveadm replicator dsync-status | grep -v 'Not connected'


which outputs :

# sync.status
Queued 'sync' requests0 

Queued 'high' requests0 

Queued 'low' requests 0 

Queued 'failed' requests  0 

Queued 'full resync' requests 0 

Waiting 'failed' requests 0 


Total number of known users   269

oh and another major thing was the replication selecting from the 
database properly.


i use postgresql

#iterate_query = SELECT user, password FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
status = True and alias_flag = False


note my db setup uses status=True for an active user and 
alias_flag=False for an alias email redirect inside postfix (fyi), you 
can ignore these based on how your database is setup.


dovecot is very intelligent if an active box gets activity on one server 
then it wont replicate to the other server if that mailbox is not 
returned to sync users list (but it will setup / activate replication on 
the server that received the email) - this took a bit to figure out as well.



# cat sync.users
doveadm replicator status '*' | grep '   y'

would only sync what was bad in the replication??

try running

# doveadm user '*' | wc
 269 2695244

on both servers the account count (269) in my case should be the same on 
both servers.



Here is my dovecot.conf config maybe it will help

replication configs (other the the server ip address) must be the same 
on both sides.



# cat dovecot.conf
# 2.3.14 (cee3cbc0d): /usr/local/etc/dovecot/dovecot.conf
# OS: FreeBSD 12.1-RELEASE amd64
# Hostname: mail18.scom.ca

auth_debug = no
auth_debug_passwords = no

default_process_limit = 16384

mail_debug = no

#lock_method = dotlock
#mail_max_lock_timeout = 300s

#mbox_read_locks = dotlock
#mbox_write_locks = dotlock

mmap_disable = yes
dotlock_use_excl = no
mail_fsync = always
mail_nfs_storage = no
mail_nfs_index = no

auth_mechanisms = plain login
auth_verbose = yes
base_dir = /data/dovecot/run/
debug_log_path = syslog
disable_plaintext_auth = no
dsync_features = empty-header-workaround

info_log_path = syslog
login_greeting = SCOM.CA Internet Services Inc. - Dovecot ready
login_log_format_elements = user=<%u> method=%m rip=%r lip=%l mpid=%e %c


mail_location = maildir:~/

mail_plugins = " virtual notify replication fts fts_lucene "
mail_prefetch_count = 20

protocols = imap pop3 lmtp sieve


protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = moni...@scom.ca
}

service lmtp {
  process_limit=1000
  vsz_limit = 512m
  client_limit=1
   unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
 group = postfix
 mode = 0600
 user = postfix
  }
}

protocol lda {
  mail_plugins = $mail_plugins sieve
}

service lda {
  process_lim

Re: Replication not working - GUIDs conflict - will be merged later

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok This is speculation but i understand the issue at a programming level

what needs to be understood is that imap's uids & ugid's are relative to 
the host server the email is coming from.


this is generally not an issue with replication on cyrus or dovecot 
because the server and the replication is being handled by the same 
server set (ie the same uids & guids etc are generated as things happen)


example replicated data :

-rw---1 vmail  vmail  uarch  185K Jul 29 09:30 
1659101404.M875201P20192.mail19.scom.ca,S=189252,W=192431:2,S
-rw---1 vmail  vmail  uarch  1.5K Jul 29 09:53 
1659102818.M268117P41331.mail18.scom.ca,S=1583,W=1639:2,S
-rw---1 vmail  vmail  uarch  1.0M Jul 29 12:52 
1659113530.M841469P58214.mail18.scom.ca,S=1095861,W=1113817:2,S
-rw---1 vmail  vmail  uarch  210K Jul 29 13:15 
1659114913.M958008P31982.mail19.scom.ca,S=215405,W=219216:2,S


you will note the originating server is in the mail file name (mail19 & 
mail18 in my case)


this is how dovecot sorts out the uids etc on the fly. (i think)

If i have read this correctly you are trying to sync to an external imap 
server that carries its own uids guids etc which will be different.


where you are saying that you are using imap sync i assume you are using 
the unix version


# imapsync
Name:

 imapsync - Email IMAP tool for syncing, copying, migrating and archiving
 email mailboxes between two imap servers, one way, and without duplicates.

Version:

 This documentation refers to Imapsync $Revision: 1.977 $


if so look at the

--useuid:


Use UIDs instead of headers as a criterion to recognize
messages. Option --usecache is then implied unless
--nousecache is used.


and the --logfile (ie run a logging file when connecting the the 
external account), it might help with any errors being generated (run 
imapsync in debug mode to get full detail)


basically using useuid deals with sometimes getting a different uid back 
from the origional server


i go through this issue more with pop3 as it returns the id list 
starting at uid 1 (for example) instead of the actual uid against the 
email on the server.


uids will force a proper sync (imap or pop3) because the uid on the 
server will always return the same uid for that email message and 
increments forward inside the account.


if so then imap sync should be sorting this out when syncing the imap 
accounts ? (ie creating new usid guids etc)


so assuming the above is happening the next question is are you using 
replication that is fully setup between the two servers or are you doing 
manual replication (ie running the doveadm command to do the sync?)


(you mentioned using the backup command which would kinda work but full 
replication does the changes on the fly and should work)


if you are running manual replication you should consider going to the 
live replication, it will sort out stuff as the imap folders sync etc. 
(or it should)


The next thing to consider is there were some issues that were fixed in 
2.3.19 replication, are you running the same dovecot versions on both 
servers ?


I do a ton of emails, reporting etc and find that replication works well 
on dovecot 2.3.19 bewteen both of my mail servers. ie it does not matter 
which one receives the email it gets sorted out. If there is an error 
the replication will sort it out on the next sync run through the 
replication process running in the background.


you can set all of the retries etc for replication in the config files.

the merged later is probably indicating that dovecot will sort stuff out 
in the background (ie a reindex etc) but that is putting extra stress on 
the server(s), i used to get the merge or duplicate uids, guids on cyrus 
and it would try to sort it out on the fly. this would occur when one 
replicated server was offline and i was forcing a sync update after 
bringing it back online, this was the case because both servers had 
received emails into the same account from seperate sources thus the 
same uid was set for two different messages on each server. (fyi)


with syrus a rebuild was the only was to sort this out

dovecot seems way more resiliant in this department.


again full replication setup would sort these issue out i expect as each 
server would handle stuff as it happens and adjust uid,guids accordingly.








Happy Sunday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 7/31/2022 8:16 AM, Patrick Westenberg wrote:


Hi everyone,

I have a weird problem with replication that I'm unable to solve.

A new account is sychronized from an external provider via imapsync.
The mails end up on my backend1. I see that the folder structure is
immediately replicated to backend2.

However, a lot of mails are missing and "doveadm replicator status" also

Re: rawlog data in a lua script

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Hi - I use this python script to capture a socket (ie the log file) and 
then send it to syslog, i use this for all the systems that do not 
really support syslogging (apache etc)


basic useage

/usr/bin/nohup /programs/common/capture -s 
/usr/local/apache2/logs/httpd-access.log -l httpd -d 10.228.0.6:514 -p 
httpd & > /dev/null


i typically run this at startup in rc.local

hope this helps :

--

## cat capture
#!/usr/local/bin/python3
# -*- coding: UTF-8 -*-


import os,sys,socket
import datetime,time
from optparse import OptionParser

from lib import *

USAGE_TEXT = '''\
usage: %%prog %s[options]
'''

parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')

parser.add_option("-s", "--socket", dest="socket_file", help="Socket 
File to Capture")
parser.add_option("-l", "--label", dest="label", help="Syslog Label to 
Insert")
parser.add_option("-d", "--destination", dest="destination", 
help="Syslog Destibnation Server:Port")

parser.add_option("-p", "--pid", dest="pid", help="PID Process Name")
#parser.add_option("-e", "--email", dest="email", help="Additional Email 
To")
#parser.add_option("-t", "--temp", dest="tempdir", help="Local Temp 
Directory")


options, args = parser.parse_args()

print (options.socket_file)
print (options.label)
print (options.destination)
print (options.pid)



if options.socket_file == None :
print ('Missing Socket File Information')
sys.exit()

if options.label == None :
print ('Missing Syslog Label Information')
sys.exit()

if options.destination == None :
print ('Missing Syslog Destination host:[port]')
sys.exit()

if options.pid == None :
print ('Missing Syslog Pid Process Name')
sys.exit()


#try local syslog (/var/run/log)

UDP_IP = options.destination.split(':')

if len(UDP_IP) == 2 : #Set Port
UDP_PORT = int(UDP_IP[1])
else :
UDP_PORT = 514 #Default

UDP_IP = UDP_IP[0]  #Server

#MESSAGE = str("<22>Mar 27 04:16:16 es-scom[12345] offsite.scom.ca su: 
Hello, World!")

#MESSAGE = str("<183>Mar 27 16:17:41 scom-live[72178]: Hello World")

print("UDP target IP: %s" % UDP_IP)
print("UDP target port: %s" % UDP_PORT)
#print("message: %s" % MESSAGE)

count = 10


#sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
#sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))
#sock.sendto(bytes(MESSAGE, "utf-8"), (UDP_IP, UDP_PORT))
#sock.close()
#sys.exit()


#def read_commands():
try:
print ("Creating read pipe... %s"  %options.socket_file )
os.mkfifo(options.socket_file)# Create pipe
print ("Pipe %s created!" %options.socket_file )
except:
print ("Pipe %s already exists" %options.socket_file )

#chmod 777 the file so everyone can talk to it
os.system('/bin/chmod 777 %s' %options.socket_file)


with open(options.socket_file, "r") as pipecmd:
while True:
time.sleep(.001)
try:
line = pipecmd.readline()
if line != '' : #New Data
if line == '\n' :
continue
print ('Raw Text : %s' %line)
encoded_string = line.encode("ascii", 
"ignore")

line = encoded_string.decode()
line = create_ascii(line)
line = line.ascii
print ('Line after ASCII : %s' %line)
print ( 'Line Count : %s' %len(line) )
#line = data
#go get my pid
pid_process = '0'
if options.pid == 'postfix' : #its a 
diverted postfix process get the actual pid from raw text
pid_process = 
line.split('[',1)[1].split(']',1)[0]


else :
command = commands('/bin/ps 
-axww | /usr/bin/grep %s' %options.pid)

print ()
#print (command.output)

for n in range 
(0,len(command.output)) :
if '/bin/ps -axww | 
/usr/bin/grep' not in command.output[n] and '/usr/bin/grep' not in 
command.output[n] and '/usr/local/bin/python3' not in command.output[n]  :
pid_process = 
( command.output.split(' ')[0] ) #whats left should be my process ?

break

print ('PID Process : %s ' %pid_process )

if options.destination == 'local' : 
#Send to log here

print ('Sending to Local S

Thunderbird / Copy to Send folder times out

[Paul Kudla (SCOM.CA Internet Services Inc.)]

I know this has been asked before in both Dovecot & Thunderbird Forums 
(i will posting this info there as well)


It seems every once in a while that copy to sent folder timeout occurs 
and you have to hit retry a few time for it to complete


I know this is a network issue (ie network speed, congestion etc does 
cause an issue)


That being said is there an imap timeout variable that would addres this 
issue and or does anyone know about a thunderbird variable


I have been unable to locate anything specific to this issue and am 
aware that we probably need to patch both sides?


ideas anyone ??

--


Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

Re: Is multi factor authentication practical/feasible?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Please note this is my opinion only

It seems any kind of dual auth will need a security app running on YOUR 
server saving toikens, logins etc etc


this is what lead to microsoft, gmail etc having their own api which 
will only work for them


this is also (mainly because of https authing the device) what makes it 
hard to proxy oauth2 etc


If you look at sogo's documentation they have a java server applet

Still working on the install to make work with my system but in general 
you need your own whatever app to track oauth2




5.7. Authenticating using C.A.S.
SOGo natively supports C.A.S. authentication. For activating C.A.S. 
authentication you need first
to make sure that the SOGoAuthenticationType setting is set to cas, 
SOGoXSRFValidationEnabled is
set to NO and that the SOGoCASServiceURL setting is configured 
appropriately.


I myself will eventually get around to implimenting this on one of my 
servers ?


logically i will have to track tokens etc via https like google etc

basically the reality is every server will have it's own token base etc 
thus preventing any kind of a standard.



Happy Sunday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 7/3/2022 9:50 AM, John Gateley wrote:




On 7/3/22 8:31 AM, John Gateley wrote:
The protocols were designed long before SAML and OIDC. SAML/OIDC give 
you more control over authn/z
and allow easily adding in MFA or other different types of auth. To do 
this right, you'd need to extend

the protocol to allow OIDC or SAML.


I did find this RFC - I haven't read it, but it applies directly:
https://datatracker.ietf.org/doc/html/rfc7628

j

Re: Multidomain ssl config ?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

John please send me a direct email address


I understand what you need and my customers are all seperate certs per domain 
on both sides


I spent over three months setting stuff up


I wil send complete instructions for both postfix & dovecot


Plus auto scripts etc


You will need to be running a postgresql database for my stuff to work without 
mods


And running python 2.xx


  
 thanks - paul 
   Paul Kudla  SCOM.CA Internet Services Inc.004-1009 Byron Street 
South   Whitby, Ontario - Canada   L1N 4S3Toronto   416.642.7266   Main   
1.866.411.7266   Fax   1.888.892.7266   

On Jun 29, 2022 at 16:39:29 EDT, John Stoffel  
wrote:

>>>>> "Maurizio" == Maurizio Caloro  writes:

Maurizio> on postfix now this seems to run, and with dovecot i need
Maurizio> also handle this two domains, but appairing this error
Maurizio> messages. like:

Why aren't you just using a single domain as the MX record for all the
domains? Then you only need one SSL cert pair for all of this, and if
you publish the right SPF records, each domain can send from the same
MX host as well.




Maurizio> Jun 29 20:49:28 Dovecot/imap-login: Info: Disconnected (no auth 
attempts in 0 secs): user=<>,
Maurizio> rip=a.b.c.d, lip=37.120.190.188, TLS handshaking: SSL_accept() 
failed: error:14094416:SSL routines:
Maurizio> ssl3_read_bytes:sslv3 alert certificate unknown: SSL alert number 46, 
session=

Maurizio> Running with Debian Buster

Maurizio> # dovecot --version
Maurizio> 2.3.4.1 (f79e8e7e4)

Maurizio> # nmail.caloro.ch
Maurizio> local_name nmail.caloro.ch {
Maurizio>  ssl_cert =   ssl_key =   }
Maurizio> # nmail.calm-ness.ch
Maurizio> local_name nmail.calm-ness.ch {
Maurizio>  ssl_cert =   ssl_key =   }

Maurizio> thanks for possible help




-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Re: Issue with one user only, exceeding connections

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok the idle connection per folder is a factor

however in thunderbird i believe it defaults to 2 simultanious connections

mine is set to 5

in thunderbird see

The solution is to reduce the maximum number of connections in 
Thunderbird. This can be done from Edit > Account Settings > Server 
Settings (under the mail account for which the setting should be 
modified) > Advanced > Maximum Number of server connections to cache.


I dont know of anything that would get it to 500?

as for outlook idle was not / is not supported past 2010 (if even that) 
you need to go into


file --> options --> advanced --> send/receive

all you can change in there is the timing which defaults to 30 minutes, 
i recommend 3 or 5


I am unaware of how outlook handles physical connections (maybe 
registery?) and google revieled nothing, outlook since 2010 just does 
not support imap, microsofts way of forcing everyone onto exchange / 
outlook 365


377,000 hits last time i googled imap issues in outlook.

Best suggestion is to run

# ps -axww | grep imap
25500  -  S  0:00.57 imap: [p...@hiscomputer.ca 172.97.150.95 
IDLE] (imap)
25530  -  S  0:00.36 imap: [p...@hiscomputer.ca 172.97.150.95 
IDLE] (imap)
26014  -  I  0:00.39 imap: [rco...@tnky.ca 172.97.128.227 IDLE] 
(imap)
26018  -  I  0:00.38 imap: [rco...@tnky.ca 172.97.128.227 IDLE] 
(imap)
26210  -  I  0:00.07 imap: [spa...@scom.ca 99.238.154.160 IDLE] 
(imap)
38911  -  S  0:00.17 imap: [marilynla...@scom.ca 142.188.149.199 
IDLE] (imap)
38912  -  S  0:00.13 imap: [marilynla...@scom.ca 142.188.149.199 
IDLE] (imap)
41306  -  S  0:00.73 imap: [ed.ha...@dssmgmt.com 204.237.48.37 
IDLE] (imap)
41312  -  S  0:00.63 imap: [ed.ha...@ekst.ca 204.237.48.37 IDLE] 
(imap)
45232  -  I  0:00.23 imap: [rco...@tnky.ca 172.97.128.227 IDLE] 
(imap)
55504  -  I  0:00.16 imap: [rco...@tnky.ca 172.97.128.227 IDLE] 
(imap)


which shows all imap connections and from where

if you are overflowing 500+ connections then it has to show up here.


Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 6/8/2022 6:41 PM, Jeremy Schaeffer wrote:
Ahhh, Ok, I did not know that and now that makes sense. I did not 
realize it held a open connection for each folder. I increased that and 
I will see what happens. I wonder if that will also effect the outlook 
issues. Thanks! - Jeremy


On 6/8/2022 14:28, Frank-Ulrich Sommer wrote:
I think if IMAP IDLE is used you need one connection per folder. If I 
remember correctly at least either Thunderbird or K9 Mail (I'm using 
both too) use one connection per selected directory. Simply increasing 
the number of connections was the easiest solution as I only have very 
few users too.


Regards
Frank

Am 8. Juni 2022 21:14:23 MESZ schrieb Jeremy Schaeffer 
:


I keep having this issue with one user, and I have to restart dovecot
several times a day to clear it. What I have is a postfix / dovecot mail
server (Centos 7) and about a dozen users. All mailboxes are imap ssl. I
monitor about 4 mailboxes on my computer and tablet. I use Thunderbird
on the computer (cache connections at 2) and K9 on the tablet, but one
user of the four I keep getting "Maximum number of connections from
user+IP exceeded" and I have the maximum at 50
"(mail_max_userip_connections=50)" so its hard for me to believe I am
actually exceeding it unless dovecot/client is not dropping connections
and keeps starting new ones until it reaches the maximum, but again,
only for one user, even though I am monitoring 4 on the same devices.
Any idea how to troubleshoot this? I don't know if I should be looking
at dovecot or the clients, or what I need to look for. It's been going
on since I put this server in use over a year ago. I also have issues
with Outlook clients disconnecting, just outlook, is there any
recommended settings to make Outlook work smoother?

Thanks! - Jeremy

Config -

# 2.2.36 (1f10bfa63): /etc/dovecot/dovecot.conf
# OS: Linux 3.10.0-1160.11.1.el7.x86_64 x86_64 CentOS Linux release
7.9.2009 (Core)
# Hostname: ***
auth_mechanisms = plain login
debug_log_path = /var/log/dovecot_debug.log
first_valid_gid = 500
last_valid_gid = 600
last_valid_uid = 600
listen = *
mail_location = maildir:~/Maildir
mbox_write_locks = fcntl
namespace inbox {
    inbox = yes
    location =
    mailbox Drafts {
      special_use = \Drafts
    }
    mailbox Junk {
      special_use = \Junk
    }
    mailbox Sent {
      special_use = \Sent
    }
    mailbox "Sent Messages" {
      special_use = \Sent
    }
    mailbox Trash {
      special_use = \Trash
  

Re: Occasional service disruptions

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok i do NOT use roundcube however trying to help

regarding :

> Jun 06 12:01:22 imap(user1)<29639>: Warning: Inotify
> instance limit for user 500 (UID vmail) exceeded, disabling. Increase
> /proc/sys/fs/inotify/max_user_instances

this to me indicates that all users are logging in under one user name / 
process at least according to dovecot?


usually when an imap connection is started it starts its own pid per 
mailbox opened process (at least for me it does)


both dovecot & cyrus work this way

number of simitanulus connections is usually handled by the client 
(thunderbird, outlook whatever)


roundcube would be considered a client thus the overflow in connections 
if it is opening everthing under one connection / user


it gets complicated but i would start by checking if different users are 
actually being logged in


if so then try closing the connection via roundqube and see if the 
connection drops off on the dovecot server.




# dovecot.who
username   # proto (pids)(ips) 

epo...@scom.ca 4 imap  (20263 74767 74743 75194) (174.114.171.16) 

install...@tomkudla.ca 7 imap  (28281 28280 69830 69832 69834 69836 
69838) (167.94.196.10) 

ditchb...@scom.ca 1 imap  (41136)   (65.39.148.2) 

recept...@clancyca.com 1 imap  (41133)   (65.39.148.2) 

e...@scom.ca 4 imap  (36344 25879 89306 89308) (204.237.48.37) 

rco...@tnky.ca 6 imap  (91131 23791 8700 16087 91176 91179) 
(172.97.128.227)
ca...@scom.ca  1 imap  (88120)   (216.58.34.142) 

p...@scom.ca   1 imap  (36202)   (69.60.225.80) 


ditchb...@clancyca.com 1 imap  (40942)   (65.39.148.2)

the max_user_instances is meant to control how many connections per user 
thus 500 is way more for multiple clients per user to log in with


i have several customers (like myself) that open connections from 
multiple locations without issues.


is it possible that roundcube is opening the same user multiple times 
and not closing the connection after a while?


I get the above logged in list above from

doveadm mailbox status -t all -u $1 '*'


something to consider.


Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 6/6/2022 11:38 AM, Nikolaos Milas wrote:


Hello,

On a server with (Postfix and) Dovecot 2.3.18 (on a VM running CentOS 7 
- 1 CPU, 5 GB RAM) with the config you will see below, we are facing 
occasional (infrequent) service disruptions: IMAP service seems 
unavailable to some users.


Jun  6 12:01:25 vweb2 roundcube: <1eecb0d4> IMAP Error: Login failed for 
imaptester against vmail2.noa.gr from 195.251.202.xxx. Could not connect 
to ssl://vmail2.noa.gr:993: Connection rejected in 
/var/webs/webmail/rcube/program/lib/Roundcube/rcube_imap.php on line 211 
(POST /?_task=login&_action=login)


At that time there was no associated logged event in dovecot log. (Other 
users are logging in and out.)


However, I see some warnings (I list the two of them closest to the 
above event):


Jun 06 12:01:22 imap(user1)<29639>: Warning: Inotify 
instance limit for user 500 (UID vmail) exceeded, disabling. Increase 
/proc/sys/fs/inotify/max_user_instances

...
Jun 06 12:01:26 imap(user2)<29793>: Warning: Inotify 
instance limit for user 500 (UID vmail) exceeded, disabling. Increase 
/proc/sys/fs/inotify/max_user_instances


(In above log excerpts I've only modified real usernames.)

Restarting Dovecot returns things back to normal.

I have tried to use "service_count = 100" in all configured services, to 
see how it goes.


Most of the config is inherited from the past (older versions) and is 
not optimized. For example one can observe different "process_limit" 
values for different services, for no apparent reason I am aware of.


Could anyone suggest changes and/or additions to the OS and/or Dovecot 
to resolve this issue?


Any additional suggestions will also be welcome.

Thanks in advance for your kind assistance.

Here is the config (I've only changed postmaster address):

===

protocols = imap pop3 sieve lmtp

login_greeting = Dovecot NOA ICXC-NIKA

log_path = /var/log/dove.log

mail_location = maildir:~/Maildir/

mail_gid = 500
mail_uid = 500

auth_mechanisms = plain login
auth_username_format = %Ln

auth_verbose = no
auth_debug = no
mail_debug = no

disable_plaintext_auth = no

mail_plugins = quota mail_log notify

protocol imap {
   imap_client_workarounds = "delay-newmail"
   mail_plugins = quota imap_quota mail_log notify
   mail_max_userip_connections = 400

   namespace inbox {
    mailbox Trash {
     autoexpunge = 15d
    }
   }
}

protocol pop3 {
   mail_max_userip_connections = 3
   mail_plugins = quota notify
   pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
   pop3_uidl_format

Re: Replicator: Panic: data stack: Out of memory

[Paul Kudla (SCOM.CA Internet Services Inc.)]

did you get this figured out / working

getting posts out of order - just wanted to make sure your ok?



Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 6/6/2022 4:03 AM, Ivan Jurišić wrote:


Dana 04.06.2022 15:34, Paul Kudla (SCOM.CA Internet Services Inc.) je 
napisao(la):

   ok thanks for the info
   from here you need to turn on full debugging and then filter the 
log by

   "replicat"


Now replication work when set vsz_limit in service aggregator and remove 
parametar replication_dsync_parameters and 
replication_full_sync_interval from my 90-replicator.conf. Now my 
configuration work for replication on another mail server.


Config file for replication /etc/dovecot/conf.d/90-replicator.conf
--

service aggregator {
   vsz_limit = 256M
   fifo_listener replication-notify-fifo {
     user = vmail
   }
   unix_listener replication-notify {
     user = vmail
   }
}

service replicator {
   process_min_avail = 1
   unix_listener replicator-doveadm {
     mode = 0600
     user = vmail
   }
}

service doveadm {
   inet_listener {
     port = 12345
     ssl = no
   }
}

replication_max_conns = 100
#replication_dsync_parameters = -d -N -l 30 -U
#replication_full_sync_interval = 1 days

doveadm_port = 12345
doveadm_password = Jados82!

plugin {
   mail_replica = tcp:imap.myserv2.local:12345
}

Re: Replicator: Panic: data stack: Out of memory

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok thanks for the info

from here you need to turn on full debugging and then filter the log by

"replicat"

It is starting to look like you have a bad email somewhere causing the crash

I have yet to see this however anything is possible, when you replicate 
anything it needs to update indexes & control files on the other side 
and vice versa


that being said its a good place to start looking?

At least see in the debug mail logs what it is trying to replicate and 
where it fails.


Im running a large email server and in 2.3.18 there was a bug that any 
account over 300+ folders would fail to replicate.


This appears to be fixed in 2.3.19

other then that TCP replication etc seems to work the best

if you look back on the mailing lists i have uploaded a sample config of 
my systems which seems to work ok.







Happy Saturday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 6/3/2022 10:46 PM, Ivan Jurišić wrote:

Ok a little more help :
vsz_limit = 0 --> means unlimited ram for allocation, change this/try 
2g etc pending avaliable ram.


I try with 524M, 1G, 2G, 4G and 8G but in any case repclicator proces 
got crash.





--
This message has been scanned for viruses and
dangerous content by *MailScanner* , and is
believed to be clean.

Re: Replicator: Panic: data stack: Out of memory

[Paul Kudla (SCOM.CA Internet Services Inc.)]

actually suggestion below is a good idea

run

ps -axww (or top)

to list active processes this will give you some hints

top is better for overall memory

i also have a perl script that will show actual memory useage, free etc

utilities like this are handy to have

also i found i had to set in dovecot.conf

default_process_limit = 16384

also are you running debug ?

auth_debug = no
auth_debug_passwords = no

mail_debug = no

ie set debug to = yes?

might give more detail if this is really a dovecot issue.

other background processes can eat memory

I run mailscanner for example and someone every one in a while tries to 
crash it!


it recovers but lord knows



mem outputs :

# mem

SYSTEM MEMORY SUMMARY:
mem_used:   16GB [ 12%] Logically used memory
mem_avail:   + 111GB [ 87%] Logically available memory
--  --- --
mem_total:   = 128GB [100%] Logically total memory

SYSTEM MEMORY INFORMATION:
mem_wire:   13GB [ 10%] Wired: disabled for paging out
mem_active:  +   0GB [  0%] Active: recently referenced
mem_inactive:+  71GB [ 57%] Inactive: recently not referenced
mem_cache:   +   0GB [  0%] Cached: almost avail. for allocation
mem_free:+  40GB [ 32%] Free: fully available for allocation
mem_gap_vm:  +   0GB [  0%] Memory gap: UNKNOWN
--  --- --
mem_all: = 124GB [100%] Total real memory managed
mem_gap_sys: +   3GBMemory gap: Kernel?!
--  ---
mem_phys:= 127GBTotal real memory available
mem_gap_hw:  +   0GBMemory gap: Segment Mappings?!
--  ---
mem_hw:  = 128GBTotal real memory installed


---
# cat /programs/common/mem
#!/usr/local/bin/perl
##
##  freebsd-memory -- List Total System Memory Usage
##  Copyright (c) 2003-2004 Ralf S. Engelschall 
##
##  Redistribution and use in source and binary forms, with or without
##  modification, are permitted provided that the following conditions
##  are met:
##  1. Redistributions of source code must retain the above copyright
## notice, this list of conditions and the following disclaimer.
##  2. Redistributions in binary form must reproduce the above copyright
## notice, this list of conditions and the following disclaimer in the
## documentation and/or other materials provided with the distribution.
##
##  THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
##  ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
##  IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 
PURPOSE

##  ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
##  FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
CONSEQUENTIAL

##  DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
##  OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
##  HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, 
STRICT
##  LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 
ANY WAY

##  OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
##  SUCH DAMAGE.
##

#   query the system through the generic sysctl(8) interface
#   (this does not require special priviledges)
my $sysctl = {};
my $sysctl_output = `/sbin/sysctl -a`;
foreach my $line (split(/\n/, $sysctl_output)) {
if ($line =~ m/^([^:]+):\s+(.+)\s*$/s) {
$sysctl->{$1} = $2;
}
}

#   round the physical memory size to the next power of two which is
#   reasonable for memory cards. We do this by first determining the
#   guessed memory card size under the assumption that usual computer
#   hardware has an average of a maximally eight memory cards installed
#   and those are usually of equal size.
sub mem_rounded {
my ($mem_size) = @_;
my $chip_size  = 1;
my $chip_guess = ($mem_size / 8) - 1;
while ($chip_guess != 0) {
$chip_guess >>= 1;
$chip_size  <<= 1;
}
my $mem_round = (int($mem_size / $chip_size) + 1) * $chip_size;
return $mem_round;
}

#   determine the individual known information
#   NOTICE: forget hw.usermem, it is just (hw.physmem - 
vm.stats.vm.v_wire_count).

#   NOTICE: forget vm.stats.misc.zero_page_count, it is just the subset of
#   vm.stats.vm.v_free_count which is already pre-zeroed.
my $mem_hw= &mem_rounded($sysctl->{"hw.physmem"});
my $mem_phys  = $sysctl->{"hw.physmem"};
my $mem_all   = $sysctl->{"vm.stats.vm.v_page_count"}  * 
$sysctl->{"hw.pagesize"};
my $mem_wire  = $sysctl->{"vm.stats.vm.v_wire_count"}  * 
$sysctl->{"hw.pagesize"};
my $mem_active= $sysctl->{"vm.stats.vm.v_active_count"}* 
$sysctl->{"hw.pagesize"};
my $mem_inactive  = $sysctl->{"vm.stats.vm.v_inactive_count"}  * 
$sysctl->{"hw.pagesize"};
my $mem_cache = $sysctl->{"vm.stats.vm

Re: Replicator: Panic: data stack: Out of memory

[Paul Kudla (SCOM.CA Internet Services Inc.)]

just an fyi domain is registered and appears to be active so there 
should not be any issues with the domain


.website is an actual domain (like .com, .ca etc)

however i did note

ZAKARIA.WEBSITE.14400   IN  MX  10 ZAKARIA.WEBSITE.

usually the mx record points to an actual sub domain like mail. or whatever

if you are running everything on one server then this is ok

its just usually better to seperate the mx record incase you want to 
goto a different server down the road.




Domain Name: ZAKARIA.WEBSITE
Registry Domain ID: D198561373-CNIC
Registrar WHOIS Server: whois.ionos.com
Registrar URL: https://ionos.com
Updated Date: 2021-11-02T01:42:25.0Z
Creation Date: 2020-08-29T09:28:59.0Z
Registry Expiry Date: 2022-08-29T23:59:59.0Z
Registrar: IONOS SE
Registrar IANA ID: 83
Domain Status: clientTransferProhibited 
https://icann.org/epp#clientTransferProhibited

Registrant Organization: 1&1 Internet Limited
Registrant State/Province: GLS
Registrant Country: GB
Registrant Email: Please query the RDDS service of the Registrar of 
Record identified in this output for information on how to contact the 
Registrant, Admin, or Tech contact of the queried domain name.
Admin Email: Please query the RDDS service of the Registrar of Record 
identified in this output for information on how to contact the 
Registrant, Admin, or Tech contact of the queried domain name.
Tech Email: Please query the RDDS service of the Registrar of Record 
identified in this output for information on how to contact the 
Registrant, Admin, or Tech contact of the queried domain name.

Name Server: NS1.ZAKARIA.WEBSITE
Name Server: NS2.ZAKARIA.WEBSITE

## nslookup ZAKARIA.WEBSITE
Server: 10.220.0.2
Address:10.220.0.2#53

Non-authoritative answer:
Name:   ZAKARIA.WEBSITE
Address: 213.171.210.111
Name:   ZAKARIA.WEBSITE
Address: 2a00:da00:1800:834c::1

## dig mx ZAKARIA.WEBSITE

; <<>> DiG 9.14.3 <<>> mx ZAKARIA.WEBSITE
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32110
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 3

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: ba2f2ec47dfcc90f458d629b4d2855567ad8dfa57bf8 (good)
;; QUESTION SECTION:
;ZAKARIA.WEBSITE.   IN  MX

;; ANSWER SECTION:
ZAKARIA.WEBSITE.14400   IN  MX  10 ZAKARIA.WEBSITE.

;; ADDITIONAL SECTION:
zakaria.website.14372   IN  A   213.171.210.111
zakaria.website.14372   IN  2a00:da00:1800:834c::1

;; Query time: 87 msec
;; SERVER: 10.220.0.2#53(10.220.0.2)
;; WHEN: Sat Jun 04 08:16:40 EDT 2022
;; MSG SIZE  rcvd: 147




Happy Saturday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 6/4/2022 6:07 AM, Marc wrote:


I think it is because of the domain, obviously you should only be running a 
website.


-Original Message-
From: dovecot  On Behalf Of
hi@zakaria.website
Sent: Saturday, 4 June 2022 11:15
To: Dovecot 
Subject: Re: Replicator: Panic: data stack: Out of memory

On 2022-06-04 02:46, Ivan Jurišić wrote:

Ok a little more help :
vsz_limit = 0 --> means unlimited ram for allocation, change
this/try 2g etc pending avaliable ram.


I try with 524M, 1G, 2G, 4G and 8G but in any case repclicator proces
got crash.


Maybe there is another service process causing OOM? e.g. check clamd,
antivirus DBs tend to be quite big and in updating for sometime becomes
double the size due to reloading.

Also, somtimes httpd service when using event worker, and its not tuned
properly, it will cause the OOM crash to other service along itself.

Re: Replicator: Panic: data stack: Out of memory

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok a little more help :

vsz_limit = 0 --> means unlimited ram for allocation, change this/try 2g 
etc pending avaliable ram.



I use :

service replicator {
  client_limit = 0
  drop_priv_before_exec = no
  idle_kill = 4294967295s
  process_limit = 1
  process_min_avail = 0
  service_count = 0
unix_listener replicator-doveadm {
mode = 0600
user = vmail
  }
  vsz_limit = 8192M
}







next :

replication_max_conns = 100

think of this as a socket that gets opened everytime you connect to the 
other server


100 means it will handle 100 seperate replication user requests

at a time (which is way more then sufficent)

if the connections overflows it just means the replication que will 
carry on when resources opens up thus not killing the server resources.


If a que overflows because too much other stuff is going on it will just 
reque and go again.


also note :

service aggregator {
  process_limit = 1000
  #vsz_limit = 1g
  fifo_listener replication-notify-fifo {
user = vmail
group = vmail
mode = 0666
  }

}







Happy Friday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 6/3/2022 5:13 AM, Ivan Jurišić wrote:


Reproduce of error (Replicator: Panic: data stack: Out of memory):
==

When replay on any mail I will got OOM.

I try add in service replicate vsz_limit = 0 but that not help (after 
proces dovecot/replicator eat all ram I got OOM)


Log error:
==

Jun 03 09:38:59 Warning: 
imap(ivan@myserv.local)<2533034>: 
replication(ivan@myserv.local): Sync failure: Timeout in 2 secs
Jun 03 09:39:03 Panic: replicator: data stack: Out of memory when 
allocating 4294967336 bytes
Jun 03 09:39:03 Error: replicator: Raw backtrace: 
/usr/lib/dovecot/libdovecot.so.0(backtrace_append+0x42) [0x7feef08f0582] 
-> /usr/lib/dovecot/libdovecot.so.0(backtrace_get+0x1e) [0x7feef08f069e] 
-> /usr/lib/dovecot/libdovecot.so.0(+0x1022fb) [0x7feef08fd2fb] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x102391) [0x7feef08fd391] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x55589) [0x7feef0850589] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x54d52) [0x7feef084fd52] -> 
/usr/lib/dovecot/libdovecot.so.0(+0xfb808) [0x7feef08f6808] -> 
/usr/lib/dovecot/libdovecot.so.0(+0x122938) [0x7feef091d938] -> 
/usr/lib/dovecot/libdovecot.so.0(+0xf76bf) [0x7feef08f26bf] -> 
/usr/lib/dovecot/libdovecot.so.0(+0xf777e) [0x7feef08f277e] -> 
/usr/lib/dovecot/libdovecot.so.0(buffer_append+0x61) [0x7feef08f2a21] -> 
dovecot/replicator(replicator_queue_push+0x15a) [0x55953ece9b7a] -> 
dovecot/replicator(+0x62b6) [0x55953ece92b6] -> 
dovecot/replicator(+0x594e) [0x55953ece894e] -> 
dovecot/replicator(+0x5b2c) [0x55953ece8b2c] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_call_io+0x69) [0x7feef0913529] 
-> /usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x132) 
[0x7feef0914c12] -> 
/usr/lib/dovecot/libdovecot.so.0(io_loop_handler_run+0x50) 
[0x7feef09135d0] -> /usr/lib/dovecot/libdovecot.so.0(io_loop_run+0x40) 
[0x7feef0913790] -> 
/usr/lib/dovecot/libdovecot.so.0(master_service_run+0x13) 
[0x7feef0886353] -> dovecot/replicator(main+0x18d) [0x55953ece7cbd] -> 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea) [0x7feef0651d0a] 
-> dovecot/replicator(_start+0x2a) [0x55953ece7d7a]
Jun 03 09:39:03 Fatal: replicator: master: service(replicator): child 
2532886 killed with signal 6 (core dumps disabled - 
https://dovecot.org/bugreport.html#coredumps)


My configuration:
==

# 2.3.19 (b3ad6004dc): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.19 (4eae2f79)
# OS: Linux 5.10.0-13-amd64 x86_64 Debian 11.3 ext4
# Hostname: mail.myserv.local
auth_mechanisms = plain login
auth_verbose = yes
debug_log_path = /var/log/dovecot.debug
default_client_limit = 5000
default_process_limit = 1000
default_vsz_limit = 2 G
deliver_log_format = msgid=%m, subject=%s, from=%f, size=%p(%w), %$
dict {
   quota = pgsql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
doveadm_password = # hidden, use -P to show it
hostname = myserv.local
lda_mailbox_autocreate = yes
lda_mailbox_autosubscribe = yes
listen = *
log_path = /var/log/dovecot.log
mail_home = /var/mail/vhosts/%d/mail/%n
mail_location = 
maildir:/var/mail/vhosts/%d/mail/%n:INDEX=/var/mail/vhosts/%d/indexes/%n

mail_plugins = " quota fts fts_solr notify replication"
mail_privileged_group = vmail
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date index ihave duplicate mime foreverypart 
extracttext imapsieve vnd.dovecot.imapsieve

namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     special_use = \Drafts
   }
   mailbox Junk {
     spec

Re: Building dovecot-core

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok

download the tarball for BOTH dovecot & pigeonhole

unzip / whatever into a build src dir

i use

/programs/src/mail

dovecot-2.3.19 for the core

dovecot-2.3-pigeonhole-0.5.19 for pigeonhole

make a currentconfig script file

containing

[13:57:42] peer1.scom.ca [paul:0] /programs/src/mail/dovecot-2.3.19.new
## cat currentconfig
./configure --with-pgsql --with-lucene


(i use pgsql & lucene for the search engine)


run the config (remember THIS IS THE SOURCE TARBALL NOT A BUILD)

ie --> ./configure --with-pgsql --with-lucene

gmake

gmake install


then make pigeonhole

same as above

currentconfig to contain

## cat currentconfig
./configure --with-dovecot=/usr/local/lib/dovecot

note you need to do a chmod +x to the currentconfig file in order to run it.

again like above

gmake
gmake install

thats the basics.





Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 6/2/2022 1:23 PM, dovecot-boun...@dovecot.org wrote:


On 2022-06-02 17:03, Dmitriy Fitisov wrote:

Hi all, do we have any info on building dovecot-core?
INSTALL.md
suggests usual way:
./configure
…

Needless to say, there is no .configure in root dir.
I think, instructions need to be updated to correct this.

Thank you.
Dmitriy

Hi there,

I think you need to run ./autogen.sh first.

Good luck.

Zakaria.

Re: Restrict IMAP login, but allow Postfix SASL

[Paul Kudla (SCOM.CA Internet Services Inc.)]

You really need to database your passwd auth

sasl supports pgsql / mysql

you can then alter the queries by selecting flags pending the access you 
want to allow


dovecot-pgsql.conf

password_query = SELECT username as user, password FROM email_users 
WHERE username = '%u' and password <> 'alias' and status = True and 
destination = '%u'


user_query = SELECT home, uid, gid FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


#iterate_query = SELECT user, password FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
status = True and alias_flag = False



postfix

smtpd.conf -> /usr/local/lib/sasl2/smtpd.conf




#Postygres
pwcheck_method: auxprop
mech_list: PLAIN LOGIN
auxprop_plugin: sql
sql_engine: pgsql
sql_hostnames: pg.scom.ca:5433
sql_database: scom_billing
sql_user: 
sql_passwd: 
sql_select: SELECT password FROM email_users WHERE username = '%u@%r' 
and password <> 'alias' and currentcount_bad < 30 and status = True


i use status to allow disabling the users ability to login but you can 
easily expand the select statement to include another flag for imap user(s)





Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 6/1/2022 7:40 AM, dovecot-boun...@dovecot.org wrote:


Hi,

we have a very simple user-/passdb (like passwd) to authenticate virtual IMAP 
users.
We also use this for Postfix authentication.  Nothing special.

But, we need to exclude some of the users from IMAP login.
This means, some users should be allowed to send mail via Postfix (submission) 
and therefore
authenticate via SASL against dovecot successfully, but they should not be 
allowed to login
to their IMAP mail box.

How could this be done?

Thanks!
-lutzn

Re: Force TCP socket disconnect on imap login failure?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

iptables (linux) & pf firewall (freebsd) do drop the packets immediately 
as the tables are updated.


I know this from experience as I use freebsd for the mail system's and 
my asterisk voip server use linux


At the end of the day the logging has to drive the updates, the only way 
to protect yourself against a brute force attack while it is happening 
is to have the logging trigger a direct ip table update in the background


It is my experience that this IS extremely system resource extensive 
(why i now run a seperate logging server)


even with dedicated hardware etc I found it impractical to try to do 
this in real time because by the time i hit the trigger, then updated 
the database and then updated pf firewall / iptables accordingly usually 
the connection was over anyways.


this issue also exists in postfix where their logging does not allow a 
signle line in syslog to indicate sasl user & ip address which makes it 
near impossible to track bad ip's / user logins. I ended up patching 
postfix sasl auth programming to add a combined line to track stuff like 
this.


In ALL cases the attack is usually over before you can do anything about 
it anyways.


Best to just plan for the future.

Below is a copy of the Auth penalty support which will help curve this 
issue but not stop it .


It seems to be a balanced approach, postfix carries similiar config's to 
acomplish the same thing.


---
from :

https://doc.dovecot.org/configuration_manual/authentication/auth_penalty/


Authentication penalty support

Dovecot anvil process tracks authentication penalties for different IPs 
to slow down brute force login attempts. The algorithm works by:


First auth failure reply will be delayed for 2 seconds (this 
happens even without auth penalty)


AUTH_PENALTY_INIT_SECS in src/auth/auth-penalty.h

The delay will be doubled for 4 -> 8 seconds, and then the upper 
limit of 15 seconds is reached.


AUTH_PENALTY_MAX_SECS and AUTH_PENALTY_MAX_PENALTY in 
src/auth/auth-penalty.h


If the IP is in login_trusted_networks (e.g. webmail), skip any 
authentication penalties


If the username+password combination is the same as one of the last 
10 login attempts, skip increasing authentication penalty.


CHECKSUM_VALUE_PTR_COUNT in src/anvil/penalty.c

The idea is that if a user has simply configured the password 
wrong, it shouldn’t keep increasing the delay.


The username+password is tracked as the CRC32 of them, so there 
is a small possibility of hash collisions


Problems:

It is still possible to do multiple auth lookups from the same IP 
in parallel.


For IPv6 it currently blocks the entire /48 block, which may or may 
not be what is wanted.


PENALTY_IPV6_MASK_BITS in auth-penalty.c

Authentication penalty tracking can be disabled completely with:

service anvil {
  unix_listener anvil-auth-penalty {
mode = 0
  }
}

Also you can have similar functionality with fail2ban.




Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/24/2022 9:55 PM, John Hardin wrote:


On Tue, 24 May 2022, Hippo Man wrote:

I have already been doing the following for the past year or so: as 
soon as
I detect (via my own, homegrown fail2ban-like log monitoring utility) 
what
I deem to be attempts to log in via imap or pop3 with a dictionary 
password

attack, I immediately do a DROP via iptables. Yes, this will block all
future connection attemps from the same host, but unfortunately, it 
doesn't

stop the following scenario, which regularly occurs on my server ...

* Hacker connects via imap or pop3 to my server.
* Hacker makes numerous login attempts one after the other with various
passwords, and without disconnecting in between attempts. I've seen 10 
and

more of these repeated attempts rapidly during a single imap or pop3
connection.

Simply using iptables to DROP or REJECT the connection does not prevent
those repeated login attempts during the original imap or pop3 session.
Again, this only prevents *future* connections via that host.


It should block all subsequent packets received from that IP address, 
immediately. An in-process connection would appear (to the client) to hang.


Either there is an ACCEPT rule for related traffic somewhere in the 
chain before your new DROP rule, which is matching first and allowing 
the existing connection's packets through, or your DROP rule is 
malformed and not actually matching the traffic.

Re: Force TCP socket disconnect on imap login failure?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

closing a socket can leave a process in an undertimed state pending how 
the code reacts


blocking in the background via iptables would just stop traffic and the 
process should die cleanly.


programming 101, network connections at best dont like the plug being 
pulled once they start to talk but if the connection just dies off then 
it is just a network timeout error with no real harm being done.


just a thought.



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/23/2022 9:25 PM, John Tulp wrote:


i googled a little, i was just curious about your question.

found a stackoverflow question which, answered, says that using gdb one
can close the fd, after using lsof to find it out.

oh, and your iptables command... you have the address aaa. etc with a
-d, i think you mean the source ip address of the connection, -s,
right ?

if you want, i can provide that link.



On Mon, 2022-05-23 at 17:16 -0400, Hippo Man wrote:

OOPS! I incorrectly copied and pasted the iptables command in my
previous message. Here is the correct iptables command:

iptables -I INPUT -p tcp -m multiport --destination-port 143,993 -d
aaa.bbb.ccc.ddd -j DROP


This command successfully blocks *future* connections to ports 143 and
993 from that IP address, but as I mentioned, it doesn't kill the
currently open connection.



--
  hippo...@gmail.com
  Take a hippopotamus to lunch today.




On Mon, May 23, 2022 at 4:54 PM Hippo Man  wrote:

 Thank you, but fail2ban doesn't do what I need. Here is
 why ...
 
 
 I have used fail2ban and also my own homegrown log monitor

 program for this purpose. In both cases, I can detect the
 failed imap logins and then cause the following command to be
 run ...
 
 
 iptables -I INPUT -p tcp --destination-port aaa.bbb.ccc.ddd -j

 DROP
 
 
 However, this does not drop connections that are existing and

 already open. It will only drop *future* connections from that
 IP address to port 143.
 
 
 
 This is why I want to kill the existing connection. Even after

 that "iptables" command is issued, the entity which is
 connected to the imap port can continue to send more and more
 imap commands.
 
 
 If I can drop the TCP connection as soon as an imap login

 fails and also issue that kind of "iptables" command, then the
 client would have to reconnect in order to retry other login
 attempts. Those future connections would then be successfully
 blocked by that iptables rule.
 
 
 And even if I issue a "tcpdrop" command instead of just the

 "iptables" command, it doesn't kill the already-open
 connection. It just force-blocks future connections.
 
 
 I'm thinking of patching the dovecot source code to create a

 personal version which immediately disconnects from the socket
 after login failure. Of course, I would prefer not to do that,
 if there is another way to accomplish this.
 
 
 
 --

  hippo...@gmail.com
  Take a hippopotamus to lunch today.
 
 
 
 
 On Mon, May 23, 2022 at 4:24 PM Jan Hugo Prins

  wrote:
 
 Look at fail2ban.

 Should be able to do that for you.
 
 Jan Hugo
 
 
 On 5/23/22 21:11, Lloyd Zusman wrote:
 
 > I'm running dovecot 2.2.13 under Debian 8.

 > I'd like to force an immediate TCP socket disconnect
 > after any imap login attempt that fails.
 >
 > Right now, if invalid credentials are supplied
 > during an imap login, the client can keep retrying
 > logins with different credentials. However, I want
 > to prevent that from occurring by causing the socket
 > connection to be closed as soon as there is any
 > failed login attempt.
 >
 > I haven't been able to find any dovecot
 > configuration setting which could control this
 > behavior, but I'm hoping that I just missed
 > something.
 >
 > Thank you very much for any suggestions.
 >
 >
 > --
 >  hippo...@gmail.com
 >  Take a hippopotamus to lunch today.
 >
 
 

Re: Force TCP socket disconnect on imap login failure?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

"-j REJECT --reject-with
tcp-reset" instead of DROP

are valid ideas

consider that if you update (in the background) and block the connection 
then dovecot (or any other process) should just block the traffic and 
timeout to close the connection anyways


fyi ??



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/24/2022 12:18 AM, Péter Márton wrote:


Just for clarification (this probably won't help achieve your primary
goal to reset the connections):
Iptables can block future connections _and_ stop existing connections
to receive (and send) packets (even the command you posted). What it
can't do is closing existing connections (sending a FIN).
If the example you show can not block existing connections you have
somewhere before the chain a RELATED, ESTABLISHED rule with ACCEPT as
target. This is a common mistake. Your fail2ban rules have to come
_before_ you check for related and established connections.

I never tested this, but you could try using "-j REJECT --reject-with
tcp-reset" instead of DROP. Then at least a RST would be sent.

Hippo Man  ezt írta (időpont: 2022. máj. 23., H, 23:17):


OOPS! I incorrectly copied and pasted the iptables command in my previous 
message. Here is the correct iptables command:

iptables -I INPUT -p tcp -m multiport --destination-port 143,993 -d 
aaa.bbb.ccc.ddd -j DROP

This command successfully blocks *future* connections to ports 143 and 993 from 
that IP address, but as I mentioned, it doesn't kill the currently open 
connection.

--
  hippo...@gmail.com
  Take a hippopotamus to lunch today.


On Mon, May 23, 2022 at 4:54 PM Hippo Man  wrote:


Thank you, but fail2ban doesn't do what I need. Here is why ...

I have used fail2ban and also my own homegrown log monitor program for this 
purpose. In both cases, I can detect the failed imap logins and then cause the 
following command to be run ...

iptables -I INPUT -p tcp --destination-port aaa.bbb.ccc.ddd -j DROP

However, this does not drop connections that are existing and already open. It 
will only drop *future* connections from that IP address to port 143.

This is why I want to kill the existing connection. Even after that "iptables" 
command is issued, the entity which is connected to the imap port can continue to send 
more and more imap commands.

If I can drop the TCP connection as soon as an imap login fails and also issue that kind 
of "iptables" command, then the client would have to reconnect in order to 
retry other login attempts. Those future connections would then be successfully blocked 
by that iptables rule.

And even if I issue a "tcpdrop" command instead of just the "iptables" command, 
it doesn't kill the already-open connection. It just force-blocks future connections.

I'm thinking of patching the dovecot source code to create a personal version 
which immediately disconnects from the socket after login failure. Of course, I 
would prefer not to do that, if there is another way to accomplish this.

--
  hippo...@gmail.com
  Take a hippopotamus to lunch today.


On Mon, May 23, 2022 at 4:24 PM Jan Hugo Prins  wrote:


Look at fail2ban.
Should be able to do that for you.

Jan Hugo


On 5/23/22 21:11, Lloyd Zusman wrote:

I'm running dovecot 2.2.13 under Debian 8.

I'd like to force an immediate TCP socket disconnect after any imap login 
attempt that fails.

Right now, if invalid credentials are supplied during an imap login, the client 
can keep retrying logins with different credentials. However, I want to prevent 
that from occurring by causing the socket connection to be closed as soon as 
there is any failed login attempt.

I haven't been able to find any dovecot configuration setting which could 
control this behavior, but I'm hoping that I just missed something.

Thank you very much for any suggestions.

--
  hippo...@gmail.com
  Take a hippopotamus to lunch today.

Re: doveadm mailbox status -u p...@scom.ca -t all (syntax)

[Paul Kudla (SCOM.CA Internet Services Inc.)]

yes did thanks

doveadm [-f formatter] mailbox status [-A|-u user|-F file] [-S 
socket_path] [-t] fields mailbox ...


My apologies for being so stupid, in the cyrus days (myhead is still 
stuck there) the mbox name was also the username


I am still seperating that in my head.

Noted for future 



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/24/2022 4:51 AM, Aki Tuomi wrote:


You might want to add `INBOX` to the command?

Aki


On 24/05/2022 00:34 Paul Kudla (SCOM.CA Internet Services Inc.)  
wrote:

  
Ok I know this is a general question and at my level I should be able to

figure this out

all i want is the complete status of an inbox ?





# doveadm mailbox status -u p...@scom.ca -t all

keeps returning 


doveadm mailbox status [-u |-A] [-S ] 
 [...]

ie no specific error.

and no status answer

just looking for default info ?


--


Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

Re: Force TCP socket disconnect on imap login failure?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

for what its worth this is a python script that i use for the database 
driven iptables updater for my asterisk server


again same ideas but it gets the job done.

It's a lot of work to get stuff like this going but may help point 
someone in the right directions balance wise pending on there system / 
network setup.


The django script is intelligent as it looks at the ip addresses already 
blacklisted and updates the list adding or subtracting ip address 
changes within the database


can answer in more detail, mainly for reference.

example iptables output :

# /sbin/iptables -L INPUT -n | more
Chain INPUT (policy ACCEPT)
target prot opt source   destination
ACCEPT all  --  92.204.135.144   0.0.0.0/0
ACCEPT all  --  104.205.0.0/16   0.0.0.0/0
ACCEPT all  --  174.95.0.0/160.0.0.0/0
ACCEPT all  --  174.94.0.0/160.0.0.0/0
ACCEPT all  --  174.93.0.0/160.0.0.0/0
ACCEPT all  --  174.92.0.0/160.0.0.0/0
ACCEPT all  --  174.91.0.0/160.0.0.0/0
ACCEPT all  --  174.90.0.0/160.0.0.0/0
ACCEPT all  --  174.89.0.0/160.0.0.0/0
ACCEPT all  --  174.88.0.0/160.0.0.0/0
ACCEPT all  --  209.171.88.0/24  0.0.0.0/0
ACCEPT all  --  72.12.174.2300.0.0.0/0
ACCEPT all  --  72.136.0.0/160.0.0.0/0
ACCEPT all  --  10.0.0.0/8   0.0.0.0/0
ACCEPT all  --  67.171.153.140   0.0.0.0/0
ACCEPT all  --  99.235.148.110   0.0.0.0/0
ACCEPT all  --  67.69.69.0/240.0.0.0/0
ACCEPT all  --  204.237.0.0/16   0.0.0.0/0
ACCEPT all  --  65.39.148.0/25   0.0.0.0/0
ACCEPT all  --  72.143.119.178   0.0.0.0/0
ACCEPT all  --  99.244.67.2440.0.0.0/0
ACCEPT all  --  69.60.225.80 0.0.0.0/0
ACCEPT all  --  198.200.68.0/24  0.0.0.0/0
ACCEPT all  --  185.58.85.0/24   0.0.0.0/0
ACCEPT all  --  172.97.0.0/160.0.0.0/0
ACCEPT all  --  184.151.0.0/16   0.0.0.0/0
DROP   tcp  --  0.0.0.0/00.0.0.0/0   tcp dpt:5038
DROP   tcp  --  0.0.0.0/00.0.0.0/0   tcp dpt:80
DROP   all  --  213.175.208.0/24 0.0.0.0/0
DROP   all  --  50.24.0.0/24 0.0.0.0/0
DROP   all  --  20.98.78.0/240.0.0.0/0
DROP   all  --  116.106.197.0/24 0.0.0.0/0
DROP   all  --  45.95.169.0/24   0.0.0.0/0
DROP   all  --  193.253.211.0/24 0.0.0.0/0
DROP   all  --  65.49.20.0/240.0.0.0/0
DROP   all  --  107.189.1.0/24   0.0.0.0/0
DROP   all  --  107.189.3.0/24   0.0.0.0/0
DROP   all  --  209.141.51.0/24  0.0.0.0/0
DROP   all  --  75.119.155.0/24  0.0.0.0/0
DROP   all  --  45.133.1.0/240.0.0.0/0
DROP   all  --  185.166.84.0/24  0.0.0.0/0
DROP   all  --  116.105.218.0/24 0.0.0.0/0
DROP   all  --  216.37.36.0/24   0.0.0.0/0
DROP   all  --  216.245.220.0/24 0.0.0.0/0
DROP   all  --  205.185.121.0/24 0.0.0.0/0


based on django model(s)

#IP Blacklistings   
class IpBlock(models.Model):
id  = models.AutoField(primary_key=True)
	ipaddress 		= models.CharField(verbose_name='IP Address', 
max_length=40, null=True, blank=False,unique=False)
	action  	= models.CharField(max_length=15, 
choices=ip_action_choices, verbose_name='Firewall', default = 'D', 
null=True, blank=True)
	syslog			= models.TextField(verbose_name='Last Syslog', 
max_length=1000, null=True, blank=True, default = '')
	whois			= models.TextField(verbose_name='Whois', max_length=1500, 
null=True, blank=True, default = '')

asterisk= models.BooleanField('Asterisk', default = 
False )
	last_datetime	= models.DateTimeField(verbose_name='Date Last Updated 
Server', null=True, blank=True, default = timezone.now)
	accountid	= models.ForeignKey(Contacts,verbose_name='Reference', 
default = '2594',null=False, blank=True,related_name = 'blacklist_soldto')
	syslog2			= models.TextField(verbose_name='Last Syslog', 
max_length=1000, null=True, blank=True, default = 'Denied due to 
Unauthorized Use')
	last_program	= models.CharField(verbose_name='Last Program', 
max_length=20, null=True, blank=True, default = '')




class Meta:
ordering = ['ipaddress',]
db_table = u'blocked_ip'
verbose_name = u"Currently Blocked IP's"
verbose_name_plural = u"Currently Blocked Ip's"

class IpCount(models.Model):
	ipaddress = models.GenericIPAddressField(verbose_name='IP Address', 
max_length=17,blank=False,primary_key=True, unique=True)
	counthour   			= models.IntegerField(verbose_name='Current IP Count 
This Hour', null=True, blank=True, default='0')
	counttotal  			= models.IntegerField(verbose_name='Total IP Count This 
Month', null=True, blank=True, default='0')
	asterisk_counthour   	= models.IntegerField(verbose_name='Asterisk 

Re: doveadm mailbox status -u p...@scom.ca -t all (syntax)

[Paul Kudla (SCOM.CA Internet Services Inc.)]

thanks




Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/23/2022 6:40 PM, Patrick Domack wrote:

doveadm mailbox status -t all -u 'p...@scom.ca' '*'

Re: doveadm mailbox status -u p...@scom.ca -t all (syntax)

[Paul Kudla (SCOM.CA Internet Services Inc.)]

&

# doveadm mailbox status -t all -u 'p...@scom.ca' 'p...@scom.ca'


doveadm(p...@scom.ca): Error: Mailbox p...@scom.ca: Failed to lookup


mailbox status: Character not allowed in mailbox name: '.'
messages=0 recent=0 unseen=0 vsize=0

&

# doveadm mailbox status -t all 'p...@scom.ca'

doveadm(root): Error: Couldn't drop privileges: User is missing UID (see 
mail_uid setting)



sorry just can not figure it out??






Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/23/2022 5:34 PM, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:



Ok I know this is a general question and at my level I should be able to 
figure this out


all i want is the complete status of an inbox ?





# doveadm mailbox status -u p...@scom.ca -t all

keeps returning 


doveadm mailbox status [-u |-A] [-S ]  
 [...]


ie no specific error.

and no status answer

just looking for default info ?

doveadm mailbox status -u p...@scom.ca -t all (syntax)

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok I know this is a general question and at my level I should be able to 
figure this out


all i want is the complete status of an inbox ?





# doveadm mailbox status -u p...@scom.ca -t all

keeps returning 


doveadm mailbox status [-u |-A] [-S ]  
 [...]


ie no specific error.

and no status answer

just looking for default info ?


--


Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

Re: Force TCP socket disconnect on imap login failure?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok would like to help and I DO understand the issue at hand.

It took me with custom programming 5 years of tinkering to get to what i 
have today.


So a few questions / comments ??

Are you running an sql database or just user-db (local) to do the lookups ?

sql gives you some major flexability on how to track and ignore ip 
blocks etc etc etc upon logins.


Issues to consider (I will explain the system i wrote for SCOM.CA as I 
went though a lot to get here and you like most are probably 
experiencing the same hack attempts.)


Note I am not trying to get off topic but you need to think about stuff 
like this through a bit.


You also have to consider people trying to send through postfix as it is 
the same issue in reverse.


Under SCOM I do the following (only meant to be a guideline):

1. i have a common syslogger running that ALL logging goes through.

2. I then have conditions against anything being logged that will 
trigger an event.


3. The event triggered in your case would be seeing a line like

   auth: sql(t...@dereilanatureinn.ca,220.194.140.110,
   <5H72HLPfTp/cwoxu>): unknown user

   pop3-login: Disconnected: Connection closed (auth failed, 1 attempts 


   in 3 secs): user=, method=PLAIN, rip=110.44.124.224,
   lip=65.39.148.18



there are other conditions but you get the idea.

Ok from here it starts getting complicated.

The idea is to keep the rift raft out and allow good users in.

Easier said then done.

I track all bad logins from all bad ip addresses and then run a seperate 
database table that tracks that.


the ip address that gets tracked lands in two places,

the firewall tables (which for me are global)

&

the user in my database.

if a user is unknown (example above) then at least the first condition 
will catch a bad hack attempt based on ip. Most hack attempts by ip 
address usually keep sending common login names (like admin, ftp, 
ftpuser etc etc) hoping to match to a common account.


for the user (which is relative here) i let 30 attempts go by every 30 
minutes and 500 per month, after which the user is blocked via auth in 
sql and will have to call to get unlocked. Usually the ip address is 
blacklisted before the username is so its not that much of a deal.


afterwhich it becomes part of the user query to lock out the userfrom 
anywhere as they are obviously getting hacked.


when an ip is doing the hacking then i count using the same formula and 
then blacklist it internal to all of my servers (thus a database makes 
it earier to track)


I run freebsd and thus use pf firewall, iptables can do the same with 
the same info you just need to build the tables and uodate them. (i 
update mine every 10 minutes, i find pf does this quicker on large 
lables (like 10,000 blocked) )



Now for the issue at hand that you are asking about:

I am sure that the c programming could be patched along the line to do 
exactly what you are asking,


However Issues that pop up.

so you hang up on the connection, they will probably just login again 
anyways which means without tracking the ip & username stats and 
updating accordingly it will really not change anything at the end of 
the day.


In my experience I see people / servers etc constantly hacking my side 
and what i generally described above turned out to be the only real fix, 
and not even really that guarenteed to work!


I do get ip's that get blacklisted by accident (i do the whole class 'c' 
as the 'c' block is usually all the same guy) but i get a good one maybe 
every few months, usually when i block it there are not many complaints 
after that.



I know the above is complicated, fyi i track postfix's sasl auth's as 
well but in that case to get a username & ip address on one syslog line 
i had to patch the sasl auth c file to get a log entry that was useable.


Postfix simply will NOT provide the info on one line.

Between both of the conditions above the server's remain fairly useable 
and secured.


the CSF firewall option below IS valid but i find you need to track IP 
address & username or you end up blocking stuff you dont want to


also on another note IPV6 (at least in canada) is becoming a pain for isp's

many cable companies, dsl providers etc are assigning an ipv6 address 
and then converting it to ipv4 on the way out the door from their 
networks using double natting ?


Issue is you can have 10,000 people all sharing that same ip address and 
if you block it then that will prevent other 'good' people from logging 
in, again back to tracking the username in this case gives you an out if 
the hacker is just using a list obtained elsewhere on the net.



Food for thought.


Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/23/2022 3:26 PM, dovecot-boun...@dovecot.org wrote:


On 2022-05-23 20:11, Lloyd Zusman wrote:

Many Thanks to the Programmers

[Paul Kudla (SCOM.CA Internet Services Inc.)]

I am still in testing mode with 2.3.19 but the replication with the 
folders over 300+ seems to be fixed


everything synced ok on its own within 30 minutes

Again MUSH APPRECIATED !

Moving to 2.3.19 was woth it!

--


Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

Re: mail_replica in userdb?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

So are you ok now

I'm a postgresql kinda guy never got around to ldap as that was mostly 
windows based!


fyi



Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/23/2022 11:18 AM, Coy Hile wrote:




On May 23, 2022, at 11:00 AM, Paul Kudla (SCOM.CA Internet Services Inc.) 
 wrote:


ok there is more to replica setup

user-db error seems to indicate the sql iterate it not setup or not returning 
stuff properly

can you specify what you are using for a sql setup (if any)

can help from there?




Ah, on the new side, I am using LDAP, rather than SQL, setup thusly in 
dovecot-ldap.conf.ext (once I removed all the comments and blank lines):

  '/^$/d'
hosts = ldap.coyhile.com
dn = uid=dovecotquery,ou=people,dc=coyhile,dc=com
dnpass = [redacted]
base = ou=People,dc=coyhile,dc=com
scope = subtree
user_filter = (&(objectClass=posixAccount)(uid=%n))
pass_attrs = uid=%n,userPassword=password
pass_filter = (&(objectClass=posixAccount)(uid=%n))
iterate_attrs = maildrop=user
iterate_filter = (objectClass=posixAccount)


--
Coy Hile
coy.h...@coyhile.com

Re: mail_replica in userdb?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

my apologies i read through the other posts after responding

you are using ldap ?




Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/23/2022 11:00 AM, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:



ok there is more to replica setup

user-db error seems to indicate the sql iterate it not setup or not 
returning stuff properly


can you specify what you are using for a sql setup (if any)

can help from there?




Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/23/2022 3:20 AM, Aki Tuomi wrote:




On 22/05/2022 02:32 Coy Hile  wrote:

I seem to have got replication working from an older instance to a 
newer one (insofar as the data are present on the new box) that I’m 
looking to stand up to replace a newer box. In this case, nothing is 
configured to talk to the new machine yet, so the replication is 
unidirectional (old -> new).


However, I’m seeing an error like like this repeatedly in syslog:

2022-05-21T23:23:15.295252+00:00 basement-imap01 dovecot: 
doveadm(h...@coyhile.com): Error: sync: User has no mail_replica in 
userdb


What setting did I miss?



plugin {
   mail_replica = 
}

https://doc.dovecot.org/settings/plugin/replication-plugin/

Aki

Re: mail_replica in userdb?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok there is more to replica setup

user-db error seems to indicate the sql iterate it not setup or not 
returning stuff properly


can you specify what you are using for a sql setup (if any)

can help from there?




Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266
Email p...@scom.ca

On 5/23/2022 3:20 AM, Aki Tuomi wrote:




On 22/05/2022 02:32 Coy Hile  wrote:

  
I seem to have got replication working from an older instance to a newer one (insofar as the data are present on the new box) that I’m looking to stand up to replace a newer box. In this case, nothing is configured to talk to the new machine yet, so the replication is unidirectional (old -> new).


However, I’m seeing an error like like this repeatedly in syslog:

2022-05-21T23:23:15.295252+00:00 basement-imap01 dovecot: 
doveadm(h...@coyhile.com): Error: sync: User has no mail_replica in userdb

What setting did I miss?



plugin {
   mail_replica = 
}

https://doc.dovecot.org/settings/plugin/replication-plugin/

Aki

Re: Duplicate messages if message is moved when using dsync

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok - question

when you say dsync are you running that manually or as a cronjob ??

or using replication via dsync

there is a difference

replication is designed to handle mailbox changes by quing the changes 
on the fly and then update in the background.


replication tracks all changes in a mailbox and then sends them one at a 
time to the target server.


i run replication with tcp (not tcps or dsync)

I find (other then the 300+folder issues) that it runs pretty good in 
real time.


if you are running dsync manually then yes timing on its own will be a 
hit and miss and it will never sync properly if something appears during 
the dsync process.


if you have the sive running as discussed that is the first step

you should setup replication so things are handled in real time.

pls advise if you have already done replication or just manually dsyncing.

Below is basic replication setup but you need to make sure various 
pre-requirements are in place a head of time.


1. you need / preferably use a database for your user auth

   you need to setup an iterate sql statement that returns the user
   list of users to be replicated

Example (i use pgsql) :

# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql password=
default_pass_scheme = PLAIN

password_query = SELECT username as user, password FROM email_users 
WHERE username = '%u' and password <> 'alias' and status = True and 
destination = '%u'


user_query = SELECT home, uid, gid FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
status = True and alias_flag = False


#iterate_query  above is what returns the user lists

2. run the command doveadm user '*' - it needs to return a list of users 
that will trigger replication (after doing config above)


3. in general here is my relative parts for tcp replication

In dovecot.conf

---

mail_plugins = " virtual notify replication fts fts_lucene "

replication_dsync_parameters = -d -N -l 300 -U

dsync_remote_cmd = ssh -l%{login} %{host} doveadm dsync-server -u%u

plugin {

.
  mail_replica = tcp:10.221.0.19:12345

#(note this is the ip address of the other server)
#it is the only config change ie server one above
#server2 would be the other server for communicating.



.

}

protocol imap {
  mail_max_userip_connections = 50
  mail_plugins = $mail_plugins notify replication
}

protocol pop3 {
  mail_max_userip_connections = 50
  mail_plugins = $mail_plugins notify replication
}

protocol imaps {
  mail_max_userip_connections = 25
  mail_plugins = $mail_plugins notify replication
}

protocol pop3s {
  mail_max_userip_connections = 25
  mail_plugins = $mail_plugins notify replication
}

replication_max_conns = 100 (optional)

replication_full_sync_interval = 1d

service replicator {
  client_limit = 0
  drop_priv_before_exec = no
  idle_kill = 4294967295s
  process_limit = 1
  process_min_avail = 0
  service_count = 0
  vsz_limit = 8g
unix_listener replicator-doveadm {
mode = 0600
user = vmail
  }
  vsz_limit = 8192M
}

service aggregator {
  process_limit = 1000
  #vsz_limit = 1g
  fifo_listener replication-notify-fifo {
user = vmail
group = vmail
mode = 0666
  }

}





that should do it?












Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/18/2022 3:39 PM, Thom Pol wrote:
Never mind. It did not work, issue still exists. I guess my network was 
a little slower when I thought it worked, so the emails were moved by my 
local client only after the sync had already happened between the servers.


So, back to the original message. Anyone any idea how this moving on 
local client while the remote servers are still running the dsync can 
work correctly without getting a duplicate?


Kind regards,

Thom Pol



On 18/05/2022 16:24, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:

Your Welcome



Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services<http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main?1.866.411.7266
Fax?1.888.892.7266


--
This message has been scanned for viruses and
dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
believed to be clean.

Upgrading from 2.3.18 --> 2.3.19

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok currently running 2.3.18

Would like to upgrade to  2.3.19 (to keep current)

Since these are production servers can i down grade back to 2.3.18 if 
nessesary?


Issues at hand is there seems to be conflicting messages about the 
replication being fixed or not ?


so would like to basically try 2.3.19 to see if it fixes some of my 
issues (large folder counts etc failing to replicate properly)


If it does or stays the same then i am ok with that for now.

if 2.3.19 causes issues can i go back to 2.3.18 as i already have 
scripts etc sorting out the sync issues?


pls advise



--


Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

Re: dovecot Digest, Vol 229, Issue 23

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Your Welcome



Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/18/2022 9:04 AM, Thom Pol wrote:
I must say I was a bit sceptical if this sieve script would resolve it, 
but it seems like it totally did! Thanks a lot for your reply, hope this 
helps someone else in the future as well.


Kind regards,

Thom Pol



On 16/05/2022 08:12, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:

ok duplicsate emails (even across dsync, replication etc) is typically
handled via a global sieve script


I use :

# cat duplicates.sieve
require "duplicate";   # for dovecot >= 2.2.18

if duplicate {
  discard;
  stop;
}

for the scripts

and setup sieve to work via my dovecot.conf file

relative parts below :



protocols = imap pop3 lmtp sieve

protocol lmtp {
mail_plugins = $mail_plugins sieve
postmaster_address =moni...@scom.ca
}


protocol lda {
mail_plugins = $mail_plugins sieve
}


plugin {
.

sieve =file:~/sieve;active=~/sieve/.dovecot.sieve

sieve_duplicate_default_period = 1h
sieve_duplicate_max_period = 1d
sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
sieve_global_dir = /usr/local/etc/dovecot/sieve
sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve

.
}

service managesieve-login {
process_limit = 1000
vsz_limit = 1g
inet_listener sieve {
  port = 4190
}
}

protocol sieve {
managesieve_implementation_string = Dovecot Pigeonhole
managesieve_max_line_length = 65536
}



--


note the sieve_before which handles duplictes during delivery etc.


Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services<http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main?1.866.411.7266
Fax?1.888.892.7266


--
This message has been scanned for viruses and
dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
believed to be clean.

Re: Can sync/migrate all mail from remote imap account except the main "INBOX"

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Assuming that the question at hand is :

My best guess at this point is the logs showing "INBOX" and 
"INBOX.INBOX" , perhaps there's something about the naming scheme that 
is throwing it off, but the "INBOX.INBOX.Drafts" and such are still 
being handled.


Thanks again for any direction!
Darren

I went through this when migrating from syrus imap

typically when dealing with special folders, cyrus liked doing everything

INBOX.Drafts
INBOX.Set

etc 

issue is it seems dovecot (only by experience) would prefer special 
folders in the ROOT of the mail folder.


to acomplish this I used this in the dovecot.conf file

-
namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Sent {
auto = subscribe
special_use = \Sent
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix =
  separator = /
}


-


However dovecot will pickup on the other folders which lead to updating 
the file subscriptions located in the mailbox folder of the user


What i had to do was add the namespaces

restart the dovecot

which will (or should) create extra special folders in the default locations

from there i had to move the emails manually from the old 
(INBOX.INBOX.Sent - for example) to the new folders


next issue is the mail client would not allow a delete of the old folder 
because it is though to be special.


in the subscriptions file (this directs what is avaliable to the client) 
i had to manually remove the entry and also delete the (now empty) dir 
for the old INBOX.INBOX.Sent (again for example)


you will probably have to do this for

Sent
Trash
Drafts

maybe junk??

I also noted (thunderbird for exanmple) that if you are running 
replication this has to be done manually on all servers.


After all is said and done trying deleting a message (should goto trash)

however note (again thunderbird) you may have to set the Trash & Sent 
folders to the new ones (why testing with deleting a message is 
nessesary to make sure stuff works correctly)


Same for the Sent

Hope this helps.





Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/13/2022 3:17 PM, Darren Mobley wrote:

Sure, thanks for the reply and suggestion [Smile]

To make testing/debugging easier, rather than using a remote provider, I 
created another domain and email account on the same server with just a 
few mails. I sent 3 jibberish mails to this new account, from this new 
account,  so they are showing in both INBOX and Sent, as well as 1 mail 
from root on the CLI (mail -v w...@hellodemo.ppl 
), These is also a draft mail saved in Drafts:


Source account file system layout:

# find /home/hellodemo/mail/hellodemo.ppl/wah/
/home/hellodemo/mail/hellodemo.ppl/wah/
/home/hellodemo/mail/hellodemo.ppl/wah/.Drafts
/home/hellodemo/mail/hellodemo.ppl/wah/.Drafts/cur
/home/hellodemo/mail/hellodemo.ppl/wah/.Drafts/cur/1652465670.M627726P3692.cent-7.darren.cpanel.net,S=341,W=353:2,S
/home/hellodemo/mail/hellodemo.ppl/wah/.Drafts/new
/home/hellodemo/mail/hellodemo.ppl/wah/.Drafts/tmp
/home/hellodemo/mail/hellodemo.ppl/wah/.Drafts/dovecot.index.log
/home/hellodemo/mail/hellodemo.ppl/wah/.Drafts/dovecot-uidlist
/home/hellodemo/mail/hellodemo.ppl/wah/.Drafts/dovecot.index.cache
/home/hellodemo/mail/hellodemo.ppl/wah/.Junk
/home/hellodemo/mail/hellodemo.ppl/wah/.Junk/cur
/home/hellodemo/mail/hellodemo.ppl/wah/.Junk/new
/home/hellodemo/mail/hellodemo.ppl/wah/.Junk/tmp
/home/hellodemo/mail/hellodemo.ppl/wah/.Junk/dovecot.index.log
/home/hellodemo/mail/hellodemo.ppl/wah/.Junk/dovecot-uidlist
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/cur
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/cur/1652465920.M799279P3808.cent-7.darren.cpanel.net,S=359,W=371:2,S
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/cur/1652465942.M469746P3832.cent-7.darren.cpanel.net,S=342,W=354:2,S
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/cur/1652465951.M751062P3906.cent-7.darren.cpanel.net,S=342,W=354:2,S
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/new
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/tmp
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/dovecot.index.log
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/dovecot-uidlist
/home/hellodemo/mail/hellodemo.ppl/wah/.Sent/dovecot.index.cache
/home/hellodemo/mail/hellodemo.ppl/wah/.Trash
/home/hellodemo/mail/hellodemo.ppl/wah/.Trash/cur
/home/hellodemo/mail/hellodemo.ppl/wah/.Trash/new
/home/hellodemo/mail/hellodemo.ppl/wah/.Trash/tmp
/home/hellodemo/mail/hellodemo.ppl/wah/.Trash/dovecot.index.log
/home/hellodemo/mail/hellodemo.ppl/wah/.Trash/dovecot-uidlist
/home/hellodemo/mail/hello

Re: TLS renegotiation issue (CVE-2011-1473) in Dovecot

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok need some more info but in general ssl setup should be as follows.

FQHN - do have have proper dns reverses setup? - this is an upstream thing

for example :

forwards :

## nslookup mail18.scom.ca
Server: 10.220.0.2
Address:10.220.0.2#53

Name:   mail18.scom.ca
Address: 65.39.148.18

reverses :

## nslookup 65.39.148.18
18.148.39.65.in-addr.arpa   name = sogo.scom.ca.
18.148.39.65.in-addr.arpa   name = mail18.scom.ca.
18.148.39.65.in-addr.arpa   name = ns2.scom.ca.
18.148.39.65.in-addr.arpa   name = mail.scom.ca.

Authoritative answers can be found from:

it needs to be understood that the reverses are usually returned by your 
upstream isp and should be set accordingly, ie you will have to get them 
to program them.


if you note above you can have several mappings for reverses

next ssl rewriting (other then sni) does simply not work so well.

also you should have a static ip (assuming you do)

mail18 is in my reverse so this error wont be thrown.

also note the server name (mail18.scom.ca) for both dovecot and postfix 
MUST match the certificate and dns for all to work.


ssl when running a masil server should be setup with a proper 
ceretificate (i use a wildcard for mine), proper forwards and proper 
reverses. Lets Encrypt (free ssl) is not a stable way to go on a busy 
server. You can typically get an ssl cert (proper one) for 10~20 us? 
pending on the provider of the cert.


also note this has to be setup properly on postfix as well as that to 
could throw a FQHN error if they are connecting to port 25/465/587 as well.


My ssl config (example) - please note i run sni for multiple domains and 
certs


i typically run with the dovecot defaults under 2.3.18 and it seems to 
work ok.



# cat sni.conf
#sni.conf
ssl = yes
verbose_ssl = yes
ssl_dh =  ssl_key = /programs/common/getssl.cert -c mail.hamletdevelopments.ca 
-q yes
  ssl_cert = /programs/common/getssl.cert -c mail.hamletdevelopments.ca 
-q yes
  ssl_ca = /programs/common/getssl.cert -c mail.hamletdevelopments.ca 
-q yes

}









Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/13/2022 10:38 PM, Elisamuel Resto wrote:


On 2022-05-13 5:02 pm, Greg Earle wrote:

Hello,

At work I'm running a Dovecot 2.3.15 server on a RHEL 7.9 system with 
OpenSSL 1.0.2k.


Our IT Security people are threatening to shut it down because of this:

We were notified of a possible TLS renegotiation vulnerability on 
[FQHN].


[Parent organization] ticket NNN is open to track efforts.

We conducted a manual test on the site for TLS Renegotiation on IMAP 
port 993.


We found that this was set to enabled.

In order to remediate we will need to either:

 1. Disable Renegotiation (preferred)
 2. Set a max aggregated renegotiation

Please remediate as soon as possible.

References:

https://support.f5.com/csp/article/K15278

https://nvd.nist.gov/vuln/detail/cve-2011-1473

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-1473


I did some Googling and among the results, I found a few old posts 
from this mailing list among them, which to summarize basically seemed 
to say "Yeah, we could write some code ... " but that was about it.


The IT Security rep sent me a reference to an ancient Red Hat article

https://access.redhat.com/articles/23543

which is hysterical - ancient history, references NSS and Tomcat, 
suggests changes to an add-on product (Red Hat Certificate Server) 
that is EOL, etc.


Is there any way to mitigate this issue?

(The only thing I can think of is to upgrade the Dovecot server to 
RHEL 8 and restrict connections to only TLSv1.3, but that ain't gonna 
happen overnight.)


Thanks,

    - Greg


Greg,

I believe this to be a configuration error, not a dovecot problem. The 
output of dovecot -n (as an attachment; look it over for any data you do 
not want publicized) would help to suggest changes to bring you back 
into compliance.



Regards,
Elisamuel Resto

Re: Fatal Error after upgrade to 2:2.3.19-2+debian11

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok the rights can be a bit confusing at times

assuming you are running virtual users (or not)

try these one at a time, i found that when dovecot starts it will adjust 
the permissions on the control files accordingly to what is set in the 
examples below, also note postfix can be a variable in this but would 
probably not be


I had to fiddle with stuff a lot

also dovecot i start in my rc.local (root startup)

the root user starts dovecot, it then changes everything rights wise as 
stated below and then changes to user dovecot (vmail whatever) to 
auctually start processing emails etc.


Again this is a pretty loose explanation but will point you in a 
direction for troubleshooting.



I typically use in dovecot.conf


-
service aggregator {
  process_limit = 1000
  #vsz_limit = 1g
  fifo_listener replication-notify-fifo {
user = vmail
group = vmail
mode = 0666
  }

}


service lmtp {
  process_limit=1000
  vsz_limit = 512m
  client_limit=1
   unix_listener /usr/home/postfix.local/private/dovecot-lmtp {
 group = postfix
 mode = 0600
 user = postfix
  }
}

service doveadm {
  process_limit = 0
  process_min_avail = 0
  idle_kill = 0
  client_limit = 1
  user = vmail
  inet_listener {
port = 12345
  }
}

service config {
  unix_listener config {
user = vmail
}
}

service anvil {
  process_limit = 1
  client_limit=5000
  vsz_limit = 512m
  unix_listener anvil {
group = vmail
mode = 0666
  }
}

service auth {
   process_limit = 1
   client_limit=5000
   vsz_limit = 1g

   unix_listener auth-userdb {
  mode = 0660
  user = vmail
  group = vmail
   }
   unix_listener /var/spool/postfix/private/auth {
  mode = 0666
   }

}

service stats {
  process_limit = 1000
  vsz_limit = 1g
  unix_listener stats-reader {
group = vmail
mode = 0666
  }
  unix_listener stats-writer {
group = vmail
mode = 0666
  }
}

-




Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/16/2022 8:09 AM, Sebastian Kroczek wrote:


Hello all,

I updated the server tonight and with it Dovecot from 2.2.27 to 
2:2.3.19-2+debian11. However, there seems to be a problem with the ACLs, 
because since then fatal errors are logged (see core dump). I suspect 
that some outdated configuration is causing this behavior, but so far I 
couldn't figure out which one it could be. I also have no clue right now 
how to debug this further.
Thank you very much for your help. If more information are needed, I 
will of course be happy to provide them.


VG
Sebastian


 Error logs =
May 16 13:33:43 Fatal: imap(us...@domain.com)<0r5YZR/fM4AfrHBI>: master: 
service(imap): child 238359 killed with signal 11 (core dumped)
May 16 13:33:46 Fatal: imap(us...@domain.com): master: 
service(imap): child 238386 killed with signal 11 (core dumped)
May 16 13:33:46 Fatal: imap(us...@domain.com): master: 
service(imap): child 238387 killed with signal 11 (core dumped)
May 16 13:34:54 Fatal: imap(us...@domain.com)<1WS6aR/fHoAfrHBI>: master: 
service(imap): child 238509 killed with signal 11 (core dumped)
May 16 13:34:54 Fatal: imap(us...@domain.com): master: 
service(imap): child 238508 killed with signal 11 (core dumped)
May 16 13:35:27 Fatal: imap(us...@domain.com): master: 
service(imap): child 238589 killed with signal 11 (core dumped)
May 16 13:35:27 Fatal: imap(us...@domain.com): master: 
service(imap): child 238590 killed with signal 11 (core dumped)

 END Error logs =

 dovecot.conf ==

# 2.3.19 (b3ad6004dc): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.19 (4eae2f79)
# OS: Linux 5.10.0-14-amd64 x86_64 Debian 11.3
# Hostname: wv-imap1.wavecloud.de
auth_mechanisms = plain login
default_vsz_limit = 4 G
dict {
   acl = mysql:/etc/dovecot/dovecot-dict-sql.conf
}
first_valid_gid = 5000
first_valid_uid = 5000
imap_capability = +XDOVECOT
last_valid_gid = 5000
last_valid_uid = 5000
listen = 10.10.115.XX
login_trusted_networks = 10.10.115.XX 10.10.115.XX
mail_location = maildir:~/
mail_log_prefix = "%s(%u)<%{session}>: "
mail_plugins = acl notify quota fts fts_solr virtual
maildir_stat_dirs = yes
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope 
encoded-character vacation subaddress comparator-i;ascii-numeric 
relational regex imap4flags copy include variables body enotify 
environment mailbox date ihave

namespace {
   inbox = yes
   location =
   mailbox Archive {
     auto = subscribe
     special_use = \Archive
   }
   mailbox Drafts {
     auto = subscribe
     special_use = \Drafts
   }
   mailbox Junk {
     special_use = \Junk
   }
   mailbox Sent {
     auto = subscribe
     special_use = \Sent
   }
   mailbox "Sen

Re: Duplicate messages if message is moved when using dsync

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok duplicsate emails (even across dsync, replication etc) is typically 
handled via a global sieve script



I use :

# cat duplicates.sieve
require "duplicate";   # for dovecot >= 2.2.18

if duplicate {
discard;
stop;
}

for the scripts

and setup sieve to work via my dovecot.conf file

relative parts below :



protocols = imap pop3 lmtp sieve

protocol lmtp {
  mail_plugins = $mail_plugins sieve
  postmaster_address = moni...@scom.ca
}


protocol lda {
  mail_plugins = $mail_plugins sieve
}


plugin {
.

  sieve = file:~/sieve;active=~/sieve/.dovecot.sieve

  sieve_duplicate_default_period = 1h
  sieve_duplicate_max_period = 1d
  sieve_extensions = +duplicate +notify +imapflags +vacation-seconds
  sieve_global_dir = /usr/local/etc/dovecot/sieve
  sieve_before = /usr/local/etc/dovecot/sieve/duplicates.sieve

.
}

service managesieve-login {
  process_limit = 1000
  vsz_limit = 1g
  inet_listener sieve {
port = 4190
  }
}

protocol sieve {
  managesieve_implementation_string = Dovecot Pigeonhole
  managesieve_max_line_length = 65536
}



--


note the sieve_before which handles duplictes during delivery etc.


Happy Monday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/15/2022 12:38 PM, Thom Pol wrote:

Hi,

Hope you are well.

We have a cluster of 2 Dovecot servers, both on v2.3.13 (89f716dc2), 
using dsync to sync the messages between them.


Previously, we used TCPS to sync the messages, but after some testing, 
we concluded that syncing over SSH resulted in a lot less failed syncs, 
so we started using SSH.


The change has been a success, but I now notice a issue when a email 
client immediately moves a messages to a separate folder while Dovecot 
is syncing, where the message is seen twice in the folder (with the 
exact same headers/content).


For example, I have set a filter in my email client, Thunderbird, to 
immediately move all emails coming from this list to a separate folder. 
When opening that folder, I do not see one, but two unread messages, 
both identical to each other.


When checking the directories on the server, I see this:
mx1:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M180050P1376677.mx1,S=19089,W=19384:2,S

mx2:
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615811.M981426P990530.mx2,S=19089,W=19384:2,S
/var/vmail/example.com/joe/Maildir/.Subdir.Subdir/cur/1652615808.M190190P990486.mx2,S=19089,W=19384:2,S

Note the difference: on mx1, one indicates mx1, and one mx2, while on 
the other server, both indicate mx2/


Any idea (other then telling end-users not to use such filters) how we 
could prevent these duplicate messages?


This is our config:
# Pigeonhole version 0.5.13 (cdd19fe3)
# OS: Linux 5.10.0-13-cloud-amd64 x86_64 Debian 11.3
# Hostname: mx1.example.com
auth_mechanisms = plain login
disable_plaintext_auth = no
dsync_remote_cmd = ssh -p 222 -l%{login} %{host} doveadm dsync-server -u%u
imap_capability = +SPECIAL-USE XLIST
listen = *,[::]
lmtp_rcpt_check_quota = yes
log_timestamp = "%Y-%m-%d %H:%M:%S "
mail_max_userip_connections = 100
mail_plugins = quota
mail_privileged_group = vmail
namespace inbox {
   inbox = yes
   location =
   mailbox Drafts {
     special_use = \Drafts
   }
   mailbox Junk {
     special_use = \Junk
   }
   mailbox Sent {
     special_use = \Sent
   }
   mailbox "Sent Messages" {
     special_use = \Sent
   }
   mailbox Trash {
     special_use = \Trash
   }
   prefix =
   separator = .
}
passdb {
   args = /etc/dovecot/dovecot-sql.conf
   driver = sql
}
plugin {
   mail_replica = remote:r...@mx2.example.com
   quota = dict:user::file:/var/vmail/%d/%n/.quotausage
   quota_status_nouser = DUNNO
   quota_status_overquota = 552 5.2.2 Mailbox is full
   quota_status_success = DUNNO
   sieve = /var/vmail/%d/%n/.sieve
   sieve_after = /var/vmail/%d/%n/.ispconfig.sieve
   sieve_before = /var/vmail/%d/%n/.ispconfig-before.sieve
   sieve_max_actions = 100
   sieve_max_redirects = 25
   sieve_max_script_size = 2M
}
protocols = imap pop3 lmtp
replication_max_conns = 50
service aggregator {
   fifo_listener replication-notify-fifo {
     mode = 0666
     user = vmail
   }
   unix_listener replication-notify {
     mode = 0666
     user = vmail
   }
}
service auth {
   unix_listener /var/spool/postfix/private/auth {
     group = postfix
     mode = 0660
     user = postfix
   }
   unix_listener auth-userdb {
     group = vmail
     mode = 0600
     user = vmail
   }
   user = root
}
service imap-login {
   client_limit = 1000
   process_limit = 512
}
service lmtp {
   unix_listener /var/spool/postfix/private

Re: Use different log files

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Robert's answer is a valid approach pending the size of your server 
networks etc.


on another note (because i run multiple servers etc)

I run a common syslog file across all servers which is what you appear 
to have now.


from there i like everything in one syslog because i am usually looking 
for something relative to a user which can occur anywhere. (imap, smtp, 
pop3, ssl etc)


that being said i wrote bash scripts that do stuff like

cat /var/log/syslog.log | grep $1

this allows everything from ALL servers going into one file for 
simplicity and then it gets seperated out when you go looking for something.


note that syslog can be programmed to divert to other servers in syslog.conf

## cat /etc/syslog.conf
*.* /var/log/all.log
*.* @10.228.0.6

10.228.0.6 is my central internal syslog capture server and all of my 
servers, routers, devices etc point to that and i go from there.


if you are having auth issues etc between dovecot & postfix this will 
show you everything related to a user, ip address etc.


Again its just a suggestion ... Logging is always relative to network 
setup more then anything else and situations vary easily.


I expanded this concept eventually into a database driven logger system 
in django, it is probably overkill for you but i am running 20+ servers 
and at the end of the day it was just easier to centralize it.


so

ssh 10.220.0.6 -q -tt /usr/home/syslog/log $1 $2 $3 $4 $5 $6 $7 $8 $9

or more spoecifically

log -t p...@hiscomputer.ca (-t was for today's date)

would give me all activity for my accounts



mail19  05-16 07:03:26 {smtphandler.py} [14475] (996976186) [14475] 
Header info data: 'hiscomputer...@em1.dereksloan.ca', 
['p...@hiscomputer.ca'] ((While
Handling File : 
/usr/home/postfix/tmp/936692CC6F0))
mail19  05-16 07:03:26 {smtphandler.py} [14475] (996976190) [14475] 
rSPF set : Received-SPF: Pass (sender SPF authorized) identity=mailfrom; 
client-
ip=167.89.21.76; 
helo=o24.email.nationbuilder.com; envelope-from=bounces+14632821-e4fc-


paul=hiscomputer...@em1.dereksloan.ca; receiver=p...@hiscomputer.ca \n 
((While Handling File :


/usr/home/postfix/tmp/936692CC6F0))
mail19  05-16 07:03:26 {smtphandler.py} [14475] (996976198) [14475] 
Checking for Spam SPF Conditions in rSPF : Received-SPF: Pass (sender 
SPF authorized)
identity=mailfrom; 
client-ip=167.89.21.76; helo=o24.email.nationbuilder.com; envelope-


from=bounces+14632821-e4fc-paul=hiscomputer...@em1.dereksloan.ca; 
receiver=p...@hiscomputer.ca \n ((While
Handling File : 
/usr/home/postfix/tmp/936692CC6F0))
mail19  05-16 07:03:26 {smtphandler.py} [14475] (996976200) [14475] 
processing TO: p...@hiscomputer.ca ((While Handling File :


/usr/home/postfix/tmp/936692CC6F0))
mail19  05-16 07:03:26 {smtphandler.py} [14475] (996976201) [14475] 
Checking if user p...@hiscomputer.ca has a mailbox ((While Handling File :


/usr/home/postfix/tmp/936692CC6F0))
mail19  05-16 07:03:26 {smtphandler.py} [14475] (996976202) [14475] 
SELECT * FROM email_users WHERE source = $$p...@hiscomputer.ca$$ ((While 
Handling File
: 
/usr/home/postfix/tmp/936692CC6F0))
mail19  05-16 07:03:28 {MailScanner}[11525] (996976259) Delivery 
of nonspam: message 936692CC6F0.AF475 from bounces+14632821-e4fc-


paul=hiscomputer...@em1.dereksloan.ca to p...@hiscomputer.ca with 
subject WHO take over!
mail19  05-16 07:03:42 {smtphandler.py} [14487] (996976373) [14487] 
Header info data: 'hiscomputer...@em1.dereksloan.ca', 
['p...@hiscomputer.ca'] ((While
Handling File : 
/usr/home/postfix/tmp/75A082CC6FE))
mail19  05-16 07:03:42 {smtphandler.py} [14487] (996976377) [14487] 
rSPF set : Received-SPF: Pass (sender SPF authorized) identity=mailfrom; 
client-
ip=167.89.21.76; 
helo=o24.email.nationbuilder.com; envelope-from=bounces+14632821-e4fc-


paul=hiscomputer...@em1.dereksloan.ca; receiver=p...@hiscomputer.ca \n 
((While Handling File :


/usr/home/postfix/tmp/75A082CC6FE))
mail19  05-16 07:03:42 {smtphandler.py} [14487] (996976385) [14487] 
Checking for Spam SPF Conditions in rSPF : Received-SPF: Pass (sender 
SPF authorized)
identity=mailfrom; 
client-ip=167.89.21.76; helo=o24.email.nationbuilder.com; envelope-


from=bounces+14632821-e4fc-paul=hiscomputer...@em1.dereksloan.ca; 
receiver=p...@hiscomputer.ca \n ((While
Handling File : 
/usr/home/postfix/tmp/75A082CC6FE))
mail19  05-16 07:03:42 {smtphandler.py} [14487] (996976387) [14487] 
processing TO: p...@hiscomputer.ca ((While

Re: how is the "Total number of known users" reported by doveadm replicator calculted?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

ok assuming that you are using a database for virtual users

(this is pgsql)

# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql 
password=x

default_pass_scheme = PLAIN

password_query = SELECT username as user, password FROM email_users 
WHERE username = '%u' and password <> 'alias' and status = True and 
destination = '%u'


user_query = SELECT home, uid, gid FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'



iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
status = True and alias_flag = False



iterate_query controls what is returned to doveadm user '*' and the 
replication service.


in the above example i only return active accounts and skip alias 
accounts (fyi)


adjust to your config.

if it is not setup propoerly then both functions will return bad info to 
the replicator


if the replicator picks up on a new user on its own it will add it to 
its own replicator db (see example below) - this is why you are getting 
different result count wise.


once the .db file is updated it will carry that user till re-created 
(see below)


to sync this you need to shut down all servers.

origionaly before i figured this out i had to delete the .db file on all 
servers, touch it (aka make a blank file) and then restart the servers


dovecot will set the rights automatically when you start it up again

best sugestion is the get the doveadm user '*' working first as it will 
be the base results.


see :

[17:05:03] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x  2 root  wheel  uarch4B May 13 10:52 .
drwxr-xr-x  4 root  wheel  uarch4B Mar  8  2021 ..
-rw-r--r--  1 root  wheel  uarch   73B May 13 10:52 instances
-rw-r--r--  1 root  wheel  uarch  161K May 13 16:50 replicator.db

replicator.db is a txt file that carries the current status of the 
replication service.


example entry (carries hash info etc that tells the replicator service 
what happens next - ie when to pull a sync) :


ditchb...@scom.ca   0   1651914641  1652433042 
1652433042  0 
AQAAALiUhhMjOhJiUHwAAM9Y3P8lOhJiAAEAJ8pBLXM8EmKfkQAAz1jc/yc6EmIEBgQAAABI1pY73z0SYp+RAADPWNz/KjoSYgABAJLdYwEjOhJiFCEBANHccmUlOhJiAAEAoEK+EyM6EmJQfAAAz1jc/yY6EmIAAQA9flMZFfIUYr7FAADR3HJlLDoSYgABAEZJQhMjOhJiUHwAAM9Y3P8jOhJiAAEAkErEKRnyFGK+xQAA0dxyZS06EmIAAQCYRnsA4D0SYp+RAADPWNz/KzoSYgABAHiN8xV7PBJin5EAAM9Y3P8oOhJiTQAAAFBNSIGABsI9EmKfkQAAz1jc/yk6EmIWGBYAAABX2T8x 
   1652433042




this db file generates the user.sync status etc

doveadm replicator status '*'






Happy Friday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/13/2022 4:26 PM, Arnaud Abélard wrote:

Hello,

I have a question regarding the "Total number of known users" displayed 
by doveadm replicator status. How is it calculated? Shouldn't it match 
the number of users reported by doveadm user '*' ?


I have 3 servers being replicated, the "total number of known users" a 
lot higher than the number of users reports by doveadm user '*' why is 
that?


Arnaud

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok update from my end

under 2.3.18 (have not upgraded production to 2.3.19 yet)

replication issues as stated before

however i need to note that i had to manually sync a user that was not 
being listed as a replicator fail


this means i have to force a full sync between servers on all accounts 
regardless of replication status


this was discovered this morning on a customers account that did not 
replicate between the servers properly and thus emails were being 
delivered days later because the client was accessing the other server.


its one thing to be 10 minutes late etc but a day late is not practical

again not complaining

I will load 2.3.19 on the test servers and try that and advise, also 
will test for the folder count replication issue as well and advise


please note NO errors are being thrown in the debug log, it reports the 
replication request, gets qued but does not complete??






Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/11/2022 12:25 AM, Cassidy B. Larson wrote:

Hi Aki,

We just installed 2.3.19, and are seeing a couple of users throwing the 
"INBOX/dovecot.index reset, view is now inconsistent" and their 
replicator status erroring out. Tried force-resync on the full mailbox, 
but to no avail just yet.  Not sure if this bug was supposedly fixed in 
2.3.19?


Thanks,

Cassidy

On Thu, Apr 28, 2022 at 5:02 AM Aki Tuomi > wrote:


2.3.19 is round the corner, so not long. I cannot yet promise an
exact date but hopefully within week or two.

Aki

 > On 28/04/2022 13:57 Paul Kudla (SCOM.CA  Internet
Services Inc.) mailto:p...@scom.ca>> wrote:
 >
 >
 > Thanks for the update.
 >
 > is this for both replication issues (folders +300 etc)
 >
 > Just Asking - Any ETA
 >
 >
 >
 >
 >
 > Happy Thursday !!!
 > Thanks - paul
 >
 > Paul Kudla
 >
 >
 > Scom.ca Internet Services >
 > 004-1009 Byron Street South
 > Whitby, Ontario - Canada
 > L1N 4S3
 >
 > Toronto 416.642.7266
 > Main 1.866.411.7266
 > Fax 1.888.892.7266
 >
 > On 4/27/2022 9:01 AM, Aki Tuomi wrote:
 > >
 > > Hi!
 > >
 > > This is probably going to get fixed in 2.3.19, this looks like
an issue we are already fixing.
 > >
 > > Aki
 > >
 > >> On 26/04/2022 16:38 Paul Kudla (SCOM.CA 
Internet Services Inc.) mailto:p...@scom.ca>> wrote:
 > >>
 > >>
 > >> Agreed there seems to be no way of posting these kinds of
issues to see
 > >> if they are even being addressed or even known about moving
forward on
 > >> new updates
 > >>
 > >> i read somewhere there is a new branch soming out but nothing
as of yet?
 > >>
 > >> 2.4 maybe 
 > >> 5.0 
 > >>
 > >> my previous replication issues (back in feb) went unanswered.
 > >>
 > >> not faulting anyone, but the developers do seem to be
disconnected from
 > >> issues as of late? or concentrating on other issues.
 > >>
 > >> I have no problem with support contracts for day to day maintence
 > >> however as a programmer myself they usually dont work as the
other end
 > >> relies on the latest source code anyways. Thus can not help.
 > >>
 > >> I am trying to take a part the replicator c programming based
on 2.3.18
 > >> as most of it does work to some extent.
 > >>
 > >> tcps just does not work (ie 600 seconds default in the c
programming)
 > >>
 > >> My thoughts are tcp works ok but fails when the replicator through
 > >> dsync-client.c when asked to return the folder list?
 > >>
 > >>
 > >> replicator-brain.c seems to control the overall process and
timing.
 > >>
 > >> replicator-queue.c seems to handle the que file that does seem
to carry
 > >> acurate info.
 > >>
 > >>
 > >> things in the source code are documented enough to figure this
out but i
 > >> am still going through all the related .h files documentation
wise which
 > >> are all over the place.
 > >>
 > >> there is no clear documentation on the .h lib files so i have
to walk
 > >> through the tree one at a time finding relative code.
 > >>
 > >> since the dsync from doveadm does see to work ok i have to
assume the
 > >> dsync-client used to compile the replicator is at fault
somehow or a
 > >> call from it upstream?
 > >>
 > >> Thanks for your input on the other issues noted below, i will
keep that
 > >> in mind when disassembling the source code.
 > >>
 > >> No sense in fixing one thing and leaving something else
behind

Re: IMAP connection lost while sending mail

[Paul Kudla (SCOM.CA Internet Services Inc.)]

   1 vmail  vmail  uarch  131K May  6 09:08 dovecot-uidlist
-rw---   1 vmail  vmail  uarch8B May  2 10:25 dovecot-uidvalidity
-r--r--r--   1 vmail  vmail  uarch0B Feb 19 17:58 
dovecot-uidvalidity.62117625

-rw---   1 vmail  vmail  uarch   45K May  6 08:41 dovecot.index
-rw---   1 vmail  vmail  uarch  1.8M May  6 10:16 dovecot.index.cache
-rw---   1 vmail  vmail  uarch   26K May  6 09:09 dovecot.index.log
-rw---   1 vmail  vmail  uarch   32K May  4 10:56 dovecot.index.log.2
-rw---   1 vmail  vmail  uarch  2.7K May  6 06:21 dovecot.list.index
-rw---   1 vmail  vmail  uarch  5.0K May  6 09:08 dovecot.list.index.log
-rw---   1 vmail  vmail  uarch  2.2K May  2 10:25 dovecot.mailbox.log
-rw---   1 vmail  vmail  uarch0B Feb 19 17:58 maildirfolder
drwx--   2 vmail  vmail  uarch2B May  6 09:08 new
drwx--   4 vmail  vmail  uarch4B Feb 27 16:44 sieve
-rw---   1 vmail  vmail  uarch  364B May  2 10:25 subscriptions
drwx--   2 vmail  vmail  uarch2B May  6 09:08 tmp

<<<

Notice the .Sent, .Trash folders are in the root.

this is what dovecot is probably returning in the login imap parameters

However if you are using INBOX/Sent then outlook is probably getting 
confusing thus causing the error ie the box would not be accessable.


another note do you have the auto creation turned on for the imap folders ?

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Sent {
auto = subscribe
special_use = \Sent
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix =
  separator = /
}


If not then outlook would be able to autocreate the Special Folders

I found that when migrating from cyrus imap i ended up with 2 sets of 
special folders which athough a mess was explanable because of what i am 
mentioning here.


if you turn on the auto create and restart dovecot chances are this 
might deal with your Special Folders as they will be re-created as per 
what / how outlook want to access them. (hopefully)




Another note is the subscriptions file in the maildir

# cat subscriptions
V   2

INBOX
Drafts
Sent
INBOX   Archives2010
INBOX   Archives2011
INBOX   Archives
INBOX   Archives2012
INBOX   Archives2013
INBOX   Archives2014
INBOX   Archives2015
INBOX   Archives2016
INBOX   Archives2017
INBOX   Archives2018
INBOX   Archives2019
INBOX   Archives2020
INBOX   Archives2021
INBOX   Junk
INBOX   Notes
Trash
INBOX   Archives2022
INBOX   ArchivesSent 2022
INBOX   Trash

this file sets the imap folder list and dir structure.

I found i had to manually fix some of these entries when moving from 
cyrus to dovecot


this file is the folder list that is sent to the IMAP email client and 
some times will not update correctly. (thunderbird as an example a lot 
of greyed out inaccessable folders were in this file and had to be removed)




Happy Hunting 




Happy Friday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/6/2022 10:30 AM, Philipp Ewald wrote:

Hello Paul and Kevin,


thanks for your replys!

Problem is, i am Administrator from mail Server and customer has there 
own administrator.

Me and the other IT don't know whats the problem is.


I will tell them too do some more testing and maybe switching 
Mailclient, Outlooks is horrible...
I will try other Ports and ask if there can clean up the mailbox (PC was 
already cleaned and still not work)


On time i was there and i had no problem with my Linux + Thunderbird... 
so network is working.



1 - Time it. Is it hitting a specific time and giving up?
It looks like there is instant problem, Outlook reporting problem Moving 
Mail into "Sent"



2 - Does the email have any specific content?
its random problem, mostly this error happen when there are some more 
recipient, but sometime problem with less recipient.


3 - Do they have anything like A/V software that might be acting to 
try and proxy / scan the email?
there told me there has tested on a new installed windows with out any 
"rules/extra software" applied on



Thanks i will report if we had some success.


Have a nice weekend!

Am 06.05.22 um 12:55 schrieb Paul Kudla (SCOM.CA Internet Services Inc.):


What version of outlook is your customer using?

and what internet provider (cell, dsl, cable, fiber etc) ?

anything past 2010 can be a crap shoot at best.

I have had many many customers try to use outlook because it is 
fancier interface wise but outlook just does not really support imap 
properly.


Last time i googled this it cam back with over 2 millions hits on 
outlook imap issues. All suggestions fell back to connect using 
exchange 

Re: IMAP connection lost while sending mail

[Paul Kudla (SCOM.CA Internet Services Inc.)]

What version of outlook is your customer using?

and what internet provider (cell, dsl, cable, fiber etc) ?

anything past 2010 can be a crap shoot at best.

I have had many many customers try to use outlook because it is fancier 
interface wise but outlook just does not really support imap properly.


Last time i googled this it cam back with over 2 millions hits on 
outlook imap issues. All suggestions fell back to connect using exchange 
protocol's there was no real answer.


I have been testing myself with 2019 office and it seems to work kinda.

On another major note with outlook it can only handle up to maybe 1000 
emails per folder or it will constantly sync and spend more time doing 
that then anything else.


Outlook also does NOT support idle. (again echange thingy)

I find thunderbird works best (even though they have issues to sort out 
as well here and there)


I run thunderbird and can comfortably run up to 1 emails in a folder.

It's a microsoft thing pushing exchange / outlook 365 against imap.

This is NO Fault of Dovecot / Cyrus or any other imap server.

Best suggestions for outlook look is to use

Port 143 for imap (you can use ssl on port 143)

Port 587 for smtp (25 is sometimes blocked by your isp) - Note on smtp 
connections there are some isp's blocking normal ports, i had to open up 
1 on my postfix to get around at&t blocking smtp sends from a 
tethered cell phone connection for one of my customers which fixed the 
issue.


Also outlook timeouts for both send and receive should be pushed up to a 
couple of minutes to cover any timing issues (although 30 seconds should 
be ok in most cases i have found outlook to fail with unknown reasons).


One more thing about outlook is to speed up the user interface in some 
cases they will give you an option on the bottom of the inbox to load 
more messages, this should be checked as i have had several customers 
forget about old emails and the inbox could just simply be going past 
1000 messages without any kind of warning. (see above)


Hope this helps






Happy Friday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 5/6/2022 3:53 AM, Philipp Ewald wrote:

Hello,

we have a strange problem.

Costumer use Outlook to connect to our IMAP and SMTP Server.
While sending Mails Outlooks reports connection lost. In Mail log there 
is no reason logged or something else.


What i dont understand is that IMAP lose connetion while using SMTP. 
Outlloks reports problem by moving mails into Send dir.


Is that a Outlook problem?

i have enables debug logging and cant see any reason :(

Network was already testet.


any ideas?

Kind regards
Philipp

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Thanks for the update

I dont push anyone when asking for updates

I am a programmer by trade as well and nothing ever goes as planned

prefer we all take our time and roll it out correctly then jumping the gun.

Why I am trying to help elsewhere as I have gotten pretty fluid with 
dovecot etc and can help users out with the day to day stuff.


I just can't help with ldap, never got around to that as i use pgsql 
databases that are replicated etc etc etc on all my configs.


Again thanks for the update.



Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/28/2022 7:02 AM, Aki Tuomi wrote:


2.3.19 is round the corner, so not long. I cannot yet promise an exact date but 
hopefully within week or two.

Aki


On 28/04/2022 13:57 Paul Kudla (SCOM.CA Internet Services Inc.)  
wrote:

  
Thanks for the update.


is this for both replication issues (folders +300 etc)

Just Asking - Any ETA





Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/27/2022 9:01 AM, Aki Tuomi wrote:


Hi!

This is probably going to get fixed in 2.3.19, this looks like an issue we are 
already fixing.

Aki


On 26/04/2022 16:38 Paul Kudla (SCOM.CA Internet Services Inc.)  
wrote:

   
Agreed there seems to be no way of posting these kinds of issues to see

if they are even being addressed or even known about moving forward on
new updates

i read somewhere there is a new branch soming out but nothing as of yet?

2.4 maybe 
5.0 

my previous replication issues (back in feb) went unanswered.

not faulting anyone, but the developers do seem to be disconnected from
issues as of late? or concentrating on other issues.

I have no problem with support contracts for day to day maintence
however as a programmer myself they usually dont work as the other end
relies on the latest source code anyways. Thus can not help.

I am trying to take a part the replicator c programming based on 2.3.18
as most of it does work to some extent.

tcps just does not work (ie 600 seconds default in the c programming)

My thoughts are tcp works ok but fails when the replicator through
dsync-client.c when asked to return the folder list?


replicator-brain.c seems to control the overall process and timing.

replicator-queue.c seems to handle the que file that does seem to carry
acurate info.


things in the source code are documented enough to figure this out but i
am still going through all the related .h files documentation wise which
are all over the place.

there is no clear documentation on the .h lib files so i have to walk
through the tree one at a time finding relative code.

since the dsync from doveadm does see to work ok i have to assume the
dsync-client used to compile the replicator is at fault somehow or a
call from it upstream?

Thanks for your input on the other issues noted below, i will keep that
in mind when disassembling the source code.

No sense in fixing one thing and leaving something else behind, probably
all related anyways.

i have two test servers avaliable so i can play with all this offline to
reproduce the issues

Unfortunately I have to make a living first, this will be addressed when
possible as i dont like systems that are live running this way and
currently only have 5 accounts with this issue (mine included)




Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/26/2022 9:03 AM, Reuben Farrelly wrote:


I ran into this back in February and documented a reproducible test case
(and sent it to this list).  In short - I was able to reproduce this by
having a valid and consistent mailbox on the source/local, creating a
very standard empty Maildir/(new|cur|tmp) folder on the remote replica,
and then initiating the replicate from the source. This consistently
caused dsync to fail replication with the error "dovecot.index reset,
view is now inconsistent" and sync aborted, leaving the replica mailbox
in a screwed up inconsistent state. Client connections on the source
replica were also dropped when this error occurred.  You can see the
error by enabling debug level logging if you initiate dsync manually on
a test mailbox.

The only workaround I found was to remove the remote Maildir and let
Dovecot create the whole thing from scratch.  Dovecot did not like any
existing folders on the destination replica even if they were the same
names as the source and completely empty.  I was able to reproduce this
the bare minimum of folders - just an INBOX!

I have no idea if any of the developers s

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Thanks for the update.

is this for both replication issues (folders +300 etc)

Just Asking - Any ETA





Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/27/2022 9:01 AM, Aki Tuomi wrote:


Hi!

This is probably going to get fixed in 2.3.19, this looks like an issue we are 
already fixing.

Aki


On 26/04/2022 16:38 Paul Kudla (SCOM.CA Internet Services Inc.)  
wrote:

  
Agreed there seems to be no way of posting these kinds of issues to see

if they are even being addressed or even known about moving forward on
new updates

i read somewhere there is a new branch soming out but nothing as of yet?

2.4 maybe 
5.0 

my previous replication issues (back in feb) went unanswered.

not faulting anyone, but the developers do seem to be disconnected from
issues as of late? or concentrating on other issues.

I have no problem with support contracts for day to day maintence
however as a programmer myself they usually dont work as the other end
relies on the latest source code anyways. Thus can not help.

I am trying to take a part the replicator c programming based on 2.3.18
as most of it does work to some extent.

tcps just does not work (ie 600 seconds default in the c programming)

My thoughts are tcp works ok but fails when the replicator through
dsync-client.c when asked to return the folder list?


replicator-brain.c seems to control the overall process and timing.

replicator-queue.c seems to handle the que file that does seem to carry
acurate info.


things in the source code are documented enough to figure this out but i
am still going through all the related .h files documentation wise which
are all over the place.

there is no clear documentation on the .h lib files so i have to walk
through the tree one at a time finding relative code.

since the dsync from doveadm does see to work ok i have to assume the
dsync-client used to compile the replicator is at fault somehow or a
call from it upstream?

Thanks for your input on the other issues noted below, i will keep that
in mind when disassembling the source code.

No sense in fixing one thing and leaving something else behind, probably
all related anyways.

i have two test servers avaliable so i can play with all this offline to
reproduce the issues

Unfortunately I have to make a living first, this will be addressed when
possible as i dont like systems that are live running this way and
currently only have 5 accounts with this issue (mine included)




Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/26/2022 9:03 AM, Reuben Farrelly wrote:


I ran into this back in February and documented a reproducible test case
(and sent it to this list).  In short - I was able to reproduce this by
having a valid and consistent mailbox on the source/local, creating a
very standard empty Maildir/(new|cur|tmp) folder on the remote replica,
and then initiating the replicate from the source. This consistently
caused dsync to fail replication with the error "dovecot.index reset,
view is now inconsistent" and sync aborted, leaving the replica mailbox
in a screwed up inconsistent state. Client connections on the source
replica were also dropped when this error occurred.  You can see the
error by enabling debug level logging if you initiate dsync manually on
a test mailbox.

The only workaround I found was to remove the remote Maildir and let
Dovecot create the whole thing from scratch.  Dovecot did not like any
existing folders on the destination replica even if they were the same
names as the source and completely empty.  I was able to reproduce this
the bare minimum of folders - just an INBOX!

I have no idea if any of the developers saw my post or if the bug has
been fixed for the next release.  But it seemed to be quite a common
problem over time (saw a few posts from people going back a long way
with the same problem) and it is seriously disruptive to clients.  The
error message is not helpful in tracking down the problem either.

Secondly, I also have had an ongoing and longstanding problem using
tcps: for replication.  For some reason using tcps: (with no other
changes at all to the config) results in a lot of timeout messages
"Error: dsync I/O has stalled, no activity for 600 seconds".  This goes
away if I revert back to tcp: instead of tcps - with tcp: I very rarely
get timeouts.  No idea why, guess this is a bug of some sort also.

It's disappointing that there appears to be no way to have these sorts
or problems addressed like there once was.  I am not using Dovecot for
commercial purposes so paying a fortune for a support contract for a
high end installation just isn'

Re: Better not post your email password on a public mailing list, was: Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

thanks

i love to share but sometime forget whats noted inside a config file

Been meaning to change this for a while anyways.




Happy Thursday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/27/2022 8:57 AM, Daniel Lange wrote:


Am 26.04.22 um 11:36 schrieb Paul Kudla (SCOM.CA Internet Services Inc.):

#imapc_host = mail.scom.ca
#imapc_password = Pk554669
#imapc_user = p...@scom.ca


I suggest to change that password immediately.

$ openssl s_client -crlf -connect mail.scom.ca:993
CONNECTED(0003)
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
LITERAL+ AUTH=PLAIN AUTH=LOGIN] SCOM.CA Internet Services Inc. - Dovecot 
ready

A login p...@scom.ca Pk554669
A OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT 
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT 
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS 
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES 
WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY 
PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE] 
Logged in

A status INBOX (messages)
* STATUS INBOX (MESSAGES 344)
A OK Status completed (0.002 + 0.000 + 0.001 secs).
^C

Kind regards,
Daniel

Re: Can I set a different certificate per listen port?

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Technically yes (under 2.3.18 & SNI Support)

here is my sni.conf file it is imported at the end of my dovecot.conf

basically sni allows for multiple different certificates inside dovecot

please note each domain (or subdomain) config below

most of this was provided via

https://doc.dovecot.org/configuration_manual/dovecot_ssl_configuration/




in dovecot.conf (at the end)

#Addition ssl config
!include sni.conf

then make an sni.conf file adjust to your certificate / ssl needs etc


# cat  sni.conf
#sni.conf
ssl = yes
verbose_ssl = yes
ssl_dh =port should not matter sni matches the incoming dns name from the sni 
compatible client (thunderbird for example)


however do a telent to ip  to make sure there is an active 
listener on the port you are trying to use.


i already listen on 143 & 993 and this works for both.

Check for dovecot

# telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
LITERAL+ STARTTLS AUTH=PLAIN AUTH=LOGIN] SCOM.CA Internet Services Inc. 
- Dovecot ready



# imap service over SSL/TLS
openssl s_client -connect imap_dns_server_name_or_ip:993

note : imap_dns_server_name_or_ip is the actual name of the certificate 
you are trying to test for.



note when run you will/should get all the certificate stuff and it 
should then end with something like above :



read R BLOCK
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
LITERAL+ AUTH=PLAIN AUTH=LOGIN] SCOM.CA Internet Services Inc. - Dovecot 
ready



postfix outgoing test for reference.

# telnet localhost 465
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mail18.scom.ca ESMTP Postfix

// Send EHLO servername gives what the server can do.

EHLO scom.ca
250-mail18.scom.ca
250-PIPELINING
250-SIZE 10
250-ETRN
250-STARTTLS
250-AUTH PLAIN LOGIN
250-AUTH=PLAIN LOGIN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250-SMTPUTF8
250 CHUNKING


sni only works if the mail client supports it so you need to run a mail 
client less then a few years old.


sni was developed with a lot of hit and misses and little documentation, 
and is still kinda being debugged but does work without issue for the 
most part inside postfix & dovecot.


i dont believe cyrus and others are fully supporting sni which makes 
this kinda a nice thing dovecot support wise - thanks to the developers.



Also note postfix also supports sni which needs to be setup seperately 
if used for outgoing emails.


main.cf

#SSL SNI
tls_server_sni_maps = hash:/usr/home/postfix/config/sni
smtpd_tls_chain_files = /etc/ssl/.scom.ca
smtpd_tls_CApath = /etc/ssl/certs

# cat sni.good
.scom.ca /etc/ssl/postfix.pem.scom
secure.mail.elirpa.com /etc/ssl/postfix.pem.elirpa

please note . at front indicates a *.scom.ca (for wildcard cert example 
above)


also note sni must be hashes into a sni.db file for postfix to be used

/usr/local/sbin/postmap -c /usr/home/postfix/config -F 
/usr/home/postfix/config/sni


will make the hash file from sni (text file)

please note i use pgsql database to store the certs etc hence the

/programs/common/getssl.cert -c mail.clancyca.com -q yes

it is from a django instance

text files would still be loaded with the < (pipe from file command)

both samples are noted above.


here is the python script i use to generate the dovecot ssl stuff

# cat /programs/common/getssl.cert
#!/usr/local/bin/python3
#update the ssl certificates for this mail server

import sys
import os
import string
import psycopg2

from optparse import OptionParser

USAGE_TEXT = '''\
usage: %%prog %s[options]
'''


parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')
parser.add_option("-c", "--cert", dest="cert", help="Domain Certificate 
Requested")

parser.add_option("-k", "--key", dest="key", help="Domain Key Requested")
parser.add_option("-r", "--crt", dest="crt", help="Domain CRT Requested")
parser.add_option("-s", "--csr", dest="csr", help="Domain CSR Requested")
parser.add_option("-i", "--inter", dest="inter", help="Domain INTER 
Requested")

parser.add_option("-x", "--pem", dest="pem", help="Domain Pem Requested")
parser.add_option("-q", "--quiet", dest="quiet", help="Quiet")

options, args = parser.parse_args()

#print (options.quiet)

if options.cert != None :
ssl = options.cert
if options.quiet == None :
print ('\nGetting Full Pem Certificate : %s\n' 
%options.cert)



if options.key != None :
ssl = options.key
if options.quiet == None :
print ('\nGetting Key Certificate : %s\n' %options.key)


if options.crt != None :
ssl = options.crt
if options.quiet == None :
print ('\nGetting CRT Certificate : %s\n' %options.crt)

if options.csr != None :
ssl = options.csr
if options.quiet == None :
print ('\nGetting CSR Certificate : %s\n' %options.csr)

if options.inter != None :
ssl = options.i

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Agreed there seems to be no way of posting these kinds of issues to see 
if they are even being addressed or even known about moving forward on 
new updates


i read somewhere there is a new branch soming out but nothing as of yet?

2.4 maybe 
5.0 

my previous replication issues (back in feb) went unanswered.

not faulting anyone, but the developers do seem to be disconnected from 
issues as of late? or concentrating on other issues.


I have no problem with support contracts for day to day maintence 
however as a programmer myself they usually dont work as the other end 
relies on the latest source code anyways. Thus can not help.


I am trying to take a part the replicator c programming based on 2.3.18 
as most of it does work to some extent.


tcps just does not work (ie 600 seconds default in the c programming)

My thoughts are tcp works ok but fails when the replicator through 
dsync-client.c when asked to return the folder list?



replicator-brain.c seems to control the overall process and timing.

replicator-queue.c seems to handle the que file that does seem to carry 
acurate info.



things in the source code are documented enough to figure this out but i 
am still going through all the related .h files documentation wise which 
are all over the place.


there is no clear documentation on the .h lib files so i have to walk 
through the tree one at a time finding relative code.


since the dsync from doveadm does see to work ok i have to assume the 
dsync-client used to compile the replicator is at fault somehow or a 
call from it upstream?


Thanks for your input on the other issues noted below, i will keep that 
in mind when disassembling the source code.


No sense in fixing one thing and leaving something else behind, probably 
all related anyways.


i have two test servers avaliable so i can play with all this offline to 
reproduce the issues


Unfortunately I have to make a living first, this will be addressed when 
possible as i dont like systems that are live running this way and 
currently only have 5 accounts with this issue (mine included)





Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/26/2022 9:03 AM, Reuben Farrelly wrote:


I ran into this back in February and documented a reproducible test case 
(and sent it to this list).  In short - I was able to reproduce this by 
having a valid and consistent mailbox on the source/local, creating a 
very standard empty Maildir/(new|cur|tmp) folder on the remote replica, 
and then initiating the replicate from the source. This consistently 
caused dsync to fail replication with the error "dovecot.index reset, 
view is now inconsistent" and sync aborted, leaving the replica mailbox 
in a screwed up inconsistent state. Client connections on the source 
replica were also dropped when this error occurred.  You can see the 
error by enabling debug level logging if you initiate dsync manually on 
a test mailbox.


The only workaround I found was to remove the remote Maildir and let 
Dovecot create the whole thing from scratch.  Dovecot did not like any 
existing folders on the destination replica even if they were the same 
names as the source and completely empty.  I was able to reproduce this 
the bare minimum of folders - just an INBOX!


I have no idea if any of the developers saw my post or if the bug has 
been fixed for the next release.  But it seemed to be quite a common 
problem over time (saw a few posts from people going back a long way 
with the same problem) and it is seriously disruptive to clients.  The 
error message is not helpful in tracking down the problem either.


Secondly, I also have had an ongoing and longstanding problem using 
tcps: for replication.  For some reason using tcps: (with no other 
changes at all to the config) results in a lot of timeout messages 
"Error: dsync I/O has stalled, no activity for 600 seconds".  This goes 
away if I revert back to tcp: instead of tcps - with tcp: I very rarely 
get timeouts.  No idea why, guess this is a bug of some sort also.


It's disappointing that there appears to be no way to have these sorts 
or problems addressed like there once was.  I am not using Dovecot for 
commercial purposes so paying a fortune for a support contract for a 
high end installation just isn't going to happen, and this list seems to 
be quite ordinary for getting support and reporting bugs nowadays....


Reuben

On 26/04/2022 7:21 pm, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:



side issue

if you are getting inconsistant dsyncs there is no real way to fix 
this in the long run.


i know its a pain (already had to my self)

i needed to do a full sync, take one server offline, delete the user 
dir (with dovecot offline) and then rsync (or somehow duplicate the 
main ser

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

self.ascii = ''
for n in range (0,len(self.list)) :
self.ascii = self.ascii + self.list[n]
self.ascii = str(self.ascii)

return


class edi_send_date_short:
def __init__(self, senddate):
self.date = senddate
self.result = self.date[2] + self.date[3] + 
self.date[5] + self.date[6] + self.date[8] + self.date[9]


def __str__(self):
return '%s' % self.result

##Return edi senddate string (long) 2011-10-31 into 20111031
class edi_send_date_long:
def __init__(self, senddate):
self.date = senddate
self.result1 = self.date[0] + self.date[1] + 
self.date[2] + self.date[3] + self.date[5] + self.date[6] + self.date[8] 
+ self.date[9]
self.result2 = self.date[2] + self.date[3] + 
self.date[5] + self.date[6] + self.date[8] + self.date[9]


def __str__(self):
return '%s' % (self.result1,self.result2)

class gpsdeg:
def __init__(self, dms):
self.dms = dms
self.is_positive = self.dms >= 0
self.dms = abs(self.dms)
self.minutes,self.seconds = divmod(self.dms*3600,60)
self.degrees,self.minutes = divmod(self.minutes,60)
self.degrees = self.degrees if self.is_positive else 
-self.degrees


def __str__(self):
return '%s' % (self.degrees,self.minutes,self.seconds)


class degdir:
def __init__(self, degrees):
self.direction_data = ['N','348.75','11.25','NNE', 
'11.25','33.75','NE','33.75','56.25','ENE', 
'56.25','78.75','E','78.75','101.25','ESE','101.25','123.75','SE','123.75','146.25','SSE','146.25','168.75','S','168.75','191.25','SSW','191.25','213.75','SW','213.75','236.25','WSW','236.25','258.75','W','258.75','281.25','WNW','281.25','303.75','NW','303.75','326.25','NNW','326.25','348.75']



def __str__(self):
return '%s' % (self.direction)


class gettime:
def __init__(self):
self.uu = time.localtime()

self.todaystime = str(self.uu[3]) #get the hr

if int(self.uu[3]) < 10: #add a zero
self.todaystime = '0' + self.todaystime
if int(self.uu[4]) < 10: #add a zero in front
self.todaystime = self.todaystime 
+":0"+str(self.uu[4])

else:
self.todaystime = self.todaystime 
+":"+str(self.uu[4])


def __str__(self):
return self.todaystime

class array2dbstring:
def __init__(self,array):
self.data = array
for self.nn in range(0,len(self.data)):
print ('Data %s \t\t %s' % 
(str(self.data[self.nn]),str( type(self.data[self.nn])) ) ) #change all 
data into strings

self.a = type(self.data[self.nn])
self.a = str(self.a)
if 'Decimal' in self.a :
self.n = str(self.data[self.nn])
#self.n = self.n.lstrip("'")
#self.n = self.n.rstrip("'")
#self.data[self.nn] = 
float(self.data[self.nn])

self.data[self.nn] = str('0.00')
print (self.n)

if 'NoneType' in self.a :
    self.data[self.nn] = ''
    if 'datetime.datetime' in self.a :
#self.data[self.nn] = 
str(self.data[self.nn])

#self.data[self.nn].replace
self.data[self.nn] = '2012-01-25 00:00:00'
self.data = str(self.data)
self.data = self.data.lstrip('[')
self.data = self.data.rstrip(']')
self.data = self.data.replace("'NULL'","NULL")
#self.data = self.data.replace(" '',", ",")
#self.data = self.data.replace(" '0.00'","'100'")

def __str__(self):
 

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

if options.quiet == None :
print ('\nGetting Inter Certificate : %s\n' %options.inter)

if options.pem != None :
ssl = options.pem
if options.quiet == None :
print ('\nGetting Pem Certificate : %s\n' %options.pem)



#sys.exit()


#from lib import *

#print ('Opening the Database ')
conn = psycopg2.connect(host='localhost', port = 5433, 
database='scom_billing', user='pgsql', password='Scom411400')

pg = conn.cursor()

#print ('Connected !')

#Ok now go get the email keys
command = ("""select domain,ssl_key,ssl_cert,ssl_csr,ssl_chain from 
email_ssl_certificates where domain = $$%s$$ """ %ssl)

#print (command)

pg.execute(command)
certs = pg.fetchone()

#print (certs)

#ok from here we have to decide the output ?
domain = certs[0]

if options.cert != None :
key = '#SSL Pem file (Key / Certificate / Intermediate) for 
%s\n\n#Key\n\n' %domain + certs[1] + '\n\n#Certificate\n' + certs[2] + 
'\n\n#Intermediate\n' + certs[4]


if options.key != None :
key = '#SSL Key file for %s\n\n' %domain + certs[1]

if options.crt != None :
key = '#SSL CERT file for %s\n\n' %domain + certs[2]

if options.csr != None :
key = '#SSL CSR Request file for %s\n\n' %domain + certs[3]

if options.inter != None :
key = '#SSL Intermediate file for %s\n\n' %domain + certs[4]

if options.pem != None :
key = '#SSL Pem (Certificate / Intermediate) file for 
%s\n\n#Certificate\n\n' %domain + certs[2] + '\n\n#Intermediate\n' + 
certs[4]



key = key.replace('\r','')


print (key)


conn.close()
sys.exit()



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/25/2022 9:13 AM, Arnaud Abélard wrote:

Hello,

On my side we are running Linux (Debian Buster).

I'm not sure my problem is actually the same as Paul or you Sebastian 
since I have a lot of boxes but those are actually small (quota of 
110MB) so I doubt any of them have more than a dozen imap folders.


The main symptom is that I have tons of full sync requests awaiting but 
even though no other sync is pending the replicator just waits for 
something to trigger those syncs.


Today, with users back I can see that normal and incremental syncs are 
being done on the 15 connections, with an occasional full sync here or 
there and lots of "Waiting 'failed' requests":


Queued 'sync' requests    0

Queued 'high' requests    0

Queued 'low' requests     0

Queued 'failed' requests  122

Queued 'full resync' requests 28785

Waiting 'failed' requests 4294

Total number of known users   42512



So, why didn't the replicator take advantage of the weekend to replicate 
the mailboxes while no user were using them?


Arnaud




On 25/04/2022 13:54, Sebastian Marske wrote:

Hi there,

thanks for your insights and for diving deeper into this Paul!

For me, the users ending up in 'Waiting for dsync to finish' all have
more than 256 Imap folders as well (ranging from 288 up to >5500; as per
'doveadm mailbox list -u  | wc -l'). For more details on my
setup please see my post from February [1].

@Arnaud: What OS are you running on?


Best
Sebastian


[1] https://dovecot.org/pipermail/dovecot/2022-February/124168.html


On 4/24/22 19:36, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:


Question having similiar replication issues

pls read everything below and advise the folder counts on the
non-replicated users?

i find  the total number of folders / account seems to be a factor and
NOT the size of the mail box

ie i have customers with 40G of emails no problem over 40 or so folders
and it works ok

300+ folders seems to be the issue

i have been going through the replication code

no errors being logged

i am assuming that the replication --> dhclient --> other server is
timing out or not reading the folder lists correctly (ie dies after X
folders read)

thus i am going through the code patching for log entries etc to find
the issues.

see

[13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x  2 root  wheel  uarch    4B Apr 24 11:11 .
drwxr-xr-x  4 root  wheel  uarch    4B Mar  8  2021 ..
-rw-r--r--  1 root  wheel  uarch   73B Apr 24 11:11 instances
-rw-r--r--  1 root  wheel  uarch  160K Apr 24 13:33 replicator.db

[13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
#

replicator.db seems to get updated ok but never processed properly.

# sync.users
n...@elirpa.com   high 00:09:41  463:47:01 -     y
k

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

side issue

if you are getting inconsistant dsyncs there is no real way to fix this 
in the long run.


i know its a pain (already had to my self)

i needed to do a full sync, take one server offline, delete the user dir 
(with dovecot offline) and then rsync (or somehow duplicate the main 
server's user data) over the the remote again.


then bring remote back up and it kind or worked worked

best suggestion is to bring the main server down at night so the copy is 
clean?


if using postfix you can enable the soft bounce option and the mail will 
back spool until everything comes back online


(needs to be enable on bother servers)

replication was still an issue on accounts with 300+ folders in them, 
still working on a fix for that.



Happy Tuesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services <http://www.scom.ca>
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/25/2022 10:01 AM, Arnaud Abélard wrote:
Ah, I'm now getting errors in the logs, that would explains the 
increasing number of failed sync requests:


dovecot: imap(x)<2961235>: Error: 
Mailbox INBOX: /vmail/l/i/x/dovecot.index reset, view is now 
inconsistent



And sure enough:

# dovecot replicator status x

x none 00:02:54  07:11:28  -    y


What could explain that error?

Arnaud



On 25/04/2022 15:13, Arnaud Abélard wrote:

Hello,

On my side we are running Linux (Debian Buster).

I'm not sure my problem is actually the same as Paul or you Sebastian 
since I have a lot of boxes but those are actually small (quota of 
110MB) so I doubt any of them have more than a dozen imap folders.


The main symptom is that I have tons of full sync requests awaiting 
but even though no other sync is pending the replicator just waits for 
something to trigger those syncs.


Today, with users back I can see that normal and incremental syncs are 
being done on the 15 connections, with an occasional full sync here or 
there and lots of "Waiting 'failed' requests":


Queued 'sync' requests    0

Queued 'high' requests    0

Queued 'low' requests 0

Queued 'failed' requests  122

Queued 'full resync' requests 28785

Waiting 'failed' requests 4294

Total number of known users   42512



So, why didn't the replicator take advantage of the weekend to 
replicate the mailboxes while no user were using them?


Arnaud




On 25/04/2022 13:54, Sebastian Marske wrote:

Hi there,

thanks for your insights and for diving deeper into this Paul!

For me, the users ending up in 'Waiting for dsync to finish' all have
more than 256 Imap folders as well (ranging from 288 up to >5500; as per
'doveadm mailbox list -u  | wc -l'). For more details on my
setup please see my post from February [1].

@Arnaud: What OS are you running on?


Best
Sebastian


[1] https://dovecot.org/pipermail/dovecot/2022-February/124168.html


On 4/24/22 19:36, Paul Kudla (SCOM.CA Internet Services Inc.) wrote:


Question having similiar replication issues

pls read everything below and advise the folder counts on the
non-replicated users?

i find  the total number of folders / account seems to be a factor and
NOT the size of the mail box

ie i have customers with 40G of emails no problem over 40 or so folders
and it works ok

300+ folders seems to be the issue

i have been going through the replication code

no errors being logged

i am assuming that the replication --> dhclient --> other server is
timing out or not reading the folder lists correctly (ie dies after X
folders read)

thus i am going through the code patching for log entries etc to find
the issues.

see

[13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x  2 root  wheel  uarch    4B Apr 24 11:11 .
drwxr-xr-x  4 root  wheel  uarch    4B Mar  8  2021 ..
-rw-r--r--  1 root  wheel  uarch   73B Apr 24 11:11 instances
-rw-r--r--  1 root  wheel  uarch  160K Apr 24 13:33 replicator.db

[13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
#

replicator.db seems to get updated ok but never processed properly.

# sync.users
n...@elirpa.com   high 00:09:41  463:47:01 -     y
ke...@elirpa.com  high 00:09:23  463:45:43 -     y
p...@scom.ca  high 00:09:41  463:46:51 -     y
e...@scom.ca    high 00:09:43  463:47:01 -     y
ed.ha...@dssmgmt.com  high 00:09:42  463:46:58 -     y
p...@paulkudla.net    high 00:09:44  463:47:03 
580:35:07

    y




so 



two things :

first to get the production stuff to work i had to write a script that
whould find the bad sync's and the force a dsync between the servers

i run this every five minutes or each server.

in crontab

*/10    *    

Re: no full syncs after upgrading to dovecot 2.3.18

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Question having similiar replication issues

pls read everything below and advise the folder counts on the 
non-replicated users?


i find  the total number of folders / account seems to be a factor and 
NOT the size of the mail box


ie i have customers with 40G of emails no problem over 40 or so folders 
and it works ok


300+ folders seems to be the issue

i have been going through the replication code

no errors being logged

i am assuming that the replication --> dhclient --> other server is 
timing out or not reading the folder lists correctly (ie dies after X 
folders read)


thus i am going through the code patching for log entries etc to find 
the issues.


see

[13:33:57] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
# ll
total 86
drwxr-xr-x  2 root  wheel  uarch4B Apr 24 11:11 .
drwxr-xr-x  4 root  wheel  uarch4B Mar  8  2021 ..
-rw-r--r--  1 root  wheel  uarch   73B Apr 24 11:11 instances
-rw-r--r--  1 root  wheel  uarch  160K Apr 24 13:33 replicator.db

[13:33:58] mail18.scom.ca [root:0] /usr/local/var/lib/dovecot
#

replicator.db seems to get updated ok but never processed properly.

# sync.users
n...@elirpa.com   high 00:09:41  463:47:01 - 
y
ke...@elirpa.com  high 00:09:23  463:45:43 - 
y
p...@scom.ca  high 00:09:41  463:46:51 - 
y
e...@scom.cahigh 00:09:43  463:47:01 - 
y
ed.ha...@dssmgmt.com  high 00:09:42  463:46:58 - 
y
p...@paulkudla.nethigh 00:09:44  463:47:03 580:35:07 
   y





so 



two things :

first to get the production stuff to work i had to write a script that 
whould find the bad sync's and the force a dsync between the servers


i run this every five minutes or each server.

in crontab

*/10****root/usr/bin/nohup 
/programs/common/sync.recover > /dev/null



python script to sort things out

# cat /programs/common/sync.recover
#!/usr/local/bin/python3

#Force sync between servers that are reporting bad?

import os,sys,django,socket
from optparse import OptionParser


from lib import *

#Sample Re-Index MB
#doveadm -D force-resync -u p...@scom.ca -f INBOX*



USAGE_TEXT = '''\
usage: %%prog %s[options]
'''

parser = OptionParser(usage=USAGE_TEXT % '', version='0.4')

parser.add_option("-m", "--send_to", dest="send_to", help="Send Email To")
parser.add_option("-e", "--email", dest="email_box", help="Box to Index")
parser.add_option("-d", "--detail",action='store_true', 
dest="detail",default =False, help="Detailed report")
parser.add_option("-i", "--index",action='store_true', 
dest="index",default =False, help="Index")


options, args = parser.parse_args()

print (options.email_box)
print (options.send_to)
print (options.detail)

#sys.exit()



print ('Getting Current User Sync Status')
command = commands("/usr/local/bin/doveadm replicator status '*'")


#print command

sync_user_status = command.output.split('\n')

#print sync_user_status

synced = []

for n in range(1,len(sync_user_status)) :
user = sync_user_status[n]
print ('Processing User : %s' %user.split(' ')[0])
if user.split(' ')[0] != options.email_box :
if options.email_box != None :
continue

if options.index == True :
command = '/usr/local/bin/doveadm -D force-resync -u %s 
-f INBOX*' %user.split(' ')[0]

command = commands(command)
command = command.output

#print user
for nn in range (len(user)-1,0,-1) :
#print nn
#print user[nn]

if user[nn] == '-' :
#print 'skipping ... %s' %user.split(' ')[0]

break



if user[nn] == 'y': #Found a Bad Mailbox
print ('syncing ... %s' %user.split(' ')[0])


if options.detail == True :
command = '/usr/local/bin/doveadm -D 
sync -u %s -d -N -l 30 -U' %user.split(' ')[0]

print (command)
command = commands(command)
command = command.output.split('\n')
print (command)
print ('Processed Mailbox for ... %s' 
%user.split(' ')[0] )
synced.append('Processed Mailbox for 
... %s' %user.split(' ')[0])

for nnn in range(len(command)):
synced.append(command[nnn] + '\n')
break


if options.detail == False :
#command = '/usr/local/bin/doveadm -D 
sync -u %s -d -N -l 30 -U' %user.split(' ')[0]

#print (command)
#command = os.system(command)
   

Re: temporary block incoming messages to specific user

[Paul Kudla (SCOM.CA Internet Services Inc.)]

this is handled in a couple of different ways


please note this is only for virtual users and assumes you are running 
postfix/dovecot



through postfix (first)

see main.cf :

# SOFT BOUNCE
#
# The soft_bounce parameter provides a limited safety net for
# testing.  When soft_bounce is enabled, mail will remain queued that
# would otherwise bounce. This parameter disables locally-generated
# bounces, and prevents the SMTP server from rejecting mail permanently
# (by changing 5xx replies into 4xx replies). However, soft_bounce
# is no cure for address rewriting mistakes or mail routing mistakes.
#

soft_bounce = yes


the above option in postfix will continually try to deliver to the end 
user, if the user does not exist it will continue trying without any 
bounceback to the sender.


this was built into postfix mainly for mail migrations between servers 
where the dovecot, cyrus server upgrades etc came into play


but should work for your situation.


next on the dovecot side (user database queries)

# cat dovecot-pgsql.conf
driver = pgsql
connect = host=localhost port=5433 dbname=scom_billing user=pgsql 
password=Scom411400

default_pass_scheme = PLAIN

password_query = SELECT username as user, password FROM email_users 
WHERE username = '%u' and password <> 'alias' and status = True and 
destination = '%u'


user_query = SELECT home, uid, gid FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


#iterate_query = SELECT user, password FROM email_users WHERE username = 
'%u' and password <> 'alias' and status = True and destination = '%u'


iterate_query = SELECT "username" as user, domain FROM email_users WHERE 
status = True and alias_flag = False



this file reflects how to query the database tables etc

note the status=true above etc, i use this to specify if the account is 
active and ok or not


if this flag is set to false, dovecot will consider this user non active 
and not allow delivery locally.


mainly used for billing but will sufice

also note i use an alias flag (from the cyrus days) to forward emails 
off to another actual active account


i am still streamlining that internally but thought i should explain why 
it is there


please note iterate_query is used for the replication users (doveadm 
user '*') query - not needed if you dont run replication but wont hurt 
anything if it is there.










Happy Wednesday !!!
Thanks - paul

Paul Kudla


Scom.ca Internet Services 
004-1009 Byron Street South
Whitby, Ontario - Canada
L1N 4S3

Toronto 416.642.7266
Main 1.866.411.7266
Fax 1.888.892.7266

On 4/20/2022 8:39 AM, Marc wrote:

Is it possible to block incoming messages from being delivered to a specific 
user in such a way that the MTA will try again later. I do not want these 
message to bounce. (eg while doing some manual maintenance on the user)?