Re: How to get a server listed in the IMAP Test wiki?

[Leander Beernaert]

Hey Timo,

Thanks for the quick turnaround, once we have the test results I'll contact you 
again.

Should I also include instructions on how to run the a self contained server 
with a dummy backend so you can independently verify our results?

Leander Beernaert
Proton AG

--- Original Message ---
On Thursday, February 23rd, 2023 at 8:59 PM, Timo Sirainen  
wrote:

> On 23. Feb 2023, at 16.13, Leander Beernaert  
> wrote:
>
>> Hey,
>>
>> We recently announced Gluon (https://github.com/ProtonMail/gluon/) our IMAP 
>> server library we are using in Proton 
>> Bridge(https://github.com/ProtonMail/proton-bridge). We would love to have 
>> it have it listed in the IMAP Server Compliancy Status wiki page 
>> (https://imapwiki.org/ImapTest/ServerStatus). What do we need to do or whom 
>> do we need to contact to make this happen?
>
> There was so much spam that we disabled all outside access to the wiki. Maybe 
> we should move it to github/sphinx similarly to doc.dovecot.org so we could 
> get pull requests instead. For now just email me what you want there and I 
> can add it.
>
>> Additionally, We have been using running imaptest 
>> (https://github.com/dovecot/imaptest) against our server library, but due to 
>> variety of configuration parameters, we would really appreciate it (if 
>> possible) if someone could point out to us the test setup used to validate 
>> each of those servers.
>
> I updated the page to specify how the different columns can be tested. It's 
> the same for all servers.

Re: creating a mailbox via imap

[Paul Kudla (SCOM.CA Internet Services Inc.)]

Ok basically (please read the entire post - its techy),

You need the username/password in the database before doing this

Then make sure dovecot config carries

___

namespace inbox {
  inbox = yes
  location =
  mailbox Drafts {
auto = subscribe
special_use = \Drafts
  }
  mailbox Sent {
auto = subscribe
special_use = \Sent
  }
  mailbox Trash {
auto = subscribe
special_use = \Trash
  }
  prefix =
  separator = /
}
___

note auto = subscribe above.

When you use (in python as per this example) the imap library calls to 
create a mailbox dovecot (like cyrus) should create the account etc


In my case I do a

CM (create mailbox) for

/INBOX
/Sent
/Trash
/Drafts

I make all of them by default, some mail clients will make these my 
default and some do not.


It's overkill but should work.

Also note this needs to be carried out on the actual mail server to
get around os system rights etc.

Please note i run a django for my admin system and ended up writing a 
"Listener" to communicate with the server over tcpip


the listener does use the create mailbox function with dovecot

I dont remember the specifics but feel free to ask if the code below has 
issues.


Listener sits in the background on a dovecot server waiting to do something.


[15:36:24] mail18.scom.ca [root:0] ~
# psx list
Displaying One Conditional ... list


 6005  -  Is   0:00.19 /usr/local/bin/python2 
/sbin/scripts/dovecot.listen (python2.7)


run in unix with the & (background command)

dovecot.listen as follows :

___
# cat /sbin/scripts/dovecot.listen
#!/usr/local/bin/python2

import os,sys
import socket
import commands
import time

from lib import *

a = onlyone ('dovecot.listen')
if a.status == 'BAD' :
print 'Another Process Is running '
sys.exit()

TCP_IP = '10.220.0.18'
TCP_PORT = 8444
BUFFER_SIZE = 1024  # Normally 1024, but we want fast response

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)

while 1 : #Process received data?
conn, addr = s.accept() #Wait for data
print 'Connection address:', addr
data = conn.recv(BUFFER_SIZE)
if not data: break
print "received data:", data
data = data.split (' ')
command = data[0]
print command

#If CM (Create Mailbox)
if command == 'CM' :
username = data[1]
print 'Creating Email Account : %s' % (username)
#Now create the mail box
#Now go make the email account
message = ''
for n in range (0,100) :
command1 = 
commands.getoutput('/usr/local/bin/doveadm mailbox create -s -u %s 
INBOX' %str(username))

print 'Command1 : %s' %command1
if 'Mailbox already exists' in command1 :
message = 'BAD'
conn.send( str(message) )
break

else :
if message == '' :
message = 'OK'
conn.send(message)
message = 'SENT'

if "User doesn't exist" in command1 :
time.sleep(2)
continue
else :
print 'Command1 : %s' %command1
message = 'SENT'
break

if message == 'SENT' : #Create the rest
command2 = 
commands.getoutput('/usr/local/bin/doveadm mailbox create -s -u %s Sent' 
%str(username))

print 'Command2 : %s' %command2
command3 = 
commands.getoutput('/usr/local/bin/doveadm mailbox create -s -u %s 
Trash' %str(username))

print 'Command3 : %s' %command3
command4 = 
commands.getoutput('/usr/local/bin/doveadm mailbox create -s -u %s 
Drafts' %str(username))

print 'Command4 : %s' %command4




if command == 'INFO' :
username = data[1]
print 'Getting Email Account Info : %s' % ( username )
command1 = commands.getoutput("/usr/local/bin/doveadm 
mailbox status -t all -u %s '*' " %str(username))

if 'Error' in command1 :
message = 'BAD'
else :
message = 'OK : ' + command1

print message
conn.send( str(message) )  # echo


if command == 'DM' :
data = data[1]
data = data.split('@')

Re: How to get a server listed in the IMAP Test wiki?

[Timo Sirainen]

On 23. Feb 2023, at 16.13, Leander Beernaert  
wrote:
> 
> Hey,
> 
> We recently announced Gluon (https://github.com/ProtonMail/gluon/) our IMAP 
> server library we are using in Proton 
> Bridge(https://github.com/ProtonMail/proton-bridge). We would love to have it 
> have it listed in the IMAP Server Compliancy Status wiki page 
> (https://imapwiki.org/ImapTest/ServerStatus). What do we need to do or whom 
> do we need to contact to make this happen?

There was so much spam that we disabled all outside access to the wiki. Maybe 
we should move it to github/sphinx similarly to doc.dovecot.org 
 so we could get pull requests instead. For now just 
email me what you want there and I can add it.

> Additionally, We have been using running imaptest 
> (https://github.com/dovecot/imaptest) against our server library, but due to 
> variety of configuration parameters, we would really appreciate it (if 
> possible) if someone could point out to us the test setup used to validate 
> each of those servers.

I updated the page to specify how the different columns can be tested. It's the 
same for all servers.

Re: creating a mailbox via imap

[dovecot]

is there any way with Dovecot to open an admin imap connection and create a 
brand new mailbox?
With Cyrus imapd I can do this by connecting as the Cyrus admin user and then create a 
folder "user/newu...@domain.tld".



Wouldn't that be dependent on how Dovecot auth worker verifies a user exist and 
is valid? Such as for one method, database queries. How would Dovecot know what 
query to run to add another user to your database? Plus update any other 
related DB tables needed for your custom setup? I would imagine too many edge 
cases for Dovecot to worry about for creating accounts.

creating a mailbox via imap

[Gabriele Bulfon]

Hi,
 
is there any way with Dovecot to open an admin imap connection and create a 
brand new mailbox?
With Cyrus imapd I can do this by connecting as the Cyrus admin user and then 
create a folder "user/newu...@domain.tld".
 
Thanks
Gabriele
 
 
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
 

Re: doveadm sending invalid AUTHENTICATE to uw-imap

[Chris Candreva]

For anyone searching in the future: The eventual cause turns out to be, as 
far as I can tell, something screwy in uw-imap or the base64 decode 
function it uses on this old Solaris server.  It seems to be something odd 
with the number of characters in the hash.  

tl;dr I'm going to do my migration with 2 different master users with 
different length passwords, which will cover all cases and allow me to 
migrate all the users. Problem solved for this case.


Now the long version for the archive:

To diagnose the problem I eventually wrote a short perl program using 
IO::Socket::SSL that prints the UW-Imap banner and the '+' second prompt, 
logs what it receives back, and exits. This let me see that in all cases, 
doveadm sends the single line AUTHENTICATE command, so that wasn't the 
problem.

It had never occurred to me to reverse the hash and see what was being 
sent. I had been testing from telnet / "openssl s_client" using a string 
"\0user@masteruser\0masterpass" . Doveadm is sending 
"user\0masteruser\0masterpass" .  Different input, different results.

Somewhere in my testing, I forgot to add the -n to 

echo -e "\0user@masteruser\-masterpass" | base64 

(so the input to base64 had a trailing newline) and to my surprise, a user 
that didn't work before worked ! That mistake is how I figured out the 
extra character in the password was somehow making a difference.

So -- since this is a one time migration, and it's repeatable, I've come 
up with the users that work with the padding, and the ones that work 
without, and will run the import each way for each list, turn off the 
legacy server and be done with this ! 

Thanks for the pointers, and if anyone else runs into this bizare 
situation hopefully they find this ! Of course if anyone knows why this 
happens I'd love to hear it.

-Chris


On Wed, 8 Feb 2023, Chris Candreva wrote:

> 
> On Wed, 8 Feb 2023, Aki Tuomi wrote:
> 
> > Can you try setting imapc_sasl_mechanisms to login, maybe it works better?
> 
> And Stephan Bosch  wrote:
> 
> > Can you make a protocol log (tcp dump of commands sent by client and 
> > replies sent by server) for one of these sessions? e.g. using ngrep if 
> > connections aren't secured.
> 
> 
> I was using imaps initially. Switching to imap over port 143 to do the 
> tcpdump had the side effect of switching to LOGIN authentication, 
> evidently uw-imap is sending different capability strings. It still 
> doesn't work though. Both from the error and the dump I can tell "doveadm" 
> is sending the user's id only without the "*masteruser" and the 
> master user password.
> 
> Plain connection banner:
> * OK [CAPABILITY IMAP4REV1 I18NLEVEL=1 LITERAL+ SASL-IR LOGIN-REFERRALS 
> STARTTLS] foo.com IMAP4rev1 2007e.404 at Wed, 8 Feb 2023 16:45:22 
> -0500 (EST)
> 
> SSL Banner on 993:
> * OK [CAPABILITY IMAP4REV1 I18NLEVEL=1 LITERAL+ SASL-IR LOGIN-REFERRALS 
> AUTH=PLAIN AUTH=LOGIN] foo.com IMAP4rev1 2007e.404 at Wed, 8 Feb 2023 
> 16:53:36 -0500 (EST)
> 
> 
> 
> > > On 08/02/2023 06:24 EET Chris Candreva  wrote:
> > > 
> > >  
> > > I'm migrating a legacy uw-imap system to Dovecot, on a Rocky (RHEL) 8 
> > > server running Dovecot 2.3.16-3 from their repos. I am using a master 
> > > user 
> > > to import all users for an imaps connection from the old server to the 
> > > new. On a trial run however, it worked for about half the users. Half are 
> > > giving an error of the form:
> > > 
> > > dsync(user): Error: imapc(host:993): 
> > > Command '1 AUTHENTICATE PLAIN ' failed 
> > > with BAD: 
> > > 1 Missing or invalid argument to AUTHENTICATE
> > > 
> > > I can't seem to get the IMAP command for the users that did work. 
> > > However, 
> > > on the face of it, that is an invalid AUTHENTICATE command. If I take 
> > > that 
> > > string and brake it up into (what I've googled is) the proper form of 
> > > multi-command form of
> > > 
> > > 1 AUTHENTICATE PLAIN
> > > +
> > > 
> > > 
> > > then the login succeeds. I have not been able to find anyone else with 
> > > this problem in my search. Is this a known issue, is there a way to force 
> > > the multi-line AUTHENTICATE, something else I'm missing ? Any help is 
> > > appreciate on this!
> > > 
> > > -Chris
> > > 
> > > 
> > > 
> > > -- 
> > > ---
> > > 
> > > Chris Candreva  --  ch...@westnet.com  --  http://www.westnet.com/~chris
> > 
> 
> 

-- 
---

Chris Candreva  --  ch...@westnet.com  --  http://www.westnet.com/~chris

How to get a server listed in the IMAP Test wiki?

[Leander Beernaert]

Hey,

We recently announced Gluon (https://github.com/ProtonMail/gluon/) our IMAP 
server library we are using in Proton 
Bridge(https://github.com/ProtonMail/proton-bridge). We would love to have it 
have it listed in the IMAP Server Compliancy Status wiki page 
(https://imapwiki.org/ImapTest/ServerStatus). What do we need to do or whom do 
we need to contact to make this happen?

Additionally, We have been using running imaptest 
(https://github.com/dovecot/imaptest) against our server library, but due to 
variety of configuration parameters, we would really appreciate it (if 
possible) if someone could point out to us the test setup used to validate each 
of those servers.

Thank you in advance for your time.

Kind Regards,
Leander Beernaert
Proton AG

Re: Redundant Database, Pgsql ?

[Brendan Kearney]

i didnt pick up on the fact that this was auth stuff, and not indexes 
(indices?) or other data.  LDAP is a hierarchical database, where the 
relationship between data is forced into a superior/subordinate 
structure.  if you ask enough people, they will say that LDAP is not an 
authentication platform, but a database.  i tend to agree, and have put 
Kerberos and LDAP together as my AuthN/Z suite.  Kerberos is truly an 
AuthN protocol, and when i can point something at it i do.  i store my 
Kerberos data in LDAP, and run n-way multi-primary replication. MIT 
Kerberos does not have an event based replication means, only 
time/schedule based.  LDAP has an event based replication mechanism, so 
when data changes all the nodes in the cluster get an immediate 
replicated update.  OpenLDAP can also proxy AuthN events to Kerberos 
with a few specific configurations set, and the password field being set 
to a specific string, {SASL}user@domain. this way something that talks 
LDAP, can point to LDAP for AuthN, and be proxied to back Kerberos.  to 
me, this maintains the ever important "single source of the truth" for 
credentials.


I run LDAP behind a HAProxy VIP too, for load balancing purposes and 
transparent failover, so apps see less impact when failures occur.  load 
balancing allows me to scale out (handle more requests in a given unit 
of time), and be fault tolerant.  take a box out of the mix for updates, 
reboots, maintenance, whatever and not interrupt processing.  i load 
balance nearly every stateful protocol, whenever possible.  i anycast 
most stateless protocols, too, as a means of load sharing.  by having 
more than one instance available to do the same work, you greatly reduce 
the "hair on fire" calls in the middle of the night, or at least shorten 
the Mean Time to Recovery.


On 2/23/23 12:55 AM, Nikolai Lusan wrote:

On Wed, 2023-02-22 at 11:08 +, Marc wrote:
> I don't even get what the advatages are of doing this with sql. If you
> use local replicated ldap and use local credential caching then your
> master ldap can go down without issues, even the local caching handle
> some local slapd issues.

Going to have to +1 this. LDAP also does multi-master replication, which
can make failover easier via DNS (like with a round robin for
ldap.mydomain), or multiple LDAP dictionaries for dovecot. The [big]
problem with OSS Postgres is that it only does master/slave replication,
with no plans to add multi-master replication to the code base (there is
Percona and 2ndQuadrant, but for small outfits, and individual there is
a price barrier there). Personally I love PGSQL as a DB, but for SSO I
use LDAP - because that's what it's designed for (i.e. read more than
written).


> I guess the local caching is also faster. Afaik were databases not
> designed for this purpose and a better fit is ldap.

This is totally true. RDBMS were not designed with this kind of use in
mind, LDAP was - it is, after all, a directory service. So unless your
auth stuff is part of some larger DB "thing" the directory type
solutions are not suitable for (how many table joins, or other extensive
SQL actions are taking place on that DB) then LDAP is the better way to
go, and extending LDAP with custom schemas is simple - just grab an IANA
number for you, or your organisation, so that you don't trample on any
other schema out there. I have a custom schema that I use for
postfix/dovecot - it's simple, quick, and efficient without the DB
overhead ... and I get the multi-master replication in OpenLDAP.

Re: Restored mail folders conflict with renamed original ones

[Nikolaos Milas]

On 22/2/2023 2:56 μ.μ., Nikolaos Milas wrote:


On 22/2/2023 2:30 μ.μ., Aki Tuomi wrote:

Can you please try

doveadm exec imap -u username_of_the_user
1 LIST "" "*"

and see if it is there?


Here is the output:

# doveadm exec imap -u userx
* PREAUTH [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE 
SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT 
MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS 
LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES 
WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY 
PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE 
QUOTA] Logged in as userx

1 LIST "" "*"
...
* LIST (\HasChildren \UnMarked) "." "Grants Active"
* LIST (\HasChildren \UnMarked) "." "Grants Active.European"
* LIST (\HasNoChildren \UnMarked) "." "Grants Active.European.URBAN 
RELEAF"

* LIST (\HasChildren \UnMarked) "." "Grants Active.European.SMURBS"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.European.SMURBS.In Kind"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.European.SMURBS.Final Reports"

* LIST (\HasNoChildren \UnMarked) "." "Grants Active.European.RI-URBANS"
* LIST (\HasNoChildren \UnMarked) "." "Grants Active.European.QA4EO"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.European.ATMO-ACCESS"

* LIST (\HasNoChildren \UnMarked) "." "Grants Active.European.IGOSP"
* LIST (\HasNoChildren \UnMarked) "." "Grants Active.European.GAUSS"
* LIST (\HasNoChildren \UnMarked) "." "Grants Active.European.E-SHAPE"
* LIST (\HasNoChildren \UnMarked) "." "Grants Active.European.CIROCCO"
* LIST (\HasNoChildren \UnMarked) "." "Grants Active.European.InCASE"
* LIST (\HasNoChildren) "." "Grants Active.European.EIFFEL"
* LIST (\HasNoChildren) "." "Grants Active.European.ARSINOE"
* LIST (\HasChildren \UnMarked) "." "Grants Active.International"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.India - 2320"

* LIST (\HasNoChildren \UnMarked) "." "Grants Active.International.2324"
* LIST (\HasNoChildren \UnMarked) "." "Grants Active.International.GAUSS"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.CIROCCO"

* LIST (\HasNoChildren \UnMarked) "." "Grants Active.International.IGOSP"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.ATMO-ACCESS"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.URBAN RELEAF"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.InCASE"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.ARSINOE"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.RI-URBANS"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.E-SHAPE"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.EIFFEL"

* LIST (\HasNoChildren \UnMarked) "." "Grants Active.International.QA4EO"
* LIST (\HasChildren \UnMarked) "." "Grants Active.International.SMURBS"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.SMURBS.Final Reports"
* LIST (\HasNoChildren \UnMarked) "." "Grants 
Active.International.SMURBS.In Kind"

...

So, yes, the Mail Folders in question are indeed listed above.


Waiting for your feedback, I decided to remove the restored folders 
(Grants Active.European*) in case it makes any difference. After 
removing them I also did:


   # doveadm -v force-resync -u userx Grants\ Active # doveadm -v
   force-resync -u userx Grants\ Active.International

Then I asked the user to access his mailbox again, but it didn't make 
any difference. As in the beginning, the only two directories appearing 
in "Grants Active.International" folder are the following:


   Grants Active.International.2324
   Grants Active.International.India

which are the two newest ones (all the others are the same with those in 
the initial "Grants Active.European" folder which was renamed to "Grants 
Active.International").


I look forward to your advice and assistance. What may be causing the 
mailbox to not open the rest of the folders?


Thanks a lot,
Nick