Re: 100 MB RAM per clinet in an imap-proxy setup ...

[Tobias Oetiker]

- On Sep 3, 2021, at 10:06 AM, Aki Tuomi aki.tu...@open-xchange.com wrote:

>> On 03/09/2021 10:25 Tobias Oetiker  wrote:
>> 
>>  
>> - On Sep 3, 2021, at 9:01 AM, Aki Tuomi aki.tu...@open-xchange.com wrote:
>> 
>> >> On 03/09/2021 09:35 Tobias Oetiker  wrote:
>> >> 
>> >> 
>> >> Hi All,
>> >> 
>> >> I spent some quality time yesterday, tuning dovecot on a server which 
>> >> serves as
>> >> an imap-proxy in front of a zimbra setup
>> >> the imap proxy does ldap client-cert authentication and works well.
>> >> 
>> >> BUT
>> >> 
>> >> I found that the imap-login processes seem to gain 100 MB per connection 
>> >> they
>> >> are accepting ... this seems pretty hefty.
>> [...]
>> > 
>> > I would strongly recommend using "high performance" configuration on your 
>> > proxy.
>> > 
>> > See 
>> > https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode
>> 
>> yes that is what we have:
>> 
>>  service imap-login {
>>   service_count = 0
>>   client_limit = 380
>>   process_limit = 10
>>   vsz_limit = 39G
>>  }
>> 
>> the machine has 180 GB ram
>> 
> 
> Which version of dovecot are you running?
>

2.3.9

I am planning to do a testsetup and maybe add mtrace to the code to see who is 
using this memory ... :)

cheers
tobi
-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch t...@oetiker.ch +41 62 775 9902

Re: 100 MB RAM per clinet in an imap-proxy setup ...

[Aki Tuomi]

> On 03/09/2021 10:25 Tobias Oetiker  wrote:
> 
>  
> - On Sep 3, 2021, at 9:01 AM, Aki Tuomi aki.tu...@open-xchange.com wrote:
> 
> >> On 03/09/2021 09:35 Tobias Oetiker  wrote:
> >> 
> >> 
> >> Hi All,
> >> 
> >> I spent some quality time yesterday, tuning dovecot on a server which 
> >> serves as
> >> an imap-proxy in front of a zimbra setup
> >> the imap proxy does ldap client-cert authentication and works well.
> >> 
> >> BUT
> >> 
> >> I found that the imap-login processes seem to gain 100 MB per connection 
> >> they
> >> are accepting ... this seems pretty hefty.
> [...]
> > 
> > I would strongly recommend using "high performance" configuration on your 
> > proxy.
> > 
> > See 
> > https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode
> 
> yes that is what we have:
> 
>  service imap-login {
>   service_count = 0
>   client_limit = 380
>   process_limit = 10
>   vsz_limit = 39G
>  }
> 
> the machine has 180 GB ram
> 
> cheers
> tobi
> 
> -- 
> Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> www.oetiker.ch t...@oetiker.ch +41 62 775 9902

Which version of dovecot are you running?

Aki

Re: 100 MB RAM per clinet in an imap-proxy setup ...

[Tobias Oetiker]

- On Sep 3, 2021, at 9:01 AM, Aki Tuomi aki.tu...@open-xchange.com wrote:

>> On 03/09/2021 09:35 Tobias Oetiker  wrote:
>> 
>> 
>> Hi All,
>> 
>> I spent some quality time yesterday, tuning dovecot on a server which serves 
>> as
>> an imap-proxy in front of a zimbra setup
>> the imap proxy does ldap client-cert authentication and works well.
>> 
>> BUT
>> 
>> I found that the imap-login processes seem to gain 100 MB per connection they
>> are accepting ... this seems pretty hefty.
[...]
> 
> I would strongly recommend using "high performance" configuration on your 
> proxy.
> 
> See 
> https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode

yes that is what we have:

 service imap-login {
  service_count = 0
  client_limit = 380
  process_limit = 10
  vsz_limit = 39G
 }

the machine has 180 GB ram

cheers
tobi

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
www.oetiker.ch t...@oetiker.ch +41 62 775 9902

Re: 100 MB RAM per clinet in an imap-proxy setup ...

[Aki Tuomi]

> On 03/09/2021 09:35 Tobias Oetiker  wrote:
> 
> 
> Hi All,
> 
> I spent some quality time yesterday, tuning dovecot on a server which serves 
> as an imap-proxy in front of a zimbra setup
> the imap proxy does ldap client-cert authentication and works well. 
> 
> BUT
> 
> I found that the imap-login processes seem to gain 100 MB per connection they 
> are accepting ... this seems pretty hefty.
> 
> It does not seem to be a leak, since the setup is stable when client_limit 
> and process_limit are set appropriately.
> 
> Any ideas what part of the code I should be looking at ? Or is this a 
> configuration problem ?
> 
> cheers
> tobi
> 
> -- 
> Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> www.oetiker.ch t...@oetiker.ch +41 62 775 9902

I would strongly recommend using "high performance" configuration on your proxy.

See https://doc.dovecot.org/admin_manual/login_processes/#high-performance-mode

Aki

100 MB RAM per clinet in an imap-proxy setup ...

[Tobias Oetiker]

Hi All, 

I spent some quality time yesterday, tuning dovecot on a server which serves as 
an imap-proxy in front of a zimbra setup 
the imap proxy does ldap client-cert authentication and works well. 

BUT 

I found that the imap-login processes seem to gain 100 MB per connection they 
are accepting ... this seems pretty hefty. 

It does not seem to be a leak, since the setup is stable when client_limit and 
process_limit are set appropriately. 

Any ideas what part of the code I should be looking at ? Or is this a 
configuration problem ? 

cheers 
tobi 

-- 
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland 
www.oetiker.ch t...@oetiker.ch +41 62 775 9902 

Dovecot Imap-Proxy: openssl_iostream_handle_error

[Urban Loesch]

Hi,

I'm running Dovecot 2.3.9-2 as an IMAP/POP3 Proxy in front of several 
Dovecotbackends. No Director, only static routing to the backends of each user.
The proxies are also working as "SSL offload engines". SystemOS: Debian Stretch 
(9.11) on LXC Virtualization

Sometimes I get the following errors in mail.err log:

...
Mar 10 16:47:24 imap1 dovecot: imap-login: Panic: file iostream-openssl.c: line 
599 (openssl_iostream_handle_error): assertion failed: (errno != 0)
Mar 10 16:47:24 imap1 dovecot: imap-login: Fatal: master: service(imap-login): 
child 30431 killed with signal 6 (core dumped)
Mar 10 16:47:38 imap1 dovecot: imap-login: Panic: file iostream-openssl.c: line 
599 (openssl_iostream_handle_error): assertion failed: (errno != 0)
Mar 10 16:47:38 imap1 dovecot: imap-login: Fatal: master: service(imap-login): 
child 30471 killed with signal 6 (core dumped)
...

Full backtrace of the coredump:
...
# gdb /usr/lib/dovecot/imap-login core.imap-login.30471

GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/dovecot/imap-login...Reading symbols from 
/usr/lib/debug/.build-id/3c/24fcde9d366e5cfd7615cc42e013a060d092e5.debug...done.
done.
[New LWP 30471]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `dovecot/imap-login'.
Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51

warning: Source file is more recent than executable.
51  }
(gdb) bt full
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
set = {__val = {0, 140427116827682, 140427117098374, 2147483649, 140427116658101, 0, 140427119440144, 140427116658101, 140427116658080, 
140427104016227, 140427119689728, 0, 140427104016288, 140427119440144, 18446744073709551615,

0}}
pid = 
tid = 
#1  0x7fb7bc5ba42a in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x7fff75b21b30, sa_sigaction = 0x7fff75b21b30}, sa_mask = {__val = {0, 140427119702945, 
94861876544520, 140427119702945, 140427120378435, 140735167994944, 94861876544520,
  140427119702945, 6008397283549395712, 140427119702945, 94861876544520, 140735167994832, 140427101843879, 3, 5, 140427119702945}}, 
sa_flags = -1130359394, sa_restorer = 0x0}

sigs = {__val = {32, 0 }}
#2  0x7fb7bca0c464 in default_fatal_finish (status=0, type=LOG_TYPE_PANIC) 
at failures.c:459
backtrace = 0x5646c0a5ae4c ""
recursed = 0
recursed = 0
#3  fatal_handler_real (ctx=, format=, 
args=) at failures.c:471
status = 0
#4  0x7fb7bca0c551 in i_internal_fatal_handler (ctx=, 
format=, args=) at failures.c:848
No locals.
#5  0x7fb7bc9622e3 in i_panic (format=format@entry=0x7fb7bb85ae70 "file %s: line 
%d (%s): assertion failed: (%s)") at failures.c:523
ctx = {type = LOG_TYPE_PANIC, exit_status = 0, timestamp = 0x0, 
timestamp_usecs = 0, log_prefix = 0x0, log_prefix_type_pos = 0}
args = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 
0x7fff75b21d20, reg_save_area = 0x7fff75b21c60}}
#6  0x7fb7bb858708 in openssl_iostream_handle_error (ssl_io=ssl_io@entry=0x5646c0a22aa0, ret=-1, 
type=type@entry=OPENSSL_IOSTREAM_SYNC_TYPE_HANDSHAKE, func_name=func_name@entry=0x7fb7bb85b1a7 "SSL_accept()") at iostream-openssl.c:599

errstr = 0x0
err = 5
__func__ = "openssl_iostream_handle_error"
#7  0x7fb7bb8588ea in openssl_iostream_handshake (ssl_io=0x5646c0a22aa0) at 
iostream-openssl.c:669
reason = 0x53621d7edb623b00 
error = 0x0
ret = 
__func__ = "openssl_iostream_handshake"
#8  0x7fb7bb8583da in openssl_iostream_more (ssl_io=ssl_io@entry=0x5646c0a22aa0, type=type@entry=OPENSSL_IOSTREAM_SYNC_TYPE_HANDSHAKE) at 
iostream-openssl.c:546

ret = 
#9  0x7fb7bb85a30f in i_stream_ssl_read_real 
(stream=stream@entry=0x5646c0a5b530) at istream-openssl.c:46
sstream = 0x5646c0a5b530
ssl_io = 0x5646c0a22aa0
buffer = 

Re: IMAP proxy

[Gandalf Corvotempesta]

2017-12-15 18:21 GMT+01:00 Aki Tuomi :
> Return from passdb, 'proxy host=your-new-host port=143 ssl=starttls'

So, instead of returning the current db output: "user, password,
userdb_mail, userdb_sieve, .."
is enough to only return "proxy host=your-new-host port=143 ssl=starttls" ?

Will dovecot automatically proxy pop3/imap on the new server ?

What about LDA ?

Re: IMAP proxy

[x9p]

On Fri, December 15, 2017 3:21 pm, Aki Tuomi wrote:
>
>> On December 15, 2017 at 6:57 PM Gandalf Corvotempesta
>>  wrote:
>>
>>
...
>> server would be proxied to the newer one automatically ?
>>
>> Any additional software or only a configuration change is required ?
>
> Return from passdb, 'proxy host=your-new-host port=143 ssl=starttls'
>
> Aki
>

Or masquerade all traffic to the new server with firewall rules:

iptables -t nat -A PREROUTING -i ethX -p tcp -m tcp --dport 143 -j DNAT
--to-destination 1.1.1.1

iptables -t nat -A POSTROUTING -d 1.1.1.1/32 -p tcp -m tcp --dport 143 -j
MASQUERADE

ethX --> interface where connections come from
1.1.1.1 --> new server IP address

Maybe smth is wrong up there, didnt tested, but I think its cool.


cheers.

--
x9p | PGP : 0x03B50AF5EA4C8D80 / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE
1524 E7EE

Re: IMAP proxy

[Aki Tuomi]

> On December 15, 2017 at 6:57 PM Gandalf Corvotempesta 
> <gandalf.corvotempe...@gmail.com> wrote:
> 
> 
> I'm migrating an old server to another old server (same dovecot
> version in both servers)
> The migration itself is straightforward, stop dovecot on the old
> server, migrate everything via rsync, start dovecot to the new server.
> 
> There is only one step left: change the dns configuration, pointing
> from the old server to the newer one.
> As most of domains are not managed by me and some other domains are
> pointing to our server via IP, I can't simply change the A record or
> wait for all users to change their domain configuration.
> 
> TL;DR: is possible to use dovecot as IMAP proxy so that even after
> changing our dns records, any user directly connecting to my old
> server would be proxied to the newer one automatically ?
> 
> Any additional software or only a configuration change is required ?

Return from passdb, 'proxy host=your-new-host port=143 ssl=starttls'

Aki

IMAP proxy

[Gandalf Corvotempesta]

I'm migrating an old server to another old server (same dovecot
version in both servers)
The migration itself is straightforward, stop dovecot on the old
server, migrate everything via rsync, start dovecot to the new server.

There is only one step left: change the dns configuration, pointing
from the old server to the newer one.
As most of domains are not managed by me and some other domains are
pointing to our server via IP, I can't simply change the A record or
wait for all users to change their domain configuration.

TL;DR: is possible to use dovecot as IMAP proxy so that even after
changing our dns records, any user directly connecting to my old
server would be proxied to the newer one automatically ?

Any additional software or only a configuration change is required ?

Re: IMAP proxy for Exchange - encrypted backend Communication?

[Thomas Koenig]

thx, I'll try it. Currently I use stunnel as a quick and dirty work around. 

Tom

Am 6. Januar 2017 18:04:57 MEZ schrieb Sami Ketola <sami.ket...@dovecot.fi>:
>
>> On 5 Jan 2017, at 22.56, tom <postu...@gmail.com> wrote:
>> 
>> Hello,
>> 
>> I try to setup a IMAP proxy for my old Exchange server.
>> Running Dovecot v2.x on Centos 7.
>> 
>> So far I follow http://wiki2.dovecot.org/HowTo/ImapcProxy and it seem
>> to work. The only but major  thing is with this setup - the
>> communication between proxy and backend is not encrypted. :(
>> 
>> To fix this, I changed the config and add:
>> imapc_ssl=imaps
>> imapc_port=993
>> 
>> but it doesnt work, because of verify failure of the self signed
>> backend certificate:
>
>you need to set:
>
>imapc_ssl_verify = no
>
>Regards,
>Sami

Re: IMAP proxy for Exchange - encrypted backend Communication?

[Sami Ketola]

> On 5 Jan 2017, at 22.56, tom <postu...@gmail.com> wrote:
> 
> Hello,
> 
> I try to setup a IMAP proxy for my old Exchange server.
> Running Dovecot v2.x on Centos 7.
> 
> So far I follow http://wiki2.dovecot.org/HowTo/ImapcProxy and it seem
> to work. The only but major  thing is with this setup - the
> communication between proxy and backend is not encrypted. :(
> 
> To fix this, I changed the config and add:
> imapc_ssl=imaps
> imapc_port=993
> 
> but it doesnt work, because of verify failure of the self signed
> backend certificate:

you need to set:

imapc_ssl_verify = no

Regards,
Sami

IMAP proxy for Exchange - encrypted backend Communication?

[tom]

Hello,

I try to setup a IMAP proxy for my old Exchange server.
Running Dovecot v2.x on Centos 7.

So far I follow http://wiki2.dovecot.org/HowTo/ImapcProxy and it seem
to work. The only but major  thing is with this setup - the
communication between proxy and backend is not encrypted. :(

To fix this, I changed the config and add:
imapc_ssl=imaps
imapc_port=993

but it doesnt work, because of verify failure of the self signed
backend certificate:

Jan  5 21:48:55 imap dovecot: imap(user1): Error:
imapc(192.168.1.1:993): Couldn't initialize SSL context: Can't verify
remote server certs without trusted CAs (ssl_client_ca_* settings)
Jan  5 21:48:55 imap dovecot: imap(user1): Error:
imapc(192.168.1.1:993): No SSL context
Jan  5 21:48:55 imap dovecot: imap(user1): Error: imapc: Command
failed: Disconnected from server
Jan  5 21:48:55 imap dovecot: imap(user1): Error: user tkoenig:
Initialization failed: Initializing mail storage from mail_location
setting failed: Mailbox list driver imapc: Failed to access imapc
backend
Jan  5 21:48:55 imap dovecot: imap(user1): Error: Invalid user
settings. Refer to server log for more information.

I didnt found anything in the documentation which tells dovcot not
verify the backend certificate.

Is there a know way to get it runing?

Many thanks for any hint!

regrds,
Tom

Re: Detect IMAP server domain name in Dovecot IMAP proxy

[Rick Romero]

 Quoting KT Walrus <ke...@my.walr.us>:


On Oct 12, 2016, at 2:07 PM, Rick Romero <ad...@vfemail.net> wrote:

Quoting KT Walrus <ke...@my.walr.us>:


I’m in the process of setting up a Dovecot IMAP proxy to handle a


number

of IMAP server domains. At the current time, I have my users divided
into 70 different groups of users (call them G1 to G70). I want each
group to configure their email client to access their mailboxes at a
domain name based on the group they belong to (e.g., g1.example.com
<http://g1.example.com/>, g2.example.com <http://g2.example.com/>, …,
g70.example.com <http://g70.example.com/>). I will only support TLS
encrypted IMAP connections to the Dovecot IMAP proxy (‘ssl=yes’ in


the

inet_listener). My SSL cert has alternate names for all 70 group domain
names.

I want the group domain to only support users that have been assigned

to

the group the domain name represents. That is, a user assigned to G23
would only be allowed to configure their email client for the IMAP
server named g23.example.com <http://g23.example.com/>.

My solution during testing has been to have the Dovecot IMAP proxy to
listen on different ports: 9930-. I plan to purchase 70 IPs, one

for

each group, and redirect traffic on port 993 to the appropriate Dovecot
IMAP proxy port based on the IP I assign to the group domain name in

the

site’s DNS. The SQL for handling the IMAP login uses the port number

of

the inet_listener

I think this could work in production, but it will cost me extra to

rent

the 70 IPs and might be a pain to manage. Eventually, I would like to
have over 5,000 groups so requiring an IP per group is less than ideal.
I also think having Dovecot IMAP proxy have 5,000 inet_listeners might
not work so well or might create too many threads/processes/ports to

fit

on a small proxy server.

I would rather have 1 public IP for each Dovecot IMAP proxy and somehow
communicate to the userdb which group domain name was configured in the
email client so only the users assigned to this group can login with
that username.

Anyone have any ideas?


Do you have a SQL userdb?
Create a table or a 'host' field for the user.

user_query = SELECT CONCAT(pw_name, '@', pw_domain) AS user, "89" as

uid,

"89" as gid, host, 'Y' AS proxy_maybe, pw_dir as home, pw_dir as
mail_home,
CONCAT('maildir:', pw_dir , '/Maildir/' ) as mail_location FROM vpopmail
WHERE pw_name = '%n' AND pw_domain = '%d'

(mine is based on qmail/vpopmail)

Then populate 'host' for each user if you don't have any other way of
programatically determining the host..


This doesn’t solve my problem. Indeed, I am doing this already:

password_query = SELECT password, 'Y' as proxy,
CONCAT_WS('@',username,domain) AS destuser, pms AS host, ’secretmaster'
AS master, ’secretpass' AS pass FROM users WHERE username='%n' and
domain='%d' and (group_id=%{lport}-9930 or %{lport}=143 or '%s'='lmtp')
and mailbox_status='active’;

This is the password_query I am using on the Dovecot IMAP proxy. This
proxy doesn’t use a user_query (only the real backend Dovecot servers
do). I allow authorizations on port 143 only for Postfix. Port 143

isn’t

exposed to the email clients (only 993 is used by email clients).

Anyway, checking the %{lport} allows only IMAP logins using the proper
domain name (IP or port) to allow the log in of the user.

I’m looking to find out the IMAP server name that the user configured
their email client with and make sure I only allow users to access their
mailboxes using their assigned IMAP server name.

Note that the problem I am trying to solve is if the user configures
their email client with the wrong IMAP server name (e.g. using
g2.example.com <http://g2.example.com/> instead of g23.example.com
<http://g23.example.com/>) and later I move G23 to another datacenter
and leave G2 in the current datacenter, they will not be able to access
their emails since the G2 datacenter doesn’t have their mailboxes any
more and the mailboxes for G23 are only in the G23 datacenter. My users
aren’t email experts and I don’t want them to have to discover that

they

made a typo in the original setup long after they have forgotten how
they set up the client in the first place.

To start with, the mailboxes will all be in the same datacenter, but I
want to be able to move some of the mailboxes to be geographically
closer to the users of those mailboxes (like Western users using Western
servers while Eastern users use a datacenter closer to the East coast).
Kevin


Gotcha.  I used g1.example.com and g2.example.com.   There are some DNS
services that will provide unique records based on the region of the caller
- but I have no experience with those.  That's what I'd prefer to do in
the long run though.

In my setup, the 'host' field still has the internal IP of the servers
physically hosting mail at g1 and g2 in order to allow the user to connect
to g1 and still be redirected to

Re: Detect IMAP server domain name in Dovecot IMAP proxy

[KT Walrus]

> On Oct 12, 2016, at 2:07 PM, Rick Romero <ad...@vfemail.net> wrote:
> 
> Quoting KT Walrus <ke...@my.walr.us>:
> 
>> I’m in the process of setting up a Dovecot IMAP proxy to handle a
> number
>> of IMAP server domains. At the current time, I have my users divided
>> into 70 different groups of users (call them G1 to G70). I want each
>> group to configure their email client to access their mailboxes at a
>> domain name based on the group they belong to (e.g., g1.example.com
>> <http://g1.example.com/>, g2.example.com <http://g2.example.com/>, …,
>> g70.example.com <http://g70.example.com/>). I will only support TLS
>> encrypted IMAP connections to the Dovecot IMAP proxy (‘ssl=yes’ in
> the
>> inet_listener). My SSL cert has alternate names for all 70 group domain
>> names.
>> 
>> I want the group domain to only support users that have been assigned to
>> the group the domain name represents. That is, a user assigned to G23
>> would only be allowed to configure their email client for the IMAP
>> server named g23.example.com <http://g23.example.com/>.
>> 
>> My solution during testing has been to have the Dovecot IMAP proxy to
>> listen on different ports: 9930-. I plan to purchase 70 IPs, one for
>> each group, and redirect traffic on port 993 to the appropriate Dovecot
>> IMAP proxy port based on the IP I assign to the group domain name in the
>> site’s DNS. The SQL for handling the IMAP login uses the port number of
>> the inet_listener
>> 
>> I think this could work in production, but it will cost me extra to rent
>> the 70 IPs and might be a pain to manage. Eventually, I would like to
>> have over 5,000 groups so requiring an IP per group is less than ideal.
>> I also think having Dovecot IMAP proxy have 5,000 inet_listeners might
>> not work so well or might create too many threads/processes/ports to fit
>> on a small proxy server.
>> 
>> I would rather have 1 public IP for each Dovecot IMAP proxy and somehow
>> communicate to the userdb which group domain name was configured in the
>> email client so only the users assigned to this group can login with
>> that username.
>> 
>> Anyone have any ideas?
>>  
> 
> Do you have a SQL userdb?
> Create a table or a 'host' field for the user.
> 
> user_query = SELECT CONCAT(pw_name, '@', pw_domain) AS user, "89" as uid,
> "89" as gid, host, 'Y' AS proxy_maybe, pw_dir as home, pw_dir as mail_home,
> CONCAT('maildir:', pw_dir , '/Maildir/' ) as mail_location FROM vpopmail
> WHERE pw_name = '%n' AND pw_domain = '%d'
> 
> (mine is based on qmail/vpopmail)
> 
> Then populate 'host' for each user if you don't have any other way of
> programatically determining the host..
> 

This doesn’t solve my problem. Indeed, I am doing this already:

password_query = SELECT password, 'Y' as proxy, CONCAT_WS('@',username,domain) 
AS destuser, pms AS host, ’secretmaster' AS master, ’secretpass' AS pass FROM 
users WHERE username='%n' and domain='%d' and (group_id=%{lport}-9930 or 
%{lport}=143 or '%s'='lmtp') and mailbox_status='active’;

This is the password_query I am using on the Dovecot IMAP proxy. This proxy 
doesn’t use a user_query (only the real backend Dovecot servers do). I allow 
authorizations on port 143 only for Postfix. Port 143 isn’t exposed to the 
email clients (only 993 is used by email clients).

Anyway, checking the %{lport} allows only IMAP logins using the proper domain 
name (IP or port) to allow the log in of the user.

I’m looking to find out the IMAP server name that the user configured their 
email client with and make sure I only allow users to access their mailboxes 
using their assigned IMAP server name.

Note that the problem I am trying to solve is if the user configures their 
email client with the wrong IMAP server name (e.g. using g2.example.com 
<http://g2.example.com/> instead of g23.example.com <http://g23.example.com/>) 
and later I move G23 to another datacenter and leave G2 in the current 
datacenter, they will not be able to access their emails since the G2 
datacenter doesn’t have their mailboxes any more and the mailboxes for G23 are 
only in the G23 datacenter. My users aren’t email experts and I don’t want them 
to have to discover that they made a typo in the original setup long after they 
have forgotten how they set up the client in the first place.

To start with, the mailboxes will all be in the same datacenter, but I want to 
be able to move some of the mailboxes to be geographically closer to the users 
of those mailboxes (like Western users using Western servers while Eastern 
users use a datacenter closer to the East coast).

Kevin

Re: Detect IMAP server domain name in Dovecot IMAP proxy

[Rick Romero]

Quoting KT Walrus <ke...@my.walr.us>:


I’m in the process of setting up a Dovecot IMAP proxy to handle a

number

of IMAP server domains. At the current time, I have my users divided
into 70 different groups of users (call them G1 to G70). I want each
group to configure their email client to access their mailboxes at a
domain name based on the group they belong to (e.g., g1.example.com
<http://g1.example.com/>, g2.example.com <http://g2.example.com/>, …,
g70.example.com <http://g70.example.com/>). I will only support TLS
encrypted IMAP connections to the Dovecot IMAP proxy (‘ssl=yes’ in

the

inet_listener). My SSL cert has alternate names for all 70 group domain
names.

I want the group domain to only support users that have been assigned to
the group the domain name represents. That is, a user assigned to G23
would only be allowed to configure their email client for the IMAP
server named g23.example.com <http://g23.example.com/>.

My solution during testing has been to have the Dovecot IMAP proxy to
listen on different ports: 9930-. I plan to purchase 70 IPs, one for
each group, and redirect traffic on port 993 to the appropriate Dovecot
IMAP proxy port based on the IP I assign to the group domain name in the
site’s DNS. The SQL for handling the IMAP login uses the port number of
the inet_listener

I think this could work in production, but it will cost me extra to rent
the 70 IPs and might be a pain to manage. Eventually, I would like to
have over 5,000 groups so requiring an IP per group is less than ideal.
I also think having Dovecot IMAP proxy have 5,000 inet_listeners might
not work so well or might create too many threads/processes/ports to fit
on a small proxy server.

I would rather have 1 public IP for each Dovecot IMAP proxy and somehow
communicate to the userdb which group domain name was configured in the
email client so only the users assigned to this group can login with
that username.

Anyone have any ideas?
 


Do you have a SQL userdb?
Create a table or a 'host' field for the user.

user_query = SELECT CONCAT(pw_name, '@', pw_domain) AS user, "89" as uid,
"89" as gid, host, 'Y' AS proxy_maybe, pw_dir as home, pw_dir as mail_home,
CONCAT('maildir:', pw_dir , '/Maildir/' ) as mail_location FROM vpopmail
WHERE pw_name = '%n' AND pw_domain = '%d'

(mine is based on qmail/vpopmail)

Then populate 'host' for each user if you don't have any other way of
programatically determining the host..

Rick

Detect IMAP server domain name in Dovecot IMAP proxy

[KT Walrus]

I’m in the process of setting up a Dovecot IMAP proxy to handle a number of 
IMAP server domains. At the current time, I have my users divided into 70 
different groups of users (call them G1 to G70). I want each group to configure 
their email client to access their mailboxes at a domain name based on the 
group they belong to (e.g., g1.example.com <http://g1.example.com/>, 
g2.example.com <http://g2.example.com/>, …, g70.example.com 
<http://g70.example.com/>). I will only support TLS encrypted IMAP connections 
to the Dovecot IMAP proxy (‘ssl=yes’ in the inet_listener). My SSL cert has 
alternate names for all 70 group domain names.

I want the group domain to only support users that have been assigned to the 
group the domain name represents. That is, a user assigned to G23 would only be 
allowed to configure their email client for the IMAP server named 
g23.example.com <http://g23.example.com/>. 

My solution during testing has been to have the Dovecot IMAP proxy to listen on 
different ports: 9930-. I plan to purchase 70 IPs, one for each group, and 
redirect traffic on port 993 to the appropriate Dovecot IMAP proxy port based 
on the IP I assign to the group domain name in the site’s DNS. The SQL for 
handling the IMAP login uses the port number of the inet_listener 

I think this could work in production, but it will cost me extra to rent the 70 
IPs and might be a pain to manage. Eventually, I would like to have over 5,000 
groups so requiring an IP per group is less than ideal. I also think having 
Dovecot IMAP proxy have 5,000 inet_listeners might not work so well or might 
create too many threads/processes/ports to fit on a small proxy server.

I would rather have 1 public IP for each Dovecot IMAP proxy and somehow 
communicate to the userdb which group domain name was configured in the email 
client so only the users assigned to this group can login with that username.

Anyone have any ideas?

For HTTP traffic, it is easy to query the host in the HTTP Request, but I don’t 
think IMAP traffic has such host info in it. Does the Dovecot IMAP proxy 
receive the hostname from the email client when exchanging SSL certs (like SNI 
for HTTPS)?

Or, maybe I should have group domain in the username used to log in with (e.g., 
username+...@example.com <mailto:username+...@example.com> or 
usern...@g23.example.com <mailto:usern...@g23.example.com>). I don’t like to 
make the user configure their email client to log in with a name that is 
different than their mailbox address. It is simpler to just have them configure 
their email client with usern...@example.com <mailto:usern...@example.com> for 
both authorization and for the from/sender headers in the messages. 

Anyway, any ideas of how to set this up in production?

Re: FTS search used / useful on an IMAP proxy?

[Timo Sirainen]

On 28 Jun 2016, at 16:07, Luca Lesinigo <l...@lm-net.it> wrote:
> 
> We are preparing an IMAP proxy based on dovecot-2.2.22, basic proxy 
> functionality is already working and I’m trying to understand if having the 
> FTS service configured on the dovecot *proxy* would be of any use.
> 
> I do suspect it would be useless, I guess dovecot in imap proxy mode just 
> forwards any command to the backend and does not bother to do anything about 
> it, but I’m failing to find a definitive answer in the documentation. If I am 
> guessing correctly, an fts service would only be useful if configured and 
> working on the actual backend.
> 
> Can anyone clarify my doubts?

If you want to use doveadm fts optimize/rescan commands via doveadm proxy, you 
need to load fts plugin on the proxy to get the commands. But otherwise there's 
no reason for it.

Re: FTS search used / useful on an IMAP proxy?

[Michael Slusarz]

> 
> On June 28, 2016 at 7:07 AM Luca Lesinigo <l...@lm-net.it> wrote:
> 
> We are preparing an IMAP proxy based on dovecot-2.2.22, basic proxy 
> functionality is already working and I’m trying to understand if having the 
> FTS service configured on the dovecot *proxy* would be of any use.
> 
> I do suspect it would be useless, I guess dovecot in imap proxy mode just 
> forwards any command to the backend and does not bother to do anything about 
> it, but I’m failing to find a definitive answer in the documentation. If I am 
> guessing correctly, an fts service would only be useful if configured and 
> working on the actual backend.
> 

FTS only makes sense on backend, where the search would be executed.

michael

FTS search used / useful on an IMAP proxy?

[Luca Lesinigo]

We are preparing an IMAP proxy based on dovecot-2.2.22, basic proxy 
functionality is already working and I’m trying to understand if having the FTS 
service configured on the dovecot *proxy* would be of any use.

I do suspect it would be useless, I guess dovecot in imap proxy mode just 
forwards any command to the backend and does not bother to do anything about 
it, but I’m failing to find a definitive answer in the documentation. If I am 
guessing correctly, an fts service would only be useful if configured and 
working on the actual backend.

Can anyone clarify my doubts?

thank you,
--
Luca Lesinigo

Re: userdb for imap proxy

[KT Walrus]

> In proxy and director configuration you can configure only the passdb lookup.

Thanks. I got my installation working yesterday. I have proxies for LMTP and 
IMAP (no POP3) backed by a farm of Dovecot servers. The IMAP proxy listens on 
70 different IPs/ports and does passdb lookups to authenticate the users based 
on the incoming IP/port. The passdb lookups select the particular backend 
server containing the user’s mailbox. SMTP (Postfix) does authentication 
through the IMAP proxy and mail delivery through the LMTP proxy. I haven’t 
bothered to set up an SMTP proxy yet, since my SMTP server will only handle 
submission and not relay. Submitted messages are queued to a Redis queue for 
importation into a MySQL database where the messages are held giving the sender 
the ability to edit/delete their messages before midnight. Messages are sent 
out to the recipient mailboxes in the early morning through another internal 
SMTP server talking to the LMTP proxy. 

For my site, I only want to delivery new messages once a day (in the early 
morning) with the sender/mailbox admin having the opportunity to edit/delete 
the messages the day it is sent by the sender.

All appears to be working well, but I’m currently only doing SSL/TLS on the 
edge (in SMTP/IMAP) and haven’t figured out how to do SSL from end to end. I’m 
not sure if end to end SSL is important for my site, but it seems to be a trend 
that should not be ignored.

Kevin

> On Jun 8, 2016, at 3:49 AM, Alessio Cecchi <ales...@skye.it> wrote:
> 
> 
> 
> Il 07/06/2016 17:42, KT Walrus ha scritto:
>> If I’m running only imap-login service in my dovecot imap proxy, do I need 
>> to configure userdb or only passdb?
>> 
> 
> In proxy and director configuration you can configure only the passdb lookup.
> -- 
> Alessio Cecchi
> Postmaster @ http://www.qboxmail.it
> https://www.linkedin.com/in/alessice

Re: userdb for imap proxy

[Alessio Cecchi]

Il 07/06/2016 17:42, KT Walrus ha scritto:

If I’m running only imap-login service in my dovecot imap proxy, do I need to 
configure userdb or only passdb?



In proxy and director configuration you can configure only the passdb 
lookup.

--
Alessio Cecchi
Postmaster @ http://www.qboxmail.it
https://www.linkedin.com/in/alessice

userdb for imap proxy

[KT Walrus]

If I’m running only imap-login service in my dovecot imap proxy, do I need to 
configure userdb or only passdb?

Re: plugin can't be loaded in imap proxy mode

[Timo Sirainen]

On 23 Mar 2016, at 07:28, Zhong, Xun <xun.zh...@hpe.com> wrote:
> 
> Hi, All
> 
> I tested [last-login] plugin, when I directly logined by pam way, the 
> last-login plugin can load and run, but when I logined  by proxy way(Dovecot 
> forward my imap request to another imap server), the last-login plugin did 
> not load and run.
> 
> Thanks
> 
> 
> From: Zhong, Xun
> Sent: Wednesday, March 23, 2016 11:47 AM
> To: 'dovecot@dovecot.org' <dovecot@dovecot.org>
> Subject: plugin can't be loaded in imap proxy mode
> 
> When dovecot v2.2.21 work in IMAP proxy mode,  plugin can not be loaded and 
> work, so my question is: plugins do not work in proxy mode?
> Anyone can help me ? Thanks a lot.

Proxying doesn't support any plugins.

RE: plugin can't be loaded in imap proxy mode

[Zhong, Xun]

Hi, All

I tested [last-login] plugin, when I directly logined by pam way, the 
last-login plugin can load and run, but when I logined  by proxy way(Dovecot 
forward my imap request to another imap server), the last-login plugin did not 
load and run.

Thanks


From: Zhong, Xun
Sent: Wednesday, March 23, 2016 11:47 AM
To: 'dovecot@dovecot.org' <dovecot@dovecot.org>
Subject: plugin can't be loaded in imap proxy mode

 When dovecot v2.2.21 work in IMAP proxy mode,  plugin can not be loaded and 
work, so my question is: plugins do not work in proxy mode?
Anyone can help me ? Thanks a lot.

plugin can't be loaded in imap proxy mode

[Zhong, Xun]

 When dovecot v2.2.21 work in IMAP proxy mode,  plugin can not be loaded and 
work, so my question is: plugins do not work in proxy mode?
Anyone can help me ? Thanks a lot.

Problem with IMAP-Proxy and M$ Exchange-Server

[Luca Bertoncello]

Hi List!

I **HATE** Exchange-Server. I think it's not able to manage E-Mails,  
but we have to use it at work...


Well, we need to read the E-Mails from outside, so I configured  
Dovecot with IMAPC to connect to the Exchange-Server.

It works, but I have a problem...

If I receive an E-Mail, and I read it from my phone (for example), I  
see in Outlook that the E-Mail as been read.
If I move the E-Mail in another folder from my phone, I see in Outlook  
the E-Mail in the new folder **AND** in the INBOX.
If I check my E-Mail from another phone (or Mail-Client), I see the  
E-Mail just in the new folder, so I think, the Exchange-Server did  
move it.


But Outlook does not get this information...

Any idea how can I solve my problem?

Thanks
Luca Bertoncello
(lucab...@lucabert.de)

Configuring LMTP/IMAP proxy

[Le Moing , Guenhaël]

Hi,
First, my version:

[root@centos1 conf.d]# dovecot --version
2.2.15
[root@centos1 conf.d]#

I have already configured dovecot that way (on one single VM, so everythinh is 
stored on the same machine)


-  A post fix server sending out to LMTP

-  LMTP is the dovecot LMTP server, configured with lmtp-proxy= no,

-  LMTP checks the users receiving messages by using Linux passwd file 
userdb  section in auth-passwdfile.conf.ext,

-  IMAP LOGINS are authenticated using DIGEST-MD5 also configured in 
auth-passwdfile.conf.ext :

-
# Authentication for passwd-file users. Included from 10-auth.conf.
#
# passwd-like file with specified location.
# doc/wiki/AuthDatabase.PasswdFile.txt

passdb {
  driver = passwd-file
  args = scheme=DIGEST-MD5 username_format=%n /etc/dovecot/users.DIGEST-MD5
#args = scheme=CRAM-MD5 username_format=%n /etc/dovecot/users.CRAM-MD5
}

userdb {
  driver = passwd

  # Default fields that can be overridden by passwd-file
  #default_fields = quota_rule=*:storage=1G
default_fields = uid=root gid=root home=/home/%n

  # Override fields from passwd-file
  #override_fields = home=/home/virtual/%u
}

This works fine, I can send messages through SMTP, and read them using IMAP 
access.

But now, I woukd like to have  amore structured architecture with :


-  A LMTP server configured as PROXY : has to identify the user, and 
proxy the request to the backen LMTP server (where its mailbox will stand)

-  On backend, I also have  to start another  LMTP serevr, but not 
configured in proxy mode of course,

-  Same for IMAP : IMAP proxy first checking LOGIN credentials, and 
then forwarding to IMAP server running on backend.

As a first step, I would like to continue using my files to authenticate uses 
(so passwd file , and users.DIGEST-MD5), but final goal will be using a MySQL 
DB.

So my main question are :


1)  I just made a test and changed  lmtp_proxy to yes in my 
20_lmtp.cong file, but, strangely, this did not cah nge anything ! The LMTP 
process still receives mails and stores them on the local host ,

2)  Are there any documentation describing this process in details ? I only 
found some pieces on wiki pages, but not enough unfortunately ...

3)  Are there some configuratiion files ready to use for my configuration.

Thanks in advance.

Cordialement.

Guenhaël.
This message contains information that may be privileged or confidential and is 
the property of the Capgemini Group. It is intended only for the person to whom 
it is addressed. If you are not the intended recipient, you are not authorized 
to read, print, retain, copy, disseminate, distribute, or use this message or 
any part thereof. If you receive this message in error, please notify the 
sender immediately and delete all copies of this message.

Re: Configuring LMTP/IMAP proxy

[Manuel Delgado]

On Mon, Dec 22, 2014 at 12:06 PM, Le Moing, Guenhaël 
guenhael.le-mo...@capgemini.com wrote:


 So my main question are :


 1)  I just made a test and changed  lmtp_proxy to yes in my
 20_lmtp.cong file, but, strangely, this did not cah nge anything ! The LMTP
 process still receives mails and stores them on the local host ,

 2)  Are there any documentation describing this process in details ? I
 only found some pieces on wiki pages, but not enough unfortunately ...

 3)  Are there some configuratiion files ready to use for my
 configuration.

 Thanks in advance.

 Cordialement.

 Guenhaël.
 This message contains information that may be privileged or confidential
 and is the property of the Capgemini Group. It is intended only for the
 person to whom it is addressed. If you are not the intended recipient, you
 are not authorized to read, print, retain, copy, disseminate, distribute,
 or use this message or any part thereof. If you receive this message in
 error, please notify the sender immediately and delete all copies of this
 message.



Hi Guenhaël

I was working in a project with 2 front-end servers (POP3/IMAP/LMTP
proxy) and 2 back-end servers (Mailbox). I configured Director[1] in the
proxies to reduce conflicts and master-password[2]. My proxies don't have
userdb nor location because they do not store emails, those configs are in
the back-end.
My backend is an standalone Dovecot but with master-password allowed in
trusted networks as described in the wiki[2][3].
Also, I configured PoolMon[4] in all proxies and added Debian/Ubuntu
scripts[5].


[... I removed some configs for short...]

 PROXY doveconf --
[...]
director_mail_servers = 10.0.0.74 10.0.0.75
director_servers = 10.0.0.72 10.0.0.73
doveadm_port = 24245
lmtp_proxy = yes
passdb {
  driver = pam
  override_fields = proxy=y ssl=any-cert master=pr...@alpha.mydomain.com
pass=PASSWORD-PROXY-MASTER
}
service director {
  fifo_listener login/proxy-notify {
mode = 0600
user = $default_login_user
  }
  inet_listener {
port = 9090
  }
  unix_listener director-userdb {
mode = 0600
  }
  unix_listener login/director {
mode = 0666
  }
}
service doveadm {
  inet_listener {
port = 24245
  }
}
service imap-login {
  executable = imap-login director
  [...]
}
service ipc {
  unix_listener ipc {
user = dovecot
  }
}
service lmtp {
  executable = lmtp -L
  [...]
}
service pop3-login {
  executable = pop3-login director
  [...]
}
protocol lmtp {
  auth_socket_path = director-userdb
  [...]
}
protocol doveadm {
  auth_socket_path = director-userdb
}
local 10.0.0.0/24/24 {
  doveadm_password = PASSWORD-DOVEADM
  doveadm_port = 24245
}
- END --


Not sure if this will work for you but maybe it helps you to get an idea.

Regards,
Manuel Delgado

---
*Usuario Linux* *#520940 http://counter.li.org/*

Bach. Computación e Informática
Universidad de Costa Rica


[1] http://wiki2.dovecot.org/Director
[2] http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy
[3] http://wiki2.dovecot.org/Authentication/MasterUsers
[4] http://www.dovecot.org/list/dovecot/2010-August/051946.html
[5] https://github.com/valarauco/poolmon

[Dovecot] IMAP proxy with master user and CRAM-MD5 auth mechanism

[ML mail]

Hello,

I would like to know if it possible to have a dovecot IMAP proxy frontend where 
CRAM-MD5 can be used as auth mechanism (assuming I would be using a master 
user/password on the dovecot IMAP backend/mailbox)? I have read a few times the 
following http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy page but 
somehow it is still unclear to me if the IMAP proxy then needs to have all the 
users and passwords in its proxy table in the database? Unfortunately there are 
no examples for that case.

Also with this case scenario is it possible to store the users' passwords in 
CRAM-MD5 format?

Any pointers or documentation to such a scenario would be great.

Thanks and regards
M.L.

Re: [Dovecot] IMAP proxy with master user and CRAM-MD5 auth mechanism

[ML mail]

Hi Trent,

Thanks for your exhaustive explanation now it is clear. What was unclear to me 
was the fact that the IMAP proxy server has to take care of the 
authentication in this case of using a master user and therefore it 
needs access to the passwords (in my case stored in the mailbox table in 
PostgreSQL). I still have two open questions:

1) Do I really need the userdb on my IMAP proxy config as you mention in 
your mail? In my understanding the passdb should be enough (which in my 
case will use a SQL query joining the result of my proxy and mailbox 
tables). 


2) 
Is this correct that the IMAP backend (the mailbox server) in this case 
scenario has to use PLAIN authentication and can NOT use CRAM-MD5?

Regards,ML

[Dovecot] Dovecot IMAP proxy for gmail IMAP server

[CM Reddy]

Hi All,

I am experimenting to use Dovecot as an IMAP proxy to GMAIL server. I would
like configure my localhost as an Dovecot IMAP proxy and trying to access
my GMAIL using the Thunderbird email client. Please share the steps to
configure and access my mails from GMAIL server.

- Thanks
CM Reddy

Re: [Dovecot] Dovecot IMAP proxy for gmail IMAP server

[Reindl Harald]

Am 17.01.2014 18:09, schrieb CM Reddy:
 I am experimenting to use Dovecot as an IMAP proxy to GMAIL server. I would
 like configure my localhost as an Dovecot IMAP proxy and trying to access
 my GMAIL using the Thunderbird email client. Please share the steps to
 configure and access my mails from GMAIL server

http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy

but what is the benfit?

you can configure encryption and whatnot different
but that does not change the connection between your
proxy and the final destination - the weakest part
stays as problem independent what you do in the middle
of the connection



signature.asc
Description: OpenPGP digital signature

[Dovecot] Message filtering capability at IMAP proxy

[CM Reddy]

Hi,
I would like process the messages at Dovecot IMAP proxy. In one of the
documents, it was mentioned that Dovecot proxy currently supports on the
fly message filtering in latest releases. Will it it possible to extend the
filtering feature to handle the following requirements.

1. Parse the email at IMAP proxy.
2. Replace the links with some other secure links.
3. Replace the attachments with some other document.


It would be great, if any one can provide some pointers to move forward.

- Thanks in advance
CM Reddy

Re: [Dovecot] Message filtering capability at IMAP proxy

[Robert Schetterer]

Am 10.01.2014 09:15, schrieb CM Reddy:
 Hi,
 I would like process the messages at Dovecot IMAP proxy. In one of the
 documents, it was mentioned that Dovecot proxy currently supports on the
 fly message filtering in latest releases. Will it it possible to extend the
 filtering feature to handle the following requirements.

dont think this is a a good idea

 
 1. Parse the email at IMAP proxy.

in general ,might be to much overlay in realtime processing

 2. Replace the links with some other secure links.
 3. Replace the attachments with some other document.

this general a job for some virus/antispam scanner etc at
smtp/lmtp/lda/sieve income stage,
additional you might get conflicting with laws and break signatures etc

 
 
 It would be great, if any one can provide some pointers to move forward.
 
 - Thanks in advance
 CM Reddy
 



Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

[Dovecot] weird situation with pop3/imap proxy and postfix authentication

[Leonardo Rodrigues]

Hello dovecot mailing list,

I have a server running postfix and dovecot. I havea configuration 
on dovecot that allows me to provide imap4/pop3 messages for local 
hosted users as well as for proxied users on other servers.


Basically, i have a simple MySQL table (imapproxy) with two fields, 
'domain' and 'host'. My password_query isa 'UNION' query, exactly like:


password_query=select endereco as user, password, '/var/spool/mail/%u' 
as userdb_home, 'maildir:/var/spool/mail/%u' as userdb_mail, 8 as 
userdb_uid, 12 as userdb_gid, concat('*:storage=', quota) as 
userdb_quota_rule, 'Trash:storage=+100M' as userdb_quota_rule2, 'Y' as 
proxy_maybe, '10.252.38.2' as host from emails where endereco = '%u' and 
ativa = '1' UNION select NULL as user, '%w' as password, NULL as 
userdb_home, NULL as userdb_mail, NULL as userdb_uid, NULL as 
userdb_gid, NULL as userdb_quota_rule, NULL as userdb_quota_rule2, 'Y' 
as proxy_maybe, imapproxy.host as host from imapproxy where 
imapproxy.dominio = '%d'


the 10.252.38.2 address, on the query, is my local server

when this query received a local user, from a domain that is NOT listed 
on the imapproxy table, results are like:


*** 1. row ***
user: localu...@domain.com.br
password: (SSHA256 encrypted password)
userdb_home: /var/spool/mail/localu...@domain.com.br
userdb_mail: maildir:/var/spool/mail/localu...@domain.com.br
userdb_uid: 8
userdb_gid: 12
 userdb_quota_rule: *:storage=51200
userdb_quota_rule2: Trash:storage=+100M
proxy_maybe: Y
host: 10.252.38.2


when it receives a proxied domain, results are:


*** 1. row ***
user: NULL
password: password
userdb_home: NULL
userdb_mail: NULL
userdb_uid: NULL
userdb_gid: NULL
 userdb_quota_rule: NULL
userdb_quota_rule2: NULL
proxy_maybe: Y
host: 10.254.116.9


This is working just fine for IMAP4 and POP3 proxying. Local users 
(which domains are NOT listed on imapproxy table) can successfully login 
to their accountsas well as users from domains listed on imapproxy table 
can successfully login to their accounts.



On SMTP authentication, tough, things are not so fine. SMTP 
authentication is provided by dovecot to postfix:


[root@correio dovecot]# postconf mail_version
mail_version = 2.7.1
[root@correio dovecot]#

smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
smtpd_sasl_path = /var/spool/postfix/private/auth
smtpd_sasl_security_options = noanonymous
smtpd_sasl_type = dovecot


service auth is defined on dovecot confs as:


service auth {
unix_listener auth-userdb {
mode = 0600
user = mail
group = mail
  }
  # Postfix smtp-auth
unix_listener /var/spool/postfix/private/auth {
mode = 0660
user = postfix
group = postfix
  }
}



and it seems to be allowing ANYuser on any domain listed on the 
imapproxy table domains to login, even if the user does not exists or 
provides wrong password. In fact, it seems dovecot returns OK to postfix 
even without trying to contact the assigned server to that domain, as i 
cannot find any password-failed-specific log to that user on the 
specific server. Example:


(a proxied domain)

[root@correio dovecot]# telnet mail.proxieddomain.com.br 110
Trying 10.254.116.9...
Connected to mail.proxieddomain.com.br (10.254.116.9).
Escape character is '^]'.
+OK Dovecot ready.
user te...@proxieddomain.com.br
+OK
pass password
-ERR Authentication failed.

(i can successfully find this auth trial and fail on 10.254.116.9 logs)


but on SMTP authentication, i have:

[root@correio dovecot]# perl -MMIME::Base64 -e \ 'print 
encode_base64(teste\@proxieddomain.com.br\0teste\@proxieddomain.com.br\0password);'

x(not the real encoded pass)=
[root@correio dovecot]#


[root@correio dovecot]# telnet localhost 587
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
220 mail.domain.com.br ESMTP
EHLO test
250-mail.domain.com.br
[ ... ]
AUTH PLAIN (encoded string returned by perl encode_base64)
235 2.7.0 Authentication successful


(and i cannot even find any authentication log, fail or success, on the 
specific server for proxieddomain.com.br)



dovecot version is:

[root@correio dovecot]# dovecot --version
2.2.2
[root@correio dovecot]#


what am i doing wrong here ? How to have dovecot to really check users 
before giving OK to postfix on SMTP authentications ?


Thanks for any hints !



--


Atenciosamente / Sincerily,
Leonardo Rodrigues
Solutti Tecnologia
http://www.solutti.com.br

Minha armadilha de SPAM, NÃO mandem email
gertru...@solutti.com.br
My SPAMTRAP, do not email it

[Dovecot] Transparent IMAP proxy

[Yonatan Broza]

Hi,

I'm considering patching Dovecot to work as a transparent (and virus 
scanning) IMAP proxy.


What is the appropriate feature to extend? (I've considered the 
following: IMAPC and reverse proxying, with IMAPC looking more promising 
since it actually parses IMAP communication).


Can anyone who is familiar with the IMAPC code recommend what are the 
most appropriate locations in the code to make the required changes? 
(I've considered hooking the storage virtual functions and making the 
settings local rather than global - but would appreciate more specific 
ideas).


Thanks.

Re: [Dovecot] Transparent IMAP proxy

[Reindl Harald]

Am 16.06.2013 15:42, schrieb Yonatan Broza:
 I'm considering patching Dovecot to work as a transparent (and virus 
 scanning) IMAP proxy

why would someone implement a virus scanner on the IMAP-level?
what happens with POP3?

this has to be done on SMTP level long before the message is stored
and not every time a client is downloading a message



signature.asc
Description: OpenPGP digital signature

Re: [Dovecot] Transparent IMAP proxy

[Timo Sirainen]

On 16.6.2013, at 16.42, Yonatan Broza yona...@cuppcomputing.com wrote:

 I'm considering patching Dovecot to work as a transparent (and virus 
 scanning) IMAP proxy.
 
 What is the appropriate feature to extend? (I've considered the following: 
 IMAPC and reverse proxying, with IMAPC looking more promising since it 
 actually parses IMAP communication).
 
 Can anyone who is familiar with the IMAPC code recommend what are the most 
 appropriate locations in the code to make the required changes? (I've 
 considered hooking the storage virtual functions and making the settings 
 local rather than global - but would appreciate more specific ideas).

imapc and http://dovecot.org/patches/2.1/mail-filter.tar.gz can do this. But 
note that the MIME structure or the parts' sizes must not change. Basically 
you'll have to replace the viruses with empty spaces or something.

imapc isn't very efficient though. It translates all IMAP commands to rather 
simple ones. So for example a SEARCH won't be passed through to the backend 
server.

Re: [Dovecot] Transparent IMAP proxy

[Eugene]

From: Reindl Harald

why would someone implement a virus scanner on the IMAP-level?
what happens with POP3?



this has to be done on SMTP level long before the message is stored
and not every time a client is downloading a message


Or, as an alternative, most desktop antivirus tools have a mail-scanning 
capability.
But SMTP is certainly better (though IMO even that is not really needed if 
you have reasonable antispam filtering and think before opening 
attachments).


Cheers
Eugene 

Re: [Dovecot] Transparent IMAP proxy

[Gedalya]

On 06/16/2013 01:31 PM, Eugene wrote:
Or, as an alternative, most desktop antivirus tools have a 
mail-scanning capability.
But SMTP is certainly better (though IMO even that is not really 
needed if you have reasonable antispam filtering and think before 
opening attachments). 
In my experience, an anti-virus is left with almost nothing to catch if 
you let spamassassin reject mail above a certain score, and so the AV 
never gets to see those.
However one benefit of delaying AV scanning is that you get later 
signatures, so you could potentially deal better with 0-hour viruses. 
But overall AV is just ineffective.

Re: [Dovecot] Transparent IMAP proxy

[Andrzej A. Filip]

On 06/16/2013 03:42 PM, Yonatan Broza wrote:
 I'm considering patching Dovecot to work as a transparent (and virus
 scanning) IMAP proxy.
 [...]

Have you considered non transparent caching proxy?

[Dovecot] IMAP proxy - can it detect parodying to itself?

[Graham Leggett]

Hi all,

I have some IMAP servers fronted with separate perdition processes, and it 
would be ideal if I could collapse this down to having dovecot do both the IMAP 
proxying and the IMAP serving at the same time on the same IP addresses.

One of the fields in my LDAP entries contains the canonical name of the server 
that hosts their mailbox, and if I follow the manual at 
http://wiki2.dovecot.org/PasswordDatabase/ExtraFields#LDAP I could add the host 
field to enable proxying.

My question is whether dovecot has the ability to notice whether dovecot is 
being asked to proxy to itself, in other words the value of host is the 
current dovecot server, and when this happens, ignore the proxy and just be a 
straight IMAP server, because the user has connected to the right box already.

Is this possible?

Regards,
Graham
--



smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] IMAP proxy - can it detect parodying to itself?

[Graham Leggett]

On 26 Nov 2012, at 4:24 PM, Graham Leggett minf...@sharp.fm wrote:

 I have some IMAP servers fronted with separate perdition processes, and it 
 would be ideal if I could collapse this down to having dovecot do both the 
 IMAP proxying and the IMAP serving at the same time on the same IP addresses.

A heartfelt thanks to Apple Autocorrect for changing parodying, oops, I meant 
parodying, aargh, I meant p r o x y i n g, to parodying in the subject of 
this message.

Regards,
Graham
--



smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] IMAP proxy - can it detect parodying to itself?

[Ben Morrow]

At  4PM +0200 on 26/11/12 you (Graham Leggett) wrote:
 Hi all,
 
 I have some IMAP servers fronted with separate perdition processes,
 and it would be ideal if I could collapse this down to having dovecot
 do both the IMAP proxying and the IMAP serving at the same time on the
 same IP addresses.
 
 One of the fields in my LDAP entries contains the canonical name of
 the server that hosts their mailbox, and if I follow the manual at
 http://wiki2.dovecot.org/PasswordDatabase/ExtraFields#LDAP I could add
 the host field to enable proxying.

You also need the 'proxy' or 'proxy_maybe' field, which is a boolean
(the field just needs to be present). If you just configure 'host' you
will get login referrals, which is not what you want.

 My question is whether dovecot has the ability to notice whether
 dovecot is being asked to proxy to itself, in other words the value of
 host is the current dovecot server, and when this happens, ignore
 the proxy and just be a straight IMAP server, because the user has
 connected to the right box already.

This is what happens if you use 'proxy_maybe' instead of 'proxy'.

Ben

[Dovecot] IMAP proxy between Office 365 client/Sun Messaging Server

[John Dalbec]

I would like to be able to migrate messages from existing end-user 
accounts on Sun Messaging Server to Office 365 using an administrator 
login and password.  Unfortunately the migration tool for Office 365 
doesn't support SASL AUTHENTICATE PLAIN login.


The online documentation I've found suggests that I should be able to 
configure dovecot as an IMAP proxy and have it log in to Sun Messaging 
Server with AUTHENTICATE PLAIN and encode_base64(user\0admin\0adminpw) 
in response to a login from the Office 365 migration tool.  I'd like to 
configure dovecot to run only the IMAP proxy if possible.  I was 
thinking of setting all accounts to use the same (strong) password in 
the proxy.


Would anyone be willing to share a sample configuration?

Thanks,
John Dalbec
ellucian Luminis system administrator
Youngstown State University

Re: [Dovecot] IMAP proxy between Office 365 client/Sun Messaging Server

[Robert Schetterer]

Am 31.10.2012 16:15, schrieb John Dalbec:
 I would like to be able to migrate messages from existing end-user
 accounts on Sun Messaging Server to Office 365 using an administrator
 login and password.  Unfortunately the migration tool for Office 365
 doesn't support SASL AUTHENTICATE PLAIN login.
 
 The online documentation I've found suggests that I should be able to
 configure dovecot as an IMAP proxy and have it log in to Sun Messaging
 Server with AUTHENTICATE PLAIN and encode_base64(user\0admin\0adminpw)
 in response to a login from the Office 365 migration tool.  I'd like to
 configure dovecot to run only the IMAP proxy if possible.  I was
 thinking of setting all accounts to use the same (strong) password in
 the proxy.
 
 Would anyone be willing to share a sample configuration?
 
 Thanks,
 John Dalbec
 ellucian Luminis system administrator
 Youngstown State University

perhaps look in this

http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy
http://wiki.dovecot.org/HowTo/ImapProxy


Best Regards
MfG Robert Schetterer

-- 
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Joerg Heidrich

Re: [Dovecot] imap proxy setup - killed with signal 11

[Timo Sirainen]

On 11.10.2012, at 20.35, btb wrote:

 i'm setting up an imap proxy in front of a novell groupwise server.  it seems 
 to so far be partially working, but dovecot is having trouble in certain 
 cases.  i expect that it's ultimately due to what i believe is a very poor 
 implementation of imap provided by groupwise [at least based on other 
 experiences in the past] - but that's a big part of why i'd like to have 
 dovecot in between it and clients.
 
 os is ubuntu 12.10 development/beta, dovecot is 2.1.7 courtesy of ubuntu's 
 packages.

There have been a couple of imapc fixes since v2.1.7. It's possible that the 
crash is fixed by one of them.

 Oct 11 13:24:52 halo dovecot: imap(jdoe): Error: imapc: Mailbox 'Trash' state 
 corrupted: Expunged message reappeared in session (uid=6282  next_uid=6283)

Could you get imapc rawlogs where this happens? Point imapc_rawlog_dir setting 
to some directory.

 #0  0x in ?? ()
 No symbol table info available.
 #1  0x7fc7f6cb611e in imap_parser_reset (parser=0x7fc7f8a0f3a0) at 
 imap-parser.c:93
 No locals.
 #2  0x7fc7f6f7ada7 in imapc_connection_input_reset 
 (conn=conn@entry=0x7fc7f8a0d270) at imapc-connection.c:664
 No locals.
 #3  0x7fc7f6f7c6f4 in imapc_connection_input_untagged 
 (conn=conn@entry=0x7fc7f8a0d270) at imapc-connection.c:908

This backtrace unfortunately doesn't make it very clear what the problem is. 
I'd guess it's trying to use already freed memory (one such bug was already 
fixed).

[Dovecot] imap proxy setup - killed with signal 11

[btb]

hi-

i'm setting up an imap proxy in front of a novell groupwise server.  it 
seems to so far be partially working, but dovecot is having trouble in 
certain cases.  i expect that it's ultimately due to what i believe is a 
very poor implementation of imap provided by groupwise [at least based 
on other experiences in the past] - but that's a big part of why i'd 
like to have dovecot in between it and clients.


below is information collected during starting of dovecot, 
opening/initial connection from a client [os x mail.app], closing of the 
client, and stopping of dovecot.


os is ubuntu 12.10 development/beta, dovecot is 2.1.7 courtesy of 
ubuntu's packages.


log entries:
Oct 11 13:24:33 halo dovecot: master: Dovecot v2.1.7 starting up
Oct 11 13:24:49 halo dovecot: imap-login: Login: user=jdoe, 
method=PLAIN, rip=10.68.40.110, lip=10.59.1.53, mpid=14171, TLS, 
session=iHgu2cvLlwAKRChu
Oct 11 13:24:50 halo dovecot: imap-login: Login: user=jdoe, 
method=PLAIN, rip=10.68.40.110, lip=10.59.1.53, mpid=14174, TLS, 
session=t1E/2cvLoQAKRChu
Oct 11 13:24:51 halo dovecot: imap-login: Login: user=jdoe, 
method=PLAIN, rip=10.68.40.110, lip=10.59.1.53, mpid=14176, TLS, 
session=OZlM2cvLqgAKRChu
Oct 11 13:24:51 halo dovecot: imap-login: Login: user=jdoe, 
method=PLAIN, rip=10.68.40.110, lip=10.59.1.53, mpid=14178, TLS, 
session=d3NU2cvLsQAKRChu

Oct 11 13:24:51 halo dovecot: imap(jdoe): Connection closed in=16 out=350
Oct 11 13:24:52 halo dovecot: imap-login: Login: user=jdoe, 
method=PLAIN, rip=10.68.40.110, lip=10.59.1.53, mpid=14180, TLS, 
session=oxNa2cvLtAAKRChu
Oct 11 13:24:52 halo dovecot: imap(jdoe): Error: imapc: Mailbox 'Trash' 
state corrupted: Expunged message reappeared in session (uid=6282  
next_uid=6283)
Oct 11 13:24:52 halo dovecot: imap(jdoe): Fatal: master: service(imap): 
child 14176 killed with signal 11 (core dumped)
Oct 11 13:24:57 halo dovecot: imap(jdoe): Error: imapc: Mailbox 'Trash' 
state corrupted: Expunged message reappeared in session (uid=6282  
next_uid=6283)
Oct 11 13:24:57 halo dovecot: imap(jdoe): Fatal: master: service(imap): 
child 14178 killed with signal 11 (core dumped)
Oct 11 13:24:57 halo dovecot: imap-login: Login: user=jdoe, 
method=PLAIN, rip=10.68.40.110, lip=10.59.1.53, mpid=14182, TLS, 
session=FUCv2cvLuAAKRChu
Oct 11 13:24:58 halo dovecot: imap(jdoe): Error: imapc: Mailbox 'Trash' 
state corrupted: Expunged message reappeared in session (uid=6282  
next_uid=6283)
Oct 11 13:24:58 halo dovecot: imap(jdoe): Fatal: master: service(imap): 
child 14180 killed with signal 11 (core dumped)
Oct 11 13:25:03 halo dovecot: imap(jdoe): Error: imapc: Mailbox 'Trash' 
state corrupted: Expunged message reappeared in session (uid=6282  
next_uid=6283)
Oct 11 13:25:03 halo dovecot: imap(jdoe): Fatal: master: service(imap): 
child 14182 killed with signal 11 (core dumped)
Oct 11 13:25:03 halo dovecot: imap-login: Login: user=jdoe, 
method=PLAIN, rip=10.68.40.110, lip=10.59.1.53, mpid=14184, TLS, 
session=DQkF2svLuwAKRChu
Oct 11 13:25:03 halo dovecot: imap(jdoe): Error: imapc: Mailbox 'Trash' 
state corrupted: Expunged message reappeared in session (uid=6282  
next_uid=6283)
Oct 11 13:25:03 halo dovecot: imap(jdoe): Fatal: master: service(imap): 
child 14184 killed with signal 11 (core dumped)
Oct 11 13:25:09 halo dovecot: imap(jdoe): Error: imapc: Mailbox 'Trash' 
state corrupted: Expunged message reappeared in session (uid=6282  
next_uid=6283)
Oct 11 13:25:09 halo dovecot: imap(jdoe): Fatal: master: service(imap): 
child 14174 killed with signal 11 (core dumped)

[repeats]
Oct 11 13:25:27 halo dovecot: dns-client: Warning: Killed with signal 15 
(by pid=1 uid=0 code=kill)
Oct 11 13:25:27 halo dovecot: dns-client: Warning: Killed with signal 15 
(by pid=1 uid=0 code=kill)
Oct 11 13:25:27 halo dovecot: master: Warning: Killed with signal 15 (by 
pid=1 uid=0 code=kill)
Oct 11 13:25:27 halo dovecot: auth: Warning: Killed with signal 15 (by 
pid=1 uid=0 code=kill)
Oct 11 13:25:27 halo dovecot: config: Warning: Killed with signal 15 (by 
pid=1 uid=0 code=kill)
Oct 11 13:25:27 halo dovecot: ssl-params: Warning: Killed with signal 15 
(by pid=1 uid=0 code=kill)
Oct 11 13:25:27 halo dovecot: anvil: Warning: Killed with signal 15 (by 
pid=1 uid=0 code=kill)
Oct 11 13:25:27 halo dovecot: log: Warning: Killed with signal 15 (by 
pid=1 uid=0 code=kill)


gdb backtrace:

~ gdb /usr/lib/dovecot/imap /var/cache/imapproxy/jdoe/core
GNU gdb (GDB) 7.5-ubuntu
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
http://gnu.org/licenses/gpl.html

This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type show copying
and show warranty for details.
This GDB was configured as x86_64-linux-gnu.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/lib/dovecot/imap...Reading symbols from 
/usr/lib/debug/usr/lib/dovecot/imap...done.

done

Re: [Dovecot] Advice for new dovecot / imap proxy? setup

[Miguel Tormo]

El Miércoles, 21 de Marzo de 2012 15:43:14 Luca Lesinigo escribió:
 Hello list.
Hello, 
 
 I'm planning a new mail servers for our company's customers to replace the 
 oldish Courier-IMAP based one, we already started to deploy some mail 
 accounts on a dovecot-2.0 server as an early test.
 I'd like to implement the new system with dovecot-2 (I'll probably go 
 straight to dovecot-2.1.x) and I'd like to get it right from the beginning so 
 I'm here asking for some advice.
 
 The issue I'm investigating right now is how to manage a single IMAP / POP / 
 SMTP / webmail  entry point for multiple mail servers... in other words an 
 IMAP proxy.
 It would be desirable for multiple reasons:
I have recently deployed a very similar setup: imap proxy, mailbox sharding... 
Although not exactly like yours. Comments below:

 - graceful migration from the current system: we'd make the mailserver 
 hostname point to the proxy (along with its SSL certificates) and then the 
 proxy would route each domain to the correct IMAP non-ssl server on our LAN. 
 No need to update customer's systems configuration and we can move one domain 
 at a time from the old to the new server, behind the scenes
This is reasonable. For example, I did this to seamless migrate lots of users 
from one server to another, migrating just a few of them at a time.
 - be ready for similar migrations in the future (eg. right now we're still 
 keeping the imap servers with the qmail MTA, but we'd like to switch to 
 postfix+dovecot in the future)
You can do the exact same thing in the future, of course.
 - be ready for sharding mail domains on multiple IMAP servers (if/when 
 current hardware reach its capacity or needs to be swapped out for new gear)
This is fairly easy to accomplish with imap proxying.
 - be ready to serve traffic over IPv6 without touching our precious mailbox 
 servers
This is doable.
 - isolate the mailbox servers from direct external access and just run IMAP 
 on them, let other systems run ssl, pop3, smtp, webmail, etc...
I don't think I understand you here. You will need to run POP3 on the mailbox 
servers if you want to give POP3 access to the mailboxes.
 
 Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) 
 and Roundcube webmail (PHP, on https) and just speak IMAP to the underlying 
 mail servers on our internal LAN.
 We'd like to support all the recent IMAP goodies to make modern users happy 
 (IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new 
 backend mailbox server to improve our operations (currently we just run du in 
 a cronjob once a day on the current mailserver, IMAP clients including the 
 webmail do not know about quota and thus cannot show amount of free space).
I didn't implement a lemonade profile nor quotas in my setup. However, I can 
confirm you that IMAP IDLE does work with imap proxy.
 
 In addition to that, customer's will hit the SMTP server running on that 
 'proxy' system and this is good to keep its configuration separated from the 
 SMTP server of the actual mail servers (which has a different configuration 
 and is restricted to get connections only from our MX systems and not from 
 outside sources).
No problem with that, but this is related to the MTA configuration, not dovecot.
 
 I'd like to know if that plan sounds reasonable or if there's something 
 stupid in it.
 Also, is the proxy going to support all kind of IMAP stuff of the backend 
 server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE 
 clients thanks to linux inotify, etc...) or will it limit me somehow?
You have my comments above, I think it is doable. In my opinion, the IMAP proxy 
part is the easiest one. MTA configuration to distribute the mails among the 
different mailbox servers can be trickier. You could use dovecot LMTP proxy and 
make the MTA deliver mails through LMTP, thus the dovecot proxy instance will 
handle the sharding for delivering and for reading mail.

Re: [Dovecot] Advice for new dovecot / imap proxy? setup

[Timo Sirainen]

On 21.3.2012, at 16.43, Luca Lesinigo wrote:

 The issue I'm investigating right now is how to manage a single IMAP / POP / 
 SMTP / webmail  entry point for multiple mail servers... in other words an 
 IMAP proxy.

Are you thinking about actual dummy proxying (which is normally what Dovecot 
proxying is about) or about the imapc backend 
(http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using 
Dovecot as backend servers, there's really no reason to use imapc proxying.

 We'd like to support all the recent IMAP goodies to make modern users happy 
 (IMAP IDLE, LEMONADE, etc)

Dovecot doesn't support the full LEMONADE yet, but I don't know if there are 
any LEMONADE clients either.

Re: [Dovecot] Advice for new dovecot / imap proxy? setup

[Luca Lesinigo]

Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto:
 Are you thinking about actual dummy proxying (which is normally what 
 Dovecot proxying is about) or about the imapc backend 
 (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're 
 using Dovecot as backend servers, there's really no reason to use imapc 
 proxying.
I actually didn't know about the two different modes. I guess I would need 
imapc to support the older Courier-IMAP server until I migrated everything away 
from it, and that I could use dummy proxying for the newer dovecot backends.
I don't know if the two can be used at the same time (eg. imapc to the older 
backend and dummy to the newer) and/or if there is any drawback in running 
everything on imapc (old and new dovecot server). I'll be investigating this

 We'd like to support all the recent IMAP goodies to make modern users happy 
 (IMAP IDLE, LEMONADE, etc)
 Dovecot doesn't support the full LEMONADE yet, but I don't know if there are 
 any LEMONADE clients either.
Oh well I included it in the list because I read about it somewhere, possibly 
on the dovecot site. But what I really meant was simply support the latest 
goodies :)

Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto:
 - isolate the mailbox servers from direct external access and just run IMAP 
 on them, let other systems run ssl, pop3, smtp, webmail, etc...
 I don't think I understand you here. You will need to run POP3 on the mailbox 
 servers if you want to give POP3 access to the mailboxes.
Don't ask me why, but I was thinking that a dovecot proxy could talk just imap 
to the backends and use that to serve both POP3 and IMAP to clients. And it's 
possibly what happens with the imapc backend, but I need to do some RTFM about 
it.

 However, I can confirm you that IMAP IDLE does work with imap proxy.

That's great, I really want to provide the best possible push-like experience 
to modern clients, and as far as I know IMAP IDLE on the protocol side plus 
some notification mechanism (as opposed to regular polling) on the backend side 
is the way to go.

 You have my comments above, I think it is doable. In my opinion, the IMAP 
 proxy part is the easiest one. MTA configuration to distribute the mails 
 among the different mailbox servers can be trickier.
Actually that part is already there. Mail enters my systems via some MX servers 
(with the usual antispam and so on) and it's finally delivered via SMTP to the 
correct mail server via postfix recipient maps (that's because I already 
receive on my MXes mail for domains not hosted on my mail server, the common 
scenario is where I route a domain's mail to the customer's exchange server). 
But right now the mail server also receives direct SMTP connections from the 
clients in addition to incoming mail from my MXes and I'd really prefer to 
separate the two things.

 You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, 
 thus the dovecot proxy instance will handle the sharding for delivering and 
 for reading mail.
On the proxy system I plan to run postfix to implement authenticated SMTP (it 
would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to 
support that :| ), but all mail will be reinjected through our MX servers to be 
scanned before final delivery (either local or external).

Thanks people for the suggestions, my next stop is getting to know imapc and 
its details, and how the various other parts will fit with that (eg. giving 
pop3 service to clients).

--
Luca Lesinigo

Re: [Dovecot] Advice for new dovecot / imap proxy? setup

[Gedalya]

On 03/23/2012 02:12 PM, Luca Lesinigo wrote:

Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto:

Are you thinking about actual dummy proxying (which is normally what Dovecot proxying 
is about) or about the imapc backend 
(http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're using Dovecot as backend 
servers, there's really no reason to use imapc proxying.

I actually didn't know about the two different modes. I guess I would need imapc to 
support the older Courier-IMAP server until I migrated everything away from it, and that 
I could use dummy proxying for the newer dovecot backends.
I don't know if the two can be used at the same time (eg. imapc to the older 
backend and dummy to the newer) and/or if there is any drawback in running 
everything on imapc (old and new dovecot server). I'll be investigating this
I'm using the dummy proxying with a very different backend, certainly 
not dovecot, and it works great. For your needs (as I understand them) 
It's a much simpler and robust solution than imapc. Try it out. The main 
potential source of trouble is possible differences in the CAPABILITY 
string, but it hasn't caused me any actual problems.



We'd like to support all the recent IMAP goodies to make modern users happy 
(IMAP IDLE, LEMONADE, etc)

Dovecot doesn't support the full LEMONADE yet, but I don't know if there are 
any LEMONADE clients either.

Oh well I included it in the list because I read about it somewhere, possibly on the 
dovecot site. But what I really meant was simply support the latest goodies :)

Il giorno 23/mar/2012, alle ore 11:38, Miguel Tormo ha scritto:

- isolate the mailbox servers from direct external access and just run IMAP on 
them, let other systems run ssl, pop3, smtp, webmail, etc...

I don't think I understand you here. You will need to run POP3 on the mailbox 
servers if you want to give POP3 access to the mailboxes.

Don't ask me why, but I was thinking that a dovecot proxy could talk just imap 
to the backends and use that to serve both POP3 and IMAP to clients. And it's 
possibly what happens with the imapc backend, but I need to do some RTFM about 
it.


The same proxy_maybe (dummy proxy) setup works great for POP3 too. Very 
simple to set up, works like a charm. Nothing much to think about.





However, I can confirm you that IMAP IDLE does work with imap proxy.

That's great, I really want to provide the best possible push-like experience 
to modern clients, and as far as I know IMAP IDLE on the protocol side plus some 
notification mechanism (as opposed to regular polling) on the backend side is the way to 
go.


It will work as well as it was working with your existing courier 
server. But it will work great for accounts migrated to native dovecot.



You have my comments above, I think it is doable. In my opinion, the IMAP proxy 
part is the easiest one. MTA configuration to distribute the mails among the 
different mailbox servers can be trickier.

Actually that part is already there. Mail enters my systems via some MX servers 
(with the usual antispam and so on) and it's finally delivered via SMTP to the 
correct mail server via postfix recipient maps (that's because I already 
receive on my MXes mail for domains not hosted on my mail server, the common 
scenario is where I route a domain's mail to the customer's exchange server). 
But right now the mail server also receives direct SMTP connections from the 
clients in addition to incoming mail from my MXes and I'd really prefer to 
separate the two things.


It's a very good idea to have completely separate machines for outgoing 
mail. Once you have imap-only boxes, you can eliminate the need for an 
MTA by using the dovecot LMTP server. Your postfix transport map can 
send mail to either smtp:imap.yourdomain.com:25 or 
lmtp:imap.yourdomain.com:2525 on a per account basis, and you can get 
rid of the MTA in due time.




You could use dovecot LMTP proxy and make the MTA deliver mails through LMTP, 
thus the dovecot proxy instance will handle the sharding for delivering and for 
reading mail.

On the proxy system I plan to run postfix to implement authenticated SMTP (it 
would authenticate on dovecot) and pop/imap-before-smtp (yes we still need to 
support that :| ), but all mail will be reinjected through our MX servers to be 
scanned before final delivery (either local or external).


Since you're sending everything back to the MX, you might as well have 
your MX use LMTP, looking up the correct protocol and host from the 
database, and spend the next couple of years telling your customers to 
change their mail client configuration to use a dedicated outgoing mail 
server. It's worth the trouble.




Thanks people for the suggestions, my next stop is getting to know imapc and 
its details, and how the various other parts will fit with that (eg. giving 
pop3 service to clients).

--
Luca Lesinigo

Re: [Dovecot] Advice for new dovecot / imap proxy? setup

[Timo Sirainen]

On 23.3.2012, at 20.24, Gedalya wrote:

 On 03/23/2012 02:12 PM, Luca Lesinigo wrote:
 Il giorno 23/mar/2012, alle ore 11:50, Timo Sirainen ha scritto:
 Are you thinking about actual dummy proxying (which is normally what 
 Dovecot proxying is about) or about the imapc backend 
 (http://www.dovecot.fi/products/105-dovecot-imap-adaptor.html)? If you're 
 using Dovecot as backend servers, there's really no reason to use imapc 
 proxying.
 I actually didn't know about the two different modes. I guess I would need 
 imapc to support the older Courier-IMAP server until I migrated everything 
 away from it, and that I could use dummy proxying for the newer dovecot 
 backends.
 I don't know if the two can be used at the same time (eg. imapc to the older 
 backend and dummy to the newer) and/or if there is any drawback in running 
 everything on imapc (old and new dovecot server). I'll be investigating 
 this
 I'm using the dummy proxying with a very different backend, certainly not 
 dovecot, and it works great. For your needs (as I understand them) It's a 
 much simpler and robust solution than imapc. Try it out. The main potential 
 source of trouble is possible differences in the CAPABILITY string, but it 
 hasn't caused me any actual problems.

Right, a lot of people have done migration from Courier - Dovecot using the 
dummy proxying. Since v2.0 the proxying automatically handles any CAPABILITY 
string issues.

[Dovecot] Advice for new dovecot / imap proxy? setup

[Luca Lesinigo]

Hello list.

I'm planning a new mail servers for our company's customers to replace the 
oldish Courier-IMAP based one, we already started to deploy some mail accounts 
on a dovecot-2.0 server as an early test.
I'd like to implement the new system with dovecot-2 (I'll probably go straight 
to dovecot-2.1.x) and I'd like to get it right from the beginning so I'm here 
asking for some advice.

The issue I'm investigating right now is how to manage a single IMAP / POP / 
SMTP / webmail  entry point for multiple mail servers... in other words an 
IMAP proxy.
It would be desirable for multiple reasons:
- graceful migration from the current system: we'd make the mailserver hostname 
point to the proxy (along with its SSL certificates) and then the proxy would 
route each domain to the correct IMAP non-ssl server on our LAN. No need to 
update customer's systems configuration and we can move one domain at a time 
from the old to the new server, behind the scenes
- be ready for similar migrations in the future (eg. right now we're still 
keeping the imap servers with the qmail MTA, but we'd like to switch to 
postfix+dovecot in the future)
- be ready for sharding mail domains on multiple IMAP servers (if/when current 
hardware reach its capacity or needs to be swapped out for new gear)
- be ready to serve traffic over IPv6 without touching our precious mailbox 
servers
- isolate the mailbox servers from direct external access and just run IMAP on 
them, let other systems run ssl, pop3, smtp, webmail, etc...

Ideally the 'proxy' system would run dovecot imap and pop3 (SSL protected) and 
Roundcube webmail (PHP, on https) and just speak IMAP to the underlying mail 
servers on our internal LAN.
We'd like to support all the recent IMAP goodies to make modern users happy 
(IMAP IDLE, LEMONADE, etc) and possibly implement Maildir quota on the new 
backend mailbox server to improve our operations (currently we just run du in a 
cronjob once a day on the current mailserver, IMAP clients including the 
webmail do not know about quota and thus cannot show amount of free space).

In addition to that, customer's will hit the SMTP server running on that 
'proxy' system and this is good to keep its configuration separated from the 
SMTP server of the actual mail servers (which has a different configuration and 
is restricted to get connections only from our MX systems and not from outside 
sources).

I'd like to know if that plan sounds reasonable or if there's something stupid 
in it.
Also, is the proxy going to support all kind of IMAP stuff of the backend 
server (IDLE, CONDSTORE, Maildir quota, immediate notification of IDLE clients 
thanks to linux inotify, etc...) or will it limit me somehow?

thanks,
--
Luca Lesinigo

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Ed W]

On 14/03/2012 10:58, Charles Marcus wrote:

On 2012-03-13 6:29 PM, Terry Carmen te...@cnysupport.com wrote:
I'm going to hope everything is OK for a while, since my goal is to 
retire

all the old Exchange servers and move all the users to dovecot/maildir
within the next couple of months.

However it's always nice to know there are options. 8-)


I'm currently looking at rolling out SOGo as part of a major reworking 
of their current infrastructure (will also include converting their 
old Courier-IMAP to dovecot 2.1.x among other things)...


SOGo, as far as I can tell, is the best truly free and open source 
'exchange clone' available that works extremely well with 
Thunderbird+Lightning (which is what my Client uses currently, but 
they are very dissatisfied with using Google Calendar for Shared 
calendars), Outlook and Apple Apps, as well as Android, Blackberry and 
Apple mobile devices - and their upcoming v2 (in beta now) will not 
only provide native Outlook support (no plugin needed), it will also 
(optionally) provide a Samba4 Active Directory server in my main 
Client's office - all with absolutely no licenses required. Commercial 
support is available from Inverse, the company created by the 
developers to provide said support services.


I also learned something very interesting yesterday concerning SOGo 
and dovecot during a sales call with a SOGo rep, but I'll wait and see 
if Timo cares to chime in on this one... ;)




If the answer is that he will write a Z-Push/Activesync module for SOGo 
then I'm all ears!  I have been watching SOGo for some time and the main 
thing I would miss is that every phone I have ever owned has largely 
limited/broken Funambol based sync and annoyingly working Activesync 
capability (I own a stream of Nokias...).  It seems that although I 
don't like it, I need activesync support if I want my contacts/calendar 
on my phone... (I think I can do caldav on some of them, but not cardav 
on my N9)


Apart from that it's a very neat system!

Ed W

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Timo Sirainen]

On Fri, 2012-03-16 at 15:22 +, Ed W wrote:
  I also learned something very interesting yesterday concerning SOGo 
  and dovecot during a sales call with a SOGo rep, but I'll wait and see 
  if Timo cares to chime in on this one... ;)
 
 
 If the answer is that he will write a Z-Push/Activesync module for SOGo 
 then I'm all ears!  I have been watching SOGo for some time and the main 
 thing I would miss is that every phone I have ever owned has largely 
 limited/broken Funambol based sync and annoyingly working Activesync 
 capability (I own a stream of Nokias...).  It seems that although I 
 don't like it, I need activesync support if I want my contacts/calendar 
 on my phone... (I think I can do caldav on some of them, but not cardav 
 on my N9)

We're also very much wishing for SOGo Activesync, but I'm not planning
on writing it myself (but maybe we'll hire someone who will). Annoyingly
Microsoft has patented Activesync, so I guess it can't be legally used
at least in USA without paying MS.

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Charles Marcus]

On 2012-03-16 11:22 AM, Ed W li...@wildgooses.com wrote:

If the answer is that he will write a Z-Push/Activesync module for SOGo
then I'm all ears!  I have been watching SOGo for some time and the main
thing I would miss is that every phone I have ever owned has largely
limited/broken Funambol based sync and annoyingly working Activesync
capability (I own a stream of Nokias...).  It seems that although I
don't like it, I need activesync support if I want my contacts/calendar
on my phone... (I think I can do caldav on some of them, but not cardav
on my N9)


While I agree it would be nice, why not just switch to a supported phone 
and be done with it? ;)


When we roll out SOGo, we'll only be supporting the officially supported 
mobile clients (android, iphone/ipad, blackberry and windows mobile)...


--

Best regards,

Charles

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Ed W]

On 16/03/2012 15:45, Charles Marcus wrote:

On 2012-03-16 11:22 AM, Ed W li...@wildgooses.com wrote:

If the answer is that he will write a Z-Push/Activesync module for SOGo
then I'm all ears!  I have been watching SOGo for some time and the main
thing I would miss is that every phone I have ever owned has largely
limited/broken Funambol based sync and annoyingly working Activesync
capability (I own a stream of Nokias...).  It seems that although I
don't like it, I need activesync support if I want my contacts/calendar
on my phone... (I think I can do caldav on some of them, but not cardav
on my N9)


While I agree it would be nice, why not just switch to a supported 
phone and be done with it? ;)


When we roll out SOGo, we'll only be supporting the officially 
supported mobile clients (android, iphone/ipad, blackberry and windows 
mobile)...




That implies you will be using cardav/caldav on those phones?  I thought 
Android support was quite weak for those?


I definitely don't like the idea of supporting activesync, but it seems 
like the only widely supported solution to pushing calendar and contacts 
updates to clients?  Caldav gets you part of the way there, but cardav 
seems badly supported and there is no push support with either...


Out of curiousity, what kind of performance are you getting out of the 
web interface and any tricks you used to improve perceived 
performance? My quick testing gave something circa 150-200ms response 
times from SOGo (forget exactly now) and as a result it was perceivable 
and just very slightly laggy (versus a desktop mail program!!).  I get 
slightly better perceived performance from Roundcube (which also seems 
more amenable to building extension plugins)


Seems a bit of a surprise that a compiled language delivers results 
slightly less quickly than PHP... Did you find any magic knobs to twist 
to get performance up there with gmail?


Cheers

Ed W

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Charles Marcus]

On 2012-03-13 6:29 PM, Terry Carmen te...@cnysupport.com wrote:

I'm going to hope everything is OK for a while, since my goal is to retire
all the old Exchange servers and move all the users to dovecot/maildir
within the next couple of months.

However it's always nice to know there are options. 8-)


I'm currently looking at rolling out SOGo as part of a major reworking 
of their current infrastructure (will also include converting their old 
Courier-IMAP to dovecot 2.1.x among other things)...


SOGo, as far as I can tell, is the best truly free and open source 
'exchange clone' available that works extremely well with 
Thunderbird+Lightning (which is what my Client uses currently, but they 
are very dissatisfied with using Google Calendar for Shared calendars), 
Outlook and Apple Apps, as well as Android, Blackberry and Apple mobile 
devices - and their upcoming v2 (in beta now) will not only provide 
native Outlook support (no plugin needed), it will also (optionally) 
provide a Samba4 Active Directory server in my main Client's office - 
all with absolutely no licenses required. Commercial support is 
available from Inverse, the company created by the developers to provide 
said support services.


I also learned something very interesting yesterday concerning SOGo and 
dovecot during a sales call with a SOGo rep, but I'll wait and see if 
Timo cares to chime in on this one... ;)


--

Best regards,

Charles

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Terry Carmen]

On 03/14/2012 06:58 AM, Charles Marcus wrote:

On 2012-03-13 6:29 PM, Terry Carmen te...@cnysupport.com wrote:
I'm going to hope everything is OK for a while, since my goal is to 
retire

all the old Exchange servers and move all the users to dovecot/maildir
within the next couple of months.

However it's always nice to know there are options. 8-)


I'm currently looking at rolling out SOGo as part of a major reworking 
of their current infrastructure (will also include converting their 
old Courier-IMAP to dovecot 2.1.x among other things)...


SOGo, as far as I can tell, is the best truly free and open source 
'exchange clone' available that works extremely well with 
Thunderbird+Lightning (which is what my Client uses currently, but 
they are very dissatisfied with using Google Calendar for Shared 
calendars), Outlook and Apple Apps, as well as Android, Blackberry and 
Apple mobile devices - and their upcoming v2 (in beta now) will not 
only provide native Outlook support (no plugin needed), it will also 
(optionally) provide a Samba4 Active Directory server in my main 
Client's office - all with absolutely no licenses required. Commercial 
support is available from Inverse, the company created by the 
developers to provide said support services.


Looks interesting.

I have currently have horde/imp/kronolith running with 
postfix/dovecot/mysql on the back end and it's been working nicely with 
all the clients and devices except for outlook.


I'll have to take a look at sogo, because I'd really like to keep 
outlook for the users that want it, to cut down on support and complaints.


Thanks!

Terry

Re: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Timo Sirainen]

On 12.3.2012, at 20.57, Terry Carmen wrote:

 If you can use userdb passwd-file and export the data to that file, it'll 
 work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile
 
 Example line:
 
 user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com
 
 Note that you can't then return any userdb fields from passdb ldap lookup.
 
 That doesn't seem to work because I can't create the passdb file containing 
 the user's password, since they're only known to the remote IMAP server that 
 I want imapproxy to connect to.

Well, you could allow users to log in with any password and then let it just 
fail later at imapc login, but that's a bit ugly.

You could also use passdb imap {} + userdb passwd-file {} with some extra work. 
The authentication would be done against the remote imap server, while the 
userdb_imapc_host would be looked up from the passwd-file.

 What would be perfect is if I could do something like this:
 
 
 
 http://wiki.dovecot.org/HowTo/ImapProxy#IMAP_and_POP3_session_proxying
 Proxy only server
..
 All I really need is a way to lookup the user's home IMAP server when given 
 the username, as above.
 
 Does imapproxy still support this 1.x feature?

This describes a regular dummy proxying setup. Sure you could still do that, 
but it's not imapc proxying. 
http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Terry Carmen]

On 03/13/2012 04:06 AM, Timo Sirainen wrote:
This describes a regular dummy proxying setup. Sure you could still do 
that, but it's not imapc proxying. 
http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy


The above URL worked beautifully and Dovecot is now running as a proxy 
for a dozen older Exchange servers on a private network.


Thanks for the help!

Terry

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Timo Sirainen]

On 13.3.2012, at 23.44, Terry Carmen wrote:

 On 03/13/2012 04:06 AM, Timo Sirainen wrote:
 This describes a regular dummy proxying setup. Sure you could still do that, 
 but it's not imapc proxying. 
 http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy
 
 The above URL worked beautifully and Dovecot is now running as a proxy for a 
 dozen older Exchange servers on a private network.

If you find out that IMAP clients still don't work nicely with Exchange 
(apparently they have random problems, especially with shared 
mailboxes/accounts), you can still put imapc proxy in front of your currently 
working Dovecot proxy. :)

Re: [Dovecot] [Solved] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Terry Carmen]

- Message from Timo Sirainen t...@iki.fi -
       Date: Wed, 14 Mar 2012 00:05:14 +0200
       From: Timo Sirainen t...@iki.fi
   Reply-To: Dovecot Mailing List dovecot@dovecot.org
   Subject: Re: [Dovecot] [Solved] Another hint from the clue box 8-)
imapc/imap proxy user mailbox server location
         To: Terry Carmen te...@cnysupport.com
         Cc: dovecot@dovecot.org

On 13.3.2012, at 23.44, Terry Carmen wrote:
On 03/13/2012 04:06 AM, Timo Sirainen wrote:This describes a  
regular dummy proxying setup. Sure you could still do that, but  
it's not imapc proxying.  
http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy
   The above URL worked beautifully and Dovecot is now running as a  
proxy for a dozen older Exchange servers on a private network.
  If you find out that IMAP clients still don't work nicely with  
Exchange (apparently they have random problems, especially with  
shared mailboxes/accounts), you can still put imapc proxy in front  
of your currently working Dovecot proxy. :)

I'm going to hope everything is OK for a while, since my goal is to retire
all the old Exchange servers and move all the users to dovecot/maildir
within the next couple of months.

However it's always nice to know there are options. 8-)

Terry

Re: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Terry Carmen]

On 03/04/2012 09:58 AM, Timo Sirainen wrote:

On 4.3.2012, at 16.48, Terry Carmen wrote:


 pass_attrs = ..., \
 msExchHomeServerName=userdb_imapc_host=%49.100$.example.com

 If the prefix differs, but all of the exchange server names have  
the same length, for example 10, you can also do:


 pass_attrs = ..., \
 msExchHomeServerName=userdb_imapc_host=%-10$.example.com
 There's no otherwise nice way to parse this string.



If by prefix, you mean the  
/O=example/OU=INT/cn=Configuration/cn=Servers/ part, then, yes,  
they're different.


OK, so if the prefix or suffix isn't always the same length you  
can't do the above.


I could export the data to a text file as  
username:homeexchangeserver (or whatever other format is needed).


homeservers.txt:
user1:exch1.example.com
user2:exch1.example.com
user3:exch1.example.com
user4:exch2.example.com

Is it possible to do a lookup in a text file to get this?



If you can use userdb passwd-file and export the data to that file,  
it'll work. http://wiki2.dovecot.org/AuthDatabase/PasswdFile


Example line:

user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com

Note that you can't then return any userdb fields from passdb ldap lookup.


That doesn't seem to work because I can't create the passdb file  
containing the user's password, since they're only known to the remote  
IMAP server that I want imapproxy to connect to.


What would be perfect is if I could do something like this:



http://wiki.dovecot.org/HowTo/ImapProxy#IMAP_and_POP3_session_proxying
Proxy only server

. . .

In this document I assume that Dovecot is installed under  
/opt/dovecot, by default it is installed under /usr/local when  
compiling from source. Examples in this document are for MySQL but  
configs do not differ much with PostgreSQL.


SQL table structure

Create SQL table like

CREATE TABLE proxy (
  user varchar(255) NOT NULL,
  host varchar(16) default NULL,
  destuser varchar(255) default NULL,
  PRIMARY KEY  (user)
);

//

All I really need is a way to lookup the user's home IMAP server when  
given the username, as above.


Does imapproxy still support this 1.x feature?

Thanks!

Terry

Re: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Timo Sirainen]

On 2.3.2012, at 0.35, Terry Carmen wrote:

 With the exchange server being returned in the msExchHomeServerName property 
 as:
 
 /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername
 
 I believe this should somehow end up in the userdb section, which currently 
 contains driver = prefetch, but can't seem to figure out specifically what 
 should be there.
..
 The only important part is cn=exchangeservername, which is the machine name 
 and would need to be prepended to example.com to get the fqdn.


Do all of the values have the same prefix? Then I guess you can do:

pass_attrs = ..., \
  msExchHomeServerName=userdb_imapc_host=%49.100$.example.com

If the prefix differs, but all of the exchange server names have the same 
length, for example 10, you can also do:

pass_attrs = ..., \
  msExchHomeServerName=userdb_imapc_host=%-10$.example.com

There's no otherwise nice way to parse this string.

Re: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Terry Carmen]

- Message from Timo Sirainen t...@iki.fi -
   nbsp; nbsp;Date: Sun, 4 Mar 2012 14:45:48 +0200
   nbsp; nbsp;From: Timo Sirainen t...@iki.fi
   Subject: Re: [Dovecot] Another hint from the clue box 8-) imapc/imap
proxy user mailbox server location
   nbsp; nbsp; nbsp;To: Terry Carmen te...@cnysupport.com
   nbsp; nbsp; nbsp;Cc: dovecot@dovecot.org

On 2.3.2012, at 0.35, Terry Carmen wrote:
With the exchange server being returned in the msExchHomeServerName  
property as:


   /O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername

   I believe this should somehow end up in the userdb section,  
which currently contains driver = prefetch, but can't seem to  
figure out specifically what should be there.
  ..   The only important part is cn=exchangeservername, which is  
the machine name and would need to be prepended to example.com to  
get the fqdn.

  Do all of the values have the same prefix? Then I guess you can do:

  pass_attrs = ..., \
  msExchHomeServerName=userdb_imapc_host=%49.100$.example.com

  If the prefix differs, but all of the exchange server names have  
the same length, for example 10, you can also do:


  pass_attrs = ..., \
  msExchHomeServerName=userdb_imapc_host=%-10$.example.com
  There's no otherwise nice way to parse this string.



If by prefix, you mean the  
/O=example/OU=INT/cn=Configuration/cn=Servers/ part, then, yes,  
they're different.


I could export the data to a text file as username:homeexchangeserver  
(or whatever other format is needed).


homeservers.txt:
user1:exch1.example.com
user2:exch1.example.com
user3:exch1.example.com
user4:exch2.example.com

Is it possible to do a lookup in a text file to get this?

Terry

Re: [Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Timo Sirainen]

On 4.3.2012, at 16.48, Terry Carmen wrote:

  pass_attrs = ..., \
  msExchHomeServerName=userdb_imapc_host=%49.100$.example.com
 
  If the prefix differs, but all of the exchange server names have the same 
 length, for example 10, you can also do:
 
  pass_attrs = ..., \
  msExchHomeServerName=userdb_imapc_host=%-10$.example.com
  There's no otherwise nice way to parse this string.
 
 
 If by prefix, you mean the /O=example/OU=INT/cn=Configuration/cn=Servers/ 
 part, then, yes, they're different.

OK, so if the prefix or suffix isn't always the same length you can't do the 
above.

 I could export the data to a text file as username:homeexchangeserver (or 
 whatever other format is needed).
 
 homeservers.txt:
 user1:exch1.example.com
 user2:exch1.example.com
 user3:exch1.example.com
 user4:exch2.example.com
 
 Is it possible to do a lookup in a text file to get this?


If you can use userdb passwd-file and export the data to that file, it'll work. 
http://wiki2.dovecot.org/AuthDatabase/PasswdFile

Example line:

user1::1000:1000::/home/user::userdb_imapc_host=exch1.example.com

Note that you can't then return any userdb fields from passdb ldap lookup.

[Dovecot] Another hint from the clue box 8-) imapc/imap proxy user mailbox server location

[Terry Carmen]

I'm running imapproxy as shown at  
http://wiki2.dovecot.org/HowTo/ImapcProxy. In fact, that's my config  
in the wiki. 8-)


It's been working, but has performance issues when the Exchange server  
that's hard-coded as imapc_host=xxx.xxx.xxx.xxx doesn't happen to be  
the user's home exchange server.


I'd like to point dovecot at the correct Exchange server based on an  
LDAP query, and in fact, have an LDAP search that works:


DC=example,DC=com

((objectCategory=person)(objectClass=user)(!(userAccountcontrol:1.2.840.113556.1.4.803:=2))(sAMAccountName=username))

With the exchange server being returned in the msExchHomeServerName  
property as:


/O=example/OU=INT/cn=Configuration/cn=Servers/cn=exchangeservername

I believe this should somehow end up in the userdb section, which  
currently contains driver = prefetch, but can't seem to figure out  
specifically what should be there.


The only important part is cn=exchangeservername, which is the  
machine name and would need to be prepended to example.com to get the  
fqdn.


Can anybody toss me a clue?

Once I get it working, I'll update the wiki.

Thanks!

Terry

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On Thu, 2012-02-23 at 01:41 +0200, Timo Sirainen wrote:

 What do you need the statistics for? I could make imap_client and
 pop3_client support some virtual methods, like user.destroy() initially,
 which would be enough for your use. I guess I could add that for v2.2.

http://dovecot.org/patches/2.2/imap-logout-plugin.c

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Ed W]

On 27/02/2012 08:34, Timo Sirainen wrote:

On Thu, 2012-02-23 at 01:41 +0200, Timo Sirainen wrote:


What do you need the statistics for? I could make imap_client and
pop3_client support some virtual methods, like user.destroy() initially,
which would be enough for your use. I guess I could add that for v2.2.

http://dovecot.org/patches/2.2/imap-logout-plugin.c




Thanks - can I assume that a pop-logout would be basically the same?

Also, how might I access the bytes in/out statistics from that context?

Thanks

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On Mon, 2012-02-27 at 08:46 +, Ed W wrote:
 On 27/02/2012 08:34, Timo Sirainen wrote:
  On Thu, 2012-02-23 at 01:41 +0200, Timo Sirainen wrote:
 
  What do you need the statistics for? I could make imap_client and
  pop3_client support some virtual methods, like user.destroy() initially,
  which would be enough for your use. I guess I could add that for v2.2.
  http://dovecot.org/patches/2.2/imap-logout-plugin.c
 
 
 
 Thanks - can I assume that a pop-logout would be basically the same?

Yes, basically s/imap/pop3.

 Also, how might I access the bytes in/out statistics from that context?

input: i_stream_get_absolute_offset(client-input)
output: client-output-offset

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Ed W]

On 22/02/2012 23:56, Ed W wrote:
I think it has potential though.  I think a lot of the current plugins 
on the website could easily be rewritten, likely without performance 
concerns, using a scripting based plugin system.  I could see that 
some other big picture pieces could potentially benefit also


One interesting test case for such a scripting hooks solution might be 
login restrictions.  There seem to be regular requests for the ability 
to setup arbitrarily complicated restrictions on users per IP, attempts 
per second, etc (and my logging interest is kind of related also).


Not trying to bump the item up any todo lists, just trying to chuck in 
some concrete ideas for actually testing a specific implementation...


I guess a substantially more performance orientated area that seems to 
get some interest would be various spam, expunge, delete ideas and the 
hooks needed for those.  These seem much more tricky to implement a 
scripting hook and still stay performant.  Again just ideas for real 
things people might want to do?


Cheers

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Jan-Frode Myklebust]

On Tue, Feb 21, 2012 at 02:33:24PM +, Ed W wrote:
 
 I think the original question was still sensible.  In your case it
 seems like the ping times are identical between:
 webmail - imap-proxy
 webmail - imap server
 
 I think your results show that a proxy has little (or negative)
 benefit in this situation, but it seems feasible that a proxy could
 eliminate several RTT trips in the event that the proxy is closer
 than the imap server?  This might happen if say the imap server is
 in a different datacenter (webmail on an office server machine?)

The webmail/imapproxy were actually running in a different datacenter to
the dovecot director/backend servers, but only about 20KM away.

Ping tests:

webmail-director:

rtt min/avg/max/mdev = 0.933/1.061/2.034/0.183 ms

director-backend:

rtt min/avg/max/mdev = 0.104/0.108/0.127/0.005 ms

webmail-localhost:

rtt min/avg/max/mdev = 0.020/0.062/1.866/0.257 ms


  -jf

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Ed W]

On 22/02/2012 08:25, Jan-Frode Myklebust wrote:

On Tue, Feb 21, 2012 at 02:33:24PM +, Ed W wrote:

I think the original question was still sensible.  In your case it
seems like the ping times are identical between:
 webmail -  imap-proxy
 webmail -  imap server

I think your results show that a proxy has little (or negative)
benefit in this situation, but it seems feasible that a proxy could
eliminate several RTT trips in the event that the proxy is closer
than the imap server?  This might happen if say the imap server is
in a different datacenter (webmail on an office server machine?)

The webmail/imapproxy were actually running in a different datacenter to
the dovecot director/backend servers, but only about 20KM away.

Ping tests:

webmail-director:

rtt min/avg/max/mdev = 0.933/1.061/2.034/0.183 ms

director-backend:

rtt min/avg/max/mdev = 0.104/0.108/0.127/0.005 ms

webmail-localhost:

rtt min/avg/max/mdev = 0.020/0.062/1.866/0.257 ms


   -jf


Hmm, not sure I understand the original numbers then?

It seems intuitive that the proxy installed locally could save you 2x 
RTT increment, which is about 0.8ms in your case.  So I might expect the 
proxy to reduce rendering times by around 1.6ms simply because it 
reduces the number of round trips to login?  Kind of curious why that's 
not achieved..?


Cheers

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Ed W]

On 21/02/2012 20:36, Timo Sirainen wrote:

On 21.2.2012, at 16.33, Ed W wrote:


I'm also pleased to see that there is little negative cost in using a proxy... I recently added 
imap-proxy to our webmail setup because I wanted to log last login + logout times.  I 
haven't quite figured out how to best log logout time (Timo, any chance of a post 
logout script? Or perhaps it's possible with the current login scripting?).

You could of course grep the logs, but other than that you'd need to write a 
Dovecot plugin. Luckily it's really simple to write a plugin. Basically:

void postlogout_init(struct module *module) { }
void postlogout_deinit(void) {
   system(/usr/local/bin/dovecot-postlogout.sh);
}

Add a few missing #includes and compile and enable for imap/pop3 and that 
should be it.



Thanks - that's really obvious and quite interesting.  I guess a simple 
log plugin makes sense.


Quick followup question - the logout log file currently logs a bunch of 
statistics such as mails read/deleted, bytes sent/received.  How might I 
access these from the _deinit context as above?  Apologies if this is a 
RTFM question?


Finally, do you see it feasible to offer a scriptable plugin 
interface, eg perhaps using some high performance scripting language 
such as lua?  Such a plugin might itself be simply a standard plugin..?  
The motivation being to offer the ability to create plugins to those who 
are nervous of using a compiler, and of course to reduce the ability of 
a badly written plugin to kill dovecot?


Cheers

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Jan-Frode Myklebust]

On Wed, Feb 22, 2012 at 09:31:55AM +, Ed W wrote:
 
 It seems intuitive that the proxy installed locally could save you
 2x RTT increment, which is about 0.8ms in your case.  So I might
 expect the proxy to reduce rendering times by around 1.6ms simply
 because it reduces the number of round trips to login?  Kind of
 curious why that's not achieved..?

Each http-request can probably trigger several IMAP requests. Maybe 
these work better in parallel directly to dovecot, than serialized (?) 
trough the imapproxy ? No idea if that's what's happening... or maybe
the imapproxy just adds more overhead than the 2xRTT + imap logins it's
supposed to save us ?


  -jf

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On 22.2.2012, at 11.38, Ed W wrote:

 void postlogout_init(struct module *module) { }
 void postlogout_deinit(void) {
   system(/usr/local/bin/dovecot-postlogout.sh);
 }
 
 Add a few missing #includes and compile and enable for imap/pop3 and that 
 should be it.
 
 Thanks - that's really obvious and quite interesting.  I guess a simple log 
 plugin makes sense.
 
 Quick followup question - the logout log file currently logs a bunch of 
 statistics such as mails read/deleted, bytes sent/received.  How might I 
 access these from the _deinit context as above?  Apologies if this is a RTFM 
 question?

You'd have to build separate plugins for POP3 and IMAP, and even then it 
becomes tricky since there's no simple hook for catching when client gets 
destroyed.

 Finally, do you see it feasible to offer a scriptable plugin interface, eg 
 perhaps using some high performance scripting language such as lua?  Such a 
 plugin might itself be simply a standard plugin..?  The motivation being to 
 offer the ability to create plugins to those who are nervous of using a 
 compiler, and of course to reduce the ability of a badly written plugin to 
 kill dovecot?

I've been thinking about adding a scripting language plugin to Dovecot. Perhaps 
even using one of the existing generators that are supposed to make this easy 
for multiple languages, such as SWIG. But this is pretty low priority 
currently..

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Ed W]

On 22/02/2012 19:49, Timo Sirainen wrote:

On 22.2.2012, at 11.38, Ed W wrote:


void postlogout_init(struct module *module) { }
void postlogout_deinit(void) {
   system(/usr/local/bin/dovecot-postlogout.sh);
}

Add a few missing #includes and compile and enable for imap/pop3 and that 
should be it.

Thanks - that's really obvious and quite interesting.  I guess a simple log 
plugin makes sense.

Quick followup question - the logout log file currently logs a bunch of 
statistics such as mails read/deleted, bytes sent/received.  How might I access 
these from the _deinit context as above?  Apologies if this is a RTFM question?

You'd have to build separate plugins for POP3 and IMAP, and even then it 
becomes tricky since there's no simple hook for catching when client gets 
destroyed.


Do you think you could keep something similar on your low priority 
backlog?  Clearly parsing log files or hacking the code is possible, but 
I think the interest in the login scripting shows there is general 
interest, and having a full log of logon/logoff/bytes is clearly 
interesting to more minority users?



Finally, do you see it feasible to offer a scriptable plugin interface, eg 
perhaps using some high performance scripting language such as lua?  Such a plugin might 
itself be simply a standard plugin..?  The motivation being to offer the ability to 
create plugins to those who are nervous of using a compiler, and of course to reduce the 
ability of a badly written plugin to kill dovecot?

I've been thinking about adding a scripting language plugin to Dovecot. Perhaps 
even using one of the existing generators that are supposed to make this easy 
for multiple languages, such as SWIG. But this is pretty low priority 
currently..


I think SWIG is for wrapping dovecot's api into the scripting language?  
(ie you could call dovecot methods from say perl/python/etc). What I had 
in mind was the reverse, ie embed LUA inside dovecot.  Whenever dovecot 
normally calls a plugin method it will also run any [lua] scripts.


I'm sure you know how to use google, but just so we are on the same 
page, top hit (below) from google shows how straight forward this is 
(lua has been built to be extremely fast and easy to embed, ie it's not 
an arbitrary choice)

http://heavycoder.com/tutorials/lua_embed.php

Cheers

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On Wed, 2012-02-22 at 22:54 +, Ed W wrote:
  Quick followup question - the logout log file currently logs a bunch of 
  statistics such as mails read/deleted, bytes sent/received.  How might I 
  access these from the _deinit context as above?  Apologies if this is a 
  RTFM question?
  You'd have to build separate plugins for POP3 and IMAP, and even then it 
  becomes tricky since there's no simple hook for catching when client gets 
  destroyed.
 
 Do you think you could keep something similar on your low priority 
 backlog?  Clearly parsing log files or hacking the code is possible, but 
 I think the interest in the login scripting shows there is general 
 interest, and having a full log of logon/logoff/bytes is clearly 
 interesting to more minority users?

What do you need the statistics for? I could make imap_client and
pop3_client support some virtual methods, like user.destroy() initially,
which would be enough for your use. I guess I could add that for v2.2.

  Finally, do you see it feasible to offer a scriptable plugin interface, 
  eg perhaps using some high performance scripting language such as lua?  
  Such a plugin might itself be simply a standard plugin..?  The motivation 
  being to offer the ability to create plugins to those who are nervous of 
  using a compiler, and of course to reduce the ability of a badly written 
  plugin to kill dovecot?
  I've been thinking about adding a scripting language plugin to Dovecot. 
  Perhaps even using one of the existing generators that are supposed to make 
  this easy for multiple languages, such as SWIG. But this is pretty low 
  priority currently..
 
 I think SWIG is for wrapping dovecot's api into the scripting language?  
 (ie you could call dovecot methods from say perl/python/etc). What I had 
 in mind was the reverse, ie embed LUA inside dovecot.  Whenever dovecot 
 normally calls a plugin method it will also run any [lua] scripts.

Yes, but I think SWIG can do that too. You'll need Dovecot's API
implemented for scripting language if you want to do anything useful in
the embedded script. That's the biggest job actually. Actually
embeddeding some scripting language to Dovecot processes would be
simple, they just couldn't really do anything useful.

Also the nice thing about generically implementing Dovecot's APIs means
that they could be used to build independent Dovecot applications, not
just plugins.

 I'm sure you know how to use google, but just so we are on the same 
 page, top hit (below) from google shows how straight forward this is 
 (lua has been built to be extremely fast and easy to embed, ie it's not 
 an arbitrary choice)
  http://heavycoder.com/tutorials/lua_embed.php

I've heard LUA being a commonly used embedded language, but I'd prefer
to instead support several very widely used languages, such as
Perl/Python.

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Ed W]

On 22/02/2012 23:41, Timo Sirainen wrote:

I've heard LUA being a commonly used embedded language, but I'd prefer
to instead support several very widely used languages, such as
Perl/Python.


I'm a perl/ruby fan myself, but I would still recommend a good look at 
lua (or python) simply because they seem to be performant, easy to use, 
and on the surface seem to have had some thought about making them 
embeddable.


My new favourite editor Sublime Text 2 has python as it's scripting 
language.  Lua has been used for some big name games amongst other things.


Perl has some memory management issues if you leave it long running, 
also writing XS code looks ok on the surface, but is an exercise in hair 
pulling in practice  Ruby is a beautiful language, but unsure of how 
easy to embed and speed + memory management is an unknown (for high 
performance applications)


I think it has potential though.  I think a lot of the current plugins 
on the website could easily be rewritten, likely without performance 
concerns, using a scripting based plugin system.  I could see that some 
other big picture pieces could potentially benefit also


Thanks for considering it

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On Wed, 2012-02-22 at 23:56 +, Ed W wrote:
 On 22/02/2012 23:41, Timo Sirainen wrote:
  I've heard LUA being a commonly used embedded language, but I'd prefer
  to instead support several very widely used languages, such as
  Perl/Python.
 
 I'm a perl/ruby fan myself, but I would still recommend a good look at 
 lua (or python) simply because they seem to be performant, easy to use, 
 and on the surface seem to have had some thought about making them 
 embeddable.

SWIG appears to generate Lua bindings just as well. But yes, I noticed
Lua 5.1 has a feature called FFI that makes calling C functions quite
easy (but then again, SWIG is supposed to do the same thing).

 Perl has some memory management issues if you leave it long running, 
 also writing XS code looks ok on the surface, but is an exercise in hair 
 pulling in practice

I've written a Perl plugin to irssi before directly with XS, and that's
my main motivation for using some kind of an easy generator this time.
It's way too much trouble to write any glue functions, especially when
it works for only a single scripting language.

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Ed W]

On 13/02/2012 19:43, Jan-Frode Myklebust wrote:

On Mon, Feb 13, 2012 at 11:08:48AM -0800, Mark Moseley wrote:

Out of curiosity, are you running dovecot locally on those webmail
servers as well, or is it talking to remote dovecot servers?

The webmail servers are talking with dovecot director servers which in
turn are talking with the backend dovecot servers. Each service running
on different servers.

Webmail-servers -  director-servers -  backend-servers



I think the original question was still sensible.  In your case it seems 
like the ping times are identical between:

webmail - imap-proxy
webmail - imap server

I think your results show that a proxy has little (or negative) benefit 
in this situation, but it seems feasible that a proxy could eliminate 
several RTT trips in the event that the proxy is closer than the imap 
server?  This might happen if say the imap server is in a different 
datacenter (webmail on an office server machine?)


I'm also pleased to see that there is little negative cost in using a 
proxy... I recently added imap-proxy to our webmail setup because I 
wanted to log last login + logout times.  I haven't quite figured out 
how to best log logout time (Timo, any chance of a post logout script? 
Or perhaps it's possible with the current login scripting?).  However, 
using imap-proxy has the benefit of clustering logins a little and 
this makes log files a little easier to understand in the face of users 
with desktop mail clients plus webmail users.  Possibly this idea useful 
to someone else...


Thanks for measuring this!

Ed W

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On 21.2.2012, at 16.33, Ed W wrote:

 I'm also pleased to see that there is little negative cost in using a 
 proxy... I recently added imap-proxy to our webmail setup because I wanted to 
 log last login + logout times.  I haven't quite figured out how to best log 
 logout time (Timo, any chance of a post logout script? Or perhaps it's 
 possible with the current login scripting?).

You could of course grep the logs, but other than that you'd need to write a 
Dovecot plugin. Luckily it's really simple to write a plugin. Basically:

void postlogout_init(struct module *module) { }
void postlogout_deinit(void) {
  system(/usr/local/bin/dovecot-postlogout.sh);
} 

Add a few missing #includes and compile and enable for imap/pop3 and that 
should be it.

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On 14.2.2012, at 5.30, Michael M Slusarz wrote:

 Actually, this could probably be safely implemented by sending all of the 
 state to the client as a string:
 
 * OK [SAVEDSTATE base64-encoded-state]
 
 There isn't a whole lot of state to be saved really. Mailbox GUID, 
 UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have the 
 language/etc. states. Clients could restore their earlier state from days 
 ago, as long as Dovecot still has the necessary .log records available 
 (similar to how QRESYNC works).
 
 Given that it is not *that* expensive to re-create the state, I don't think 
 the ability to recreate state from several days ago would be worth the effort 
 of storing in the log/index files.

There wouldn't be *any* additional state stored. All the necessary state is 
already stored to .log files for other purposes. The base64-encoded-state would 
be a self-contained description of the IMAP connection's entire state.

 As for base-64-encoded state: if other IMAP servers ever wanted to implement 
 a similar protocol trying to coordinate the data structure would be a 
 nightmare.  Keeping it to something like a MODSEQ value would hide the state 
 ID - data abstraction entirely within dovecot.  And would allow you to 
 change your mind in the future if
 you come up with a better way to represent state.

The client doesn't need to care about the data structure. The client simply 
reads a string and sends it later back to server if it wants to restore that 
state. Only the server needs to verify that the string looks reasonable. The 
same string wouldn't be sent to a different server implementation, so there 
wouldn't be any interoperability issues. Each server can implement it in 
whatever way they want to (although there should be some checks in case the 
same string is sent to different versions of the same server).

[Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Jan-Frode Myklebust]

We've been collecting some stats to see what kind of benefits
UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is
running in High-performance mode http://wiki2.dovecot.org/LoginProcess
with authentication caching http://wiki2.dovecot.org/Authentication/Caching

During the weekend two servers (webmail3 and webmail4) has been running
with local imapproxy and two servers without (webmail1 and webmail2). Each
server has served about 1 million http requests, over 3 days. 

server  avg. response time  # requests

webmail1.example.net   0.3704111092386
webmail2.example.net   0.3742271045141
webmail3.example.net   0.3780971043919  imapproxy
webmail4.example.net   0.3785931028653  imapproxy


ONLY requests that took more than 5 seconds to process:

server  avg. response time  # requests

webmail1.example.net   26.048  1125
webmail2.example.net   26.2997 1080
webmail3.example.net   28.5596 808  imapproxy
webmail4.example.net   27.1004 964  imapproxy

ONLY requests that took more than 10 seconds to process:

server  avg. response time  # requests

webmail1.example.net   49.1407 516
webmail2.example.net   53.0139 459
webmail3.example.net   59.7906 333  imapproxy
webmail4.example.net   58.167  384  imapproxy

The responstimes are not very fast, but they do seem to support
the claim that an imapproxy isn't needed for dovecot.


  -jf

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Mark Moseley]

On Mon, Feb 13, 2012 at 5:54 AM, Jan-Frode Myklebust janfr...@tanso.net wrote:
 We've been collecting some stats to see what kind of benefits
 UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is
 running in High-performance mode http://wiki2.dovecot.org/LoginProcess
 with authentication caching http://wiki2.dovecot.org/Authentication/Caching

 During the weekend two servers (webmail3 and webmail4) has been running
 with local imapproxy and two servers without (webmail1 and webmail2). Each
 server has served about 1 million http requests, over 3 days.

 server          avg. response time      # requests
 
 webmail1.example.net   0.370411        1092386
 webmail2.example.net   0.374227        1045141
 webmail3.example.net   0.378097        1043919  imapproxy
 webmail4.example.net   0.378593        1028653  imapproxy


 ONLY requests that took more than 5 seconds to process:

 server          avg. response time      # requests
 
 webmail1.example.net   26.048          1125
 webmail2.example.net   26.2997         1080
 webmail3.example.net   28.5596         808      imapproxy
 webmail4.example.net   27.1004         964      imapproxy

 ONLY requests that took more than 10 seconds to process:

 server          avg. response time      # requests
 
 webmail1.example.net   49.1407         516
 webmail2.example.net   53.0139         459
 webmail3.example.net   59.7906         333      imapproxy
 webmail4.example.net   58.167          384      imapproxy

 The responstimes are not very fast, but they do seem to support
 the claim that an imapproxy isn't needed for dovecot.

Out of curiosity, are you running dovecot locally on those webmail
servers as well, or is it talking to remote dovecot servers? I ask
because I'm looking at moving our webmail from an on-box setup to a
remote pool to support director and was going to look into whether
running imapproxyd would help there. We don't bother with it in the
local setup, since dovecot is so fast, but remote (but still on a LAN)
might be different. Though imapproxyd seems to make (wait for it...)
squirrelmail unhappy (complains about IMAP errors, when sniffing shows
none), though I've not bothered to debug it yet.

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Jan-Frode Myklebust]

On Mon, Feb 13, 2012 at 04:14:22PM +0200, Timo Sirainen wrote:
  The responstimes are not very fast, but they do seem to support
  the claim that an imapproxy isn't needed for dovecot.
 
 That's what I always suspected, but good to have someone actually test it. :) 
 This is with Maildir?

Yes, this is maildirs (on GPFS).

 
 Other things that would be interesting to try out (both from latency and disk 
 IO usage point of view):
 
  - maildir_very_dirty_syncs

We already have

$ doveconf maildir_very_dirty_syncs
maildir_very_dirty_syncs = yes

but I don't think this gave the advantage I was expecting.. Was
expecting this to move most iops to the index-luns, but the maildir
luns seems just as busy.

  - mail_prefetch_count (Linux+maildir only, v2.1+)

Will look into if this works with GPFS when we upgrade to v2.1. It has
it's own page cache, so I have no idea if it will respect
POSIX_FADV_WILLNEED or if one will need to use it's own API's for
hinting:


http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r4.gpfs300.doc%2Fbl1adm_mlacrge.html


  -jf

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Jan-Frode Myklebust]

On Mon, Feb 13, 2012 at 11:08:48AM -0800, Mark Moseley wrote:
 
 Out of curiosity, are you running dovecot locally on those webmail
 servers as well, or is it talking to remote dovecot servers?

The webmail servers are talking with dovecot director servers which in
turn are talking with the backend dovecot servers. Each service running
on different servers.

Webmail-servers - director-servers - backend-servers

 I ask because I'm looking at moving our webmail from an on-box setup to a
 remote pool to support director and was going to look into whether
 running imapproxyd would help there. We don't bother with it in the
 local setup, since dovecot is so fast, but remote (but still on a LAN)
 might be different.

Doesn't seem so to us...

 Though imapproxyd seems to make (wait for it...)
 squirrelmail unhappy (complains about IMAP errors, when sniffing shows
 none), though I've not bothered to debug it yet.

:-)


  -jf

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On 13.2.2012, at 21.36, Jan-Frode Myklebust wrote:

 Other things that would be interesting to try out (both from latency and 
 disk IO usage point of view):
 
 - maildir_very_dirty_syncs
 
 We already have
 
   $ doveconf maildir_very_dirty_syncs
   maildir_very_dirty_syncs = yes
 
 but I don't think this gave the advantage I was expecting.. Was
 expecting this to move most iops to the index-luns, but the maildir
 luns seems just as busy.

This setting should get rid of almost all readdir() calls. If it doesn't, 
something's not working right.

 - mail_prefetch_count (Linux+maildir only, v2.1+)
 
 Will look into if this works with GPFS when we upgrade to v2.1. It has
 it's own page cache, so I have no idea if it will respect
 POSIX_FADV_WILLNEED or if one will need to use it's own API's for
 hinting:
 
   
 http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=%2Fcom.ibm.cluster.gpfs.v3r4.gpfs300.doc%2Fbl1adm_mlacrge.html

I guess if there's an easy way to lookup filename or fd - blockNumber that 
wouldn't be difficult to implement with a plugin.

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Michael M Slusarz]

Quoting Jan-Frode Myklebust janfr...@tanso.net:


We've been collecting some stats to see what kind of benefits
UP/SquirrelMail's IMAP Proxy in for our SOGo webmail users. Dovecot is
running in High-performance mode http://wiki2.dovecot.org/LoginProcess
with authentication caching http://wiki2.dovecot.org/Authentication/Caching

During the weekend two servers (webmail3 and webmail4) has been running
with local imapproxy and two servers without (webmail1 and webmail2). Each
server has served about 1 million http requests, over 3 days.

server  avg. response time  # requests

webmail1.example.net   0.3704111092386
webmail2.example.net   0.3742271045141
webmail3.example.net   0.3780971043919  imapproxy
webmail4.example.net   0.3785931028653  imapproxy


ONLY requests that took more than 5 seconds to process:

server  avg. response time  # requests

webmail1.example.net   26.048  1125
webmail2.example.net   26.2997 1080
webmail3.example.net   28.5596 808  imapproxy
webmail4.example.net   27.1004 964  imapproxy

ONLY requests that took more than 10 seconds to process:

server  avg. response time  # requests

webmail1.example.net   49.1407 516
webmail2.example.net   53.0139 459
webmail3.example.net   59.7906 333  imapproxy
webmail4.example.net   58.167  384  imapproxy

The responstimes are not very fast, but they do seem to support
the claim that an imapproxy isn't needed for dovecot.


Except you are most likely NOT leveraging the truly interesting part  
of imapproxy - the ability to restore the IMAP connection state via  
the XPROXYREUSE status response.  This is a significant performance  
improvement since it also reduces processing load on the client side  
(everything before/including authentication needs to be done whether  
using imapproxy or not, so there is no client-side savings for these  
commands).


For further information, see, e.g.:

http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html
http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html

These posts neglect the fact that you don't need to issue a CAPABILITY  
command if the connection is reused either, so that's an additional  
advantage. Note that the XPROXYREUSE-enabled MUA must be the exclusive  
user of the imapproxy instance for this feature to work correctly.


Somewhat topical, since Timo was just mentioning support for some sort  
of IMAP state save/restore feature possibly making it into 2.2.


michael

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Jan-Frode Myklebust]

On Mon, Feb 13, 2012 at 09:57:31PM +0200, Timo Sirainen wrote:
  
  $ doveconf maildir_very_dirty_syncs
  maildir_very_dirty_syncs = yes
  
  but I don't think this gave the advantage I was expecting.. Was
  expecting this to move most iops to the index-luns, but the maildir
  luns seems just as busy.
 
 This setting should get rid of almost all readdir() calls. If it doesn't, 
 something's not working right.


With maildir_very_dirty_syncs = yes:

ReadMB/s  WriteMB/s F_open  f_close reads   writes  rdirinode
  1.5 0.0   96  92  514 73  9   7
  1.2 0.0   59  43  367 18  4   76
  1.7 0.0   66  61  477 67  2   6
  1.2 0.0   54  50  348 31  1   145
  3.0 0.0   113 90  860 59  7   8
  2.9 0.0   107 99  840 58  5   11
  4.0 0.0   131 101 111777  2   65

With maildir_very_dirty_syncs = no (same node, a few seconds later):

ReadMB/s  WriteMB/s F_open  f_close reads   writes  rdirinode
  4.6 0.9   125 91  1161109641  6
  2.3 0.7   200 170 697 127 86  16
  1.1 0.6   124 99  406 61  48  109
  2.7 0.1   212 144 755 114 74  15
  2.7 0.0   159 133 818 70  78  194
  0.8 1.2   86  73  225 60  16  9
  1.9 0.0   124 116 573 53  30  6

So it seems to be working, good :-)


  -jf

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Jan-Frode Myklebust]

On Mon, Feb 13, 2012 at 01:24:25PM -0700, Michael M Slusarz wrote:
 
 Except you are most likely NOT leveraging the truly interesting part
 of imapproxy - the ability to restore the IMAP connection state via
 the XPROXYREUSE status response.  This is a significant performance
 improvement since it also reduces processing load on the client side
 (everything before/including authentication needs to be done whether
 using imapproxy or not, so there is no client-side savings for these
 commands).

Thanks for this info, good to know. I'll check with inverse/sogo if this
is something they use/intend to use..

 
 additional advantage. Note that the XPROXYREUSE-enabled MUA must be
 the exclusive user of the imapproxy instance for this feature to
 work correctly.

Not a problem. Assuming it doesn't also need to be the only imap user of
the account/folder.

BTW: do you also have information on the state of select caching in the
up-imapproxy? I got some very negative comments when googling it, and the
changelog didn't suggest there had been any improvements since..


  -jf

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On 13.2.2012, at 22.24, Michael M Slusarz wrote:

 http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html
 http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html
 
 These posts neglect the fact that you don't need to issue a CAPABILITY 
 command if the connection is reused either, so that's an additional 
 advantage. Note that the XPROXYREUSE-enabled MUA must be the exclusive user 
 of the imapproxy instance for this feature to work correctly.
 
 Somewhat topical, since Timo was just mentioning support for some sort of 
 IMAP state save/restore feature possibly making it into 2.2.

Perhaps a way for (trusted) clients to be able to do this? :)

a logout save
* OK [SAVEDSTATE 1234567890]
* BYE logged out
a OK

...

b login (SAVEDSTATE 1234567890) user pass
* OK [RESTOREDSTATE 1234567890]
b ok

vs.

b login (SAVEDSTATE 1234567890) user pass
* NO [UNKNOWNSTATE 1234567890]
b ok

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On 13.2.2012, at 23.32, Timo Sirainen wrote:

 Perhaps a way for (trusted) clients to be able to do this? :)
 
 a logout save
 * OK [SAVEDSTATE 1234567890]
 * BYE logged out
 a OK

Actually, this could probably be safely implemented by sending all of the state 
to the client as a string:

* OK [SAVEDSTATE base64-encoded-state]

There isn't a whole lot of state to be saved really. Mailbox GUID, UIDVALIDITY, 
HIGHESTMODSEQ gives the mailbox state. Then you have the language/etc. states. 
Clients could restore their earlier state from days ago, as long as Dovecot 
still has the necessary .log records available (similar to how QRESYNC works).

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Michael M Slusarz]

Quoting Jan-Frode Myklebust janfr...@tanso.net:


BTW: do you also have information on the state of select caching in the
up-imapproxy? I got some very negative comments when googling it, and the
changelog didn't suggest there had been any improvements since..


I wouldn't trust it.  IIRC, it was added years ago and given the  
syntax changes to the SELECT/EXAMINE call since then (e.g. QRESYNC  
extentsion), it is doubtful the code has been updated to handle these  
situations.  And in a server-caching IMAP server like dovecot, all  
that SELECT information is not expensive anyway, so any gains would be  
miniscule.


michael

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Michael M Slusarz]

Quoting Timo Sirainen t...@iki.fi:


On 13.2.2012, at 22.24, Michael M Slusarz wrote:


http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052316.html
http://lists.horde.org/archives/imp/Week-of-Mon-20110523/052317.html

These posts neglect the fact that you don't need to issue a  
CAPABILITY command if the connection is reused either, so that's an  
additional advantage. Note that the XPROXYREUSE-enabled MUA must be  
the exclusive user of the imapproxy instance for this feature to  
work correctly.


Somewhat topical, since Timo was just mentioning support for some  
sort of IMAP state save/restore feature possibly making it into 2.2.


Perhaps a way for (trusted) clients to be able to do this? :)

a logout save
* OK [SAVEDSTATE 1234567890]
* BYE logged out
a OK

...

b login (SAVEDSTATE 1234567890) user pass
* OK [RESTOREDSTATE 1234567890]
b ok

vs.

b login (SAVEDSTATE 1234567890) user pass
* NO [UNKNOWNSTATE 1234567890]
b ok


I guess the drawback for this approach is that you are explicitly  
breaking the LOGIN definition.  And you don't allow reviving the state  
if using the AUTHENTICATE command.


The alternative would be to have an additional pre-authentication  
command that sets the desired state.  But that adds the overhead of  
sending/parsing another command.


michael

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Timo Sirainen]

On 14.2.2012, at 5.19, Michael M Slusarz wrote:

 b login (SAVEDSTATE 1234567890) user pass
 
 I guess the drawback for this approach is that you are explicitly breaking 
 the LOGIN definition.  

No breaking, extending :)

 And you don't allow reviving the state if using the AUTHENTICATE command.

Just as easy:

b authenticate (savedstate ..) plain ..

 The alternative would be to have an additional pre-authentication command 
 that sets the desired state.  But that adds the overhead of sending/parsing 
 another command.

Yeah. Although not bad either, since you can pipeline that command + 
login/authenticate.

Re: [Dovecot] IMAP-proxy or not with sogo webmail and dovecot backend

[Michael M Slusarz]

Quoting Timo Sirainen t...@iki.fi:


On 13.2.2012, at 23.32, Timo Sirainen wrote:


Perhaps a way for (trusted) clients to be able to do this? :)

a logout save
* OK [SAVEDSTATE 1234567890]
* BYE logged out
a OK


Actually, this could probably be safely implemented by sending all  
of the state to the client as a string:


* OK [SAVEDSTATE base64-encoded-state]

There isn't a whole lot of state to be saved really. Mailbox GUID,  
UIDVALIDITY, HIGHESTMODSEQ gives the mailbox state. Then you have  
the language/etc. states. Clients could restore their earlier state  
from days ago, as long as Dovecot still has the necessary .log  
records available (similar to how QRESYNC works).


Given that it is not *that* expensive to re-create the state, I don't  
think the ability to recreate state from several days ago would be  
worth the effort of storing in the log/index files.  For a  
disconnected client (e.g. webmail), there's going to necessarily need  
to be startup costs of initializing the session anyway so re-creating  
the state can be buried in this other work.


Re-creating becomes more important when users are doing actions they  
expect immediate (or at least very fast) responses for.  Things like  
listing messages in a mailbox, viewing a message, or polling  
mailboxes.  Saving 50ms per request becomes important in these  
situations since re-creating state now takes 10% of the total request  
time.


So I don't think states don't need a terribly long lifetime.  I almost  
think of something like an internal Dovecot IDLE queue - after a  
logout is received, state is stored for 30 minutes after which it is  
discarded.  (Although not knowing anything about internal Dovecot  
state, don't know if this is overly resource intensive.)


As for base-64-encoded state: if other IMAP servers ever wanted to  
implement a similar protocol trying to coordinate the data structure  
would be a nightmare.  Keeping it to something like a MODSEQ value  
would hide the state ID - data abstraction entirely within dovecot.   
And would allow you to change your mind in the future if

you come up with a better way to represent state.

michael