Some questions about code in lib-storage/index/maildir
Hi, Recently, I have read some code in dovecot 2.3.13. And I have 2 questions
about some lines of code in lib-storage/index/maildir Firstly, in the function,
maildir_scan_dir, from lib-storage/index/maildir/maildir-sync.c around line 575
(https://github.com/dovecot/core/blob/6986a435e31dfe5bcd14985534b76b3c735f7831/src/lib-storage/index/maildir/maildir-sync.c#L575)
I would like to know that after successfully stat the cur directory, does it
assign the max value between st.st_mtime and start_time intensionally to
new_check_time? Or maybe should it assign the value to the cur_check_time. if
(stat(ctx->cur_dir, ) == 0) { ctx->mbox->maildir_hdr.new_check_time =
I_MAX(st.st_mtime, start_time); ctx->mbox->maildir_hdr.cur_mtime = st.st_mtime;
ctx->mbox->maildir_hdr.cur_mtime_nsecs = ST_MTIME_NSEC(st); } Secondly, in the
function, maildir_uidlist_sync_next_uid, from
lib-storage/index/maildir/maildir-uidlist.c
(https://github.com/dovecot/core/blob/6986a435e31dfe5bcd14985534b76b3c735f7831/s
rc/lib-storage/index/maildir/maildir-uidlist.c#L1760) Is it ok to remove
MAILDIR_UIDLIST_REC_FLAG_NEW_DIR flag from the record for the record that is
not found in ctx->files just like the behavior
(https://github.com/dovecot/core/blob/6986a435e31dfe5bcd14985534b76b3c735f7831/src/lib-storage/index/maildir/maildir-uidlist.c#L1723)in
the function, maildir_uidlist_sync_next_partial
(https://github.com/dovecot/core/blob/6986a435e31dfe5bcd14985534b76b3c735f7831/src/lib-storage/index/maildir/maildir-uidlist.c#L1723),
from the same file? I think when it gets to the function,
maildir_uidlist_sync_next_uid, the mail file have already been moved to cur
directory. And maybe it is ok to remove MAILDIR_UIDLIST_REC_FLAG_NEW_DIR flag
from the record. That is, is it ok to add this line rec->flags &=
ENUM_NEGATE(MAILDIR_UIDLIST_REC_FLAG_NEW_DIR); just before rec->flags =
(rec->flags | flags) & ENUM_NEGATE(MAILDIR_UIDLIST_REC_FLAG_NONSYNCED); in
maildir_uidlist_sync_next_uid? Thank you in advance to he
lp me get more understanding to these questions, Sony
Re: Some questions about mail_crypt setups
On 2021-02-22 2:25 am, Aki Tuomi wrote: > On 22/02/2021 00:20 deano-dovecot@areyes.comwrote: Some questions about > mail_crypt setups I have global mail encryption working nicely, and > replication works nicely between two systems. The main problem is that the > private and public keys are *right there* on the server in > /etc/dovecot/private ... Fine for a completely controlled system, but not so > fine when on a rented VPS etc. When are the keys read in by dovecot ? Are > they ever read in again while dovecot is running, or does it cache them in > ram until dovecot is restarted ? Would it be possible for dovecot to read the > keys as output from a script ? I'm thinking of a small script that would > reach out to an authentication service like Authy or Okta or similar. Admin > gets an alert on their phone, taps OK, UNLOCK and the two keys are returned > to the script, which then hands them back to dovecot and away it goes. The > mail_crypt config normally contains mail_crypt_global_private_key = > Recently one solution used was to provide per-user global keypair, which is > used to encrypt everything for a user. This can be easier than using the > managed keys and encrypting the user's key with password. Any examples around ? DC
Re: Some questions about mail_crypt setups
> On 22/02/2021 00:20 deano-dove...@areyes.com wrote:
>
>
> Some questions about mail_crypt setups
> I have global mail enecryption working nicely, and replication works nicely
> between two systems. The main problem is that the private and public keys are
> *right there* on the server in /etc/dovecot/private ... Fine for a completely
> controlled system, but not so fine when on a rented VPS etc.
> When are the keys read in by dovecot ? Are they ever read in again while
> dovecot is running, or does it cache them in ram until dovecot is restarted ?
> Would it be possible for dovecot to read the keys as output from a script ?
> I'm thinking of a small script that would reach out to an authentication
> service like Authy or Okta or similar. Admin gets an alert on their phone,
> taps OK, UNLOCK and the two keys are returned to the script, which then hands
> them back to dovecot and away it goes.
> The mail_crypt config normally contains
> > mail_crypt_global_private_key = > mail_crypt_global_public_key = Perhaps add another variable like
> > mail_crypt_global_script = That script would run and feed the two keys back into dovecot (no matter how
> it got to them).
>
> So I started looking into per-user/per-folder encryption to see how that
> would work, and I have that setup nicely too. The config looks like this
> > # /etc/dovecot/conf.d/99-mailcrypt.conf
> > #--
> > mail_attribute_dict = file:%h/Maildir/dovecot-attributes
> > plugin {
> > mail_crypt_require_encrypted_user_key = yes
> > mail_crypt_save_version = 2
> > mail_crypt_curve = secp521r1
> > }
> >
> > # /etc/dovecot/dovecot-sql.conf.ext
> > #--
> > # CREATE TABLE IF NOT EXISTS `users` (
> > # `username` varchar(64) character set utf8 collate utf8_bin NOT NULL
> > COMMENT 'localpart of email-address',
> > # `domain` varchar(64) character set utf8 collate utf8_bin NOT NULL COMMENT
> > 'domain-part of email-address',
> > # `name` varchar(64) character set utf8 collate utf8_bin NOT NULL COMMENT
> > 'Full name of user',
> > # `password` varchar(128) character set utf8 collate utf8_bin NOT NULL
> > COMMENT 'base64-encoded SHA512 hash of password',
> > # PRIMARY KEY (`username`,`domain`)
> > # ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='users';
> > driver = mysql
> > connect = host=/var/run/mysqld/mysqld.sock dbname=emailers user=dovecot
> > password=RandomPassword
> > default_pass_scheme = SHA512-CRYPT
> > password_query = SELECT username, password, '%w' AS
> > userdb_mail_crypt_private_password, '/var/mail/%d/%n' AS userdb_home,
> > 'vmail' AS userdb_uid, 'vmail' AS userdb_gid FROM users WHERE username =
> > '%n' AND domain = '%d'
> > # For LDA:
> > user_query = SELECT '/var/mail/%d/%n' AS home, 'vmail' AS uid, 'vmail' AS
> > gid FROM users WHERE username = '%n' AND domain = '%d'
> > # For using doveadm -A:
> > iterate_query = SELECT username, domain FROM users
>
> Except that replication doesn't work due to the user password not being
> available. Actually, indexing fails too for the same reason.
> > Feb 21 14:02:13
> > indexer-worker(testu...@example.com)<120846>:
> > Error: Mailbox INBOX: UID=1: read() failed:
> > read(/var/mail/example.com/testuser/Maildir/INBOX/new/1613934133.M132059P120842.dove1,S=2568,W=2624)
> > failed: Private key not available: Cannot decrypt key
> > f64e7c12a60b3df12ebf865a70bec57fedd3e9b4fd98df93205f1096db14fda7: Cannot
> > decrypt key
> > eca099273f525ca46b2f5640253770ad19e0578543244d8cd34bde183e996bd5: Password
> > not available (read reason=fts indexing)
> > Feb 21 14:02:13
> > indexer-worker(testu...@example.com)<120846>:
> > Error: Failed to read mailbox INBOX mail UID=1 stream: Mailbox INBOX:
> > UID=1: read() failed:
> > read(/var/mail/example.com/testuser/Maildir/INBOX/new/1613934133.M132059P120842.dove1,S=2568,W=2624)
> > failed: Private key not available: Cannot decrypt key
> > f64e7c12a60b3df12ebf865a70bec57fedd3e9b4fd98df93205f1096db14fda7: Cannot
> > decrypt key
> > eca099273f525ca46b2f5640253770ad19e0578543244d8cd34bde183e996bd5: Password
> > not available (read reason=fts indexing)
> > Feb 21 14:02:13
> > indexer-worker(testu...@example.com)<120846>:
> > Error: Mailbox INBOX: Mail search failed: Internal error occurred. Refer
> > to server log for more information. [2021-02-21 14:02:13]
> > Feb 21 14:02:13
> > indexer-worker(testu...@example.com)<120846>:
> > Error: Mailbox INBOX: Transaction comm
Re: Some questions about mail_crypt setups
On Sun, Feb 21, 2021 at 05:20:59PM -0500, deano-dove...@areyes.com wrote: > I have global mail enecryption working nicely, and replication works > nicely between two systems. The main problem is that the private and > public keys are *right there* on the server in /etc/dovecot/private > ... Fine for a completely controlled system, but not so fine when on > a rented VPS etc. I'm not running a Dovecot instance myself at the moment, but I have been wondering about the above. My current understanding is that Dovecot, like any other piece of software that needs to decrypt data from disk, will inevitably need to either: - keep the private keys in memory for at least *some* time, in order to be able to perform decryption using the CPU; OR - use an HSM (or equivalent, such as maybe a TPM or an OpenPGP Card) to perform decryption as needed. In a case where there is no HSM (or equivalent), any attacker who gains root or hypervisor privileges over the machine can in principle extract the key from memory irrespective of whether the private key is on disk. They can then decrypt messages at their leisure. In such a case, the security is already quite low and little additional security is lost by keeping the private key in a local file on disk that is readable only by root (and perhaps also readable by one other carefully-chosen account if necessary). The above applies to rented VPSes. You are vulnerable to the VPS provider, because they have hypervisor privileges. So, if you want the email store to be private, the first thing to do is have it on your own hardware. In the better case where you have your own hardware, then the concern becomes: how to avoid attackers accessing the private keys if they gain root, or if they gain physical access. Here, an HSM (or equivalent) will help, by keeping the private keys off the filesystem and out of RAM/cache/etc. A properly-implemented HSM or smartcard will make it infeasible for an attacker to obtain the private key even if they gain root; and will make it expensive for an attacker to obtain the private key even if they gain physical access. Can Dovecot utilise an HSM (or equivalent)? I'm not sure. I look forward to finding out. > Would it be possible for dovecot to read the keys as output from a > script ? I'm thinking of a small script that would reach out to an > authentication service like Authy or Okta or similar. Making your own ability to access the email store dependent upon an untrustworthy third-party like Okta is, IMO, even worse than using a VPS. Not only are you leaving the door open to an attacker should that service provider prove to be either compromised or malicious; you also leave yourself vulnerable to a whole new class of DoS attacks. (Okta is mostly security theatre. The basic premise is bad enough, but auditing various Okta deployments, and meeting and speaking with Okta technical staff, left me with an even worse impression of that company.) Sam -- A: When it messes up the order in which people normally read text. Q: When is top-posting a bad thing? () ASCII ribbon campaign. Please avoid HTML emails & proprietary /\ file formats. (Why? See e.g. https://v.gd/jrmGbS ). Thank you.
Some questions about mail_crypt setups
Some questions about mail_crypt setups
I have global mail enecryption working nicely, and replication works
nicely between two systems. The main problem is that the private and
public keys are *right there* on the server in /etc/dovecot/private ...
Fine for a completely controlled system, but not so fine when on a
rented VPS etc.
When are the keys read in by dovecot ? Are they ever read in again while
dovecot is running, or does it cache them in ram until dovecot is
restarted ?
Would it be possible for dovecot to read the keys as output from a
script ? I'm thinking of a small script that would reach out to an
authentication service like Authy or Okta or similar. Admin gets an
alert on their phone, taps OK, UNLOCK and the two keys are returned to
the script, which then hands them back to dovecot and away it goes.
The mail_crypt config normally contains
> mail_crypt_global_private_key = mail_crypt_global_public_key = mail_crypt_global_script = # /etc/dovecot/conf.d/99-mailcrypt.conf
> #--
> mail_attribute_dict = file:%h/Maildir/dovecot-attributes
> plugin {
> mail_crypt_require_encrypted_user_key = yes
> mail_crypt_save_version = 2
> mail_crypt_curve = secp521r1
> }
>
> # /etc/dovecot/dovecot-sql.conf.ext
> #--
> # CREATE TABLE IF NOT EXISTS `users` (
> # `username` varchar(64) character set utf8 collate utf8_bin NOT NULL COMMENT
> 'localpart of email-address',
> # `domain` varchar(64) character set utf8 collate utf8_bin NOT NULL COMMENT
> 'domain-part of email-address',
> # `name` varchar(64) character set utf8 collate utf8_bin NOT NULL COMMENT
> 'Full name of user',
> # `password` varchar(128) character set utf8 collate utf8_bin NOT NULL
> COMMENT 'base64-encoded SHA512 hash of password',
> # PRIMARY KEY (`username`,`domain`)
> # ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='users';
>
> driver = mysql
> connect = host=/var/run/mysqld/mysqld.sock dbname=emailers user=dovecot
> password=RandomPassword
> default_pass_scheme = SHA512-CRYPT
>
> password_query = SELECT username, password, '%w' AS
> userdb_mail_crypt_private_password, '/var/mail/%d/%n' AS userdb_home, 'vmail'
> AS userdb_uid, 'vmail' AS userdb_gid FROM users WHERE username = '%n' AND
> domain = '%d'
>
> # For LDA:
> user_query = SELECT '/var/mail/%d/%n' AS home, 'vmail' AS uid, 'vmail' AS gid
> FROM users WHERE username = '%n' AND domain = '%d'
>
> # For using doveadm -A:
> iterate_query = SELECT username, domain FROM users
Except that replication doesn't work due to the user password not being
available. Actually, indexing fails too for the same reason.
> Feb 21 14:02:13
> indexer-worker(testu...@example.com)<120846>:
> Error: Mailbox INBOX: UID=1: read() failed:
> read(/var/mail/example.com/testuser/Maildir/INBOX/new/1613934133.M132059P120842.dove1,S=2568,W=2624)
> failed: Private key not available: Cannot decrypt key
> f64e7c12a60b3df12ebf865a70bec57fedd3e9b4fd98df93205f1096db14fda7: Cannot
> decrypt key eca099273f525ca46b2f5640253770ad19e0578543244d8cd34bde183e996bd5:
> Password not available (read reason=fts indexing)
>
> Feb 21 14:02:13
> indexer-worker(testu...@example.com)<120846>:
> Error: Failed to read mailbox INBOX mail UID=1 stream: Mailbox INBOX: UID=1:
> read() failed:
> read(/var/mail/example.com/testuser/Maildir/INBOX/new/1613934133.M132059P120842.dove1,S=2568,W=2624)
> failed: Private key not available: Cannot decrypt key
> f64e7c12a60b3df12ebf865a70bec57fedd3e9b4fd98df93205f1096db14fda7: Cannot
> decrypt key eca099273f525ca46b2f5640253770ad19e0578543244d8cd34bde183e996bd5:
> Password not available (read reason=fts indexing)
>
> Feb 21 14:02:13
> indexer-worker(testu...@example.com)<120846>:
> Error: Mailbox INBOX: Mail search failed: Internal error occurred. Refer to
> server log for more information. [2021-02-21 14:02:13]
>
> Feb 21 14:02:13
> indexer-worker(testu...@example.com)<120846>:
> Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit
> failed: transaction context (attempted to index 1 messages (UIDs 1..1))
>
> Feb 21 14:02:13 dsync-local(testu...@example.com):
> Error: Mailbox INBOX: UID=1: read() failed:
> read(/var/mail/example.com/testuser/Maildir/INBOX/new/1613934133.M132059P120842.dove1,S=2568,W=2624)
> failed: Private key not available: Cannot decrypt key
> f64e7c12a60b3df12ebf865a70bec57fedd3e9b4fd98df93205f1096db14fda7: Cannot
> decrypt key eca099273f525ca46b2f5640253770ad19e0578543244d8cd34bde183e996bd5:
> Password not available (read reason=prefetch)
What are the options here for providing the decryption password or key ?
The user password is already stored in the mysql database as a
SHA512-CRYPT so we
Re: Some questions
> Le 10 juil. 2019 à 14:06, Bardot Jérôme via dovecot a > écrit : > > Le 09/07/2019 à 17:28, Daniel Miller via dovecot a écrit : >> >> On 7/9/2019 6:17 AM, Jérôme Bardot via dovecot wrote: >>> Hello, >>> >>> This is my first email here. >>> I want to understand well how dovecot is integrate with ldap in a >>> postfix/dovecot/ldap setup. >>> I use a debian server. >> >> Perfectly! >> >>> >>> More specifically what dovecot need in ldap to work. >>> I saw we can use several "mode" related to virtual domain, etc. For >>> "start" i only need one domain with several address. >>> I currently use fusiondirectory for manage my ldap users. i guess i >>> can use that schema to auto create users email >>> (name.firstn...@domain.tld for ie) ? >>> I also want to setup some aliases and share directory based on ldap >>> group/role can i do it ? >>> >>> An other question is can we have two domain name for imap.domain.tld >>> && smtp.domain.tld ? >> >> Yes. > There is some documentation somewhere on it ? >> >> Dovecot & Postfix have no "hard" schema, or database definition, or >> particular fields. You need to create map files which tell each server >> how to use the information from LDAP (or any other database). Each >> server (Postfix & Dovecot) have their own configuration which is >> separate from each other. So you need to start with one or the other. >> Postfix questions should be asked on the Postfix list. >> >> Everything you asked for above is easily doable - just start with one >> step at a time. Ask specific questions when you get stuck. > > The map part stuck me at this time. Can i found somewhere a list of > field should/can be map ? I think i’m ok with postfix conf if i > understand well i can delegate pretty all stuff to dovecot/ldap. > > > An other question is : > > For all vitual stuff i always use a new user (system) with a custom > home, all stuff i read are not clear for me about this point. There is > some diagram with technical stuff about dovecot ? > You should start by reading https://wiki2.dovecot.org/AuthDatabase/LDAP/Userdb The main point if you use a single user is: "If you're using a single UID and GID for all the users, you can specify them globally with mail_uid and mail_gid settings instead of returning them from LDAP."
Re: Some questions
Le 09/07/2019 à 17:28, Daniel Miller via dovecot a écrit : > > On 7/9/2019 6:17 AM, Jérôme Bardot via dovecot wrote: >> Hello, >> >> This is my first email here. >> I want to understand well how dovecot is integrate with ldap in a >> postfix/dovecot/ldap setup. >> I use a debian server. > > Perfectly! > >> >> More specifically what dovecot need in ldap to work. >> I saw we can use several "mode" related to virtual domain, etc. For >> "start" i only need one domain with several address. >> I currently use fusiondirectory for manage my ldap users. i guess i >> can use that schema to auto create users email >> (name.firstn...@domain.tld for ie) ? >> I also want to setup some aliases and share directory based on ldap >> group/role can i do it ? >> >> An other question is can we have two domain name for imap.domain.tld >> && smtp.domain.tld ? > > Yes. There is some documentation somewhere on it ? > > Dovecot & Postfix have no "hard" schema, or database definition, or > particular fields. You need to create map files which tell each server > how to use the information from LDAP (or any other database). Each > server (Postfix & Dovecot) have their own configuration which is > separate from each other. So you need to start with one or the other. > Postfix questions should be asked on the Postfix list. > > Everything you asked for above is easily doable - just start with one > step at a time. Ask specific questions when you get stuck. The map part stuck me at this time. Can i found somewhere a list of field should/can be map ? I think i’m ok with postfix conf if i understand well i can delegate pretty all stuff to dovecot/ldap. An other question is : For all vitual stuff i always use a new user (system) with a custom home, all stuff i read are not clear for me about this point. There is some diagram with technical stuff about dovecot ? Thx > > -- > Daniel > 0x053A41EF03878A98.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Some questions
On 7/9/2019 6:17 AM, Jérôme Bardot via dovecot wrote: Hello, This is my first email here. I want to understand well how dovecot is integrate with ldap in a postfix/dovecot/ldap setup. I use a debian server. Perfectly! More specifically what dovecot need in ldap to work. I saw we can use several "mode" related to virtual domain, etc. For "start" i only need one domain with several address. I currently use fusiondirectory for manage my ldap users. i guess i can use that schema to auto create users email (name.firstn...@domain.tld for ie) ? I also want to setup some aliases and share directory based on ldap group/role can i do it ? An other question is can we have two domain name for imap.domain.tld && smtp.domain.tld ? Yes. Dovecot & Postfix have no "hard" schema, or database definition, or particular fields. You need to create map files which tell each server how to use the information from LDAP (or any other database). Each server (Postfix & Dovecot) have their own configuration which is separate from each other. So you need to start with one or the other. Postfix questions should be asked on the Postfix list. Everything you asked for above is easily doable - just start with one step at a time. Ask specific questions when you get stuck. -- Daniel
Some questions
Hello, This is my first email here. I want to understand well how dovecot is integrate with ldap in a postfix/dovecot/ldap setup. I use a debian server. More specifically what dovecot need in ldap to work. I saw we can use several "mode" related to virtual domain, etc. For "start" i only need one domain with several address. I currently use fusiondirectory for manage my ldap users. i guess i can use that schema to auto create users email (name.firstn...@domain.tld for ie) ? I also want to setup some aliases and share directory based on ldap group/role can i do it ? An other question is can we have two domain name for imap.domain.tld && smtp.domain.tld ? IF some questions are out of topic let me know. thx jerome
[Dovecot] Dovecot 2.1.13: some questions about the mail-filter plugin
Looking at mail_filter_mailbox_allocated() of
src/plugins/mail-filter/mail-filter-plugin.c:
if ((class_flags MAIL_STORAGE_CLASS_FLAG_OPEN_STREAMS) == 0
(class_flags MAIL_STORAGE_CLASS_FLAG_BINARY_DATA) != 0
muser-out_socket_path != NULL)
v-save_begin = mail_filter_mail_save_begin;
it seems that a backend with MAIL_STORAGE_CLASS_FLAG_OPEN_STREAMS unset and
MAIL_STORAGE_CLASS_FLAG_BINARY_DATA set is expected by the plugin.
Technically, this would thus exclude the mbox backend, since its flag
configuration seems to be exactly the opposite:
struct mail_storage mbox_storage = {
.name = MBOX_STORAGE_NAME,
.class_flags = MAIL_STORAGE_CLASS_FLAG_MAILBOX_IS_FILE |
MAIL_STORAGE_CLASS_FLAG_OPEN_STREAMS |
MAIL_STORAGE_CLASS_FLAG_HAVE_MAIL_GUIDS,
[...]
Is this a correct interpretation?
If yes, what is the rationale?
I couldn't find any info related to such a limitation in the docs; the README
file of http://dovecot.org/patches/2.2/mail-filter.tar.gz even says: Mail
filter plugin can read an email from any storage supported by Dovecot and
modify the mail in some way...
All of this stems from my attempts to have working mail filters when I was
trying to better understand the behaviors Stanislas was describing in a recent
thread...
Hence another question; in sdbox-storage.c, the definition of the alias for
sdbox comes with:
struct mail_storage dbox_storage = {
.name = dbox, /* alias */
.class_flags = MAIL_STORAGE_CLASS_FLAG_FILE_PER_MSG,
[...]
The initialization of member class_flags seems to be incomplete, when compared
to the one for the name sdbox. A typo?
On the other hand, Stanislas showed a configuration with
mail_location = dbox:/mailboxes/%Ld/%Ln
and seemed to have a working mail filter.
If above interpretation is correct, this would mean that the alias definition
isn't used, but that the one for sdbox is used instead.
Out of curiosity, how/where is that substitution achieved?
Regards,
Axel
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
Le 1 août 2013 à 18:05, Timo Sirainen a écrit : On 1.8.2013, at 19.02, Axel Luttgens wrote: [...] If yes, could it be that it is never called in my case? If not, then there's definitely some problem :) [...] Could I try to break somewhere earlier in the call chain? It should definitely stop in hook_mail_user_created, which should call quota_mail_user_created as one of the hooks. If not, the user then doesn't actually have quota plugin enabled.. And I'm definitely not a gdb guru. :-( Still trying to have it provide me with some enlightening info, but if I may in the meantime paraphrase one of my initial questions on this thread: What makes doveadm-quota/lmtp and quota-status different? As a reminder: doveadm-quota and lmtp correctly understand my quota-related settings, and over-qauota users are handled as such. On the other hand, quota-status always returns action=OK for any existing user, whether over-quota or not. According to the logs, the userdb queries correctly return all needed quota-related info for the user; on the other hand, the dict service never gets launched. It could thus be inferred that quota-status is following a slightly different path for fetching/handling quota information. Knowing the difference could help to focus my miserable gdb investigations and/or to understand what may be at the fringe in my config. TIA, Axel
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
On Fri, 2013-08-02 at 10:30 +0200, Axel Luttgens wrote: Le 1 août 2013 à 18:05, Timo Sirainen a écrit : On 1.8.2013, at 19.02, Axel Luttgens wrote: [...] If yes, could it be that it is never called in my case? If not, then there's definitely some problem :) [...] Could I try to break somewhere earlier in the call chain? It should definitely stop in hook_mail_user_created, which should call quota_mail_user_created as one of the hooks. If not, the user then doesn't actually have quota plugin enabled.. And I'm definitely not a gdb guru. :-( I'd just do: 1) start quota-status service by e.g. connecting to it via telnet 2) gdb -p `pidof quota-status` b hook_mail_user_created cont 3) recipient=user 4) does it stop?.. if yes, keep hitting s to see if it goes to quota code. Still trying to have it provide me with some enlightening info, but if I may in the meantime paraphrase one of my initial questions on this thread: What makes doveadm-quota/lmtp and quota-status different? Not much.. doveadm-quota and lmtp correctly understand my quota-related settings, and over-qauota users are handled as such. On the other hand, quota-status always returns action=OK for any existing user, whether over-quota or not. I've no idea. Send your current doveconf -n and I'll see if I can reproduce the problem with it?
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
Le 2 août 2013 à 14:13, Timo Sirainen a écrit : I'd just do: 1) start quota-status service by e.g. connecting to it via telnet 2) gdb -p `pidof quota-status` b hook_mail_user_created cont 3) recipient=user 4) does it stop?.. if yes, keep hitting s to see if it goes to quota code. To be sure, tried again, but still getting quite anarchistic behaviors, requiring some luck for retrieving useful info... Could be a clang vs gdb thing; I also tried to compile the quota plugin without optimization in the hope to bring some consistency back, without much success. Still trying to have it provide me with some enlightening info, but if I may in the meantime paraphrase one of my initial questions on this thread: What makes doveadm-quota/lmtp and quota-status different? Not much.. This is what I was tempted to believe, until... ;-) doveadm-quota and lmtp correctly understand my quota-related settings, and over-qauota users are handled as such. On the other hand, quota-status always returns action=OK for any existing user, whether over-quota or not. I've no idea. Send your current doveconf -n and I'll see if I can reproduce the problem with it? Thank you for your kind proposal; it would be such a relief, should you find something I'm overlooking. I provided that info at the very beginning of the thread, but I may have changed one detail or another in the meantime; I'll thus send you my current config privately. Best Regards, Axel
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
On 2.8.2013, at 16.18, Axel Luttgens axelluttg...@swing.be wrote: Le 2 août 2013 à 14:13, Timo Sirainen a écrit : I'd just do: 1) start quota-status service by e.g. connecting to it via telnet 2) gdb -p `pidof quota-status` b hook_mail_user_created cont 3) recipient=user 4) does it stop?.. if yes, keep hitting s to see if it goes to quota code. To be sure, tried again, but still getting quite anarchistic behaviors, requiring some luck for retrieving useful info... Could be a clang vs gdb thing; I also tried to compile the quota plugin without optimization in the hope to bring some consistency back, without much success. Optimization always makes things rather annoying. Especially with clang -O2 makes it just about impossible for gdb to do anything useful. You'd probably need to disable optimization for lib-storage also, not just quota plugin.
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
On 19.7.2013, at 16.02, Axel Luttgens axelluttg...@swing.be wrote:
Le 18 juil. 2013 à 11:25, Axel Luttgens a écrit :
[...]
It is to be noted that no lines in the log are related to possible problems
encountered for launching [the dict server]. It is a bit as if quota_check()
in src/plugins/quota/quota-status.c always immediately returned with 1 at
the first test.
[...]
Tracing with gdb, it appears this is indeed the case.
Here's the beginning of quota_check():
static int
quota_check(struct mail_user *user, uoff_t mail_size, const char
**error_r)
{
struct quota_user *quser = QUOTA_USER_CONTEXT(user);
[...]
if (quser == NULL) {
/* no quota for user */
return 1;
}
[...]
and one has for quser:
(gdb) p quser
$1 = (struct quota_user *) 0x0
And if you're still stuck with this, set a breakpoint to
quota_mail_user_created and step through it to figure out why
MODULE_CONTEXT_SET() isn't being called.
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
Le 1 août 2013 à 14:29, Timo Sirainen a écrit : And if you're still stuck with this, set a breakpoint to quota_mail_user_created and step through it to figure out why MODULE_CONTEXT_SET() isn't being called. Yes, still stuck. :-( Did you mean function quota_mail_user_created from quota-storage.c? If yes, could it be that it is never called in my case? Desperately trying to have the program break there, without success... Could I try to break somewhere earlier in the call chain? TIA, Axel
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
On 1.8.2013, at 19.02, Axel Luttgens axelluttg...@swing.be wrote: Le 1 août 2013 à 14:29, Timo Sirainen a écrit : And if you're still stuck with this, set a breakpoint to quota_mail_user_created and step through it to figure out why MODULE_CONTEXT_SET() isn't being called. Yes, still stuck. :-( Did you mean function quota_mail_user_created from quota-storage.c? Yes. If yes, could it be that it is never called in my case? If not, then there's definitely some problem :) Desperately trying to have the program break there, without success... Could I try to break somewhere earlier in the call chain? It should definitely stop in hook_mail_user_created, which should call quota_mail_user_created as one of the hooks. If not, the user then doesn't actually have quota plugin enabled..
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
Le 18 juil. 2013 à 11:25, Axel Luttgens a écrit :
[...]
It is to be noted that no lines in the log are related to possible problems
encountered for launching [the dict server]. It is a bit as if quota_check()
in src/plugins/quota/quota-status.c always immediately returned with 1 at the
first test.
[...]
Tracing with gdb, it appears this is indeed the case.
Here's the beginning of quota_check():
static int
quota_check(struct mail_user *user, uoff_t mail_size, const char
**error_r)
{
struct quota_user *quser = QUOTA_USER_CONTEXT(user);
[...]
if (quser == NULL) {
/* no quota for user */
return 1;
}
[...]
and one has for quser:
(gdb) p quser
$1 = (struct quota_user *) 0x0
Yet, struct user passed as argument doesn't show obvious problems (but I have
to confess the details are faaar beyond me); I reproduce it at the end of this
message.
So, either my users aren't recognized as being subjected to quotas, or
something goes wrong with macro QUOTA_USER_CONTEXT (which in turn translates
into macro MODULE_CONTEXT which I just don't understand), or both.
As a reminder, with the same configs, doveadm quota and lmtp do not show such
a behavior: they both take quotas into account for my users.
Any ideas?
TIA,
Axel
(gdb) p *user
$2 = {
pool = 0x7fed9b829020,
v = {
deinit = 0x10b190dd0 quota_user_deinit
},
vlast = 0x7fed9b82a188,
refcount = 1,
username = 0x7fed9b829110 john@example.com,
_home = 0x7fed9b829e08 /_Mailstores/john.doe,
uid = 999,
gid = 999,
service = 0x7fed9b829e30 quota-status,
local_ip = 0x0,
remote_ip = 0x0,
auth_token = 0x0,
var_expand_table = 0x7fed9b829e40,
error = 0x0,
set_info = 0x7fed9b814d60,
unexpanded_set = 0x7fed9b829138,
set = 0x7fed9b829770,
namespaces = 0x7fed9b4046b0,
storages = 0x7fed9b404780,
hooks = {
arr = {
buffer = 0x7fed9b82a130,
element_size = 8
},
v = 0x7fed9b82a130,
v_modifiable = 0x7fed9b82a130
},
mountpoints = 0x0,
default_normalizer = 0x10b0c1d00 uni_utf8_to_decomposed_titlecase,
_attr_dict = 0x0,
module_contexts = {
arr = {
buffer = 0x7fed9b829da8,
element_size = 8
},
v = 0x7fed9b829da8,
v_modifiable = 0x7fed9b829da8
},
nonexistent = 0,
home_looked_up = 1,
anonymous = 0,
autocreated = 0,
initialized = 1,
mail_debug = 1,
inbox_open_error_logged = 0,
fuzzy_search = 0,
dsyncing = 0,
attr_dict_failed = 0
}
Re: [Dovecot] 2.2.4 - Some questions about and needing help with quota-status
Hello, I ended my previous message with : [...] Q3. What am I doing wrong? [...] Given the details discussed in another thread (http://www.dovecot.org/list/dovecot/2013-July/091309.html), I tried by changing the user_query's SELECT from: user_query = \ SELECT DISTINCT \ nickname AS user, \ mail_home AS home, \ mail_location AS mail, \ quota_rule AS quota_rule \ FROM \ [...] to: user_query = \ SELECT DISTINCT \ nickname AS user, \ coalesce(mail_home, '/_Mailstores/' || nickname) AS home, \ mail_location AS mail, \ 'dict:Quota utilisateur:' || nickname || ':proxy::sql_quota' AS quota, \ quota_rule AS quota_rule \ FROM \ [...] while keeping dovecot.conf unchanged (see my previous message). The idea is to not rely anymore on the expansion of %u (or %n or %d) in dovecot.conf, while still keeping the ability to have per-user settings. Currently, a doveadm quota get -u john.doe yields: Quota nameTypeValue Limit % Quota utilisateur STORAGE3134 91 Quota utilisateur MESSAGE23 - 0 So, let's ask quota-status what it believes about a message with a size of 10. The reply is still action=OK, the dict server still isn't launched, and the log shows: auth: Debug: userdb out: USER 1 john@example.com home=/_Mailstores/john.doe quota=dict:Quota utilisateur:john.doe:proxy::sql_quota quota_rule=*:storage=35000b quota-status: Debug: auth input: john@example.com home=/_Mailstores/john.doe quota=dict:Quota utilisateur:john.doe:proxy::sql_quota quota_rule=*:storage=35000b quota-status: Debug: Added userdb setting: plugin/quota=dict:Quota utilisateur:john.doe:proxy::sql_quota quota-status: Debug: Added userdb setting: plugin/quota_rule=*:storage=35000b quota-status(john@example.com): Debug: Effective uid=999, gid=999, home=/_Mailstores/john.doe quota-status(john@example.com): Debug: Quota root: name=Quota utilisateur backend=dict args=john.doe:proxy::sql_quota quota-status(john@example.com): Debug: Quota rule: root=Quota utilisateur mailbox=* bytes=35000 messages=0 quota-status(john@example.com): Debug: Quota grace: root=Quota utilisateur bytes=3500 (10%) quota-status(john@example.com): Debug: dict quota: user=john.doe, uri=proxy::sql_quota, noenforcing=0 quota-status(john@example.com): Debug: fs: root=/_Mailstores/john.doe/mboxes, index=, indexpvt=, control=, inbox=/_Mailstores/john.doe/mboxes/inbox, alt= To rule out any other side-effects potentially introduced by the user_query, I even tried with the nickname AS user removed from the SELECT. Even with that, the reply is action=OK, the dict server still isn't launched, and the lines written to the log are undistinguishable from above ones... It is to be noted that no lines in the log are related to possible problems encountered for launching it. It is a bit as if quota_check() in src/plugins/quota/quota-status.c always immediately returned with 1 at the first test. Anyway, I'm still stuck. And still very interested in replies to Q1, Q2 and Q3. ;-) TIA, Axel
[Dovecot] 2.2.4 - Some questions about and needing help with quota-status
Help! I'm stuck. :-(
The config of my experimental setup appears at the end of this message; I'm
providing hereafter some more info that may not be immediately obvious.
This is dovecot 2.2.4 with changesets 9091d0f2d971 and 2be295a0b64f.
All involved databases are sqlite ones.
passdb and userdb are devised so as to change usernames.
For example, I could have a user with addresses d...@oldexample.com,
jo...@oldexample.com and john@example.com needing to be able to log in
as jdoe or u123456; all db lookups for that user then end with name
john.doe.
This is a single mail user setup (user/group dovemailer, uid/gid 999).
Service lmtp has been configured to run as that user; that required a slight
adjustment at the auth-userdb socket level.
As a general rule, the quota dict appears to be correctly updated upon message
arrivals and removals (thru lmtp, pop, imap), and to be correctly
queried/interpreted by the various parts of the server.
For example, thru the userdb query, user john.doe has been given an even lower
quota limit than the already low default defined for testings:
$ sudo doveadm quota get -u john.doe
Quota nameTypeValue Limit %
Quota utilisateur STORAGE20 5 400
Quota utilisateur MESSAGE14 - 0
and is clearly recognized as being over-quota by lmtp:
$ telnet /_ROOT/var/run/dovecot/lmtp
Trying /_ROOT/var/run/dovecot/lmtp...
Connected to (null).
Escape character is '^]'.
220 almba.local Dovecot ready.
mail from:mary.j...@example.com
250 2.1.0 OK
rcpt to:john@example.com
250 2.1.5 OK
data
354 OK
Subject: test
.
552 5.2.2 john@example.com Quota exceeded (mailbox for user is
full)
Note that both services config and dict are launched if they aren't running.
Since it has been previously seen that running quota-status as root comes with
its own problems, and since it is a single user setup anyway, I'm trying to run
it as dovemailer as well.
So, let's try to see what quota-status thinks about john.doe:
$ sudo -u _postfix telnet /_ROOT/var/spool/postfix/private/quota-policyd
Trying /_ROOT/var/spool/postfix/private/quota-policyd...
Connected to (null).
Escape character is '^]'.
Connection closed by foreign host.
Clearly, not much...
Looking in the log:
dovecot[10554]: quota-status: Fatal: Error reading configuration:
net_connect_unix(/_ROOT/var/run/dovecot/config) failed: Permission denied
This thus raises a first question:
Q1. What makes lmtp and quota-status different? How does lmtp manage to fetch
all needed info, while quota-status seems to require an access to the config
socket?
Let's then slightly adjust dovecot.conf, in the hope to make quota-status happy:
service config {
unix_listener config {
group = dovemailer
mode = 0660
}
}
Q2. Should the above really be needed, wouldn't there be a better way?
I ask, because it seems to me that I'm starting to seriously lose the benefits
of privilege separation...
Anyway, let's ask quota-status again:
$ sudo -u _postfix telnet /_ROOT/var/spool/postfix/private/quota-policyd
Password:
Trying /_ROOT/var/spool/postfix/private/quota-policyd...
Connected to (null).
Escape character is '^]'.
recipient=john@example.com
size=1
action=OK
^]
telnet quit
Connection closed.
In the log:
dovecot[11050]: auth: Debug: userdb out: USER 1
john@example.comquota_rule=*:storage=5k
dovecot[11050]: quota-status: Debug: auth input: john@example.com
quota_rule=*:storage=5k
dovecot[11050]: quota-status: Debug: Added userdb setting:
plugin/quota_rule=*:storage=5k
dovecot[11050]: quota-status(john@example.com): Debug: Effective
uid=999, gid=999, home=/_Mailstores/john.doe
dovecot[11050]: quota-status(john@example.com): Debug: Quota root:
name=Quota utilisateur backend=dict args=john.doe:proxy::sql_quota
dovecot[11050]: quota-status(john@example.com): Debug: Quota rule:
root=Quota utilisateur mailbox=* bytes=5120 messages=0
dovecot[11050]: quota-status(john@example.com): Debug: Quota grace:
root=Quota utilisateur bytes=512 (10%)
dovecot[11050]: quota-status(john@example.com): Debug: dict quota:
user=john.doe, uri=proxy::sql_quota, noenforcing=0
dovecot[11050]: quota-status(john@example.com): Debug: fs:
root=/_Mailstores/john.doe/mboxes, index=, indexpvt=, control=,
inbox=/_Mailstores/john.doe/mboxes/inbox, alt=
It is to be noted that the config server is now launched as expected, but that
the dict server still isn't.
Trying a dirsize backend instead
Re: [Dovecot] some questions on AOX or rather a mail system setup (ignore)
Oops... that was obivously not intended for dovecot but AOX mailing list,... where I ask around similar questions. Sorry for the noise =) Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
[Dovecot] some questions on AOX or rather a mail system setup
Hi folks. Perhaps you find some time to look into this,... I'm trying to plan my mail system and would have some questions. The overall idea is about the following: - There is a (internet) server, which is the MTA (which will be postfix) and imap and/or pop3 server (which shall be, guess, dovecot ... AOX comes in later). - Any spam filtering, virus-scanning, mail filtering happens on that server. - I have not yet decided on whether to use maildrop for this or Sieve. Maildrop is nice, but has one major deficiency which I don't know how to work around. - A few clients (that means I won't serve 100 of users) connect to that server via imap and should see all mails, etc. already in some fancy sorted hierarchy (that means filtered into different directories). So far nothing complicated. But now... - Apart from spam, I never delete mail; and because I'm subscribed to many lists, I get a lot of mail. - Storage on my server is limited and it's located somewhere at my ISP, so I generally do not trust it with respect to safety... For both reasons, I want the canonical archive of all mail to be at home at some local server. - The local server should also be an imap server, so that I can access the archive from may computers at home. - The local server won't be available from the internet. - The local archive should have the same folder hierarchy as the internet server (I'd prefer not to filter twice). - The local archive might be AOX... Mail Flow: - I generally want to have _all_ mail (which is not sorted out because of being spam) to be archived at the local server. - But(!) I want to selectively keep (in addition) mail at the internet server. For example I may want to select the folder that contains all mail form some friend to be kept online completely. But I may want to decide that mailinglists keep only the last 10 days and/or 1000 messages of mail. - The idea is, that the local server regularly (when it is online/running) catches new mail from the internet server... and stores it in the archive. - So apart from new mail that has not yet been read, that local archive contains always all mails that are also on the internet server... the later may contain (for specific directories) the same, or just parts of. - The MUAs will then have two imap accounts, one to the internet server and one to the local archive,... each one being usable, depending on where I am. 1) This is where my first problem arises: How can I implement that mail flow, especially: - How do I secure that all mail is read from the internet server (i.e. that nothing is forgotten? - How do I make sure that no mails are retrieved twice (or more)? A problem which I often had with pop, when the mail client crashed during sync? - Further it must be secured, that when I delete something on the internet server, it is NOT deleted on the local server (on the next mail-fetching) this is why I don't use the word sync. a) One stupid solution would be, that I duplicate all mail on the online server,... one part is for staying online, one part is for being fetched to the local archive. As soon as it was fetched... that copy gets removed (always). That solution would give a clean and secured separation of both? b) I don't think offlineimap or any other caching-like solution is the right thing... especially as one must always fear that such a cache may be accidentally wiped. Are there better solutions than (a)? 2) Problem would be already a refinement of a working solution for (1) (but obviously not when using (1).(a) ). When e.g. reply to or forward a mail using the online server,... and that mail had already been fetched,... can I make the flag synced? 3) Is AOX suitable for the local server? - I couldn't use maildir locally, because I loose just to much space to the block fragmentation. But I guess I won't have this problem with the DB backend. a) Is AOX able to manage a really big folder hierarchy that basically ever keeps growing... with easily several 100k mails per folder... and that is in total already over 100GB? I read that e.g. dovecot would have more performance problems with that. b) I would prefer to have fast full text search. Does AOX provide this? I read that IMAP has limitations which make full text search not really usable via it? Further, I remember some dovecot wiki page that showed a comparison which said that both do not perfectly implement imap. Thanks a lot, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] some questions on AOX or rather a mail system setup
Am 11.10.2012 04:10, schrieb Christoph Anton Mitterer: 3) Is AOX suitable for the local server? - I couldn't use maildir locally, because I loose just to much space to the block fragmentation. But I guess I won't have this problem with the DB backend. a) Is AOX able to manage a really big folder hierarchy that basically ever keeps growing... with easily several 100k mails per folder... and that is in total already over 100GB? I read that e.g. dovecot would have more performance problems with that. b) I would prefer to have fast full text search. Does AOX provide this? I read that IMAP has limitations which make full text search not really usable via it? Further, I remember some dovecot wiki page that showed a comparison which said that both do not perfectly implement imap. Christoph, sorry, what exact is AOX, and what is its relation to the dovecot list youre looking to an overall mail setup so split your questions up to software you wanna use and ask the related mail list, perhaps hire some mail consultant advice you -- Best Regards MfG Robert Schetterer
Re: [Dovecot] some questions on AOX or rather a mail system setup (ignore)
Am 11.10.2012 04:18, schrieb Christoph Anton Mitterer: Oops... that was obivously not intended for dovecot but AOX mailing list,... where I ask around similar questions. Sorry for the noise =) Cheers, Chris. ups answered exact about this *g -- Best Regards MfG Robert Schetterer
Re: [Dovecot] some questions on AOX or rather a mail system setup
Am 11.10.2012 14:56, schrieb Robert Schetterer: Am 11.10.2012 04:10, schrieb Christoph Anton Mitterer: 3) Is AOX suitable for the local server? [..] Christoph, sorry, what exact is AOX, and what is its relation to the dovecot list I suppose he meant Archiveopteryx (another IMAP server). -- Frerich Raabe - ra...@froglogic.com www.froglogic.com - Multi-Platform GUI Testing
Re: [Dovecot] some questions on AOX or rather a mail system setup
Hi. Sorry folks for the stupid postings... At first I posted what should go to the AOX list accidentally here and then our institute's MTA hat some issues yesterday, so the mail[0] where I already tried to explain the wrong posting, came much earlier than the wrong post itself. Guess you see why I need a better mail system ;) Sorry, Chris. [0] http://dovecot.org/pipermail/dovecot/2012-October/068740.html smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] some questions on dovecot or rather a mail system setup
Am 08.10.2012 23:37, schrieb Christoph Anton Mitterer: - Apart from spam, I never delete mail; and because I'm subscribed to many lists, I get a lot of mail. - Storage on my server is limited and it's located somewhere at my ISP, so I generally do not trust it with respect to safety... For both reasons, I want the canonical archive of all mail to be at home at some local server. sorry you questioned very complex, try to ask more simple there are many tools which may help you bcc_copy with postfix imapsync rsync dsync getmail you may use filters too like sieve, maildrop , procmail etc at the end that should solve nearly all what you might goal its not that much a dovecot question, it more depends if you find that general layout which fits best to your ideas however there is no magical imap/pop3 server more flexibel to configure then dovecot, if your ideas dont work with it, your ideas are broken -- Best Regards MfG Robert Schetterer
Re: [Dovecot] some questions on dovecot or rather a mail system setup
On 10/8/2012 4:37 PM, Christoph Anton Mitterer wrote: The proper way to accomplish your goals, or at least the big ones. - I generally want to have _all_ mail (which is not sorted out because of being spam) to be archived at the local server. http://www.postfix.org/postconf.5.html#always_bcc - But(!) I want to selectively keep (in addition) mail at the internet server. For example I may want to select the folder that contains all mail form some friend to be kept online completely. See above. But I may want to decide that mailinglists keep only the last 10 days and/or 1000 messages of mail. http://wiki2.dovecot.org/Plugins/Expire Does age based deletion, but not folder message count based. You must use your MUA, TBird, for the latter. It's far easier to configure this in TBird than in Dovecot config files. You seem like the type who wants flexibility so you can change things often, so use TBird to be happy here. - The idea is, that the local server regularly (when it is online/running) catches new mail from the internet server... and stores it in the archive. This is not an option. The system must be up and connected to the internet 24x7x365. It must have an MX record associated and a valid domain, or a VPN tunnel and entries in both systems hosts files, along with a Postfix transport table, and other tweaks. http://www.postfix.org/transport.5.html If you refuse to run this local server 24x7x365 then you will have to use a fetchmail based solution, which will not work well, and whose configuration will prompt you to kill yourself. I cannot help you with any of that. - So apart from new mail that has not yet been read, that local archive contains always all mails that are also on the internet server... the later may contain (for specific directories) the same, or just parts of. No. Mail arriving at the colo/VPS host is immediately sent to the always_bcc address, an address and mailbox on your home server. You will create a duplicate IMAP folder structure on the home server by hand in your MUA. Once this is completed you will write individual user sieve scripts that sort the mail into folder just as it is sorted on the colo/VPS server. Basically, home server Dovecot IMAP config is identical in structure to colo/VPS setup, only the mailbox account names differ. Folder tree, folders, sieve scripts identical, retention policy different. - The MUAs will then have two imap accounts, one to the internet server and one to the local archive,... each one being usable, depending on where I am. Yep. 1) This is where my first problem arises: How can I implement that mail flow, especially: - How do I secure that all mail is read from the internet server (i.e. that nothing is forgotten? Done: always_bcc - How do I make sure that no mails are retrieved twice (or more)? A problem which I often had with pop, when the mail client crashed during sync? Done: always_bcc - Further it must be secured, that when I delete something on the internet server, it is NOT deleted on the local server (on the next mail-fetching) this is why I don't use the word sync. Done: always_bcc a) One stupid solution would be, that I duplicate all mail on the online server,... one part is for staying online, one part is for being fetched to the local archive. Done: always_bcc And yes that is stupid. As soon as it was fetched... that copy gets removed (always). That solution would give a clean and secured separation of both? b) I don't think offlineimap or any other caching-like solution is the right thing... especially as one must always fear that such a cache may be accidentally wiped. Are there better solutions than (a)? Yes. Already done: always_bcc 2) Problem would be already a refinement of a working solution for (1) (but obviously not when using (1).(a) ). When e.g. reply to or forward a mail using the online server,... and that mail had already been fetched,... can I make the flag synced? No. Your stated goal is that the local server is a mail archive put into service due to limited space on your colo/VPS server. An archive is an archive, not a secondary online server. It should only be accessed, read only, when you want to search and read an old message. And in fact, since this is an archive, you should implement the zlib plugin with dbox so all this archived mail is compressed in real time. Make up your mind. You can't have it both ways. I hear the iPhone5 can do anything automatically, no setup. Get one of those, problem solved. ;) 3) Is dovecot suitable for the local server? Yes. Probably more than any other IMAP server. - I couldn't sue maildir locally, because I loose just to much space to the block fragmentation. Maildir causes the least filesystem fragmentation. You must be thinking of mbox, which causes heavy fragmentation due to constant appends past EOF. As I said you need dbox. One email per file, similar to maildir, but
Re: [Dovecot] some questions on dovecot or rather a mail system setup
On 10/9/2012 2:57 AM, Stan Hoeppner wrote: http://www.postfix.org/postconf.5.html#always_bcc Correction. In your case you'll need to use: http://www.postfix.org/postconf.5.html#recipient_bcc_maps Because you said you only want to archive email for some users, not simply all mail received by the colo/VPS server. -- Stan
[Dovecot] some questions on dovecot or rather a mail system setup
Hi folks. Perhaps you find some time to look into this,... if you think I should better direct this do some dovecot mailing list,.. just tell :) I'm trying to plan my mail system and would have some questions. The overall idea is about the following: - There is a (internet) server, which is the MTA (which will be postfix) and imap and/or pop3 server (which shall be, guess, dovecot!). - Any spam filtering, virus-scanning, mail filtering happens on that server. - I have not yet decided on whether to use maildrop for this or Sieve. Maildrop is nice, but has one major deficiency which I don't know how to work around. - A few clients (that means I won't serve 100 of users) connect to that server via imap and should see all mails, etc. already in some fancy sorted hierarchy (that means filtered into different directories). So far nothing complicated. But now... - Apart from spam, I never delete mail; and because I'm subscribed to many lists, I get a lot of mail. - Storage on my server is limited and it's located somewhere at my ISP, so I generally do not trust it with respect to safety... For both reasons, I want the canonical archive of all mail to be at home at some local server. - The local server should also be an imap server, so that I can access the archive from may computers at home. - The local server won't be available from the internet. - The local archive should have the same folder hierarchy as the internet server (I'd prefer not to filter twice). Mail Flow: - I generally want to have _all_ mail (which is not sorted out because of being spam) to be archived at the local server. - But(!) I want to selectively keep (in addition) mail at the internet server. For example I may want to select the folder that contains all mail form some friend to be kept online completely. But I may want to decide that mailinglists keep only the last 10 days and/or 1000 messages of mail. - The idea is, that the local server regularly (when it is online/running) catches new mail from the internet server... and stores it in the archive. - So apart from new mail that has not yet been read, that local archive contains always all mails that are also on the internet server... the later may contain (for specific directories) the same, or just parts of. - The MUAs will then have two imap accounts, one to the internet server and one to the local archive,... each one being usable, depending on where I am. 1) This is where my first problem arises: How can I implement that mail flow, especially: - How do I secure that all mail is read from the internet server (i.e. that nothing is forgotten? - How do I make sure that no mails are retrieved twice (or more)? A problem which I often had with pop, when the mail client crashed during sync? - Further it must be secured, that when I delete something on the internet server, it is NOT deleted on the local server (on the next mail-fetching) this is why I don't use the word sync. a) One stupid solution would be, that I duplicate all mail on the online server,... one part is for staying online, one part is for being fetched to the local archive. As soon as it was fetched... that copy gets removed (always). That solution would give a clean and secured separation of both? b) I don't think offlineimap or any other caching-like solution is the right thing... especially as one must always fear that such a cache may be accidentally wiped. Are there better solutions than (a)? 2) Problem would be already a refinement of a working solution for (1) (but obviously not when using (1).(a) ). When e.g. reply to or forward a mail using the online server,... and that mail had already been fetched,... can I make the flag synced? 3) Is dovecot suitable for the local server? - Ideally of course, I would use dovecot there, too... because that would mean one piece of software less to understand. - I couldn't sue maildir locally, because I loose just to much space to the block fragmentation. - I'd prefer not to use dbox (the thing that the indices are crucial scares me a bit off). a) When using mbox... is dovecot able to manage a really big folder hierarchy that basically ever keeps growing... with easily several 100k mails per folder... and that is in total already over 100GB? - I would prefer to have fast full text search. Does dovecot provide this? I was looking into database backed mail systems (again,... just for the local archive)... namely dbmail and archiveopteryx (are there other open source solutions?)... Not sure which of the two... or whether it's a good idea at all. I remember some dovecot wiki page that showed a comparison which said that both do not perfectly implement imap. Any suggestions with respect to that? Or is there even some SQL backend planned for dovecot? Thanks a lot, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] Some questions about mdbox
Hello everybody I have found that with bzip compression level 6 I have a better balance between compression and decompression speed with dovecot+mdbox, but still too slow compared with zlib. Does anybody have any other hint about my original questions (see below) Regards Maria Arrea - Original Message - From: Maria Arrea Sent: 02/15/11 09:16 PM To: dovecot@dovecot.org Subject: [Dovecot] Some questions about mdbox Hello I am starting testing with dovecot 2.0.9. In my //etc/dovecot/dovecot.conf/ I have the following line: /mail_location = mdbox:/mydomain.org/%2.26Hn/%2.200Hn/%n:INDEX=/mydomain.org/indexes/%2.26Hn/%2.200Hn/%n/ These are my questions (a long mail): *1º How should I restore from backup with mdbox separate indexes?* I have read in dovecot list archive about restoring the full user mailbox to a temp location and then run something like: /doveadm import mdbox:/tmp/restored_mailbox/ savedsince 2011-01-01 / In my schema indexes and mail stores are in separated paths. What (and where!) should I copy to the temp location and how do I restore it? *2º Disaster recovery: indexes mail store mismatch* When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent? What would happens if I lose indexes? Can I still recover the messages? Reading the doc it seems I would lose all the message flags and all the messages would be in INBOX, is this right? *3º Performance tuning with mdbox* I have read in dovecot archive about using mdbox_rotate_size mdbox_rotate_interval to balance between a lot of smaill files (like Maildir) or really big files (a variant of mbox). I have tested with a big vaule of mdbox_rotate_size (100 MB) and is a complete disaster for backup:if a user delete a message, then a nightly cronjob purges the mailbox (refcount=0) a new mdbox .m file is created!. So deleting a 1 k message from the mailbox of an user is a new 99 MB file to backup. Any hint about this? I am also using ext4 (rw,noatime,nodiratime,commit=15,data=writeback) and I have read about mdbox_preallocate_space. I will try it, any hint about this setup? This is not still in production, so I have no idea about IOPs and user concurrency, but I have several thousand users. *4º Compression with mdbox* I am reading about compression in mdbox (zlib/bzlib). I am star ting with zlib, has anyone experience with bzlib? How much load is bzlib against zlib and how much delay adds to user experience (and space savings vs zlib???) ? *5º Recover a message with refcount=0* Is there any way to recover a message with refcount=0 without using a backup? I am thinking about the following plan: /1º a user gets her mail via pop3 (or delete a message via imap and empties trash) 2º the message is still stored in my mail server, but has refcount=0 3º nightly backup is performed 4º nightly cronjob purges the mailbox (deleting messaged with refcount=0)/ Using this schema the backup is bigger, but If I can recover files with refcount=0 from backup we can give our users a safeguard against accidentaly deleted pop3 mail. Thank you very much for your responsed and sorry for my (awful) english! Maria
Re: [Dovecot] Some questions about mdbox
On 15.2.2011, at 22.16, Maria Arrea wrote: *1º How should I restore from backup with mdbox separate indexes?* I have read in dovecot list archive about restoring the full user mailbox to a temp location and then run something like: /doveadm import mdbox:/tmp/restored_mailbox/ savedsince 2011-01-01 / In my schema indexes and mail stores are in separated paths. What (and where!) should I copy to the temp location and how do I restore it? The import parameter works exactly the same as mail_location. So if you for example restore user's files to: /tmp/backup/mdbox/storage/m.* /tmp/backup/indexes/dovecot.index* You can use doveadm import mdbox:/tmp/backup/mdbox:INDEX=/tmp/backup/indexes. *2º Disaster recovery: indexes mail store mismatch* When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent? Well, this isn't ideal.. Better would probably be if indexes were backed up after. Now it's possible that when restoring a mail there are some mails that aren't in index yet. You could run doveadm force-resync to make sure they're all added. What would happens if I lose indexes? Can I still recover the messages? Reading the doc it seems I would lose all the message flags and all the messages would be in INBOX, is this right? doveadm force-resync makes all messages visible, even if all flags are lost. The messages will show up where they were saved to the first time (INBOX and Sent mainly, also if Sieve fileinto was used then those too). So mainly if user/client copied/moved messages around, those changes would be lost. Anyway, since you have backups and Dovecot also generates dovecot.index.backup files, you should always have some versions of them available. And mdbox index rebuilding tries really hard to use those files to avoid losing changes. *3º Performance tuning with mdbox* I have read in dovecot archive about using mdbox_rotate_size mdbox_rotate_interval to balance between a lot of smaill files (like Maildir) or really big files (a variant of mbox). I have tested with a big vaule of mdbox_rotate_size (100 MB) and is a complete disaster for backup:if a user delete a message, then a nightly cronjob purges the mailbox (refcount=0) a new mdbox .m file is created!. So deleting a 1 k message from the mailbox of an user is a new 99 MB file to backup. Any hint about this? Well, that's how it is intended to work.. Typically users would delete only recent mails, so if you rotate files daily, then deletions would happen only from the file that hasn't yet even been backed up. Of course, with daily rotations you rarely get even close to 100 MB files. *5º Recover a message with refcount=0* Is there any way to recover a message with refcount=0 without using a backup? I am thinking about the following plan: /1º a user gets her mail via pop3 (or delete a message via imap and empties trash) 2º the message is still stored in my mail server, but has refcount=0 3º nightly backup is performed 4º nightly cronjob purges the mailbox (deleting messaged with refcount=0)/ Using this schema the backup is bigger, but If I can recover files with refcount=0 from backup we can give our users a safeguard against accidentaly deleted pop3 mail. There is no clean way to do this currently. An ugly way would be to delete dovecot.map.index* files and rebuild index. The rebuilding sees that there are some mails that don't exist in any mailbox indexes, and assigns them to the mailbox where they initially were saved to. No (flag) changes are lost by doing this.
Re: [Dovecot] Some questions about mdbox
On 2011-02-16 11:42 AM, Timo Sirainen wrote: On 15.2.2011, at 22.16, Maria Arrea wrote: *2º Disaster recovery: indexes mail store mismatch* When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent? Well, this isn't ideal.. Better would probably be if indexes were backed up after. Now it's possible that when restoring a mail there are some mails that aren't in index yet. You could run doveadm force-resync to make sure they're all added. Best would be to store your mail on an LVM partition (or something else) that supports snapshots, and take a snapshot, back it up, then release it. -- Best regards, Charles
[Dovecot] Some questions about mdbox
Hello I am starting testing with dovecot 2.0.9. In my //etc/dovecot/dovecot.conf/ I have the following line: /mail_location = mdbox:/mydomain.org/%2.26Hn/%2.200Hn/%n:INDEX=/mydomain.org/indexes/%2.26Hn/%2.200Hn/%n/ These are my questions (a long mail): *1º How should I restore from backup with mdbox separate indexes?* I have read in dovecot list archive about restoring the full user mailbox to a temp location and then run something like: /doveadm import mdbox:/tmp/restored_mailbox/ savedsince 2011-01-01 / In my schema indexes and mail stores are in separated paths. What (and where!) should I copy to the temp location and how do I restore it? *2º Disaster recovery: indexes mail store mismatch* When creating backups, indexes and mail store are in different paths. My backup takes several hours, so indexes are copied BEFORE mail stores. Is the backup consistent? What would happens if I lose indexes? Can I still recover the messages? Reading the doc it seems I would lose all the message flags and all the messages would be in INBOX, is this right? *3º Performance tuning with mdbox* I have read in dovecot archive about using mdbox_rotate_size mdbox_rotate_interval to balance between a lot of smaill files (like Maildir) or really big files (a variant of mbox). I have tested with a big vaule of mdbox_rotate_size (100 MB) and is a complete disaster for backup:if a user delete a message, then a nightly cronjob purges the mailbox (refcount=0) a new mdbox .m file is created!. So deleting a 1 k message from the mailbox of an user is a new 99 MB file to backup. Any hint about this? I am also using ext4 (rw,noatime,nodiratime,commit=15,data=writeback) and I have read about mdbox_preallocate_space. I will try it, any hint about this setup? This is not still in production, so I have no idea about IOPs and user concurrency, but I have several thousand users. *4º Compression with mdbox* I am reading about compression in mdbox (zlib/bzlib). I am starting with zlib, has anyone experience with bzlib? How much load is bzlib against zlib and how much delay adds to user experience (and space savings vs zlib???) ? *5º Recover a message with refcount=0* Is there any way to recover a message with refcount=0 without using a backup? I am thinking about the following plan: /1º a user gets her mail via pop3 (or delete a message via imap and empties trash) 2º the message is still stored in my mail server, but has refcount=0 3º nightly backup is performed 4º nightly cronjob purges the mailbox (deleting messaged with refcount=0)/ Using this schema the backup is bigger, but If I can recover files with refcount=0 from backup we can give our users a safeguard against accidentaly deleted pop3 mail. Thank you very much for your responsed and sorry for my (awful) english! Maria
Re: [Dovecot] Some questions about Shared mailboxes
On Fri, 2010-08-27 at 09:58 +0300, Mihajlin Evgenij wrote: Hi, again. 1. Can somebody explain me about dovecot's shared mailboxes? What specifically is unclear in http://wiki2.dovecot.org/SharedMailboxes/Shared that could be improved? 2. What does it do while searching shared mailboxes (debug_log cannot explain it)? What files is it looking for? User's mail files.. I don't really know what to answer to this. 3. Need i set dovecot-shared file into directory which are shared (if i want to have separate flags for separate users)? No. 4. Some errors are hard to understand. For example, # pwd /var/spool/vmail/domains/badmltd.dn.ua # ls -l drwx-- 3 mailnull mail 4096 Авг 27 09:24 admin drwx-- 3 mailnull mail 4096 Июн 3 10:51 exim drwx-- 3 mailnull mail 4096 Мар 26 15:08 iif drwx-w 3 mailnull mail 4096 Мар 11 15:53 jack drwx-- 2 mailnull mail 4096 Авг 27 09:28 Maildir drwx-w 3 mailnull mail 4096 Авг 27 09:25 test Why did dovecot create Maildir folder??? It shouldn't have done it. Delete it. If you can reproduce it, let me know what IMAP command did that.
[Dovecot] Some questions about Shared mailboxes
Hi, again. 1. Can somebody explain me about dovecot's shared mailboxes? 2. What does it do while searching shared mailboxes (debug_log cannot explain it)? What files is it looking for? 3. Need i set dovecot-shared file into directory which are shared (if i want to have separate flags for separate users)? 4. Some errors are hard to understand. For example, # pwd /var/spool/vmail/domains/badmltd.dn.ua # ls -l drwx-- 3 mailnull mail 4096 Авг 27 09:24 admin drwx-- 3 mailnull mail 4096 Июн 3 10:51 exim drwx-- 3 mailnull mail 4096 Мар 26 15:08 iif drwx-w 3 mailnull mail 4096 Мар 11 15:53 jack drwx-- 2 mailnull mail 4096 Авг 27 09:28 Maildir drwx-w 3 mailnull mail 4096 Авг 27 09:25 test Why did dovecot create Maildir folder??? And what mean this lines at the end of debug.log. Aug 27 09:28:42 imap(za...@badmltd.dn.ua): Debug: Namespace : type=shared, prefix=shared/%n/, sep=/, inbox=no, hidden=no, list=children, subscriptions=no Aug 27 09:28:42 imap(za...@badmltd.dn.ua): Debug: shared: root=/var/run/dovecot, index=, control=, inbox= Aug 27 09:28:42 imap(za...@badmltd.dn.ua): Debug: acl: initializing backend with data: vfile Aug 27 09:28:42 imap(za...@badmltd.dn.ua): Debug: acl: acl username = za...@badmltd.dn.ua Aug 27 09:28:42 imap(za...@badmltd.dn.ua): Debug: acl: owner = 0 Aug 27 09:28:42 imap(za...@badmltd.dn.ua): Debug: acl vfile: Global ACL directory: (null) Aug 27 09:28:46 imap(za...@badmltd.dn.ua): Debug: Namespace : Using permissions from /var/spool/vmail/domains/badmltd.dn.ua/zakaz/Maildir: mode=0700 gid=-1 Aug 27 09:28:46 imap(za...@badmltd.dn.ua): Debug: acl vfile: file /var/spool/vmail/domains/badmltd.dn.ua/zakaz/Maildir/dovecot-acl not found Aug 27 09:28:46 imap(za...@badmltd.dn.ua): Debug: maildir++: root=/var/spool/vmail/domains/badmltd.dn.ua//Maildir, index=/var/spool/vmail/domains/badmltd.dn.ua/zakaz/shared/@badmltd.dn.ua, control=, inbox=/var/spool/vmail/domains/badmltd.dn.ua//Maildir Aug 27 09:28:46 imap(za...@badmltd.dn.ua): Debug: Namespace shared//: Permission lookup failed from /var/spool/vmail/domains/badmltd.dn.ua//Maildir Aug 27 09:28:46 imap(za...@badmltd.dn.ua): Debug: Namespace shared//: Using permissions from /var/spool/vmail/domains/badmltd.dn.ua//Maildir: mode=0700 gid=-1 Thanks.
Re: [Dovecot] Newbee, some questions
On 11/22/2009 12:39 PM, Spyros Tsiolis wrote: Due to reasons beyond me (mainly my clients demanding more for their buck / things like webmail etc.), I was forced to start searching for (always) open source alternatives. We used Postfix only for a long time (SMTP/POP3), back in '07 I started researching, built a test rig in early '08 and we switched full over to Postfix/Dovecot in late spring of '08. You'll have a lot of reading ahead of you and I recommend registering a domain or two to use as a test bed on the new system before you start adding the real domains and repointing MX records at the new box. 1. Do multiple domain handling. This can be done with virtual domains and users. We found it easier to go with virtual users instead of system users for our multi-domain setup. http://wiki.dovecot.org/VirtualUsers http://wiki.dovecot.org/SystemUsers Personally, we use PostfixAdmin (a web-based tool) along with its databases (in PostgreSQL) to store our virtual domains and to manage domains/accounts. Our Postfix (which handles the SMTP side) and Dovecot (which handles the POP3/IMAP side) query this database for domain/user information. We used to use a system users setup, which had the advantage (and disadvantage) that Fred could receive email as f...@anyofourdomains without having to do anything special. With the virtual user setup, we had to put fred@ into one domain, and then setup aliases in the other domains that rewrote f...@otherdomain into f...@homedomain. In the long run, I'm happier, because most of our users really didn't need to be addressable as u...@anyofourdomains. 2. Have a centralized user base 3. Have a centralized mail repository for each user (like exchange, only without the admin/maintenance pain that comes with it) We store user email in Dovecot's Maildir setup, usually under: /var/vmail/domain/user/ There are lots of sub-folders below that point specific to the MailDir implementation. I personally have IMAP mailboxes with hundreds of thousands of messages spread across dozens of folders and a total size of over 2GB. 4. be able to do IMAP/POP3 and not SMTP/POP3 (or do I need all three of them ?) SMTP servers (postfix, sendmail) handle accepting mail from the outside world before handing it off to a LDA (local delivery agent) like Dovecot. The SMTP server also handles taking mail from a mail client (submitted via SMTP) and either handing it to the LDA for local delivery or contacting foreign SMTP servers to deliver to other domains. (Postfix also has an LDA component, and a POP3 component, but you can plug other LDA servers in like Dovecot.) POP3/IMAP access to the mailbox location is usually the job of Dovecot. 5. Interface with things like web-based mail software (LAMP ?) We use SquirrelMail here for our webmail. I'm pretty sure that it talks to the Dovecot IMAP server in order to access the user's mailbox. Once you have IMAP access to your mailboxes configured, you can use lots of different tools to talk to it.
Re: [Dovecot] Newbee, some questions
On Sun, Nov 22, 2009 at 01:55:22PM -0500, Thomas Harold wrote: We used Postfix only for a long time (SMTP/POP3), ... Um, no, Postfix does not serve POP3. -- Offlist mail to this address is discarded unless /dev/rob0 or not-spam is in Subject: header
Re: [Dovecot] Newbee, some questions
On 11/23/2009 2:12 PM, /dev/rob0 wrote: On Sun, Nov 22, 2009 at 01:55:22PM -0500, Thomas Harold wrote: We used Postfix only for a long time (SMTP/POP3), ... Um, no, Postfix does not serve POP3. Thanks for catching that. I wonder what the Solaris admin was using to serve up POP3 access to the mbox files? Maybe it was qpopper, that sounds familiar... (I'd have to dig back through my SSH session log files to know for sure.)
[Dovecot] Newbee, some questions
Hello list, OK, I just registered with the list, so please be tolerant :-) I've been using another MTA for about, oh, I'd say nine years now. Due to reasons beyond me (mainly my clients demanding more for their buck / things like webmail etc.), I was forced to start searching for (always) open source alternatives. The last MTA was only POP3/SMTP. It does no webmail. It interfaces with other software with a lot of pain and difficulty. I'll cut to the chase. I would like to start using some kind of software that can do the following : 1. Do multiple domain handling. 2. Have a centralized user base 3. Have a centralized mail repository for each user (like exchange, only without the admin/maintenance pain that comes with it) 4. be able to do IMAP/POP3 and not SMTP/POP3 (or do I need all three of them ?) 5. Interface with things like web-based mail software (LAMP ?) Am I in the right place ? I would certainly like to think so. However, the feedback belongs to you . Thank you kindly for your hospitality and patience, S. Tsiolis
Re: [Dovecot] Newbee, some questions
On 11/22/2009 12:39 PM, Spyros Tsiolis wrote: Hello list, OK, I just registered with the list, so please be tolerant :-) I've been using another MTA for about, oh, I'd say nine years now. Ummm... dovecot is not an MTA, it is a POP/IMAP server. I'd recommend the postfix list...
Re: [Dovecot] Newbee, some questions
On Sun, Nov 22, 2009 at 05:39:15PM +, Spyros Tsiolis wrote: OK, I just registered with the list, so please be tolerant :-) I've been using another MTA for about, oh, I'd say nine years now. I think we may have some terminology issues here. Dovecot is not a Mail Transfer Agent (MTA). It is a POP3/IMAP server and can also act as a Local Delivery Agent (LDA) for an MTA. It can also be used to provide authentication services to an MTA for SMTP authentication. Due to reasons beyond me (mainly my clients demanding more for their buck / things like webmail etc.), I was forced to start searching for (always) open source alternatives. The last MTA was only POP3/SMTP. It does no webmail. It interfaces with other software with a lot of pain and difficulty. Dovecot does not do webmail, but it can provide the IMAP access required by some web based User Agents (MUA) such as SquirrelMail http://squirrelmail.org/ or roundcube http://roundcube.net/. I'll cut to the chase. I would like to start using some kind of software that can do the following : 1. Do multiple domain handling. 2. Have a centralized user base 3. Have a centralized mail repository for each user (like exchange, only without the admin/maintenance pain that comes with it) 4. be able to do IMAP/POP3 and not SMTP/POP3 (or do I need all three of them ?) 5. Interface with things like web-based mail software (LAMP ?) Am I in the right place ? I would certainly like to think so. However, the feedback belongs to you . I think so. See http://wiki.dovecot.org/ for more info. -- Mark Sapiro mark at msapiro net The highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan
Re: [Dovecot] Newbee, some questions
Spyros Tsiolis pisze: Hello list, 1. Do multiple domain handling. 2. Have a centralized user base 3. Have a centralized mail repository for each user (like exchange, only without the admin/maintenance pain that comes with it) 4. be able to do IMAP/POP3 and not SMTP/POP3 (or do I need all three of them ?) 5. Interface with things like web-based mail software (LAMP ?) Am I in the right place ? I would certainly like to think so. However, the feedback belongs to you . Dovecot can do IMAP/POP3. If you want to achieve your points you need to put a lot of other things together. Better try something like Zimbra http://www.zimbra.com/community/ -- Tomasz
Re: [Dovecot] Some questions about deliver
Le 30 mai 09 à 00:04, I wrote : [...] I really believe that it would be worth to engrave that behavior somewhere in the docs. This could prove extremely useful to people considering to replace their existing LDA in their existing setup, by making explicit some points to take care of. As a side-effect, this might also act as a reminder of the behaviors that should not be altered in case of code rewriting. Last but not least, no more irritating questions. ;-) Just noticed that the info has been added to the wiki some days ago, under the Return values heading. :-) Many, many thanks Timo, Axel
Re: [Dovecot] Some questions about deliver
Le 28 mai 09 à 23:51, Timo Sirainen a écrit : On Tue, 2009-05-26 at 14:35 +0200, Axel Luttgens wrote: [...] That looks like a server configuration mistake. No, it's just a Postfix system quickly and dirtily brought to life for the sole purpose of testing deliver from within a MTA. ;-) On that occasion, I just took Postfix' default behaviors for illustrative purposes. But it could have been Sendmail, qmail, Exim... as well, or even a very esoteric homegrown system: any of those systems, especially the already operational and fine-tuned ones, have precise expectations on how their native LDA behaves. Replacing that LDA by another one is thus something far from benign and to be done with care, if possible with the help of as much info as possible about how the new LDA will behave. Hence my investigations, then my dumb questions about deliver: how does it behave, is that behavior configurable and if yes to what extent, have I missed something in the docs or in the source code, and so on. [...] At least, it would be nice to have a very precise description of how deliver is supposed to behave when facing various conditions. This would then be a documentation matter. Maybe.. Although it can be summarized pretty easily: - Invalid command line parameter gives EX_USAGE - Invalid configuration gives EX_CONFIG - User-over-quota optionally bounces or exits with EX_NOPERM with a mention about the -e option and the quota_full_tempfail setting - Anything else is EX_TEMPFAIL. Fine! I really believe that it would be worth to engrave that behavior somewhere in the docs. This could prove extremely useful to people considering to replace their existing LDA in their existing setup, by making explicit some points to take care of. As a side-effect, this might also act as a reminder of the behaviors that should not be altered in case of code rewriting. Last but not least, no more irritating questions. ;-) Thanks a lot, Axel
Re: [Dovecot] Some questions about deliver
On Tue, 2009-05-26 at 14:35 +0200, Axel Luttgens wrote: postfix/local[8643]: 1AFE4CA5D97: to=w...@almbp.local, relay=local, delay=0.08, delays=0.01/0.01/0/0.06, dsn=5.2.0, status=bounced (cannot update mailbox /Library/WebServer/_inbox/ mailspool for user www. unable to create lock file /Library/WebServer/ _inbox/mailspool.lock: No such file or directory) That looks like a server configuration mistake. So, Postfix' default behavior is to bounce the message; strictly speaking, deliver can't thus be viewed as a transparent substitute. If I was running a mail server, I'd prefer Postfix not to bounce the message in that situation. At least, it would be nice to have a very precise description of how deliver is supposed to behave when facing various conditions. This would then be a documentation matter. Maybe.. Although it can be summarized pretty easily: - Invalid command line parameter gives EX_USAGE - Invalid configuration gives EX_CONFIG - User-over-quota optionally bounces or exits with EX_NOPERM - Anything else is EX_TEMPFAIL. Anyway, let's consider this (stupid) one: mailbox_command = /usr/local/dovecot/libexec/dovecot/deliver -e -n -x Here, deliver immediately returns with EX_USAGE and, by default, Postix will reject the message; but this is something I could have noticed and fixed in the meantime. It could be argued that Postfix shouldn't be bouncing the message. :) EX_USAGE is meant exactly for that error, so I don't think it's a good idea for deliver to fail with EX_TEMPFAIL just to work around a Postfix issue. On the other hand, it could also be argued that an over-quota recipient is fixable as well: just pick the phone and ask the recipient to clean his mailbox. And one could thus conclude that deliver's default behavior isn't the right one... The difference is that over-quota can be caused by a user. Configuration mistakes can't be caused by a user. Dovecot in general uses this logic in writing errors. Users can't (well, shouldn't) ever cause Dovecot to log warnings/errors. And BTW, is that function guaranteed to be always called with an EX_* value? Seems to be the case, but... ;-) What do you mean? It's called with FATAL_* values and it replaces them with EX_TEMPFAIL. Sorry, some kind of mental short circuit occurred here; I meant: guaranteed to always return with status set to an EX_* value?. Pretty much, yes. I asked because of the default case in failure_exit_callback(), which just returns and leaves status as it was on entry. Conceptually, one could thus enter and leave the function with status set to a value differing from one of the FATAL_* and the EX_*, that value being ultimately returned to the caller. May this happen in practice? If yes, when and which values? The code looks like that to allow some future code change or plugin to exit with other values. Currently it never happens. But note that all the i_fatal_status(EX_USAGE, ..) etc. calls also go through that failure_exit_callback(). So it can't convert everything to EX_TEMPFAIL anyway, it would have to catch the used EX_* values at least. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Some questions about deliver
Le 26 mai 09 à 02:08, Timo Sirainen a écrit : On Sun, 2009-05-24 at 18:35 +0200, Axel Luttgens wrote: 1. Shouldn't deliver honor the first_valid_uid setting? I'm not sure. Somehow enforcing it there seems like a bad idea to me. Yes, could well be that I overlooked some possible side-effects. On the other hand, this would help to enforce the deliver+Dovecot pair consistency (if John Doe can't fetch his mails, should he receive mails?). Or help to enforce, even if inefficiently, some local policies not easily implemented otherwise. Someone else? ;-) 2. What exactly is the -e option supposed to do? .. Note that the question may somehow be re-phrased as: when invoked without the -e option, under which circumstances will deliver send a rejection message? Only when user is over quota. OK, this would thus be intended design. If you don't want messages to be delivered to some users, Postfix shouldn't call deliver for the user. I agree that waiting to have a message enqueued for rejecting it afterwards is somewhat inefficient. But... ;-) Let's first consider deliver as a replacement of Postfix' mailbox delivery, because of the nice benefits (indexing, headers sanitization, plugins...) it comes with. With Postfix' own mailbox delivery, i.e. mailbox_command = one gets with my previous example: postfix/smtpd[8639]: connect from localhost[127.0.0.1] postfix/smtpd[8639]: 1AFE4CA5D97: client=localhost[127.0.0.1] postfix/cleanup[8642]: 1AFE4CA5D97: message-id=029ee72b-b412-437f-a211-33c3597c8...@almbp.local postfix/qmgr[8637]: 1AFE4CA5D97: from=testu...@almbp.local, size=560, nrcpt=1 (queue active) postfix/local[8643]: 1AFE4CA5D97: to=w...@almbp.local, relay=local, delay=0.08, delays=0.01/0.01/0/0.06, dsn=5.2.0, status=bounced (cannot update mailbox /Library/WebServer/_inbox/ mailspool for user www. unable to create lock file /Library/WebServer/ _inbox/mailspool.lock: No such file or directory) postfix/cleanup[8642]: 2C455CA5D99: message-id=20090526075000.2c455ca5...@almbp.local postfix/qmgr[8637]: 2C455CA5D99: from=, size=2896, nrcpt=1 (queue active) postfix/bounce[8644]: 1AFE4CA5D97: sender non-delivery notification: 2C455CA5D99 postfix/local[8643]: 2C455CA5D99: to=testu...@almbp.local, relay=local, delay=0.01, delays=0/0/0/0, dsn=2.0.0, status=sent (delivered to mailbox) postfix/qmgr[8637]: 1AFE4CA5D97: removed postfix/qmgr[8637]: 2C455CA5D99: removed postfix/smtpd[8639]: disconnect from localhost[127.0.0.1] So, Postfix' default behavior is to bounce the message; strictly speaking, deliver can't thus be viewed as a transparent substitute. Now, let's consider the default behavior of Postfix when facing an over-quota recipient: postfix/smtpd[8977]: connect from localhost[127.0.0.1] postfix/smtpd[8977]: 91CEECA5FDF: client=localhost[127.0.0.1] postfix/cleanup[8980]: 91CEECA5FDF: message-id=b46cadd6-e42d-493c-ab04-46b0bff3f...@swing.be postfix/qmgr[8938]: 91CEECA5FDF: from=axelluttg...@swing.be, size=12634, nrcpt=1 (queue active) postfix/local[8981]: 91CEECA5FDF: to=testu...@almbp.local, relay=local, delay=0.04, delays=0.02/0.01/0/0.01, dsn=5.2.2, status=bounced (cannot update mailbox /Volumes/ALMbpSpare/People/a/ testuser/_inbox/mailspool for user testuser. error writing message: File too large) postfix/cleanup[8980]: 980EECA5FE1: message-id=20090526085344.980eeca5...@almbp.fusl.ac.be postfix/bounce[8982]: 91CEECA5FDF: sender non-delivery notification: 980EECA5FE1 postfix/qmgr[8938]: 91CEECA5FDF: removed postfix/qmgr[8938]: 980EECA5FE1: from=, size=2702, nrcpt=1 (queue active) postfix/smtp[8983]: 980EECA5FE1: to=axelluttg...@swing.be, relay=in.mx.skynet.be[195.238.5.129]:25, delay=2.8, delays=0.01/0.01/2.8/0.05, dsn=2.0.0, status=sent (250 ok: Message 446634039 accepted) postfix/qmgr[8938]: 980EECA5FE1: removed postfix/smtpd[8977]: disconnect from localhost[127.0.0.1] In this case, by default, Posfix adopts the same behavior as deliver. But I could have needed for some administrative reason to configure Postfix with, for example, soft_bounce = yes and then again face a problem when considering to make use of deliver as mailbox transport. More generally, since one of deliver's goal is to replace an MTA's local delivery agent, it would be nice to have some ways to fine tune deliver's behavior. That would allow to transparently integrate deliver into an existing MTA setup. Or even to augment the capabilities of that setup, for example by refining local policies. At least, it would be nice to have a very precise description of how deliver is supposed to behave when facing various conditions. This would then be a documentation matter. Of course, the source code always is the ultimate documentation :-) 3. Doesn't failure_exit_callback() in deliver.c tend to merge many (possibly dissimilar) errors
Re: [Dovecot] Some questions about deliver
On Sun, 2009-05-24 at 18:35 +0200, Axel Luttgens wrote: 1. Shouldn't deliver honor the first_valid_uid setting? I'm not sure. Somehow enforcing it there seems like a bad idea to me. 2. What exactly is the -e option supposed to do? .. Note that the question may somehow be re-phrased as: when invoked without the -e option, under which circumstances will deliver send a rejection message? Only when user is over quota. If you don't want messages to be delivered to some users, Postfix shouldn't call deliver for the user. 3. Doesn't failure_exit_callback() in deliver.c tend to merge many (possibly dissimilar) errors into the single EX_TEMPFAIL one? Yes. EX_TEMPFAIL is the safest choice almost always. If something breaks, you typically want to fix it and get the mail delivered again, instead of being rejected the first time and never seeing the mail. And BTW, is that function guaranteed to be always called with an EX_* value? Seems to be the case, but... ;-) What do you mean? It's called with FATAL_* values and it replaces them with EX_TEMPFAIL. signature.asc Description: This is a digitally signed message part
[Dovecot] Some questions about deliver
While investigating Dovecot's deliver with Postfix, I encountered some behaviors making me wonder whether I really understand the purpose of that binary. So, if you allow... This is from Postfix' main.cf: mailbox_command = /usr/local/dovecot/libexec/dovecot/deliver -e -n This is my quick setup for Dovecot: # 1.2.rc4: /usr/local/etc/dovecot.conf # OS: Darwin 9.7.0 i386 protocols: pop3 ssl: no disable_plaintext_auth: no login_dir: /usr/local/var/run/dovecot/login login_executable: /usr/local/dovecot-1.2.rc4/libexec/dovecot/pop3- login first_valid_uid: 2001 mail_location: mbox:~/_mailboxes:INBOX=~/_inbox/mailspool mbox_read_locks: flock mbox_write_locks: flock dotlock mail_executable: /usr/local/dovecot-1.2.rc4/libexec/dovecot/pop3 mail_plugin_dir: /usr/local/dovecot-1.2.rc4/lib/dovecot/pop3 pop3_lock_session: yes pop3_uidl_format: %08Xv%08Xu auth default: passdb: driver: pam args: * userdb: driver: passwd Here is an excerpt from mail.log: postfix/smtpd[4117]: connect from localhost[127.0.0.1] postfix/smtpd[4117]: 163E1CA2675: client=localhost[127.0.0.1] postfix/cleanup[4120]: 163E1CA2675: message-id=27002a3f-af44-410d-a500-368d04f5f...@almbp.local postfix/qmgr[3232]: 163E1CA2675: from=testu...@almbp.local, size=2650, nrcpt=1 (queue active) dovecot[4129]: deliver(www): mail_location: mbox: mkdir(/Library/ WebServer/_mailboxes) failed: Permission denied (euid=70(_www) egid=70(_www) missing +w perm: /Library/WebServer) dovecot[4129]: deliver(www): Fatal: Namespace initialization failed postfix/local[4122]: 163E1CA2675: to=w...@almbp.local, relay=local, delay=0.09, delays=0.02/0/0/0.07, dsn=4.3.0, status=deferred (temporary failure) postfix/smtpd[4117]: disconnect from localhost[127.0.0.1] 1. Shouldn't deliver honor the first_valid_uid setting? After all, if some user isn't allowed to login, is that user supposed to receive mail? 2. What exactly is the -e option supposed to do? User _www clearly isn't configured for receiving mail, but Postfix will nevertheless try to deliver the message until maximal_queue_lifetime has expired. From the log excerpt, it appears that Postfix has been led to consider a temporary failure; unless I'm wrong, that's because Postfix didn't receive neither a D.S.N text string nor an EX_* fatal return code. From deliver.c, it seems that many calls to i_fatal() are liable to be executed before the -e option is taken into account (through the local variable stderr_rejection). As a result, I'm wondering under which circumstances exactly deliver is liable to return EX_NOPERM, so that Postfix may consider a permanent failure and bounce the message. Note that the question may somehow be re-phrased as: when invoked without the -e option, under which circumstances will deliver send a rejection message? 3. Doesn't failure_exit_callback() in deliver.c tend to merge many (possibly dissimilar) errors into the single EX_TEMPFAIL one? And BTW, is that function guaranteed to be always called with an EX_* value? Seems to be the case, but... ;-) TIA for your patience, Axel
