Re: Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Mon, 8 Feb 2016, Charles Marcus wrote: My problem is, one of my most used folders, which was working fine up until a week or so ago, stopped loading the messages, and after some frustrating troubleshooting via email with people who don't listen very well, I finally got a tarball of this folder, and they are using maildir. There are about 24,000 messages in there (non-zero-byte files). This number sounds about right. All other folders (including INBOX, Sent, etc) are still working fine. The problem, though, is there are over 815,000 zero-byte-files in the cur directory, all showing as hardlinks (looks like maybe a whole bunch of duplicates for each of the real message files in the cur directory). "zero-byte-files ... showing as hardlinks" You mean this: hrw-r--r-- user/group 0 2016-02-09 07:26 ./2 link to ./1 ? This is a pseudo-notation of tar to indicate hardlinks. This is no zero-byte file. yes, these entries are duplicates of other messages. Note, https://en.wikipedia.org/wiki/Hard_link if two files are hardlinked together, there is no "to" or "from". You cannot tell, which existed before. They just indicate that those directory entries point to the same physical file with the same access rights and times and data. Extract the tar file to a Unix-like, inode-based filesystem supporting hardlinks to see. There are also 43 non-zero-byte message files in the new directory, and 1,515 of these zero-byte hardlinks (to message files in the new directory). There are also no non-zero-byte message files in the tmp directory, but there are 52 of the hardlinks, linked to something in the new directory. if there is such entry in the tmp directory, it indicates a failed delivery attempt. If one entry in "tmp" is hardlinked to one entry in "new" of the same mailbox, it may indicate that the message was to spool into another mailbox (via hardlink, too), which failed fatally. Is it possible that those messages are messages from your hoster and the message was to spool to many user mailboxes? I've never seen any of these kinds of zero-byte files before on the one server I managed for a long time (not shared, just used for a single domain). See above. Anyone ever seen this before? What does "stopped loading messages" mean? The MUA cannot download messages? Check if the server returns OK selecting the mailbox and if the numbers match, see http://wiki2.dovecot.org/TestInstallation You could use a select INBOX b copy 1 "mailbox-name" to copy a new message there and re-select the broken mailbox and compare the numbers. Also you could test, if the server crashes on a message in the mailbox, try c fetch 1:* BODY.PEEK[HEADER.FIELDS (SUBJECT)] c FETCH 1:* FLAGS c FETCH 1:* BODY[TEXT] Would running: doveadm index -u myuser * only, if the index is corrupt. or doveadm force-resync -u myuser * you can run doveadm, but cannot doveconf on the server? be appropriate commands to try to repair the damage (whatever it is)? Any other commands I could suggest running? Thanks. I know I haven't given much to go on. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQEVAwUBVrmPFHz1H7kL/d9rAQKGUAgAllyxylzcN+4+jvnB7rlxPwFF0/QbxbZb hHCVbLI5J0nL4BVaj8De1uY3TGW09HIf5p6DLoX0O0k+4tmvSKBSJASNZypF9Dco ELQbSoJCXL+fhOodsXxHXzfMJFVAM79Ly/2IPLsvHQclEUklrKKK7BXvpkqQmVKY Bos1ZWi0Ctl2pCZzG//dyz7ZRgkyr2j6MF/LaHRcmK0kOZT9fM8lfxPcYOY3ynOm xEjqDTP6iZtTMrpqm4YOMMhtXho0JmGVnLlO4HCdb9bMJzSwe/ZBw2Y2WoyuXwiL 4dmZ2r6WRQ+OM18aWGkDWQ3STenmuZUT4q7U3t1ObhnJw2xnLt0AJg== =oCQf -END PGP SIGNATURE-
Re: autoexpunge - different settings for different users
Hi Tom,
Looks like you are missing 'mailbox' keyword:
namespace/inbox/mailbox/Drafts/autoexpunge=3d
Regards,
Leon
On Tue, Feb 9, 2016 at 12:51 PM Tom Johnson wrote:
>
> > On Feb 7, 2016, at 10:59 AM, Timo Sirainen wrote:
> >
> > You can override any mail-related setting from userdb. So for example if
> you have:
> >
> > namespace inbox {
> > mailbox Trash {
> >autoexpunge = 30d
> > }
> > }
> >
> > You can have your userdb return namespace/inbox/Trash/autoexpunge=90d
> extra field for other users.
> http://wiki2.dovecot.org/UserDatabase/ExtraFields has some examples.
>
>
> I have my query returning the extra field, with the namespace entries
> separated by tabs. If I do a doveadm, I see it:
> # doveadm user
> field value
> uid 1005
> gid 1005
> home
> mail
> namespace/inbox/Drafts/autoexpunge 3d
> namespace/inbox/Quarantine/autoexpunge 3d
> quota_rule *:storage=1M
>
>
> I turned on debugging, and tried to deliver some email to see if
> autoexpunge is called, and I'm seeing this:
>
> 2016-02-09T01:41:23+ s01.lax.mailroute.net dovecot: doveadm(
> t...@terramar.net): Debug: Unknown userdb setting:
> plugin/namespace/inbox/Drafts/autoexpunge=3d
> 2016-02-09T01:41:23+ s01.lax.mailroute.net dovecot: doveadm(
> t...@terramar.net): Debug: Unknown userdb setting:
> plugin/namespace/inbox/Quarantine/autoexpunge=3d
>
>
>
> Anybody know what am I missing?
>
> Thanks-
>
> Tom
>
Re: autoexpunge - different settings for different users
> On Feb 7, 2016, at 10:59 AM, Timo Sirainen wrote:
>
> You can override any mail-related setting from userdb. So for example if you
> have:
>
> namespace inbox {
> mailbox Trash {
>autoexpunge = 30d
> }
> }
>
> You can have your userdb return namespace/inbox/Trash/autoexpunge=90d extra
> field for other users. http://wiki2.dovecot.org/UserDatabase/ExtraFields has
> some examples.
I have my query returning the extra field, with the namespace entries separated
by tabs. If I do a doveadm, I see it:
# doveadm user
field value
uid 1005
gid 1005
home
mail
namespace/inbox/Drafts/autoexpunge 3d
namespace/inbox/Quarantine/autoexpunge 3d
quota_rule *:storage=1M
I turned on debugging, and tried to deliver some email to see if autoexpunge is
called, and I'm seeing this:
2016-02-09T01:41:23+ s01.lax.mailroute.net dovecot:
doveadm(t...@terramar.net): Debug: Unknown userdb setting:
plugin/namespace/inbox/Drafts/autoexpunge=3d
2016-02-09T01:41:23+ s01.lax.mailroute.net dovecot:
doveadm(t...@terramar.net): Debug: Unknown userdb setting:
plugin/namespace/inbox/Quarantine/autoexpunge=3d
Anybody know what am I missing?
Thanks-
Tom
Re: Found bug in quota-status + patch
> Le 7 févr. 2016 à 23:26, Thierry Coppey a écrit : > > Hello, > > I've found a bug in the quota-status util (Dovecot 2.2.21, and probably > below): it always return > OK (sufficient quota) because it fails to load the user properly. More > specifically, the branch > " if (quser == NULL) return 1; " in src/plugins/quota/quota-status.c:59 is > always taken. > (at least with postgresql backend, test your config with the (adapted) > command below: > printf "recipient=mail...@example.com\nsize=100\n\n" | nc > inet:mailstore.example.com 12340 > it should reject, unless you have no quota, or more than 10G). > > […] Hello Thierry, Could it be that you found an explanation for a similar problem I encountered as early as with 2.2.4 and with a sqlite backend (see thread "2.2.4 - Some questions about and needing help with quota-status" started July 16, 2013)? It would be marvelous, as I’m currently stuck with quite an ugly workaround. Thanks, Axel
Re: autoexpunge - different settings for different users
> On Feb 8, 2016, at 2:02 AM, Timo Sirainen wrote:
>
>>
>> On 08 Feb 2016, at 06:46, Tom Johnson wrote:
>>
>> Can you return something like "namespace/inbox/*/autoexpuge=90d" to cover
>> all the namespaces? And then override individual ones if necessary?
>
> It would mean the same as:
>
> namespace inbox {
> mailbox * {
>autoexpunge = 90d
> }
> }
>
> So the question is about whether the above configuration works. Currently it
> doesn't, but it probably will in future.
Does this mean that you can't use autoexpunge on the main inbox for a user at
all?
Re: Dovecot with pam authentication and user@domain
Sorry for bringing up this issue again, but I still have no solution.
Is the describtion of my problem unclear? I suppose this setup is not
uncommon...
Greetings
Christian
On Samstag, 16. Januar 2016 00:51:27 CET Christian Schneider wrote:
> Hello all,
> I'm trying to setup dovecot for local users with pam authentication.
> The passdb and userdb entries are as follows:
>
> passdb {
> args = username_format=%n
> driver = pam
> }
> userdb {
> args = username_format=%n
> driver = passwd
> }
>
> Using "doveadm user chriss" returns the user record as expected, but
> "doveadm user chr...@testmail.ch-sc.de" gives an error:
> field valueuserdb lookup: user chr...@testmail.ch-sc.de doesn't exist
>
> As far as I understand, username_format=%n should drop the domain part and
> only search for the user in the userdb, but it doesn't. What am I missing?
>
> Greetings
> Christian
signature.asc
Description: This is a digitally signed message part.
Re: Permissions issue with Debian 8, dovecot 2.21, vpopmail
Am 08.02.2016 um 21:01 schrieb Stephen: There is a permissions issue in dovecot 2.21 in the ~/run directory (mine is /var/run/dovecot) at least in Debian 8. Files created by dovecot imap: srw--- 1 rootroot 0 Feb 3 16:46 imap-hibernate srw--- 1 rootroot 0 Feb 3 16:46 imap-master changed to make work: srw-rw-rw- 1 rootroot 0 Feb 3 16:46 imap-hibernate srw-rw-rw- 1 rootroot 0 Feb 3 16:46 imap-master Errors were reported in /var/log/dovecot.log which we use mainly for fail2ban (it gets huge, but ...) vpopmail version 5.4.33 with one patch I don't think this is a Debian issue ... possibly could be a vpopmail issue, but it's probably caused by the fact that imap daemons are running as vpopmail: vpopmail 30182 9187 0 15:02 ?00:00:00 dovecot/imap What purpose does vpopmail serve that dovecot alone does not? Isn't it just an ancient leftover from historic qmail times? Alexander
Permissions issue with Debian 8, dovecot 2.21, vpopmail
There is a permissions issue in dovecot 2.21 in the ~/run directory (mine is /var/run/dovecot) at least in Debian 8. Files created by dovecot imap: srw--- 1 rootroot 0 Feb 3 16:46 imap-hibernate srw--- 1 rootroot 0 Feb 3 16:46 imap-master changed to make work: srw-rw-rw- 1 rootroot 0 Feb 3 16:46 imap-hibernate srw-rw-rw- 1 rootroot 0 Feb 3 16:46 imap-master Errors were reported in /var/log/dovecot.log which we use mainly for fail2ban (it gets huge, but ...) vpopmail version 5.4.33 with one patch I don't think this is a Debian issue ... possibly could be a vpopmail issue, but it's probably caused by the fact that imap daemons are running as vpopmail: vpopmail 30182 9187 0 15:02 ?00:00:00 dovecot/imap
Re: Released Pigeonhole v0.4.12 for Dovecot v2.2.21
Am 08.02.2016 um 18:49 schrieb Stephan Bosch: > Op 2/7/2016 om 8:11 AM schreef Robert Schetterer: >> Am 06.02.2016 um 11:32 schrieb Stephan Bosch: >>> Hello Dovecot users, >>> >>> Here is the final v0.4.12 release of Pigeonhole for Dovecot v2.2.21. >>> Nothing changed since the RC. >>> >>> Changelog v0.4.12: >>> >>> + Implemented the Sieve extracttext extension (RFC 5703; Section 7). It >>> is now possible to extract body text from a message into a variable. >>> * Increased ABI version due to changes in the Sieve interpreter's object >>> definitions. >>> - multiscript: Fixed bug in handling of (implicit) keep; final keep >>> action was always executed as though there was a failure. This caused >>> the keep action to revert back to the initial message, causing >>> editheader actions to be ignored. >>> - managesieve-login: Fixed proxy to allow SASL mechanisms other than >>> PLAIN. Before, the proxy would fail if the server did not support the >>> PLAIN mechanism. >>> - ldap storage: Prevent segfault occurring when assigning certain >>> (global) configuration options. >>> >>> The release is available as follows: >>> >>> http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.12.tar.gz >>> http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.12.tar.gz.sig >>> >>> Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for >>> more information. Have fun testing this release and don't hesitate to >>> notify me when there are any problems. >>> >>> Regards, >>> >> Hi Stephan, looks like >> the last update at http://xi.rename-it.nl/ >> was on 04-Feb-2016 perhaps an additional build is need >> with Pigeonhole v0.4.12 ? > > As I mentioned earlier, Xi currently does not track release branches; it > only follows master which has placeholder version 2.2.devel (Pigionhole > has 0.4.devel). So, unless I intervene manually, base versions are > stuck at where they were last: Dovecot v2.2.21 and Pigeonhole v0.4.11. > Of course, all changes are in there, but the versions are not updated > with recent releases. This will be implemented once Dovecot v2.2.22 is > released, so that I can test this properly. > > Currently, as a workaround, you can use the git hash in the version > output from `dovecot -n` to find which master commits are part of your > package and thereby get an idea of what approximate version you are > actually running. > > Regards, > > Stephan. > thx Stephan , i am thinking i am fine now with dovecot_2.2.21-1~auto+115 looks like all sieve patches i want to test are included reading git changelog Best Regards MfG Robert Schetterer -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Marc Schiffbauer Aufsichtsratsvorsitzender: Florian Kirstein
Re: Released Pigeonhole v0.4.12 for Dovecot v2.2.21
Op 2/7/2016 om 8:11 AM schreef Robert Schetterer: > Am 06.02.2016 um 11:32 schrieb Stephan Bosch: >> Hello Dovecot users, >> >> Here is the final v0.4.12 release of Pigeonhole for Dovecot v2.2.21. >> Nothing changed since the RC. >> >> Changelog v0.4.12: >> >> + Implemented the Sieve extracttext extension (RFC 5703; Section 7). It >> is now possible to extract body text from a message into a variable. >> * Increased ABI version due to changes in the Sieve interpreter's object >> definitions. >> - multiscript: Fixed bug in handling of (implicit) keep; final keep >> action was always executed as though there was a failure. This caused >> the keep action to revert back to the initial message, causing >> editheader actions to be ignored. >> - managesieve-login: Fixed proxy to allow SASL mechanisms other than >> PLAIN. Before, the proxy would fail if the server did not support the >> PLAIN mechanism. >> - ldap storage: Prevent segfault occurring when assigning certain >> (global) configuration options. >> >> The release is available as follows: >> >> http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.12.tar.gz >> http://pigeonhole.dovecot.org/releases/2.2/dovecot-2.2-pigeonhole-0.4.12.tar.gz.sig >> >> Refer to http://pigeonhole.dovecot.org and the Dovecot v2.x wiki for >> more information. Have fun testing this release and don't hesitate to >> notify me when there are any problems. >> >> Regards, >> > Hi Stephan, looks like > the last update at http://xi.rename-it.nl/ > was on 04-Feb-2016 perhaps an additional build is need > with Pigeonhole v0.4.12 ? As I mentioned earlier, Xi currently does not track release branches; it only follows master which has placeholder version 2.2.devel (Pigionhole has 0.4.devel). So, unless I intervene manually, base versions are stuck at where they were last: Dovecot v2.2.21 and Pigeonhole v0.4.11. Of course, all changes are in there, but the versions are not updated with recent releases. This will be implemented once Dovecot v2.2.22 is released, so that I can test this properly. Currently, as a workaround, you can use the git hash in the version output from `dovecot -n` to find which master commits are part of your package and thereby get an idea of what approximate version you are actually running. Regards, Stephan.
Lots of zero-byte hard link files in cur (and new/tmp), cannot see messages in folder
Hello, I have an el-cheapo shared hosting account on Dreamhost, and have had it for a very long time. For the most part everything usually works fairly well, considering I do keep a lot of folders, and mail, on some of my accounts. They are running dovecot, but still don't have a response as to the version, or doveconf -n output yet. My problem is, one of my most used folders, which was working fine up until a week or so ago, stopped loading the messages, and after some frustrating troubleshooting via email with people who don't listen very well, I finally got a tarball of this folder, and they are using maildir. There are about 24,000 messages in there (non-zero-byte files). This number sounds about right. All other folders (including INBOX, Sent, etc) are still working fine. The problem, though, is there are over 815,000 zero-byte-files in the cur directory, all showing as hardlinks (looks like maybe a whole bunch of duplicates for each of the real message files in the cur directory). There are also 43 non-zero-byte message files in the new directory, and 1,515 of these zero-byte hardlinks (to message files in the new directory). There are also no non-zero-byte message files in the tmp directory, but there are 52 of the hardlinks, linked to something in the new directory. I've never seen any of these kinds of zero-byte files before on the one server I managed for a long time (not shared, just used for a single domain). Anyone ever seen this before? Would running: doveadm index -u myuser * or doveadm force-resync -u myuser * be appropriate commands to try to repair the damage (whatever it is)? Any other commands I could suggest running? Thanks. I know I haven't given much to go on. Charles /* */
Re: "mail_shared_explicit_inbox = no", but shared INBOX has a "\Noselect" tag?
Not solved, but found the reason of the problem: Obviously, the plugin "listescape" breaks the sharing of the INBOX of user as "Shared/" for another user. At least with usernames containing a dot "." as in "u...@example.com", which are used as the name of the filesystem Maildir home directory. (IMAP separator "/".) When using this plugin, a "\Noselect" tag is added then to the output of the response to the IMAP command 'list "" "Shared/ow...@example.com"'. Best Regards, R.Koerner
Re: ACL user
Am 2016-02-08 um 11:50 schrieb Timo Sirainen:
On 05 Feb 2016, at 17:42, Peter Chiochetti wrote:
How would I go, If I wanted ACL processing to start with
%{auth_user} instead of %{user} when determining rights?
You could kludge it by returning master_user=%{auth_user} in userdb,
but that might affect other things..
[…]
I tested the kludge: I put userdb_master_user=someone into the static
passwd file for a certain auth_user and now global ACLs apply; as an
extra bonus now userdb_acl_groups=somegroup starts to be applied too for
that account!
Observations:
- my virtual users start with no rights
- I add rights in the global dovecot-acl file
- changes work immediately, no restart necessary
- only users with a master_user set are affected
- in the future a single stance in local.conf will apply to all users
I could not put master_user=%{auth_user} into the userdb stance (nor the
passwd file), because the parser does not expand the variable, possibly
a formatting error on my side: "doveadm -D acl debug -u myname INBOX"
then prints:
Debug: Added userdb setting: plugin/master_user=auth_user}
After all, once more
A happy dovecot user!
--
peter
Re: Send Dovecot logs to rsyslog
Great ! I was using both "log_path" and "syslog_facility" so "syslog_facility" didn't work. No I have in dovecot.conf : # Logging. Reference: http://wiki2.dovecot.org/Logging #log_path = /var/log/dovecot.log syslog_facility = local5 And in rsyslog.conf : local5.*@IP_RSYSLOG_SERVER:PORT local5.*-/var/log/dovecot.log Thanks a lot Nicolas 2016-02-08 11:06 GMT+01:00 Phil Carmody : > On Sat, Feb 06, 2016 at 07:22:33PM +0100, Nicolas Fo wrote: > > Just tested, same result :( > > It may be a silly question, but is one of log_path, info_log_path, or > debug_log_path set to "syslog"? log_path defaults to it, so if you've > changed that, the syslog service wouldn't be used. Dovecot's use of > syslog is very vanilla, there's not much room for it to do much wrong. > > Phil > > > 2016-02-06 16:44 GMT+01:00 Oscar del Rio : > > > On 05/02/2016 9:42 AM, Nicolas Fo wrote: > > >> I set in dovecot.conf : syslog_facility = local5.info > > > > > > Try: > > > > > > syslog_facility = local5 > > > > > > Source: > > > http://wiki.dovecot.org/Logging#Syslog_Example >
Found bug in quota-status + patch
Hello, I've found a bug in the quota-status util (Dovecot 2.2.21, and probably below): it always return OK (sufficient quota) because it fails to load the user properly. More specifically, the branch " if (quser == NULL) return 1; " in src/plugins/quota/quota-status.c:59 is always taken. (at least with postgresql backend, test your config with the (adapted) command below: printf "recipient=mail...@example.com\nsize=100\n\n" | nc inet:mailstore.example.com 12340 it should reject, unless you have no quota, or more than 10G). Therefore it is pointless to configure the quota-status service as described in http://wiki2.dovecot.org/Quota because of that bug: the mail would be always accepted from the client and rejected at the end of the (Postfix) processing by lda. It is possible to reject directly at SMTP level when the mailbox is full using Postfix and main.cf: smtpd_recipient_restrictions = ... reject_unauth_destination, ... check_policy_service inet:mailstore.example.com:12340 smtpd_end_of_data_restrictions = check_policy_service inet:mailstore.example.com:12340 However there remain a *small inconsistency* as lda will count the size of message including headers whereas smtpd_end_of_data_restrictions seems not to count them towards the message size (or additional headers might be added by other preprocessing e.g. antispam etc...) I propose the patch below to fix quota-status. Could an expert double-check that it doesn't have memory leaks? (it shouldn't but I'm not yet comfortable enough with Dovecot memory management). Best regards, Thierry C. quota-status.patch Description: Binary data
Re: RFC: HTTP based storage API
> On 08 Feb 2016, at 13:28, Nagy, Attila wrote: > > On 02/08/16 12:07, Timo Sirainen wrote: >> On 08 Feb 2016, at 12:56, Nagy, Attila wrote: >>> On 02/08/16 11:16, Timo Sirainen wrote: Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement. >>> Yeah, the opposite, in this case a jmap backend to Dovecot. >>> BTW, I think jmap is too high level and implementing a jmap server is very >>> much like implementing an IMAP one. >>> >>> I much more think of a pluggable, easy (remote) storage API, which has much >>> less to do with IMAP, but can offer capabilities, which can help Dovecot >>> (like the search and indexes). >> What kind of use cases are you thinking for this remote storage API? What >> kind of remote storages would implement it, and what kind of installations >> would use them? >> > Interfacing non-email systems to e-mail protocols and implementing other > storage options (I can't say a better analogue than what FUSE is). > Using Dovecot as a protocol implementation, but not for the underlying > storage method, which may be custom to the given solution. > A flexible way of implementing and adapting anything to IMAP. > > For example if I want to store really old and archived e-mails in a costly > compression format somewhere in the cloud, but hot mails locally, based on my > custom policies. > In an international company there may be some regulations about what should > go where, so for example if I can handle object placement myself, I can use > the same installation to store non-EU mails in non-EU countries and others in > US or EU clouds if law permits. You could already implement these with mail-filter plugin. The locally stored mail would be just an object pointer, which the mail-filter plugin reads from the remote storage. Although I think mail-filter is lacking error handling right now. > Intermixing several systems into one (like merging two IMAP accounts into > one) etc. If I bothered to implement per-namespace imapc-settings, this could be done with virtual plugin. > Some of these are very costly to develop in Dovecot, while just a few lines > in -say- Python or go. I think it might not be that easy. Although I've made lib-storage backends easier and easier to implement, it's still not exactly trivial. Implementing it behind some API could simplify it somewhat, but it wouldn't really remove all the difficult work that needs to be done. For example some backends might want to be read-only, others read-write. Some might want Dovecot to store all the message flags and other metadata locally, while others might want to store it themselves (and that would require two-way syncing between them). Some backends would support searching, fetching some headers fast and in general support different kinds of optimizations, while others wouldn't. The imapc backend is of course already implementing a lot of this functionality, but it's also getting complex. The backend would also have to guarantee some things to be compatible with IMAP, mainly never modifying existing mails. One alternative might be to add scripting support to Dovecot. I've been thinking about that several times over the years. So instead of implementing plugins with C, you could implement them with Python or some other language. Long time ago I tried to do this with SWIG, but it didn't work out. The main problem was function pointers in structs, but nowadays those aren't directly called and I think it could be implemented a bit differently.
Re: RFC: HTTP based storage API
On 2016-02-08 12:31:57 +0100, Nagy, Attila wrote: > On 02/08/16 11:44, Marcus Rueckert wrote: > >that sounds like s3 API based backend. > > > >also note that dovecot comes with some "non local mail storage plugins" > >in the commercial part: > > > >``` > >In this release, we support Windows Azure, Amazon S3, Scality and > >Dropbox systems. > >``` > > > >so that should come close to what you want. for a local version you > >could use e.g. radosdb/ceph or riak. both can provide the s3 API for > >you. > > > Close, but this may be too low level, for example it may hide whose e-mail > do you store (I don't know the exact implementation, it's not open source > AFAIK). True ... but that might be the hook point you can use for your more fancy implementation. and some of the mail backends are public. like mdbox and so on. the api towards dovecot should be the same. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
Re: RFC: HTTP based storage API
On 02/08/16 11:44, Marcus Rueckert wrote: that sounds like s3 API based backend. also note that dovecot comes with some "non local mail storage plugins" in the commercial part: ``` In this release, we support Windows Azure, Amazon S3, Scality and Dropbox systems. ``` so that should come close to what you want. for a local version you could use e.g. radosdb/ceph or riak. both can provide the s3 API for you. Close, but this may be too low level, for example it may hide whose e-mail do you store (I don't know the exact implementation, it's not open source AFAIK).
Re: RFC: HTTP based storage API
On 02/08/16 12:07, Timo Sirainen wrote: On 08 Feb 2016, at 12:56, Nagy, Attila wrote: On 02/08/16 11:16, Timo Sirainen wrote: Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement. Yeah, the opposite, in this case a jmap backend to Dovecot. BTW, I think jmap is too high level and implementing a jmap server is very much like implementing an IMAP one. I much more think of a pluggable, easy (remote) storage API, which has much less to do with IMAP, but can offer capabilities, which can help Dovecot (like the search and indexes). What kind of use cases are you thinking for this remote storage API? What kind of remote storages would implement it, and what kind of installations would use them? Interfacing non-email systems to e-mail protocols and implementing other storage options (I can't say a better analogue than what FUSE is). Using Dovecot as a protocol implementation, but not for the underlying storage method, which may be custom to the given solution. A flexible way of implementing and adapting anything to IMAP. For example if I want to store really old and archived e-mails in a costly compression format somewhere in the cloud, but hot mails locally, based on my custom policies. In an international company there may be some regulations about what should go where, so for example if I can handle object placement myself, I can use the same installation to store non-EU mails in non-EU countries and others in US or EU clouds if law permits. Intermixing several systems into one (like merging two IMAP accounts into one) etc. Some of these are very costly to develop in Dovecot, while just a few lines in -say- Python or go.
Re: RFC: HTTP based storage API
On 08 Feb 2016, at 12:56, Nagy, Attila wrote: > > On 02/08/16 11:16, Timo Sirainen wrote: >> Oh, you were thinking about ability to provide IMAP/etc support for other >> random servers, and have Dovecot act as kind of a middleware and translate >> the requests. Maybe the answer is still jmap though? It would require jmap >> lib-storage backend similar to imapc, which would be doable, although not >> really something we're right now planning to implement. >> > Yeah, the opposite, in this case a jmap backend to Dovecot. > BTW, I think jmap is too high level and implementing a jmap server is very > much like implementing an IMAP one. > > I much more think of a pluggable, easy (remote) storage API, which has much > less to do with IMAP, but can offer capabilities, which can help Dovecot > (like the search and indexes). What kind of use cases are you thinking for this remote storage API? What kind of remote storages would implement it, and what kind of installations would use them?
Re: RFC: HTTP based storage API
On 02/08/16 11:16, Timo Sirainen wrote: On 08 Feb 2016, at 11:59, Timo Sirainen wrote: On 08 Feb 2016, at 11:01, Nagy, Attila wrote: Hi, Nearly every popular programming language has an LMTP/POP/IMAP implementation, most of them suck in many different ways. I don't know any server or library which provides a well-established, compatible protocol frontend with an open backend API, which could be used to easily make a custom storage backend for the LMTP/POP/IMAP frontend in any language, in any programming paradigm, irregardless of the frontend implementation. Dovecot is very close to this: it speaks IMAP to the backend, but you still have to make a nearly complete IMAP implementation, which is a PITA. Therefore I am curious about your opinion: what about an HTTP-based open backend (lib-storage) API to Dovecot? We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close enough to how Dovecot works internally (although we'd still need the threading support) and also at least attempting to become a standard. We haven't really started it yet though, so now would be a good time to give any alternative suggestions or complaints about jmap :) Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement. Yeah, the opposite, in this case a jmap backend to Dovecot. BTW, I think jmap is too high level and implementing a jmap server is very much like implementing an IMAP one. I much more think of a pluggable, easy (remote) storage API, which has much less to do with IMAP, but can offer capabilities, which can help Dovecot (like the search and indexes).
Re: ACL user
On 05 Feb 2016, at 17:42, Peter Chiochetti wrote:
>
> How would I go, If I wanted ACL processing to
> start with %{auth_user} instead of %{user}
> when determining rights?
You could kludge it by returning master_user=%{auth_user} in userdb, but that
might affect other things.. But since this has been asked a few times, I
finally implemented it in a bit cleaner way:
https://github.com/dovecot/core/commit/dd5683e19979085fdfe9f269876f2a91ea604679
So you can do:
plugin {
acl_user = %{auth_user}
}
Re: RFC: HTTP based storage API
that sounds like s3 API based backend. also note that dovecot comes with some "non local mail storage plugins" in the commercial part: ``` In this release, we support Windows Azure, Amazon S3, Scality and Dropbox systems. ``` so that should come close to what you want. for a local version you could use e.g. radosdb/ceph or riak. both can provide the s3 API for you. hth darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
Re: Dovecot (LDAP) quota only if user in backend?
On Wed, Feb 3, 2016 at 6:06 PM, Andrey Fesenko wrote: > Hello, > > Is it possible to get the quotas for users served by the backend? > > My env CentOS, dovecot-2.2.10-5.el7, users in LDAP > > > BackendHost may bee backend1.wibble.net backend2.wibble.net ... if I'm > get quota in backend1.wibble.net > > doveadm -f tab quota get -A > After fix dovecot-ldap.conf user_filter = (&(objectClass=posixAccount)(uid=%u)(BackendHost=backend1.wibble.net)) pass_attrs = uid=user,userPassword=password pass_filter = (&(objectClass=posixAccount)(uid=%u)) # For using doveadm -A: iterate_attrs = uid=user iterate_filter = (&(objectClass=posixAccount)(BackendHost=backend1.wibble.net)) Work perfect > doveadm: Error: Failed to iterate through some users Need more search limit in LDAP
Re: RFC: HTTP based storage API
> On 08 Feb 2016, at 11:59, Timo Sirainen wrote: > > On 08 Feb 2016, at 11:01, Nagy, Attila wrote: >> >> Hi, >> >> Nearly every popular programming language has an LMTP/POP/IMAP >> implementation, most of them suck in many different ways. >> I don't know any server or library which provides a well-established, >> compatible protocol frontend with an open backend API, which could be used >> to easily make a custom storage backend for the LMTP/POP/IMAP frontend in >> any language, in any programming paradigm, irregardless of the frontend >> implementation. >> >> Dovecot is very close to this: it speaks IMAP to the backend, but you still >> have to make a nearly complete IMAP implementation, which is a PITA. >> >> Therefore I am curious about your opinion: what about an HTTP-based open >> backend (lib-storage) API to Dovecot? > > We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close > enough to how Dovecot works internally (although we'd still need the > threading support) and also at least attempting to become a standard. We > haven't really started it yet though, so now would be a good time to give any > alternative suggestions or complaints about jmap :) Oh, you were thinking about ability to provide IMAP/etc support for other random servers, and have Dovecot act as kind of a middleware and translate the requests. Maybe the answer is still jmap though? It would require jmap lib-storage backend similar to imapc, which would be doable, although not really something we're right now planning to implement.
Re: Send Dovecot logs to rsyslog
On Sat, Feb 06, 2016 at 07:22:33PM +0100, Nicolas Fo wrote: > Just tested, same result :( It may be a silly question, but is one of log_path, info_log_path, or debug_log_path set to "syslog"? log_path defaults to it, so if you've changed that, the syslog service wouldn't be used. Dovecot's use of syslog is very vanilla, there's not much room for it to do much wrong. Phil > 2016-02-06 16:44 GMT+01:00 Oscar del Rio : > > On 05/02/2016 9:42 AM, Nicolas Fo wrote: > >> I set in dovecot.conf : syslog_facility = local5.info > > > > Try: > > > > syslog_facility = local5 > > > > Source: > > http://wiki.dovecot.org/Logging#Syslog_Example
Re: autoexpunge - different settings for different users
> On 08 Feb 2016, at 06:46, Tom Johnson wrote:
>
>> You can override any mail-related setting from userdb. So for example if you
>> have:
>>
>> namespace inbox {
>> mailbox Trash {
>> autoexpunge = 30d
>> }
>> }
>>
>> You can have your userdb return namespace/inbox/Trash/autoexpunge=90d extra
>> field for other users. http://wiki2.dovecot.org/UserDatabase/ExtraFields has
>> some examples.
>
> Thank you, Timo!
>
> Can you return something like "namespace/inbox/*/autoexpuge=90d" to cover all
> the namespaces? And then override individual ones if necessary?
It would mean the same as:
namespace inbox {
mailbox * {
autoexpunge = 90d
}
}
So the question is about whether the above configuration works. Currently it
doesn't, but it probably will in future.
Re: RFC: HTTP based storage API
On 08 Feb 2016, at 11:01, Nagy, Attila wrote: > > Hi, > > Nearly every popular programming language has an LMTP/POP/IMAP > implementation, most of them suck in many different ways. > I don't know any server or library which provides a well-established, > compatible protocol frontend with an open backend API, which could be used to > easily make a custom storage backend for the LMTP/POP/IMAP frontend in any > language, in any programming paradigm, irregardless of the frontend > implementation. > > Dovecot is very close to this: it speaks IMAP to the backend, but you still > have to make a nearly complete IMAP implementation, which is a PITA. > > Therefore I am curious about your opinion: what about an HTTP-based open > backend (lib-storage) API to Dovecot? We're planning on implementing http://jmap.io/ to Dovecot v2.3. It's close enough to how Dovecot works internally (although we'd still need the threading support) and also at least attempting to become a standard. We haven't really started it yet though, so now would be a good time to give any alternative suggestions or complaints about jmap :)
RFC: HTTP based storage API
Hi, Nearly every popular programming language has an LMTP/POP/IMAP implementation, most of them suck in many different ways. I don't know any server or library which provides a well-established, compatible protocol frontend with an open backend API, which could be used to easily make a custom storage backend for the LMTP/POP/IMAP frontend in any language, in any programming paradigm, irregardless of the frontend implementation. Dovecot is very close to this: it speaks IMAP to the backend, but you still have to make a nearly complete IMAP implementation, which is a PITA. Therefore I am curious about your opinion: what about an HTTP-based open backend (lib-storage) API to Dovecot? Some thoughts on this: 1. it should be a simple REST API 2. it should support efficient operations on both sides (client, server), for example listing a multi-million folder shouldn't be a single JSON response, but a JSON stream, which could be produced and parsed individually or in smaller batches efficiently 3. it should support capabilities, somewhat like the current imapc backend, so if the backend service suppports search, it should announce this capability and dovecot could offload these requests to it, but if it doesn't, dovecot could build its indices. 4. only the needed amount of POP/IMAP legacy should appear in the protocol 5. the operations should be asynchronous on Dovecot's side (to be scalable) 6. maybe even Dovecot's index-queries could be offloaded to the backend service (this way the local file system wouldn't be used at all and all caches could be stored in a distributed dictionary), this could also be a capability 7. it could support some kind of push (for IDLE-ing and the like) 8. it should be stateless If Dovecot could make this right, it could be the FUSE of the mail protocols, acting as a glue for experimental or production ready mail storage backends, without the need to write these backends in "Dovecot C" (I mean it's process and API structure). Of course HTTP here is not a strict requirement. It could be anything, which can be easy to implement and understand and is able to fulfill the needs. An example could be Dovecot's own dictionary protocol. What do you think about this?
