Re: [Dovecot] v2.1 memory usage
On 2012-11-12 20:44, Timo Sirainen wrote: > On 12.11.2012, at 6.13, Daniel L. Miller wrote: >> The tiny bit of Googling I've done tells me GnuTLS >> seems to be a more standards-compliant implementation, and MAY be >> "safer" than OpenSSL. However, as OpenSSL is the de-facto standard used >> by most Linux programs, acceptance of GnuTLS is quite limited. I've been >> intrigued by what I've read about it, and took a quick look at enabling >> support in Dovecot for GnuTLS directly - but while it didn't seem overly >> heavy at first glance the fact that Timo doesn't want to do it tells me >> I'm underestimating the complexity. > > I already once wrote GnuTLS support for Dovecot, but GnuTLS changed its APIs > since then and it was probably originally already buggy. I think the only > somewhat "special" APIs that Dovecot needs nowadays are related to reading > cert/keys from memory instead of from files. If GnuTLS can do that, I don't > think there's anything special in supporting it. Although it might be a bit > complex to make it work properly asynchronously. istream-openssl was a bit > annoying in that way (all the data read from the fd must be parsed and > decoded all the way through to the SSL istream, regardless of any max buffer > limits). A while ago, I started working on GnuTLS support for Dovecot. While I didn't finish the implementation due to time constraints (the "abstract" API of lib-ssl-iostream is made for OpenSSL and you have to emulate some things), I think it would be possible to add GnuTLS support and with GnuTLS it would be possible to load X.509 certificates from memory. Regards, Matthias-Christian
Re: [Dovecot] dovecot support ms-tnef mail parser?
On 2012-06-25 04:52, DongYu.Zhen wrote: > How to parse ms-tnef mail by dovecot? You can't do that directly in Dovecot. What you can do is to use a utility called tnef [1] (available in major GNU/Linux distributions) on the client to extract the data on the client. Otherwise you could use ytnef [2] with procmail or Dovecot sieve_pipe [3] on the server (see [4]). I tried tnef on rare occasions and it worked. Regards, Matthias-Christian [1] http://sourceforge.net/projects/tnef/ [2] http://sourceforge.net/projects/ytnef/ [3] http://wiki2.dovecot.org/Pigeonhole/Sieve/Plugins/Pipe [4] http://wiki.clug.org.za/wiki/Automatic_winmail.dat_decoding
Re: [Dovecot] [ Re: best practises for mail systems]
On 2012-06-09 16:11, Timo Sirainen wrote: > On 9.6.2012, at 4.55, Matthias-Christian Ott wrote: > >> Yes, there is. You have to replicate the entire state of the IMAP >> session (protocol states, buffers, TLS state etc.) and the TCP state of >> the connection. The state of the IMAP session is (in theory) easily >> replicable (although you probably have to rely on internals of the TLS >> implementation; OpenSSL can serialise TLS sessions from/into ASN.1 via >> i2d_SSL_SESSION, though this is meant to resume session via TLS) > > Interesting! I thought OpenSSL didn't have a way to [de]serialize the session > state. The first time I wanted to do that was 13 years ago. I see there are > some google hits for i2d_SSL_SESSION, but do you already know a good web page > / example code I could look at? The Apache httpd module mod_ssl uses it. GnuTLS has similar functions with gnutls_db_*, although it's also only intended to be used to resume a session. Have look at the Apache httpd module mod_gnutls. Regards, Matthias-Christian
Re: [Dovecot] [ Re: best practises for mail systems]
On 2012-06-05 23:43, Timo Sirainen wrote: > On 5.6.2012, at 23.33, Michescu Andrei wrote: > >>> I agree, in practice this is not an issue compared to the unavailability >>> of the service, but on longer IMAP sessions (e.g. transferring a big >>> file) the connection loss is noticeable. >> >> It is noticeable for somebody that really waits for a large email. > > And there is actually some (any!) way this could be avoided?... One server > dies, another continues sending the mail? Yes, there is. You have to replicate the entire state of the IMAP session (protocol states, buffers, TLS state etc.) and the TCP state of the connection. The state of the IMAP session is (in theory) easily replicable (although you probably have to rely on internals of the TLS implementation; OpenSSL can serialise TLS sessions from/into ASN.1 via i2d_SSL_SESSION, though this is meant to resume session via TLS) and for TCP there is RTCP [1]. RTCP intercepts the TCP session is able to recover the TCP state. It works without any modification of the operating system (at the moment limited to Linux). If this would be implemented in Dovecot it would really set it apart from other IMAP servers and software that I've seen so far. Being able to transparently handle failover of a TCP connection is unique. > I have had some thoughts about transferring idling Dovecot connections > between processes / servers so that clients wouldn't notice it, but I haven't > even thought about moving active (long-running) connections. Load rebalancing would probably be another feature that separates Dovecot from other IMAP servers. Regards, Matthias-Christian [1] http://rtcp.sourceforge.net/
Re: [Dovecot] [ Re: best practises for mail systems]
On 2012-06-05 17:33, Michescu Andrei wrote: > The fear of lossing the imap session does not make sense (at least to me) > as the client will reconnect automatically in the background. I agree, in practice this is not an issue compared to the unavailability of the service, but on longer IMAP sessions (e.g. transferring a big file) the connection loss is noticeable. > Like this you have no SPOF and no split-brain and you get the flexibility > (if needed) to geographically distribute your servers in the the future. > > Keep each server with its own ip, connect to them via DNS (round robin etc > etc). This depends on the resolver, operating systems and clients you want to support, because I read that not all networks generate proper ICMP/ICMPv6 Destination Unreachable messages and instead simple drop the packets, so that the clients first try to connect to the failed server until timeout and then connects to the second server. Since IMAP is a stateful protocol the latency of the initial connect to the failed server can be ignored, but if you want to eliminate this, you can use dynamic DNS to automatically remove the corresponding RRs (depending on your situation you need an external monitoring server for this to avoid problems in case of net splits). > We are currently experimenting with a setup similar to this one, but with > geographically distributed servers (trans-continental) (bandwidth limited > and high cost). I also have some plans for a similar setup in the near future. Can you share your results on the mailing list? I'm especially interested if failover via DNS works in practice (I did some searches, but I'm not fully convinced of it, but it seems quite simple compared to other solutions). Regards, Matthias-Christian
Re: [Dovecot] best practises for mail systems
On 2012-06-05 05:14, Костырев Александр Алексеевич wrote: > On each host system we created one VM and passed through 3x2TB disks into it. > > > > In guests vms on top of this disks we made XFS and fired up glusterfs with > distributed replicated volumes for our mailstorage. > > so it looks like this: > > > > vm1replicate vm2 > > disk1 > disk4 > > disk2 > disk5 > > disk3 > disk6 > > > > in each vm we mounted glusterfs and pointed dovecot to that dir for mail > creation (as ltmp) and imap4 user access. > > also we use exim as smtp. > > > > So, with glusterfs as mailstorage we can go for LVS to load balancing for > exim and dovecot. > > so wherenever one of host systems (hence one of mail vms) goes down, users > don't notice that > > 'cause LVS points them to working smtp and imap4 servers > > and they get their mail 'cause of glusterfs. > [...] > Cons: > > - not quite sure if glusterfs is production ready solution 'cause I've > experienced split-brains during setting it up > > - IO performance issue. Though we didn't yet run any io tests, but glusterfs > uses fuse to mount on clients. And guys on #gluster told me writing to the > glusterfs mount will not be strictly local io. I'm not familiar with LVS, but from the project description it seems that you need a "front server" that does the load balancing, so you either have to run at least two of these servers in parallel or add to your cons that you introduced a single point of failure. But you mentioned that you only have two servers, so you really can do this. I would rather ensure high availability by running the two servers as masters and using either IP address takeover or DNS failover (with dynamic DNS) and either use Dovecot's replication (I haven't tested it yet and I'm not sure what happens in case of IP address takeover) or a file system that can handle these kinds of errors (e.g. Coda). You could do load balancing via round-robin DNS. This only protects you against the failure of single machine and because IMAP sessions are not replicated between the servers, connections will get reset if one server fails, but it's cost-effective and uses software that already exists. Regards, Matthias-Christian
Re: [Dovecot] TLS X.509 CRLs
On Mon, 14 May 2012 18:37:37 +0300, Timo Sirainen wrote: > On Sun, 2012-05-13 at 18:43 +0200, Matthias-Christian Ott wrote: > > > according to the documentation file referenced by ssl_ca must > > contain the Client certificate CA and the corresponding CRL. Thus > > dovecot would have to receive SIGHUP to reload a new CRL. Did I > > understand this correctly? > > Yeah. Thanks for the confirmation. I think this should suffice for my use case (I control the CA, so I can immediately upload a new CRL once I have revoked a certificate), but it doesn't sound practical if you have a bigger deployment, there will be a delay between the revocation and the update of the CRL on the mail servers. OCSP could solve this (though X.509 in the current form is broken and it is not clear whether it is worth the effort). Regards, Matthias-Christian
[Dovecot] TLS X.509 CRLs
Hi, according to the documentation file referenced by ssl_ca must contain the Client certificate CA and the corresponding CRL. Thus dovecot would have to receive SIGHUP to reload a new CRL. Did I understand this correctly? Regards, Matthias-Christian
[Dovecot] Replace message headers
Hi, I'm looking for a way to replace specific message headers of a mail via the mail storage API. Any suggestions? Regards, Matthias-Christian
Re: [Dovecot] Dovecot and Bogofilter
Steffen Kaiser wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Mon, 10 Nov 2008, Matthias-Christian Ott wrote: > >>> If you want something and Dovecot's deliver doesn't fit your needs, >>> consider maildrop, <http://www.courier-mta.org/maildrop/> >> >> Looks a bit bloated to me. > > I'm not sure if I missed that part about the shell script, someone > suggested. > > You could use something like: > > #!/bin/bash > > bogofilter | dovecot-deliver "$@" > > replacing the orginal dovecot-deliver in the conf. > If bogofilter does not support to pipe the message through, there will be > other ways. > > Besides that you have just one additional exec() (the shell) for each > delivery and you already inject the modified message to deliver. Or did I > misunderstood the problem, and you want the antispam to change the > message, when a mail is copied into / out of the SPAM folder via IMAP? Yes, but currently it doesn't matter, because I decided to not use spam filtering at the moment. Mutt just sucks and all this crappy mail software that's still around. I can't see this from a pragmatic point of view anymore, it's so fucked up. It just annoys me, I'll keep my mail server at low traffic and avoid to use all these things. This whole issue just showed me yet another time that UNIX is really dead and forgotten. I have better things to do than just configuring my computers, finding workarounds etc. I wish we could somehow turn the clocks back 30 years and do all this crap right. In my opinion the whole software industry (including the Free Software community which degenerated into a periphery position) is just a bloated bubble that will disastrously collapse at some point of time. I'm yet another time glad that I don't have to work in this business. So I think we can regard this issue as resolved. Sorry for annoying you with my pessimistic opinion about software, but it just bothers that after nearly four centuries of UNIX that nobody can get simple tasks like mail delivery right (Yes, I know there are workarounds, but there are tons of magazines that deal with workarounds for some big proprietary operating system and it's still pain to use it). > Bye, > > - -- Steffen Kaiser Regards, Matthias-Christian
Re: [Dovecot] Dovecot and Bogofilter
Patrick Nagel wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi Matthias
>
> Matthias Andree wrote:
> > On Mon, 10 Nov 2008, Patrick Nagel wrote:
> >> The problem with modifying the mail after reclassification persists, hope
> >> this
> >> can be solved. Then I could also finally move from
> >> postfix-procmail-bogofilter-cron-dovecot to
> >> postfix-deliver-antispam-bogofilter-dovecot. :)
> >
> > Hi,
> >
> > it's not quite clear to me how this would work with bogofilter as a
> > mailbox_transport - bogofilter isn't designed to do final delivery.
> >
> > It's also not quite clear to me why people would use procmail. Although
> > defended by its maintainers, it's an unusable and unconfigurable piece
> > of software from ancient past -- getting error handling right in
> > procmail is next to impossible, requires forfeiting :e rules and
> > bloating procmailrc with explicit error handling recipes.
>
> You're right, and that's why I don't want to continue using it. It had a lot
> of
> security issues, and the syntax of procmailrc looked like a bad joke to me
> when
> I encountered it for the first time.
I think the syntax is not that worse if got a bit used to it. Even this
exitcode issues are just a matter of standardisation. But look at this
(taken from procmail.c):
[...]
int main(argc,argv)int argc;const char*const argv[];
{ register char*chp,*chp2;
#if 0 /* enable this if you want to trace procmail */
kill(getpid(),SIGSTOP);/*raise(SIGSTOP);*/
#endif
newid();
;{ int presenviron,override;char*fromwhom=0;
const char*idhint=0;gid_t egid=getegid();
presenviron=Deliverymode=mailfilter=override=0;
Openlog(procmailn,LOG_PID,LOG_MAIL); /* for the syslogd */
if(argc) /* sanity check, any argument at all? */
{ Deliverymode=!!strncmp(lastdirsep(argv0=argv[0]),procmailn,
STRLEN(procmailn));
[...]
The whole source code looks like this. I mean, who is supposed to
read this and who wrote this? I don't want to use such software, it's
a nightmare.
> Not being able to pipe mail through an arbitrary program surely makes sieve
> more secure, but it also makes things much more complicated sometimes. Also I
> think beginners tend to use procmail, just because in many guides / tutorials
> /
> howtos it's the LDA of choice. (I, for example, started out with this howto:
> http://www.gentoo-wiki.info/HOWTO_Email_System_for_the_Home_Network )
>
> > If you want something and Dovecot's deliver doesn't fit your needs,
> > consider maildrop,
Re: [Dovecot] Dovecot and Bogofilter
Matthias Andree wrote: > On Mon, 10 Nov 2008, Patrick Nagel wrote: > > > Thorsten Vollmer wrote: > > > On Sun, 2008-11-09 at 22:39 +0100, Matthias-Christian Ott wrote: > > >> An other problem is that each mail needs to be initially classified and > > >> due to the fact that sieve is not able to execute external programmes, > > >> deliver has to do this task. I'm currently thinking of possibilities to > > >> implement this, so far I came up with the following: > > >> > > >> 1. Write a generic pipe plugin which can execute an arbitrary number of > > >> programmes. The problem with this is that I'm not sure how to > > >> integrate this is in Dovecot's configuration file. I thought of > > >> something like this: pipe = prg1 | prg2 > > > > > > You do not need a plugin if you do the classification before the > > > delivery: MTA | classification | LDA > > > > With postfix, just add bogofilter as transport to master.cf and make it the > > 'mailbox_transport'. > > > > The problem with modifying the mail after reclassification persists, hope > > this > > can be solved. Then I could also finally move from > > postfix-procmail-bogofilter-cron-dovecot to > > postfix-deliver-antispam-bogofilter-dovecot. :) > > Hi, > > it's not quite clear to me how this would work with bogofilter as a > mailbox_transport - bogofilter isn't designed to do final delivery. > > It's also not quite clear to me why people would use procmail. Although > defended by its maintainers, it's an unusable and unconfigurable piece > of software from ancient past -- getting error handling right in > procmail is next to impossible, requires forfeiting :e rules and > bloating procmailrc with explicit error handling recipes. If you haven't puked today, just look at their source code and programming style. I think the configuration and exit code handling is not the worst problem in this software. > If you want something and Dovecot's deliver doesn't fit your needs, > consider maildrop, <http://www.courier-mta.org/maildrop/> Looks a bit bloated to me. > Bogofilter has an "integrating-with-postfix" document in the doc/ > directory that shows how to use Postfix's content_filter and does not > need procmail. Unfortunately, it does not show how to integrate updates; > there are several approaches to achieve that. One way is to use separate > mailboxes where users can send mail to and where they are picked up by > cron - best when using Dovecot is probably to make users move spam into > particular folders via IMAP. I read that document some days ago, but the content_filter approach looks strange to me, because they use sendmail to reinject the E-Mail in the queue. Regards, Matthias-Christian
Re: [Dovecot] Dovecot and Bogofilter
Patrick Nagel wrote: > Thorsten Vollmer wrote: > > On Sun, 2008-11-09 at 22:39 +0100, Matthias-Christian Ott wrote: > >> An other problem is that each mail needs to be initially classified and > >> due to the fact that sieve is not able to execute external programmes, > >> deliver has to do this task. I'm currently thinking of possibilities to > >> implement this, so far I came up with the following: > >> > >> 1. Write a generic pipe plugin which can execute an arbitrary number of > >> programmes. The problem with this is that I'm not sure how to > >> integrate this is in Dovecot's configuration file. I thought of > >> something like this: pipe = prg1 | prg2 > > > > You do not need a plugin if you do the classification before the > > delivery: MTA | classification | LDA > > With postfix, just add bogofilter as transport to master.cf and make it the > 'mailbox_transport'. Does Postfix understand the pipe syntax? If so everything would be fine. Regards, Matthias-Christian
[Dovecot] Dovecot and Bogofilter
Hi, on my small Xen-virtualised server with 48 MiB RAM I use Postfix and Dovecot, because the Debian administrators dislike qmail [1], which is in my opinion despite some maintainability and code quality issues a quite well designed software, because it mostly follows the UNIX principles. Postfix is not able to sort my E-Mail into different Maildir folders and after I looked at procmail's source code, I decided to use Dovecot's LDA, because it supports sieve via a plugin. All in all it worked after some tweaking and Dovecot is now responsible for authentication and delivery. Furthermore I want to setup a spam filter and due to the memory constraints I decided in favour of bogofilter (I made no real world tests, but the authors claim that it's fast and has a small memory footprint). My first idea was train bogofilter via a cronjob that runs bogofilter for each E-Mail in my spam folder and deletes them afterwards. Then I stumbled upon the dovecot antispam [2] plugin and quickly wrote a backend for bogofilter. During the development I realised that bogofilter has to update the X-Bogosity header of the reclassified E-Mails. Johannes Berg told me that he wasn't sure whether dovecot would be able to do this and recommended to ask this on the mailing list. An other problem is that each mail needs to be initially classified and due to the fact that sieve is not able to execute external programmes, deliver has to do this task. I'm currently thinking of possibilities to implement this, so far I came up with the following: 1. Write a generic pipe plugin which can execute an arbitrary number of programmes. The problem with this is that I'm not sure how to integrate this is in Dovecot's configuration file. I thought of something like this: pipe = prg1 | prg2 2. Write a bogofilter-specific plugin for this. Anyhow I would like to take this opportunity to criticise Dovecot for being a bloated (unnecessary abstractions, verbose, pseudo object-oriented, ...), non-suckless [3] and non-unixish (plugin architecture, monolithic design, ...) software which in return works quite well and stable so far. I appreciate any suggestions, ideas or criticism. I would really like to finish configuring my mail server as soon as possible. Regards, Matthias-Christian [1] http://smarden.org/pape/Debian/1215531259.4854_332.werc [2] http://johannes.sipsolutions.net/Projects/dovecot-antispam [3] http://www.suckless.org/common/
