Re: [Dovecot] Managesieve patch for Dovecot v1.2.10 on Dovecot v1.2.11

[Andreas Schulze]

Am 17.03.2010 17:45 schrieb Ivica Glavocic:
> Can i use Managesieve patch for Dovecot v1.2.10
> on new Dovecot version v1.2.11
at least: it compiles ;-)
I will try these days

-- 
Andreas Schulze
Internetdienste | P532

DATEV eG
90329 Nürnberg | Telefon +49 911 319-0 | Telefax +49 911 319-3196
E-Mail info @datev.de | Internet www.datev.de
Sitz: 90429 Nürnberg, Paumgartnerstr. 6-14 | Registergericht Nürnberg, GenReg 
Nr.70
Vorstand
Prof. Dieter Kempf (Vorsitzender)
Dipl.-Kfm. Wolfgang Stegmann (stellvertretender Vorsitzender)
Dipl.-Kfm. Michael Leistenschneider
Jörg Rabe v. Pappenheim
Dipl.-Vw. Eckhard Schwarzer
Vorsitzender des Aufsichtsrates: Reinhard Verholen



GnuPG-Signatur.asc
Description: digitale Signatur dieser Nachricht von Andreas Schulze

Re: [Dovecot] .mailboxlist location

[Dave Brenner]

On 3/16/2010 11:08 AM, Timo Sirainen wrote:

> mail_location = mbox:~/mail:CONTROL=~/:SUBSCRIPTIONS=.mailboxlist
>
> This works, because the only control file that mbox uses is the
> subscriptions file. In future this might cause some trouble if some
> other mbox control files show up..

This seems to work quite well.  We're using a passwd-file for 
userdb_mail fields and we had already recompiled for .mailboxlist, so 
all I had to add to the end of each line was :CONTROL=~/


Thank you.

Re: [Dovecot] quota usage

[Pascal Volk]

On 03/17/2010 11:24 PM Papp Tamás wrote:
> …
> The problem is that the mailbox is almost empty:
> $ du -sh .
> 228K.
> 
> But the quota usage show something else:
> 
> mysql> select bytes,messages from quota where username='';
> +---+--+
> | bytes | messages |
> +---+--+
> | 217699358 |  987 |
> +---+--+
> 
> What could be a problem, how can I fix this?

mysql> DELETE FROM quota WHERE username='';
mysql> \q
Bye

telnet host imap2
0 login  
1 GETQUOTAROOT INBOX
3 LOGOUT


Regards,
Pascal
-- 
The trapper recommends today: c01dcafe.1007...@localdomain.org

Re: [Dovecot] v1.2.11 released (managesieve updated)

[Stephan Bosch]

Stephan Bosch wrote:

Timo Sirainen wrote:

http://dovecot.org/releases/1.2/dovecot-1.2.11.tar.gz
http://dovecot.org/releases/1.2/dovecot-1.2.11.tar.gz.sig

mbox users really should upgrade, because by sending a message with a
huge header you could basically cause a DoS (this problem exists only
with v1.2.x, not with v1.0 or v1.1).

- mbox: Message header reading was unnecessarily slow. Fetching a
  huge header could have resulted in Dovecot eating a lot of CPU.
  Also searching messages was much slower than necessary.
- mbox, dbox, cydir: Mail root directory was created with 0770
  permissions, instead of 0700.
- maildir: Reading uidlist could have ended up in an infinite loop.
- IMAP IDLE: v1.2.7+ caused extra load by checking changes every
  0.5 seconds after a change had occurred in mailbox



I have a paper deadline this Friday, so a new release of Pigeonhole 
will be delayed until this weekend.



I've refreshed the ManageSieve patch for v1.2:

http://www.rename-it.nl/dovecot/1.2/dovecot-1.2.11-managesieve-0.11.11.diff.gz
http://www.rename-it.nl/dovecot/1.2/dovecot-1.2.11-managesieve-0.11.11.diff.gz.sig

Sieve release will take more time.

Regards,

Stephan.

Re: [Dovecot] I stream read - stale NFS file handle (reboot of server)

[Joe]

On 3/17/2010 2:59 PM, Edgar Fuß wrote:

These web servers also use the nullfs mount which is actually a NFS
mount via the host machine to get the content that it serves to the
reverse proxy.
 

But Web servers usually don't delete or rename files, do they?
   
Some can depending on whether any of the sites use php's session 
handler. Generally speaking, none of the sites i host do any 
deleting/renaming of files. I should have mentioned, for what its worth, 
that i also use cronolog to manage the apache log files.

[Dovecot] quota usage

[Papp Tamás]

hi All!

Dovecot's version is 1.2.11-0~auto+0 from http://xi.rename-it.nl/debian/ .

dovecot.conf:

dict {
 quota = mysql:/data/apps/dovecot/dict-sql.conf
 #expire = db:/var/lib/dovecot/expire.db
}


/data/apps/dovecot/dict-sql.conf:

connect = host=127.0.0.1 dbname=db user=user password=pw

map {
 pattern = priv/quota/storage
 table = quota
 username_field = username
 value_field = bytes
}
map {
 pattern = priv/quota/messages
 table = quota
 username_field = username
 value_field = messages
}

The problem is that the mailbox is almost empty:
$ du -sh .
228K.

But the quota usage show something else:

mysql> select bytes,messages from quota where username='';
+---+--+
| bytes | messages |
+---+--+
| 217699358 |  987 |
+---+--+

What could be a problem, how can I fix this?

Thank you very much,

tamas

Re: [Dovecot] Overly long email of miscellaneous Dovecot migration questions

[Mark Moseley]

On Wed, Mar 17, 2010 at 1:09 PM, Timo Sirainen  wrote:
> On Wed, 2010-03-17 at 12:15 -0700, Mark Moseley wrote:

>> From the wiki and from the
>> thread, it sounds like this just affects index files. One thing I
>> didn't see in the thread (though it'd be easy to miss in a thread that
>> long) is whether performance suffered horribly with 'noac' on a
>> *separate* index mount (the mentions I can find where they tried
>> 'noac', it was on the entire mail store, not a separate index mount)
>> -- though doing 'noac' on anything seems like a recipe for disaster.
>
> noac increases the NFS traffic for indexes by something like 10x,
> regardless of how it's mounted.

Yeah, this would be something of a last act of desperation. Hopefully
if it comes to that, having a separate indexes-only NFS mount that's
the *only* thing mounted with noac would make it barely tolerable. Is
it safe to say that with 'noac' but using 'sync' wouldn't help? And
I'm assuming it's file attributes being cached that leads to
corruptions, so would leaving acdirmin/max at their default values but
lowering acregmax to 1 (not sure if 0 disables it completely) prevent
corruption in all but the most extremely concurrent cases? Again this
would be a last ditch thing, esp considering how hot our Filers run
already -- adding another chunk of cpu% would put the nail in the
coffin.

It sounds like a local cache would be far preferable. Since it's
*just* IMAP, I'm guessing it won't be too huge. I'm thinking I
dedicate a local partition, mount it *with* atime, and then cron
something to delete all index files that haven't been accessed in a
certain amount of time (a week, 2 weeks, a month, whatever). I'd use
something like this:
mail_location=:INDEX=/path/to/index/partition/Indexes/%2Mu/%2.2Mu/%u
to keep from getting too many entries in a single
directory/subdirectory.


>> Another extreme option might be to just keep indexes on local disk
>> only. Obviously users hitting another server wouldn't benefit from the
>> other server and it'd incur updates to the indexes but they seem
>> somewhat minimal -- in sane cases at least where the number of
>> newer-than-the-index messages is reasonable. With aggressive load
>> balancer stickiness at maybe a /24 netblock level, I'd be able to keep
>> people localized for a little while (with no way around people
>> accessing the same mailbox from home and work). Am I underestimating
>> the costs of incremental index updates on local disks? In testing, tt
>> seemed like dovecot was pretty conservative in disk accesses/reads
>> when updating. I'm hoping the 95%-of-the-time worst case will be
>> someone hitting the same mailbox from two locations, so they'd be
>> hitting at most two servers. With local index caches, I'm mildly sure
>> I could live with that (esp if I can talk my way into an SSD on each
>> one). Am I being too naive?
>
> I don't know if anyone has run local indexes in larger setups, so I
> can't really give any good answers. The worst case of rebuilding the
> whole index isn't anyway any worse than what Courier has to do every
> time.

I'll check it out. And if we go that route and I'm still sane at the
end, I'll be sure to report back :)


>> I'll play around with that more. So it should be possible to have more
>> than one private 'hidden=no; list=yes' namespace that'd show up in a
>> mail client's subscribable folders list?
>
> Yes. And probably hidden=yes is better, since it doesn't really need to
> show up as a separate namespace for clients (even though most would
> ignore it anyway, but some might show it a bit weirdly).
>
>> Offhand, before I start sifting
>> through source code, is Dovecot on a 32-bit system limited to a 32-bit
>> -- i.e. 2 gig -- limit on quota size like Courier is?
>
> Quota is always 64bit. And Dovecot usually tries to enable 64 bit file
> offsets for 32 bit systems too, and then there aren't any 32 bit file
> size limits anywhere.

Excellent. Not that I really want to give out +2gb mailboxes but now I
won't have to explain for the 30th time to marketing folks why I can't
raise someone over the 2gig quota.


>> >> If we're going to have to live with
>> >> users complaining about a one-time redownload of just post-conversion
>> >> mail, I'll need to get started convincing the higher-ups that that's
>> >> life.
>> >
>> > Check what the new Courier POP3 UIDLs look like. If all of them are 
>> > maildir base filename, setting pop3_uidl_format=%f should make this 
>> > conversion easy. Just run it through once to get old UIDLs converted and 
>> > then you don't need to worry about it, because both Courier and Dovecot 
>> > give the same UIDLs. But I never really understood how Courier assigns new 
>> > UIDLs, sometimes it seems to use the filename and sometimes not..
>>
>> So do you mean keep pop3_uidl_format=%f basically forever? That'd work
>> for me, but are there any big implications, performance-wise or other,
>> of not using the default "%08Xu%08Xv"?
>
> %f is at least

Re: [Dovecot] dovecot 2.0beta3 installation

[Timo Sirainen]

On Wed, 2010-03-17 at 21:45 +0100, Renaud Allard wrote:

> > userdb {
> >driver = passwd
> > }
> > userdb {
> >args = /etc/passwd
> >driver = passwd-file
> > }
..
> userdb {
>driver = passwd
>args = /etc/passwd
> }
> 
> 
> Is there really a need for 2 userdb config groups?

No, and using /etc/passwd as passwd-file isn't really a good idea
either. But you should remove args=/etc/passwd from userdb passwd, it's
not needed.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] dovecot 2.0beta3 installation

[Renaud Allard]

On 17/03/10 21:25, Timo Sirainen wrote:


Recent hg versions convert v1.2 configs to v2.0 configs automatically.
That is what it outputs with your config:

auth_mechanisms = digest-md5 cram-md5 ntlm login plain
passdb {
   args = /etc/cram.pwd
   driver = passwd-file
}
service auth {
   unix_listener /var/run/dovecot/auth-client {
 mode = 0666
   }
   user = exim
}
userdb {
   driver = passwd
}
userdb {
   args = /etc/passwd
   driver = passwd-file
}



I see a difference with what I have configured.
My config is:
auth_mechanisms = digest-md5 cram-md5 ntlm login plain
passdb {
  driver = passwd-file
  args = /etc/cram.pwd
}
userdb {
  driver = passwd
  args = /etc/passwd
}


Is there really a need for 2 userdb config groups?
Reading mails was working fine, only sending was causing problems.



smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Dovecot] dovecot 2.0beta3 installation

[Renaud Allard]

On 17/03/10 21:25, Timo Sirainen wrote:

Thanks Timo, so I suppose the daily snapshot should be better for me.


So do you mean you really tried beta3 release, not the nightly
snapshot / hg version? Because I thought that error didn't happen with
beta3.


Yes, I really tried beta3
I am also using sieve in dovecot-2-0-pigeonhole-35a382739679, maybe this 
is the culprit?




Recent hg versions convert v1.2 configs to v2.0 configs automatically.


That's really a nice feature I must admit.




service auth {
unix_listener /var/run/dovecot/auth-client {
  mode = 0666
}
}


I don't really see anything wrong with that..


Neither do I (except maybe for the user), hence my question.



smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Dovecot] Overly long email of miscellaneous Dovecot migration questions

[Timo Sirainen]

On Wed, 2010-03-17 at 13:27 -0700, Mark Moseley wrote:

> I'll definitely keep that in mind. I should be able to keep things
> pretty segregated in terms of POP3 alone or IMAP alone but my big
> worry is that Courier POP3+Dovecot IMAP scenario. 

I know a large installation that was (is?) using Dovecot IMAP + Courier
POP3 for years. That works fine. Or there were some problems, but I
think they were mainly related to some external tools that assumed
things worked the way Dovecot did them, but Courier messed up that
expectation.. I can't remember specifics.

Anyway afterward when migrating POP3 to Dovecot, the migration script
should also work fine as long as you delete courierimapuidlist files
before running it, so that it'll only try to preserve POP3 UIDLs. Of
course, testing it well is still a good idea..



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] Overly long email of miscellaneous Dovecot migration questions

[Mark Moseley]

On Wed, Mar 17, 2010 at 12:43 PM, Tony Rutherford  wrote:
> On 3/16/2010 7:36 PM, Timo Sirainen wrote:
>>
>> On 17.3.2010, at 1.01, Mark Moseley wrote:
>>
>>
>>>
>>> * Since Dovecot 2.0 seems like it's just around the corner, that's all
>>> I've been testing, and indeed all I've even looked at.
>>>
>>
>> Yes, hopefully it's coming soon :)
>>
>>
>>>
>>> * Our #1 main motivation for looking Dovecot is relief for our
>>> currently overtaxed NFS servers, mostly in the form of the index
>>> files. Benchmarking dovecot looks great, even with the index files in
>>> the maildir.
>>>
>>
>> Have you read the thread starting from
>> http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a
>> month or so? That provides a good view of potential problems with NFS.
>>
>>
>>>
>>> * Exim: We currently deliver all of our mail via Exim on separate
>>> servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail
>>> servers delivering to maildirs only do Exim. From what I've seen in
>>> the docs and various threads, from what I can gather, the best thing
>>> to do in that case would be to use Exim's built-in maildir handling,
>>> instead of using 'deliver'. That would be my preference anyway, but I
>>> wanted to make sure I didn't misinterpret things.
>>>
>>
>> v2.0 supports also LMTP server, so you could deliver to Dovecot that way.
>>
>>
>>>
>>> * Any problems running Courier POP3 and Dovecot IMAP for a while,
>>> possibly Courier IMAP and Dovecot IMAP concurrently?
>>>
>>
>> Courier POP + Dovecot IMAP is fine. But concurrently running both POPs or
>> both IMAPs is just going to cause trouble because of conflicting UIDs. You
>> might be able to make both Dovecot and Courier use the same POP3 UIDLs, but
>> I wouldn't really trust it.
>>
>> One possibility would be to just run the migration script on login, so
>> users would migrate to Dovecot as they log in.
>>
>>
>
> I would add a big warning as well to running Courier and Dovecot
> concurrently.  If users are strictly segregated to using one or the other
> (consistently), then strictly running Courier and Dovecot on the same
> machine is not an issue (on different ports or whatever).  But, I would NOT
> recommend having it such that a user could hit Courier one time and Dovecot
> another...I think that is begging for big trouble.  As Timo mentions, most
> of the trouble will be with syncing the UIDs.  Even running POP Courier and
> Dovecot IMAP might be a problem...especially if your users can use both POP
> and IMAPCourier has 2 uidl files,  (at least the version we were
> migrating from)..one for POP and the other for IMAP, and they get combined
> to one in Dovecot (dovecot-uidlist).  Then if a user comes in via
> Dovecot...and updates the dovecot-uidlist...by definition the Courier files
> would be out of date.  I would really steer away from that approach.

I'll definitely keep that in mind. I should be able to keep things
pretty segregated in terms of POP3 alone or IMAP alone but my big
worry is that Courier POP3+Dovecot IMAP scenario. I'm thinking
especially of people using POP3 in one place and webmail in another
location. I'm guessing that people using POP3 in one location but
direct IMAP in another are fairly rare. Whatever the case,  it sounds
like I need to do some log analysis to see what % of people access in
both ways. Thanks for the advice!

Re: [Dovecot] dovecot 2.0beta3 installation

[Timo Sirainen]

On Wed, 2010-03-17 at 21:21 +0100, Renaud Allard wrote:
> >> dovecot_delivery transport output: lda: Fatal: User lookup failed:
> >> net_connect_unix(/opt/dovecot/etc/dovecot/dovecot.conf) failed: No such
> >> file or directory
> >
> > Yeah, I broke this yesterday. Fixed it today.
> >
> 
> Thanks Timo, so I suppose the daily snapshot should be better for me.

So do you mean you really tried beta3 release, not the nightly
snapshot / hg version? Because I thought that error didn't happen with
beta3.

> Any idea on the second part of my mail?
> I didn't really find out how to convert the v1.2 config snippet:

Recent hg versions convert v1.2 configs to v2.0 configs automatically.
That is what it outputs with your config:

auth_mechanisms = digest-md5 cram-md5 ntlm login plain
passdb {
  args = /etc/cram.pwd
  driver = passwd-file
}
service auth {
  unix_listener /var/run/dovecot/auth-client {
mode = 0666
  }
  user = exim
}
userdb {
  driver = passwd
}
userdb {
  args = /etc/passwd
  driver = passwd-file
}

> I tried the following, but it does not seem to work as I get an 
> authentication socket protocol error
> 
> service auth {
>unix_listener /var/run/dovecot/auth-client {
>  mode = 0666
>}
> }

I don't really see anything wrong with that..


signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] Overly long email of miscellaneous Dovecot migration questions

[Timo Sirainen]

On Wed, 2010-03-17 at 13:19 -0700, Mark Moseley wrote:

> Since Exim wouldn't be touching the index files, is it safe to leave
> exim as-is and let it handle the deliveries to the maildirs natively?
> Exim's already got access to everything it needs including the quota.
> I just want to make sure I won't horribly corrupt anything leaving it
> as-is -- but I'm also desirous to not have to mess with how deliveries
> are currently being done.

That's fine.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] dovecot 2.0beta3 installation

[Renaud Allard]

On 17/03/10 21:11, Timo Sirainen wrote:

On Wed, 2010-03-17 at 21:09 +0100, Renaud Allard wrote:


dovecot_delivery transport output: lda: Fatal: User lookup failed:
net_connect_unix(/opt/dovecot/etc/dovecot/dovecot.conf) failed: No such
file or directory


Yeah, I broke this yesterday. Fixed it today.



Thanks Timo, so I suppose the daily snapshot should be better for me.

Any idea on the second part of my mail?
I didn't really find out how to convert the v1.2 config snippet:
auth default {
  mechanisms = digest-md5 cram-md5 ntlm login plain
  passdb passwd-file {
args = /etc/cram.pwd
  }
  userdb passwd {
  }
  userdb passwd-file {
args = /etc/passwd
  }
  socket listen {
client {
  path = /var/run/dovecot/auth-client
  mode = 0666
}
  }
  user = exim
}

I tried the following, but it does not seem to work as I get an 
authentication socket protocol error


service auth {
  unix_listener /var/run/dovecot/auth-client {
mode = 0666
  }
}



smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Dovecot] Overly long email of miscellaneous Dovecot migration questions

[Mark Moseley]

Oops, forgot to ask one other thing

On Tue, Mar 16, 2010 at 4:36 PM, Timo Sirainen  wrote:
> On 17.3.2010, at 1.01, Mark Moseley wrote:

>> * Exim: We currently deliver all of our mail via Exim on separate
>> servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail
>> servers delivering to maildirs only do Exim. From what I've seen in
>> the docs and various threads, from what I can gather, the best thing
>> to do in that case would be to use Exim's built-in maildir handling,
>> instead of using 'deliver'. That would be my preference anyway, but I
>> wanted to make sure I didn't misinterpret things.
>
> v2.0 supports also LMTP server, so you could deliver to Dovecot that way.

Since Exim wouldn't be touching the index files, is it safe to leave
exim as-is and let it handle the deliveries to the maildirs natively?
Exim's already got access to everything it needs including the quota.
I just want to make sure I won't horribly corrupt anything leaving it
as-is -- but I'm also desirous to not have to mess with how deliveries
are currently being done.

Re: [Dovecot] dovecot 2.0beta3 installation

[Timo Sirainen]

On Wed, 2010-03-17 at 21:09 +0100, Renaud Allard wrote:

> dovecot_delivery transport output: lda: Fatal: User lookup failed: 
> net_connect_unix(/opt/dovecot/etc/dovecot/dovecot.conf) failed: No such 
> file or directory

Yeah, I broke this yesterday. Fixed it today.



signature.asc
Description: This is a digitally signed message part

Re: [Dovecot] Overly long email of miscellaneous Dovecot migration questions

[Timo Sirainen]

On Wed, 2010-03-17 at 12:15 -0700, Mark Moseley wrote:
> >> * Our #1 main motivation for looking Dovecot is relief for our
> >> currently overtaxed NFS servers, mostly in the form of the index
> >> files. Benchmarking dovecot looks great, even with the index files in
> >> the maildir.
> >
> > Have you read the thread starting from 
> > http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a 
> > month or so? That provides a good view of potential problems with NFS.
> 
> I'd read through that thread and subsequently forgot the implications.
> Presumably the same pitfalls apply to 2.0? 

Yes.

> From the wiki and from the
> thread, it sounds like this just affects index files. One thing I
> didn't see in the thread (though it'd be easy to miss in a thread that
> long) is whether performance suffered horribly with 'noac' on a
> *separate* index mount (the mentions I can find where they tried
> 'noac', it was on the entire mail store, not a separate index mount)
> -- though doing 'noac' on anything seems like a recipe for disaster.

noac increases the NFS traffic for indexes by something like 10x,
regardless of how it's mounted.

> Another extreme option might be to just keep indexes on local disk
> only. Obviously users hitting another server wouldn't benefit from the
> other server and it'd incur updates to the indexes but they seem
> somewhat minimal -- in sane cases at least where the number of
> newer-than-the-index messages is reasonable. With aggressive load
> balancer stickiness at maybe a /24 netblock level, I'd be able to keep
> people localized for a little while (with no way around people
> accessing the same mailbox from home and work). Am I underestimating
> the costs of incremental index updates on local disks? In testing, tt
> seemed like dovecot was pretty conservative in disk accesses/reads
> when updating. I'm hoping the 95%-of-the-time worst case will be
> someone hitting the same mailbox from two locations, so they'd be
> hitting at most two servers. With local index caches, I'm mildly sure
> I could live with that (esp if I can talk my way into an SSD on each
> one). Am I being too naive?

I don't know if anyone has run local indexes in larger setups, so I
can't really give any good answers. The worst case of rebuilding the
whole index isn't anyway any worse than what Courier has to do every
time.

> I'll play around with that more. So it should be possible to have more
> than one private 'hidden=no; list=yes' namespace that'd show up in a
> mail client's subscribable folders list?

Yes. And probably hidden=yes is better, since it doesn't really need to
show up as a separate namespace for clients (even though most would
ignore it anyway, but some might show it a bit weirdly).

> Offhand, before I start sifting
> through source code, is Dovecot on a 32-bit system limited to a 32-bit
> -- i.e. 2 gig -- limit on quota size like Courier is?

Quota is always 64bit. And Dovecot usually tries to enable 64 bit file
offsets for 32 bit systems too, and then there aren't any 32 bit file
size limits anywhere.

> >> If we're going to have to live with
> >> users complaining about a one-time redownload of just post-conversion
> >> mail, I'll need to get started convincing the higher-ups that that's
> >> life.
> >
> > Check what the new Courier POP3 UIDLs look like. If all of them are maildir 
> > base filename, setting pop3_uidl_format=%f should make this conversion 
> > easy. Just run it through once to get old UIDLs converted and then you 
> > don't need to worry about it, because both Courier and Dovecot give the 
> > same UIDLs. But I never really understood how Courier assigns new UIDLs, 
> > sometimes it seems to use the filename and sometimes not..
> 
> So do you mean keep pop3_uidl_format=%f basically forever? That'd work
> for me, but are there any big implications, performance-wise or other,
> of not using the default "%08Xu%08Xv"? 

%f is at least more reliable, because it's not fully guaranteed that
UIDVALIDITY+UID never changes (although in practice they don't). If they
do, then POP3 UIDL changes and clients redownload mail. The only
disadvantage is the somewhat larger UIDLs causing a bit more and network
bandwidth usage.

It's also possible to set pop3_save_uidl=yes, and that's probably a good
idea also, so that whenever UIDL is sent to client, it's also saved to
dovecot-uidlist. That allows you to later change pop3_uidl_format
without changing existing UIDLs.

> Does that require a
> corresponding change to the conversion script? 

No.

> So I'd have Dovecot use
> the Courier-style UIDLs, run the conversion to get the old UIDLs (i.e.
> anything not using the Courier "%f", which is a bit) into
> dovecot-uidlist, and then not worry about any new mail, since clients
> will see the same UIDLs (and dovecot will just update dovecot-uidlist
> with those UIDLs the first time they log into Dovecot's POP3)?

Yeah.


signature.asc
Description: This is a digitally signed message part

[Dovecot] dovecot 2.0beta3 installation

[Renaud Allard]

Hello,

I tried to install dovecot 2.0 beta3 and deliver seems to look for its 
config file in unusual places.

In my exim logs, I get:
010-03-17 13:39:24 [31257] 1NrsWt-0004T0-1K : 
dovecot_delivery transport output: lda: Fatal: User lookup failed: 
net_connect_unix(/opt/dovecot/etc/dovecot/dovecot.conf) failed: No such 
file or directory
2010-03-17 13:39:24 [31257] 1NrsWt-0004T0-1K == em...@example.com 
(em...@example.com)  R=local_user T=dovecot_delivery 
defer (0): Child process of dovecot_delivery transport returned 75 
(could mean temporary error) from command: 
/opt/dovecot/libexec/dovecot/deliver


Dovecot have been started with the following command:
/opt/dovecot/sbin/dovecot -F -c /etc/dovecot/dovecot2.conf
Furthermore, it has been compiled with:
--prefix=/opt/dovecot --sysconfdir=/etc
I don't see why it searches for /opt/dovecot/etc/dovecot/dovecot.conf



Also, I didn't really find out how to convert the v1.2 config snippet:
auth default {
  mechanisms = digest-md5 cram-md5 ntlm login plain
  passdb passwd-file {
args = /etc/cram.pwd
  }
  userdb passwd {
  }
  userdb passwd-file {
args = /etc/passwd
  }
  socket listen {
client {
  path = /var/run/dovecot/auth-client
  mode = 0666
}
  }
  user = exim
}

I tried the following, but it does not seem to work as I get an 
authentication socket protocol error


service auth {
  unix_listener /var/run/dovecot/auth-client {
mode = 0666
  }
}

Thanks for you help



smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Dovecot] Overly long email of miscellaneous Dovecot migration questions

[Tony Rutherford]

On 3/16/2010 7:36 PM, Timo Sirainen wrote:

On 17.3.2010, at 1.01, Mark Moseley wrote:

   

* Since Dovecot 2.0 seems like it's just around the corner, that's all
I've been testing, and indeed all I've even looked at.
 

Yes, hopefully it's coming soon :)

   

* Our #1 main motivation for looking Dovecot is relief for our
currently overtaxed NFS servers, mostly in the form of the index
files. Benchmarking dovecot looks great, even with the index files in
the maildir.
 

Have you read the thread starting from 
http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a month 
or so? That provides a good view of potential problems with NFS.

   

* Exim: We currently deliver all of our mail via Exim on separate
servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail
servers delivering to maildirs only do Exim. From what I've seen in
the docs and various threads, from what I can gather, the best thing
to do in that case would be to use Exim's built-in maildir handling,
instead of using 'deliver'. That would be my preference anyway, but I
wanted to make sure I didn't misinterpret things.
 

v2.0 supports also LMTP server, so you could deliver to Dovecot that way.

   

* Any problems running Courier POP3 and Dovecot IMAP for a while,
possibly Courier IMAP and Dovecot IMAP concurrently?
 

Courier POP + Dovecot IMAP is fine. But concurrently running both POPs or both 
IMAPs is just going to cause trouble because of conflicting UIDs. You might be 
able to make both Dovecot and Courier use the same POP3 UIDLs, but I wouldn't 
really trust it.

One possibility would be to just run the migration script on login, so users 
would migrate to Dovecot as they log in.

   
I would add a big warning as well to running Courier and Dovecot 
concurrently.  If users are strictly segregated to using one or the 
other (consistently), then strictly running Courier and Dovecot on the 
same machine is not an issue (on different ports or whatever).  But, I 
would NOT recommend having it such that a user could hit Courier one 
time and Dovecot another...I think that is begging for big trouble.  As 
Timo mentions, most of the trouble will be with syncing the UIDs.  Even 
running POP Courier and Dovecot IMAP might be a problem...especially if 
your users can use both POP and IMAPCourier has 2 uidl files,  (at 
least the version we were migrating from)..one for POP and the other for 
IMAP, and they get combined to one in Dovecot (dovecot-uidlist).  Then 
if a user comes in via Dovecot...and updates the dovecot-uidlist...by 
definition the Courier files would be out of date.  I would really steer 
away from that approach.

* Union mailboxes: I'm pretty sure in a fairly recent thread that Timo
said that something like a 'union' mailbox (at least with maildir)
wasn't possible.
 

I also had a thought where you could do that if you wrote some scripts and such 
that copied the mails to the other storage and replaced the original file with 
a symlink. But that of course has some potential race conditions and other 
problems with either side of the symlink disappearing but the other not.

Single-dbox would really be the best solution for this in future. It's 
currently somewhat broken in v2.0 tree, but it probably won't be too long until 
I'm going to start doing a migration from Maildir to dbox for a similar NFS 
system than yours.

   

I tried messing with multiple 'private' namespaces
(i.e. a namespace called "ARCHIVE" with a "location" different than
the INBOX location, ideally placed on slower but denser NFS servers)
but even with 'hidden=no' and 'list=yes', only the main INBOX folders
would show up, so I'm guessing that's not going to work.
 

You can create more namespaces, so I guess that was some kind of a 
configuration problem.

   

That would be
a killer feature, to be able to serve an alternate namespace that
would show up in a mail client's subscribable list that could be on
slower storage than the main inbox (though I'm not sure a mail client
can even handle multiple namespaces).
 

The clients don't need to be aware of namespaces. And you should be able to do 
that already. But do you think users would actually move their mails to there?

   

* Any problems with keeping only quota limits in db and not current
quota numbers? Our limits come out of a SQL table but  the current
counts just live in the maildir file.
 

That's how most people do it.

   

* Any problem with leaving the namespace in "Courier compatibility"
mode? I.e. in namespace 'private', leaving "prefix = INBOX.". With 4
million mailboxes, FAQs all over the place, and support reps trained
in a particular way of doing things in IMAP, it'd be hellish to try to
change the prefix (I know I could leave the courier namespace around
with 'hidden=yes' but retraining support staff is perhaps better left
to phase #2). Do I lose anything besides tidiness by not changing it
to "prefix =" as if I was deploying d

Re: [Dovecot] Overly long email of miscellaneous Dovecot migration questions

[Mark Moseley]

First off, thanks for the reply. I appreciate it greatly!

On Tue, Mar 16, 2010 at 4:36 PM, Timo Sirainen  wrote:
> On 17.3.2010, at 1.01, Mark Moseley wrote:
>
>> * Since Dovecot 2.0 seems like it's just around the corner, that's all
>> I've been testing, and indeed all I've even looked at.
>
> Yes, hopefully it's coming soon :)
>
>> * Our #1 main motivation for looking Dovecot is relief for our
>> currently overtaxed NFS servers, mostly in the form of the index
>> files. Benchmarking dovecot looks great, even with the index files in
>> the maildir.
>
> Have you read the thread starting from 
> http://dovecot.org/list/dovecot/2010-January/046106.html and spanning a month 
> or so? That provides a good view of potential problems with NFS.

I'd read through that thread and subsequently forgot the implications.
Presumably the same pitfalls apply to 2.0? From the wiki and from the
thread, it sounds like this just affects index files. One thing I
didn't see in the thread (though it'd be easy to miss in a thread that
long) is whether performance suffered horribly with 'noac' on a
*separate* index mount (the mentions I can find where they tried
'noac', it was on the entire mail store, not a separate index mount)
-- though doing 'noac' on anything seems like a recipe for disaster.
Another extreme option might be to just keep indexes on local disk
only. Obviously users hitting another server wouldn't benefit from the
other server and it'd incur updates to the indexes but they seem
somewhat minimal -- in sane cases at least where the number of
newer-than-the-index messages is reasonable. With aggressive load
balancer stickiness at maybe a /24 netblock level, I'd be able to keep
people localized for a little while (with no way around people
accessing the same mailbox from home and work). Am I underestimating
the costs of incremental index updates on local disks? In testing, tt
seemed like dovecot was pretty conservative in disk accesses/reads
when updating. I'm hoping the 95%-of-the-time worst case will be
someone hitting the same mailbox from two locations, so they'd be
hitting at most two servers. With local index caches, I'm mildly sure
I could live with that (esp if I can talk my way into an SSD on each
one). Am I being too naive?


>> * Exim: We currently deliver all of our mail via Exim on separate
>> servers. Our POP3/IMAP servers only do POP3/IMAP and the Exim mail
>> servers delivering to maildirs only do Exim. From what I've seen in
>> the docs and various threads, from what I can gather, the best thing
>> to do in that case would be to use Exim's built-in maildir handling,
>> instead of using 'deliver'. That would be my preference anyway, but I
>> wanted to make sure I didn't misinterpret things.
>
> v2.0 supports also LMTP server, so you could deliver to Dovecot that way.
>
>> * Any problems running Courier POP3 and Dovecot IMAP for a while,
>> possibly Courier IMAP and Dovecot IMAP concurrently?
>
> Courier POP + Dovecot IMAP is fine. But concurrently running both POPs or 
> both IMAPs is just going to cause trouble because of conflicting UIDs. You 
> might be able to make both Dovecot and Courier use the same POP3 UIDLs, but I 
> wouldn't really trust it.
>
> One possibility would be to just run the migration script on login, so users 
> would migrate to Dovecot as they log in.

Good to know. I'll definitely limit everything IMAP to just dovecot
then (POP3 was a given). Looking back, I forgot about the fact that
folder subscriptions could get out of whack too, in a mixed IMAP
environment, if someone updated their subscriptions on one platform.


>> * Union mailboxes: I'm pretty sure in a fairly recent thread that Timo
>> said that something like a 'union' mailbox (at least with maildir)
>> wasn't possible.
>
> I also had a thought where you could do that if you wrote some scripts and 
> such that copied the mails to the other storage and replaced the original 
> file with a symlink. But that of course has some potential race conditions 
> and other problems with either side of the symlink disappearing but the other 
> not.
>
> Single-dbox would really be the best solution for this in future. It's 
> currently somewhat broken in v2.0 tree, but it probably won't be too long 
> until I'm going to start doing a migration from Maildir to dbox for a similar 
> NFS system than yours.

My main concern with doing anything with symlinks is how easy it'd be
for users to inadvertently undo, by moving messages around or
deleting/recreating folders.


>> I tried messing with multiple 'private' namespaces
>> (i.e. a namespace called "ARCHIVE" with a "location" different than
>> the INBOX location, ideally placed on slower but denser NFS servers)
>> but even with 'hidden=no' and 'list=yes', only the main INBOX folders
>> would show up, so I'm guessing that's not going to work.
>
> You can create more namespaces, so I guess that was some kind of a 
> configuration problem.

I'll play around with that more. So it should

Re: [Dovecot] Testing EXTERNAL AUTHENTICATION

[Stephen Feyrer]

Hi.

It works, I'm in!  Authentication mechanism set to External and five  
colons after {PLAIN} the command "a AUTHENTICATE EXTERNAL =" worked.


It was brute force that did the trick, after reading as much as I could  
find about /etc/passwd file formats I was still none the wiser.


passwd file looks like this:
Stephen:{PLAIN}:nopassword=y

Anyway the result is below:-

---
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE  
AUTH=EXTERNAL] Dovecot ready.

a AUTHENTICATE EXTERNAL =
a OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT  
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE  
CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC  
ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS] Logged in

a list "" *
* LIST (\HasNoChildren) "/" "INBOX"
a OK List completed.
a select INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags  
permitted.

* 0 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1268850687] UIDs valid
* OK [UIDNEXT 1] Predicted next UID
* OK [HIGHESTMODSEQ 1] Highest
a OK [READ-WRITE] Select completed.
DONE


Perhaps in the great game of snakes and ladders, I have now finally  
reached square one?


--
A big Thank YOU!

Stephen Feyrer.

On Tue, 16 Mar 2010 23:10:57 -, Timo Sirainen  wrote:


On 17.3.2010, at 1.09, Stephen Feyrer wrote:


Hi.

I tried:

Stephen:{EXTERNAL}nopassword=y


{EXTERNAL} is never going to work anywhere, because there's no such  
password scheme.




and

Stephen:{PLAIN}nopassword=y


This is the wrong format. It's more like:

Stephen:{PLAIN}nopassword=y

Just figure out the correct number of : characters (based on the wiki  
page I gave or just brute force).

Re: [Dovecot] I stream read - stale NFS file handle (reboot of server)

[Edgar Fuß]

> These web servers also use the nullfs mount which is actually a NFS
> mount via the host machine to get the content that it serves to the
> reverse proxy.
But Web servers usually don't delete or rename files, do they?

[Dovecot] Managesieve patch for Dovecot v1.2.10 on Dovecot v1.2.11

[Ivica Glavocic]

Hi all

Can i use Managesieve patch for Dovecot v1.2.10
http://www.rename-it.nl/dovecot/1.2/dovecot-1.2.10-managesieve-0.11.11.diff.gz

on new Dovecot version v1.2.11
http://www.dovecot.org/releases/1.2/dovecot-1.2.11.tar.gz ?

Is Dovecot gonna work correctly after compiled with SIEVE support and 
this patch, or do I have to use Dovecot version 1.2.10?


Thanks, regards

*Ivica Glavočić*

Laser Line d.o.o.
Tribje 17, 52470 Umag
tel.: +385 52 725 600
fax: +385 52 725 610
OIB: 26680017138
mail: ivica.glavo...@laserline.hr 
mail: s...@laserline.hr 
web: http://www.laserline.hr

Re: [Dovecot] INBOX of a shared namespace appears to be always subscribed

[Holger Richter]

There was an other problem. I forgot to remove the .INBOX  
subdirectories. They were a rudiment of earlier tests. Well, I've  
removed them, and the original problem is resolved. But now I run into  
another trouble. The users cannot subscribe to the inboxes other  
users. The inbox paths in the shared namespace point to the namespace  
prefixes of the users' shared namespace. And they are marked with the  
\Noselect flag. Thus the inboxes are unaccessible via the shared  
namespace.


BTW, I've set subscriptions=no in the config file, dovecot -n lists  
the subscription parameter only if it was set to yes.


Holger

Re: [Dovecot] sieve fileinto rule (pigeonhole)

[Oliver Eales]

Stephan Bosch schrieb:
> Oliver Eales wrote:
>>
>> the mailbox gets created but not subscribed to. (I didn't make any
>> difference if i use the :create sieve command or leave or it out)
>>
>> Am i doing something wrong ?
>>   
> Not that I can see. I've tested the behavior at my end and all works
> well. Note that the subscription is only performed when the folder is
> actually created, not when it already exists. I must say, if this is
> truly a bug somehow, I am not quite sure how to debug this. You should
> post your dovecot -n output here so that we can check your config. 
Hello Stephan,
sorry for the false alarm. I tried to reproduce it today and everything
was working. No clue what i messed up last time...

Thanks for your help!

Regards,
Oliver Eales
 

Re: [Dovecot] qmail-secretary plugin for dovecot deliver

[Rajkumar S]

On Tue, Mar 16, 2010 at 9:14 PM, Timo Sirainen  wrote:
> I haven't tried to really understand what your plugin does, but looks
> like it does most of its work in mail_alloc(). deliver does the mail
> delivery via:
>
> typedef int deliver_mail_func_t(struct mail_namespace *namespaces,
>                                struct mail_storage **storage_r,
>                                struct mail *mail,
>                                const char *destaddr, const char *mailbox);
> extern deliver_mail_func_t *deliver_mail;
>
> Can your plugin use that instead? the -a address is in destaddr
> parameter.

Actually I tried to use this first, as follows:

static int lda_group_deliver_mail
(struct mail_namespace *namespaces, struct mail_storage **storage_r,
struct mail *mail, const char *destaddr, const char *mailbox)
{
i_info("ping");
}

void lda_group_plugin_init(void)
{
i_info("INIT");
n_deliver_mail = deliver_mail;
deliver_mail = lda_group_deliver_mail;
}

void lda_group_plugin_deinit(void)
{
deliver_mail = n_deliver_mail;
}

This works when I am not using sieve plugin, but fails when sieve
plugin is also used.

> I guess I mainly just haven't wanted to spend too much time thinking
> about this yet.

I shall go with using a separate conf for plugin and query ldap when
plugin is called.

with regards,

raj

Re: [Dovecot] problem with master db and dovecot-2.0.beta3

[Oliver Eales]

Timo Sirainen wrote:
>> auth: Fatal: Master passdb can't have pass=yes if there are no passdbs
>> master: Error: service(auth): command startup failed, throttling
>> 
>
> I fixed that a few days ago in hg.
>   
The original error is gone but it is still not working for me. Now i get
in the debug log:

Mar 17 12:06:50 auth: Info: passdb(masteru...@vodafone.de,::1,master):
Attempted master login with no master passdbs (trying to log in as user:
10...@vodafone.de)

With 1.2.x it is working, but not with current 2.0-hg-10937
excerpt from dovecot -n

# 2.0.beta3: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 2.6.27.42-0.1-default x86_64 SUSE Linux Enterprise Server 11
(x86_64)
auth_debug = yes
auth_default_realm = vodafone.de
auth_master_user_separator = *
auth_mechanisms = plain login
auth_socket_path = /usr/local/var/run/dovecot/auth-userdb
auth_verbose = yes
disable_plaintext_auth = no
log_path = /var/log/dovecot.log
passdb {
  args = /usr/local/etc/dovecot/passwd.masterusers
  driver = passwd-file
  master = yes
  pass = yes
}
passdb {
  args = /usr/local/etc/dovecot/dovecot-ldap.conf
  driver = ldap
}

And working 1.2.11 config:

# 1.2.11: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.27.42-0.1-default i686 SUSE Linux Enterprise Server 11
(i586)
auth default:
  default_realm: vodafone.de
  cache_size: 16
  user: ngmail
  master_user_separator: *
  verbose: yes
  debug: yes
  passdb:
driver: passwd-file
args: /etc/dovecot/passwd.masterusers
pass: yes
master: yes
  passdb:
driver: ldap
args: /etc/dovecot/dovecot-ldap.conf
  userdb:
driver: ldap
args: /etc/dovecot/dovecot-ldap.conf


Is there something wrong with the config for the 2.0 ?

Regards and thanks!
Oliver Eales

Re: [Dovecot] [xmail] XMail + Dovecot

[Sabahattin Gucukoglu]

What, exactly, is the problem?  How do you want Dovecot and XMail to work 
together?

I am setting up Dovecot to read from standard Maildirs in user home 
directories, because I am using a separate delivery agent (TMDA but would work 
for maildrop, Dovecot's deliver program and others) to deliver mails using 
mailproc.tab.  So, even while XMail runs as root, Dovecot need not and there is 
no problem getting it to work just like with other MTAs.  XMail can't help with 
delivery to Maildir because it runs as root, and unless you mess about with 
packet filters to non-root it you'll still end up using for instance XACLs to 
allow Dovecot to read the mails.

For authentication of XMail to Dovecot you would have to write external 
authentication helpers that use Dovecot's protocol.  Or, you can use 
checkpassword or PAM-POP3 or something else to try getting Dovecot to 
authenticate against the active XMail list of users.  Or you can just do what I 
do, maintain two databases and keep them in sync.

Cheers,
Sabahattin



smime.p7s
Description: S/MIME cryptographic signature

Re: [Dovecot] I stream read - stale NFS file handle (reboot of server)

[Edgar Fuß]

> I was part of the discussion group for NFSv4 spec
> the short comings of v2 and v3 have been fixed
I'm a bit surprised by this. Which "discussion group"?

> NFSD (v2/v3) is stateless other than the information provided by
> mountd (mount requests) and lockd (file locking).
NFS is stateless save the state information remembered by statd. After all, 
that's the whole point of statd (and the NLM grace period).

> A long time ago FileSystemHandle would stay the same between reboots
> and you wouldn't get this problem other than on an individual file.
My XNFS says:
> 7.1.3 Stateless Servers and Idempotency 
[...]
> With a few exceptions, rebooting the server must not invalidate
> distributed state information. 
(with the exceptions being unstable writes, teporary files etc.)

> To handle deleted files which are in use by NFS clients some servers
> rename them to .nfs* because if one client deleted and other clients
> where accessing the file then they would get Stale NFS handle.
This rename takes place on the client on a REMOVE, not on the server.
Cf. either McKusick et. al. or RFC 1813, 3.3.12, IMPLEMENTATION.

> Once in a while a NFS server will do find $dir -type f -name .nfs\*
> -mtime +7 -mount -exec rm -f {} \; to clean up.
Do you have any reference for that?

> If you do not get the Stale File Handle error when the server
> reboots, it most likely means the FileSystemHandle is not changing
> between reboots
I.e., it conforms to the specs.

> but then you may have more security issues.
Could you please elaborate on this "secutity issues"?

Re: [Dovecot] Design: Asynchronous I/O for single/multi-dbox

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Tue, Mar 16, 2010 at 01:19:01PM -0400, Frank Cusack wrote:

[...]

> And when you don't want to block on I/O.  Threads are almost always
> easier than AIO, and especially easy (ie, no scary complexity issues
> eg deadlock) if you aren't sharing data structures.
   ^^

Such a little phrase... ;-)

Yes, if you aren't sharing data structures you may as well use separate
processes.

Honestly, I do prefer the explicit complexity of async IO to the
implicit complexity of locking. At least in IO-centered server
processes. Threads just /seem/ easy.

And as to the latency/performance, watch the small/fast HTTP servers
moving to async IO models.

I'm not the one to be taken too seriously on it, but I feel that Timo
has the right hunch here.

Regards
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFLoJjjBcgs9XrR2kYRAlz+AJ9J1m3GiX8xAXyIyRa/2yyvejfc0QCdEhLy
an2q0IxgRP/FYp4gVda/22Q=
=exbZ
-END PGP SIGNATURE-