[Dovecot] auth-worker ownership issue
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Greetings, I installed 2.0.7 this evening and I'm getting these messages in /var/log/maillog : Nov 8 23:36:53 myserver dovecot: auth: Fatal: net_connect_unix(auth-worker) in directory /var/run/dovecot failed: Permission denied (euid=89(vpopmail) egid=89(vchkpw) missing +r perm: auth-worker, euid is not dir owner) Nov 8 23:36:53 myserver dovecot: master: Error: service(auth): command startup failed, throttling If I change the ownership of /var/run/dovecot/auth-worker to vpopmail, dovecot starts working properly. However, restarting the service results in this file being re-owned by dovecot. Am I missing a config option somewhere? I'm using dovecot with qmail and vpopmail 5.4.32. Thanks, - --- Jason 'XenoPhage' Frisvold xenoph...@godshell.com - --- "Any sufficiently advanced magic is indistinguishable from technology." - - Niven's Inverse of Clarke's Third Law -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.14 (Darwin) iEYEARECAAYFAkzY1OUACgkQ8CjzPZyTUTQVtQCgi3E0N9IL/kBdT59p0CxpU11u 5YMAoIA+2338GzFkRDt8ymusl73tJM07 =cVkA -END PGP SIGNATURE-
[Dovecot] Managesieve + procmail
Hello, I have an Ubuntu Lucid Postfix/Dovecot server. I have a web ui to manage my websites, emails and so that makes use of Procmail but would I like my users to be able to use Managesieve filters. It it possible to "chain" them ? How? Best regards. -- Lox lox@knc.nc
Re: [Dovecot] emails getting mangled when dragging from Exchange account to IMAP shared folders
Looks like there was some more discussion of this via the digest, but to keep the original thread intact, I'll just reply here. I'm still a little confused over whether or not my issue is an Outlook/Exchange problem or something else, since I'm not certain how I should be interpreting this raw log. I've attached the rawlog to this email (domain names and a few surnames redacted, and about 40K crap between the tags removed, but everything else intact). Can anyone take a peek at this and see if it indicates a problem upstream or downstream from the rawlog? Thanks much. Also note the other oddity that stands out is that some of the envelope recipients are broken (actually, all the recipients in my domain). For example, lines 19 - 21: > To: "Roy", "Mandred", "Jerry" ...don't show any email addresses -- just the names. I know that this is a problem, but I'm just not sure if it is a separate issue, or related to the email mangling problem. At any rate, line 28 shows the ~1134 character References header, with the breakage at the end there. Note that this is the rawlog.out, not the rawlog.in. My confusion here lies in the fact that the breakage seems to be in the out-log only. I just don't know how to read these logs. The in-log is basically exactly the same as the out-log, except that it doesn't contain the first 38 lines that the out-log contains... so does that mean the email came in just fine, but didn't go "out" ok? Or does this still confirm that the email was mangled from outside of Dovecot? Thanks ahead of time. I can send the in-log if you want. On Thu, Nov 4, 2010 at 8:08 AM, Timo Sirainen wrote: > On Wed, 2010-11-03 at 16:18 -0700, Scott Goodwin wrote: > > FYI, I got rawlog working and it shows the same break in the raw logs as > in > > the broken headers. > > Below is a snippet from the rawlog (names and other identifiers > redacted). > > The offending sequence is always in the References headers section, and > you > > can see the line breaks there that show this. So it sounds like this > can't > > be an issue with Dovecot, am I right? > > Yeah, sounds like Outlook breaks with huge headers. That's one huge > References header you have. > > > 20101108-160201-28857.out Description: Binary data
Re: [Dovecot] v2.0.7 released
On 8.11.2010, at 20.44, Daniel L. Miller wrote: > On 11/8/2010 11:51 AM, Timo Sirainen wrote: >> http://dovecot.org/releases/2.0/dovecot-2.0.7.tar.gz >> http://dovecot.org/releases/2.0/dovecot-2.0.7.tar.gz.sig > Does this include the fix for the SIS error logs previously mentioned? Yeah. Whenever I give some URL to hg it's always included in next release.
Re: [Dovecot] v2.0.7 released
On 11/8/2010 11:51 AM, Timo Sirainen wrote: http://dovecot.org/releases/2.0/dovecot-2.0.7.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.7.tar.gz.sig Does this include the fix for the SIS error logs previously mentioned? -- Daniel
[Dovecot] v2.0.7 released
http://dovecot.org/releases/2.0/dovecot-2.0.7.tar.gz http://dovecot.org/releases/2.0/dovecot-2.0.7.tar.gz.sig * master: default_process_limit wasn't actually used anywhere, rather the default was unlimited. Now that it is enforced, you might notice that the default limit is too low and you need to increase it. Dovecot logs a warning when this happens. * mail-log plugin: Log mailbox name as virtual name rather than physical name (e.g. namespace prefix is included in the name) + doveadm dump: Added imapzlib type to uncompress IMAP's COMPRESS DEFLATE I/O traffic (e.g. from rawlog). - IMAP: Fixed LIST-STATUS when listing subscriptions with subscriptions=no namespaces. - IMAP: Fixed SELECT QRESYNC not to crash on mailbox close if a lot of changes were being sent. - quota: Don't count virtual mailboxes in quota - doveadm expunge didn't always actually do the physical expunging - Fixed some index reading optimizations introduced by v2.0.5. - LMTP proxying fixes
[Dovecot] v1.2.16 released
http://dovecot.org/releases/1.2/dovecot-1.2.16.tar.gz http://dovecot.org/releases/1.2/dovecot-1.2.16.tar.gz.sig Just some small fixes. - imap: Fixed SELECT QRESYNC not to crash on mailbox close if a lot of changes were being sent. - pop3: Fixed a potential hang - mbox: Creating new mailboxes should base permissions on mail root dir, not always use 0600. - auth: Disable auth caching entirely for master users.
Re: [Dovecot] 2.0, hourly performance stats
* Timo Sirainen : > Attached a new script. Here's another output of what I get myself: I switched back to 2.0 and will try keeping it going until 12:00 tomorrow -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: [Dovecot] 2.0, hourly performance stats
* Timo Sirainen :
> The managesieve-login is pretty high here too. Much worse than auth
> process. Were there tons of logins at that time? Or some brute force
> password guessing, or some other DoSing?
No, the port is not reachable from the outside.
> Perhaps the problem is actually managesieve-login alone? Did you set
> also service managesieve-login { service_count=0 } ?
>
> I'm currently seeing in my test machine where imaptest is runnng
> something like:
>
> type realusersys recla faults swaps bin boutsignals
> volcs involcs
> master1252.14 0.582.70138271 0 0 0 16
> 0 30101 69
> anvil 1252.13 0.270.23336 0 0 0 0 0
> 16739 6
> imap 2.340.110.2015480 0 0 856 0
> 22 144
> imap-lo 0.110.000.80622 0 0 0 0
> 0 6 19
> auth 1248.45 1.260.91841 16 0 32480 0
> 51559 118
> log 1252.84 0.861.12347 0 0 8 45600
> 47245 27
> config1252.12 13.57 0.5910610 0 0 0
> 0 36574 727
> lmtp 41.43 0.400.80495 11 0 23280 0
> 25 14
>
> The config process's high user CPU% is expected. Master is doing a lot
> of page reclaims, which I'd guess is because it's forking a lot.
>
--
Ralf Hildebrandt
Geschäftsbereich IT | Abteilung Netzwerk
Charité - Universitätsmedizin Berlin
Campus Benjamin Franklin
Hindenburgdamm 30 | D-12203 Berlin
Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
ralf.hildebra...@charite.de | http://www.charite.de
Re: [Dovecot] 2.0, hourly performance stats
On Mon, 2010-11-08 at 19:07 +, Timo Sirainen wrote: > I'm currently seeing in my test machine where imaptest is runnng > something like: Well, that was stupid. Looks like I suck at Perl and couldn't write an actually working script. I was looking at the low user/sys CPU usages, but since they looked ok for some processes, I just thought that Dovecot is so awesomely fast elsewhere too :) So the problem was that it reported only the first process's values, which worked well for long running processes but not for short lived processes.. Attached a new script. Here's another output of what I get myself: typerealusersys reclaims faults swaps bin boutsignals volcs involcs count master 1252.14 0.582.70138271 0 0 0 16 0 30101 69 1 anvil 1252.13 0.270.23336 0 0 0 0 0 16739 6 1 imap21566.3 2166.63 2718.56 17362595 1240 14088 6512656 0 212187 771221 9224 imap-lo 2271.96 4332.52 3043.32 5730026 3 0 328 0 0 59963 187152 9220 auth3585.92 3.522.581729 54 0 59040 0 145214 244 2 log 1252.84 0.861.12347 0 0 8 45600 47245 27 1 config 3590.06 36.98 1.892739 3660 20872 0 0 103559 20522 lmtp604.16 2.161.042749 11 0 2328264 0 235 42 4 Most of this data is for running "rm -rf mdbox;imaptest secs=2" in a loop and looks like imap-login was with service_count=1. logparse.pl Description: Perl program
Re: [Dovecot] 2.0, hourly performance stats
On Mon, 2010-11-08 at 08:34 +0100, Ralf Hildebrandt wrote:
> I'm getting constantly high numbers of page reclaims & involuntary
> context switches for dovecot/auth.
>
> page reclaims = minor faults = cpu switching back to system-mode, But
> why is the auth process doing that so excessively? Same for the large
> number of involuntary context switches...
Hmm. "A page reclaim occurs when a requested page exists on the free
list. A page reclaim results in a page fault being satisfied in
memory."
> Date: Mon, 08 Nov 2010 01:00:01 +0100
>
> type realusersys reclaim faults swaps bin bout
> signals volcs involcs
> auth 421.98 1.321.6624216 0 0 0 0
> 0 267 723
> managesieve-lo18616.9 86.77 32.49 319768 1 0 168
> 0 0 63448 48838
The managesieve-login is pretty high here too. Much worse than auth
process. Were there tons of logins at that time? Or some brute force
password guessing, or some other DoSing? Perhaps the problem is actually
managesieve-login alone? Did you set also service managesieve-login
{ service_count=0 } ?
I'm currently seeing in my test machine where imaptest is runnng
something like:
typerealusersys recla faults swaps bin boutsignals
volcs involcs
master 1252.14 0.582.70138271 0 0 0 16 0
30101 69
anvil 1252.13 0.270.23336 0 0 0 0 0
16739 6
imap2.340.110.2015480 0 0 856 0
22 144
imap-lo 0.110.000.80622 0 0 0 0 0
6 19
auth1248.45 1.260.91841 16 0 32480 0
51559 118
log 1252.84 0.861.12347 0 0 8 45600
47245 27
config 1252.12 13.57 0.5910610 0 0 0 0
36574 727
lmtp41.43 0.400.80495 11 0 23280 0
25 14
The config process's high user CPU% is expected. Master is doing a lot
of page reclaims, which I'd guess is because it's forking a lot.
Re: [Dovecot] Ongoing performance issues with 2.0.x
Stan, On 11/8/10 10:39 AM, "Stan Hoeppner" wrote: > > However, if CONFIG_HZ=1000 you're generating WAY too many interrupts/sec > to the timer, ESPECIALLY on an 8 core machine. This will exacerbate the > high context switching problem. On an 8 vCPU (and physical CPU) machine > you should have CONFIG_HZ=100 or a tickless kernel. You may get by > using 250, but anything higher than that is trouble. On modern kernels you can boot with "divider=10" to take the HZ from 1000 down to 100 at boot time - no rebuilding necessary. -Brad
Re: [Dovecot] Use a different usename than the recipient address (that require a lookup)?
> -Original Message-
> From: dovecot-bounces+yzhou=medplus@dovecot.org [mailto:dovecot-
> bounces+yzhou=medplus@dovecot.org] On Behalf Of Mauricio Tavares
> Sent: Monday, November 08, 2010 1:55 PM
> To: Dovecot Mailing List
> Subject: Re: [Dovecot] Use a different usename than the recipient
> address (that require a lookup)?
>
> On 11/08/2010 01:42 PM, Zhou, Yan wrote:
> > Hi there,
> >
> > I am integrating Postfix and DoveCot, both use LDAP to look up
> > user/email address info. My question is this: can I create a user
> > "jsmith" to get all emails with the recipient address:
> > dovecotd...@domain? Because of using LDAP, I need to do a lookup to
> > know that the user "jsmith" is getting emails for
> "dovecotd...@domain",
> > NOT "dovecotdemo" (there will be no such user).
> >
> Most of my users just use their usernames as their mail
> directories.
> But, I have one I had to do something like
>
>userdb passwd-file {
> args = /etc/dovecot/users
>}
>
> and then define in /etc/dovecot/users for that user where his maildir
> is.
>
> Another option would be to use aliases in postfix.
>
> > In postfix's master.cf, I can use LDA to deliver to DoveCot, which
> needs
> > to know the user name where the message should be delivered to. But
I
> > would not know that at the time because I need to do a lookup in
LDAP.
> > Does the DoveCot delivery agent have any way to allow me return a
> > different user name after LDAP lookup?
> >
> If you use virtual domains, you could specify where the emails
go
> using
> virtual_mailbox_maps.
>
> > I would not want to modify a couple of files to accomplish that,
> because
> > I would have to do that for every email address created.
> >
> Why?
>
We are using SMTP/DoveCot to deliver messages between two systems. It is
best if any modification can be handled automatically, rather than
manually.
NOTICE: This email message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy all
copies of the original message.
This message has been content scanned by the Axway MailGate.
MailGate uses policy enforcement to scan for known viruses, spam, undesirable
content and malicious code. For more information on Axway products please visit
www.axway.com.
Re: [Dovecot] Use a different usename than the recipient address (that require a lookup)?
On 11/08/2010 01:42 PM, Zhou, Yan wrote:
Hi there,
I am integrating Postfix and DoveCot, both use LDAP to look up
user/email address info. My question is this: can I create a user
"jsmith" to get all emails with the recipient address:
dovecotd...@domain? Because of using LDAP, I need to do a lookup to
know that the user "jsmith" is getting emails for "dovecotd...@domain",
NOT "dovecotdemo" (there will be no such user).
Most of my users just use their usernames as their mail directories.
But, I have one I had to do something like
userdb passwd-file {
args = /etc/dovecot/users
}
and then define in /etc/dovecot/users for that user where his maildir is.
Another option would be to use aliases in postfix.
In postfix's master.cf, I can use LDA to deliver to DoveCot, which needs
to know the user name where the message should be delivered to. But I
would not know that at the time because I need to do a lookup in LDAP.
Does the DoveCot delivery agent have any way to allow me return a
different user name after LDAP lookup?
If you use virtual domains, you could specify where the emails go using
virtual_mailbox_maps.
I would not want to modify a couple of files to accomplish that, because
I would have to do that for every email address created.
Why?
Thanks,
Yan
NOTICE: This email message is for the sole use of the intended recipient(s)
and may contain confidential and privileged information. Any unauthorized
review, use, disclosure or distribution is prohibited. If you are not the
intended recipient, please contact the sender by reply email and destroy all
copies of the original message.
This message has been content scanned by the Axway MailGate.
MailGate uses policy enforcement to scan for known viruses, spam, undesirable
content and malicious code. For more information on Axway products please visit
www.axway.com.
Re: [Dovecot] Expunge -A cannot find users
On 11/08/2010 10:29 PM, A.L.E.C wrote: > On 08.11.2010 06:00, Kilburn Abrahams wrote: > >> doveadm expunge -u t...@planetseafood.com mailbox Archive ON '8-Nov-2010' >> >> But in this format doveadm expunge -A mailbox Archive ON '8-Nov-2010', >> it will list all the users in the database and display the following and >> no messages are expunged. >> >> doveadm(t...@planetseafood.com@planetseafood.com): Info: User no longer >> exists, skipping. >> >> The iterate_query = SELECT username AS username, domain FROM mailbox >> which is correct for the DB layout. > What returns this query? Maybe you should just use "-u test" (without > domain part)? > Changed iterate_query = SELECT username FROM mailbox and it works. Sorry for the noise.
Re: [Dovecot] Ongoing performance issues with 2.0.x
* Stan Hoeppner : > Does this machine have more than 4GB of RAM? You do realize that merely > utilizing PAE will cause an increase in context switching, whether on > bare medal or in a VM guest. It will probably actually be much higher > with a VM guest running a PAE kernel. Also, please tell me the ESX > kernel you're running is native 64 bit, not 32 bit. If the VMWare > kernel itself is doing PAE, as well as the guest Linux kernel, this may > fully explain the performance disaster you have on your hands, if it is > indeed due to context switching. It sure work with 1.2.x now, so that's not really the problem > The bigger question is, why does this problem surface so readily while > running Dovecot 2.0.x and not while running Dovecot 1.2.x? EXACTLY > Is 1.2.x merely tickling the dragon's chin, whereas 2.0.x is sticking > it's head into the dragon's mouth? I'd say the difference between 1.2 and 2.0 is so dramatic that it's probably something else. > This very well may be the case. You need to also look at the CONFIG_HZ= > value of the Linux kernel of the guest. If it's a tickless kernel you > should be fine. If tickless, IIRC, you should see CONFIG_NO_HZ=y. # fgrep HZ config-2.6.32-23-generic-pae CONFIG_NO_HZ=y # CONFIG_HZ_100 is not set CONFIG_HZ_250=y # CONFIG_HZ_300 is not set # CONFIG_HZ_1000 is not set CONFIG_HZ=250 CONFIG_MACHZ_WDT=m -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
[Dovecot] Use a different usename than the recipient address (that require a lookup)?
Hi there, I am integrating Postfix and DoveCot, both use LDAP to look up user/email address info. My question is this: can I create a user "jsmith" to get all emails with the recipient address: dovecotd...@domain? Because of using LDAP, I need to do a lookup to know that the user "jsmith" is getting emails for "dovecotd...@domain", NOT "dovecotdemo" (there will be no such user). In postfix's master.cf, I can use LDA to deliver to DoveCot, which needs to know the user name where the message should be delivered to. But I would not know that at the time because I need to do a lookup in LDAP. Does the DoveCot delivery agent have any way to allow me return a different user name after LDAP lookup? I would not want to modify a couple of files to accomplish that, because I would have to do that for every email address created. Thanks, Yan NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. This message has been content scanned by the Axway MailGate. MailGate uses policy enforcement to scan for known viruses, spam, undesirable content and malicious code. For more information on Axway products please visit www.axway.com.
Re: [Dovecot] Ongoing performance issues with 2.0.x
Udo Wolter put forth on 11/8/2010 4:45 AM: > * Ralf Hildebrandt : >>> And I'm guessing you're running a 32bit PAE kernel because VMWare ESX >>> still doesn't officially support 64bit guests, correct? >> >> No, it's supported, but I don'T want to change the whole system. > > That's right, we cannot switch without having several hours downtime. This is > not acceptable. I'm thinking of a way for switching to 64 bit with exchanging > disks etc. But I don't know if this will work, I have to test it first. Does this machine have more than 4GB of RAM? You do realize that merely utilizing PAE will cause an increase in context switching, whether on bare medal or in a VM guest. It will probably actually be much higher with a VM guest running a PAE kernel. Also, please tell me the ESX kernel you're running is native 64 bit, not 32 bit. If the VMWare kernel itself is doing PAE, as well as the guest Linux kernel, this may fully explain the performance disaster you have on your hands, if it is indeed due to context switching. The bigger question is, why does this problem surface so readily while running Dovecot 2.0.x and not while running Dovecot 1.2.x? Is 1.2.x merely tickling the dragon's chin, whereas 2.0.x is sticking it's head into the dragon's mouth? >>> Is this the only guest on this host or do you have others? >> >> only guest > > Yes, the VM-system has 8 CPUs and that's all the ESX has. Of course, there are > times, when the ESX doesn't have that much stress so the DRS moves 1 or 2 > other > machines onto it. But since we got that high load, the rest of the machines > all > had been moved off the ESX. > >>> If this is the only guest, you have 2 dual core dies in that Xeon CPU, >>> 4 cores total. I assume you've assigned 4 virtual CPUs to this Debian >>> VM? >> >> Yes, something like that > > 8. Ralf gave me the model number of that server and said it was a single CPU machine. I looked up the specs, and if that is the case, there are 4 cores total in that Xeon. And, IIRC, that Xeon does not have the HyperThreading circuitry. So, are there two physical CPUs in the machine with 4 cores each, or 1 CPU with 4 cores and HT, appearing as 8 cores? If it's one 4 core CPU with HT enabled, reboot the machine and disable HT in the BIOS. HT itself also contributes to high context switching. HT is more of a hindrance to ESX performance than a benefit. www.vmware.com/pdf/vi_performance_tuning.pdf >>> You may want to run top in the hypervisor console itself (or an SSH >>> session into the hypervisor) and watch the %CPU of the hypervisor's >>> kernel threads. That might tell us something as well. >> >> Udo has to answer that, but from what he told me it was fully using >> all cpus with 2.0, and now it's idling with 1.2 >> >> More details to follow (from him) > > As I said in the other mail: as long as the load isn't high enough we cannot > see any problems in the ESX. Only, if we step over some kind of specific > barrier. I think, it's when even the ESX runs out of possibilities to handle > so > many interrupts. This very well may be the case. You need to also look at the CONFIG_HZ= value of the Linux kernel of the guest. If it's a tickless kernel you should be fine. If tickless, IIRC, you should see CONFIG_NO_HZ=y. However, if CONFIG_HZ=1000 you're generating WAY too many interrupts/sec to the timer, ESPECIALLY on an 8 core machine. This will exacerbate the high context switching problem. On an 8 vCPU (and physical CPU) machine you should have CONFIG_HZ=100 or a tickless kernel. You may get by using 250, but anything higher than that is trouble. -- Stan
Re: [Dovecot] Ongoing performance issues with 2.0.x
* Ralf Hildebrandt : > > And I'm guessing you're running a 32bit PAE kernel because VMWare ESX > > still doesn't officially support 64bit guests, correct? > > No, it's supported, but I don'T want to change the whole system. That's right, we cannot switch without having several hours downtime. This is not acceptable. I'm thinking of a way for switching to 64 bit with exchanging disks etc. But I don't know if this will work, I have to test it first. > > Is this the only guest on this host or do you have others? > > only guest Yes, the VM-system has 8 CPUs and that's all the ESX has. Of course, there are times, when the ESX doesn't have that much stress so the DRS moves 1 or 2 other machines onto it. But since we got that high load, the rest of the machines all had been moved off the ESX. > > If this is the only guest, you have 2 dual core dies in that Xeon CPU, > > 4 cores total. I assume you've assigned 4 virtual CPUs to this Debian > > VM? > > Yes, something like that 8. > > You may want to run top in the hypervisor console itself (or an SSH > > session into the hypervisor) and watch the %CPU of the hypervisor's > > kernel threads. That might tell us something as well. > > Udo has to answer that, but from what he told me it was fully using > all cpus with 2.0, and now it's idling with 1.2 > > More details to follow (from him) As I said in the other mail: as long as the load isn't high enough we cannot see any problems in the ESX. Only, if we step over some kind of specific barrier. I think, it's when even the ESX runs out of possibilities to handle so many interrupts. Bye, Udo -- Udo Wolter Geschäftsbereich IT | Abt. System Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570847 | Fax +49 30 450 7570600 udo.wol...@charite.de | http://www.charite.de smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] 2.0, hourly performance stats
* Ralf Hildebrandt : > I'm getting constantly high numbers of page reclaims & involuntary > context switches for dovecot/auth. > > page reclaims = minor faults = cpu switching back to system-mode, But > why is the auth process doing that so excessively? Same for the large > number of involuntary context switches... Some additions: The last time we had 2.0 at the start we came into big trouble which could also be seen on the VMware ESX side. The CPU load was about 95% constantly and on the VM side the processes showed up in top at mainly using kernel space (system load). Now we didn't have that high load in the morning, of course processes had been in the kernel space too often. But: until the load isn't getting too high the ESX doesn't show any problems, even the stats went up and down (what they didn't do the last time we had the real problems, they just stayed in an even upper line...). Of course we could test it during the main noon time but in that case the mailsystem begins to stumble on high load and users might complain. We also have no real test scenario because it's not easy to get a "real" pressure on the machine, so we have to test it in the production line. But I cannot switch on 2.0 permanently this would cause too many problems. Anyway, even if it runs without making problems on the ESX side we can see the processes in the kernel space. They're way too long there and Ralf seems to find the reason: too many page faults. That's all we can say now. Regards, Udo -- Udo Wolter Geschäftsbereich IT | Abt. System Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570847 | Fax +49 30 450 7570600 udo.wol...@charite.de | http://www.charite.de smime.p7s Description: S/MIME cryptographic signature
Re: [Dovecot] Estimación de horas del cambio a Dovec ot 2.0
Sorry for this mail. It's not for this mail-list. El 8 de noviembre de 2010 16:00, Marcus Rueckert escribió: > and now in english please. > >darix > > -- > openSUSE - SUSE Linux is my linux > openSUSE is good for you > www.opensuse.org >
Re: [Dovecot] Estimación de horas del cambio a Dovec ot 2.0
and now in english please. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org
[Dovecot] Estimación de horas del cambio a Dovec ot 2.0
Actualmente tenemos dovecot 1.2.15 que ha demostrado ser bastante estable,
pero con el que podemos tener problemas de corrupción en ficheros de índices
de Dovecot (dovecot-uidlist). Este problema, podría llegar a ocasionar la
descarga de todos los correos desde el cliente del usuario.
En Dovecot 2.0 tenemos una mejor gestión sobre NFS mediante un nuevo sistema
llamado director. Con este director se consigue que dovecot haga a su vez de
proxy/balanceador e intente direccionar las conexiones de un usuario a un
mismo servidor si este permite conexiones. En resumen, conseguimos evitar
escrituras "concurrentes" sobre cachés de NFS que podrían ocasionar
corrupción en determinados índices de dovecot.
Para este cambio básicamente deberíamos repetir ciertas pruebas que hemos
realizado para dovecot 1.2 así como ciertas tareas.
1. Actualizar configuración
2. Actualizar plugin TrataSPAM (MarcaSPAM/DesmarcaSPAM) Usando skel creado
por Timo durante las sesiones de Málaga.
3. Pruebas de tiempos de indexado con SQUAT
4. Búsquedas concurrentes.
5. Pruebas de IMAP IDLE.
6. Migración { Keywords, suscriptions, flags, status...}
7. Condiciones de estrés.
8. Actualizar Doc.
En una primera estimación. El total de estas tareas no debería superar las
45-50 horas.
A cambio obtendríamos:
1. Un mejor soporte por parte de Timo.
2. doveadm para realizar búsquedas, gestión del director, pruebas de
depuración...
3. Mejoras en el comportamiento ante storage en NFS.
4. Mejoras en la coordinación con SOLR para cuando comencemos con el
buscador avanzado con ficheros adjuntos.
5. Mejoras para conexiones concurrentes.
La razón por la que no empezamos directamente con dovecot 2.0 es porque no
preveíamos que fuera relativamente estable en poco tiempo. Observando la
lista de correos y después de la conversación con Timo, la última release de
dovecot 2.0 es lo suficientemente estable como para usarse en sistemas de
producción.
Si lo creéis necesario, podemos organizar una reunión para tomar la
decisión, aunque sí que esta reunión debería ser para esta semana o a
principios de la siguiente.
Re: [Dovecot] dovecot + sieve
Op 8-11-2010 13:27, maximatt wrote:
In ower mail system we use postfix and dovecot v1.2.10 with sieve plugin and
i think is work fine but.
i need to have access to variables values like in these test script
require ["enotify", "variables", "envelope"];
keep;
notify :importance "1" :message "TEST SIEVE ${from}: ${subject}"
"mailto:i_not_ex...@hotmail.com";;
but when i receive message, the values for "${from}" and "${subject}" are
empty.
i don't know the reasons about these (i'm some lost with these issue)..
Where did you get this example? Sieve does not predefine such
variables. You'll need additional steps to achieve that. For example:
if header :matches "subject" "*" { set "subject" "${1}" }
if envelope :matches "from" "*" { set "from" "${1}" }
You can also choose to use the From header instead of the envelope.
Regards,
Stephan
[Dovecot] Virtual Folder by user.
Can i only set a virtual folder for especific users ? dovecot 2.0.6. Thanks! []'sf.rique
Re: [Dovecot] dovecot Digest, Vol 91, Issue 18
On 08/11/2010 13:16, William Blunn wrote: But surely you can fold header records, e.g. References: For the avoidance of doubt, the second and third lines of that header record are supposed to have an additional space character at the beginning. I did type the required space characters, but obviously the creators of the e-mail client software I use don't think it's important to pass such things through without breaking them. (Whilst I am responsible for choosing that particular e-mail client software, I would venture to suggest that it is hard to find an e-mail client software which is any good.) Bill
Re: [Dovecot] dovecot Digest, Vol 91, Issue 18
On 08/11/2010 12:47, Alan Brown wrote: From: Timo Sirainen Subject: Re: [Dovecot] emails getting mangled when dragging from Exchange account to IMAP shared folders On Wed, 2010-11-03 at 16:18 -0700, Scott Goodwin wrote: FYI, I got rawlog working and it shows the same break in the raw logs as in the broken headers. Below is a snippet from the rawlog (names and other identifiers redacted). The offending sequence is always in the References headers section, and you can see the line breaks there that show this. So it sounds like this can't be an issue with Dovecot, am I right? Yeah, sounds like Outlook breaks with huge headers. That's one huge References header you have. Summarising mail standards (WRT headers) There is a limit of 4096 characters per line. But surely you can fold header records, e.g. References: meaning you could have a "References" header record of any length. If Outlook is breaking at less than that it's a bug. I take it you mean breaking as in malfunctioning as opposed to breaking as in folding. If the References: line is longer than that then it should have been truncated by the sending MUA - there's no provision for multiline References headers (which are a non-standardised import from Usenet anyway). Surely the MUA could just fold the References header record on to multiple lines? When you say "multiline References headers", do you mean "multiple 'References' header records" or "'References' header records folded on to multiple lines"? Bill
Re: [Dovecot] 2.0, hourly performance stats
* Timo Sirainen : > On 8.11.2010, at 7.34, Ralf Hildebrandt wrote: > > > I'm getting constantly high numbers of page reclaims & involuntary > > context switches for dovecot/auth. > > Is it intentional that you have both passdb shadow and passdb pam enabled? If I leave out on of them, the masteruser feature isn't working anymore. Which one should I be using? > > typerealusersys reclaim faults swaps bin > > boutsignals volcs involcs > > auth205.93 1.322.7625280 0 0 0 > > 0 0 261 1425 > > It's still only 2.76 seconds of system CPU from the total of 205 seconds it > was running, so that can't really cause the big system CPU usages you see.. Please note that I was measuring this in the morning. I'll have to keep it running a whole day in order to get "crtiical" values. Then I'll suffer a beating from my colleagues for thrashing the server :) > What about after you stop Dovecot? Then you should see lines from the > permanently running processes (master, anvil, log). -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: [Dovecot] 2.0, hourly performance stats
On 8.11.2010, at 7.34, Ralf Hildebrandt wrote: > I'm getting constantly high numbers of page reclaims & involuntary > context switches for dovecot/auth. Is it intentional that you have both passdb shadow and passdb pam enabled? > page reclaims = minor faults = cpu switching back to system-mode, But > why is the auth process doing that so excessively? Same for the large > number of involuntary context switches... I'm not sure. But: > type realusersys reclaim faults swaps bin bout > signals volcs involcs > auth 205.93 1.322.7625280 0 0 0 0 > 0 261 1425 It's still only 2.76 seconds of system CPU from the total of 205 seconds it was running, so that can't really cause the big system CPU usages you see.. > ssl-params288.71 0.400.48468 0 0 8 0 > 0 8 1 > pop3 0.590.800.20651 0 0 48 16 > 0 12 13 > managesieve 0.510.000.28576 0 0 416 0 > 0 13 7 > imap 4677.82 0.190.8914800 0 488 16 > 0 23550 136 > pop3-login181.77 0.320.28780 0 0 8 0 > 0 46 9 Same for the rest of these processes. What about after you stop Dovecot? Then you should see lines from the permanently running processes (master, anvil, log).
Re: [Dovecot] dovecot Digest, Vol 91, Issue 18
From: Timo Sirainen Subject: Re: [Dovecot] emails getting mangled when dragging from Exchange account to IMAP shared folders On Wed, 2010-11-03 at 16:18 -0700, Scott Goodwin wrote: FYI, I got rawlog working and it shows the same break in the raw logs as in the broken headers. Below is a snippet from the rawlog (names and other identifiers redacted). The offending sequence is always in the References headers section, and you can see the line breaks there that show this. So it sounds like this can't be an issue with Dovecot, am I right? Yeah, sounds like Outlook breaks with huge headers. That's one huge References header you have. Summarising mail standards (WRT headers) There is a limit of 4096 characters per line. If Outlook is breaking at less than that it's a bug. If the References: line is longer than that then it should have been truncated by the sending MUA - there's no provision for multiline References headers (which are a non-standardised import from Usenet anyway).
[Dovecot] dovecot + sieve
Hi,
In ower mail system we use postfix and dovecot v1.2.10 with sieve plugin and
i think is work fine but.
i need to have access to variables values like in these test script
require ["enotify", "variables", "envelope"];
keep;
notify :importance "1" :message "TEST SIEVE ${from}: ${subject}"
"mailto:i_not_ex...@hotmail.com";;
but when i receive message, the values for "${from}" and "${subject}" are
empty.
i don't know the reasons about these (i'm some lost with these issue)..
if help ... i put in master.cf (postfix) these value for dovecot delivery:
dovecot unix - n n - - pipe
flags=DRhu user=mail:mail
argv=/etc/dovecot/libexec/deliver -f ${sender} -d ${recipient}
¿is here the problem or what could be?
thanks in advance!
*Salu2 ;)*
Re: [Dovecot] Expunge -A cannot find users
On 08.11.2010 06:00, Kilburn Abrahams wrote: > doveadm expunge -u t...@planetseafood.com mailbox Archive ON '8-Nov-2010' > > But in this format doveadm expunge -A mailbox Archive ON '8-Nov-2010', > it will list all the users in the database and display the following and > no messages are expunged. > > doveadm(t...@planetseafood.com@planetseafood.com): Info: User no longer > exists, skipping. > > The iterate_query = SELECT username AS username, domain FROM mailbox > which is correct for the DB layout. What returns this query? Maybe you should just use "-u test" (without domain part)? -- Aleksander 'A.L.E.C' Machniak LAN Management System Developer [http://lms.org.pl] Roundcube Webmail Developer [http://roundcube.net] --- PGP: 19359DC1 @@ GG: 2275252 @@ WWW: http://alec.pl
Re: [Dovecot] 2.0, hourly performance stats
* Daniel L. Miller : > On 11/7/2010 11:34 PM, Ralf Hildebrandt wrote: > >Attached is my "dovecot -n" output. > > > Here's another attempt to insert lower supportive appendage into open > breathing & consumption orifice. > > Just comparing your doveconf -n output with mine... > > 1. Why does it appear you have multiple "listen" lines - and they're > not IP specific? I don't have any at all. You mean those? listen(default): * listen(imap): * listen(pop3): * listen(managesieve): *:4190 > 2. Based on your original post, and inferring from your 1.2 config, > it looks like you're trying to run the "high-performance" login mode. Yes > I only saw the service_count parm listed in your 2.0 config 1.2 was also using the "high-performance" login mode > - maybe specifying a couple of the others may have an impact? Can you > try removing your "service imap-login" section, or specifying > "service_count = 1" to, and try running in the default "high-security" > mode and see if that makes a difference? Right now I had to turn 2.0 off because load is getting uneasy :) -- Ralf Hildebrandt Geschäftsbereich IT | Abteilung Netzwerk Charité - Universitätsmedizin Berlin Campus Benjamin Franklin Hindenburgdamm 30 | D-12203 Berlin Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962 ralf.hildebra...@charite.de | http://www.charite.de
Re: [Dovecot] 2.0, hourly performance stats
On 11/7/2010 11:34 PM, Ralf Hildebrandt wrote: Attached is my "dovecot -n" output. Here's another attempt to insert lower supportive appendage into open breathing & consumption orifice. Just comparing your doveconf -n output with mine... 1. Why does it appear you have multiple "listen" lines - and they're not IP specific? I don't have any at all. 2. Based on your original post, and inferring from your 1.2 config, it looks like you're trying to run the "high-performance" login mode. I only saw the service_count parm listed in your 2.0 config - maybe specifying a couple of the others may have an impact? Can you try removing your "service imap-login" section, or specifying "service_count = 1" to, and try running in the default "high-security" mode and see if that makes a difference? -- Daniel
