Re: [Dovecot] Outlook (2010) -> Dovecot (IMAP) >10x slower with high network load and many folders

2012-04-09 Thread Stuart Henderson
On 2012-04-06, Thomas von Eyben  wrote:
> I am seeing a >10x as slow performance when trying to complete a
> "send/receive" from an Outlook 2010 client to Dovecot via IMAP, but
> only when the LAN is fully loaded with other traffic, EG file copying.
> It seems the problem is when outlook is trying to identify folders
> that have changed since last "send/receive" thus traversing the
> hierachy.

Not sure why it would only affect Outlook clients, but if your
switches are managed, you might like to check if flow control is
enabled and, if so, try disabling it.




Re: [Dovecot] Outlook (2010) -> Dovecot (IMAP) >10x slower with high network load and many folders

2012-04-07 Thread Thomas von Eyben
On Sat, Apr 7, 2012 at 3:16 AM, Willie Gillespie
 wrote:
> On 4/6/2012 3:52 AM, Thomas von Eyben wrote:
>>
>> Test results:
>> CLIENT-1 is having the problems when CLIENT-2 is using all the
>> (100Mbps) bandwidth eg. copying files to MAIL-SRV.
>> If I move CLIENT-1 to CLIENT-3 then almost all the delay is gone.
>> NB.: I have not (yet) tested if the problem also exists when CLIENT-2
>> generates traffic to MAIL-SRV as opposed to OTHER-SRV (but I am
>> expecting the same problems).
>
>
> So the link between your 100 Mbps switch and the 1 Gbps switch is saturated
> by CLIENT-2, so CLIENT-1 is just getting the leftovers?
>
> Since CLIENT-3 doesn't go through that 100 Mbps switch, it obviously doesn't
> see that issue.

Yes - that's my current "workaround" (perhaps also solution), I'm
wondering if the performance is really expected to be _so_ bad when
other users are utilizing the LAN.
(You seem to indicate that what I am observing is expected and is
"just" caused by [un-intended] semi-bad behavior from other users…)

BR TvE


Re: [Dovecot] Outlook (2010) -> Dovecot (IMAP) >10x slower with high network load and many folders

2012-04-06 Thread Willie Gillespie

On 4/6/2012 3:52 AM, Thomas von Eyben wrote:

Test results:
CLIENT-1 is having the problems when CLIENT-2 is using all the
(100Mbps) bandwidth eg. copying files to MAIL-SRV.
If I move CLIENT-1 to CLIENT-3 then almost all the delay is gone.
NB.: I have not (yet) tested if the problem also exists when CLIENT-2
generates traffic to MAIL-SRV as opposed to OTHER-SRV (but I am
expecting the same problems).


So the link between your 100 Mbps switch and the 1 Gbps switch is 
saturated by CLIENT-2, so CLIENT-1 is just getting the leftovers?


Since CLIENT-3 doesn't go through that 100 Mbps switch, it obviously 
doesn't see that issue.


Re: [Dovecot] Outlook (2010) -> Dovecot (IMAP) >10x slower with high network load and many folders

2012-04-06 Thread Adrian Minta
It seems that you have packets lost in the network. MAC and Windows have 
different network stacks so this may count for the different behavior.


--
Best regards,
Adrian Minta



[Dovecot] Outlook (2010) -> Dovecot (IMAP) >10x slower with high network load and many folders

2012-04-06 Thread Thomas von Eyben
Hi,

I am seeing a >10x as slow performance when trying to complete a
"send/receive" from an Outlook 2010 client to Dovecot via IMAP, but
only when the LAN is fully loaded with other traffic, EG file copying.
It seems the problem is when outlook is trying to identify folders
that have changed since last "send/receive" thus traversing the
hierachy.

Observations:
- Apple's Mail.app does not have problems when exposed to the same environment.
- Eliminating Outlook is not an option.
- The tests have been performed during the easter vacation, thus
almost no other clients are using the network/mailserver. Under normal
load there are 2-300 users using the network.


Description of the environment:
Server sw: Mac OS X Server 10.6.8 running dovecot: 1.1.20apple0.5 (OS
= fully updated 10.6.x)
Server hw: Xserve Quadcore intel Xeon 2.26 GHz 12 GB RAM
- Never having a cpu load > 20%, mail is stored on a Promise vtrack
RAID connected via Fibre Channel

Client: Windows 7, Outlook 2010
The client have maybe 50 folders (and each folder about a handfull of
subfolders) the size of the mailbox is around 3-5 GB.


Test results:
CLIENT-1 is having the problems when CLIENT-2 is using all the
(100Mbps) bandwidth eg. copying files to MAIL-SRV.
If I move CLIENT-1 to CLIENT-3 then almost all the delay is gone.
NB.: I have not (yet) tested if the problem also exists when CLIENT-2
generates traffic to MAIL-SRV as opposed to OTHER-SRV (but I am
expecting the same problems).


When dumping the traffic on CLIENT-1 (with Wireshark) one thing is
catching my eyes for the 'bad' case:
There is a very long delay after each "REQUEST IDLE" until the next
"REQUEST DONE" ca. 0,3 seconds.
Comparing to the 'good' setup the pause at the same place in the
communication is less than 1 ms !

So why this delay?
Where/how shall I continue my debugging?
- Run dtruss/dtrace scripts on the server?
- Get dovecot to output more debug info (I guess it's doing it's best
so no problems will be seen there…)?

Are there any dovecot configurations that can be altered to workaround
[what I think is a limitation in Outlook's IMAP implementation]?
The current dovecot configuration is Apple's defaults (+POP3 disabled))

I am seing one warning from dovecotd -n though:
- But I see the same warning on a clean installed not-yet-configured
OS X Server so I guess it's not 'that' bad:
Warning: fd limit 256 is lower than what Dovecot can use under full
load (more than 456). Either grow the limit or change
login_max_processes_count and max_mail_processes settings



Physical setup:
+---+  +--+    +-+
| CLIENT-1      |  | CLIENT-2                     |    | CLIENT-3        |
| Outlook "bad" |  | Traffic generator, eg. Samba |    | Outlook  "good" |
+---+  +--+    +-+
        |                       |                              |
+-+             |
|           100Mbps switch (clients)              |             |
+-+             |
                     |                                         |
++
|                     1Gbps switch (for servers)                         |
++
              |                         |
+-+    +---+
| MAIL-SRV: Dovecot, AFP, SMB |    | OTHER-SRV |
+-+    +---+




Wireshark dump - IMAP communication:
|Time     | 10.211.55.3                           |
|         |                   | 10.0.0.10         |
|9.851    |         Response: gatq OK L           |IMAP: Response:
gatq OK Logged in.
|         |(49433)  <--  (143)    |
|9.852    |         Request: o47u SELEC           |IMAP: Request: o47u
SELECT "1_GROUPS"
|         |(49433)  -->  (143)    |
|9.853    |         Response: * FLAGS (           |IMAP: Response: *
FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
|         |(49433)  <--  (143)    |
|9.854    |         Request: 3y4b NOOP            |IMAP: Request: 3y4b NOOP
|         |(49433)  -->  (143)    |
|9.854    |         Response: 3y4b OK N           |IMAP: Response:
3y4b OK NOOP completed.
|         |(49433)  <--  (143)    |
|9.855    |         Request: 4vlj IDLE            |IMAP: Request: 4vlj IDLE
|         |(49433)  -->  (143)    |
|9.856    |         Response: + idling            |IMAP: Response: + idling
|         |(49433)  <--  (143)    |
|10.108   |         Request: DONE                 |IMAP: Request: DONE
|         |(49433)  -->  (143)    |
|10.108   |         Response: 4vlj OK I           |IMAP: Response:
4vlj OK Idle completed.
|         |(49433)  <--  (143)    |
|10.108   |