Hello,
we observed a performance drop in our IPoIB NFS backup
infrastructure since we switched to machines with newer
kernels. As I do not know where to start I hope someone
on this list can give me hint where to dig for more details.
To make a long story short. We use ConnectX cards with the
standard kernel drivers on version 2.6.32 (Ubuntu 10.04), 3.5
(Ubuntu 12.04) and 3.10 (Fedora 19). The very simple and not
scientific test consists of mounting a NFS share using IPoIB UD
network interfaces at MTU of 2044. Afterwards read a large file
on the client side with dd if=file of=/dev/null bs=256K.
During the transfer we run a tcpdump on the ibX interface on
the NFS server side. No special settings for kernel parameters
until now.
When doing the test with a 2.6.32 kernel based client we see the
following packet sequence. More or less a lot of transferd blocks
from the NFS server to the client with sometimes an ACK package
from the client to the server:
16:16:45.050930 IP server.nfs > cli_2_6_32.896:
Flags [.], seq 8909853:8913837, ack 1154149,
win 604, options [nop,nop,TS val 1640401415
ecr 3881919089], length 3984
16:16:45.050936 IP server.nfs > cli_2_6_32.896:
Flags [.], seq 8913837:8917821, ack 1154149,
win 604, options [nop,nop,TS val 1640401415
ecr 3881919089], length 3984
... 8 more ...
16:16:45.050976 IP cli_2_6_32.896 > server.nfs:
Flags [.], ack 8909853, win 24574, options
[nop,nop,TS val 3881919089 ecr 1640401415],
length 0
...
After switchng to a client with a newer kernel (3.5 or 3.10) the
sequence all of a sudden gives just the opposite behaviour.
One should note that this is the same server as in the test
above. The server sends bigger packets (I guess TSO is doing
the rest of the work). After each packet the client sends
several ACK packages back.
16:15:21.038782 IP server.nfs > cli_3_5_0.928:
Flags [.], seq 9612429:9652269, ack 372776,
win 5815, options [nop,nop,TS val 1640380412
ecr 560111379], length 39840
16:15:21.038806 IP cli_3_5_0.928 > server.nfs:
Flags [.], ack 9542205, win 16384, options
[nop,nop,TS val 560111379 ecr 1640380412],
length 0
16:15:21.038812 IP cli_3_5_0.928 > server.nfs:
Flags [.], ack 9546077, win 16384, options
[nop,nop,TS val 560111379 ecr 1640380412],
length 0
... 6-8 more ...
The visible side effects of this changed processing include:
- NIC interrupts on the NFS servers raise by a factor of 8.
- Transfer speed lowers by 50% (400->200 MB/sec)
Best regards.
Markus
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
