Re: Strange NFS client ACK behaviour
On Mon, Sep 9, 2013 at 11:51 PM, Markus Stockhausen wrote: >> Von: Wendy Cheng [s.wendy.ch...@gmail.com] >> Gesendet: Montag, 9. September 2013 22:03 >> An: Markus Stockhausen >> Cc: linux-rdma@vger.kernel.org >> Betreff: Re: Strange NFS client ACK behaviour >> >> On Sun, Sep 8, 2013 at 11:24 AM, Markus Stockhausen >> wrote: >> >> >> > we observed a performance drop in our IPoIB NFS backup >> >> > infrastructure since we switched to machines with newer >> >> > kernels. >> > >> >> Not sure how your backup infrastructure works but the symptoms seem to >> match with this discussion: >> http://www.spinics.net/lists/linux-nfs/msg38980.html >> >> If you know how to recompile nfs kmod, Trond's patch does worth a try. >> Or open an Ubuntu support ticket, let them build you a test kmod. >> >> -- Wendy > > Thanks for pointing into that direction. From my understanding this > patch goes into the NFS client side. I built a patched module for my > Fedora 19 client (3.10 kernel). Nevertheless the behaviour ist still > the same. If I get the patch right it is about forked childs that > access a page of a mmapped file round robin and the kernel issues > tons of write requests to the file. > > My case is only about ACK transmissions for a single writer. > > Markus > So you have to go back to the drawing board :(. Have you tried to profile it ? http://oprofile.sourceforge.net/about/ -- Wendy -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
AW: Strange NFS client ACK behaviour
> Von: Wendy Cheng [s.wendy.ch...@gmail.com] > Gesendet: Montag, 9. September 2013 22:03 > An: Markus Stockhausen > Cc: linux-rdma@vger.kernel.org > Betreff: Re: Strange NFS client ACK behaviour > > On Sun, Sep 8, 2013 at 11:24 AM, Markus Stockhausen > wrote: > > >> > we observed a performance drop in our IPoIB NFS backup > >> > infrastructure since we switched to machines with newer > >> > kernels. > > > > Not sure how your backup infrastructure works but the symptoms seem to > match with this discussion: > http://www.spinics.net/lists/linux-nfs/msg38980.html > > If you know how to recompile nfs kmod, Trond's patch does worth a try. > Or open an Ubuntu support ticket, let them build you a test kmod. > > -- Wendy Thanks for pointing into that direction. From my understanding this patch goes into the NFS client side. I built a patched module for my Fedora 19 client (3.10 kernel). Nevertheless the behaviour ist still the same. If I get the patch right it is about forked childs that access a page of a mmapped file round robin and the kernel issues tons of write requests to the file. My case is only about ACK transmissions for a single writer. Markus Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497
Re: Strange NFS client ACK behaviour
On Sun, Sep 8, 2013 at 11:24 AM, Markus Stockhausen wrote: >> > we observed a performance drop in our IPoIB NFS backup >> > infrastructure since we switched to machines with newer >> > kernels. > Not sure how your backup infrastructure works but the symptoms seem to match with this discussion: http://www.spinics.net/lists/linux-nfs/msg38980.html If you know how to recompile nfs kmod, Trond's patch does worth a try. Or open an Ubuntu support ticket, let them build you a test kmod. -- Wendy -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
AW: Strange NFS client ACK behaviour
> Von: Wendy Cheng [s.wendy.ch...@gmail.com] > Gesendet: Donnerstag, 5. September 2013 18:42 > An: Markus Stockhausen > Cc: linux-rdma@vger.kernel.org; linux-...@vger.kernel.org > Betreff: Re: Strange NFS client ACK behaviour > > CC linux-nfs .. maybe this is obvious to someone there ... Two > comments inlined below. > > On Tue, Sep 3, 2013 at 11:28 AM, Markus Stockhausen > wrote: > > Hello, > > > > we observed a performance drop in our IPoIB NFS backup > > infrastructure since we switched to machines with newer > > kernels. As I do not know where to start I hope someone > > on this list can give me hint where to dig for more details. > > In case of no other reply, I would start w/ a socket program (or a > network performance measuring tool) on the interface that does similar > logic as "dd" you described below; that is, send a 256K message in a > fixed number of loops (so total transfer size somewhere close to your > file size) between client and server, followed by comparing the > interrupt counters (cat /proc/interrtups) on both kernels. If the > interrupt count differs as you described, the problem is most likely > with the IB driver, not NFS layer. > > > > > To make a long story short. We use ConnectX cards with the > > standard kernel drivers on version 2.6.32 (Ubuntu 10.04), 3.5 > > (Ubuntu 12.04) and 3.10 (Fedora 19). The very simple and not > > scientific test consists of mounting a NFS share using IPoIB UD > > network interfaces at MTU of 2044. Afterwards read a large file > > on the client side with dd if=file of=/dev/null bs=256K. > > During the transfer we run a tcpdump on the ibX interface on > > the NFS server side. No special settings for kernel parameters > > until now. > > I don't know much about ConnectX. Not sure what "IPoIB UD" means ? > "Datagram vs. CM" or "TCP vs. UDP" ? > Hello, thanks for stopping by. I followed your advise and compared the behaviour to a tcpdump of a netserver/netperf session. That reads more normal: 20:06:50.397472 IP server.43845 > client.58489: seq ... length 63744 20:06:50.397531 IP client.58489 > server.43845: ack ... length 0 20:06:50.397567 IP server.43845 > client.58489: seq ... length 63744 20:06:50.397595 IP server.43845 > client.58489: seq ... length 63744 20:06:50.397622 IP server.43845 > client.58489: seq ... length 63744 20:06:50.397632 IP client.58489 > server.43845: ack ... length 0 20:06:50.397667 IP server.43845 > client.58489: seq ... length 63744 20:06:50.397715 IP client.58489 > server.43845: ack ... length 0 20:06:50.397723 IP client.58489 > server.43845: ack ... length 0 It is not really comparable as we se transfers to several ports but at least a much better ratio between send and ack packets. Seems like further digging in NFS behaviour is required. But that may be another story. Markus Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497
Re: Strange NFS client ACK behaviour
CC linux-nfs .. maybe this is obvious to someone there ... Two comments inlined below. On Tue, Sep 3, 2013 at 11:28 AM, Markus Stockhausen wrote: > Hello, > > we observed a performance drop in our IPoIB NFS backup > infrastructure since we switched to machines with newer > kernels. As I do not know where to start I hope someone > on this list can give me hint where to dig for more details. In case of no other reply, I would start w/ a socket program (or a network performance measuring tool) on the interface that does similar logic as "dd" you described below; that is, send a 256K message in a fixed number of loops (so total transfer size somewhere close to your file size) between client and server, followed by comparing the interrupt counters (cat /proc/interrtups) on both kernels. If the interrupt count differs as you described, the problem is most likely with the IB driver, not NFS layer. > > To make a long story short. We use ConnectX cards with the > standard kernel drivers on version 2.6.32 (Ubuntu 10.04), 3.5 > (Ubuntu 12.04) and 3.10 (Fedora 19). The very simple and not > scientific test consists of mounting a NFS share using IPoIB UD > network interfaces at MTU of 2044. Afterwards read a large file > on the client side with dd if=file of=/dev/null bs=256K. > During the transfer we run a tcpdump on the ibX interface on > the NFS server side. No special settings for kernel parameters > until now. I don't know much about ConnectX. Not sure what "IPoIB UD" means ? "Datagram vs. CM" or "TCP vs. UDP" ? > > When doing the test with a 2.6.32 kernel based client we see the > following packet sequence. More or less a lot of transferd blocks > from the NFS server to the client with sometimes an ACK package > from the client to the server: > > 16:16:45.050930 IP server.nfs > cli_2_6_32.896: > Flags [.], seq 8909853:8913837, ack 1154149, > win 604, options [nop,nop,TS val 1640401415 > ecr 3881919089], length 3984 > 16:16:45.050936 IP server.nfs > cli_2_6_32.896: > Flags [.], seq 8913837:8917821, ack 1154149, > win 604, options [nop,nop,TS val 1640401415 > ecr 3881919089], length 3984 > > ... 8 more ... > > 16:16:45.050976 IP cli_2_6_32.896 > server.nfs: > Flags [.], ack 8909853, win 24574, options > [nop,nop,TS val 3881919089 ecr 1640401415], > length 0 > ... > > After switchng to a client with a newer kernel (3.5 or 3.10) the > sequence all of a sudden gives just the opposite behaviour. > One should note that this is the same server as in the test > above. The server sends bigger packets (I guess TSO is doing > the rest of the work). After each packet the client sends > several ACK packages back. > > 16:15:21.038782 IP server.nfs > cli_3_5_0.928: > Flags [.], seq 9612429:9652269, ack 372776, > win 5815, options [nop,nop,TS val 1640380412 > ecr 560111379], length 39840 > 16:15:21.038806 IP cli_3_5_0.928 > server.nfs: > Flags [.], ack 9542205, win 16384, options > [nop,nop,TS val 560111379 ecr 1640380412], > length 0 > 16:15:21.038812 IP cli_3_5_0.928 > server.nfs: > Flags [.], ack 9546077, win 16384, options > [nop,nop,TS val 560111379 ecr 1640380412], > length 0 > > ... 6-8 more ... > > The visible side effects of this changed processing include: > - NIC interrupts on the NFS servers raise by a factor of 8. > - Transfer speed lowers by 50% (400->200 MB/sec) > > Best regards. > > Markus -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Strange NFS client ACK behaviour
Hello, we observed a performance drop in our IPoIB NFS backup infrastructure since we switched to machines with newer kernels. As I do not know where to start I hope someone on this list can give me hint where to dig for more details. To make a long story short. We use ConnectX cards with the standard kernel drivers on version 2.6.32 (Ubuntu 10.04), 3.5 (Ubuntu 12.04) and 3.10 (Fedora 19). The very simple and not scientific test consists of mounting a NFS share using IPoIB UD network interfaces at MTU of 2044. Afterwards read a large file on the client side with dd if=file of=/dev/null bs=256K. During the transfer we run a tcpdump on the ibX interface on the NFS server side. No special settings for kernel parameters until now. When doing the test with a 2.6.32 kernel based client we see the following packet sequence. More or less a lot of transferd blocks from the NFS server to the client with sometimes an ACK package from the client to the server: 16:16:45.050930 IP server.nfs > cli_2_6_32.896: Flags [.], seq 8909853:8913837, ack 1154149, win 604, options [nop,nop,TS val 1640401415 ecr 3881919089], length 3984 16:16:45.050936 IP server.nfs > cli_2_6_32.896: Flags [.], seq 8913837:8917821, ack 1154149, win 604, options [nop,nop,TS val 1640401415 ecr 3881919089], length 3984 ... 8 more ... 16:16:45.050976 IP cli_2_6_32.896 > server.nfs: Flags [.], ack 8909853, win 24574, options [nop,nop,TS val 3881919089 ecr 1640401415], length 0 ... After switchng to a client with a newer kernel (3.5 or 3.10) the sequence all of a sudden gives just the opposite behaviour. One should note that this is the same server as in the test above. The server sends bigger packets (I guess TSO is doing the rest of the work). After each packet the client sends several ACK packages back. 16:15:21.038782 IP server.nfs > cli_3_5_0.928: Flags [.], seq 9612429:9652269, ack 372776, win 5815, options [nop,nop,TS val 1640380412 ecr 560111379], length 39840 16:15:21.038806 IP cli_3_5_0.928 > server.nfs: Flags [.], ack 9542205, win 16384, options [nop,nop,TS val 560111379 ecr 1640380412], length 0 16:15:21.038812 IP cli_3_5_0.928 > server.nfs: Flags [.], ack 9546077, win 16384, options [nop,nop,TS val 560111379 ecr 1640380412], length 0 ... 6-8 more ... The visible side effects of this changed processing include: - NIC interrupts on the NFS servers raise by a factor of 8. - Transfer speed lowers by 50% (400->200 MB/sec) Best regards. Markus Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497
