Re[2]: Disk access/MPT under ESX3.5
Hello, Daniel Can you execute camcontrol with some parameters again? For example, `camcontrol negotiate 0:0 -W 16 -O 127 -R 40.000'. -- Shunsuke SHINOMIYA [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Disk access/MPT under ESX3.5
da0 at mpt0 bus 0 target 0 lun 0 da0: VMware Virtual disk 1.0 Fixed Direct Access SCSI-2 device da0: 3.300MB/s transfers da0: Command Queueing Enabled da0: 34816MB (71303168 512 byte sectors: 255H 63S/T 4438C) Can you re-negotiate transfer rate, using camcontrol? `camcontrol negotiate 0:0 -W 16' might improve that transfer settings. camcontrol needs passthrough device support in a kernel. -- Shunsuke SHINOMIYA [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
reenable tagged queueing of VMware virtual disk.
Hi,
Tagged queueing feature of a VMware virtual disk connected with SCSI
has been disabled since the cam quirk was added.
# this quirk is needed to find the drive by SCSI HA.
I modified this quirk to allow tagged queueing while finding out causes
of low disk performance.
This modification improve the throughput of the disk I/O on VMware ESX
Server 3, at least. (4MBps - 18Mbps, by some easy measurements using dd).
I think following modification to be applicable to RELENG_[456] and MAIN.
I applied this patch to my RELENG_5 (virtual) box and do 'make -j4
buildworld' now and at present, the problem has not occurred.
--- sys/cam/cam_xpt.c.orig Fri Sep 8 18:10:14 2006
+++ sys/cam/cam_xpt.c Fri Sep 8 17:43:57 2006
@@ -359,7 +359,7 @@
{
/* Does not support other than LUN 0 */
{ T_DIRECT, SIP_MEDIA_FIXED, VMware*, *, * },
- CAM_QUIRK_NOLUNS, /*mintags*/0, /*maxtags*/0
+ CAM_QUIRK_NOLUNS, /*mintags*/2, /*maxtags*/255
},
thank you.
--
Shunsuke SHINOMIYA [EMAIL PROTECTED]
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Tuning for router performance
Hi, Paul. Do sysctl kern.random.sys.harvest.ethernet=0 sysctl kern.random.sys.harvest.interrupt=0 affect to your router's performance? and, disabling interrupt moderation of em(4) affects the router too? Please see the previous discussion of interrupt moderation at freebsd-current. http://docs.freebsd.org/cgi/getmsg.cgi?fetch=1279567+0+archive/2004/freebsd-current/20041121.freebsd-current http://docs.freebsd.org/cgi/getmsg.cgi?fetch=215083+0+archive/2004/freebsd-current/20041128.freebsd-current http://www.freebsd.org/cgi/query-pr.cgi?pr=74986 -- Shunsuke SHINOMIYA [EMAIL PROTECTED] ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re[2]: if_em int_throttle_ceil patch
Hi Jeremie and Lists, Great work Shunsuke, thank you ! Any chance to get this patch reviewed and commited ? Thank you. I did send-pr as kern/74986. http://www.freebsd.org/cgi/query-pr.cgi?pr=74986 I think that the appropriate value of interrupt moderation depends on each system, but tuning this value is worth to bring out a performance of the system. I hope that this patch help to bring out the performance of your system. -- Shunsuke SHINOMIYA [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
if_em int_throttle_ceil patch
Hi, Jeremie and Lists. The previous patch to change if_em's int_throttle_ceil into sysctl-able cause a kernel panic. If you set em's int_throttle_ceil=0 and then reconfigure the em, it cause a diveded by zero panic. This patch for original if_em.[ch] which is attached to this mail is corrected this problem. -- Shunsuke SHINOMIYA [EMAIL PROTECTED] if_em.diff Description: Binary data ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re[5]: serious networking (em) performance (ggate and NFS) problem
I did FastEthernet throughput test by Smartbits with SmartApp. It's simpler than TCP throughput measurement. :) This Smartbits has some FastEthernet ports, has no GbE ports. The router is consist of single Xeon 2.4GHz which is HTT enabled and two on-boarded em interfaces. The kernel is 5.3-RELEASE with option SMP. if_em.c is version 1.44.2.3 with the sysctl-able patch. kern.random.sys.harvest.ethernet is set to 0. `Interrupt Moderation on' means the combination of parameters, hw.em?.rx_int_delay: 0 hw.em?.tx_int_delay: 66 hw.em?.rx_abs_int_delay: 66 hw.em?.tx_abs_int_delay: 66 hw.em?.int_throttle_ceil: 8000. `Interrupt Moderation off' means the combination of parameters, hw.em?.rx_int_delay: 0 hw.em?.tx_int_delay: 0 hw.em?.rx_abs_int_delay: 0 hw.em?.tx_abs_int_delay: 0 hw.em?.int_throttle_ceil: 0. `on/off' means input side interface(em0)'s moderation is `on' and output side interface(em1)'s moderation is `off'. hmm ..., the results of off/on are unstable. (A) ... Passed Rate(%) (B) ... Passed Rate(pps) Interrupt Moderation on/on Frame 1st Trial 2nd Trial 3rd Trial Size (A) (B) (A) (B) (A) (B) 64 28.14 41876 28.14 41876 28.14 41876 128 50.08 42301 49.50 41805 49.50 41806 256 93.24 42229 92.00 41666 92.62 41946 512100.00 23496 100.00 23496 100.00 23496 1024100.00 11973 100.00 11973 100.00 11973 1280100.009615 100.009615 100.009615 1518100.008127 100.008127 100.008127 Interrupt Moderation on/off Frame 1st Trial 2nd Trial 3rd Trial Size (A) (B) (A) (B) (A) (B) 64 28.14 41876 27.50 40916 28.14 41876 128 50.08 42301 48.85 41254 49.50 41806 256 93.24 42229 92.00 41666 92.62 41946 512100.00 23496 100.00 23496 100.00 23496 1024100.00 11973 100.00 11973 100.00 11973 1280100.009615 100.009615 100.009615 1518100.008127 100.008127 100.008127 Interrupt Moderation off/on Frame 1st Trial 2nd Trial 3rd Trial Size (B)(C) (B) (C) (B) (C) 64100.00 148807 100.00 148807 99.41 147927 128100.00 84458 55.74 47080 61.41 51867 256100.00 45289 98.75 44722 100.00 45289 512100.00 23496 100.00 23496 100.00 23496 1024100.00 11973 100.00 11973 100.00 11973 1280100.009615 100.009615 100.009615 1518100.008127 100.008127 100.008127 Interrupt Moderation off/off Frame 1st Trial 2nd Trial 3rd Trial Size (B)(C) (B) (C) (B) (C) 64 81.55 121358 81.55 121358 82.35 122547 128100.00 84458 100.00 84458 100.00 84458 256100.00 45289 100.00 45289 100.00 45289 512100.00 23496 100.00 23496 100.00 23496 1024100.00 11973 100.00 11973 100.00 11973 1280100.009615 100.009615 100.009615 1518100.008127 100.008127 100.008127 -- Shunsuke SHINOMIYA [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re[2]: serious networking (em) performance (ggate and NFS) problem
Jeremie, thank you for your comment. I did simple benchmark at some settings. I used two boxes which are single Xeon 2.4GHz with on-boarded em. I measured a TCP throughput by iperf. These results show that the throughput of TCP increased if Interrupt Moderation is turned OFF. At least, adjusting these parameters affected TCP performance. Other appropriate combination of parameter may exist. The settings are some combinations of hw.em0.rx_int_delay hw.em0.tx_int_delay hw.em0.rx_abs_int_delay hw.em0.tx_abs_int_delay hw.em0.int_throttle_ceil. In this mail, A setting, hw.em0.rx_int_delay: 0 hw.em0.tx_int_delay: 66 hw.em0.rx_abs_int_delay: 66 hw.em0.tx_abs_int_delay: 66 hw.em0.int_throttle_ceil: 8000 is abbreviated to (0, 66, 66, 66, 8000). TCP window size was not adjusted by iperf's options. It mean that iperf is used by default setting. sender : default(0, 66, 66, 66, 8000), receiver : default(0, 66, 66, 66, 8000) 1st trial 852Mbps 2nd trial 861Mbps 3rd trial 822Mbps 4th trial 791Mbps 5th trial 826Mbps average 830.4Mbps, std. dev. 27.6Mbps sender : (0, 0, 0, 0, 8000), receiver : (0, 0, 0, 0, 8000) 1st trial 787Mbps 2nd trial 793Mbps 3rd trial 843Mbps 4th trial 771Mbps 5th trial 848Mbps average 808.4Mbps, std. dev. 34.9Mbps sender : off(0, 0, 0, 0, 0), receiver : off(0, 0, 0, 0, 0) 1st trial 902Mbps 2nd trial 901Mbps 3rd trial 899Mbps 4th trial 894Mbps 5th trial 900Mbps average 899.2Mbps, std. dev. 3.1Mbps -- Shunsuke SHINOMIYA [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re[4]: serious networking (em) performance (ggate and NFS) problem
Thank you, Matt. Very interesting, but the only reason you get lower results is simply because the TCP window is not big enough. That's it. Yes, I knew that adjusting TCP window size is important to use up a link. However I wanted to show adjusting the parameters of Interrupt Moderation affects network performance. And I think a packet loss was occured by enabled Interrupt Moderation. The mechanism of a packet loss in this case is not cleared, but I think inappropriate TCP window size is not the only reason. I found TCP throuput improvement at disabled Interrupt Moderation is related to congestion avoidance phase of TCP. Because these standard deviations are decreased when Interrupt Moderation is disabled. The following two results are outputs of `iperf -P 10'. without TCP window size adjustment too. I think, the difference of each throughput at same measurement shows congestion avoidance worked. o with default setting of Interrupt Moderation. [ ID] Interval Transfer Bandwidth [ 13] 0.0-10.0 sec 80.1 MBytes 67.2 Mbits/sec [ 11] 0.0-10.0 sec 121 MBytes 102 Mbits/sec [ 12] 0.0-10.0 sec 98.9 MBytes 83.0 Mbits/sec [ 4] 0.0-10.0 sec 91.8 MBytes 76.9 Mbits/sec [ 7] 0.0-10.0 sec 127 MBytes 106 Mbits/sec [ 5] 0.0-10.0 sec 106 MBytes 88.8 Mbits/sec [ 6] 0.0-10.0 sec 113 MBytes 94.4 Mbits/sec [ 10] 0.0-10.0 sec 117 MBytes 98.2 Mbits/sec [ 9] 0.0-10.0 sec 113 MBytes 95.0 Mbits/sec [ 8] 0.0-10.0 sec 93.0 MBytes 78.0 Mbits/sec [SUM] 0.0-10.0 sec 1.04 GBytes 889 Mbits/sec o with disabled Interrupt Moderation. [ ID] Interval Transfer Bandwidth [ 7] 0.0-10.0 sec 106 MBytes 88.9 Mbits/sec [ 10] 0.0-10.0 sec 107 MBytes 89.7 Mbits/sec [ 8] 0.0-10.0 sec 107 MBytes 89.4 Mbits/sec [ 9] 0.0-10.0 sec 107 MBytes 90.0 Mbits/sec [ 11] 0.0-10.0 sec 106 MBytes 89.2 Mbits/sec [ 12] 0.0-10.0 sec 104 MBytes 87.6 Mbits/sec [ 4] 0.0-10.0 sec 106 MBytes 88.7 Mbits/sec [ 13] 0.0-10.0 sec 106 MBytes 88.9 Mbits/sec [ 5] 0.0-10.0 sec 106 MBytes 88.9 Mbits/sec [ 6] 0.0-10.0 sec 107 MBytes 89.9 Mbits/sec [SUM] 0.0-10.0 sec 1.04 GBytes 891 Mbits/sec But, By decreasing TCP windows size, it could avoid. o with default setting of Interrupt Moderation and iperf -P 10 -w 28.3k [ ID] Interval Transfer Bandwidth [ 12] 0.0-10.0 sec 111 MBytes 93.0 Mbits/sec [ 4] 0.0-10.0 sec 106 MBytes 88.8 Mbits/sec [ 11] 0.0-10.0 sec 107 MBytes 89.9 Mbits/sec [ 9] 0.0-10.0 sec 109 MBytes 91.6 Mbits/sec [ 5] 0.0-10.0 sec 109 MBytes 91.5 Mbits/sec [ 13] 0.0-10.0 sec 108 MBytes 90.8 Mbits/sec [ 10] 0.0-10.0 sec 107 MBytes 89.7 Mbits/sec [ 8] 0.0-10.0 sec 110 MBytes 92.3 Mbits/sec [ 6] 0.0-10.0 sec 111 MBytes 93.2 Mbits/sec [ 7] 0.0-10.0 sec 108 MBytes 90.6 Mbits/sec [SUM] 0.0-10.0 sec 1.06 GBytes 911 Mbits/sec Measureing TCP throughput was not appropriate way to indicate an effect of Interrupt Moderation clearly. It's my mistake. TCP is too complicated. :) -- Shunsuke SHINOMIYA [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re[2]: serious networking (em) performance (ggate and NFS) problem
Hi, Jeremie, how is this? To disable Interrupt Moderation, sysctl hw.em?.int_throttle_valve=0. However, because this patch is just made now, it is not fully tested. if you suppose your computer has sufficient performance, please try to disable or adjust parameters of Interrupt Moderation of em. Nice ! It would be even better if there was a boot-time sysctl to configure the behaviour of this feature, or something like ifconfig link0 option of the fxp(4) driver. -- Shunsuke SHINOMIYA [EMAIL PROTECTED] if_em.diff Description: Binary data ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re[2]: serious networking (em) performance (ggate and NFS) problem
Hi list,
if you suppose your computer has sufficient performance, please try to
disable or adjust parameters of Interrupt Moderation of em.
In my router(Xeon 2.4GHz and on-board two em interfaces) case, it
improves a router's packet forwarding performance. I think the
interrupt delay by Interrupt Moderation caused NIF's input buffer
overflow or output buffer underrun in this case.
In order to disable Interrupt Moderation, modify src/sys/dev/em/if_em.c
like the following patch and set hw.em.{rx,tx}_{,abs_}int_delay zero by
sysctl.
*** if_em.c-1.44.2.3.orig Fri Nov 19 11:22:48 2004
--- if_em.c Fri Nov 19 11:23:39 2004
*** em_initialize_receive_unit(struct adapte
*** 2611,2618
--- 2611,2622
/* Set the interrupt throttling rate. Value is calculated
* as DEFAULT_ITR = 1/(MAX_INTS_PER_SEC * 256ns) */
+ #if 1
+ #define DEFAULT_ITR 0
+ #else
#define MAX_INTS_PER_SEC8000
#define DEFAULT_ITR 10/(MAX_INTS_PER_SEC * 256)
+ #endif
E1000_WRITE_REG(adapter-hw, ITR, DEFAULT_ITR);
}
--
Shunsuke SHINOMIYA [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
