Hello all, Here are more results, where each section's heading indicates server-client size (e.g. 32-65 is a 32-bit server and a 64-bit client), only related to userspace; that is both machines are running (identical) 64-bit kernels. Each row is headed by the particular kernel which was booted on the two machines (If the kernel is suffixed with svn, then it indicates that the OpenIB kernel components were used. If not, then it is a mainline tree without modification). The userpace (and kernel, where applicable) svn revision is listed following the kernel version.
This e-mail is ridiculously long, but there is *a lot* of data here... Enjoy! FYI, all of the errors in the footnotes are from the client-side. If you would like to see what the server-side said, please just ask. netpipe over IB 32-32 avg b/w (Mbps) peak b/w (Mbps) rdma_write 2.6.15-rc7-git4-svn (4662) 1036.73 1839.97 2.6.15-rc7-git5-svn (4663) 1036.18 1839.98 2.6.15-rc7-git6-svn (4670) 1036.08 1839.99 2.6.15-rc7-git6-svn (4692) 1035.65 1839.97 2.6.15-svn (4714) 1036.25 1839.93 2.6.15-svn (4785) 1035.81 1840 2.6.15-svn (4789) 1036.48 1839.97 2.6.15 (4789) 1035.46 1839.98 2.6.15-svn (4803) 1035.57 1840.01 2.6.15 (4803) 1035.67 1839.97 rdma_write with immediate Errors across the board [1] send_recv Errors across the board [2] send_recv with immediate Errors across the board [3] 32-64 avg b/w (Mbps) peak b/w (Mbps) rdma_write Errors across the board [4] rdma_write with immediate Errors across the board [1] send_recv Errors across the board [2] send_recv with immediate Errors across the board [3] 64-32 avg b/w (Mbps) peak b/w (Mbps) rdma_write Errors across the board [4] rdma_write with immediate Errors across the board [1] send_recv Errors across the board [2] send_recv with immediate Errors across the board [3] 64-64 avg b/w (Mbps) peak b/w (Mbps) rdma_write 2.6.15-rc7-git4-svn (4662) 1037.22 1840.01 2.6.15-rc7-git5-svn (4663) Errors [5] 2.6.15-rc7-git6-svn (4670) 1037.58 1839.99 2.6.15-rc7-git6-svn (4692) 1037.28 1839.97 2.6.15-svn (4714) 1036.83 1839.98 2.6.15-svn (4785) 1037.32 1840.01 2.6.15-svn (4789) 1036.99 1840 2.6.15 (4789) 1036.74 1839.97 2.6.15-svn (4803) 1037.82 1839.95 2.6.15 (4803) 1038.05 1839.95 rdma_write with immediate Errors across the board [1] send_recv Errors across the board [2] send_recv with immediate Errors across the board [3] pingpong 32-32 b/w (Mbps) rc 2.6.15-rc7-git4-svn (4662) 962.96 2.6.15-rc7-git5-svn (4663) 959.49 2.6.15-rc7-git6-svn (4670) 956.59 2.6.15-rc7-git6-svn (4692) 961.30 2.6.15-svn (4714) 957.58 2.6.15-svn (4785) 961.83 2.6.15-svn (4789) 962.79 2.6.15-svn (4803) 960.01 2.6.15 (4789) 883.95 2.6.15 (4803) 957.97 srq 2.6.15-rc7-git4-svn (4662) 3318.61 2.6.15-rc7-git5-svn (4663) 3247.41 2.6.15-rc7-git6-svn (4670) 3320.13 2.6.15-rc7-git6-svn (4692) 3294.26 2.6.15-svn (4714) 3267.16 2.6.15-svn (4785) 3293.27 2.6.15-svn (4789) 3347.09 2.6.15 (4789) 3277.46 2.6.15-svn (4803) 3386.00 2.6.15 (4803) 3144.87 uc 2.6.15-rc7-git4-svn (4662) 966.24 2.6.15-rc7-git5-svn (4663) 968.46 2.6.15-rc7-git6-svn (4670) 965.45 2.6.15-rc7-git6-svn (4692) 964.91 2.6.15-svn (4714) 970.53 2.6.15-svn (4785) 962.19 2.6.15-svn (4789) 967.02 2.6.15 (4789) 969.64 2.6.15-svn (4803) 964.62 2.6.15 (4803) 967.62 ud 2.6.15-rc7-git4-svn (4662) 466.30 2.6.15-rc7-git5-svn (4663) 465.32 2.6.15-rc7-git6-svn (4670) 465.64 2.6.15-rc7-git6-svn (4692) 465.42 2.6.15-svn (4714) 463.22 2.6.15-svn (4785) 464.58 2.6.15-svn (4789) 465.56 2.6.15 (4789) 465.97 2.6.15-svn (4803) 462.77 2.6.15 (4803) 462.94 32-64 b/w (Mbps) rc 2.6.15-rc7-git4-svn (4662) 975.59 2.6.15-rc7-git5-svn (4663) 971.97 2.6.15-rc7-git6-svn (4670) 972.89 2.6.15-rc7-git6-svn (4692) 974.41 2.6.15-svn (4714) 968.46 2.6.15-svn (4785) 972.20 2.6.15-svn (4789) 967.26 2.6.15 (4789) 973.15 2.6.15-svn (4803) 968.55 2.6.15 (4803) 968.26 srq 2.6.15-rc7-git4-svn (4662) 3347.26 2.6.15-rc7-git5-svn (4663) 3301.56 2.6.15-rc7-git6-svn (4670) Error [5] 2.6.15-rc7-git6-svn (4692) 3330.42 2.6.15-svn (4714) 3337.54 2.6.15-svn (4785) 334.15 2.6.15-svn (4789) 3378.32 2.6.15 (4789) 3397.06 2.6.15-svn (4803) 3427.79 2.6.15 (4803) 3429.95 uc 2.6.15-rc7-git4-svn (4662) 977.55 2.6.15-rc7-git5-svn (4663) 963.25 2.6.15-rc7-git6-svn (4670) 979.67 2.6.15-rc7-git6-svn (4692) 977.61 2.6.15-svn (4714) 976.04 2.6.15-svn (4785) 974.37 2.6.15-svn (4789) 974.40 2.6.15 (4789) 978.41 2.6.15-svn (4803) 975.75 2.6.15 (4803) 977.76 ud 2.6.15-rc7-git4-svn (4662) 469.33 2.6.15-rc7-git5-svn (4663) 468.64 2.6.15-rc7-git6-svn (4670) 468.90 2.6.15-rc7-git6-svn (4692) 468.07 2.6.15-svn (4714) 468.14 2.6.15-svn (4785) 467.97 2.6.15-svn (4789) 467.30 2.6.15 (4789) 468.58 2.6.15-svn (4803) 462.95 2.6.15 (4803) 466.87 64-32 b/w (Mbps) rc 2.6.15-rc7-git4-svn (4662) 974.95 2.6.15-rc7-git5-svn (4663) 972.01 2.6.15-rc7-git6-svn (4670) 974.17 2.6.15-rc7-git6-svn (4692) 970.75 2.6.15-svn (4714) 970.04 2.6.15-svn (4785) 964.25 2.6.15-svn (4789) 973.72 2.6.15 (4789) 974.80 2.6.15-svn (4803) 969.02 2.6.15 (4803) 968.62 srq 2.6.15-rc7-git4-svn (4662) 3312.58 2.6.15-rc7-git5-svn (4663) 3354.46 2.6.15-rc7-git6-svn (4670) 3344.36 2.6.15-rc7-git6-svn (4692) 3300.56 2.6.15-svn (4714) 3337.71 2.6.15-svn (4785) 3364.79 2.6.15-svn (4789) Error [5] 2.6.15 (4789) 3307.39 2.6.15-svn (4803) 3430.13 2.6.15 (4803) 3415.47 uc 2.6.15-rc7-git4-svn (4662) 973.99 2.6.15-rc7-git5-svn (4663) 975.11 2.6.15-rc7-git6-svn (4670) 981.45 2.6.15-rc7-git6-svn (4692) 978.91 2.6.15-svn (4714) 977.61 2.6.15-svn (4785) 974.28 2.6.15-svn (4789) 976.14 2.6.15 (4789) 975.51 2.6.15-svn (4803) 973.96 2.6.15 (4803) 972.13 ud 2.6.15-rc7-git4-svn (4662) 469.70 2.6.15-rc7-git5-svn (4663) 467.81 2.6.15-rc7-git6-svn (4670) Error [5] 2.6.15-rc7-git6-svn (4692) 467.97 2.6.15-svn (4714) 469.01 2.6.15-svn (4785) 468.09 2.6.15-svn (4789) 468.41 2.6.15 (4789) 468.94 2.6.15-svn (4803) 467.44 2.6.15 (4803) 467.53 64-64 b/w (Mbps) rc 2.6.15-rc7-git4-svn (4662) 980.93 2.6.15-rc7-git5-svn (4663) Error [5] 2.6.15-rc7-git6-svn (4670) 982.37 2.6.15-rc7-git6-svn (4692) 982.76 2.6.15-svn (4714) 968.68 2.6.15-svn (4785) 983.86 2.6.15-svn (4789) 982.61 2.6.15 (4789) 868.61 2.6.15-svn (4803) 982.68 2.6.15 (4803) 981.84 srq 2.6.15-rc7-git4-svn (4662) 3379.36 2.6.15-rc7-git5-svn (4663) Error [5] 2.6.15-rc7-git6-svn (4670) 3303.73 2.6.15-rc7-git6-svn (4692) 3354.80 2.6.15-svn (4714) 3345.55 2.6.15-svn (4785) 3376.58 2.6.15-svn (4789) 3408.89 2.6.15 (4789) 2546.47 2.6.15-svn (4803) 3461.84 2.6.15 (4803) 3450.72 uc 2.6.15-rc7-git4-svn (4662) 987.23 2.6.15-rc7-git5-svn (4663) Error [5] 2.6.15-rc7-git6-svn (4670) 989.21 2.6.15-rc7-git6-svn (4692) 989.24 2.6.15-svn (4714) 985.30 2.6.15-svn (4785) 984.91 2.6.15-svn (4789) 986.27 2.6.15 (4789) 983.80 2.6.15-svn (4803) 985.37 2.6.15 (4803) 990.13 ud 2.6.15-rc7-git4-svn (4662) 472.34 2.6.15-rc7-git5-svn (4663) Error [5] 2.6.15-rc7-git6-svn (4670) 471.73 2.6.15-rc7-git6-svn (4692) 471.38 2.6.15-svn (4714) 471.28 2.6.15-svn (4785) 471.79 2.6.15-svn (4789) 472.37 2.6.15 (4789) 471.68 2.6.15-svn (4803) 471.04 2.6.15 (4803) 470.53 perftest latency [6] 32-32 typical (us) min max rdma_lat 2.6.15-rc7-git4-svn (4662) 3.26954e+09 3.21451e+09 9.55805e+10 2.6.15-rc7-git5-svn (4663) 3.26015e+09 3.20109e+09 4.08654e+10 2.6.15-rc7-git6-svn (4670) 3.26552e+09 3.21586e+09 4.27914e+10 2.6.15-rc7-git6-svn (4692) 3.2682e+09 3.2172e+09 4.30169e+10 2.6.15-svn (4714) 3.26686e+09 3.20512e+09 4.5814e+10 2.6.15 (4785) 3.27491e+09 3.21988e+09 4.00909e+10 2.6.15-svn (4789) 3.26283e+09 3.20378e+09 4.42879e+10 2.6.15 (4789) 3.26686e+09 3.21586e+09 4.35323e+10 2.6.15-svn (4803) 3.27491e+09 3.22659e+09 5.65581e+10 2.6.15 (4803) 3.27625e+09 3.2172e+09 3.89313e+10 read_lat 2.6.15-svn (4803) 7274600857.60 7041062010.88 21923123691.52 2.6.15 (4803) 7282653921.28 7065221201.92 20535312384.00 send_lat 2.6.15-svn (4803) 4954647429.12 4771440230.40 43512045240.21 2.6.15 (4803) 4969411379.20 4799625953.28 46604421693.32 write_lat 2.6.15-svn (4803) 3291018690.56 3235989422.08 44894487838.72 2.6.15 (4803) 3286992158.72 3234647244.80 42607417753.60 32-64 typical (us) min max rdma_lat 2.6.15-rc7-git4-svn (4662) 0.750625 0.7375 10.2784 2.6.15-rc7-git5-svn (4663) 0.753437 0.73875 10.1547 2.6.15-rc7-git6-svn (4670) 0.754687 0.740938 10.4453 2.6.15-rc7-git6-svn (4692) 0.753437 0.74 9.07281 2.6.15-svn (4714) 0.75625 0.744375 9.80438 2.6.15-svn (4785) 0.742812 0.74125 9.77406 2.6.15-svn (4789) 0.756875 0.744687 9.83906 2.6.15 (4789) 0.753125 0.74 9.15125 2.6.15-svn (4803) 0.753437 0.741875 12.2256 2.6.15 (4803) 0.75875 0.745 11.3766 read_lat 2.6.15-svn (4803) 1.63 1.58 8.59 2.6.15 (4803) 1.63 1.59 8.98 send_lat 2.6.15-svn (4803) 1.12 1.09 10.51 2.6.15 (4803) 1.13 1.09 9.44 write_lat 2.6.15-svn (4803) 0.75 0.74 10.24 2.6.15 (4803) 0.76 0.74 11.84 64-32 typical (us) min max rdma_lat 2.6.15-rc7-git4-svn (4662) 3.22659e+09 3.17693e+09 4.30155e+10 2.6.15-rc7-git5-svn (4663) 3.22257e+09 3.16485e+09 4.63898e+10 2.6.15-rc7-git6-svn (4670) 3.21586e+09 3.1662e+09 4.29565e+10 2.6.15-rc7-git6-svn (4692) 3.23733e+09 3.17022e+09 4.34558e+10 2.6.15-svn (4714) 3.22794e+09 3.17425e+09 1.09938e+11 2.6.15-svn (4785) 3.23499e+09 3.17828e+09 4.08023e+10 2.6.15-svn (4789) 3.22928e+09 3.17425e+09 5.84103e+10 2.6.15 (4789) 3.2172e+09 3.17425e+09 5.45797e+10 2.6.15-svn (4803) 3.22391e+09 3.17291e+09 4.00372e+10 2.6.15 (4803) 3.23331e+09 3.17559e+09 1.30541e+12 read_lat 2.6.15-svn (4803) 7296075694.08 7041062010.88 71653476270.08 2.6.15 (4803) 7282653921.28 7046430720.00 20787641712.64 send_lat 2.6.15-svn (4803) 4818416435.20 4674803466.24 46003126271.89 2.6.15 (4803) 4821100789.76 4689567416.32 42077257727.89 write_lat 2.6.15-svn (4803) 3245384663.04 3201092812.80 44909251788.80 2.6.15 (4803) 3244042485.76 3193039749.12 59760443392.00 64-64 typical (us) min max rdma_lat 2.6.15-rc7-git4-svn (4662) Errors [5] 2.6.15-rc7-git5-svn (4663) Errors [5] 2.6.15-rc7-git6-svn (4670) 0.745625 0.731563 10.5003 2.6.15-rc7-git6-svn (4692) 0.745313 0.73375 10.3 2.6.15-svn (4714) 0.747812 0.736875 24.7016 2.6.15-svn (4785) 0.745 0.735313 10.5237 2.6.15-svn (4789) 0.7425 0.73125 9.94781 2.6.15 (4789) 0.745938 0.736563 11.0213 2.6.15-svn (4803) 0.742188 0.730938 9.91563 2.6.15 (4803) 0.7425 0.730625 15.0122 read_lat 2.6.15-svn (4803) 1.63 1.58 8.61 2.6.15 (4803) 1.63 1.58 44.82 send_lat 2.6.15-svn (4803) 1.10 1.07 11.21 2.6.15 (4803) 1.10 1.07 10.89 write_lat 2.6.15-svn (4803) 0.74 0.73 11.00 2.6.15 (4803) 0.75 0.74 10.53 perftest bandwidth 32-32 peak (MBps) avg rdma_bw 2.6.15-rc7-git4-svn (4662) 4.34461e-07 4.34461e-07 2.6.15-rc7-git5-svn (4663) 4.34457e-07 4.34457e-07 2.6.15-rc7-git6-svn (4670) 4.34455e-07 4.34454e-07 2.6.15-rc7-git6-svn (4692) 4.3451e-07 4.3451e-07 2.6.15-svn (4714) 4.34508e-07 4.34508e-07 2.6.15 (4785) 4.34518e-07 4.34518e-07 2.6.15-svn (4789) 4.34452e-07 4.34452e-07 2.6.15 (4789) 4.34453e-07 4.34453e-07 2.6.15-svn (4803) 4.34503e-07 4.34502e-07 2.6.15 (4803) 4.34446e-07 4.34446e-07 read_bw 2.6.15-svn (4803) 0.00 0.00 2.6.15 (4803) 0.00 0.00 send_bw 2.6.15-svn (4803) 0.00 0.00 2.6.15 (4803) 0.00 0.00 write_bw 2.6.15-svn (4803) 0.00 0.00 2.6.15 (4803) 0.00 0.00 32-64 peak (MBps) avg rdma_bw 2.6.15-rc7-git4-svn (4662) 1866.19 1866.17 2.6.15-rc7-git5-svn (4663) 1866.16 1866.16 2.6.15-rc7-git6-svn (4670) 1865.88 1865.87 2.6.15-rc7-git6-svn (4692) 1865.98 1865.97 2.6.15-svn (4714) 1865.88 1865.86 2.6.15-svn (4785) 1866.16 1866.14 2.6.15-svn (4789) 1865.95 1865.95 2.6.15 (4789) 1865.95 1865.92 2.6.15-svn (4803) 1865.92 1865.89 2.6.15 (4803) 1865.92 1865.9 read_bw 2.6.15-svn (4803) 1840.20 1840.19 2.6.15 (4803) 1840.74 1840.71 send_bw 2.6.15-svn (4803) 1840.16 1840.16 2.6.15 (4803) 1840.37 1840.36 write_bw 2.6.15-svn (4803) 1841.55 1841.52 2.6.15 (4803) 1841.32 1841.31 64-32 peak (MBps) avg rdma_bw 2.6.15-rc7-git4-svn (4662) 4.3449e-07 4.34489e-07 2.6.15-rc7-git5-svn (4663) 4.34456e-07 4.34456e-07 2.6.15-rc7-git6-svn (4670) 4.34458e-07 4.34458e-07 2.6.15-rc7-git6-svn (4692) 4.34507e-07 4.34507e-07 2.6.15-svn (4714) 4.34463e-07 4.34463e-07 2.6.15-svn (4785) 4.34454e-07 4.34454e-07 2.6.15-svn (4789) 4.34455e-07 4.34455e-07 2.6.15 (4789) 4.34443e-07 4.34443e-07 2.6.15-svn (4803) 4.34458e-07 4.34458e-07 2.6.15 (4803) 4.34501e-07 4.34501e-07 read_bw 2.6.15-svn (4803) 0.00 0.00 2.6.15 (4803) 0.00 0.00 send_bw 2.6.15-svn (4803) 0.00 0.00 2.6.15 (4803) 0.00 0.00 write_bw 2.6.15-svn (4803) 0.00 0.00 2.6.15 (4803) 0.00 0.00 64-64 peak (MBps) avg rdma_bw 2.6.15-rc7-git4-svn (4662) Errors [5] 2.6.15-rc7-git5-svn (4663) Errors [5] 2.6.15-rc7-git6-svn (4670) 1865.95 1865.93 2.6.15-rc7-git6-svn (4692) 1865.95 1865.94 2.6.15-svn (4714) 1865.92 1865.91 2.6.15-svn (4785) 1865.92 1865.9 2.6.15-svn (4789) 1865.92 1865.89 2.6.15 (4789) 1866.16 1866.15 2.6.15-svn (4803) 1866.19 1866.16 2.6.15 (4803) 1865.95 1865.93 read_bw 2.6.15-svn (4803) 1840.50 1840.48 2.6.15 (4803) 1840.47 1840.46 send_bw 2.6.15-svn (4803) 1840.40 1840.38 2.6.15 (4803) 1840.20 1840.20 write_bw 2.6.15-svn (4803) 1841.32 1841.31 2.6.15 (4803) 1841.35 1841.32 Thanks, Nish [1] Preposting asynchronous receives (required for Infiniband) Error, local polling may only be used with RDMA Write. Try using vapi polling or event completion Using RDMA Write communications with immediate data [2] Preposting asynchronous receives (required for Infiniband) Error, local polling may only be used with RDMA Write. Try using vapi polling or event completion Using Send/Receive communications [3] Preposting asynchronous receives (required for Infiniband) Error, local polling may only be used with RDMA Write. Try using vapi polling or event completion Using Send/Receive communications with immediate data [4] Preposting asynchronous receives (required for Infiniband) NetPIPE: error writing or reading synchronization string: Connection reset by peer Using RDMA Write communications This seems to be an actual error, not necessarily a programmatic one like [1]-[3]. It seems netPIPE over IB verbs is unhappy with mixing the size of the server client (does ok when they are the same, though). [5] These errors are due to our test grid, and thus are not indicative of errors in the code. [6] Just like with the results I posted earlier, all the perftest results are seriously wrong for 32-bit clients (with both 32-bit and 64-bit servers). I am not sure who else to notify beyond the general list (is there a corresponding MAINTAINERS files like in the kernel proper for the OpenIB code?) _______________________________________________ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general