Are you using truescale IB interfaces ?
There is a known truescale bug in rhel/centos 6.6 kernels. You should
try kernel 2.6.32-504.23.4 or newer. Some details of the bug are in
LU-6698 and RHSA-2015-1081.
regards,
chris hunter
yale hpc group
From: "Lassus, Magnus" <magnus.las...@wartsila.com>
To: "lustre-discuss@lists.lustre.org"
<lustre-discuss@lists.lustre.org>
Subject: [lustre-discuss] o2ib (ib_qib) with 2.7.0 rpms on centos 6.6:
LNetError: kiblnd_init_rdma: Src buffer exhausted: 1 frags
Message-ID:
<he1pr04mb1273c36e676e1824d8b2e3a494...@he1pr04mb1273.eurprd04.prod.outlook.com>
Content-Type: text/plain; charset="us-ascii"
Hi,
I fail to understand where I go wrong in getting o2ib working using 2.7.0 rpms
on top of CentOS 6.6. Running selftest I see:
Nov 17 18:22:40 ss08 kernel: LNet: Added LNI 10.165.32.18@o2ib [8/256/0/180]
Nov 17 18:24:40 ss08 kernel: LNetError:
12532:0:(o2iblnd_cb.c:1123:kiblnd_init_rdma()) Src buffer exhausted: 1 frags
Nov 17 18:24:40 ss08 kernel: LustreError:
12553:0:(brw_test.c:212:brw_check_page()) Bad data in page ffffea0070c20800:
0xbeefbeefbeefbeef, 0xeeb0eeb1eeb2eeb3 expec
Nov 17 18:24:40 ss08 kernel: LustreError:
12553:0:(brw_test.c:238:brw_check_bulk()) Bulk page ffffea0070c20800 (0/256) is
corrupted!
Nov 17 18:24:40 ss08 kernel: LustreError:
12553:0:(brw_test.c:343:brw_client_done_rpc()) Bulk data from
12345-10.165.32.18@o2ib is corrupted!
Nov 17 18:24:40 ss08 kernel: LNetError:
12532:0:(o2iblnd_cb.c:1690:kiblnd_reply()) Can't setup rdma for GET from
10.165.32.18@o2ib: -71
Nov 17 18:25:31 ss08 kernel: LNetError:
12529:0:(o2iblnd_cb.c:3036:kiblnd_check_txs_locked()) Timed out tx: active_txs,
0 seconds
Nov 17 18:25:31 ss08 kernel: LNetError:
12529:0:(o2iblnd_cb.c:3099:kiblnd_check_conns()) Timed out RDMA with
10.165.32.18@o2ib (0): c: 7, oc: 0, rc: 7
Nov 17 18:25:31 ss08 kernel: LustreError:
12558:0:(brw_test.c:388:brw_bulk_ready()) BRW bulk WRITE failed for RPC from
12345-10.165.32.18@o2ib: -103
Nov 17 18:25:31 ss08 kernel: LustreError:
12558:0:(brw_test.c:362:brw_server_rpc_done()) Bulk transfer from
12345-10.165.32.18@o2ib has failed: -5
Nov 17 18:25:48 ss08 kernel: LNet:
12581:0:(rpc.c:1077:srpc_client_rpc_expired()) Client RPC expired: service 11,
peer 12345-10.165.32.18@o2ib, timeout 64.
Nov 17 18:25:48 ss08 kernel: LustreError:
12555:0:(brw_test.c:318:brw_client_done_rpc()) BRW RPC to
12345-10.165.32.18@o2ib failed with -110
# rpm -qa | egrep 'lustre|kernel' | sort
dracut-kernel-004-356.el6.noarch
kernel-2.6.32-504.8.1.el6_lustre.x86_64
kernel-devel-2.6.32-504.8.1.el6_lustre.x86_64
kernel-firmware-2.6.32-504.8.1.el6_lustre.x86_64
kernel-headers-2.6.32-504.8.1.el6_lustre.x86_64
lustre-2.7.0-2.6.32_504.8.1.el6_lustre.x86_64.x86_64
lustre-iokit-2.7.0-2.6.32_504.8.1.el6_lustre.x86_64.x86_64
lustre-modules-2.7.0-2.6.32_504.8.1.el6_lustre.x86_64.x86_64
lustre-osd-ldiskfs-2.7.0-2.6.32_504.8.1.el6_lustre.x86_64.x86_64
lustre-osd-ldiskfs-mount-2.7.0-2.6.32_504.8.1.el6_lustre.x86_64.x86_64
lustre-tests-2.7.0-2.6.32_504.8.1.el6_lustre.x86_64.x86_64
perf-2.6.32-504.8.1.el6_lustre.x86_64
python-perf-2.6.32-504.8.1.el6_lustre.x86_64
Using latest 2.7.63 build on 6.7 works.
Any pointers are warmly welcome as I'd prefer to use 2.7.0.
Regards,
Magnus
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_pipermail_lustre-2Ddiscuss-2Dlustre.org_attachments_20151118_bc19b61a_attachment.html&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=d_G2h_sZYG4xtHMeKo8QgjDmOcMVdQvYgM-5Dri1AOY&m=yntd6s6FbhcK6yz7f--sTQB8uauio2sPpZXJO07_GMM&s=fmaW2S-MSdcgBPqEnTVELb9GaBrR0zwaQlFI9_QrbYw&e=
>
------------------------------
Subject: Digest Footer
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.lustre.org_listinfo.cgi_lustre-2Ddiscuss-2Dlustre.org&d=AwICAg&c=-dg2m7zWuuDZ0MUcV7Sdqw&r=d_G2h_sZYG4xtHMeKo8QgjDmOcMVdQvYgM-5Dri1AOY&m=yntd6s6FbhcK6yz7f--sTQB8uauio2sPpZXJO07_GMM&s=XPhf61e64WjkcxWw05wudsYWLfRBfsN0OiJF8O2DYE4&e=
------------------------------
End of lustre-discuss Digest, Vol 116, Issue 9
**********************************************
_______________________________________________
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
