Hi Parag - I've been away from this field for awhile, but this message may be the most helpful at this point:
modprobe: ERROR: could not insert 'ko2iblnd': Invalid argument Last I knew, there were two IB releases - Mellanox/MOFED and Intel (formerly QLogic) - and the Lustre kernel you use needs to be compiled against the IB release that is installed on the node. So, I see a couple possibilities of why the Lustre IB module will not load: 1. Since you are using MOFED, the kernel you have may instead support Intel IB (thus the invalid argument). 2. One of the options provided may not be valid for the module, or may conflict with another provided option: require_privileged_port=0 use_privileged_port=0 timeout=150 retry_count=7 map_on_demand=32 peer_credits=63 concurrent_sends=63 ntx=32768 credits=32768 fmr_pool_size=8193 As item 1 is probably a binary, and item 2 a matrix, confirming that the kernel that you have supports the module that you are trying to load seems to be a better starting point. :) As Intel HPDD seems to provide a kernel that matches the version are using, item 1 seems likely. https://downloads.hpdd.intel.com/public/lustre/lustre-2.9.0/el7.3.1611/server/RPMS/x86_64/ If that is the case, a couple options include using the Intel IB, or compiling a Lustre kernel that supports the Mellanox OFED. Hope this helps. Brett On Sun, Nov 12, 2017 at 1:16 AM, Parag Khuraswar <para...@citilindia.com> wrote: > Hi Brett, > > > > I am using MOFED “MLNX_OFED_LINUX-4.1-1.0.2.0” with kernel > “3.10.0-514.el7.x86_64” > > > > o/p of modprobe –v ko2iblnd > > > > [root@mds1 ~]# modprobe -v ko2iblnd > > install /usr/sbin/ko2iblnd-probe require_privileged_port=0 > use_privileged_port=0 timeout=150 retry_count=7 map_on_demand=32 > peer_credits=63 concurrent_sends=63 ntx=32768 credits=32768 > fmr_pool_size=8193 > > insmod /lib/modules/3.10.0-514.el7.x86_64/extra/lustre/net/ko2iblnd.ko > require_privileged_port=0 use_privileged_port=0 timeout=150 retry_count=7 > map_on_demand=32 peer_credits=63 concurrent_sends=63 ntx=32768 > credits=32768 fmr_pool_size=8193 > > modprobe: ERROR: could not insert 'ko2iblnd': Invalid argument > > modprobe: ERROR: Error running install command for ko2iblnd > > modprobe: ERROR: could not insert 'ko2iblnd': Operation not permitted > > [root@mds1 ~]# > > > > > > Regards, > > Parag > > > > > > *From:* Brett Lee [mailto:brettlee.lus...@gmail.com] > *Sent:* Saturday, November , 2017 8:08 PM > *To:* Parag Khuraswar > *Cc:* Mannthey, Keith; lustre-discuss@lists.lustre.org > *Subject:* Re: [lustre-discuss] error while configuring lnet > > > > Hi Parag, > > > > You may need to confirm that the in-kernel IB and the IB in the kernel > module "match" (are compatible). I think that loading the module (`sudo > modprobe -v ko2iblnd`) may be sufficient to verify the match (it's been a > while, others may correct me). > > > > Please indicate which kernel and which IB you are using. > > > Brett > > -- > > Protect yourself against cybercrime > > PDS Software Solutions > > https://www.TrustPDS.com <https://www.trustpds.com/> > > > > On Fri, Nov 10, 2017 at 8:30 PM, Parag Khuraswar <para...@citilindia.com> > wrote: > > Hi Keith, > > > > Below errors I am getting while adding lnet and mounting mdt. > > > > dmesg logs while adding lnet > > ========================================= > > [317831.432182] LNetError: 28362:0:(api-ni.c:1861:lnet_startup_lndnet()) > Can't load LND o2ib, module ko2iblnd, rc=256 > > ========================================= > > > > > > > > dmesg logs while mounting mdt > > ========================================== > > [290476.172602] LNetError: 23040:0:(api-ni.c:1861:lnet_startup_lndnet()) > Can't load LND o2ib, module ko2iblnd, rc=256 > > [317478.730515] LDISKFS-fs (dm-1): mounted filesystem with ordered data > mode. Opts: errors=remount-ro > > [317480.166277] LDISKFS-fs (dm-1): mounted filesystem with ordered data > mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc > > [317480.313296] LustreError: 28268:0:(ldlm_lib.c:483:client_obd_setup()) > can't add initial connection > > [317480.313600] LustreError: 28268:0:(obd_config.c:608:class_setup()) > setup MGC10.2.1.204@o2ib failed (-2) > > [317480.313603] LustreError: 28268:0:(obd_mount.c:202:lustre_start_simple()) > MGC10.2.1.204@o2ib setup error -2 > > [317480.313632] LustreError: > 28268:0:(obd_mount_server.c:1573:server_put_super()) > no obd home-MDT0000 > > [317480.313635] LustreError: > 28268:0:(obd_mount_server.c:132:server_deregister_mount()) > home-MDT0000 not registered > > [317480.433934] Lustre: server umount home-MDT0000 complete > > [317480.433940] LustreError: 28268:0:(obd_mount.c:1504:lustre_fill_super()) > Unable to mount (-2) > > ========================================== > > > > > > Regards, > > Parag > > > > > > *From:* Mannthey, Keith [mailto:keith.mannt...@intel.com] > *Sent:* Saturday, November , 2017 2:06 AM > > > *To:* Parag Khuraswar; lustre-discuss@lists.lustre.org > *Subject:* RE: [lustre-discuss] error while configuring lnet > > > > If you have ib0 device check dmesg for more hints on what is going wrong. > > > > Thanks, > > Keith > > *From:* Parag Khuraswar [mailto:para...@citilindia.com] > *Sent:* Friday, November 10, 2017 10:59 AM > *To:* Mannthey, Keith <keith.mannt...@intel.com>; > lustre-discuss@lists.lustre.org > *Subject:* RE: [lustre-discuss] error while configuring lnet > > > > Hi, > > > > Basically I am trying to add lnet. Deleting is just try whether it is > happing or not. > > Main is I want to add o2ib network. Which is giving error “invalid > argument ” > > ================== > > [root@mds2 ~]# lnetctl net add --net o2ib --if ib0 > > add: > > - net: > > errno: -22 > > descr: "cannot add network: Invalid argument" > > ================ > > I am really not able to understand what argument is invalid in my command. > > I am able to ping ib0 network > > > > Regards, > > Parag > > > > > > *From:* Mannthey, Keith [mailto:keith.mannt...@intel.com > <keith.mannt...@intel.com>] > *Sent:* Friday, November , 2017 10:51 PM > *To:* Parag Khuraswar; lustre-discuss@lists.lustre.org > *Subject:* RE: [lustre-discuss] error while configuring lnet > > > > What are you trying to accomplish? > > > > From below: > > > > 10.1.1.205@tcp is on 0@lo not eno1 and in general you should not need the > “—if” option to delete a fabric. > > > > Try: # lnetctl net del --net tcp > > > > Can you do a normal ping over ib0? > > > > “dmesg” can sometime provide greater details about errors like this. > > > > Thanks, > > Keith > > > > > > *From:* lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org > <lustre-discuss-boun...@lists.lustre.org>] *On Behalf Of *Parag Khuraswar > *Sent:* Friday, November 10, 2017 9:10 AM > *To:* lustre-discuss@lists.lustre.org > *Subject:* [lustre-discuss] error while configuring lnet > > > > Hi, > > > > I am trying to add lnet but getting below error. > > ====================== > > [root@mds2 ~]# lnetctl net show > > net: > > - net type: lo > > local NI(s): > > - nid: 0@lo > > status: up > > - net type: tcp > > local NI(s): > > - nid: 10.1.1.205@tcp > > status: up > > [root@mds2 ~]# lnetctl net add --net o2ib --if ib0 > > add: > > - net: > > errno: -22 > > descr: "cannot add network: Invalid argument" > > [root@mds2 ~]# lnetctl net del --net tcp --if eno1 > > del: > > - net: > > errno: -22 > > descr: "cannot del network: Invalid argument" > > [root@mds2 ~]# lctl list_nids > > 10.1.1.205@tcp > > [root@mds2 ~]# > > ==================================== > > > > > > Regards, > > Parag > > > > > > > _______________________________________________ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org > > >
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
