Bob, Our max locked memory was already set to unlimited.
thanks K. Can ----- Original Message ----- From: Robert D. Russell <r...@iol.unh.edu> To: Ker Can <kercan7...@yahoo.com> Cc: "ewg@lists.openfabrics.org" <ewg@lists.openfabrics.org> Sent: Friday, March 23, 2012 5:08 PM Subject: Re: [ewg] ibv_create_qp fails with enomem Ker: In your /etc/security/limits.conf file, try adding the lines * hard memlock unlimited * soft memlock unlimited We hit the same problem and this cured it. Bob Russell r...@iol.unh.edu On Fri, 23 Mar 2012, Ker Can wrote: > Sorry about all the formatting - trying again with plain text. > > We updated to MLNX_OFED_LINUX-1.5.3-3.0.0 from OFED-1.5.2_1 recently and > we're running into issues where we get a ENOMEM error from ibv_create_qp(). > Earlier with this OFED version we were running into a ENOMEM with > ibv_reg_mr() but that was resolved by setting log_num_mtt to 24 as described > in this link > http://h10025.www1.hp.com/ewfrf/wc/document?cc=us&lc=en&dlc=en&docname=c03113904 > All the other settings in /sys/module/mlx4_core/parameters are the default > values. The node itself has 96GB of memory we're nowhere close to using > that. Is there any way to figure out whats going on ? > > thanks > K. Can > > here's the output from ibv_devinfo -v: > > hca_id: mlx4_0 > transport: InfiniBand (0) > fw_ver: 2.9.1000 > node_guid: 0002:c903:004b:d5e6 > sys_image_guid: 0002:c903:004b:d5e9 > vendor_id: 0x02c9 > vendor_part_id: 26428 > hw_ver: 0xB0 > board_id: HP_0180000009 > phys_port_cnt: 2 > max_mr_size: 0xffffffffffffffff > page_size_cap: 0xfffffe00 > max_qp: 260032 > max_qp_wr: 16351 > device_cap_flags: 0x007c9c76 > max_sge: 32 > max_sge_rd: 0 > max_cq: 65408 > max_cqe: 4194303 > max_mr: 524272 > max_pd: 32764 > max_qp_rd_atom: 16 > max_ee_rd_atom: 0 > max_res_rd_atom: 4160512 > max_qp_init_rd_atom: 128 > max_ee_init_rd_atom: 0 > atomic_cap: ATOMIC_HCA (1) > max_ee: 0 > max_rdd: 0 > max_mw: 0 > max_raw_ipv6_qp: 0 > max_raw_ethy_qp: 2 > max_mcast_grp: 8192 > max_mcast_qp_attach: 120 > max_total_mcast_qp_attach: 983040 > max_ah: 0 > max_fmr: 0 > max_srq: 65472 > max_srq_wr: 16383 > max_srq_sge: 31 > max_pkeys: 128 > local_ca_ack_delay: 15 > port: 1 > state: PORT_ACTIVE (4) > max_mtu: 2048 (4) > active_mtu: 2048 (4) > sm_lid: 9 > port_lid: 311 > port_lmc: 0x00 > link_layer: IB > max_msg_sz: 0x40000000 > port_cap_flags: 0x02510868 > max_vl_num: 8 (4) > bad_pkey_cntr: 0x0 > qkey_viol_cntr: 0x0 > sm_sl: 0 > pkey_tbl_len: 128 > gid_tbl_len: 128 > subnet_timeout: 18 > init_type_reply: 0 > active_width: 4X (2) > active_speed: 10.0 Gbps (4) > phys_state: LINK_UP (5) > GID[ 0]: > fe80:0000:0000:0000:0002:c903:004b:d5e7 > > port: 2 > state: PORT_ACTIVE (4) > max_mtu: 2048 (4) > active_mtu: 2048 (4) > sm_lid: 9 > port_lid: 158 > port_lmc: 0x00 > link_layer: IB > max_msg_sz: 0x40000000 > port_cap_flags: 0x02510868 > max_vl_num: 8 (4) > bad_pkey_cntr: 0x0 > qkey_viol_cntr: 0x0 > sm_sl: 0 > pkey_tbl_len: 128 > gid_tbl_len: 128 > subnet_timeout: 18 > init_type_reply: 0 > active_width: 4X (2) > active_speed: 10.0 Gbps (4) > phys_state: LINK_UP (5) > GID[ 0]: > fe80:0000:0000:0000:0002:c903:004b:d5e8 > _______________________________________________ > ewg mailing list > ewg@lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > _______________________________________________ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg