-----Original Message----- From: Steve Wise [mailto:[email protected]] Sent: Friday, July 10, 2009 11:46 AM To: pandit ib Cc: [email protected]; Felix Marti Subject: Re: [ofa-general] Memory registration limit of 16GB with Chelsio
Looks like you're running out of adapter resources for storing the physical buffers lists (PBLs). The cards have 32MB of memory for storing PBLs. 32MB of memory can store 4MB of 8B physical addresses. For user mode, your page size is 4KB, so that maps out to 4MB * 4KB = 16GB of memory. Hey Felix, Can we somehow adjust the PMRX memory map dynamically? For instance, maybe drop the number of TPTs to allow for more PBL memory? [felix] Not dynamically, but it can surely be changed. DM, can it be controlled through a cxgbtool? Steve pandit ib wrote: > Hi, > > In our testing we are hitting a memory registration limit of 16GB with > the Chelsio cards. > > Here are some sample results for different region sizes. > > region size Number of Successful registration Total > 2MB 8178 > ~16GB > 16MB 1021 > ~16GB > 128MB 127 > ~16GB > > Are there any knobs that we can use to raise that limit? > > For example, on mlx4 we use "log_num_mtt" module load parameter to > increase the limit. > > Here is a sample output of the test: > > [r...@lv2 ~]# ./mr-test 16777216 > > mr-test: bufsize 16777216 > device # 0 name="cxgb3_0" guid="00074305ca890000" > ibv_open_device() context=0x77b5100 > ibv_alloc_pd() pd=0x77b50e0 > alloc: 1021 > ibv_reg_mr failed:: Cannot allocate memory > fw_ver: 7.4.0 > max_mr_size 0x100000000 > max_mr: 32768, could only register 1021 regions; Region size 16MB > sleep 5 sec > free: 0 > done > _______________________________________________ > general mailing list > [email protected] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
