[ofa-general] Memory registration limit of 16GB with Chelsio

[pandit ib]

Hi,

In our testing we are hitting a memory registration limit of 16GB with
the Chelsio cards.

Here are some sample results for different region sizes.

region size                 Number of Successful registration        Total
2MB                          8178
             ~16GB
16MB                        1021
            ~16GB
128MB                       127
            ~16GB

Are there any knobs that we can use to raise that limit?

For example, on mlx4 we use "log_num_mtt" module load parameter to
increase the limit.

Here is a sample output of the test:

[r...@lv2 ~]# ./mr-test 16777216

mr-test: bufsize 16777216
device # 0 name="cxgb3_0" guid="00074305ca890000"
        ibv_open_device() context=0x77b5100
        ibv_alloc_pd() pd=0x77b50e0
        alloc: 1021
        ibv_reg_mr failed:: Cannot allocate memory
        fw_ver: 7.4.0
        max_mr_size 0x100000000
        max_mr: 32768, could only register 1021 regions; Region size 16MB
        sleep 5 sec
        free: 0
done
_______________________________________________
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general