Hi All, I am trying to see whether hugetlbfs is improving the latency of communication with a small send receive program.
mpirun -np 2 --map-by core --bind-to core --mca pml ucx --mca opal_common_ucx_tls any --mca opal_common_ucx_devices any -mca pml_base_verbose 10 --mca mtl_base_verbose 10 -x OMPI_MCA_pml_ucx_verbose=10 -x UCX_LOG_LEVEL=debu g -x UCX_PROTO_INFO=y send_recv 1000 1 But the internal buffer allocation in ucx is unable to select the hugetlbfs. [1688297246.205092] [lib-ssp-04:4022755:0] ucp_context.c:1979 UCX DEBUG allocation method[2] is 'huge' [1688297246.208660] [lib-ssp-04:4022755:0] mm_sysv.c:97 UCX DEBUG mm failed to allocate 8447 bytes with hugetlb---------> I checked the code, this is a valid failure as the size is small compared to huge page size of 2MB [1688297246.208704] [lib-ssp-04:4022755:0] mm_sysv.c:97 UCX DEBUG mm failed to allocate 4292720 bytes with hugetlb [1688297246.210048] [lib-ssp-04:4022755:0] mm_posix.c:332 UCX DEBUG shared memory mmap(addr=(nil), length=6291456, flags= HUGETLB, fd=29) failed: Invalid argument [1688297246.211451] [lib-ssp-04:4022754:0] ucp_context.c:1979 UCX DEBUG allocation method[2] is 'huge' [1688297246.214849] [lib-ssp-04:4022754:0] mm_sysv.c:97 UCX DEBUG mm failed to allocate 8447 bytes with hugetlb [1688297246.214888] [lib-ssp-04:4022754:0] mm_sysv.c:97 UCX DEBUG mm failed to allocate 4292720 bytes with hugetlb [1688297246.216235] [lib-ssp-04:4022754:0] mm_posix.c:332 UCX DEBUG shared memory mmap(addr=(nil), length=6291456, flags= HUGETLB, fd=29) failed: Invalid argument Can someone suggest what are the steps to be done to enable hugetlbfs [I cannot run my application as root] ? Is using hugetlbfs for the internal buffers is recommended? --Arun
