Hi Howard,
Thank you very much for the reply.
Ucx is trying to setup the FIFO for shared memory communication using both sysv
and posix.
By default, these allocations are failing when tried with hugetlbfs
a) Failure log from strace(Pasting only for rank0):
[pid 3541286] shmget(IPC_PRIVATE, 6291456,
IPC_CREAT|IPC_EXCL|SHM_HUGETLB|0660) = -1 EPERM (Operation not permitted)
[pid 3541286] mmap(NULL, 6291456, PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_HUGETLB, 29, 0) = -1 EINVAL (Invalid argument)
b) I was able to overcome the failure for shmget allocation with hugetlbfs by
adding my gid to "/proc/sys/vm/hugetlb_shm_group"
[pid 3541465] shmget(IPC_PRIVATE, 6291456,
IPC_CREAT|IPC_EXCL|SHM_HUGETLB|0660) = 2916410--> success
[pid 3541465] mmap(NULL, 6291456, PROT_READ|PROT_WRITE,
MAP_SHARED|MAP_HUGETLB, 29, 0) = -1 EINVAL (Invalid argument) --> still fail
But mmap with " MAP_SHARED|MAP_HUGETLB" is still failing. Any clues?
I am aware of the advantages of huge pagetables, I am asking from the openmpi
library perspective,
Should I use it for openmpi internal buffers and data structures or leave it
for the applications to use?
What are the community recommendations in this regard?
--Arun
-----Original Message-----
From: Pritchard Jr., Howard <[email protected]>
Sent: Thursday, July 20, 2023 9:36 PM
To: Open MPI Users <[email protected]>; Florent GERMAIN
<[email protected]>
Cc: Chandran, Arun <[email protected]>
Subject: Re: [EXTERNAL] Re: [OMPI users] How to use hugetlbfs with openmpi and
ucx
HI Arun,
Its going to be chatty, but you may want to see if strace helps in diagnosing:
mpirun -np 2 (all your favorite mpi args) strace -f send_recv 1000 1
huge pages often helps reduce pressure on a NIC's I/O MMU widget and speeds up
resolving va to pa memory addresses.
On 7/19/23, 9:24 PM, "users on behalf of Chandran, Arun via users"
<[email protected] <mailto:[email protected]> on
behalf of [email protected] <mailto:[email protected]>> wrote:
Good luck,
Howard
Hi,
I am trying to use static huge pages, not transparent huge pages.
Ucx is allowed to allocate via hugetlbfs.
$ ./bin/ucx_info -c | grep -i huge
UCX_SELF_ALLOC=huge,thp,md,mmap,heap
UCX_TCP_ALLOC=huge,thp,md,mmap,heap
UCX_SYSV_HUGETLB_MODE=try --->It is trying this and failing
UCX_SYSV_FIFO_HUGETLB=n UCX_POSIX_HUGETLB_MODE=try---> it is trying this and
failing UCX_POSIX_FIFO_HUGETLB=n
UCX_ALLOC_PRIO=md:sysv,md:posix,huge,thp,md:*,mmap,heap
UCX_CMA_ALLOC=huge,thp,mmap,heap
It is failing even though I have static hugepages available in my system.
$ cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 20
THP is also enabled:
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
--Arun
-----Original Message-----
From: Florent GERMAIN <[email protected]
<mailto:[email protected]>>
Sent: Wednesday, July 19, 2023 7:51 PM
To: Open MPI Users <[email protected]
<mailto:[email protected]>>; Chandran, Arun <[email protected]
<mailto:[email protected]>>
Subject: RE: How to use hugetlbfs with openmpi and ucx
Hi,
You can check if there are dedicated huge pages on your system or if
transparent huge pages are allowed.
Transparent huge pages on rhel systems :
$cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never
-> this means that transparent huge pages are selected through mmap +
-> madvise always = always try to aggregate pages on thp (for large
-> enough allocation with good alignment) never = never try to aggregate
-> pages on thp
Dedicated huge pages on rhel systems :
$ cat /proc/meminfo | grep HugePages_Total
HugePages_Total: 0
-> no dedicated huge pages here
It seems that ucx tries to use dedicated huge pages (mmap(addr=(nil),
length=6291456, flags= HUGETLB, fd=29)).
If there are no dedicated huge pages available, mmap fails.
Huge pages can accelerate virtual address to physical address translation and
reduce TLB consumption.
It may be useful for large and frequently used buffers.
Regards,
Florent
-----Message d'origine-----
De : users <[email protected]
<mailto:[email protected]>> De la part de Chandran, Arun via
users Envoyé : mercredi 19 juillet 2023 15:44 À : [email protected]
<mailto:[email protected]> Cc : Chandran, Arun <[email protected]
<mailto:[email protected]>> Objet : [OMPI users] How to use hugetlbfs with
openmpi and ucx
Hi All,
I am trying to see whether hugetlbfs is improving the latency of communication
with a small send receive program.
mpirun -np 2 --map-by core --bind-to core --mca pml ucx --mca
opal_common_ucx_tls any --mca opal_common_ucx_devices any -mca pml_base_verbose
10 --mca mtl_base_verbose 10 -x OMPI_MCA_pml_ucx_verbose=10 -x
UCX_LOG_LEVEL=debu g -x UCX_PROTO_INFO=y send_recv 1000 1
But the internal buffer allocation in ucx is unable to select the hugetlbfs.
[1688297246.205092] [lib-ssp-04:4022755:0] ucp_context.c:1979 UCX DEBUG
allocation method[2] is 'huge'
[1688297246.208660] [lib-ssp-04:4022755:0] mm_sysv.c:97 UCX DEBUG mm failed to
allocate 8447 bytes with hugetlb---------> I checked the code, this is a valid
failure as the size is small compared to huge page size of 2MB
[1688297246.208704] [lib-ssp-04:4022755:0] mm_sysv.c:97 UCX DEBUG mm failed to
allocate 4292720 bytes with hugetlb [1688297246.210048] [lib-ssp-04:4022755:0]
mm_posix.c:332 UCX DEBUG shared memory mmap(addr=(nil), length=6291456, flags=
HUGETLB, fd=29) failed: Invalid argument [1688297246.211451]
[lib-ssp-04:4022754:0] ucp_context.c:1979 UCX DEBUG allocation method[2] is
'huge'
[1688297246.214849] [lib-ssp-04:4022754:0] mm_sysv.c:97 UCX DEBUG mm failed to
allocate 8447 bytes with hugetlb [1688297246.214888] [lib-ssp-04:4022754:0]
mm_sysv.c:97 UCX DEBUG mm failed to allocate 4292720 bytes with hugetlb
[1688297246.216235] [lib-ssp-04:4022754:0] mm_posix.c:332 UCX DEBUG shared
memory mmap(addr=(nil), length=6291456, flags= HUGETLB, fd=29) failed: Invalid
argument
Can someone suggest what are the steps to be done to enable hugetlbfs [I cannot
run my application as root] ? Is using hugetlbfs for the internal buffers is
recommended?
--Arun
