I'm writing a unit test that takes 12 GiB of RAM, and want to test whether it is adequate to run this test.
In a first attempt, I wrote: if (physmem_total () / INT_MAX < 6.0) /* run the test */ but this resulted in the test not being run, although my machine has plenty of RAM: $ ./test-vasnprintf-big avail = 4534.64 MiB Skipping test: not enough memory available $ free total used free shared buff/cache available Mem: 63579 22159 4533 1184 36887 39533 Swap: 19072 2566 16506 The reason is that physmem_available(), like sysconf (_SC_AVPHYS_PAGES), returns, as documented in the glibc manual [1] "the amount of memory the application can use without hindering any other process (given that no other process increases its memory usage)" and that is the number of "free" pages. In my case, with 4.5 GiB "free" pages plus 39 GiB "available" pages, it is completely adequate to run a test that takes 12 GiB of RAM. The Linux memory management usually keeps the number of free pages relatively small over time (because free memory is wasted memory). So, to determine whether it is OK to take a certain amount of memory — of course, while slowing down other processes a bit, but without crashing the system — we must look at the other categories of memory. The figures from /proc/meminfo are quite detailed: $ cat /proc/meminfo MemTotal: 65105348 kB MemFree: 4488440 kB MemAvailable: 40354724 kB Buffers: 2738416 kB Cached: 30930548 kB SwapCached: 239900 kB Active: 15246788 kB Inactive: 39476552 kB Active(anon): 1143740 kB Inactive(anon): 21123360 kB Active(file): 14103048 kB Inactive(file): 18353192 kB Unevictable: 208 kB Mlocked: 208 kB SwapTotal: 19530748 kB SwapFree: 16902908 kB Dirty: 100 kB Writeback: 0 kB AnonPages: 20842972 kB Mapped: 1855324 kB Shmem: 1212724 kB KReclaimable: 4129976 kB Slab: 4644636 kB SReclaimable: 4129976 kB SUnreclaim: 514660 kB KernelStack: 64720 kB PageTables: 165636 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 52083420 kB Committed_AS: 49274636 kB VmallocTotal: 34359738367 kB VmallocUsed: 116336 kB VmallocChunk: 0 kB Percpu: 23040 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 21034368 kB DirectMap2M: 36941824 kB DirectMap1G: 9437184 kB These figures are documented in [2][3][4][5]. In particular, the "Inactive(file)" number is memory that is used in the file cache. When pages from this pool are reclaimed, of course some other processes will slow down. But we know that reclaiming less than 100% of the pages from the file cache will not bring the machine to its knees. So, here is a patch that adds another function physmem_claimable, which returns a larger value than physmem_available. A call physmem_claimable (0.5) returns 4 GiB + 0.5 * 18 GiB = 13 GiB, which is sufficient for running the 12 GiB test. [1] https://www.gnu.org/software/libc/manual/html_node/Query-Memory-Parameters.html [2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-proc-meminfo [3] https://man7.org/linux/man-pages/man5/proc.5.html [4] https://superuser.com/questions/521551/cat-proc-meminfo-what-do-all-those-numbers-mean [5] https://www.baeldung.com/linux/proc-meminfo 2024-04-24 Bruno Haible <br...@clisp.org> physmem: Port better to Linux. * lib/physmem.h (physmem_total, physmem_available): Add documentation. (physmem_claimable): New declaration. * lib/physmem.c: Include <fcntl.h>, <stdio.h>, full-read.h. (get_meminfo): New function. (physmem_claimable): Renamed from physmem_available. Add logic for aggressivity > 0. (physmem_available): New function. * modules/physmem (Depends-on): Add full-read. diff --git a/lib/physmem.h b/lib/physmem.h index 957e629180..542dbf214c 100644 --- a/lib/physmem.h +++ b/lib/physmem.h @@ -25,9 +25,27 @@ extern "C" { #endif +/* Returns the total amount of physical memory. + This value is more or less a hard limit for the working set. */ double physmem_total (void); + +/* Returns the amount of physical memory available. + This value is the amount of memory the application can use without hindering + any other process (assuming that no other process increases its memory + usage). */ double physmem_available (void); +/* Returns the amount of physical memory that can be claimed, with a given + aggressivity. + For AGGRESSIVITY == 0.0, the result is like physmem_available (): the amount + of memory the application can use without hindering any other process. + For AGGRESSIVITY == 1,0, the result is the amount of memory the application + can use, while causing memory shortage to other processes, but without + bringing the machine into an out-of-memory state. + Values in between, for example AGGRESSIVITY == 0.5, are a reasonable middle + ground. */ +double physmem_claimable (double aggressivity); + #ifdef __cplusplus } diff --git a/lib/physmem.c b/lib/physmem.c index e6eb26b5f3..5c226b8abf 100644 --- a/lib/physmem.c +++ b/lib/physmem.c @@ -22,25 +22,28 @@ #include "physmem.h" +#include <fcntl.h> +#include <stdio.h> #include <unistd.h> -#if HAVE_SYS_PSTAT_H +#if HAVE_SYS_PSTAT_H /* HP-UX */ # include <sys/pstat.h> #endif -#if HAVE_SYS_SYSMP_H +#if HAVE_SYS_SYSMP_H /* IRIX */ # include <sys/sysmp.h> #endif #if HAVE_SYS_SYSINFO_H +/* Linux, AIX, HP-UX, IRIX, OSF/1, Solaris, Cygwin, Android */ # include <sys/sysinfo.h> #endif -#if HAVE_MACHINE_HAL_SYSINFO_H +#if HAVE_MACHINE_HAL_SYSINFO_H /* OSF/1 */ # include <machine/hal_sysinfo.h> #endif -#if HAVE_SYS_TABLE_H +#if HAVE_SYS_TABLE_H /* OSF/1 */ # include <sys/table.h> #endif @@ -51,13 +54,16 @@ #endif #if HAVE_SYS_SYSCTL_H && !(defined __GLIBC__ && defined __linux__) +/* Linux/musl, macOS, *BSD, IRIX, Minix */ # include <sys/sysctl.h> #endif -#if HAVE_SYS_SYSTEMCFG_H +#if HAVE_SYS_SYSTEMCFG_H /* AIX */ # include <sys/systemcfg.h> #endif +#include "full-read.h" + #ifdef _WIN32 # define WIN32_LEAN_AND_MEAN @@ -203,11 +209,84 @@ physmem_total (void) return 64 * 1024 * 1024; } -/* Return the amount of physical memory available. */ +#if defined __linux__ + +/* Get the amount of free memory and of inactive file cache memory, and + return 0. Upon failure, return -1. */ +static int +get_meminfo (unsigned long long *mem_free_p, + unsigned long long *mem_inactive_file_p) +{ + /* While the sysinfo() system call returns mem_total, mem_free, and a few + other numbers, the only way to get mem_inactive_file is by reading + /proc/meminfo. */ + int fd = open ("/proc/meminfo", O_RDONLY); + if (fd >= 0) + { + char buf[4096]; + size_t buf_size = full_read (fd, buf, sizeof (buf)); + close (fd); + if (buf_size > 0) + { + char *buf_end = buf + buf_size; + unsigned long long mem_free = 0; + unsigned long long mem_inactive_file = 0; + + /* Iterate through the lines. */ + char *line = buf; + for (;;) + { + char *p; + for (p = line; p < buf_end; p++) + if (*p == '\n') + break; + if (p == buf_end) + break; + *p = '\0'; + if (sscanf (line, "MemFree: %llu kB", &mem_free) == 1) + { + mem_free *= 1024; + } + if (sscanf (line, "Inactive(file): %llu kB", &mem_inactive_file) == 1) + { + mem_inactive_file *= 1024; + } + line = p + 1; + } + if (mem_free > 0 && mem_inactive_file > 0) + { + *mem_free_p = mem_free; + *mem_inactive_file_p = mem_inactive_file; + return 0; + } + } + } + return -1; +} + +#endif + +/* Return the amount of physical memory that can be claimed, with a given + aggressivity. */ double -physmem_available (void) +physmem_claimable (double aggressivity) { #if defined _SC_AVPHYS_PAGES && defined _SC_PAGESIZE +# if defined __linux__ + /* On Linux, sysconf (_SC_AVPHYS_PAGES) returns the amount of "free" memory. + The Linux memory management system attempts to keep only a small amount + of memory (something like 5% to 10%) as free, because memory is better + used in the file cache. + We compute the "claimable" memory as + (free memory) + aggressivity * (inactive memory in the file cache). */ + if (aggressivity > 0.0) + { + unsigned long long mem_free; + unsigned long long mem_inactive_file; + if (get_meminfo (&mem_free, &mem_inactive_file) == 0) + return (double) mem_free + aggressivity * (double) mem_inactive_file; + } +# endif { /* This works on linux-gnu, kfreebsd-gnu, solaris2, and cygwin. */ double pages = sysconf (_SC_AVPHYS_PAGES); double pagesize = sysconf (_SC_PAGESIZE); @@ -312,6 +391,12 @@ physmem_available (void) return physmem_total () / 4; } +/* Return the amount of physical memory available. */ +double +physmem_available (void) +{ + return physmem_claimable (0.0); +} #if DEBUG diff --git a/modules/physmem b/modules/physmem index 2cb7e7f64a..e1a4a76700 100644 --- a/modules/physmem +++ b/modules/physmem @@ -8,6 +8,7 @@ m4/physmem.m4 Depends-on: unistd +full-read configure.ac: gl_PHYSMEM