On Wed, Jul 31, 2024, at 7:56 AM, Yury V. Zaytsev wrote:
> So it appears to be not as trivial as “mmap on AIX has been broken for
> ages”, but rather it seems that up to some version of AIX (4-5?), it
> used to require unmapping the address first. After that IBM still kept
> it as the default behavior, but added spec-compliant behavior to the
> set of behavioral changes invoked by the XPG_SUS_ENV variable, most
> likely out of backwards compatibility concerns…
Thanks for digging into this, Yury. I have access to AIX machines via
the GCC compile farm but I would not have been able to find this.
> So, the bottom line, I guess, is still that the autoconf test is
> correct in terms of checking for spec compliant behavior. It’s of no
> help to developers who want mmap on AIX, but at least it prevents
> the worst.
>
> Probably breaking the macro down as Zack has suggested would be
> helpful so that MAP_FIXED behavior could be examined separately if
> needed, but still, for my use case, it’s better to just get rid of
> mmap altogether.
Independently, I have written a new test program that probes for a
superset of the things that the existing AC_FUNC_MMAP test probes for.
My goal for this program is to determine which problems are still
present on current-generation operating systems, and how far back in
time we have to go to encounter the problems that no longer show up
today. It's attached to this email, and also posted for convenient
download at
<https://paste.sr.ht/~zackw/69f6fe801335583216de012b2d2f86c7d43afd76>.
I ran this program on all the operating systems conveniently
available to me, all of which are reasonably up to date. A fully
correct implementation of mmap (as far as the test goes) will print
something like
note: using 4096 bytes as page size.
note: define MMAP_PROBE_PAGESIZE if this is incorrect.
test: zero-fill after EOF (shared mapping)... ok
test: zero-fill after EOF (private mapping)... ok
test: writes visible via shared map... ok
test: shared map alterations visible via read... ok
test: shared map alterations visible via second map... ok
test: private read-write map of file opened read-only... ok
test: alter private map of file opened read-only... ok
test: alter private map of file opened read-write... ok
test: replace existing mapping... ok
I get this result on:
x86_64-pc-linux-gnu
x86_64-unknown-freebsd13.3
aarch64c-unknown-freebsd14.0
x86_64-unknown-netbsd10.0
sparc-sun-solaris2.10
sparc-sun-solaris2.11
powerpc-ibm-aix7.1.5.0 (with XPG_SUS_ENV=ON)
powerpc-ibm-aix7.3.1.0 (with XPG_SUS_ENV=ON)
I get the previously discussed MAP_FIXED failure on both powerpc-ibm-aix7
machines if I don't set XPG_SUS_ENV=ON.
Finally, I got two failures I wasn't expecting to see at all:
test: writes visible via shared map... FAIL: data mismatch
test: shared map alterations visible via read... FAIL: data mismatch
amd64-unknown-openbsd7.5
mips64-unknown-openbsd7.5
AC_FUNC_MMAP currently doesn't test MAP_SHARED at all, but maybe it
*should*...
For right now, I'd like to ask everyone who reads this message to run
the test program on operating systems not listed above, and on older
versions of operating systems that *are* listed above. The most
interesting reports will be of current OSes that fail one or more
tests, and of version thresholds where the set of failing tests
*changes* for some OS. No-longer-developed OSes whose final version
fails some tests are also interesting. Testing contemporary OSes on
CPUs I didn't try is probably *not* going to turn up anything
interesting since the VM layer should be mostly machine-independent
these days, but you never know.
Please also pass the program and the request along to anyone you know
who might have weird old Unixes to try.
zw
/* Test for correct functionality of several corner cases of mmap().
Copyright 2000-2017, 2020-2024 Free Software Foundation, Inc.
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version. See <https://www.gnu.org/licenses>
for precise terms and conditions.
Originally part of Autoconf, original design of this test by Mike
Haertel and Jim Avera, expanded substantially by Zack Weinberg. */
/* This program tests for a variety of bugs which were known
to exist in real implementations of mmap, mostly in the 1990s:
- If a file is mapped with MAP_SHARED, modifications to the mapped
memory area should be visible to read(), and modifications to
the file via write() should be visible in the mapped area.
Current versions of OpenBSD fail this test.
Old versions of FreeBSD and NetBSD may also fail this test,
according to notes in the guts of autoconf's code.
- If a file is mapped twice, both times with MAP_SHARED,
modifications to one mapped area should be visible in the other.
Unknown whether any historical systems fail this test.
- When a file is mapped with MAP_PRIVATE, it should be possible
to make that mapping read-write even if the file was opened
read-only. Unknown whether any historical systems fail this test.
- If a file is mapped read-write with MAP_PRIVATE, modifications
to the mapped memory area should *not* be visible to read(),
whether or not the file was opened read-write. Some ports of
System V Release 4 to x86-32 are believed to have failed this
test; unknown whether there are others.
Note: the converse--whether modifications via write() after
creation of a MAP_PRIVATE mapping are visible in the mapped
memory area--is unspecified and therefore not tested.
- If a file whose length is not a multiple of the page size is
mapped, bytes of the mapped memory area beyond the end of the
file, up to the next page boundary, should all read as zero.
Believed very likely that some historical systems would
fail this test, but which ones are not known.
- If memory region [a, b) is mapped, mmap(a, b-a, ..., MAP_FIXED|..., ...)
should succeed and replace the old mapping with a new one.
We only test MAP_FIXED|MAP_PRIVATE and we do not test partial
replacement of an existing mapping.
Known to fail on *current* versions of AIX unless an environment
variable is set to request standard-compliant behavior from the
C library; unknown what version introduced that environment
variable. Autoconf's documentation says this also fails on
HP-UX 11, but this has not been verified recently.
Only file mappings are tested, not anonymous mappings, because
POSIX and XSI do not specify MAP_ANON(YMOUS) or /dev/zero.
This program should compile using any ISO C 1990 compliant hosted
implementation which also provides <sys/mman.h>, <sys/stat.h>,
<sys/types.h>, <fcntl.h>, <unistd.h>, open, close, umask, write,
read, mmap, and munmap.
It attempts to use sysconf(_SC_PAGESIZE) to query the system page
size; if this does not work, or if <unistd.h> does not define
_SC_PAGESIZE, it falls back to guessing the page size is 4096
bytes. If sysconf doesn't work and the guess is wrong, define
MMAP_PROBE_PAGESIZE on the command line to the correct value.
(It does not attempt to use getpagesize(), nor various #defines
from <sys/param.h>, because there is no practical way to detect the
existence of getpagesize or sys/param.h from within the program.)
You may also find it necessary to define feature selection
macros; in particular, if you see "SKIP: MAP_FIXED not defined"
in the output, try recompiling with -D_XOPEN_SOURCE=700.
In order to avoid using C1999 extensions to printf, this program
assumes that both size_t values and pointers to char can be cast to
unsigned long without loss of information. Failure of this
assumption will only affect diagnostic output. Casts in the
opposite direction are not used.
The current official specification of mmap is here:
https://pubs.opengroup.org/onlinepubs/9699919799/functions/mmap.html
You may find it helpful to (re)read this specification in order to
understand this program fully. */
#include <errno.h>
#include <stdarg.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>
static int status = 0;
static void
test_begin (const char *what)
{
printf ("test: %s... ", what);
fflush (stdout);
}
static void
test_skip (const char *what, const char *why)
{
printf ("test: %s... SKIP: %s\n", what, why);
status = 1;
}
static void
test_ok (void)
{
puts ("ok");
}
static void
test_fail (const char *msg, ...)
{
va_list ap;
va_start (ap, msg);
fputs ("FAIL: ", stdout);
vprintf (msg, ap);
putchar ('\n');
va_end (ap);
status = 1;
}
static void
test_fail_oserr (const char *tag)
{
printf ("FAIL: %s: %s\n", tag, strerror (errno));
status = 1;
}
static void
perror_exit (const char *msg)
{
printf ("fatal error: %s: %s\n", msg, strerror (errno));
exit (1);
}
static int
read_full_or_fail (int fd, char *buf, size_t nbytes)
{
do
{
ssize_t nread = read (fd, buf, nbytes);
if (nread == -1)
{
test_fail_oserr ("read");
return 1;
}
if (nread == 0)
{
test_fail ("read: early end of file, expected %lu bytes more",
(unsigned long) nbytes);
return 1;
}
buf += (size_t) nread;
nbytes -= (size_t) nread;
}
while (nbytes > 0);
return 0;
}
static int
write_full_or_fail (int fd, const char *buf, size_t nbytes)
{
do
{
ssize_t nwrote = write (fd, buf, nbytes);
if (nwrote == -1)
{
test_fail_oserr ("write");
return 1;
}
if (nwrote == 0)
{
test_fail ("write: zero-byte write with %lu bytes pending",
(unsigned long) nbytes);
return 1;
}
buf += (size_t) nwrote;
nbytes -= (size_t) nwrote;
}
while (nbytes > 0);
return 0;
}
static void
write_full_or_exit (int fd, const char *buf, size_t nbytes, const char *tag)
{
do
{
ssize_t nwrote = write (fd, buf, nbytes);
if (nwrote == -1)
perror_exit (tag);
if (nwrote == 0)
{
printf ("fatal error: %s: zero-byte write with %lu bytes pending",
tag, (unsigned long) nbytes);
exit (1);
}
buf += (size_t) nwrote;
nbytes -= (size_t) nwrote;
}
while (nbytes > 0);
}
static size_t
mmap_page_size (void)
{
#ifdef MMAP_PROBE_PAGESIZE
return MMAP_PROBE_PAGESIZE;
#else
#ifdef _SC_PAGESIZE
long ps = sysconf (_SC_PAGESIZE);
if (ps > 0)
return (size_t) ps;
#endif
return 4096;
#endif
}
#define TEST_FILE_1 "conftst1.dat"
#define TEST_FILE_2 "conftst2.dat"
static void
cleanup_test_file_1 (void)
{
remove (TEST_FILE_1);
}
static void
cleanup_test_file_2 (void)
{
remove (TEST_FILE_2);
}
static void
create_test_files (size_t pagesize, char **pdata1, char **pdata2)
{
char *data1, *data2;
int fd;
size_t i;
/* Ensure files are created with exactly the permissions we want. */
umask (0);
/* File 1 is exactly one byte long and that byte is a binary zero. */
data1 = malloc (1);
if (!data1)
perror_exit ("malloc (data 1)");
data1[0] = 0;
fd = open (TEST_FILE_1, O_RDWR | O_CREAT | O_EXCL, 0600);
if (fd < 0)
perror_exit ("creating " TEST_FILE_1);
atexit (cleanup_test_file_1);
write_full_or_exit (fd, data1, 1, "filling " TEST_FILE_1);
close (fd);
/* File 2 is exactly one page long and filled with ascending byte values. */
data2 = malloc (pagesize);
if (!data2)
perror_exit ("malloc (data 2)");
for (i = 0; i < pagesize; i++)
data2[i] = (char) (unsigned char) i;
fd = open (TEST_FILE_2, O_RDWR | O_CREAT | O_EXCL, 0600);
if (fd < 0)
perror_exit ("creating " TEST_FILE_2);
atexit (cleanup_test_file_2);
write_full_or_exit (fd, data2, pagesize, "filling " TEST_FILE_2);
close (fd);
*pdata1 = data1;
*pdata2 = data2;
}
static void
test_tail_fill (size_t pagesize, const char *fname)
{
char *map;
int fd;
size_t i;
test_begin ("zero-fill after EOF (shared mapping)");
fd = open (fname, O_RDONLY);
if (fd == -1)
{
test_fail_oserr (fname);
test_skip ("zero-fill after EOF (private mapping)", "previous failure");
return;
}
map = mmap (0, pagesize, PROT_READ, MAP_SHARED, fd, 0L);
if (map != MAP_FAILED)
{
for (i = 0; i < pagesize; i++)
if (map[i] != 0)
{
test_fail ("byte %lu has value 0x%02x",
(unsigned long) i, (unsigned char) map[i]);
goto cleanup_1;
}
test_ok ();
cleanup_1:
munmap (map, pagesize);
}
else
test_fail_oserr ("mmap");
test_begin ("zero-fill after EOF (private mapping)");
map = mmap (0, pagesize, PROT_READ, MAP_PRIVATE, fd, 0L);
if (map != MAP_FAILED)
{
for (i = 0; i < pagesize; i++)
if (map[i] != 0)
{
test_fail ("byte %lu has value 0x%02x",
(unsigned long) i, (unsigned char) map[i]);
goto cleanup_2;
}
test_ok ();
cleanup_2:
munmap (map, pagesize);
}
else
test_fail_oserr ("mmap");
close (fd);
}
static void
test_shared_mapping_write (size_t pagesize, const char *fname)
{
static const char new_contents[] =
"this string should appear in the mapping";
char *map;
int fd;
size_t i;
test_begin ("writes visible via shared map");
fd = open (fname, O_RDWR);
if (fd == -1)
{
test_fail_oserr (fname);
return;
}
map = mmap (0, pagesize, PROT_READ, MAP_SHARED, fd, 0L);
if (map == MAP_FAILED)
{
test_fail_oserr ("mmap");
close (fd);
return;
}
if (write_full_or_fail (fd, new_contents, sizeof new_contents))
goto cleanup;
for (i = 0; i < sizeof new_contents; i++)
if (map[i] != new_contents[i])
{
test_fail ("data mismatch at byte %lu: 0x%02x != 0x%02x",
(unsigned long) i, (unsigned char) map[i],
(unsigned char) new_contents[i]);
goto cleanup;
}
test_ok ();
cleanup:
munmap (map, pagesize);
close (fd);
}
static void
test_shared_mapping_read (size_t pagesize, const char *fname)
{
static const char contents[] =
"this string should not be what we read back after modification";
char *map, *readback;
int fd;
size_t i;
test_begin ("shared map alterations visible via read");
fd = open (fname, O_RDWR);
if (fd == -1)
{
test_fail_oserr (fname);
return;
}
if (write_full_or_fail (fd, contents, sizeof contents))
{
close (fd);
return;
}
map = mmap (0, pagesize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0L);
if (map == MAP_FAILED)
{
test_fail_oserr ("mmap");
close (fd);
return;
}
for (i = 0; i < sizeof contents; i++)
map[i] += 1;
readback = malloc (sizeof contents);
if (!readback)
perror_exit ("malloc");
lseek (fd, 0, SEEK_SET);
if (read_full_or_fail (fd, readback, sizeof contents))
goto cleanup;
for (i = 0; i < sizeof contents; i++)
if (readback[i] != map[i])
{
test_fail ("data mismatch at byte %lu: 0x%02x != 0x%02x",
(unsigned long) i, (unsigned char) readback[i],
(unsigned char) map[i]);
goto cleanup;
}
test_ok ();
cleanup:
free (readback);
munmap (map, pagesize);
close (fd);
}
static void
test_shared_mapping_two_maps (size_t pagesize, const char *fname)
{
static const char contents[] =
"This String Should Not Be What We Read Back After Modification";
char *map1, *map2;
int fd1, fd2;
size_t i;
test_begin ("shared map alterations visible via second map");
fd1 = open (fname, O_RDWR);
if (fd1 == -1)
{
test_fail_oserr (fname);
return;
}
if (write_full_or_fail (fd1, contents, sizeof contents))
{
close (fd1);
return;
}
map1 = mmap (0, pagesize, PROT_READ | PROT_WRITE, MAP_SHARED, fd1, 0L);
if (map1 == MAP_FAILED)
{
test_fail_oserr ("mmap 1");
close (fd1);
return;
}
/* Open the file again to catch out caches that aren't coherent
across open file descriptions. */
fd2 = open (fname, O_RDWR);
if (fd2 == -1)
{
test_fail_oserr (fname);
munmap (map1, pagesize);
close (fd1);
return;
}
map2 = mmap (0, pagesize, PROT_READ, MAP_SHARED, fd2, 0L);
if (map2 == MAP_FAILED)
{
test_fail_oserr ("mmap 2");
munmap (map1, pagesize);
close (fd1);
close (fd2);
return;
}
for (i = 0; i < sizeof contents; i++)
map1[i] += 1;
for (i = 0; i < pagesize; i++)
if (map2[i] != map1[i])
{
test_fail ("data mismatch at byte %lu: 0x%02x != 0x%02x",
(unsigned long) i, (unsigned char) map2[i],
(unsigned char) map1[i]);
goto cleanup;
}
test_ok ();
cleanup:
munmap (map1, pagesize);
munmap (map2, pagesize);
close (fd1);
close (fd2);
}
static void
test_alter_private_mapping_rw (size_t pagesize, char *data, const char *fname)
{
char *map, *readback;
size_t i;
int fd;
test_begin ("alter private map of file opened read-write");
fd = open (fname, O_RDWR);
if (fd == -1)
{
test_fail_oserr (fname);
return;
}
map = mmap (0, pagesize, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0L);
if (map == MAP_FAILED)
{
test_fail_oserr ("mmap");
close (fd);
return;
}
for (i = 0; i < pagesize; i++)
map[i] += 1;
munmap (map, pagesize);
readback = malloc (pagesize);
if (!readback)
perror_exit ("malloc");
if (read_full_or_fail (fd, readback, pagesize))
goto cleanup;
for (i = 0; i < pagesize; i++)
if (readback[i] != data[i])
{
test_fail ("data mismatch at byte %lu: 0x%02x != 0x%02x",
(unsigned long) i, (unsigned char) readback[i],
(unsigned char) data[i]);
/* Correct the expected contents of the file for subsequent tests. */
memcpy (data, readback, pagesize);
goto cleanup;
}
test_ok ();
cleanup:
free (readback);
close (fd);
}
static void
test_alter_private_mapping_ro (size_t pagesize, char *data, const char *fname)
{
int fd;
char *map, *readback;
size_t i;
test_begin ("private read-write map of file opened read-only");
fd = open (fname, O_RDONLY);
if (fd == -1)
{
test_fail_oserr (fname);
test_skip ("alter private map of file opened read-only",
"previous failure");
return;
}
map = mmap (0, pagesize, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0L);
if (map == MAP_FAILED)
{
test_fail_oserr ("mmap");
test_skip ("alter private map of file opened read-only",
"previous failure");
close (fd);
return;
}
test_ok ();
test_begin ("alter private map of file opened read-only");
for (i = 0; i < pagesize; i++)
map[i] += 1;
munmap (map, pagesize);
readback = malloc (pagesize);
if (!readback)
perror_exit ("malloc");
if (read_full_or_fail (fd, readback, pagesize))
goto cleanup;
for (i = 0; i < pagesize; i++)
if (readback[i] != data[i])
{
test_fail ("data mismatch at byte %lu: 0x%02x != 0x%02x",
(unsigned long) i, (unsigned char) readback[i],
(unsigned char) data[i]);
/* Correct the expected contents of the file for subsequent tests. */
memcpy (data, readback, pagesize);
goto cleanup;
}
test_ok ();
cleanup:
free (readback);
close (fd);
}
static void
test_replace_mapping (size_t pagesize, const char *data, const char *fn1,
const char *fn2)
{
#ifdef MAP_FIXED
char *map1, *map2;
int fd;
size_t i;
test_begin ("replace existing mapping");
fd = open (fn2, O_RDONLY);
if (fd == -1)
{
test_fail_oserr (fn2);
return;
}
map1 = mmap (0, pagesize, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0L);
if (map1 == MAP_FAILED)
{
test_fail_oserr ("initial mmap");
return;
}
close (fd);
fd = open (fn1, O_RDONLY);
if (fd == -1)
{
test_fail_oserr (fn1);
munmap (map1, pagesize);
return;
}
map2 = mmap (map1, pagesize, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_FIXED,
fd, 0L);
if (map2 == MAP_FAILED)
{
test_fail_oserr ("mmap(MAP_FIXED)");
munmap (map1, pagesize);
return;
}
close (fd);
if (map2 != map1)
{
test_fail ("new mapping at address 0x%08lx != 0x%08lx",
(unsigned long) map2, (unsigned long) map1);
munmap (map1, pagesize);
munmap (map2, pagesize);
return;
}
for (i = 0; i < pagesize; i++)
if (map2[i] != data[i])
{
test_fail ("data mismatch at byte %lu: 0x%02x != 0x%02x",
(unsigned long) i, (unsigned char) map2[i],
(unsigned char) data[i]);
munmap (map2, pagesize);
return;
}
test_ok ();
munmap (map2, pagesize);
#else
(void) pagesize;
(void) data;
(void) fn1;
(void) fn2;
test_skip ("replace existing mapping", "MAP_FIXED not defined");
#endif
}
int
main (void)
{
char *data1, *data2;
size_t pagesize;
setvbuf (stdout, 0, _IOLBF, 0);
pagesize = mmap_page_size ();
printf ("note: using %lu bytes as page size.\n"
"note: define MMAP_PROBE_PAGESIZE if this is incorrect.\n",
(unsigned long) pagesize);
create_test_files (pagesize, &data1, &data2);
test_tail_fill (pagesize, TEST_FILE_1);
test_shared_mapping_write (pagesize, TEST_FILE_1);
test_shared_mapping_read (pagesize, TEST_FILE_1);
test_shared_mapping_two_maps (pagesize, TEST_FILE_1);
test_alter_private_mapping_ro (pagesize, data2, TEST_FILE_2);
test_alter_private_mapping_rw (pagesize, data2, TEST_FILE_2);
test_replace_mapping (pagesize, data2, TEST_FILE_2, TEST_FILE_1);
free (data1);
free (data2);
return status;
}
