Since commit 86f04735ac ("linux-user: Fix brk() to release pages") it's
possible for userspace applications to reduce memory footprint by
calling brk() with a lower address and free up memory. Before that guest
heap memory never was unmapped.But the Linux kernel prohibits to reduce brk() below the initial memory address which is set at startup by the program, while this check was missed in commit 86f04735ac. This patch adds the missing check by storing the initial brk value in initial_target_brk and verify new brk addresses against that value. Tested with the i386 upx binary from https://github.com/upx/upx/releases/download/v4.0.2/upx-4.0.2-i386_linux.tar.xz Signed-off-by: Helge Deller <del...@gmx.de> Tested-by: Markus F.X.J. Oberhumer <notificati...@github.com> Fixes: 86f04735ac ("linux-user: Fix brk() to release pages") Buglink: https://github.com/upx/upx/issues/683 --- linux-user/syscall.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/linux-user/syscall.c b/linux-user/syscall.c index f877156ed3..92d146f8fb 100644 --- a/linux-user/syscall.c +++ b/linux-user/syscall.c @@ -801,12 +801,13 @@ static inline int host_to_target_sock_type(int host_type) return target_type; } -static abi_ulong target_brk; +static abi_ulong target_brk, initial_target_brk; static abi_ulong brk_page; void target_set_brk(abi_ulong new_brk) { target_brk = new_brk; + initial_target_brk = new_brk; brk_page = HOST_PAGE_ALIGN(target_brk); } @@ -824,6 +825,11 @@ abi_long do_brk(abi_ulong brk_val) return target_brk; } + /* do not allow to shrink below initial brk value */ + if (brk_val < initial_target_brk) { + brk_val = initial_target_brk; + } + new_brk = TARGET_PAGE_ALIGN(brk_val); new_host_brk_page = HOST_PAGE_ALIGN(brk_val); -- 2.41.0
