Re: [PATCH 4/6] linux-user: Fix signed math overflow in brk() syscall

On 7/18/23 00:02, Philippe Mathieu-Daudé wrote: On 17/7/23 23:35, Helge Deller wrote: Fix the math overflow when calculating the new_malloc_size. new_host_brk_page and brk_page are unsigned integers. If userspace reduces the heap, new_host_brk_page is lower than brk_page which results in a huge

Re: [PATCH 4/6] linux-user: Fix signed math overflow in brk() syscall

On 17/7/23 23:35, Helge Deller wrote: Fix the math overflow when calculating the new_malloc_size. new_host_brk_page and brk_page are unsigned integers. If userspace reduces the heap, new_host_brk_page is lower than brk_page which results in a huge positive number (but should actually be negative

[PATCH 4/6] linux-user: Fix signed math overflow in brk() syscall

Fix the math overflow when calculating the new_malloc_size. new_host_brk_page and brk_page are unsigned integers. If userspace reduces the heap, new_host_brk_page is lower than brk_page which results in a huge positive number (but should actually be negative). Fix it by adding a proper check and