Hello,
in mpz_realloc, we check if new_alloc > INT_MAX and produce an error in
that case.
In mul.c, we use __GMP_ALLOCATE_FUNC_LIMBS directly and store "ALLOC (w) =
wsize". I think that may explain why we silently get a wrong result in
https://stackoverflow.com/q/79483605/1918193 . We should probably add a
check, or see if we can use one of the macros/functions that already do
the check.
On windows, with a 32-bit long (and thus mp_size_t), it looks like
usize+vsize could instead overflow and make mpn_mul write in unallocated
memory, but that seems to be the case throughout the library, and at least
compiling with -fsanitize=undefined should stop it (whereas it does not
mind a long->int conversion).
--
Marc Glisse
_______________________________________________
gmp-bugs mailing list
[email protected]
https://gmplib.org/mailman/listinfo/gmp-bugs
