Re: Unsigned integer overflow in `toom_eval_pm2.c`

2023-09-04 Thread Vincent Lefevre
On 2023-09-04 09:52:23 +0200, marco.bodr...@tutanota.com wrote: > Should the value ~0 be written as ~0U to be correctly assigned to an > unsigned? IMHO, this would be better as this would make the signedness of the types more consistent (the goal being to have an unsigned int for all these

Re: Unsigned integer overflow in `toom_eval_pm2.c`

2023-09-04 Thread Niels Möller
marco.bodr...@tutanota.com writes: > Would a comment "It returns 0 or ~0, depending on the sign of the result" > added to all the mpn_toom_eval_ functions suffice? Yes. It would be nice if there were some reasonable place to document that only once, but don't know where that could be. > From

Re: Unsigned integer overflow in `toom_eval_pm2.c`

2023-09-04 Thread marco . bodrato
Ciao, 4 set 2023, 08:02 da ni...@lysator.liu.se: > As Vincent suggested, it would be good to document somewhere what the > convention is (values 0 or ~0). > Would a comment "It returns 0 or ~0, depending on the sign of the result" added to all the mpn_toom_eval_ functions suffice? > And maybe

Re: Unsigned integer overflow in `toom_eval_pm2.c`

2023-09-04 Thread Andrew Teylu
On Sun, Sep 3, 2023 at 11:40 PM wrote: > I attach a possible patch. > As an outsider, does it make sense to also change instances of "neg" to be "sign" (if that's the interpretation) for consistency? For someone who isn't literate in the GMP code, neg vs. sign seems to be used slightly

Re: Unsigned integer overflow in `toom_eval_pm2.c`

2023-09-04 Thread Niels Möller
marco.bodr...@tutanota.com writes: > Yes, unsigned is the best choice, it used to store a positive or > negative number, but now it actually is a mask: 0 or ~0. > > I attach a possible patch. Makes sense, I think. As Vincent suggested, it would be good to document somewhere what the convention