Re: [PATCH v6 4/8] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

2020-07-07 Thread Heinrich Schuchardt
On 18.06.20 16:23, Heiko Stuebner wrote: > From: Heiko Stuebner > > When calculating rrtmp/rr rsa_gen_key_prop() tries to make > (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and > (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] > with rlen being num_bits * 2 > > On a 4096bit key

Re: [PATCH v6 4/8] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

2020-06-25 Thread Simon Glass
On Thu, 18 Jun 2020 at 08:23, Heiko Stuebner wrote: > > From: Heiko Stuebner > > When calculating rrtmp/rr rsa_gen_key_prop() tries to make > (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and > (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] > with rlen being num_bits * 2 > > On

[PATCH v6 4/8] lib: rsa: fix allocated size for rr and rrtmp in rsa_gen_key_prop()

2020-06-18 Thread Heiko Stuebner
From: Heiko Stuebner When calculating rrtmp/rr rsa_gen_key_prop() tries to make (((rlen + 31) >> 5) + 1) steps in the rr uint32_t array and (((rlen + 7) >> 3) + 1) / 4 steps in uint32_t rrtmp[] with rlen being num_bits * 2 On a 4096bit key this comes down to to 257 uint32_t elements in rr and