I think this could be an issue with the system's /dev/urandom or entropy, as I've observed similar infinite loops in BN_prime when I changed OpenSSL code to always return the same sequence of bytes from its PRNG (for testing purposes). It could also be a genuine bug in OpenSSL, or both. I'll let others comment on that.
On Thu, Jun 18, 2020 at 9:47 AM Ronny Meeus <ronny.me...@gmail.com> wrote: > Hello > > we are in the process of upgrading our openssl to version 1.1.1g. > On one of our architectures (Cavium MIPS, running kernel 4.9) we have > an issue in the ssh-keygen tool. It keeps on consuming 100% CPU of 1 > core. > On other architectures we do not see the issue at all. > > I instrumented the openssl library with some traces and observed that > it keeps on looping in the "probable prime" function. > At the end of the function the "BN_num_bits" check is done and if the > return value is not equal to "bits" it basically starts all over > again. > > } > if (!BN_add_word(rnd, delta)) > return 0; > if (BN_num_bits(rnd) != bits) { > printf("%s BN_num_bits %d %d\n", __FUNCTION__, BN_num_bits(rnd), > bits); > goto again; > } > bn_check_top(rnd); > return 1; > } > > I added the print function and the result of the print is as follows: > probable_prime BN_num_bits 1473 1536 > This trace keeps on going forever and the values never change. > > Any idea what could be the underlying root-cause? > > Many thanks and best regards, > Ronny >