Op do 18 jun. 2020 om 18:27 schreef Ronny Meeus <ronny.me...@gmail.com>: > > Op do 18 jun. 2020 om 18:13 schreef Salz, Rich <rs...@akamai.com>: > > > > > BN_bin2bn assumes that the size of a BN_ULONG (the type of a bn->d) is > > BN_BYTES. You have already told us that sizeof(*d) is 4. So BN_BYTES > > should also be 4. If BN_BYTES is being incorrectly set to 8 on your > > platform then that would explain the discrepancy. Can you check? > > > > This seems HIGHLY likely since Ronny said earlier that the same > > config/toolchain is used for 32bit userspace and 64bit kernel, right? > > > > I used the following print statement so the sizeof is actual of the *d > and not of the pointer :-). > printf("BN_num data (size=%d top=%d neg=%d flags=%d, sizeof(*d)=%d) > BN_BYTES=%d: ", rnd->dmax, rnd->top, rnd->neg, rnd->flags, > sizeof(*rnd->d), BN_BYTES); > > The output clearly shows that BN_BYTES is 8: > > BN_num data (size=24 top=24 neg=0 flags=13, sizeof(*d)=4) BN_BYTES=8: > efe838eb2cf627a7cf1944d5969e17602474f949d4e04f33263e49cdc9917b5f4f71f4f27eb06b2f41930dbac791ded7fae69fa604ec65686b412ef048e91cfd8c976e74ff237112406371b6cb2770588328f8db400718366665b6bca733a19c > > I think we are getting close to the root-cause here ... >
I adapted the code so that the BN_BYTES is 4 in all cases (just a hack). Now the issue is gone and the loop is not observed anymore but the tool still fails to generate the keys: LD_LIBRARY_PATH=/tmp ssh-keygen -A ssh-keygen: generating new host keys: RSA probable_prime 0 loop=0 again=0 bnrand bits=1536 top=1 bottom=1 bytes=192 data: db d2 2c 9e d8 99 08 af d0 f9 da 7c 19 2a 0d 05 94 e2 ba fd f5 60 38 74 81 aa 1a 1c cf 02 70 0d a6 97 51 c6 4a e4 13 98 39 99 7b c4 30 21 97 a0 32 89 3e 2c ff a2 fb 3f b3 bd f9 6b 59 00 f5 48 41 b7 57 cc a5 37 f9 d8 be ba f9 3d f0 6e 86 58 27 fb ae ca 9c 7b 2e 3f 4c 33 16 74 23 b0 c7 d7 74 19 8a ad 3e 5d 76 73 a5 f5 4a 95 d5 b0 da 5f a0 34 c8 ee 57 fb 88 22 be 5a f5 92 6e 47 08 68 e5 21 6c 57 b2 1b 1d 50 c9 89 d8 a3 76 24 b3 2d f3 b6 26 bc 35 33 b7 d0 73 09 8c 1b 2d 90 05 5a 75 24 05 1a de 04 31 ef 5b b1 55 62 33 86 8a 72 3d bc 5b c9 a1 55 e4 6a 4f cd 2e 28 5f 1a 09 25 probable_prime BN_num_bits=1536 bits=1536 BN_num data (size=48 top=48 neg=0 flags=13, sizeof(*d)=4) BN_BYTES=4: 5f1a09734fcd2e28a155e46a3dbc5bc933868a725bb15562de0431ef7524051a2d90055a73098c1b3533b7d0f3b626bc7624b32dc989d8a3b21b1d50e5216c576e470868be5af59257fb8822a034c8eed5b0da5fa5f54a953e5d767374198aad23b0c7d74c3316749c7b2e3f27fbaecaf06e8658bebaf93da537f9d841b757cc5900f548b3bdf96bffa2fb3f32893e2c302197a039997bc44ae41398a69751c6cf02700d81aa1a1cf560387494e2bafd192a0d05d0f9da7cd89908afdbd22c9e sshkey_generate failed: error in libcrypto ssh-keygen: generating new host keys: DSA sshkey_generate failed: error in libcrypto ssh-keygen: generating new host keys: ECDSA sshkey_generate failed: memory allocation failed ssh-keygen: generating new host keys: ED25519 I also have the impression that it takes a very long time for the tool to complete. On other CPUs (ARM) the tool finishes much quicker (and successful). > Ronny