When building with GCC 12: ../tools/../lib/rsa/rsa-verify.c:275:11: warning: ‘*db’ may be used uninitialized [-Wmaybe-uninitialized] 275 | db[0] &= 0xff >> leftmost_bits; | ~~^~~
If msg_len or hash_len were nonsense values, db_len would end up invalid and the rest of the function wil not work, so detect this case and return early. If this was host code we could assert, but as this is target code print an error and return. Fixes: 061daa0b61f0 ("rsa: add support of padding pss") Signed-off-by: Joel Stanley <j...@jms.id.au> --- lib/rsa/rsa-verify.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/rsa/rsa-verify.c b/lib/rsa/rsa-verify.c index 1d95cfbdee0c..a7c87e40afa2 100644 --- a/lib/rsa/rsa-verify.c +++ b/lib/rsa/rsa-verify.c @@ -234,6 +234,11 @@ int padding_pss_verify(struct image_sign_info *info, uint8_t leftmost_mask; struct checksum_algo *checksum = info->checksum; + if (db_len <= 0) { + printf("%s: invalid db length\n", __func__); + return -EINVAL; + } + /* first, allocate everything */ db_mask = malloc(db_len); db = malloc(db_len); -- 2.35.1