anatol opened a new pull request, #21646:
URL: https://github.com/apache/nuttx/pull/21646
## Summary
`TD4` holds `uint8_t`, which is promoted to a signed `int`, so `<< 24` of
a byte >= 0x80 shifts into the sign bit. UBSan flags this as
`shift-out-of-bounds`. It reports on every rtl8188f WPA decrypt (201
reports in `TestSanitizedImages/ubsan`).
The shifted value is only used as part of the `uint32_t` result of the XOR
with the round key, so the behavior is harmless in practice. This change adds
`__attribute__((no_sanitize("shift")))` to
`rijndaeldecrypt()` so UBSan no longer checks shifts in that function. It
also adds a comment explaining why.
I used the attribute because it is the smallest change and leaves the
cipher code itself untouched. If maintainers prefer, I can instead cast the
`TD4[...]` lookups to `uint32_t` before shifting.
## Impact
- Users: none. Generated code is unchanged apart from the removed UBSan
instrumentation in this one function.
- Build process: none for non-sanitized builds. With
`-fsanitize=undefined` the `shift` check is skipped for `rijndaeldecrypt()`
only. All other checks stay enabled there.
- Hardware / documentation / security / compatibility: no impact.
## Testing
Host: Arch Linux x86_64, GCC 16.2
Board: K230
Build: with `-fsanitize=undefined` gcc flag
Ran the UBSan-enabled image on the board and exercised the wireless WPA
decrypt path.
Before (without this change), UBSan reports on each decrypt:
`UBSAN: shift-out-of-bounds in crypto/rijndael.c`
After (with this change): no UBSan reports from `crypto/rijndael.c` on the
same run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]