Division by zero may occur in rare constellation of conditions if: 1. not TrueColor mode on the client side tight_detect_smooth_image16() and tight_detect_smooth_image32(), defined by macro DEFINE_DETECT_FUNCTION()2, are affected. 2. if all pixels on the screen are equal, then pixels == stats[0]
Found by Linux Verification Center (linuxtesting.org) with SVACE. Signed-off-by: Dmitry Frolov <fro...@swemel.ru> --- ui/vnc-enc-tight.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c index 41f559eb83..f1249ab136 100644 --- a/ui/vnc-enc-tight.c +++ b/ui/vnc-enc-tight.c @@ -284,6 +284,9 @@ tight_detect_smooth_image24(VncState *vs, int w, int h) for (; c < 256; c++) { \ errors += stats[c] * (c * c); \ } \ + if (pixels == stats[0]) { \ + return 0; \ + } \ errors /= (pixels - stats[0]); \ \ return errors; \ -- 2.34.1