Public bug reported: We are running x11vnc 0.9.16-8 on a FIPS enabled Ubuntu 22.04 with libvncserver1 0.9.13+dfsg-3build2 and libssl3 3.0.2-0ubuntu1.12+Fips1.
With the fips=1 kernel parameter enabling FIPS, it seems that the password hashing is broken and only a clear text password is written: $ cat /proc/sys/crypto/fips_enabled 1 $ x11vnc -storepasswd Abc /tmp/.testpw && cat /tmp/.testpw stored passwd in file: /tmp/.testpw Abc Any connection attempt fails with a 'password check failed!' error. Running x11vnc with sudo /usr/bin/x11vnc -auth guess -forever -localhost -loop -noxdamage -repeat -rfbauth /root/.vncpasswd -rfbport 5900 -shared logs the following: Got connection from client 127.0.0.1 0 other clients Normal socket connection check_access: client 127.0.0.1 matches host 127.0.0.1 incr accepted_client=1 for 127.0.0.1:54968 sock=10 Client Protocol Version 3.8 Protocol version sent 3.8, using 3.8 rfbProcessClientSecurityType: executing handler for type 2 Couldn't read password file: /root/.vncpasswd rfbAuthProcessClientMessage: password check failed rfbClientSendString("password check failed!") client_count: 0 Client 127.0.0.1 gone By turning off FIPS with fips=0 in the kernel, it works as expected: $ cat /proc/sys/crypto/fips_enabled 0 $ x11vnc -storepasswd Abc /tmp/.testpw && cat /tmp/.testpw stored passwd in file: /tmp/.testpw �97l܊ ** Affects: x11vnc (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2058354 Title: Enabling FIPS breaks password hashing To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/x11vnc/+bug/2058354/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs