Entropy error blocks lang/python38 installation
I've been trying to install lang/python38, critically important for further progress building and installing packages from pkgsrc System is NetBSD-9.99.82 amd64. Build goes through and most of the way through installation, but ends with a green console message [ 386.3532035] entropy: pid 24617 (python) blocking due to lack of entropy This time, I tried again (after Ctrl-C and "make install"), got the same message but with different numbers. This is an awful strange message about failing to install a package. End of build log file was /usr/bin/install -c -o root -g wheel -m 644 ./Lib/curses/ascii.py /usr/pkg/lib/python3.8/curses /usr/bin/install -c -o root -g wheel -m 644 ./Lib/curses/has_key.py /usr/pkg/lib/python3.8/curses /usr/bin/install -c -o root -g wheel -m 644 ./Lib/curses/panel.py /usr/pkg/lib/python3.8/curses /usr/bin/install -c -o root -g wheel -m 644 ./Lib/curses/textpad.py /usr/pkg/lib/python3.8/curses /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/__init__.py /usr/pkg/lib/python3.8/pydoc_data /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/_pydoc.css /usr/pkg/lib/python3.8/pydoc_data /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/topics.py /usr/pkg/lib/python3.8/pydoc_data /usr/bin/install -c -o root -g wheel -m 644 `cat pybuilddir.txt`/_sysconfigdata_netbsd9.py /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8; /usr/bin/install -c -o root -g wheel -m 644 ./LICENSE /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8/LICENSE.txt if test -d /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8/distutils/tests; then /usr/bin/install -c -o root -g wheel -m 644 ./Modules/xxmodule.c /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8/distutils/tests ; fi PYTHONPATH=/pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8 LD_LIBRARY_PATH=/pkgsrc/lang/python38/work10.amd64/Python-3.8.10 ./python -E -Wi /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8/compileall.py -j0 -d /usr/pkg/lib/python3.8 -f -x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8 On other computer, system crashes and reboots (dirty, so I have to fsck_ffs after reboot) at about this point. What does this entropy error message mean, and is there anything I can do except wait some months, update the system, and try again? Inability to build python38 is a bone in the throat, so much depends on Python. I am at an impasse on NetBSD. Tom
Re: Entropy error blocks lang/python38 installation
Hi, This problem was reported/discussed several time I believe and I guess it does not have final solution yet (you can check this thread: http://mail-index.netbsd.org/tech-security/2021/01/11/msg001100.html). However, there are workarounds to build python38 now. One described here https://mail-index.netbsd.org/current-users/2020/05/01/msg038495.html, you can fool system into believing that it has enough entropy: dd if=/dev/urandom of=/dev/random bs=32 count=1 sysctl -w kern.entropy.consolidate=1 Another workaround which also makes python to compile was described in pkg/55847: https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=55847 Regards, Andrius V On Mon, Jun 14, 2021 at 11:15 AM Thomas Mueller wrote: > > I've been trying to install lang/python38, critically important for further > progress building and installing packages from pkgsrc > > System is NetBSD-9.99.82 amd64. > > Build goes through and most of the way through installation, but ends with a > green console message > > [ 386.3532035] entropy: pid 24617 (python) blocking due to lack of entropy > > This time, I tried again (after Ctrl-C and "make install"), got the same > message but with different numbers. > > This is an awful strange message about failing to install a package. > > End of build log file was > > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/curses/ascii.py > /usr/pkg/lib/python3.8/curses > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/curses/has_key.py > /usr/pkg/lib/python3.8/curses > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/curses/panel.py > /usr/pkg/lib/python3.8/curses > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/curses/textpad.py > /usr/pkg/lib/python3.8/curses > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/__init__.py > /usr/pkg/lib/python3.8/pydoc_data > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/_pydoc.css > /usr/pkg/lib/python3.8/pydoc_data > /usr/bin/install -c -o root -g wheel -m 644 ./Lib/pydoc_data/topics.py > /usr/pkg/lib/python3.8/pydoc_data > /usr/bin/install -c -o root -g wheel -m 644 `cat > pybuilddir.txt`/_sysconfigdata_netbsd9.py > /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8; > /usr/bin/install -c -o root -g wheel -m 644 ./LICENSE > /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8/LICENSE.txt > if test -d > /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8/distutils/tests; > then /usr/bin/install -c -o root -g wheel -m 644 ./Modules/xxmodule.c > /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8/distutils/tests > ; fi > PYTHONPATH=/pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8 > LD_LIBRARY_PATH=/pkgsrc/lang/python38/work10.amd64/Python-3.8.10 ./python -E > -Wi > /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8/compileall.py > -j0 -d /usr/pkg/lib/python3.8 -f -x > 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' > /pkgsrc/lang/python38/work10.amd64/.destdir/usr/pkg/lib/python3.8 > > > On other computer, system crashes and reboots (dirty, so I have to fsck_ffs > after reboot) at about this point. > > What does this entropy error message mean, and is there anything I can do > except wait some months, update the system, and try again? > > Inability to build python38 is a bone in the throat, so much depends on > Python. I am at an impasse on NetBSD. > > Tom >
Re: Entropy error blocks lang/python38 installation
On Mon, Jun 14, 2021 at 08:15:27AM +, Thomas Mueller wrote: > [ 386.3532035] entropy: pid 24617 (python) blocking due to lack of entropy This means your setup is incomplete. You need to seed entropy, e.g. by one of the methods in entropy(7) in the "Adding entropy" subsection. Python is probably stupid, but that is a harder battle to fight. Martin
Re: Entropy error blocks lang/python38 installation
> Hi, > This problem was reported/discussed several time I believe and I guess > it does not have final solution yet (you can check this thread: > http://mail-index.netbsd.org/tech-security/2021/01/11/msg001100.html). > However, there are workarounds to build python38 now. > One described here > https://mail-index.netbsd.org/current-users/2020/05/01/msg038495.html, > you can fool system into believing that it has enough entropy: > dd if=/dev/urandom of=/dev/random bs=32 count=1 > sysctl -w kern.entropy.consolidate=1 > Another workaround which also makes python to compile was described in > pkg/55847: https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=55847 > Regards, > Andrius V > On Mon, Jun 14, 2021 at 08:15:27AM +, Thomas Mueller wrote: > > [ 386.3532035] entropy: pid 24617 (python) blocking due to lack of entropy > This means your setup is incomplete. You need to seed entropy, e.g. > by one of the methods in entropy(7) in the "Adding entropy" subsection. > Python is probably stupid, but that is a harder battle to fight. > Martin Thanks for the suggestions! I will surely try when I get the chance. I am now in a different partition/installation that I don't want to mess up. I want to be able to build packages besides and beyond lang/python38. When was this entropy issue introduced (what NetBSD version)? Does any other OS have such an entropy issue? On the other computer, I am now on FreeBSD-current mass-building packages with synth. Another old bug has popped up that I thought was fixed some time ago for NetBSD-current and releng-9: [ 190470.442443] wd1d: error writing fsbn 2316146760 of 2316146760-2316146791 (wd1 bn 2316146760; cn 2297764 tn 10 sn 18) [ 190480.442412] wd1d: device timeout writing fsbn 2316146792 of 2316146792-2316146823 (wd1 bn 2316146792; cn 2297764 tn 10 sn 50), xfer 40, retry 0 [ 190480.442412] wd1d: device timeout writing fsbn 2316146824 of 2316146824-2316146855 (wd1 bn 2316146824; cn 2297764 tn 11 sn 19), xfer e0, retry 0 [ 190480.442412] wd1d: device timeout writing fsbn 2316146856 of 2316146856-2316146887 (wd1 bn 2316146856; cn 2297764 tn 11 sn 51), xfer 180, retry 0 [ 190480.442412] wd1d: device timeout writing fsbn 2316146888 of 2316146888-2316146919 (wd1 bn 2316146888; cn 2297764 tn 12 sn 20), xfer 220, retry 0 [ 190490.942379] wd1d: device timeout writing fsbn 2316146888 of 2316146888-2316146919 (wd1 bn 2316146888; cn 2297764 tn 12 sn 20), xfer 220, retry 1 [ 190490.942379] wd1d: device timeout writing fsbn 2316146856 of 2316146856-2316146887 (wd1 bn 2316146856; cn 2297764 tn 11 sn 51), xfer 180, retry 1 [ 190490.942379] wd1d: device timeout writing fsbn 2316146824 of 2316146824-2316146855 (wd1 bn 2316146824; cn 2297764 tn 11 sn 19), xfer e0, retry 1 [ 190490.942379] wd1d: device timeout writing fsbn 2316146792 of 2316146792-2316146823 (wd1 bn 2316146792; cn 2297764 tn 10 sn 50), xfer 40, retry 1 [ 190501.442347] wd1d: device timeout writing fsbn 2316146792 of 2316146792-2316146823 (wd1 bn 2316146792; cn 2297764 tn 10 sn 50), xfer 40, retry 2 [ 190501.442347] wd1d: device timeout writing fsbn 2316146824 of 2316146824-2316146855 (wd1 bn 2316146824; cn 2297764 tn 11 sn 19), xfer e0, retry 2 [ 190501.442347] wd1d: device timeout writing fsbn 2316146856 of 2316146856-2316146887 (wd1 bn 2316146856; cn 2297764 tn 11 sn 51), xfer 180, retry 2 [ 190501.442347] wd1d: device timeout writing fsbn 2316146888 of 2316146888-2316146919 (wd1 bn 2316146888; cn 2297764 tn 12 sn 20), xfer 220, retry 2 [ 190511.942314] wd1d: device timeout writing fsbn 2316146888 of 2316146888-2316146919 (wd1 bn 2316146888; cn 2297764 tn 12 sn 20), xfer 220, retry 3 [ 190511.942314] wd1d: device timeout writing fsbn 2316146856 of 2316146856-2316146887 (wd1 bn 2316146856; cn 2297764 tn 11 sn 51), xfer 180, retry 3 [ 190511.942314] wd1d: device timeout writing fsbn 2316146824 of 2316146824-2316146855 (wd1 bn 2316146824; cn 2297764 tn 11 sn 19), xfer e0, retry 3 [ 190511.942314] wd1d: device timeout writing fsbn 2316146792 of 2316146792-2316146823 (wd1 bn 2316146792; cn 2297764 tn 10 sn 50), xfer 40, retry 3 [ 190522.442281] wd1d: device timeout writing fsbn 2316146792 of 2316146792-2316146823 (wd1 bn 2316146792; cn 2297764 tn 10 sn 50), xfer 40, retry 4 [ 190532.442250] wd1d: device timeout writing fsbn 2316146824 of 2316146824-2316146855 (wd1 bn 2316146824; cn 2297764 tn 11 sn 19), xfer e0, retry 4 [ 190542.442218] wd1d: device timeout writing fsbn 2316146856 of 2316146856-2316146887 (wd1 bn 2316146856; cn 2297764 tn 11 sn 51), xfer 180, retry 4 I have to go into the debugger (Ctrl-Alt-Esc) if I can and reboot, otherwise Reset button. Tom
Re: Entropy error blocks lang/python38 installation
> Hi, > This problem was reported/discussed several time I believe and I guess > it does not have final solution yet (you can check this thread: > http://mail-index.netbsd.org/tech-security/2021/01/11/msg001100.html). > However, there are workarounds to build python38 now. > One described here > https://mail-index.netbsd.org/current-users/2020/05/01/msg038495.html, > you can fool system into believing that it has enough entropy: > dd if=/dev/urandom of=/dev/random bs=32 count=1 > sysctl -w kern.entropy.consolidate=1 -> These two lines enabled me to build lang/python38 where otherwise it would lead to a system crash and unclean reboot. Now onto other packages. I have a file /extra/entropy.txt on the partition where I keep pkgsrc tree, so I know what to do and don't have to remember perfectly. > Another workaround which also makes python to compile was described in > pkg/55847: https://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=55847 > Regards, > Andrius V > On Mon, Jun 14, 2021 at 08:15:27AM +, Thomas Mueller wrote: > > [ 386.3532035] entropy: pid 24617 (python) blocking due to lack of entropy > This means your setup is incomplete. You need to seed entropy, e.g. > by one of the methods in entropy(7) in the "Adding entropy" subsection. > Python is probably stupid, but that is a harder battle to fight. > Martin Thanks for the suggestions! I will surely try when I get the chance. -> succeeded. I want to be able to build packages besides and beyond lang/python38. This entropy thing seems crazy to me, but then having to sanitize Linux kernel headers to be usable for userspace programs also impressed me as crazy. On the other computer, I was on FreeBSD-current mass-building packages with synth. Partially successful, some sloppy bugs, packages failed to install, and graphical web browsers all were skipped because lang/rust, a dependency, failed. Tom
Re: Entropy error blocks lang/python38 installation
On Tue, Jun 15, 2021 at 09:24:01AM +, Thomas Mueller wrote: > This entropy thing seems crazy to me It is! But it also is: - a one time only thing (supposed to happen when you set the machine up for the first time) - a current only issue - only affecting some machines - a blocker for the netbsd-10 branch Unfortunatley there is no consensus yet how to deal with it better. Martin
Re: Entropy error blocks lang/python38 installation
from Thomas Klausner: > On Mon, Jun 14, 2021 at 10:24:23PM +, Thomas Mueller wrote: > > Another old bug has popped up that I thought was fixed some time ago for > > NetBSD-current and releng-9: > > [ 190470.442443] wd1d: error writing fsbn 2316146760 of > > 2316146760-2316146791 (wd1 bn 2316146760; cn 2297764 tn 10 sn 18) > > [ 190480.442412] wd1d: device timeout writing fsbn 2316146792 of > > 2316146792-2316146823 (wd1 bn 2316146792; cn 2297764 tn 10 sn 50), xfer 40, > > retry 0 > These look like hardware errors to me - the disk starts failing. > Thomas This is the same error/bug that hit me in NetBSD 8.99.51 and earlier 8.99.x, but never with 7.99.1 or FreeBSD. Tom
Re: Entropy error blocks lang/python38 installation
from Martin Musemann: > On Tue, Jun 15, 2021 at 09:24:01AM +, Thomas Mueller wrote: > > This entropy thing seems crazy to me > It is! But it also is: > - a one time only thing (supposed to happen when you set the machine up >for the first time) > - a current only issue > - only affecting some machines > - a blocker for the netbsd-10 branch > Unfortunatley there is no consensus yet how to deal with it better. > Martin I believe I must apply the fix/workaround every time. meaning every booted session where I would build packages from pkgsrc. Had I known the next point, I might have tried to upgrade to releng-9 (NetBSD 9.2_STABLE) instead of HEAD, if the cvs checkout and update didn't get stuck on a broken pipe. But I don't really know if that would have helped regarding pkgsrc problems. NetBSD 9.99.82 has reduced functionality compared to 8.99.51. I can still use some old packages, but much less than with 8.99.51. Last time I looked at the releng wiki page, I noticed that there were issues that prevented the netbsd-10 branch, and no time estimate for the branch. Entropy bug was not mentioned specifically, but clearly it is a blocker. Tom
Re: Entropy error blocks lang/python38 installation
On Wed, Jun 16, 2021 at 03:42:35AM +, Thomas Mueller wrote: > I believe I must apply the fix/workaround every time. The entropy state gets stored on shutdown and reloaded on next boot. Fixing it once is enough. > Last time I looked at the releng wiki page, I noticed that there were > issues that prevented the netbsd-10 branch, and no time estimate for > the branch. > > Entropy bug was not mentioned specifically, but clearly it is a blocker. It is there, in the top list of blockers: - handling of randomness and seed file need to be decided (more below, waiting for randot) Martin
Re: Entropy error blocks lang/python38 installation
On Wed, Jun 16, 2021 at 06:13:23AM +0200, Martin Husemann wrote: > On Wed, Jun 16, 2021 at 03:42:35AM +, Thomas Mueller wrote: > > I believe I must apply the fix/workaround every time. > > The entropy state gets stored on shutdown and reloaded on next boot. > Fixing it once is enough. ...assuming that people actually use shutdown and don't just reboot. Joerg
Re: Entropy error blocks lang/python38 installation
On Wed, Jun 16, 2021 at 11:10:34AM +0200, Joerg Sonnenberger wrote: > On Wed, Jun 16, 2021 at 06:13:23AM +0200, Martin Husemann wrote: > > On Wed, Jun 16, 2021 at 03:42:35AM +, Thomas Mueller wrote: > > > I believe I must apply the fix/workaround every time. > > > > The entropy state gets stored on shutdown and reloaded on next boot. > > Fixing it once is enough. > > ...assuming that people actually use shutdown and don't just reboot. Kinda - but the instructions in the man page are quite explicit and ask you to save the entropy state at least once manually, which should avoid the blocking behaviour in all cases. Martin
Re: Entropy error blocks lang/python38 installation
At Wed, 16 Jun 2021 11:18:02 +0200, Martin Husemann wrote: Subject: Re: Entropy error blocks lang/python38 installation > > On Wed, Jun 16, 2021 at 11:10:34AM +0200, Joerg Sonnenberger wrote: > > On Wed, Jun 16, 2021 at 06:13:23AM +0200, Martin Husemann wrote: > > > On Wed, Jun 16, 2021 at 03:42:35AM +, Thomas Mueller wrote: > > > > I believe I must apply the fix/workaround every time. > > > > > > The entropy state gets stored on shutdown and reloaded on next boot. > > > Fixing it once is enough. > > > > ...assuming that people actually use shutdown and don't just reboot. > > Kinda - but the instructions in the man page are quite explicit and > ask you to save the entropy state at least once manually, which should > avoid the blocking behaviour in all cases. That's not an acceptable regression. Previously no manual operations were ever necessary -- the blocking was never permanent. -- Greg A. Woods Kelowna, BC +1 250 762-7675 RoboHack Planix, Inc. Avoncote Farms pgpekRmVN6MUp.pgp Description: OpenPGP Digital Signature
