Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
On Sat, Apr 27, 2019 at 03:16:54PM +0200, Stefano Zacchiroli wrote: > On Sun, Apr 21, 2019 at 05:52:36PM +0200, Santiago Vila wrote: > > In other words: My "merge request" is below. > > Thanks Santiago, patch applied --- both in the master branch in git and > in a dedicated minimal upload targeting sid. I'll request an unblock as > soon as the upload hits the archive. Thanks a lot! There is a similar bug (FTBFS randomly because of failing tests). in beets, also maintained by "Python Applications Packaging Team". Could you take a look at it? https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=855092 (The usual uploader for this one is another Stefano, but the package is maintained by the same team). Thanks.
Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
On Sun, Apr 21, 2019 at 05:52:36PM +0200, Santiago Vila wrote: > In other words: My "merge request" is below. Thanks Santiago, patch applied --- both in the master branch in git and in a dedicated minimal upload targeting sid. I'll request an unblock as soon as the upload hits the archive. Cheers -- Stefano Zacchiroli . z...@upsilon.cc . upsilon.cc/zack . . o . . . o . o Computer Science Professor . CTO Software Heritage . . . . . o . . . o o Former Debian Project Leader & OSI Board Director . . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: PGP signature
Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
tags 923606 + patch
thanks
On Tue, 9 Apr 2019, Stefano Zacchiroli wrote:
> On Tue, Apr 09, 2019 at 10:38:10AM +0200, Santiago Vila wrote:
> > I obviously have interest to fix this before the release, because I
> > can build 99.99% of packages in my setup and this is an inconvenient
> > exception for me. I don't like NMUs. Should I join the Python
> > Applications Packaging Team? Please tell me how I would proceed.
>
> either way is fine, and also just submitting a merge request would do
> (although please let this bug log know if you do so, to make sure it
> doesn't get overlooked).
Well, could we do it the old-fashioned way for this time?
In other words: My "merge request" is below.
Thanks.
diff -Nru beancount-2.2.0/debian/changelog beancount-2.2.0/debian/changelog
--- beancount-2.2.0/debian/changelog2019-01-14 09:01:37.0 +
+++ beancount-2.2.0/debian/changelog2019-04-21 15:00:36.0 +
@@ -1,3 +1,11 @@
+beancount (2.2.0-3) unstable; urgency=medium
+
+ [ Santiago Vila ]
+ * patches/0003: Ignore FileNotFoundError from self.tmpdir.cleanup().
+Fixes a FTBFS problem which happens randomly (Closes: #923606)
+
+ -- Stefano Zacchiroli Sun, 21 Apr 2019 17:00:36 +0200
+
beancount (2.2.0-2) unstable; urgency=medium
* rules: do not ship *.picklecache files with Python examples (fixes
diff -Nru
beancount-2.2.0/debian/patches/0003-Ignore-FileNotFoundError-from-self.tmpdir.cleanup.patch
beancount-2.2.0/debian/patches/0003-Ignore-FileNotFoundError-from-self.tmpdir.cleanup.patch
---
beancount-2.2.0/debian/patches/0003-Ignore-FileNotFoundError-from-self.tmpdir.cleanup.patch
1970-01-01 00:00:00.0 +
+++
beancount-2.2.0/debian/patches/0003-Ignore-FileNotFoundError-from-self.tmpdir.cleanup.patch
2019-04-21 15:00:36.0 +
@@ -0,0 +1,18 @@
+From: Santiago Vila
+Date: Sun, 21 Apr 2019 16:48:40 +0200
+Subject: Ignore FileNotFoundError from self.tmpdir.cleanup()
+
+--- a/beancount/utils/encryption_test.py
b/beancount/utils/encryption_test.py
+@@ -98,7 +98,10 @@
+ self.run_gpg('--import', stdin=TEST_SECRET_KEY.encode('ascii'))
+
+ def tearDown(self):
+-self.tmpdir.cleanup()
++try:
++self.tmpdir.cleanup()
++except FileNotFoundError:
++pass
+
+ def run_gpg(self, *args, **kw):
+ command = ('gpg',
diff -Nru beancount-2.2.0/debian/patches/series
beancount-2.2.0/debian/patches/series
--- beancount-2.2.0/debian/patches/series 2019-01-14 09:01:37.0
+
+++ beancount-2.2.0/debian/patches/series 2019-04-21 15:00:36.0
+
@@ -1,2 +1,3 @@
0001-Remove-fonts.googleapis.com-links-for-the-bean-web-t.patch
0002-make-test_version-more-lax-to-accept-non-git-hg-vers.patch
+0003-Ignore-FileNotFoundError-from-self.tmpdir.cleanup.patch
Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
Hi Santiago, On Tue, Apr 09, 2019 at 10:38:10AM +0200, Santiago Vila wrote: > I obviously have interest to fix this before the release, because I > can build 99.99% of packages in my setup and this is an inconvenient > exception for me. I don't like NMUs. Should I join the Python > Applications Packaging Team? Please tell me how I would proceed. either way is fine, and also just submitting a merge request would do (although please let this bug log know if you do so, to make sure it doesn't get overlooked). Thanks! -- Stefano Zacchiroli . z...@upsilon.cc . upsilon.cc/zack . . o . . . o . o Computer Science Professor . CTO Software Heritage . . . . . o . . . o o Former Debian Project Leader & OSI Board Director . . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club »
Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
On Tue, Mar 05, 2019 at 11:10:10AM +0100, Stefano Zacchiroli wrote: > Thanks a lot for this. I don't think it's needed to determine the actual > fix, which is fairly obvious. My main issue is that I also want to > update to latest upstream release for the next package upload, which is > a minor version update, but in which the code layout changed a bit, > requiring other changes in the packaging. I will not have time to do so > in time for the freeze, so I'm just trying to postpone this work > post-freeze, and I'll address this then. > > If you, or others, have time and interest to fix this before, by all > means go ahead. The package is team-maintained, and I'm very happy to > receive useful NMUs to "my" packages anyway. I obviously have interest to fix this before the release, because I can build 99.99% of packages in my setup and this is an inconvenient exception for me. I don't like NMUs. Should I join the Python Applications Packaging Team? Please tell me how I would proceed. Thanks.
Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
Hi Santiago, On Tue, Mar 05, 2019 at 10:52:47AM +0100, Santiago Vila wrote: > Also, I hope you will surely agree that telling me that you can build > the package in your computer does not help me at all to build the > package in my computer. In this case I'm just doing QA, but if I > wanted to modify the software, the end result is that I could not. sure, I agree with you on this, and I'm also very grateful for the QA work you're doing. > So, while we are not violating the DFSG in strict sense, as we are > still distributing the source, the user still can't exercise their > right to modify the software in practical terms. Here is where I disagree with you, on the perceived severity of this issue. I still haven't seen any other build setup than yours where this build failure happens. So, while I totally trust you and concede that my view of the matter might be biased; I think yours might be too, in the opposite sense. Hence, as an independent test, I'm relying on the official Debian build infrastructure (and associated services, like reproducible build), as a proxy of what users are most likely to encounter when trying to rebuild the package. That's all I'm saying. If you disagree with this, which is totally fine, the severity can be raised and I'll happily punt to the release team to decide what to do with beancount. > (I'm not changing the severity, and I'm still offering a test machine > for you to reproduce the issue if you need it). Thanks a lot for this. I don't think it's needed to determine the actual fix, which is fairly obvious. My main issue is that I also want to update to latest upstream release for the next package upload, which is a minor version update, but in which the code layout changed a bit, requiring other changes in the packaging. I will not have time to do so in time for the freeze, so I'm just trying to postpone this work post-freeze, and I'll address this then. If you, or others, have time and interest to fix this before, by all means go ahead. The package is team-maintained, and I'm very happy to receive useful NMUs to "my" packages anyway. I hope this clarifies, Cheers -- Stefano Zacchiroli . z...@upsilon.cc . upsilon.cc/zack . . o . . . o . o Computer Science Professor . CTO Software Heritage . . . . . o . . . o o Former Debian Project Leader & OSI Board Director . . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: PGP signature
Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
On Sun, Mar 03, 2019 at 11:38:32PM +0100, Stefano Zacchiroli wrote: > Similarly to the other bug you mention, I think the incidence of this > FTBFS is low... at least elsewhere w.r.t. your build environment. I've > personal never witnessed this, and I'm aware of a bunch of other people > who have built from scratch without issues. More to the point, > reproducible building the package is working fine on at least four > architectures, including the one you are testing on: > https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/beancount.html I fear that this kind of reasoning may be leading us to discriminating users based on how close/far their building systems are to reproducible-builds.org or buildd.debian.org regarding things not covered by Debian Policy (for example, overall computer speed). Also, I hope you will surely agree that telling me that you can build the package in your computer does not help me at all to build the package in my computer. In this case I'm just doing QA, but if I wanted to modify the software, the end result is that I could not. So, while we are not violating the DFSG in strict sense, as we are still distributing the source, the user still can't exercise their right to modify the software in practical terms. How can we ensure that the user can rebuild any package if we start downgrading bugs like this one? Maybe by telling them that they have to use another computer? That would be, imo, slighly more obnoxious than telling them that have to install a missing build-depends, something which is already a serious bug. (I'm not changing the severity, and I'm still offering a test machine for you to reproduce the issue if you need it). Thanks.
Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
severity -1 important thanks Hi Santiago, On Sat, Mar 02, 2019 at 06:03:10PM +, Santiago Vila wrote: > I tried to build this package in buster but it failed: > [...] > try: > > os.unlink(entry.name, dir_fd=topfd) > E FileNotFoundError: [Errno 2] No such file or directory: > 'S.gpg-agent.browser' [...] > The above is just how the build ends and not necessarily the most relevant > part, > so I've put a bunch of failed build logs here: > > https://people.debian.org/~sanvila/build-logs/beancount/ > > The failure happens randomly. Sometimes it happens, sometimes it does not, > but the failure rate is so high that we can't really say that the package > "builds from source and without failure". thanks for this bug report, I had no idea about this flakiness in the build. > This failure is similar to another one which I reported before in another > package: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=906335 > > but I'm not really sure if that helps. > > If you need a test machine where this happens almost all the time, please > contact me > privately and I will gladly offer ssh access. Similarly to the other bug you mention, I think the incidence of this FTBFS is low... at least elsewhere w.r.t. your build environment. I've personal never witnessed this, and I'm aware of a bunch of other people who have built from scratch without issues. More to the point, reproducible building the package is working fine on at least four architectures, including the one you are testing on: https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/beancount.html On that basis, I'm lowering the priority to important. Still, I think the bug should be fixed, and I will do so (most likely ignoring the rm error, as that means no cleanup is needed). But I will not necessarily do this in time for the next release. Cheers. -- Stefano Zacchiroli . z...@upsilon.cc . upsilon.cc/zack . . o . . . o . o Computer Science Professor . CTO Software Heritage . . . . . o . . . o o Former Debian Project Leader & OSI Board Director . . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: PGP signature
Bug#923606: beancount: FTBFS randomly (gpg-related race condition)
Package: src:beancount
Version: 2.2.0-2
Severity: serious
Tags: ftbfs
Dear maintainer:
I tried to build this package in buster but it failed:
[...]
debian/rules binary-indep
dh binary-indep --with python3,elpa --buildsystem pybuild
dh_update_autotools_config -i -O--buildsystem=pybuild
dh_autoreconf -i -O--buildsystem=pybuild
dh_auto_configure -i -O--buildsystem=pybuild
I: pybuild base:217: python3.7 setup.py config
running config
debian/rules override_dh_auto_build
make[1]: Entering directory '/<>'
dh_auto_build
I: pybuild base:217: /usr/bin/python3 setup.py build
running build
running build_py
creating /<>/.pybuild/cpython3_3.7/build/beancount
[... snipped ...]
is_dir = False
if is_dir:
try:
dirfd = os.open(entry.name, os.O_RDONLY, dir_fd=topfd)
except OSError:
onerror(os.open, fullname, sys.exc_info())
else:
try:
if os.path.samestat(orig_st, os.fstat(dirfd)):
_rmtree_safe_fd(dirfd, fullname, onerror)
try:
os.rmdir(entry.name, dir_fd=topfd)
except OSError:
onerror(os.rmdir, fullname, sys.exc_info())
else:
try:
# This can only happen if someone replaces
# a directory with a symlink after the call to
# os.scandir or stat.S_ISDIR above.
raise OSError("Cannot call rmtree on a symbolic
"
"link")
except OSError:
onerror(os.path.islink, fullname,
sys.exc_info())
finally:
os.close(dirfd)
else:
try:
> os.unlink(entry.name, dir_fd=topfd)
E FileNotFoundError: [Errno 2] No such file or directory:
'S.gpg-agent.browser'
/usr/lib/python3.7/shutil.py:447: FileNotFoundError
=== warnings summary ===
/usr/lib/python3/dist-packages/bottle.py:87
/usr/lib/python3/dist-packages/bottle.py:87: DeprecationWarning: Using or
importing the ABCs from 'collections' instead of from 'collections.abc' is
deprecated, and in 3.8 it will stop working
from collections import MutableMapping as DictMixin
.pybuild/cpython3_3.7/build/beancount/ingest/regression_test.py::TestImporterTestGenerators::test_compare_sample_files__no_directory
/tmp/TestImporterTestGenerators.wuw61umy:170: DeprecationWarning: Call to
deprecated function compare_sample_files: Use
beancount.ingest.regression_pytest instead
-- Docs: https://docs.pytest.org/en/latest/warnings.html
== 1 failed, 1632 passed, 22 skipped, 2 xfailed, 2 warnings in 137.49 seconds ==
E: pybuild pybuild:338: test: plugin distutils failed with: exit code=1: cd
/<>/.pybuild/cpython3_3.7/build; python3.7 -m pytest -v
dh_auto_test: pybuild --test --test-pytest -i python{version} -p 3.7 returned
exit code 13
make: *** [debian/rules:13: binary-indep] Error 25
dpkg-buildpackage: error: debian/rules binary-indep subprocess returned exit
status 2
The above is just how the build ends and not necessarily the most relevant part,
so I've put a bunch of failed build logs here:
https://people.debian.org/~sanvila/build-logs/beancount/
The failure happens randomly. Sometimes it happens, sometimes it does not,
but the failure rate is so high that we can't really say that the package
"builds from source and without failure".
This failure is similar to another one which I reported before in another
package:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=906335
but I'm not really sure if that helps.
If you need a test machine where this happens almost all the time, please
contact me
privately and I will gladly offer ssh access.
Thanks.
