[Python-modules-team] Bug#776443: sphinx: please make output reproducible
Chris Lamb:
The attached patch attempts to remedy these issues. Once applied, many
packages that use sphinx--but alas not sphinx itself yet!--can be built
reproducibly in our current experimental
framework.
Attached is a full patch of the package uploaded in the reproducible
experimental toolchain [1]. It also fixes a minor issue with Chris'
original patch regarding domains ordering.
[1]: https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain
--
Lunar.''`.
lu...@debian.org: :Ⓐ : # apt-get install anarchism
`. `'`
`-
diff -Nru sphinx-1.2.3+dfsg/debian/changelog sphinx-1.2.3+dfsg/debian/changelog
--- sphinx-1.2.3+dfsg/debian/changelog 2014-09-12 10:33:57.0 +0200
+++ sphinx-1.2.3+dfsg/debian/changelog 2015-01-28 15:39:42.0 +0100
@@ -1,3 +1,11 @@
+sphinx (1.2.3+dfsg-1.0~reproducible1) UNRELEASED; urgency=low
+
+ [ Chris Lamb ]
+ * Add remove_non_determinism.diff to make Sphinx output reproducible
+from one build to the other (closes: #776443).
+
+ -- Jérémy Bobbio lu...@debian.org Wed, 28 Jan 2015 14:38:24 +
+
sphinx (1.2.3+dfsg-1) unstable; urgency=medium
* New upstream bugfix release.
diff -Nru sphinx-1.2.3+dfsg/debian/patches/remove_non_determinism.diff sphinx-1.2.3+dfsg/debian/patches/remove_non_determinism.diff
--- sphinx-1.2.3+dfsg/debian/patches/remove_non_determinism.diff 1970-01-01 01:00:00.0 +0100
+++ sphinx-1.2.3+dfsg/debian/patches/remove_non_determinism.diff 2015-01-28 16:03:13.0 +0100
@@ -0,0 +1,133 @@
+Description: remove non-determinism
+ To enable packages using Sphinx to build reproducibly, its output
+ needs to be the same from one build to another.
+ .
+ Its output now strips memory references such as:
+ .
+ __main__.A at 0x7f68cb685710
+ .
+ In addition, various generated files (objects.inv, searchindex.js,
+ translations) are now written with their keys in a determinstic order.
+Author: Chris Lamb la...@debian.org
+
+--- sphinx-1.2.3+dfsg.orig/sphinx/builders/html.py
sphinx-1.2.3+dfsg/sphinx/builders/html.py
+@@ -269,7 +269,8 @@ class StandaloneHTMLBuilder(Builder):
+ # html_domain_indices can be False/True or a list of index names
+ indices_config = self.config.html_domain_indices
+ if indices_config:
+-for domain in self.env.domains.itervalues():
++for domain_name in sorted(self.env.domains.keys()):
++domain = self.env.domains[domain_name]
+ for indexcls in domain.indices:
+ indexname = '%s-%s' % (domain.name, indexcls.name)
+ if isinstance(indices_config, list):
+@@ -808,7 +809,7 @@ class StandaloneHTMLBuilder(Builder):
+ compressor = zlib.compressobj(9)
+ for domainname, domain in self.env.domains.iteritems():
+ for name, dispname, type, docname, anchor, prio in \
+-domain.get_objects():
++sorted(domain.get_objects()):
+ if anchor.endswith(name):
+ # this can shorten the inventory by as much as 25%
+ anchor = anchor[:-len(name)] + '$'
+--- sphinx-1.2.3+dfsg.orig/sphinx/ext/autodoc.py
sphinx-1.2.3+dfsg/sphinx/ext/autodoc.py
+@@ -60,7 +60,6 @@ class DefDict(dict):
+
+ identity = lambda x: x
+
+-
+ class Options(dict):
+ A dict/attribute hybrid that returns None on nonexisting keys.
+ def __getattr__(self, name):
+@@ -975,7 +974,8 @@ class FunctionDocumenter(DocstringSignat
+ argspec = getargspec(self.object.__init__)
+ if argspec[0]:
+ del argspec[0][0]
+-args = inspect.formatargspec(*argspec)
++args = inspect.formatargspec(*argspec,
++ formatvalue=lambda x: '=' + safe_repr(x))
+ # escape backslashes for reST
+ args = args.replace('\\', '')
+ return args
+@@ -1030,7 +1030,8 @@ class ClassDocumenter(ModuleLevelDocumen
+ return None
+ if argspec[0] and argspec[0][0] in ('cls', 'self'):
+ del argspec[0][0]
+-return inspect.formatargspec(*argspec)
++return inspect.formatargspec(*argspec,
++ formatvalue=lambda x: '=' + safe_repr(x))
+
+ def format_signature(self):
+ if self.doc_as_attr:
+@@ -1229,7 +1230,8 @@ class MethodDocumenter(DocstringSignatur
+ argspec = getargspec(self.object)
+ if argspec[0] and argspec[0][0] in ('cls', 'self'):
+ del argspec[0][0]
+-args = inspect.formatargspec(*argspec)
++args = inspect.formatargspec(*argspec,
++ formatvalue=lambda x: '=' + safe_repr(x))
+ # escape backslashes for reST
+ args = args.replace('\\', '')
+ return args
+---
[Python-modules-team] Bug#776443: sphinx: please make output reproducible
Control: forwarded -1 https://github.com/sphinx-doc/sphinx/pull/1694 I have now forwarded your patch upstream (needed a bit of rebasing), let's see what upstream thinks about it. As we are in freeze now, I guess this can wait until the new upstream release, right? -- Dmitry Shachnev signature.asc Description: OpenPGP digital signature ___ Python-modules-team mailing list Python-modules-team@lists.alioth.debian.org http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/python-modules-team
[Python-modules-team] Bug#776443: sphinx: please make output reproducible
Source: sphinx
Version: 1.2.3+dfsg-1
Severity: wishlist
Tags: patch
User: reproducible-bui...@lists.alioth.debian.org
Usertags: toolchain randomness
X-Debbugs-Cc: reproducible-bui...@lists.alioth.debian.org
Hi,
While working on the reproducible builds effort [1], we have noticed
that sphinx is generating documentation that is not repoducible.
For example, its output includes non-deterministic memory references
such as:
__main__.A at 0x7f68cb685710
In addition, various generated (objects.inv, searchindex.js,
translations) do not output their keys in a determinstic order,
resulting in further randomness.
The attached patch attempts to remedy these issues. Once applied, many
packages that use sphinx--but alas not sphinx itself yet!--can be built
reproducibly in our current experimental
framework.
[1]: https://wiki.debian.org/ReproducibleBuilds
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-
diff --git a/sphinx/builders/html.py b/sphinx/builders/html.py
index 9c039e3..f489a35 100644
--- a/sphinx/builders/html.py
+++ b/sphinx/builders/html.py
@@ -269,7 +269,7 @@ class StandaloneHTMLBuilder(Builder):
# html_domain_indices can be False/True or a list of index names
indices_config = self.config.html_domain_indices
if indices_config:
-for domain in self.env.domains.itervalues():
+for domain in sorted(self.env.domains.itervalues()):
for indexcls in domain.indices:
indexname = '%s-%s' % (domain.name, indexcls.name)
if isinstance(indices_config, list):
@@ -808,7 +808,7 @@ class StandaloneHTMLBuilder(Builder):
compressor = zlib.compressobj(9)
for domainname, domain in self.env.domains.iteritems():
for name, dispname, type, docname, anchor, prio in \
-domain.get_objects():
+sorted(domain.get_objects()):
if anchor.endswith(name):
# this can shorten the inventory by as much as 25%
anchor = anchor[:-len(name)] + '$'
diff --git a/sphinx/ext/autodoc.py b/sphinx/ext/autodoc.py
index 423f921..721fbb4 100644
--- a/sphinx/ext/autodoc.py
+++ b/sphinx/ext/autodoc.py
@@ -60,7 +60,6 @@ class DefDict(dict):
identity = lambda x: x
-
class Options(dict):
A dict/attribute hybrid that returns None on nonexisting keys.
def __getattr__(self, name):
@@ -975,7 +974,8 @@ class FunctionDocumenter(DocstringSignatureMixin,
ModuleLevelDocumenter):
argspec = getargspec(self.object.__init__)
if argspec[0]:
del argspec[0][0]
-args = inspect.formatargspec(*argspec)
+args = inspect.formatargspec(*argspec,
+ formatvalue=lambda x: '=' + safe_repr(x))
# escape backslashes for reST
args = args.replace('\\', '')
return args
@@ -1030,7 +1030,8 @@ class ClassDocumenter(ModuleLevelDocumenter):
return None
if argspec[0] and argspec[0][0] in ('cls', 'self'):
del argspec[0][0]
-return inspect.formatargspec(*argspec)
+return inspect.formatargspec(*argspec,
+ formatvalue=lambda x: '=' + safe_repr(x))
def format_signature(self):
if self.doc_as_attr:
@@ -1229,7 +1230,8 @@ class MethodDocumenter(DocstringSignatureMixin,
ClassLevelDocumenter):
argspec = getargspec(self.object)
if argspec[0] and argspec[0][0] in ('cls', 'self'):
del argspec[0][0]
-args = inspect.formatargspec(*argspec)
+args = inspect.formatargspec(*argspec,
+ formatvalue=lambda x: '=' + safe_repr(x))
# escape backslashes for reST
args = args.replace('\\', '')
return args
diff --git a/sphinx/search/__init__.py b/sphinx/search/__init__.py
index bd95ecc..760b137 100644
--- a/sphinx/search/__init__.py
+++ b/sphinx/search/__init__.py
@@ -268,13 +268,13 @@ class IndexBuilder(object):
if fn in fn2index:
rv[k] = fn2index[fn]
else:
-rv[k] = [fn2index[fn] for fn in v if fn in fn2index]
+rv[k] = sorted([fn2index[fn] for fn in v if fn in
fn2index])
return rvs
def freeze(self):
Create a usable data structure for serializing.
-filenames = self._titles.keys()
-titles = self._titles.values()
+filenames = sorted(self._titles.keys())
+titles = sorted(self._titles.values())
fn2index = dict((f, i) for (i, f) in enumerate(filenames))
terms, title_terms = self.get_terms(fn2index)
diff --git a/sphinx/util/inspect.py b/sphinx/util/inspect.py
index cdbfea7..e04f1fa 100644
--- a/sphinx/util/inspect.py
+++ b/sphinx/util/inspect.py
@@
