On 2021-01-09, Lisandro Damián Nicanor Pérez Meyer wrote: > Note: in case we do not agree on this topic this will be the text I'll > send to the > tech-ctte.
Thanks for taking the time to draft some text. If we can come closer to agreement on the proposed text, that would probably take a bit of the load off of the tech-ctte. Hopefully some of my comments move in that direction, although I also have added some counterpoint text as well... > Let me start by noting that I have nothing against reproducibility. In fact > quite the opposite: I love the idea... as long as it's properly implemented. It is even "recommended" in Debian Policy policy to build reproducibly with different build paths. > The problem here is that __FILE__ is a public, well defined API with very > valid > use cases (more on this below), even if the current implementation of all the > major compilers go against reproducibility. At least two major compilers, GCC and Clang provide the -ffile-prefix-map feature to do exactly that(which is what dpkg is enabling), so it seems a bit overstated to say that all major compilers "go against" reproducibility. They do not enable reproducibility by default. > So if we want to mangle __FILE__ > (thus breaking API) this should be done by an opt-in, and **never** by opting > out. Else we risk breaking a valid implementation, be it already on the > archive > any new package added afterwards. While I understand that may be how you feel, it would be appreciated if we could use something a little less loaded than "mangle". perhaps: So if we want to enable features using __FILE__ in a way which arguably breaks API > Even more: we library maintainers continuously ask our upstreams to keep their > API as stable as possible, and if not possible following some rules > (SONAME change, for example). I will present an option to do the same > ourselves > without breaking API/ABI. > > # __FILE__ is a public, well defined API > > During the course of #876901 many reasons were used to both say that __FILE__ > is or not a well defined API. In fact one of the evidences used where the > compiler's documentation. For example > https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html > (enphasis mine): > > This macro expands to the name of the current input file, in the form of a C > string constant. **This is the path by which the preprocessor opened > the file**, > not the short name specified in ‘#include’ or as the input file name > argument. > For example, "/usr/local/include/myheader.h" is a possible expansion of this > macro. > > This definition says that it's up to the preprocessor to define the path. So, > what has been the behaviour of all major compilers during all these years? > Using > the full path. The proof of this is Qt itself. Check > https://sources.debian.org/src/qtbase-opensource-src/5.15.2+dfsg-2/src/testlib/qtestcase.h/#L216 > > This code has been working on **every** platform Qt works without any change. > Qt is compiled in a myriad of OSs, using a myriad of compilers. They all do > the exact same thing. So developers depend on a very stable definition > of an API. > > And we all know that breaking API is bad, except if done carefully (read > below). > > # Doing the right thing > > This is just an idea of "doing the right thing", like bumping SONAME > on a library. > It definitely doesn't have to be the only one. I understand your position is fundamentally about "Doing the right thing" but I would say that we are all trying to do the right thing. > I understand that the reproducibility people do not want to consider > providing the > same build path. While this is arguable I do understand that > reproducibility without > depending on the build path is a good thing. So I've tried to come up with a > path to achieve this. > ## New macro and warning (if they do not exist already) > > This would be the first step. The idea is to provide a new macro that, > by definition > and documentation, it can be mangled with the help of the build > system, much as you > are currently doing with __FILE__ now. > > The second step in this is to make compilers create a reproducibility warning > if > some code uses __FILE__. This will have three effects effects: We haven't proposed an alternate macro, which would surely take years at best, possibly decades. That doesn't seem too realistic. > - discouraging it's use > - creating awareness on reproducibility. We've discouraged the use of __FILE__ for years, have done plenty of outreach on the subject of reproducibility, and gotten traction with various upstream projects. > - making other distros jump into reproducibility in a much easier way. Arguably some distros (e.g. archlinux) are passing us by when it comes to real-world reproducibility; I'm not sure Debian is the example by which all other distros should be measured anymore. Which is good in some ways, but somewhat disappointing to see Debian start something great and then lag behind. > ## Mangling by opting-in, or what to do in the meantime Maybe drop the phrase "Mangling"? Perhaps Opt-in to the fixfilepath feature, ... Or maybe: Keeping the opt-in status quo, ... > Of course we want to make things reproducible as fast as possible too. This is > why mangling __FILE__ was considered and implemented. But this is breaking > API, > no matter how you look at it. Well, that's the crux of the argument, isn't it? Is making an assumption about correct behavior by reading ambigous documentation that can be interpreted either way, and then examining default compiler behavior ... really an API? > But that doesn't means it can not be used: if a > package can be built mangling __FILE__ in order to get a reproducible way then > the maintainer should really consider opting *in* for the change, but never > ever should the maintainer be forced to opt *out* using a broken API. > > In fact I consider all the "fixes" of opting in a bug, and forcing this change > a couple of weeks before freeze even worse. So I propose you to undo the > change > in dpkg and, after the release, starting to work on letting maintainers opting > in. It has been an option to opt-in to this feature for at least two years; and some packages have done so. Adding boilerplate one-line changes to hundreds or thousands of packages in Debian seems suboptimal to actually effect change. I don't know if size of the issue is relevent to tech-ctte, but it seems worth noting that the archive-wide rebuild revealed less than 30 packages broken by this change, most of which have been patched to opt-out of this feature. Most or all of the reamaining issues are due to QFINDTESTDATA in test suites. And for completeness, It was speculated that hundreds more packages which currently disable test suites would need to also opt-out of this feature in order to enable test suites, or come up with some other way to remove these flags during tests. It also seems worth asking what are the negative impacts of enabling this, other than a handful of packages that need changes to debian/rules in order to build? Thanks for taking the time with all this everyone! live well, vagrant
signature.asc
Description: PGP signature