Re: [PATCH] gnu: Add Nmap.
taylanbayi...@gmail.com (Taylan Ulrich Bayırlı/Kammer) skribis: l...@gnu.org (Ludovic Courtès) writes: [...] The whole functional approach things means that bindings are static That's a neat analogy. :-) It’s not just an analogy, it’s really what happens. However, we needn't recompile C code when there's an update to some dynlang library the program uses, do we? Yes we do. Also, configure scripts that check for such dynlang libraries' presence might become a problem. I imagine one could provide some for-building version of said library that's merely of the same major version, to shut up the configure script, since it only gets in the way in this case. (Complaining about the lack of a run-time dependency, at build-time... Yes, above I said *not* doing this is probably a bug in Nmap's build process, but for us it would become an annoyance.) Alternatively, the script might simply take a flag telling it to enable a feature even if it can't find a run-time dependency; I don't how many configure scripts have this. Most of the time, packages check at configure-time whether their perquisites are available. Then, their test suite use these specific dependencies. Finally, in the case of DSOs, the RUNPATH makes sure that these specific dependencies are used at run time. Then there’s the case of dynamic languages with no RUNPATH equivalent. In that case we arrange with ‘propagated-inputs’ or with wrappers to make sure the “right” dependencies are used at run time. I’m not sure if that answers your concerns, does it? Same thing as above really. Until the ABI of a dylib changes, updates to it needn't ignite recompilation of C code which uses that lib; it won't affect the compiler output anyway. (Tell me if I'm technically wrong; not an expert on C compilation etc.) At the very least, the RUNPATH will be different. But then, keep in mind we’re aiming for complete reproducibility. So we can’t rely on things like “oh, developers say this version is ABI-compatible with the previous one.” It does sound like breaking functional purity, but is it really so when an input has *no* effect on a build process other than the test suite? That’s an important effect: it means that the test suite is known to work this particular input. Once the test suite is separated, the real package and for-building package become operationally equivalent under the given build process, so providing one in place of the other would be a correctness preserving optimization. :-) Yeah, I see where you’re going. :-) Eelco Dolstra’s PhD thesis on Nix had that idea of “equivalence classes”. It’s not very practical, though. In that case, we should already be able to do at least this, no? (The other ideas are sadly high up in the air.) It could have prevented world-rebuilds upon the bash security patches, for example. (Given it's acceptable to use a buggy (even security-holy) Bash for some ./configure scripts and the like, so long as it doesn't break specifically them.) See “Security Updates” in the manual for that. Runtime dependencies are found by scanning binaries for the hash parts of Nix store paths (such as r8vvq9kq…). This sounds risky, but it works extremely well. Perhaps we already do that? Yes. That means that run-time dependencies are a strict subset of build-time dependencies. I see. One confusion remains: wouldn't that mean it would drop a dependency such as the Python GTK module in our case, because its hash doesn't appear anywhere, and instead it's only referenced by module name which is expected to be in some Python load path at run-time? Correct. That’s why we’d either use ‘propagated-inputs’ or ‘wrap-program’, depending on the situation. I hope this clarifies things. So, where were we with this nmap patch? :-) Ludo’.
Re: [PATCH] gnu: Add Nmap.
l...@gnu.org (Ludovic Courtès) writes: [large snip] I hope this clarifies things. Yes! Thanks for your patience. It now dawns on me that our primary aim isn't byte-equal build process outputs (in terms of file contents); it's immutable run-time behavior of a package. That is to say, if /gnu/store/...-foo/bin/foo works fine today, it should work fine any day; it's unacceptable to let it load a different .so than its designated one, unless we define some very strict criteria for the .so and prove that a replacement satisfies them... Similarly for some Python module the program uses. I was chasing a reproducible builds red herring. That being said, I still think it's wasteful that C code gets recompiled when there's an update to a dynamic component. A rough idea for the future: save some intermediate build output (say from the 'build' phase), then when there's an update to a run-time component such as a DSO or dynlang module, make a new package that merely receives some tweaks to the saved intermediate output, such as fixing RUNPATH, re-patching shebang lines, etc. Users will still need to download these new packages; that in turn could be improved via binary diffs one day. So, where were we with this nmap patch? :-) Heh, indeed. I thought I'd wait for feedback from the Nmap ML but I guess that's not necessary. I'll continue working on it tomorrow. Taylan
Re: [PATCH] gnu: Add Nmap.
On Tue, Nov 25, 2014 at 5:25 PM, Taylan Ulrich Bayırlı/Kammer taylanbayi...@gmail.com wrote: That being said, I still think it's wasteful that C code gets recompiled when there's an update to a dynamic component. A rough idea for the future: save some intermediate build output (say from the 'build' phase) I added the ccache package a little while ago. I had wondered whether it would be possible to run all our builds through ccache so that updates to core packages might not be as much of a time burden (at the cost of some extra disk space). Might be something interesting to try. `~Eric -- Please avoid sending me Word or PowerPoint attachments. See http://www.gnu.org/philosophy/no-word-attachments.html
Re: [PATCH] gnu: Add Nmap.
taylanbayi...@gmail.com (Taylan Ulrich Bayırlı/Kammer) skribis: l...@gnu.org (Ludovic Courtès) writes: This is normally dealt with by using multiple outputs (info (guix) Packages with Multiple Outputs). An example of that is Git: the Tcl GUIs are moved to a separate output, and so is git-svn support, such that the main output does not depend on Tcl, libx11, Subversion, etc. OK, will have a go at this. Thanks, that seems like the best option. It seems Zenmap doesn't need X11/GTK libraries (rather headers) at build time because it only uses a Python GTK module. This raises two general questions for me: 1) Is it OK if users have to install additional packages for a given component of a package to work, or should all dependencies, even if purely run-time, be inputs? Yes. (There may be rare exceptions, but this is not one of them.) Here you want users to be able to run ‘zenmap’ (or whatever the command is called) and have it Just Work, regardless of whether they happen to have installed Python and Python-GTK as well. So that means that the zenmap (or nmap) package basically needs to “close over” these inputs, probably using ‘wrap-program’. 2) If purely-run-time dependencies are inputs, won't that trigger unnecessary rebuilds of the package when a run-time dependency is updated? It’s rarely be “purely run-time”. Normally, at the very least there’s a configure script that makes sure at build-time that the dependencies are satisfied, and a test suite that makes sure things work. After some pondering, I would say: 1) There should be a way to run-time-depend on another package without it being a build input at all. The whole functional approach things means that bindings are static (again, there may be exceptions, but zenmap has nothing exceptional here.) 2) When interface files of a dylib are needed during compilation of a static lang (e.g. C headers), a special for-building package should be used as input, thus the actual dylib can be updated without causing rebuilds. (Until ABI compatibility breaks I guess.) You’re describing an imperative packaging system. This is fine, but it defeats the whole idea of functional packaging. :-) See http://nixos.org/docs/papers.html and http://gnu.org/s/guix/#documentation for the rationale. 3) Similarly, when a program is needed purely at build-time, like Bash or SCons, a special for-building package should be used as input, thus the actual program can be updated without causing rebuilds. (The for-building package would be updated only when doing so will improve the builds, like when a newer GCC version optimizes better.) Build tools are listed in ‘native-inputs’. Runtime dependencies are found by scanning binaries for the hash parts of Nix store paths (such as r8vvq9kq…). This sounds risky, but it works extremely well. Perhaps we already do that? Yes. That means that run-time dependencies are a strict subset of build-time dependencies. I hope this clarifies things. Otherwise let me know! Thanks, Ludo’.
Re: [PATCH] gnu: Add Nmap.
l...@gnu.org (Ludovic Courtès) writes: So that means that the zenmap (or nmap) package basically needs to “close over” these inputs, probably using ‘wrap-program’. Thanks for the pointer; I'll look into 'wrap-program'. 2) If purely-run-time dependencies are inputs, won't that trigger unnecessary rebuilds of the package when a run-time dependency is updated? It’s rarely be “purely run-time”. Normally, at the very least there’s a configure script that makes sure at build-time that the dependencies are satisfied, and a test suite that makes sure things work. You're right; now that I think of it, it's probably a bug that Nmap's build process doesn't check for the presence of a GTK module. I'll talk to their developers to clarify. (ML thread should pop up at the bottom of http://seclists.org/nmap-dev/2014/q4/index.html soon.) 1) There should be a way to run-time-depend on another package without it being a build input at all. The whole functional approach things means that bindings are static That's a neat analogy. :-) However, we needn't recompile C code when there's an update to some dynlang library the program uses, do we? Then again, we want to re-run a potential test suite that might be using that library. I guess it's OK to be conservative for now, until One Day™ we can be more precise and separate testing from building... (If that day comes though, we better find a way to automatically discern run-time-only inputs. (Regarding test suites to be run-time too.) Otherwise we'll have a lot to do.) Also, configure scripts that check for such dynlang libraries' presence might become a problem. I imagine one could provide some for-building version of said library that's merely of the same major version, to shut up the configure script, since it only gets in the way in this case. (Complaining about the lack of a run-time dependency, at build-time... Yes, above I said *not* doing this is probably a bug in Nmap's build process, but for us it would become an annoyance.) Alternatively, the script might simply take a flag telling it to enable a feature even if it can't find a run-time dependency; I don't how many configure scripts have this. 2) When interface files of a dylib are needed during compilation of a static lang (e.g. C headers), a special for-building package should be used as input, thus the actual dylib can be updated without causing rebuilds. (Until ABI compatibility breaks I guess.) You’re describing an imperative packaging system. This is fine, but it defeats the whole idea of functional packaging. :-) See http://nixos.org/docs/papers.html and http://gnu.org/s/guix/#documentation for the rationale. Same thing as above really. Until the ABI of a dylib changes, updates to it needn't ignite recompilation of C code which uses that lib; it won't affect the compiler output anyway. (Tell me if I'm technically wrong; not an expert on C compilation etc.) But again, test suites... It does sound like breaking functional purity, but is it really so when an input has *no* effect on a build process other than the test suite? Once the test suite is separated, the real package and for-building package become operationally equivalent under the given build process, so providing one in place of the other would be a correctness preserving optimization. :-) 3) Similarly, when a program is needed purely at build-time, like Bash or SCons, a special for-building package should be used as input, thus the actual program can be updated without causing rebuilds. (The for-building package would be updated only when doing so will improve the builds, like when a newer GCC version optimizes better.) Build tools are listed in ‘native-inputs’. Ah, thanks for clarifying. (BTW 'native-inputs' seems undocumented.) In that case, we should already be able to do at least this, no? (The other ideas are sadly high up in the air.) It could have prevented world-rebuilds upon the bash security patches, for example. (Given it's acceptable to use a buggy (even security-holy) Bash for some ./configure scripts and the like, so long as it doesn't break specifically them.) Runtime dependencies are found by scanning binaries for the hash parts of Nix store paths (such as r8vvq9kq…). This sounds risky, but it works extremely well. Perhaps we already do that? Yes. That means that run-time dependencies are a strict subset of build-time dependencies. I see. One confusion remains: wouldn't that mean it would drop a dependency such as the Python GTK module in our case, because its hash doesn't appear anywhere, and instead it's only referenced by module name which is expected to be in some Python load path at run-time? Thanks a lot and sorry for taking your time! I might be dumping out many dubious ideas. Taylan
Re: [PATCH] gnu: Add Nmap.
On Sat, Nov 22, 2014 at 11:05:35PM +0100, Taylan Ulrich Bayırlı/Kammer wrote: Many distros separate Zenmap from the Nmap package. Should we do the same? If yes, I'll need some help. This recipe builds it together with the rest of the suite. Our general policy is to build as intended by upstream with as little changes as possible. So unless there is a compelling reason to separate the two, I would not do it. Andreas
Re: [PATCH] gnu: Add Nmap.
Andreas Enge andr...@enge.fr writes: On Sat, Nov 22, 2014 at 11:05:35PM +0100, Taylan Ulrich Bayırlı/Kammer wrote: Many distros separate Zenmap from the Nmap package. Should we do the same? If yes, I'll need some help. This recipe builds it together with the rest of the suite. Our general policy is to build as intended by upstream with as little changes as possible. So unless there is a compelling reason to separate the two, I would not do it. Hm, zenmap would pull in X.Org and all sorts of GUI things, which would be very annoying for someone who wants nmap on a headless server, embedded system, or the like. Well, I didn't need to add any such inputs, and it built zenmap anyway, but of course it won't run without those. Does that mean I made a mistake, or is it fine that users have to know what run-time dependencies to install for certain components of packages to work? It seems Debian recently split of ndiff as well, after which the nmap package itself won't depend on Python either, and only has very slim dependencies. (I *did* have to input python-2 for it to build ndiff and zenmap...) This topic seems to tie in well with Gammel Holte's recent post on the ML regarding how optional run-time dependencies will be handled in Guix. Maybe we should wait for the outcome of that. Or at least I'll need to read up on Nix/Guix dependency handling, because I'm confused on what should and shouldn't be an input. Taylan
Re: [PATCH] gnu: Add Nmap.
taylanbayi...@gmail.com (Taylan Ulrich Bayırlı/Kammer) skribis: Andreas Enge andr...@enge.fr writes: On Sat, Nov 22, 2014 at 11:05:35PM +0100, Taylan Ulrich Bayırlı/Kammer wrote: Many distros separate Zenmap from the Nmap package. Should we do the same? If yes, I'll need some help. This recipe builds it together with the rest of the suite. Our general policy is to build as intended by upstream with as little changes as possible. So unless there is a compelling reason to separate the two, I would not do it. Hm, zenmap would pull in X.Org and all sorts of GUI things, which would be very annoying for someone who wants nmap on a headless server, embedded system, or the like. Indeed. This is normally dealt with by using multiple outputs (info (guix) Packages with Multiple Outputs). An example of that is Git: the Tcl GUIs are moved to a separate output, and so is git-svn support, such that the main output does not depend on Tcl, libx11, Subversion, etc. Would it work here? Thanks, Ludo’.
Re: [PATCH] gnu: Add Nmap.
l...@gnu.org (Ludovic Courtès) writes: This is normally dealt with by using multiple outputs (info (guix) Packages with Multiple Outputs). An example of that is Git: the Tcl GUIs are moved to a separate output, and so is git-svn support, such that the main output does not depend on Tcl, libx11, Subversion, etc. OK, will have a go at this. It seems Zenmap doesn't need X11/GTK libraries (rather headers) at build time because it only uses a Python GTK module. This raises two general questions for me: 1) Is it OK if users have to install additional packages for a given component of a package to work, or should all dependencies, even if purely run-time, be inputs? 2) If purely-run-time dependencies are inputs, won't that trigger unnecessary rebuilds of the package when a run-time dependency is updated? (In this special case only question 2 applies, because the whole of Zenmap is useless without GTK, not just a component of it.) After some pondering, I would say: 1) There should be a way to run-time-depend on another package without it being a build input at all. (The installation of these needn't be forced on the user, though in some cases like Zenmap it's senseless not to do so; we could have dependencies and recommendations and possibly more, like in Debian.) 2) When interface files of a dylib are needed during compilation of a static lang (e.g. C headers), a special for-building package should be used as input, thus the actual dylib can be updated without causing rebuilds. (Until ABI compatibility breaks I guess.) 3) Similarly, when a program is needed purely at build-time, like Bash or SCons, a special for-building package should be used as input, thus the actual program can be updated without causing rebuilds. (The for-building package would be updated only when doing so will improve the builds, like when a newer GCC version optimizes better.) 4) The for-building packages in #3 should obviously not be installed on the user's machine (unless they build locally instead of using binary substitutes), meaning build inputs and run-time dependencies are fully orthogonal. In the Nix manual I see the following, which possibly fixes #4 without needing to separate build inputs from run-time dependencies in recipes: Runtime dependencies are found by scanning binaries for the hash parts of Nix store paths (such as r8vvq9kq…). This sounds risky, but it works extremely well. Perhaps we already do that? But otherwise, it seems like we could save gigawatts of electricity over time with #1-3. Please tell me if I'm missing something obvious. :-) Taylan
