Re: [Nix-dev] Source URL breakage – please can we improve the determinism
On 9 May 2016 at 20:21, Eelco Dolstra wrote: > * The mirroring script only copies tarballs of packages in the Nixpkgs Hydra > jobset. Since moreutils is not part of the jobset, it's not mirrored. This can > be fixed by adding a meta.platforms attribute to moreutils. Thank you! (Completely forgot about meta.platforms...) - Bjørn ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Source URL breakage – please can we improve the determinism
Hi, On 08/05/16 02:19, J. R. Haigh wrote: > Dear Nix project leaders, I don't find NixOS to be very deterministic at all, > and it's nearly always for the same reason: source files on random servers > scattered across the Internet going walkabouts. We have a content-addressed tarball mirror at http://tarballs.nixos.org for this purpose. "fetchurl" will automatically use this mirror to obtain files by hash. However: * The mirroring script was not running lately. I've revived it so 16.03 tarballs are mirrored now (https://github.com/NixOS/nixos-org-configurations/commit/a17ccf87deae4fb86639c8d34ab5938edd68d8c4). * The mirroring script only copies tarballs of packages in the Nixpkgs Hydra jobset. Since moreutils is not part of the jobset, it's not mirrored. This can be fixed by adding a meta.platforms attribute to moreutils. -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Source URL breakage – please can we improve the determinism
On Sun, 8 May 2016 18:11:09 +0300 Игорь Пашев wrote: > Can't be nix cache used for this? Source tarballs are kind of > derivation too. Somehow I was under the impression that the output of "fetchFromUrl" or "fetchFromGit" was also a result worth storing in the nix-store. And thus also part of the 'binary'-cache. Or is that only when configuring a private binary-cache/hydra? - Arnold signature.asc Description: PGP signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Source URL breakage - please can we improve the determinism
We have also run into this issue with spotify and similar unfree / non-redistributable packages. The consensus last time we had upstream URLs disappear was that it is not permissible to cache and redistribute even the original source. There may be a partial solution for this problem, but it will not solve the issue for nonfree packages. On Sun, May 8, 2016 at 10:11 AM Игорь Пашев wrote: > Can't be nix cache used for this? Source tarballs are kind of derivation > too. > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev > ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Source URL breakage – please can we improve the determinism
Can't be nix cache used for this? Source tarballs are kind of derivation too. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Source URL breakage – please can we improve the determinism
On 8 May 2016 at 02:19, J. R. Haigh wrote: > Dear Nix project leaders, > I don't find NixOS to be very deterministic at all, and it's nearly > always for the same reason: source files on random servers scattered across > the Internet going walkabouts. > Please can we have all source files hosted by the Nix project and all > source URLs in nixpkgs and nixpkgs-channels replaced or accompanied by the > Nix-hosted copies. +1. Except I don't think we should change the URLs in nixpkgs, those should still point to upstream. IMO, Nix fetchers should be configurable to try Nix mirrors before (or after) the given URLs in Nix package files. > Ideally there should be a way to specify multiple URLs for each source file, > so both the original and the Nix-hosted locations can be specified, as well > as any other mirrors if there are any. That is already possible. For instance, fetchurl can take a list of urls in the "urls" attribute (note the 's'). (But again, I'd prefer to not have to specify these Nix provided backup URLs manually for each package, better do that globally.) > In this specific instance, please could someone with commit access resolve > this breakage a.s.a.p.?: > https://github.com/NixOS/nixpkgs-channels/blob/nixos-15.09/pkgs/tools/misc/moreutils/default.nix That was fixed just a couple of days ago. But unfortunately it takes a while for the change to hit the channel :-( (Until then, build from git.) Best regards, Bjørn Forsman ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Source URL breakage – please can we improve the determinism
On Sun, 8 May 2016, 1:19 a.m. J. R. Haigh, wrote: > Dear Nix project leaders, > I don't find NixOS to be very deterministic at all, and it's > nearly always for the same reason: source files on random servers scattered > across the Internet going walkabouts. > While I agree this is a problem, this is not a problem with determinism. If the source code can't be found as it is expected, then the build fails - that *is* deterministic. Please can we have all source files hosted by the Nix project and > all source URLs in nixpkgs and nixpkgs-channels replaced or accompanied by > the Nix-hosted copies. Ideally there should be a way to specify multiple > URLs for each source file, so both the original and the Nix-hosted > locations can be specified, as well as any other mirrors if there are any. > We do have the mirror:// scheme which knows to look in many locations, but I imagine the source you're referring to is not mirrored? Ollie ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Source URL breakage – please can we improve the determinism
Its definitely annoying. There's discussion about it involving ipfs. One stopgap solution is to rollback just for only that package. You can do that with nix-env -f on a tar.gz archive of nixpkgs from a commit hash on github. On 08/05/2016 10:19 AM, "J. R. Haigh" wrote: > Dear Nix project leaders, > I don't find NixOS to be very deterministic at all, and it's > nearly always for the same reason: source files on random servers scattered > across the Internet going walkabouts. > Please can we have all source files hosted by the Nix project and > all source URLs in nixpkgs and nixpkgs-channels replaced or accompanied by > the Nix-hosted copies. Ideally there should be a way to specify multiple > URLs for each source file, so both the original and the Nix-hosted > locations can be specified, as well as any other mirrors if there are any. > As for the duration of the hosting, I think that at least every > release build that goes through Hydra to build live images and such should > keep hosted files (source or binary alike) indefinitely. Anything that was > not one of these Hydra builds I don't mind too much if they go missing > because I nearly always stick to the release builds, but it would still be > appreciated if all source files could be hosted indefinitely to aid with > binary searches of regressions or other changes. > Better still, if magnet URIs could be specified and Nix would > support downloading from torrents then it would be very easy to avoid > breakage due to missing source files. > > In this specific instance, please could someone with commit access > resolve this breakage a.s.a.p.?: > https://github.com/NixOS/nixpkgs-channels/blob/nixos-15.09/pkgs/tools/misc/moreutils/default.nix > Perhaps it could be done by updating it from 0.57 to 0.58 or by > finding it on another mirror somewhere. The Debian source archive at > http://ftp.de.debian.org/debian/pool/main/m/moreutils/ lists 0.47, 0.52, > and 0.58, but not 0.57. Moreutils is depended on by Mercurial and Mercurial > is a build dependency of many things, so that one source breakage has > broken a lot of things from building in the head of the NixOS 15.09 release > channel branch. > > Best regards, > James R. Haigh. > -- > Sent from Claws Mail on NixOS. > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev > ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Source URL breakage – please can we improve the determinism
Dear Nix project leaders, I don't find NixOS to be very deterministic at all, and it's nearly always for the same reason: source files on random servers scattered across the Internet going walkabouts. Please can we have all source files hosted by the Nix project and all source URLs in nixpkgs and nixpkgs-channels replaced or accompanied by the Nix-hosted copies. Ideally there should be a way to specify multiple URLs for each source file, so both the original and the Nix-hosted locations can be specified, as well as any other mirrors if there are any. As for the duration of the hosting, I think that at least every release build that goes through Hydra to build live images and such should keep hosted files (source or binary alike) indefinitely. Anything that was not one of these Hydra builds I don't mind too much if they go missing because I nearly always stick to the release builds, but it would still be appreciated if all source files could be hosted indefinitely to aid with binary searches of regressions or other changes. Better still, if magnet URIs could be specified and Nix would support downloading from torrents then it would be very easy to avoid breakage due to missing source files. In this specific instance, please could someone with commit access resolve this breakage a.s.a.p.?: https://github.com/NixOS/nixpkgs-channels/blob/nixos-15.09/pkgs/tools/misc/moreutils/default.nix Perhaps it could be done by updating it from 0.57 to 0.58 or by finding it on another mirror somewhere. The Debian source archive at http://ftp.de.debian.org/debian/pool/main/m/moreutils/ lists 0.47, 0.52, and 0.58, but not 0.57. Moreutils is depended on by Mercurial and Mercurial is a build dependency of many things, so that one source breakage has broken a lot of things from building in the head of the NixOS 15.09 release channel branch. Best regards, James R. Haigh. -- Sent from Claws Mail on NixOS. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev