Re: [Nix-dev] unequal build hash
On Fri, Jul 07, 2017 at 07:59:46AM +0200, Vladimír Čunát wrote: > On 07/06/2017 07:35 PM, Harmen via nix-dev wrote: > > Does that makes sense? Did I forget a 'name' somewhere? > > When you use things like > src = ./.; > the directory gets copied into nix store and the resulting path's name > is based on the name of the directory. I see. That was rather unobvious and unexpected to me. Anyone knows a (creative) way around that? I would like to have it build code from ./., but where the name of the directory you happened to have checked the project out in doesn't matter (because then the binary cache works). Thanks, Harmen > > --Vladimir ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] unequal build hash
On Thu, Jul 06, 2017 at 06:35:15PM +0200, Harmen via nix-dev wrote:
> Hello all,
> another day, another problem :/
>
> I'm trying to figure out why a build generates different build IDs in
> different
> contexts.
> I have a nix expression and some code. If I copy that from machine A
> to machine B they both give the same build hash. All's fine there.
> If on machine B I run nix-build in a docker container it won't give the same
> build hash.
> `nix-hash .` gives the same hash for B and B-docker. Same nix version, same
> nixpkgs version.
I'm getting a bit further, and it worked on different machines because I
happened to name the directory the same on both machines. Seems like the name
of the directory is used in the build process. As long as I checkout the repo
in a subdir with the same name everything is fine. If I rename the checkout dir
nix will rebuild. This my current default.nix:
with pkgs; buildGoPackage rec {
name = "mypackage";
src = builtins.filterSource
(name: type:
(lib.hasPrefix (toString ./vendor) name) ||
(lib.hasPrefix (toString ./rzutil) name) ||
(lib.hasPrefix (toString ./util) name) ||
(lib.hasPrefix (toString ./datatype) name) ||
(lib.hasPrefix (toString ./one) name)
) ./.;
goPackagePath = "example.com/mypackage";
meta = with stdenv.lib; {
description = "My First Package";
homepage = https://github.com/example-com/mypackage;
platforms = platforms.unix;
};
}
If I do the checkout in /tmp/fooo part of the .drv gives:
...
["/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh","/nix/store/zw7ax2gxc8qr5cx5a7byjpxdshx73297-fooo"]
...
Does that makes sense? Did I forget a 'name' somewhere?
Thanks!
>
> Any hints what to look for?
> Thanks!
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] unequal build hash
Hello all, another day, another problem :/ I'm trying to figure out why a build generates different build IDs in different contexts. I have a nix expression and some code. If I copy that from machine A to machine B they both give the same build hash. All's fine there. If on machine B I run nix-build in a docker container it won't give the same build hash. `nix-hash .` gives the same hash for B and B-docker. Same nix version, same nixpkgs version. Any hints what to look for? Thanks! ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] nix-daemon and private git repos
On Tue, Jul 04, 2017 at 08:10:09PM +, zimbatm wrote: Thanks for the suggestions. I took away from this that it's best to not have Nix deal with the checkouts. For now I'll keep things as separate repos, to keep things easier. Eventually I would like to go to one pkgs tree for all repos, but I'll think about how to do that best. It also ties into integration and system testing, which is not just Nix related. thanks! Harmen ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] nix-daemon and private git repos
On Mon, Jul 03, 2017 at 03:27:34PM +0200, Tomas Hlavaty wrote: > Hi Harmen, > > On Mon 03 Jul 2017 at 15:19, Harmen <har...@lijzij.de> wrote: > > I can't be the first to want to use fetchgitPrivate with a sandboxed > > nix-daemon. Any experiences or tips? > > I had it working but there are several cases which needs extra setup > that I recommend to avoid fetchgitPrivate and use source variables, > e.g. src = ; This works the same in any context, e.g. in > hydra, without hydra, etc. You just need to set the proprietarySrc > accordingly. Thanks for the help Tomas, could you be a bit more specific what you mean? Thanks! Harmen ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] nix-daemon and private git repos
Hi all, I'm struggling to get fetchgitPrivate to work on nix-daemon installation (no NixOS, these are Ubuntu machines with nix). I can make it work on my dev machine, with is a non-daemon install, by setting NIX_PATH=ssh-config-file=/my/ssh/config:$NIX_PATH But that doesn't work in sandboxed daemon mode, because the nixbld* users can't read that file (both because of access rights, and because of the sandbox). Nix has this warning in fetchgitPrivate: > Note that the config file and any keys it points to must be readable > by the build user, which depending on your nix configuration means making it > readable by the build-users-group, the user of the running nix-daemon, or the > user calling the nix command which started the build. Similarly, if using an > ssh agent ssh-auth-sock must point to a socket the build user can access. > You may need StrictHostKeyChecking=no in the config file. Since ssh > will refuse to use a group-readable private key, if using build-users you will > likely want to use something like IdentityFile /some/directory/%u/key and have > a directory for each build user accessible to that user. from https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/fetchgit/private.nix which sounds reasonable, but it I don't see how to do that since I don't know exactly which user will run the build. Also because of the sandbox the build can't read the ssh config file at all. So next option is to generate the configfile in my expression, a-la https://www.mpscholten.de/nixos/2016/07/07/private-github-repositories-and-nixos.html but I don't know how to set nix.path from inside an expression. It would also require bundling the key with the expression, but if that works... I can't be the first to want to use fetchgitPrivate with a sandboxed nix-daemon. Any experiences or tips? Thanks! Harmen ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] drone and nix (sort of solved)
On Sun, Jun 18, 2017 at 05:46:03AM +, Wout Mertens wrote: > Nice! Why is it not perfect? Idealy I would not have to do the one-time setup on every machine. But it's good enough. > > On Fri, Jun 16, 2017 at 1:21 PM Harmen <har...@lijzij.de> wrote: > > > Hi all, > > > > I while ago here I asked about using nix with drone.io (a CI system which > > uses docker), to deal with push based tests. I previously reported back > > with a solution with gitlab, but I did end up with a workable (not perfect) > > solution for drone. > > > > On the 'drone agent' machine run this once: > > > > # docker volume create nix > > # docker run --rm -v nix:/newnix nixos/nix cp -a /nix/store /nix/var > > /newnix > > > > in the .drone.yml files: > > > > pipeline: > > build: > > image: nixos/nix > > commands: > > ... > > volumes: > > - nix:/nix/ > > > > And whenever you want to update nixchannel: > > > > # docker run -v nix:/nix/ --rm -ti nixos/nix nix-channel --update > > > > > > This will share the /nix directory with every build, so rebuild (and > > downloads!) are kept to a minimum. > > > > > > Maybe it helps someone. > > Thanks! > > Harmen > > > > ___ > > nix-dev mailing list > > nix-dev@lists.science.uu.nl > > https://mailman.science.uu.nl/mailman/listinfo/nix-dev > > ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] drone and nix (sort of solved)
Hi all, I while ago here I asked about using nix with drone.io (a CI system which uses docker), to deal with push based tests. I previously reported back with a solution with gitlab, but I did end up with a workable (not perfect) solution for drone. On the 'drone agent' machine run this once: # docker volume create nix # docker run --rm -v nix:/newnix nixos/nix cp -a /nix/store /nix/var /newnix in the .drone.yml files: pipeline: build: image: nixos/nix commands: ... volumes: - nix:/nix/ And whenever you want to update nixchannel: # docker run -v nix:/nix/ --rm -ti nixos/nix nix-channel --update This will share the /nix directory with every build, so rebuild (and downloads!) are kept to a minimum. Maybe it helps someone. Thanks! Harmen ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Node example?
On Wed, May 31, 2017 at 08:29:03AM +, Benno Fünfstück wrote: > Hi Harmen, > > the reason it works on your test machine but doesn't work on CI is probably > because of build sandboxing (see `build-use-sandbox` nix option: that > disables network access & access to certain FS paths for nix builders). That fixed it! I'm still not happy with the nix expression, but at least I have something to work with now. Thanks! ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Node example?
(I got a reply offlist, I'll just answer here hoping the sender doesn't mind) node2nix works fine, but I do indeed need to build some frontend, using webpack. Currently I have two errors: Npm can't download the precompiled node-webpack binary, but rebuilds it from source every time I run nix. Downloading works fine on my laptop, but not on my CI machine (gitlab builder, nix-daemon). The error is 'getaddrinfo ENOTFOUND github.com github.com:443'. Should not be fatal (npm will build it from source), but looks fishy to me. The other, final, error is: sh: /nix/store/qvrv30xb597zr1ryw9d3n700pywxz3q9-node-my-dashboard-0.0.1/lib/node_modules/my-dashboard/node_modules/.bin/webpack: /usr/bin/env: bad interpreter: No such file or directory which, again, works fine on my test machine(s). Same nix version, same unstable nixpkgs nix-channel version. Thanks! ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Node example?
Hi all, does anyone have an example of a nix expression to build a (react based) nodejs project? I'm having some problems with writing the nix expression and can't really formulate a proper question, but maybe some good example will help me on my way. Ideally something which uses webpack to build static html (node is only used at build time to generate the static files, it's not needed at runtime). Thanks! Harmen ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] nix rebuilds
On Fri, May 12, 2017 at 01:31:00PM +0100, Kevin Cox wrote: > On 05/12/2017 01:23 PM, Harmen wrote: > > I'm building a few ruby and node projects with nix. It works well enough, > > but > > nix does too many rebuilds, also when nothing changed. The question is: how > > I > > can figure out why it does a rebuild? Maybe some checksum has changed, but > > can > > I somehow figure out which one? > I suspect that at a bare minimum the ./result symlink has changed. > > What I tend to do is use builtins.filterSource in order to ensure I am only > including paths that I need. > > You can see an example at > https://github.com/kevincox/sog/blob/master/default.nix > > src = builtins.filterSource (name: type: > (lib.hasPrefix (toString ./BUILD) name) || > (lib.hasPrefix (toString ./sog) name) || > (lib.hasPrefix (toString ./test) name) || > (lib.hasPrefix (toString ./tools) name) || > (lib.hasPrefix (toString ./WORKSPACE) name)) ./.; Works great, thanks! ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] github triggered builds
On Tue, May 09, 2017 at 10:44:53PM +0200, Harmen wrote: > On Tue, May 09, 2017 at 01:08:08PM +0200, Maarten Hoogendoorn wrote: > > Unfortunately this does not really help with sharing derivations that have > > been built before. > > Thanks for all the replies. Seems like I didn't miss anything obvious. > > My current plan is to try gitlab with my own runner, which has a 'shell > executor' option. > https://docs.gitlab.com/runner/executors/shell.html > That looks like to be exactly what I need for nix based builds. I tried it and it works rather well. All the console output from the build ends up on gitlab, easily visible in the merge request. As long as there are only pure integration tests involved (no databases) this seems a nice and easy to set up solution for simple CI on private git repos. Thanks for the suggestions. Harmen ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] nix rebuilds
I'm building a few ruby and node projects with nix. It works well enough, but nix does too many rebuilds, also when nothing changed. The question is: how I can figure out why it does a rebuild? Maybe some checksum has changed, but can I somehow figure out which one? I put the .nix file in the repo next to the code, so I don't refer to an external repo with a nice checksum, but like this: src = ./.; This is for a Ruby on Rails project build with bundix. Any thoughts? Thanks! Harmen ___ nix-dev mailing list nix-dev@lists.science.uu.nl https://mailman.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] github triggered builds
On Tue, May 09, 2017 at 01:08:08PM +0200, Maarten Hoogendoorn wrote:
> Unfortunately this does not really help with sharing derivations that have
> been built before.
Thanks for all the replies. Seems like I didn't miss anything obvious.
My current plan is to try gitlab with my own runner, which has a 'shell
executor' option.
https://docs.gitlab.com/runner/executors/shell.html
That looks like to be exactly what I need for nix based builds.
Is this use-case of nix so uncommon (push-triggered builds)? Do most people go
for Hydra?
Thanks!
Harmen
>
> 2017-05-09 10:02 GMT+02:00 zimbatm <zimb...@zimbatm.com>:
>
> > Travis CI also has support for nix builds and might be easier to setup.
> >
> > On Mon, 8 May 2017, 18:17 Tomasz Czyż, <tomasz.c...@gmail.com> wrote:
> >
> >> https://nixos.org/hydra/
> >>
> >> and
> >>
> >> https://github.com/hercules-ci/hercules ( looks like still in heavy
> >> development but maybe usable :))
> >>
> >> 2017-05-08 18:14 GMT+01:00 Harmen <har...@lijzij.de>:
> >>
> >>> Hi,
> >>>
> >>> I'm trying to see how I can make my build processes easier with nix. So
> >>> far
> >>> it's going pretty good and it's fun, although there was a lot of
> >>> searching
> >>> online for scattered documents.
> >>>
> >>> Want I want to do (as the first thing to change to nix in production) is
> >>> to
> >>> port the building of some docker images I use for testing. The idea is to
> >>> have docker images build, tagged with their branch they come from, when
> >>> someone
> >>> pushes something. The building and pushing an sich work. The .nix files
> >>> live in
> >>> the repo, and with a `make docker` the image is build and uploaded. I'm
> >>> very
> >>> happy to be able to build docker images without actually having to use
> >>> docker
> >>> ;)
> >>>
> >>> So, what would be the recommended way to trigger the building process?
> >>> I'm
> >>> currently using drone.io, but that works with containers. It works with
> >>> nix,
> >>> when I give it the nixos/nix docker image, but building a node project
> >>> takes
> >>> about 5 minutes, and drags in way too much from cache.nixos.org. I
> >>> tried to
> >>> have it make a local nix binary-cache, but there are some problems
> >>> there, but
> >>> drone also just doesn't fit the problem nicely. Nix solves the problem
> >>> of
> >>> versioning so much nicer than containers that I would prefer to use
> >>> something
> >>> simpler. Hydra could work, but I'm a bit intimidated by that, and would
> >>> like to
> >>> have something simpler for now.
> >>>
> >>> The LT;DR: question: is there a simple nix based build system which can
> >>> be
> >>> triggered via git{hub,lab} hooks?
> >>>
> >>>
> >>> Thanks!
> >>> Harmen
> >>> (If there is a better place to ask this, let me know)
> >>> ___
> >>> nix-dev mailing list
> >>> nix-dev@lists.science.uu.nl
> >>> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
> >>>
> >>
> >>
> >>
> >> --
> >> Tomasz Czyż
> >> ___
> >> nix-dev mailing list
> >> nix-dev@lists.science.uu.nl
> >> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
> >>
> >
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > https://mailman.science.uu.nl/mailman/listinfo/nix-dev
> >
> >
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] github triggered builds
Hi,
I'm trying to see how I can make my build processes easier with nix. So far
it's going pretty good and it's fun, although there was a lot of searching
online for scattered documents.
Want I want to do (as the first thing to change to nix in production) is to
port the building of some docker images I use for testing. The idea is to
have docker images build, tagged with their branch they come from, when someone
pushes something. The building and pushing an sich work. The .nix files live in
the repo, and with a `make docker` the image is build and uploaded. I'm very
happy to be able to build docker images without actually having to use docker
;)
So, what would be the recommended way to trigger the building process? I'm
currently using drone.io, but that works with containers. It works with nix,
when I give it the nixos/nix docker image, but building a node project takes
about 5 minutes, and drags in way too much from cache.nixos.org. I tried to
have it make a local nix binary-cache, but there are some problems there, but
drone also just doesn't fit the problem nicely. Nix solves the problem of
versioning so much nicer than containers that I would prefer to use something
simpler. Hydra could work, but I'm a bit intimidated by that, and would like to
have something simpler for now.
The LT;DR: question: is there a simple nix based build system which can be
triggered via git{hub,lab} hooks?
Thanks!
Harmen
(If there is a better place to ask this, let me know)
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev
