Re: [Nix-dev] "IO" in Nix

[Ericson, John]

Not sure how relevant this is, but sometimes I think about a sort of
"quoting monad" where Q a = a derivation producing a nix expression with
"type" a. import-with-derivation would form the crux of the bind operation
of the monad.

On Tue, May 24, 2016 at 3:52 PM, Rickard Nilsson 
wrote:

> Hi Dan,
>
> On 05/23/2016 05:30 PM, Daniel Peebles wrote:
> > Hi all,
> >
> > I've been experimenting with a pattern recently that I'd say is fairly
> > close to a Haskell notion of IO. I have a collection of Nix-centered
> > scripts that need to perform fairly restricted side effects:
> >
> >   * Build an AMI (image) for AWS
> >   * Write some disk image stuff to a raw device
> >   * Run some tests that talk to the outside world
> >   * Build some stuff that depends in a fairly constrained way on mutable
> > external state (e.g., RPM update repos, as opposed to the
> > deterministic base repos)
> >   * Deploy stuff
> >
> > These things all have the following in common:
> >
> >   * I build as much stuff possible in "pure" Nix
> >   * I write out scripts (in the store) that all start with a
> > ${stdenv.shell} shebang and an explicit `export PATH` clause to
> > avoid inheriting an environment from the outside.
> >   * The scripts rarely take arguments at runtime; instead I run them
> > with similarly to `sudo $(nix-build -A foobar)` or sometimes without
> > `sudo`.
> >
> > I'm wondering if anyone else is doing something similar. It's basically
> > (right now) analogous to a non-composable IO value in Haskell: pure code
> > producing a "script" for some external impure interpreter to execute.
> >
> > I can think of a few next steps from here: start building composition
> > operators (like bind) to chain together these impure actions without
> > wanting to pull your hair out. I think Shea Levy's nix-exec already has
> > a monad-flavored API for IO but I haven't seen anything like it outside
> > of nix-exec.
>
> I'm interested in hearing a bit more about the composition operators you
> envision. I do a lot of shell script building like you describe above,
> and I tend to compose them by building other shell scripts. I assume
> that is what you mean by pulling your hair out?
>
> Sometimes I also do composition by building Makefiles, which is kind of
> nice when you want to assemble many tasks that could possibly run in
> parallel.
>
> To have useful composition operators, wouldn't you need more precise
> types for your IO actions?
>
>
> Regards,
>Rickard
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Ericson, John]

>
> If https://github.com/NixOS/hydra/pull/277 is merged, then there is one
> step left to get automatic PR management: An input plugin which fetches
> information about open PRs on a repo. I'll write it if 277 ever gets in,
> but without 277 or something like it there's no point.
>

Well, it go merged! I don't actually know anything about the relevant APIs,
but shouldn't github be able to push this information to hydra? I'd think
notifying on newly-opened PRs and changes to master would be sufficient.

On Thu, Apr 28, 2016 at 3:42 AM, Domen Kožar <do...@dev.si> wrote:

> That's a good point, but I don't see how a separate tab would contrain
> hydra from adding extra input parameters to the eval. It's just a perl
> "controller" for UI separation.
>
> /cc aszlig
>
> On Thu, Apr 28, 2016 at 11:37 AM, <s...@shealevy.com> wrote:
>
>> IMO putting the logic completely into hydra will be too limiting. If
>> hydra just gives a nix expression some info about the PRs, then the nix
>> expression can do arbitrary things (e.g. only build PRs from trusted users,
>> or that don't change stdenv, or whatever). If it's a jobset tab then you
>> have to add every possible variation into the hydra UI.
>>
>>
>>
>> - Original Message -
>> From:
>> "Domen Kožar" <do...@dev.si>
>>
>> To:
>> "Shea Levy" <s...@shealevy.com>
>> Cc:
>> "Graham Christensen" <gra...@grahamc.com>, "Ericson John" <
>> john_eric...@brown.edu>, "nix-dev" <nix-dev@lists.science.uu.nl>
>> Sent:
>> Thu, 28 Apr 2016 11:32:53 +0100
>>
>> Subject:
>> Re: [Nix-dev] Continuous Integration
>>
>>
>> I was talking to Aszlig about this and it would be best if PRs would be a
>> tab on the jobset. Since jobset defines the inputs, it would be tested for
>> PRs against that specific branch.
>>
>> On Thu, Apr 28, 2016 at 11:30 AM, <s...@shealevy.com> wrote:
>>
>>> It can set pull request statuses (e.g. see
>>> https://github.com/shlevy/hydra-github-status-test/pull/3), but
>>> currently you have to manually create the jobset corresponding to the PR
>>> and ensure that the relevant jobs are captured by the status plugin.
>>>
>>> If https://github.com/NixOS/hydra/pull/277 is merged, then there is one
>>> step left to get automatic PR management: An input plugin which fetches
>>> information about open PRs on a repo. I'll write it if 277 ever gets in,
>>> but without 277 or something like it there's no point.
>>>
>>>
>>> - Original Message -
>>> From:
>>> "Graham Christensen" <gra...@grahamc.com>
>>>
>>> To:
>>> "Domen Kožar" <do...@dev.si>
>>> Cc:
>>> "Ericson John" <john_eric...@brown.edu>, "nix-dev" <
>>> nix-dev@lists.science.uu.nl>
>>> Sent:
>>> Wed, 27 Apr 2016 18:23:55 -0500
>>> Subject:
>>> Re: [Nix-dev] Continuous Integration
>>>
>>>
>>>
>>>
>>>
>>> Domen Kožar <do...@dev.si> writes:
>>>
>>> > It doesn't build PRs, just updates the status.
>>> >
>>>
>>> Does this mean it can build specific branches (master, release-16.04..)
>>> and set the commit status on those commits? ie: doesn't have anything to
>>> do with pull request statuses?
>>>
>>> Best,
>>> Graham
>>> ___
>>> nix-dev mailing list
>>> nix-dev@lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>
>>>
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] all-packages.nix and the unoverridable self.

[Ericson, John]

Nicolas intentionally kept `self` for such aliases as these. See the
security updates PR (https://github.com/NixOS/nixpkgs/pull/10851) for the
reasons why.

On Sun, May 8, 2016 at 10:44 AM,  wrote:

> I haven't heard back from Nicolas yet.
>
> Perhaps I should put together a PR that removes the instances of self that
> I think are wrong so that we can discuss the issue.
>
>
> On Sun, 8 May 2016, Daniel Peebles wrote:
>
> Did you figure this out? I'm curious whether something got messed up in
>> the recent changes.
>>
>> On Mon, May 2, 2016 at 2:23 AM,  wrote:
>>   Something seems wrong / bizzare with the recursion in
>> all-packages.nix
>>
>>   I've been trying to override linux_4_4 in my configuration.nix with
>> the
>>   following:
>>
>>nixpkgs.config.packageOverrides = super:
>> { linux_4_4 = super.linux_4_4.override { extraConfig =
>> "CHROME_PLATFORMS y";
>>  kernelPatches = [ {
>> name = "f10_sysrq"; patch = ./f10_sysrq.patch; } ]; };
>> };
>>
>>   but this no longer works.  I've spent a few hours studying the
>> issue.
>>   If I do the following chain of overrides upto the linuxPackages
>>   attribute
>>
>>nixpkgs.config.packageOverrides = super: rec
>> { linux_4_4 = super.linux_4_4.override { extraConfig =
>> "CHROME_PLATFORMS y";
>>  kernelPatches = [ {
>> name = "f10_sysrq"; patch = ./f10_sysrq.patch; } ]; };
>>   linuxPackages_4_4 = super.recurseIntoAttrs
>> (super.linuxPackagesFor
>>   linux_4_4 linuxPackages_4_4);
>>   linuxPackages = linuxPackages_4_4;
>>   linux = linuxPackages.kernel;
>> };
>>
>>   then it does work.
>>
>>   I couldn't for the life of me understand why copying what is
>> essentially
>>   the exact definitions of linuxPackages_4_4 and linuxPackages into my
>>   packageOverrides caused it to work.  The whole point of the
>>   packageOverride mechanism is to invoke late-binding so that I don't
>> have
>>   to override long chains.
>>
>>   I traced the issue to the following strange set of definitions:
>>
>>   all-packages.nix begins with something like this
>>
>>   { ... }:
>>   self: pkgs:
>>
>>   with pkgs;
>>
>>   { ... }
>>
>>   It is a function of three arguments, (1) a set of parameters, (2) a
>>   binding for self, (3) a binding for pkgs, and the with pkgs; bring
>> all
>>   the definitions from pkgs into scope.
>>
>>   This is called from top-level/default.nix with the following
>>   expression:
>>
>>   allPackages = self: super:
>> let res = import ./all-packages.nix allPackagesArgs res
>> self;
>> in res;
>>
>>   So allPackageArgs contains the parameters, self gets bound to res,
>> and
>>   pkgs get bound to self.
>>
>>   The upshot of this is that within all-packages.nix self (which is
>> bound
>>   to res) is the result of only evaluating all-packanges *with no
>>   overrides* while pkgs (which is bound to self) ends up late-bound
>> and is
>>   the set of packges *with all overrides*
>>
>>   So when linux and linuxPackages get bound using self in
>> all-packages:
>>
>>   # The current default kernel / kernel modules.
>>   linuxPackages = self.linuxPackages_4_4;
>>   linux = self.linuxPackages.kernel;
>>
>>   The use of self here (and throughout the linuxPackage definitions)
>> means
>>   that we are making reference to the *unoverridden package set*.
>> This is
>>   why my packageOverrides of linux_4_4 did nothing, because the
>> references
>>   to linux_4_4 inn all-packages.nix are prefixed with "self." which
>> means
>>   it always gets the unoverriden packages.
>>
>>   Is this really the desired behaviour?  I think that
>> all-packages.nix is
>>   full of many questionable uses of the "self." prefix.  I suspect
>> that
>>   people don't really understand that "self." means "give me the
>>   unoverriden version of packages".  I think renaming "self" in
>>   all-packages.nix to "unoverridenPackages" would be a better name.
>>
>>   --
>>   Russell O'Connor  <
>> http://r6.ca/>
>>   ``All talk about `theft,''' the general counsel of the American
>> Graphophone
>>   Company wrote, ``is the merest claptrap, for there exists no
>> property in
>>   ideas musical, literary or artistic, except as defined by statute.''
>>   ___
>>   nix-dev mailing list
>>   nix-dev@lists.science.uu.nl
>>   http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
>>
>>
>>
> --
> Russell O'Connor  
> ``All talk about `theft,''' the general 

Re: [Nix-dev] Changing how we compose all-packages and stdenvs

[Ericson, John]

Understood, and no offense taken :). Also, If it is more convenient to just
review the first N commits, the PR is easy to break up and merge in pieces.

John

On Sat, May 7, 2016 at 1:59 PM, Vladimír Čunát <vcu...@gmail.com> wrote:

> On 05/07/2016 10:52 PM, Ericson, John wrote:
> > I have a PR[1] at that makes simplifies things [...]
>
> I've had it in my queue for some time, but it's not easy to
> (immediately) find enough time to review such large things...
>
> --Vladimir
>
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Changing how we compose all-packages and stdenvs

[Ericson, John]

I have a PR[1] at that makes simplifies things by plunging in stdenvs to
top-level rather than calling back and forth. Additionally @Mathnerd314 has
a few extra commits on top, that look good to me. [See the PR description
and comment thread for a clarification on all of that.]

The PR hasn't received any attention in the last week or so, and
furthermore as a fairly fundamental (if invisible) change to the way
nixpkgs works, I think it should get as many eyes as possible, especially
from those that maintain the stdenvs today. Hence I am emailing the list.

Thanks,

John

[1]: https://github.com/NixOS/nixpkgs/pull/15043
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Stripped down Linux distribution based on Nix/NixOS

[Ericson, John]

Glad to here you switched to NixOS!

For your hydra checksum concerns see https://github.com/NixOS/hydra/pull/277
--- in short it would be nice if we had "non-deterministic derivations" for
this, but in the Eelco has just added a `fetchGit` builtin in the meantime.

I hope NixOS can somebody impose no policy and the user other than using
Nix, but rather just provide support code for every conceivable setup, so
please I hope you continue using it rather than rolling your own with Nix!
Interesting use-cases are exactly needed to pull NixOS in this direction.

On Wed, May 4, 2016 at 8:23 AM, Nahum Shalman  wrote:

> On Wed, May 4, 2016 at 10:25 AM,  wrote:
>
>> On Tuesday, May 03, 2016 11:31:37 Nahum Shalman wrote:
>>
>> > I think the two most critical areas for us to work on next are:
>>
>> > 1. Shipping a kernel that enables selinux rather than apparmor. Any
>>
>> > suggestions about how to do this?
>>
>>
>>
>> I'm the person who effectively made the choice in favor of apparmor.
>> "Enabling selinux" is trivial in the sense of turning on the feature in the
>> kernel.
>>
>
> Sadly that's proving tricky to me and making me feel rather clumsy... My
> cerana-test5 branch attempts to do that but fails. I can't yet figure out
> why it's remaining disabled in spite of my changes.
>
>
>> Shipping a working and useful policy would be hard.
>>
>
> For the Cerana project it's not so bad as the Nix store will be very small
> and limited in how much it's doing so the policy for the core system will
> be very limited in scope. For the rest of the software running on the
> system we will have tools automatically generating the appropriate policies
> for sotware that will be living in the ZFS pool rather than in the Nix
> store.
>
>
>> A relevant discussion:
>>
>> http://lists.science.uu.nl/pipermail/nix-dev/2013-May/011091.html
>>
>
> I am by no means suggesting that NixOS should switch to selinux. Just that
> I want my downstream project to be able to use it.
>
> Thanks!
> -Nahum
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Incremental recompilation

[Ericson, John]

Nicolas is correct. One complication however one is that one has to be
careful when files contain relative paths. The cleanest treatment of the
recursive hashing would normalize any nix store path within the source
location to hash of just that subtree, but that breaks the relative paths.
Note that its not always possible to resolve all relative paths while
hashing either, as they may form cycles.

Back to Volkhov's original point, I think that compilers should work like
Nix itself -- explicit graph of computation all data is content-addressed
and all function application is cached on the hashes of the arguments (and
function itself). Its a lot of work to refactor compilers to work this way
of course, but practicalities aside I think that's the cleanest way to go.

To cache across derivations, let's say, the compiler and Nix should be able
to use the same content-address store---that's half the battle. The other
half is the compiler (also like Nix) needs to map function applications to
their evaluations. As I said above the arguments are already
content-addressed, but the functions need to be given some sort of abstract
names. Perhaps indices prefixed with the derivation path would do, and then
Nix and the compiler can share the same store for their evaluation maps too.

As it turns out, Rust is exploring incremental compilation
https://github.com/rust-lang/rfcs/blob/master/text/1298-incremental-compilation.md
and their plan is, well, kinda a first approximation of the above. I'd be
really interested in trying to put this plan into action with rustc---this
is our best shot given the feasibility of refactoring GCC, LLVM, GHC, etc
on our own.

John


On Sat, Apr 30, 2016 at 8:39 AM, Nicolas Pierron <
nicolas.b.pier...@gmail.com> wrote:

> On Fri, Apr 29, 2016 at 7:56 PM, stewart mackenzie 
> wrote:
> > IMHO, incremental compilation is a strong boon for nix adoption. It is
> the
> > only serious issue I can see which prevents Nix being a Make replacement.
>
> I agree with you on this point, Nix has the potential of been a better
> ccache & distcc in one shot.
>
> Note, this is not the same issue as mention in the Volkhov email.  The
> email suggest caching artifacts of the compilation spawn by Nix and
> independently of the build system, while doing a make replacement
> involves replacing/wrapping existing build system and using Nix for
> caching artifacts of the compilation spawned in user-land.  But, I
> guess the second problem's solution could be a mean for solving the
> first problem.
>
> Also, I admit I thought on how to solve this issue in the first place,
> but the main problem I got is not even building, this is only about
> making a copy of the source directory into the nix store.
> When your project sources are larger than 1 GB (without the VCS), you
> don't want to be reading all the bytes before being able to start any
> compilation.
>
> Thus, none of the approaches proposed before are practical for larger
> projects.
> So, how can we make it such that it will scale?
>
> The only approach I can think of would be to:
>  - Upload each individual file to the Nix store, when they would be needed.
>  - Make a symlink fram to re-build fake source directories.
>  - Use hash-by-content to prevent recompilation.  (when using  "gcc
> -M"  to filter out directory content, you want to make sure you get
> the same set of headers, and only cause a recompilation if some of the
> header changed)
>
> --
> Nicolas Pierron
> http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Ericson, John]

Support for hydra to build github PR has been added for a few weeks now in
https://github.com/NixOS/hydra/pull/280 . What's the next step for actually
using this with nixpkgs and hydra.nixos.org?

On Tue, Feb 23, 2016 at 4:46 PM, Ericson, John <john_eric...@brown.edu>
wrote:

> > S3 and Hydra PR support
>
> I asked Domen earlier on IRC, but in case anyone else is interested,
> is there any way to contribute to these? If this will go most the way
> towards continuously updating channel(s) I'll give an arm and a leg.
>
> > IMHO it's very often a fault of some builds or tests failing (sometimes
> > transiently). > IMHO it's very often a fault of some builds or tests
> failing (sometimes
> > transiently).
>
> Well my proposal very much does matter if honestly-failing
> builds/tests are a problem---let's not have one bad commit hold up
> good commits. If most of the failures are transitive however, well
> that's very unfortunate. I hear bad things about timeouts but would
> love more details on the transitive errors we face and what can be
> done about them.
>
> John
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] haskell structure for all of nixpkgs

[Ericson, John]

Well, besides working on other langauges' integration, I think there are
some Petisms™ that all of nixpkgs could benefit from. A good example is
pervasive use of `self: super: ...` and fix+extend rather than ad-hoc hacks
and the scary reflection used in `lib/customization.nix`.

I'd say https://github.com/NixOS/nixpkgs/pull/14000 was the first big step
in this direction, and hopefully
https://github.com/NixOS/nixpkgs/issues/10874 will lead to the second.

On Mon, Apr 25, 2016 at 10:26 AM, stewart mackenzie 
wrote:

> Okay Domen's a +1, maybe the guys and girls who implemented haskell
> like PL level package systems could weigh in with insight. For example
> ICIUC, Erlang packages adopts the same approach. The gained knowledge
> could be helpful to start with this document.
>
> Could someone with experience please write, in this thread, a few
> words about this approach? Keep it small and simple.
>
> Hopefully this gets more people interested by understanding it. Domen
> says it's advanced and simple... why is it advanced and simple?
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Did we just get windows support for free?

[Ericson, John]

Yeah this mode seems to be pretty cordoned off. For example, I'm not sure
that Linux processes will be able spawn windows executables. I'd say it
will be a useful to use a Linux binary cache on windows, but not at all a
replacement for a msys/cygwin-based windows port.

On Wed, Mar 30, 2016 at 5:42 PM, Mathnerd314 
wrote:

> The talk is finished. There's also a pre-recorded demo:
> https://channel9.msdn.com/Events/Build/2016/P488
>
> Apparently the files are stored in a hidden user directory as normal NTFS
> files. It's still unclear why the filesystem appears as read-only, but
> apt-get does work, so I imagine Nix will work fine too, unless it uses one
> of the unimplemented syscalls. it's not going to be available for another
> 1-2 weeks, and then only to Windows Insiders (but it's free to join that
> program).
>
> I don't think Nix should drop MinGW or Cygwin support though (what little
> works), since it will still be useful for compiling GUI things (e.g. SDL
> games).
>
> -- Mathnerd314
>
> On Wed, Mar 30, 2016 at 1:01 PM, Mathnerd314 
> wrote:
>
>> The talk was scheduled before today; I think it's too early for April.
>>
>> https://channel9.msdn.com/Events/Build/2016/C906
>>
>> What's not clear is how the linux files are stored; it looks like the
>> root filesystem is read-only in Dustin Kirkland's post, which would
>> preclude using a package manager.
>>
>> -- Mathnerd314
>>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Commit access

[Ericson, John]

> This is kind of where things are at with hydra channels.

Hydra currently *polls* master periodically, whereas with this github
*pushes* to hydra/travis/etc, and they git push back to master if the PR
passes. This distinction is really important if we want to never break
master.

The catch is that it would effectively require serializing the merge
> queue's test runs...


But most PRs are independent/commute, and we cache well enough to take
advantage of that, so I don't think we will need any special tricks here.

Anyways I started a thread on exactly this and more
https://www.mail-archive.com/nix-dev@lists.science.uu.nl/msg18287.html
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Commit access

[Ericson, John]

Another -1 for multiple repos. We've been fairly good at abstraction and
refactoring, and this would serious impair our abilities with regard to
both.

On Mon, Feb 29, 2016 at 10:02 AM, stewart mackenzie 
wrote:

> Clear support for Layus' points.
>
> I'll mention another point more clearly.
>
> Eschew hierarchy & consensus
>
> Adopt the Advice Process:
> Before making a decision seek advice from expertise, then seek advice
> from those you'll affect. Neither group can veto your decision.
> Maintainers who do not do this or are unwilling to do this are loose
> cannons and are removed. The bar to becoming a maintainer: 1 PR.
> Evidence will amount to see whether the advice process is used or not.
>
> It's a good thing more people feel part of the community. Remove the
> loose cannons, the code base then stabilizes greatly.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Continuous Integration

[Ericson, John]

> S3 and Hydra PR support

I asked Domen earlier on IRC, but in case anyone else is interested,
is there any way to contribute to these? If this will go most the way
towards continuously updating channel(s) I'll give an arm and a leg.

> IMHO it's very often a fault of some builds or tests failing (sometimes
> transiently). > IMHO it's very often a fault of some builds or tests failing 
> (sometimes
> transiently).

Well my proposal very much does matter if honestly-failing
builds/tests are a problem---let's not have one bad commit hold up
good commits. If most of the failures are transitive however, well
that's very unfortunate. I hear bad things about timeouts but would
love more details on the transitive errors we face and what can be
done about them.

John
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] On nixpkgs and reasonable code size

[Ericson, John]

And now this leads into my thread
https://www.mail-archive.com/nix-dev@lists.science.uu.nl/msg18287.html
:).

I'm totally for that, but do note that there is an equivalent problem
in deciding who gets "PR approval" access. But, I suppose since we
control hydra/whatever and not github, we can roll our own logic like
allowing people to only approve PRs for automatic merging that affect
certain files / subtrees.

On Tue, Feb 23, 2016 at 7:12 AM, Matthias Beyer  wrote:
> On 23-02-2016 16:08:37, Matthias Beyer wrote:
>> On 21-02-2016 15:28:08, Bjørn Forsman wrote:
>> > On 21 February 2016 at 15:17, zimbatm  wrote:
>> > > tl,td; I think that we should split nixpkgs/pkgs in two
>> >
>> > Another way to do it is the Linux kernel way. Instead of splitting the
>> > (git) repository in two (or more) pieces, split _maintenance
>> > responsibility_ into a hierarchy. This is opposite to the flat
>> > responsibility model NixOS development use today.
>>
>> I completely second this. The problem is IMHO _not_ that the repo gets big
>> (there are other repos which are way, way bigger than nixpkgs) but the
>> development model. AFAIK I said that before on this list. The problem is that
>> everyone who wants to be a contributer gets push access to master. It just
>> screams at you "I won't scale"!
>>
>
> Let me add: In my opinion, nobody should be able to push to master. Merging 
> into
> master should be done either automatically (by a build-bot or something like
> this) or by a "merge this PR"-click in github, after some checks are succeeded
> (something like CI or lgtm.co (not proposing tools here but functionality)).
>
>
> --
> Mit freundlichen Grüßen,
> Kind regards,
> Matthias Beyer
>
> Proudly sent with mutt.
> Happily signed with gnupg.
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Continuous Integration

[Ericson, John]

As everyone knows, while updates to nixpkgs roll in 'round the clock,
the unstable channels are usually stuck for days or weeks before
lurching forward. In other words, we don't do continuous integration.
This does not seem sustainable as nixpkgs grows larger and changes
more rapidly.

When I asked about this on IRC previously, I was told that the problem
was intractable while we only had one mac mini, but now we have more
Darwin build machines, and I believe the Darwin nixpkgs infrastructure
has also matured leading to less mass-rebuilds. Thus, I hope there is
no longer a fatal imbalance between our Linux and Darwin needs and
capabilities.

So perhaps now is good time to consider: can we commit to ensuring
that nixos-unsable == nixpkgs-unstable == master? In particular this
means ensuring that every PR passes before merging, and rebuilding
previously-passing PRs whenever master changes. With most build
systems, this would unleash a thundering heard of rebuilding PRs every
time master changes, but since we cache very well and most PRs
commute, I believe this won't be a problem. Mass-rebuilds of course
violate this, but they are a bottleneck no matter what we do.

Note that Travis doesn't rebuild previously-passing PRs. Also, we'd
probably want some way to bless a bunch of PRs so they are
automatically merged as soon as their builds pass. I'd love to
integrate Hydra with github directly, but in the meantime the Rust
community wrote https://github.com/barosl/homu to address these
issues, and it might be worth looking at. [I follow Rust development
so I've seen it in action. But I haven't used it myself.]

What I think this boils down to is the difference between between
master and the *-unstable channels branches dictating hydra's queue,
and the set of open PRs doing so. The order in which PRs get merged is
mostly-arbitrary, and thus the linearity that imposes is mostly
worthless: Hydra must sign off on one commit before moving on to the
next even if the failing commit doesn't interact with the succeeding
commits at all.

Finally, there is more that I don't know than I do know about hydra
and the status quo, so I hope above all else this kicks off a good
discussion.

John
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] A call for hardware configurations

[Ericson, John]

This is a great idea, and I'll be happy to contribute to it! I hope
this could really be a killer draw for new users, maybe even those new
to Linux. While the NixOS installation procedure isn't so fancy, being
reliable over a GUI install that's easy until something hard to debug
goes wrong give us some advantage. Somethings things to think about:

1. It might be good to use `findFirst` to select from stable/default
and unstable/bespoke arguments. For example on my laptop I am
currently using `linuxPackages_latest` because 4.4 seems to be
necessary, and I hope the next version will be fix more things. But
rather than bake in `linuxPackage_4_4`, or `linuxPackages_latest`, I'd
think the most robust thing would be something like:

  findFirst
(p: builtins.compareVersions p.version "4.4" >= 0)
(assert false "no adiquate kernel avilable"; null)
[ linuxPackages linuxPackages_4_4 linuxPackages_latest ]

That way we only break from the default/stable when necessary.

2. Beware of caching generated results. Already I have a branch of
just stock-generated configuration.nix I merge my config team to time,
lest some default I kept around is no longer considered best practice.
Likewise I try to see what I can do so the generated
hardware-configuration.nix is valid, and I now don't even both to
stick it in version control as I've had good luck with it on multiple
machines. There's a tension between wanting to really freeze a good
configuration in place (the logical extreme would also include pinning
a version of nixpkgs), and the maintenance burden of little-used
configs assuming the generated defaults that best/apply to older
nixpkgs. I know we try to be backwards compatible with nixos settings,
and that's great, but I'd feel bad making the need for backwards
comparability greater. Not really sure what the ideal solution is.

3. For PCs, many things are contingent on firmware settings which
unfortunately we do not (can not?) control. A convention for guides on
how to setup ones firmware before installing nix would be nice.

On Mon, Feb 8, 2016 at 3:50 AM, Emery Hemingway  wrote:
> Garbas and I were talking about hardware at the last Berlin Nix meetup and
> the idea came up that the community might be ready for a collection of
> model specific hardware configurations. Git repositories are vastly easier
> to manage than wikis, so I went and created one.
>
> https://github.com/ehmry/nixos-hardware
>
> This will just be a collection of standalone modules to import into
> configuration.nix, and is not intended to merge with nixpkgs. Suggestions
> are solicited for shipping the configurations to machines. Importing by
> a channel and <...> would be nice, but I don't know how to do it.
>
> I'm not a hardware guru, but I'm willing to handle pull requests.
>
> Emery
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] dist-upgrade? define channel declaratively?

[Ericson, John]

Hmm, I think we should strive to keep nix as minimal as possible. This is
definitely the killer app for the builtin.fetch*.

On Fri, Jan 15, 2016 at 8:51 AM, stewart mackenzie 
wrote:

> On 15 Jan 2016 19:17, "Eelco Dolstra"  wrote:
> > Another possibility would be to build the module system into Nix, so
> module
> > evaluation wouldn't depend on Nixpkgs.
>
> this seems to be a better longterm solution.
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Non-root access to nixos-container

[Ericson, John]

The `--private-users=` flag to systemd-nspawn will create a container with
its own user namespace. In principle, this should allow creating systemd
container by non-root. That in turn would alleviates the security problems
(it's a far more grievous bug for a normal to gain root powers absent an
screw-ups on root's powers than root to accidentally back-door itself). See
the "Fully Unprivileged Container Payload" section at
https://wiki.freedesktop.org/www/Software/systemd/ContainerInterface/

Maybe we should look into (optionally) using `--private-users=` with
nixos-container?

On Wed, Dec 16, 2015 at 4:33 PM,  wrote:

> On Wed, Dec 16, 2015, at 08:46 PM, rohit yadav wrote:
> > Hi,
>
> Hi,
>
> > After trying docker, rkt etc, I have found nixos-container to be best
> > suited for my application. However, I find ​a warning that root access to
> > the container should not be provided to any untrusted user. I am
> > wondering
> > if I can create a normal user in a declarative container, would that be
> > safe? This may be a trivial question, I just want to be clear on this.
>
> Depending on your setup, having root in the container may be equivalent
> to having root on the host. Compared to that situation, executing as an
> unprivileged user within the container appears to improve security. That
> said, if a container solution CAN adversely affect the host system, it
> is prudent to assume that a malicious user will find a way to make that
> happen (whether anyone will care to try is another matter). This caveat
> very much applies to NixOS containers, which are implemented by
> executing `systemd-nspawn` as root on the host system.
>
> Systemd-nspawn upstream explicitly states that lightweight containers
> are insecure and not to be relied on to do much beyond preventing
> accidental damage to the host system. If security is your only reason
> for using containers, consider whether you're meaningfully improving
> security compared to running the service as an unprivileged user on the
> host and not actually making things worse by introducing additional
> complexity.
>
> HTH,
> Joachim
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How can we make import-from-derivation more useful?

[Ericson, John]

Shea's changes sound good, and I once wrote up a plan for making
`--dry-run` play nicer with this https://github.com/NixOS/nix/issues/666 .
That said, I rather just start using import-from-derivation in nixpkgs
immediately, and let the fallout be more motivation to improve nix. IMO,
anybody using the eval-everything portions of nix-env is a masochist
anyways :).

On Mon, Dec 28, 2015 at 8:12 AM, Shea Levy  wrote:

> https://github.com/NixOS/nix/pull/31 may be relevant.
>
> On 2015-12-28 11:10, Daniel Peebles wrote:
> > A few days ago, I proposed importing from a derivation [1] to save us
> > from having to manually manage autogenerated firefox/thunderbird
> > fixed-output derivaiton hash files and junk up the nixpkgs repository
> > with them. In response, Vladimír Čunát pointed out that nix-env would
> > currently force a download at evaluation time as a result of that
> > change, and that's undesirable.
> >
> > I see it as inevitable that we'll have to start importing from
> > derivations to make external package ecosystems more manageable.
> > Currently, haskellPackages is already eating up a large chunk of the
> > overall repository size, and as we start adopting similar automated
> > processes to manage other ecosystems, I see no way to keep the repo
> > size manageable. It also feels like a bit of an abuse of a VCS to be
> > putting autogenerated files in it, as we do today. Especially when
> > nix
> > is generally so good at doing that sort of thing for us.
> >
> > It seems like the main obstacle standing in the way of this pattern
> > is nix-env, given its habit (also unsustainable) of forcing the
> > evaluation of a large chunk of the packages expression. People have
> > expressed a desire to deprecate that sort of functionality, but I
> > don't know what sort of timeline such a change could be made on, so
> > I'm looking for ways to make it more manageable in the meantime.
> >
> > Does anyone have ideas on how to improve this? Or am I wrong about
> > things getting out of control if we don't? Are there other options?
> >
> > Thanks,
> > Dan
> >
> >
> >
> > Links:
> > --
> > [1]
> > https://github.com/NixOS/nixpkgs/pull/11319#issuecomment-167144900
> >
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Publish All of Hackage

[Ericson, John]

Yeah longer term https://github.com/NixOS/nix/issues/520 can be of use, but
for now normal fetchgit should be fine for the reasons Oliver mentions. 52
megs is something, but its only a compile-time dep, and in quite long term
I hope things like IPFS means the hashes can be checked out incrementally.

> Honestly, though, IMO the nix/nixpkgs model of "determine the build
exactly from the expressions" simply does not match the modern
language-specific package manager model of "dynamically query a web service
for the latest versions that meet my constraints". I think something
leveraging nix-exec [2] might be a good short-term solution, and in the
long term better management of impurities (at the expression level,
resulting in still-pure *builds*) is needed.

That incongruity is real, but the things like Stack, and Rust's Cargo's and
Ruby's bundler's lockfiles convince me that perhaps those tools are a bit
too non-deterministic for even more mainstream tastes. Not quite sure what
the optimal setup looks like but I trust we will get there.

Lastly, I mailed list before about this
https://www.mail-archive.com/nix-dev@lists.science.uu.nl/msg17048.html The
big points are all already in this thread, but a few ideas on the details
there.

On Fri, Nov 20, 2015 at 6:42 AM, Oliver Charles 
wrote:

> On Fri, Nov 20, 2015 at 1:58 PM Shea Levy  wrote:
>
>> The problem with doing this with import-from-derivation is we still
>> need the hashes of every tarball ahead of time (though that's much
>> smaller than all of hackage, and we really just need the hash of the
>> file that contains all the hashes in nixpkgs itself). If we have that,
>> then we don't need to generate all of hackage every time, just the bits
>> needed to build the specific requested package.
>>
>
> What I would expect is that the "generate Nix expressions" Nix expression
> - i.e., the thing that generates what is now hackage-packages.nix - would
> have to be given known inputs. As far as I understand, hackage2nix simply
> requires a checkouts of
> https://github.com/commercialhaskell/all-cabal-hashes and
> https://github.com/commercialhaskell/all-cabal-hashes. I would expect
> that those could be provided by fetchgit, so updating the set of Haskell
> packages simply requires Peter or other commiters to change the pinned Git
> revisions.
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] error in unpack-bootstrap-tools.sh

[Ericson, John]

That's unfortunate, but I suppose only what's to be expected given Debian
Squeeze.

If you are stick stuck on your earlier plan of attack, did you see on the
wiki about "bootstrapping" a custom-compiled nix with proot? That might
help.

John

On Tue, Nov 10, 2015 at 8:31 AM, Matei David <ma...@cs.toronto.edu> wrote:

> On Fri, 30 Oct 2015 08:56:29 -0700
> "Ericson, John" <john_eric...@brown.edu> wrote:
>
> > `unshare -urm` should do the trick if you have it. Also check if `man
> > 2 unshare` describes a CLONE_NEWUSR flag. `man 7 user_namespaces` is
> > also a good resource.
>
> Sorry for the delay. I looked into this, and the unshare version
> available on my cluster does not support -r. Furthermore,
> lxc-checkconfig reports:
>
> User namespace: missing
>
> So I think the kernel is too old for such things.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Ideas for a NixOS-related bachelors thesis?

[Ericson, John]

This is perhaps a bigger project, and one that would require C++, but I
think a lack of the intentional store[1] is the single biggest missing
feature for Nix.

On 29-10-2015 13:28:10, Erik Rybakken wrote:

>  The IPFS-thing would be a channel with a binary cache, not a
> /nix/store!
>

I really want IPFS too, but we can only use it correctly once we have the
intentional store. Then the IPFS store *is* the nix store, and the trust
relation from derivations->files is dealt with IPNS. We can also ditch
SQLite by just storing everything in IPFS. (The data in the sqlite acyclic
and immutable, and thus *better* suited for IPFS, except for the
"referrers" relation, which just an inverted view on the reference
relation. And how much do we need the referrers relation anyways?) Only
issue is we'd need to ditch NAR-based hashing to conform to IPFS, and that
would require changing every hash in nixpkgs.

[1]: https://github.com/NixOS/nix/issues/296
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Ideas for a NixOS-related bachelors thesis?

[Ericson, John]

Point taken. I find IPFS + Intentional store more elegant, but IPFS +
extensional store is a step in the right direction, both in terms of
conception and implementation work.

On Sun, Nov 1, 2015 at 10:51 AM, Nicolas Pierron <
nicolas.b.pier...@gmail.com> wrote:

> On the other hand, if we want to go to a model where the cache is not
> built by a central authority, which sounds a bit more interesting and
> challenging, then we would need a way to map a derivation to the output.
>

Hmm either way we map drv->output, just with extensional store the store
hash != the NAR hash. Also I can't think of easier way to verify the
drv->build mapping other than re-performing the build oneself, so I don't
think we can do better than a system based on trust.


> Note, that the intentional would not help in any way.  The intentional
> model compute hashes after moving the self-references hash to be indexes,
> which means that you effectively got a different hash than just blindly
> hashing the content.
>

If I recall Eelco's thesis correctly, the derivation is built in a
temporary location, the build is hashed, and then the binary is mangled so
the temp path is replaced with the proper /path/to/store/hash/ path. Well
if we just stick the data in IPFS before the path replacement, and let the
other side do the mangling we dodge that problem.

N.B. I'll admit that neither my above plan nor Eelco's original plan are
very satisfactory---say the path strings are compressed in the binary for
example, then the mangling will fail. I was once thinking about about an
alternate solution of mounting `/self/` or something per-process to avoid
needing mangling at all. But that will break with at least dynamically
linked libraries and probably other things.

>
> Also, you probably don't want to upload nar files as-is, even if nothing
> prevents you from doing so, because you probably want to take advantage of
> the merkle-dag for sharing similar files, in the same way as we optimize
> the nix-store with hard links.
>

Yes! This + normalizing "${mDrv}/sub/path" to a hash of the subdirectory
could reduce/eliminate the need for multiple-output derivations.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] error in unpack-bootstrap-tools.sh

[Ericson, John]

`unshare -urm` should do the trick if you have it. Also check if `man 2
unshare` describes a CLONE_NEWUSR flag. `man 7 user_namespaces` is also a
good resource.

Root has been able to make containers for far longer on Linux, so the
availability of LXC does not mean so much.

Sorry for not CC'ing the list in my initial reply.

John

On Fri, Oct 30, 2015 at 7:41 AM, Matei David <ma...@cs.toronto.edu> wrote:

> On Wed, 28 Oct 2015 20:56:18 -0700
> "Ericson, John" <john_eric...@brown.edu> wrote:
>
> > I assume your kernel is too old for this if you are on Debian
> > squeeze, but just in case I am wrong, have you looked at unshare/user
> > namespaces? Basically you can do things the easy way way, and use
> > unshare to give you a new virtual filesystem where you have
> > permission to mount /nix, rather than using proot.
>
> Hi,
>
> Thanks for your suggestion. I checked, and unshare indeed exists. I've
> never used it so I'm not sure how it works. Wouldn't I still need root
> to mount /my-home/nix to /nix?
>
> The unshare man page only mentions mount namespaces, I'm less sure about
> user namespaces, I mean I don't know if they are implemented in the
> squeeze kernel. According to this
> https://wiki.debian.org/LXC
> LXC is supported, but I don't know to what extent. It's definitely not
> installed yet, but I can see a package being available.
>
> I googled a bit but I can't find a relevant howto about these things,
> do you know of any good reference?
>
> Thanks,
> Matei
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev