Re: [Nix-dev] [***SPAM***] unzipNLS

[Freddy Rietdijk via nix-dev]

Thanks! bd129c2b3e033bd92d5c93cca4d516b021d408e6

On Sun, Jul 9, 2017 at 4:01 AM, Karn Kallio via nix-dev <
nix-dev@lists.science.uu.nl> wrote:

>
> The unzip Nixpkgs expression does not build with the NLS attribute
> selected because the patch conditionally applied conflicts with the
> earlier patch [dont-hardcode-cc.patch].
>
> The attached patch fixes the build by increasing the patch fuzz to the
> size of the context.
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Auto-generated expressions for applications

[Freddy Rietdijk]

> Current approach seems to be doing the job except notifying people when
dependency is updated.

What do you base that on? With the generated data sets for applications we
have another issue which I didn't mention, and that is the sometimes overly
conservative pinning of versions by the application developer. Aside from
reporting such issues upstream, how to handle that?

Other distributions face similar issues but, at least with Python, on a
smaller scale, because they cannot have multiple versions of packages and
therefore having separate package sets is just not an option for them.

Maybe I should have written in the initial mail that I'm looking for a
policy on how we're going to deal with these ki nd of situations.

On Tue, May 30, 2017 at 2:10 PM, Tomasz Czyż  wrote:

> Current approach seems to be doing the job except notifying people when
> dependency is updated. Previously we had monitor to do some similar stuff,
> and I think vulnix can check that without much effort so I wouldn't worry
> about having duplicated packages for apps.
> I think focusing on improving CI process and security notifications
> process is the way to go.
>
> Probably we could set another process/job in hydra to check all apps for
> security issues/updates. (I'm not sure if security team doesn't have that
> already).
>
> 2017-05-30 10:17 GMT+01:00 Marc Weber :
>
>> Let met try to sum up what I remember:
>>
>> - There 3+ solutions to update "sources" documented on the wiki
>>   somewhere - ideas from comparing versions with other distributions up
>>   to adding scrapers getting latest version from web sites if I recall
>>   correctly
>>
>> - Putting automatically generated code into nixpkgs doesn't solve all
>>   issues, for corner cases you have to duplicate dependencies using
>>   different version constraints.
>>
>>   -> overhead
>>
>> - Its not always quite clear how stable the user wants to be
>>   (gimp/inskscape) case, master sometimes has new features.
>>
>>   So which versions to support ?
>>
>> - Whatever we do, we don't solve anything for other distros (which
>>   suffer the same problem), unless we switch point of view:
>>
>>   The solution would be a cross platform cross language dependency
>>   management system allowing to declare dependencies in a file so that
>>   you can even install from gihtub automatically.
>>
>>   systemPackages = [ (fromGithub "user/package" "HEAD") ] # sort rest out
>> on your own, thanks
>>
>> package A could be working, package B could be working, but [A B] in the
>> same environment not (because they both depend on executable C)
>>
>> After all we want nixpkgs to be at least stable enough to have broken
>> packages marked as broken and expect everything else to at least
>> compile/install.
>>
>> Which are the best short/middle/long term solutions ?
>>
>> Marc Weber
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>>
>
>
>
> --
> Tomasz Czyż
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Auto-generated expressions for applications

[Freddy Rietdijk]

Hi,

At several places in Nixpkgs we use auto-generated data, mostly for the
larger package sets like Haskell. Sometimes we also use auto-generated sets
for applications that may need different versions than are offered in the
main package sets. In the past months several issues/PR's have been opened
to add rather large Python applications to Nixpkgs, generated using
`pypi2nix`.

Using such tool to generate the expressions has some very big advantages.
However, there's also a disadvantage. The expressions shouldn't only be
updated when there's a new release of the application, but also whenever
there's security updates in any of its generated dependencies, which the
application maintainer now has to keep track of. Therefore, I find it quite
a risk to have separate package sets. At the same time, we'll also likely
run behind in the main package sets every once in a while.

What do you think of this issue? Any suggestions how we can improve this?
Maybe we could have a server/bot that runs update scripts and opens a PR
whenever there's an actual diff?

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python 3.6 as default Python 3 interpreter

[Freddy Rietdijk]

You may have already noticed that 3.6 is now the default 3.x interpreter on
Nixpkgs Unstable.

Freddy

On Thu, Apr 20, 2017 at 11:04 AM, Lancelot SIX <lsix@lancelotsix.com>
wrote:

> Hi Freddy,
>
> This sounds good to me !
>
> Lancelot.
>
> On 20/04/2017 10:58, Freddy Rietdijk wrote:
>
> Hi,
>
> Currently the default Python 3 interpreter in Nixpkgs is version 3.5.
> Version 3.6 was released in December and a first maintenance-release this
> March. Unless there are good objections I would like to change the default
> Python 3 interpreter to version 3.6.
>
> This will affect a couple of Python applications. I tested several and
> haven't run into any issues. Of course, maintainers can always explicitly
> choose a different version, like 3.5, if there's a need.
>
> Freddy
>
>
> ___
> nix-dev mailing 
> listnix-...@lists.science.uu.nlhttps://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python: Provide your own buildPhase and installPhase.

[Freddy Rietdijk]

Hi,

What are you trying to achieve? You want to package panda3d?

The `buildPythonPackage` function builds by default setuptools/distutils
Python packages. When passed `format="flit"` it builds from Flit source,
and when passed `format="wheel'` it takes the wheel and installs it. The
fourth option is, as you mentioned, `format="other"`. In this case, it uses
the `buildPhase` and `installPhase` from `stdenv.mkDerivation` and is
therefore useful for when a Makefile is provided and the package has
scripts that need to be wrapped or provides Python modules.

An example of `format="other"` is how we provide the `tkinter` module from
the standard library.
You don't have to copy the code of the `buildPythonPackage` function, just
call it and pass in the arguments.

Freddy



On Fri, May 26, 2017 at 10:03 PM, Nawal Husnoo  wrote:

> Hi,
>
> Could anyone point me to an example where I can see this in action please?
>
> https://github.com/NixOS/nixpkgs/blob/master/pkgs/
> development/interpreters/python/build-python-package.nix
> # "other" : Provide your own buildPhase and installPhase.
>
> Say I copy the build-python-package-flit.nix file as
> build-python-package-panda3d.nix - how do I get my default.nix to use
> this?
>
> My current attempt:
>
> with import  {};
>
> # nix-shell
>
> # python2.7 makepanda/makepanda.py --everything --wheel --no-egl --no-gles
> --no-gles2 --no-opencv  --use-x11 --use-python --use-gl --use-fftw
> --use-gtk2 --python-incdir 
> /nix/store/3n3r10921c0mzfrq64hc4m2jzjk13qpr-python-2.7.13/include
> --python-libdir /nix/store/3n3r10921c0mzfrq64hc4m2jzjk13qpr-python-2.7.13/lib
>
> # pip install --prefix /home/nawal/usr 
> panda3d-1.10.0-cp27-cp27mu-linux_x86_64.whl
>
> # export PYTHONPATH=/home/nawal/usr/lib/python2.7/site-packages/:$
> PYTHONPATH
>
> (let
>
> panda3d = pkgs.pythonPackages.buildPythonPackage rec{
> name = "panda3d-${version}";
> version = "";
> src = ./panda3d;
>
>
> propagatedBuildInputs =  [
> xorg.libX11
> xorg.xproto
> pkgconfig
> python
> ffmpeg-full
> assimp
> freetype
> harfbuzz
> glfw
> ode
> openal
> libtiff
> openexr
> vrpn
> bullet
> libvorbis
> libjpeg
> openjpeg
> libpng
> openssl
> zlib
> gtk2-x11
> opusTools
> x11
> xorg.libX11
> xorg.xproto
> pythonPackages.pip
> ];
> };
>
> in python.buildEnv.override {
> extraLibs = [
> panda3d
> pkgs.pythonPackages.gnureadline
> ];
> ignoreCollisions = true;
> }).env
>
>
>
>
>
>
>
>
>
>
>
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Installing manpages

[Freddy Rietdijk]

Function for getting all manpages of all pkgs:

```
manpkgs = pkgs: builtins.map (x: lib.getOutput "man" x) pkgs;
```
https://github.com/NixOS/nixpkgs/blob/5989515b9410f3d557e1aba037325a218f4338f5/lib/attrsets.nix#L450

You might want to concatenate the generated list to `pkgs`.

On Thu, May 25, 2017 at 12:31 PM, Sergey Mironov  wrote:

> Hi! Is there a possibility to install manpages for all packages I have
> in NixOS. Possibly, a combinator which would convert a list of `with
> pkgs; [foo bar]` to `with pkgs; [foo foo.man bar bar.man]` with checks
> of .man existing ?
>
> Regards,
> Sergey
>
>
> PS
> Sorry If I am missing the topic in documentation
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] How to build all of nixpkgs?

[Freddy Rietdijk]

If I am correct Hydra first checks which jobs evaluate, and then builds
those. It checks using the findJobs function in this file
https://github.com/NixOS/hydra/blob/master/src/hydra-eval-jobs/hydra-eval-jobs.cc

On May 22, 2017 17:35, "Bjørn Forsman"  wrote:

> On 22 May 2017 at 17:21, Tomasz Czyż  wrote:
> > Look there, that's how all packages are build by CI:
> > http://hydra.nixos.org/jobset/nixpkgs/trunk#tabs-configuration
> > It uses pkgs/top-level/release.nix to build all of that.
> > Also it passes nixpkgs. So maybe you need to do something like
>
> I have looked there. That's why I'm wondering what magic Hydra does,
> because I cannot get this thing to work with nix-build:
>
> $ nix-build -I nixpkgs=/path/to/my/nixpkgs pkgs/top-level/release.nix
> error: value is a set while a Boolean was expected
>
> $ nix-build pkgs/top-level/release.nix --arg nixpkgs \"./.\" --arg
> officialRelease false
> error: value is a set while a Boolean was expected
>
> Please let me know if you manage to build that with nix-build.
>
> Best regards,
> Bjørn Forsman
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] wrapProgram: also pass name as NIX_PROGRAM_NAME

[Freddy Rietdijk]

>
> The problem with such an environment variable is that it's inherited by
> child
> processes, who might get confused. (E.g. some code might run both in a
> wrapped
> and unwrapped context, so it wouldn't be able to rely on $NIX_PROGRAM_NAME
> unambiguously.)


In the case of Python we might be able to inject some code where we unset
the variable. Currently we inject the path to the script, but this is wrong
in case a wrapper is used.

I think in general a) wrappers should be avoided; and b) nested wrappers
> should
> *definitely* be avoided. Wrappers cannot really be avoided for the "delayed
> composition" use case (like firefox-wrapper), but in such a case perhaps
> the
> inner wrapper can be eliminated. For example, if the inner wrapper is used
> to
> set $PYTHONPATH, this can be moved into the underlying Python script by
> setting
> sys.path at the start.


When possible I would prefer to avoid wrappers as well. Unfortunately,
there's not really a way around it with Python (2.x).
Sometimes we need to add a program to PATH and for that we need a wrapper.
When we add the wrapper, the name is wrong. if the wrapper is added in the
same derivation, then we can insert sys.path like we do now.

However, often we also need composition: creating an environment consisting
of the interpreter and multiple packages. In that case, they need to be
able to find each other, and that means again wrappers (although with 3.x
we might get away with pyvenv.cfg).

On Mon, May 22, 2017 at 2:36 PM, Eelco Dolstra <eelco.dols...@logicblox.com>
wrote:

> Hi,
>
> On 05/19/2017 01:35 PM, Freddy Rietdijk wrote:
>
> > Therefore, I propose we set an environment variable `NIX_PROGRAM_NAME`
> that the
> > scripts check for. To prevent creating yet another `wrapProgram` variant
> I
> > propose we extend `wrapProgram` to always set this variable.
>
> The problem with such an environment variable is that it's inherited by
> child
> processes, who might get confused. (E.g. some code might run both in a
> wrapped
> and unwrapped context, so it wouldn't be able to rely on $NIX_PROGRAM_NAME
> unambiguously.)
>
> But maybe $NIX_PROGRAM_NAME can be used only in the case where one wrapper
> script calls another, so the final wrapper can unset it. However, if we
> have
> wrapper scripts detecting that the wrapped program is also a wrapper
> script, we
> might as well eliminate the execve into the next wrapper (thus preserving
> the
> original argv[0]).
>
> > This, however, doesn't solve the issue with nested wrappers yet
>
> I think in general a) wrappers should be avoided; and b) nested wrappers
> should
> *definitely* be avoided. Wrappers cannot really be avoided for the "delayed
> composition" use case (like firefox-wrapper), but in such a case perhaps
> the
> inner wrapper can be eliminated. For example, if the inner wrapper is used
> to
> set $PYTHONPATH, this can be moved into the underlying Python script by
> setting
> sys.path at the start.
>
> --
> Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] wrapProgram: also pass name as NIX_PROGRAM_NAME

[Freddy Rietdijk]

If a Python script is run, then `sys.argv[0]` is set to the script name (or
path, depending on platform). Some scripts use this name or path to start a
new instance of itself. Unfortunately, exec -a doesn't set `sys.argv[0]`.
Therefore, when running a wrapped Python script, `sys.argv[0]` points not
to the wrapped script, but to the original one.

On Mon, May 22, 2017 at 9:41 AM, zimbatm <zimb...@zimbatm.com> wrote:

> Hi Freddy,
>
> Can you expand a bit on why this is a problem? If a program is executed
> using python I would expect this to be reflected in the process name. Argv0
> would contain a path to python and argv1 the path to the script.
>
> Cheers,
> z
>
> On Fri, 19 May 2017, 12:35 Freddy Rietdijk, <freddyrietd...@fridh.nl>
> wrote:
>
>> Hi,
>>
>> When we create a wrapper we use `exec -a` so that the wrapped program
>> uses the name of the original program. This isn't respected by all
>> programs, e.g. Python ignores this. If I am correct this is the case with
>> all interpreters. How do we handle this with other interpreters?
>>
>> Currently we 'solve' the issue for Python by patching the scripts setting
>> `sys.argv[0]` to the original script name, including the full path. This is
>> sufficient for scripts that call themselves, however, it is troublesome
>> when such a script is wrapped in another derivation, because when the
>> script now calls itself, it will call the unwrapped version.
>>
>> Therefore, I propose we set an environment variable `NIX_PROGRAM_NAME`
>> that the scripts check for. To prevent creating yet another `wrapProgram`
>> variant I propose we extend `wrapProgram` to always set this variable.
>>
>> This, however, doesn't solve the issue with nested wrappers yet, because
>> if we would use `set`, it would still refer to the name passed by the
>> innermost wrapper. Either we try and update wrappers, or, which has my
>> preference, we do not use `set` but use `prefix` and make the script pick
>> the last item from the list.
>>
>> What do you think? I like to also refer to the following gist where I've
>> been documenting some of the issues and how we can rewrite the Python infra
>> to solve those issues.
>> https://gist.github.com/FRidh/c3e974e34eac738405af699786b462d5
>>
>>
>> Freddy
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] wrapProgram: also pass name as NIX_PROGRAM_NAME

[Freddy Rietdijk]

Hi,

When we create a wrapper we use `exec -a` so that the wrapped program uses
the name of the original program. This isn't respected by all programs,
e.g. Python ignores this. If I am correct this is the case with all
interpreters. How do we handle this with other interpreters?

Currently we 'solve' the issue for Python by patching the scripts setting
`sys.argv[0]` to the original script name, including the full path. This is
sufficient for scripts that call themselves, however, it is troublesome
when such a script is wrapped in another derivation, because when the
script now calls itself, it will call the unwrapped version.

Therefore, I propose we set an environment variable `NIX_PROGRAM_NAME` that
the scripts check for. To prevent creating yet another `wrapProgram`
variant I propose we extend `wrapProgram` to always set this variable.

This, however, doesn't solve the issue with nested wrappers yet, because if
we would use `set`, it would still refer to the name passed by the
innermost wrapper. Either we try and update wrappers, or, which has my
preference, we do not use `set` but use `prefix` and make the script pick
the last item from the list.

What do you think? I like to also refer to the following gist where I've
been documenting some of the issues and how we can rewrite the Python infra
to solve those issues.
https://gist.github.com/FRidh/c3e974e34eac738405af699786b462d5


Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Python 3.6 as default Python 3 interpreter

[Freddy Rietdijk]

Hi,

Currently the default Python 3 interpreter in Nixpkgs is version 3.5.
Version 3.6 was released in December and a first maintenance-release this
March. Unless there are good objections I would like to change the default
Python 3 interpreter to version 3.6.

This will affect a couple of Python applications. I tested several and
haven't run into any issues. Of course, maintainers can always explicitly
choose a different version, like 3.5, if there's a need.

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] IPython in Nixpkgs

[Freddy Rietdijk]

Hi,

Currently we have IPython 5.3.0 in Unstable and 17.03. Soon IPython 6 will
be released which will only support Python 3.x, and that might mean the
Notebook will also no longer support a Python 2.x kernel.

As maintainer of the package my intention is to just upgrade to this latest
version on Unstable. Users that do want to use IPython / Notebook with a
2.x kernel can simply use the 17.03 version.

Now, I know there are some users of IPython / Notebook with the 2.x kernel,
so I do would like to hear what you think of this.

Having an older and newer version simultaneously is likely not going to
work because of different versions of dependencies that would be needed,
although I haven't checked yet whether that is really the case. In any
case, I do not plan to maintain the soon-to-be-older version.

Kind regards,

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Inkscape pythonEnv - .sif export - PYTHONPATH wrapping

[Freddy Rietdijk]

You wrote the following to me directly:


> Can you try regression exporting .sif files?
> How to: Export -> select filter sif.
> A simple rectangle is enough, the error I get is "lxml" not found.
> Have a look at your commit: You left the comment there, but dropped teh
> wrapProgram (which actually fixes the filter).
>  # Make sure PyXML modules can be found at run-time.
> -for i in "$out/bin/"*
> -do
> -  wrapProgram "$i" --prefix PYTHONPATH :  \
> -   "$(toPythonPath ${pyxml}):$(toPythonPath ${lxml}):$(toPythonPath
> ${numpy})"  \
> -   --prefix PATH : ${python}/bin ||  \
> -exit 2
> -done


I get a SyntaxError when trying to export this, and it seems that's due to
an impurity; removing my Python installation resulted in a different error.
Apparently somewhere certain parts of Inkscape try to just call `python`
instead of the one that was passed as a `propagatedBuildInput` (note that
we shouldn't propagate the env here). Either we fix those references, or
add `${pythonEnv}/bin` to PATH with wrapProgram.

The reason those lines are removed is because an environment constructed
with symbolic links is used now instead. Therefore, there shouldn't be a
need anymore

On Fri, Mar 3, 2017 at 3:38 PM, Marc Weber  wrote:

> A commit turning the list of pythonpackages into pythonEnv broke the
> export. The problem is I'm unsure how to suggest a proper fix to
> fr...@fridh.nl
> because there are many ways to fix:
>
> makeWrapper requires shell function toPythonPath provided by
> ${python}/nix-support/..
> which is "gone" using pythonEnv. Possibilities to fix
>
>   1) revert
>   2) "rescue" adding python2 to buildInputs (providing toPythonPath)
>   see PATCH below
>   3) improve pythonEnv lifting toPythonPath from shell to nix world
>   4a) improve pythonEnv adding a run-in-python-env sot hat the wrapped
>   script looks like
>   #!/bin/sh
>   /nix/store/bin/python-env/bin/run-in-env /nix/store/...inkscape
> "$@"
>
>   4b) make it composable / sourcable
>
>   #!/bin/sh
>   source /nix/store/bin/python-env/nix-support/env
>   /nix/store/...inkscape "$@"
>
> No idea which solution is best - thoughts?
>
> People who are maintaining python might want to take care of.
>
> Marc Weber
>
> immediate fix: PATCH
>
> diff --git a/pkgs/applications/graphics/inkscape/default.nix
> b/pkgs/applications/graphics/inkscape/default.nix
> index 2f72d00..53d612f 100644
> --- a/pkgs/applications/graphics/inkscape/default.nix
> +++ b/pkgs/applications/graphics/inkscape/default.nix
> @@ -31,12 +32,23 @@ stdenv.mkDerivation rec {
>  libxml2 libxslt glib gtkmm2 glibmm libsigcxx lcms boost gettext
>  makeWrapper intltool gsl poppler imagemagick libwpg librevenge
>  libvisio libcdr libexif automake114x cmake
> +
> +python2 # provides toPythonPath
>];
>
>enableParallelBuilding = true;
>
>postInstall = ''
>  # Make sure PyXML modules can be found at run-time.
> +for i in "$out/bin/"*
> +do
> +  wrapProgram "$i" \
> +--prefix PYTHONPATH : $(toPythonPath "${python2Env}") \
> +--prefix PATH   : ${python2Env}/bin || \
> +exit 2
> +done
>  rm "$out/share/icons/hicolor/icon-theme.cache"
>'';
>
> diff --git a/pkgs/applications/graphics/inkscape/default.nix
> b/pkgs/applications/graphics/inkscape/default.nix
> index 2f72d00..53d612f 100644
> --- a/pkgs/applications/graphics/inkscape/default.nix
> +++ b/pkgs/applications/graphics/inkscape/default.nix
> @@ -31,12 +32,23 @@ stdenv.mkDerivation rec {
>  libxml2 libxslt glib gtkmm2 glibmm libsigcxx lcms boost gettext
>  makeWrapper intltool gsl poppler imagemagick libwpg librevenge
>  libvisio libcdr libexif automake114x cmake
> +
> +python2 # provides toPythonPath
>];
>
>enableParallelBuilding = true;
>
>postInstall = ''
>  # Make sure PyXML modules can be found at run-time.
> +
> +# Make sure PyXML modules can be found at run-time.
> +for i in "$out/bin/"*
> +do
> +  wrapProgram "$i" \
> +--prefix PYTHONPATH : $(toPythonPath "${python2Env}") \
> +--prefix PATH   : ${python2Env}/bin || \
> +exit 2
> +done
>  rm "$out/share/icons/hicolor/icon-theme.cache"
>'';
>
> diff --git a/pkgs/applications/graphics/inkscape/default.nix
> b/pkgs/applications/graphics/inkscape/default.nix
> index 2f72d00..53d612f 100644
> --- a/pkgs/applications/graphics/inkscape/default.nix
> +++ b/pkgs/applications/graphics/inkscape/default.nix
> @@ -31,12 +32,23 @@ stdenv.mkDerivation rec {
>  libxml2 libxslt glib gtkmm2 glibmm libsigcxx lcms boost gettext
>  makeWrapper intltool gsl poppler imagemagick libwpg librevenge
>  libvisio libcdr libexif automake114x cmake
> +
> +python2 # provides toPythonPath
>];
>
>enableParallelBuilding = true;
>
>postInstall = ''
>  # Make sure PyXML modules can be found at run-time.
> +
> +# Make sure 

Re: [Nix-dev] Fwd: Python 3 as default

[Freddy Rietdijk]

> It may silently break external users, though.

It may do so and likely will. Even so, users of Nixpkgs have always had the
choice to explicitly pin a version, both minor and major.

> BTW, I just noticed that the Python 3 package in 17.03 already defines a
"python
-> python3" symlink (introduced by [1]). IMHO, we should revert that, since
it
can only lead to pointless environment collision errors/warnings.

The collision warnings has been taken care of by changing the priority. It
might be considered messy though, `python` pointing to `python3` when we
only have 3 in a profile, yet pointing to `python2` when 2 is in the
profile.

The symbolic link was added already 4,5 months ago and there haven't been
any complaints since. The only mention of a potential issue came after I
sent this e-mail with Peter mentioning his scripts pointing to
`/usr/bin/env python`. But here the issue is simple: according to PEP 394
[1] one should update the shebangs to point to the version of the
interpreter that is needed, and only point to `python` when both 2 and 3
are valid. Therefore, this is an issue that package/script developers are
supposed to solve.

> Also, we probably don't want to promote the use of the "python"
executable name because it's unversioned

One should use `/bin/python` only when it can handle 2 and 3 and use a more
specific shebang when it is needed.
The reason why `python` also points to `/bin/python3` is because many
packages haven't yet adopted the earlier referenced PEP 394 [1] even though
their packages are already compatible with Python 3.

Earlier I said we don't follow the PEP, but actually we do, with the
noticeable addition that we also point to `/bin/python3` when there is no
`python2` interpreter.

[1] http://legacy.python.org/dev/peps/pep-0394/#id5


On Mon, Feb 27, 2017 at 4:20 PM, Eelco Dolstra <eelco.dols...@logicblox.com>
wrote:

> Hi,
>
> On 02/24/2017 07:54 AM, Freddy Rietdijk wrote:
>
> >> I'd be happy to help with the process of mass-renaming, if somebody
> provided a
> > list of packages wanting 2 or 3, however I suspect the real work is in
> > determining those package sets.
> > So, if a query-replace for `pkgs.python` -> `pkgs.python2` /
> `pkgs.python3`
> > within nixpkgs would cause so much trouble, how would a rename in
> > `all-packages.nix` fare any better? Are we ready to throw the switch yet?
> >
> > All the packages that explicitly need Python 2 are already pinned to use
> Python
> > 2. That was a matter of converting, testing, and moving on. About 150
> packages
> > were pinned to `python2`. That is why a `python = python3` won't break
> anything
> > (or nearly nothing) in Nixpkgs anymore.
>
> It may silently break external users, though. So removing the "python"
> attribute
> is preferable to aliasing it to "python3".
>
> BTW, I just noticed that the Python 3 package in 17.03 already defines a
> "python
> -> python3" symlink (introduced by [1]). IMHO, we should revert that,
> since it
> can only lead to pointless environment collision errors/warnings. Also, we
> probably don't want to promote the use of the "python" executable name
> because
> it's unversioned (so when Python 4 breaks the language again, we'll have
> to go
> through this whole process again).
>
> [1] https://github.com/NixOS/nixpkgs/commit/708822250e017f0256b0bace5af27a
> 7232e6e1fd
>
> --
> Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Fwd: Python 3 as default

[Freddy Rietdijk]

> I'd be happy to help with the process of mass-renaming, if somebody
provided a list of packages wanting 2 or 3, however I suspect the real work
is in determining those package sets.
So, if a query-replace for `pkgs.python` -> `pkgs.python2` / `pkgs.python3`
within nixpkgs would cause so much trouble, how would a rename in
`all-packages.nix` fare any better? Are we ready to throw the switch yet?

All the packages that explicitly need Python 2 are already pinned to use
Python 2. That was a matter of converting, testing, and moving on. About
150 packages were pinned to `python2`. That is why a `python = python3`
won't break anything (or nearly nothing) in Nixpkgs anymore.

Explicitly pinning to `python3` means converting every expression that has
`python` to `python3`. No breakage should occur anymore, except when one
makes a mistake in converting expressions only partially. There are about
1000 packages that need to be converted.

The best method I can think of is searching for all the files that contain
`python` without leading and trailing alphanumerical characters. Those can
typically be converted to `python3`. Exceptions are
`pkgs/top-level/python-packages.nix`,
`pkgs/development/interpreters/python`,
 `pkgs/development/python-modules/*` and when the `python` argument is
overriden in, say, `all-packages.nix`. Another exception could be `python =
python3.withPackages(...)` but I think that is rare.

On Fri, Feb 24, 2017 at 4:59 AM, Herwig Hochleitner <hhochleit...@gmail.com>
wrote:

> And a copy for the list as well + a one person applause for our
> friend, reply-all.
>
> -- Forwarded message --
> From: Herwig Hochleitner <hhochleit...@gmail.com>
> Date: 2017-02-24 4:54 GMT+01:00
> Subject: Re: [Nix-dev] Python 3 as default
> To: Freddy Rietdijk <freddyrietd...@fridh.nl>
>
>
> First off, for context: I'm very grateful for your work to get our
> python stuff over to python3.
>
> I also support python3 providing `/bin/python`, _if_ we ensure that
> python2's `/bin/python` takes precedence in all collisions (right
> until python2's EOL). I think the PEP's symlinking guideline doesn't
> directly apply here, since our transitive dependencies don't collide
> (horray!).
>
> I'm against changing pkgs.python to mean python3, though. Let me break
> down why:
>
> 2017-02-22 16:32 GMT+01:00 Freddy Rietdijk <freddyrietd...@fridh.nl>:
> >
> >
> > If we can't break stuff we might as well not make any changes at all.
>
>
> I see your point, but I'd like to point out that additive changes are
> less cumbersome than breaking changes, and explicit breakage is less
> cumbersome than implicit breakage.
>
> > the fact that upstream wants everyone to move away from Python 2
>
> to be fair, upstream has been wanting that for a decade
>
> >
> > incompatibilities also exist between minor versions
>
>
> i think users are customarily expected to keep up with point releases.
>
> > explicitly changing parameters and arguments ... considering how much
> Python is used throughout Nixpkgs that's just unrealistic.
> >
> > I'm still open to getting rid of the `python` attribute, ... If this is
> the route we want to take, then its very unlikely we will be able to switch
> to 3.5 as the default for 17.03,
>
>
> I'd be happy to help with the process of mass-renaming, if somebody
> provided a list of packages wanting 2 or 3, however I suspect the real
> work is in determining those package sets.
> So, if a query-replace for `pkgs.python` -> `pkgs.python2` /
> `pkgs.python3` within nixpkgs would cause so much trouble, how would a
> rename in `all-packages.nix` fare any better? Are we ready to throw
> the switch yet?
>
> > or 3.6 for 17.09.
>
>
> Are there python projects, stuck at 3.5, as well?
>
> kind regards
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python 3 as default

[Freddy Rietdijk]

> I would also vote for not changing "default" python as it will break
stuff.

If we can't break stuff we might as well not make any changes at all.

There are definitely differences between Python 2 and 3 causing
incompatibilities, but incompatibilities also exist between minor versions,
so I don't think it is a very strong argument.

> but the nix expressions only have the version suffixed attributes to
increase explicitness and improve readability.
> Plus then we will be ready for python 4 

That means explicitly changing parameters and arguments each time we want
to upgrade Python (major) version. That's okay with certain packages, but
considering how much Python is used throughout Nixpkgs that's just
unrealistic.

I'm still open to getting rid of the `python` attribute, but I just don't
think it is very realistic. If this is the route we want to take, then its
very unlikely we will be able to switch to 3.5 as the default for 17.03, or
3.6 for 17.09.

On Wed, Feb 15, 2017 at 9:10 PM, Ricardo M. Correia 
wrote:

>
> On Tue, Feb 14, 2017 at 2:34 PM, Profpatsch  wrote:
>
>> On 17-02-14 01:18pm, Peter Simons wrote:
>> > I hardly ever use nix-shell and I don't want to, to
>> > be honest
>>
>> Completely off-discussion: Why is that?
>> Convenience? nix-shell does too many strange things?
>>
>> Personally I use nix-shell for nearly everything I do
>> nowadays because it’s so convenient.
>> Even a buildFHSUserEnv is quickly set up for strange software.
>>
>
> Personally, I try to avoid using "nix-shell" as much as possible because I
> want "nixos-rebuild switch" to atomically install (and upgrade) all the
> software that I usually need, including development packages.
>
> With nix-shell, I risk having the packages I need become outdated (without
> me noticing) and/or become garbage collected (which I do every day
> automatically) and unavailable if I lose Internet access (e.g. because I'm
> traveling).
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Explicitly selecting sources for "src" in stdenv.mkDerivation?

[Freddy Rietdijk]

> src = [ ./subproject-A/schema.sql ./subproject-A/lib ];

Each of the files you mentioned here will be stored in a separate store
path. You could write a simple function that takes a list of derivations
and copies the contents of those derivations in a new output.

On Thu, Feb 16, 2017 at 1:14 PM, Thomas Hunger  wrote:

> Hi,
>
> I am consistently struggling with the following in nix: I have a
> repository and I want to specify derivations for some local sub-projects.
> The obvious solution is
>
>   src = ./subproject-A;
>
> But that pulls in everything in that directory, including build artifacts,
> or random intermediate data files.
>
> Another solution is
>
>   src = sourceFilesBySuffices ./subproject-A [".cabal" ".hs"];
>
> Which works reasonably well but introduces this weird dance where I suffix
> files so they can be matched by sourceFilesBySuffices. Mostly I want to do
> this:
>
>   src = [ ./subproject-A/schema.sql ./subproject-A/lib ];
>
> Or even get all the files checked into git:
>
>   src = gitFiles ./subproject-A; # does not work
>
> I was wondering whether any of you have this issue, and if you do: how do
> you solve it?
>
> ~
>
> (see also https://github.com/NixOS/nix/issues/885)
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python 3 as default

[Freddy Rietdijk]

It would two mean things:

   1.  alias `python3 to `python`
   2.  add a symlink that points to the `python` program, as discussed
   before in this thread.

Just removing the `python` attribute is indeed perhaps a better solution
than 1). It's explicit, and like you said, it won't silently break people's
expressions. If anyone is willing to contribute their sed-fu to do such
mass-replacement then I don't see any reason why not.




On Tue, Feb 14, 2017 at 4:22 PM, Eelco Dolstra <eelco.dols...@logicblox.com>
wrote:

> Hi,
>
> On 02/13/2017 10:08 AM, Freddy Rietdijk wrote:
>
> > As you may be aware the plan is to change to Python 3.5 as default for
> 17.03.
>
> What does it mean to make Python 3 the default? If it's changing the
> "python"
> attribute or command to alias "python3", I don't really see the advantage
> in
> doing that. It will just unnecessarily and silently break stuff (like user
> scripts that have "#!/usr/bin/env python").
>
> We could remove the "python" and "pythonPackages" attributes from Nixpkgs,
> forcing use of "python2" or "python3". Then at least people's Nix
> expressions
> won't change meaning silently.
>
> --
> Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python 3 as default

[Freddy Rietdijk]

Peter, that issue we can fix by setting the priority for Python 2 higher.
That should also be done in `python.buildEnv`.

So aside from symlinking, there are I think two other methods. One, fixing
all the shebangs ourselves. Two, modify `patchShebangs` to have an
exception for Python so it will rewrite to whatever `python{X}` is on PATH.
The first option is likely what we ought to do, and the second option won't
happen.

As to your question what the advantages are. Well, aside from the fact that
upstream wants everyone to move away from Python 2, there aren't any,
really. It makes sense for them, they don't want to support old code. Sure,
the Python 3 interpreter can be faster, but then again, the closure is
somewhat bigger. Furthermore, there's a good chance you need now both 2 and
3 on your system. If you want to develop with 3, you already can (and
should), so there isn't any advantage there either.

So when you just want to have your system with your programs and services,
there aren't any real advantages. But for those developing and maintaining
Python packages there definitely are: not having to keep supporting both 2
and 3 because users/distro's don't want to change.


On Tue, Feb 14, 2017 at 1:25 PM, Peter Simons  wrote:

> Hi Freddy,
>
>  > Peter, could you explain how it broke your setup?
>
> I have plenty of scripts that run "python" expecting to get python2.
>
> I can fix those scripts, of course. I probably should, and I will.
> Still, from a user's point of view, the experience of them suddenly
> misbehaving was not a nice one. Especially since it's not quite
> obviously to me what the *benefits* that change are.
>
> Best regards,
> Peter
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python 3 as default

[Freddy Rietdijk]

Peter, could you explain how it broke your setup?

On Tue, Feb 14, 2017 at 1:14 PM, Peter Simons  wrote:

> Hi Richard,
>
>  > Out of interest can I ask what the rationale for that decision is?
>  >
>  > I'd like to try to persuade you not to symlink python to python3,
>  > because doing so potentially breaks existing python2 programs.
>
> I, too, wonder why we sym-link "python" to "python3". I'm not sure what
> the benefits are, but this change certainly broke my setup, so I am well
> aware of the disadvantages.
>
> Best regards,
> Peter
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python 3 as default

[Freddy Rietdijk]

Sure. As I said the PEP doesn't provide any method for distributions that
would already like to make the switch. In an ideal scenario all developers
would update their requirements/shebangs. That just doesn't happen.

What we perhaps ought to do is instead fix all those shebangs ourselves,
before we run `patchShebang/wrapPythonPrograms`. Considering the amount of
(boring) work that takes I don't see anyone stepping up doing that.
Therefore, symlinking `python` *also* to `python3` is a pragmatic solution.

As Moritz described, we already pass a specific interpreter as a buildInput
to a package. Therefore, there shouldn't be any issues with our programs.
The only issue that could arise is when one installs both interpreters in
one profile. In that case a collision occurs. Maybe we should set a
priority here so that in that case `python` will refer to `python2`.













On Tue, Feb 14, 2017 at 12:11 PM, Richard Ipsum <richardip...@fastmail.co.uk
> wrote:

> On Tue, Feb 14, 2017 at 11:29:32AM +0100, Freddy Rietdijk wrote:
> > Hi Richard,
> >
> > Thanks for sharing the PEP.
> >
> > Actually, we, or at least I, chose to ignore that PEP, and so we do
> > symlink.
> >
>
> Out of interest can I ask what the rationale for that decision is?
>
> I'd like to try to persuade you not to symlink python to python3,
> because doing so potentially breaks existing python2 programs.
>
> But since I'm not an active contributor to this project it's not my place
> to argue here really. :)
>
> Thanks,
> Richard
>
> [snip]
> >
> >
> > On Tue, Feb 14, 2017 at 10:40 AM, Richard Ipsum <
> richardip...@fastmail.co.uk
> > > wrote:
> >
> > > On Mon, Feb 13, 2017 at 10:08:07AM +0100, Freddy Rietdijk wrote:
> > > > Hi,
> > > >
> > > > As you may be aware the plan is to change to Python 3.5 as default
> for
> > > > 17.03. There were a couple of things that had to be done, and I think
> > > we're
> > > > ready now to make the change. Packages that require Python 2 are now
> > > pinned
> > > > to do so.
> > > >
> > > > Now, obviously I haven't been able to test all packages myself, but
> > > judging
> > > > from what Hydra reports and from working from the `python3` branch I
> > > don't
> > > > expect any major issues.
> > > >
> > > > The Python 2.7 interpreter had quite some patches, among one to make
> the
> > > > interpreter and Python packages more deterministic. I say more,
> because
> > > > they're not fully deterministic yet. I have some patches lining up
> for
> > > the
> > > > 3.5 interpreter to improvement determinism there as well.
> > > >
> > > > In any case, I would like to merge this as soon as possible. Any
> > > objections?
> > > >
> > > > Freddy
> > >
> > > Hi, I'm not really an active contributor here but I follow NixOS
> because
> > > I think it's a really cool project.
> > >
> > > Am I to understand that NixOS is considering replacing the python2
> > > interpreter with the python3 interpreter, as arch did?
> > >
> > > If that's the case it may be worth considering this pep[1] first.
> > >
> > > If I've misunderstood and there's no intention to symlink 'python' to
> > > 'python3'
> > > then please ignore this. :)
> > >
> > > [1]: http://legacy.python.org/dev/peps/pep-0394/#id5
> > > ___
> > > nix-dev mailing list
> > > nix-dev@lists.science.uu.nl
> > > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> > >
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python 3 as default

[Freddy Rietdijk]

Hi Richard,

Thanks for sharing the PEP.

Actually, we, or at least I, chose to ignore that PEP, and so we do
symlink.

The PEP describes what distributions should do

> for the time being, all distributions should ensure that python refers to
the same target as python2.

> The more general python command should be installed whenever any version
of Python 2 is installed and should invoke the same version of Python as
the python2 command

and what script maintainers should do

> in preparation for an eventual change in the default version of Python,
Python 2 only scripts should either be updated to be source compatible with
Python 3 or else to use python2 in the shebang line.

We have a lot of packages that are compatible with both Python 2 and 3. The
PEP says about such case

> One exception to this is scripts that are deliberately written to be
source compatible with both Python 2.x and 3.x. Such scripts may continue
to use python on their shebang line without affecting their portability.

So now we have `python2` for Python 2 only, `python3` for Python3 only,
`python` for packages that support only Python 2, or both 2 and 3.

The PEP however doesn't provide any method for distributions that already
would like to make the switch to Python 3, which is the reason this PEP
came to being

> however, end users should be aware that python refers to python3 on at
least Arch Linux (that change is what prompted the creation of this
PEP), so python should be used in the shebang line only for scripts that
are source compatible with both Python 2 and 3.


Freddy


On Tue, Feb 14, 2017 at 10:40 AM, Richard Ipsum <richardip...@fastmail.co.uk
> wrote:

> On Mon, Feb 13, 2017 at 10:08:07AM +0100, Freddy Rietdijk wrote:
> > Hi,
> >
> > As you may be aware the plan is to change to Python 3.5 as default for
> > 17.03. There were a couple of things that had to be done, and I think
> we're
> > ready now to make the change. Packages that require Python 2 are now
> pinned
> > to do so.
> >
> > Now, obviously I haven't been able to test all packages myself, but
> judging
> > from what Hydra reports and from working from the `python3` branch I
> don't
> > expect any major issues.
> >
> > The Python 2.7 interpreter had quite some patches, among one to make the
> > interpreter and Python packages more deterministic. I say more, because
> > they're not fully deterministic yet. I have some patches lining up for
> the
> > 3.5 interpreter to improvement determinism there as well.
> >
> > In any case, I would like to merge this as soon as possible. Any
> objections?
> >
> > Freddy
>
> Hi, I'm not really an active contributor here but I follow NixOS because
> I think it's a really cool project.
>
> Am I to understand that NixOS is considering replacing the python2
> interpreter with the python3 interpreter, as arch did?
>
> If that's the case it may be worth considering this pep[1] first.
>
> If I've misunderstood and there's no intention to symlink 'python' to
> 'python3'
> then please ignore this. :)
>
> [1]: http://legacy.python.org/dev/peps/pep-0394/#id5
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python 3 as default

[Freddy Rietdijk]

The branch with fixes for reproducible builds can be found at
https://github.com/NixOS/nixpkgs/pull/22585.
Note that neither the build of the 2.7 nor the 3.5 interpreter is
reproducible afterwards. In the case of the 2.7 there's 4 files that just
refuse to use the fixed timestamp in their bytecode, and in case of 3.5 all
timestamps are deterministic, but code using sets is not.

On Mon, Feb 13, 2017 at 10:34 AM, Jörg Thalheim <jo...@higgsboson.tk> wrote:

> No objections from my side. Do you have a public WIP branch of your
> commits to make python 3 reproducible?
>
>
> On 2017-02-13 10:08, Freddy Rietdijk wrote:
> > Hi,
> >
> > As you may be aware the plan is to change to Python 3.5 as default for
> 17.03. There were a couple of things that had to be done, and I think we're
> ready now to make the change. Packages that require Python 2 are now pinned
> to do so.
> >
> > Now, obviously I haven't been able to test all packages myself, but
> judging from what Hydra reports and from working from the `python3` branch
> I don't expect any major issues.
> >
> > The Python 2.7 interpreter had quite some patches, among one to make the
> interpreter and Python packages more deterministic. I say more, because
> they're not fully deterministic yet. I have some patches lining up for the
> 3.5 interpreter to improvement determinism there as well.
> >
> > In any case, I would like to merge this as soon as possible. Any
> objections?
> >
> > Freddy
> >
> >
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Python 3 as default

[Freddy Rietdijk]

Hi,

As you may be aware the plan is to change to Python 3.5 as default for
17.03. There were a couple of things that had to be done, and I think we're
ready now to make the change. Packages that require Python 2 are now pinned
to do so.

Now, obviously I haven't been able to test all packages myself, but judging
from what Hydra reports and from working from the `python3` branch I don't
expect any major issues.

The Python 2.7 interpreter had quite some patches, among one to make the
interpreter and Python packages more deterministic. I say more, because
they're not fully deterministic yet. I have some patches lining up for the
3.5 interpreter to improvement determinism there as well.

In any case, I would like to merge this as soon as possible. Any objections?

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Why having releases if you break things in it often

[Freddy Rietdijk]

> So except you want security updates you dont have to update your system? I
think automated tests could fix that...
>
> something like systemctl status flexget | grep running or something like
that.

Packages and modules preferably have tests. There were a couple of packages
broken because of the update, and if I recall correctly none had tests...

On Mon, Jan 23, 2017 at 4:09 PM, Stefan Huchler <stefan.huch...@mail.de>
wrote:

> Hello Freddy,
>
> yes I think that html5lib thing would it be. So it was at least a
> security fix, so you dont just update stuff to update it, which would
> make releases pretty useless concept :)
>
> So except you want security updates you dont have to update your system?
> I think automated tests could fix that...
>
> something like systemctl status flexget | grep running or something like
> that.
>
> Of course you cant write a test for every cornercase, but that bug seems
> pretty obvious and easy to reproduce (install/upgrade flexget).
>
> Sorry I formulated that message a bit trollish, but just wanted to learn
> why how releases are done in nixos.
>
> Also a hint why list-derivations and boot options in grub dont are the
> same would be interesting? Maybe when I run gc or optimise they vanish
> from grub?
>
> Stefan
>
>
> Freddy Rietdijk <freddyrietd...@fridh.nl> writes:
>
> > Hi Stefan,
> >
> > Regarding flexget. There were some security issues with an (indirect)
> dependency, html5lib, and thus
> > html5lib was upgraded. Maybe that broke flexget, I don't know.
> >
> > The main issue is just a lack of maintainers. It's relatively
> straightforward to add a package to Nixpkgs, but
> > maintaining a package set this size that also keeps growing is becoming
> increasingly harder.
> >
> > Freddy
> >
> > On Mon, Jan 23, 2017 at 10:07 AM, Stefan Huchler <stefan.huch...@mail.de>
> wrote:
> >
> >  So because I dont need always newest versions on all of my boxes, I
> >  selected the 16.xx chhannel.
> >
> >  There are here and there some minor issues as example kodi here and
> >  there crashes maybe 1-3 times a week. Could be extentions or something.
> >
> >  For that and other reasons I update here and there all few weeks maybe
> >  the maschine.
> >
> >  So one advantage of course is that if I notice that something does not
> >  work I can boot a old configuration, so I dont have to deal with some
> >  updates that broke stuff or rollback.
> >
> >  But I wonder how you can break relativly often stuff (at the moment
> >  there seems to be a python dependency problem with flexget, that makes
> >  the daemon crash), in a "stable" release channel.
> >
> >  I mean if I use debian, and stick to my "channel"/release, normaly
> >  nothing breaks, as long as I use only their package installer, pip
> >  updates of course broke stuff. If I use fedora, well I get maybe some
> >  upstream changes like new kernel versions, but normaly they brake also
> >  nothing.
> >
> >  So if "stable" channel makes updates that are not needed (the older
> >  version of flexget works fine), whats the point or the criterias of
> >  those releases? I could then just use the newest version, if I have to
> >  relay on rollback / boot old versions anyway, I dont really see the
> >  point of "stable" channels.
> >
> >  I had pretty good experiences with using the rolling channel, but had
> >  many times break stuff in the stable channel.
> >
> >  Also the tools around generations / boot-generations is very confusing,
> >  why do I have 3 4 options in the nix-env --list-generation overview but
> >  20 in the boot menu.
> >
> >  But thats a 2nd different issue I guess.
> >
> >  Just wonder what your policies are.
> >
> >  Other stuff that broke on me in the past, was latex packages as example.
> >
> >  ___
> >  nix-dev mailing list
> >  nix-dev@lists.science.uu.nl
> >  http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Why having releases if you break things in it often

[Freddy Rietdijk]

Hi Stefan,

Regarding flexget. There were some security issues with an (indirect)
dependency, html5lib, and thus html5lib was upgraded. Maybe that broke
flexget, I don't know.

The main issue is just a lack of maintainers. It's relatively
straightforward to add a package to Nixpkgs, but maintaining a package set
this size that also keeps growing is becoming increasingly harder.

Freddy

On Mon, Jan 23, 2017 at 10:07 AM, Stefan Huchler 
wrote:

> So because I dont need always newest versions on all of my boxes, I
> selected the 16.xx chhannel.
>
> There are here and there some minor issues as example kodi here and
> there crashes maybe 1-3 times a week. Could be extentions or something.
>
> For that and other reasons I update here and there all few weeks maybe
> the maschine.
>
> So one advantage of course is that if I notice that something does not
> work I can boot a old configuration, so I dont have to deal with some
> updates that broke stuff or rollback.
>
> But I wonder how you can break relativly often stuff (at the moment
> there seems to be a python dependency problem with flexget, that makes
> the daemon crash), in a "stable" release channel.
>
> I mean if I use debian, and stick to my "channel"/release, normaly
> nothing breaks, as long as I use only their package installer, pip
> updates of course broke stuff. If I use fedora, well I get maybe some
> upstream changes like new kernel versions, but normaly they brake also
> nothing.
>
> So if "stable" channel makes updates that are not needed (the older
> version of flexget works fine), whats the point or the criterias of
> those releases? I could then just use the newest version, if I have to
> relay on rollback / boot old versions anyway, I dont really see the
> point of "stable" channels.
>
> I had pretty good experiences with using the rolling channel, but had
> many times break stuff in the stable channel.
>
> Also the tools around generations / boot-generations is very confusing,
> why do I have 3 4 options in the nix-env --list-generation overview but
> 20 in the boot menu.
>
> But thats a 2nd different issue I guess.
>
> Just wonder what your policies are.
>
> Other stuff that broke on me in the past, was latex packages as example.
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Channels, LetsEncrypt, and Security fixes from 2016-12-23 01:26 UTC

[Freddy Rietdijk]

Those extra failures seem to be transients. At least, the one that I
tested. It is also very unlikely that the only change, a Python 2.7
maintenance version bump, would cause those extra failures.

Building 16.09 after the Python bump went fast. There is still a large
queue (nixpkgs:trunk/nixos:trunk-combined) but that should be solved in my
opinion by reconsidering the jobsets so that nixpkgs:trunk and
nixos:trunk-combined are synchronized.



On Sun, Dec 25, 2016 at 6:25 PM, zimbatm  wrote:

> Checking Hydra as well. Exim has been built but both the large and small
> channels are stuck.
>
> Unfortunately there are 130 (!!!) failing jobs on http://hydra.nixos.org/
> jobset/nixos/release-16.09 . How did we get there?
>
> The small build was blocked by a *new* package that pointed a missing
> file, I reverted that change and the channel should advance in the next
> hour unless somebody else is being reckless again.
>
> We really need pierron's security framework so the security releases can
> be pushed regardless of the branch's state. Or prevent users from pushing
> to master unless the builds are passing.
>
> On Sun, 25 Dec 2016 at 13:10 Graham Christensen 
> wrote:
>
> zimbatm  writes:
>
> > Thanks Graham.
> >
> > I pushed the Exim updates for CVE-2016-9963 as well.
> >
> > master: 352e167c224: exim: 4.87 -> 4.88 for CVE-2016-9963
> > release-16.09: d6bff30c96ed6: exim: 4.87 -> 4.87.1 for CVE-2016-9963
>
>
> Thank you so much! I have sent notice to the security list. Checking
> Hydra, this channel update should be out very soon.
>
> Graham
>
> --
> You received this message because you are subscribed to the Google Groups
> "nix-security-announce" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nix-security-announce+unsubscr...@googlegroups.com.
> To post to this group, send email to nix-security-announce@
> googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/nix-security-announce/87zijkyx1v.fsf%40grahamc.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Using/packaging Python-modules

[Freddy Rietdijk]

Hi Roland,

I've fixed the issues you described with
https://github.com/NixOS/nixpkgs/commit/e0d1691ba3ceffe51efb0dfde285a3e29b0002fe

I couldn't find
>   - http://nixos.org/nixpkgs/manual/#ssec-python-development
>This anchor des not exist; maybe #development-mode is meant.
Could you tell me where you found this?

There are some issues with the links because this part of the docs is
written in Markdown and converted with Pandoc to Docbook. The Markdown part
includes some internal references but because Markdown does not support
that they are instead references to the docs online.


Freddy






On Wed, Dec 21, 2016 at 3:18 PM, Roland Koebler <
rk-l...@simple-is-better.org> wrote:

> Hi,
>
> I'm trying to create nix-packages from some of my own Python-modules.
> So, I followed the Nixpkgs manual, but it did not work as described:
>
> Section 8.10.1.2.1. Python packaging on Nix:
>
> - 1st shell.nix-example: local toolz package [1]
>
>   1. variable "licenses" is not found:
>
>  user@host:~/tmp/_$ nix-shell
>  error: undefined variable ‘licenses’ at /home/rk/tmp/_/shell.nix:15:15
>  (use ‘--show-trace’ to show detailed location information)
>
>   2. after removing the license-line, I can start a nix-shell, but the
> toolz
>  package is not available:
>
>  user@host:~/tmp$ nix-shell
>  ...
>  [nix-shell:~/tmp]$ which python3.5
>  /nix/store/c5c9fs9vh12rr0lvbss1a813xwlwv0
> zh-python3-3.5.2/bin/python3.5
>
>  [nix-shell:~/tmp]$ python3.5
>  Python 3.5.2 (default, Jun 25 2016, 21:38:40)
>  [GCC 5.4.0] on linux
>  Type "help", "copyright", "credits" or "license" for more information.
>  >>> import toolz
>  Traceback (most recent call last):
>File "", line 1, in 
>  ImportError: No module named 'toolz'
>
> - 2nd shell.nix-examle: local toolz package + numpy [2]
>
>   1. variable "licenses" is not found (as above)
>   2. tests fail:
>
> ...
> 
> ==
> ERROR: exceptions (unittest.loader._FailedTest)
> 
> --
> AttributeError: module 'toolz.curried' has no attribute
> 'exceptions'
>
> 
> ==
> ERROR: exceptions (unittest.loader._FailedTest)
> 
> --
> AttributeError: module 'toolz.curried' has no attribute
> 'exceptions'
>
> 
> --
> Ran 2 tests in 0.001s
>
> FAILED (errors=2)
> builder for 
> ‘/nix/store/pa77cfky1b0f2knzn2k2czdaxz6wdym6-python3.5-toolz-0.7.4.drv’
> failed with exit code 1
> cannot build derivation 
> ‘/nix/store/pi89rb0v178avb0jjdm352vad42a1f7a-python3-3.5.2-env.drv’:
> 1 dependencies couldn't be built
> error: build of 
> ‘/nix/store/pi89rb0v178avb0jjdm352vad42a1f7a-python3-3.5.2-env.drv’
> failed
> /home/user/.nix-profile/bin/nix-shell: failed to build all
> dependencies
>
>
> So, how can I develop with local Python packages or package local Python
> modules?
>
>
> Additionally, I've found several broken links:
>   I found several broken links:
>   - buildPythonPackage https://github.com/NixOS/nixpkgs/blob/master/pkgs/
> development/python-modules/generic/default.nix
>   - http://pythonhosted.org/setuptools/setuptools.html#development-mode
> should probably point to http://setuptools.readthedocs.
> io/en/latest/setuptools.html#development-mode
>   - http://nixos.org/nixpkgs/manual/#ssec-python-development
> This anchor des not exist; maybe #development-mode is meant.
>   - A link-checker (https://validator.w3.org/checklink) reported even
> more broken links.
>
> best regards
> Roland
>
>
> [1]
> with import  {};
>
> pkgs.python35Packages.buildPythonPackage rec {
>   name = "toolz-${version}";
>   version = "0.7.4";
>
>   src = pkgs.fetchurl{
> url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
> sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed0
> 5afd";
>   };
>
>   meta = {
> homepage = "http://github.com/pytoolz/toolz/;;
> description = "List processing tools and functional utilities";
> license = licenses.bsd3;
> maintainers = with maintainers; [ fridh ];
>   };
> }
>
>
> [2]
> with import  {};
>
> ( let
> toolz = pkgs.python35Packages.buildPythonPackage rec {
>   name = "toolz-${version}";
>   version = "0.7.4";
>
>   src = pkgs.fetchurl{
> url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
> sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc8
> 2f7db8e13378be7c78d6c14a5f8ed05afd";
>   };
>
>   meta = {
> homepage = "http://github.com/pytoolz/toolz/;;
> description = "List processing tools and functional utilities";
> license = licenses.bsd3;
> 

Re: [Nix-dev] Reproducibility testing in Hydra

[Freddy Rietdijk]

That is a very nice improvement, thanks!

Is there a command line tool for checking whether a derivation is
reproducible? And when requested generate the diffoscoe output. That would
be very convenient to have!

Regarding pip. I think we need to use the --build flag to explicitly set a
build location.
http://pip.readthedocs.io/en/stable/reference/pip_install/#cmdoption-b
Testing right now.





On Tue, Dec 13, 2016 at 2:15 PM, stewart mackenzie 
wrote:

> Massive achievement! Congrats to all involved!
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Extra dependencies for Steam games?

[Freddy Rietdijk]

Are you on NixOS? If so, did you add

hardware.opengl = { driSupport = true; driSupport32Bit = true; };

On Mon, Dec 12, 2016 at 9:28 PM, Mike Cooper  wrote:

> Thanks for the replies everyone. As for the meta issue, I think that
> having a steam package that is compatible with all games is a good idea,
> instead of requiring overrides.
>
> Unfortunately, it didn't seem to have worked. I tried adding mesa, which
> seems to contain the missing library) to each of commonTargetPkgs and
> multipkgs in turn, but neither makes the error when launching the game go
> away.
>
> To try and debug this further, I ran steam through strace and saw these
> lines:
>
> [pid  2901] 
> open("/nix/store/16vmqiwq4jpcbsfqm4vn48p7wxzgkpbv-acl-2.2.52/lib/libc.so.6",
> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> [pid  2901] 
> open("/nix/store/m7a3k9cgci9mz114c8497f89w28c5dzn-glibc-2.24/lib/libc.so.6",
> O_RDONLY|O_CLOEXEC) = 3
> [pid  2901] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\
> 0\0\3\0>\0\1\0\0\0\0\4\2\0\0\0\0\0"..., 832) = 832
> [pid  2901] fstat(3, {st_mode=S_IFREG|0555, st_size=1908040, ...}) = 0
> [pid  2901] mmap(NULL, 3791104, PROT_READ|PROT_EXEC,
> MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fef3778e000
> [pid  2901] mprotect(0x7fef37923000, 2093056, PROT_NONE) = 0
> [pid  2901] mmap(0x7fef37b22000, 24576, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x194000) = 0x7fef37b22000
> [pid  2901] mmap(0x7fef37b28000, 14592, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fef37b28000
> [pid  2901] close(3)= 0
> [pid  2901] open("/nix/store/m7a3k9cgci9mz114c8497f89w28c5d
> zn-glibc-2.24/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such
> file or directory)
> [pid  2901] 
> open("/nix/store/m7a3k9cgci9mz114c8497f89w28c5dzn-glibc-2.24/lib/libGL.so.1",
> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
> [pid  2901] writev(2, [{iov_base="/usr/bin/uname", iov_len=14},
> {iov_base=": ", iov_len=2}, {iov_base="error while loading shared
> libra"..., iov_len=36}, {iov_base=": ", iov_len=2}, {iov_base="libGL.so.1",
> iov_len=10}, {iov_base=": ", iov_len=2}, {iov_base="cannot open shared
> object file", iov_len=30}, {iov_base=": ", iov_len=2}, {iov_base="No such
> file or directory", iov_len=25}, {iov_base="\n", iov_len=1}], 10) = 124
> [pid  2646] <... read resumed> "/usr/bin/uname: error while load"...,
> 8192) = 124
> [pid  2901] exit_group(127) = ?
> [pid  2646] write(1, "/usr/bin/uname: error while load"...,
> 124/usr/bin/uname: error while loading shared libraries: libGL.so.1: cannot
> open shared object file: No such file or directory
> ) = 124
> [pid  2646] write(3, "/usr/bin/uname: error while load"..., 124
> 
> [pid  2901] +++ exited with 127 +++
>
> It looks to me that it is looking in the wrong place for the library,
> looking in the lib directory of glibc, not mesa.
>
> Any ideas?
>
> On Fri, Dec 9, 2016 at 12:46 AM Kevin Cox  wrote:
>
>> On 09/12/16 10:45, Nikolay Amiantov wrote:
>> > To clarify: I was talking not about Steam Runtime (the guaranteed set)
>> > but rather extra libraries that some games need (a small set defined in
>> > `multiPkgs`, under "Not formally in runtime but needed by some games"
>> > comment). Removing something from the runtime seems a bad idea to me.
>> Oh I see. I didn't realize games did that. It seems like it would cause
>> a lot of problems.
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Extra dependencies for Steam games?

[Freddy Rietdijk]

If you would move certain libraries out because of whatever reason, then I
think we need to introduce a set that defines per game the extra libraries
and introduce something like a steam.withGameLibs.

On Fri, Dec 9, 2016 at 11:07 AM, Nikolay Amiantov  wrote:

> commonTargetPkgs is not the right place for this: it's for binaries (not
> libraries) that are needed both for steam-env and steam. Difference
> between targetPkgs (binaries) and multiPkgs (libraries) is that latter
> are installed for both architectures on 64-bit systems; you'd need that
> for a Steam game most likely as a lot of them are compiled for i686.
>
> On 12/09/2016 01:04 PM, Moritz Ulrich wrote:
> >
> > I think the easiest / most useful solution is adding the libraries to
> > `commonTargetPkgs` in `pkgs/games/steam/chrootenv.nix`. You can do this
> > locally by cloning `nixpkgs.git`, modify the file, then running `nix-env
> > -f . -iA steam` inside the `nixpkgs` directory to install steam in your
> > user-env (which takes priority over `environment.systemPackages`).
> >
> > Feel free to open a Pull Request (or an issue describing the missing
> > libraries) for changes like this. Right now a "fat" steam chrootenv
> > seems like the most useful solution to me.
> >
> > Mike Cooper  writes:
> >
> >> The Steam game SHENZHEN I/O fials to start with errors about missing
> >> libraries. One solution is to symlink the missing libraries into the
> >> directory of the game, but that doesn't seem like a very good idea.
> >>
> >> I tried to add an override for steam to add extra dependencies, but I
> was
> >> able to make any headway on this. Does anyone have a good way of getting
> >> Steam games that need extra libraries to work?
> >>
> >> -Mythmon
> >> ___
> >> nix-dev mailing list
> >> nix-dev@lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
>
>
> --
> Nikolay.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Multiple outputs: is a first output necessary?

[Freddy Rietdijk]

Hi,

I'm using Nix and now also multiple outputs for my data analysis and
simulations. Many scripts generate several files and instead of having them
together in the same store path I would prefer to put them in separate
store paths.

I'm not using any of the common outputs like $out and $bin. The multiple
outputs feature seems to require that $out is used. Therefore I now just
put a single file in $out. What is the reason this is needed? I saw in the
documentation that leftover files are typically put in $out.

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Python package set now uses fixed-point combinator

[Freddy Rietdijk]

Hi,

This is to inform those of you that override Python package sets. A
fixed-point combinator is now used, and examples are included in the
documentation.
https://github.com/NixOS/nixpkgs/pull/20893

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] pypi2nix: generate full PYTHONPATH from requirements?

[Freddy Rietdijk]

Check e.g. the mopidy service
https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/audio/mopidy.nix

Instead of `python.withPackages` `python.buildEnv.override{extralibs=...;}`
is used. `withPackages` is just a wrapper around `buildEnv`.

On Sat, Nov 5, 2016 at 10:09 PM, Daniel Hlynskyi <abcz2.upr...@gmail.com>
wrote:

> pypi2nix supports `python.withPackages`. But how should I use the created
> "environment" in NixOS service?
>
> path = [ python-env ];
>
> doesn't work
>
> 2016-11-05 22:59 GMT+02:00 Freddy Rietdijk <freddyrietd...@fridh.nl>:
>
>> I am not familiar with the details of pypi2nix, but try and use
>> `python.withPackages` to create an environment where all packages can be
>> found by the interpreter. PYTHONPATH is still used a lot, especially with
>> modules, but we shouldn't be using it.
>>
>> http://nixos.org/nixpkgs/manual/#python
>>
>> On Sat, Nov 5, 2016 at 9:50 PM, Daniel Hlynskyi <abcz2.upr...@gmail.com>
>> wrote:
>>
>>> I'm trying to use pypi2nix-generated expression to create Kallithea
>>> service.
>>>
>>>  systemd.services.kalithea = {
>>>   after = [ "network.target" ];
>>>   wantedBy = [ "multi-user.target" ];
>>>   script =
>>>   let ppath =
>>> concatMapStringsSep ":" (x: 
>>> "${x}/${python.interpreter.python.sitePackages}")
>>> (let self = python.packages; in [
>>>   # package
>>>   self."Kallithea"
>>>   # deps
>>>   self."Babel"
>>>   self."Beaker"
>>>   self."FormEncode"
>>>   self."Mako"
>>>   self."Markdown"
>>>   self."Pygments"
>>>   self."Pylons"
>>>   self."Routes"
>>>   self."SQLAlchemy"
>>>   self."URLObject"
>>>   self."WebHelpers"
>>>   self."WebOb"
>>>   self."WebTest"
>>>   self."Whoosh"
>>>   self."celery"
>>>   self."docutils"
>>>   self."dulwich"
>>>   self."mercurial"
>>>   self."mock"
>>>   self."py-bcrypt"
>>>   self."python-dateutil"
>>>   self."waitress"
>>>   # second level deps
>>>   self."MarkupSafe"
>>>   self."funcsigs"
>>>   self."pbr"
>>>   self."six"
>>>   # TODO: add all the others
>>> ]);
>>>   in ''
>>> export PYTHONPATH=${ppath}
>>> ${pythonPackages.pasteScript}/bin/paster serve
>>> ${kallithea1}/etc/development.ini;
>>>   '';
>>> };
>>>
>>> So I need to feed PYTHONPATH into service, but I don't know a nice way
>>> to recursively aggregate all the propagatedBuildInputs from Kallithea into
>>> full PYTHONPATH
>>> (and service wants all of them, as far as I understand)
>>>
>>> So, is there a way to do this? I'm pretty new to Python infrastructure
>>> and perhaps miss some obvious thing.
>>>
>>> PS. Pinging my previous ask about PERL5LIB, which is an example of exact
>>> this problem, but with Perl.
>>>
>>> https://www.mail-archive.com/nix-dev@lists.science.uu.nl/msg22577.html
>>>
>>> The solution I found breaks on Travis (I guess, it is of kind "import
>>> from derivation", so it won't work generally)
>>>
>>> ___
>>> nix-dev mailing list
>>> nix-dev@lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>
>>>
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] pypi2nix: generate full PYTHONPATH from requirements?

[Freddy Rietdijk]

I am not familiar with the details of pypi2nix, but try and use
`python.withPackages` to create an environment where all packages can be
found by the interpreter. PYTHONPATH is still used a lot, especially with
modules, but we shouldn't be using it.

http://nixos.org/nixpkgs/manual/#python

On Sat, Nov 5, 2016 at 9:50 PM, Daniel Hlynskyi 
wrote:

> I'm trying to use pypi2nix-generated expression to create Kallithea
> service.
>
>  systemd.services.kalithea = {
>   after = [ "network.target" ];
>   wantedBy = [ "multi-user.target" ];
>   script =
>   let ppath =
> concatMapStringsSep ":" (x: 
> "${x}/${python.interpreter.python.sitePackages}")
> (let self = python.packages; in [
>   # package
>   self."Kallithea"
>   # deps
>   self."Babel"
>   self."Beaker"
>   self."FormEncode"
>   self."Mako"
>   self."Markdown"
>   self."Pygments"
>   self."Pylons"
>   self."Routes"
>   self."SQLAlchemy"
>   self."URLObject"
>   self."WebHelpers"
>   self."WebOb"
>   self."WebTest"
>   self."Whoosh"
>   self."celery"
>   self."docutils"
>   self."dulwich"
>   self."mercurial"
>   self."mock"
>   self."py-bcrypt"
>   self."python-dateutil"
>   self."waitress"
>   # second level deps
>   self."MarkupSafe"
>   self."funcsigs"
>   self."pbr"
>   self."six"
>   # TODO: add all the others
> ]);
>   in ''
> export PYTHONPATH=${ppath}
> ${pythonPackages.pasteScript}/bin/paster serve
> ${kallithea1}/etc/development.ini;
>   '';
> };
>
> So I need to feed PYTHONPATH into service, but I don't know a nice way to
> recursively aggregate all the propagatedBuildInputs from Kallithea into
> full PYTHONPATH
> (and service wants all of them, as far as I understand)
>
> So, is there a way to do this? I'm pretty new to Python infrastructure and
> perhaps miss some obvious thing.
>
> PS. Pinging my previous ask about PERL5LIB, which is an example of exact
> this problem, but with Perl.
>
> https://www.mail-archive.com/nix-dev@lists.science.uu.nl/msg22577.html
>
> The solution I found breaks on Travis (I guess, it is of kind "import from
> derivation", so it won't work generally)
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Bot that replies to PR with review points based on files that were touched

[Freddy Rietdijk]

Hi,

Currently we have the mention-bot that checks which files PR modifies, and
then mentions users that have modified those files before or are explicitly
watching those files. These contributors can then check the PR and maybe
come up with specific questions regarding the PR.

I'm looking for a bot that checks which files are touched, and that gives a
pre-configured message depending on the files that are touched. E.g., when
Python files are touched I would like the bot to reply to the PR with a
message that includes an additional checklist with questions like whether
the modifications were tested with all interpreters, and whether tests are
run.

Such a bot I think would be extremely useful since, even though our manual
is getting more extensive and includes a lot of points that contributors
should consider, that is also the problem; it becomes overwhelming. Both
new as well as longtime contributors will miss points that should be
checked.

Do you happen to know such a bot? If so, please let me know. Otherwise, I
think we will have to build one or maybe extend the mention-bot with this
feature.

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Python: getting rid of PYTHONPATH in Nixpkgs

[Freddy Rietdijk]

Hi,

Currently we use PYTHONPATH
 a lot in
Nixpkgs to let applications and the interpreter find Python modules. This
typically works fine, but there are problems with this method and so I
would like to get rid of it and use only `python.withPackages` which uses
`python.buildEnv`.

The two main issues with the use of PYTHONPATH:

   1. Before we used `--prefix $PYTHONPATH`, but this would leak PYTHONPATH
   into subprocesses, which is especially problematic when both parent and
   child depend on Python but of different versions.  `--set $PYTHONPATH`
   would solve that issue, but that makes it impossible to add other (impure)
   paths, which is an important feature. This also breaks alternative shells
   like `ipython`. This issue is currently solved by extending `sys.path` in
   the Python applications.
   2. Limits the use of multiple outputs. When moving a module of a package
   into a separate output it becomes problematic to load this again, since
   just adding the module to PYTHONPATH typically doesn't work because the
   order matters. While Python modules are typically small, and build fast,
   rebuilding can take a lot of time in cases like `matplotlib` which supports
   multiple backends and is depended on by quite some packages.

A method that is more reliable is building an environment with symbolic
links to all the modules, and wrapping the applications with a wrapper that
sets PYTHONHOME
. This is
exactly what `python.buildEnv` does, and it solves both 1) and 2).

`PYTHONPATH` is mainly constructed with the Python interpreter setupHook.
It is used in `buildPythonPackage` for building the package, and after
installing it is extended so the tests can run. Furthermore, `PYTHONPATH`
is set by the `setupHook` when using `nix-shell` like `nix-shell -p
pythonPackages.numpy`.

I think we can get rid of the setupHook. For the building we can create an
env. This would be able to support multiple outputs as inputs, but will be
more expensive than setting PYTHONPATH. For the tests we do add the newly
constructed package to PYTHONPATH; there's no way around it and it doesn't
cause any problems either.

The biggest impact will be on how `nix-shell` is used. It won't be possible
anymore to use it as shown before, instead `nix-shell -p
'python.withPackages(ps:[ps.numpy])'` would have to be used. While it is
possible to keep the setupHook (or use it as a shellHook) it will be
unreliable in the case of multiple outputs.

What do you think about this change? Do you see any problems with it? Any
other suggestions?

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Nix with fetchTarball that supports hash

[Freddy Rietdijk]

Hi,

On July the 26th support was added to `fetchTarball` for a `sha256`
attribute [1]. Since that commit, it is also possible to use `fetchTarball`
with the hash in restricted mode, thereby making it possible to load e.g.
an external repository with urls and hashes.

`nixUnstable` currently refers to a build from March 24th [1] and therefore
doesn't include this change yet. When could we expect a `nixUnstable` that
has this change, and when about would that become `nixStable`? Because of
the impact this change can have on automating version updates it would be
nice to get this in as soon as possible.

[1]
https://github.com/NixOS/nix/commit/06bbfb6004942bfcddd930e746ee7a2bfe5c3872
[2]
https://github.com/NixOS/nix/commit/3b81b2645706d7f8d9c0ec228426dee8ef1fc7ac

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Python: a bit fatter interpreter

[Freddy Rietdijk]

> What's the impact on the Python closure size?

The closure size of `python27` went up from 79 MB at 16.09 to 107 MB on
stable, and that of `python35` down from 138 MB to 120 MB (because
`tkinter` was moved out). Therefore, when we make the switch to Python 3
the closure size will go up a net 41 MB while at the same time the amount
of headaches will be reduced significantly.

On Wed, Oct 26, 2016 at 5:43 PM, Eelco Dolstra <eelco.dols...@logicblox.com>
wrote:

> Hi,
>
> On 10/26/2016 04:36 PM, Freddy Rietdijk wrote:
>
> > Today I merged staging into master, and this brings some changes to
> CPython.
> > Of importance is that the different versions of the interpreter now come
> with
> > the whole standard library, except for tkinter. This module can be found
> at
> > pythonPackages.tkinter. Therefore, if you get evaluation errors that the
> > attributes readline or sqlite3 or one of the others is missing, then
> this is why.
>
> What's the impact on the Python closure size?
>
> --
> Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Python: a bit fatter interpreter

[Freddy Rietdijk]

Hi,

Today I merged staging into master, and this brings some changes to CPython.
Of importance is that the different versions of the interpreter now come
with the whole standard library, except for tkinter. This module can be
found at pythonPackages.tkinter. Therefore, if you get evaluation errors
that the attributes readline or sqlite3 or one of the others is missing,
then this is why.

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Eval breakage in master

[Freddy Rietdijk]

> error: while querying the derivation named ‘gnome-encfs-manager-1.8.16’:

I've marked this package as broken for now.

On Wed, Oct 26, 2016 at 3:44 PM, Bjørn Forsman <bjorn.fors...@gmail.com>
wrote:

> On 26 October 2016 at 15:03, Freddy Rietdijk <freddyrietd...@fridh.nl>
> wrote:
> > Hi,
> >
> > I am looking into this. The error was already on staging. The problem is
> > line 1390 in all-packages.nix. Something likely gone wrong with a merge.
>
> I see you fixed it. Thanks.
>
> However, there is more eval breakage on master:
>
> $ nix-env -f . -qa '*' --meta --xml --drv-path --show-trace >/dev/null
> [...]
> error: while querying the derivation named ‘gnome-encfs-manager-1.8.16’:
> while evaluating the attribute ‘nativeBuildInputs’ of the derivation
> ‘gnome-encfs-manager-1.8.16’ at
> /home/bfo/proj/code/forks/nixpkgs3/pkgs/tools/security/
> gencfsm/default.nix:6:3:
> while evaluating ‘getOutput’ at
> /home/bfo/proj/code/forks/nixpkgs3/lib/attrsets.nix:453:23, called
> from undefined position:
> attribute ‘libgee_1’ missing, at
> /home/bfo/proj/code/forks/nixpkgs3/pkgs/tools/security/
> gencfsm/default.nix:14:27
>
> Best regards,
> Bjørn Forsman
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Eval breakage in master

[Freddy Rietdijk]

Hi,

I am looking into this. The error was already on staging. The problem is
line 1390 in all-packages.nix. Something likely gone wrong with a merge.

Freddy

On Wed, Oct 26, 2016 at 3:00 PM, Bjørn Forsman 
wrote:

> Hi all,
>
> $ nix-env -f . -qa '*' --meta --xml --drv-path --show-trace >/dev/null
> [...]
> error: while querying the derivation named ‘diffoscope-52’:
> while evaluating the attribute ‘propagatedNativeBuildInputs’ of the
> derivation ‘diffoscope-52’ at
> /home/bfo/proj/code/forks/nixpkgs3/pkgs/development/
> interpreters/python/mk-python-derivation.nix:56:3:
> cannot coerce a function to a string, at
> /home/bfo/proj/code/forks/nixpkgs3/pkgs/development/
> interpreters/python/mk-python-derivation.nix:56:3
>
> Best regards,
> Bjørn Forsman
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] buildFHSUserEnv and matlab impure install

[Freddy Rietdijk]

I guess the approach here is

   1. Make sure you have all the installation files offline in an archive.
   2. Prepare file for silent installation
   https://sysadminblogger.wordpress.com/tag/matlab-offline-installation/
   3. Write a Nix expression that uses `buildFHSUserEnv` and use
   `requireFile` with the name and hash of the archive
   4. Find out all the packages you need in the env


On Thu, Oct 20, 2016 at 2:20 PM, Ruben Astudillo 
wrote:

> Hello Nix'ers
>
> I recently got a copy of (unix) matlab. The usual method of install
> includes running as root a install script, this will launch a popup
> dialog which you can "next next" until the installation is
> finished. Matlab assumes in various part a FHS structure, so I've meant
> to not bother with patching and use buildFHSUserEnv to replicate that
> hierarchy. I can launch a nix-shell such structure, yet I have doubts on
> how to proceed, should I
>
>   * Try writing a default.nix for matlab, and somehow fill the
> interactive parts on a script?
>   * Try to install all the parts on a folder under $HOME, such that each
> time I want to use it, I must be in the correct nix-shell?
>
> As I didn't care much, I wanted to follow the 2ª option, yet trying to
> run with sudo gets the following error
>
> sudo: setuid-wrapper.c:47: main: Assertion `(st.st_mode & S_ISUID)
> == 0 || (st.st_uid == geteuid())' failed
>
> Thus I think this isn't how is supposed to be used. Anybody with an idea
> on how to use the second method? Ideally I would like to this to work,
> so I can `nix-build` such hierarchy and use matlab from there...
>
> -- Ruben
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Remove Python 2.6 and 3.3

[Freddy Rietdijk]

This is about the old Python 2.6, `python26`. The default version in
Nixpkgs is Python 2.7. Python 2.6 has been broken for some time, and hasn't
been used in Nixpkgs for a long time either. Therefore, if you use Python
2.6, you will know, because you're doing so explicitly.

On Mon, Sep 26, 2016 at 1:29 PM, Matthias Beyer <m...@beyermatthias.de>
wrote:

> I have 702 paths in /nix/store which contain the string "python2"
>
> Can you tell me how to check properly whether I need python2
> support/packages? I'm not so familiar with the python infrastructure.
>
> On 24-09-2016 10:15:12, Freddy Rietdijk wrote:
> > Hi Nixers,
> >
> > I've just opened a PR to remove Python 2.6 and 3.3 from Nixpkgs. If you
> > object, say because you still use them, then please drop a comment on the
> > PR.
> > https://github.com/NixOS/nixpkgs/pull/18917
> >
> > Freddy
>
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
> --
> Mit freundlichen Grüßen,
> Kind regards,
> Matthias Beyer
>
> Proudly sent with mutt.
> Happily signed with gnupg.
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Limiting access to only maintained packages and ensuring core packages are maintained

[Freddy Rietdijk]

We could let the mention-bot ignore certain files

  "fileBlacklist": ["*.md"], // mention-bot will ignore any files that
match these file globs

On Sat, Sep 3, 2016 at 1:03 PM, Shea Levy  wrote:

> No, not something automated like that. Though mention-bot seems to work
> decently well, it doesn't know for example that changes to
> all-packages.nix are much less useful than changes to a default.nix file
> in a package directory. A human can know that, and additionally look at
> the log messages to see the nature of the changes to see if they're
> likely to be relevant.
>
> Bardur Arantsson  writes:
>
> > On 2016-09-02 23:16, Shea Levy wrote:
> >> Why can't people use the commit logs to see who is knowledgeable?
> >
> > Are you thinking of something like https://github.com/facebook/
> mention-bot ?
> >
> > This fails in the case where someone does a big cross-cutting (i.e. not
> > concerned with particular packages) refactor which happens to touch a
> > lot of code. Suddenly you'd get a huge number of notifications about
> > *every package under the sun*.
> >
> > (Why, yes, I *have* been the 'victim' of this type of thing in a GitHub
> > repo where the project uses mention-bot. Now, some of that may simply be
> > due to the algorithm that mention-bot uses, but it's pretty hard to
> > avoid unless you're *somehow* able to algorithmically distinguish
> > large-scale cross-cutting refactors from actual package maintenance
> > changes.)
> >
> > Regards,
> >
> >
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Multiple stores

[Freddy Rietdijk]

My understanding is that whenever a file is needed it is copied in the
store. All these files are needed, at some point. The size likely wouldn't
be an issue if Nix could build and garbage-collect at the same time as the
eventual size of the data that I do want to keep in the store is expected
to be about 200 GB.

For the initial data reduction I could us a Nix store on the slower drive
(either with proot or a VM) and then copy the resulting files into the
store on the faster disk.

On Fri, Aug 26, 2016 at 9:45 PM, Roland Koebler <
rk-l...@simple-is-better.org> wrote:

> Hi,
>
> On Fri, Aug 26, 2016 at 02:51:02PM +0200, Freddy Rietdijk wrote:
> > I would like to use Nix for a certain data analysis where I will have
> > several thousand files which are all together about two TB in size.
> Do these 2 TB of data really have to be stored in the nix-store?
> And if yes, how about storing the data on the harddisk and symlinking
> it (or a subdirectory containing the data) to the store?
>
>
> Roland
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Multiple stores

[Freddy Rietdijk]

Hi,

I would like to use Nix for a certain data analysis where I will have
several thousand files which are all together about two TB in size. At the
moment NixOS is installed on a SSD which is a quarter that size. Would it
be possible to install a second store elsewhere, in this case on a slower
HDD?

I couldn't find anything about this so I guess it isn't possible. Do you
have in that case any other suggestions? I suppose the most obvious
solution would be to install Nix on the slower HDD. Using LVM with multiple
SSD's gets a bit too expensive.

How does Nix perform on a HDD?

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Changes to Python infrastructure

[Freddy Rietdijk]

Hi,

Some time ago I opened a PR to use a fixed-point combinator for Python [1].
It works fine and is, aside from documenting the change (especially
regarding overriding), ready.

However, before documenting and merging I would like to have some feedback
on how it will look like.

Right now, we have `python`, `python27`, `python3`, `python27Packages`, and
so on. We also have `pypy` to consider. In the PR I put everything under
`pythonNew` and so you e.g.
- `pythonNew.cpython27.interpreter`
- `pythonNew.cpython35.buildEnv`
- `pythonNew.pypy27.withPackages`
- `pythonNew.cpython.pkgs.numpy`

Furthermore, there are for compatibility references from `python` and so on
to these new attributes.

It's a bit more verbose but I think it makes sense to have it like this.
Now, I am a bit unhappy with the name `pythonNew` and would rather have
`python` here. This will however break a lot of code. What do you think? Do
you have any suggestions?
Another option I consider is to append `pythonNew` to `pkgs` instead.

Kind regards,

Frederik

[1] https://github.com/NixOS/nixpkgs/pull/17428
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] multiple versions of the same software? / collisions

[Freddy Rietdijk]

A method to have access to all interpreters is to create a derivation that
has symbolic links to each of the interpreters and install that one. But as
has been said before nix-shell is the recommended method.



On Thu, Aug 4, 2016 at 2:13 PM, Rok Garbas  wrote:

> for the python development don't use nix-env but create per project
> default.nix scripts and use nix-shell.
>
> make sure you read the python section in nixpkgs manual ->
> http://nixos.org/nixpkgs/manual/
>
> if you're looking to generate nix expressions from requirements.txt
> checkout pypi2nix -> https://github.com/garbas/pypi2nix. also know
> that pypi2nix is in early stage - still it is giving good results for
> some - and i'm looking for feedback of packages that break since this
> is the only way i can fix it.
>
>
> On Thu, Aug 4, 2016 at 2:07 PM, Guillaume Maudoux (Layus)
>  wrote:
> > Hi,
> >
> > I think you are putting your efforts in the wrong direction.
> > Nix is not designed to install both python 3.4 and 3.5 in the same
> > environment.
> >
> > What it can do however is cache both in the store, so you can get an
> > environment with python 3.4 with
> > nix-shell -p python34
> > and an environment with python 3.5 with
> > nix-shell -p python35
> >
> > Nix can provide you with as many different environments at the same time.
> > You can have a shell with python 3.4 running besides a shell with python
> > 3.5.
> > You should however not install both in your default environment (nix-env
> > -i) as it won't work.
> >
> > May I refer you to https://nixos.org/wiki/Development_Environments ?
> >
> > Regards,
> > -- Layus.
> >
> >
> > Le 03/08/16 à 16:43, Roland Koebler a écrit :
> >> Hi,
> >>
> >> I think I've now found out (at least partially), although this could
> >> result in a package being installed multiple times.
> >>
> >>> It appears that the flag setting only works on already installed
> packages.
> >> Yes, and if I use
> >>
> >> $ nix-env -i python3-3.5.2
> >> ...
> >> $ nix-env --set-flag priority 10 python3-3.5.2
> >> ...
> >> $ nix-env -i python3-3.4.5 python3-3.5.2
> >> replacing old ‘python3-3.5.2’
> >> installing ‘python3-3.4.5’
> >> installing ‘python3-3.5.2’
> >>
> >> it does not work, since the last command first removes python3-3.5.2,
> >> then probably the flag is lost, and then installs it again.
> >>
> >>> Also I'm not sure, but even if there are name collisions in the user
> >>> environment, it should still be possible to install both packages but
> only
> >>> use one of the names, and then create your name alias to the non-named
> >>> package. However I do not know how this can be done.
> >> I think I've found out. The following works:
> >>
> >> $ nix-env -i python3-3.5.2
> >> $ nix-env --set-flag priority 10 python3-3.5.2
> >> $ nix-env -i -P python3-3.4.5
> >>
> >> --preserve-installed / -P was the relevant switch.
> >> However, this can have some strange side-effects, since -P can
> >> cause a package to be installed "multiple times":
> >>
> >> $ nix-env -i -P python3-3.4.5
> >> ...
> >> $ nix-env -i -P python3-3.4.5
> >> ...
> >> $ nix-env -i -P python3-3.4.5
> >> ...
> >> $ nix-env -q
> >> ...
> >> python3-3.4.5
> >> python3-3.4.5
> >> python3-3.4.5
> >> python3-3.4.5
> >> ...
> >>
> >> This could then be "resolved" by:
> >>
> >> $ nix-env -e python3-3.4.5
> >> uninstalling ‘python3-3.4.5’
> >> uninstalling ‘python3-3.4.5’
> >> uninstalling ‘python3-3.4.5’
> >> uninstalling ‘python3-3.4.5’
> >> $ nix-env -P -i python3-3.4.5
> >> installing ‘python3-3.4.5’
> >>
> >> But it still looks strange.
> >> Is this intentional?
> >>
> >> Roland
> >>
> >> ___
> >> nix-dev mailing list
> >> nix-dev@lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
>
> --
> Rok Garbas
> https://garbas.si
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Ruby Development on Nix

[Freddy Rietdijk]

Hi Taeer,

For instance, the python model of
>   python27Packages.package1
>   python27Packages.package2
> is very different from the haskell model of
>   (haskellPackages.ghcWithPackages (h: with h; [
> package1
> package2
>   ])


Actually, they're quite similar. For Python we also have `python.buildEnv`
and `python.withPackages` of which the use is identical to your Haskell
example. In certain cases it is possible to install or use individual
Python packages but using `python.buildEnv` or `python.withPackages`
generally works better.



On Fri, Jul 15, 2016 at 9:41 PM, Taeer Bar-Yam  wrote:

> Dear Knowledgable Nixers,
>
> *Introduction*
> I have been trying to get ruby gems installed in any way on my nixos
> machine. I have been reading up on the wikis, and I understand that this is
> not yet well put together. However, it seems that even things that aught to
> be working are not.
>
> *Problem*
> I put "bundler" in my configuration.nix, and it installs fine. However,
> when I try to actually run it, I get the error message:
> /nix/store/n5gqlhs1nmadxkbibv42wnxq3m7f404k-ruby-2.3.0-p0/lib/ruby/2.3.0/rubygems.rb:241:in
> `bin_path': can't find gem bundler (>= 0) (Gem::GemNotFoundException)
> from /run/current-system/sw/bin/bundler:14:in `'
>
> I tried to add the gem I wanted to my configuration.nix directly using
> buildRubyGem, but ruby was not able to find it.
>
> The method the manual prescribes is dependent on already having bundler
> installed, so I have not had a chance to test that.
>
> *Request/Solution*
> I am interested in contributing to the nix community by developing a
> better system for ruby gem installation, so I guess what I am looking for
> is the following:
>
> 1) For now, what is the current simplest way to install a ruby gem so that
> I can, in the terminal, type
> $ irb
> > require 'desired_gem'
> and it will work?
> This can include simply installing bundler and using that locally with a
> gemfile, but I would prefer that any system-wide installation be done
> through configuration.nix
>
> 2) If I'm going to add a system for installing ruby gems, what would be
> the right way to design it? I understand that programming languages
> sometimes have structure that prevents us from always using the ideal path,
> so can y'all point me to a programming language that is considered "done
> right"?
>
> For instance, the python model of
>   python27Packages.package1
>   python27Packages.package2
>
> is very different from the haskell model of
>   (haskellPackages.ghcWithPackages (h: with h; [
> package1
> package2
>   ])
> My understanding is that the haskell model is a less-than-ideal kludge
> because haskell wont allow the python method. Is that right? Is there an
> even better way than python?
>
> Lastly, is there anyone already working on this that I should get in touch
> with instead of splitting efforts?
>
> Thank you,
>   --Taeer
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Install Nix on OSX, install nixops -> runs python2.7-pytest tests

[Freddy Rietdijk]

The nixpkgs-unstable channel, which includes OSX packages, hasn't updated
in 25 days whereas the nixos-unstable channel was updated 4 days ago.
http://howoldis.herokuapp.com/

On Tue, Jun 7, 2016 at 2:54 PM, Maarten Hoogendoorn 
wrote:

> Ah, I see ;) Now it makes sense.
>
> 2016-06-07 14:42 GMT+02:00 zimbatm :
>
>> It's possible, the nixpkgs-unstable and nixos-unstable both evolve
>> independently as they have different "success" conditions.
>>
>> On Tue, 7 Jun 2016, 13:38 Maarten Hoogendoorn, 
>> wrote:
>>
>>> Ah, that could be very well the cause of this. Is the OS X build lagging
>>> behind the NixOS one? I thought if some package was present in the unstable
>>> channel, that hydra would have build it and uploaded it to the binary cache?
>>>
>>> 2016-06-07 14:30 GMT+02:00 zimbatm :
>>>
 In general I found that there are less packages available from the
 cache on OSX. What probably happened is that the packages had to be built.
 If doCheck = true then tests are run after the build and this is the
 default for python packages.

 On Tue, 7 Jun 2016, 13:19 Maarten Hoogendoorn, 
 wrote:

> Hi there,
>
> I have been a really happy Nix{,pkgs,os,ops} for some time. In fact,
> I'm writing two intro blog posts about nix during working hours for
> Container Solutions. Since most of the developers that will read the
> blogpost will be running OS X, I've borrowed a spare macbook, and 
> installed
> nix and nixops.
>
> To my surprise, this caused python tests to run. Is this the expected
> behavior?
>
> Thanks,
> Maarten
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>

>>>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Separate repository for autogenerated lists of urls and hashes

[Freddy Rietdijk]

Autogenerating packages is one step further. But it would be nice if there
was one source where you could get more of such data from. Maybe
https://libraries.io/ some day?

To come back to the main topic, autogenerated lists in Nixpkgs. I could
imagine it would be nice if we would have also, next to `pkgs` and `lib`, a
`srcs`. This you could use to select the source archive of a package for a
certain version. For example, `src = fetchurl
srcs.github.${name}.${version}` or `src = fetchurl
srcs.pypi.${name}.${version} but also `meta = srcs.pypi.${name}.meta`.

On Sun, Apr 24, 2016 at 7:11 PM, Kosyrev Serge <_deepf...@feelingofgreen.ru>
wrote:

> Marc Weber  writes:
> > I personally think that "autogenerating" packages is the next huge step
> > to be taken. This requires a database and dependency information for all
> > information.
> >
> > Then all distros can benefit, not just nixos.org
>
> ..and share the maintenance burden..
>
> Which would raise more centralisation-related questions, but those would
> be problems we'd be happy to have, I think.. : -)
>
> --
> с уважениeм / respectfully,
> Косырев Сергей
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Separate repository for autogenerated lists of urls and hashes

[Freddy Rietdijk]

Hi,

Autogenerating lists with urls and hashes has been discussed before
[1,2,3], but I like to bring it up again as I'm working on autogenerating a
list with urls and hashes for Python's PyPI packages along with other meta
data [4].

At this point I chose to only retrieve data for specific packages instead
of all of PyPI. I think that's the sensible thing to do do since PyPI
contains a lot of junk. Even so, the autogenerated list with urls, hashes
and meta data can become quite large, especially as I chose to retrieve the
data for all versions of a package.

What I propose is that we split Nixpkgs basically in two, we keep Nixpkgs
that contains expressions, and we create another repository, Nixsrcs, that
contains scripts and autogenerated lists. These autogenerated lists will
contain *all* versions of a package. Nixpkgs would import these lists as
derivations (so we do need one small autogenerated list in Nixpkgs).

When building packages we can optionally specify a version. If a version is
not specified the latest available version will be built. Likely we do need
to limit somewhat the amount of versions in the autogenerated lists, but
that is something we need to find out. One nice feature to have is version
comparison, so we can just write the major version and let Nix determine
the latest minor version.

What do you think? Can it be done? Am I missing something? I guess it's a
matter of 'just doing'.

Freddy



[1] https://github.com/NixOS/nixpkgs/issues/14897
[2] http://thread.gmane.org/gmane.linux.distributions.nixos/18932
[3] http://thread.gmane.org/gmane.linux.distributions.nixos/18595
[4] https://github.com/NixOS/nixpkgs/pull/14927
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Upcoming PyPi URL Scheme Change

[Freddy Rietdijk]

Thanks for the update.

There are indeed some things we can automate. Before, I experimented with
using one of the API's to get out as much metadata as possible. We could
also use pypi2nix, which can give for more information, but requires
downloading all files.
Unfortunately, the old site still uses MD5 so I quit my effort using the
API. The new site (https://warehouse.python.org/) uses SHA256 though.

If this change in URL scheme is really going to happen I think we should
start using the API to find the correct file, version, hash, description
and license. Optionally, we should make it possible to run pypi2nix to
extract more, and more precise, information.

See also https://github.com/NixOS/nixpkgs/issues/11587.

On Thu, Apr 21, 2016 at 12:31 AM, Profpatsch  wrote:

> On 16-04-20 11:41am, Graham Christensen wrote:
> > I recently got word that PyPi is changing their URL scheme.
> >
> > Old example:
> >
> https://pypi.python.org/packages/source/a/ansible/ansible-1.8.2.tar.gz#md5=c2ac0e5a4c092dfa84c7e9e51cd45095
> >
> > New example:
> >
> https://pypi.python.org/packages/62/18/91f0e5059373e9b87588c2a1c3b4c3c08ee89e0443aa2017469a4cdae41c/SCRY-1.1.2-py2-none-any.whl#md5=a3c636c4e94df1f0644b6917a9c05e67
>
> This is going to be a lot of work.
>
> > Yet another option is to run a sort of "translator" service that can
> consume
> > the PyPI JSON API and will output the URLs in whatever format best
> suites you.
> > An example of this is pypi.debian.net (which I don't know where the
> code base
> > for it is, but the proof of concept I wrote for it is at
> > https://github.com/dstufft/pypi-debian). These translators are
> fairly simple,
> > they take an URL, pull the project and filename out of it and then
> use the JSON
> > API to figure out the "real" URL and then just simply redirects to
> that.
>
> Maybe it’s time to automate what we can? Similar to Hackage?
>
> --
> Proudly written in Mutt with Vim on NixOS.
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es
> May take up to five days to read your message. If it’s urgent, call me.
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Fwd: Fwd: Wiki is dead

[Freddy Rietdijk]

It makes sense not to have multiple formats inside a single document. As
Eelco mentioned, it makes it harder to move around fragments, and, as I
experienced now by using the `toDocbook` function, you still end up with
XML errors. Therefore, instead of having to debug errors related to one
format you now have to do so for two.

What format then? Is there a single format that would fit? Should there be
a single format for all our documentation? I actually don't think that's
necessary. No, we better not have multiple formats in a single document.
But how about multiple documents, with each possibly having another format?

We have to think about what type of documentation we need, who each type is
meant for, and actually also who would write that documentation. I think we
are going to need multiple documents, as we have now, but reorganized.

The most relevant one for many of us from the contributor point of view is
going to be a User Guide, which the way I see it, would contain reference
material to specific items in Nixpkgs (modules, functions, ...) and
introductions. Many people should contribute to this guide, and
contributing to it should therefore be sufficiently accessible.

If we now consider formats again then Commonmark is very accessible, almost
everyone knows it. Being a well-specified dialect of Markdown it doesn't
suffer from ambiguity between implementations. However, it still lacks one
important feature: references to sections. You cannot write a manual
without internal references.

Sphinx with reStructuredText has this feature, and many others that are
also covered by Docbook but without the verbosity that comes with it.
Furthermore, the step from Commonmark/Markdown to reStructuredText is
small. Markdown is close to being valid rST.



On Fri, Feb 26, 2016 at 9:29 AM, Vladimír Čunát  wrote:

> On 02/25/2016 01:39 PM, zimbatm wrote:
> > The output file wasn't exactly right, I had to replace ` > id="something">` to `` tags and wrap it in a  tag.
>
> That's because pandoc uses an older version of docbook where some tags
> are different. (IIRC I looked a few months ago and it didn't support the
> newer one yet.)
>
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Fwd: Wiki is dead

[Freddy Rietdijk]

Adding documentation is one thing, maintaining it is a second. It's great
if someone is willing to convert contributions to Docbook, and I'm thankful
for that. However, that does not help much if you want to modify parts of
it. And as Nixpkgs is updated, so should the documentation.

I agree writing docs isn't peoples favorite activity, but somehow, when I
see the amount of content that's on the Wiki, I do wonder: how come people
chose to put it there instead of in the manual? Was it because there never
really was a manual where the contents would fit in? Or was it because it
was more convenient to contribute to the wiki instead of the manual?
I haven't been around long enough with this project, so I don't know. Some
of you will have a better idea.

Being able to convert to Docbook like is done with the Haskell guide makes
it easier for others to contribute, at least, if you know how that
conversion is done.

Some time ago I began using Nix because I like it. Python packaging is a
mess as Domen explained during NixCon, but with Nix it works much better.
I've contributed Python packages, and now I would like to share how it is
done. As acoustician/researcher my work activities are quite different from
I bet most other Nix users. Learning Nix was a big investment, but is I
think worth it. Learning Docbook on the other hand...

If you want to scare away contributors by keeping the barrier high, then
keep the documentation the way it is.



On Sat, Feb 20, 2016 at 2:04 PM, Thomas Hunger <tehun...@gmail.com> wrote:

> I'm tool agnostic but +1 on having a cookbook in git for the
> review-workflow (avoids wiki spam). I have a number of snippets (how to
> remove gc roots, haskell profiling, how to use ihaskell properly, many
> more) but no good place to put them.
>
> I've started a git-book thing [1] a while back to collect these but didn't
> get very far. I'd much rather contribute to a common, low-barrier-to-entry
> repository than rolling my own.
>
> In my experience just providing the structure will eventually attract
> content because adding a small snippet is the path of least resistance for
> each individual contributor. Maybe we could add a banner "This is how you
> add a snippet" and buttons "File a bug that this is wrong / outdated" to
> each snippet?
>
> Rok - I know it's not free software but maybe it's worth setting up a
> google docs spreadsheet for coordinating the migration once we've settled
> on a tool? I will contribute.
>
> ~
>
> [1]
> https://github.com/WeAreWizards/nixbyexample
>
> On 20 February 2016 at 12:06, Freddy Rietdijk <freddyrietd...@fridh.nl>
> wrote:
>
>> I agree with Vladimir that we already have the infrastructure, the
>> Nixpkgs repository.
>>
>> What is needed is a clearer way where to put certain documentation and a
>> lower barrier for contributing. In `Redesign of documentation` I came with
>> a proposal of how to structure the documentation. A wiki has a low barrier
>> for contributing, however, it also has many disadvantages, which you would
>> not have if we use, say, the Nixpkgs repository.
>>
>> A big barrier, in my opinion and I'm pretty sure also in that of others,
>> is the current format of the Nixpkgs manual. I can understand why docbook
>> is used, and I think it should be used for say the Nix manual, but for a
>> User's Guide to which many of us would/should contribute, the barrier is
>> just too high.
>>
>> On the branch https://github.com/FRidh/nixpkgs/tree/usersguide I
>> implemented the proposed redesign.
>> The documentation is split into two documents, the User's Guide, and the
>> Contributor's Guide. The User's Guide is divided into three parts:
>>
>>- Introduction
>>- Reference
>>- Cookbook
>>
>> There have been several proposals already for tools/implementations
>> (Matthias' Jekyll implementation, Domens Sphinx docs) . Here I chose to go
>> also for the Sphinx documentation generator. It allows you to write
>> RestructuredText. With a small plugin, which I enabled, you can also
>> include CommonMark/Markdown. Nix highlighting is supported.
>> The result can be found at http://nixpkgs.readthedocs.org/en/latest/ .
>> It's mostly empty still.
>>
>> Now this is only a proposal, and I'm open to other ways. But I really
>> think we should do something about the current state of the docs, in both
>> content and lowering the barrier. ReStructuredText/Markdown obviously
>> doesn't have as much possibilities as Docbook but what matters eventually
>> is whether it is enough, and I think it will be, at least for now.
>>
>>
>> On Sat, Feb 20, 2016 at 12:30 PM, Vladimír Čunát <vcu...@gmail.com>
>&

[Nix-dev] Fwd: Wiki is dead

[Freddy Rietdijk]

I agree with Vladimir that we already have the infrastructure, the Nixpkgs
repository.

What is needed is a clearer way where to put certain documentation and a
lower barrier for contributing. In `Redesign of documentation` I came with
a proposal of how to structure the documentation. A wiki has a low barrier
for contributing, however, it also has many disadvantages, which you would
not have if we use, say, the Nixpkgs repository.

A big barrier, in my opinion and I'm pretty sure also in that of others, is
the current format of the Nixpkgs manual. I can understand why docbook is
used, and I think it should be used for say the Nix manual, but for a
User's Guide to which many of us would/should contribute, the barrier is
just too high.

On the branch https://github.com/FRidh/nixpkgs/tree/usersguide I
implemented the proposed redesign.
The documentation is split into two documents, the User's Guide, and the
Contributor's Guide. The User's Guide is divided into three parts:

   - Introduction
   - Reference
   - Cookbook

There have been several proposals already for tools/implementations
(Matthias' Jekyll implementation, Domens Sphinx docs) . Here I chose to go
also for the Sphinx documentation generator. It allows you to write
RestructuredText. With a small plugin, which I enabled, you can also
include CommonMark/Markdown. Nix highlighting is supported.
The result can be found at http://nixpkgs.readthedocs.org/en/latest/ . It's
mostly empty still.

Now this is only a proposal, and I'm open to other ways. But I really think
we should do something about the current state of the docs, in both content
and lowering the barrier. ReStructuredText/Markdown obviously doesn't have
as much possibilities as Docbook but what matters eventually is whether it
is enough, and I think it will be, at least for now.


On Sat, Feb 20, 2016 at 12:30 PM, Vladimír Čunát  wrote:

> On 02/20/2016 12:52 AM, zimbatm wrote:
> > It's a great staging area for content where people can edit without
> > asking for permission. [...]
>
> What are the advantages in comparison to standard pull requests with
> discussion underneath? We already have lots of infrastructure advantages
> in there, such as merging changes at once with documentation for them,
> auto-mention bot, etc.
>
>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Fwd: Redesign of documentation

[Freddy Rietdijk]

On Tue, Feb 16, 2016 at 10:50 AM, Christian Kauhaus <k...@flyingcircus.io>
wrote:

>
> In my opinion it would be helpful to cover the Nix language (language
> syntax,
> features and builtins) in a separate section.
>
>
I agree the Nix language should also be covered. Where do you think this
should be? For example an introduction to the Nix language in the User's
manual after or before explaining the introduction to the Nix package
manager?

On Tue, Feb 16, 2016 at 10:50 AM, Christian Kauhaus <k...@flyingcircus.io>
wrote:

> Am 15.02.2016 um 18:57 schrieb Freddy Rietdijk:
> > So, that's my idea about the documentation. What are your views about
> this
> > issue and the proposal? And remember, no talking about
> implementations/formats
> > just yet!
>
> In my opinion it would be helpful to cover the Nix language (language
> syntax,
> features and builtins) in a separate section.
>
> Regards
>
> Christian
>
> --
> Dipl-Inf. Christian Kauhaus <>< · k...@flyingcircus.io · +49 345 219401-0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Redesign of documentation

[Freddy Rietdijk]

Hi Nixers,

Since NixCon last year UX and documentation are in the spotlight, and
especially since the Wiki has gone to read-only there seems to be even more
pressure to come up with a good solution to the documentation challenge.

I would like to bring up a discussion on what type of documentation we
need, and what the requirements are, mostly regarding content. Let's NOT
discuss implementations and formats at this point. That's something for
later.

Currently the following guides exist:

   1. Nix Package Manager guide 
   2. NixOS manual 
   3. Nixpkgs Contributors guide 

as well as a (mostly outdated) read-only wiki (and documentation for Hydra,
NixOps and Disnix but I think we can leave that out of this discussion).

What clearly is missing is introductory material. What kind of introductory
material do we need? And what other type of documentation do we need? Also,
who do we want to address with the documentation? Furthermore, how do we
keep the quality high? Contributions to the guides are currently mostly
peer-reviewed since they go via GitHub.


Now, let's move on to my proposal. I propose we create the following guides

   1. Nix Package Manager guide 
   2. NixOS manua l
   3. Nix user guide
   4. Nixpkgs contributor guide

where the Nix user guide is a guide containing both introductory material
and reference material

   - Introduction to
  - Nix ecosystem 
  - Nix package manager
  - Nixpkgs
 - Nixpkgs Hydra
  - NixOS
  - Hydra (NOT https://hydra.nixos.org.)
  - Python on Nix
  
  - Haskell on Nix
  - ...
   - Reference
  - Modules
  - Python
 - buildPythonPackage
  - Haskell
  - ...

and the Nixpkgs contributor guide would then only contain details about
Nixpkgs style and how to contribute. Furthermore, I can imagine a third
part with How To's could be added to the Nix user guide.

I think we need to keep peer-review to keep the quality high, and I think
the barrier for contributing to guides 3) and 4) should be sufficiently low
to hopefully attract more Nixpkgs contributors.

So, that's my idea about the documentation. What are your views about this
issue and the proposal? And remember, no talking about
implementations/formats just yet!

Kind regards,

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Python setuptools upgrade and possibly failing builds

[Freddy Rietdijk]

Hi Nixers,

We're upgrading Pythons setuptools to the latest version [1]. In the latest
version setuptools changed behavior with regards to how tests are run, and
this might affect one or more of your packages.

Up till now we had builds passing the test phase even when no tests were
run, often because no test runner was explicitly specified. With the new
version, setuptools will always try to run tests, even when the required
test runner is not present. Therefore, if you haven't specified correctly
how the tests should run the build will fail. And it will fail occasionally
with some very vague errors.

This change has a big impact on nixpkgs, with many Python package builds
now failing that before silently passed the checkPhase. We've fixed a large
set of the failing builds, but definitely not all of them.

So what should you do if you want to fix a package you need? Well, simple,
if you notice that a package you need is not building, you have to fix the
test runner. Generally that is enough.

If however you cannot get it to pass the checkPhase and you want to disable
the tests, then please explain in the code why you disabled the tests.

Tomorrow we'll likely merge this PR. Until this is merged, please don't
make any changes to `python-packages.nix`.

Thanks!

Freddy

[1] https://github.com/NixOS/nixpkgs/pull/12552
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Store nixpkgs licenses as JSON instead of .nix

[Freddy Rietdijk]

Hi Nixers,

The licenses that are used in nixpkgs are specified in .nix format.
While it is possible to use builtins.toJSON to convert to JSON on-demand, I
propose that we store the licenses in JSON format instead, separating the
data from code.

A practical use case is that I would like to match (using Python) raw
license information to the licenses we have in nixpkgs. I don't want to
depend on the user having Nix available (just the nixpkgs repository) to
convert libraries.nix to JSON.

That is my argumentation. What do you think?

Kind regards,

Freddy Rietdijk
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] NixOS: entry in configuration to list packages that should not be gc-ed

[Freddy Rietdijk]

Dear Nixers,

There are a bunch of packages I would always like to have available in the
Nix store, but that I do not want directly in my primary environment. These
packages I would like to specify in `/etc/nixos/configuration.nix` so that
on a rebuild these are always available without downloading/building when I
need them.

I know that with `nix-shell` it is possible to prevent packages from being
gc-ed, however, I would like to specify the list of packages in
`configuration.nix`. Is there any such way?

Kind regards,

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Several updates to rather important Python packages

[Freddy Rietdijk]

Hi Nixers,

For some time now I've been submitting new Python packages as well as
updates. Occasionally, an update arrives with a very large impact, e.g. the
currently available numpy update to 1.10.0 (#10255
).

For some of the cases, like this one, I don't have the computational power
available to build all derivations. And updates like these are likely to
break some dependents. I am wondering what to do in such case. Make a PR
for staging and ask for it to be submitted, even though it might break
dependents?

Sometimes it happens that updating the dependents is sufficient. However,
there are cases this could be a lot of work, updating each and every
failing dependent till there are no failures.

An example is the PR I linked to, I update numpy to a version which allows
it to run with Python 3.5. However, many new derivations that can now be
build will likely fail due to Python 3.5 incompatibilities. What should we
do in such case? Not update? Or update and have many failures? I think
that, because a new NixOS release is still far away, now is the time to
push such updates. It is then up to the maintainers of the dependents to
fix their packages.

The same goes for scipy in #9146
.

And especially markupsafe. The current version is 0.15 which is from July
2011. Updating this one to the latest affects not only many Python
packages, but also most of KDE.

Kind regards,

Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] Trouble packaging llvmlite, Python bindings for LLVM

[Freddy Rietdijk]

Hi Thomas,

It's good to hear you also gave it a try. I reported the issue
https://github.com/numba/llvmlite/issues/93

Freddy

On Sun, Aug 30, 2015 at 12:30 PM, Thomas Hunger <tehun...@gmail.com> wrote:

> Hi Freddy,
>
> I ran into the numba segfault as well [1]. I tried to debug the issue with
> a numba core dev when in the same room but we didn't make any progress in
> an hour. Potentially a double-free bug. Probably worth filing an issue
> which I forgot to do.
>
> ~
>
> [1]
> In [1]: import numba
> *** Error in
> `/nix/store/pbi1lgank10fy0xpjckbdpgacqw34dsz-python-2.7.9/bin/python2.7':
> free(): invalid pointer: 0x7f501151f3c0 ***
> === Backtrace: =
>
> /nix/store/6k9z1sfl7kghmagwd205k3i81pbcw57s-glibc-2.21/lib/libc.so.6(+0x70d2f)[0x7f502aa60d2f]
>
> /nix/store/6k9z1sfl7kghmagwd205k3i81pbcw57s-glibc-2.21/lib/libc.so.6(+0x760be)[0x7f502aa660be]
> 
>
> On 30 August 2015 at 09:56, Freddy Rietdijk <freddyrietd...@fridh.nl>
> wrote:
>
>> Dear Nixers,
>>
>> In the last couple of weeks I've been packaging a bunch of Python
>> packages, mostly packages that I use for my research. There is one package
>> though (or in fact, two), that I can't seem to build correctly: llvmlite
>> and numba. The first is a Python binding to LLVM, and the other is a JIT
>> that uses the binding.
>>
>> llvmlite can be build, but unfortunately segfaults. I don't know too much
>> about compiling, let alone compilers. I was able to build a working version
>> on Debian though.
>>
>> If you could have a look at it, I would very much appreciate it:
>> https://github.com/NixOS/nixpkgs/pull/9516
>> and a direct link to the build log:
>> https://s3.amazonaws.com/archive.travis-ci.org/jobs/77894848/log.txt
>> I would be very hapy to get these working, since then I could finally
>> switch to NixOS on my main machine.
>>
>> Kind regards,
>>
>> Frederik
>>
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] Trouble packaging llvmlite, Python bindings for LLVM

[Freddy Rietdijk]

Dear Nixers,

In the last couple of weeks I've been packaging a bunch of Python packages,
mostly packages that I use for my research. There is one package though (or
in fact, two), that I can't seem to build correctly: llvmlite and numba.
The first is a Python binding to LLVM, and the other is a JIT that uses the
binding.

llvmlite can be build, but unfortunately segfaults. I don't know too much
about compiling, let alone compilers. I was able to build a working version
on Debian though.

If you could have a look at it, I would very much appreciate it:
https://github.com/NixOS/nixpkgs/pull/9516
and a direct link to the build log:
https://s3.amazonaws.com/archive.travis-ci.org/jobs/77894848/log.txt
I would be very hapy to get these working, since then I could finally
switch to NixOS on my main machine.

Kind regards,

Frederik
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev