Re: [Nix-dev] Hydra not building a package, fetchSubmodules fault?
On 7 June 2014 05:23, Mateusz Kowalczyk wrote: > Hi, > > I packaged the PSP emulator called PPSSPP a while ago and it has been in > nixpkgs for little over a week[1]. When I try to nix-env -i PPSSPP > however my system tells me that it will build it rather than fetch it. I think, in this case, it is because the package lacks meta.platforms. (I don't know why vice is built, even though it lacks meta.platforms.) Another reason for such behaviour is that the channel lags git master. Check the version of your channel sources with nixos-version, the last part of the version number output is the git commit id. Best regards, Bjørn Forsman ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
[Nix-dev] Hydra not building a package, fetchSubmodules fault?
Hi, I packaged the PSP emulator called PPSSPP a while ago and it has been in nixpkgs for little over a week[1]. When I try to nix-env -i PPSSPP however my system tells me that it will build it rather than fetch it. I thought that maybe I needed to explicitly specify platforms but the ‘vice’ package (also an emulator) doesn't do it and Hydra builds it. I used the search box on Hydra page but there are no results for PPSSPP. Can someone tell me why Hydra won't acknowledge this package? The only out-of-ordinary thing in it might be that I'm using fetchgit in it but I can't think why this should be a problem considering that specific revision and sha is specified so in the end it's no different than a tarball. In fact the ‘vobsub2srt’ package (found with grep) also uses fetchgit and Hydra seems to have no problem with that. I do note that I'm using fetchSubmodules = true and only other package in nixpkgs that uses it too (opendylan) seems to not be acknowledged by Hydra either. Should this be considered a bug and if yes, of which tool, Hydra/nix/fetchgit/…? [1]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/misc/emulators/ppsspp/default.nix -- Mateusz K. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
Hi, On 06/06/14 13:57, Mathijs Kwik wrote: > So server systems need to start following a channel that does not run > lengthy tests? The NixOS system tests actually aren't very lengthy (though they do suffer from non-deterministic failures). It's more the latency of having to wait for 20,000 package builds. Obviously, a better solution would be to: * Throw more hardware at the problem. * Start mirroring NARs to cache.nixos.org before the jobset eval has finished building. -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
On 06/06/2014 07:59 AM, Ertugrul Söylemez wrote: > On Thu, 05 Jun 2014 23:39:34 +0200 > Vladimír Čunát wrote: > >> Hydra has and uses priorities. Anyway, building OpenSSL itself is very >> quick, but rebuilding all that (transitively) depends on it is worse. >> And there are CVE fixes for stdenv stuff sometimes (glibc)... > Yes, and the basic idea is that you could have high priority packages like > OpenSSL, OpenVPN and nginx. Whenever Hydra sees a job of higher priority it > starts doing it (potentially aborting whatever it is currently doing). Once > all jobs of the same priority are done, it runs the tests of the same > priority and updates the channel. Then it goes to the next highest priority. > That way security updates won't take longer than necessary. > > When we use priorities generously we could avoid a lot of delay even in less > critical cases. > > > Greets, > Ertugrul > So you would provide a separate channel then which is updated incrementally, and the original channel is updated once all packages are being built? Does the nix package manager allow to measure how many packages it would have to build for a system's current configuration and for how many packages there would be binary package available? If so, it would help to decide whether to wait for Hydra to build the needed packages, or when to start building the remaining locally. Best, Thomas ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
Eelco Dolstra writes: > Hi, > > On 06/06/14 13:29, Mathijs Kwik wrote: > >> How are people supposed to use that channel? >> I don't think I can _add_ a secondary channel which provides a >> conflicting source (nixos). Switching back and forth doesn't sound >> easy. > > Switching to another channel is very easy: > > $ nix-channel --add http://nixos.org/channels/ nixos The command is not the uneasy part :) I just don't think that switching back&forth for security-updates is a sane thing to do. Leaving the system at the "security" channel means you will end up with untested stuff that probably fails the default channel build. I would rather have a secondary channel that gets "merged" (packageOverrides style) into nixpkgs, which only contains overrides that haven't made it into the main channel yet. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
Hi, On 06/06/14 13:14, Mathijs Kwik wrote: > That still doesn't solve the question "how to do a half build". > If there have been 5 commits since the last channel build, and you add > an important security fix, there is no way to only build that fix > without building the other stuff. Well, if it's really urgent, you can always revert those 5 commits :-) -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
Michael Raskin <7c6f4...@mail.ru> writes: Calculating the transitive closure for all nixos modules / services run by systemd is one way to prioritize. A populatiry contest could be added to that. >>> >>> Maybe having a channel which is a subset of the main channel and >>> includes at least ssh, apache, nginx, postgresql, mysql, and some ftp >>> server would be a nice start? >> >>How are people supposed to use that channel? >>I don't think I can _add_ a secondary channel which provides a >>conflicting source (nixos). Switching back and forth doesn't sound >>easy. >> >>Also, this would just make your system start to build all the additional >>packages (not built by this new channel) by itself. No matter how much >>stuff gets pre-built, a channel always contains a specific nixpkgs >>version, so with or without binary archives, a nixos-rebuild _will_ >>build everything for that release. > > This will be a channel for server systems. You will checkout master and > do a rebuild, and this channel would provide you with prebuilt packages > for most of your server needs. So server systems need to start following a channel that does not run lengthy tests? And I think people don't usually use local git repos on their servers. The nixpkgs/nixos tree gets provided by the channel. I just want to run `nix-channel --update && nixos-rebuild switch` on my server. Not supply it with a local git checkout that I need to keep up to date and do custom rebuilds against. I do see what you are aiming for, but I don't think there's a clean way to combine it with the default workflow (nicely tested channel which provides nixpkgs/nixos sources and binaries). If there's a way to be able to add _both_ channels, this would be great. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
Hi, On 06/06/14 13:29, Mathijs Kwik wrote: > How are people supposed to use that channel? > I don't think I can _add_ a secondary channel which provides a > conflicting source (nixos). Switching back and forth doesn't sound > easy. Switching to another channel is very easy: $ nix-channel --add http://nixos.org/channels/ nixos -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
>>>Calculating the transitive closure for all nixos modules / services run by >>>systemd is one way to prioritize. A populatiry contest could be added to >>>that. >> >> Maybe having a channel which is a subset of the main channel and >> includes at least ssh, apache, nginx, postgresql, mysql, and some ftp >> server would be a nice start? > >How are people supposed to use that channel? >I don't think I can _add_ a secondary channel which provides a >conflicting source (nixos). Switching back and forth doesn't sound >easy. > >Also, this would just make your system start to build all the additional >packages (not built by this new channel) by itself. No matter how much >stuff gets pre-built, a channel always contains a specific nixpkgs >version, so with or without binary archives, a nixos-rebuild _will_ >build everything for that release. This will be a channel for server systems. You will checkout master and do a rebuild, and this channel would provide you with prebuilt packages for most of your server needs. ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
Michael Raskin <7c6f4...@mail.ru> writes: When we use priorities generously we could avoid a lot of delay even in less critical cases. >>> >>> The main problem I see is that normally you don't want to release a >>> channel until *all* parts have rebuilt. >>> >> >>+1 Rebuilding for a server that runs, say ssh, apache, nginx, postfix and a >>few such services takes maybe 2% of the time required to build a full >>desktop distribution. >> >>I think being able to release packages used on public facing servers could >>be prioritized over, say LibreOffice, Qt, Webkit etc. >> >>If the system environment is not "polluted" by the desktop packages, it >>could be possible to upgrade the system environment before user >>environments that needs one or two orders of magnitude more time to compile. >> >>Calculating the transitive closure for all nixos modules / services run by >>systemd is one way to prioritize. A populatiry contest could be added to >>that. > > Maybe having a channel which is a subset of the main channel and > includes at least ssh, apache, nginx, postgresql, mysql, and some ftp > server would be a nice start? How are people supposed to use that channel? I don't think I can _add_ a secondary channel which provides a conflicting source (nixos). Switching back and forth doesn't sound easy. Also, this would just make your system start to build all the additional packages (not built by this new channel) by itself. No matter how much stuff gets pre-built, a channel always contains a specific nixpkgs version, so with or without binary archives, a nixos-rebuild _will_ build everything for that release. > > > > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
Hi, On 06/06/14 13:13, Michael Raskin wrote: > Maybe having a channel which is a subset of the main channel and > includes at least ssh, apache, nginx, postgresql, mysql, and some ftp > server would be a nice start? Yeah, that sounds like a good idea. -- Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/ ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
Alexander Kjeldaas writes: > On Fri, Jun 6, 2014 at 10:20 AM, Vladimír Čunát wrote: > >> On 06/06/2014 08:59 AM, Ertugrul Söylemez wrote: >> >>> When we use priorities generously we could avoid a lot of delay even in >>> less critical cases. >>> >> >> The main problem I see is that normally you don't want to release a >> channel until *all* parts have rebuilt. >> > > +1 Rebuilding for a server that runs, say ssh, apache, nginx, postfix and a > few such services takes maybe 2% of the time required to build a full > desktop distribution. > > I think being able to release packages used on public facing servers could > be prioritized over, say LibreOffice, Qt, Webkit etc. > > If the system environment is not "polluted" by the desktop packages, it > could be possible to upgrade the system environment before user > environments that needs one or two orders of magnitude more time to compile. > > Calculating the transitive closure for all nixos modules / services run by > systemd is one way to prioritize. A populatiry contest could be added to > that. That still doesn't solve the question "how to do a half build". If there have been 5 commits since the last channel build, and you add an important security fix, there is no way to only build that fix without building the other stuff. A build takes a full nixpkgs checkout and builds that. In its output, it includes a tarball with the nixpkgs tree that was used. The only way to _only_ get the fix is to have a separate git branch (on top of what? release-14.04?) with just these changes and a separate hydra target/job that only builds a few important server packages and does not perform the expensive run-in-vm tests. But even then, I don't know how this would magically end up on people's systems, because I don't think you can "merge 2 channels" that both provide nixpkgs/nixos. > > Alexander > > >> >> We do have meta.schedulingPriority, but it's used little, and from earlier >> discussions I think it's really hard to objectively determine which >> packages are more important than others ;-) >> >> BTW, aborting jobs would need to be done very carefully, because we have >> some jobs that run for hours, so that could lead to wasting lots of time. >> >> >> Vlada >> >> >> >> ___ >> nix-dev mailing list >> nix-dev@lists.science.uu.nl >> http://lists.science.uu.nl/mailman/listinfo/nix-dev >> >> > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
>>> When we use priorities generously we could avoid a lot of delay even in >>> less critical cases. >>> >> >> The main problem I see is that normally you don't want to release a >> channel until *all* parts have rebuilt. >> > >+1 Rebuilding for a server that runs, say ssh, apache, nginx, postfix and a >few such services takes maybe 2% of the time required to build a full >desktop distribution. > >I think being able to release packages used on public facing servers could >be prioritized over, say LibreOffice, Qt, Webkit etc. > >If the system environment is not "polluted" by the desktop packages, it >could be possible to upgrade the system environment before user >environments that needs one or two orders of magnitude more time to compile. > >Calculating the transitive closure for all nixos modules / services run by >systemd is one way to prioritize. A populatiry contest could be added to >that. Maybe having a channel which is a subset of the main channel and includes at least ssh, apache, nginx, postgresql, mysql, and some ftp server would be a nice start? ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] setfacl: /var/log/journal: Operation not supported
Hey Ricardo!
On Wed, Jun 04, 2014 at 06:24:55PM +0200, Ricardo M. Correia wrote:
> If the problem is indeed the setfacl failure (as it appears to be), I think
> you need to enable ACLs in ZFS by doing:
>
> "zfs set acltype=posixacl ", where is the ZFS dataset/filesystem
> which contains your systemd journal.
Okay, thank you! I will look into that and test whether it resolves
the problem.
> I'm not sure if the default ZFS version (0.6.2) contains the acltype
> property already... if not, you may have to use a recent git version. You
> can do that simply by adding "boot.zfs.useGit = true;" to your
> configuration.nix.
Done already.
> For performance reasons it's also advisable to do "zfs set xattr=sa ".
> This allows ZFS to use a more performant on-disk representation of ACLs.
> However, be careful because I think (but I could be wrong) that there have
> been reports of filesystem corruption using this option.
>
> See for example this issue: https://github.com/zfsonlinux/zfs/issues/2214
> If I'm not mistaken our git version of ZFS, even in the unstable NixOS
> channel, does not include the fix for that yet.
>
> I may be wrong but I think that if you're not going to be using ACLs apart
> from the systemd journal, it's OK to not set "xattr=sa", as the systemd
> journal's ACL will probably always be cached in memory due to constant
> access, so it shouldn't be cause a large performance impact.
I will investigate that issue and ensure a recent backup before doing
anything ;)
Thanks!
> On Wed, Jun 4, 2014 at 2:11 PM, _1126 wrote:
>
> > Hello all!
> >
> > I recently re-installed my NixOS configuration on a ZFS root. Since
> > then I am unable to access systemd's journal. My user is part of the
> > systemd-journal group, but the journal file aint.
> >
> > -rw-r- 1 root root 117440512 Jun 4 14:07 system.journal
> >
> > This is might be due to the the error reported in the subject:
> >
> > setfacl: /var/log/journal: Operation not supported
> >
> > The nixos/modules/system/boot/systemd.nix module tries the following
> > (line 743):
> >
> > ${pkgs.acl}/bin/setfacl -nm g:wheel:rx,d:g:wheel:rx,g:adm:rx,d:g:adm:rx
> > /var/log/journal
> >
> > This apparently fails. Has anyone an idea what that is the case? Or am
> > I wrong in suspecting that and the reason for the wrong permissions is
> > to be found somewhere else?
> >
> > Greetings
> > _1126 (Christian Lask)
> >
> >
> >
> >
> >
> > ___
> > nix-dev mailing list
> > nix-dev@lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
On Fri, Jun 6, 2014 at 10:20 AM, Vladimír Čunát wrote: > On 06/06/2014 08:59 AM, Ertugrul Söylemez wrote: > >> When we use priorities generously we could avoid a lot of delay even in >> less critical cases. >> > > The main problem I see is that normally you don't want to release a > channel until *all* parts have rebuilt. > +1 Rebuilding for a server that runs, say ssh, apache, nginx, postfix and a few such services takes maybe 2% of the time required to build a full desktop distribution. I think being able to release packages used on public facing servers could be prioritized over, say LibreOffice, Qt, Webkit etc. If the system environment is not "polluted" by the desktop packages, it could be possible to upgrade the system environment before user environments that needs one or two orders of magnitude more time to compile. Calculating the transitive closure for all nixos modules / services run by systemd is one way to prioritize. A populatiry contest could be added to that. Alexander > > We do have meta.schedulingPriority, but it's used little, and from earlier > discussions I think it's really hard to objectively determine which > packages are more important than others ;-) > > BTW, aborting jobs would need to be done very carefully, because we have > some jobs that run for hours, so that could lead to wasting lots of time. > > > Vlada > > > > ___ > nix-dev mailing list > nix-dev@lists.science.uu.nl > http://lists.science.uu.nl/mailman/listinfo/nix-dev > > ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
On 06/06/2014 08:59 AM, Ertugrul Söylemez wrote: When we use priorities generously we could avoid a lot of delay even in less critical cases. The main problem I see is that normally you don't want to release a channel until *all* parts have rebuilt. We do have meta.schedulingPriority, but it's used little, and from earlier discussions I think it's really hard to objectively determine which packages are more important than others ;-) BTW, aborting jobs would need to be done very carefully, because we have some jobs that run for hours, so that could lead to wasting lots of time. Vlada smime.p7s Description: S/MIME Cryptographic Signature ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Openssl and fast security updates
On Thu, 05 Jun 2014 23:39:34 +0200 Vladimír Čunát wrote: > Hydra has and uses priorities. Anyway, building OpenSSL itself is very > quick, but rebuilding all that (transitively) depends on it is worse. > And there are CVE fixes for stdenv stuff sometimes (glibc)... Yes, and the basic idea is that you could have high priority packages like OpenSSL, OpenVPN and nginx. Whenever Hydra sees a job of higher priority it starts doing it (potentially aborting whatever it is currently doing). Once all jobs of the same priority are done, it runs the tests of the same priority and updates the channel. Then it goes to the next highest priority. That way security updates won't take longer than necessary. When we use priorities generously we could avoid a lot of delay even in less critical cases. Greets, Ertugrul -- Ertugrul Söylemez ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
