Re: [Nix-dev] Making a custom installation tarball
I’m no expert but this totally looks like an ordinary derivation, just `nix-build` it. On Fri Dec 26 2014 at 12:14:51 Nikita Karetnikov nik...@karetnikov.org wrote: I’d like to make an installation tarball with a custom set of packages. I’ve been pointed to [1], which is a Hydra job. Is it possible without Hydra? Could anyone explain in detail what I need to do? (It’d be great to create a wiki page for this.) [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/top- level/make-tarball.nix ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Making a custom installation tarball
… passing in your `nixpkgs` and `false` of course. On Sat Dec 27 2014 at 11:14:31 Kirill Elagin kirela...@gmail.com wrote: I’m no expert but this totally looks like an ordinary derivation, just `nix-build` it. On Fri Dec 26 2014 at 12:14:51 Nikita Karetnikov nik...@karetnikov.org wrote: I’d like to make an installation tarball with a custom set of packages. I’ve been pointed to [1], which is a Hydra job. Is it possible without Hydra? Could anyone explain in detail what I need to do? (It’d be great to create a wiki page for this.) [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/ make-tarball.nix ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Modifying the init system (introducing S6 supervisionsuite)
On 2014-12-27 09:57, Michael Raskin wrote: I'm afraid I fail to see how S6 is better than systemd, except that the author is totally worried about the future of Linux with systemd. S6 has logging without the unfixable bugs of JournalD S6 doesn't insist on breaking xinit/startx-like functionality. S6 doesn't barf if you put your swap into fstab. SystemD has its unique WONTFIX bugs that no other init system shares (sometimes not even UseLessD, although it is a minimalistic fork of SystemD). For me these bugs are showstoppers. I recently had a task at work that was _much_ simpler by running 3 identical X servers. Please provide links to specific criticsms, bugs, etc. rather than just speaking in generalities. How does systemd insist on breaking xinit/startx-like functionality, for instance? Are there open WONTFIX bugs on it? If so, please link. You haven't backed up bart if you put your swap into fstab. (And so on.) I don't particularly care whether you personally like or dislike systemd, but can we please at least keep critism founded on some sort of factual basis rather than just asserting things? (Links to bug reports, etc.) ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] Dataflow
Oh so that there are harmless forms of non-determinism. I totally forgot about things like that. So there is no way for Nix to automatically know if the package is built correctly? You still need the user to try the package and see if it functions correctly. I know that some packages come with tests. Mongrel2's built from source, comes with a bunch of tests, that you can run at the end, to verify that the built version works according to its tests. On 27/12/2014 8:10 AM, Vladimír Čunát wrote: On 12/26/2014 04:23 AM, Roger Qiu wrote: Given that the build isn't deterministic, is there any provisions for Nix to force it to be deterministic? We are blocking some frequent sources of non-determinism (e.g. __DATE__), but it's practically impossible to get rid of all (e.g. UID). Fortunately, some upstreams also find it's good to have a (more) deterministic output. Vladimir ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev -- Founder of Matrix AI http://matrix.ai/ +61420925975 ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Yay! The idea is even better than I expected! 2014-12-27 4:48 GMT-02:00 Ertugrul Söylemez ert...@gmx.de: Note that systemd is not just an init system. This is my first problem - systemd is a bit bloated and mysterious. I don't like the idea of big blobs doing everything on a system. It becomes harder to maintain and modify, and even to modularize. But, modifyng the way we treat about services and modules is revolutionary! It is just like Nix is for package management, applied to system maintainance :) As a final bonus this is so difficult and ugly to solve with systemd that we would feel a much greater temptation to get rid of it. =) Let me know what you think. If this idea finds acceptance, I'm happy to develop a formal model and a prototype. Well, we need also to do it in a way we won't be handcuffed by a particular init/supervisor implementation. We can even to switch the supervisor/init without worries about a complete rewriting of services. There is some I am searching about: * http://www.gentoo.org/proj/en/base/openrc/ * http://supervisord.org/ * http://uselessd.darknedgy.net/ * http://skarnet.org/software/s6/ Some comparisons: * http://wiki.gentoo.org/wiki/Comparison_of_init_systems * https://lwn.net/Articles/578209/ * https://lwn.net/Articles/578210/ I am not even a junior-seasoned programmer, not even on functional paradigm, but I wanna help. Greets, Ertugrul Greets! Anderson ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
On Sat, Dec 27, 2014 at 7:48 AM, Ertugrul Söylemez ert...@gmx.de wrote:
I think as a first step to get rid of systemd and gain a much more
sensible services model as well we should move away from NixOS modules
for services.
The module system is made in such a way that one can build on top of
the work of the other without having any computer scientist knowledge.
This is also the reason why I am pushing to abolish functions as
option definitions. Most of the time if you want to use a function,
you can express it as a set of input (option declaration) which are
being processed to provide a lower level of abstraction given by
another module (option definitions).
The module system does not prevent us from doing anything, it provides
a new way of formatting which makes things declarative and extensible.
If a module does not serve what you expect, you can always write
another module and use it instead of the previous one. Having
multiple instances of a service is an issue with the module, not a
limitation in the module system.
Being able to reason about the resulting system is very
important.
Have you tried nixos-option ?
I would prefer and do propose an algebraic solution (view services as a
toolbox from which you can pick the services you want and compose them
together). A services configuration then might look like this:
services =
bitlbee { port = 1; stateDir = /var/lib/bitlbee1; }
bitlbee { port = 10001; stateDir = /var/lib/bitlbee2; }
nginx { httpConfig = ...; }
postfix { postMasterAlias = blah; }
First, unamed instance are not modular. This is why we are moving away
from types.listOf and that the suggested alternative is a backward
compatible version named types.loaOf (list or attribute set of).
Second, I see 2 additional concepts and I do not think I would be able
to explain these to my grand-mother, Can you?
I think what you are trying to express can be done with the module
system, maybe you can try to make a prototype using the module system
and a new top-level system-proto option which is declaring submodule
for every service that you want to declare. Maybe we can learn
something from this experiment, in order to better design future
services.
In any cases, I do not think we want to switch to something without
having a working prototype to experiment with. Do not take that
personally, this was also the case with the module system [1].
[1] http://comments.gmane.org/gmane.linux.distributions.nixos/1187
--
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Hi all,
At Zalora, we are gradually switching to using defnix
https://github.com/zalora/defnix. It is still very much under active
development (read: may change at any time), and the underlying implementation
does use systemd, but the service/functionality interface (check out the
defnixos directory) are specifically designed to be init system agnostic. In
particular, behaviors like socket activation, cron jobs, restart on failure,
etc. are all managed via independent programs/library functions that don’t
require any specific functionality from the init system. Eventually even “start
a bunch of services” will be managed like that as well.
~Shea
On Dec 27, 2014, at 6:48 AM, Ertugrul Söylemez ert...@gmx.de wrote:
Hi there Anderson,
I want to know if it's possible to use an alternate to Systemd as init
system on NixOS.
this has been asked many times now, and I'm in favour of switching to an
alternative. Unfortunately it's not easy, because a considerable
portion of NixOS currently depends on systemd, most notably the services
infrastructure and pretty much all container-related stuff. Note that
systemd is not just an init system.
One thing most of us seem to agree about is that conceptually NixOS does
not depend as much on something like systemd as most other distributions
do. In principle most of what systemd does could be handled by the
activation script and perhaps a custom daemon (or even just a set of
scripts) that allows us to manage the services that are part of the
current system derivation. NixOS can deal with things like cgroups or
ACPI events in a more sensible, repeatable and transparent way.
I would go as far as to say that systemd is the number one thing that
cripples us in many ways. Switching to it was a huge step back from the
ideals of NixOS, because it represents all the traditional views on what
a system should look like: a giant pile of interconnected mutable
variables. Remember that immutability is not only about the filesystem.
How to get rid of systemd
=
I think as a first step to get rid of systemd and gain a much more
sensible services model as well we should move away from NixOS modules
for services. At this point we are trapped in the traditional view that
services preexist and we get to enable/disable them. Our system is
defined by a large set of switches and global settings. This needlessly
robs us of a lot of flexibility, for example to run multiple instances
of the same daemon or to give us a choice to use our already running
preferred webserver when installing a web application as a service.
It also corrupts our ability to reason about the resulting system,
because enabling a certain service might trigger settings in some other
part of the system configuration to change. This has actually bitten me
in the past. Being able to reason about the resulting system is very
important.
I would prefer and do propose an algebraic solution (view services as a
toolbox from which you can pick the services you want and compose them
together). A services configuration then might look like this:
services =
bitlbee { port = 1; stateDir = /var/lib/bitlbee1; }
bitlbee { port = 10001; stateDir = /var/lib/bitlbee2; }
nginx { httpConfig = ...; }
postfix { postMasterAlias = blah; }
The big difference is that now services are first class values and form
a monoid under composition, which brings a number of advantages:
* It allows us to run multiple instances of a daemon with slightly
different configurations, as those are really just similar but
otherwise unrelated services.
* Services can take services as arguments, which would become the
primary way to construct a dependency graph. Think of a web
application depending on a webserver or the other way around.
* We could develop a fully automated way for services in different
containers to communicate with each other.
* Now people like me can actually use equational reasoning to
understand and make predictions about the resulting system. I can
use the fact that the `` operator is associative, that certain
services are idempotent with respect to `` and also that certain
functions are morphisms in a rich category. For example if the
nginx function is a monoid morphism, then I know that regardless of
how I compose multiple webservers, the result will be a sensible
combination of either separate daemons or multiple configurations.
This would be extremely valuable.
The last point is very important, because it basically tells that a
service is not directly related to what programs will run on the
resulting system. Even if you compose ten nginx services, there will be
only one nginx *program* with a suitable configuration file to represent
this composition. It also means that when a service itself comes with
nginx, then adding
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
On 27 December 2014 at 07:48, Ertugrul Söylemez ert...@gmx.de wrote: One thing most of us seem to agree about [...] Maybe that's true (I don't see a consensus on the list, only the usual sparse FUD. I don't follow IRC) but arguments that start this way always give me the willies. [...] is that conceptually NixOS does not depend as much on something like systemd as most other distributions do. Something like systemd? Nix doesn't magically make *nix less *nix, for better or worse. The meat of systemd is still service management, which NixOS needs as much as any distribution. Switching to it was a huge step back from the ideals of NixOS, because it represents all the traditional views on what a system should look like: a giant pile of interconnected mutable variables. That's what a running system is, though. Immutability *is* just for file systems. You plant a pretty tree, and hope it describes a sane system that will do the right thing most of the time. Then, something will happen that will kick all your pretty algebraic assumptions off the table very quickly. Most of your OS is in buggy C, whether you like it or not... How to get rid of systemd = Uh. You seem to have some very nice ideas about describing services (or something better), if still very rough around the edges. But if you can't implement your ideas on top of systemd, so much for init-agnosticism. And probably conceptual purity, too. What's left of your argument then, besides I don't like the damn thing? You'll be writing your own daemon supervision code before you know it. With more bugs. Those who don't understand systemd are doomed to reinvent it. Poorly. :-) As a final bonus this is so difficult and ugly to solve with systemd that we would feel a much greater temptation to get rid of it. =) Because your plan is centred entirely around it not being systemd. Regards, T G-R ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Hi Nicolas, I think as a first step to get rid of systemd and gain a much more sensible services model as well we should move away from NixOS modules for services. The module system is made in such a way that one can build on top of the work of the other without having any computer scientist knowledge. I hope you realise that my proposal makes everything *simpler* both for module/service writers and configuration authors. Services will be simpler and less tedious to write (simple functions). They will compose naturally. The algebraic part is *design* and perhaps terminology. It enables *those who want* to exploit the additional structure especially for equational reasoning and containerisation, but understanding abstract algebra is in no way a requirement for the regular NixOS user. Let me respond to your objection anyway: If we can have a configuration DSL that gives us a major benefit, including much simpler modules, provable safety, full automation of tedious tasks, etc., then I'm sorry for the I don't want to learn something new cavemen, but I'm more than happy to leave them far behind. Exploring and using new ideas require acquiring new knowledge from time to time. NixOS itself is a great example of this. One thing I'd like to see *by default* in NixOS is containerisation, even for the regular desktop user who only uses Firefox and LibreOffice, in the spirit of Qubes OS. It's technically possible, supported by Linux by and does not require setup. This is the motivating purpose for my proposal. Let me give you a very compelling example: services = withBridge bridgeConfig (br: container (nginx ... openssh ...) container (openvpn ... br ...)) openssh ... This simple configuration would define a whole network of containerised applications. This is also the reason why I am pushing to abolish functions as option definitions. Most of the time if you want to use a function, you can express it as a set of input (option declaration) which are being processed to provide a lower level of abstraction given by another module (option definitions). Perhaps NixOS with its *functional* package management and *functional* configuration language is just the wrong distribution for you. ;) /joke Jokes aside, when a function makes sense, I go for it, because functions make things simpler. Please stop pushing that way. For those who aren't programmers, functions are just syntax anyway. The module system does not prevent us from doing anything, it provides a new way of formatting which makes things declarative and extensible. I won't go as far as to argue that declarative is a meaningless term. Also I'm not sure what you mean by formatting (syntax?). The extensibility of the current module system is limited in the same way global variables limit the extensibility of a program. Everything is global, and every module can affect every other module in completely unrestricted and unpredictable ways. Any kind of data feedback requires something you would classify as computer scientist knowledge (least-defined fixed points of nonstrict recursive functions). Again, this has bitten me in the past. There is no way to reason about your configuration other than to look into the resulting derivation. This is a bad situation. The more modules and options we get, the more difficult it will get to keep everything together. If a module does not serve what you expect, you can always write another module and use it instead of the previous one. Having multiple instances of a service is an issue with the module, not a limitation in the module system. I guess we have very different philosophical views, so it is difficult to communicate the advantage of my proposal to you without actually implementing a prototype. I admit that to a non-FP programmer the concept I propose must look very similar to what we have today except that you can add the same service multiple times. The advantage is only apparent when you start ot use the functional and algebraic features. Otherwise all that changes is that the syntax becomes simpler (still a small benefit though). Being able to reason about the resulting system is very important. Have you tried nixos-option ? This question suggests that you are unfamiliar with equational reasoning. To answer your question: Yes, I'm aware of nixos-option, and it's orthogonal to the issue I'm raising. Also I should point out explicitly that I'm not proposing to replace the module system. I'm proposing not to use the module system for services. If my proposal works as intended and finds acceptance we will quickly find other things that would benefit from this, for example the network configuration. I would prefer and do propose an algebraic solution (view services as a toolbox from which you can pick the services you want and compose them together). A services configuration then might look like this:
Re: [Nix-dev] How to get rid of systemd (was: Modifying the initsystem (introducing S6 supervision suite))
Hi Michael, Well, with NixPkgs having multiple instances of a package installed requires zero support from the package expression itself. I do think that services becoming more like packages would be a good thing, but currently I have no idea how to do this except by just giving up the module system. I don't think we need to or even should give up the module system. Some settings are global (like the hostname and most boot options), and the module system makes those a lot easier to manage. This proposal is only for services and service-like entities. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
Hi there Tobias, One thing most of us seem to agree about [...] Maybe that's true (I don't see a consensus on the list, only the usual sparse FUD. I don't follow IRC) but arguments that start this way always give me the willies. I understand why, but I'm not very good at English rhetoric. Please don't let that overshadow the rest of the mail. [...] is that conceptually NixOS does not depend as much on something like systemd as most other distributions do. Something like systemd? Nix doesn't magically make *nix less *nix, for better or worse. The meat of systemd is still service management, which NixOS needs as much as any distribution. NixOS needs service *management*, but it doesn't necessarily need a service *manager*. In traditional distributions services are managed by destructive update (enable/disable). Normally we don't have that. Switching to it was a huge step back from the ideals of NixOS, because it represents all the traditional views on what a system should look like: a giant pile of interconnected mutable variables. That's what a running system is, though. Immutability *is* just for file systems. You plant a pretty tree, and hope it describes a sane system that will do the right thing most of the time. Then, something will happen that will kick all your pretty algebraic assumptions off the table very quickly. Most of your OS is in buggy C, whether you like it or not... The assumptions are not that the programs will do what I expect them to do, but rather that two instances of nginx each with one virtual host is equivalent one instance of nginx with two virtual hosts. This does not assume anything about the correctness of nginx itself, but it does help building systems, containers and networks componentwise. You seem to have some very nice ideas about describing services (or something better), if still very rough around the edges. But if you can't implement your ideas on top of systemd, so much for init-agnosticism. And probably conceptual purity, too. This may be a misunderstanding. Init-agnosticism means that the complexity of translating this concept to an actual set of services that systemd can work with is the responsibility of a single function/script/whatever. We achieve separation of concerns. Should we, at any point, decide to turn our back on systemd or even just provide the option to use another manager, then all we need to do is to reimplement this one function/script/whatever. Those who don't understand systemd are doomed to reinvent it. Poorly. :-) I do understand systemd very well and still don't like it. However, reinventing it is certainly not something I have planned for the near future. =) As a final bonus this is so difficult and ugly to solve with systemd that we would feel a much greater temptation to get rid of it. =) Because your plan is centred entirely around it not being systemd. Not at all. It would be a nice bonus, but I doubt that it's going to happen. The motivation for my proposal, as the motivation for most of the things I do right now, is easy, composable, transparent containerisation as well as a configuration DSL that is easy to reason about. Greets, Ertugrul ___ nix-dev mailing list nix-dev@lists.science.uu.nl http://lists.science.uu.nl/mailman/listinfo/nix-dev
Re: [Nix-dev] How to get rid of systemd (was: Modifying the init system (introducing S6 supervision suite))
I'll just jump in and say that if you is going to donate time to do this, Ertugrul, then I'm all for it. It seems like it is possible to make the service system a lot leaner, smaller, and nicer. This is especially true when services are in containers, or NixOS is used in docker containers. For those that don't really get what Ertugrul describes, looking at what is called orchestration systems for docker gives a sense of what I think he is talking about. An orchestration system is a way of defining how multiple docker containers are connected to each other. The orchestration system itself doesn't necessarily need to keep the state of the system, because the cloud system it operates on can just spin up new instances, or the cloud system has a service discovery mechanism, or which service is active is handled by a load balancer. Similarly, on a host, when we have cgroups/containers/namespaces, we can spin up a new service in a container and redirect traffic to it by fiddling with namespaces, routing settings etc. We can make I am mostly interested in seeing how the monoid On Sat, Dec 27, 2014 at 6:53 PM, Ertugrul Söylemez ert...@gmx.de wrote: Hi there Tobias, One thing most of us seem to agree about [...] Maybe that's true (I don't see a consensus on the list, only the usual sparse FUD. I don't follow IRC) but arguments that start this way always give me the willies. I understand why, but I'm not very good at English rhetoric. Please don't let that overshadow the rest of the mail. [...] is that conceptually NixOS does not depend as much on something like systemd as most other distributions do. Something like systemd? Nix doesn't magically make *nix less *nix, for better or worse. The meat of systemd is still service management, which NixOS needs as much as any distribution. NixOS needs service *management*, but it doesn't necessarily need a service *manager*. In traditional distributions services are managed by destructive update (enable/disable). Normally we don't have that. Switching to it was a huge step back from the ideals of NixOS, because it represents all the traditional views on what a system should look like: a giant pile of interconnected mutable variables. That's what a running system is, though. Immutability *is* just for file systems. You plant a pretty tree, and hope it describes a sane system that will do the right thing most of the time. Then, something will happen that will kick all your pretty algebraic assumptions off the table very quickly. Most of your OS is in buggy C, whether you like it or not... The assumptions are not that the programs will do what I expect them to do, but rather that two instances of nginx each with one virtual host is equivalent one instance of nginx with two virtual hosts. This does not assume anything about the correctness of nginx itself, but it does help building systems, containers and networks componentwise. You seem to have some very nice ideas about describing services (or something better), if still very rough around the edges. But if you can't implement your ideas on top of systemd, so much for init-agnosticism. And probably conceptual purity, too. This may be a misunderstanding. Init-agnosticism means that the complexity of translating this concept to an actual set of services that systemd can work with is the responsibility of a single function/script/whatever. We achieve separation of concerns. Should we, at any point, decide to turn our back on systemd or even just provide the option to use another manager, then all we need to do is to reimplement this one function/script/whatever. We can use a monoid system to construct configurations, but the socket activation standard for example, is centered on optimizing the activation script itself. What are your thoughts on the activation script? I can easily see that using systemd might be overkill and way too complex for a container-based system, so I think there is something to research here. I also think upgrading services doesn't really seem to work in systemd or in the current setup. Similarly to how, when using a distributed docker setup, we have a load-balancer that atomically switches instances, we should not need to take down the old instance service before the new one is created. Rather, the upgraded service should be started in isolation (using containers), and after ensuring that it has started, is working etc, then should the switch happen using namespaces, routing entries etc. This should be the preferred way to deal with non-transactional services (i.e. non-database stuff). The idea that the old service must be stopped before the new one is started is based on what I think is a conflation of concerns, namely treating the whole service state as global state. Instead a lot of services can be treated as a sequence of isolated containers, and a small set of load-balanced, mutable service entry
