Re: Status of "GuixOps"?
Hi! "Thompson, David" skribis: > On Sun, Sep 17, 2017 at 2:34 PM, Hartmut Goebel > wrote: [...] >> Q1: I did not follow the development closely, but I seem to recall that >> there is some guix sub-command for configuring a remote system. But >> grepping the manual for "remote", I did not find it, neither one of the >> commands did attract me. How is it called? > > I haven't tried it but I think Ludo said that it was going to be a > flag to 'guix system'. Has that code been merged? Not being able to > easily connect to a remote daemon is what blocked me when I first > tried writing 'guix deploy'. The idea that I had in mind is to support: guix system reconfigure --target=host.example.org config.scm which would build locally and send the new system (or build it directly on the target), and then run remotely the effectful bits of ‘reconfigure’: running activation scripts, upgrading Shepherd services. To get there, the idea I have is to: 1. Isolate the effectful bits in (guix scripts system) and wrap them in gexps. 2. Provide ‘eval-gexp’ (which would combine ‘build-derivations’ and ‘eval’) and ‘eval-gexp-remotely’ (likewise, but it would use Guile-SSH’s ‘node-eval’ to evaluate the gexp remotely.) 3. Change (guix scripts system) to use ‘eval-gexp’ or ‘eval-gexp-remotely’ depending on whether --target was found. This is just a subset of the functionality in you envision for ‘guix deploy’, David, but it’s a good building block and useful in its own. > I don't think that is the best approach, or at least it should not be > an approach that GuixSD users *have to* use. I use Chef at work, and > the equivalent technology there is called "encrypted data bags" > (terrible name, I know). They are a pain and thus we do not manage > secrets with it. You're right, everything in the store is > world-readable, therefore you should never store secrets there. The > most important thing to know is that secrets are stateful, which > contrasts with many things in Guix that are stateless. Secrets can't > be configured at build-time, it has to be done at runtime. So how do > we get secrets onto machines? I think this is a job for GuixSD system > services. Let's take for example a web application that connects to a > MariaDB database. When the web application service starts, it will > fetch the database password from the secret store and initialize the > application with it by whatever means is supported (command line flag, > environment variable, etc.). The secrets store could be a local > GPG-encrypted file (getting the decryption key onto the machine is > non-trivial, of course) or an external service. To give you an idea > of the possibilities available when it comes to secrets management, > I'll give a more complex example. All of my experience with this is > with Amazon web services, so if you don't know AWS stuff you can skip > this part but I'll describe it anyway for people that might be > interested: EC2 instances are assigned an IAM role that allows them to > decrypt a set of secrets that have been encrypted using a KMS key. > The encrypted secrets happen to be stored in a DynamoDB table. A > command line tool for fetching the secrets is installed on the > instances and is integrated into the application startup processes > that need secrets. When we want to rotate a key, we replace the > DynamoDB record with a new one and restart the relevant services. > Whether the secrets are stored in an encrypted file on each machine or > in some external database, the basic process is the same: applications > will receive their secrets when they are started, not when the system > is configured. Sounds interesting. We don’t have a good story on how to deal with secrets currently, and this is problematic for deployments. >> Q3: One of DepOps' main features for me is easy use and the automatic >> refresh of Let's Encrypt certificates. Basically I just say: "Create >> certificates for hostnames A, B, C" and everything happens >> automatically: Configuration of nginx, creating the CSR, requesting the >> certificate, renewal, etc. What is the status for something like this >> for GuixSD? > > We have an nginx service, so you can write a configuration that points > to certs generated by Let's Encrypt. The cron service could be > configured to periodically run the Let's Encrypt tool that refreshes > the certificates periodically. The pieces are all there, you just > need to put them together. This is what I plan to do when I finally > get around to switching my VPS from Debian to GuixSD. Since this is > all just Scheme code, we could write abstractions that make this > common use-case "just work" for people so they don't have to worry > about the details. One step at a time. :) There’s even a patch that brings a Cerbot service! https://bugs.gnu.org/26685 People commented on it but the patch still needs love to reach its final form. Maybe you can help? :-) Thanks, Ludo
Re: Status of "GuixOps"?
On Sun, Sep 17, 2017 at 2:34 PM, Hartmut Goebel wrote: > Hi, > > in Ludo's presentation at GHM he presented "GuixOps" on a slide. What is > the status of this approach? I'm very interested in trying it out and > contributing. I made a wip-deploy branch years ago that allowed you to describe a cluster of machines and then launch them all simultaneously with qemu. It was an interesting prototype but nothing has been done since. I have since become a full-time DevOps person and have some opinions about how 'guix deploy' ought to work. The most important thing is that there needs to be a generic interface for creating/updating/destroying machines that can be implemented for bare metal machines on some LAN, VMs at some VPS provider, VMs at AWS, VMs on OpenStack, etc. I think it's also important to support both in-place updates and "immutable deploys". A bare metal machine would need an in-place update, but an Amazon EC2 instance in an auto-scaling group could be replaced entirely. As a long term thing, I think that we need to leave a door open for supporting the all-inclusive infrastructure creation tools like OpenStack's HEAT templates and Amazon's CloudFormation. I use CloudFormation extensively at my day job and a tool that just managed EC2 instances for me wouldn't cut it. In summary I think that we need to think carefully about what a convenient interface is for someone managing a few personal servers and implement that without making it impossible to scale up to more "enterprise" use-cases later. > I contributed to DebOps when it was "young". So my point of view is > influenced by how DebOps works. DebOps is a collection of interoperating > role/recipes for Ansible. Debops has become quite complex and I would > like to migrate to GuixSD for new systems. Yeah, anything based on the status quo configuration management tools is bound to be overly complex. With Guix we can do s much better, we just need some free time, a plan, and a few good hackers. :) > Q1: I did not follow the development closely, but I seem to recall that > there is some guix sub-command for configuring a remote system. But > grepping the manual for "remote", I did not find it, neither one of the > commands did attract me. How is it called? I haven't tried it but I think Ludo said that it was going to be a flag to 'guix system'. Has that code been merged? Not being able to easily connect to a remote daemon is what blocked me when I first tried writing 'guix deploy'. > Q2: DebOps has some tooling to securely store credentials, certificates, > etc. It uses a gpg-encrypted container which is mounted using FUSE. When > I unlock this container, the appropriate data is transferred to the > target system. How can this be handled with GuixSD? AFAIU with GuixSD > all data in the system-configuration is world-readable in the store. So > how can I automatically transfer e.g. passwords and private keys the the > target system? I don't think that is the best approach, or at least it should not be an approach that GuixSD users *have to* use. I use Chef at work, and the equivalent technology there is called "encrypted data bags" (terrible name, I know). They are a pain and thus we do not manage secrets with it. You're right, everything in the store is world-readable, therefore you should never store secrets there. The most important thing to know is that secrets are stateful, which contrasts with many things in Guix that are stateless. Secrets can't be configured at build-time, it has to be done at runtime. So how do we get secrets onto machines? I think this is a job for GuixSD system services. Let's take for example a web application that connects to a MariaDB database. When the web application service starts, it will fetch the database password from the secret store and initialize the application with it by whatever means is supported (command line flag, environment variable, etc.). The secrets store could be a local GPG-encrypted file (getting the decryption key onto the machine is non-trivial, of course) or an external service. To give you an idea of the possibilities available when it comes to secrets management, I'll give a more complex example. All of my experience with this is with Amazon web services, so if you don't know AWS stuff you can skip this part but I'll describe it anyway for people that might be interested: EC2 instances are assigned an IAM role that allows them to decrypt a set of secrets that have been encrypted using a KMS key. The encrypted secrets happen to be stored in a DynamoDB table. A command line tool for fetching the secrets is installed on the instances and is integrated into the application startup processes that need secrets. When we want to rotate a key, we replace the DynamoDB record with a new one and restart the relevant services. Whether the secrets are stored in an encrypted file on each machine or in some external database, the basic process is the same: applications will rece
Re: Status of "GuixOps"?
Hartmut Goebel writes: > Hi, > > in Ludo's presentation at GHM he presented "GuixOps" on a slide. What is > the status of this approach? I'm very interested in trying it out and > contributing. > > I contributed to DebOps when it was "young". So my point of view is > influenced by how DebOps works. DebOps is a collection of interoperating > role/recipes for Ansible. Debops has become quite complex and I would > like to migrate to GuixSD for new systems. > > Q1: I did not follow the development closely, but I seem to recall that > there is some guix sub-command for configuring a remote system. But > grepping the manual for "remote", I did not find it, neither one of the > commands did attract me. How is it called? There's a vey out of date branch on git origin called wip-deploy. It needs a lot more work! > Q2: DebOps has some tooling to securely store credentials, certificates, > etc. It uses a gpg-encrypted container which is mounted using FUSE. When > I unlock this container, the appropriate data is transferred to the > target system. How can this be handled with GuixSD? AFAIU with GuixSD > all data in the system-configuration is world-readable in the store. So > how can I automatically transfer e.g. passwords and private keys the the > target system? Not sure the right answer for this one :) But the right system might be user-hackable since Guix is Just Scheme (TM)? Probably the right route is to remote-copy the files while pushing the new state of the system over. Maybe having a loopback device with that data mounted in it is indeed a good idea, I don't know. > Q3: One of DepOps' main features for me is easy use and the automatic > refresh of Let's Encrypt certificates. Basically I just say: "Create > certificates for hostnames A, B, C" and everything happens > automatically: Configuration of nginx, creating the CSR, requesting the > certificate, renewal, etc. What is the status for something like this > for GuixSD? There's a wip-lets-encrypt branch on origin too! In fact I'm using it on a server! I'd really like to work on guix-deploy but I won't be able to until next year. It sounds like you have experience hacking similar systems; maybe look at wip-deploy and read David Thompson's old thread about it? (I'm too tired to look it up...) Happy hacking! I'm off for happy sleeping. :) - Chris
Re: Status of "GuixOps"?
Hi Hartmut, > in Ludo's presentation at GHM he presented "GuixOps" on a slide. What is > the status of this approach? GuixOps does not exist yet. > Q1: I did not follow the development closely, but I seem to recall that > there is some guix sub-command for configuring a remote system. But > grepping the manual for "remote", I did not find it, neither one of the > commands did attract me. How is it called? It does not exist yet, but it’s one of the first targets. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net
Status of "GuixOps"?
Hi, in Ludo's presentation at GHM he presented "GuixOps" on a slide. What is the status of this approach? I'm very interested in trying it out and contributing. I contributed to DebOps when it was "young". So my point of view is influenced by how DebOps works. DebOps is a collection of interoperating role/recipes for Ansible. Debops has become quite complex and I would like to migrate to GuixSD for new systems. Q1: I did not follow the development closely, but I seem to recall that there is some guix sub-command for configuring a remote system. But grepping the manual for "remote", I did not find it, neither one of the commands did attract me. How is it called? Q2: DebOps has some tooling to securely store credentials, certificates, etc. It uses a gpg-encrypted container which is mounted using FUSE. When I unlock this container, the appropriate data is transferred to the target system. How can this be handled with GuixSD? AFAIU with GuixSD all data in the system-configuration is world-readable in the store. So how can I automatically transfer e.g. passwords and private keys the the target system? Q3: One of DepOps' main features for me is easy use and the automatic refresh of Let's Encrypt certificates. Basically I just say: "Create certificates for hostnames A, B, C" and everything happens automatically: Configuration of nginx, creating the CSR, requesting the certificate, renewal, etc. What is the status for something like this for GuixSD? -- Regards Hartmut Goebel | Hartmut Goebel | h.goe...@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible |