Re: Current best practices for system configuration management?

[David Christensen]

On 6/22/24 11:33, Dmitrii Odintcov wrote:

Hi all,


Sorry to resurrect an old-ish thread, but I am facing the exact same
task, minus the know-how.

Basically I am looking to pre-configure a number of Debian setups -
let's say, "server", "laptop" and "PC" - that would contain sets of
packages to install (or uninstall), configuration files (including but
not limited to /etc/), and possibly arbitrary scripts to execute upon
installation (e.g. to make more precise edits to configs).

I would like to store these in some central, git-controlled location
where I could pull them from to my target machines. This kind of setup
could be rather low-level, such as configuring APT sources and
preferences, installing and configuring systemd-networkd, etc.
Ideally, I would also be able to incorporate these into a Debian
installer so I could get a new machine "up and running" with my
defaults in one go.

It is worth adding that I am not looking to build an "infrastructure",
and no setup *between* these machines (networking, file sharing, etc.)
is necessary.

Generally, I would rather avoid complicated tools like Ansible and
those with large dependencies such as interpreted languages (except
Perl).

So far, equivs and config-package-dev appear most relevant, but the
former is somewhat lacking in documentation (or I am in
documentation-finding ability), and the latter seems to be focused on
config editing/deployment rather than package installation.


Would be grateful for some advice!



I think the "best" answer depends upon the scale of your installation.


I have a SOHO network with a dozen or so Debian, Windows, macOS, and iOS 
clients, a FreeBSD/ZFS CVS, SSH, and Samba server, and a FreeBSD/ZFS 
backup server.  For system administration, including configuration 
management, I have gone down the do-it-yourself (DIY) path using 
lowest-common-denominator FOSS command line tools.



After the network, ssh(1), and rsync(1), the most valuable tool for 
system administration (including configuration management) has been a 
version control system.  I prefer CVS over Git because CVS provides 
monotonically-increasing MAJOR.MINOR version numbers via keywords (e.g. 
$Revision$) that can be included inside managed plaintext files.  I 
create a CVS project for everything I want to manage.  For OS's, I check 
in a sysadmin log file with my notes and console sessions, a list of 
packages installed, various reports that I have run, and any system 
configuration files that I have modified, added, or deleted.



After version control, the next most valuable tool has been scripting. 
Over the years, I have written numerous scripts to automate repetitive 
chores.  When the needs are simple, I write Bourne shell scripts.  When 
I want more power, I upgrade to Perl.  All of the scripts are checked in 
to CVS.



I expect Ansible, Puppet, etc., would work at my scale, but are designed 
for large installations.



David

Re: Current best practices for system configuration management?

[Dmitrii Odintcov]

Hi all,


Sorry to resurrect an old-ish thread, but I am facing the exact same
task, minus the know-how.

Basically I am looking to pre-configure a number of Debian setups -
let's say, "server", "laptop" and "PC" - that would contain sets of
packages to install (or uninstall), configuration files (including but
not limited to /etc/), and possibly arbitrary scripts to execute upon
installation (e.g. to make more precise edits to configs).

I would like to store these in some central, git-controlled location
where I could pull them from to my target machines. This kind of setup
could be rather low-level, such as configuring APT sources and
preferences, installing and configuring systemd-networkd, etc.
Ideally, I would also be able to incorporate these into a Debian
installer so I could get a new machine "up and running" with my
defaults in one go.

It is worth adding that I am not looking to build an "infrastructure",
and no setup *between* these machines (networking, file sharing, etc.)
is necessary.

Generally, I would rather avoid complicated tools like Ansible and
those with large dependencies such as interpreted languages (except
Perl).

So far, equivs and config-package-dev appear most relevant, but the
former is somewhat lacking in documentation (or I am in
documentation-finding ability), and the latter seems to be focused on
config editing/deployment rather than package installation.


Would be grateful for some advice!

Re: Current best practices for system configuration management?

[Mike Castle]

On Sat, Apr 20, 2024 at 4:40 PM Mike Castle  wrote:
> Thanks for all of the commentary so far.
>
> Once I get something working, I will *try* to remember to follow up
> here with what I've managed to cobble together.

I have done quite a bit of research and experimentation and finally
settled on a solution that seems like it will work for me:

Plain old debian/* control files along with config-package-dev.

Effectively, I've abandoned `equivs` and now just using plain old
`debhelper` with a small wrapper script.

One thing I've learned along the way is that debian/control files can
build multiple .deb files.  But the "control" files that `equivs` uses
are not really the same as regular control files, so need one per
package.  Since I created a hierarchy of packages, I needed several
configs for equivs.  Now, I can use just one, and it is MUCH faster
(about 8 seconds total).

I'm still working on replacing my old equivs stuff, but, I do have
some things I'm installing now.  Essentially my setup looks like this:
debian/control:
Maintainer: Mike Castle 
Source: mrc
Build-Depends: debhelper-compat (= 12), config-package-dev
Rules-Requires-Root: no
Standards-Version: 4.5.1
Section: metapackages
Priority: optional

Package: mrc-base
Architecture: all
Depends: ${misc:Depends},
 bc,
 cron,
 ...
 zip,

Package: mrc-mozilla
Architecture: all
Description: MRC's Mozilla apt configuration
 Lots of files under /etc/apt.
Depends: ${misc:Depends},

Package: mrc-desktop
Architecture: all
Description: MRC's desktop installation
 Graphical stuff.
Depends: ${misc:Depends},
 mrc-base,
 mrc-games,
 mrc-python,
 mrc-mozilla,

... more meta packages ...

Package: mrc-mars
Architecture: all
Description: MRC's host mars
 Host specific deps.
Provides: ${diverted-files}
Conflicts: ${diverted-files}
Depends: ${misc:Depends},
 mrc-desktop,
 mrc-development,
 mrc-virtual-machines,

... most host packages ...

A few other debian control files:
$ find debian/ -type f
debian/source/format
debian/mrc-mars.install
debian/mrc-mars.displace
debian/mrc-mozilla.install
debian/copyright
debian/changelog
debian/rules
debian/control

$ cat debian/rules
#!/usr/bin/make -f

%:
dh $@ --with=config-package

The files that match .* are the magic.  For example:
$ cat debian/mrc-mozilla.install
files/mozilla/* /

$ find files/mozilla -type f
files/mozilla/usr/share/lintian/overrides/mrc-mozilla
files/mozilla/etc/apt/sources.list.d/mozilla.list
files/mozilla/etc/apt/preferences.d/mozilla
files/mozilla/etc/apt/keyrings/packages.mozilla.org.asc

The etc/apt files come from
https://support.mozilla.org/en-US/kb/install-firefox-linux#w_install-firefox-deb-package-for-debian-based-distributions
and the lintian overrides is:
$ cat files/mozilla/usr/share/lintian/overrides/mrc-mozilla
mrc-mozilla: package-installs-apt-sources
mrc-mozilla: package-installs-apt-preferences

For the host packages, it is slightly more complicated, but not much
once I figured it out.  Again, a basic "copy in everything under this
directory" install file:
$ cat debian/mrc-mars.install
files/mars/* /

which is currently this:
$ find files/mars -type f
files/mars/etc/hostname.mrc

Then the interesting bit:
$ cat debian/mrc-mars.displace
/etc/hostname.mrc

This "dispace" file is something that the `config-package` addon takes
care of.  It sets up backing up the existing file name (/etc/hostname)
and symlinks in my version:
$ ls -l /etc/hostname*
lrwxrwxrwx 1 root root 12 May 27 08:57 /etc/hostname -> hostname.mrc
-rw-r--r-- 1 root root  5 May 27 06:52 /etc/hostname.mrc
-rw-r--r-- 1 root root  6 May 27 07:37 /etc/hostname.mrc-orig

If I purge mrc-mars, then /etc/hostname.mrc-orig is moved back to /etc/hostname.

(The way config-package-dev works is, it takes the first portion of
the package names, in this case "mrc", to identify what file names to
use.)

To drive it all, I currently use a simple wrapper script:
$ cat doit.sh
#!/bin/bash

set -e

work_dir=$(mktemp -d)
find -depth | cpio -pdm ${work_dir}/work
cd ${work_dir}/work
debuild --no-conf --no-sign --lintian-opts --info

OUTPUT=/srv/deb/packages
rm -rf $OUTPUT
mkdir -p $OUTPUT
cp ${work_dir}/*.deb $OUTPUT
cd $OUTPUT

dpkg-scanpackages . > Packages

echo "work_dir was ${work_dir}"

Not the fanciest, but "works for me" (so far).  Essentially it builds
things in /tmp because the debian packaging tool chain always drops
the files into the parent directory and leaves things that *I*
currently do not care about littering my disk.  Also, it drops
intermediate files into the debian/* directory, making it more
difficult to know what needs to go under SCM.  With this approach,
"git status" will be easier to read.

The only drawback so far, over my previous approach of one `equivs`
config per package, is that now, all built debs share the same version
number.  With the individual approach, I could add one package to a
metapackage, and when I do "apt update && apt upgrade", only the
single package (and its new deps) are processed.  Now, 

Re: Current best practices for system configuration management?

[Linux-Fan]

Mike Castle writes:


Hah!

https://lists.debian.org/debian-user/2013/08/msg00042.html


Yes, that was me > 10a ago. Transitioning from these scripts to ant allowed  
came with a few improvements:


* I switched all package building to `debuild` in favor of using more
  low-level tools for `raw` packages before.

* I was able to establish some sort of “buld upon new commit” poor man's
  local CI -- I experimented with some fully-fledged systems like Concurse CI
  and GitLab, too, but they were all too much for my local-first
  activities

* When resources were required from external locations (e.g. source code
  downloads), that set of scripts did not provide for any automatism in this
  regard. One of my continuous goals is to split between “my settings” and
  some re-downloadable files. It was only achieved partially back then and
  the ant-approach takes it much further :)

Still, in many ways the ant does not do much more than the scripts linked at the  
message above. If you are about to create your own scripts (which is  
probably a better idea than trying to use my scripts anyways :) ) the  
shellscripts may be easier to understand.


HTH
Linux-Fan

öö


pgpEtXWCbBgkx.pgp
Description: PGP signature

Mesh VPN on Debian (Was Re: Current best practices for system configuration management?)

[Andy Smith]

Hi,

On Sat, Apr 20, 2024 at 04:40:24PM -0700, Mike Castle wrote:
> Like Alex, one of my physical machines is a laptop that is not always
> on the home network.  Though I'm usually connected to *something*.
> I'm still debating whether to bother with a VPN or trying something
> like a tailnet.

For mesh VPN I really like Yggdrasil (packaged in Debian, but widely
available).

It does quite a lot of the things that people use Tailscale for, but
has the advantages of:

- Completely FOSS
- No need to contact a central authority - your nodes all
  self-organise
- Thus no limit on how many nodes you can have for free (though
  Tailscale's limit is very generous)

Like Tailscale it will detect other instances of itself on your LAN
so local traffic remains local (avoid a VPN hairpin) while you still
use the same Yggdrasil IP addresses to talk to things.

Downsides compared to Tailscale are things like:

- Not as polished a product so no hand-holding; you need to read the
  docs

- Not available on as many platforms.

  It is a single static Go binary so it's not hard to deploy if you
  can compile it, but I don't know what the story is on things like
  mobile platforms, whereas there's Tailscale apps for everything.

- I don't have personal experience but possibly it's more energy
  intensive than Tailscale which would matter a lot on mobile
  devices

There is a good introduction and comparison with some other
solutions here:


https://www.complete.org/easily-accessing-all-your-stuff-with-a-zero-trust-mesh-vpn/

I still wouldn't want to automated a config push/pull to a laptop
over a mesh VPN I think, but others have mentioned that you can do
Ansible in a pull mode.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Current best practices for system configuration management?

[Tim Woodall]

On Thu, 18 Apr 2024, Mike Castle wrote:


Now, I would like to expand that into also setting up various config
files that I currently do manually, for example, the `/etc/apt/*`
configs I need to make the above work.  For a single set of files,
manual isn't bad, but as I want to get into setting up LDAP, autofs,
and so on, it is time to explore solutions.  I only have four systems
at the moment (two physical and two virtual), so I don't think I need
something too fancy.


I do this - but it needs a bit of thought as to how you want to do it.

I'm assuming you don't care about debian policy ...

I do one of four things in packages depending on what I want to achieve
and what the package I'm configuring supports.

1.(easiest) - just drop a config file into a conf.d directory. I
use this for maintaining sources.list. Back in the buster days I was
using openssh from backports that supported this as maintaining the
config is so much easier than any of the other ways and so much easier
to fix bugs in your packages - a failing postinst script can be a pain
to resolve.

apt-mirror@aptmirror19:~ (none)$ apt-cache search bookworm-sources
bookworm-sources - Meta-package to pull in list of apt-sources for debian

(I generate these packages automatically from what is available on
deb.debian.org.)

FWIW, I also have a dev repo and a fast repo - the dev repo is a clone
where I can test out substantial changes and the fast repo only has a
couple of packages but can rebuild "the world" very quickly when I'm
testing. So I have a number of extra levels of indirection to support
this:

$ apt-cache depends bookworm-sources
bookworm-sources
  Depends: bookworm-local-sources

That is the normal package that I would install.

apt-mirror@aptmirror19:/mnt/mirror/local/main/o (master)$ apt-cache depends 
bookworm-dev-sources
bookworm-dev-sources
  Depends: bookworm-dev-main-sources

That lets me use the dev repo - and thanks to pinning, the packages in
it will replace (even downgrade) packages from local.

Ditto for my fast repo.

2. (next easiest) - add a file in /usr/share/ and then overwrite the
file in /etc by copying in postinst.

3. edit the config file without diverting it using a sed script.

4. divert the config and edit/replace it in the package.

3. and 4. in particular can make upgrades more difficult if/when you
find bugs in the way you're handling diversions and uninstalls.

Re: Current best practices for system configuration management?

[Mike Castle]

Hah!

https://lists.debian.org/debian-user/2013/08/msg00042.html

Re: Current best practices for system configuration management?

[John Crawley]

On 21/04/2024 08:40, Mike Castle wrote:

One thing Linux-Fan mentioned was `config-package-dev`.  In my OP, I
commented about ``slightly old to really old tools'', and that was one
I was thinking of.  It looks like it hasn't been touched in seven
years, and I wasn't sure if it still worked.  But that drive by
comments lends some hope.  Using it would help address Alex's concern
about modifying existing config files.  That debhelper extension is
designed precisely for that situation.  But, its age is pretty much
what inspired me to start this thread.


I've been using config-package-dev for some years now - to add some customized 
configs to existing installed packages - and it still seems to be working 
perfectly. It's just a wrapper on top of dpkg diverts so there's not all that 
much to go wrong.

--
John

Re: Current best practices for system configuration management?

[Mike Castle]

Thanks for all the suggestions so far.

Like Alex, one of my physical machines is a laptop that is not always
on the home network.  Though I'm usually connected to *something*.
I'm still debating whether to bother with a VPN or trying something
like a tailnet.

Heck, before I adopted Debian and ran my own
Linux-from-Scratch-started-before-LFS-or-Debian-existed
"distribution", I used to run $HOME over NFS over WiFi.  Sharing
configs that way would be acceptable for me.  I also used to keep my
configs in CVS and pushed them using rsync-over-ssh-as-root, hence
"current best practices".  :->

Linux-Fan's MDVL stuff looks interesting.  Everything can always be
solved by one more layer of redirection.  :->  I just have to suss out
the Java and Ant stuff to understand the basics.  Personally, I'm a
big fan of NIH-Syndrome (otherwise I wouldn't have run a LFS type
system for ~15 years).  So it is likely I would reinvent what is
happening there.

One thing Linux-Fan mentioned was `config-package-dev`.  In my OP, I
commented about ``slightly old to really old tools'', and that was one
I was thinking of.  It looks like it hasn't been touched in seven
years, and I wasn't sure if it still worked.  But that drive by
comments lends some hope.  Using it would help address Alex's concern
about modifying existing config files.  That debhelper extension is
designed precisely for that situation.  But, its age is pretty much
what inspired me to start this thread.

Interestingly enough, I sent the OP just before I went to a BayLISA
talk about Apple's PKL.  Since I tend to only use whatever comes out
of the Debian repos (my only exceptions being Firefox and an emacs
package the maintainers won't fix for bookworm), I'm unlikely to do
more than look at it,  But, between it and MDVL, I think I'm
definitely going to try to make sure I don't configure myself into a
corner.  :->

Thanks for all of the commentary so far.

Once I get something working, I will *try* to remember to follow up
here with what I've managed to cobble together.

Cheers!
mrc

PS: Actually, I used to share $HOME (and /usr) over PLIP, so, it is
probably obvious that FS speed is not always a concern for me.

Re: Current best practices for system configuration management?

[Andy Smith]

Hi,

On Thu, Apr 18, 2024 at 05:05:14PM -0700, Mike Castle wrote:
> Anyway, suggestions based upon actually experience would be appreciated.

As others have mentioned, Ansible can be a good choice for as little
as one machine as long as you don't object to installing Python and
a bunch of Python apps. Probably best to do it in a Python
virtualenv rather than try to use OS packages as it does still
change quite often, but I've found backwards compatibility to be
decent.

I used Puppet for quite a few years. It was a nicer, more expressive
and powerful language. It's also a lot faster than Ansible when you
have tens of hosts. It probably does not make sense for only a
couple of hosts though.

I stopped using it because the rate of change (requiring changes to
its server and client on all hosts) was too much for me.

These days I think the original developers consider it "done" and
have moved on, selling the lot to Perforce who I suspect will just
coast along milking the community for what it can. I would not
consider Puppet for a new setup at this time, mainly for that
reason.

Chef, never used it but did consider it when wanting to move away
from Puppet. Again probably not make sense for small number of
hosts. Likely has more of a future than Puppet considering Cinc.

We all hear great things about Nix, but I want something that
manages Debian packages, not replaces Debian packages.

Ansible is still viable for me but I fully expect Red Hat to ruin it
one day in search of profit. Which is their right.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Current best practices for system configuration management?

[Linux-Fan]

Mike Castle writes:


For a while now, I've been using `equivs-build` for maintaining a
hierarchy of metapackages to control what is installed on my various
machines.  Generally, I can do `apt install mrc-$(hostname -s)` and
I'm golden.

Now, I would like to expand that into also setting up various config
files that I currently do manually, for example, the `/etc/apt/*`
configs I need to make the above work.  For a single set of files,


[...]


My first thought was to simply add a `Files:` section to *.control
files I use for my metapackages.  After all, for configs going into
*.d directories, they are usually easy to just drop in and remove, no
editing in place required.  But, that is when I discovered that all
files under `/etc` are treated specially.


[...]

Hello,

I can confirm from experience that Ansible can indeed scale down to as  
little as the one local machine that it is running from. It has a learning  
curve and at least to me it always felt a little clumsy to learn a YAML  
based scripting language for this purpose, but its a solid choice.


Continuing the package-based approach is what I do because once some wrapper  
around the `debuild` commands was established, it became acceptably easy to  
use. I even maintain my “dotfiles” (not under $HOME but under /etc, but to  
a similar effect) this way: https://masysma.net/32/conf-cli.xhtml.


With `config-package-dev` there are some tricks to even allow changing  
(config) files supplied by other packages.


The disadvantage with the package-based approach is that it is heavily  
distribution-specific and also if you mess anything up, a core component of  
the OS (package management) can become broken - I luckily never broke it to  
the extent that recovery was impossible, but in the beginning ran a  
dedicated test VM to validate all package changes prior to installing them  
on my main system


I have also heard good things about Nix and if I had to start again from  
scratch today, I'd probably invest time into learning that technology. Right  
now I am sufficiently satisfied with the package-based approach to not look  
into it yet.


HTH
Linux-Fan

öö


pgplLIRpEGPqu.pgp
Description: PGP signature

Re: Current best practices for system configuration management?

[Alexandre Rossi]

Hi,

> > and so on, it is time to explore solutions.  I only have four systems
> > at the moment (two physical and two virtual), so I don't think I need
> > something too fancy.

I am in the same situation with an extra constraint: some are laptops
and not always connected.

> > My first thought was to simply add a `Files:` section to *.control
> > files I use for my metapackages.  After all, for configs going into
> > *.d directories, they are usually easy to just drop in and remove, no
> > editing in place required.  But, that is when I discovered that all
> > files under `/etc` are treated specially.

The limitation of this is that you cannot modify existing configuration
files, which is required sometimes.

> > Anyway, suggestions based upon actually experience would be appreciated.
> 
> The easy end of single-machine is etckeeper, which just checks
> your /etc (and whatever else you specify) into a local git. The
> high end of single machine is Nix, which has a complete language
> designed to capture the complete configuration of a system (and
> has spawned NixOS, a complete distribution).
> 
> The easy end of multi-machine systems is cdist and itamae. You might be
> quite happy with those, and itamae is reputedly very Chef-like.

I can also mention ansible which is ubiquitous nowadays and is relevant
even for a single machine. I've worked around my laptop deployment
requirement with a wrapper script around ansible-pull and a systemd timer
to regularly pull the conf from a git repository. It works well but
the complete configuration is known to all machines.

The Nix mention is highly relevant, but I did not get the chance to play
with it yet. The big advantage over  ansible and probably many others
is that if you remove a package installation from your configuration,
it will get removed from the host upon configuration deployment. Whereas
with ansible, you must add a explicit uninstallation rule. This is fine
for cloud host deployment where you always start from scratch, but for
physical machines and user laptops, Nix enforces consistency (but then
you need to learn something that is not Debian...).

Cheers,

Alex

Re: Current best practices for system configuration management?

[Dan Ritter]

Mike Castle wrote: 
> and so on, it is time to explore solutions.  I only have four systems
> at the moment (two physical and two virtual), so I don't think I need
> something too fancy.
> 
> My first thought was to simply add a `Files:` section to *.control
> files I use for my metapackages.  After all, for configs going into
> *.d directories, they are usually easy to just drop in and remove, no
> editing in place required.  But, that is when I discovered that all
> files under `/etc` are treated specially.
> 
> I've found a lot of documentation out there, but, of course, much of
> it is out of date.  https://wiki.debian.org/ConfigPackages , for
> example, seems to recommend slightly old to really old tools.  Tools
> like `ansible`, `puppet` and so on seem, at first blush, aimed at
> larger installations than mine.  But maybe other's experience with
> them will show they scale down fine?
> 
> Anyway, suggestions based upon actually experience would be appreciated.

Chef or Puppet are good, but probably not worth the bother for
4-5 systems. Definitely worthwhile for 10+ systems, and scalable
to thousands. https://cinc.sh is producing a fully open release
of Chef (with the new name, Cinc is not chef). I use Chef at
work.

The easy end of single-machine is etckeeper, which just checks
your /etc (and whatever else you specify) into a local git. The
high end of single machine is Nix, which has a complete language
designed to capture the complete configuration of a system (and
has spawned NixOS, a complete distribution).

The easy end of multi-machine systems is cdist and itamae. You might be
quite happy with those, and itamae is reputedly very Chef-like.


-dsr-

Current best practices for system configuration management?

[Mike Castle]

For a while now, I've been using `equivs-build` for maintaining a
hierarchy of metapackages to control what is installed on my various
machines.  Generally, I can do `apt install mrc-$(hostname -s)` and
I'm golden.

Now, I would like to expand that into also setting up various config
files that I currently do manually, for example, the `/etc/apt/*`
configs I need to make the above work.  For a single set of files,
manual isn't bad, but as I want to get into setting up LDAP, autofs,
and so on, it is time to explore solutions.  I only have four systems
at the moment (two physical and two virtual), so I don't think I need
something too fancy.

My first thought was to simply add a `Files:` section to *.control
files I use for my metapackages.  After all, for configs going into
*.d directories, they are usually easy to just drop in and remove, no
editing in place required.  But, that is when I discovered that all
files under `/etc` are treated specially.

I've found a lot of documentation out there, but, of course, much of
it is out of date.  https://wiki.debian.org/ConfigPackages , for
example, seems to recommend slightly old to really old tools.  Tools
like `ansible`, `puppet` and so on seem, at first blush, aimed at
larger installations than mine.  But maybe other's experience with
them will show they scale down fine?

Anyway, suggestions based upon actually experience would be appreciated.

Cheers,
mrc