Re: [systemd-devel] systemd-resolve and name servers order
On Wed, 11 Oct 2023 at 09:37, Marc wrote: > Having 3 different nameservers reporting different results? > An example I have seen quite frequently is where there is an internal DNS which resolves local (internal) server resources and forwards anything else to an external server such as 8.8.8.8. When the internal DNS fails (typically because it's been taken down for server updates) it would be useful for clients to still have access to the external DNS - even if they can't access local resources they can still access external ones, and in reality they probably already had the internal resource IPs cached anyway. So having client DNS configured as 192.168.x.x (primary) and 8.8.8.8 (secondary) makes sense. But as you say this isn't how resolv.conf was designed (nor other DNS clients in my experience), and configuring a client that way will cause a lot of internal requests to fail even when the local DNS is operational. It's a shame that DNS clients can't have server configurations much like MX records where a random choice is made between servers of the same priority, but lower priority options can be configured too (as that would also allow load balancing of requests across multiple servers as now). I don't know of any clients that do this, though. Obviously there are other solutions to the problem described above (eg having multiple internal servers, although my experience was in the SOHO environment where that would be excessive). If as Rafał says Windows prioritises the first DNS option then I'm pretty sure that wasn't always the case, but it's been well over a decade since I got out of Windows IT support! But there are certainly use cases for allowing DNS server prioritisation in this way. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0344 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
Re: [systemd-devel] Starting a service before any networking
On Thu, 28 Sept 2023 at 11:16, Mark Rogers wrote: > DefaultDependencies=no > FWIW I tried: DefaultDependencies=no Before=network-pre.target Wants=network-pre.target and DefaultDependencies=no Before=network-pre.target Wants=network-pre.target local-fs.target .. and in both cases my unit still started after dhcpcd (albeit that with my "ip link set eth0 down/up" hack the network worked OK). I have left it as DefaultDependencies=no Before=network-pre.target dhcpcd.service Wants=network-pre.target local-fs.target .. although a lot of that seems redundant, really it's just the Before=dhcpcd.service that seems to be achieving anything. -- Mark Rogers
Re: [systemd-devel] Starting a service before any networking
On Wed, 27 Sept 2023 at 14:09, Jetchko Jekov wrote: > A good example of a service that needs to be started before networking > is the firewall service. > You can take a look at what your distro of choice is providing for hints. > Good idea, thanks > DefaultDependencies=no > This looks like the most important bit I was unaware of, although it now seems that I was looking in the wrong direction in thinking that dhcpcd was starting before it had a configuration file as fixing that doesn't seem to have helped. -- Mark Rogers
Re: [systemd-devel] Starting a service before any networking
On Wed, 27 Sept 2023 at 11:31, Silvio Knizek wrote: > Why does this sounds like https://github.com/raspberrypi/linux/issues/3195? > Maybe you find starting there some more information. > I agree it does sound similar. That said I am on a Pi3 (not a Pi4) and later kernel (4.19.97-v7+). But it is possible that something in the thread (and the ones it links to) might offer more clues so I'll dig a bit deeper, thank you. -- Mark Rogers
Re: [systemd-devel] Starting a service before any networking
On Wed, 27 Sept 2023 at 10:18, Mantas Mikulėnas wrote: > So now I'm curious: if the first command you run is to bring the interface > *down*, then what exactly brought it up? > Good question. The reason for down/up was that this was working as a way to reset the connection after boot, so I just transferred that to the ExecStartPre. Looking at the "journalctl -u dhcpcd" output, this is what I see from my last boot: Feb 14 10:12:05 pi systemd[1]: Starting dhcpcd on all interfaces... Feb 14 10:12:05 pi ip[372]: 2: eth0: mtu 1500 qdisc noop state DOWN group default qlen 1000 Feb 14 10:12:05 pi ip[372]: link/ether b8:27:eb:0d:ee:bb brd ff:ff:ff:ff:ff:ff Feb 14 10:12:05 pi ip[383]: 2: eth0: mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 Feb 14 10:12:05 pi ip[383]: link/ether b8:27:eb:0d:ee:bb brd ff:ff:ff:ff:ff:ff Feb 14 10:12:06 pi dhcpcd[385]: wlan0: starting wpa_supplicant Feb 14 10:12:36 pi dhcpcd[385]: timed out Feb 14 10:12:36 pi systemd[1]: Started dhcpcd on all interfaces. Feb 14 10:12:37 pi systemd[1]: Stopping dhcpcd on all interfaces... Feb 14 10:12:37 pi dhcpcd[519]: sending signal TERM to pid 466 Feb 14 10:12:37 pi dhcpcd[519]: waiting for pid 466 to exit Feb 14 10:12:38 pi systemd[1]: dhcpcd.service: Succeeded. Feb 14 10:12:38 pi systemd[1]: Stopped dhcpcd on all interfaces. Feb 14 10:12:38 pi systemd[1]: Starting dhcpcd on all interfaces... Feb 14 10:12:38 pi ip[524]: 2: eth0: mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 Feb 14 10:12:38 pi ip[524]: link/ether b8:27:eb:0d:ee:bb brd ff:ff:ff:ff:ff:ff Feb 14 10:12:38 pi ip[529]: 2: eth0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 Feb 14 10:12:38 pi ip[529]: link/ether b8:27:eb:0d:ee:bb brd ff:ff:ff:ff:ff:ff Feb 14 10:12:38 pi dhcpcd[530]: wlan0: starting wpa_supplicant Feb 14 10:12:49 pi dhcpcd[530]: Too few arguments. Feb 14 10:12:49 pi dhcpcd[530]: Too few arguments. Feb 14 10:12:49 pi systemd[1]: Started dhcpcd on all interfaces. (I deleted the "ip addr" output from the interfaces other than eth0 for brevity.) The interesting thing is surely that dhcpcd is being started twice. Assuming that was always happening then that suggests dhcpcd was bringing the network up early (and failing but leaving it in a "stuck" state) and then again later (where it was unable to recover from the first failure, but now can)? -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0344 251 1450 Registered in England (0456 0902) 21 Drakes Mews, Milton Keynes, MK8 0ER
Re: [systemd-devel] Starting a service before any networking
On Wed, 27 Sept 2023 at 09:39, Mantas Mikulėnas wrote: > It might be an issue with the kernel driver for your Ethernet interface, > then (as setting the interface 'up/down' usually reinitializes the > controller) – or possibly a physical issue with your cable or your switch, > but it doesn't seem like the kind of issue that userspace configuration > should be *able* to lead to in the first place. (...except maybe for EEE > "power saving" stuff that might tip over a really marginal link.) > What doesn't make sense is that this had previously worked, although it's possible that the network hardware has changed since it was previously tested. > (It's sort of like blaming a segfault crash on the user: if a program > crashes, that's inherently a bug regardless of configuration. Here it's > similar: if the Ethernet cable is really connected but the driver still > reports "no carrier", that's either an interface issue or – if you see the > same on multiple Pi's – perhaps a NIC driver issue, but it's not something > that configuration ought to be *able* to do.) > OK, in that case if this persists I'll have to look at upgrading the whole system, which I'm trying to avoid doing. But: > Use the "drop-in" system (dhcpcd.service.d/*.conf), e.g. via `systemctl > edit dhcpcd5`. Add a few ExecStartPre= commands in [Service] to have it > "manually" bring the interface up, then down (possibly with a 'sleep .5' > after each), and hopefully when dhcpcd brings it up the /second/ time it > will work. > This has worked: [Service] ExecStartPre=ip addr ExecStartPre=ip link set eth0 down ExecStartPre=ip link set eth0 up ExecStartPre=ip addr (the "ip addr" calls are just to log the before/after state to journal). It's booted in that state several times now successfully. I'll need to do more testing yet but I am inclined to leave it at that (I hate workarounds rather than actually fixing the issue but I suspect this is far as I'll get). Thank you (massively!) for your assistance on this. -- Mark Rogers
Re: [systemd-devel] Starting a service before any networking
On Tue, 26 Sept 2023 at 20:41, Mark Rogers wrote: > (I should be able to find another Pi to test for any physical hardware > issues, I'll try that tomorrow.) > I have today tested on a different Pi, different PSU, different cable, all with exactly the same results. There is definitely something about the early boot stages which is different from later on that means bringing the network up early (as happens now) will usually fail. (Some more background: This is a heavily modified install for a specific application so it's almost certainly something I have broken somewhere. However it has worked for years, I'm trying to resolve an issue on a unit that was returned because of physical damage to the SD card, so I've rebuilt it from an old image and now have this problem. I just need to break down the boot sequence to find out which step is causing the interface to get into a state where it fails like this. Systemd version is 241.) Alternatively I guess there's the workaround option: detect the condition at a later stage of the boot and run the down/up sequence to fix it. If I try that, where is likely the best place in the sequence to put it? If I wanted to make it, in effect, part of the dhcpcd unit (in that when dhcpcd starts it first runs a down/up script), how should I do that without modifying system dhcpcd unit files? -- Mark Rogers
Re: [systemd-devel] Starting a service before any networking
On Tue, 26 Sept 2023 at 19:38, Mantas Mikulėnas wrote: > That's not a race condition; it's a fault in the network interface > itself. "NO-CARRIER" means it's physically unable to establish the > Ethernet link – an external condition that the service ordering has no > effect on. > That's interesting. In that case, how is it that ip link set eth0 down ip link set eth0 up .. consistently brings the network up? (I have tested that sequence dozens of times.) What is that doing that isn't happening during a normal boot? This being a Raspberry Pi I believe that the Ethernet port hangs of the USB bus internally in case that's relevant. (I should be able to find another Pi to test for any physical hardware issues, I'll try that tomorrow.) -- Mark Rogers
Re: [systemd-devel] Starting a service before any networking
On Tue, 26 Sept 2023 at 13:44, Mantas Mikulėnas wrote: > I think you're confusing two different states, which have similar > indications – "administrative" up/down that you control (the "" flag, > with nothing shown when down) and "operational" up/down that represents the > actual interface status (the "" vs "" flags and/or the > "state XXX" field). > Yes I am, thanks for clarifying. > "state DOWN" is *not* directly controlled by `ip link set up` – it's the > result of the interface being operative for any other reason even though it > is administratively (i.e. turned on). > > I'm still not entirely sure of the situation but right now it sounds like > the configuration is okay but the Ethernet interface is failing to > establish a physical link on the first try. Does it also show > "" within the interface flags? > eth0: mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000 I've done a lot more testing now and there's a race condition somewhere as it does sometimes (rarely) boot OK and get an IP address with no config changes. > `systemctl cat` for direct configuration and `systemctl list-dependencies > --after` (if I remember it right) should be a good start. > So here is what I now have. My unit is now this: [Unit] Before=network-pre.target dhcpcd.service Wants=network-pre.target [Service] Type=oneshot ExecStart=/path/to/script [Install] RequiredBy=network.target Note I added dhcpcd.service to Before as it consistently starts too early otherwise. The dhcpcd unit config is (I haven't changed anything here): [Unit] Wants=network.target Before=network.target [Service] Type=forking PIDFile=/run/dhcpcd.pid ExecStart=/usr/lib/dhcpcd5/dhcpcd -q -b ExecStop=/sbin/dhcpcd -x [Install] WantedBy=multi-user.target Alias=dhcpcd5.service In this state dhcpcd consistently starts after my script but the DHCP issue I'm trying to fix continues, so the race may not related to dhcpcd after all. -- Mark Rogers
Re: [systemd-devel] Starting a service before any networking
On Tue, 26 Sept 2023 at 13:08, Mantas Mikulėnas wrote:
> Depends on what exactly runs dhcpcd and wpa_supplicant. Is that done by
> networking.service (ifupdown)? NetworkManager? Are they standalone services?
>
How do I tell?
(System is a Pi running an elderly Raspbian. The issue I am having is that
the device is not getting an IP address - if i wait until booted I have to
issue "ip link set eth0 down" and "ip link set eth0 up" to get it to retry
the DHCP request ("up" alone isn't sufficient, despite "ip addr" showing
the interface as DOWN. I am assuming that this is because the config file
isn't in place when dhcpcd starts but I may be mistaken.)
> I would generally expect Before/Wants=network-pre.target to work, but that
> relies on your network services themselves being set up correctly – they
> too need to order themselves After that target.
>
In that case I should probably return to Before/Wants=network-pre.target
and work out what is breaking it, but same question as above: how do I
figure that out?
--
Mark Rogers
[systemd-devel] Starting a service before any networking
I'm sure this is trivial but I've gone round in circles without success. I have a script which reads from an SQLite database and generates various system configuration files - at the moment these are dhcpcd.conf and wpa_supplicant.conf but this might grow in future. As such the only dependency the script has is that the filesystem is up and running. But the script must complete before anything that the script manages the configuration file for. My current unit looks like this: [Unit] Before=networking.service After=local-fs.target [Service] Type=oneshot ExectStart=/path/to/script [Install] RequiredBy=network.target Where am I going wrong and what is the right way to do this? I've also tried Before=network-pre.target and Wants=network-pre.target without success - it was that not working that set me off trying to fix it. -- Mark Rogers
[systemd-devel] systemd-rfkill.service: Failed to set up special execution directory in /var/lib: Read-only file system
I have a Raspberry Pi-based development kiosk system which has been working fine for ages. But without having made any changes that I can relate to the boot sequence, it now errors on startup with multiple: [FAILED] Failed to start Load/Save RF Kill Switch Status errors, and also a single [FAILED] Failed to start Set console font and keymap error. Clearly these relate to the read-only filesystem but that hasn't changed so I don't see why I am now getting errors unless there was always a race condition that I'm now experiencing for the first time? Any suggestions for how to resolve these? (Do I even need rfkill in a kiosk environment? I tried to disable it but that doesn't appear to have any impact.) -- Mark Rogers ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Ensuring that a unit starts before any networking
On Tue, 30 Jun 2020 at 12:13, Simon McVittie wrote: > > I would have expected this to be done in some "larger" network management > component that is responsible for bringing up network interfaces according > to your configuration, rather than necessarily dhcpcd.service itself. In which case one would expect dhcpcd to depend on that instead; I can't see any obvious dependencies that would apply but I'm somewhat out of my depth here. > You mentioned Raspbian, which is a Debian derivative? Yes, Raspbian is Debian a derivative. > Typically server-class systems will use either ifupdown or > systemd-networkd, which are ideal for relatively static network > configurations that are set up by a sysadmin. It's definitely not systemd-networkd, to my untrained eye it looks like ifupdown is there but how do I confirm? dhcpcd seems to start very early in the boot and doesn't seem to be After anything on my Raspbian box. networking.service certainly includes After=network-pre.target and Before=network.target, but dhcpcd also includes Before=network.target - I can't see any combination of dependencies that would put dhcpcd.service later than networking.service. It's quite plausible that the "bug" here isn't necessarily the need for After=network-pre.target but as far as I can tell it is missing something. > If Raspbian does its own thing rather than recycling Debian components > for this, then it might need a separate bug report. I don't have a Debian box to check dhcpcd dependencies but I would assume that they're lifted from Debian, albeit that they may or may not be default Debian components. I have raised a Raspbian bug, which references this thread, so I'll see where that takes me. -- Mark Rogers ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Ensuring that a unit starts before any networking
On Tue, 30 Jun 2020 at 10:27, Lennart Poettering wrote: > Yes, this is a bug in the the distro packaging. Please file a bug > against your distro, so that they add After=network-pre.target. > Will do. > My educated guess is that, it's not trvial to get this right: we > document what network-pre.target is for in systemd.special(7) man > page, but I figure not everyone looks there. And i guess one most know > a certain level of systemd to understand what this actually means for > a specific implementation. > Fair enough. >From that, is it reasonable to conclude that this is just an omission that has no potential negative effects from being fixed? I can't see any reason why adding After=network-pre.target would have direct a negative impact (it's plausible that on some systems it could subtly alter startup ordering and expose a bug that was previously hidden by luck, but that's not a reason to avoid doing it) and so there is no good reason not to do this, but my knowledge of this isn't adequate to make an informed statement on that matter. Before I make the argument for it being fixed I want to be sure of my argument! -- Mark Rogers ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Ensuring that a unit starts before any networking
On Sat, 27 Jun 2020 at 13:04, Andrei Borzenkov wrote: > All other units that implement networking must have > After=network-pre.target for the above to do anything. Do they? .. and .. On Sat, 27 Jun 2020 at 13:08, Michael Chapman wrote: > It could very well be because of the dhcpcd.service you're using. > > I don't know what it's like on Raspbian, but on Debian this unit appears > to have no ordering with respect to network.target or network-pre.target > at all. Ah, therein lies my failing - I assumed OS supplied units wouldn't need adapting... Adding After=network-pre.target to dhcpcd.service seems to have resolved it, and taught me a lesson in the process. Is there an obvious reason I'm missing why these aren't distro defaults? (Is this a "bug" in the network management tools' unit files? Would "fixing" this at the distro level have any likely side effects?) -- Mark Rogers ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Ensuring that a unit starts before any networking
On Sat, 27 Jun 2020 at 11:51, Dave Howorth wrote: > A Pi doesn't normally have an RTC, so the mixup usually takes place > when the time is updated via NTP I believe. Do you have an RTC? Yes I do have an RTC. At that point in the logs dhcpcd hasn't started so it must be from the RTC (although nothing seems to appear in the logs from a quick glance). But I only mentioned it in passing because otherwise it looked like the logs were from different dates despite being from the same boot. > > also note my db2config script doesn't run until after hostname is set > > which I would assume is set by the network startup?) > > Well that depends how you've set the Pi up, so you tell us, don't > assume. If your script doesn't start until hostname is set and hostname > is set by dhcp then you have a fundamental problem. My script doesn't care about hostname. Hostname is set in /etc/hostname but I don't know when that is read (nor care unless it's a clue to why the script is running after networking starts despite my efforts! -- Mark Rogers ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Ensuring that a unit starts before any networking
On Sat, 27 Jun 2020 at 11:06, Zbigniew Jędrzejewski-Szmek wrote: > You should use Before=network-pre.target, Wants=network-pre.target. Thanks, tried that but still not working: $ journalctl -b | grep -Ei '(db2config|dhcpcd)' Feb 14 10:12:03 localhost systemd[1]: Starting dhcpcd on all interfaces... Feb 14 10:12:03 localhost dhcpcd[341]: read_config: fopen `/etc/dhcpcd.conf': No such file or directory [...] Jun 27 10:19:39 localhost dhcpcd[341]: wlan0: /etc/wpa_supplicant.conf does not exist [...] Jun 27 10:19:39 localhost dhcpcd[341]: read_config: fopen `/etc/dhcpcd.conf': No such file or directory [...] Jun 27 10:19:40 localhost dhcpcd[341]: eth0: soliciting an IPv6 router Jun 27 10:19:40 localhost dhcpcd[341]: eth0: soliciting a DHCP lease Jun 27 10:19:41 mypi db2config.py[325]: 2020-06-27 10:19:41 db2config Creating /tmp/sys//etc/dhcpcd.conf Jun 27 10:19:41 mypi db2config.py[325]: 2020-06-27 10:19:41 db2config Creating /tmp/sys//etc/wpa_supplicant/wpa_supplicant.conf (Comments about that extract: the jump from Feb to Jun I assume is the clock getting updated from RTC, it's all from the same boot obviously; also note my db2config script doesn't run until after hostname is set which I would assume is set by the network startup?) Unit file is currently: [Unit] Description=Config generation from DB Before=network-pre.target Wants=network-pre.target [Service] Type=oneshot ExecStart=/home/mark/bin/db2config.py [Install] RequiredBy=network.target After any changes I'm using $ sudo systemctl daemon-reload $ sudo systemctl reenable db2config.service ... although that's another area I'm not entirely clear about what exactly is required after a unit file change. PS: Is list etiquette around here to CC on reply? Some love it, some hate it, others don't care... -- Mark Rogers ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Ensuring that a unit starts before any networking
This feels like something I should be easily able to answer from documentation/Google, and failing that from somewhere like StackOverflow, without troubling systemd-devel, but all my efforts have thus far failed [1] What is the correct way to ensure a script runs to completion before any networking units start? "The Internet" is abound with people asking the opposite question which hasn't helped my keyword-based searches. My use case (on a Raspberry Pi running Raspbian if it matters) is that I have network settings stored in a database from which I generate network configuration files (eg dhcpcd.conf) on boot, therefore they obviously need to be in place before dhcpcd starts. (Similarly wpa_supplicant.) Ideally I'd like to be agnostic about the actual network stack in my unit configuration (so not setting dependencies relating to dhcpcd, for example) but at this point I've tried pretty much everything without success so I'm obviously doing something stupid. I have tried multiple approaches so far but by current service file looks like this: [Unit] Description=Config generation from DB Before=networking.service [Service] Type=oneshot ExecStart=/home/mark/bin/db2config.py [Install] RequiredBy=network.target [1] https://stackoverflow.com/questions/62574482/ensuring-that-a-systemd-unit-starts-before-any-networking - no responses there to date, feel free to respond there for reputation or else I'll update it when I solve it. -- Mark Rogers ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/systemd-devel
