On Fri, 29 Apr 2022 at 10:43:40 +0800, Paul Wise wrote: > Does that mean that autopkgtests that directly or indirectly use the > dbus-run-session tool to setup a temporary D-Bus session are buggy?
There's no single answer to that. For unit-test-style testing, it's often more convenient to start a session bus that's fully under the control of the test suite, particularly if the tests need special configuration (rejecting certain D-Bus messages, or setting limits, or making mock D-Bus services activatable, or something like that). For tests of the overall system (closer to an integration test), the fact that the service is genuinely activatable on a real system seems like something that the autopkgtest framework should not prevent you from testing. You don't necessarily need to actually test this - perhaps your priorities are elsewhere, and 100% test coverage is rarely realistic - but I think it would be a design flaw if autopkgtest prevented you from testing it, by forcing you to use a mock XDG_RUNTIME_DIR or a mock home directory or anything else that would interfere with your tests' access to our real system integration layer. If you run dbus-run-session without the --config-file option, it does use the real /usr/share/dbus-1/services, so you can test traditional (non-systemd) session service activation like this. You can't test systemd service activation this way, because that only works for the dbus-daemon that was launched by systemd and told to integrate with it. dbus has debian/tests/system-bus which does a brief integration test to check that the system bus works, but it doesn't yet have an equivalent for dbus-user-session - contributions welcome, if you are interested in expanding coverage. autopkgtest tries to set up a "real" system login session by running su, even if it's already running as the target uid, to make sure that some of the side-effects of login (in particular a PAM stack) have already happened; but I suspect this doesn't have full coverage. autopkgtest-virt-ssh is probably the most "realistic" of the backends, by running test commands under a ssh session that is genuinely a login entry point (meaning processes have a loginuid and so on), and one of the things on my increasingly intractable to-do list is to add a way to launch qemu VMs on-demand and use ssh to log in to them, rather than using a privileged serial console like virt-qemu does. In some backends it isn't really feasible to run the tests in a "real" system login session, so integration tests for things that rely on the system's session infrastructure might have to have the 'skippable' restriction and detect whether they are running in a login session at runtime. This is also very likely to be something that some autopkgtest backends just cannot provide: -chroot and -schroot don't have their own init systems (and neither would a Docker backend, review/finishing of which is also on my to-do list), so they cannot be expected to run their own system services. > The other thing is testing graphical tools; should that be done in a > mock X11/Wayland session (using xvfb-run), or a real desktop session? At the moment a mock X11 session using xvfb-run or a mock Wayland session using weston --backend=headless-backend.so (see debian/tests/run-with-display in src:gtk4) are the only realistic options for this. Ideally it'd be possible to test graphical tools under a real desktop session, but that's not straightforward to set up even under something like autopkgtest-virt-qemu (the test framework would have to script a login to a display manager using techniques similar to openQA, or configure the display manager to autologin and then reboot to make the autologin happen), and I'm not sure whether this would be feasible to set up under other backends at all. There's a cost/benefit tradeoff here: running tests under a real desktop session is likely to be more difficult, slower, and less robust than xvfb-run, so we should only do it if the benefit is worth those costs. In particular, we can are not going to be able to use this as a QA gate unless someone puts a lot of time and effort into making it completely reliable. GNOME's installed-tests convention[1], which is a frequent source of autopkgtests for Debian, *does* call for tests to be run in a real, active GUI desktop session, although in practice most packages' installed-tests do not actually need this. [1] https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests smcv