Hello Ken. Thank you for looking into this issue. I fully agree with the approach to reuse as much of the existing QA infrastructure as possible.
> I am unfamiliar with autopkgtests, could you please describe: > - the goals of autopkgtests (specifically what level of testing confidence is > aimed for) The slightly different purpose of the autopkgtests is the reason for me making the effort of providing the additional tests. Autopkgtests are intented to be a suite of rather light-weight integration/acceptance level tests rather than functionality tests. Their goal is usually to smoke-test the system integration pieces after any changes done to the packaging (including the software dependencies), init script management, security policies. Hence, one of the tests provided checks if e.g. pmcd has been started correctly, no default firewall policies prevent it from being accessed (including those introduced in the future) etc. > - the environment it runs in (post build, with all PCP packages installed?) It runs in a clean, minimal environment - usually in a VM or a container. I'm not sure how does it look in Debian, but in case of Ubuntu it is run every time a Stable Release Update is being prepared for release to the -proposed pocket. Only packages listed as 'Depends:' in debian/tests/control are installed on top of the minimal environment. A clean environment is used for every 'Test:' listed in the control file, so ideally they should not depend on each other. > - the way it is invoked and the results it must pass back The test scripts are run as a non-root user with debian/tests set as CWD. They are considered successful as long as nothing is written to stderr and return 0. There are some configuration options changing this behaviour (more info at [1]). > I'll bet a thin wrapper between the autopkgtests test harness and the > existing PCP QA infrastructure will more than match the requirements and keep > the test maintenance effort with the upstream PCP development community. I believe it would a a good idea to include the sanity group as an additional step of autopkgtests. Moreover, I have already done some tests in that area. Sadly, I noticed that the test 354 fails consistently while running in a lxd container with lxd exec <container> /bin/bash. Running on metal or in a KVM seems to be perfectly fine). Looks like its result depends on pipe fd numbering assingned by the system which seems to differ slightly in this case (please check the attached diff). Thanks, Dariusz [1] http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/tree/doc/README.package-tests.rst
PMDA probe: pminfo -h pcp1 -f sample.milliseconds PMDA probe: pminfo -h pcp1 -f sampledso.milliseconds PMDA probe: pminfo -h pcp1 -f simple.numfetch _change_config: Error: cannot change config "pmcd on" 354 - output mismatch (see 354.out.bad) 21c21 < f_config: NULL f_ipc: 6 pid: pmlogger-pid-1 status: -1 --- > f_config: NULL f_ipc: 7 pid: pmlogger-pid-1 status: -1 24c24 < f_config: NULL f_ipc: 7 pid: pmlogger-pid-2 status: -1 --- > f_config: NULL f_ipc: 8 pid: pmlogger-pid-2 status: -1 27c27 < f_config: NULL f_ipc: 8 pid: pmlogger-pid-3 status: -1 --- > f_config: NULL f_ipc: 10 pid: pmlogger-pid-3 status: -1 30c30 < f_config: NULL f_ipc: 9 pid: pmlogger-pid-4 status: -1 --- > f_config: NULL f_ipc: 11 pid: pmlogger-pid-4 status: -1 72c72 < ps: pmlogger-pid-2 pmlogger -x 6 -h localhost --- > ps: pmlogger-pid-2 pmlogger -x 7 -h localhost 208c208 < f_config: NULL f_ipc: 7 pid: pmlogger-pid-1 status: -1 --- > f_config: NULL f_ipc: 8 pid: pmlogger-pid-1 status: -1 211c211 < f_config: NULL f_ipc: 8 pid: pmlogger-pid-2 status: -1 --- > f_config: NULL f_ipc: 10 pid: pmlogger-pid-2 status: -1 214c214 < f_config: NULL f_ipc: 9 pid: pmlogger-pid-3 status: -1 --- > f_config: NULL f_ipc: 11 pid: pmlogger-pid-3 status: -1 217c217 < f_config: NULL f_ipc: 10 pid: pmlogger-pid-4 status: -1 --- > f_config: NULL f_ipc: 12 pid: pmlogger-pid-4 status: -1 259c259 < ps: pmlogger-pid-2 pmlogger -x 7 -h localhost --- > ps: pmlogger-pid-2 pmlogger -x 8 -h localhost Check local PMCD is still alive ... PMDA probe: pminfo -h pcp1 -f sample.milliseconds PMDA probe: pminfo -h pcp1 -f sampledso.milliseconds PMDA probe: pminfo -h pcp1 -f simple.numfetch Failures: 354 Failed 1 of 1 tests
