Hello, When CI fails, I get an email and can chase things up with a look at the output files. This is something I like a lot. As a user, though, especially for scientific packages I would want to see the files that have been created. This way I could confirm that, e.g., the generated PDFs have usable fonts and if a test follows an established workflow for a well-known dataset, I can also compare the results generated during testing with the ones expected. "Expected" could be the files a Mac user generated when installing the package locally, a less technical user may know a tutorial (a nice one that I want to address is on https://docs.qiime2.org/2021.4/tutorials/moving-pictures/) and would then compare the CI-generated image with the one presented online by upstream.
There are many problems with this suggestion, to mind come * amount of data to be handled as input * amount of data to be stored * unclear acceptance by users * too much extra compute * ... please add what comes to your mind ... My sketch of an idea was that we could possibly have in analogy to the debian/install files an extension to the test instructions that define a set of files, preferably together with a description, that may be worthwhile to be inspected by end users as a proof that the package works. A test report would then be generated with links to these files. Is this anything we would want to have? Do we have this already and I just failed to find this? Many thanks! Steffen