I set up a raspberry pi as a device under test with contest, and ran the contest controller in a mode that needed no mysql, and it took about 14 minutes.
I'd still recommend looking at contest. I've found it very easy to use and I am very new to it. On Tue, Oct 12, 2021 at 7:02 AM Patrick Georgi <[email protected]> wrote: > > Hi Ricardo, > > sorry for the late response, and that your project fell a bit by the wayside. > I guess discussion configuration frameworks is more attractive to this > community than testing frameworks (which also explains why we have ~3 config > frameworks and only ~1 testing frameworks ;-) ) > > So yes, testing is something we really need to improve on. I'm not sure if > "contest" is the right solution to your particular problem though. The first > thing I see when opening up its page is something about mysql, and scrolling > down, something about submitting jobs to a server. That seems more like a > potential replacement to our Jenkins install (qa.coreboot.org) rather than > something to easily write end-to-end tests for our userland tools. > > Looking for options, my first instinct was to go for expect(1), but that's > really best for interactive uses - might be interesting if we ever grow > interactive tools, but so far we manage to stick to clean and tidy CLI. Then > I ran into https://github.com/bats-core/bats-core. That seems maintained, > reasonably minimal in its dependencies, it emits TAP which is as good as > JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse > the results and give meaningful feedback on review.coreboot.org), and it > seems to be fairly widely used for similar things to what you're doing, see > https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points > to many, many code examples, e.g. > https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats > which should cover the basic "call some command, see what it did" scenario > quite nicely. > > Of course in the end it's all a matter of taste, and that's why I opened the > can of worms again that is Python-use in coreboot land. As python hasn't seen > a warmer reception than last time, I'd look for alternatives. > Maybe Bats could do? Of course, I haven't _actually_ used it yet and if > writing tests with that makes you want to scream and run away, we'd have to > look for other options (please, don't run away!) > > > Regards, > Patrick > > Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada > <[email protected]>: >> >> Hi all, >> >> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way >> to go forward? >> I just need to run one of the Coreboot's utils (in this case "elogtool"), >> and make sure the output is the expected one. >> >> Should I use Contest, as suggested by Ron Minnich? >> >> Thanks! >> >> >> >> On Thu, Sep 30, 2021 at 10:18 AM ron minnich <[email protected]> wrote: >>> >>> Speaking as the person who wrote the first few config tools in python, >>> and was happy to see the python dependency gone, I think bringing >>> python back in would be a mistake. >>> >>> Every single python test framework I've worked with works until there >>> is a problem, and I then find myself having to walk back a python call >>> trace, because what inevitably breaks is the test framework tooling. >>> It's why so many projects are removing python test frameworks. >>> >>> If you want a good test framework, get the linuxboot fork of >>> facebook's contest github.com/linuxboot/contest, written in Go, in use >>> at scale near you. >>> >>> It's easy to let the joy of building a build system overwhelm the >>> actual goals of a project. coreboot is not about being a build system. >>> It's easy to fall into the trap of creating an ever more complex >>> system that is more than is needed. >>> >>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot >>> <[email protected]> wrote: >>> > >>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal >>> > <[email protected]>: >>> >> >>> >> With respect to Kconfig, we (at Google) encountered a lovely build flake >>> >> after the Kconfig uprev last month in the coreboot tree that took a >>> >> couple of weeks to sort out and resolve. Some sort of automated >>> >> validation that the code is working could have possibly helped. Of >>> >> course, the C implementation of Kconfig has no tests at all. Some tests >>> >> is better than nothing. >>> > >>> > >>> > Let me put the record straight: >>> > >>> > - The last kconfig "uprev" hasn't been a simple update the way >>> > https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire >>> > build system integration to ease maintenance >>> > - That issue sprang up because before the kconfig update, we were >>> > shipping prebuilt parser files (C code) while now we made bison and flex >>> > hard dependencies for our build >>> > - Tests covering the C code wouldn't have helped, because the issue >>> > wasn't in the C code >>> > - The issue we were facing has been an external dependency (namely: the >>> > Chromium OS development environment shipping a broken version of bison(1)) >>> > - Fixing bison in CrOS wouldn't have helped any because we have to assume >>> > that other users come with the same kind of broken bison tool >>> > - The fix has been to ship pre-built files again to remove an external >>> > dependency >>> > >>> > The alternative that we did actually consider was to add support for >>> > building bison and flex to util/crossgcc/buildgcc. For three files that's >>> > sheer madness, so we went back to precompiled files instead. >>> > >>> > In relation to your proposal to adopt kconfiglib: We can run into any >>> > kind of external tool demonstrating weird behavior. That's true for bison >>> > (as seen here) just as it can be true for arbitrary python versions (even >>> > when specified to be "python 3.6+" or whatever): Linux distributions do >>> > strange things to their packages, and we're not a Linux-only project, so >>> > even official, unchanged, straight-from-the-server python might behave >>> > unexpectedly on less well exercised platforms. >>> > >>> > The best way to reduce issues on that end is to avoid external >>> > dependencies - like bison, like flex... like python. >>> > I'd _love_ to avoid the dependency on the host compiler as well, but >>> > that's one of those "sheer madness" moments when you support a multitude >>> > of operating systems on as many architectures. >>> > >>> > Introducing kconfiglib (and through it, a deep reliance on python) just >>> > doesn't seem to provide comparable benefits. >>> > >>> > >>> > Patrick >>> > -- >>> > Google Germany GmbH, ABC-Str. 19, 20354 Hamburg >>> > Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: >>> > Hamburg >>> > Geschäftsführer: Paul Manicle, Halimah DeLaine Prado >>> > _______________________________________________ >>> > coreboot mailing list -- [email protected] >>> > To unsubscribe send an email to [email protected] > > > > -- > Google Germany GmbH, ABC-Str. 19, 20354 Hamburg > Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: > Hamburg > Geschäftsführer: Paul Manicle, Halimah DeLaine Prado _______________________________________________ coreboot mailing list -- [email protected] To unsubscribe send an email to [email protected]
