Thanks for the answers! Regarding the end-to-end testing framework, perhaps that's a topic that needs further discussion / consensus (?). E.g: - discuss policy: what parts of coreboot should have end-to-end testing. Should it be forced? recommended? optional? - do some requirement gathering: not only from the firmware but also for the coreboot/utils - Update Jenkins accordingly
In the meantime, would it make sense, as Jack mentioned, to land my change [1] as it is? It is small/simple and it only has ~160 LoC Python. For comparison, other util are using Python: util/qualcomm has ~3500 LoC Python [2] I'll happily migrate + integrate my test once a end-to-end test has been chosen. Thanks! [1]: https://review.coreboot.org/c/coreboot/+/57869 [2]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm On Tue, Oct 12, 2021 at 8:30 AM ron minnich <[email protected]> wrote: > contest is a coming standard. The companies I work with have moved away > from python test frameworks. Familiarity did not breed respect for python > test frameworks. > > On Tue, Oct 12, 2021 at 7:31 AM Jack Rosenthal <[email protected]> > wrote: > >> Both bats and expect seem problematic for Ricardo's use case ... >> generating the elog binary format in these tools seems difficult, bash >> wasn't really meant for generating C structs. >> >> One huge advantage of pytest is that it's fairly industry standard at >> this point. There's a good number of people who have used it, and there's a >> large community around it (i.e., finding Q&A on sites like stack overflow >> is possible). I wasn't able to find a single stack overflow question about >> ConTest, although I may be looking in the wrong places (looks like they do >> have a slack group chat). >> >> I believe Ricardo had an open question on the CL if it was OK to submit >> as an optional test suite for elogtool. I.e., tests don't need to run on CI >> tools or anything, if there's developers that want to run the tests they >> can if they have Python. >> >> >> >> On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi <[email protected]> >> wrote: >> >>> Hi Ricardo, >>> >>> sorry for the late response, and that your project fell a bit by the >>> wayside. I guess discussion configuration frameworks is more attractive to >>> this community than testing frameworks (which also explains why we have ~3 >>> config frameworks and only ~1 testing frameworks ;-) ) >>> >>> So yes, testing is something we really need to improve on. I'm not sure >>> if "contest" is the right solution to your particular problem though. The >>> first thing I see when opening up its page is something about mysql, and >>> scrolling down, something about submitting jobs to a server. That seems >>> more like a potential replacement to our Jenkins install ( >>> qa.coreboot.org) rather than something to easily write end-to-end tests >>> for our userland tools. >>> >>> Looking for options, my first instinct was to go for expect(1), but >>> that's really best for interactive uses - might be interesting if we ever >>> grow interactive tools, but so far we manage to stick to clean and tidy >>> CLI. Then I ran into https://github.com/bats-core/bats-core. That seems >>> maintained, reasonably minimal in its dependencies, it emits TAP which is >>> as good as JUnit in terms of Jenkins-integration (so we can have >>> qa.coreboot.org parse the results and give meaningful feedback on >>> review.coreboot.org), and it seems to be fairly widely used for similar >>> things to what you're doing, see >>> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it >>> points to many, many code examples, e.g. >>> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats >>> which should cover the basic "call some command, see what it did" scenario >>> quite nicely. >>> >>> Of course in the end it's all a matter of taste, and that's why I opened >>> the can of worms again that is Python-use in coreboot land. As python >>> hasn't seen a warmer reception than last time, I'd look for alternatives. >>> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if >>> writing tests with that makes you want to scream and run away, we'd have to >>> look for other options (please, don't run away!) >>> >>> >>> Regards, >>> Patrick >>> >>> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada < >>> [email protected]>: >>> >>>> Hi all, >>>> >>>> Regarding Patrick's 2nd point (end-to-end testing), what's the >>>> recommend way to go forward? >>>> I just need to run one of the Coreboot's utils (in this case >>>> "elogtool"), and make sure the output is the expected one. >>>> >>>> Should I use Contest, as suggested by Ron Minnich? >>>> >>>> Thanks! >>>> >>>> >>>> >>>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich <[email protected]> >>>> wrote: >>>> >>>>> Speaking as the person who wrote the first few config tools in python, >>>>> and was happy to see the python dependency gone, I think bringing >>>>> python back in would be a mistake. >>>>> >>>>> Every single python test framework I've worked with works until there >>>>> is a problem, and I then find myself having to walk back a python call >>>>> trace, because what inevitably breaks is the test framework tooling. >>>>> It's why so many projects are removing python test frameworks. >>>>> >>>>> If you want a good test framework, get the linuxboot fork of >>>>> facebook's contest github.com/linuxboot/contest, written in Go, in use >>>>> at scale near you. >>>>> >>>>> It's easy to let the joy of building a build system overwhelm the >>>>> actual goals of a project. coreboot is not about being a build system. >>>>> It's easy to fall into the trap of creating an ever more complex >>>>> system that is more than is needed. >>>>> >>>>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot >>>>> <[email protected]> wrote: >>>>> > >>>>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal < >>>>> [email protected]>: >>>>> >> >>>>> >> With respect to Kconfig, we (at Google) encountered a lovely build >>>>> flake after the Kconfig uprev last month in the coreboot tree that took a >>>>> couple of weeks to sort out and resolve. Some sort of automated validation >>>>> that the code is working could have possibly helped. Of course, the C >>>>> implementation of Kconfig has no tests at all. Some tests is better than >>>>> nothing. >>>>> > >>>>> > >>>>> > Let me put the record straight: >>>>> > >>>>> > - The last kconfig "uprev" hasn't been a simple update the way >>>>> https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the >>>>> entire build system integration to ease maintenance >>>>> > - That issue sprang up because before the kconfig update, we were >>>>> shipping prebuilt parser files (C code) while now we made bison and flex >>>>> hard dependencies for our build >>>>> > - Tests covering the C code wouldn't have helped, because the issue >>>>> wasn't in the C code >>>>> > - The issue we were facing has been an external dependency (namely: >>>>> the Chromium OS development environment shipping a broken version of >>>>> bison(1)) >>>>> > - Fixing bison in CrOS wouldn't have helped any because we have to >>>>> assume that other users come with the same kind of broken bison tool >>>>> > - The fix has been to ship pre-built files again to remove an >>>>> external dependency >>>>> > >>>>> > The alternative that we did actually consider was to add support for >>>>> building bison and flex to util/crossgcc/buildgcc. For three files that's >>>>> sheer madness, so we went back to precompiled files instead. >>>>> > >>>>> > In relation to your proposal to adopt kconfiglib: We can run into >>>>> any kind of external tool demonstrating weird behavior. That's true for >>>>> bison (as seen here) just as it can be true for arbitrary python versions >>>>> (even when specified to be "python 3.6+" or whatever): Linux distributions >>>>> do strange things to their packages, and we're not a Linux-only project, >>>>> so >>>>> even official, unchanged, straight-from-the-server python might behave >>>>> unexpectedly on less well exercised platforms. >>>>> > >>>>> > The best way to reduce issues on that end is to avoid external >>>>> dependencies - like bison, like flex... like python. >>>>> > I'd _love_ to avoid the dependency on the host compiler as well, but >>>>> that's one of those "sheer madness" moments when you support a multitude >>>>> of >>>>> operating systems on as many architectures. >>>>> > >>>>> > Introducing kconfiglib (and through it, a deep reliance on python) >>>>> just doesn't seem to provide comparable benefits. >>>>> > >>>>> > >>>>> > Patrick >>>>> > -- >>>>> > Google Germany GmbH, ABC-Str. 19, 20354 Hamburg >>>>> > Registergericht und -nummer: Hamburg, HRB 86891, Sitz der >>>>> Gesellschaft: Hamburg >>>>> > Geschäftsführer: Paul Manicle, Halimah DeLaine Prado >>>>> > _______________________________________________ >>>>> > coreboot mailing list -- [email protected] >>>>> > To unsubscribe send an email to [email protected] >>>>> >>>> >>> >>> -- >>> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg >>> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: >>> Hamburg >>> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado >>> >> >> >> -- >> >> Jack Rosenthal (he/him/his) >> >> Software Engineer - Chrome OS >> >> Google Boulder >> >> [email protected] >> >> I value feedback from others. Please feel free to contact me directly, or >> file it anonymously at go/jrosenth-feedback >> <https://goto.google.com/jrosenth-feedback>. >> >
_______________________________________________ coreboot mailing list -- [email protected] To unsubscribe send an email to [email protected]
