contest is a coming standard. The companies I work with have moved away from python test frameworks. Familiarity did not breed respect for python test frameworks.
On Tue, Oct 12, 2021 at 7:31 AM Jack Rosenthal <[email protected]> wrote: > Both bats and expect seem problematic for Ricardo's use case ... > generating the elog binary format in these tools seems difficult, bash > wasn't really meant for generating C structs. > > One huge advantage of pytest is that it's fairly industry standard at this > point. There's a good number of people who have used it, and there's a > large community around it (i.e., finding Q&A on sites like stack overflow > is possible). I wasn't able to find a single stack overflow question about > ConTest, although I may be looking in the wrong places (looks like they do > have a slack group chat). > > I believe Ricardo had an open question on the CL if it was OK to submit as > an optional test suite for elogtool. I.e., tests don't need to run on CI > tools or anything, if there's developers that want to run the tests they > can if they have Python. > > > > On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi <[email protected]> wrote: > >> Hi Ricardo, >> >> sorry for the late response, and that your project fell a bit by the >> wayside. I guess discussion configuration frameworks is more attractive to >> this community than testing frameworks (which also explains why we have ~3 >> config frameworks and only ~1 testing frameworks ;-) ) >> >> So yes, testing is something we really need to improve on. I'm not sure >> if "contest" is the right solution to your particular problem though. The >> first thing I see when opening up its page is something about mysql, and >> scrolling down, something about submitting jobs to a server. That seems >> more like a potential replacement to our Jenkins install (qa.coreboot.org) >> rather than something to easily write end-to-end tests for our userland >> tools. >> >> Looking for options, my first instinct was to go for expect(1), but >> that's really best for interactive uses - might be interesting if we ever >> grow interactive tools, but so far we manage to stick to clean and tidy >> CLI. Then I ran into https://github.com/bats-core/bats-core. That seems >> maintained, reasonably minimal in its dependencies, it emits TAP which is >> as good as JUnit in terms of Jenkins-integration (so we can have >> qa.coreboot.org parse the results and give meaningful feedback on >> review.coreboot.org), and it seems to be fairly widely used for similar >> things to what you're doing, see >> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it >> points to many, many code examples, e.g. >> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats >> which should cover the basic "call some command, see what it did" scenario >> quite nicely. >> >> Of course in the end it's all a matter of taste, and that's why I opened >> the can of worms again that is Python-use in coreboot land. As python >> hasn't seen a warmer reception than last time, I'd look for alternatives. >> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if >> writing tests with that makes you want to scream and run away, we'd have to >> look for other options (please, don't run away!) >> >> >> Regards, >> Patrick >> >> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada < >> [email protected]>: >> >>> Hi all, >>> >>> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend >>> way to go forward? >>> I just need to run one of the Coreboot's utils (in this case >>> "elogtool"), and make sure the output is the expected one. >>> >>> Should I use Contest, as suggested by Ron Minnich? >>> >>> Thanks! >>> >>> >>> >>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich <[email protected]> wrote: >>> >>>> Speaking as the person who wrote the first few config tools in python, >>>> and was happy to see the python dependency gone, I think bringing >>>> python back in would be a mistake. >>>> >>>> Every single python test framework I've worked with works until there >>>> is a problem, and I then find myself having to walk back a python call >>>> trace, because what inevitably breaks is the test framework tooling. >>>> It's why so many projects are removing python test frameworks. >>>> >>>> If you want a good test framework, get the linuxboot fork of >>>> facebook's contest github.com/linuxboot/contest, written in Go, in use >>>> at scale near you. >>>> >>>> It's easy to let the joy of building a build system overwhelm the >>>> actual goals of a project. coreboot is not about being a build system. >>>> It's easy to fall into the trap of creating an ever more complex >>>> system that is more than is needed. >>>> >>>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot >>>> <[email protected]> wrote: >>>> > >>>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal < >>>> [email protected]>: >>>> >> >>>> >> With respect to Kconfig, we (at Google) encountered a lovely build >>>> flake after the Kconfig uprev last month in the coreboot tree that took a >>>> couple of weeks to sort out and resolve. Some sort of automated validation >>>> that the code is working could have possibly helped. Of course, the C >>>> implementation of Kconfig has no tests at all. Some tests is better than >>>> nothing. >>>> > >>>> > >>>> > Let me put the record straight: >>>> > >>>> > - The last kconfig "uprev" hasn't been a simple update the way >>>> https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the >>>> entire build system integration to ease maintenance >>>> > - That issue sprang up because before the kconfig update, we were >>>> shipping prebuilt parser files (C code) while now we made bison and flex >>>> hard dependencies for our build >>>> > - Tests covering the C code wouldn't have helped, because the issue >>>> wasn't in the C code >>>> > - The issue we were facing has been an external dependency (namely: >>>> the Chromium OS development environment shipping a broken version of >>>> bison(1)) >>>> > - Fixing bison in CrOS wouldn't have helped any because we have to >>>> assume that other users come with the same kind of broken bison tool >>>> > - The fix has been to ship pre-built files again to remove an >>>> external dependency >>>> > >>>> > The alternative that we did actually consider was to add support for >>>> building bison and flex to util/crossgcc/buildgcc. For three files that's >>>> sheer madness, so we went back to precompiled files instead. >>>> > >>>> > In relation to your proposal to adopt kconfiglib: We can run into any >>>> kind of external tool demonstrating weird behavior. That's true for bison >>>> (as seen here) just as it can be true for arbitrary python versions (even >>>> when specified to be "python 3.6+" or whatever): Linux distributions do >>>> strange things to their packages, and we're not a Linux-only project, so >>>> even official, unchanged, straight-from-the-server python might behave >>>> unexpectedly on less well exercised platforms. >>>> > >>>> > The best way to reduce issues on that end is to avoid external >>>> dependencies - like bison, like flex... like python. >>>> > I'd _love_ to avoid the dependency on the host compiler as well, but >>>> that's one of those "sheer madness" moments when you support a multitude of >>>> operating systems on as many architectures. >>>> > >>>> > Introducing kconfiglib (and through it, a deep reliance on python) >>>> just doesn't seem to provide comparable benefits. >>>> > >>>> > >>>> > Patrick >>>> > -- >>>> > Google Germany GmbH, ABC-Str. 19, 20354 Hamburg >>>> > Registergericht und -nummer: Hamburg, HRB 86891, Sitz der >>>> Gesellschaft: Hamburg >>>> > Geschäftsführer: Paul Manicle, Halimah DeLaine Prado >>>> > _______________________________________________ >>>> > coreboot mailing list -- [email protected] >>>> > To unsubscribe send an email to [email protected] >>>> >>> >> >> -- >> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg >> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: >> Hamburg >> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado >> > > > -- > > Jack Rosenthal (he/him/his) > > Software Engineer - Chrome OS > > Google Boulder > > [email protected] > > I value feedback from others. Please feel free to contact me directly, or > file it anonymously at go/jrosenth-feedback > <https://goto.google.com/jrosenth-feedback>. >
_______________________________________________ coreboot mailing list -- [email protected] To unsubscribe send an email to [email protected]
