[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

Both bats and expect seem problematic for Ricardo's use case ... generating
the elog binary format in these tools seems difficult, bash wasn't really
meant for generating C structs.

One huge advantage of pytest is that it's fairly industry standard at this
point. There's a good number of people who have used it, and there's a
large community around it (i.e., finding Q&A on sites like stack overflow
is possible). I wasn't able to find a single stack overflow question about
ConTest, although I may be looking in the wrong places (looks like they do
have a slack group chat).

I believe Ricardo had an open question on the CL if it was OK to submit as
an optional test suite for elogtool. I.e., tests don't need to run on CI
tools or anything, if there's developers that want to run the tests they
can if they have Python.



On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi  wrote:

> Hi Ricardo,
>
> sorry for the late response, and that your project fell a bit by the
> wayside. I guess discussion configuration frameworks is more attractive to
> this community than testing frameworks (which also explains why we have ~3
> config frameworks and only ~1 testing frameworks ;-) )
>
> So yes, testing is something we really need to improve on. I'm not sure if
> "contest" is the right solution to your particular problem though. The
> first thing I see when opening up its page is something about mysql, and
> scrolling down, something about submitting jobs to a server. That seems
> more like a potential replacement to our Jenkins install (qa.coreboot.org)
> rather than something to easily write end-to-end tests for our userland
> tools.
>
> Looking for options, my first instinct was to go for expect(1), but that's
> really best for interactive uses - might be interesting if we ever grow
> interactive tools, but so far we manage to stick to clean and tidy CLI.
> Then I ran into https://github.com/bats-core/bats-core. That seems
> maintained, reasonably minimal in its dependencies, it emits TAP which is
> as good as JUnit in terms of Jenkins-integration (so we can have
> qa.coreboot.org parse the results and give meaningful feedback on
> review.coreboot.org), and it seems to be fairly widely used for similar
> things to what you're doing, see
> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it
> points to many, many code examples, e.g.
> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
> which should cover the basic "call some command, see what it did" scenario
> quite nicely.
>
> Of course in the end it's all a matter of taste, and that's why I opened
> the can of worms again that is Python-use in coreboot land. As python
> hasn't seen a warmer reception than last time, I'd look for alternatives.
> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if
> writing tests with that makes you want to scream and run away, we'd have to
> look for other options (please, don't run away!)
>
>
> Regards,
> Patrick
>
> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <
> ricar...@google.com>:
>
>> Hi all,
>>
>> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend
>> way to go forward?
>> I just need to run one of the Coreboot's utils (in this case "elogtool"),
>> and make sure the output is the expected one.
>>
>> Should I use Contest, as suggested by Ron Minnich?
>>
>> Thanks!
>>
>>
>>
>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>>
>>> Speaking as the person who wrote the first few config tools in python,
>>> and was happy to see the python dependency gone, I think bringing
>>> python back in would be a mistake.
>>>
>>> Every single python test framework I've worked with works until there
>>> is a problem, and I then find myself having to walk back a python call
>>> trace, because what inevitably breaks is the test framework tooling.
>>> It's why so many projects are removing python test frameworks.
>>>
>>> If you want a good test framework, get the linuxboot fork of
>>> facebook's contest github.com/linuxboot/contest, written in Go, in use
>>> at scale near you.
>>>
>>> It's easy to let the joy of building a build system overwhelm the
>>> actual goals of a project. coreboot is not about being a build system.
>>> It's easy to fall into the trap of creating an ever more complex
>>> system that is more than is needed.
>>>
>>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
>>>  wrote:
>>> >
>>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
>>> jrose...@google.com>:
>>> >>
>>> >> With respect to Kconfig, we (at Google) encountered a lovely build
>>> flake after the Kconfig uprev last month in the coreboot tree that took a
>>> couple of weeks to sort out and resolve. Some sort of automated validation
>>> that the code is working could have possibly helped. Of course, the C
>>> implementation of Kconfig has no tests at all. Some tests is better than
>>> nothing.
>>> >
>>> >
>>> > Let me put the record 

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Thu, Sep 30, 2021 at 8:37 AM Nico Huber  wrote:

> Hi Jack,
>
> On 30.09.21 15:22, Jack Rosenthal via coreboot wrote:
> > On Thu, Sep 30, 2021 at 2:27 AM Arthur Heymans 
> wrote:
> >
> >> As a rule of thumb, any project involving a substantial amount of Python
> >>> always ends up needing a Docker container to build. So I'm in the "no"
> camp
> >>> for making Python a dependency, however I think it's fine to keep
> things
> >>> as-is where it can be used for helper scripts and utilities for
> specific
> >>> purposes such that they aren't critical to building the tree.
> >>>
> >>
> >> I'm on the same side here. Building the documentation with python sphinx
> >> is a pain and I ended up needing docker.
> >> The same can be said about edk2/tianocore which also uses a lot of
> python
> >> in critical parts of its build system.
> >>
> >
> > For just Kconfiglib, the requirements are Python 2.7 or Python 3.2+,
> > including no additional library installation (standard libraries only).
> > Using docker in this scenario would be completely ridiculous.
>
> please keep in mind that this thread tries to discuss a general python
> requirement. Not Kconfiglib in particular. I suppose nobody doubts that
> Kconfiglib is one of the better/easier to integrate python projects.
>
> IMHO, we should discuss every added dependency separately. E.g. not just
> all of python at once but every piece of code that we can't fix in our
> own repository. I guess, in the case of Kconfiglib that would be:
>

Agreed. Some sort of specification of what the dependency is we are adding
is absolutely required.

As an example, we could say "Python 3.6 or newer, no additional libraries
required". (this is just an example, the actual version numbers etc we
settle on should be discussed).

I bet most distributions are shipping with 3.6+ these days too...


>
> * python (the interpreter)
>   * std libs (I guess part of the above)
> * Kconfiglib itself
>   unless we pull it into our repo like we did with Kconfig
>

I believe the previous discussion around Kconfiglib was to pull it into the
coreboot repo directly so we don't need to make it a dependency for people.
We don't need to clone the entire repository into our repos, everything is
pretty much self-contained to kconfiglib.py, as well as any UIs we want to
enable (e.g., menuconfig.py and guiconfig.py).


> >
> > Looking at the current Kconfiglib implementation we would be replacing
> the
> >> C code with 21873 lines of Python code that is now taking the code to
> >> deviate from what the Linux kernel is doing. I am having a hard time
> seeing
> >> a "net benefit" in this scenario. Given the mess that Python 2 to
> Python 3
> >> conversion has been (and still is), this is just inviting a lot of
> trouble
> >> into what has been a fairly stable part of coreboot for the last decade.
> >>
> >
> > 21,873 lines of code is including the tests. Wouldn't it be nice if the C
> > Kconfig implementation even had tests? :P
>
> Not sure, I never looked close enough at the code to tell if it's bad
> enough to justify the overhead of tests.
>
> >
> > IMO, any codebase is significantly easier and safer to maintain if there
> > are tests.
>
> In my experience that is only true for projects that don't focus enough
> on code reviews and refactoring. I often see the necessity of tests as
> lack of review. In some projects tests are even added to provide false
> confidence that one would need less review. IMO, the opposite is the
> case. Tests encourage less experienced developers to push patches with-
> out looking much into the code, because all tests are green, right? In
> such a scenario the effort to integrate a patch shifts from authors to
> reviewers, IMO not good.
>
> Nico
>

Tests and code review serve two separate functions: tests allowing
automatic validation that a change won't break everything, and code review
to keep the code-quality high of the code that lands. Obviously, as you
point out, there is some overlap, and tests cannot ever be 100% perfect.
But enough with the philosophy.

With respect to Kconfig, we (at Google) encountered a lovely build flake
after the Kconfig uprev last month in the coreboot tree that took a couple
of weeks to sort out and resolve. Some sort of automated validation that
the code is working could have possibly helped. Of course, the C
implementation of Kconfig has no tests at all. Some tests is better than
nothing.


-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
<https://goto.google.com/jrosenth-feedback>.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Thu, Sep 30, 2021 at 7:43 AM Patrick Georgi  wrote:

> Am Do., 30. Sept. 2021 um 15:22 Uhr schrieb Jack Rosenthal <
> jrose...@google.com>:
>
>> IMO, any codebase is significantly easier and safer to maintain if there
>> are tests.
>>
> Since we kinda-sorta support SPARK in our toolchain (not for the host
> though at this time), maybe we should evaluate doing a rewrite in SPARK
> then: Formal verification beats spot tests which may or may not cover the
> troublesome situations. ;-)
> (What I mean is that it's always possible to up the ante, not that we
> should actually do that. I'd rather get rid of the kconfig language, and
> even that idea has dubious value in this context.)
>

Yeah. Let's stay practical. Removing the Kconfig language entirely from the
tree wouldn't exactly be feasible.

 Both the C Kconfig and the Kconfiglib implementations are implementations
that exist today, let's evaluate which one of them is better from a
maintenance & stability perspective.


>
> As for the idea of a Python 4 you seem to have here (or if it does come,
>> repeating the massive language differences we had between 2 and 3), it's
>> unlikely to happen. Guido says that a "Python 4" at the scale Python 3 was
>> is unlikely to happen
>> 
>> .
>>
> If Python 3->4 is only half as painful as 2->3, this would still be true -
> and still mean misery for 5+ years. No thanks.
>

I don't even think even half is in domain... if you give the article to
read, Guido basically says we might just stay on Python 3 forever.


>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>


-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Thu, Sep 30, 2021 at 2:27 AM Arthur Heymans  wrote:

> As a rule of thumb, any project involving a substantial amount of Python
>> always ends up needing a Docker container to build. So I'm in the "no" camp
>> for making Python a dependency, however I think it's fine to keep things
>> as-is where it can be used for helper scripts and utilities for specific
>> purposes such that they aren't critical to building the tree.
>>
>
> I'm on the same side here. Building the documentation with python sphinx
> is a pain and I ended up needing docker.
> The same can be said about edk2/tianocore which also uses a lot of python
> in critical parts of its build system.
>

For just Kconfiglib, the requirements are Python 2.7 or Python 3.2+,
including no additional library installation (standard libraries only).
Using docker in this scenario would be completely ridiculous.

Looking at the current Kconfiglib implementation we would be replacing the
> C code with 21873 lines of Python code that is now taking the code to
> deviate from what the Linux kernel is doing. I am having a hard time seeing
> a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
> conversion has been (and still is), this is just inviting a lot of trouble
> into what has been a fairly stable part of coreboot for the last decade.
>

21,873 lines of code is including the tests. Wouldn't it be nice if the C
Kconfig implementation even had tests? :P

IMO, any codebase is significantly easier and safer to maintain if there
are tests.


>
> In terms of Kconfiglib's stability and track record: I think it has it
>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>> without any issues at all.
>>
>
> I am failing to see how anybody involved in coreboot would sign up for and
> commit to porting 20k lines of Python code to the next version, when it
> arrives. My indication is that not even the python code that is currently
> in the tree has been ported to python3 yet.
>

As for the idea of a Python 4 you seem to have here (or if it does come,
repeating the massive language differences we had between 2 and 3), it's
unlikely to happen. Guido says that a "Python 4" at the scale Python 3 was
is unlikely to happen

.

-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Wed, Sep 29, 2021 at 9:57 AM Patrick Georgi  wrote:

> Am Mi., 29. Sept. 2021 um 16:43 Uhr schrieb Jack Rosenthal <
> jrose...@google.com>:
>
>> Overall I think introducing Python to the build would provide net
>> benefit, mainly from Kconfiglib, but could also find other good uses in e2e
>> tests like Ricardo was working on. Most people's Linux distros ship with a
>> Python interpreter too, so most developers would be unlikely to notice the
>> extra dependency introduction.
>>
> This is assuming that all python environments are alike. Experience
> disagrees.
>

Kconfiglib only uses Python standard libraries, and works with both Python
3.3+ and 2.7. I think it'd be pretty hard to find a Python environment it
didn't work with.


>
>
>> In terms of Kconfiglib, we have a lot to gain by switching away from the
>> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
>> forked from the Linux tree and hacked in our own customizations
>> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
>> a miniature Python script
>> 
>> that we use to handle our custom header format, and a stable API to work
>> off of so that we can uprev Kconfiglib without needing to change our
>> scripts.
>>
> The customizations are a set of patches with simple maintenance (as
> documented in https://review.coreboot.org/c/coreboot/+/57879).
>

This is true, but the maintenance burden could be even less.

Additionally, the last Kconfig uprev didn't go so well, at least from the
Chrome OS perspective, we saw a build flake that sat around for a few weeks
bothering people.


>
> In terms of Kconfiglib's stability and track record: I think it has it
>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>> without any issues at all.
>>
> This project deals in 20 year timelines. Zephyr is at 5.5 years, while
> depthcharge is used exclusively in a tightly controlled environment (and
> even there we had - and have - our share of pythonic problems).
>

IMO 5.5 years of stability is a pretty good point to start considering
adopting a new technology. If we wait for 20 years of it, there'll be
something better than Kconfiglib out there ;)


>
> At a minimum, I think we should consider introducing Python on an optional
>> basis (i.e., the C Kconfig implementation only gets used if a Python
>> interpreter is unavailable), but making it required would be even better.
>>
> That idea is... horrible. Instead of only bringing in the python
> dependency we'd also create an environmental dependency that can silently
> introduce behavioral differences.
>
> This project has the ability of building bit-identical firmware images on
> hosts spanning all kinds of CPU register sizes, endianness and operating
> system families. If you ever heard of "hermetic builds" being a good idea:
> the trustworthiness of our build is stronger - and part of making that
> possible is being (relatively) careful with picking our dependencies.
>

Fair, require python then. Most people have it on their systems anyways.


>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>


-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

Overall I think introducing Python to the build would provide net benefit,
mainly from Kconfiglib, but could also find other good uses in e2e tests
like Ricardo was working on. Most people's Linux distros ship with a Python
interpreter too, so most developers would be unlikely to notice the extra
dependency introduction.

In terms of Kconfiglib, we have a lot to gain by switching away from the
Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
forked from the Linux tree and hacked in our own customizations
(KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
a miniature Python script

that we use to handle our custom header format, and a stable API to work
off of so that we can uprev Kconfiglib without needing to change our
scripts.

In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
without any issues at all.

At a minimum, I think we should consider introducing Python on an optional
basis (i.e., the C Kconfig implementation only gets used if a Python
interpreter is unavailable), but making it required would be even better.

On Wed, Sep 29, 2021 at 5:39 AM Rao G  wrote:

> Hi Patrick,
>
> That's good to hear, would there be change to "make menuconfig" with
> kconfiglib
>

No, we'd make it so that all the "make *config" commands run the Kconfiglib
alternatives without any user action required.

The "menuconfig" interface in specific is very similar to the
lxdialog-based interface that the C Kconfig uses, except it's a bit more
polished and refined feeling.


>
> Thanks
> Ranga
>
> On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
> coreboot@coreboot.org> wrote:
>
>> Hi everybody,
>>
>> Historically, coreboot avoided depending on python too much (we got rid
>> of an entire python based configuration and build system, for example),
>> with few minor exceptions.
>>
>> The main reason has been that while python code is quick to slap
>> together, it has demonstrated a penchant for breaking in all kinds of
>> mysterious ways (python 2->3 really was just a slightly bigger instance of
>> what's going on in python all the time), and its users demonstrate a
>> disregard for their fellow developers as demonstrated by endless stack
>> traces on trivial errors (or is the language too complicated to properly
>> catch them all?)
>>
>> While probably nice for one-off prototypes, long term maintenance is a
>> concern: this project has over 20 years of history under its belt, with
>> more to come.
>>
>> That said, python makes its way back into the tree every now and then
>> (typically as small snippets to compute and add hashes to binaries as
>> needed by ARM SoCs). Uncanny, but typically not a big deal.
>>
>> There are two bigger initiatives proposed that would significantly
>> increase our python footprint:
>> 1. Replacing Linux's kconfig with kconfiglib (
>> https://review.coreboot.org/c/coreboot/+/48679)
>> 2. Using pytest for end-to-end testing utilities (
>> https://review.coreboot.org/c/coreboot/+/57869)
>>
>> Compared to the "inject a hash value at a fixed location" scripts, these
>> would probably be here to stay, and sufficiently integrated that everybody
>> will have to deal with them.
>>
>> People spending time working on python code when it has no chance to land
>> isn't a good use of their time and we should avoid that in the project.
>>
>> People spending time arguing that python shouldn't be used (to avoid
>> the other outcome) even though the project's culture shifted and is now
>> accepting Python isn't creating a great community for anybody.
>>
>> To avoid these scenarios, could we possibly nail down the policy on
>> python in coreboot?
>>
>>
>> Regards,
>> Patrick
>> --
>> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
>> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
>> Hamburg
>> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>> ___
>> coreboot mailing list -- coreboot@coreboot.org
>> To unsubscribe send an email to coreboot-le...@coreboot.org
>>
>

-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org