[coreboot] Re: There is a python in our toolchain?!?

[ron minnich]

Still working on getting the setup, but 9elements reports: "We have it
running as CI System attached to a Github repo and do build and boot
testing on Qemu und real hardware with it."

On Sat, Oct 16, 2021 at 1:39 AM Patrick Georgi  wrote:
>
> Am Sa., 16. Okt. 2021 um 02:40 Uhr schrieb ron minnich :
>>
>> Contest is easy to set up, easy to run, it's
>> getting contributors. I understand it's a commitment of a day or so to
>> figure out, but it's worth it in our experience. It's just not that
>> hard.
>>
>> I believe starting down the python path is a bad start, and I'd rather
>> not make it.
>
> It is, but that's the code proposed on the repo.
> I asked you about using Contest in a setting that comes without client/server 
> architecture, SQL database and a dedicated system under test. Any news on 
> that end?
>
>> I realize "it's only 160 lines", now; but that's how these things
>> always start. They don't end well.
>
> It's not a license for adding more of the same.
>
> I consider it:
> 1. A signal that we care for testing our tools (as in: we like contributions 
> that improve our test coverage).
> 2. A signal of encouragement towards Ricardo who haplessly ran into the trap 
> of coreboot discouraging (ahem) python (we should document that!) while 
> trying to improve the project's general posture.
>
> As soon as you bring up an alternative path that's acceptable to the project 
> at large, Ricardo offered to rewrite this specific test in whatever we'll use 
> officially (I suppose there's a caveat of it being "within reason": don't get 
> out your brainf*ck-based e2e testing framework!), and I hereby offer to be 
> the rewriter-of-last-resort in case Ricardo is gone by then.
>
> As is, Content looks like a solution for a _different_ problem. As soon as we 
> can clear up that confusion, I'm all for using it, and the faster we get that 
> done, the faster this python code is removed again.
>
> Therefore I guess one could say that I also consider this change (and my 
> proposal of merging it):
>
> 3. A rather blunt tool to get you to resolve the open questions regarding 
> Contest ;-)
>
>
> Patrick
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Am Sa., 16. Okt. 2021 um 02:40 Uhr schrieb ron minnich :

> Contest is easy to set up, easy to run, it's
> getting contributors. I understand it's a commitment of a day or so to
> figure out, but it's worth it in our experience. It's just not that
> hard.
>
> I believe starting down the python path is a bad start, and I'd rather
> not make it.

It is, but that's the code proposed on the repo.
I asked you about using Contest in a setting that comes without
client/server architecture, SQL database and a dedicated system under test.
Any news on that end?

I realize "it's only 160 lines", now; but that's how these things
> always start. They don't end well.
>
It's not a license for adding more of the same.

I consider it:
1. A signal that we care for testing our tools (as in: we like
contributions that improve our test coverage).
2. A signal of encouragement towards Ricardo who haplessly ran into the
trap of coreboot discouraging (ahem) python (we should document that!)
while trying to improve the project's general posture.

As soon as you bring up an alternative path that's acceptable to the
project at large, Ricardo offered to rewrite this specific test in whatever
we'll use officially (I suppose there's a caveat of it being "within
reason": don't get out your brainf*ck-based e2e testing framework!), and I
hereby offer to be the rewriter-of-last-resort in case Ricardo is gone by
then.

As is, Content looks like a solution for a _different_ problem. As soon as
we can clear up that confusion, I'm all for using it, and the faster we get
that done, the faster this python code is removed again.

Therefore I guess one could say that I also consider this change (and my
proposal of merging it):

3. A rather blunt tool to get you to resolve the open questions regarding
Contest ;-)


Patrick
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[ron minnich]

I would rather we not start depending on pytest. Just my take.There's
a difference between a utility for one chipset and global
infrastructure, and pytest could become the latter.

There's a pretty big testing activity starting up in OCP, involving
many companies, and python is not something we will be using, for all
the usual reasons. Some of those companies had python test infra and
they are moving on. Contest is easy to set up, easy to run, it's
getting contributors. I understand it's a commitment of a day or so to
figure out, but it's worth it in our experience. It's just not that
hard.

I believe starting down the python path is a bad start, and I'd rather
not make it. I like to know programs have errors in the source code
before I run them. I spend too much time debugging python call traces,
having it work back to a missing module, to have confidence in it.
Just recently I ran a program which has worked for months, and got
this:
Traceback (most recent call last):
  File "/Users/rminnich/projects/python-dlipower/scripts/dlipower",
line 30, in 
from dlipower import PowerSwitch
ModuleNotFoundError: No module named 'dlipower'

What changed? nothing. Why did it break? I don't know. This sucks, but
it super sucks when it happens on a batch test system and you're on a
deadline. no no no no no.

Again, speaking as the person who wrote the first three device tree
parsers in python, I was not as opposed to python for infra back then,
but I am now. Python is always a headache every time I have to deal
with it for things like test.

I realize "it's only 160 lines", now; but that's how these things
always start. They don't end well.

On Fri, Oct 15, 2021 at 11:06 AM Patrick Georgi  wrote:
>
> Am Fr., 15. Okt. 2021 um 19:50 Uhr schrieb Ricardo Quesada 
> :
>>
>> In the meantime, would it make sense, as Jack mentioned, to land my change 
>> [1] as it is? It is small/simple and it only has  ~160 LoC Python.
>> For comparison, other util are using Python: util/qualcomm has ~3500 LoC 
>> Python [2]
>> I'll happily migrate + integrate my test once a end-to-end test has been 
>> chosen.
>
> I'm fine getting it in for now, but it won't see testing every build.
>
> Will wait for other people to chime in, though. If I don't see pushback 
> against getting this in as-is (with the understanding that it might be out 
> quickly again once we decide on a plan of action) by next Thursday at noon in 
> Germany (UTC+0200), I'll see that I submit it.
>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: 
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Am Fr., 15. Okt. 2021 um 19:50 Uhr schrieb Ricardo Quesada <
ricar...@google.com>:

> In the meantime, would it make sense, as Jack mentioned, to land my change
> [1] as it is? It is small/simple and it only has  ~160 LoC Python.
> For comparison, other util are using Python: util/qualcomm has ~3500 LoC
> Python [2]
> I'll happily migrate + integrate my test once a end-to-end test has been
> chosen.
>
I'm fine getting it in for now, but it won't see testing every build.

Will wait for other people to chime in, though. If I don't see pushback
against getting this in as-is (with the understanding that it might be out
quickly again once we decide on a plan of action) by next Thursday at noon
in Germany (UTC+0200), I'll see that I submit it.


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Ricardo Quesada via coreboot]

Thanks for the answers!

Regarding the end-to-end testing framework, perhaps that's a topic that
needs further discussion / consensus (?). E.g:
  - discuss policy: what parts of coreboot should have end-to-end testing.
Should it be forced? recommended? optional?
  - do some requirement gathering: not only from the  firmware but also for
the coreboot/utils
  - Update Jenkins accordingly

In the meantime, would it make sense, as Jack mentioned, to land my change
[1] as it is? It is small/simple and it only has  ~160 LoC Python.
For comparison, other util are using Python: util/qualcomm has ~3500 LoC
Python [2]
I'll happily migrate + integrate my test once a end-to-end test has been
chosen.

Thanks!

[1]: https://review.coreboot.org/c/coreboot/+/57869
[2]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm

On Tue, Oct 12, 2021 at 8:30 AM ron minnich  wrote:

> contest is a coming standard. The companies I work with have moved away
> from python test frameworks. Familiarity did not breed respect for python
> test frameworks.
>
> On Tue, Oct 12, 2021 at 7:31 AM Jack Rosenthal 
> wrote:
>
>> Both bats and expect seem problematic for Ricardo's use case ...
>> generating the elog binary format in these tools seems difficult, bash
>> wasn't really meant for generating C structs.
>>
>> One huge advantage of pytest is that it's fairly industry standard at
>> this point. There's a good number of people who have used it, and there's a
>> large community around it (i.e., finding Q&A on sites like stack overflow
>> is possible). I wasn't able to find a single stack overflow question about
>> ConTest, although I may be looking in the wrong places (looks like they do
>> have a slack group chat).
>>
>> I believe Ricardo had an open question on the CL if it was OK to submit
>> as an optional test suite for elogtool. I.e., tests don't need to run on CI
>> tools or anything, if there's developers that want to run the tests they
>> can if they have Python.
>>
>>
>>
>> On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi 
>> wrote:
>>
>>> Hi Ricardo,
>>>
>>> sorry for the late response, and that your project fell a bit by the
>>> wayside. I guess discussion configuration frameworks is more attractive to
>>> this community than testing frameworks (which also explains why we have ~3
>>> config frameworks and only ~1 testing frameworks ;-) )
>>>
>>> So yes, testing is something we really need to improve on. I'm not sure
>>> if "contest" is the right solution to your particular problem though. The
>>> first thing I see when opening up its page is something about mysql, and
>>> scrolling down, something about submitting jobs to a server. That seems
>>> more like a potential replacement to our Jenkins install (
>>> qa.coreboot.org) rather than something to easily write end-to-end tests
>>> for our userland tools.
>>>
>>> Looking for options, my first instinct was to go for expect(1), but
>>> that's really best for interactive uses - might be interesting if we ever
>>> grow interactive tools, but so far we manage to stick to clean and tidy
>>> CLI. Then I ran into https://github.com/bats-core/bats-core. That seems
>>> maintained, reasonably minimal in its dependencies, it emits TAP which is
>>> as good as JUnit in terms of Jenkins-integration (so we can have
>>> qa.coreboot.org parse the results and give meaningful feedback on
>>> review.coreboot.org), and it seems to be fairly widely used for similar
>>> things to what you're doing, see
>>> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it
>>> points to many, many code examples, e.g.
>>> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
>>> which should cover the basic "call some command, see what it did" scenario
>>> quite nicely.
>>>
>>> Of course in the end it's all a matter of taste, and that's why I opened
>>> the can of worms again that is Python-use in coreboot land. As python
>>> hasn't seen a warmer reception than last time, I'd look for alternatives.
>>> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if
>>> writing tests with that makes you want to scream and run away, we'd have to
>>> look for other options (please, don't run away!)
>>>
>>>
>>> Regards,
>>> Patrick
>>>
>>> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <
>>> ricar...@google.com>:
>>>
 Hi all,

 Regarding Patrick's 2nd point (end-to-end testing), what's the
 recommend way to go forward?
 I just need to run one of the Coreboot's utils (in this case
 "elogtool"), and make sure the output is the expected one.

 Should I use Contest, as suggested by Ron Minnich?

 Thanks!



 On Thu, Sep 30, 2021 at 10:18 AM ron minnich 
 wrote:

> Speaking as the person who wrote the first few config tools in python,
> and was happy to see the python dependency gone, I think bringing
> python back in would be a mistake.
>
> Every single python test fr

[coreboot] Re: There is a python in our toolchain?!?

[ron minnich]

contest is a coming standard. The companies I work with have moved away
from python test frameworks. Familiarity did not breed respect for python
test frameworks.

On Tue, Oct 12, 2021 at 7:31 AM Jack Rosenthal  wrote:

> Both bats and expect seem problematic for Ricardo's use case ...
> generating the elog binary format in these tools seems difficult, bash
> wasn't really meant for generating C structs.
>
> One huge advantage of pytest is that it's fairly industry standard at this
> point. There's a good number of people who have used it, and there's a
> large community around it (i.e., finding Q&A on sites like stack overflow
> is possible). I wasn't able to find a single stack overflow question about
> ConTest, although I may be looking in the wrong places (looks like they do
> have a slack group chat).
>
> I believe Ricardo had an open question on the CL if it was OK to submit as
> an optional test suite for elogtool. I.e., tests don't need to run on CI
> tools or anything, if there's developers that want to run the tests they
> can if they have Python.
>
>
>
> On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi  wrote:
>
>> Hi Ricardo,
>>
>> sorry for the late response, and that your project fell a bit by the
>> wayside. I guess discussion configuration frameworks is more attractive to
>> this community than testing frameworks (which also explains why we have ~3
>> config frameworks and only ~1 testing frameworks ;-) )
>>
>> So yes, testing is something we really need to improve on. I'm not sure
>> if "contest" is the right solution to your particular problem though. The
>> first thing I see when opening up its page is something about mysql, and
>> scrolling down, something about submitting jobs to a server. That seems
>> more like a potential replacement to our Jenkins install (qa.coreboot.org)
>> rather than something to easily write end-to-end tests for our userland
>> tools.
>>
>> Looking for options, my first instinct was to go for expect(1), but
>> that's really best for interactive uses - might be interesting if we ever
>> grow interactive tools, but so far we manage to stick to clean and tidy
>> CLI. Then I ran into https://github.com/bats-core/bats-core. That seems
>> maintained, reasonably minimal in its dependencies, it emits TAP which is
>> as good as JUnit in terms of Jenkins-integration (so we can have
>> qa.coreboot.org parse the results and give meaningful feedback on
>> review.coreboot.org), and it seems to be fairly widely used for similar
>> things to what you're doing, see
>> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it
>> points to many, many code examples, e.g.
>> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
>> which should cover the basic "call some command, see what it did" scenario
>> quite nicely.
>>
>> Of course in the end it's all a matter of taste, and that's why I opened
>> the can of worms again that is Python-use in coreboot land. As python
>> hasn't seen a warmer reception than last time, I'd look for alternatives.
>> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if
>> writing tests with that makes you want to scream and run away, we'd have to
>> look for other options (please, don't run away!)
>>
>>
>> Regards,
>> Patrick
>>
>> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <
>> ricar...@google.com>:
>>
>>> Hi all,
>>>
>>> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend
>>> way to go forward?
>>> I just need to run one of the Coreboot's utils (in this case
>>> "elogtool"), and make sure the output is the expected one.
>>>
>>> Should I use Contest, as suggested by Ron Minnich?
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>>>
 Speaking as the person who wrote the first few config tools in python,
 and was happy to see the python dependency gone, I think bringing
 python back in would be a mistake.

 Every single python test framework I've worked with works until there
 is a problem, and I then find myself having to walk back a python call
 trace, because what inevitably breaks is the test framework tooling.
 It's why so many projects are removing python test frameworks.

 If you want a good test framework, get the linuxboot fork of
 facebook's contest github.com/linuxboot/contest, written in Go, in use
 at scale near you.

 It's easy to let the joy of building a build system overwhelm the
 actual goals of a project. coreboot is not about being a build system.
 It's easy to fall into the trap of creating an ever more complex
 system that is more than is needed.

 On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
  wrote:
 >
 > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
 jrose...@google.com>:
 >>
 >> With respect to Kconfig, we (at Google) encountered a lovely build
 flake after the Kc

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

Both bats and expect seem problematic for Ricardo's use case ... generating
the elog binary format in these tools seems difficult, bash wasn't really
meant for generating C structs.

One huge advantage of pytest is that it's fairly industry standard at this
point. There's a good number of people who have used it, and there's a
large community around it (i.e., finding Q&A on sites like stack overflow
is possible). I wasn't able to find a single stack overflow question about
ConTest, although I may be looking in the wrong places (looks like they do
have a slack group chat).

I believe Ricardo had an open question on the CL if it was OK to submit as
an optional test suite for elogtool. I.e., tests don't need to run on CI
tools or anything, if there's developers that want to run the tests they
can if they have Python.



On Tue, Oct 12, 2021 at 8:02 AM Patrick Georgi  wrote:

> Hi Ricardo,
>
> sorry for the late response, and that your project fell a bit by the
> wayside. I guess discussion configuration frameworks is more attractive to
> this community than testing frameworks (which also explains why we have ~3
> config frameworks and only ~1 testing frameworks ;-) )
>
> So yes, testing is something we really need to improve on. I'm not sure if
> "contest" is the right solution to your particular problem though. The
> first thing I see when opening up its page is something about mysql, and
> scrolling down, something about submitting jobs to a server. That seems
> more like a potential replacement to our Jenkins install (qa.coreboot.org)
> rather than something to easily write end-to-end tests for our userland
> tools.
>
> Looking for options, my first instinct was to go for expect(1), but that's
> really best for interactive uses - might be interesting if we ever grow
> interactive tools, but so far we manage to stick to clean and tidy CLI.
> Then I ran into https://github.com/bats-core/bats-core. That seems
> maintained, reasonably minimal in its dependencies, it emits TAP which is
> as good as JUnit in terms of Jenkins-integration (so we can have
> qa.coreboot.org parse the results and give meaningful feedback on
> review.coreboot.org), and it seems to be fairly widely used for similar
> things to what you're doing, see
> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it
> points to many, many code examples, e.g.
> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
> which should cover the basic "call some command, see what it did" scenario
> quite nicely.
>
> Of course in the end it's all a matter of taste, and that's why I opened
> the can of worms again that is Python-use in coreboot land. As python
> hasn't seen a warmer reception than last time, I'd look for alternatives.
> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if
> writing tests with that makes you want to scream and run away, we'd have to
> look for other options (please, don't run away!)
>
>
> Regards,
> Patrick
>
> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <
> ricar...@google.com>:
>
>> Hi all,
>>
>> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend
>> way to go forward?
>> I just need to run one of the Coreboot's utils (in this case "elogtool"),
>> and make sure the output is the expected one.
>>
>> Should I use Contest, as suggested by Ron Minnich?
>>
>> Thanks!
>>
>>
>>
>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>>
>>> Speaking as the person who wrote the first few config tools in python,
>>> and was happy to see the python dependency gone, I think bringing
>>> python back in would be a mistake.
>>>
>>> Every single python test framework I've worked with works until there
>>> is a problem, and I then find myself having to walk back a python call
>>> trace, because what inevitably breaks is the test framework tooling.
>>> It's why so many projects are removing python test frameworks.
>>>
>>> If you want a good test framework, get the linuxboot fork of
>>> facebook's contest github.com/linuxboot/contest, written in Go, in use
>>> at scale near you.
>>>
>>> It's easy to let the joy of building a build system overwhelm the
>>> actual goals of a project. coreboot is not about being a build system.
>>> It's easy to fall into the trap of creating an ever more complex
>>> system that is more than is needed.
>>>
>>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
>>>  wrote:
>>> >
>>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
>>> jrose...@google.com>:
>>> >>
>>> >> With respect to Kconfig, we (at Google) encountered a lovely build
>>> flake after the Kconfig uprev last month in the coreboot tree that took a
>>> couple of weeks to sort out and resolve. Some sort of automated validation
>>> that the code is working could have possibly helped. Of course, the C
>>> implementation of Kconfig has no tests at all. Some tests is better than
>>> nothing.
>>> >
>>> >
>>> > Let me put the record 

[coreboot] Re: There is a python in our toolchain?!?

[ron minnich]

I set up a raspberry pi as a device under test with contest, and ran
the contest controller in a mode that needed no mysql, and it took
about 14 minutes.

I'd still recommend looking at contest. I've found it very easy to use
and I am very new to it.

On Tue, Oct 12, 2021 at 7:02 AM Patrick Georgi  wrote:
>
> Hi Ricardo,
>
> sorry for the late response, and that your project fell a bit by the wayside. 
> I guess discussion configuration frameworks is more attractive to this 
> community than testing frameworks (which also explains why we have ~3 config 
> frameworks and only ~1 testing frameworks ;-) )
>
> So yes, testing is something we really need to improve on. I'm not sure if 
> "contest" is the right solution to your particular problem though. The first 
> thing I see when opening up its page is something about mysql, and scrolling 
> down, something about submitting jobs to a server. That seems more like a 
> potential replacement to our Jenkins install (qa.coreboot.org) rather than 
> something to easily write end-to-end tests for our userland tools.
>
> Looking for options, my first instinct was to go for expect(1), but that's 
> really best for interactive uses - might be interesting if we ever grow 
> interactive tools, but so far we manage to stick to clean and tidy CLI. Then 
> I ran into https://github.com/bats-core/bats-core. That seems maintained, 
> reasonably minimal in its dependencies, it emits TAP which is as good as 
> JUnit in terms of Jenkins-integration (so we can have qa.coreboot.org parse 
> the results and give meaningful feedback on review.coreboot.org), and it 
> seems to be fairly widely used for similar things to what you're doing, see 
> https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points 
> to many, many code examples, e.g. 
> https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
>  which should cover the basic "call some command, see what it did" scenario 
> quite nicely.
>
> Of course in the end it's all a matter of taste, and that's why I opened the 
> can of worms again that is Python-use in coreboot land. As python hasn't seen 
> a warmer reception than last time, I'd look for alternatives.
> Maybe Bats could do? Of course, I haven't _actually_ used it yet and if 
> writing tests with that makes you want to scream and run away, we'd have to 
> look for other options (please, don't run away!)
>
>
> Regards,
> Patrick
>
> Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada 
> :
>>
>> Hi all,
>>
>> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend way 
>> to go forward?
>> I just need to run one of the Coreboot's utils (in this case "elogtool"), 
>> and make sure the output is the expected one.
>>
>> Should I use Contest, as suggested by Ron Minnich?
>>
>> Thanks!
>>
>>
>>
>> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>>>
>>> Speaking as the person who wrote the first few config tools in python,
>>> and was happy to see the python dependency gone, I think bringing
>>> python back in would be a mistake.
>>>
>>> Every single python test framework I've worked with works until there
>>> is a problem, and I then find myself having to walk back a python call
>>> trace, because what inevitably breaks is the test framework tooling.
>>> It's why so many projects are removing python test frameworks.
>>>
>>> If you want a good test framework, get the linuxboot fork of
>>> facebook's contest github.com/linuxboot/contest, written in Go, in use
>>> at scale near you.
>>>
>>> It's easy to let the joy of building a build system overwhelm the
>>> actual goals of a project. coreboot is not about being a build system.
>>> It's easy to fall into the trap of creating an ever more complex
>>> system that is more than is needed.
>>>
>>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
>>>  wrote:
>>> >
>>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal 
>>> > :
>>> >>
>>> >> With respect to Kconfig, we (at Google) encountered a lovely build flake 
>>> >> after the Kconfig uprev last month in the coreboot tree that took a 
>>> >> couple of weeks to sort out and resolve. Some sort of automated 
>>> >> validation that the code is working could have possibly helped. Of 
>>> >> course, the C implementation of Kconfig has no tests at all. Some tests 
>>> >> is better than nothing.
>>> >
>>> >
>>> > Let me put the record straight:
>>> >
>>> > - The last kconfig "uprev" hasn't been a simple update the way 
>>> > https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire 
>>> > build system integration to ease maintenance
>>> > - That issue sprang up because before the kconfig update, we were 
>>> > shipping prebuilt parser files (C code) while now we made bison and flex 
>>> > hard dependencies for our build
>>> > - Tests covering the C code wouldn't have helped, because the issue 
>>> > wasn't in the C code
>>> > - The issue we were facing has been an external depende

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Hi Ricardo,

sorry for the late response, and that your project fell a bit by the
wayside. I guess discussion configuration frameworks is more attractive to
this community than testing frameworks (which also explains why we have ~3
config frameworks and only ~1 testing frameworks ;-) )

So yes, testing is something we really need to improve on. I'm not sure if
"contest" is the right solution to your particular problem though. The
first thing I see when opening up its page is something about mysql, and
scrolling down, something about submitting jobs to a server. That seems
more like a potential replacement to our Jenkins install (qa.coreboot.org)
rather than something to easily write end-to-end tests for our userland
tools.

Looking for options, my first instinct was to go for expect(1), but that's
really best for interactive uses - might be interesting if we ever grow
interactive tools, but so far we manage to stick to clean and tidy CLI.
Then I ran into https://github.com/bats-core/bats-core. That seems
maintained, reasonably minimal in its dependencies, it emits TAP which is
as good as JUnit in terms of Jenkins-integration (so we can have
qa.coreboot.org parse the results and give meaningful feedback on
review.coreboot.org), and it seems to be fairly widely used for similar
things to what you're doing, see
https://github.com/bats-core/bats-core/wiki/Projects-Using-Bats - it points
to many, many code examples, e.g.
https://github.com/docker/machine/blob/master/test/integration/core/core-commands.bats
which should cover the basic "call some command, see what it did" scenario
quite nicely.

Of course in the end it's all a matter of taste, and that's why I opened
the can of worms again that is Python-use in coreboot land. As python
hasn't seen a warmer reception than last time, I'd look for alternatives.
Maybe Bats could do? Of course, I haven't _actually_ used it yet and if
writing tests with that makes you want to scream and run away, we'd have to
look for other options (please, don't run away!)


Regards,
Patrick

Am Mo., 4. Okt. 2021 um 18:59 Uhr schrieb Ricardo Quesada <
ricar...@google.com>:

> Hi all,
>
> Regarding Patrick's 2nd point (end-to-end testing), what's the recommend
> way to go forward?
> I just need to run one of the Coreboot's utils (in this case "elogtool"),
> and make sure the output is the expected one.
>
> Should I use Contest, as suggested by Ron Minnich?
>
> Thanks!
>
>
>
> On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:
>
>> Speaking as the person who wrote the first few config tools in python,
>> and was happy to see the python dependency gone, I think bringing
>> python back in would be a mistake.
>>
>> Every single python test framework I've worked with works until there
>> is a problem, and I then find myself having to walk back a python call
>> trace, because what inevitably breaks is the test framework tooling.
>> It's why so many projects are removing python test frameworks.
>>
>> If you want a good test framework, get the linuxboot fork of
>> facebook's contest github.com/linuxboot/contest, written in Go, in use
>> at scale near you.
>>
>> It's easy to let the joy of building a build system overwhelm the
>> actual goals of a project. coreboot is not about being a build system.
>> It's easy to fall into the trap of creating an ever more complex
>> system that is more than is needed.
>>
>> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
>>  wrote:
>> >
>> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
>> jrose...@google.com>:
>> >>
>> >> With respect to Kconfig, we (at Google) encountered a lovely build
>> flake after the Kconfig uprev last month in the coreboot tree that took a
>> couple of weeks to sort out and resolve. Some sort of automated validation
>> that the code is working could have possibly helped. Of course, the C
>> implementation of Kconfig has no tests at all. Some tests is better than
>> nothing.
>> >
>> >
>> > Let me put the record straight:
>> >
>> > - The last kconfig "uprev" hasn't been a simple update the way
>> https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the
>> entire build system integration to ease maintenance
>> > - That issue sprang up because before the kconfig update, we were
>> shipping prebuilt parser files (C code) while now we made bison and flex
>> hard dependencies for our build
>> > - Tests covering the C code wouldn't have helped, because the issue
>> wasn't in the C code
>> > - The issue we were facing has been an external dependency (namely: the
>> Chromium OS development environment shipping a broken version of bison(1))
>> > - Fixing bison in CrOS wouldn't have helped any because we have to
>> assume that other users come with the same kind of broken bison tool
>> > - The fix has been to ship pre-built files again to remove an external
>> dependency
>> >
>> > The alternative that we did actually consider was to add support for
>> building bison and flex to util/crossgcc/buildgcc. For 

[coreboot] Re: There is a python in our toolchain?!?

[Ricardo Quesada via coreboot]

Hi all,

Regarding Patrick's 2nd point (end-to-end testing), what's the recommend
way to go forward?
I just need to run one of the Coreboot's utils (in this case "elogtool"),
and make sure the output is the expected one.

Should I use Contest, as suggested by Ron Minnich?

Thanks!



On Thu, Sep 30, 2021 at 10:18 AM ron minnich  wrote:

> Speaking as the person who wrote the first few config tools in python,
> and was happy to see the python dependency gone, I think bringing
> python back in would be a mistake.
>
> Every single python test framework I've worked with works until there
> is a problem, and I then find myself having to walk back a python call
> trace, because what inevitably breaks is the test framework tooling.
> It's why so many projects are removing python test frameworks.
>
> If you want a good test framework, get the linuxboot fork of
> facebook's contest github.com/linuxboot/contest, written in Go, in use
> at scale near you.
>
> It's easy to let the joy of building a build system overwhelm the
> actual goals of a project. coreboot is not about being a build system.
> It's easy to fall into the trap of creating an ever more complex
> system that is more than is needed.
>
> On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
>  wrote:
> >
> > Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
> jrose...@google.com>:
> >>
> >> With respect to Kconfig, we (at Google) encountered a lovely build
> flake after the Kconfig uprev last month in the coreboot tree that took a
> couple of weeks to sort out and resolve. Some sort of automated validation
> that the code is working could have possibly helped. Of course, the C
> implementation of Kconfig has no tests at all. Some tests is better than
> nothing.
> >
> >
> > Let me put the record straight:
> >
> > - The last kconfig "uprev" hasn't been a simple update the way
> https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire
> build system integration to ease maintenance
> > - That issue sprang up because before the kconfig update, we were
> shipping prebuilt parser files (C code) while now we made bison and flex
> hard dependencies for our build
> > - Tests covering the C code wouldn't have helped, because the issue
> wasn't in the C code
> > - The issue we were facing has been an external dependency (namely: the
> Chromium OS development environment shipping a broken version of bison(1))
> > - Fixing bison in CrOS wouldn't have helped any because we have to
> assume that other users come with the same kind of broken bison tool
> > - The fix has been to ship pre-built files again to remove an external
> dependency
> >
> > The alternative that we did actually consider was to add support for
> building bison and flex to util/crossgcc/buildgcc. For three files that's
> sheer madness, so we went back to precompiled files instead.
> >
> > In relation to your proposal to adopt kconfiglib: We can run into any
> kind of external tool demonstrating weird behavior. That's true for bison
> (as seen here) just as it can be true for arbitrary python versions (even
> when specified to be "python 3.6+" or whatever): Linux distributions do
> strange things to their packages, and we're not a Linux-only project, so
> even official, unchanged, straight-from-the-server python might behave
> unexpectedly on less well exercised platforms.
> >
> > The best way to reduce issues on that end is to avoid external
> dependencies - like bison, like flex... like python.
> > I'd _love_ to avoid the dependency on the host compiler as well, but
> that's one of those "sheer madness" moments when you support a multitude of
> operating systems on as many architectures.
> >
> > Introducing kconfiglib (and through it, a deep reliance on python) just
> doesn't seem to provide comparable benefits.
> >
> >
> > Patrick
> > --
> > Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> > Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
> Hamburg
> > Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
> > ___
> > coreboot mailing list -- coreboot@coreboot.org
> > To unsubscribe send an email to coreboot-le...@coreboot.org
>
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[ron minnich]

Speaking as the person who wrote the first few config tools in python,
and was happy to see the python dependency gone, I think bringing
python back in would be a mistake.

Every single python test framework I've worked with works until there
is a problem, and I then find myself having to walk back a python call
trace, because what inevitably breaks is the test framework tooling.
It's why so many projects are removing python test frameworks.

If you want a good test framework, get the linuxboot fork of
facebook's contest github.com/linuxboot/contest, written in Go, in use
at scale near you.

It's easy to let the joy of building a build system overwhelm the
actual goals of a project. coreboot is not about being a build system.
It's easy to fall into the trap of creating an ever more complex
system that is more than is needed.

On Thu, Sep 30, 2021 at 9:11 AM Patrick Georgi via coreboot
 wrote:
>
> Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal 
> :
>>
>> With respect to Kconfig, we (at Google) encountered a lovely build flake 
>> after the Kconfig uprev last month in the coreboot tree that took a couple 
>> of weeks to sort out and resolve. Some sort of automated validation that the 
>> code is working could have possibly helped. Of course, the C implementation 
>> of Kconfig has no tests at all. Some tests is better than nothing.
>
>
> Let me put the record straight:
>
> - The last kconfig "uprev" hasn't been a simple update the way 
> https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire 
> build system integration to ease maintenance
> - That issue sprang up because before the kconfig update, we were shipping 
> prebuilt parser files (C code) while now we made bison and flex hard 
> dependencies for our build
> - Tests covering the C code wouldn't have helped, because the issue wasn't in 
> the C code
> - The issue we were facing has been an external dependency (namely: the 
> Chromium OS development environment shipping a broken version of bison(1))
> - Fixing bison in CrOS wouldn't have helped any because we have to assume 
> that other users come with the same kind of broken bison tool
> - The fix has been to ship pre-built files again to remove an external 
> dependency
>
> The alternative that we did actually consider was to add support for building 
> bison and flex to util/crossgcc/buildgcc. For three files that's sheer 
> madness, so we went back to precompiled files instead.
>
> In relation to your proposal to adopt kconfiglib: We can run into any kind of 
> external tool demonstrating weird behavior. That's true for bison (as seen 
> here) just as it can be true for arbitrary python versions (even when 
> specified to be "python 3.6+" or whatever): Linux distributions do strange 
> things to their packages, and we're not a Linux-only project, so even 
> official, unchanged, straight-from-the-server python might behave 
> unexpectedly on less well exercised platforms.
>
> The best way to reduce issues on that end is to avoid external dependencies - 
> like bison, like flex... like python.
> I'd _love_ to avoid the dependency on the host compiler as well, but that's 
> one of those "sheer madness" moments when you support a multitude of 
> operating systems on as many architectures.
>
> Introducing kconfiglib (and through it, a deep reliance on python) just 
> doesn't seem to provide comparable benefits.
>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft: 
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
> ___
> coreboot mailing list -- coreboot@coreboot.org
> To unsubscribe send an email to coreboot-le...@coreboot.org
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Thu, Sep 30, 2021 at 8:37 AM Nico Huber  wrote:

> Hi Jack,
>
> On 30.09.21 15:22, Jack Rosenthal via coreboot wrote:
> > On Thu, Sep 30, 2021 at 2:27 AM Arthur Heymans 
> wrote:
> >
> >> As a rule of thumb, any project involving a substantial amount of Python
> >>> always ends up needing a Docker container to build. So I'm in the "no"
> camp
> >>> for making Python a dependency, however I think it's fine to keep
> things
> >>> as-is where it can be used for helper scripts and utilities for
> specific
> >>> purposes such that they aren't critical to building the tree.
> >>>
> >>
> >> I'm on the same side here. Building the documentation with python sphinx
> >> is a pain and I ended up needing docker.
> >> The same can be said about edk2/tianocore which also uses a lot of
> python
> >> in critical parts of its build system.
> >>
> >
> > For just Kconfiglib, the requirements are Python 2.7 or Python 3.2+,
> > including no additional library installation (standard libraries only).
> > Using docker in this scenario would be completely ridiculous.
>
> please keep in mind that this thread tries to discuss a general python
> requirement. Not Kconfiglib in particular. I suppose nobody doubts that
> Kconfiglib is one of the better/easier to integrate python projects.
>
> IMHO, we should discuss every added dependency separately. E.g. not just
> all of python at once but every piece of code that we can't fix in our
> own repository. I guess, in the case of Kconfiglib that would be:
>

Agreed. Some sort of specification of what the dependency is we are adding
is absolutely required.

As an example, we could say "Python 3.6 or newer, no additional libraries
required". (this is just an example, the actual version numbers etc we
settle on should be discussed).

I bet most distributions are shipping with 3.6+ these days too...


>
> * python (the interpreter)
>   * std libs (I guess part of the above)
> * Kconfiglib itself
>   unless we pull it into our repo like we did with Kconfig
>

I believe the previous discussion around Kconfiglib was to pull it into the
coreboot repo directly so we don't need to make it a dependency for people.
We don't need to clone the entire repository into our repos, everything is
pretty much self-contained to kconfiglib.py, as well as any UIs we want to
enable (e.g., menuconfig.py and guiconfig.py).


> >
> > Looking at the current Kconfiglib implementation we would be replacing
> the
> >> C code with 21873 lines of Python code that is now taking the code to
> >> deviate from what the Linux kernel is doing. I am having a hard time
> seeing
> >> a "net benefit" in this scenario. Given the mess that Python 2 to
> Python 3
> >> conversion has been (and still is), this is just inviting a lot of
> trouble
> >> into what has been a fairly stable part of coreboot for the last decade.
> >>
> >
> > 21,873 lines of code is including the tests. Wouldn't it be nice if the C
> > Kconfig implementation even had tests? :P
>
> Not sure, I never looked close enough at the code to tell if it's bad
> enough to justify the overhead of tests.
>
> >
> > IMO, any codebase is significantly easier and safer to maintain if there
> > are tests.
>
> In my experience that is only true for projects that don't focus enough
> on code reviews and refactoring. I often see the necessity of tests as
> lack of review. In some projects tests are even added to provide false
> confidence that one would need less review. IMO, the opposite is the
> case. Tests encourage less experienced developers to push patches with-
> out looking much into the code, because all tests are green, right? In
> such a scenario the effort to integrate a patch shifts from authors to
> reviewers, IMO not good.
>
> Nico
>

Tests and code review serve two separate functions: tests allowing
automatic validation that a change won't break everything, and code review
to keep the code-quality high of the code that lands. Obviously, as you
point out, there is some overlap, and tests cannot ever be 100% perfect.
But enough with the philosophy.

With respect to Kconfig, we (at Google) encountered a lovely build flake
after the Kconfig uprev last month in the coreboot tree that took a couple
of weeks to sort out and resolve. Some sort of automated validation that
the code is working could have possibly helped. Of course, the C
implementation of Kconfig has no tests at all. Some tests is better than
nothing.


-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Am Do., 30. Sept. 2021 um 17:29 Uhr schrieb Jack Rosenthal <
jrose...@google.com>:

> With respect to Kconfig, we (at Google) encountered a lovely build flake
> after the Kconfig uprev last month in the coreboot tree that took a couple
> of weeks to sort out and resolve. Some sort of automated validation that
> the code is working could have possibly helped. Of course, the C
> implementation of Kconfig has no tests at all. Some tests is better than
> nothing.
>

Let me put the record straight:

- The last kconfig "uprev" hasn't been a simple update the way
https://review.coreboot.org/c/coreboot/+/57880 is, but rebuilt the entire
build system integration to ease maintenance
- That issue sprang up because before the kconfig update, we were shipping
prebuilt parser files (C code) while now we made bison and flex hard
dependencies for our build
- Tests covering the C code wouldn't have helped, because the issue wasn't
in the C code
- The issue we were facing has been an external dependency (namely: the
Chromium OS development environment shipping a broken version of bison(1))
- Fixing bison in CrOS wouldn't have helped any because we have to assume
that other users come with the same kind of broken bison tool
- The fix has been to ship pre-built files again to remove an external
dependency

The alternative that we did actually consider was to add support for
building bison and flex to util/crossgcc/buildgcc. For three files that's
sheer madness, so we went back to precompiled files instead.

In relation to your proposal to adopt kconfiglib: We can run into any kind
of external tool demonstrating weird behavior. That's true for bison (as
seen here) just as it can be true for arbitrary python versions (even when
specified to be "python 3.6+" or whatever): Linux distributions do strange
things to their packages, and we're not a Linux-only project, so even
official, unchanged, straight-from-the-server python might behave
unexpectedly on less well exercised platforms.

The best way to reduce issues on that end is to avoid external dependencies
- like bison, like flex... like python.
I'd _love_ to avoid the dependency on the host compiler as well, but that's
one of those "sheer madness" moments when you support a multitude of
operating systems on as many architectures.

Introducing kconfiglib (and through it, a deep reliance on python) just
doesn't seem to provide comparable benefits.


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Nico Huber]

Hi Jack,

On 30.09.21 15:22, Jack Rosenthal via coreboot wrote:
> On Thu, Sep 30, 2021 at 2:27 AM Arthur Heymans  wrote:
>
>> As a rule of thumb, any project involving a substantial amount of Python
>>> always ends up needing a Docker container to build. So I'm in the "no" camp
>>> for making Python a dependency, however I think it's fine to keep things
>>> as-is where it can be used for helper scripts and utilities for specific
>>> purposes such that they aren't critical to building the tree.
>>>
>>
>> I'm on the same side here. Building the documentation with python sphinx
>> is a pain and I ended up needing docker.
>> The same can be said about edk2/tianocore which also uses a lot of python
>> in critical parts of its build system.
>>
>
> For just Kconfiglib, the requirements are Python 2.7 or Python 3.2+,
> including no additional library installation (standard libraries only).
> Using docker in this scenario would be completely ridiculous.

please keep in mind that this thread tries to discuss a general python
requirement. Not Kconfiglib in particular. I suppose nobody doubts that
Kconfiglib is one of the better/easier to integrate python projects.

IMHO, we should discuss every added dependency separately. E.g. not just
all of python at once but every piece of code that we can't fix in our
own repository. I guess, in the case of Kconfiglib that would be:

* python (the interpreter)
  * std libs (I guess part of the above)
* Kconfiglib itself
  unless we pull it into our repo like we did with Kconfig

If we decided now that we'd allow the above that shouldn't mean that
we can add any python related dependency in the future without further
discussion.

For any piece of code that we can't fix in our own repository, we should
have people that know the respective upstreaming process and can react
rather quickly to any issue, IMO.

>
> Looking at the current Kconfiglib implementation we would be replacing the
>> C code with 21873 lines of Python code that is now taking the code to
>> deviate from what the Linux kernel is doing. I am having a hard time seeing
>> a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
>> conversion has been (and still is), this is just inviting a lot of trouble
>> into what has been a fairly stable part of coreboot for the last decade.
>>
>
> 21,873 lines of code is including the tests. Wouldn't it be nice if the C
> Kconfig implementation even had tests? :P

Not sure, I never looked close enough at the code to tell if it's bad
enough to justify the overhead of tests.

>
> IMO, any codebase is significantly easier and safer to maintain if there
> are tests.

In my experience that is only true for projects that don't focus enough
on code reviews and refactoring. I often see the necessity of tests as
lack of review. In some projects tests are even added to provide false
confidence that one would need less review. IMO, the opposite is the
case. Tests encourage less experienced developers to push patches with-
out looking much into the code, because all tests are green, right? In
such a scenario the effort to integrate a patch shifts from authors to
reviewers, IMO not good.

Nico
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Thu, Sep 30, 2021 at 7:43 AM Patrick Georgi  wrote:

> Am Do., 30. Sept. 2021 um 15:22 Uhr schrieb Jack Rosenthal <
> jrose...@google.com>:
>
>> IMO, any codebase is significantly easier and safer to maintain if there
>> are tests.
>>
> Since we kinda-sorta support SPARK in our toolchain (not for the host
> though at this time), maybe we should evaluate doing a rewrite in SPARK
> then: Formal verification beats spot tests which may or may not cover the
> troublesome situations. ;-)
> (What I mean is that it's always possible to up the ante, not that we
> should actually do that. I'd rather get rid of the kconfig language, and
> even that idea has dubious value in this context.)
>

Yeah. Let's stay practical. Removing the Kconfig language entirely from the
tree wouldn't exactly be feasible.

 Both the C Kconfig and the Kconfiglib implementations are implementations
that exist today, let's evaluate which one of them is better from a
maintenance & stability perspective.


>
> As for the idea of a Python 4 you seem to have here (or if it does come,
>> repeating the massive language differences we had between 2 and 3), it's
>> unlikely to happen. Guido says that a "Python 4" at the scale Python 3 was
>> is unlikely to happen
>> 
>> .
>>
> If Python 3->4 is only half as painful as 2->3, this would still be true -
> and still mean misery for 5+ years. No thanks.
>

I don't even think even half is in domain... if you give the article to
read, Guido basically says we might just stay on Python 3 forever.


>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>


-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Am Do., 30. Sept. 2021 um 15:22 Uhr schrieb Jack Rosenthal <
jrose...@google.com>:

> IMO, any codebase is significantly easier and safer to maintain if there
> are tests.
>
Since we kinda-sorta support SPARK in our toolchain (not for the host
though at this time), maybe we should evaluate doing a rewrite in SPARK
then: Formal verification beats spot tests which may or may not cover the
troublesome situations. ;-)
(What I mean is that it's always possible to up the ante, not that we
should actually do that. I'd rather get rid of the kconfig language, and
even that idea has dubious value in this context.)

As for the idea of a Python 4 you seem to have here (or if it does come,
> repeating the massive language differences we had between 2 and 3), it's
> unlikely to happen. Guido says that a "Python 4" at the scale Python 3 was
> is unlikely to happen
> 
> .
>
If Python 3->4 is only half as painful as 2->3, this would still be true -
and still mean misery for 5+ years. No thanks.


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Thu, Sep 30, 2021 at 2:27 AM Arthur Heymans  wrote:

> As a rule of thumb, any project involving a substantial amount of Python
>> always ends up needing a Docker container to build. So I'm in the "no" camp
>> for making Python a dependency, however I think it's fine to keep things
>> as-is where it can be used for helper scripts and utilities for specific
>> purposes such that they aren't critical to building the tree.
>>
>
> I'm on the same side here. Building the documentation with python sphinx
> is a pain and I ended up needing docker.
> The same can be said about edk2/tianocore which also uses a lot of python
> in critical parts of its build system.
>

For just Kconfiglib, the requirements are Python 2.7 or Python 3.2+,
including no additional library installation (standard libraries only).
Using docker in this scenario would be completely ridiculous.

Looking at the current Kconfiglib implementation we would be replacing the
> C code with 21873 lines of Python code that is now taking the code to
> deviate from what the Linux kernel is doing. I am having a hard time seeing
> a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
> conversion has been (and still is), this is just inviting a lot of trouble
> into what has been a fairly stable part of coreboot for the last decade.
>

21,873 lines of code is including the tests. Wouldn't it be nice if the C
Kconfig implementation even had tests? :P

IMO, any codebase is significantly easier and safer to maintain if there
are tests.


>
> In terms of Kconfiglib's stability and track record: I think it has it
>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>> without any issues at all.
>>
>
> I am failing to see how anybody involved in coreboot would sign up for and
> commit to porting 20k lines of Python code to the next version, when it
> arrives. My indication is that not even the python code that is currently
> in the tree has been ported to python3 yet.
>

As for the idea of a Python 4 you seem to have here (or if it does come,
repeating the massive language differences we had between 2 and 3), it's
unlikely to happen. Guido says that a "Python 4" at the scale Python 3 was
is unlikely to happen

.

-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Arthur Heymans]

>
> As a rule of thumb, any project involving a substantial amount of Python
> always ends up needing a Docker container to build. So I'm in the "no" camp
> for making Python a dependency, however I think it's fine to keep things
> as-is where it can be used for helper scripts and utilities for specific
> purposes such that they aren't critical to building the tree.
>

I'm on the same side here. Building the documentation with python sphinx is
a pain and I ended up needing docker.
The same can be said about edk2/tianocore which also uses a lot of python
in critical parts of its build system.

At a minimum, I think we should consider introducing Python on an optional
> basis (i.e., the C Kconfig implementation only gets used if a Python
> interpreter is unavailable), but making it required would be even better.
>

No. I don't like 'optional' things which do more or less the same as
existing code. It's a great way to make the maintenance cost explode.
This is true in general (not just python tooling): adding alternative
codepaths to do more or less the same as existing code creates a
combinatorial explosion.
We have (had) 2 different bootblocks codepathds, multiple different cache
as ram teardown mechanism, 2 resource different allocators, 2 MP init
codepaths, 3 SMM init codepaths, ...
Tooling is not the place you want to add new optional codepaths (maybe
toolchains can be an exception as different compilers catch different
errors which improves code quality).
'Optional' things under the pretext to not break existing things looks like
a nice idea at first sight, but it's a very bad strategy in the long run.
I'd much rather stick to 1 thing, even
if it breaks the master branch, quickly fix things that broke and move on.
I don't want to hijack the thread which is about python, but 'optional'
python is worse than both no
python or mandatory python.

On Thu, Sep 30, 2021 at 7:01 AM David Hendricks 
wrote:

> As a rule of thumb, any project involving a substantial amount of Python
> always ends up needing a Docker container to build. So I'm in the "no" camp
> for making Python a dependency, however I think it's fine to keep things
> as-is where it can be used for helper scripts and utilities for specific
> purposes such that they aren't critical to building the tree.
>
> On Wed, Sep 29, 2021 at 2:58 AM Patrick Georgi via coreboot <
> coreboot@coreboot.org> wrote:
>
>> That said, python makes its way back into the tree every now and then
>> (typically as small snippets to compute and add hashes to binaries as
>> needed by ARM SoCs). Uncanny, but typically not a big deal.
>>
> ...
>> To avoid these scenarios, could we possibly nail down the policy on
>> python in coreboot?
>>
>
> The policy should be simple: The CI system (Jenkins) must be able to build
> every target in its default configuration.
>
> If we introduce Python as a dependency, then all Python in the tree must
> be compatible with whatever version Jenkins uses. And if we're going to
> impose the burden of fixing Python on everyone, then all developers must
> have the ability to install a compatible version in their OS. Given the
> experiences many of us in this thread have had and how widely distros vary
> in Python support, I don't see this as tenable.
>
> Another thing to keep in mind is that we have these sorts of helper
> scripts from multiple vendors/parties over several years, and we'll likely
> see more in the future. Pushing them all to use whatever version(s) of
> Python we decide to build with does not seem realistic.
>
> All that said, I'm fine with Python being used for helper scripts and such
> as we've done in the past. It gives developers/vendors/etc. freedom to use
> whatever works for their purposes without imposing a huge burden on
> everyone else.
> ___
> coreboot mailing list -- coreboot@coreboot.org
> To unsubscribe send an email to coreboot-le...@coreboot.org
>
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[David Hendricks]

As a rule of thumb, any project involving a substantial amount of Python
always ends up needing a Docker container to build. So I'm in the "no" camp
for making Python a dependency, however I think it's fine to keep things
as-is where it can be used for helper scripts and utilities for specific
purposes such that they aren't critical to building the tree.

On Wed, Sep 29, 2021 at 2:58 AM Patrick Georgi via coreboot <
coreboot@coreboot.org> wrote:

> That said, python makes its way back into the tree every now and then
> (typically as small snippets to compute and add hashes to binaries as
> needed by ARM SoCs). Uncanny, but typically not a big deal.
>
...
> To avoid these scenarios, could we possibly nail down the policy on python
> in coreboot?
>

The policy should be simple: The CI system (Jenkins) must be able to build
every target in its default configuration.

If we introduce Python as a dependency, then all Python in the tree must be
compatible with whatever version Jenkins uses. And if we're going to impose
the burden of fixing Python on everyone, then all developers must have the
ability to install a compatible version in their OS. Given the experiences
many of us in this thread have had and how widely distros vary in Python
support, I don't see this as tenable.

Another thing to keep in mind is that we have these sorts of helper scripts
from multiple vendors/parties over several years, and we'll likely see more
in the future. Pushing them all to use whatever version(s) of Python we
decide to build with does not seem realistic.

All that said, I'm fine with Python being used for helper scripts and such
as we've done in the past. It gives developers/vendors/etc. freedom to use
whatever works for their purposes without imposing a huge burden on
everyone else.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Peter Stuge]

Thanks for bringing this to the list, Patrick.

Stefan Reinauer wrote:
> Given the mess that Python 2 to Python 3 conversion has been (and
> still is), this is just inviting a lot of trouble into what has
> been a fairly stable part of coreboot for the last decade.

I strongly agree.

On a more general note I find it super sad that distributions purge
Python 2 only because it is EOL upstream when it still has many use
cases. It's not a problem for me but it does demonstrate how little
critical thinking happens within distributions.

So projects must be extra critical about dependencies.


I prefer changing nothing (change != progress) or banning Python
for the coreboot build. Utilities (and payloads) not needed for the
actual coreboot build can continue to use it.

openssl + dd could probably calculate and inject hashes.


Thanks

//Peter
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Matt B]

> What coreboot problems that we have seen in the past are we actually
solving with these rewrites?
To be a bit more blunt, what is expected to be improved by writing it in
python?
Utilities, eg. to analyse blobs?
Menu and configuration of builds?
Packing blobs for flashing?

I could see some things being harmless, like using python to drive menus
and check dependencies. I could potentially see a case that it's preferable
to complicated #ifdef logic and for handling of board variants.

On the other hand, I could definitely see python adding new ways that
reproducible builds fail, unless you compile a specific python interpreter
on the spot. Who knows what subtle differences there might be in python's
handling of binary data, especially on different arches and systems with
different endianness? Maybe I'm paranoid, or maybe this stuff could be
caught well enough in automated testing (and $someone could fix it).

Perhaps user-driven utils like those that extract blobs or examine MSRs to
discover interrupt routing on proprietary firmware are a safer middle
ground between the two extremes, but those are already written in C aren't
they?

>Python is already being used, in the "util" directory. util/qualcomm has
~3500 LoC in Python [1] (this is not just a "small snippet" of code).
How was it introduced? Was this from a vendor dump? Those seem to
occasionally bend the rules of what would be "ideal."

-Matt


On Wed, Sep 29, 2021 at 5:15 PM Ricardo Quesada via coreboot <
coreboot@coreboot.org> wrote:

> Thanks Patrick for bringing this in.
>
> Regarding Pytest for utils (https://review.coreboot.org/c/coreboot/+/57869
> ), what's the recommended way to test the tools that are placed in "util/*"
> ?
> I noticed that most of them don't have any kind of end-to-end test (I
> couldn't find a single one, but perhaps I missed them).
>
> Using PyTest (or any other easy to install / easy to use framework) to
> create tests for util/* seems natural to me.
> Although any other test framework can be used. Suggestions?
>
> Besides all the things that have been said in favor or against Python, I'd
> like to add that:
>Python is already being used, in the "util" directory. util/qualcomm
> has ~3500 LoC in Python [1] (this is not just a "small snippet" of code).
>(compare it with PyTests for elogutil[2]: It has ~160 LoC ).
>
> [1]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm
> [2]: https://review.coreboot.org/c/coreboot/+/57869
>
>
> Thanks.
>
>
> On Wed, Sep 29, 2021 at 12:09 PM Stefan Reinauer <
> stefan.k.reina...@gmail.com> wrote:
>
>> On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal 
>> wrote:
>>
>>> Overall I think introducing Python to the build would provide net
>>> benefit, mainly from Kconfiglib, but could also find other good uses in e2e
>>> tests like Ricardo was working on. Most people's Linux distros ship with a
>>> Python interpreter too, so most developers would be unlikely to notice the
>>> extra dependency introduction.
>>>
>>> In terms of Kconfiglib, we have a lot to gain by switching away from the
>>> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
>>> forked from the Linux tree and hacked in our own customizations
>>> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
>>> a miniature Python script
>>> 
>>> that we use to handle our custom header format, and a stable API to work
>>> off of so that we can uprev Kconfiglib without needing to change our
>>> scripts.
>>>
>>
>> Looking at the current Kconfiglib implementation we would be replacing
>> the C code with 21873 lines of Python code that is now taking the code to
>> deviate from what the Linux kernel is doing. I am having a hard time seeing
>> a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
>> conversion has been (and still is), this is just inviting a lot of trouble
>> into what has been a fairly stable part of coreboot for the last decade.
>>
>> In terms of Kconfiglib's stability and track record: I think it has it
>>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>>> without any issues at all.
>>>
>>
>> I am failing to see how anybody involved in coreboot would sign up for
>> and commit to porting 20k lines of Python code to the next version, when it
>> arrives. My indication is that not even the python code that is currently
>> in the tree has been ported to python3 yet.
>>
>>
>>> At a minimum, I think we should consider introducing Python on an
>>> optional basis (i.e., the C Kconfig implementation only gets used if a
>>> Python interpreter is unavailable), but making it required would be even
>>> better.
>>>
>>
>> Come on, we do not need two types of Kconfig parsers in the tree. Let's
>> focus coreboot on actually booting cores, not collections of
>> implementations in different programming languages. There are better

[coreboot] Re: There is a python in our toolchain?!?

[Ricardo Quesada via coreboot]

Thanks Patrick for bringing this in.

Regarding Pytest for utils (https://review.coreboot.org/c/coreboot/+/57869
), what's the recommended way to test the tools that are placed in "util/*"
?
I noticed that most of them don't have any kind of end-to-end test (I
couldn't find a single one, but perhaps I missed them).

Using PyTest (or any other easy to install / easy to use framework) to
create tests for util/* seems natural to me.
Although any other test framework can be used. Suggestions?

Besides all the things that have been said in favor or against Python, I'd
like to add that:
   Python is already being used, in the "util" directory. util/qualcomm has
~3500 LoC in Python [1] (this is not just a "small snippet" of code).
   (compare it with PyTests for elogutil[2]: It has ~160 LoC ).

[1]: https://github.com/coreboot/coreboot/tree/master/util/qualcomm
[2]: https://review.coreboot.org/c/coreboot/+/57869


Thanks.


On Wed, Sep 29, 2021 at 12:09 PM Stefan Reinauer <
stefan.k.reina...@gmail.com> wrote:

> On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal 
> wrote:
>
>> Overall I think introducing Python to the build would provide net
>> benefit, mainly from Kconfiglib, but could also find other good uses in e2e
>> tests like Ricardo was working on. Most people's Linux distros ship with a
>> Python interpreter too, so most developers would be unlikely to notice the
>> extra dependency introduction.
>>
>> In terms of Kconfiglib, we have a lot to gain by switching away from the
>> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
>> forked from the Linux tree and hacked in our own customizations
>> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
>> a miniature Python script
>> 
>> that we use to handle our custom header format, and a stable API to work
>> off of so that we can uprev Kconfiglib without needing to change our
>> scripts.
>>
>
> Looking at the current Kconfiglib implementation we would be replacing the
> C code with 21873 lines of Python code that is now taking the code to
> deviate from what the Linux kernel is doing. I am having a hard time seeing
> a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
> conversion has been (and still is), this is just inviting a lot of trouble
> into what has been a fairly stable part of coreboot for the last decade.
>
> In terms of Kconfiglib's stability and track record: I think it has it
>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>> without any issues at all.
>>
>
> I am failing to see how anybody involved in coreboot would sign up for and
> commit to porting 20k lines of Python code to the next version, when it
> arrives. My indication is that not even the python code that is currently
> in the tree has been ported to python3 yet.
>
>
>> At a minimum, I think we should consider introducing Python on an
>> optional basis (i.e., the C Kconfig implementation only gets used if a
>> Python interpreter is unavailable), but making it required would be even
>> better.
>>
>
> Come on, we do not need two types of Kconfig parsers in the tree. Let's
> focus coreboot on actually booting cores, not collections of
> implementations in different programming languages. There are better
> projects for that, such as https://github.com/mame/quine-relay
>
> What coreboot problems that we have seen in the past are we actually
> solving with these rewrites?
>
> Stefan
>
>
>>
>> On Wed, Sep 29, 2021 at 5:39 AM Rao G  wrote:
>>
>>> Hi Patrick,
>>>
>>> That's good to hear, would there be change to "make menuconfig" with
>>> kconfiglib
>>>
>>
>> No, we'd make it so that all the "make *config" commands run the
>> Kconfiglib alternatives without any user action required.
>>
>> The "menuconfig" interface in specific is very similar to the
>> lxdialog-based interface that the C Kconfig uses, except it's a bit more
>> polished and refined feeling.
>>
>>
>>>
>>> Thanks
>>> Ranga
>>>
>>> On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
>>> coreboot@coreboot.org> wrote:
>>>
 Hi everybody,

 Historically, coreboot avoided depending on python too much (we got rid
 of an entire python based configuration and build system, for example),
 with few minor exceptions.

 The main reason has been that while python code is quick to slap
 together, it has demonstrated a penchant for breaking in all kinds of
 mysterious ways (python 2->3 really was just a slightly bigger instance of
 what's going on in python all the time), and its users demonstrate a
 disregard for their fellow developers as demonstrated by endless stack
 traces on trivial errors (or is the language too complicated to properly
 catch them all?)

 While probably nice for one-off prototypes, long term maintenance is a
 concern: this project has over 20 ye

[coreboot] Re: There is a python in our toolchain?!?

[Stefan Reinauer]

On Wed, Sep 29, 2021 at 7:44 AM Jack Rosenthal  wrote:

> Overall I think introducing Python to the build would provide net benefit,
> mainly from Kconfiglib, but could also find other good uses in e2e tests
> like Ricardo was working on. Most people's Linux distros ship with a Python
> interpreter too, so most developers would be unlikely to notice the extra
> dependency introduction.
>
> In terms of Kconfiglib, we have a lot to gain by switching away from the
> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
> forked from the Linux tree and hacked in our own customizations
> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
> a miniature Python script
> 
> that we use to handle our custom header format, and a stable API to work
> off of so that we can uprev Kconfiglib without needing to change our
> scripts.
>

Looking at the current Kconfiglib implementation we would be replacing the
C code with 21873 lines of Python code that is now taking the code to
deviate from what the Linux kernel is doing. I am having a hard time seeing
a "net benefit" in this scenario. Given the mess that Python 2 to Python 3
conversion has been (and still is), this is just inviting a lot of trouble
into what has been a fairly stable part of coreboot for the last decade.

In terms of Kconfiglib's stability and track record: I think it has it
> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
> without any issues at all.
>

I am failing to see how anybody involved in coreboot would sign up for and
commit to porting 20k lines of Python code to the next version, when it
arrives. My indication is that not even the python code that is currently
in the tree has been ported to python3 yet.


> At a minimum, I think we should consider introducing Python on an optional
> basis (i.e., the C Kconfig implementation only gets used if a Python
> interpreter is unavailable), but making it required would be even better.
>

Come on, we do not need two types of Kconfig parsers in the tree. Let's
focus coreboot on actually booting cores, not collections of
implementations in different programming languages. There are better
projects for that, such as https://github.com/mame/quine-relay

What coreboot problems that we have seen in the past are we actually
solving with these rewrites?

Stefan


>
> On Wed, Sep 29, 2021 at 5:39 AM Rao G  wrote:
>
>> Hi Patrick,
>>
>> That's good to hear, would there be change to "make menuconfig" with
>> kconfiglib
>>
>
> No, we'd make it so that all the "make *config" commands run the
> Kconfiglib alternatives without any user action required.
>
> The "menuconfig" interface in specific is very similar to the
> lxdialog-based interface that the C Kconfig uses, except it's a bit more
> polished and refined feeling.
>
>
>>
>> Thanks
>> Ranga
>>
>> On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
>> coreboot@coreboot.org> wrote:
>>
>>> Hi everybody,
>>>
>>> Historically, coreboot avoided depending on python too much (we got rid
>>> of an entire python based configuration and build system, for example),
>>> with few minor exceptions.
>>>
>>> The main reason has been that while python code is quick to slap
>>> together, it has demonstrated a penchant for breaking in all kinds of
>>> mysterious ways (python 2->3 really was just a slightly bigger instance of
>>> what's going on in python all the time), and its users demonstrate a
>>> disregard for their fellow developers as demonstrated by endless stack
>>> traces on trivial errors (or is the language too complicated to properly
>>> catch them all?)
>>>
>>> While probably nice for one-off prototypes, long term maintenance is a
>>> concern: this project has over 20 years of history under its belt, with
>>> more to come.
>>>
>>> That said, python makes its way back into the tree every now and then
>>> (typically as small snippets to compute and add hashes to binaries as
>>> needed by ARM SoCs). Uncanny, but typically not a big deal.
>>>
>>> There are two bigger initiatives proposed that would significantly
>>> increase our python footprint:
>>> 1. Replacing Linux's kconfig with kconfiglib (
>>> https://review.coreboot.org/c/coreboot/+/48679)
>>> 2. Using pytest for end-to-end testing utilities (
>>> https://review.coreboot.org/c/coreboot/+/57869)
>>>
>>> Compared to the "inject a hash value at a fixed location" scripts, these
>>> would probably be here to stay, and sufficiently integrated that everybody
>>> will have to deal with them.
>>>
>>> People spending time working on python code when it has no chance to
>>> land isn't a good use of their time and we should avoid that in the project.
>>>
>>> People spending time arguing that python shouldn't be used (to avoid
>>> the other outcome) even though the project's culture shifted and is now
>>> accepting Python isn't creating a grea

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Am Mi., 29. Sept. 2021 um 19:47 Uhr schrieb Jack Rosenthal <
jrose...@google.com>:

> At a minimum, I think we should consider introducing Python on an optional
>>> basis (i.e., the C Kconfig implementation only gets used if a Python
>>> interpreter is unavailable), but making it required would be even better.
>>>
>> That idea is... horrible. Instead of only bringing in the python
>> dependency we'd also create an environmental dependency that can silently
>> introduce behavioral differences.
>>
> Fair, require python then. Most people have it on their systems anyways.
>
Well, the question I put up on the mailing list, for the community to
decide, is if we should lift our quasi-ban on python in the project. I
added you and the other contributors proposing python changes in Cc to this
email as a courtesy because you're not all registered on the project's
discussion venue.

The default alternative to "let's do something horrible" isn't "let's do
this other thing instead", it's "don't change anything at all". Don't look
further than the requirement for proposed changes to make it through
review: Without "don't change anything" as default, we could just open up
write access to the main repo and branch for everybody and everything - we
don't.

Only if we decide that extensive python use in the coreboot tree is now
okay, kconfiglib is even eligible for discussion.

So far the only argument in favor of python in _any_ way has been "it's
installed everywhere". That was true 20 years ago as well, and yet still we
had a fair amount of trouble with python. Look no further than the recent
uptick in support requests on IRC by people trying to build coreboot +
"stable" Tianocore: It fails because python moved on while the tagged
Tianocore version did not and its build tooling relies on python. So I
agree that "installed everywhere" is a necessary condition, but given how
badly python's maintainers historically treated their user base, I don't
think it's a sufficient condition, by far.

tl;dr: people of coreboot, opinions?


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

On Wed, Sep 29, 2021 at 9:57 AM Patrick Georgi  wrote:

> Am Mi., 29. Sept. 2021 um 16:43 Uhr schrieb Jack Rosenthal <
> jrose...@google.com>:
>
>> Overall I think introducing Python to the build would provide net
>> benefit, mainly from Kconfiglib, but could also find other good uses in e2e
>> tests like Ricardo was working on. Most people's Linux distros ship with a
>> Python interpreter too, so most developers would be unlikely to notice the
>> extra dependency introduction.
>>
> This is assuming that all python environments are alike. Experience
> disagrees.
>

Kconfiglib only uses Python standard libraries, and works with both Python
3.3+ and 2.7. I think it'd be pretty hard to find a Python environment it
didn't work with.


>
>
>> In terms of Kconfiglib, we have a lot to gain by switching away from the
>> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
>> forked from the Linux tree and hacked in our own customizations
>> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
>> a miniature Python script
>> 
>> that we use to handle our custom header format, and a stable API to work
>> off of so that we can uprev Kconfiglib without needing to change our
>> scripts.
>>
> The customizations are a set of patches with simple maintenance (as
> documented in https://review.coreboot.org/c/coreboot/+/57879).
>

This is true, but the maintenance burden could be even less.

Additionally, the last Kconfig uprev didn't go so well, at least from the
Chrome OS perspective, we saw a build flake that sat around for a few weeks
bothering people.


>
> In terms of Kconfiglib's stability and track record: I think it has it
>> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
>> without any issues at all.
>>
> This project deals in 20 year timelines. Zephyr is at 5.5 years, while
> depthcharge is used exclusively in a tightly controlled environment (and
> even there we had - and have - our share of pythonic problems).
>

IMO 5.5 years of stability is a pretty good point to start considering
adopting a new technology. If we wait for 20 years of it, there'll be
something better than Kconfiglib out there ;)


>
> At a minimum, I think we should consider introducing Python on an optional
>> basis (i.e., the C Kconfig implementation only gets used if a Python
>> interpreter is unavailable), but making it required would be even better.
>>
> That idea is... horrible. Instead of only bringing in the python
> dependency we'd also create an environmental dependency that can silently
> introduce behavioral differences.
>
> This project has the ability of building bit-identical firmware images on
> hosts spanning all kinds of CPU register sizes, endianness and operating
> system families. If you ever heard of "hermetic builds" being a good idea:
> the trustworthiness of our build is stronger - and part of making that
> possible is being (relatively) careful with picking our dependencies.
>

Fair, require python then. Most people have it on their systems anyways.


>
>
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>


-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Jack Rosenthal via coreboot]

Overall I think introducing Python to the build would provide net benefit,
mainly from Kconfiglib, but could also find other good uses in e2e tests
like Ricardo was working on. Most people's Linux distros ship with a Python
interpreter too, so most developers would be unlikely to notice the extra
dependency introduction.

In terms of Kconfiglib, we have a lot to gain by switching away from the
Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
forked from the Linux tree and hacked in our own customizations
(KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
a miniature Python script

that we use to handle our custom header format, and a stable API to work
off of so that we can uprev Kconfiglib without needing to change our
scripts.

In terms of Kconfiglib's stability and track record: I think it has it
covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
without any issues at all.

At a minimum, I think we should consider introducing Python on an optional
basis (i.e., the C Kconfig implementation only gets used if a Python
interpreter is unavailable), but making it required would be even better.

On Wed, Sep 29, 2021 at 5:39 AM Rao G  wrote:

> Hi Patrick,
>
> That's good to hear, would there be change to "make menuconfig" with
> kconfiglib
>

No, we'd make it so that all the "make *config" commands run the Kconfiglib
alternatives without any user action required.

The "menuconfig" interface in specific is very similar to the
lxdialog-based interface that the C Kconfig uses, except it's a bit more
polished and refined feeling.


>
> Thanks
> Ranga
>
> On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
> coreboot@coreboot.org> wrote:
>
>> Hi everybody,
>>
>> Historically, coreboot avoided depending on python too much (we got rid
>> of an entire python based configuration and build system, for example),
>> with few minor exceptions.
>>
>> The main reason has been that while python code is quick to slap
>> together, it has demonstrated a penchant for breaking in all kinds of
>> mysterious ways (python 2->3 really was just a slightly bigger instance of
>> what's going on in python all the time), and its users demonstrate a
>> disregard for their fellow developers as demonstrated by endless stack
>> traces on trivial errors (or is the language too complicated to properly
>> catch them all?)
>>
>> While probably nice for one-off prototypes, long term maintenance is a
>> concern: this project has over 20 years of history under its belt, with
>> more to come.
>>
>> That said, python makes its way back into the tree every now and then
>> (typically as small snippets to compute and add hashes to binaries as
>> needed by ARM SoCs). Uncanny, but typically not a big deal.
>>
>> There are two bigger initiatives proposed that would significantly
>> increase our python footprint:
>> 1. Replacing Linux's kconfig with kconfiglib (
>> https://review.coreboot.org/c/coreboot/+/48679)
>> 2. Using pytest for end-to-end testing utilities (
>> https://review.coreboot.org/c/coreboot/+/57869)
>>
>> Compared to the "inject a hash value at a fixed location" scripts, these
>> would probably be here to stay, and sufficiently integrated that everybody
>> will have to deal with them.
>>
>> People spending time working on python code when it has no chance to land
>> isn't a good use of their time and we should avoid that in the project.
>>
>> People spending time arguing that python shouldn't be used (to avoid
>> the other outcome) even though the project's culture shifted and is now
>> accepting Python isn't creating a great community for anybody.
>>
>> To avoid these scenarios, could we possibly nail down the policy on
>> python in coreboot?
>>
>>
>> Regards,
>> Patrick
>> --
>> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
>> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
>> Hamburg
>> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>> ___
>> coreboot mailing list -- coreboot@coreboot.org
>> To unsubscribe send an email to coreboot-le...@coreboot.org
>>
>

-- 

Jack Rosenthal (he/him/his)

Software Engineer - Chrome OS

Google Boulder

jrose...@google.com

I value feedback from others. Please feel free to contact me directly, or
file it anonymously at go/jrosenth-feedback
.
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Patrick Georgi via coreboot]

Am Mi., 29. Sept. 2021 um 16:43 Uhr schrieb Jack Rosenthal <
jrose...@google.com>:

> Overall I think introducing Python to the build would provide net benefit,
> mainly from Kconfiglib, but could also find other good uses in e2e tests
> like Ricardo was working on. Most people's Linux distros ship with a Python
> interpreter too, so most developers would be unlikely to notice the extra
> dependency introduction.
>
This is assuming that all python environments are alike. Experience
disagrees.


> In terms of Kconfiglib, we have a lot to gain by switching away from the
> Linux C implementation of Kconfig, mainly the ~30kloc of C code that we've
> forked from the Linux tree and hacked in our own customizations
> (KCONFIG_NEGATIVES). With Kconfiglib, these customizations get turned into
> a miniature Python script
> 
> that we use to handle our custom header format, and a stable API to work
> off of so that we can uprev Kconfiglib without needing to change our
> scripts.
>
The customizations are a set of patches with simple maintenance (as
documented in https://review.coreboot.org/c/coreboot/+/57879).

In terms of Kconfiglib's stability and track record: I think it has it
> covered. We adopted Kconfiglib in both Zephyr OS and in Depthcharge already
> without any issues at all.
>
This project deals in 20 year timelines. Zephyr is at 5.5 years, while
depthcharge is used exclusively in a tightly controlled environment (and
even there we had - and have - our share of pythonic problems).

At a minimum, I think we should consider introducing Python on an optional
> basis (i.e., the C Kconfig implementation only gets used if a Python
> interpreter is unavailable), but making it required would be even better.
>
That idea is... horrible. Instead of only bringing in the python dependency
we'd also create an environmental dependency that can silently introduce
behavioral differences.

This project has the ability of building bit-identical firmware images on
hosts spanning all kinds of CPU register sizes, endianness and operating
system families. If you ever heard of "hermetic builds" being a good idea:
the trustworthiness of our build is stronger - and part of making that
possible is being (relatively) careful with picking our dependencies.


Patrick
-- 
Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org

[coreboot] Re: There is a python in our toolchain?!?

[Rao G]

Hi Patrick,

That's good to hear, would there be change to "make menuconfig" with
kconfiglib

Thanks
Ranga

On Wed, Sep 29, 2021 at 10:58 AM Patrick Georgi via coreboot <
coreboot@coreboot.org> wrote:

> Hi everybody,
>
> Historically, coreboot avoided depending on python too much (we got rid of
> an entire python based configuration and build system, for example), with
> few minor exceptions.
>
> The main reason has been that while python code is quick to slap together,
> it has demonstrated a penchant for breaking in all kinds of mysterious ways
> (python 2->3 really was just a slightly bigger instance of what's going on
> in python all the time), and its users demonstrate a disregard for their
> fellow developers as demonstrated by endless stack traces on trivial errors
> (or is the language too complicated to properly catch them all?)
>
> While probably nice for one-off prototypes, long term maintenance is a
> concern: this project has over 20 years of history under its belt, with
> more to come.
>
> That said, python makes its way back into the tree every now and then
> (typically as small snippets to compute and add hashes to binaries as
> needed by ARM SoCs). Uncanny, but typically not a big deal.
>
> There are two bigger initiatives proposed that would significantly
> increase our python footprint:
> 1. Replacing Linux's kconfig with kconfiglib (
> https://review.coreboot.org/c/coreboot/+/48679)
> 2. Using pytest for end-to-end testing utilities (
> https://review.coreboot.org/c/coreboot/+/57869)
>
> Compared to the "inject a hash value at a fixed location" scripts, these
> would probably be here to stay, and sufficiently integrated that everybody
> will have to deal with them.
>
> People spending time working on python code when it has no chance to land
> isn't a good use of their time and we should avoid that in the project.
>
> People spending time arguing that python shouldn't be used (to avoid
> the other outcome) even though the project's culture shifted and is now
> accepting Python isn't creating a great community for anybody.
>
> To avoid these scenarios, could we possibly nail down the policy on python
> in coreboot?
>
>
> Regards,
> Patrick
> --
> Google Germany GmbH, ABC-Str. 19, 20354 Hamburg
> Registergericht und -nummer: Hamburg, HRB 86891, Sitz der Gesellschaft:
> Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
> ___
> coreboot mailing list -- coreboot@coreboot.org
> To unsubscribe send an email to coreboot-le...@coreboot.org
>
___
coreboot mailing list -- coreboot@coreboot.org
To unsubscribe send an email to coreboot-le...@coreboot.org