Re: Requirement Document generator tool

[Christian Mauderer]

Hello Chris,

please let me pick up the discussion again so that we manage to come
to a conclusion. Like Matthias said oflist, we are trying to address
the problem of the missing project overview in a separate thread. But
it would be really relevant to get a result for the python topics.

Again: In my view the discussion has been triggered by the
pre-qualification efforts but it isn't limited to it. Having standards
for how to handle python tools would make adding and maintaining new
tools simpler in any case.

Although we will have some more topics (test approach, static
analysis, documentation style, ...) I'll try to concentrate on the
style issues for now. The other topics are much less relevant for the
existing code.

I already said that in another mail but please let me repeat it again:
My target wouldn't be to reformat existing code. My target is to find
something that would be acceptable for all new code and that is
similar enough that existing code is still readable and maybe can
converted when a change is necessary for some other reason. PEP 8 even
suggests to not reformat code if it predates a coding standard:

https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds

Google style doesn't express it that directly but there are "Parting
Words" too that tell basically the same:

https://google.github.io/styleguide/pyguide.html#4-parting-words

Below I added some answers to the code locations that you specifically
mentioned before.

On 28/01/2020 11:41, Chris Johns wrote:
> On 28/1/20 8:14 pm, Christian Mauderer wrote:
>> On 28/01/2020 01:42, Chris Johns wrote:
>>> On 28/1/20 10:48 am, Joel Sherrill wrote:
 On Mon, Jan 27, 2020 at 4:18 PM Chris Johns >>> > wrote:

 On 24/1/20 9:57 pm, Jose Valdez wrote:
 > In fact these tools target the pre-qualified project.

 Do you see this as different to the RTEMS project?

 > Since it was Sebastian who suggested to create this set of
python tools,

 I think Sebastian is wanting a smooth path for these tools into
the project.

 > I think the idea was to standardize the use of python not
only for this
 project, but also for other python written code in RTEMS
community. This has
 the advantages that every written python code is standard, but
has the
 drawbacks:
 >
 > -> old written code would need to be adapted to the standards.

 How different to the proposed coding standard is the existing
code? Why not base
 the coding standard on what exists in the code base?

 This is a very important question.

 Have you evaluated the size of the task to update the existing
code? How would
 get such changes for the rtems-tools and the RSB be tested and
integrated back
 into the project? This apporach seems like a huge review task
for me.

 It could be or it may turn out that there isn't much changed.
Without someone
 running the reformatter and reporting, we won't know.
>>>
>>> Running a reformatter may give you an idea of the scale however I
have some
>>> concerns as Python's logic is based on indent levels and a missed
level changes
>>> the logic. I am not sure how you know such a tool is safe to use
unless you
>>> review all the changes.
>>
>> To get some statistics for that I tried using yapf on the rtems-tools
>> repo. I excluded everything with  "patch-gdb-python", "tftpy" and
>> "asciidoc" in the name. You can see the yapf call in the commit message.
>
> The doxygen is from waf. Thomas knows how to write python.
>
>> Google style:
>>
https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/f8202a53ca043d23732d79ab217071e975dc35ad/0001-FIXME-Format-in-Google-style.patch
>>
>> PEP8 Style:
>>
https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/c97c77762f65041d9244bc8bc64b90ec698735bf/0001-FIXME-Reformat-using-PEP8.patch
>>
>> In both cases it's about 3000 lines removed and 3700 lines added.
>>
>> Without having a look at each line: Most seems to be just the normal
>> formatting stuff. Some lines are too long. Some spaces missing around
>> operators or spaces where they shouldn't be.
>
> And then there is removal of spaces on default args ...
>
> -def section_macro_map(self, section, nesting_level = 0):
> +def section_macro_map(self, section, nesting_level=0):

How strong do you feel about these changes? Like noted in another mail
it's one of the official Python PEPs (PEP 8) that suggests to have no
spaces here. See:

https://www.python.org/dev/peps/pep-0008/#other-recommendations

>
> I prefer spaces as it is consistent with other uses of spaces around
operators.
>
>> Some lists written with
>> more or less line breaks.
>
> Yes to make them readable. This is on purpose. I do not agree with this ..
>
> -return cfgs + ['objcopy',
> -   'arc

Re: Requirement Document generator tool

[Sebastian Huber]

- Am 29. Jan 2020 um 23:48 schrieb Chris Johns chr...@rtems.org:

> On 29/1/20 7:40 pm, Christian Mauderer wrote:
>> On 28/01/2020 23:15, Chris Johns wrote:
>>> On 28/1/20 10:22 pm, Christian Mauderer wrote:
[...]
>> If I understood you correctly your main problem with both suggested ones
>> (PEP8 and Google) is that multi line lists have been put together into
>> one liners and dictionaries with aligned fields are not aligned anymore?
> 
> They cannot be touched ...
> 
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/darwin.py#n41
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/freebsd.py#n50
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/linux.py#n54
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/linux.py#n104
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/linux.py#n104
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/options.py#n56
> https://git.rtems.org/rtems-libbsd/tree/libbsd.py#n59
> https://git.rtems.org/rtems-libbsd/tree/libbsd.py#n108
> https://git.rtems.org/rtems-libbsd/tree/libbsd.py#n148
> 
> I could go on but I think this list should highlight the issue.

The YAPF guide explicitly mentions this issue (Why does YAPF destroy my awesome 
formatting?):

https://github.com/google/yapf#why-does-yapf-destroy-my-awesome-formatting

You can disable the formatting in selected areas with

# yapf: disable/enable

comments.

black has a similar feature:

https://github.com/psf/black#the-black-code-style
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Chris Johns]

On 3/2/20 4:19 am, Christian Mauderer wrote:
> like discussed: A ticket for it: https://devel.rtems.org/ticket/3860
> Please take a look at it. Feel free to add something if I missed it.

Thank you. I have added a possible hurdle, defining how we test each change as
the tool is introduced.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Christian Mauderer]

On 31/01/2020 19:44, Gedare Bloom wrote:
 But I think implementing that would be a lot bigger task than the one we
 are just discussing. One of the first problems would be finding a style
 checker that can produce _exactly_ the coding style that is defined for
 score.
>>> Agreed.
>>>
 And also I'm happy to discuss approaches for something like that
 it would start to be of-topic for this thread. So if you want to discuss
 details about automatic style checking for score we should create an
 extra topic. Maybe we could even create a GSoC project for something
 like that?
>>> That is a good idea. Would you like to raise a task ticket?
>> I can do that. I think it could be a nice project. If it works we gain
>> quite a value from it. If it doesn't work we don't lose anything.
>>
> +1 as a potential GSoC project. Please ping me with the ticket # so I
> can review it.

Hello Chris and Gedare,

like discussed: A ticket for it: https://devel.rtems.org/ticket/3860
Please take a look at it. Feel free to add something if I missed it.

Best regards

Christian
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Gedare Bloom]

On Fri, Jan 31, 2020 at 12:39 AM Christian Mauderer
 wrote:
>
> On 30/01/2020 23:13, Chris Johns wrote:
> > On 30/1/20 8:35 pm, Christian Mauderer wrote:
> >> On 29/01/2020 23:48, Chris Johns wrote:
> >>> On 29/1/20 7:40 pm, Christian Mauderer wrote:
>  On 28/01/2020 23:15, Chris Johns wrote:
> > On 28/1/20 10:22 pm, Christian Mauderer wrote:
> >> I'm quite indifferent which style we use
> >
> > But you are arguing a position, such as ...
> >
> >> but do you really think that it
> >> is a good idea to roll our own RTEMS-Python-style instead of using one
> >> of the big ones (like PEP8, Google or some other that might is a better
> >> fit for existing code). Rolling our own would lead to:
> >>
> >> 1. Long discussions about what is THE RIGHT STYLE (already started in
> >> this mail).
> >>
> >> 2. A lot of problems that there is no tool that formats or checks
> >> whether code is in THE RIGHT STYLE.
> >>
> >> 3. More discussions later because some other developer thinks that THE
> >> RIGHT STYLE is wrong.
> 
>  You are right that I argue against home-grown styles. I learned that
>  style discussions are quite similar to religious discussions. Everyone
>  beliefs that his style is the only right one. Using a big one makes it
>  simpler to avoid discussions because a lot of people know it.
> >>>
> >>> And yet you inherit issues a large organisation cannot correct because
> >>> historically a style is difficult to change and they are wedged. They 
> >>> need to be
> >>> reviewed and evaluated.
> >>
> >> That's why I originally thought of defining a style for _new_ code with
> >> an optional _slow_ migration if some existing code is touched. Of course
> >> the style should be compatible with the views how the old code should
> >> look in the future too.
> >
> > I do not think this works as I pointed out earlier. We leave a libaility in 
> > the
> > old code that someone needs to clean up or we just ignore the rules for the 
> > old
> > code. Either are not great.
>
> I agree that it's not a great solution. But we have existing code. And
> like you said yourself: It's not a good idea to just blindly reformat
> it. And I assume even if it would be in the scope of some project, we
> wouldn't entirely trust a new developer to do it right because we don't
> know how careful he works.
>
> >
> >> The patches were just for an estimate how much
> >> influence the suggested styles could have. I would never expect that any
> >> big patch with automated changes would be acceptable without a thorough
> >> review. I wouldn't accept that myself.
> >
> > I understand this.
> >
>  Please don't get me wrong: I'm OK with another style too. The relevant
>  result is that we have _one_ style in RTEMS that can be automatically
>  checked.
> >>>
> >>> I find it interesting this view is OK for Python and the tools and yet we 
> >>> do not
> >>> have automatic checking for the score's C. Or do we?
> >>
> >> It would be great if we would have automatic checking for C too. There
> >> are big projects that have something like that and it's really
> >> convenient. For example for Linux before you send a patch you can just
> >> run "./scripts/checkpatch.pl some.patch" and it will give you everything
> >> you should fix before sending it to the list. That removes at least one
> >> round of review.
> >
> > My point is taking say Google C or C++ coding standard and applying it to 
> > the
> > score. I would prefer we wrote up the rules we have in place and check 
> > those.
>
> Ah OK. I missed your point. In that case: I'm a bit torn. I wouldn't say
> that it is always a bad idea to migrate to some adapted style rules if
> there are good reasons to drop the old ones (except from a revision
> control system view where such changes are always annoying). But please
> also note that the differences would be a lot bigger.
>
> I counted the lines in the files that I touched with the formatting tool:
>
> wc -l `find -name "*.py" | grep -v "patch-gdb-python" | grep -v "tftpy"
> | grep -v "asciidoc"`
>
> It's a total of 18193 lines. The patch touched about 3000 of them. So
> it's not a small part but it's not everything either. I assume that we
> can reduce it as soon as we worked out some rules and solutions for
> formats that can't be touched to the extend that these coding standards
> would.
>
> For RTEMS score and nearly every C coding style I know it would be more
> of 90 to 100 percent of the code due to the indentation style alone.
>
> >
> >> But I think implementing that would be a lot bigger task than the one we
> >> are just discussing. One of the first problems would be finding a style
> >> checker that can produce _exactly_ the coding style that is defined for
> >> score.
> >
> > Agreed.
> >
> >> And also I'm happy to discuss approaches for something like that
> >> it would start to be of-topic for this thread. So if you want to discus

Re: Requirement Document generator tool

[Christian Mauderer]

On 30/01/2020 23:13, Chris Johns wrote:
> On 30/1/20 8:35 pm, Christian Mauderer wrote:
>> On 29/01/2020 23:48, Chris Johns wrote:
>>> On 29/1/20 7:40 pm, Christian Mauderer wrote:
 On 28/01/2020 23:15, Chris Johns wrote:
> On 28/1/20 10:22 pm, Christian Mauderer wrote:
>> I'm quite indifferent which style we use
>
> But you are arguing a position, such as ...
>
>> but do you really think that it
>> is a good idea to roll our own RTEMS-Python-style instead of using one
>> of the big ones (like PEP8, Google or some other that might is a better
>> fit for existing code). Rolling our own would lead to:
>>
>> 1. Long discussions about what is THE RIGHT STYLE (already started in
>> this mail).
>>
>> 2. A lot of problems that there is no tool that formats or checks
>> whether code is in THE RIGHT STYLE.
>>
>> 3. More discussions later because some other developer thinks that THE
>> RIGHT STYLE is wrong.

 You are right that I argue against home-grown styles. I learned that
 style discussions are quite similar to religious discussions. Everyone
 beliefs that his style is the only right one. Using a big one makes it
 simpler to avoid discussions because a lot of people know it.
>>>
>>> And yet you inherit issues a large organisation cannot correct because
>>> historically a style is difficult to change and they are wedged. They need 
>>> to be
>>> reviewed and evaluated.
>>
>> That's why I originally thought of defining a style for _new_ code with
>> an optional _slow_ migration if some existing code is touched. Of course
>> the style should be compatible with the views how the old code should
>> look in the future too. 
> 
> I do not think this works as I pointed out earlier. We leave a libaility in 
> the
> old code that someone needs to clean up or we just ignore the rules for the 
> old
> code. Either are not great.

I agree that it's not a great solution. But we have existing code. And
like you said yourself: It's not a good idea to just blindly reformat
it. And I assume even if it would be in the scope of some project, we
wouldn't entirely trust a new developer to do it right because we don't
know how careful he works.

> 
>> The patches were just for an estimate how much
>> influence the suggested styles could have. I would never expect that any
>> big patch with automated changes would be acceptable without a thorough
>> review. I wouldn't accept that myself.
> 
> I understand this.
> 
 Please don't get me wrong: I'm OK with another style too. The relevant
 result is that we have _one_ style in RTEMS that can be automatically
 checked. 
>>>
>>> I find it interesting this view is OK for Python and the tools and yet we 
>>> do not
>>> have automatic checking for the score's C. Or do we?
>>
>> It would be great if we would have automatic checking for C too. There
>> are big projects that have something like that and it's really
>> convenient. For example for Linux before you send a patch you can just
>> run "./scripts/checkpatch.pl some.patch" and it will give you everything
>> you should fix before sending it to the list. That removes at least one
>> round of review.
> 
> My point is taking say Google C or C++ coding standard and applying it to the
> score. I would prefer we wrote up the rules we have in place and check those.

Ah OK. I missed your point. In that case: I'm a bit torn. I wouldn't say
that it is always a bad idea to migrate to some adapted style rules if
there are good reasons to drop the old ones (except from a revision
control system view where such changes are always annoying). But please
also note that the differences would be a lot bigger.

I counted the lines in the files that I touched with the formatting tool:

wc -l `find -name "*.py" | grep -v "patch-gdb-python" | grep -v "tftpy"
| grep -v "asciidoc"`

It's a total of 18193 lines. The patch touched about 3000 of them. So
it's not a small part but it's not everything either. I assume that we
can reduce it as soon as we worked out some rules and solutions for
formats that can't be touched to the extend that these coding standards
would.

For RTEMS score and nearly every C coding style I know it would be more
of 90 to 100 percent of the code due to the indentation style alone.

> 
>> But I think implementing that would be a lot bigger task than the one we
>> are just discussing. One of the first problems would be finding a style
>> checker that can produce _exactly_ the coding style that is defined for
>> score.
> 
> Agreed.
> 
>> And also I'm happy to discuss approaches for something like that
>> it would start to be of-topic for this thread. So if you want to discuss
>> details about automatic style checking for score we should create an
>> extra topic. Maybe we could even create a GSoC project for something
>> like that?
> 
> That is a good idea. Would you like to raise a task ticket?

I can do that. I think it could be a nice 

Re: Requirement Document generator tool

[Chris Johns]

On 30/1/20 8:35 pm, Christian Mauderer wrote:
> On 29/01/2020 23:48, Chris Johns wrote:
>> On 29/1/20 7:40 pm, Christian Mauderer wrote:
>>> On 28/01/2020 23:15, Chris Johns wrote:
 On 28/1/20 10:22 pm, Christian Mauderer wrote:
> I'm quite indifferent which style we use

 But you are arguing a position, such as ...

> but do you really think that it
> is a good idea to roll our own RTEMS-Python-style instead of using one
> of the big ones (like PEP8, Google or some other that might is a better
> fit for existing code). Rolling our own would lead to:
>
> 1. Long discussions about what is THE RIGHT STYLE (already started in
> this mail).
>
> 2. A lot of problems that there is no tool that formats or checks
> whether code is in THE RIGHT STYLE.
>
> 3. More discussions later because some other developer thinks that THE
> RIGHT STYLE is wrong.
>>>
>>> You are right that I argue against home-grown styles. I learned that
>>> style discussions are quite similar to religious discussions. Everyone
>>> beliefs that his style is the only right one. Using a big one makes it
>>> simpler to avoid discussions because a lot of people know it.
>>
>> And yet you inherit issues a large organisation cannot correct because
>> historically a style is difficult to change and they are wedged. They need 
>> to be
>> reviewed and evaluated.
> 
> That's why I originally thought of defining a style for _new_ code with
> an optional _slow_ migration if some existing code is touched. Of course
> the style should be compatible with the views how the old code should
> look in the future too. 

I do not think this works as I pointed out earlier. We leave a libaility in the
old code that someone needs to clean up or we just ignore the rules for the old
code. Either are not great.

> The patches were just for an estimate how much
> influence the suggested styles could have. I would never expect that any
> big patch with automated changes would be acceptable without a thorough
> review. I wouldn't accept that myself.

I understand this.

>>> Please don't get me wrong: I'm OK with another style too. The relevant
>>> result is that we have _one_ style in RTEMS that can be automatically
>>> checked. 
>>
>> I find it interesting this view is OK for Python and the tools and yet we do 
>> not
>> have automatic checking for the score's C. Or do we?
> 
> It would be great if we would have automatic checking for C too. There
> are big projects that have something like that and it's really
> convenient. For example for Linux before you send a patch you can just
> run "./scripts/checkpatch.pl some.patch" and it will give you everything
> you should fix before sending it to the list. That removes at least one
> round of review.

My point is taking say Google C or C++ coding standard and applying it to the
score. I would prefer we wrote up the rules we have in place and check those.

> But I think implementing that would be a lot bigger task than the one we
> are just discussing. One of the first problems would be finding a style
> checker that can produce _exactly_ the coding style that is defined for
> score.

Agreed.

> And also I'm happy to discuss approaches for something like that
> it would start to be of-topic for this thread. So if you want to discuss
> details about automatic style checking for score we should create an
> extra topic. Maybe we could even create a GSoC project for something
> like that?

That is a good idea. Would you like to raise a task ticket?

>>> That also will make patch review simpler because we as
>>> maintainers don't have to do the style review by hand.
>>
>> This is often more about those coding than those reviewing.
> 
> Sorry, I think I don't fully understand you here.

The reviewers often know the rules and can code to them, it is those writing the
code that do not and need the help. A formatter helps them.

 I wrote the code, I know it and I need to maintain it. All I am asking is 
 if
 pre-qual wants rules then I suggest they find some common ground.
>>>
>>> I agree that it is a good idea to find something that is acceptable for all.
>>>

 I wonder how libbsd.py goes with these rules. Also all the wscript files 
 which
 are also python and also the RSB would be covered.
>>>
>>> Most likely not well. But that's the problem: We already have a bunch of
>>> different styles. Let's try to agree on one before we get lots of extra
>>> code. A lot of pre-qualification code will be python. Therefore now is
>>> the best time.
>>
>> You may know more than me about this. We have been given no information about
>> the project's plans or these tools and is becoming frustrating. Joel and I 
>> have
>> asked on a number occasions. The silence is testing our patience.
> 
> I'm really sorry for that. Like I mentioned earlier I'm not really very
> good with the overview over this project too. I'll try to discuss this
> with Sebastian

Re: Requirement Document generator tool

[Christian Mauderer]

On 29/01/2020 23:48, Chris Johns wrote:
> On 29/1/20 7:40 pm, Christian Mauderer wrote:
>> On 28/01/2020 23:15, Chris Johns wrote:
>>> On 28/1/20 10:22 pm, Christian Mauderer wrote:
 I'm quite indifferent which style we use
>>>
>>> But you are arguing a position, such as ...
>>>
 but do you really think that it
 is a good idea to roll our own RTEMS-Python-style instead of using one
 of the big ones (like PEP8, Google or some other that might is a better
 fit for existing code). Rolling our own would lead to:

 1. Long discussions about what is THE RIGHT STYLE (already started in
 this mail).

 2. A lot of problems that there is no tool that formats or checks
 whether code is in THE RIGHT STYLE.

 3. More discussions later because some other developer thinks that THE
 RIGHT STYLE is wrong.
>>
>> You are right that I argue against home-grown styles. I learned that
>> style discussions are quite similar to religious discussions. Everyone
>> beliefs that his style is the only right one. Using a big one makes it
>> simpler to avoid discussions because a lot of people know it.
> 
> And yet you inherit issues a large organisation cannot correct because
> historically a style is difficult to change and they are wedged. They need to 
> be
> reviewed and evaluated.

That's why I originally thought of defining a style for _new_ code with
an optional _slow_ migration if some existing code is touched. Of course
the style should be compatible with the views how the old code should
look in the future too. The patches were just for an estimate how much
influence the suggested styles could have. I would never expect that any
big patch with automated changes would be acceptable without a thorough
review. I wouldn't accept that myself.

> 
>> Please don't get me wrong: I'm OK with another style too. The relevant
>> result is that we have _one_ style in RTEMS that can be automatically
>> checked. 
> 
> I find it interesting this view is OK for Python and the tools and yet we do 
> not
> have automatic checking for the score's C. Or do we?

It would be great if we would have automatic checking for C too. There
are big projects that have something like that and it's really
convenient. For example for Linux before you send a patch you can just
run "./scripts/checkpatch.pl some.patch" and it will give you everything
you should fix before sending it to the list. That removes at least one
round of review.

But I think implementing that would be a lot bigger task than the one we
are just discussing. One of the first problems would be finding a style
checker that can produce _exactly_ the coding style that is defined for
score. And also I'm happy to discuss approaches for something like that
it would start to be of-topic for this thread. So if you want to discuss
details about automatic style checking for score we should create an
extra topic. Maybe we could even create a GSoC project for something
like that?

> 
>> That also will make patch review simpler because we as
>> maintainers don't have to do the style review by hand.
> 
> This is often more about those coding than those reviewing.

Sorry, I think I don't fully understand you here.

> 
>>> I wrote the code, I know it and I need to maintain it. All I am asking is if
>>> pre-qual wants rules then I suggest they find some common ground.
>>
>> I agree that it is a good idea to find something that is acceptable for all.
>>
>>>
>>> I wonder how libbsd.py goes with these rules. Also all the wscript files 
>>> which
>>> are also python and also the RSB would be covered.
>>
>> Most likely not well. But that's the problem: We already have a bunch of
>> different styles. Let's try to agree on one before we get lots of extra
>> code. A lot of pre-qualification code will be python. Therefore now is
>> the best time.
> 
> You may know more than me about this. We have been given no information about
> the project's plans or these tools and is becoming frustrating. Joel and I 
> have
> asked on a number occasions. The silence is testing our patience.

I'm really sorry for that. Like I mentioned earlier I'm not really very
good with the overview over this project too. I'll try to discuss this
with Sebastian and Thomas as soon as Sebastian is back from vacation.

> 
> How about we put a python rules discussion on hold until the intentions of the
> pre-qual project are presented publicly and in a manner we can all digest. I
> have spent enough time and energy discussing this stuff and it is not clear to
> me we are in a position to accept what is to be written.

If you want to stop the discussion, I have to accept that. Please just
answer with "this needs further discussion but not now" for all parts
where you think we should stop right now.

But please note that our current discussion is in my opinion more of a
basic discussion that is only triggered by the pre-qualification
efforts. It's not really relevant whether new tools are added b

Re: Requirement Document generator tool

[Chris Johns]

On 29/1/20 7:40 pm, Christian Mauderer wrote:
> On 28/01/2020 23:15, Chris Johns wrote:
>> On 28/1/20 10:22 pm, Christian Mauderer wrote:
>>> I'm quite indifferent which style we use
>>
>> But you are arguing a position, such as ...
>>
>>> but do you really think that it
>>> is a good idea to roll our own RTEMS-Python-style instead of using one
>>> of the big ones (like PEP8, Google or some other that might is a better
>>> fit for existing code). Rolling our own would lead to:
>>>
>>> 1. Long discussions about what is THE RIGHT STYLE (already started in
>>> this mail).
>>>
>>> 2. A lot of problems that there is no tool that formats or checks
>>> whether code is in THE RIGHT STYLE.
>>>
>>> 3. More discussions later because some other developer thinks that THE
>>> RIGHT STYLE is wrong.
> 
> You are right that I argue against home-grown styles. I learned that
> style discussions are quite similar to religious discussions. Everyone
> beliefs that his style is the only right one. Using a big one makes it
> simpler to avoid discussions because a lot of people know it.

And yet you inherit issues a large organisation cannot correct because
historically a style is difficult to change and they are wedged. They need to be
reviewed and evaluated.

> Please don't get me wrong: I'm OK with another style too. The relevant
> result is that we have _one_ style in RTEMS that can be automatically
> checked. 

I find it interesting this view is OK for Python and the tools and yet we do not
have automatic checking for the score's C. Or do we?

> That also will make patch review simpler because we as
> maintainers don't have to do the style review by hand.

This is often more about those coding than those reviewing.

>> I wrote the code, I know it and I need to maintain it. All I am asking is if
>> pre-qual wants rules then I suggest they find some common ground.
> 
> I agree that it is a good idea to find something that is acceptable for all.
> 
>>
>> I wonder how libbsd.py goes with these rules. Also all the wscript files 
>> which
>> are also python and also the RSB would be covered.
> 
> Most likely not well. But that's the problem: We already have a bunch of
> different styles. Let's try to agree on one before we get lots of extra
> code. A lot of pre-qualification code will be python. Therefore now is
> the best time.

You may know more than me about this. We have been given no information about
the project's plans or these tools and is becoming frustrating. Joel and I have
asked on a number occasions. The silence is testing our patience.

How about we put a python rules discussion on hold until the intentions of the
pre-qual project are presented publicly and in a manner we can all digest. I
have spent enough time and energy discussing this stuff and it is not clear to
me we are in a position to accept what is to be written.

> If I understood you correctly your main problem with both suggested ones
> (PEP8 and Google) is that multi line lists have been put together into
> one liners and dictionaries with aligned fields are not aligned anymore?

They cannot be touched ...

https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/darwin.py#n41
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/freebsd.py#n50
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/linux.py#n54
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/linux.py#n104
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/linux.py#n104
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/options.py#n56
https://git.rtems.org/rtems-libbsd/tree/libbsd.py#n59
https://git.rtems.org/rtems-libbsd/tree/libbsd.py#n108
https://git.rtems.org/rtems-libbsd/tree/libbsd.py#n148

I could go on but I think this list should highlight the issue.

> About correct?

The lack of space around the arguments is also a change that is silly and I do
not want changed.

The biggest challenge who ever finally proposes these rules faces is the RSB. It
is very difficult to know if you have broken it. It has taken years to converge
to the level of stability we have. All changes in here would need a close review
and that something I am not excited about.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Christian Mauderer]

On 28/01/2020 23:15, Chris Johns wrote:
> On 28/1/20 10:22 pm, Christian Mauderer wrote:
>> I'm quite indifferent which style we use
> 
> But you are arguing a position, such as ...
> 
>> but do you really think that it
>> is a good idea to roll our own RTEMS-Python-style instead of using one
>> of the big ones (like PEP8, Google or some other that might is a better
>> fit for existing code). Rolling our own would lead to:
>>
>> 1. Long discussions about what is THE RIGHT STYLE (already started in
>> this mail).
>>
>> 2. A lot of problems that there is no tool that formats or checks
>> whether code is in THE RIGHT STYLE.
>>
>> 3. More discussions later because some other developer thinks that THE
>> RIGHT STYLE is wrong.

You are right that I argue against home-grown styles. I learned that
style discussions are quite similar to religious discussions. Everyone
beliefs that his style is the only right one. Using a big one makes it
simpler to avoid discussions because a lot of people know it.

Please don't get me wrong: I'm OK with another style too. The relevant
result is that we have _one_ style in RTEMS that can be automatically
checked. That also will make patch review simpler because we as
maintainers don't have to do the style review by hand.

> 
> I wrote the code, I know it and I need to maintain it. All I am asking is if
> pre-qual wants rules then I suggest they find some common ground.

I agree that it is a good idea to find something that is acceptable for all.

> 
> I wonder how libbsd.py goes with these rules. Also all the wscript files which
> are also python and also the RSB would be covered.

Most likely not well. But that's the problem: We already have a bunch of
different styles. Let's try to agree on one before we get lots of extra
code. A lot of pre-qualification code will be python. Therefore now is
the best time.

If I understood you correctly your main problem with both suggested ones
(PEP8 and Google) is that multi line lists have been put together into
one liners and dictionaries with aligned fields are not aligned anymore?
About correct?

Best regards

Christian

> 
> Chris
> 

-- 

embedded brains GmbH
Herr Christian Mauderer
Dornierstr. 4
D-82178 Puchheim
Germany
email: christian.maude...@embedded-brains.de
Phone: +49-89-18 94 741 - 18
Fax:   +49-89-18 94 741 - 08
PGP: Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Christian Mauderer]

On 28/01/2020 23:15, Joel Sherrill wrote:
> 
> 
> On Tue, Jan 28, 2020, 5:22 AM Christian Mauderer
>  > wrote:
> 
> On 28/01/2020 11:41, Chris Johns wrote:
> >
> >
> > On 28/1/20 8:14 pm, Christian Mauderer wrote:
> >> On 28/01/2020 01:42, Chris Johns wrote:
> >>> On 28/1/20 10:48 am, Joel Sherrill wrote:
>  On Mon, Jan 27, 2020 at 4:18 PM Chris Johns  
>  >> wrote:
> 
>      On 24/1/20 9:57 pm, Jose Valdez wrote:
>      > In fact these tools target the pre-qualified project.
> 
>      Do you see this as different to the RTEMS project?
> 
>      > Since it was Sebastian who suggested to create this set
> of python tools,
> 
>      I think Sebastian is wanting a smooth path for these tools
> into the project.
> 
>      > I think the idea was to standardize the use of python not
> only for this
>      project, but also for other python written code in RTEMS
> community. This has
>      the advantages that every written python code is standard,
> but has the
>      drawbacks:
>      >
>      > -> old written code would need to be adapted to the
> standards.
> 
>      How different to the proposed coding standard is the
> existing code? Why not base
>      the coding standard on what exists in the code base?
> 
>  This is a very important question.
> 
>      Have you evaluated the size of the task to update the
> existing code? How would
>      get such changes for the rtems-tools and the RSB be tested
> and integrated back
>      into the project? This apporach seems like a huge review
> task for me.
> 
>  It could be or it may turn out that there isn't much changed.
> Without someone
>  running the reformatter and reporting, we won't know.
> >>>
> >>> Running a reformatter may give you an idea of the scale however
> I have some
> >>> concerns as Python's logic is based on indent levels and a
> missed level changes
> >>> the logic. I am not sure how you know such a tool is safe to use
> unless you
> >>> review all the changes.
> >>
> >> To get some statistics for that I tried using yapf on the rtems-tools
> >> repo. I excluded everything with  "patch-gdb-python", "tftpy" and
> >> "asciidoc" in the name. You can see the yapf call in the commit
> message.
> >
> > The doxygen is from waf. Thomas knows how to write python.
> 
> OK. I missed that one. The patches were just a test to get some
> orientation about what it would mean.
> 
> Please also note: I don't want to doubt the python knowledge of anyone.
> 
> >
> >> Google style:
> >>
> 
> https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/f8202a53ca043d23732d79ab217071e975dc35ad/0001-FIXME-Format-in-Google-style.patch
> >>
> >> PEP8 Style:
> >>
> 
> https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/c97c77762f65041d9244bc8bc64b90ec698735bf/0001-FIXME-Reformat-using-PEP8.patch
> >>
> >> In both cases it's about 3000 lines removed and 3700 lines added.
> 
> 
> Is it possible to make changes we think are ok and in both styles? That
> would start to whittle down the style issues 
> 
> 
> >>
> >> Without having a look at each line: Most seems to be just the normal
> >> formatting stuff. Some lines are too long. Some spaces missing around
> >> operators or spaces where they shouldn't be.
> >
> > And then there is removal of spaces on default args ...
> >
> > -    def section_macro_map(self, section, nesting_level = 0):
> > +    def section_macro_map(self, section, nesting_level=0):
> >
> > I prefer spaces as it is consistent with other uses of spaces
> around operators.
> 
> In my view these are detail style discussions. Same is true for most of
> your notes further below (although I agree with some of your opinions).
> But I didn't wanted to trigger any of these. My intention was to give
> some impression about how big such changes would be.
> 
> I know that there is quite some python code already in RTEMS. And I'm
> also sure that there are already quite some different python styles in
> RTEMS. At least as many as programmers touching python code. But
> currently the python code is still few enough that it would be possible
> to get a uniform style. And pre-qualification or not: I'm really
> convinced that a fixed style would be a good thing for the code base in
> general.
> 
> I'm quite indifferent which style we use but do you really think that it
> is a good idea to 

Re: Requirement Document generator tool

[Chris Johns]

On 28/1/20 10:22 pm, Christian Mauderer wrote:
> I'm quite indifferent which style we use

But you are arguing a position, such as ...

> but do you really think that it
> is a good idea to roll our own RTEMS-Python-style instead of using one
> of the big ones (like PEP8, Google or some other that might is a better
> fit for existing code). Rolling our own would lead to:
> 
> 1. Long discussions about what is THE RIGHT STYLE (already started in
> this mail).
> 
> 2. A lot of problems that there is no tool that formats or checks
> whether code is in THE RIGHT STYLE.
> 
> 3. More discussions later because some other developer thinks that THE
> RIGHT STYLE is wrong.

I wrote the code, I know it and I need to maintain it. All I am asking is if
pre-qual wants rules then I suggest they find some common ground.

I wonder how libbsd.py goes with these rules. Also all the wscript files which
are also python and also the RSB would be covered.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Joel Sherrill]

On Tue, Jan 28, 2020, 5:22 AM Christian Mauderer <
christian.maude...@embedded-brains.de> wrote:

> On 28/01/2020 11:41, Chris Johns wrote:
> >
> >
> > On 28/1/20 8:14 pm, Christian Mauderer wrote:
> >> On 28/01/2020 01:42, Chris Johns wrote:
> >>> On 28/1/20 10:48 am, Joel Sherrill wrote:
>  On Mon, Jan 27, 2020 at 4:18 PM Chris Johns   > wrote:
> 
>  On 24/1/20 9:57 pm, Jose Valdez wrote:
>  > In fact these tools target the pre-qualified project.
> 
>  Do you see this as different to the RTEMS project?
> 
>  > Since it was Sebastian who suggested to create this set of
> python tools,
> 
>  I think Sebastian is wanting a smooth path for these tools into
> the project.
> 
>  > I think the idea was to standardize the use of python not only
> for this
>  project, but also for other python written code in RTEMS
> community. This has
>  the advantages that every written python code is standard, but
> has the
>  drawbacks:
>  >
>  > -> old written code would need to be adapted to the standards.
> 
>  How different to the proposed coding standard is the existing
> code? Why not base
>  the coding standard on what exists in the code base?
> 
>  This is a very important question.
> 
>  Have you evaluated the size of the task to update the existing
> code? How would
>  get such changes for the rtems-tools and the RSB be tested and
> integrated back
>  into the project? This apporach seems like a huge review task for
> me.
> 
>  It could be or it may turn out that there isn't much changed. Without
> someone
>  running the reformatter and reporting, we won't know.
> >>>
> >>> Running a reformatter may give you an idea of the scale however I have
> some
> >>> concerns as Python's logic is based on indent levels and a missed
> level changes
> >>> the logic. I am not sure how you know such a tool is safe to use
> unless you
> >>> review all the changes.
> >>
> >> To get some statistics for that I tried using yapf on the rtems-tools
> >> repo. I excluded everything with  "patch-gdb-python", "tftpy" and
> >> "asciidoc" in the name. You can see the yapf call in the commit message.
> >
> > The doxygen is from waf. Thomas knows how to write python.
>
> OK. I missed that one. The patches were just a test to get some
> orientation about what it would mean.
>
> Please also note: I don't want to doubt the python knowledge of anyone.
>
> >
> >> Google style:
> >>
> https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/f8202a53ca043d23732d79ab217071e975dc35ad/0001-FIXME-Format-in-Google-style.patch
> >>
> >> PEP8 Style:
> >>
> https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/c97c77762f65041d9244bc8bc64b90ec698735bf/0001-FIXME-Reformat-using-PEP8.patch
> >>
> >> In both cases it's about 3000 lines removed and 3700 lines added.
>

Is it possible to make changes we think are ok and in both styles? That
would start to whittle down the style issues


>>
> >> Without having a look at each line: Most seems to be just the normal
> >> formatting stuff. Some lines are too long. Some spaces missing around
> >> operators or spaces where they shouldn't be.
> >
> > And then there is removal of spaces on default args ...
> >
> > -def section_macro_map(self, section, nesting_level = 0):
> > +def section_macro_map(self, section, nesting_level=0):
> >
> > I prefer spaces as it is consistent with other uses of spaces around
> operators.
>
> In my view these are detail style discussions. Same is true for most of
> your notes further below (although I agree with some of your opinions).
> But I didn't wanted to trigger any of these. My intention was to give
> some impression about how big such changes would be.
>
> I know that there is quite some python code already in RTEMS. And I'm
> also sure that there are already quite some different python styles in
> RTEMS. At least as many as programmers touching python code. But
> currently the python code is still few enough that it would be possible
> to get a uniform style. And pre-qualification or not: I'm really
> convinced that a fixed style would be a good thing for the code base in
> general.
>
> I'm quite indifferent which style we use but do you really think that it
> is a good idea to roll our own RTEMS-Python-style instead of using one
> of the big ones (like PEP8, Google or some other that might is a better
> fit for existing code). Rolling our own would lead to:
>

In spite of your points below, taking one of those with a few exceptions
isn't a big deal. We have lots of people who have written C code for RTEMS
and only a few who have written any Python.

One thing from the pre-qualification effort that you have forgotten is that
we are supposed to document our decisions and rationale. If we say we use
Python style X with 

Re: Requirement Document generator tool

[Christian Mauderer]

On 28/01/2020 11:41, Chris Johns wrote:
> 
> 
> On 28/1/20 8:14 pm, Christian Mauderer wrote:
>> On 28/01/2020 01:42, Chris Johns wrote:
>>> On 28/1/20 10:48 am, Joel Sherrill wrote:
 On Mon, Jan 27, 2020 at 4:18 PM Chris Johns >>> > wrote:

 On 24/1/20 9:57 pm, Jose Valdez wrote:
 > In fact these tools target the pre-qualified project.

 Do you see this as different to the RTEMS project?

 > Since it was Sebastian who suggested to create this set of python 
 tools,

 I think Sebastian is wanting a smooth path for these tools into the 
 project.

 > I think the idea was to standardize the use of python not only for 
 this
 project, but also for other python written code in RTEMS community. 
 This has
 the advantages that every written python code is standard, but has the
 drawbacks:
 >
 > -> old written code would need to be adapted to the standards.

 How different to the proposed coding standard is the existing code? 
 Why not base
 the coding standard on what exists in the code base?

 This is a very important question.

 Have you evaluated the size of the task to update the existing code? 
 How would
 get such changes for the rtems-tools and the RSB be tested and 
 integrated back
 into the project? This apporach seems like a huge review task for me.

 It could be or it may turn out that there isn't much changed. Without 
 someone
 running the reformatter and reporting, we won't know.
>>>
>>> Running a reformatter may give you an idea of the scale however I have some
>>> concerns as Python's logic is based on indent levels and a missed level 
>>> changes
>>> the logic. I am not sure how you know such a tool is safe to use unless you
>>> review all the changes.
>>
>> To get some statistics for that I tried using yapf on the rtems-tools
>> repo. I excluded everything with  "patch-gdb-python", "tftpy" and
>> "asciidoc" in the name. You can see the yapf call in the commit message.
> 
> The doxygen is from waf. Thomas knows how to write python.

OK. I missed that one. The patches were just a test to get some
orientation about what it would mean.

Please also note: I don't want to doubt the python knowledge of anyone.

> 
>> Google style:
>> https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/f8202a53ca043d23732d79ab217071e975dc35ad/0001-FIXME-Format-in-Google-style.patch
>>
>> PEP8 Style:
>> https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/c97c77762f65041d9244bc8bc64b90ec698735bf/0001-FIXME-Reformat-using-PEP8.patch
>>
>> In both cases it's about 3000 lines removed and 3700 lines added.
>>
>> Without having a look at each line: Most seems to be just the normal
>> formatting stuff. Some lines are too long. Some spaces missing around
>> operators or spaces where they shouldn't be.
> 
> And then there is removal of spaces on default args ...
> 
> -def section_macro_map(self, section, nesting_level = 0):
> +def section_macro_map(self, section, nesting_level=0):
> 
> I prefer spaces as it is consistent with other uses of spaces around 
> operators.

In my view these are detail style discussions. Same is true for most of
your notes further below (although I agree with some of your opinions).
But I didn't wanted to trigger any of these. My intention was to give
some impression about how big such changes would be.

I know that there is quite some python code already in RTEMS. And I'm
also sure that there are already quite some different python styles in
RTEMS. At least as many as programmers touching python code. But
currently the python code is still few enough that it would be possible
to get a uniform style. And pre-qualification or not: I'm really
convinced that a fixed style would be a good thing for the code base in
general.

I'm quite indifferent which style we use but do you really think that it
is a good idea to roll our own RTEMS-Python-style instead of using one
of the big ones (like PEP8, Google or some other that might is a better
fit for existing code). Rolling our own would lead to:

1. Long discussions about what is THE RIGHT STYLE (already started in
this mail).

2. A lot of problems that there is no tool that formats or checks
whether code is in THE RIGHT STYLE.

3. More discussions later because some other developer thinks that THE
RIGHT STYLE is wrong.

> 
>> Some lists written with
>> more or less line breaks.
> 
> Yes to make them readable. This is on purpose. I do not agree with this ..
> 
> -return cfgs + ['objcopy',
> -   'arch',
> -   'vendor',
> -   'board',
> -   'config_name',
> -   'first_stage',
> -   'second_stage',
> -   'kernel_conv

Re: Requirement Document generator tool

[Chris Johns]

On 28/1/20 8:14 pm, Christian Mauderer wrote:
> On 28/01/2020 01:42, Chris Johns wrote:
>> On 28/1/20 10:48 am, Joel Sherrill wrote:
>>> On Mon, Jan 27, 2020 at 4:18 PM Chris Johns >> > wrote:
>>>
>>> On 24/1/20 9:57 pm, Jose Valdez wrote:
>>> > In fact these tools target the pre-qualified project.
>>>
>>> Do you see this as different to the RTEMS project?
>>>
>>> > Since it was Sebastian who suggested to create this set of python 
>>> tools,
>>>
>>> I think Sebastian is wanting a smooth path for these tools into the 
>>> project.
>>>
>>> > I think the idea was to standardize the use of python not only for 
>>> this
>>> project, but also for other python written code in RTEMS community. 
>>> This has
>>> the advantages that every written python code is standard, but has the
>>> drawbacks:
>>> >
>>> > -> old written code would need to be adapted to the standards.
>>>
>>> How different to the proposed coding standard is the existing code? Why 
>>> not base
>>> the coding standard on what exists in the code base?
>>>
>>> This is a very important question.
>>>
>>> Have you evaluated the size of the task to update the existing code? 
>>> How would
>>> get such changes for the rtems-tools and the RSB be tested and 
>>> integrated back
>>> into the project? This apporach seems like a huge review task for me.
>>>
>>> It could be or it may turn out that there isn't much changed. Without 
>>> someone
>>> running the reformatter and reporting, we won't know.
>>
>> Running a reformatter may give you an idea of the scale however I have some
>> concerns as Python's logic is based on indent levels and a missed level 
>> changes
>> the logic. I am not sure how you know such a tool is safe to use unless you
>> review all the changes.
> 
> To get some statistics for that I tried using yapf on the rtems-tools
> repo. I excluded everything with  "patch-gdb-python", "tftpy" and
> "asciidoc" in the name. You can see the yapf call in the commit message.

The doxygen is from waf. Thomas knows how to write python.

> Google style:
> https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/f8202a53ca043d23732d79ab217071e975dc35ad/0001-FIXME-Format-in-Google-style.patch
> 
> PEP8 Style:
> https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/c97c77762f65041d9244bc8bc64b90ec698735bf/0001-FIXME-Reformat-using-PEP8.patch
> 
> In both cases it's about 3000 lines removed and 3700 lines added.
> 
> Without having a look at each line: Most seems to be just the normal
> formatting stuff. Some lines are too long. Some spaces missing around
> operators or spaces where they shouldn't be.

And then there is removal of spaces on default args ...

-def section_macro_map(self, section, nesting_level = 0):
+def section_macro_map(self, section, nesting_level=0):

I prefer spaces as it is consistent with other uses of spaces around operators.

> Some lists written with
> more or less line breaks.

Yes to make them readable. This is on purpose. I do not agree with this ..

-return cfgs + ['objcopy',
-   'arch',
-   'vendor',
-   'board',
-   'config_name',
-   'first_stage',
-   'second_stage',
-   'kernel_converter']
+return cfgs + [
+'objcopy', 'arch', 'vendor', 'board', 'config_name', 'first_stage',
+'second_stage', 'kernel_converter'
+]


> Some extra line breaks added between functions.

Yes, this seems to be some that happens. No spaces when setting an argument and
then these spaces between functions and methods.

> But I also noted a problem for python 3 compatibility: There are some
> tab to space conversions. As far as I know python 3 throws at least a
> warning or maybe an error if it finds tabs.

Yes this happens and there are some cases in the code still.

I do not like this ...

-argsp.add_argument('--net-boot-file',
-   help = 'network boot file (default: %(default)r).',
-   type = str, default = 'rtems.img')
-argsp.add_argument('--net-boot-fdt',
-   help = 'network boot load a fdt file (default:
%(default)r).',
-   type = str, default = None)
+argsp.add_argument(
+'--net-boot-file',
+help='network boot file (default: %(default)r).',
+type=str,
+default='rtems.img')
+argsp.add_argument(
+'--net-boot-fdt',
+help='network boot load a fdt file (default: %(default)r).',
+type=str,
+default=None)

This is a blocker for me ...

-'_ncpus': ('none','none', ncpus),
-'_os':('none','none', 'darwin'),
-'_host':  ('triplet', 're

Re: Requirement Document generator tool

[Christian Mauderer]

On 28/01/2020 01:42, Chris Johns wrote:
> On 28/1/20 10:48 am, Joel Sherrill wrote:
>> On Mon, Jan 27, 2020 at 4:18 PM Chris Johns > > wrote:
>>
>> On 24/1/20 9:57 pm, Jose Valdez wrote:
>> > In fact these tools target the pre-qualified project.
>>
>> Do you see this as different to the RTEMS project?
>>
>> > Since it was Sebastian who suggested to create this set of python 
>> tools,
>>
>> I think Sebastian is wanting a smooth path for these tools into the 
>> project.
>>
>> > I think the idea was to standardize the use of python not only for this
>> project, but also for other python written code in RTEMS community. This 
>> has
>> the advantages that every written python code is standard, but has the
>> drawbacks:
>> >
>> > -> old written code would need to be adapted to the standards.
>>
>> How different to the proposed coding standard is the existing code? Why 
>> not base
>> the coding standard on what exists in the code base?
>>
>> This is a very important question.
>>
>> Have you evaluated the size of the task to update the existing code? How 
>> would
>> get such changes for the rtems-tools and the RSB be tested and 
>> integrated back
>> into the project? This apporach seems like a huge review task for me.
>>
>> It could be or it may turn out that there isn't much changed. Without someone
>> running the reformatter and reporting, we won't know.
> 
> Running a reformatter may give you an idea of the scale however I have some
> concerns as Python's logic is based on indent levels and a missed level 
> changes
> the logic. I am not sure how you know such a tool is safe to use unless you
> review all the changes.

To get some statistics for that I tried using yapf on the rtems-tools
repo. I excluded everything with  "patch-gdb-python", "tftpy" and
"asciidoc" in the name. You can see the yapf call in the commit message.

Google style:
https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/f8202a53ca043d23732d79ab217071e975dc35ad/0001-FIXME-Format-in-Google-style.patch

PEP8 Style:
https://gist.githubusercontent.com/c-mauderer/18746eaa8f3a077adddfe35c1f7b5194/raw/c97c77762f65041d9244bc8bc64b90ec698735bf/0001-FIXME-Reformat-using-PEP8.patch

In both cases it's about 3000 lines removed and 3700 lines added.

Without having a look at each line: Most seems to be just the normal
formatting stuff. Some lines are too long. Some spaces missing around
operators or spaces where they shouldn't be. Some lists written with
more or less line breaks. Some extra line breaks added between functions.

But I also noted a problem for python 3 compatibility: There are some
tab to space conversions. As far as I know python 3 throws at least a
warning or maybe an error if it finds tabs.

> 
>> I tend to think it is worth knowing if this is a monster or a mouse before 
>> making
>> a decision. 
> 
> Yes this is important and also if it is a monster which specific rules make it
> so? It maybe most of the rules are fine.
> 
> If these rules are important to the qual effort I hope they see the value in
> find these answers for us.
> 
>> > Another option could be leave it as it is and only do this for new 
>> written
>> code.
>>
>> It would be confusing to any new user to the code to have code written 
>> to a
>> standard and code that is not? If you edit the old code is it to the new
>> standard? If you edit an old file do you need to update the whole file?
>>
>> If we accept a standard, then it is all or nothing. I'm going to sound like 
>> a 
>> cranky old man but we have said things like this before and regretted it 
>> every time. Consistency is critical.
> 
> If you are a cranky old man then that must make me one as well. Ah the youth 
> of
> today ... :)
> 
>> Quick run of sloccount for a baseline
>>
>> + rtems-tools - 
>> Totals grouped by language (dominant language first):
>> ansic:        47237 (49.86%)
>> cpp:          25837 (27.27%)
>> python:       21227 (22.40%)
>> sh:             442 (0.47%)
> 
> Nice start. A more accurate report on this code would mean removing the 
> imported
> pieces.
> 
>> + rtems-source-builder/source-builder - 
>> SLOC    Directory       SLOC-by-Language (Sorted)
>> 14314   sb              python=14169,sh=145
>> 65      top_dir         sh=65
>> 0       config          (none)
> 
> There are the .cfg and .bset files.
> 
>> 0       patches         (none)
>>
>> So we have about 35K SLOC or Python by that.
>>
>> No idea how the new standard versus the old looks. I thought Python had a 
>> consistent
>> style but I could be very wrong. :(
> 
> I am not sure, I have never looked. My python skills have developed producing
> these tools and this code. I know doc comments are missing.
> 
>> > -> at some point some tools need to be upgraded (ex: python 3.7 will
>> become unusable in 2030 Operating systems).
>>
>> I am not sure how this relates. Yes it wil

Re: Requirement Document generator tool

[Chris Johns]

On 28/1/20 10:48 am, Joel Sherrill wrote:
> On Mon, Jan 27, 2020 at 4:18 PM Chris Johns  > wrote:
> 
> On 24/1/20 9:57 pm, Jose Valdez wrote:
> > In fact these tools target the pre-qualified project.
> 
> Do you see this as different to the RTEMS project?
> 
> > Since it was Sebastian who suggested to create this set of python tools,
> 
> I think Sebastian is wanting a smooth path for these tools into the 
> project.
> 
> > I think the idea was to standardize the use of python not only for this
> project, but also for other python written code in RTEMS community. This 
> has
> the advantages that every written python code is standard, but has the
> drawbacks:
> >
> > -> old written code would need to be adapted to the standards.
> 
> How different to the proposed coding standard is the existing code? Why 
> not base
> the coding standard on what exists in the code base?
> 
> This is a very important question.
> 
> Have you evaluated the size of the task to update the existing code? How 
> would
> get such changes for the rtems-tools and the RSB be tested and integrated 
> back
> into the project? This apporach seems like a huge review task for me.
> 
> It could be or it may turn out that there isn't much changed. Without someone
> running the reformatter and reporting, we won't know.

Running a reformatter may give you an idea of the scale however I have some
concerns as Python's logic is based on indent levels and a missed level changes
the logic. I am not sure how you know such a tool is safe to use unless you
review all the changes.

> I tend to think it is worth knowing if this is a monster or a mouse before 
> making
> a decision. 

Yes this is important and also if it is a monster which specific rules make it
so? It maybe most of the rules are fine.

If these rules are important to the qual effort I hope they see the value in
find these answers for us.

> > Another option could be leave it as it is and only do this for new 
> written
> code.
> 
> It would be confusing to any new user to the code to have code written to 
> a
> standard and code that is not? If you edit the old code is it to the new
> standard? If you edit an old file do you need to update the whole file?
> 
> If we accept a standard, then it is all or nothing. I'm going to sound like a 
> cranky old man but we have said things like this before and regretted it 
> every time. Consistency is critical.

If you are a cranky old man then that must make me one as well. Ah the youth of
today ... :)

> Quick run of sloccount for a baseline
> 
> + rtems-tools - 
> Totals grouped by language (dominant language first):
> ansic:        47237 (49.86%)
> cpp:          25837 (27.27%)
> python:       21227 (22.40%)
> sh:             442 (0.47%)

Nice start. A more accurate report on this code would mean removing the imported
pieces.

> + rtems-source-builder/source-builder - 
> SLOC    Directory       SLOC-by-Language (Sorted)
> 14314   sb              python=14169,sh=145
> 65      top_dir         sh=65
> 0       config          (none)

There are the .cfg and .bset files.

> 0       patches         (none)
> 
> So we have about 35K SLOC or Python by that.
> 
> No idea how the new standard versus the old looks. I thought Python had a 
> consistent
> style but I could be very wrong. :(

I am not sure, I have never looked. My python skills have developed producing
these tools and this code. I know doc comments are missing.

> > -> at some point some tools need to be upgraded (ex: python 3.7 will
> become unusable in 2030 Operating systems).
> 
> I am not sure how this relates. Yes it will need to update however we 
> need to
> support python2 for user facing tools for a while yet. A lot of what we 
> do and
> how we work is historically driven.
> 
> CentOS 8 was just released in October. None of the big organization users I
> see are using it yet. 
> 
> We need to make a LTS release with 5 on Python 2 as a minimum. I feel strongly
> about that.

And I suspect longer.

> As long as the tools are written in a python agnostic manner, the version 
> won't
> matter.

Yes I agree, we should aim for a middle ground and avoid hitting any of issues
that can arise.

> We need some test cases for the tools to verify them

Yes.

Chris

> > I hope soon to formalize our suggestion to you and then you may review 
> it
> (and propose changes if you find appropriate).
> 
> I suggest working in the open and with us will be more beneficial in the
> long term.
> 
> 
> +1 I can't agree strongly enough. 
> 
> 
> 
> Note, I am assuming the remainder of the email was Christian's. The 
> quoting from
> your email client made it difficult to tell.
> 
> Chris
> ___
> devel mailing list
> devel@rtems.org 
> http://lists.rtems.org/mailman/listinf

Re: Requirement Document generator tool

[Joel Sherrill]

On Mon, Jan 27, 2020 at 4:18 PM Chris Johns  wrote:

> On 24/1/20 9:57 pm, Jose Valdez wrote:
> > In fact these tools target the pre-qualified project.
>
> Do you see this as different to the RTEMS project?
>
> > Since it was Sebastian who suggested to create this set of python tools,
>
> I think Sebastian is wanting a smooth path for these tools into the
> project.
>
> > I think the idea was to standardize the use of python not only for this
> project, but also for other python written code in RTEMS community. This
> has the advantages that every written python code is standard, but has the
> drawbacks:
> >
> > -> old written code would need to be adapted to the standards.
>
> How different to the proposed coding standard is the existing code? Why
> not base
> the coding standard on what exists in the code base?
>

This is a very important question.


>
> Have you evaluated the size of the task to update the existing code? How
> would
> get such changes for the rtems-tools and the RSB be tested and integrated
> back
> into the project? This apporach seems like a huge review task for me.
>

It could be or it may turn out that there isn't much changed. Without
someone running
the reformatter and reporting, we won't know.

I tend to think it is worth knowing if this is a monster or a mouse before
making
a decision.


>
> > Another option could be leave it as it is and only do this for new
> written code.
>
> It would be confusing to any new user to the code to have code written to a
> standard and code that is not? If you edit the old code is it to the new
> standard? If you edit an old file do you need to update the whole file?
>

If we accept a standard, then it is all or nothing. I'm going to sound like
a
cranky old man but we have said things like this before and regretted it
every time. Consistency is critical.

Quick run of sloccount for a baseline

+ rtems-tools -
Totals grouped by language (dominant language first):
ansic:47237 (49.86%)
cpp:  25837 (27.27%)
python:   21227 (22.40%)
sh: 442 (0.47%)

+ rtems-source-builder/source-builder -
SLOCDirectory   SLOC-by-Language (Sorted)
14314   sb  python=14169,sh=145
65  top_dir sh=65
0   config  (none)
0   patches (none)

So we have about 35K SLOC or Python by that.

No idea how the new standard versus the old looks. I thought Python had a
consistent
style but I could be very wrong. :(



>
> > -> at some point some tools need to be upgraded (ex: python 3.7 will
> become unusable in 2030 Operating systems).
>
> I am not sure how this relates. Yes it will need to update however we need
> to
> support python2 for user facing tools for a while yet. A lot of what we do
> and
> how we work is historically driven.
>

CentOS 8 was just released in October. None of the big organization users I
see are using it yet.

We need to make a LTS release with 5 on Python 2 as a minimum. I feel
strongly
about that.

As long as the tools are written in a python agnostic manner, the version
won't matter.

We need some test cases for the tools to verify them


>
> > I hope soon to formalize our suggestion to you and then you may review
> it (and propose changes if you find appropriate).
>
> I suggest working in the open and with us will be more beneficial in the
> long term.
>

+1 I can't agree strongly enough.



> Note, I am assuming the remainder of the email was Christian's. The
> quoting from
> your email client made it difficult to tell.
>
> Chris
> ___
> devel mailing list
> devel@rtems.org
> http://lists.rtems.org/mailman/listinfo/devel
>
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Chris Johns]

On 24/1/20 9:57 pm, Jose Valdez wrote:
> In fact these tools target the pre-qualified project.

Do you see this as different to the RTEMS project?

> Since it was Sebastian who suggested to create this set of python tools,

I think Sebastian is wanting a smooth path for these tools into the project.

> I think the idea was to standardize the use of python not only for this 
> project, but also for other python written code in RTEMS community. This has 
> the advantages that every written python code is standard, but has the 
> drawbacks:
> 
> -> old written code would need to be adapted to the standards. 

How different to the proposed coding standard is the existing code? Why not base
the coding standard on what exists in the code base?

Have you evaluated the size of the task to update the existing code? How would
get such changes for the rtems-tools and the RSB be tested and integrated back
into the project? This apporach seems like a huge review task for me.

> Another option could be leave it as it is and only do this for new written 
> code.

It would be confusing to any new user to the code to have code written to a
standard and code that is not? If you edit the old code is it to the new
standard? If you edit an old file do you need to update the whole file?

> -> at some point some tools need to be upgraded (ex: python 3.7 will become 
> unusable in 2030 Operating systems).

I am not sure how this relates. Yes it will need to update however we need to
support python2 for user facing tools for a while yet. A lot of what we do and
how we work is historically driven.

> I hope soon to formalize our suggestion to you and then you may review it 
> (and propose changes if you find appropriate).

I suggest working in the open and with us will be more beneficial in the long 
term.

Note, I am assuming the remainder of the email was Christian's. The quoting from
your email client made it difficult to tell.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

RE: Requirement Document generator tool

[Jose Valdez]

Hello Everyone,

Sorry for my late reply in this topic, but I have not been able to work on 
this. I will try to reach all the topics/problems raised.

In fact these tools target the pre-qualified project. Since it was Sebastian 
who suggested to create this set of python tools, I think the idea was to 
standardize the use of python not only for this project, but also for other 
python written code in RTEMS community. This has the advantages that every 
written python code is standard, but has the drawbacks:

-> old written code would need to be adapted to the standards. Another option 
could be leave it as it is and only do this for new written code.
-> at some point some tools need to be upgraded (ex: python 3.7 will become 
unusable in 2030 Operating systems).

I hope soon to formalize our suggestion to you and then you may review it (and 
propose changes if you find appropriate).

José



-Original Message-
From: Christian Mauderer [mailto:christian.maude...@embedded-brains.de]
Sent: quinta-feira, 23 de janeiro de 2020 06:42
To: Chris Johns; j...@rtems.org
Cc: Jose Valdez; Gedare Bloom; sebastian huber; devel@rtems.org
Subject: Re: Requirement Document generator tool

On 23/01/2020 00:19, Chris Johns wrote:
> On 22/1/20 7:20 pm, Christian Mauderer wrote:
>> On 22/01/2020 02:39, Chris Johns wrote:
>>> On 21/1/20 8:10 pm, Christian Mauderer wrote:
>>>> On 20/01/2020 22:37, Chris Johns wrote:
>>>>> On 14/1/20 7:18 pm, Christian Mauderer wrote:
>>>>>> On 13/01/2020 18:03, Joel Sherrill wrote:
>>>>>>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>>>>>>> >>>>>> <mailto:christian.maude...@embedded-brains.de>> wrote:
>>>>>>>
>>>>>>> Hello Jose,
>>>>>>>
>>>>>>> On 10/01/2020 17:33, Jose Valdez wrote:
>>>>>>> > Hello,
>>>>>>> >
>>>>>>> > Regarding this topic and to start to define the python tools that
>>>>>>> are more appropriate for the RTEMS community python developments, I
>>>>>>> would like to propose the following tools (to be placed in RTEMS
>>>>>>> Software Engineering manual):
>>>>>>> >
>>>>>>> > Python language version: 3.6 (minimum version for Doorstop 
>>>>>>> 2.0.post2)
>>>>>>> > Python coding style/standard - Google (as suggested by Gedare)
>>>>>>> > Python documentation style - Google docstrings (to be in
>>>>>>> accordance with coding style. Note it is supported by sphinx)
>>>>>>> > Python test approach - unittest (seems to be the standard used by
>>>>>>> python)
>>>>>>> > Python static analysis - pylint (recommended by Google style, see
>>>>>>> http://google.github.io/styleguide/pyguide.html) and mypy (catch
>>>>>>> additional errors) Python code coverage - Coverage.py (seems to be
>>>>>>> the standard used by python)
>>>>>>> > Python third party packages - For now we are using: paramiko,
>>>>>>> pyserial, pexpect, gitpython, but the list is expected to be
>>>>>>> changed. I think here any third package is allowed, as long as the
>>>>>>> license of the package complies with RTEMS project license.
>>>>>>>
>>>>>>> To be future-proof: python3 compatibility would be nearly a 
>>>>>>> must-have
>>>>>>> for any new library.
>>>>>>>
>>>>>>>
>>>>>>> We discussed this somewhere earlier and I thought we decided that if
>>>>>>> it was a core tool required for a "normal user", then it would have to
>>>>>>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
>>>>>>> "pre-qualification user" (pick a name for this role), can be 3.x only
>>>>>>> mainly because there was so much infrastructure required for the
>>>>>>> "requirements user" that already needed Python 3.x.
>>>>>
>>>>> Yes this is correct. Python is used in the rtems-tools repo and any 
>>>>> program in
>>>>> the rtems-tools repo needs to support Python 2 and Python 3 no matter 
>>>>> what role
>>>>> it performs. The rtems-tools repo is not the 

Re: Requirement Document generator tool

[Christian Mauderer]

On 23/01/2020 00:19, Chris Johns wrote:
> On 22/1/20 7:20 pm, Christian Mauderer wrote:
>> On 22/01/2020 02:39, Chris Johns wrote:
>>> On 21/1/20 8:10 pm, Christian Mauderer wrote:
 On 20/01/2020 22:37, Chris Johns wrote:
> On 14/1/20 7:18 pm, Christian Mauderer wrote:
>> On 13/01/2020 18:03, Joel Sherrill wrote:
>>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>>> >> > wrote:
>>>
>>> Hello Jose,
>>>
>>> On 10/01/2020 17:33, Jose Valdez wrote:
>>> > Hello,
>>> >
>>> > Regarding this topic and to start to define the python tools that
>>> are more appropriate for the RTEMS community python developments, I
>>> would like to propose the following tools (to be placed in RTEMS
>>> Software Engineering manual):
>>> >
>>> > Python language version: 3.6 (minimum version for Doorstop 
>>> 2.0.post2)
>>> > Python coding style/standard - Google (as suggested by Gedare)
>>> > Python documentation style - Google docstrings (to be in
>>> accordance with coding style. Note it is supported by sphinx)
>>> > Python test approach - unittest (seems to be the standard used by
>>> python)
>>> > Python static analysis - pylint (recommended by Google style, see
>>> http://google.github.io/styleguide/pyguide.html) and mypy (catch
>>> additional errors) Python code coverage - Coverage.py (seems to be
>>> the standard used by python)
>>> > Python third party packages - For now we are using: paramiko,
>>> pyserial, pexpect, gitpython, but the list is expected to be
>>> changed. I think here any third package is allowed, as long as the
>>> license of the package complies with RTEMS project license.
>>>
>>> To be future-proof: python3 compatibility would be nearly a 
>>> must-have
>>> for any new library.
>>>
>>>
>>> We discussed this somewhere earlier and I thought we decided that if
>>> it was a core tool required for a "normal user", then it would have to
>>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
>>> "pre-qualification user" (pick a name for this role), can be 3.x only 
>>> mainly because there was so much infrastructure required for the
>>> "requirements user" that already needed Python 3.x.
>
> Yes this is correct. Python is used in the rtems-tools repo and any 
> program in
> the rtems-tools repo needs to support Python 2 and Python 3 no matter 
> what role
> it performs. The rtems-tools repo is not the same as doorstop or similar 
> tools
> used to develop RTEMS, they are user tools used to develop application to 
> run on
> RTEMS.

 Does that mean you don't want to have any pre-qualification tools in the
 rtems-tools repo?
>>>
>>> This is not what I am saying. All I am doing is listing what is currently 
>>> needed
>>> when a tool is added to rtems-tools. If a pre-qual tool meets the 
>>> requirements
>>> for inclusion in the tools repo then it can be added.
>>>
 If yes: Where else?
>>>
>>> I do not know. I am not sure what these tools are to do, what interfaces 
>>> they
>>> provide to their users, their complexity and who are the maintainers. Until
>>> these questions are answered I cannot say.
>>>
 I fully agree that every tool that is thought for every day use for 99%
 of the RTEMS users should have Python 2 and 3 support. With no official
 Python 2 support since this year I don't think that this would be a good
 requirement for new specialized areas like pre-qualification. I think
 you agree there?
>>>
>>> I believe Joel has looked at Python 3 on long life operating systems like 
>>> CentOS
>>> and it can be installed and made to work. I am sure there will be cases we 
>>> have
>>> missed and this assumption may not hold.
>>
>> I'm aware that CentOS 7 still has a python 2 as default version. I think
>> CentOS 8 changed that. So we will have to support python 2 at least till
>> EOL of CentOS 7 which is 30 June 2024.
>>
>>>
>>> My experience with Python 2 and 3 is most things we would use can be 
>>> handled by
>>> future imports. With a little care programs can be portable. Doorstop has 
>>> not
>>> been written this way so we have said Python 3 is required when developing
>>> RTEMS. It was never made a requirement for users.
>>
>> OK.
>>
>>>
 On the other hand I wouldn't know a better place then rtems-tools for
 everything produced through the pre-qualification effort. So I'm not a
 fan of saying: Everything in rtems-tools should be Python 2 + 3.
>>>
>>> Let me put you in the position of the rtems-tools maintainer and Python 3 is
>>> mandated plus a number of new pre-qual tools with a variable number of 
>>> packages
>>> are added.
>>>
>>> How would you manage these new dependencies the

Re: Requirement Document generator tool

[Chris Johns]

On 22/1/20 7:20 pm, Christian Mauderer wrote:
> On 22/01/2020 02:39, Chris Johns wrote:
>> On 21/1/20 8:10 pm, Christian Mauderer wrote:
>>> On 20/01/2020 22:37, Chris Johns wrote:
 On 14/1/20 7:18 pm, Christian Mauderer wrote:
> On 13/01/2020 18:03, Joel Sherrill wrote:
>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>> > > wrote:
>>
>> Hello Jose,
>>
>> On 10/01/2020 17:33, Jose Valdez wrote:
>> > Hello,
>> >
>> > Regarding this topic and to start to define the python tools that
>> are more appropriate for the RTEMS community python developments, I
>> would like to propose the following tools (to be placed in RTEMS
>> Software Engineering manual):
>> >
>> > Python language version: 3.6 (minimum version for Doorstop 
>> 2.0.post2)
>> > Python coding style/standard - Google (as suggested by Gedare)
>> > Python documentation style - Google docstrings (to be in
>> accordance with coding style. Note it is supported by sphinx)
>> > Python test approach - unittest (seems to be the standard used by
>> python)
>> > Python static analysis - pylint (recommended by Google style, see
>> http://google.github.io/styleguide/pyguide.html) and mypy (catch
>> additional errors) Python code coverage - Coverage.py (seems to be
>> the standard used by python)
>> > Python third party packages - For now we are using: paramiko,
>> pyserial, pexpect, gitpython, but the list is expected to be
>> changed. I think here any third package is allowed, as long as the
>> license of the package complies with RTEMS project license.
>>
>> To be future-proof: python3 compatibility would be nearly a must-have
>> for any new library.
>>
>>
>> We discussed this somewhere earlier and I thought we decided that if
>> it was a core tool required for a "normal user", then it would have to
>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
>> "pre-qualification user" (pick a name for this role), can be 3.x only 
>> mainly because there was so much infrastructure required for the
>> "requirements user" that already needed Python 3.x.

 Yes this is correct. Python is used in the rtems-tools repo and any 
 program in
 the rtems-tools repo needs to support Python 2 and Python 3 no matter what 
 role
 it performs. The rtems-tools repo is not the same as doorstop or similar 
 tools
 used to develop RTEMS, they are user tools used to develop application to 
 run on
 RTEMS.
>>>
>>> Does that mean you don't want to have any pre-qualification tools in the
>>> rtems-tools repo?
>>
>> This is not what I am saying. All I am doing is listing what is currently 
>> needed
>> when a tool is added to rtems-tools. If a pre-qual tool meets the 
>> requirements
>> for inclusion in the tools repo then it can be added.
>>
>>> If yes: Where else?
>>
>> I do not know. I am not sure what these tools are to do, what interfaces they
>> provide to their users, their complexity and who are the maintainers. Until
>> these questions are answered I cannot say.
>>
>>> I fully agree that every tool that is thought for every day use for 99%
>>> of the RTEMS users should have Python 2 and 3 support. With no official
>>> Python 2 support since this year I don't think that this would be a good
>>> requirement for new specialized areas like pre-qualification. I think
>>> you agree there?
>>
>> I believe Joel has looked at Python 3 on long life operating systems like 
>> CentOS
>> and it can be installed and made to work. I am sure there will be cases we 
>> have
>> missed and this assumption may not hold.
> 
> I'm aware that CentOS 7 still has a python 2 as default version. I think
> CentOS 8 changed that. So we will have to support python 2 at least till
> EOL of CentOS 7 which is 30 June 2024.
> 
>>
>> My experience with Python 2 and 3 is most things we would use can be handled 
>> by
>> future imports. With a little care programs can be portable. Doorstop has not
>> been written this way so we have said Python 3 is required when developing
>> RTEMS. It was never made a requirement for users.
> 
> OK.
> 
>>
>>> On the other hand I wouldn't know a better place then rtems-tools for
>>> everything produced through the pre-qualification effort. So I'm not a
>>> fan of saying: Everything in rtems-tools should be Python 2 + 3.
>>
>> Let me put you in the position of the rtems-tools maintainer and Python 3 is
>> mandated plus a number of new pre-qual tools with a variable number of 
>> packages
>> are added.
>>
>> How would you manage these new dependencies the RSB would encounter?
>>
>> How would you manage the errors at runtime the python programs throw if a
>> dependent package is not install?
>>
>> How would you manag

Re: Requirement Document generator tool

[Christian Mauderer]

On 22/01/2020 02:39, Chris Johns wrote:
> On 21/1/20 8:10 pm, Christian Mauderer wrote:
>> On 20/01/2020 22:37, Chris Johns wrote:
>>> On 14/1/20 7:18 pm, Christian Mauderer wrote:
 On 13/01/2020 18:03, Joel Sherrill wrote:
> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>  > wrote:
>
> Hello Jose,
>
> On 10/01/2020 17:33, Jose Valdez wrote:
> > Hello,
> >
> > Regarding this topic and to start to define the python tools that
> are more appropriate for the RTEMS community python developments, I
> would like to propose the following tools (to be placed in RTEMS
> Software Engineering manual):
> >
> > Python language version: 3.6 (minimum version for Doorstop 
> 2.0.post2)
> > Python coding style/standard - Google (as suggested by Gedare)
> > Python documentation style - Google docstrings (to be in
> accordance with coding style. Note it is supported by sphinx)
> > Python test approach - unittest (seems to be the standard used by
> python)
> > Python static analysis - pylint (recommended by Google style, see
> http://google.github.io/styleguide/pyguide.html) and mypy (catch
> additional errors) Python code coverage - Coverage.py (seems to be
> the standard used by python)
> > Python third party packages - For now we are using: paramiko,
> pyserial, pexpect, gitpython, but the list is expected to be
> changed. I think here any third package is allowed, as long as the
> license of the package complies with RTEMS project license.
>
> To be future-proof: python3 compatibility would be nearly a must-have
> for any new library.
>
>
> We discussed this somewhere earlier and I thought we decided that if
> it was a core tool required for a "normal user", then it would have to
> be Python 2.x and 3.x compatible. Tools for "requirements user" or
> "pre-qualification user" (pick a name for this role), can be 3.x only 
> mainly because there was so much infrastructure required for the
> "requirements user" that already needed Python 3.x.
>>>
>>> Yes this is correct. Python is used in the rtems-tools repo and any program 
>>> in
>>> the rtems-tools repo needs to support Python 2 and Python 3 no matter what 
>>> role
>>> it performs. The rtems-tools repo is not the same as doorstop or similar 
>>> tools
>>> used to develop RTEMS, they are user tools used to develop application to 
>>> run on
>>> RTEMS.
>>
>> Does that mean you don't want to have any pre-qualification tools in the
>> rtems-tools repo?
> 
> This is not what I am saying. All I am doing is listing what is currently 
> needed
> when a tool is added to rtems-tools. If a pre-qual tool meets the requirements
> for inclusion in the tools repo then it can be added.
> 
>> If yes: Where else?
> 
> I do not know. I am not sure what these tools are to do, what interfaces they
> provide to their users, their complexity and who are the maintainers. Until
> these questions are answered I cannot say.
> 
>> I fully agree that every tool that is thought for every day use for 99%
>> of the RTEMS users should have Python 2 and 3 support. With no official
>> Python 2 support since this year I don't think that this would be a good
>> requirement for new specialized areas like pre-qualification. I think
>> you agree there?
> 
> I believe Joel has looked at Python 3 on long life operating systems like 
> CentOS
> and it can be installed and made to work. I am sure there will be cases we 
> have
> missed and this assumption may not hold.

I'm aware that CentOS 7 still has a python 2 as default version. I think
CentOS 8 changed that. So we will have to support python 2 at least till
EOL of CentOS 7 which is 30 June 2024.

> 
> My experience with Python 2 and 3 is most things we would use can be handled 
> by
> future imports. With a little care programs can be portable. Doorstop has not
> been written this way so we have said Python 3 is required when developing
> RTEMS. It was never made a requirement for users.

OK.

> 
>> On the other hand I wouldn't know a better place then rtems-tools for
>> everything produced through the pre-qualification effort. So I'm not a
>> fan of saying: Everything in rtems-tools should be Python 2 + 3.
> 
> Let me put you in the position of the rtems-tools maintainer and Python 3 is
> mandated plus a number of new pre-qual tools with a variable number of 
> packages
> are added.
> 
> How would you manage these new dependencies the RSB would encounter?
> 
> How would you manage the errors at runtime the python programs throw if a
> dependent package is not install?
> 
> How would you manage the deployment of tools as binary packages by those users
> who want to do this? We have users doing this now.
> 
> Is the increase in on-boarding complexity

Re: Requirement Document generator tool

[Chris Johns]

On 21/1/20 8:10 pm, Christian Mauderer wrote:
> On 20/01/2020 22:37, Chris Johns wrote:
>> On 14/1/20 7:18 pm, Christian Mauderer wrote:
>>> On 13/01/2020 18:03, Joel Sherrill wrote:
 On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
 >>> > wrote:

 Hello Jose,

 On 10/01/2020 17:33, Jose Valdez wrote:
 > Hello,
 >
 > Regarding this topic and to start to define the python tools that
 are more appropriate for the RTEMS community python developments, I
 would like to propose the following tools (to be placed in RTEMS
 Software Engineering manual):
 >
 > Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
 > Python coding style/standard - Google (as suggested by Gedare)
 > Python documentation style - Google docstrings (to be in
 accordance with coding style. Note it is supported by sphinx)
 > Python test approach - unittest (seems to be the standard used by
 python)
 > Python static analysis - pylint (recommended by Google style, see
 http://google.github.io/styleguide/pyguide.html) and mypy (catch
 additional errors) Python code coverage - Coverage.py (seems to be
 the standard used by python)
 > Python third party packages - For now we are using: paramiko,
 pyserial, pexpect, gitpython, but the list is expected to be
 changed. I think here any third package is allowed, as long as the
 license of the package complies with RTEMS project license.

 To be future-proof: python3 compatibility would be nearly a must-have
 for any new library.


 We discussed this somewhere earlier and I thought we decided that if
 it was a core tool required for a "normal user", then it would have to
 be Python 2.x and 3.x compatible. Tools for "requirements user" or
 "pre-qualification user" (pick a name for this role), can be 3.x only 
 mainly because there was so much infrastructure required for the
 "requirements user" that already needed Python 3.x.
>>
>> Yes this is correct. Python is used in the rtems-tools repo and any program 
>> in
>> the rtems-tools repo needs to support Python 2 and Python 3 no matter what 
>> role
>> it performs. The rtems-tools repo is not the same as doorstop or similar 
>> tools
>> used to develop RTEMS, they are user tools used to develop application to 
>> run on
>> RTEMS.
> 
> Does that mean you don't want to have any pre-qualification tools in the
> rtems-tools repo?

This is not what I am saying. All I am doing is listing what is currently needed
when a tool is added to rtems-tools. If a pre-qual tool meets the requirements
for inclusion in the tools repo then it can be added.

> If yes: Where else?

I do not know. I am not sure what these tools are to do, what interfaces they
provide to their users, their complexity and who are the maintainers. Until
these questions are answered I cannot say.

> I fully agree that every tool that is thought for every day use for 99%
> of the RTEMS users should have Python 2 and 3 support. With no official
> Python 2 support since this year I don't think that this would be a good
> requirement for new specialized areas like pre-qualification. I think
> you agree there?

I believe Joel has looked at Python 3 on long life operating systems like CentOS
and it can be installed and made to work. I am sure there will be cases we have
missed and this assumption may not hold.

My experience with Python 2 and 3 is most things we would use can be handled by
future imports. With a little care programs can be portable. Doorstop has not
been written this way so we have said Python 3 is required when developing
RTEMS. It was never made a requirement for users.

> On the other hand I wouldn't know a better place then rtems-tools for
> everything produced through the pre-qualification effort. So I'm not a
> fan of saying: Everything in rtems-tools should be Python 2 + 3.

Let me put you in the position of the rtems-tools maintainer and Python 3 is
mandated plus a number of new pre-qual tools with a variable number of packages
are added.

How would you manage these new dependencies the RSB would encounter?

How would you manage the errors at runtime the python programs throw if a
dependent package is not install?

How would you manage the deployment of tools as binary packages by those users
who want to do this? We have users doing this now.

Is the increase in on-boarding complexity for new RTEMS users warranted for a
very small and highly skill section of the user community?

The current rtems-tools model combined with the RSB currently handle these
things. That is, the dependencies for the RSB are simple, small and known. The
RTEMS tools once built can be copied to a new machine (the tar packaging options
in the RSB have been present from the start).

The exper

Re: Requirement Document generator tool

[Gedare Bloom]

On Tue, Jan 21, 2020 at 2:10 AM Christian Mauderer
 wrote:
>
> On 20/01/2020 22:37, Chris Johns wrote:
> > On 14/1/20 7:18 pm, Christian Mauderer wrote:
> >> On 13/01/2020 18:03, Joel Sherrill wrote:
> >>>
> >>>
> >>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
> >>>  >>> > wrote:
> >>>
> >>> Hello Jose,
> >>>
> >>> On 10/01/2020 17:33, Jose Valdez wrote:
> >>> > Hello,
> >>> >
> >>> > Regarding this topic and to start to define the python tools that
> >>> are more appropriate for the RTEMS community python developments, I
> >>> would like to propose the following tools (to be placed in RTEMS
> >>> Software Engineering manual):
> >>> >
> >>> > Python language version: 3.6 (minimum version for Doorstop 
> >>> 2.0.post2)
> >>> > Python coding style/standard - Google (as suggested by Gedare)
> >>> > Python documentation style - Google docstrings (to be in
> >>> accordance with coding style. Note it is supported by sphinx)
> >>> > Python test approach - unittest (seems to be the standard used by
> >>> python)
> >>> > Python static analysis - pylint (recommended by Google style, see
> >>> http://google.github.io/styleguide/pyguide.html) and mypy (catch
> >>> additional errors) Python code coverage - Coverage.py (seems to be
> >>> the standard used by python)
> >>> > Python third party packages - For now we are using: paramiko,
> >>> pyserial, pexpect, gitpython, but the list is expected to be
> >>> changed. I think here any third package is allowed, as long as the
> >>> license of the package complies with RTEMS project license.
> >>>
> >>> To be future-proof: python3 compatibility would be nearly a must-have
> >>> for any new library.
> >>>
> >>>
> >>> We discussed this somewhere earlier and I thought we decided that if
> >>> it was a core tool required for a "normal user", then it would have to
> >>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
> >>> "pre-qualification user" (pick a name for this role), can be 3.x only
> >>> mainly because there was so much infrastructure required for the
> >>> "requirements user" that already needed Python 3.x.
> >
> > Yes this is correct. Python is used in the rtems-tools repo and any program 
> > in
> > the rtems-tools repo needs to support Python 2 and Python 3 no matter what 
> > role
> > it performs. The rtems-tools repo is not the same as doorstop or similar 
> > tools
> > used to develop RTEMS, they are user tools used to develop application to 
> > run on
> > RTEMS.
>
> Does that mean you don't want to have any pre-qualification tools in the
> rtems-tools repo? If yes: Where else?
>

Maybe we need a new repo, e.g., 'rtems-devtools.git' to hold
developer-only tooling.

> I fully agree that every tool that is thought for every day use for 99%
> of the RTEMS users should have Python 2 and 3 support. With no official
> Python 2 support since this year I don't think that this would be a good
> requirement for new specialized areas like pre-qualification. I think
> you agree there?
>
> On the other hand I wouldn't know a better place then rtems-tools for
> everything produced through the pre-qualification effort. So I'm not a
> fan of saying: Everything in rtems-tools should be Python 2 + 3.
>
> >
> > It is not clear to me what role the original python specification Jose 
> > listed is
> > for. I will restrict my comments to rtems-tools or more generally any tool
> > installed into the user's tools prefix for use when developing RTEMS 
> > applications.
> >
> > For a user focused tool:
> >
> > - Python is 2 and 3
> > - No external third party packages, all support needs to be internal.
>
> What about stuff like pygdb (which is already used in rtems-tools)?
>
> > - Fully portable for all supported host platforms, that is Windows, MacOS,
> > FreeBSD and Linux.
> >
> > The second item is important. If user installed tools start to depend on 3rd
> > party packages the deployment complexity for us rises and the process 
> > becomes
> > complex.
> >
> > While on the topic of 3rd party packages I do not agree with Jose's view any
> > package can be used if the license it suitable. Package complexity, it's
> > dependencies and portability need to be considered, for example gitpython 
> > uses
> > gitdb and this is python code to directly read and write a git repo. I 
> > would not
> > like any of our repo's being modified by that code so I would prefer we do 
> > not
> > use it. If we need to extract data I prefer the solution in the rtems-tools
> > toolkit so we should also consider using existing support first.
>
> OK. So rule for third party is:
>
> For everything that is used by 99% of our users: Basically don't use any
> third party. Exceptions for that user group have to be discussed (and
> are most likely rejected) on the mailing list.
>
> For special use cases:
> - Should be avoided if a build-in can

Re: Requirement Document generator tool

[Christian Mauderer]

On 20/01/2020 22:37, Chris Johns wrote:
> On 14/1/20 7:18 pm, Christian Mauderer wrote:
>> On 13/01/2020 18:03, Joel Sherrill wrote:
>>>
>>>
>>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>>> >> > wrote:
>>>
>>> Hello Jose,
>>>
>>> On 10/01/2020 17:33, Jose Valdez wrote:
>>> > Hello,
>>> >
>>> > Regarding this topic and to start to define the python tools that
>>> are more appropriate for the RTEMS community python developments, I
>>> would like to propose the following tools (to be placed in RTEMS
>>> Software Engineering manual):
>>> >
>>> > Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
>>> > Python coding style/standard - Google (as suggested by Gedare)
>>> > Python documentation style - Google docstrings (to be in
>>> accordance with coding style. Note it is supported by sphinx)
>>> > Python test approach - unittest (seems to be the standard used by
>>> python)
>>> > Python static analysis - pylint (recommended by Google style, see
>>> http://google.github.io/styleguide/pyguide.html) and mypy (catch
>>> additional errors) Python code coverage - Coverage.py (seems to be
>>> the standard used by python)
>>> > Python third party packages - For now we are using: paramiko,
>>> pyserial, pexpect, gitpython, but the list is expected to be
>>> changed. I think here any third package is allowed, as long as the
>>> license of the package complies with RTEMS project license.
>>>
>>> To be future-proof: python3 compatibility would be nearly a must-have
>>> for any new library.
>>>
>>>
>>> We discussed this somewhere earlier and I thought we decided that if
>>> it was a core tool required for a "normal user", then it would have to
>>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
>>> "pre-qualification user" (pick a name for this role), can be 3.x only 
>>> mainly because there was so much infrastructure required for the
>>> "requirements user" that already needed Python 3.x.
> 
> Yes this is correct. Python is used in the rtems-tools repo and any program in
> the rtems-tools repo needs to support Python 2 and Python 3 no matter what 
> role
> it performs. The rtems-tools repo is not the same as doorstop or similar tools
> used to develop RTEMS, they are user tools used to develop application to run 
> on
> RTEMS.

Does that mean you don't want to have any pre-qualification tools in the
rtems-tools repo? If yes: Where else?

I fully agree that every tool that is thought for every day use for 99%
of the RTEMS users should have Python 2 and 3 support. With no official
Python 2 support since this year I don't think that this would be a good
requirement for new specialized areas like pre-qualification. I think
you agree there?

On the other hand I wouldn't know a better place then rtems-tools for
everything produced through the pre-qualification effort. So I'm not a
fan of saying: Everything in rtems-tools should be Python 2 + 3.

> 
> It is not clear to me what role the original python specification Jose listed 
> is
> for. I will restrict my comments to rtems-tools or more generally any tool
> installed into the user's tools prefix for use when developing RTEMS 
> applications.
> 
> For a user focused tool:
> 
> - Python is 2 and 3
> - No external third party packages, all support needs to be internal.

What about stuff like pygdb (which is already used in rtems-tools)?

> - Fully portable for all supported host platforms, that is Windows, MacOS,
> FreeBSD and Linux.
> 
> The second item is important. If user installed tools start to depend on 3rd
> party packages the deployment complexity for us rises and the process becomes
> complex.
> 
> While on the topic of 3rd party packages I do not agree with Jose's view any
> package can be used if the license it suitable. Package complexity, it's
> dependencies and portability need to be considered, for example gitpython uses
> gitdb and this is python code to directly read and write a git repo. I would 
> not
> like any of our repo's being modified by that code so I would prefer we do not
> use it. If we need to extract data I prefer the solution in the rtems-tools
> toolkit so we should also consider using existing support first.

OK. So rule for third party is:

For everything that is used by 99% of our users: Basically don't use any
third party. Exceptions for that user group have to be discussed (and
are most likely rejected) on the mailing list.

For special use cases:
- Should be avoided if a build-in can be used instead.
- Only if there isn't another package for the same purpose already in use.
- Open source license that is not putting restrictions on our code (like
a code generator that would put the generated code under GPL).
- Portable to all mayor host systems (for RTEMS that's Windows, Linux,
FreeBSD, MacOS).
- Libraries that touch critical infrastructure would ha

Re: Requirement Document generator tool

[Chris Johns]

On 14/1/20 7:18 pm, Christian Mauderer wrote:
> On 13/01/2020 18:03, Joel Sherrill wrote:
>>
>>
>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>> > > wrote:
>>
>> Hello Jose,
>>
>> On 10/01/2020 17:33, Jose Valdez wrote:
>> > Hello,
>> >
>> > Regarding this topic and to start to define the python tools that
>> are more appropriate for the RTEMS community python developments, I
>> would like to propose the following tools (to be placed in RTEMS
>> Software Engineering manual):
>> >
>> > Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
>> > Python coding style/standard - Google (as suggested by Gedare)
>> > Python documentation style - Google docstrings (to be in
>> accordance with coding style. Note it is supported by sphinx)
>> > Python test approach - unittest (seems to be the standard used by
>> python)
>> > Python static analysis - pylint (recommended by Google style, see
>> http://google.github.io/styleguide/pyguide.html) and mypy (catch
>> additional errors) Python code coverage - Coverage.py (seems to be
>> the standard used by python)
>> > Python third party packages - For now we are using: paramiko,
>> pyserial, pexpect, gitpython, but the list is expected to be
>> changed. I think here any third package is allowed, as long as the
>> license of the package complies with RTEMS project license.
>>
>> To be future-proof: python3 compatibility would be nearly a must-have
>> for any new library.
>>
>>
>> We discussed this somewhere earlier and I thought we decided that if
>> it was a core tool required for a "normal user", then it would have to
>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
>> "pre-qualification user" (pick a name for this role), can be 3.x only 
>> mainly because there was so much infrastructure required for the
>> "requirements user" that already needed Python 3.x.

Yes this is correct. Python is used in the rtems-tools repo and any program in
the rtems-tools repo needs to support Python 2 and Python 3 no matter what role
it performs. The rtems-tools repo is not the same as doorstop or similar tools
used to develop RTEMS, they are user tools used to develop application to run on
RTEMS.

It is not clear to me what role the original python specification Jose listed is
for. I will restrict my comments to rtems-tools or more generally any tool
installed into the user's tools prefix for use when developing RTEMS 
applications.

For a user focused tool:

- Python is 2 and 3
- No external third party packages, all support needs to be internal.
- Fully portable for all supported host platforms, that is Windows, MacOS,
FreeBSD and Linux.

The second item is important. If user installed tools start to depend on 3rd
party packages the deployment complexity for us rises and the process becomes
complex.

While on the topic of 3rd party packages I do not agree with Jose's view any
package can be used if the license it suitable. Package complexity, it's
dependencies and portability need to be considered, for example gitpython uses
gitdb and this is python code to directly read and write a git repo. I would not
like any of our repo's being modified by that code so I would prefer we do not
use it. If we need to extract data I prefer the solution in the rtems-tools
toolkit so we should also consider using existing support first.

> 
> Yes, sorry. I used the wrong wording. What I meant was that we should
> avoid libraries that support only python 2. But you have added a lot of
> relevant details to this.
> 
>>
>> I think this discussion should also include "documentation producer"
>> because that requires Python 3.x as well.
>>
>> We just need to be conscious of the user roles that the tools are
>> supporting and justify the minimum. This gives us at least 3 categories
>> of tools with potentially different minimum hosting requirements. For
>> example, we may not care if a certain set of tools runs on Mingw.
>>
>> Obviously documented somewhere.
>>  
>>
>>
>> I don't think that there are heavy license problems. The python packages
>> are most likely only used in tools. So as long as they are open enough
>> to use them on all host platforms for all development situations that
>> shouldn't be a problem. Exception: If the package is used to generate
>> code with a specific license. That generated code would have to be
>> compatible with the RTEMS license.
>>
>>
>> +1 and very much non-viral and no advertising for generated code.
>>
>>
>> >
>> > Please tell me your opinions for each python tool and then we can
>> start to agree on what to use. Note that once this is defined, our
>> Qualification software will comply with the RTEMS community chosen
>> tools.
>>
>> If no one objects I would suggest to start documenting them in the RTEMS
>> Software Engineering man

Re: Requirement Document generator tool

[Christian Mauderer]

On 13/01/2020 18:03, Joel Sherrill wrote:
> 
> 
> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>  <mailto:christian.maude...@embedded-brains.de>> wrote:
> 
> Hello Jose,
> 
> On 10/01/2020 17:33, Jose Valdez wrote:
> > Hello,
> >
> > Regarding this topic and to start to define the python tools that
> are more appropriate for the RTEMS community python developments, I
> would like to propose the following tools (to be placed in RTEMS
> Software Engineering manual):
> >
> > Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
> > Python coding style/standard - Google (as suggested by Gedare)
> > Python documentation style - Google docstrings (to be in
> accordance with coding style. Note it is supported by sphinx)
> > Python test approach - unittest (seems to be the standard used by
> python)
> > Python static analysis - pylint (recommended by Google style, see
> http://google.github.io/styleguide/pyguide.html) and mypy (catch
> additional errors) Python code coverage - Coverage.py (seems to be
> the standard used by python)
> > Python third party packages - For now we are using: paramiko,
> pyserial, pexpect, gitpython, but the list is expected to be
> changed. I think here any third package is allowed, as long as the
> license of the package complies with RTEMS project license.
> 
> To be future-proof: python3 compatibility would be nearly a must-have
> for any new library.
> 
> 
> We discussed this somewhere earlier and I thought we decided that if
> it was a core tool required for a "normal user", then it would have to
> be Python 2.x and 3.x compatible. Tools for "requirements user" or
> "pre-qualification user" (pick a name for this role), can be 3.x only 
> mainly because there was so much infrastructure required for the
> "requirements user" that already needed Python 3.x.

Yes, sorry. I used the wrong wording. What I meant was that we should
avoid libraries that support only python 2. But you have added a lot of
relevant details to this.

> 
> I think this discussion should also include "documentation producer"
> because that requires Python 3.x as well.
> 
> We just need to be conscious of the user roles that the tools are
> supporting and justify the minimum. This gives us at least 3 categories
> of tools with potentially different minimum hosting requirements. For
> example, we may not care if a certain set of tools runs on Mingw.
> 
> Obviously documented somewhere.
>  
> 
> 
> I don't think that there are heavy license problems. The python packages
> are most likely only used in tools. So as long as they are open enough
> to use them on all host platforms for all development situations that
> shouldn't be a problem. Exception: If the package is used to generate
> code with a specific license. That generated code would have to be
> compatible with the RTEMS license.
> 
> 
> +1 and very much non-viral and no advertising for generated code.
> 
> 
> >
> > Please tell me your opinions for each python tool and then we can
> start to agree on what to use. Note that once this is defined, our
> Qualification software will comply with the RTEMS community chosen
> tools.
> 
> If no one objects I would suggest to start documenting them in the RTEMS
> Software Engineering manual. A small example project in the rtems-tools
> repository that uses the suggested code checkers (meaning: calls them
> during a waf build or a special 'waf check' target) would be great too.
> That could be used for all future python based tools that are added to
> rtems-tools (which includes the whole qualification tools).
> 
> 
> Yes. It must be enforced. Whether it is a special step or not is up for
> discussion.
> My first thought is that a special step is easy to skip.
> 
> Also can "check" be done as a pre-commit hook? 

Possible. But I'm not sure. We might can try it as soon as we have it
running in an example.

> 
> And when the style is agreed to, should existing code be auto reformatted
> to meet it?

I think this could be migrated on a per-tool basis. If we have one
project that is a sample for all future tools the checkers can be back
ported to existing tools when there is time or a good opportunity.

> 
> --joel

Best regards

Christian

> 
> 
> Best regards
> 
>     Christian
> 
> >
> > Best regards
> >
> > José
> >
> > -Original M

Re: Requirement Document generator tool

[Gedare Bloom]

On Mon, Jan 13, 2020 at 9:06 AM Christian Mauderer
 wrote:
>
> Hello Jose,
>
> On 10/01/2020 17:33, Jose Valdez wrote:
> > Hello,
> >
> > Regarding this topic and to start to define the python tools that are more 
> > appropriate for the RTEMS community python developments, I would like to 
> > propose the following tools (to be placed in RTEMS Software Engineering 
> > manual):
> >
> > Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
> > Python coding style/standard - Google (as suggested by Gedare)
> > Python documentation style - Google docstrings (to be in accordance with 
> > coding style. Note it is supported by sphinx)
> > Python test approach - unittest (seems to be the standard used by python)
> > Python static analysis - pylint (recommended by Google style, see 
> > http://google.github.io/styleguide/pyguide.html) and mypy (catch additional 
> > errors) Python code coverage - Coverage.py (seems to be the standard used 
> > by python)
> > Python third party packages - For now we are using: paramiko, pyserial, 
> > pexpect, gitpython, but the list is expected to be changed. I think here 
> > any third package is allowed, as long as the license of the package 
> > complies with RTEMS project license.
>
> To be future-proof: python3 compatibility would be nearly a must-have
> for any new library.
>
> I don't think that there are heavy license problems. The python packages
> are most likely only used in tools. So as long as they are open enough
> to use them on all host platforms for all development situations that
> shouldn't be a problem. Exception: If the package is used to generate
> code with a specific license. That generated code would have to be
> compatible with the RTEMS license.
>
Yes, I would like something similar to Christian's wording instead of
specifing a list of 3rd party packages.

I think we can start with the proposed set of tools/requirements. If
problems in the coding style arise, we can evolve. The Google Python
Style Guide is given as a single markdown file in
https://github.com/google/styleguide/pyguide.md. It is licensed CC-By
3.0. We should mirror it on our infrastructure and track the
version/changes.

> >
> > Please tell me your opinions for each python tool and then we can start to 
> > agree on what to use. Note that once this is defined, our Qualification 
> > software will comply with the RTEMS community chosen tools.
>
> If no one objects I would suggest to start documenting them in the RTEMS
> Software Engineering manual. A small example project in the rtems-tools
> repository that uses the suggested code checkers (meaning: calls them
> during a waf build or a special 'waf check' target) would be great too.
> That could be used for all future python based tools that are added to
> rtems-tools (which includes the whole qualification tools).
>
> Best regards
>
> Christian
>
> >
> > Best regards
> >
> > José
> >
> > -Original Message-
> > From: devel [mailto:devel-boun...@rtems.org] On Behalf Of Chris Johns
> > Sent: quinta-feira, 9 de janeiro de 2020 20:57
> > To: Gedare Bloom; sebastian huber
> > Cc: devel@rtems.org
> > Subject: Re: Requirement Document generator tool
> >
> > On 10/1/20 4:45 am, Gedare Bloom wrote:
> >> On Wed, Jan 8, 2020 at 11:51 PM Sebastian Huber
> >>  wrote:
> >>>
> >>> On 08/01/2020 19:31, Gedare Bloom wrote:
> >>>>>
> >>>>> I agree completely on the proposed approach with Python tools.
> >>>>>
> >>>> Yes. Reading it I'm actually reminded about Google's approach toward
> >>>> Python which includes many of the elements mentioned. Although their
> >>>> guide is probably more comprehensive and verbose that what we need, it
> >>>> might be a useful reference for developing a set of guidelines
> >>>> suitable for Python code in RTEMS (mainly, rtems-tools).  Here is a
> >>>> link:
> >>>> http://google.github.io/styleguide/pyguide.html
> >>>>
> >>>> I think most of the existing style has been determined and driving by
> >>>> Chris Johns. So I would also give him some credit to develop/approve
> >>>> how we plan to use Python at a project level. (**Hands Chris an "RTEMS
> >>>> Python Maintainer" hat**);)
> >>>
> >>> I think the Google guide would be a good start. We can always add
> >>> project-specific exceptions/clarifications if necessary. My aim is to
> >>>

Re: Requirement Document generator tool

[Joel Sherrill]

On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer <
christian.maude...@embedded-brains.de> wrote:

> Hello Jose,
>
> On 10/01/2020 17:33, Jose Valdez wrote:
> > Hello,
> >
> > Regarding this topic and to start to define the python tools that are
> more appropriate for the RTEMS community python developments, I would like
> to propose the following tools (to be placed in RTEMS Software Engineering
> manual):
> >
> > Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
> > Python coding style/standard - Google (as suggested by Gedare)
> > Python documentation style - Google docstrings (to be in accordance with
> coding style. Note it is supported by sphinx)
> > Python test approach - unittest (seems to be the standard used by python)
> > Python static analysis - pylint (recommended by Google style, see
> http://google.github.io/styleguide/pyguide.html) and mypy (catch
> additional errors) Python code coverage - Coverage.py (seems to be the
> standard used by python)
> > Python third party packages - For now we are using: paramiko, pyserial,
> pexpect, gitpython, but the list is expected to be changed. I think here
> any third package is allowed, as long as the license of the package
> complies with RTEMS project license.
>
> To be future-proof: python3 compatibility would be nearly a must-have
> for any new library.
>

We discussed this somewhere earlier and I thought we decided that if
it was a core tool required for a "normal user", then it would have to
be Python 2.x and 3.x compatible. Tools for "requirements user" or
"pre-qualification user" (pick a name for this role), can be 3.x only
mainly because there was so much infrastructure required for the
"requirements user" that already needed Python 3.x.

I think this discussion should also include "documentation producer"
because that requires Python 3.x as well.

We just need to be conscious of the user roles that the tools are
supporting and justify the minimum. This gives us at least 3 categories
of tools with potentially different minimum hosting requirements. For
example, we may not care if a certain set of tools runs on Mingw.

Obviously documented somewhere.


>
> I don't think that there are heavy license problems. The python packages
> are most likely only used in tools. So as long as they are open enough
> to use them on all host platforms for all development situations that
> shouldn't be a problem. Exception: If the package is used to generate
> code with a specific license. That generated code would have to be
> compatible with the RTEMS license.
>

+1 and very much non-viral and no advertising for generated code.

>
> >
> > Please tell me your opinions for each python tool and then we can start
> to agree on what to use. Note that once this is defined, our Qualification
> software will comply with the RTEMS community chosen tools.
>
> If no one objects I would suggest to start documenting them in the RTEMS
> Software Engineering manual. A small example project in the rtems-tools
> repository that uses the suggested code checkers (meaning: calls them
> during a waf build or a special 'waf check' target) would be great too.
> That could be used for all future python based tools that are added to
> rtems-tools (which includes the whole qualification tools).
>

Yes. It must be enforced. Whether it is a special step or not is up for
discussion.
My first thought is that a special step is easy to skip.

Also can "check" be done as a pre-commit hook?

And when the style is agreed to, should existing code be auto reformatted
to meet it?

--joel

>
> Best regards
>
> Christian
>
> >
> > Best regards
> >
> > José
> >
> > -Original Message-
> > From: devel [mailto:devel-boun...@rtems.org] On Behalf Of Chris Johns
> > Sent: quinta-feira, 9 de janeiro de 2020 20:57
> > To: Gedare Bloom; sebastian huber
> > Cc: devel@rtems.org
> > Subject: Re: Requirement Document generator tool
> >
> > On 10/1/20 4:45 am, Gedare Bloom wrote:
> >> On Wed, Jan 8, 2020 at 11:51 PM Sebastian Huber
> >>  wrote:
> >>>
> >>> On 08/01/2020 19:31, Gedare Bloom wrote:
> >>>>>
> >>>>> I agree completely on the proposed approach with Python tools.
> >>>>>
> >>>> Yes. Reading it I'm actually reminded about Google's approach toward
> >>>> Python which includes many of the elements mentioned. Although their
> >>>> guide is probably more comprehensive and verbose that what we need, it
> >>>> might be a useful reference for developing a set of gu

Re: Requirement Document generator tool

[Christian Mauderer]

Hello Jose,

On 10/01/2020 17:33, Jose Valdez wrote:
> Hello,
> 
> Regarding this topic and to start to define the python tools that are more 
> appropriate for the RTEMS community python developments, I would like to 
> propose the following tools (to be placed in RTEMS Software Engineering 
> manual):
> 
> Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
> Python coding style/standard - Google (as suggested by Gedare)
> Python documentation style - Google docstrings (to be in accordance with 
> coding style. Note it is supported by sphinx)
> Python test approach - unittest (seems to be the standard used by python)
> Python static analysis - pylint (recommended by Google style, see 
> http://google.github.io/styleguide/pyguide.html) and mypy (catch additional 
> errors) Python code coverage - Coverage.py (seems to be the standard used by 
> python)
> Python third party packages - For now we are using: paramiko, pyserial, 
> pexpect, gitpython, but the list is expected to be changed. I think here any 
> third package is allowed, as long as the license of the package complies with 
> RTEMS project license.

To be future-proof: python3 compatibility would be nearly a must-have
for any new library.

I don't think that there are heavy license problems. The python packages
are most likely only used in tools. So as long as they are open enough
to use them on all host platforms for all development situations that
shouldn't be a problem. Exception: If the package is used to generate
code with a specific license. That generated code would have to be
compatible with the RTEMS license.

> 
> Please tell me your opinions for each python tool and then we can start to 
> agree on what to use. Note that once this is defined, our Qualification 
> software will comply with the RTEMS community chosen tools.

If no one objects I would suggest to start documenting them in the RTEMS
Software Engineering manual. A small example project in the rtems-tools
repository that uses the suggested code checkers (meaning: calls them
during a waf build or a special 'waf check' target) would be great too.
That could be used for all future python based tools that are added to
rtems-tools (which includes the whole qualification tools).

Best regards

Christian

> 
> Best regards
> 
> José
> 
> -Original Message-
> From: devel [mailto:devel-boun...@rtems.org] On Behalf Of Chris Johns
> Sent: quinta-feira, 9 de janeiro de 2020 20:57
> To: Gedare Bloom; sebastian huber
> Cc: devel@rtems.org
> Subject: Re: Requirement Document generator tool
> 
> On 10/1/20 4:45 am, Gedare Bloom wrote:
>> On Wed, Jan 8, 2020 at 11:51 PM Sebastian Huber
>>  wrote:
>>>
>>> On 08/01/2020 19:31, Gedare Bloom wrote:
>>>>>
>>>>> I agree completely on the proposed approach with Python tools.
>>>>>
>>>> Yes. Reading it I'm actually reminded about Google's approach toward
>>>> Python which includes many of the elements mentioned. Although their
>>>> guide is probably more comprehensive and verbose that what we need, it
>>>> might be a useful reference for developing a set of guidelines
>>>> suitable for Python code in RTEMS (mainly, rtems-tools).  Here is a
>>>> link:
>>>> http://google.github.io/styleguide/pyguide.html
>>>>
>>>> I think most of the existing style has been determined and driving by
>>>> Chris Johns. So I would also give him some credit to develop/approve
>>>> how we plan to use Python at a project level. (**Hands Chris an "RTEMS
>>>> Python Maintainer" hat**);)
>>>
>>> I think the Google guide would be a good start. We can always add
>>> project-specific exceptions/clarifications if necessary. My aim is to
>>> use it for new code, e.g. code produced for the pre-qualification
>>> activity. For the code format I strongly want to use a tool for this. I
>>> don't want to spend review time on code formatting issues.
>>>
>>> Using standard guidelines makes the RTEMS Project more attractive for
>>> new contributors and GSoC students. I think it increases your job
>>> opportunities if you can refer to a successful GSoC project and it shows
>>> that you used standard engineering practices in the project. This is
>>> usually not something a university education includes.
>>>
>> OK, both points make sense. I'd be happy with the Google guide, I hope
>> Chris will comment when he can.
> 
> Sorry about the erratic access. I have been knee deep in painting and flooring
> this summer as I avoid any smoke from the fires we are ha

RE: Requirement Document generator tool

[Jose Valdez]

Hello,

Regarding this topic and to start to define the python tools that are more 
appropriate for the RTEMS community python developments, I would like to 
propose the following tools (to be placed in RTEMS Software Engineering manual):

Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
Python coding style/standard - Google (as suggested by Gedare)
Python documentation style - Google docstrings (to be in accordance with coding 
style. Note it is supported by sphinx)
Python test approach - unittest (seems to be the standard used by python)
Python static analysis - pylint (recommended by Google style, see 
http://google.github.io/styleguide/pyguide.html) and mypy (catch additional 
errors) Python code coverage - Coverage.py (seems to be the standard used by 
python)
Python third party packages - For now we are using: paramiko, pyserial, 
pexpect, gitpython, but the list is expected to be changed. I think here any 
third package is allowed, as long as the license of the package complies with 
RTEMS project license.

Please tell me your opinions for each python tool and then we can start to 
agree on what to use. Note that once this is defined, our Qualification 
software will comply with the RTEMS community chosen tools.

Best regards

José

-Original Message-
From: devel [mailto:devel-boun...@rtems.org] On Behalf Of Chris Johns
Sent: quinta-feira, 9 de janeiro de 2020 20:57
To: Gedare Bloom; sebastian huber
Cc: devel@rtems.org
Subject: Re: Requirement Document generator tool

On 10/1/20 4:45 am, Gedare Bloom wrote:
> On Wed, Jan 8, 2020 at 11:51 PM Sebastian Huber
>  wrote:
>>
>> On 08/01/2020 19:31, Gedare Bloom wrote:
>>>>
>>>> I agree completely on the proposed approach with Python tools.
>>>>
>>> Yes. Reading it I'm actually reminded about Google's approach toward
>>> Python which includes many of the elements mentioned. Although their
>>> guide is probably more comprehensive and verbose that what we need, it
>>> might be a useful reference for developing a set of guidelines
>>> suitable for Python code in RTEMS (mainly, rtems-tools).  Here is a
>>> link:
>>> http://google.github.io/styleguide/pyguide.html
>>>
>>> I think most of the existing style has been determined and driving by
>>> Chris Johns. So I would also give him some credit to develop/approve
>>> how we plan to use Python at a project level. (**Hands Chris an "RTEMS
>>> Python Maintainer" hat**);)
>>
>> I think the Google guide would be a good start. We can always add
>> project-specific exceptions/clarifications if necessary. My aim is to
>> use it for new code, e.g. code produced for the pre-qualification
>> activity. For the code format I strongly want to use a tool for this. I
>> don't want to spend review time on code formatting issues.
>>
>> Using standard guidelines makes the RTEMS Project more attractive for
>> new contributors and GSoC students. I think it increases your job
>> opportunities if you can refer to a successful GSoC project and it shows
>> that you used standard engineering practices in the project. This is
>> usually not something a university education includes.
>>
> OK, both points make sense. I'd be happy with the Google guide, I hope
> Chris will comment when he can.

Sorry about the erratic access. I have been knee deep in painting and flooring
this summer as I avoid any smoke from the fires we are having.

I am fine with using a standard guide however we need to review it and to make
sure it fits. As an example the C++ guide from Google had some good points as
well as some parts I did not think offered any value but did create a certain
level of pain. We should also consider capturing the guide as a public one
belonging to someone else can and will change and that effects us. I am not sure
how we could do this.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Chris Johns]

On 10/1/20 4:45 am, Gedare Bloom wrote:
> On Wed, Jan 8, 2020 at 11:51 PM Sebastian Huber
>  wrote:
>>
>> On 08/01/2020 19:31, Gedare Bloom wrote:

 I agree completely on the proposed approach with Python tools.

>>> Yes. Reading it I'm actually reminded about Google's approach toward
>>> Python which includes many of the elements mentioned. Although their
>>> guide is probably more comprehensive and verbose that what we need, it
>>> might be a useful reference for developing a set of guidelines
>>> suitable for Python code in RTEMS (mainly, rtems-tools).  Here is a
>>> link:
>>> http://google.github.io/styleguide/pyguide.html
>>>
>>> I think most of the existing style has been determined and driving by
>>> Chris Johns. So I would also give him some credit to develop/approve
>>> how we plan to use Python at a project level. (**Hands Chris an "RTEMS
>>> Python Maintainer" hat**);)
>>
>> I think the Google guide would be a good start. We can always add
>> project-specific exceptions/clarifications if necessary. My aim is to
>> use it for new code, e.g. code produced for the pre-qualification
>> activity. For the code format I strongly want to use a tool for this. I
>> don't want to spend review time on code formatting issues.
>>
>> Using standard guidelines makes the RTEMS Project more attractive for
>> new contributors and GSoC students. I think it increases your job
>> opportunities if you can refer to a successful GSoC project and it shows
>> that you used standard engineering practices in the project. This is
>> usually not something a university education includes.
>>
> OK, both points make sense. I'd be happy with the Google guide, I hope
> Chris will comment when he can.

Sorry about the erratic access. I have been knee deep in painting and flooring
this summer as I avoid any smoke from the fires we are having.

I am fine with using a standard guide however we need to review it and to make
sure it fits. As an example the C++ guide from Google had some good points as
well as some parts I did not think offered any value but did create a certain
level of pain. We should also consider capturing the guide as a public one
belonging to someone else can and will change and that effects us. I am not sure
how we could do this.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Gedare Bloom]

On Wed, Jan 8, 2020 at 11:51 PM Sebastian Huber
 wrote:
>
> On 08/01/2020 19:31, Gedare Bloom wrote:
> >>
> >> I agree completely on the proposed approach with Python tools.
> >>
> > Yes. Reading it I'm actually reminded about Google's approach toward
> > Python which includes many of the elements mentioned. Although their
> > guide is probably more comprehensive and verbose that what we need, it
> > might be a useful reference for developing a set of guidelines
> > suitable for Python code in RTEMS (mainly, rtems-tools).  Here is a
> > link:
> > http://google.github.io/styleguide/pyguide.html
> >
> > I think most of the existing style has been determined and driving by
> > Chris Johns. So I would also give him some credit to develop/approve
> > how we plan to use Python at a project level. (**Hands Chris an "RTEMS
> > Python Maintainer" hat**);)
>
> I think the Google guide would be a good start. We can always add
> project-specific exceptions/clarifications if necessary. My aim is to
> use it for new code, e.g. code produced for the pre-qualification
> activity. For the code format I strongly want to use a tool for this. I
> don't want to spend review time on code formatting issues.
>
> Using standard guidelines makes the RTEMS Project more attractive for
> new contributors and GSoC students. I think it increases your job
> opportunities if you can refer to a successful GSoC project and it shows
> that you used standard engineering practices in the project. This is
> usually not something a university education includes.
>
OK, both points make sense. I'd be happy with the Google guide, I hope
Chris will comment when he can.

> --
> Sebastian Huber, embedded brains GmbH
>
> Address : Dornierstr. 4, D-82178 Puchheim, Germany
> Phone   : +49 89 189 47 41-16
> Fax : +49 89 189 47 41-09
> E-Mail  : sebastian.hu...@embedded-brains.de
> PGP : Public key available on request.
>
> Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Sebastian Huber]

On 08/01/2020 19:31, Gedare Bloom wrote:


I agree completely on the proposed approach with Python tools.


Yes. Reading it I'm actually reminded about Google's approach toward
Python which includes many of the elements mentioned. Although their
guide is probably more comprehensive and verbose that what we need, it
might be a useful reference for developing a set of guidelines
suitable for Python code in RTEMS (mainly, rtems-tools).  Here is a
link:
http://google.github.io/styleguide/pyguide.html

I think most of the existing style has been determined and driving by
Chris Johns. So I would also give him some credit to develop/approve
how we plan to use Python at a project level. (**Hands Chris an "RTEMS
Python Maintainer" hat**);)


I think the Google guide would be a good start. We can always add 
project-specific exceptions/clarifications if necessary. My aim is to 
use it for new code, e.g. code produced for the pre-qualification 
activity. For the code format I strongly want to use a tool for this. I 
don't want to spend review time on code formatting issues.


Using standard guidelines makes the RTEMS Project more attractive for 
new contributors and GSoC students. I think it increases your job 
opportunities if you can refer to a successful GSoC project and it shows 
that you used standard engineering practices in the project. This is 
usually not something a university education includes.


--
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail  : sebastian.hu...@embedded-brains.de
PGP : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Requirement Document generator tool

[Gedare Bloom]

On Wed, Jan 8, 2020 at 9:47 AM Joel Sherrill  wrote:
>
>
>
> On Fri, Jan 3, 2020 at 7:26 AM Sebastian Huber 
>  wrote:
>>
>> Hello Jose,
>>
>> On 03/01/2020 11:52, Jose Valdez wrote:
>>  Should be in rtems-tools? If so, in which place?
>> >>> I think this is the best place given the information I have. I would
>> >>> need to have more detail before I could provide any specific direction 
>> >>> here.
>> >>
>> >> All host tools are in rtems-tools so at this point, there isn't another 
>> >> place.
>> >>
>> >> But I agree with Chris. We need more information.
>> >>
>> > We need to know:
>> > * Language(s) used for the tool
>> > * Host requirements to run the tool
>> > * Licensing of the tool
>> >
>> > Putting the pre-qual tools into rtems-tools seems reasonable, but a 
>> > discussion should happen within the RTEMS Project community/maintainers to 
>> > do that.
>> >
>> > JV02012019, Joel and Gedare:
>> >
>> > We need to know:
>> > * Language(s) used for the tool - python (>= 3.7)
>> > * Host requirements to run the tool - Debian 10
>> > * Licensing of the tool - BSD-2-Clause
>> >
>> > Please read the section 4 of the document SDD-301.pdf, I am sending in 
>> > attach (removed to go to rtems devel mailing list!!). This provides an 
>> > overview of the tool (I think we missed to give you this overview).
>> > For more information you may also take a look in the SRS-300.pdf and 
>> > TI-003 (just the conclusions on each section).
>> > Note that the information presented in these documents is subjected to be 
>> > changed, but as I said, the idea is for you to have a better understanding 
>> > about the big picture of the project. The details will be discussed (and 
>> > iterated) alongside the project execution.
>>
>> the aim is to write all tools in the ESA pre-qualification activity in
>> Python. Derivatives of existing stuff will use the respective license.
>> New stuff will use the BSD-2-Clause license for code and CC-BY-SA-4.0
>> for documentation.
>>
>>  From an RTEMS Project point of view, project-specific documentation is
>> irrelevant in general. It can be used as a complementary material to
>> explain things intended for integration, however, all content relevant
>> to things integrated in the RTEMS Project must move to the right places
>> in the RTEMS Project.
>>
>> We have to settle on a specific Python version. Since Doorstop 2.0.post2
>> is already selected as the tool for requirements management and it
>> requires at least Python 3.6 I think it makes sense to use this version
>> as well for new tools. In contrast to the build system I think a Python
>> 2 compatibility is unnecessary.
>>
>> Tools which belong also to the work flow of an RTEMS user, e.g. the
>> RTEMS Tester, should still work with Python 2.
>>
>> The decisions, justification, and requirements with respect to the tool
>> language selection should be recorded in the RTEMS Software Engineering
>> manual.
>>
>> Once a tool language is selected. We need a coding style and standard.
>> We should choose an existing one, e.g.
>>
>> https://www.python.org/dev/peps/pep-0008/
>>
>> There should be tools to check the coding style automatically. I don't
>> want the situation we have with the RTEMS sources. For Python there are
>> plenty of tools, guides, and documentation available. We just have to
>> pick some for the RTEMS Project and document this in the RTEMS Software
>> Engineering manual.
>>
>> The compatible Python versions (e.g. 3.6+), coding style (e.g. black -l
>> 80), documentation style (e.g. pydocstyle), test approach (standard
>> Python unittest and unittest.mock), static analysis (e.g. mypy and
>> pylint), code coverage, use of third-party packages, etc. need to be
>> discussed with the RTEMS Project and the conclusions should be added to
>> the RTEMS Software Engineering manual.
>>
>> Running the checkers, test suites, code coverage, etc. should be done
>> though the build system (in rtems-tools this is waf). The Doorstop
>> project is a good example how this can be set up.
>>
>> If you look at the current RTEMS Software Engineering manual you will
>> find next to nothing with respect to Python code. You can set now the
>> standards (after discussion on devel@rtems.org). You should do this
>> before you send the first line of Python code for review.
>
>
> I agree completely on the proposed approach with Python tools.
>
Yes. Reading it I'm actually reminded about Google's approach toward
Python which includes many of the elements mentioned. Although their
guide is probably more comprehensive and verbose that what we need, it
might be a useful reference for developing a set of guidelines
suitable for Python code in RTEMS (mainly, rtems-tools).  Here is a
link:
http://google.github.io/styleguide/pyguide.html

I think most of the existing style has been determined and driving by
Chris Johns. So I would also give him some credit to develop/approve
how we plan to use Python at a project level. (**Hands Chris an "RTEMS
Python Main

Re: Requirement Document generator tool

[Joel Sherrill]

On Fri, Jan 3, 2020 at 7:26 AM Sebastian Huber <
sebastian.hu...@embedded-brains.de> wrote:

> Hello Jose,
>
> On 03/01/2020 11:52, Jose Valdez wrote:
>  Should be in rtems-tools? If so, in which place?
> >>> I think this is the best place given the information I have. I would
> >>> need to have more detail before I could provide any specific direction
> here.
> >>
> >> All host tools are in rtems-tools so at this point, there isn't another
> place.
> >>
> >> But I agree with Chris. We need more information.
> >>
> > We need to know:
> > * Language(s) used for the tool
> > * Host requirements to run the tool
> > * Licensing of the tool
> >
> > Putting the pre-qual tools into rtems-tools seems reasonable, but a
> discussion should happen within the RTEMS Project community/maintainers to
> do that.
> >
> > JV02012019, Joel and Gedare:
> >
> > We need to know:
> > * Language(s) used for the tool - python (>= 3.7)
> > * Host requirements to run the tool - Debian 10
> > * Licensing of the tool - BSD-2-Clause
> >
> > Please read the section 4 of the document SDD-301.pdf, I am sending in
> attach (removed to go to rtems devel mailing list!!). This provides an
> overview of the tool (I think we missed to give you this overview).
> > For more information you may also take a look in the SRS-300.pdf and
> TI-003 (just the conclusions on each section).
> > Note that the information presented in these documents is subjected to
> be changed, but as I said, the idea is for you to have a better
> understanding about the big picture of the project. The details will be
> discussed (and iterated) alongside the project execution.
>
> the aim is to write all tools in the ESA pre-qualification activity in
> Python. Derivatives of existing stuff will use the respective license.
> New stuff will use the BSD-2-Clause license for code and CC-BY-SA-4.0
> for documentation.
>
>  From an RTEMS Project point of view, project-specific documentation is
> irrelevant in general. It can be used as a complementary material to
> explain things intended for integration, however, all content relevant
> to things integrated in the RTEMS Project must move to the right places
> in the RTEMS Project.
>
> We have to settle on a specific Python version. Since Doorstop 2.0.post2
> is already selected as the tool for requirements management and it
> requires at least Python 3.6 I think it makes sense to use this version
> as well for new tools. In contrast to the build system I think a Python
> 2 compatibility is unnecessary.
>
> Tools which belong also to the work flow of an RTEMS user, e.g. the
> RTEMS Tester, should still work with Python 2.
>
> The decisions, justification, and requirements with respect to the tool
> language selection should be recorded in the RTEMS Software Engineering
> manual.
>
> Once a tool language is selected. We need a coding style and standard.
> We should choose an existing one, e.g.
>
> https://www.python.org/dev/peps/pep-0008/
>
> There should be tools to check the coding style automatically. I don't
> want the situation we have with the RTEMS sources. For Python there are
> plenty of tools, guides, and documentation available. We just have to
> pick some for the RTEMS Project and document this in the RTEMS Software
> Engineering manual.
>
> The compatible Python versions (e.g. 3.6+), coding style (e.g. black -l
> 80), documentation style (e.g. pydocstyle), test approach (standard
> Python unittest and unittest.mock), static analysis (e.g. mypy and
> pylint), code coverage, use of third-party packages, etc. need to be
> discussed with the RTEMS Project and the conclusions should be added to
> the RTEMS Software Engineering manual.
>
> Running the checkers, test suites, code coverage, etc. should be done
> though the build system (in rtems-tools this is waf). The Doorstop
> project is a good example how this can be set up.
>
> If you look at the current RTEMS Software Engineering manual you will
> find next to nothing with respect to Python code. You can set now the
> standards (after discussion on devel@rtems.org). You should do this
> before you send the first line of Python code for review.
>

I agree completely on the proposed approach with Python tools.


>
> Once the frame for Python development is settled. We need a high level
> description of the purpose of the tools, the inputs and the outputs.
> This high level description should be integrated in the RTEMS Project
> documentation. If you want to get things integrated in the RTEMS Project
> then you have make sure that it is good and general enough so that
> others can continue to work on it. In the worst case everything should
> be maintainable by the RTEMS Project without you in the future.
>

Describing the purpose, inputs and outputs, and high level description
should
go a long way to having a theory of operation for the tools. It is
important to
know what software engineering purpose the tool satisfies and its role in
the pre-qualification workflow.

T

Re: Requirement Document generator tool

[Joel Sherrill]

On Fri, Jan 3, 2020, 4:49 AM Jose Valdez  wrote:

> Hello Gedare and Joel,
>
> Please see my answers below for both of you (see JV02012019).
>
> Please tell me, if you need further clarification.
>
> Best regards
>
> José
>
> -Original Message-
> From: Gedare Bloom [mailto:ged...@rtems.org]
> Sent: quinta-feira, 2 de janeiro de 2020 17:21
> To: Joel Sherrill
> Cc: Jose Valdez; devel@rtems.org
> Subject: Re: Requirement Document generator tool
>
> Hello José et al.,
>
> I have only a few points I would like clarified, below:
>
> On Mon, Dec 16, 2019 at 3:35 PM Joel Sherrill  wrote:
> >
> >
> >
> > On Mon, Dec 16, 2019 at 7:29 AM Jose Valdez 
> wrote:
> >>
> >> Hello Chris,
> >>
> >> Thank you for your reply.
> >>
> >> Please find below my answers.
> >>
> >> José
> >>
> >> -----Original Message-----
> >> From: Chris Johns [mailto:chr...@rtems.org]
> >> Sent: sexta-feira, 13 de dezembro de 2019 19:09
> >> To: Jose Valdez; devel@rtems.org
> >> Subject: Re: Requirement Document generator tool
> >>
> >> On 14/12/19 3:09 am, Jose Valdez wrote:
> >> > In the scope of ESA's RTEMS SMP project, we are developing a tool
> which will
> >> > allow to generate RTEMS software documentation
> >>
> >> I assume this is the pre-qual effort.
> >>
> >> JV: That's correct.
> >>
> >> > (Requirements, Design, Test Specification and other software
> documents).
> >>
> >> How do these documents relate to the project's current documentation?
> It may
> >> help to explain the purpose of each of these documents and the target
> user.
> >
> >
> >
> >>
> >> JV: The idea of the project is to allow an user to pre-qualify RTEMS
> for space applications. In that sense, the goal of the tool is to save the
> pre-qualification effort, by creating automatically the parts of the
> necessary documentation, which may be automatized. Generically parts which
> may be automatized are:
> >> -> Traceabilities (ex: Requirements against Architecture)
> >> -> Requirement coverage (Tests against requirements)
> >> -> Test results verification
> >> -> Creation of document templates
> >> -> source code metrics collection
> >> -> etc.
> >>
> >> The current scope is to create the documentation in accordance with ESA
> standard (ECSS), which defines the following necessary documents (and also
> the template) for a software product (note: the description text of each
> document was taken from ECSS, for helping you to understand the goal of
> each document):
> >
> >
> > This list of documents has some of which I would expect to
> > be hand-written, one-time documents. Others related to requirements,
> traceability,
> > and tests, I would expect to be generated. Can you clarify which
> documents
> > fall into which category?
> >
> > And how these documents related to RTEMS Pre-Qualification. For example,
> I don't see the
> > need for a Software Reuse File for RTEMS. That would seem to be
> something a project adopting
> > RTEMS might need.
> >
> > These documents also show an intentional bias to ECSS which is OK for
> you guys but
> > doesn't help in the RTEMS.org broader goal of having technical data for
> qualification which
> > fulfills the needs for multiple standards (NASA Quality, DO-178,
> automotive, etc.) Please
> > keep in mind that RTEMS is a world-wide project, independent of ESA and
> we would
> > like this effort to be able to address the multiple stakeholders.
> >
> > Some of these have the same names as other quality standard, others
> don't.
>
>
> JV02012019: Regarding your first question, a document itself may have both
> parts hand-written and generated. I understand that probably we missed to
> give you the full technical scope of the project (which is difficult by
> only exchanging e-mails). Please wait for our first input (Requirement
> Manager), which will include instructions on how to use and then you may
> perform your assessments and clarify your technical doubts.
>

One of the things that has unfortunately been lost over the years in
computer documentation is a Theory of Operations. This is independent of
the implementation.

In this case, you guys are trying to automate parts of an information flow
and traceability of that flow. What would help is an implementation
independent description of that information flow so we can say f

Re: Requirement Document generator tool

[Sebastian Huber]

Hello Jose,

On 03/01/2020 11:52, Jose Valdez wrote:

Should be in rtems-tools? If so, in which place?

I think this is the best place given the information I have. I would
need to have more detail before I could provide any specific direction here.


All host tools are in rtems-tools so at this point, there isn't another place.

But I agree with Chris. We need more information.


We need to know:
* Language(s) used for the tool
* Host requirements to run the tool
* Licensing of the tool

Putting the pre-qual tools into rtems-tools seems reasonable, but a discussion 
should happen within the RTEMS Project community/maintainers to do that.

JV02012019, Joel and Gedare:

We need to know:
* Language(s) used for the tool - python (>= 3.7)
* Host requirements to run the tool - Debian 10
* Licensing of the tool - BSD-2-Clause

Please read the section 4 of the document SDD-301.pdf, I am sending in attach 
(removed to go to rtems devel mailing list!!). This provides an overview of the 
tool (I think we missed to give you this overview).
For more information you may also take a look in the SRS-300.pdf and TI-003 
(just the conclusions on each section).
Note that the information presented in these documents is subjected to be 
changed, but as I said, the idea is for you to have a better understanding 
about the big picture of the project. The details will be discussed (and 
iterated) alongside the project execution.


the aim is to write all tools in the ESA pre-qualification activity in 
Python. Derivatives of existing stuff will use the respective license. 
New stuff will use the BSD-2-Clause license for code and CC-BY-SA-4.0 
for documentation.


From an RTEMS Project point of view, project-specific documentation is 
irrelevant in general. It can be used as a complementary material to 
explain things intended for integration, however, all content relevant 
to things integrated in the RTEMS Project must move to the right places 
in the RTEMS Project.


We have to settle on a specific Python version. Since Doorstop 2.0.post2 
is already selected as the tool for requirements management and it 
requires at least Python 3.6 I think it makes sense to use this version 
as well for new tools. In contrast to the build system I think a Python 
2 compatibility is unnecessary.


Tools which belong also to the work flow of an RTEMS user, e.g. the 
RTEMS Tester, should still work with Python 2.


The decisions, justification, and requirements with respect to the tool 
language selection should be recorded in the RTEMS Software Engineering 
manual.


Once a tool language is selected. We need a coding style and standard. 
We should choose an existing one, e.g.


https://www.python.org/dev/peps/pep-0008/

There should be tools to check the coding style automatically. I don't 
want the situation we have with the RTEMS sources. For Python there are 
plenty of tools, guides, and documentation available. We just have to 
pick some for the RTEMS Project and document this in the RTEMS Software 
Engineering manual.


The compatible Python versions (e.g. 3.6+), coding style (e.g. black -l 
80), documentation style (e.g. pydocstyle), test approach (standard 
Python unittest and unittest.mock), static analysis (e.g. mypy and 
pylint), code coverage, use of third-party packages, etc. need to be 
discussed with the RTEMS Project and the conclusions should be added to 
the RTEMS Software Engineering manual.


Running the checkers, test suites, code coverage, etc. should be done 
though the build system (in rtems-tools this is waf). The Doorstop 
project is a good example how this can be set up.


If you look at the current RTEMS Software Engineering manual you will 
find next to nothing with respect to Python code. You can set now the 
standards (after discussion on devel@rtems.org). You should do this 
before you send the first line of Python code for review.


Once the frame for Python development is settled. We need a high level 
description of the purpose of the tools, the inputs and the outputs. 
This high level description should be integrated in the RTEMS Project 
documentation. If you want to get things integrated in the RTEMS Project 
then you have make sure that it is good and general enough so that 
others can continue to work on it. In the worst case everything should 
be maintainable by the RTEMS Project without you in the future.


--
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail  : sebastian.hu...@embedded-brains.de
PGP : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

RE: Requirement Document generator tool

[Jose Valdez]

Same message with no attachs.

-Original Message-
From: Jose Valdez
Sent: sexta-feira, 3 de janeiro de 2020 10:49
To: 'Gedare Bloom'; Joel Sherrill
Cc: devel@rtems.org
Subject: RE: Requirement Document generator tool

Hello Gedare and Joel,

Please see my answers below for both of you (see JV02012019).

Please tell me, if you need further clarification.

Best regards

José

-Original Message-
From: Gedare Bloom [mailto:ged...@rtems.org]
Sent: quinta-feira, 2 de janeiro de 2020 17:21
To: Joel Sherrill
Cc: Jose Valdez; devel@rtems.org
Subject: Re: Requirement Document generator tool

Hello José et al.,

I have only a few points I would like clarified, below:

On Mon, Dec 16, 2019 at 3:35 PM Joel Sherrill  wrote:
>
>
>
> On Mon, Dec 16, 2019 at 7:29 AM Jose Valdez  wrote:
>>
>> Hello Chris,
>>
>> Thank you for your reply.
>>
>> Please find below my answers.
>>
>> José
>>
>> -Original Message-
>> From: Chris Johns [mailto:chr...@rtems.org]
>> Sent: sexta-feira, 13 de dezembro de 2019 19:09
>> To: Jose Valdez; devel@rtems.org
>> Subject: Re: Requirement Document generator tool
>>
>> On 14/12/19 3:09 am, Jose Valdez wrote:
>> > In the scope of ESA's RTEMS SMP project, we are developing a tool
>> > which will allow to generate RTEMS software documentation
>>
>> I assume this is the pre-qual effort.
>>
>> JV: That's correct.
>>
>> > (Requirements, Design, Test Specification and other software documents).
>>
>> How do these documents relate to the project's current documentation?
>> It may help to explain the purpose of each of these documents and the target 
>> user.
>
>
>
>>
>> JV: The idea of the project is to allow an user to pre-qualify RTEMS for 
>> space applications. In that sense, the goal of the tool is to save the 
>> pre-qualification effort, by creating automatically the parts of the 
>> necessary documentation, which may be automatized. Generically parts which 
>> may be automatized are:
>> -> Traceabilities (ex: Requirements against Architecture) Requirement
>> -> coverage (Tests against requirements) Test results verification
>> -> Creation of document templates source code metrics collection etc.
>>
>> The current scope is to create the documentation in accordance with ESA 
>> standard (ECSS), which defines the following necessary documents (and also 
>> the template) for a software product (note: the description text of each 
>> document was taken from ECSS, for helping you to understand the goal of each 
>> document):
>
>
> This list of documents has some of which I would expect to be
> hand-written, one-time documents. Others related to requirements,
> traceability, and tests, I would expect to be generated. Can you
> clarify which documents fall into which category?
>
> And how these documents related to RTEMS Pre-Qualification. For
> example, I don't see the need for a Software Reuse File for RTEMS.
> That would seem to be something a project adopting RTEMS might need.
>
> These documents also show an intentional bias to ECSS which is OK for
> you guys but doesn't help in the RTEMS.org broader goal of having
> technical data for qualification which fulfills the needs for multiple
> standards (NASA Quality, DO-178, automotive, etc.) Please keep in mind
> that RTEMS is a world-wide project, independent of ESA and we would like this 
> effort to be able to address the multiple stakeholders.
>
> Some of these have the same names as other quality standard, others don't.


JV02012019: Regarding your first question, a document itself may have both 
parts hand-written and generated. I understand that probably we missed to give 
you the full technical scope of the project (which is difficult by only 
exchanging e-mails). Please wait for our first input (Requirement Manager), 
which will include instructions on how to use and then you may perform your 
assessments and clarify your technical doubts.

JV02012019: The documents which are not applicable to RTEMS project (like 
Software Reuse File) will be created, but will not have any content (the 
sections will have text like "Not Applicable").

JV02012019: I understand your concern about having the tool producing output 
for other standards/applications. Unfortunately this is not possible in this 
project (due to budget). However, the philosophy for this project should be to 
open the possibility to extend the tool, and hence other users may add the 
functionality to produce the documentation according with their standards.

>
>>
>>
>> 

Re: Requirement Document generator tool

[Gedare Bloom]

Hello José et al.,

I have only a few points I would like clarified, below:

On Mon, Dec 16, 2019 at 3:35 PM Joel Sherrill  wrote:
>
>
>
> On Mon, Dec 16, 2019 at 7:29 AM Jose Valdez  wrote:
>>
>> Hello Chris,
>>
>> Thank you for your reply.
>>
>> Please find below my answers.
>>
>> José
>>
>> -Original Message-
>> From: Chris Johns [mailto:chr...@rtems.org]
>> Sent: sexta-feira, 13 de dezembro de 2019 19:09
>> To: Jose Valdez; devel@rtems.org
>> Subject: Re: Requirement Document generator tool
>>
>> On 14/12/19 3:09 am, Jose Valdez wrote:
>> > In the scope of ESA's RTEMS SMP project, we are developing a tool which 
>> > will
>> > allow to generate RTEMS software documentation
>>
>> I assume this is the pre-qual effort.
>>
>> JV: That's correct.
>>
>> > (Requirements, Design, Test Specification and other software documents).
>>
>> How do these documents relate to the project's current documentation? It may
>> help to explain the purpose of each of these documents and the target user.
>
>
>
>>
>> JV: The idea of the project is to allow an user to pre-qualify RTEMS for 
>> space applications. In that sense, the goal of the tool is to save the 
>> pre-qualification effort, by creating automatically the parts of the 
>> necessary documentation, which may be automatized. Generically parts which 
>> may be automatized are:
>> -> Traceabilities (ex: Requirements against Architecture)
>> -> Requirement coverage (Tests against requirements)
>> -> Test results verification
>> -> Creation of document templates
>> -> source code metrics collection
>> -> etc.
>>
>> The current scope is to create the documentation in accordance with ESA 
>> standard (ECSS), which defines the following necessary documents (and also 
>> the template) for a software product (note: the description text of each 
>> document was taken from ECSS, for helping you to understand the goal of each 
>> document):
>
>
> This list of documents has some of which I would expect to
> be hand-written, one-time documents. Others related to requirements, 
> traceability,
> and tests, I would expect to be generated. Can you clarify which documents
> fall into which category?
>
> And how these documents related to RTEMS Pre-Qualification. For example, I 
> don't see the
> need for a Software Reuse File for RTEMS. That would seem to be something a 
> project adopting
> RTEMS might need.
>
> These documents also show an intentional bias to ECSS which is OK for you 
> guys but
> doesn't help in the RTEMS.org broader goal of having technical data for 
> qualification which
> fulfills the needs for multiple standards (NASA Quality, DO-178, automotive, 
> etc.) Please
> keep in mind that RTEMS is a world-wide project, independent of ESA and we 
> would
> like this effort to be able to address the multiple stakeholders.
>
> Some of these have the same names as other quality standard, others don't.
>
>>
>>
>> 
>> Software Development Plan (SDP) - Its purpose is to describe the established 
>> management and development approach for the software items to be defined by 
>> a software supplier to set up a software project in accordance with the 
>> customer requirements.
>>
>> Software Product Assurance Plan (SPAP) - The purpose of the software product 
>> assurance plan is to provide information on the organizational aspects and 
>> the technical approach to the execution of the software product assurance 
>> programme
>>
>> Software Configuration Management Plan (SCMP) - The objective of the 
>> configuration management plan is to provide in a single document all 
>> elements necessary to ensure that the implementation of configuration 
>> management meets customer requirements and is commensurate with the 
>> programme or project, organization, and management structure.
>>
>> Software Configuration File (SCF) - The objective of the software 
>> configuration file is to provide the configuration status of the software 
>> configuration item. It controls its evolution during the programme or 
>> project life cycle.
>>
>> Software Reuse File (SRF) - Its purpose is to document the analysis to be 
>> performed on existing software intended to be reused.
>> The global objectives of the software reuse file are to document all the 
>> information used to decide about the reuse (or

Re: Requirement Document generator tool

[Joel Sherrill]

On Mon, Dec 16, 2019 at 7:29 AM Jose Valdez  wrote:

> Hello Chris,
>
> Thank you for your reply.
>
> Please find below my answers.
>
> José
>
> -Original Message-
> From: Chris Johns [mailto:chr...@rtems.org]
> Sent: sexta-feira, 13 de dezembro de 2019 19:09
> To: Jose Valdez; devel@rtems.org
> Subject: Re: Requirement Document generator tool
>
> On 14/12/19 3:09 am, Jose Valdez wrote:
> > In the scope of ESA's RTEMS SMP project, we are developing a tool which
> will
> > allow to generate RTEMS software documentation
>
> I assume this is the pre-qual effort.
>
> JV: That's correct.
>
> > (Requirements, Design, Test Specification and other software documents).
>
> How do these documents relate to the project's current documentation? It
> may
> help to explain the purpose of each of these documents and the target user.
>



> JV: The idea of the project is to allow an user to pre-qualify RTEMS for
> space applications. In that sense, the goal of the tool is to save the
> pre-qualification effort, by creating automatically the parts of the
> necessary documentation, which may be automatized. Generically parts which
> may be automatized are:
> -> Traceabilities (ex: Requirements against Architecture)
> -> Requirement coverage (Tests against requirements)
> -> Test results verification
> -> Creation of document templates
> -> source code metrics collection
> -> etc.
>
> The current scope is to create the documentation in accordance with ESA
> standard (ECSS), which defines the following necessary documents (and also
> the template) for a software product (note: the description text of each
> document was taken from ECSS, for helping you to understand the goal of
> each document):
>

This list of documents has some of which I would expect to
be hand-written, one-time documents. Others related to requirements,
traceability,
and tests, I would expect to be generated. Can you clarify which documents
fall into which category?

And how these documents related to RTEMS Pre-Qualification. For example, I
don't see the
need for a Software Reuse File for RTEMS. That would seem to be something a
project adopting
RTEMS might need.

These documents also show an intentional bias to ECSS which is OK for you
guys but
doesn't help in the RTEMS.org broader goal of having technical data for
qualification which
fulfills the needs for multiple standards (NASA Quality, DO-178,
automotive, etc.) Please
keep in mind that RTEMS is a world-wide project, independent of ESA and we
would
like this effort to be able to address the multiple stakeholders.

Some of these have the same names as other quality standard, others don't.


>
>
> 
> Software Development Plan (SDP) - Its purpose is to describe the
> established management and development approach for the software items to
> be defined by a software supplier to set up a software project in
> accordance with the customer requirements.
>
> Software Product Assurance Plan (SPAP) - The purpose of the software
> product assurance plan is to provide information on the organizational
> aspects and the technical approach to the execution of the software product
> assurance programme
>
> Software Configuration Management Plan (SCMP) - The objective of the
> configuration management plan is to provide in a single document all
> elements necessary to ensure that the implementation of configuration
> management meets customer requirements and is commensurate with the
> programme or project, organization, and management structure.
>
> Software Configuration File (SCF) - The objective of the software
> configuration file is to provide the configuration status of the software
> configuration item. It controls its evolution during the programme or
> project life cycle.
>
> Software Reuse File (SRF) - Its purpose is to document the analysis to be
> performed on existing software intended to be reused.
> The global objectives of the software reuse file are to document all the
> information used to decide about the reuse (or not) of existing software
> and to plan the specific actions undertaken to ensure that the reused
> software meets the project requirements.
> The SRF is also used to document software developed for intended reuse,
> such that it is ready when the software is actually reused.
>
> Software User Manual (SUM) - Its purpose is to provide instructions for
> the users of the software.
> For flight software, the relevant parts of the SUM are a contribution to
> the flight operation manual (FOM).
>
> Software Validation Specification (SVS) - The purpose of this DRD is to
> des

RE: Requirement Document generator tool

[Jose Valdez]

Hello Chris,

Thank you for your reply.

Please find below my answers.

José

-Original Message-
From: Chris Johns [mailto:chr...@rtems.org] 
Sent: sexta-feira, 13 de dezembro de 2019 19:09
To: Jose Valdez; devel@rtems.org
Subject: Re: Requirement Document generator tool

On 14/12/19 3:09 am, Jose Valdez wrote:
> In the scope of ESA's RTEMS SMP project, we are developing a tool which will
> allow to generate RTEMS software documentation 

I assume this is the pre-qual effort.

JV: That's correct.

> (Requirements, Design, Test Specification and other software documents).

How do these documents relate to the project's current documentation? It may
help to explain the purpose of each of these documents and the target user.

JV: The idea of the project is to allow an user to pre-qualify RTEMS for space 
applications. In that sense, the goal of the tool is to save the 
pre-qualification effort, by creating automatically the parts of the necessary 
documentation, which may be automatized. Generically parts which may be 
automatized are:
-> Traceabilities (ex: Requirements against Architecture)
-> Requirement coverage (Tests against requirements)
-> Test results verification
-> Creation of document templates
-> source code metrics collection
-> etc.

The current scope is to create the documentation in accordance with ESA 
standard (ECSS), which defines the following necessary documents (and also the 
template) for a software product (note: the description text of each document 
was taken from ECSS, for helping you to understand the goal of each document):


Software Development Plan (SDP) - Its purpose is to describe the established 
management and development approach for the software items to be defined by a 
software supplier to set up a software project in accordance with the customer 
requirements.

Software Product Assurance Plan (SPAP) - The purpose of the software product 
assurance plan is to provide information on the organizational aspects and the 
technical approach to the execution of the software product assurance programme

Software Configuration Management Plan (SCMP) - The objective of the 
configuration management plan is to provide in a single document all elements 
necessary to ensure that the implementation of configuration management meets 
customer requirements and is commensurate with the programme or project, 
organization, and management structure.

Software Configuration File (SCF) - The objective of the software configuration 
file is to provide the configuration status of the software configuration item. 
It controls its evolution during the programme or project life cycle.

Software Reuse File (SRF) - Its purpose is to document the analysis to be 
performed on existing software intended to be reused.
The global objectives of the software reuse file are to document all the 
information used to decide about the reuse (or not) of existing software and to 
plan the specific actions undertaken to ensure that the reused software meets 
the project requirements.
The SRF is also used to document software developed for intended reuse, such 
that it is ready when the software is actually reused.

Software User Manual (SUM) - Its purpose is to provide instructions for the 
users of the software.
For flight software, the relevant parts of the SUM are a contribution to the 
flight operation manual (FOM).

Software Validation Specification (SVS) - The purpose of this DRD is to 
describe the testing, analysis, inspection and
review of design specifications, and is used to document:
-> the software validation specification with respect to the technical 
specification (TS), and
-> the software validation specification with respect to the requirements 
baseline (RB).
It provides a unique template for the software validation specification 
document, to be instantiated for, either the technical specification, or the
requirement baseline. The acronym SVS w.r.t. TS is used to designate the 
software validation specification with respect to the technical specification 
whilst SVS w.r.t. RB is used to designate the software validation specification 
with respect to the requirement baseline.

Software Design Document (SDD) - It provides description of the software 
architectural design and the software detailed design. Internal interfaces 
design is also included in this document.

Software Release Document (SRelD) - Its purpose is to describe a given software 
version in terms of known problems, limitations or restrictions with respect to 
its approved baseline.

Interface Control Document (ICD) - It describes all the (preliminary and 
update) external interfaces.

Software Product Assurance Milestone Report (SPAMR) - The main purpose of the 
software product assurance milestone report is to collect and present at 
project milestones the reporting on the 

Re: Requirement Document generator tool

[Chris Johns]

On 14/12/19 3:09 am, Jose Valdez wrote:
> In the scope of ESA’s RTEMS SMP project, we are developing a tool which will
> allow to generate RTEMS software documentation 

I assume this is the pre-qual effort.

> (Requirements, Design, Test Specification and other software documents).

How do these documents relate to the project's current documentation? It may
help to explain the purpose of each of these documents and the target user.

> Our first task will be to develop and integrate the SRS Manager into RTEMS
> project. 

What does SRS mean? What is it managing?

Could you please explain in more detail how you are planing to do the
development? As an open source project we prefer to see things that end up in
RTEMS being developed publicly and on our devel@ list. It makes the process of
merging smoother.

> This SRS Manager reads requirements written in doorstop format (each
> topic is a doorstop document) and produces the SRS document according with 
> ECSS
> format (the output of the tool are sphinx files, which then are compiled to 
> pdf
> document).

I do not understand what is meant by "doorstop document"? A work flow may help
where we see the types of data used. I ask this because a document generator
needs to have the requirements data being held somewhere and I do not know where
that is.

Does the tool generate ReST format output?

Is the ECSS format a template given to sphinx? We would need to be able to
support other formats.

We typically require HTML _and_ PDF.

> Could you indicate how I can integrate our tool into RTEMS git repository?

Commit access to the repositories is given to people who work with the project
and are active for a period of time. You will need to post changes and get the
support of a core development before the code can be merged.

> Should be in rtems-tools? If so, in which place?

I think this is the best place given the information I have. I would need to
have more detail before I could provide any specific direction here.

I would like to clearly state I fully support the pre-qual effort and I thank
ESA for making this happen. It is exciting for RTEMS to have this activity.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel