RE: Tool Roadmap for the RTEMS Pre-Qualification

[Jose Valdez]

Hello Gedare,

In the sequence of your last e-mail, I spent some time to putting to work again 
the CLANG Static Analyzer.
About one year ago, I run the CLANG Static Analyzer version 7.0.0 in my Debian 
9 machine. I followed the next steps to install it:

tar xvf llvm-7.0.0.src.tar.xz
mv llvm-7.0.0.src llvm
cd llvm/tools/
tar xvf ../../cfe-7.0.0.src.tar.xz
mv cfe-7.0.0.src/ clang
cd clang/tools/
tar xvf ../../../../clang-tools-extra-7.0.0.src.tar.xz
mv clang-tools-extra-7.0.0.src/ extra
cd ../../../../
mkdir build
cd build/
cmake -G "Unix Makefiles" ../llvm
make

Meanwhile, I upgraded my machine to Debian 10 and I tried to install the CLANG 
Static Analyzer head version (11).
This version did not work for me and I tried (successfully) the default CLANG 
package provided in the system (version 7.0.1-8):

sudo apt-get install clang-format clang-tidy clang-tools clang libc++-dev 
libc++1 libc++abi-dev libc++abi1 libclang-dev libclang1 liblldb-dev 
libllvm-ocaml-dev libomp-dev libomp5 lld lldb llvm-dev llvm-runtime llvm 
python-clang

Note that the "build from source" CLANG installation (steps above) takes about 
10 hours! If your system provides a default CLANG package, I suggest to use it, 
to avoid the long-time build from source.

After installing the CLANG, you need to install the rtems source builder, clone 
and bootstrap RTEMS (as usual).
After this you shall follow the following steps, the example here is for gr740 
which is a 4-core SPARC target:

1) Change the file rtems-master/cpukit/include/rtems/score/cpustdatomic.h, by 
removing the "const" qualifier on the parameter "*obj" of the header of the 
functions _CPU_atomic_Load_uint, _CPU_atomic_Load_ulong and 
_CPU_atomic_Load_uintptr.

-> I cannot explain you very well the rationale of this step, but there
-> is a conflict with the "const" qualifier in these variables when
-> CLANG scans RTEMS


2) Set the C_INCLUDE_PATH environment variable: export 
C_INCLUDE_PATH=/sparc-rtems5/include/:/lib/gcc/sparc-rtems5/7.3.0/include/
Note that the order of the two paths matter (do not revert the order of these 
two paths).

-> This is for CLANG to know the path of the RTEMS included files from the 
source builder tools.

3) Add the toolchain binary folder to the path variable:
export PATH=/bin:${PATH}

4) In the build directory (placed alongside RTEMS) execute the following 
commands:

scan-build --use-cc=/bin/sparc-rtems5-gcc 
--use-c++=/bin/sparc-rtems5-g++ 
--analyzer-target=sparc-rtems-unknown ../rtems/configure --target=sparc-rtems5 
--disable-posix --enable-smp --enable-cxx --disable-networking --enable-drvmgr 
--disable-tests --prefix= CFLAGS="-D__CLANG_ATOMICS 
-D__rtems__"  --enable-rtemsbsp="gr740"

scan-build --use-cc=/bin/sparc-rtems5-gcc 
--use-c++=/bin/sparc-rtems5-g++ 
--analyzer-target=sparc-rtems-unknown make all

-> Note that each normal RTEMS step is preceded by " scan-build 
--use-cc=/bin/sparc-rtems5-gcc 
--use-c++=/bin/sparc-rtems5-g++ 
--analyzer-target=sparc-rtems-unknown ". If you install the CLANG Static 
Analyzer from the source (i.e not use the sytem one), then in the scan-build 
command you shall precede it with its complete path. The flags 
CFLAGS="-D__CLANG_ATOMICS -D__rtems__" for the scan-build to run (they are not 
used in a normal RTEMS compilation).

The e-mail with the results of scan-build was blocked in your server.

Tell me if you are able to build it.

For the

"It might be better to have a parseable intermediate output for the results in 
a well-structured format (e.g., XML, yaml, etc.). The report in sphinx makes 
sense, but there should be a structured output file that could be used for 
example by other report generators."

I agree that it might be convenient, and probably not hard to do.

José

-Original Message-
From: Gedare Bloom [mailto:ged...@rtems.org]
Sent: quarta-feira, 19 de fevereiro de 2020 19:03
To: Jose Valdez
Cc: sebastian huber; rtems-de...@rtems.org
Subject: Re: Tool Roadmap for the RTEMS Pre-Qualification

On Wed, Feb 19, 2020 at 8:44 AM Jose Valdez  wrote:
>
> Hello Gedare,
>
> Thank you for your review.
>
> Please find my answers below.
>
> Best regards
>
> José
>
> -Original Message-
> From: Gedare Bloom [mailto:ged...@rtems.org]
> Sent: quarta-feira, 19 de fevereiro de 2020 15:05
> To: Jose Valdez
> Cc: sebastian huber; rtems-de...@rtems.org
> Subject: Re: Tool Roadmap for the RTEMS Pre-Qualification
>
> Hi Jose,
>
> Thank you for the detailed information. I have a few questions:
> 1) Will the output of test analysis and report generator tools be
> plaintext, and will it be structured or unstructured (flat file)?
>
>
> [Jose] The final output will be in sphinx (rst) format.
>

It might be better to have a parseable intermediate output for the results in a 
well-structured format (e.g.,

RE: Tool Roadmap for the RTEMS Pre-Qualification

[Jose Valdez]

Hello Gedare,

Thank you for your review.

Please find my answers below.

Best regards

José

-Original Message-
From: Gedare Bloom [mailto:ged...@rtems.org]
Sent: quarta-feira, 19 de fevereiro de 2020 15:05
To: Jose Valdez
Cc: sebastian huber; rtems-de...@rtems.org
Subject: Re: Tool Roadmap for the RTEMS Pre-Qualification

Hi Jose,

Thank you for the detailed information. I have a few questions:
1) Will the output of test analysis and report generator tools be
plaintext, and will it be structured or unstructured (flat file)?


[Jose] The final output will be in sphinx (rst) format.

2) Do you consider the covoar tool as viable for code coverage? That
is what we have been using in the community. Joel may like to
weigh-in.


[Jose] We don't know yet. Since we are currently analyzing the RTEMS Tester, we 
will also analyze the covoar and comeback with the conclusions.
If the covoar presents good results I don't see why to not use it.

3) I want to see Clang Static Analyzer working. If there is any
possibility you all can help move us toward that direction, I think it
would be a great positive to the community. Coverity is difficult and
the license we can get freely is limited. I also have plans to work on
both Coverity and clang-analyzer improvements soon.


[Jose] In the past (2018)we were able to run the CLANG static analyzer on RTEMS 
(we had to do a minor "dirty" modification) and we get interesting results.
Maybe I could try to do it again and send you the steps to run the CLANG Static 
Analyzer, if you would like.

Gedare

On Wed, Feb 19, 2020 at 6:05 AM Jose Valdez  wrote:
>
> Hello,
>
> Following the e-mail sent by Sebastian, please find here the missing 
> information, regarding the tools for RTEMS pre qualification.
>
> Best regards
>
> José
>
> === Test Executor ===
>
> The Test Executor will be the software that manages the execution of the 
> RTEMS Tests.
> It will set the necessary hardware, run the executable tests, control the 
> execution and gather the output.
> The following capabilities will be available:
>
> * Send commands to the target platform.
>   A subset of commands will be made available, but it shall be possible to 
> add more commands.
> * Send commands to auxiliary test programs, which run in external devices.
>   This will be done by a "command interface" that will allow the Test 
> Executor to communicate with external components in a generic way.
>   This "command interface" is intended to be generic to fit any functionality 
> need.
>   Note that this capability is still to be evaluated if its implementation 
> complexity fits the the RTEMS SMP project budget.
> * Load and execute RTEMS executables to the target platform.
> * Get and store the output (log) from the running executable.
> * Wait for end of execution of the RTEMS executable.
>   This includes both cases of test termination (Successfully/Unsuccessfully) 
> and to stop execution with a timeout mechanism.
>
> Currently RTEMS Tester offers part of these functionalities.
> An evaluation in undergoing to assess how RTEMS Tester fits these 
> capabilities and the possibility to add the missing features.
> The functionalities described above will be subject to validation to make 
> sure that the Test Executor is suitable to pre qualify RTEMS for critical 
> missions.
>
> In the scope of RTEMS SMP project, the following specific functionalities are 
> required from the above generic functionality:
>
> * Send a board reset command (to allow to set the board to boot-up 
> conditions).
> * Load and execute tests in the tsim-leon2, tsim-leon3, gr712rc and gr740 
> targets.
> * Use an auxiliary PC to test MIL-STD-1553 interface.
>   The need for this PC is due that the gr712rc and gr740 targets only support 
> a single MIL-STD-1553 interface, which makes impossible loopback tests.
>   This auxiliary PC will contain a MIL-STD-1553 interface and will run an 
> application which will translate the "command interface" messages into 
> command to this auxiliary hardware.
>   Note that this activity depends on the assessment of the possibility to 
> have this command interface.
> * Receive test output either by UART or the standard output.
>
>
> === Test Report Analyser and Report Generator ===
>
> The Test Analyser will receive the reports produced by the Test Executor, 
> which will follow the `Test Framework 
> <https://lists.rtems.org/pipermail/devel/2019-March/025178.html>`_ structure 
> and assess the status of the tests.
> The possible status for each test are presented  `here 
> <https://docs.rtems.org/branches/master/user/testing/tests.html#test-controls>`_.
> For each test report, the Test Report Analyser will scan the requirements 
> targeted by 

RE: Tool Roadmap for the RTEMS Pre-Qualification

[Jose Valdez]

Hello,

Following the e-mail sent by Sebastian, please find here the missing 
information, regarding the tools for RTEMS pre qualification.

Best regards

José

=== Test Executor ===

The Test Executor will be the software that manages the execution of the RTEMS 
Tests.
It will set the necessary hardware, run the executable tests, control the 
execution and gather the output.
The following capabilities will be available:

* Send commands to the target platform.
  A subset of commands will be made available, but it shall be possible to add 
more commands.
* Send commands to auxiliary test programs, which run in external devices.
  This will be done by a "command interface" that will allow the Test Executor 
to communicate with external components in a generic way.
  This "command interface" is intended to be generic to fit any functionality 
need.
  Note that this capability is still to be evaluated if its implementation 
complexity fits the the RTEMS SMP project budget.
* Load and execute RTEMS executables to the target platform.
* Get and store the output (log) from the running executable.
* Wait for end of execution of the RTEMS executable.
  This includes both cases of test termination (Successfully/Unsuccessfully) 
and to stop execution with a timeout mechanism.

Currently RTEMS Tester offers part of these functionalities.
An evaluation in undergoing to assess how RTEMS Tester fits these capabilities 
and the possibility to add the missing features.
The functionalities described above will be subject to validation to make sure 
that the Test Executor is suitable to pre qualify RTEMS for critical missions.

In the scope of RTEMS SMP project, the following specific functionalities are 
required from the above generic functionality:

* Send a board reset command (to allow to set the board to boot-up conditions).
* Load and execute tests in the tsim-leon2, tsim-leon3, gr712rc and gr740 
targets.
* Use an auxiliary PC to test MIL-STD-1553 interface.
  The need for this PC is due that the gr712rc and gr740 targets only support a 
single MIL-STD-1553 interface, which makes impossible loopback tests.
  This auxiliary PC will contain a MIL-STD-1553 interface and will run an 
application which will translate the "command interface" messages into command 
to this auxiliary hardware.
  Note that this activity depends on the assessment of the possibility to have 
this command interface.
* Receive test output either by UART or the standard output.


=== Test Report Analyser and Report Generator ===

The Test Analyser will receive the reports produced by the Test Executor, which 
will follow the `Test Framework 
`_ structure 
and assess the status of the tests.
The possible status for each test are presented  `here 
`_.
For each test report, the Test Report Analyser will scan the requirements 
targeted by the test and, depending on the result of the test, will set the 
requirement as OK or Not OK.
Note also the result of a requirement tested by several tests will be the 
logical conjunction of the result of the associated tests (that is, a 
requirement is considered passed only if all the tests result is the expected 
for that requirement).

The Report Generator will gather the information from the Test Report Analyser 
and will also check for `other validation 
`_
 performed to the requirements.
These validation items are written in doorstop format for each requirement, 
containing the result of the validation activity performed to the requirement, 
including the final assessment (OK or Not OK, as for testing and respective 
justification.
The Report Generator will read each of these validattion and, as for the 
requirements validated by test, set the requirement as OK or Not OK, depending 
on the result of the validation.


=== Test Plan Generator ===

The Test Plan will read the following information written in doorstop format:

* `Test Suite Specifications 
`_, which 
will contain the testsuite general description.
* `Test Procedure Specifications 
`_, 
which will contain the set-up for each test configuration.
* `Test Case Specifications 
`_, which 
will contain the steps for each test.

The information present in the above specifications will be printed in the Test 
Plan document, which will contain all the tests specifications and the 
necessary set-ups in order to run the complete test suite for RTEMS SMP.
Note that the contents of the above specifications are still to be refined.
The Test Procedure Specifications may also have the following additional 
information:

* configurations - 

RE: Requirement Document generator tool

[Jose Valdez]

Hello Everyone,

Sorry for my late reply in this topic, but I have not been able to work on 
this. I will try to reach all the topics/problems raised.

In fact these tools target the pre-qualified project. Since it was Sebastian 
who suggested to create this set of python tools, I think the idea was to 
standardize the use of python not only for this project, but also for other 
python written code in RTEMS community. This has the advantages that every 
written python code is standard, but has the drawbacks:

-> old written code would need to be adapted to the standards. Another option 
could be leave it as it is and only do this for new written code.
-> at some point some tools need to be upgraded (ex: python 3.7 will become 
unusable in 2030 Operating systems).

I hope soon to formalize our suggestion to you and then you may review it (and 
propose changes if you find appropriate).

José



-Original Message-
From: Christian Mauderer [mailto:christian.maude...@embedded-brains.de]
Sent: quinta-feira, 23 de janeiro de 2020 06:42
To: Chris Johns; j...@rtems.org
Cc: Jose Valdez; Gedare Bloom; sebastian huber; devel@rtems.org
Subject: Re: Requirement Document generator tool

On 23/01/2020 00:19, Chris Johns wrote:
> On 22/1/20 7:20 pm, Christian Mauderer wrote:
>> On 22/01/2020 02:39, Chris Johns wrote:
>>> On 21/1/20 8:10 pm, Christian Mauderer wrote:
>>>> On 20/01/2020 22:37, Chris Johns wrote:
>>>>> On 14/1/20 7:18 pm, Christian Mauderer wrote:
>>>>>> On 13/01/2020 18:03, Joel Sherrill wrote:
>>>>>>> On Mon, Jan 13, 2020 at 10:04 AM Christian Mauderer
>>>>>>> >>>>>> <mailto:christian.maude...@embedded-brains.de>> wrote:
>>>>>>>
>>>>>>> Hello Jose,
>>>>>>>
>>>>>>> On 10/01/2020 17:33, Jose Valdez wrote:
>>>>>>> > Hello,
>>>>>>> >
>>>>>>> > Regarding this topic and to start to define the python tools that
>>>>>>> are more appropriate for the RTEMS community python developments, I
>>>>>>> would like to propose the following tools (to be placed in RTEMS
>>>>>>> Software Engineering manual):
>>>>>>> >
>>>>>>> > Python language version: 3.6 (minimum version for Doorstop 
>>>>>>> 2.0.post2)
>>>>>>> > Python coding style/standard - Google (as suggested by Gedare)
>>>>>>> > Python documentation style - Google docstrings (to be in
>>>>>>> accordance with coding style. Note it is supported by sphinx)
>>>>>>> > Python test approach - unittest (seems to be the standard used by
>>>>>>> python)
>>>>>>> > Python static analysis - pylint (recommended by Google style, see
>>>>>>> http://google.github.io/styleguide/pyguide.html) and mypy (catch
>>>>>>> additional errors) Python code coverage - Coverage.py (seems to be
>>>>>>> the standard used by python)
>>>>>>> > Python third party packages - For now we are using: paramiko,
>>>>>>> pyserial, pexpect, gitpython, but the list is expected to be
>>>>>>> changed. I think here any third package is allowed, as long as the
>>>>>>> license of the package complies with RTEMS project license.
>>>>>>>
>>>>>>> To be future-proof: python3 compatibility would be nearly a 
>>>>>>> must-have
>>>>>>> for any new library.
>>>>>>>
>>>>>>>
>>>>>>> We discussed this somewhere earlier and I thought we decided that if
>>>>>>> it was a core tool required for a "normal user", then it would have to
>>>>>>> be Python 2.x and 3.x compatible. Tools for "requirements user" or
>>>>>>> "pre-qualification user" (pick a name for this role), can be 3.x only
>>>>>>> mainly because there was so much infrastructure required for the
>>>>>>> "requirements user" that already needed Python 3.x.
>>>>>
>>>>> Yes this is correct. Python is used in the rtems-tools repo and any 
>>>>> program in
>>>>> the rtems-tools repo needs to support Python 2 and Python 3 no matter 
>>>>> what role
>>>>> it performs. The rtems-tools repo is not the 

RE: Requirement Document generator tool

[Jose Valdez]

Hello,

Regarding this topic and to start to define the python tools that are more 
appropriate for the RTEMS community python developments, I would like to 
propose the following tools (to be placed in RTEMS Software Engineering manual):

Python language version: 3.6 (minimum version for Doorstop 2.0.post2)
Python coding style/standard - Google (as suggested by Gedare)
Python documentation style - Google docstrings (to be in accordance with coding 
style. Note it is supported by sphinx)
Python test approach - unittest (seems to be the standard used by python)
Python static analysis - pylint (recommended by Google style, see 
http://google.github.io/styleguide/pyguide.html) and mypy (catch additional 
errors) Python code coverage - Coverage.py (seems to be the standard used by 
python)
Python third party packages - For now we are using: paramiko, pyserial, 
pexpect, gitpython, but the list is expected to be changed. I think here any 
third package is allowed, as long as the license of the package complies with 
RTEMS project license.

Please tell me your opinions for each python tool and then we can start to 
agree on what to use. Note that once this is defined, our Qualification 
software will comply with the RTEMS community chosen tools.

Best regards

José

-Original Message-
From: devel [mailto:devel-boun...@rtems.org] On Behalf Of Chris Johns
Sent: quinta-feira, 9 de janeiro de 2020 20:57
To: Gedare Bloom; sebastian huber
Cc: devel@rtems.org
Subject: Re: Requirement Document generator tool

On 10/1/20 4:45 am, Gedare Bloom wrote:
> On Wed, Jan 8, 2020 at 11:51 PM Sebastian Huber
>  wrote:
>>
>> On 08/01/2020 19:31, Gedare Bloom wrote:

 I agree completely on the proposed approach with Python tools.

>>> Yes. Reading it I'm actually reminded about Google's approach toward
>>> Python which includes many of the elements mentioned. Although their
>>> guide is probably more comprehensive and verbose that what we need, it
>>> might be a useful reference for developing a set of guidelines
>>> suitable for Python code in RTEMS (mainly, rtems-tools).  Here is a
>>> link:
>>> http://google.github.io/styleguide/pyguide.html
>>>
>>> I think most of the existing style has been determined and driving by
>>> Chris Johns. So I would also give him some credit to develop/approve
>>> how we plan to use Python at a project level. (**Hands Chris an "RTEMS
>>> Python Maintainer" hat**);)
>>
>> I think the Google guide would be a good start. We can always add
>> project-specific exceptions/clarifications if necessary. My aim is to
>> use it for new code, e.g. code produced for the pre-qualification
>> activity. For the code format I strongly want to use a tool for this. I
>> don't want to spend review time on code formatting issues.
>>
>> Using standard guidelines makes the RTEMS Project more attractive for
>> new contributors and GSoC students. I think it increases your job
>> opportunities if you can refer to a successful GSoC project and it shows
>> that you used standard engineering practices in the project. This is
>> usually not something a university education includes.
>>
> OK, both points make sense. I'd be happy with the Google guide, I hope
> Chris will comment when he can.

Sorry about the erratic access. I have been knee deep in painting and flooring
this summer as I avoid any smoke from the fires we are having.

I am fine with using a standard guide however we need to review it and to make
sure it fits. As an example the C++ guide from Google had some good points as
well as some parts I did not think offered any value but did create a certain
level of pain. We should also consider capturing the guide as a public one
belonging to someone else can and will change and that effects us. I am not sure
how we could do this.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

RE: Requirement Document generator tool

[Jose Valdez]

Same message with no attachs.

-Original Message-
From: Jose Valdez
Sent: sexta-feira, 3 de janeiro de 2020 10:49
To: 'Gedare Bloom'; Joel Sherrill
Cc: devel@rtems.org
Subject: RE: Requirement Document generator tool

Hello Gedare and Joel,

Please see my answers below for both of you (see JV02012019).

Please tell me, if you need further clarification.

Best regards

José

-Original Message-
From: Gedare Bloom [mailto:ged...@rtems.org]
Sent: quinta-feira, 2 de janeiro de 2020 17:21
To: Joel Sherrill
Cc: Jose Valdez; devel@rtems.org
Subject: Re: Requirement Document generator tool

Hello José et al.,

I have only a few points I would like clarified, below:

On Mon, Dec 16, 2019 at 3:35 PM Joel Sherrill  wrote:
>
>
>
> On Mon, Dec 16, 2019 at 7:29 AM Jose Valdez  wrote:
>>
>> Hello Chris,
>>
>> Thank you for your reply.
>>
>> Please find below my answers.
>>
>> José
>>
>> -Original Message-
>> From: Chris Johns [mailto:chr...@rtems.org]
>> Sent: sexta-feira, 13 de dezembro de 2019 19:09
>> To: Jose Valdez; devel@rtems.org
>> Subject: Re: Requirement Document generator tool
>>
>> On 14/12/19 3:09 am, Jose Valdez wrote:
>> > In the scope of ESA's RTEMS SMP project, we are developing a tool
>> > which will allow to generate RTEMS software documentation
>>
>> I assume this is the pre-qual effort.
>>
>> JV: That's correct.
>>
>> > (Requirements, Design, Test Specification and other software documents).
>>
>> How do these documents relate to the project's current documentation?
>> It may help to explain the purpose of each of these documents and the target 
>> user.
>
>
>
>>
>> JV: The idea of the project is to allow an user to pre-qualify RTEMS for 
>> space applications. In that sense, the goal of the tool is to save the 
>> pre-qualification effort, by creating automatically the parts of the 
>> necessary documentation, which may be automatized. Generically parts which 
>> may be automatized are:
>> -> Traceabilities (ex: Requirements against Architecture) Requirement
>> -> coverage (Tests against requirements) Test results verification
>> -> Creation of document templates source code metrics collection etc.
>>
>> The current scope is to create the documentation in accordance with ESA 
>> standard (ECSS), which defines the following necessary documents (and also 
>> the template) for a software product (note: the description text of each 
>> document was taken from ECSS, for helping you to understand the goal of each 
>> document):
>
>
> This list of documents has some of which I would expect to be
> hand-written, one-time documents. Others related to requirements,
> traceability, and tests, I would expect to be generated. Can you
> clarify which documents fall into which category?
>
> And how these documents related to RTEMS Pre-Qualification. For
> example, I don't see the need for a Software Reuse File for RTEMS.
> That would seem to be something a project adopting RTEMS might need.
>
> These documents also show an intentional bias to ECSS which is OK for
> you guys but doesn't help in the RTEMS.org broader goal of having
> technical data for qualification which fulfills the needs for multiple
> standards (NASA Quality, DO-178, automotive, etc.) Please keep in mind
> that RTEMS is a world-wide project, independent of ESA and we would like this 
> effort to be able to address the multiple stakeholders.
>
> Some of these have the same names as other quality standard, others don't.


JV02012019: Regarding your first question, a document itself may have both 
parts hand-written and generated. I understand that probably we missed to give 
you the full technical scope of the project (which is difficult by only 
exchanging e-mails). Please wait for our first input (Requirement Manager), 
which will include instructions on how to use and then you may perform your 
assessments and clarify your technical doubts.

JV02012019: The documents which are not applicable to RTEMS project (like 
Software Reuse File) will be created, but will not have any content (the 
sections will have text like "Not Applicable").

JV02012019: I understand your concern about having the tool producing output 
for other standards/applications. Unfortunately this is not possible in this 
project (due to budget). However, the philosophy for this project should be to 
open the possibility to extend the tool, and hence other users may add the 
functionality to produce the documentation according with their standards.

>
>>
>>
>> -
>> --

RE: Requirement Document generator tool

[Jose Valdez]

Hello Chris,

Thank you for your reply.

Please find below my answers.

José

-Original Message-
From: Chris Johns [mailto:chr...@rtems.org] 
Sent: sexta-feira, 13 de dezembro de 2019 19:09
To: Jose Valdez; devel@rtems.org
Subject: Re: Requirement Document generator tool

On 14/12/19 3:09 am, Jose Valdez wrote:
> In the scope of ESA's RTEMS SMP project, we are developing a tool which will
> allow to generate RTEMS software documentation 

I assume this is the pre-qual effort.

JV: That's correct.

> (Requirements, Design, Test Specification and other software documents).

How do these documents relate to the project's current documentation? It may
help to explain the purpose of each of these documents and the target user.

JV: The idea of the project is to allow an user to pre-qualify RTEMS for space 
applications. In that sense, the goal of the tool is to save the 
pre-qualification effort, by creating automatically the parts of the necessary 
documentation, which may be automatized. Generically parts which may be 
automatized are:
-> Traceabilities (ex: Requirements against Architecture)
-> Requirement coverage (Tests against requirements)
-> Test results verification
-> Creation of document templates
-> source code metrics collection
-> etc.

The current scope is to create the documentation in accordance with ESA 
standard (ECSS), which defines the following necessary documents (and also the 
template) for a software product (note: the description text of each document 
was taken from ECSS, for helping you to understand the goal of each document):


Software Development Plan (SDP) - Its purpose is to describe the established 
management and development approach for the software items to be defined by a 
software supplier to set up a software project in accordance with the customer 
requirements.

Software Product Assurance Plan (SPAP) - The purpose of the software product 
assurance plan is to provide information on the organizational aspects and the 
technical approach to the execution of the software product assurance programme

Software Configuration Management Plan (SCMP) - The objective of the 
configuration management plan is to provide in a single document all elements 
necessary to ensure that the implementation of configuration management meets 
customer requirements and is commensurate with the programme or project, 
organization, and management structure.

Software Configuration File (SCF) - The objective of the software configuration 
file is to provide the configuration status of the software configuration item. 
It controls its evolution during the programme or project life cycle.

Software Reuse File (SRF) - Its purpose is to document the analysis to be 
performed on existing software intended to be reused.
The global objectives of the software reuse file are to document all the 
information used to decide about the reuse (or not) of existing software and to 
plan the specific actions undertaken to ensure that the reused software meets 
the project requirements.
The SRF is also used to document software developed for intended reuse, such 
that it is ready when the software is actually reused.

Software User Manual (SUM) - Its purpose is to provide instructions for the 
users of the software.
For flight software, the relevant parts of the SUM are a contribution to the 
flight operation manual (FOM).

Software Validation Specification (SVS) - The purpose of this DRD is to 
describe the testing, analysis, inspection and
review of design specifications, and is used to document:
-> the software validation specification with respect to the technical 
specification (TS), and
-> the software validation specification with respect to the requirements 
baseline (RB).
It provides a unique template for the software validation specification 
document, to be instantiated for, either the technical specification, or the
requirement baseline. The acronym SVS w.r.t. TS is used to designate the 
software validation specification with respect to the technical specification 
whilst SVS w.r.t. RB is used to designate the software validation specification 
with respect to the requirement baseline.

Software Design Document (SDD) - It provides description of the software 
architectural design and the software detailed design. Internal interfaces 
design is also included in this document.

Software Release Document (SRelD) - Its purpose is to describe a given software 
version in terms of known problems, limitations or restrictions with respect to 
its approved baseline.

Interface Control Document (ICD) - It describes all the (preliminary and 
update) external interfaces.

Software Product Assurance Milestone Report (SPAMR) - The main purpose of the 
software product assurance milestone report is to collect and present at 
project milestones the reporting on the software produc

Requirement Document generator tool

[Jose Valdez]

Hello,

In the scope of ESA's RTEMS SMP project, we are developing a tool which will 
allow to generate RTEMS software documentation (Requirements, Design, Test 
Specification and other software documents).

Our first task will be to develop and integrate the SRS Manager into RTEMS 
project. This SRS Manager reads requirements written in doorstop format (each 
topic is a doorstop document) and produces the SRS document according with ECSS 
format (the output of the tool are sphinx files, which then are compiled to pdf 
document).

Could you indicate how I can integrate our tool into RTEMS git repository? 
Should be in rtems-tools? If so, in which place?

Best regards

José
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

RE: Design Tools for RTEMS Qualification Toolchain

[Jose Valdez]

Hello Joel,

From the statement of work of the RTEMS Qualification, we have that:

“The Software Design Document (SDD) has to cover ECSS-E-ST-40C clauses 
5.4.3.1.a, 5.4.3.2.a, 5.4.3.3.a, 5.4.3.4.a, 5.4.3.5.a, 5.4.3.6.c, 5.5.2.1.a, 
5.5.2.1.b, 5.5.2.1.c, 5.5.2.2.a, 5.5.2.3.a, 5.5.2.4.a, 5.5.2.5.a, 5.5.2.6.a, 
5.5.2.7.a, 5.5.3.1.a, 5.5.3.2.a, 5.5.3.2.b, 5.8.3.3.a, and 5.8.3.4.a and 
ECSS-Q-ST-80C clause 7.2.2.3.a.”

The Software Design Document is the document which will contain Architectural 
Design (“high level”) and Detailed Design (“low level”) of RTEMS. This means 
that the RTEMS Qualification Toolchain shall cover as much as possible the 
points above. I expect that some of the points can be done with the help of a 
tool (e.g. production of detailed component and sequence diagrams) but I 
haven’t seen a tool that does every content automatically – although of course, 
this would be great. My best guess is that the qualification toolchain will 
pick up the design that is made, probably bits of sphynx modules, aggregate 
them, do some automatic work (e.g. traceabilities, interfaces) and then 
generate a final Sphynx that could be converted to pdf. For the points above 
which cannot be automated, they have to be done manually.

I hope that this explanation answers your questions. Note also for your 
question below: “This is a goal for improving the RTEMS Project processes. Are 
these being captured?”, yes, we will do it

Best regards

José


From: Joel Sherrill [mailto:j...@rtems.org]
Sent: terça-feira, 26 de março de 2019 14:32
To: Jose Valdez
Cc: rtems-de...@rtems.org
Subject: Re: Design Tools for RTEMS Qualification Toolchain



On Mon, Mar 25, 2019 at 12:20 PM Jose Valdez 
mailto:jose.val...@edisoft.pt>> wrote:
Hello Joel,

Thank you for your reply.

I meant by design tools, UML tools (I am targeting component and sequence 
diagrams). It seems that you meant that you consider design tools the ones that 
are able to produce code from the design. Indeed was not with that goal in 
mind, because:

==> RTEMS source code is already defined (although this would not be a critical 
point, since reverse engineering would be possible, like will do the doxygen)

==> As far as I know, most of these tools target code production for 
object-oriented languages. This is not the case for RTEMS.

The term design tools covers so much territory that my first Google turned up 
SysML and MathCAD.
It is just too broad a term.


What do you mean by a use-case? An example? I tried to quick capture the main 
features of the tools and their applicability to our project and, as you may 
read in the previous e-mail, the PlantUML and blockdiag are the far the most 
suitable.

The definition of use case is " a specific situation in which a product or 
service could potentially be used."
I think that covers my intent. Define how these tools will be used and what's 
expected.

I agree that we are not generating code and are just capturing existing design.
But I still don't know what technical data/artifacts are required by the ESA 
quality standard
so can only base the expected use cases on what I would expect.

+ Document high level architecture in a new document
+ Document sequences, flows, logic primarily to enhance Doxygen output

Any diagrams could also be useful in Classic API Users Guide to document
schedulers,  etc to users.



I agree with the approach to generate detail design data from source code and 
then for the top level architecture, to use the Design selected tool. Of course 
part of the work will be to incorporate functionality that if an architectural 
change in the code is made, the user shall be warned to update the 
Architectural Design. This shall be done by linking (traceability) 
architectural components source code (that’s why defining architectural 
components in text files will be useful).

This is a a goal for improving the RTEMS Project processes. Are these being
captured?

And I would expect a fair number of false positives. I will have 30 years with 
RTEMS
this summer and the basic architecture figures rarely need to change for 
technical
reasons.

I'm not trying to be argumentative. Just looking a well-defined process for 
these
tool evaluations:

+ What will the tool(s) be used for?
+ What artifacts produced?
+ What "requirements" of the various quality standards is it meeting?
+ Does it need to integrate with build processes?

I am assuming a high level design document would be natural to go in the
RTEMS Documentation Suite. Today, I am sure the rtems-docs build system
supports both PlantUML and ditaa. If it doesn't support graphviz/dot and mscgen
(used by our Doxygen), then that's has to be addressed.

--joel


Best regards

José

From: Joel Sherrill [mailto:j...@rtems.org<mailto:j...@rtems.org>]
Sent: segunda-feira, 25 de março de 2019 16:31
To: Jose Valdez
Cc: rtems-de...@rtems.org<mailto:rtems-de...@rtems.org>
Subject: Re: Design Tools for RTEMS Quali

RE: Design Tools for RTEMS Qualification Toolchain

[Jose Valdez]

Hello Joel,

Thank you for your reply.

I meant by design tools, UML tools (I am targeting component and sequence 
diagrams). It seems that you meant that you consider design tools the ones that 
are able to produce code from the design. Indeed was not with that goal in 
mind, because:

è RTEMS source code is already defined (although this would not be a critical 
point, since reverse engineering would be possible, like will do the doxygen)

è As far as I know, most of these tools target code production for 
object-oriented languages. This is not the case for RTEMS.

What do you mean by a use-case? An example? I tried to quick capture the main 
features of the tools and their applicability to our project and, as you may 
read in the previous e-mail, the PlantUML and blockdiag are the far the most 
suitable.

I agree with the approach to generate detail design data from source code and 
then for the top level architecture, to use the Design selected tool. Of course 
part of the work will be to incorporate functionality that if an architectural 
change in the code is made, the user shall be warned to update the 
Architectural Design. This shall be done by linking (traceability) 
architectural components source code (that’s why defining architectural 
components in text files will be useful).

Best regards

José

From: Joel Sherrill [mailto:j...@rtems.org]
Sent: segunda-feira, 25 de março de 2019 16:31
To: Jose Valdez
Cc: rtems-de...@rtems.org
Subject: Re: Design Tools for RTEMS Qualification Toolchain

You haven't defined what you mean by design tool. The implication seems to be
something to do UML-ish things in.

PlantUML with DITAA, mscgen, and dot (e.g. graphviz) as backups is a pretty
good combination for drawing diagrams.

But they are not design tools IMO. They are tools to produce diagrams which
can be used in design documents. When you say design tools, I think tools
like Enterprise Architect, Magic Draw, etc.

And for design, there are multiple levels. As a minimum, I would expect a
higher level architectural view and a low level software design. The former
is primarily going to be abstract with figures and descriptions. Probably a
document when it is done. The low level design should be able to be
captured via Doxygen and the drawing tools above.

Please define the use cases for the tool before dropping into specific tools.

--joel

On Mon, Mar 25, 2019 at 10:52 AM Jose Valdez 
mailto:jose.val...@edisoft.pt>> wrote:
Hello,

I have been doing the Design Tools investigation. I had investigated several 
tools with the following features in mind:


==> Open Source and active project

==> Text-file based tool, which allows git control and manipulation of the 
files with other tools (to add extra functionality)

==> Diagram export to image and possibly to ascii art (if we want to 
incorporate some diagrams into source code)

==> User friendly

==> Both component and sequence diagrams available

After my investigation I made the following short list:


==> PlantUML: It has all characteristics above listed. Unfortunately, the 
export to ascii art diagrams feature is only available in the PlantUML online 
editor, also, there is no GUI (at least which I am aware of, but no big 
problem) for editing diagrams. Note that although the tool is called PlantUML, 
PlantUML is itself a standard, so there are several tools, which can handle it.

==> Modelio: More GUI based than text file based. Import/Export from/to html 
like files is possible, however in a hard-to-understand format

==> Violet: Also a GUI based. It has a simpler interface, but slower (annoying) 
than Modelio. The model is saved into html in a hard-to-read format.

==> UMLet: Also GUI interface (seems not 100 % functional). It outputs to html, 
but this html only contains the position of the elements in the UML drawing and 
not the relation between components

==> ASCIIFlow: Not properly an UML design tool, but it allows to quickly design 
simple component diagrams in ascii art, which can be included in code. Also 
allows to import and edit already existing ascii drawings. I believe It will 
not be used, but was put here as a reference.

==> blockdiag and seqdiag: (http://blockdiag.com/en/), blockdiag provides two 
packages (shall be installed separately), blockdiag and seqdiag to, 
respectively, draw block and sequence diagrams. The input is text file which 
allows to generate the diagrams. Has also interface with sphinx and probably it 
is possible to represent the diagrams in asci art.

I would say that the best candidates are PlantUML and blockdiag.

I found other tools which are either dead projects (most of them) or have 
missing features. I will write down the list, If anyone is interested to have a 
look:

==> ARGOUML

==> UMBRELLO

==> Dia

==> StartUML

==> UMPLE

==> ZenUML

I know that there is the idea to use the doxygen to generate architectural 
components from the

RE: RTEMS Requirement Management Tools

[Jose Valdez]

Hello All,

Sorry for my late reply and thanks to the people who answered these questions. 
I believe these positive answers reinforces the choice for doorstop.

For the requirement analysis, I meant that this feature is descoped (too much 
complexity to make it) and it will be the responsibility of the user to define 
good requirements. Of course, in future, if a good requirement analysis tool 
shows up, it could be integrated in the Qualification Toolchain.

Best regards

José

From: devel [mailto:devel-boun...@rtems.org] On Behalf Of Joel Sherrill
Sent: sexta-feira, 22 de março de 2019 20:18
To: sebastian huber
Cc: RTEMS
Subject: Re: RTEMS Requirement Management Tools



On Fri, Mar 22, 2019 at 5:28 AM Sebastian Huber 
mailto:sebastian.hu...@embedded-brains.de>> 
wrote:
Hello,

I searched a bit for Doorstop use cases. I found this:

https://arxiv.org/abs/1807.05422

It was used for this NASA mission:

https://wfirst.gsfc.nasa.gov/observatory.html

Ironically enough, WFIRST uses RTEMS for at least one subsystem.

Plus we have the report on the list from DLR.

--joel



--
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail  : 
sebastian.hu...@embedded-brains.de
PGP : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

RE: Requirement Management tools

[Jose Valdez]

Hello Mikail,

Thank you for contributing to this discussion.

I have been also myself investigating the requirement management tools. 
Currently, I converged into the three tools:


è rmToo

è papyrus

è doorstop

During this week,  I took a deeper look into rmToo and doorstop and today I am 
into papyrus. My current conclusion of the analysis of rmToo vs doorstop is 
that the doorstop is more generic (it allows trace between other items besides 
requirements, ex: source code and tests), but it has the price that it has not 
so many specific functionalities for requirements. For example rmToo has the 
following that doorstep does not have:

è requirement analysis (for example looks for forbidden words/expressions in 
the requirements)

è more information in the requirements that doorstop (some useful, other maybe 
not), for example: author of the requirement, effort, status, priority, etc

è constraints can be added (although using like some of formal language written 
in python)

For the remaining features, I would say that the tools are similar (they have 
the same philosophy, although rmToo seems to be more complete but with no 
capability to trace to other items as doorstop).

Best regards

José

From: Mikail Yayla [mailto:mikail.ya...@tu-dortmund.de]
Sent: sexta-feira, 15 de março de 2019 10:33
To: Jose Valdez
Cc: sebastian huber; devel@rtems.org
Subject: Re: Requirement Management tools

Hello all,

I am helping Sebastian trying out different requirement management tools.
We are currently evaluating doorstop and how well it can be used.

As an example we created a draft of a possible requirements management document 
structure in doorstop for the queue creation with rtems_message_queue_create.
Here is the created document structure: https://depot.tu-dortmund.de/rfek8

In the folder pub there are the automatically generated HTML files of the 
requirement documents which can be viewed in the browser. In the rtems folder 
are a few source files.  The other folders (header, include, llt, reqs, source) 
are for doorstop documents.

I have tried out ProR and Papyrus as well, and in comparison to those two, 
getting accustomed to doorstop was much simpler. There are only a few command 
line directives, and using them allows creating a decent overview of the 
document structure in HTML in a short time.

Here are the pros and cons for doorstop from my view:

Pros:
- Easy to use, either using the command line interface or creating and editing 
the document files by hand.
- Automatic generation of the HTML overview. We can work on the requirement 
document and see the changes immediately in the HTML using the publish 
directive.
- Traceability is enabled by the file structure with fingerprints. When changes 
occur in the documents, these fingerprints are changed. The doorstop files live 
inside a version controlled folder, so changes can be traced.
- Markup can be used in the text fields of the document files

Cons:
- It is a bit inconvenient to use a stringent way of naming files. I stopped 
using the command line interface and started creating files by hand because of 
this. The author also has this issue in this example: 
http://jacebrowning.github.io/doorstop/index.html in 
http://jacebrowning.github.io/doorstop/REQ.html
- Files can only reference one (source-)file. For links between doorstop files, 
when a file has multiple child files, there are problems in the HTML. Because 
of this I did not use links between files yet (links are used to detect changes 
in the parent, as these could have an influence on the children)

Possible bugs:
- For the ref field, in the documentation it says doorstop will first search 
the file in the ref field (e.g. 'message.h') in the root and its subdirs, and 
if no file was found it will search for matches in text files. However, for 
INCLUDE001, the file 'rtems.h' is searched first although the file 'message.h' 
exists. I'm still trying to find out the issue.
- In the item traceability table, some links disappear, and I haven't found out 
yet why. For example, the files for HEADER are now missing. I think it has to 
do with creating links between files, in the case when many children refer to a 
single mutual parent file.
- Sometimes the HTML has to be cleaned by hand from old HTML files from 
documents which were already deleted.

Other Points and ideas to extend doorstop:
- We could try to adapt doorstop such that it accepts reST in the text fields, 
then we would use text from the documentation.
- A visualization of the document structure as a graph would be neat
- There is a GUI but it is experimental, can only be used to view the document.

If you have some other ideas or other things we should try with doorstop, 
please let us know.

Here is the summary of the materials:
The paper with some use cases for doortstop: 
https://www.researchgate.net/publication/276044183_Doorstop_Text-Based_Requirements_Management_Using_Version_Control
The example from the author: http

RE: Requirement Management tools

[Jose Valdez]

Hello Sebastian,

Thank you for creating this ticket. I believe this will help, since now the 
RTEMS community can contribute with other tools and add other tool selection 
criteria.

Just a note that some selection criteria can be classified as "hard" (that 
means must have or must not have) and other as soft ("desirable" having/not 
having).

I believe that you saw (but devel rtems did not), Andre Ribeiro proposed other 
tools:
-> http://testlink.org
->http://www.eclipse.org/osee

As far as I have seen, they use database, so they cannot be used, but maybe I 
can add them on the report for reference. Also myself take a look into another 
tools. From my research I concluded that almost tools were projects that 
started some years ago, but are deprecated now.

Best regards

José
 

-Original Message-
From: Sebastian Huber [mailto:sebastian.hu...@embedded-brains.de] 
Sent: quarta-feira, 13 de março de 2019 06:47
To: Jose Valdez; devel@rtems.org
Subject: Re: Requirement Management tools

Hello,

I added a ticket and a wiki page for this topic:

https://devel.rtems.org/ticket/3726

https://devel.rtems.org/wiki/Developer/RequirementsEngineering

-- 
Sebastian Huber, embedded brains GmbH

Address : Dornierstr. 4, D-82178 Puchheim, Germany
Phone   : +49 89 189 47 41-16
Fax : +49 89 189 47 41-09
E-Mail  : sebastian.hu...@embedded-brains.de
PGP : Public key available on request.

Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG.

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Requirement Management tools

[Jose Valdez]

Hello all,

I am José Valdez from EDISOFT.

I am working for the RTEMS SMP project which aims to pre-qualify RTEMS SMP for 
space missions.

I am investigating tools which may partially automate the software development 
process, i.e: Requirements Management, Design and Unit test tools and source 
code analysis and metrics tools.

Currently I have been investigating myself and also requested Sebastian Huber 
for some tools he knows. With this information I am building an excel sheet 
with the summarized information about the tools (not yet complete). Could you 
please help in this discussion, indicating, as far as you know:

è if do you know other open source Requirement Management Tools

è if do you know the tools provided and if they are/are not suitable for RTEMS 
SMP

è if do you remember any tool feature not listed in excel sheet, that is 
relevant or if do you know any feature not listed that may block its selection 
for RTEMS SMP

As far as I have currently discussed with Sebastian Huber, we have the 
following ideas:

è Tools which use databases are to be avoided

è Deprecated tools ("dead projects") are to be avoided

è Could be interesting to have a tool which can import/export ReqIf standard 
file

Looking forward for your suggestions

Best regards

José


requirement tools.xls
Description: requirement tools.xls
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel