Re: Discussion: How to handle HALs, SDKs and libraries

[Kinsey Moore]

On Tue, May 23, 2023 at 2:26 AM Christian MAUDERER <
christian.maude...@embedded-brains.de> wrote:

> Hello,
>
> I recently updated the HAL in the i.MXRT BSP. I used the same approach
> that we use for a lot of similar cases: Import the sources into RTEMS
> and adapt them slightly so that they work for us. So basically a
> Clone-and-Own approach.
>
> During the discussion of the patches, some concerns were raised, whether
> we should find a better solution to handle HALs, SDKs and similar cases.
> We should start discussing a solution that can be used after the 6
> release so that maybe someone can start to work on a prototype.
>
> Some example cases are:
>
> - the mcux_sdk in the imxrt BSP
> - the hal in the stm32h7 BSP
> - general ARM CMSIS files
> - zlib
> - libfdt
>
> One solution could be to build these libraries external and only link
> RTEMS with them. There are disadvantages to this aproach:
>
> - Also in my experience, the API of the HALs / SDKs / libraries seems to
> be quite stable, it's possible that there are combinations where some
> unexpected change breaks a driver or makes it impossible to link the
> applications.
>
> - BSPs rely on basic drivers from these libraries (like console or clock
> driver). If we link against the libraries, the testsuite wouldn't build
> any more without preinstalled libraries.
>
> Another solution could be to include libraties like that as submodules
> and build them using the RTEMS build system. We could clone the repos
> onto the RTEMS git server, and add necessary patches. Advantage would be
> that it is more similar to the process that we currently have. Another
> advantage is that we have a known-working version of the files. Upstream
> updates could be either merged or we could rebase our patches to a new
> version.
>
>  From my point of view, the second option would be the better one
> especially because we have a tested, fixed version of the library
> instead telling the user to just use some random version that might or
> might not work.
>
> Regardless which aproach we use: We have to think about how to handle
> that on releases. In the link aproach (first case), we have to somehow
> archive source tar balls and some kind of build recipe. In the submodule
> aproach, we could checkout all submodules and pack the files into the
> RTEMS release tar ball. So I would expect that the second aproach has
> less impact here too.
>
> Comments? Improvements? Better suggestions?
>

I would definitely prefer the submodule approach over the linking approach
to avoid the test issues since some of these HALs bring core functionality.
The Xilinx driver framework (embeddedsw repo on Github) would be
well-suited to the submodule approach since it is already broken out into
the shared driver space because it can apply to at least 3 architectures
(ARM, AArch64, MicroBlaze).

One issue with either approach is the need to modify the HAL source to suit
RTEMS. As far as I'm aware, there is no tooling in place in git for
applying patches to submodules and in the external build scenario we'd end
up maintaining a branch of the origin repo with patches applied.
Upstreaming the changes would be ideal, but I wouldn't expect them to
accept RTEMS-specific patches. The Xilinx NAND driver already requires a
minor modification because that driver doesn't expose an option and instead
has a defined macro that determines how many chip selects are usable to
address different parts of the NAND chip. Technically, this particular
change could be worked around with some include path trickery to leave the
original sources unmodified, but many other changes would not be suited by
that type of workaround and it makes the source less maintainable. We would
need to come up with our own tooling for submodule patch application and
silencing of warnings about dirty submodule trees due to applied patches.

Kinsey
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Discussion: How to handle HALs, SDKs and libraries

[Christian MAUDERER]

On 2023-05-23 16:11, Kinsey Moore wrote:
On Tue, May 23, 2023 at 2:26 AM Christian MAUDERER 
> wrote:


Hello,

I recently updated the HAL in the i.MXRT BSP. I used the same approach
that we use for a lot of similar cases: Import the sources into RTEMS
and adapt them slightly so that they work for us. So basically a
Clone-and-Own approach.

During the discussion of the patches, some concerns were raised,
whether
we should find a better solution to handle HALs, SDKs and similar
cases.
We should start discussing a solution that can be used after the 6
release so that maybe someone can start to work on a prototype.

Some example cases are:

- the mcux_sdk in the imxrt BSP
- the hal in the stm32h7 BSP
- general ARM CMSIS files
- zlib
- libfdt

One solution could be to build these libraries external and only link
RTEMS with them. There are disadvantages to this aproach:

- Also in my experience, the API of the HALs / SDKs / libraries
seems to
be quite stable, it's possible that there are combinations where some
unexpected change breaks a driver or makes it impossible to link the
applications.

- BSPs rely on basic drivers from these libraries (like console or
clock
driver). If we link against the libraries, the testsuite wouldn't build
any more without preinstalled libraries.

Another solution could be to include libraties like that as submodules
and build them using the RTEMS build system. We could clone the repos
onto the RTEMS git server, and add necessary patches. Advantage
would be
that it is more similar to the process that we currently have. Another
advantage is that we have a known-working version of the files.
Upstream
updates could be either merged or we could rebase our patches to a new
version.

  From my point of view, the second option would be the better one
especially because we have a tested, fixed version of the library
instead telling the user to just use some random version that might or
might not work.

Regardless which aproach we use: We have to think about how to handle
that on releases. In the link aproach (first case), we have to somehow
archive source tar balls and some kind of build recipe. In the
submodule
aproach, we could checkout all submodules and pack the files into the
RTEMS release tar ball. So I would expect that the second aproach has
less impact here too.

Comments? Improvements? Better suggestions?


I would definitely prefer the submodule approach over the linking 
approach to avoid the test issues since some of these HALs bring core 
functionality. The Xilinx driver framework (embeddedsw repo on Github) 
would be well-suited to the submodule approach since it is already 
broken out into the shared driver space because it can apply to at least 
3 architectures (ARM, AArch64, MicroBlaze).


One issue with either approach is the need to modify the HAL source to 
suit RTEMS. As far as I'm aware, there is no tooling in place in git for 
applying patches to submodules and in the external build scenario we'd 
end up maintaining a branch of the origin repo with patches applied. 
Upstreaming the changes would be ideal, but I wouldn't expect them to 
accept RTEMS-specific patches. The Xilinx NAND driver already requires a 
minor modification because that driver doesn't expose an option and 
instead has a defined macro that determines how many chip selects are 
usable to address different parts of the NAND chip. Technically, this 
particular change could be worked around with some include path trickery 
to leave the original sources unmodified, but many other changes would 
not be suited by that type of workaround and it makes the source less 
maintainable. We would need to come up with our own tooling for 
submodule patch application and silencing of warnings about dirty 
submodule trees due to applied patches.


Kinsey


I would suggest that we maintain a branch on the git.rtems.org if 
adaptions are necessary and patches can't be upstreamed. We can either 
merge upstream changes or rebase that branch onto the latest upstream if 
we update. I think that is more intuitive and robust compared to a tool 
that applies RTEMS specific patches after cloning the submodule.


Best regards

Christian
--

embedded brains GmbH & Co. KG
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email:  christian.maude...@embedded-brains.de
phone:  +49-89-18 94 741 - 18
mobile: +49-176-152 206 08

Registergericht: Amtsgericht München
Registernummer: HRA 117265
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
ht

Re: Discussion: How to handle HALs, SDKs and libraries

[Chris Johns]

Hi Christian,

Thanks for raising this topic. It is a tough one.

On 24/5/2023 12:11 am, Kinsey Moore wrote:
> On Tue, May 23, 2023 at 2:26 AM Christian MAUDERER
>  > wrote:
> 
> Hello,
> 
> I recently updated the HAL in the i.MXRT BSP. I used the same approach
> that we use for a lot of similar cases: Import the sources into RTEMS
> and adapt them slightly so that they work for us. So basically a
> Clone-and-Own approach.
> 
> During the discussion of the patches, some concerns were raised, whether
> we should find a better solution to handle HALs, SDKs and similar cases.
> We should start discussing a solution that can be used after the 6
> release so that maybe someone can start to work on a prototype.
> 
> Some example cases are:
> 
> - the mcux_sdk in the imxrt BSP
> - the hal in the stm32h7 BSP
> - general ARM CMSIS files
> - zlib
> - libfdt
> 
> One solution could be to build these libraries external and only link
> RTEMS with them. There are disadvantages to this aproach:
> 
> - Also in my experience, the API of the HALs / SDKs / libraries seems to
> be quite stable, it's possible that there are combinations where some
> unexpected change breaks a driver or makes it impossible to link the
> applications.

Xilinx with the more complex devices like the Versal have been moving things
about. The Versal SMC call set is fluid and the PM (platform manager) seems to
functionally align to Xilinx tools releases plus Petalinux versions. For example
there are stable defined API calls in Versal Linux (XRT/zocl) that depends on PM
code that is commented in the code as "to be removed".

When I first used the Zynq I used Xilinx's drivers like OAR is currently with
the Microblaze. I could not release the results because of the license at the
time. I quickly found the drivers lacked functionality for general use and broke
under high loads and boundary conditions. The fixes are part of a project and
cannot be released because the license at the time made it impossible. What I
leant from the exercise is to not depend on their drivers.

I feel what we considered stable will depend on the origin of the code and that
will be case by case.

> - BSPs rely on basic drivers from these libraries (like console or clock
> driver). If we link against the libraries, the testsuite wouldn't build
> any more without preinstalled libraries.

Yes the mutual dependence if built externally and before RTEMS is not easy to
solve. The idea of the HAL code being supplied as .h and a .a does let a user
update the drivers without needing an RTEMS version update.

> Another solution could be to include libraties like that as submodules
> and build them using the RTEMS build system. We could clone the repos
> onto the RTEMS git server, and add necessary patches. Advantage would be
> that it is more similar to the process that we currently have. Another
> advantage is that we have a known-working version of the files. Upstream
> updates could be either merged or we could rebase our patches to a new
> version.

See below for the problems this creates.

>  From my point of view, the second option would be the better one
> especially because we have a tested, fixed version of the library
> instead telling the user to just use some random version that might or
> might not work.

This is important. We need to define what a release is and it is a requirement
we provide all code as tarball files. This implies the release process knows how
to create the tarfiles.

> Regardless which aproach we use: We have to think about how to handle
> that on releases. In the link aproach (first case), we have to somehow
> archive source tar balls and some kind of build recipe. In the submodule
> aproach, we could checkout all submodules and pack the files into the
> RTEMS release tar ball. So I would expect that the second aproach has
> less impact here too.
> 
> Comments? Improvements? Better suggestions?
> 
> I would definitely prefer the submodule approach over the linking approach to
> avoid the test issues since some of these HALs bring core functionality. The
> Xilinx driver framework (embeddedsw repo on Github) would be well-suited to 
> the
> submodule approach since it is already broken out into the shared driver space
> because it can apply to at least 3 architectures (ARM, AArch64, MicroBlaze).

I suggest you avoid making that repo a submodule of anything. The code in that
repo is "over the wall" and there is no continuity. I have it as a submodule in
my XRT repo and a Xilinx push of the next release of tools broke the code. What
I had depended on was removed and moved somewhere else. The Xilinx updates are
based on the release cycle of their tools and they do not respond to issues or
PRs. They are free to make what ever changes they like and the

Re: Discussion: How to handle HALs, SDKs and libraries

[Christian MAUDERER]

Hello Chris,

On 2023-05-24 03:44, Chris Johns wrote:

Hi Christian,

Thanks for raising this topic. It is a tough one.

On 24/5/2023 12:11 am, Kinsey Moore wrote:

On Tue, May 23, 2023 at 2:26 AM Christian MAUDERER
mailto:christian.maude...@embedded-brains.de>> wrote:

 Hello,

 I recently updated the HAL in the i.MXRT BSP. I used the same approach
 that we use for a lot of similar cases: Import the sources into RTEMS
 and adapt them slightly so that they work for us. So basically a
 Clone-and-Own approach.

 During the discussion of the patches, some concerns were raised, whether
 we should find a better solution to handle HALs, SDKs and similar cases.
 We should start discussing a solution that can be used after the 6
 release so that maybe someone can start to work on a prototype.

 Some example cases are:

 - the mcux_sdk in the imxrt BSP
 - the hal in the stm32h7 BSP
 - general ARM CMSIS files
 - zlib
 - libfdt

 One solution could be to build these libraries external and only link
 RTEMS with them. There are disadvantages to this aproach:

 - Also in my experience, the API of the HALs / SDKs / libraries seems to
 be quite stable, it's possible that there are combinations where some
 unexpected change breaks a driver or makes it impossible to link the
 applications.


Xilinx with the more complex devices like the Versal have been moving things
about. The Versal SMC call set is fluid and the PM (platform manager) seems to
functionally align to Xilinx tools releases plus Petalinux versions. For example
there are stable defined API calls in Versal Linux (XRT/zocl) that depends on PM
code that is commented in the code as "to be removed".

When I first used the Zynq I used Xilinx's drivers like OAR is currently with
the Microblaze. I could not release the results because of the license at the
time. I quickly found the drivers lacked functionality for general use and broke
under high loads and boundary conditions. The fixes are part of a project and
cannot be released because the license at the time made it impossible. What I
leant from the exercise is to not depend on their drivers.


That sounds like a quite bad case. So it's a good example for that 
discussion. Thanks for bringing it up.




I feel what we considered stable will depend on the origin of the code and that
will be case by case.


Agreed.




 - BSPs rely on basic drivers from these libraries (like console or clock
 driver). If we link against the libraries, the testsuite wouldn't build
 any more without preinstalled libraries.


Yes the mutual dependence if built externally and before RTEMS is not easy to
solve. The idea of the HAL code being supplied as .h and a .a does let a user
update the drivers without needing an RTEMS version update.


 Another solution could be to include libraties like that as submodules
 and build them using the RTEMS build system. We could clone the repos
 onto the RTEMS git server, and add necessary patches. Advantage would be
 that it is more similar to the process that we currently have. Another
 advantage is that we have a known-working version of the files. Upstream
 updates could be either merged or we could rebase our patches to a new
 version.


See below for the problems this creates.


  From my point of view, the second option would be the better one
 especially because we have a tested, fixed version of the library
 instead telling the user to just use some random version that might or
 might not work.


This is important. We need to define what a release is and it is a requirement
we provide all code as tarball files. This implies the release process knows how
to create the tarfiles.


 Regardless which aproach we use: We have to think about how to handle
 that on releases. In the link aproach (first case), we have to somehow
 archive source tar balls and some kind of build recipe. In the submodule
 aproach, we could checkout all submodules and pack the files into the
 RTEMS release tar ball. So I would expect that the second aproach has
 less impact here too.

 Comments? Improvements? Better suggestions?

I would definitely prefer the submodule approach over the linking approach to
avoid the test issues since some of these HALs bring core functionality. The
Xilinx driver framework (embeddedsw repo on Github) would be well-suited to the
submodule approach since it is already broken out into the shared driver space
because it can apply to at least 3 architectures (ARM, AArch64, MicroBlaze).


I suggest you avoid making that repo a submodule of anything. The code in that
repo is "over the wall" and there is no continuity. I have it as a submodule in
my XRT repo and a Xilinx push of the next release of tools broke the code. What
I had depended on was removed and moved somewhere else. The Xilinx updates are
based on the release cycle of 

Re: Discussion: How to handle HALs, SDKs and libraries

[Chris Johns]

On 24/5/2023 5:07 pm, Christian MAUDERER wrote:
> Hello Chris,
> 
> On 2023-05-24 03:44, Chris Johns wrote:
>> Hi Christian,
>>
>> Thanks for raising this topic. It is a tough one.
>>
>> On 24/5/2023 12:11 am, Kinsey Moore wrote:
>>> On Tue, May 23, 2023 at 2:26 AM Christian MAUDERER
>>> >> > wrote:
>>>
>>>  Hello,
>>>
>>>  I recently updated the HAL in the i.MXRT BSP. I used the same approach
>>>  that we use for a lot of similar cases: Import the sources into RTEMS
>>>  and adapt them slightly so that they work for us. So basically a
>>>  Clone-and-Own approach.
>>>
>>>  During the discussion of the patches, some concerns were raised, 
>>> whether
>>>  we should find a better solution to handle HALs, SDKs and similar 
>>> cases.
>>>  We should start discussing a solution that can be used after the 6
>>>  release so that maybe someone can start to work on a prototype.
>>>
>>>  Some example cases are:
>>>
>>>  - the mcux_sdk in the imxrt BSP
>>>  - the hal in the stm32h7 BSP
>>>  - general ARM CMSIS files
>>>  - zlib
>>>  - libfdt
>>>
>>>  One solution could be to build these libraries external and only link
>>>  RTEMS with them. There are disadvantages to this aproach:
>>>
>>>  - Also in my experience, the API of the HALs / SDKs / libraries seems 
>>> to
>>>  be quite stable, it's possible that there are combinations where some
>>>  unexpected change breaks a driver or makes it impossible to link the
>>>  applications.
>>
>> Xilinx with the more complex devices like the Versal have been moving things
>> about. The Versal SMC call set is fluid and the PM (platform manager) seems 
>> to
>> functionally align to Xilinx tools releases plus Petalinux versions. For 
>> example
>> there are stable defined API calls in Versal Linux (XRT/zocl) that depends 
>> on PM
>> code that is commented in the code as "to be removed".
>>
>> When I first used the Zynq I used Xilinx's drivers like OAR is currently with
>> the Microblaze. I could not release the results because of the license at the
>> time. I quickly found the drivers lacked functionality for general use and 
>> broke
>> under high loads and boundary conditions. The fixes are part of a project and
>> cannot be released because the license at the time made it impossible. What I
>> leant from the exercise is to not depend on their drivers.
> 
> That sounds like a quite bad case. So it's a good example for that discussion.
> Thanks for bringing it up.

I view the repo as open but not open source ... if that sentence makes sense?

> 
>>
>> I feel what we considered stable will depend on the origin of the code and 
>> that
>> will be case by case.
> 
> Agreed.
> 
>>
>>>  - BSPs rely on basic drivers from these libraries (like console or 
>>> clock
>>>  driver). If we link against the libraries, the testsuite wouldn't build
>>>  any more without preinstalled libraries.
>>
>> Yes the mutual dependence if built externally and before RTEMS is not easy to
>> solve. The idea of the HAL code being supplied as .h and a .a does let a user
>> update the drivers without needing an RTEMS version update.
>>
>>>  Another solution could be to include libraties like that as submodules
>>>  and build them using the RTEMS build system. We could clone the repos
>>>  onto the RTEMS git server, and add necessary patches. Advantage would 
>>> be
>>>  that it is more similar to the process that we currently have. Another
>>>  advantage is that we have a known-working version of the files. 
>>> Upstream
>>>  updates could be either merged or we could rebase our patches to a new
>>>  version.
>>
>> See below for the problems this creates.
>>
>>>   From my point of view, the second option would be the better one
>>>  especially because we have a tested, fixed version of the library
>>>  instead telling the user to just use some random version that might or
>>>  might not work.
>>
>> This is important. We need to define what a release is and it is a 
>> requirement
>> we provide all code as tarball files. This implies the release process knows 
>> how
>> to create the tarfiles.
>>
>>>  Regardless which aproach we use: We have to think about how to handle
>>>  that on releases. In the link aproach (first case), we have to somehow
>>>  archive source tar balls and some kind of build recipe. In the 
>>> submodule
>>>  aproach, we could checkout all submodules and pack the files into the
>>>  RTEMS release tar ball. So I would expect that the second aproach has
>>>  less impact here too.
>>>
>>>  Comments? Improvements? Better suggestions?
>>>
>>> I would definitely prefer the submodule approach over the linking approach 
>>> to
>>> avoid the test issues since some of these HALs bring core functionality. The
>>> Xilinx driver framework (embeddedsw repo on Github) would be well-suited to

Re: Discussion: How to handle HALs, SDKs and libraries

[Christian MAUDERER]

On 2023-05-25 01:57, Chris Johns wrote:

On 24/5/2023 5:07 pm, Christian MAUDERER wrote:

Hello Chris,

On 2023-05-24 03:44, Chris Johns wrote:

Hi Christian,

Thanks for raising this topic. It is a tough one.

On 24/5/2023 12:11 am, Kinsey Moore wrote:

On Tue, May 23, 2023 at 2:26 AM Christian MAUDERER
mailto:christian.maude...@embedded-brains.de>> wrote:

  Hello,

  I recently updated the HAL in the i.MXRT BSP. I used the same approach
  that we use for a lot of similar cases: Import the sources into RTEMS
  and adapt them slightly so that they work for us. So basically a
  Clone-and-Own approach.

  During the discussion of the patches, some concerns were raised, whether
  we should find a better solution to handle HALs, SDKs and similar cases.
  We should start discussing a solution that can be used after the 6
  release so that maybe someone can start to work on a prototype.

  Some example cases are:

  - the mcux_sdk in the imxrt BSP
  - the hal in the stm32h7 BSP
  - general ARM CMSIS files
  - zlib
  - libfdt

  One solution could be to build these libraries external and only link
  RTEMS with them. There are disadvantages to this aproach:

  - Also in my experience, the API of the HALs / SDKs / libraries seems to
  be quite stable, it's possible that there are combinations where some
  unexpected change breaks a driver or makes it impossible to link the
  applications.


Xilinx with the more complex devices like the Versal have been moving things
about. The Versal SMC call set is fluid and the PM (platform manager) seems to
functionally align to Xilinx tools releases plus Petalinux versions. For example
there are stable defined API calls in Versal Linux (XRT/zocl) that depends on PM
code that is commented in the code as "to be removed".

When I first used the Zynq I used Xilinx's drivers like OAR is currently with
the Microblaze. I could not release the results because of the license at the
time. I quickly found the drivers lacked functionality for general use and broke
under high loads and boundary conditions. The fixes are part of a project and
cannot be released because the license at the time made it impossible. What I
leant from the exercise is to not depend on their drivers.


That sounds like a quite bad case. So it's a good example for that discussion.
Thanks for bringing it up.


I view the repo as open but not open source ... if that sentence makes sense?



I think I understand what you mean. But it's still a good example for 
the discussion. If a solution theoretically works with that case, it 
should work with a lot of other cases too.






I feel what we considered stable will depend on the origin of the code and that
will be case by case.


Agreed.




  - BSPs rely on basic drivers from these libraries (like console or clock
  driver). If we link against the libraries, the testsuite wouldn't build
  any more without preinstalled libraries.


Yes the mutual dependence if built externally and before RTEMS is not easy to
solve. The idea of the HAL code being supplied as .h and a .a does let a user
update the drivers without needing an RTEMS version update.


  Another solution could be to include libraties like that as submodules
  and build them using the RTEMS build system. We could clone the repos
  onto the RTEMS git server, and add necessary patches. Advantage would be
  that it is more similar to the process that we currently have. Another
  advantage is that we have a known-working version of the files. Upstream
  updates could be either merged or we could rebase our patches to a new
  version.


See below for the problems this creates.


   From my point of view, the second option would be the better one
  especially because we have a tested, fixed version of the library
  instead telling the user to just use some random version that might or
  might not work.


This is important. We need to define what a release is and it is a requirement
we provide all code as tarball files. This implies the release process knows how
to create the tarfiles.


  Regardless which aproach we use: We have to think about how to handle
  that on releases. In the link aproach (first case), we have to somehow
  archive source tar balls and some kind of build recipe. In the submodule
  aproach, we could checkout all submodules and pack the files into the
  RTEMS release tar ball. So I would expect that the second aproach has
  less impact here too.

  Comments? Improvements? Better suggestions?

I would definitely prefer the submodule approach over the linking approach to
avoid the test issues since some of these HALs bring core functionality. The
Xilinx driver framework (embeddedsw repo on Github) would be well-suited to the
submodule approach since it is already broken out into the shared driver space
because it can apply to at 

Re: Discussion: How to handle HALs, SDKs and libraries

[Karel Gardas]

Hello,

are we really that close to RTEMS 6 release that none of this is 
acceptable to do now?


Asking since I'd also like to update stm32h7 HAL. I may do that manually 
or I may do that submodule way which may perhaps save some of the manual 
work involved as some of the patches may not be needed anymore. It 
depends on resolution to following questions:


- stm32h7 hal contains Sebastian's modification/additions of Doxygen 
tags. Do we need to preserve that in the submodule? I guess not, but I'm 
not sure and this question of Doxygen markups was not solved in this 
thread yet AFAIK.


- all ARMs HALs probably also depends on particular version of CMSIS 
files, so I guess we should use separate submodules of CMSIS per HAL. At 
least STM recommends that: 
https://htmlpreview.github.io/?https://github.com/STMicroelectronics/STM32CubeH7/blob/master/Release_Notes.html 
-- this would certainly be improvement in comparison with current state 
of the art besides update of CMSIS alone we would also support 
manufacture tested CMSIS version for their HAL...


- where to put submodules? What is preferred location? For stm32h7 case!
  (i) hals/arm/stm32h7
 or
  (ii) deps/arm/stm32h7
 or
  (iii) contrib/arm/stm32h7
 or
  (iv) anything else preferred?


I may do stm32h7 hal + its CMSIS dependency now (or next week), but only 
if the chance of inclusion into RTEMS 6 is reasonably high. Otherwise I 
would need to go more safer route of manual in-tree HAL update assuming 
this is still acceptable for RTEMS 6 release.


Testing of stm32h7 changes is on my shoulders, hardware is here and 
customer is using the platform in field. Offering this as an opportunity 
to test the submodule vehicle ASAP on some part of RTEMS...


Thanks,
Karel

On 5/23/23 09:26, Christian MAUDERER wrote:

Hello,

I recently updated the HAL in the i.MXRT BSP. I used the same approach 
that we use for a lot of similar cases: Import the sources into RTEMS 
and adapt them slightly so that they work for us. So basically a 
Clone-and-Own approach.


During the discussion of the patches, some concerns were raised, whether 
we should find a better solution to handle HALs, SDKs and similar cases. 
We should start discussing a solution that can be used after the 6 
release so that maybe someone can start to work on a prototype.


Some example cases are:

- the mcux_sdk in the imxrt BSP
- the hal in the stm32h7 BSP
- general ARM CMSIS files
- zlib
- libfdt

One solution could be to build these libraries external and only link 
RTEMS with them. There are disadvantages to this aproach:


- Also in my experience, the API of the HALs / SDKs / libraries seems to 
be quite stable, it's possible that there are combinations where some 
unexpected change breaks a driver or makes it impossible to link the 
applications.


- BSPs rely on basic drivers from these libraries (like console or clock 
driver). If we link against the libraries, the testsuite wouldn't build 
any more without preinstalled libraries.


Another solution could be to include libraties like that as submodules 
and build them using the RTEMS build system. We could clone the repos 
onto the RTEMS git server, and add necessary patches. Advantage would be 
that it is more similar to the process that we currently have. Another 
advantage is that we have a known-working version of the files. Upstream 
updates could be either merged or we could rebase our patches to a new 
version.


 From my point of view, the second option would be the better one 
especially because we have a tested, fixed version of the library 
instead telling the user to just use some random version that might or 
might not work.


Regardless which aproach we use: We have to think about how to handle 
that on releases. In the link aproach (first case), we have to somehow 
archive source tar balls and some kind of build recipe. In the submodule 
aproach, we could checkout all submodules and pack the files into the 
RTEMS release tar ball. So I would expect that the second aproach has 
less impact here too.


Comments? Improvements? Better suggestions?

Best regards

Christian


___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Discussion: How to handle HALs, SDKs and libraries

[Karel Gardas]

 Hello,

I've created setup where I've put updated STM32H7 HAL consisting of two 
submodules:


- stm32h7xx_hal_driver
- cmsis_device_h7

and ARM's

- CMSIS_5

into hals/arm/stm32h7 to follow related BSP code location.

I've also completely removed HAL code from the BSP directory and also 
CMSIS v4 files from bsps/arm/include to verify compilation is really 
using submodules.


If you like to see how this is looking, just:

$ git clone https://github.com/karelfv/rtems.git rtems
$ cd rtems
$ git checkout rtems-stm32h7-hal-git-submodules
$ git submodule update --init

And see bsps/arm/stm32h7/ and hals/arm/stm32h7/ and bsps/arm/include 
subdirectories.


You can also compile if you have arm tools:

$ ./waf bspdefaults --rtems-bsps=arm/stm32h757i-eval > stm32h757i-eval.ini
$ ./waf configure --rtems-bsps=arm/stm32h757i-eval 
--rtems-config=./stm32h757i-eval.ini --rtems-tools= --prefix=

$ ./waf

CAVEAT: only STM32H757i-eval BSP is supported and only compilation. 
Installation is not supported yet.


Note: HAL is updated to latest STMicro code and all related changes done 
in the past by Sebastian and Christian are merged into submodules 
projects except Doxygen tag change done by Sebastian. This is to be 
discussed topic if doxygen changes belongs to submodule or are not 
needed anymore...


hello.exe/ticker.exe/paranoia.exe runs fine.

Comments welcome!

Karel


On 7/14/23 13:53, Karel Gardas wrote:


Hello,

are we really that close to RTEMS 6 release that none of this is 
acceptable to do now?


Asking since I'd also like to update stm32h7 HAL. I may do that manually 
or I may do that submodule way which may perhaps save some of the manual 
work involved as some of the patches may not be needed anymore. It 
depends on resolution to following questions:


- stm32h7 hal contains Sebastian's modification/additions of Doxygen 
tags. Do we need to preserve that in the submodule? I guess not, but I'm 
not sure and this question of Doxygen markups was not solved in this 
thread yet AFAIK.


- all ARMs HALs probably also depends on particular version of CMSIS 
files, so I guess we should use separate submodules of CMSIS per HAL. At 
least STM recommends that: 
https://htmlpreview.github.io/?https://github.com/STMicroelectronics/STM32CubeH7/blob/master/Release_Notes.html -- this would certainly be improvement in comparison with current state of the art besides update of CMSIS alone we would also support manufacture tested CMSIS version for their HAL...


- where to put submodules? What is preferred location? For stm32h7 case!
   (i) hals/arm/stm32h7
  or
   (ii) deps/arm/stm32h7
  or
   (iii) contrib/arm/stm32h7
  or
   (iv) anything else preferred?


I may do stm32h7 hal + its CMSIS dependency now (or next week), but only 
if the chance of inclusion into RTEMS 6 is reasonably high. Otherwise I 
would need to go more safer route of manual in-tree HAL update assuming 
this is still acceptable for RTEMS 6 release.


Testing of stm32h7 changes is on my shoulders, hardware is here and 
customer is using the platform in field. Offering this as an opportunity 
to test the submodule vehicle ASAP on some part of RTEMS...


Thanks,
Karel

On 5/23/23 09:26, Christian MAUDERER wrote:

Hello,

I recently updated the HAL in the i.MXRT BSP. I used the same approach 
that we use for a lot of similar cases: Import the sources into RTEMS 
and adapt them slightly so that they work for us. So basically a 
Clone-and-Own approach.


During the discussion of the patches, some concerns were raised, 
whether we should find a better solution to handle HALs, SDKs and 
similar cases. We should start discussing a solution that can be used 
after the 6 release so that maybe someone can start to work on a 
prototype.


Some example cases are:

- the mcux_sdk in the imxrt BSP
- the hal in the stm32h7 BSP
- general ARM CMSIS files
- zlib
- libfdt

One solution could be to build these libraries external and only link 
RTEMS with them. There are disadvantages to this aproach:


- Also in my experience, the API of the HALs / SDKs / libraries seems 
to be quite stable, it's possible that there are combinations where 
some unexpected change breaks a driver or makes it impossible to link 
the applications.


- BSPs rely on basic drivers from these libraries (like console or 
clock driver). If we link against the libraries, the testsuite 
wouldn't build any more without preinstalled libraries.


Another solution could be to include libraties like that as submodules 
and build them using the RTEMS build system. We could clone the repos 
onto the RTEMS git server, and add necessary patches. Advantage would 
be that it is more similar to the process that we currently have. 
Another advantage is that we have a known-working version of the 
files. Upstream updates could be either merged or we could rebase our 
patches to a new version.


 From my point of view, the second option would be the better one 
especially because we have a tested, fixed version of 

Re: Discussion: How to handle HALs, SDKs and libraries

[oss]

Hello Karel,

thanks for trying that. I wouldn't object to add it to 6 but I wouldn't 
expect it either because it means a lot of adaptions to the release 
scripts. Chris most likely can tell you more about whether it's even 
thinkable at that point in the release cycle or not.


Am 15.07.23 um 01:48 schrieb Karel Gardas:


  Hello,

I've created setup where I've put updated STM32H7 HAL consisting of two 
submodules:


- stm32h7xx_hal_driver
- cmsis_device_h7

and ARM's

- CMSIS_5

into hals/arm/stm32h7 to follow related BSP code location.


I think I would prefer "contrib/hals", "external/hals" or some other 
more generic top directory name. It would allow to use that for other 
parts like for example zlib in "contrib/libs/zlib" in the future (if we 
agree on having libs like that there too).



At the moment, you have absolute URLs for your submodules:

  url = https://github.com/ARM-software/CMSIS_5.git
  url = https://github.com/karelfv/stm32h7xx_hal_driver.git
  url = https://github.com/karelfv/cmsis_device_h7.git

If we use submodules, I would go for relative paths and clone the 
repositories to our git.rtems.org and to all mirrors. It has the 
advantage that even if some upstream URL changes, we won't be affected.



I like submodules because they are well-supported by the usual tools. 
Nonetheless, we should take a look at some of the reasons why other 
projects decided against them. For example Google has the "repo" tool 
used in Android. Zephyr has the "west" tool. OpenWRT uses "feeds" for 
something similar. I'm sure there are more.


  https://gerrit.googlesource.com/git-repo/
  https://docs.zephyrproject.org/latest/develop/west/why.html#
  https://openwrt.org/docs/guide-developer/feeds

All solve a similar problem. Part of them even overlap already with 
functionality from our waf based build system. We should consider 
whether we need any of the functionality of these tools and whether we 
want to re-implement it or whether we just want to use one of the tools 
and integrate with it.




I've also completely removed HAL code from the BSP directory and also 
CMSIS v4 files from bsps/arm/include to verify compilation is really 
using submodules.


I think the CMSIS headers are used by some other BSPs too. So that's 
fine for a prototype. But for the final solution, we need something 
different.




If you like to see how this is looking, just:

$ git clone https://github.com/karelfv/rtems.git rtems
$ cd rtems
$ git checkout rtems-stm32h7-hal-git-submodules
$ git submodule update --init

And see bsps/arm/stm32h7/ and hals/arm/stm32h7/ and bsps/arm/include 
subdirectories.


You can also compile if you have arm tools:

$ ./waf bspdefaults --rtems-bsps=arm/stm32h757i-eval > stm32h757i-eval.ini
$ ./waf configure --rtems-bsps=arm/stm32h757i-eval 
--rtems-config=./stm32h757i-eval.ini --rtems-tools= 
--prefix=

$ ./waf

CAVEAT: only STM32H757i-eval BSP is supported and only compilation. 
Installation is not supported yet.


Note: HAL is updated to latest STMicro code and all related changes done 
in the past by Sebastian and Christian are merged into submodules 
projects except Doxygen tag change done by Sebastian. This is to be 
discussed topic if doxygen changes belongs to submodule or are not 
needed anymore...


I agree that the doxygen shouldn't be necessary any more. But we should 
make sure that our Doxyfile doesn't pick up the submodules. So that 
would need some of the EXCLUDE* options set to the path of the submodules.


Best regards

Christian



hello.exe/ticker.exe/paranoia.exe runs fine.

Comments welcome!

Karel


On 7/14/23 13:53, Karel Gardas wrote:


Hello,

are we really that close to RTEMS 6 release that none of this is 
acceptable to do now?


Asking since I'd also like to update stm32h7 HAL. I may do that 
manually or I may do that submodule way which may perhaps save some of 
the manual work involved as some of the patches may not be needed 
anymore. It depends on resolution to following questions:


- stm32h7 hal contains Sebastian's modification/additions of Doxygen 
tags. Do we need to preserve that in the submodule? I guess not, but 
I'm not sure and this question of Doxygen markups was not solved in 
this thread yet AFAIK.


- all ARMs HALs probably also depends on particular version of CMSIS 
files, so I guess we should use separate submodules of CMSIS per HAL. 
At least STM recommends that: 
https://htmlpreview.github.io/?https://github.com/STMicroelectronics/STM32CubeH7/blob/master/Release_Notes.html -- this would certainly be improvement in comparison with current state of the art besides update of CMSIS alone we would also support manufacture tested CMSIS version for their HAL...


- where to put submodules? What is preferred location? For stm32h7 case!
   (i) hals/arm/stm32h7
  or
   (ii) deps/arm/stm32h7
  or
   (iii) contrib/arm/stm32h7
  or
   (iv) anything else preferred?


I may do stm32h7 hal + its CMSIS dependency now (or next week), but 
only if the chance of

Re: Discussion: How to handle HALs, SDKs and libraries

[Karel Gardas]

On 7/15/23 09:47, o...@c-mauderer.de wrote:

Am 15.07.23 um 01:48 schrieb Karel Gardas:


  Hello,

I've created setup where I've put updated STM32H7 HAL consisting of 
two submodules:


- stm32h7xx_hal_driver
- cmsis_device_h7

and ARM's

- CMSIS_5

into hals/arm/stm32h7 to follow related BSP code location.


I think I would prefer "contrib/hals", "external/hals" or some other 
more generic top directory name. It would allow to use that for other 
parts like for example zlib in "contrib/libs/zlib" in the future (if we 
agree on having libs like that there too).


Indeed, name is not the problem here. Both looks fine, somewhat prefer 
external a bit.



At the moment, you have absolute URLs for your submodules:

   url = https://github.com/ARM-software/CMSIS_5.git
   url = https://github.com/karelfv/stm32h7xx_hal_driver.git
   url = https://github.com/karelfv/cmsis_device_h7.git

If we use submodules, I would go for relative paths and clone the 
repositories to our git.rtems.org and to all mirrors. It has the 
advantage that even if some upstream URL changes, we won't be affected.


This is dirty PoC not a final solution. Due to this I've settled on 
absolute urls and not relative. For final solution relative are needed, 
but both url (relative/abolute) and name/location in RTEMS tree is very 
easy to change based on decision.


I like submodules because they are well-supported by the usual tools. 


Honestly. I like submodules idea, but hate its implementation. Some reasons:

- submodules were added as a light weight feature and during the 
development of git/submodule feature they accumulated some more.


- submodules started as way to get commit id of the external project to 
the tree. But only master branch was supported. (not suitable for RTEMS)


- later submodules added capability to use tag (still not suitable for 
RTEMS)


- later submodules added capability to use different branch from the 
project, but only a head of it. (finally what RTEMS need).


- in the field you can see gits supporting different sets of submodule 
features described above.


- submodules are easy to be broken. Personal experience from Haskell 
compiler (GHC) project which uses submodules a lot. I have git 2.25 and 
I always have an issue to pull from GHC repo since alwyas for me one 
submodule (when changed) will be broken for me. Conclusion: clone with 
submodules may kind of work, but later pull may be tricky.


- besides submodules are not supported in 'got' which is 3rd party 
independent implementation of git like SCM with more sane UI


- due to submodules trickiness development policy for external 
integration with project code modification (aka HAL patches) needs to be 
very well written. I smell horrible increase in cognitive load here...


Nonetheless, we should take a look at some of the reasons why other 
projects decided against them. For example Google has the "repo" tool 
used in Android. Zephyr has the "west" tool. OpenWRT uses "feeds" for 
something similar. I'm sure there are more.


   https://gerrit.googlesource.com/git-repo/
   https://docs.zephyrproject.org/latest/develop/west/why.html#
   https://openwrt.org/docs/guide-developer/feeds

All solve a similar problem. Part of them even overlap already with 
functionality from our waf based build system. We should consider 
whether we need any of the functionality of these tools and whether we 
want to re-implement it or whether we just want to use one of the tools 
and integrate with it.


Oh, do I sense starting academic debate now? The point of my dirty PoC 
was to ground that a bit and show real stuff to get a feel of possibilities.


Anyway, based on my PoC experience I would probably prefer something 
more KISS and also more flexible by tool(s) being also aware of RTEMS BSPs


- first lets start with just 'boot' and 'update' scripts hosted in the 
repo and used to clone/update externals would be needed. And put 
'external' into git ignore.


- later boot/update my be made BSP aware. I mean when doing development 
on H7 I do not need HALs for Xilinx and other ARM/v8/Rx stuff. Neither I 
need HAL for NXP/Microchip chips.


- may this boot/update functionality be made part of ./waf tooling? 
Perhaps it may


So based on this I can see how Zephyr gravitated to west...


I've also completely removed HAL code from the BSP directory and also 
CMSIS v4 files from bsps/arm/include to verify compilation is really 
using submodules.


I think the CMSIS headers are used by some other BSPs too. So that's 
fine for a prototype. But for the final solution, we need something 
different.


The PoC needs to prove some things hence removal of CMSIS from the tree. 
Nothing more. It's a "gain experience" vehicle, not a final solution. 
And btw, you will not be able to find final solution without 
touching/testing stuff and getting your hands dirty with it.


Everything has its own price: (i) adaptation/support of west for RTEMS 
or (ii) writing homegrown script(s) 

Re: Discussion: How to handle HALs, SDKs and libraries

[oss]

Hello Karel,

Am 15.07.23 um 12:17 schrieb Karel Gardas:

On 7/15/23 09:47, o...@c-mauderer.de wrote:

Am 15.07.23 um 01:48 schrieb Karel Gardas:


  Hello,

I've created setup where I've put updated STM32H7 HAL consisting of 
two submodules:


- stm32h7xx_hal_driver
- cmsis_device_h7

and ARM's

- CMSIS_5

into hals/arm/stm32h7 to follow related BSP code location.


I think I would prefer "contrib/hals", "external/hals" or some other 
more generic top directory name. It would allow to use that for other 
parts like for example zlib in "contrib/libs/zlib" in the future (if 
we agree on having libs like that there too).


Indeed, name is not the problem here. Both looks fine, somewhat prefer 
external a bit.



At the moment, you have absolute URLs for your submodules:

   url = https://github.com/ARM-software/CMSIS_5.git
   url = https://github.com/karelfv/stm32h7xx_hal_driver.git
   url = https://github.com/karelfv/cmsis_device_h7.git

If we use submodules, I would go for relative paths and clone the 
repositories to our git.rtems.org and to all mirrors. It has the 
advantage that even if some upstream URL changes, we won't be affected.


This is dirty PoC not a final solution. Due to this I've settled on 
absolute urls and not relative. For final solution relative are needed, 
but both url (relative/abolute) and name/location in RTEMS tree is very 
easy to change based on decision.


I like submodules because they are well-supported by the usual tools. 


Honestly. I like submodules idea, but hate its implementation. Some 
reasons:


- submodules were added as a light weight feature and during the 
development of git/submodule feature they accumulated some more.


- submodules started as way to get commit id of the external project to 
the tree. But only master branch was supported. (not suitable for RTEMS)


- later submodules added capability to use tag (still not suitable for 
RTEMS)


- later submodules added capability to use different branch from the 
project, but only a head of it. (finally what RTEMS need).


I don't like using branches instead of fixed commit IDs. Otherwise, it's 
not possible to reliable re-build a fixed version.




- in the field you can see gits supporting different sets of submodule 
features described above.




Yes, it will force us to use a minimum git version that supports the 
necessary features. From my point of view, it's mainly something we 
should document.


- submodules are easy to be broken. Personal experience from Haskell 
compiler (GHC) project which uses submodules a lot. I have git 2.25 and 
I always have an issue to pull from GHC repo since alwyas for me one 
submodule (when changed) will be broken for me. Conclusion: clone with 
submodules may kind of work, but later pull may be tricky.




Agreed: It's easy to miss that a submodule is out of date.

- besides submodules are not supported in 'got' which is 3rd party 
independent implementation of git like SCM with more sane UI




I would say that this is a problem of the third-party client. We should 
make sure to support the official git client. Maybe additionally some 
(python?) git library if it is necessary for our tools.


- due to submodules trickiness development policy for external 
integration with project code modification (aka HAL patches) needs to be 
very well written. I smell horrible increase in cognitive load here...


Nonetheless, we should take a look at some of the reasons why other 
projects decided against them. For example Google has the "repo" tool 
used in Android. Zephyr has the "west" tool. OpenWRT uses "feeds" for 
something similar. I'm sure there are more.


   https://gerrit.googlesource.com/git-repo/
   https://docs.zephyrproject.org/latest/develop/west/why.html#
   https://openwrt.org/docs/guide-developer/feeds

All solve a similar problem. Part of them even overlap already with 
functionality from our waf based build system. We should consider 
whether we need any of the functionality of these tools and whether we 
want to re-implement it or whether we just want to use one of the 
tools and integrate with it.


Oh, do I sense starting academic debate now? The point of my dirty PoC 
was to ground that a bit and show real stuff to get a feel of 
possibilities.




I think the Proof of Concept is great and important. Thanks for doing 
it. I'm fully OK with submodules as a general direction. I just wanted 
to point out that we should take at least a short look at other systems 
that solve similar problems. If they work around serious limitations of 
submodules we can learn from them and avoid that we hit the same 
limitations.


I used submodules myself in a lot of cases. And there are mainly two 
problems that I have:


1. I often miss pushing a submodule when I push the base repo. And I 
only note it way later if I try to update the repo on another PC. That 
happened to me multiple times when working with submodules. But another 
tool will just have other problems so that's

Re: Discussion: How to handle HALs, SDKs and libraries

[Karel Gardas]

On 7/15/23 14:33, o...@c-mauderer.de wrote:
I like submodules because they are well-supported by the usual tools. 


Honestly. I like submodules idea, but hate its implementation. Some 
reasons:


- submodules were added as a light weight feature and during the 
development of git/submodule feature they accumulated some more.


- submodules started as way to get commit id of the external project 
to the tree. But only master branch was supported. (not suitable for 
RTEMS)


- later submodules added capability to use tag (still not suitable for 
RTEMS)


- later submodules added capability to use different branch from the 
project, but only a head of it. (finally what RTEMS need).


I don't like using branches instead of fixed commit IDs. Otherwise, it's 
not possible to reliable re-build a fixed version.


And this is most important detail, so let's discuss this alone.

I have a hal library, let's consider this to be 
https://github.com/karelfv/stm32h7xx_hal_driver -- just one project for now.


This is a fork of ST Micro project of the same name which is a slow 
development project. Few accumulated commits over one year let say.


This project needs few RTEMS related patches and there is a chance they 
also need to kind of adapt to the new development. So let's branch as:


- master branch -- this is fork of ST Micro
- rtems-6-branch -- this is a branch from master at one point of time 
accumulating RTEMS related changes.


Now, let's assume this was done 6 months ago and now you would like to 
update from ST Micro upstream. How would you do that? I see:


- git fetch upstream inside the master
- git push master

that's update of master in fork and now:

- pull/merge master
or
- rebase master

in the rtems-6-branch? I'm afraid rebase is dangerous in shared setup so 
probably pull/merge?


Anyway, still this is a branch based workflow. How would you do that 
workflow with just fixed commit IDs?


I'm asking since so far commit IDs on submodules were always obtain from 
master branch which is what we basically fork and does not give us 
ability to patch with RTEMS changes...


So I'm curious how would you do that?

Thanks!
Karel


___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Discussion: How to handle HALs, SDKs and libraries

[oss]

Hello Karel,

Am 15.07.23 um 15:03 schrieb Karel Gardas:

On 7/15/23 14:33, o...@c-mauderer.de wrote:
I like submodules because they are well-supported by the usual tools. 


Honestly. I like submodules idea, but hate its implementation. Some 
reasons:


- submodules were added as a light weight feature and during the 
development of git/submodule feature they accumulated some more.


- submodules started as way to get commit id of the external project 
to the tree. But only master branch was supported. (not suitable for 
RTEMS)


- later submodules added capability to use tag (still not suitable 
for RTEMS)


- later submodules added capability to use different branch from the 
project, but only a head of it. (finally what RTEMS need).


I don't like using branches instead of fixed commit IDs. Otherwise, 
it's not possible to reliable re-build a fixed version.


And this is most important detail, so let's discuss this alone.


OK.



I have a hal library, let's consider this to be 
https://github.com/karelfv/stm32h7xx_hal_driver -- just one project for 
now.


This is a fork of ST Micro project of the same name which is a slow 
development project. Few accumulated commits over one year let say.


This project needs few RTEMS related patches and there is a chance they 
also need to kind of adapt to the new development. So let's branch as:


- master branch -- this is fork of ST Micro
- rtems-6-branch -- this is a branch from master at one point of time 
accumulating RTEMS related changes.


Now, let's assume this was done 6 months ago and now you would like to 
update from ST Micro upstream. How would you do that? I see:


- git fetch upstream inside the master
- git push master

that's update of master in fork and now:

- pull/merge master
or
- rebase master

in the rtems-6-branch? I'm afraid rebase is dangerous in shared setup so 
probably pull/merge?


We should avoid rebasing and overwriting the old branch because we have 
to preserve a version that can be used with older RTEMS versions. So for 
rebasing, we would have to use multiple branches. Instead of 
rtems-6-branch, it could be a rtems- or something 
similar.


I think that depends heavily on how the upstream project works. If the 
upstream project uses a typical git workflow and is quite active, 
merging is most likely the better solution because we don't get big 
amounts of branches with that.


On the other hand I know that some semiconductor manufacturers tend to 
just dump code into a repo and they don't care whether files move around 
or change heavily. They happily mix changes with reformatting the 
complete code. In that case you have no chance of a useful merge-commit. 
I think in these cases it's most likely better to create a new branch on 
every update and rebase the patches.




Anyway, still this is a branch based workflow. How would you do that 
workflow with just fixed commit IDs?


I'm asking since so far commit IDs on submodules were always obtain from 
master branch which is what we basically fork and does not give us 
ability to patch with RTEMS changes...


So I'm curious how would you do that?


I think I have put that in the wrong words. I assumed you mean that you 
want to use some feature that always uses the latest commit of a branch 
(also I don't know whether git even has that feature).


What I wanted to say is that if I check out a RTEMS version from two 
months back, it should check out the revisions of the submodules that 
have been used in RTEMS two months back too. It must not use the latest 
HEAD of the branch from today. It's just part of having a version of the 
code that can be reproduced later.


Of course the revision from two months back has to be available in the 
repo. It doesn't really matter whether it's on master because the code 
just worked or whether it's a RTEMS specific branch because adaptions 
have been necessary.


Best regards

Christian



Thanks!
Karel



___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Discussion: How to handle HALs, SDKs and libraries

[Karel Gardas]

On 7/15/23 16:00, o...@c-mauderer.de wrote:
We should avoid rebasing and overwriting the old branch because we have 
to preserve a version that can be used with older RTEMS versions. So for 
rebasing, we would have to use multiple branches. Instead of 
rtems-6-branch, it could be a rtems- or something 
similar.


Thanks for confirmation. Have the same feeling about it.

I think that depends heavily on how the upstream project works. If the 
upstream project uses a typical git workflow and is quite active, 
merging is most likely the better solution because we don't get big 
amounts of branches with that.


But this means we still merge to some rtems specific branch since master 
is used to sync with upstream, right?


On the other hand I know that some semiconductor manufacturers tend to 
just dump code into a repo and they don't care whether files move around 
or change heavily. They happily mix changes with reformatting the 
complete code. In that case you have no chance of a useful merge-commit. 
I think in these cases it's most likely better to create a new branch on 
every update and rebase the patches.




Anyway, still this is a branch based workflow. How would you do that 
workflow with just fixed commit IDs?


I'm asking since so far commit IDs on submodules were always obtain 
from master branch which is what we basically fork and does not give 
us ability to patch with RTEMS changes...


So I'm curious how would you do that?


I think I have put that in the wrong words. I assumed you mean that you 
want to use some feature that always uses the latest commit of a branch 
(also I don't know whether git even has that feature).


What I wanted to say is that if I check out a RTEMS version from two 
months back, it should check out the revisions of the submodules that 
have been used in RTEMS two months back too. It must not use the latest 
HEAD of the branch from today. It's just part of having a version of the 
code that can be reproduced later.


Right! Now crucial question: is that even supported in git? I'm asking 
since so far I've seen only:


- use fixed commit ids, but those are obtained only from master branch
or
- use branch and then this is always branch HEAD

I've not seen anything like "use fixed commit ID from branch XXX". So 
question for you as an git expert: are you absolutely sure that this 
workflow is supported by git at all? And by this I mean have you tested 
that actually? :-)


Of course the revision from two months back has to be available in the 
repo. It doesn't really matter whether it's on master because the code 
just worked or whether it's a RTEMS specific branch because adaptions 
have been necessary.


You "it doesn't really matter" worries me a little bit. I'm not sure 
this functionality is even supported hence my question above if you have 
tested it yourself.


Thanks!
Karel

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Discussion: How to handle HALs, SDKs and libraries

[oss]

Hello Karel,

Am 15.07.23 um 19:22 schrieb Karel Gardas:

On 7/15/23 16:00, o...@c-mauderer.de wrote:
We should avoid rebasing and overwriting the old branch because we 
have to preserve a version that can be used with older RTEMS versions. 
So for rebasing, we would have to use multiple branches. Instead of 
rtems-6-branch, it could be a rtems- or 
something similar.


Thanks for confirmation. Have the same feeling about it.

I think that depends heavily on how the upstream project works. If the 
upstream project uses a typical git workflow and is quite active, 
merging is most likely the better solution because we don't get big 
amounts of branches with that.


But this means we still merge to some rtems specific branch since master 
is used to sync with upstream, right?




Yes. That would be my suggested workflow: Track the upstream master or 
main branch and merge it into an RTEMS specific one.


On the other hand I know that some semiconductor manufacturers tend to 
just dump code into a repo and they don't care whether files move 
around or change heavily. They happily mix changes with reformatting 
the complete code. In that case you have no chance of a useful 
merge-commit. I think in these cases it's most likely better to create 
a new branch on every update and rebase the patches.




Anyway, still this is a branch based workflow. How would you do that 
workflow with just fixed commit IDs?


I'm asking since so far commit IDs on submodules were always obtain 
from master branch which is what we basically fork and does not give 
us ability to patch with RTEMS changes...


So I'm curious how would you do that?


I think I have put that in the wrong words. I assumed you mean that 
you want to use some feature that always uses the latest commit of a 
branch (also I don't know whether git even has that feature).


What I wanted to say is that if I check out a RTEMS version from two 
months back, it should check out the revisions of the submodules that 
have been used in RTEMS two months back too. It must not use the 
latest HEAD of the branch from today. It's just part of having a 
version of the code that can be reproduced later.


Right! Now crucial question: is that even supported in git? I'm asking 
since so far I've seen only:


- use fixed commit ids, but those are obtained only from master branch
or
- use branch and then this is always branch HEAD

I've not seen anything like "use fixed commit ID from branch XXX". So 
question for you as an git expert: are you absolutely sure that this 
workflow is supported by git at all? And by this I mean have you tested 
that actually? :-)


I know only of the method that git uses a fixed commit ID in a 
submodule. It expects that the commit ID is on some branch in the remote 
repository. In my experience, it's not relevant whether the commit is on 
the branch `main`, `master`, `foo-bar`, `my-cool-new-feature` or any 
other branch. And yes: I use that regularly.


Additionally, git support adding an optional "branch" parameter to each 
submodule. But as far as I know, that is only used during


  git submodule update --remote

which basically tells git to pull the latest version of the branch 
instead of a special commit ID. I usually don't use that.


See https://stackoverflow.com/a/18797720 for a better explanation.



Of course the revision from two months back has to be available in the 
repo. It doesn't really matter whether it's on master because the code 
just worked or whether it's a RTEMS specific branch because adaptions 
have been necessary.


You "it doesn't really matter" worries me a little bit. I'm not sure 
this functionality is even supported hence my question above if you have 
tested it yourself.


Yes, I use that regularly. For an (old) example, take a look at a branch 
in the repo that I use to test GSoC work:


  https://gitlab.com/c-mauderer/rtems-bbb/-/tree/6

I have a libs/rtems-libbsd submodule there that uses the revision 
c9474c0b which is on a cm/20200726_fix branch in my clone of 
rtems-libbsd. In the .gitmodules the branch is still set to "master". 
Despite that, git can find thre right commit ID.


Best regards

Christian



Thanks!
Karel


___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Discussion: How to handle HALs, SDKs and libraries

[Karel Gardas]

On 7/16/23 10:03, o...@c-mauderer.de wrote:
Of course the revision from two months back has to be available in 
the repo. It doesn't really matter whether it's on master because the 
code just worked or whether it's a RTEMS specific branch because 
adaptions have been necessary.


You "it doesn't really matter" worries me a little bit. I'm not sure 
this functionality is even supported hence my question above if you 
have tested it yourself.


Yes, I use that regularly. For an (old) example, take a look at a branch 
in the repo that I use to test GSoC work:


   https://gitlab.com/c-mauderer/rtems-bbb/-/tree/6

I have a libs/rtems-libbsd submodule there that uses the revision 
c9474c0b which is on a cm/20200726_fix branch in my clone of 
rtems-libbsd. In the .gitmodules the branch is still set to "master". 
Despite that, git can find thre right commit ID.


Cool! So it didn't work for me due to (i) my git bug or (ii) my git 
doesn't supporting whatever or simply due to (iii) me making silly 
mistake somewhere.


Will need to recheck this. Thanks for the info!

Karel


___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Discussion: How to handle HALs, SDKs and libraries

[oss]

Hello Karel,

Am 16.07.23 um 12:37 schrieb Karel Gardas:

On 7/16/23 10:03, o...@c-mauderer.de wrote:
Of course the revision from two months back has to be available in 
the repo. It doesn't really matter whether it's on master because 
the code just worked or whether it's a RTEMS specific branch because 
adaptions have been necessary.


You "it doesn't really matter" worries me a little bit. I'm not sure 
this functionality is even supported hence my question above if you 
have tested it yourself.


Yes, I use that regularly. For an (old) example, take a look at a 
branch in the repo that I use to test GSoC work:


   https://gitlab.com/c-mauderer/rtems-bbb/-/tree/6

I have a libs/rtems-libbsd submodule there that uses the revision 
c9474c0b which is on a cm/20200726_fix branch in my clone of 
rtems-libbsd. In the .gitmodules the branch is still set to "master". 
Despite that, git can find thre right commit ID.


Cool! So it didn't work for me due to (i) my git bug or (ii) my git 
doesn't supporting whatever or simply due to (iii) me making silly 
mistake somewhere.


Will need to recheck this. Thanks for the info!

Karel




Seems that we are not the only ones with a waf based build system that 
want to manipulate submodules:


https://github.com/ArduPilot/ardupilot/pull/3592

The final solution seems to be a bit different. Their wscript in the 
latest version has a "--no-submodule-update" parameter. So they decided 
to just update the submodules every time. Can be interesting for users 
because they don't have to care for outdated submodules if they pull a 
new version.


But I think the problem with adding it as a fixed step is, that we have 
a problem with the releases. A release tar ball is not a git repo 
anymore. So a "git submodule init --update" or similar will not work. We 
would need some mechanism to detect whether it's a git repo or not.


Best regards

Christian
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Westernization of RTEMS/stm32H7 [was: Re: Discussion: How to handle HALs, SDKs and libraries]

[Karel Gardas]

  Hello,

Christian mentioned Zephyr west tool as one of good/possible candidates 
for RTEMS tree modularization especially with regarding to HALs and 
other 3rd party libraries.


For comparison with my previous git submodulization of RTEMS/stm32H7 
I've also westernized the same in a similar way. Yes, it is intended to 
remove CMSIS v4 *just* to proof compilation is working with CMSIS 5 from 
outside module! This is just dirty PoC nothing more. A tree structure 
looks a bit different as (folowing zephyr example) I keep modules 
outside the rtems tree. Hence result for west init command below is tree:


modules/hal/stm32h7/stm32h7xx_hal_driver
modules/hal/stm32h7/cmsis_device_h7
modules/hal/stm32h7/CMSIS_5
rtems/...


So if you like to see thing, then install west:

$ pip3 install --user west
$ export PATH=$PATH:$HOME/.local/bin

go into some testing directory and:

$ west init -m https://github.com/karelfv/rtems --mr 
rtems-west-stm32h7-hal rtems-workspace


$ cd rtems-workspace/
$ west update

Compilation again produces working samples. In comparison with git 
submodule approach I find west a bit more civilized (user friendly). 
Nothing is shooting in my own feet etc.


What may resonate in RTEMS devs is:

- yaml format for module configuration
- west implemented in python3
- west extensible

It probably will not be big issue to implement custom:

- west bsplist
- west bspdefaults
- west configure
- west build

which would just delegate to waf for actual work if needed...

Comments welcome!

Karel

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: Westernization of RTEMS/stm32H7 [was: Re: Discussion: How to handle HALs, SDKs and libraries]

[oss]

Hello Karel,

Am 16.07.23 um 00:07 schrieb Karel Gardas:


   Hello,

Christian mentioned Zephyr west tool as one of good/possible candidates 
for RTEMS tree modularization especially with regarding to HALs and 
other 3rd party libraries.


I mentioned it as one of three tools that I know that solve similar 
problems (Google `repo` from Android, `west` from Zephyr and the feeds 
from OpenWRT). And clearly I haven't made clear enough what my goal was:


I don't want to suggest that we jump on a random tool and use that. What 
I wanted is that we take a look at the reasons why the tools exist and 
evaluate whether we - as RTEMS project - have similar problems that have 
been solved with the tool. You wanted to avoid the academic debate in an 
earlier mail, so I didn't start to go into more details of every reason. 
But now there is a second prototype with a different tool so most likely 
we have to compare some more.


West has the best documentation why they created that tool:

  https://docs.zephyrproject.org/latest/develop/west/why.html

R1. Keeping code out of the main repo. That has advantages and 
disadvantages. And I'm not sure what's more important. Main advantage is 
that it makes more clear what is RTEMS and what is some external source. 
 Main disadvantage is most likely that it is a bit more difficult to 
integrate it into the release process. If it's only one repo, the 
release will be to check out the submodules and archive everything as 
one big file. With multiple separate repos, we have to create one file 
for every repo. Additionally, a user has to download all archives to 
build the release version. So most likely that adds complexity.


R2. Tool, that users can use. Depends quite heavily on the workflow and 
use case of the user. RTEMS has a tendency to not adopt new tools 
easily, especially if they have to be used by the end user. So that 
could be a disadvantage.


R3. Allow to override repos in custom distributions. Not sure whether 
that is useful for us as a project.


R4. Continuous tracking of a HEAD or commit-based tracking. Git does the 
commit-based tracking and I already said that I wouldn't be happy about 
continuous tracking of a branch. So that wouldn't be a reason for me.


From my point of view, what we can learn from that tool is two things:

A) We should take a look whether putting the code in the repo is a 
disadvantage in some situations. I think size of releases and licenses 
could be two. Licenses can be most likely solved with a clear directory 
structure where external libs are in one subdirectory with a good 
description.


B) Do we need to support use cases, where a user wants to replace one 
submodule version by another version without having to create a private 
fork of RTEMS?


C) Do we need to support use cases where we don't want to track fixed 
commit IDs but the HEAD of a branch instead?



For Google repo, I didn't find the original reasons. What I did find are 
some comparisons with submodules like 
https://www.edureka.co/blog/git-submodules-versus-googles-repo-tool


Sounds like the main intention is to make working with multiple repos 
simpler and integrate well with Gerrit and Jenkins. Beneath that, they 
heavily support tracking HEADs of branches.


Main point to learn:

Again: C)

D) We should be careful how we handle submodules so that the usage is as 
simple as possible. So that's part of how we integrate it in our build 
system.


E) We maybe can get problems on a future CI/CD integration.


The OpenWRT Feeds: https://openwrt.org/docs/guide-developer/feeds

I think the main reason why they use that are:

1. Allow to track a HEAD of a branch instead of a certain commit ID. Not 
something that I would want for HALs.


2. Allow a user to add additional feeds. That's similar to the point 3 
of `west`.


I think the main point to learn is basically the same as B) and C).

Best regards

Christian




For comparison with my previous git submodulization of RTEMS/stm32H7 
I've also westernized the same in a similar way. Yes, it is intended to 
remove CMSIS v4 *just* to proof compilation is working with CMSIS 5 from 
outside module! This is just dirty PoC nothing more. A tree structure 
looks a bit different as (folowing zephyr example) I keep modules 
outside the rtems tree. Hence result for west init command below is tree:


modules/hal/stm32h7/stm32h7xx_hal_driver
modules/hal/stm32h7/cmsis_device_h7
modules/hal/stm32h7/CMSIS_5
rtems/...


So if you like to see thing, then install west:

$ pip3 install --user west
$ export PATH=$PATH:$HOME/.local/bin

go into some testing directory and:

$ west init -m https://github.com/karelfv/rtems --mr 
rtems-west-stm32h7-hal rtems-workspace


$ cd rtems-workspace/
$ west update

Compilation again produces working samples. In comparison with git 
submodule approach I find west a bit more civilized (user friendly). 
Nothing is shooting in my own feet etc.


What may resonate in RTEMS devs is:

- yaml format for modu