[PATCH 1/3] build: Format build items
Use yaml.dump(data, default_flow_style=False, allow_unicode=True) to format all build items. --- spec/build/bsps/aarch64/a53/optramlen.yml | 2 +- spec/build/bsps/aarch64/a53/tsta53.yml| 22 +++- spec/build/bsps/aarch64/a72/optramlen.yml | 2 +- spec/build/bsps/aarch64/a72/tsta72.yml| 22 +++- spec/build/bsps/aarch64/optmmupages.yml | 6 +-- spec/build/bsps/aarch64/raspberrypi/abi.yml | 5 +- .../aarch64/raspberrypi/bspraspberrypi4.yml | 52 +-- .../bsps/aarch64/xilinx-versal/optloadoff.yml | 2 +- .../aarch64/xilinx-versal/optnocachelen.yml | 2 +- .../bsps/aarch64/xilinx-versal/optramlen.yml | 2 +- .../bsps/aarch64/xilinx-versal/optramori.yml | 4 +- .../bsps/aarch64/xilinx-versal/tstaiedge.yml | 2 - .../bsps/aarch64/xilinx-versal/tstqemu.yml| 2 - .../bsps/aarch64/xilinx-versal/tstvck190.yml | 2 - .../bsps/aarch64/xilinx-zynqmp/optloadoff.yml | 2 +- .../aarch64/xilinx-zynqmp/optnocachelen.yml | 2 +- .../bsps/aarch64/xilinx-zynqmp/optramlen.yml | 2 +- .../bsps/aarch64/xilinx-zynqmp/optramori.yml | 4 +- .../bsps/aarch64/xilinx-zynqmp/tstqemu.yml| 22 +++- .../bsps/aarch64/xilinx-zynqmp/tstzu3eg.yml | 2 - spec/build/bsps/arm/atsam/optnullsz.yml | 10 ++-- spec/build/bsps/arm/imxrt/linkcmds.yml| 6 +-- spec/build/bsps/arm/imxrt/linkcmdsmemory.yml | 6 +-- spec/build/bsps/arm/imxrt/optlinkcmds.yml | 12 ++--- spec/build/bsps/arm/imxrt/optmemdtcmsz.yml| 14 ++--- .../bsps/arm/imxrt/optmemextramnocachesz.yml | 12 ++--- .../bsps/arm/imxrt/optmemextramorigin.yml | 14 ++--- spec/build/bsps/arm/imxrt/optmemextramsz.yml | 16 +++--- .../build/bsps/arm/imxrt/optmemflashcfgsz.yml | 16 +++--- .../build/bsps/arm/imxrt/optmemflashivtsz.yml | 16 +++--- .../bsps/arm/imxrt/optmemflashorigin.yml | 16 +++--- spec/build/bsps/arm/imxrt/optmemflashsz.yml | 16 +++--- spec/build/bsps/arm/imxrt/optmemitcmsz.yml| 16 +++--- spec/build/bsps/arm/imxrt/optmemnullsz.yml| 16 +++--- .../bsps/arm/imxrt/optmemocramnocachesz.yml | 12 ++--- spec/build/bsps/arm/imxrt/optmemocramsz.yml | 16 +++--- spec/build/bsps/arm/optgiccpuif.yml | 2 +- spec/build/bsps/arm/optgicdist.yml| 4 +- spec/build/bsps/arm/optgicredist.yml | 4 +- spec/build/bsps/arm/optgtfreq.yml | 2 +- spec/build/bsps/arm/optgtsysbase.yml | 2 +- spec/build/bsps/arm/optgtsyscntcr.yml | 2 +- spec/build/bsps/arm/stm32h7/abi.yml | 10 ++-- spec/build/bsps/arm/stm32h7/linkcmds.yml | 6 +-- spec/build/bsps/arm/stm32h7/linkcmdsflash.yml | 6 +-- .../bsps/arm/stm32h7/linkcmdsflashsdram.yml | 6 +-- .../build/bsps/arm/stm32h7/linkcmdsmemory.yml | 6 +-- spec/build/bsps/arm/stm32h7/linkcmdssdram.yml | 6 +-- spec/build/bsps/arm/stm32h7/linkcmdssram.yml | 6 +-- .../bsps/arm/stm32h7/linkcmdssramsdram.yml| 6 +-- spec/build/bsps/arm/stm32h7/optbootcore.yml | 18 +++ spec/build/bsps/arm/stm32h7/optenmpualign.yml | 14 ++--- spec/build/bsps/arm/stm32h7/optenuart4.yml| 10 ++-- spec/build/bsps/arm/stm32h7/optenuart5.yml| 10 ++-- spec/build/bsps/arm/stm32h7/optenuart7.yml| 10 ++-- spec/build/bsps/arm/stm32h7/optenuart8.yml| 10 ++-- spec/build/bsps/arm/stm32h7/optenuart9.yml| 10 ++-- spec/build/bsps/arm/stm32h7/optenusart1.yml | 10 ++-- spec/build/bsps/arm/stm32h7/optenusart10.yml | 10 ++-- spec/build/bsps/arm/stm32h7/optenusart2.yml | 11 ++-- spec/build/bsps/arm/stm32h7/optenusart3.yml | 10 ++-- spec/build/bsps/arm/stm32h7/optenusart6.yml | 10 ++-- .../bsps/arm/stm32h7/optethgpiobregs.yml | 10 ++-- .../bsps/arm/stm32h7/optethgpiogregs.yml | 10 ++-- spec/build/bsps/arm/stm32h7/opthse.yml| 8 +-- spec/build/bsps/arm/stm32h7/optlinkcmds.yml | 14 ++--- spec/build/bsps/arm/stm32h7/optmemdtcmsz.yml | 12 ++--- .../bsps/arm/stm32h7/optmemflashlatency.yml | 10 ++-- .../bsps/arm/stm32h7/optmemflashorigin.yml| 14 ++--- spec/build/bsps/arm/stm32h7/optmemflashsz.yml | 15 +++--- spec/build/bsps/arm/stm32h7/optmemitcmsz.yml | 16 +++--- spec/build/bsps/arm/stm32h7/optmemnandsz.yml | 10 ++-- spec/build/bsps/arm/stm32h7/optmemnorsz.yml | 10 ++-- spec/build/bsps/arm/stm32h7/optmemnullsz.yml | 12 ++--- .../bsps/arm/stm32h7/optmemperipheralsz.yml | 12 ++--- .../bsps/arm/stm32h7/optmemquadspisz.yml | 10 ++-- .../build/bsps/arm/stm32h7/optmemsdram1sz.yml | 10 ++-- .../build/bsps/arm/stm32h7/optmemsdram2sz.yml | 10 ++-- spec/build/bsps/arm/stm32h7/optmemsram1sz.yml | 14 ++--- spec/build/bsps/arm/stm32h7/optmemsram2sz.yml | 14 ++--- spec/build/bsps/arm/stm32h7/optmemsram3sz.yml | 14 ++--- spec/build/bsps/arm/stm32h7/optmemsram4sz.yml | 14 ++--- .../bsps/arm/stm32h7/optmemsramaxisz.yml | 14 ++--- .../bsps/arm/stm32h7/optmemsrambackupsz.yml | 12 ++--- .../bsps/arm/stm32h7/optprintkinstance.yml| 10 ++-- spec/build/bsps/arm/stm32h7/optpwrsupply.yml
Re: [PATCH 1/3] build: Format build items
On 12.01.23 15:44, Kinsey Moore wrote: The other two patches look fine to me. The use of dump() that results in this patch does several things: * Removal of whitespace This is fine for whitespace at the base level of indentation. Whitespace within an indented block may be more important for readability. * Removal of comments This is not good as they are exclusively used to annotate manually ordered blocks of test result expectations * Rearrangement of items in alphabetical order In general, rearrangement of top-level sections is good. For indented sections specifically in tst*.yml, this is bad for the above reaso One goal of the new build system was to be able to alter the data through scripts. This requires that the build items are human and machine readable and writable. The Python YAML import/export does not preserve white space and comments. This there is a need for comments, then the data format should be improved. For example, the test state list could be changed to also include a reason for a categorization. * Alteration of integer representation to base-10 in all cases I'd say this is a net-negative as most items represented in hex are memory-related and should stay that way. Yes, this is a bit annoying. To mitigate this issue the options have a format attribute so that the numbers are correctly formatted in user visible content, for example the configuration files or generated header and linker command files. The build item maintainer has to use a calculator or whatever to look at the numbers in the right format. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 12.01.23 16:52, Kinsey Moore wrote: Annotation of expected test states with a description could get quite repetitive, but I suppose that's better than losing the information. I'm fine with this going in for now with a ticket to address this issue and we can pull the annotations back in when the ticket is dealt with. It could look like this: SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause actions: - set-test-state: - reason: | expected to fail, don't compile these state: exclude tests: - minimum - reason: | tests that are passing intermittently state: indeterminate tests: - psx12 - psxtimes01 - rtmonuse - rtmonusxtimes01 - sp04 - sp20 - sp68 - sp69 - spcpucounter01 - spedfsched02 - spedfsched04 - sprmsched01 - sptimecounter02 - sptimecounter04 - ttest02 - reason: | tests that pass nominally, but fail under Qemu when the host is under heavy load state: indeterminate tests: - spintrcritical03 - spintrcritical04 - spintrcritical05 build-type: option copyrights: - Copyright (C) 2020 On-Line Applications Research (OAR) default: [] description: '' enabled-by: true links: [] type: build -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 1/12/2023 10:38 AM, Sebastian Huber wrote: On 12.01.23 16:52, Kinsey Moore wrote: Annotation of expected test states with a description could get quite repetitive, but I suppose that's better than losing the information. I'm fine with this going in for now with a ticket to address this issue and we can pull the annotations back in when the ticket is dealt with. It could look like this: SPDX-License-Identifier: CC-BY-SA-4.0 OR BSD-2-Clause actions: - set-test-state: - reason: | expected to fail, don't compile these state: exclude tests: - minimum - reason: | tests that are passing intermittently state: indeterminate tests: - psx12 - psxtimes01 - rtmonuse - rtmonusxtimes01 - sp04 - sp20 - sp68 - sp69 - spcpucounter01 - spedfsched02 - spedfsched04 - sprmsched01 - sptimecounter02 - sptimecounter04 - ttest02 - reason: | tests that pass nominally, but fail under Qemu when the host is under heavy load state: indeterminate tests: - spintrcritical03 - spintrcritical04 - spintrcritical05 build-type: option copyrights: - Copyright (C) 2020 On-Line Applications Research (OAR) default: [] description: '' enabled-by: true links: [] type: build That seems like a reasonable format for test state expectations. Kinsey ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 13/1/2023 1:54 am, Sebastian Huber wrote: > On 12.01.23 15:44, Kinsey Moore wrote: >> The other two patches look fine to me. The use of dump() that results in this >> patch does several things: >> * Removal of whitespace >> This is fine for whitespace at the base level of indentation. Whitespace >> within an indented block may be more important for readability. >> >> * Removal of comments >> This is not good as they are exclusively used to annotate manually ordered >> blocks of test result expectations >> >> * Rearrangement of items in alphabetical order >> In general, rearrangement of top-level sections is good. For indented >> sections >> specifically in tst*.yml, this is bad for the above reaso > One goal of the new build system was to be able to alter the data through > scripts. This requires that the build items are human and machine readable and > writable. The Python YAML import/export does not preserve white space and > comments. Can someone edit the file and add a hex number? > This there is a need for comments, then the data format should be improved. > For > example, the test state list could be changed to also include a reason for a > categorization. > >> >> * Alteration of integer representation to base-10 in all cases >> I'd say this is a net-negative as most items represented in hex are >> memory-related and should stay that way. > > Yes, this is a bit annoying. To mitigate this issue the options have a format > attribute so that the numbers are correctly formatted in user visible content, > for example the configuration files or generated header and linker command > files. The build item maintainer has to use a calculator or whatever to look > at > the numbers in the right format. I cannot see an example of a format attribute? Do the patches contain an example? Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 16.01.23 01:35, Chris Johns wrote: On 13/1/2023 1:54 am, Sebastian Huber wrote: On 12.01.23 15:44, Kinsey Moore wrote: The other two patches look fine to me. The use of dump() that results in this patch does several things: * Removal of whitespace This is fine for whitespace at the base level of indentation. Whitespace within an indented block may be more important for readability. * Removal of comments This is not good as they are exclusively used to annotate manually ordered blocks of test result expectations * Rearrangement of items in alphabetical order In general, rearrangement of top-level sections is good. For indented sections specifically in tst*.yml, this is bad for the above reaso One goal of the new build system was to be able to alter the data through scripts. This requires that the build items are human and machine readable and writable. The Python YAML import/export does not preserve white space and comments. Can someone edit the file and add a hex number? Yes, you can manually use whatever format is understood by the YAML loader. When you write the file with the YAML dumper, then the standard representation is used. I changed the representer to use the format attribute, see v2 of the patch: https://lists.rtems.org/pipermail/devel/2023-January/074094.html This there is a need for comments, then the data format should be improved. For example, the test state list could be changed to also include a reason for a categorization. * Alteration of integer representation to base-10 in all cases I'd say this is a net-negative as most items represented in hex are memory-related and should stay that way. Yes, this is a bit annoying. To mitigate this issue the options have a format attribute so that the numbers are correctly formatted in user visible content, for example the configuration files or generated header and linker command files. The build item maintainer has to use a calculator or whatever to look at the numbers in the right format. I cannot see an example of a format attribute? Do the patches contain an example? grep 'format:' -r spec -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 16/1/2023 6:56 pm, Sebastian Huber wrote: > On 16.01.23 01:35, Chris Johns wrote: >> On 13/1/2023 1:54 am, Sebastian Huber wrote: >>> On 12.01.23 15:44, Kinsey Moore wrote: The other two patches look fine to me. The use of dump() that results in this patch does several things: * Removal of whitespace This is fine for whitespace at the base level of indentation. Whitespace within an indented block may be more important for readability. * Removal of comments This is not good as they are exclusively used to annotate manually ordered blocks of test result expectations * Rearrangement of items in alphabetical order In general, rearrangement of top-level sections is good. For indented sections specifically in tst*.yml, this is bad for the above reaso >>> One goal of the new build system was to be able to alter the data through >>> scripts. This requires that the build items are human and machine readable >>> and >>> writable. The Python YAML import/export does not preserve white space and >>> comments. >> Can someone edit the file and add a hex number? > > Yes, you can manually use whatever format is understood by the YAML loader. > When > you write the file with the YAML dumper, then the standard representation is > used. Are there python modules in rtems.git someone could import that reads and writes the YAML spec files? If not should we provide a module to do this? It could be `spec` so a user can use `import spec` after setting the import path. That is, if external scripts (and I encourage this) all used a common read and write type functionality the format consistency is relative to specific version of rtems.git being used. Updates become read then write. > I changed the representer to use the format attribute, see v2 of the patch: > > https://lists.rtems.org/pipermail/devel/2023-January/074094.html > I see and thanks. How many format strings would cover the majority of formats we have? I am wondering if `format:` is checked against a standard list and those are part of the "writer" support? For example `address`, `address32`, `hex64` etc? I am concerned about repeated common format strings being placed through all the spec files. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 17.01.23 03:48, Chris Johns wrote: On 16/1/2023 6:56 pm, Sebastian Huber wrote: On 16.01.23 01:35, Chris Johns wrote: On 13/1/2023 1:54 am, Sebastian Huber wrote: On 12.01.23 15:44, Kinsey Moore wrote: The other two patches look fine to me. The use of dump() that results in this patch does several things: * Removal of whitespace This is fine for whitespace at the base level of indentation. Whitespace within an indented block may be more important for readability. * Removal of comments This is not good as they are exclusively used to annotate manually ordered blocks of test result expectations * Rearrangement of items in alphabetical order In general, rearrangement of top-level sections is good. For indented sections specifically in tst*.yml, this is bad for the above reaso One goal of the new build system was to be able to alter the data through scripts. This requires that the build items are human and machine readable and writable. The Python YAML import/export does not preserve white space and comments. Can someone edit the file and add a hex number? Yes, you can manually use whatever format is understood by the YAML loader. When you write the file with the YAML dumper, then the standard representation is used. Are there python modules in rtems.git someone could import that reads and writes the YAML spec files? If not should we provide a module to do this? It could be `spec` so a user can use `import spec` after setting the import path. That is, if external scripts (and I encourage this) all used a common read and write type functionality the format consistency is relative to specific version of rtems.git being used. Updates become read then write. The Python modules to work with specification items are in rtems-central.git. This repository contains also a format specification of the build items. We could add an action to a Github work flow to check the build item format for pull requests and commits. I changed the representer to use the format attribute, see v2 of the patch: https://lists.rtems.org/pipermail/devel/2023-January/074094.html I see and thanks. How many format strings would cover the majority of formats we have? I am wondering if `format:` is checked against a standard list and those are part of the "writer" support? For example `address`, `address32`, `hex64` etc? I am concerned about repeated common format strings being placed through all the spec files. The format string is a standard Python format string. This is easy to implement and flexible. We could replace this with a fixed set of formats. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 17/1/2023 6:39 pm, Sebastian Huber wrote: > On 17.01.23 03:48, Chris Johns wrote: >> On 16/1/2023 6:56 pm, Sebastian Huber wrote: >>> On 16.01.23 01:35, Chris Johns wrote: On 13/1/2023 1:54 am, Sebastian Huber wrote: > On 12.01.23 15:44, Kinsey Moore wrote: >> The other two patches look fine to me. The use of dump() that results in >> this >> patch does several things: >> * Removal of whitespace >> This is fine for whitespace at the base level of indentation. Whitespace >> within an indented block may be more important for readability. >> >> * Removal of comments >> This is not good as they are exclusively used to annotate manually >> ordered >> blocks of test result expectations >> >> * Rearrangement of items in alphabetical order >> In general, rearrangement of top-level sections is good. For indented >> sections >> specifically in tst*.yml, this is bad for the above reaso > One goal of the new build system was to be able to alter the data through > scripts. This requires that the build items are human and machine > readable and > writable. The Python YAML import/export does not preserve white space and > comments. Can someone edit the file and add a hex number? >>> >>> Yes, you can manually use whatever format is understood by the YAML loader. >>> When >>> you write the file with the YAML dumper, then the standard representation is >>> used. >> >> Are there python modules in rtems.git someone could import that reads and >> writes >> the YAML spec files? If not should we provide a module to do this? It could >> be >> `spec` so a user can use `import spec` after setting the import path. >> >> That is, if external scripts (and I encourage this) all used a common read >> and >> write type functionality the format consistency is relative to specific >> version >> of rtems.git being used. Updates become read then write. > > The Python modules to work with specification items are in rtems-central.git. > This repository contains also a format specification of the build items. And how does that help here with this repo? I suggest you reconsider the accessibility of those modules before pushing scripted generation changes like this into rtems.git. > We > could add an action to a Github work flow to check the build item format for > pull requests and commits. Thanks for including github in this thread. It has now confirmed my position of no github automations in our repos (including rtems-central). >>> I changed the representer to use the format attribute, see v2 of the patch: >>> >>> https://lists.rtems.org/pipermail/devel/2023-January/074094.html >>> >> >> I see and thanks. How many format strings would cover the majority of >> formats we >> have? >> >> I am wondering if `format:` is checked against a standard list and those are >> part of the "writer" support? For example `address`, `address32`, `hex64` >> etc? I >> am concerned about repeated common format strings being placed through all >> the >> spec files. > > The format string is a standard Python format string. This is easy to > implement > and flexible. We could replace this with a fixed set of formats. Maybe if you have scripting support which this repo does not. I think the formats should be controlled however I see the whole discussion and patch as academic until rtems.git has scripting support in python modules. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 2023-01-17 08:39 +0100, Sebastian Huber wrote: > > The Python modules to work with specification items are in > rtems-central.git. This repository contains also a format specification > of the build items. We could add an action to a Github work flow to > check the build item format for pull requests and commits. I don't chime in here very often but after seeing the recent changes I'm extremely confused. There are now tools that sit outside of the RTEMS repository that now change source files in RTEMS? I have never seen it done this way before. What version of this tool have you used? How can anyone in the future possibly debug this? What should happen here is this tools should sit in the main repository. Any required changes made and then a command run to generate new output: ./waf This lets us know that the output in this commit was generated by code existing in the previous commit. There is no need to version anything it's already done. Having it sitting outside of the source repository makes debugging impossible. Case in point is the commit message here. No explanation as to where these changes have come from. No commit version. I spent a very long time figuring out what was going on and I could not figure it out without digging through the lists. How is anyone in 5, 10, 15 years down the road going to sort through this? What if I want to make a change in 10 years and regenerate. It is impossible as I have no idea what's been done let alone where to look. I'm not against the purpose of the tool or the work has been done but this is not the correct way to handle generated files within a repository if we want to maintain the ability to make changes or debug years down the line. Amar. ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 19/1/2023 8:58 am, Amar Takhar wrote: > On 2023-01-17 08:39 +0100, Sebastian Huber wrote: > >> >> The Python modules to work with specification items are in >> rtems-central.git. This repository contains also a format specification >> of the build items. We could add an action to a Github work flow to >> check the build item format for pull requests and commits. > > I don't chime in here very often but after seeing the recent changes I'm > extremely confused. > > There are now tools that sit outside of the RTEMS repository that now change > source files in RTEMS? > > I have never seen it done this way before. What version of this tool have > you > used? How can anyone in the future possibly debug this? > > What should happen here is this tools should sit in the main repository. Any > required changes made and then a command run to generate new output: > > ./waf > > This lets us know that the output in this commit was generated by code > existing > in the previous commit. There is no need to version anything it's already > done. > > Having it sitting outside of the source repository makes debugging > impossible. > Case in point is the commit message here. No explanation as to where these > changes have come from. No commit version. I spent a very long time > figuring > out what was going on and I could not figure it out without digging through > the > lists. > > How is anyone in 5, 10, 15 years down the road going to sort through this? > What > if I want to make a change in 10 years and regenerate. It is impossible as I > have no idea what's been done let alone where to look. > > I'm not against the purpose of the tool or the work has been done but this is > not the correct way to handle generated files within a repository if we want > to > maintain the ability to make changes or debug years down the line. Yes these are good points. The rtems-central repo is not a requirement for rtems.git and never will be. The separation was agreed to before any qual work started. The rtems.git repo can exist and operate without rtems-central however the rtems-central repo is meaningless without rtems.git so Amar I agree the dependency is the wrong way around. I have agreed to generated content for headers, some tests, and documentation because the generator input is in rtems-central so the effort to maintain that rtems-central data with any manual changes in rtems.git etc is part of the overhead of maintaining rtems-central. The build spec files are in rtems.git and this relationship is very different. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 18.01.23 22:58, Amar Takhar wrote: On 2023-01-17 08:39 +0100, Sebastian Huber wrote: The Python modules to work with specification items are in rtems-central.git. This repository contains also a format specification of the build items. We could add an action to a Github work flow to check the build item format for pull requests and commits. I don't chime in here very often but after seeing the recent changes I'm extremely confused. There are now tools that sit outside of the RTEMS repository that now change source files in RTEMS? I have never seen it done this way before. What version of this tool have you used? How can anyone in the future possibly debug this? In rtems-central.git there are Python modules and scripts which generate source, header, and documentation files from specification items. This repository contains the pre-qualification support for RTEMS. By accident, a part of the build system uses also specification items. So, if you want to change the format of the items, you can use the tooling available in rtems-central.git to do this. An example is the recent merge of the "default" and "default-by-variant" attributes. What should happen here is this tools should sit in the main repository. Any required changes made and then a command run to generate new output: ./waf This lets us know that the output in this commit was generated by code existing in the previous commit. There is no need to version anything it's already done. Having it sitting outside of the source repository makes debugging impossible. Case in point is the commit message here. No explanation as to where these changes have come from. No commit version. I spent a very long time figuring out what was going on and I could not figure it out without digging through the lists. The changes were made by a 10 liner throw away script. I could have done the changes also manually it would be just more work. How is anyone in 5, 10, 15 years down the road going to sort through this? What if I want to make a change in 10 years and regenerate. It is impossible as I have no idea what's been done let alone where to look. There is nothing to regenerate. The patch set contains simple format changes and a transformation of attributes. However, your concern is still valid for the generated files, for example https://github.com/RTEMS/rtems/blob/master/cpukit/include/rtems.h which is generated from https://github.com/RTEMS/rtems-central/blob/master/spec/rtems/if/header.yml Most of the Classic API header files are generated from specification items. If nobody takes care that rtems-central.git is used and stays up to date the specification will get out of synchronization with rtems.git. I'm not against the purpose of the tool or the work has been done but this is not the correct way to handle generated files within a repository if we want to maintain the ability to make changes or debug years down the line. I would use a single repository for RTEMS just like the FreeBSD base system, but this is another discussion. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 2023-01-19 08:21 +0100, Sebastian Huber wrote: > > In rtems-central.git there are Python modules and scripts which generate > source, header, and documentation files from specification items. This > repository contains the pre-qualification support for RTEMS. By > accident, a part of the build system uses also specification items. So, > if you want to change the format of the items, you can use the tooling > available in rtems-central.git to do this. An example is the recent > merge of the "default" and "default-by-variant" attributes. Why are these in 'rtems-central' and not the RTEMS repository? Why are we using tools with unpinned versioning generating files in RTEMS? > The changes were made by a 10 liner throw away script. I could have done > the changes also manually it would be just more work. This does not alter my above point. I understand that in this case it was a script but the rest wasn't generated by a throw-away script. > There is nothing to regenerate. The patch set contains simple format > changes and a transformation of attributes. > > However, your concern is still valid for the generated files, for example > > https://github.com/RTEMS/rtems/blob/master/cpukit/include/rtems.h > > which is generated from > > https://github.com/RTEMS/rtems-central/blob/master/spec/rtems/if/header.yml Yes I do not like this at all both the software and header.yml should be sitting in our main repository. > Most of the Classic API header files are generated from specification > items. If nobody takes care that rtems-central.git is used and stays up > to date the specification will get out of synchronization with rtems.git. Okay but why is this sitting in rtems-central.git and not in rtems.git. You've just made the argument that it's very important we don't want anything to be out of sync. > > I'm not against the purpose of the tool or the work has been done but this > > is > > not the correct way to handle generated files within a repository if we > > want to > > maintain the ability to make changes or debug years down the line. > > I would use a single repository for RTEMS just like the FreeBSD base > system, but this is another discussion. FreeBSD also uses a vendor system which brings all this code under a central repository. They aren't maintaining code outside and then requiring FreeBSD to be built with it. They even bring the compiler in this has an astounding impact on the usability of FreeBSD. Recently I took a FreeBSD 3.1 image and rebuilt it for a customer who needed a change it was used in an appliance. It just worked. If I want to make a chance to rtems.h it's now generated from a header.yml with no version pinning how am I going to find the first header.yml to make a single change in 20 years? rtems.h is a small file where we can get rid of it and move to another tool. It's integral and anything that works with it is integral this is arguably different. Amar. ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 20/1/2023 6:01 am, Amar Takhar wrote: > On 2023-01-19 08:21 +0100, Sebastian Huber wrote: >> >> In rtems-central.git there are Python modules and scripts which generate >> source, header, and documentation files from specification items. This >> repository contains the pre-qualification support for RTEMS. By >> accident, a part of the build system uses also specification items. So, >> if you want to change the format of the items, you can use the tooling >> available in rtems-central.git to do this. An example is the recent >> merge of the "default" and "default-by-variant" attributes. > > Why are these in 'rtems-central' and not the RTEMS repository? Why are we > using > tools with unpinned versioning generating files in RTEMS? My view is below. >> The changes were made by a 10 liner throw away script. I could have done >> the changes also manually it would be just more work. > > This does not alter my above point. I understand that in this case it was a > script but the rest wasn't generated by a throw-away script. It is good to know the amount of python is small. It should be easy to add :) >> There is nothing to regenerate. The patch set contains simple format >> changes and a transformation of attributes. >> >> However, your concern is still valid for the generated files, for example >> >> https://github.com/RTEMS/rtems/blob/master/cpukit/include/rtems.h >> >> which is generated from >> >> https://github.com/RTEMS/rtems-central/blob/master/spec/rtems/if/header.yml > > Yes I do not like this at all both the software and header.yml should be > sitting > in our main repository. I have been OK with the headers and tests being generated this way because the agreement is files in rtems.git can be manually edited and rtems-central has to track those changes. The sources and code to generate the files need to be located together. >> Most of the Classic API header files are generated from specification >> items. If nobody takes care that rtems-central.git is used and stays up >> to date the specification will get out of synchronization with rtems.git. > > Okay but why is this sitting in rtems-central.git and not in rtems.git. > You've > just made the argument that it's very important we don't want anything to be > out > of sync. The specification maintenance is a separate responsibility to the function and role of rtems.git and related repos. The bar to make changes in rtems.git is at the source level because it is the norm for open source projects and those working with the code. The separation also helps make it clear how funding of the qual related work happens. If we bring those specifications items down into rtems.git the responsibility moves to that repo and that raises the bar on being able to make a change. Our view on this may change as we learn now the pre-qual effort effects what we do. We have YAML files to run our build system and it is awesome. It also allows automation so I think it makes sense to provide an interface to read and write these files to aid automation rather than a disjointed set of personal scripts, tools etc that may break or clash. >>> I'm not against the purpose of the tool or the work has been done but this >>> is >>> not the correct way to handle generated files within a repository if we >>> want to >>> maintain the ability to make changes or debug years down the line. >> >> I would use a single repository for RTEMS just like the FreeBSD base >> system, but this is another discussion. > > FreeBSD also uses a vendor system which brings all this code under a central > repository. They aren't maintaining code outside and then requiring FreeBSD > to > be built with it. They even bring the compiler in this has an astounding > impact > on the usability of FreeBSD. > > Recently I took a FreeBSD 3.1 image and rebuilt it for a customer who needed > a > change it was used in an appliance. It just worked. > > If I want to make a chance to rtems.h it's now generated from a header.yml > with > no version pinning how am I going to find the first header.yml to make a > single > change in 20 years? > > rtems.h is a small file where we can get rid of it and move to another tool. > It's integral and anything that works with it is integral this is arguably > different. The FreeBSD single repo is about the kernel and base runtime. The ports are not part of this so the analogy breaks down. Bringing the RSB, RTEMS tools and documentation into a single repo creates new complexities I am not sure we are prepared enough to handle those. Maybe patch review and CI would help here and how they effect a single repo is something I have not given much consideration too. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
Sorry to hijack that thread, but correction is needed here. On 1/20/23 01:03, Chris Johns wrote: The FreeBSD single repo is about the kernel and base runtime. The ports are not part of this so the analogy breaks down. Certainly all BSDs have separated ports repos, but AFAIK all of them maintain so called base and in base there is kernel, libs, bin/sbin commands and most importantly *all* tool-chain required to build *all* this code to form whole OS. IMHO this rebuild whole OS distro is discussed here, isn't it? If not, I'm sorry for noise here... Karel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 2023-01-20 11:03 +1100, Chris Johns wrote: > It is good to know the amount of python is small. It should be easy to add :) Agreed. > I have been OK with the headers and tests being generated this way because the > agreement is files in rtems.git can be manually edited and rtems-central has > to > track those changes. The sources and code to generate the files need to be > located together. Yes. To be clear in this case if we have in rtems.git and in rtems-central.git where you need to edit to generate then both the tool *and* somefile.yml need to be in rtems.git. If what you are saying is the requirement is reversed. rtems-central.git needs rtems.git to work that's fine and the order it should be. > The specification maintenance is a separate responsibility to the function and > role of rtems.git and related repos. The bar to make changes in rtems.git is > at > the source level because it is the norm for open source projects and those > working with the code. The separation also helps make it clear how funding of > the qual related work happens. If we bring those specifications items down > into > rtems.git the responsibility moves to that repo and that raises the bar on > being > able to make a change. Our view on this may change as we learn now the > pre-qual > effort effects what we do. I think the important part here is it's evolving. Change happens sometimes you do one thing one way to discover you really want or need to do it another. > We have YAML files to run our build system and it is awesome. It also allows > automation so I think it makes sense to provide an interface to read and write > these files to aid automation rather than a disjointed set of personal > scripts, > tools etc that may break or clash. Agreed on this. > The FreeBSD single repo is about the kernel and base runtime. The ports are > not > part of this so the analogy breaks down. Bringing the RSB, RTEMS tools and > documentation into a single repo creates new complexities I am not sure we are > prepared enough to handle those. Maybe patch review and CI would help here and > how they effect a single repo is something I have not given much > consideration too. In theory I'm not against bringing everything together into one repository it allows for a lot of inter connectivity we can't do right now. Regardless even if we had been doing it this way I would still advocate for the change being in the /rtems/ directory versus /rtems-central/ Amar. ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 20/1/2023 11:22 am, Karel Gardas wrote: > > Sorry to hijack that thread, but correction is needed here. > > On 1/20/23 01:03, Chris Johns wrote: >> The FreeBSD single repo is about the kernel and base runtime. The ports are >> not >> part of this so the analogy breaks down. > > Certainly all BSDs have separated ports repos, but AFAIK all of them maintain > so > called base and in base there is kernel, libs, bin/sbin commands and most > importantly *all* tool-chain required to build *all* this code to form whole > OS. Yes the ports place executables in /usr/local. The kernel and base is considered a working set so the user commands under /bin /usr/bin /sbin etc match the kernel and it is tested as a whole. We will never have a single repo of all sources such as gcc etc like FreeBSD has. > IMHO this rebuild whole OS distro is discussed here, isn't it? If not, I'm > sorry > for noise here... Ah ok, the discussion is about the how we manage over a long period of time scripts, tools etc that read and write the spec files in rtems.git. This patch adds a format option to aid automation via scripting however there is no script support included so is that left as an exercise for those in the know on how to do it or should we require support being brought into rtems.git and maintained along with the file format? If there is no provided scripting support who does this patch serve or benefit and what value does it bring other than overheads in maintaining these spec files? [1] The spec files are close or the same as the YAML spec files rtems-central has and that repo has scripting support so those working with rtems-central are ok because that single repo has consistent support. The history and agreement for the qual effort is these repos remain separated and nothing has changed here. The rtems.git repo will always take precedence over rtems-central.git. In relation to a single RTEMS repo it helps make deployment easier for some use cases and for others it makes no difference. It all depends on how you want to tool up for deployment but it is not essential. Many repos or a single repo does not change what is in a release and that is where the project's efforts are focused. Chris [1] the patch in this thread was pushed while I am still questioning it (hmm) ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 20/1/2023 11:53 am, Amar Takhar wrote: > On 2023-01-20 11:03 +1100, Chris Johns wrote: > >> I have been OK with the headers and tests being generated this way because >> the >> agreement is files in rtems.git can be manually edited and rtems-central has >> to >> track those changes. The sources and code to generate the files need to be >> located together. > > Yes. To be clear in this case if we have in rtems.git and > in rtems-central.git where you need to edit to > generate then both the tool *and* somefile.yml need to be in > rtems.git. Placing the API header spec files into rtems.git complicates being able to update or add a new API by editing the header files. With the spec files to generate the headers in the same repo they would need to be updated and that means learning about the whole qual workflow, spec formats etc. This is why the separation is done this way. As things are the rtems-central could be left as is and rtems.git is not effected. We just continue to maintain the headers manually. Those interested in qualifications can fund support for rtems-central independently of the other parts of RTEMS. > If what you are saying is the requirement is reversed. rtems-central.git > needs > rtems.git to work that's fine and the order it should be. The rtems-central specs files are cleverly put together so the headers, documentation and other pieces are all handled from one place. This means all pieces would need to be brought together. The rtems-central repo is the central clearing house for the specifications. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 1/3] build: Format build items
On 17.01.23 22:52, Chris Johns wrote: On 17/1/2023 6:39 pm, Sebastian Huber wrote: On 17.01.23 03:48, Chris Johns wrote: On 16/1/2023 6:56 pm, Sebastian Huber wrote: On 16.01.23 01:35, Chris Johns wrote: On 13/1/2023 1:54 am, Sebastian Huber wrote: On 12.01.23 15:44, Kinsey Moore wrote: The other two patches look fine to me. The use of dump() that results in this patch does several things: * Removal of whitespace This is fine for whitespace at the base level of indentation. Whitespace within an indented block may be more important for readability. * Removal of comments This is not good as they are exclusively used to annotate manually ordered blocks of test result expectations * Rearrangement of items in alphabetical order In general, rearrangement of top-level sections is good. For indented sections specifically in tst*.yml, this is bad for the above reaso One goal of the new build system was to be able to alter the data through scripts. This requires that the build items are human and machine readable and writable. The Python YAML import/export does not preserve white space and comments. Can someone edit the file and add a hex number? Yes, you can manually use whatever format is understood by the YAML loader. When you write the file with the YAML dumper, then the standard representation is used. Are there python modules in rtems.git someone could import that reads and writes the YAML spec files? If not should we provide a module to do this? It could be `spec` so a user can use `import spec` after setting the import path. That is, if external scripts (and I encourage this) all used a common read and write type functionality the format consistency is relative to specific version of rtems.git being used. Updates become read then write. The Python modules to work with specification items are in rtems-central.git. This repository contains also a format specification of the build items. And how does that help here with this repo? I suggest you reconsider the accessibility of those modules before pushing scripted generation changes like this into rtems.git. It was not my idea to use multiple repositories. Do you prefer now to move everything back into rtems.git (tools, documentation)? -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
