Re: [PATCH] wscript: Add post-build processing support
On 5/3/21 6:29 am, Sebastian Huber wrote: > On 04/03/2021 20:15, Kinsey Moore wrote: > >> -Original Message- >> From: Sebastian Huber >> Sent: Thursday, March 4, 2021 12:57 >> To: Kinsey Moore ; devel@rtems.org >> Subject: Re: [PATCH] wscript: Add post-build processing support >> >>> On 04/03/2021 17:18, Kinsey Moore wrote: >>> >>>> This allows packaging of the compiled binaries to occur automatically >>>> and be bundled with other scripts and build items as convenient. >>> This is not how a build system should work. When an item changes it is >>> the job of the build system to automatically rebuild all its dependents. >>> >>> I think this issue was already discussed with Joel and Chris. The >>> discussed approach is to add an optional post-build script and run this >>> script in a post-link step. >> I thought this might be the case. I'll try to find some time to rework it >> into a more proper patch. > I discussed this with Chis in November last year. We did. Seems like it was yesterday. A > Unfortunately we didn't add a ticket for this. I recently raise a ticket as a GSoC Open project ... https://devel.rtems.org/ticket/4272 > The general idea was to let BSPs add an optional post-link > script which is installed as /bin/post-link. For the test > executables, please have a look at: > > https://waf.io/blog/2016/09/waf-strip-binaries.html Please do not add any post processing to the rtems.git build. Let me explain ... 1. The previous approach in the autotool build system performed a range of post link steps and most of the builds done by developers never ran so that whole step was a nop. The step consumed an ever growing amount of time and disk space and as a result the post link step was turned off. 2. The autotool build system post-link solution was available to only those users who built with make and used the Makefile.inc support. Today we have a number of build systems so this is no longer a viable solution. A command line tool can be integrated into any build system and we need to maintain it as a public user interface across releases. 3. The post link steps are the domain of users and their set up. It is not possible for RTEMS to capture and included everything users need to do. For example the previous post-link processing generated a map file and I have no idea why when we have an ELF executable image at hand. There are a number of tools that can create a map file from it. 4. The rtems-test command has the `target_pretest_command` hook that is used to convert the ELF executable into a suitable format for loading on to the target. We should be educating and encouraging our user to on-demand convert. 5. The post-processing can only depend on the RTEMS ecosystem support provided. For example `mkimage` is a Linux and u-boot command not available on all hosts. We do have the `mkimage.py` script so that could be used. Yes this creates some sort term pain but the long term benefit is important. What I would like to see happen is: a) A script added to each BSP that needs post processing. b) A reference added to the BSP's YAML spec support to indicate there is a post-processing script. c) Have waf build the script into the BSP build tree applying some form of yet to be defined filtering. For example a base address or some other setting the script and BSP depend on. I have no idea how this could be implemented and if it is possible but the few places we handle some of these settings the better. For example if a BSP config changes a setting the build and a script uses we need to see the change in the script as well. d) Have waf install the script into the BSP specific area under the $PREFIX/$ARCH-rtems$RTEMS_VERSION/$BSP/ path. e) Provide and install a command to $PREFIX/bin with a formal name such rtems-post-link that finds and runs the BSP installed script. This implies an interface between the BSP script and this command. The BSP scripts can provide extensions but there needs to be a core level of functionality provided. An example of an extension is creating a second image type such as the bootloader for the motorola_powerpc BSP. f) Support all major hosts we support. Again please do not load up the build of rtems with post processing that is not used most of the time. Post process on demand when you need it. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH] wscript: Add post-build processing support
On 04/03/2021 20:15, Kinsey Moore wrote: -Original Message- From: Sebastian Huber Sent: Thursday, March 4, 2021 12:57 To: Kinsey Moore ; devel@rtems.org Subject: Re: [PATCH] wscript: Add post-build processing support On 04/03/2021 17:18, Kinsey Moore wrote: This allows packaging of the compiled binaries to occur automatically and be bundled with other scripts and build items as convenient. This is not how a build system should work. When an item changes it is the job of the build system to automatically rebuild all its dependents. I think this issue was already discussed with Joel and Chris. The discussed approach is to add an optional post-build script and run this script in a post-link step. I thought this might be the case. I'll try to find some time to rework it into a more proper patch. I discussed this with Chis in November last year. Unfortunately we didn't add a ticket for this. The general idea was to let BSPs add an optional post-link script which is installed as /bin/post-link. For the test executables, please have a look at: https://waf.io/blog/2016/09/waf-strip-binaries.html -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
RE: [PATCH] wscript: Add post-build processing support
-Original Message- From: Sebastian Huber Sent: Thursday, March 4, 2021 12:57 To: Kinsey Moore ; devel@rtems.org Subject: Re: [PATCH] wscript: Add post-build processing support >On 04/03/2021 17:18, Kinsey Moore wrote: > >> This allows packaging of the compiled binaries to occur automatically >> and be bundled with other scripts and build items as convenient. > > This is not how a build system should work. When an item changes it is > the job of the build system to automatically rebuild all its dependents. > > I think this issue was already discussed with Joel and Chris. The > discussed approach is to add an optional post-build script and run this > script in a post-link step. I thought this might be the case. I'll try to find some time to rework it into a more proper patch. Thanks, Kinsey ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH] wscript: Add post-build processing support
On Thu, Mar 4, 2021 at 11:57 AM Sebastian Huber wrote: > > On 04/03/2021 17:18, Kinsey Moore wrote: > > > This allows packaging of the compiled binaries to occur automatically > > and be bundled with other scripts and build items as convenient. > > This is not how a build system should work. When an item changes it is > the job of the build system to automatically rebuild all its dependents. > > I think this issue was already discussed with Joel and Chris. The > discussed approach is to add an optional post-build script and run this > script in a post-link step. > Should it be integrated as a waf command? ./waf post-link or something or rolled up in install? It is a kind of "installation" step to prepare a binary for loading maybe. > -- > embedded brains GmbH > Herr Sebastian HUBER > Dornierstr. 4 > 82178 Puchheim > Germany > email: sebastian.hu...@embedded-brains.de > phone: +49-89-18 94 741 - 16 > fax: +49-89-18 94 741 - 08 > > Registergericht: Amtsgericht München > Registernummer: HRB 157899 > Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler > Unsere Datenschutzerklärung finden Sie hier: > https://embedded-brains.de/datenschutzerklaerung/ > > ___ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH] wscript: Add post-build processing support
On 04/03/2021 17:18, Kinsey Moore wrote: This allows packaging of the compiled binaries to occur automatically and be bundled with other scripts and build items as convenient. This is not how a build system should work. When an item changes it is the job of the build system to automatically rebuild all its dependents. I think this issue was already discussed with Joel and Chris. The discussed approach is to add an optional post-build script and run this script in a post-link step. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
RE: [PATCH] wscript: Add post-build processing support
Hi Gedare, -Original Message- From: Gedare Bloom Sent: Thursday, March 4, 2021 11:21 To: Kinsey Moore Cc: devel@rtems.org Subject: Re: [PATCH] wscript: Add post-build processing support > Hi Kinsey, > > On Thu, Mar 4, 2021 at 9:32 AM Kinsey Moore wrote: >> >> Currently, I’m using it to perform automated u-boot packaging of all the >> test executables in my development branch and CI versus having to manually >> generate these images or managing external scripts to do the automated >> packaging. I'm on the fence as to whether I submit that portion for merging >> since it adds extra dependencies on the u-boot tools and doesn't/can't build >> images for non-test applications. I thought this might be useful to have in >> the public repo but if not that's fine, too. >> > > This sounds a lot like the old bsp-post-link target we had. Has that > gone away in the new build system, and is this a suggestion to replace > it? That's the kind of functionality I was going for. I didn't see a way to accomplish this in the waf build system, hence the patch. It's entirely possible I just missed the functionality somewhere, though. > As a side note, I don't intend to ask questions to discourage merging, > I just want clarification and often the historical reasons to include > things are important later on for others who want to dig back to > figure out why something exists. Understood, no problem. Kinsey ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH] wscript: Add post-build processing support
Hi Kinsey, On Thu, Mar 4, 2021 at 9:32 AM Kinsey Moore wrote: > > Currently, I’m using it to perform automated u-boot packaging of all the test > executables in my development branch and CI versus having to manually > generate these images or managing external scripts to do the automated > packaging. I'm on the fence as to whether I submit that portion for merging > since it adds extra dependencies on the u-boot tools and doesn't/can't build > images for non-test applications. I thought this might be useful to have in > the public repo but if not that's fine, too. > This sounds a lot like the old bsp-post-link target we had. Has that gone away in the new build system, and is this a suggestion to replace it? As a side note, I don't intend to ask questions to discourage merging, I just want clarification and often the historical reasons to include things are important later on for others who want to dig back to figure out why something exists. > Kinsey > > -Original Message- > From: Gedare Bloom > Sent: Thursday, March 4, 2021 10:25 > To: Kinsey Moore > Cc: devel@rtems.org > Subject: Re: [PATCH] wscript: Add post-build processing support > > Can you explain this a bit more? What is the need this solves, and how > will it be used? > > On Thu, Mar 4, 2021 at 9:18 AM Kinsey Moore wrote: > > > > This allows packaging of the compiled binaries to occur automatically > > and be bundled with other scripts and build items as convenient. > > --- > > wscript | 22 ++ > > 1 file changed, 22 insertions(+) > > > > diff --git a/wscript b/wscript > > index 6626fafb74..e263681716 100755 > > --- a/wscript > > +++ b/wscript > > @@ -223,6 +223,12 @@ class Item(object): > > p.build(bld, bic) > > self.do_build(bld, bic) > > > > +def post_build(self, bld, bic): > > +if _is_enabled(bld.env.ENABLE, self.get_enabled_by()): > > +for p in self.links(): > > +p.post_build(bld, bic) > > +self.do_post_build(bld, bic) > > + > > def do_defaults(self, variant): > > return > > > > @@ -238,6 +244,9 @@ class Item(object): > > def do_build(self, bld, bic): > > return > > > > +def do_post_build(self, bld, bic): > > +return > > + > > def substitute(self, ctx, value): > > if isinstance(value, str): > > try: > > @@ -1074,6 +1083,15 @@ class ScriptItem(Item): > > if script: > > exec(script) > > > > +def do_post_build(self, bld, bic): > > +script = False > > +try: > > +script = self.data["do-post-build"] > > +except KeyError: > > +pass > > +if script: > > +exec(script) > > + > > > > class ConfigItemContext(object): > > def __init__(self, cp, path_list): > > @@ -1485,6 +1503,10 @@ def build(bld): > > bsps[bld.env.ARCH][bld.env.BSP_BASE].build(bld, bic) > > items[bld.env.TOPGROUP].build(bld, bic) > > > > +def post_build(bld): > > +bsps[bld.env.ARCH][bld.env.BSP_BASE].post_build(bld, bic) > > +bld.add_post_fun(post_build) > > + > > > > def add_log_filter(name): > > msg = "'" + name + "' finished successfully" > > -- > > 2.20.1 > > > > ___ > > devel mailing list > > devel@rtems.org > > http://lists.rtems.org/mailman/listinfo/devel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
RE: [PATCH] wscript: Add post-build processing support
Currently, I’m using it to perform automated u-boot packaging of all the test executables in my development branch and CI versus having to manually generate these images or managing external scripts to do the automated packaging. I'm on the fence as to whether I submit that portion for merging since it adds extra dependencies on the u-boot tools and doesn't/can't build images for non-test applications. I thought this might be useful to have in the public repo but if not that's fine, too. Kinsey -Original Message- From: Gedare Bloom Sent: Thursday, March 4, 2021 10:25 To: Kinsey Moore Cc: devel@rtems.org Subject: Re: [PATCH] wscript: Add post-build processing support Can you explain this a bit more? What is the need this solves, and how will it be used? On Thu, Mar 4, 2021 at 9:18 AM Kinsey Moore wrote: > > This allows packaging of the compiled binaries to occur automatically > and be bundled with other scripts and build items as convenient. > --- > wscript | 22 ++ > 1 file changed, 22 insertions(+) > > diff --git a/wscript b/wscript > index 6626fafb74..e263681716 100755 > --- a/wscript > +++ b/wscript > @@ -223,6 +223,12 @@ class Item(object): > p.build(bld, bic) > self.do_build(bld, bic) > > +def post_build(self, bld, bic): > +if _is_enabled(bld.env.ENABLE, self.get_enabled_by()): > +for p in self.links(): > +p.post_build(bld, bic) > +self.do_post_build(bld, bic) > + > def do_defaults(self, variant): > return > > @@ -238,6 +244,9 @@ class Item(object): > def do_build(self, bld, bic): > return > > +def do_post_build(self, bld, bic): > +return > + > def substitute(self, ctx, value): > if isinstance(value, str): > try: > @@ -1074,6 +1083,15 @@ class ScriptItem(Item): > if script: > exec(script) > > +def do_post_build(self, bld, bic): > +script = False > +try: > +script = self.data["do-post-build"] > +except KeyError: > +pass > +if script: > +exec(script) > + > > class ConfigItemContext(object): > def __init__(self, cp, path_list): > @@ -1485,6 +1503,10 @@ def build(bld): > bsps[bld.env.ARCH][bld.env.BSP_BASE].build(bld, bic) > items[bld.env.TOPGROUP].build(bld, bic) > > +def post_build(bld): > +bsps[bld.env.ARCH][bld.env.BSP_BASE].post_build(bld, bic) > +bld.add_post_fun(post_build) > + > > def add_log_filter(name): > msg = "'" + name + "' finished successfully" > -- > 2.20.1 > > ___ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH] wscript: Add post-build processing support
Can you explain this a bit more? What is the need this solves, and how will it be used? On Thu, Mar 4, 2021 at 9:18 AM Kinsey Moore wrote: > > This allows packaging of the compiled binaries to occur automatically > and be bundled with other scripts and build items as convenient. > --- > wscript | 22 ++ > 1 file changed, 22 insertions(+) > > diff --git a/wscript b/wscript > index 6626fafb74..e263681716 100755 > --- a/wscript > +++ b/wscript > @@ -223,6 +223,12 @@ class Item(object): > p.build(bld, bic) > self.do_build(bld, bic) > > +def post_build(self, bld, bic): > +if _is_enabled(bld.env.ENABLE, self.get_enabled_by()): > +for p in self.links(): > +p.post_build(bld, bic) > +self.do_post_build(bld, bic) > + > def do_defaults(self, variant): > return > > @@ -238,6 +244,9 @@ class Item(object): > def do_build(self, bld, bic): > return > > +def do_post_build(self, bld, bic): > +return > + > def substitute(self, ctx, value): > if isinstance(value, str): > try: > @@ -1074,6 +1083,15 @@ class ScriptItem(Item): > if script: > exec(script) > > +def do_post_build(self, bld, bic): > +script = False > +try: > +script = self.data["do-post-build"] > +except KeyError: > +pass > +if script: > +exec(script) > + > > class ConfigItemContext(object): > def __init__(self, cp, path_list): > @@ -1485,6 +1503,10 @@ def build(bld): > bsps[bld.env.ARCH][bld.env.BSP_BASE].build(bld, bic) > items[bld.env.TOPGROUP].build(bld, bic) > > +def post_build(bld): > +bsps[bld.env.ARCH][bld.env.BSP_BASE].post_build(bld, bic) > +bld.add_post_fun(post_build) > + > > def add_log_filter(name): > msg = "'" + name + "' finished successfully" > -- > 2.20.1 > > ___ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
[PATCH] wscript: Add post-build processing support
This allows packaging of the compiled binaries to occur automatically and be bundled with other scripts and build items as convenient. --- wscript | 22 ++ 1 file changed, 22 insertions(+) diff --git a/wscript b/wscript index 6626fafb74..e263681716 100755 --- a/wscript +++ b/wscript @@ -223,6 +223,12 @@ class Item(object): p.build(bld, bic) self.do_build(bld, bic) +def post_build(self, bld, bic): +if _is_enabled(bld.env.ENABLE, self.get_enabled_by()): +for p in self.links(): +p.post_build(bld, bic) +self.do_post_build(bld, bic) + def do_defaults(self, variant): return @@ -238,6 +244,9 @@ class Item(object): def do_build(self, bld, bic): return +def do_post_build(self, bld, bic): +return + def substitute(self, ctx, value): if isinstance(value, str): try: @@ -1074,6 +1083,15 @@ class ScriptItem(Item): if script: exec(script) +def do_post_build(self, bld, bic): +script = False +try: +script = self.data["do-post-build"] +except KeyError: +pass +if script: +exec(script) + class ConfigItemContext(object): def __init__(self, cp, path_list): @@ -1485,6 +1503,10 @@ def build(bld): bsps[bld.env.ARCH][bld.env.BSP_BASE].build(bld, bic) items[bld.env.TOPGROUP].build(bld, bic) +def post_build(bld): +bsps[bld.env.ARCH][bld.env.BSP_BASE].post_build(bld, bic) +bld.add_post_fun(post_build) + def add_log_filter(name): msg = "'" + name + "' finished successfully" -- 2.20.1 ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
