Re: Automated gbuild -> SCons convertibility now at 88.57%
Am 05.07.20 um 08:00 schrieb Damjan Jovanovic: I think there is are some more tools that we need to examine before we can drop Cygwin. http://www.openoffice.org/tools/build_env_tools.html That list is missing idlc, if not others. And what does that mean? Those tools are built by the "Executable" targets which we already support. With the changes I just pushed, AllLangResTarget from main/uui built fine in "cmd" on my first try (with Python and SCons installed outside Cygwin), though I do want to check that the paths passed to rsc.exe are correct. My MSVC is broken, so I can't test the binary targets. Do you need the MSVC Installation media? We can share that for reinstall if you like. All the best Peter - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Automated gbuild -> SCons convertibility now at 88.57%
On Sun, Jul 5, 2020 at 7:27 AM Peter wrote: > Awesome Damjan, > > Thank you :) > I had a talk with SCons on freenode IRC. they liked the plan to move to > SCons a lot. > > And did support us right a way. :) So I am poretty confidant that SCons > will do the trick. > > How did you talk to SCon people? > > That's great. We talked on the scons-us...@scons.org mailing list. > > I think there is are some more tools that we need to examine before we > can drop Cygwin. > > http://www.openoffice.org/tools/build_env_tools.html > > That list is missing idlc, if not others. And what does that mean? Those tools are built by the "Executable" targets which we already support. With the changes I just pushed, AllLangResTarget from main/uui built fine in "cmd" on my first try (with Python and SCons installed outside Cygwin), though I do want to check that the paths passed to rsc.exe are correct. My MSVC is broken, so I can't test the binary targets. > > I have to look at the SCons. I plan to build GSICheck also with SCons, > even if it is a total simple tool. > > Ok great. > > All the best. > > Peter > > You too Damjan > > Am 04.07.20 um 20:44 schrieb Damjan Jovanovic: > > Hi > > > > In the scons-build branch, I've just pushed a set of 11 commits that > > theoretically get 93 out of 105 gbuild modules (88.57%) automatically > > converting to gbuild. > > > > The "gotoSCons" converter and the SCons infrastructure in that branch > have > > now been developed to such a level that a module can be automatically > > converted from gbuild to SCons, from where it can use SCons for all of > the > > following: > > Building C/C++ objects > > Linking shared libraries, static libraries, and executables > > Building JUnit tests and running them > > Building Google Tests and running them > > Building .component files with XSLT > > Running Ant sub-builds > > Delivering "package" files such as headers > > Even doing the impossibly difficult 5 step "AllLangResTarget" (.src -> > > merged .src -> .srs -> .res for each language). > > > > I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, > but > > I think only Jar and Zip are worth implementing automatic conversion for, > > as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget > in > > only 2 places, which makes manual conversion for them easier. The hardest > > conversions are already done. > > > > Where does this leave us? > > > > The gotoSCons converter can't support a number of features, like > > non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, > etc. A > > module can only be automatically converted if it doesn't use the > > unsupported features. Currently, 35 modules use only supported features, > > and can be converted automatically (this should increase to 39 modules > when > > I add Jar and Zip conversion). > > > > Another 58 modules use non-deterministic constructs or custom make rules. > > Converting those 58 could be done through a semi-automated process, which > > involves editing the gbuild files to remove the unsupported features, > > running the automated conversion on what is left, then manually patching > > what was removed into the conversion results. Sometimes this is quick and > > easy, at other times probably not. > > > > The final 12 modules use unsupported targets requiring a longer and > mostly > > manual conversion to SCons, though even there the supported targets could > > be converted automatically. > > > > The SCons infrastructure does require some cleanup, as I was learning > while > > developing, and we still need library naming conversions, tests on > > Linux/WIndows/Mac, etc. > > > > The more I've used SCons, the more I've liked it. I've even started using > > it in my own projects at work now. I've found a way to solve every > problem > > I've encountered, and the SCons developers have been helpful when I asked > > them questions. Complex functionality like header dependency scanning, > > automatic directory creation for output files, using @responsefile for > long > > command lines when necessary, and other features gbuild implements > > manually, all work in SCons automatically. In 1816 lines of code, our > SCons > > infrastructure implements what took gbuild 9418 lines, and SCons is far > > more readable and maintainable (even in its current messy state). > > > > The plan isn't to merge this to trunk any time soon. Rather the idea is > to > > develop the converter even further, then when it's complete enough, > convert > > as many gbuild modules to SCons. Then measure performance of building > those > > modules en-masse with SCons alone - if there are performance problems at > > that stage, they are only going to get worse with more modules. > > > > The real test however is converting the other 78 dmake modules to SCons. > 37 > > of them are 3rd party "externals" like jpeg and zlib, which have their > own > > build systems that we just call, so conversion is relatively easy. The > >
Re: Automated gbuild -> SCons convertibility now at 88.57%
Awesome Damjan, I had a talk with SCons on freenode IRC. they liked the plan to move to SCons a lot. And did support us right a way. :) So I am poretty confidant that SCons will do the trick. How did you talk to SCon people? I think there is are some more tools that we need to examine before we can drop Cygwin. http://www.openoffice.org/tools/build_env_tools.html I have to look at the SCons. I plan to build GSICheck also with SCons, even if it is a total simple tool. All the best. Peter Am 04.07.20 um 20:44 schrieb Damjan Jovanovic: Hi In the scons-build branch, I've just pushed a set of 11 commits that theoretically get 93 out of 105 gbuild modules (88.57%) automatically converting to gbuild. The "gotoSCons" converter and the SCons infrastructure in that branch have now been developed to such a level that a module can be automatically converted from gbuild to SCons, from where it can use SCons for all of the following: Building C/C++ objects Linking shared libraries, static libraries, and executables Building JUnit tests and running them Building Google Tests and running them Building .component files with XSLT Running Ant sub-builds Delivering "package" files such as headers Even doing the impossibly difficult 5 step "AllLangResTarget" (.src -> merged .src -> .srs -> .res for each language). I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, but I think only Jar and Zip are worth implementing automatic conversion for, as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget in only 2 places, which makes manual conversion for them easier. The hardest conversions are already done. Where does this leave us? The gotoSCons converter can't support a number of features, like non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, etc. A module can only be automatically converted if it doesn't use the unsupported features. Currently, 35 modules use only supported features, and can be converted automatically (this should increase to 39 modules when I add Jar and Zip conversion). Another 58 modules use non-deterministic constructs or custom make rules. Converting those 58 could be done through a semi-automated process, which involves editing the gbuild files to remove the unsupported features, running the automated conversion on what is left, then manually patching what was removed into the conversion results. Sometimes this is quick and easy, at other times probably not. The final 12 modules use unsupported targets requiring a longer and mostly manual conversion to SCons, though even there the supported targets could be converted automatically. The SCons infrastructure does require some cleanup, as I was learning while developing, and we still need library naming conversions, tests on Linux/WIndows/Mac, etc. The more I've used SCons, the more I've liked it. I've even started using it in my own projects at work now. I've found a way to solve every problem I've encountered, and the SCons developers have been helpful when I asked them questions. Complex functionality like header dependency scanning, automatic directory creation for output files, using @responsefile for long command lines when necessary, and other features gbuild implements manually, all work in SCons automatically. In 1816 lines of code, our SCons infrastructure implements what took gbuild 9418 lines, and SCons is far more readable and maintainable (even in its current messy state). The plan isn't to merge this to trunk any time soon. Rather the idea is to develop the converter even further, then when it's complete enough, convert as many gbuild modules to SCons. Then measure performance of building those modules en-masse with SCons alone - if there are performance problems at that stage, they are only going to get worse with more modules. The real test however is converting the other 78 dmake modules to SCons. 37 of them are 3rd party "externals" like jpeg and zlib, which have their own build systems that we just call, so conversion is relatively easy. The other 41 modules are hard to convert, but gbuild is one of the reasons that they were hard, and where SCons is expected to make the greatest difference. Only once we are building without dmake, without gbuild, without build.pl, without Cygwin, on all platforms, then it would be the right time to merge to trunk. If at some stage in this process we are unhappy with SCons, and some better build system can be found, it shouldn't be difficult to change to it. The converter could output files for that other build system instead; at present it's only 498 lines of code in 1 file, that are involved in writing SCons build files, all we would need is a similar file for that other build system. Build systems are not the most exciting part of development, but bad build systems make development painful, and as a large multi-platform project, we build a lot. We answer build-related questions on the mailing lists too often,
Re: Automated gbuild -> SCons convertibility now at 88.57%
On 7/4/2020 12:53 PM, Damjan Jovanovic wrote: scripting is a dmake module with no symbols package is a gbuild module with symbols So yes, as we move off dmake, more and more modules should have working debug symbols. I have suddenly become a convert to moving off dmake. This is the case on *nix too, where dmake doesn't provide full paths to filenames, breaking debugging. On Sat, Jul 4, 2020 at 9:35 PM Patricia Shanahan wrote: On 7/4/2020 12:24 PM, Damjan Jovanovic wrote: Given how I've only developed on FreeBSD so far, anything Windows is probably at negative infinity :) > Do you have some example modules with that symbols problem, or at least know whether they are gbuild or dmake modules? c:\OpenOfficeDev\openoffice\main\scripting\source\protocolhandler\scripthandler.cxx does not have symbols. c:\OpenOfficeDev\openoffice\main\package\source\xstor\xstorage.cxx does have symbols generated. On Sat, Jul 4, 2020 at 9:13 PM Patricia Shanahan wrote: In the new build system, what is the status of automatic symbol generation, needed for easy debug. It is badly broken in 4.1.7, with most modules not getting symbols generated despite --enable-symbols in the configure parameters. This has cost me weeks of work on a debug project. On 7/4/2020 11:44 AM, Damjan Jovanovic wrote: Hi In the scons-build branch, I've just pushed a set of 11 commits that theoretically get 93 out of 105 gbuild modules (88.57%) automatically converting to gbuild. The "gotoSCons" converter and the SCons infrastructure in that branch have now been developed to such a level that a module can be automatically converted from gbuild to SCons, from where it can use SCons for all of the following: Building C/C++ objects Linking shared libraries, static libraries, and executables Building JUnit tests and running them Building Google Tests and running them Building .component files with XSLT Running Ant sub-builds Delivering "package" files such as headers Even doing the impossibly difficult 5 step "AllLangResTarget" (.src -> merged .src -> .srs -> .res for each language). I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, but I think only Jar and Zip are worth implementing automatic conversion for, as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget in only 2 places, which makes manual conversion for them easier. The hardest conversions are already done. Where does this leave us? The gotoSCons converter can't support a number of features, like non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, etc. A module can only be automatically converted if it doesn't use the unsupported features. Currently, 35 modules use only supported features, and can be converted automatically (this should increase to 39 modules when I add Jar and Zip conversion). Another 58 modules use non-deterministic constructs or custom make rules. Converting those 58 could be done through a semi-automated process, which involves editing the gbuild files to remove the unsupported features, running the automated conversion on what is left, then manually patching what was removed into the conversion results. Sometimes this is quick and easy, at other times probably not. The final 12 modules use unsupported targets requiring a longer and mostly manual conversion to SCons, though even there the supported targets could be converted automatically. The SCons infrastructure does require some cleanup, as I was learning while developing, and we still need library naming conversions, tests on Linux/WIndows/Mac, etc. The more I've used SCons, the more I've liked it. I've even started using it in my own projects at work now. I've found a way to solve every problem I've encountered, and the SCons developers have been helpful when I asked them questions. Complex functionality like header dependency scanning, automatic directory creation for output files, using @responsefile for long command lines when necessary, and other features gbuild implements manually, all work in SCons automatically. In 1816 lines of code, our SCons infrastructure implements what took gbuild 9418 lines, and SCons is far more readable and maintainable (even in its current messy state). The plan isn't to merge this to trunk any time soon. Rather the idea is to develop the converter even further, then when it's complete enough, convert as many gbuild modules to SCons. Then measure performance of building those modules en-masse with SCons alone - if there are performance problems at that stage, they are only going to get worse with more modules. The real test however is converting the other 78 dmake modules to SCons. 37 of them are 3rd party "externals" like jpeg and zlib, which have their own build systems that we just call, so conversion is relatively easy. The other 41 modules are hard to convert, but gbuild is one of the reasons that they were hard, and where SCons is expected
Re: Automated gbuild -> SCons convertibility now at 88.57%
scripting is a dmake module with no symbols package is a gbuild module with symbols So yes, as we move off dmake, more and more modules should have working debug symbols. This is the case on *nix too, where dmake doesn't provide full paths to filenames, breaking debugging. On Sat, Jul 4, 2020 at 9:35 PM Patricia Shanahan wrote: > On 7/4/2020 12:24 PM, Damjan Jovanovic wrote: > > Given how I've only developed on FreeBSD so far, anything Windows is > > probably at negative infinity :) > > > Do you have some example modules with that symbols problem, or at least > > know whether they are gbuild or dmake modules? > > c:\OpenOfficeDev\openoffice\main\scripting\source\protocolhandler\scripthandler.cxx > > does not have symbols. > > c:\OpenOfficeDev\openoffice\main\package\source\xstor\xstorage.cxx does > have symbols generated. > > > > On Sat, Jul 4, 2020 at 9:13 PM Patricia Shanahan wrote: > > > >> In the new build system, what is the status of automatic symbol > >> generation, needed for easy debug. > >> > >> It is badly broken in 4.1.7, with most modules not getting symbols > >> generated despite --enable-symbols in the configure parameters. This has > >> cost me weeks of work on a debug project. > >> > >> On 7/4/2020 11:44 AM, Damjan Jovanovic wrote: > >>> Hi > >>> > >>> In the scons-build branch, I've just pushed a set of 11 commits that > >>> theoretically get 93 out of 105 gbuild modules (88.57%) automatically > >>> converting to gbuild. > >>> > >>> The "gotoSCons" converter and the SCons infrastructure in that branch > >> have > >>> now been developed to such a level that a module can be automatically > >>> converted from gbuild to SCons, from where it can use SCons for all of > >> the > >>> following: > >>> Building C/C++ objects > >>> Linking shared libraries, static libraries, and executables > >>> Building JUnit tests and running them > >>> Building Google Tests and running them > >>> Building .component files with XSLT > >>> Running Ant sub-builds > >>> Delivering "package" files such as headers > >>> Even doing the impossibly difficult 5 step "AllLangResTarget" (.src -> > >>> merged .src -> .srs -> .res for each language). > >>> > >>> I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, > >> but > >>> I think only Jar and Zip are worth implementing automatic conversion > for, > >>> as SdiTarget and UnoApi are only used in 5 places each, and > WinResTarget > >> in > >>> only 2 places, which makes manual conversion for them easier. The > hardest > >>> conversions are already done. > >>> > >>> Where does this leave us? > >>> > >>> The gotoSCons converter can't support a number of features, like > >>> non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, > >> etc. A > >>> module can only be automatically converted if it doesn't use the > >>> unsupported features. Currently, 35 modules use only supported > features, > >>> and can be converted automatically (this should increase to 39 modules > >> when > >>> I add Jar and Zip conversion). > >>> > >>> Another 58 modules use non-deterministic constructs or custom make > rules. > >>> Converting those 58 could be done through a semi-automated process, > which > >>> involves editing the gbuild files to remove the unsupported features, > >>> running the automated conversion on what is left, then manually > patching > >>> what was removed into the conversion results. Sometimes this is quick > and > >>> easy, at other times probably not. > >>> > >>> The final 12 modules use unsupported targets requiring a longer and > >> mostly > >>> manual conversion to SCons, though even there the supported targets > could > >>> be converted automatically. > >>> > >>> The SCons infrastructure does require some cleanup, as I was learning > >> while > >>> developing, and we still need library naming conversions, tests on > >>> Linux/WIndows/Mac, etc. > >>> > >>> The more I've used SCons, the more I've liked it. I've even started > using > >>> it in my own projects at work now. I've found a way to solve every > >> problem > >>> I've encountered, and the SCons developers have been helpful when I > asked > >>> them questions. Complex functionality like header dependency scanning, > >>> automatic directory creation for output files, using @responsefile for > >> long > >>> command lines when necessary, and other features gbuild implements > >>> manually, all work in SCons automatically. In 1816 lines of code, our > >> SCons > >>> infrastructure implements what took gbuild 9418 lines, and SCons is far > >>> more readable and maintainable (even in its current messy state). > >>> > >>> The plan isn't to merge this to trunk any time soon. Rather the idea is > >> to > >>> develop the converter even further, then when it's complete enough, > >> convert > >>> as many gbuild modules to SCons. Then measure performance of building > >> those > >>> modules en-masse with SCons alone - if there are performance problems > at > >>> that stage, they are only
Re: Automated gbuild -> SCons convertibility now at 88.57%
On 7/4/2020 12:24 PM, Damjan Jovanovic wrote: Given how I've only developed on FreeBSD so far, anything Windows is probably at negative infinity :) > Do you have some example modules with that symbols problem, or at least know whether they are gbuild or dmake modules? c:\OpenOfficeDev\openoffice\main\scripting\source\protocolhandler\scripthandler.cxx does not have symbols. c:\OpenOfficeDev\openoffice\main\package\source\xstor\xstorage.cxx does have symbols generated. On Sat, Jul 4, 2020 at 9:13 PM Patricia Shanahan wrote: In the new build system, what is the status of automatic symbol generation, needed for easy debug. It is badly broken in 4.1.7, with most modules not getting symbols generated despite --enable-symbols in the configure parameters. This has cost me weeks of work on a debug project. On 7/4/2020 11:44 AM, Damjan Jovanovic wrote: Hi In the scons-build branch, I've just pushed a set of 11 commits that theoretically get 93 out of 105 gbuild modules (88.57%) automatically converting to gbuild. The "gotoSCons" converter and the SCons infrastructure in that branch have now been developed to such a level that a module can be automatically converted from gbuild to SCons, from where it can use SCons for all of the following: Building C/C++ objects Linking shared libraries, static libraries, and executables Building JUnit tests and running them Building Google Tests and running them Building .component files with XSLT Running Ant sub-builds Delivering "package" files such as headers Even doing the impossibly difficult 5 step "AllLangResTarget" (.src -> merged .src -> .srs -> .res for each language). I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, but I think only Jar and Zip are worth implementing automatic conversion for, as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget in only 2 places, which makes manual conversion for them easier. The hardest conversions are already done. Where does this leave us? The gotoSCons converter can't support a number of features, like non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, etc. A module can only be automatically converted if it doesn't use the unsupported features. Currently, 35 modules use only supported features, and can be converted automatically (this should increase to 39 modules when I add Jar and Zip conversion). Another 58 modules use non-deterministic constructs or custom make rules. Converting those 58 could be done through a semi-automated process, which involves editing the gbuild files to remove the unsupported features, running the automated conversion on what is left, then manually patching what was removed into the conversion results. Sometimes this is quick and easy, at other times probably not. The final 12 modules use unsupported targets requiring a longer and mostly manual conversion to SCons, though even there the supported targets could be converted automatically. The SCons infrastructure does require some cleanup, as I was learning while developing, and we still need library naming conversions, tests on Linux/WIndows/Mac, etc. The more I've used SCons, the more I've liked it. I've even started using it in my own projects at work now. I've found a way to solve every problem I've encountered, and the SCons developers have been helpful when I asked them questions. Complex functionality like header dependency scanning, automatic directory creation for output files, using @responsefile for long command lines when necessary, and other features gbuild implements manually, all work in SCons automatically. In 1816 lines of code, our SCons infrastructure implements what took gbuild 9418 lines, and SCons is far more readable and maintainable (even in its current messy state). The plan isn't to merge this to trunk any time soon. Rather the idea is to develop the converter even further, then when it's complete enough, convert as many gbuild modules to SCons. Then measure performance of building those modules en-masse with SCons alone - if there are performance problems at that stage, they are only going to get worse with more modules. The real test however is converting the other 78 dmake modules to SCons. 37 of them are 3rd party "externals" like jpeg and zlib, which have their own build systems that we just call, so conversion is relatively easy. The other 41 modules are hard to convert, but gbuild is one of the reasons that they were hard, and where SCons is expected to make the greatest difference. Only once we are building without dmake, without gbuild, without build.pl, without Cygwin, on all platforms, then it would be the right time to merge to trunk. If at some stage in this process we are unhappy with SCons, and some better build system can be found, it shouldn't be difficult to change to it. The converter could output files for that other build system instead; at present it's only 498 lines of code in 1 file,
Re: Automated gbuild -> SCons convertibility now at 88.57%
Given how I've only developed on FreeBSD so far, anything Windows is probably at negative infinity :). Do you have some example modules with that symbols problem, or at least know whether they are gbuild or dmake modules? On Sat, Jul 4, 2020 at 9:13 PM Patricia Shanahan wrote: > In the new build system, what is the status of automatic symbol > generation, needed for easy debug. > > It is badly broken in 4.1.7, with most modules not getting symbols > generated despite --enable-symbols in the configure parameters. This has > cost me weeks of work on a debug project. > > On 7/4/2020 11:44 AM, Damjan Jovanovic wrote: > > Hi > > > > In the scons-build branch, I've just pushed a set of 11 commits that > > theoretically get 93 out of 105 gbuild modules (88.57%) automatically > > converting to gbuild. > > > > The "gotoSCons" converter and the SCons infrastructure in that branch > have > > now been developed to such a level that a module can be automatically > > converted from gbuild to SCons, from where it can use SCons for all of > the > > following: > > Building C/C++ objects > > Linking shared libraries, static libraries, and executables > > Building JUnit tests and running them > > Building Google Tests and running them > > Building .component files with XSLT > > Running Ant sub-builds > > Delivering "package" files such as headers > > Even doing the impossibly difficult 5 step "AllLangResTarget" (.src -> > > merged .src -> .srs -> .res for each language). > > > > I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, > but > > I think only Jar and Zip are worth implementing automatic conversion for, > > as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget > in > > only 2 places, which makes manual conversion for them easier. The hardest > > conversions are already done. > > > > Where does this leave us? > > > > The gotoSCons converter can't support a number of features, like > > non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, > etc. A > > module can only be automatically converted if it doesn't use the > > unsupported features. Currently, 35 modules use only supported features, > > and can be converted automatically (this should increase to 39 modules > when > > I add Jar and Zip conversion). > > > > Another 58 modules use non-deterministic constructs or custom make rules. > > Converting those 58 could be done through a semi-automated process, which > > involves editing the gbuild files to remove the unsupported features, > > running the automated conversion on what is left, then manually patching > > what was removed into the conversion results. Sometimes this is quick and > > easy, at other times probably not. > > > > The final 12 modules use unsupported targets requiring a longer and > mostly > > manual conversion to SCons, though even there the supported targets could > > be converted automatically. > > > > The SCons infrastructure does require some cleanup, as I was learning > while > > developing, and we still need library naming conversions, tests on > > Linux/WIndows/Mac, etc. > > > > The more I've used SCons, the more I've liked it. I've even started using > > it in my own projects at work now. I've found a way to solve every > problem > > I've encountered, and the SCons developers have been helpful when I asked > > them questions. Complex functionality like header dependency scanning, > > automatic directory creation for output files, using @responsefile for > long > > command lines when necessary, and other features gbuild implements > > manually, all work in SCons automatically. In 1816 lines of code, our > SCons > > infrastructure implements what took gbuild 9418 lines, and SCons is far > > more readable and maintainable (even in its current messy state). > > > > The plan isn't to merge this to trunk any time soon. Rather the idea is > to > > develop the converter even further, then when it's complete enough, > convert > > as many gbuild modules to SCons. Then measure performance of building > those > > modules en-masse with SCons alone - if there are performance problems at > > that stage, they are only going to get worse with more modules. > > > > The real test however is converting the other 78 dmake modules to SCons. > 37 > > of them are 3rd party "externals" like jpeg and zlib, which have their > own > > build systems that we just call, so conversion is relatively easy. The > > other 41 modules are hard to convert, but gbuild is one of the reasons > that > > they were hard, and where SCons is expected to make the greatest > > difference. Only once we are building without dmake, without gbuild, > > without build.pl, without Cygwin, on all platforms, then it would be the > > right time to merge to trunk. > > > > If at some stage in this process we are unhappy with SCons, and some > better > > build system can be found, it shouldn't be difficult to change to it. The > > converter could output files for that other build system
Re: Automated gbuild -> SCons convertibility now at 88.57%
In the new build system, what is the status of automatic symbol generation, needed for easy debug. It is badly broken in 4.1.7, with most modules not getting symbols generated despite --enable-symbols in the configure parameters. This has cost me weeks of work on a debug project. On 7/4/2020 11:44 AM, Damjan Jovanovic wrote: Hi In the scons-build branch, I've just pushed a set of 11 commits that theoretically get 93 out of 105 gbuild modules (88.57%) automatically converting to gbuild. The "gotoSCons" converter and the SCons infrastructure in that branch have now been developed to such a level that a module can be automatically converted from gbuild to SCons, from where it can use SCons for all of the following: Building C/C++ objects Linking shared libraries, static libraries, and executables Building JUnit tests and running them Building Google Tests and running them Building .component files with XSLT Running Ant sub-builds Delivering "package" files such as headers Even doing the impossibly difficult 5 step "AllLangResTarget" (.src -> merged .src -> .srs -> .res for each language). I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, but I think only Jar and Zip are worth implementing automatic conversion for, as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget in only 2 places, which makes manual conversion for them easier. The hardest conversions are already done. Where does this leave us? The gotoSCons converter can't support a number of features, like non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, etc. A module can only be automatically converted if it doesn't use the unsupported features. Currently, 35 modules use only supported features, and can be converted automatically (this should increase to 39 modules when I add Jar and Zip conversion). Another 58 modules use non-deterministic constructs or custom make rules. Converting those 58 could be done through a semi-automated process, which involves editing the gbuild files to remove the unsupported features, running the automated conversion on what is left, then manually patching what was removed into the conversion results. Sometimes this is quick and easy, at other times probably not. The final 12 modules use unsupported targets requiring a longer and mostly manual conversion to SCons, though even there the supported targets could be converted automatically. The SCons infrastructure does require some cleanup, as I was learning while developing, and we still need library naming conversions, tests on Linux/WIndows/Mac, etc. The more I've used SCons, the more I've liked it. I've even started using it in my own projects at work now. I've found a way to solve every problem I've encountered, and the SCons developers have been helpful when I asked them questions. Complex functionality like header dependency scanning, automatic directory creation for output files, using @responsefile for long command lines when necessary, and other features gbuild implements manually, all work in SCons automatically. In 1816 lines of code, our SCons infrastructure implements what took gbuild 9418 lines, and SCons is far more readable and maintainable (even in its current messy state). The plan isn't to merge this to trunk any time soon. Rather the idea is to develop the converter even further, then when it's complete enough, convert as many gbuild modules to SCons. Then measure performance of building those modules en-masse with SCons alone - if there are performance problems at that stage, they are only going to get worse with more modules. The real test however is converting the other 78 dmake modules to SCons. 37 of them are 3rd party "externals" like jpeg and zlib, which have their own build systems that we just call, so conversion is relatively easy. The other 41 modules are hard to convert, but gbuild is one of the reasons that they were hard, and where SCons is expected to make the greatest difference. Only once we are building without dmake, without gbuild, without build.pl, without Cygwin, on all platforms, then it would be the right time to merge to trunk. If at some stage in this process we are unhappy with SCons, and some better build system can be found, it shouldn't be difficult to change to it. The converter could output files for that other build system instead; at present it's only 498 lines of code in 1 file, that are involved in writing SCons build files, all we would need is a similar file for that other build system. Build systems are not the most exciting part of development, but bad build systems make development painful, and as a large multi-platform project, we build a lot. We answer build-related questions on the mailing lists too often, and new contributors are put off by the current build system. A lot of what we want to do with AOO, such as ports to Win64, AArch64, newer MSVC versions, and so on, also involve build-related changes. Damjan -- This email has
Automated gbuild -> SCons convertibility now at 88.57%
Hi In the scons-build branch, I've just pushed a set of 11 commits that theoretically get 93 out of 105 gbuild modules (88.57%) automatically converting to gbuild. The "gotoSCons" converter and the SCons infrastructure in that branch have now been developed to such a level that a module can be automatically converted from gbuild to SCons, from where it can use SCons for all of the following: Building C/C++ objects Linking shared libraries, static libraries, and executables Building JUnit tests and running them Building Google Tests and running them Building .component files with XSLT Running Ant sub-builds Delivering "package" files such as headers Even doing the impossibly difficult 5 step "AllLangResTarget" (.src -> merged .src -> .srs -> .res for each language). I still have to implement Jar, Zip, UnoApi, WinResTarget and SdiTarget, but I think only Jar and Zip are worth implementing automatic conversion for, as SdiTarget and UnoApi are only used in 5 places each, and WinResTarget in only 2 places, which makes manual conversion for them easier. The hardest conversions are already done. Where does this leave us? The gotoSCons converter can't support a number of features, like non-deterministic constructs (ifeq ($(GUI),UNX)), custom make rules, etc. A module can only be automatically converted if it doesn't use the unsupported features. Currently, 35 modules use only supported features, and can be converted automatically (this should increase to 39 modules when I add Jar and Zip conversion). Another 58 modules use non-deterministic constructs or custom make rules. Converting those 58 could be done through a semi-automated process, which involves editing the gbuild files to remove the unsupported features, running the automated conversion on what is left, then manually patching what was removed into the conversion results. Sometimes this is quick and easy, at other times probably not. The final 12 modules use unsupported targets requiring a longer and mostly manual conversion to SCons, though even there the supported targets could be converted automatically. The SCons infrastructure does require some cleanup, as I was learning while developing, and we still need library naming conversions, tests on Linux/WIndows/Mac, etc. The more I've used SCons, the more I've liked it. I've even started using it in my own projects at work now. I've found a way to solve every problem I've encountered, and the SCons developers have been helpful when I asked them questions. Complex functionality like header dependency scanning, automatic directory creation for output files, using @responsefile for long command lines when necessary, and other features gbuild implements manually, all work in SCons automatically. In 1816 lines of code, our SCons infrastructure implements what took gbuild 9418 lines, and SCons is far more readable and maintainable (even in its current messy state). The plan isn't to merge this to trunk any time soon. Rather the idea is to develop the converter even further, then when it's complete enough, convert as many gbuild modules to SCons. Then measure performance of building those modules en-masse with SCons alone - if there are performance problems at that stage, they are only going to get worse with more modules. The real test however is converting the other 78 dmake modules to SCons. 37 of them are 3rd party "externals" like jpeg and zlib, which have their own build systems that we just call, so conversion is relatively easy. The other 41 modules are hard to convert, but gbuild is one of the reasons that they were hard, and where SCons is expected to make the greatest difference. Only once we are building without dmake, without gbuild, without build.pl, without Cygwin, on all platforms, then it would be the right time to merge to trunk. If at some stage in this process we are unhappy with SCons, and some better build system can be found, it shouldn't be difficult to change to it. The converter could output files for that other build system instead; at present it's only 498 lines of code in 1 file, that are involved in writing SCons build files, all we would need is a similar file for that other build system. Build systems are not the most exciting part of development, but bad build systems make development painful, and as a large multi-platform project, we build a lot. We answer build-related questions on the mailing lists too often, and new contributors are put off by the current build system. A lot of what we want to do with AOO, such as ports to Win64, AArch64, newer MSVC versions, and so on, also involve build-related changes. Damjan
