Re: [cmake-developers] Fortran support for Ninja generator
On 03/27/2015 03:20 PM, Brad King wrote: > I've drafted the attached document in Markdown to make it easy to paste > into a Github issue or comment for discussion with Ninja upstream. > > Review on our side should be about whether the proposal is sufficient I've updated the "Automatic Module Dependencies" to split the `scan` tool execution out into its own step. The depgen file needs to be an output of a rule so it can be regenerated if it is deleted. -Brad Ninja and Fortran = Ninja has excellent support for implicit dependencies of languages that use the `C` preprocessor. Fortran adds another level of complexity for build systems because its "module" system adds implicit *outputs* to compilation nodes that are referenced as implicit dependencies of other compilation nodes. These implicit outputs and dependencies may be discovered only during the build process when all (possibly generated) files are available for a given source file to be preprocessed. Below we introduce Fortran modules and their build system requirements. Then we propose new Ninja features to support them. Fortran Modules --- The Fortran module system is defined in section 11.2 of the [Fortran 2008 Standard][ISO/IEC 1539-1:2010], also available as a [Draft PDF][]. [ISO/IEC 1539-1:2010]: http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=50459 [Draft PDF]: http://www.j3-fortran.org/doc/year/10/10-007.pdf Each Fortran source file may provide zero or more modules: $ cat provider.F90 MODULE mymod ... END and also consume zero or more modules: $ cat consumer.F90 USE MODULE mymod ... Compilation of a source file consuming a module requires the `.mod` file to be present: $ f95 -c consumer.F90 ... Fatal Error: Can't open module file 'mymod.mod' for reading... Compilation of a source file providing a module generates a `.mod` file for it: $ f95 -c provider.F90 $ ls *.o *.mod provider.o mymod.mod Once the module has been provided then it can be used: $ f95 -c consumer.F90 $ ls *.o *.mod consumer.o provider.o mymod.mod If the source providing a module is modified and recompiled, the module file *may or may not* be updated, depending on whether the interface exposed in the module changes. In general modules used by a source file may be found anywhere in a search path (e.g. `-I` paths). They may be provided by another source in the same build or may be found in external directories existing on the system. The only restriction on module dependencies is that they are acyclic. Manual Module Dependencies -- The above example source files may be built correctly by Ninja using manual specification of the module dependencies. We'll ignore preprocessor implicit dependencies for now: $ cat build.ninja rule FC command = f95 -o $OBJ -c $in restat = $RESTAT build provider.o mymod.mod: FC provider.F90 OBJ = provider.o RESTAT = 1 build consumer.o: FC consumer.F90 | mymod.mod OBJ = consumer.o We use the `OBJ` variable in `FC` because `$out` would contain the module as well as the object file. We use `restat` for the provider because `mymod.mod` may not be updated by the compiler. We can see that the above works in a test session: $ ninja -v consumer.o [1/2] f95 -o provider.o -c provider.F90 [2/2] f95 -o consumer.o -c consumer.F90 $ ninja -v consumer.o ninja: no work to do. $ touch provider.F90 $ ninja -v consumer.o [1/2] f95 -o provider.o -c provider.F90 $ ninja -v consumer.o ninja: no work to do. This example serves as a conceptual model for the basic dependency structure Fortran needs. In practice Fortran source files provide and consume many modules so maintaining the dependencies by hand is impractical. The build system must be able to discover these implicit outputs and implicit dependencies automatically. Automatic Module Dependencies - In order to scan for modules provided or consumed by a source file, it must first be preprocessed. Therefore we split compilation into three steps: 1. A `FP` rule to preprocess a source file: rule FP command = f95 -o $out -E $in -MMD -MT $out -MF $out.d deps = gcc depfile = $out.d This rule uses normal `depfile` for its implicit preprocessing dependencies. Preprocessing does not require the `.mod` files to exist. 2. A `FD` rule to scan for module dependency information and tell Ninja to load it using a `depgen` feature proposed below. rule FD command = scan $in $OBJ $out depgen = $out The user-provided `scan` tool will read the preprocessed source from `$in`, detect the implicit outputs and dependencies of the object file named by `$OBJ`, and write the build plan updates to the `$out` file also specified as
Re: [cmake-developers] Fortran support for Ninja generator
On 03/26/2015 08:53 AM, Brad King wrote: > I'm working on a write-up of the whole thing to present to Ninja > folks. Once it's drafted I'll post it here for discussion. I've drafted the attached document in Markdown to make it easy to paste into a Github issue or comment for discussion with Ninja upstream. Review on our side should be about whether the proposal is sufficient to satisfy our needs rather than about specific syntax and such. Thanks, -Brad Ninja and Fortran = Ninja has excellent support for implicit dependencies of languages that use the `C` preprocessor. Fortran adds another level of complexity for build systems because its "module" system adds implicit *outputs* to compilation nodes that are referenced as implicit dependencies of other compilation nodes. These implicit outputs and dependencies may be discovered only during the build process when all (possibly generated) files are available for a given source file to be preprocessed. Below we introduce Fortran modules and their build system requirements. Then we propose new Ninja features to support them. Fortran Modules --- The Fortran module system is defined in section 11.2 of the [Fortran 2008 Standard][ISO/IEC 1539-1:2010], also available as a [Draft PDF][]. [ISO/IEC 1539-1:2010]: http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=50459 [Draft PDF]: http://www.j3-fortran.org/doc/year/10/10-007.pdf Each Fortran source file may provide zero or more modules: $ cat provider.F90 MODULE mymod ... END and also consume zero or more modules: $ cat consumer.F90 USE MODULE mymod ... Compilation of a source file consuming a module requires the `.mod` file to be present: $ f95 -c consumer.F90 ... Fatal Error: Can't open module file 'mymod.mod' for reading... Compilation of a source file providing a module generates a `.mod` file for it: $ f95 -c provider.F90 $ ls *.o *.mod provider.o mymod.mod Once the module has been provided then it can be used: $ f95 -c consumer.F90 $ ls *.o *.mod consumer.o provider.o mymod.mod If the source providing a module is modified and recompiled, the module file *may or may not* be updated, depending on whether the interface exposed in the module changes. In general modules used by a source file may be found anywhere in a search path (e.g. `-I` paths). They may be provided by another source in the same build or may be found in external directories existing on the system. The only restriction on module dependencies is that they are acyclic. Manual Module Dependencies -- The above example source files may be built correctly by Ninja using manual specification of the module dependencies. We'll ignore preprocessor implicit dependencies for now: $ cat build.ninja rule FC command = f95 -o $OBJ -c $in restat = $RESTAT build provider.o mymod.mod: FC provider.F90 OBJ = provider.o RESTAT = 1 build consumer.o: FC consumer.F90 | mymod.mod OBJ = consumer.o We use the `OBJ` variable in `FC` because `$out` would contain the module as well as the object file. We use `restat` for the provider because `mymod.mod` may not be updated by the compiler. We can see that the above works in a test session: $ ninja -v consumer.o [1/2] f95 -o provider.o -c provider.F90 [2/2] f95 -o consumer.o -c consumer.F90 $ ninja -v consumer.o ninja: no work to do. $ touch provider.F90 $ ninja -v consumer.o [1/2] f95 -o provider.o -c provider.F90 $ ninja -v consumer.o ninja: no work to do. This example serves as a conceptual model for the basic dependency structure Fortran needs. In practice Fortran source files provide and consume many modules so maintaining the dependencies by hand is impractical. The build system must be able to discover these implicit outputs and implicit dependencies automatically. Automatic Module Dependencies - In order to scan for modules provided or consumed by a source file, it must first be preprocessed. Therefore we split compilation into two steps: 1. A `FP` rule to preprocess a source file. This can use a normal `depfile` for its implicit preprocessing dependencies. We do not need `.mod` files to exist for preprocessing. The `FP` rule will use some kind of `scan` tool to read the preprocessed output and generate dependencies for Ninja to load using a `depgen` feature proposed below. 2. A `FC` rule will compile the already-preprocessed source to produce an object file and possibly some `.mod` files. Implicit outputs and dependencies of compilation nodes using this rule must be added on the fly by reading the `depgen` file generated by preprocessing nodes using the `FP` rule. Our `build.ninja` file now contains: rule FP command = f95 -o $out -E $in -MMD -MT $out &&
Re: [cmake-developers] Fortran support for Ninja generator
On 03/25/2015 05:12 PM, Alexander Neundorf wrote: >> rule Fortran_DEPENDS_SCAN >> command = f90deps $in $out >> description = Write Fortran dependencies to $out >> >> rule Fortran_COMPILER >> depfile = $DEP_FILE >> deps = gcc >> command = /usr/bin/f95 $DEFINES $FLAGS -c $in -o $out >> description = Building Fortran object $out Thanks. I have a similar structure in mind. I think it will end up being a bit more complicated because the dependencies have to be scanned and loaded on the fly. Currently Ninja loads all depfiles up front and dependencies generated from scanning are only considered on the next invocation. We will need build-time incremental addition of dependencies and even new build nodes to represent the module files that sit between providers and consumers. I'm working on a write-up of the whole thing to present to Ninja folks. Once it's drafted I'll post it here for discussion. >> So, to do a similar thing with cmake, one would need to expose the >> cmake Fortran dependency scanner so that it can be invoked as e.g.: >> >> rule Fortran_DEPENDS_SCAN >> command = cmake scan_dependencies $in $out >> description = Write Fortran dependencies to $out >> >> but I don't know if turning the cmake executable into a build-time >> dependency scanner is feasible and/or acceptable. > > as Brad wrote, this is already the case for the makefile generator, > so I guess in general this should be Ok. Yes, the Makefile generator already uses an undocumented cmake command line tool to scan dependencies during the build. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Fortran support for Ninja generator
On Wednesday, March 25, 2015 14:37:00 Steven Vancoillie wrote: > On Mar 24 [11:21], Brad King wrote: > > On 03/24/2015 05:46 AM, Nils Gladitz wrote: > > > I am not very familiar with fortran myself but there was this discussion > > > on the ninja mailing list that implied that this might also require > > > changes to ninja itself: > > > > > > https://groups.google.com/d/msg/ninja-build/b1-AF3pRJuE/NkPDsO0C2IUJ > > > > That discussion concludes assuming that CMake scans source files > > while generating the build files to generate the ordering dependencies. > > It does not. There could be generated source files or header files > > that are needed to get the ordering right. In the Makefile generator > > we have a step to scan dependencies for a target after all its > > dependencies are finished and its custom commands have executed. > > This ensures generated files are available. Then CMake puts the > > dependency scanning results in a place used by the actual compile > > and link rules. > > > > [snip] > > > > I've been thinking about some ideas on how build.ninja rules > > could express the dynamic scanning and update we need. Rather > > than posting them now, perhaps the design process would benefit > > from any independently developed ideas you may have. > > From what I understood from the conclusion of the discussion is indeed > that it would require cmake to generate dependency files at build > time. In a test setup, I have the following: > > build CMakeFiles/testninja.dir/prog.o.d: Fortran_DEPENDS_SCAN ../prog.f90 > build CMakeFiles/testninja.dir/prog.o: Fortran_COMPILER ../prog.f90 | > CMakeFiles/testninja.dir/prog.o.d DEP_FILE = > CMakeFiles/testninja.dir/prog.o.d > FLAGS = -cpp -I../ > OBJECT_DIR = CMakeFiles/testninja.dir > OBJECT_FILE_DIR = CMakeFiles/testninja.dir > build ... > build ... > > I skipped all the other files, but the pattern is the same. > Then, there are these rules: > > rule Fortran_DEPENDS_SCAN > command = f90deps $in $out > description = Write Fortran dependencies to $out > > rule Fortran_COMPILER > depfile = $DEP_FILE > deps = gcc > command = /usr/bin/f95 $DEFINES $FLAGS -c $in -o $out > description = Building Fortran object $out > > In this case, a script 'f90deps' does the dependencies on the fly. > Note that the Fortran_COMPILER rule is now missing any -MMD/-MT stuff. > > So, to do a similar thing with cmake, one would need to expose the > cmake Fortran dependency scanner so that it can be invoked as e.g.: > > rule Fortran_DEPENDS_SCAN > command = cmake scan_dependencies $in $out > description = Write Fortran dependencies to $out > > but I don't know if turning the cmake executable into a build-time > dependency scanner is feasible and/or acceptable. as Brad wrote, this is already the case for the makefile generator, so I guess in general this should be Ok. Alex -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Fortran support for Ninja generator
On Mar 24 [11:21], Brad King wrote: > On 03/24/2015 05:46 AM, Nils Gladitz wrote: > > I am not very familiar with fortran myself but there was this discussion > > on the ninja mailing list that implied that this might also require > > changes to ninja itself: > > > > https://groups.google.com/d/msg/ninja-build/b1-AF3pRJuE/NkPDsO0C2IUJ > > That discussion concludes assuming that CMake scans source files > while generating the build files to generate the ordering dependencies. > It does not. There could be generated source files or header files > that are needed to get the ordering right. In the Makefile generator > we have a step to scan dependencies for a target after all its > dependencies are finished and its custom commands have executed. > This ensures generated files are available. Then CMake puts the > dependency scanning results in a place used by the actual compile > and link rules. > > [snip] > > I've been thinking about some ideas on how build.ninja rules > could express the dynamic scanning and update we need. Rather > than posting them now, perhaps the design process would benefit > from any independently developed ideas you may have. >From what I understood from the conclusion of the discussion is indeed that it would require cmake to generate dependency files at build time. In a test setup, I have the following: build CMakeFiles/testninja.dir/prog.o.d: Fortran_DEPENDS_SCAN ../prog.f90 build CMakeFiles/testninja.dir/prog.o: Fortran_COMPILER ../prog.f90 | CMakeFiles/testninja.dir/prog.o.d DEP_FILE = CMakeFiles/testninja.dir/prog.o.d FLAGS = -cpp -I../ OBJECT_DIR = CMakeFiles/testninja.dir OBJECT_FILE_DIR = CMakeFiles/testninja.dir build ... build ... I skipped all the other files, but the pattern is the same. Then, there are these rules: rule Fortran_DEPENDS_SCAN command = f90deps $in $out description = Write Fortran dependencies to $out rule Fortran_COMPILER depfile = $DEP_FILE deps = gcc command = /usr/bin/f95 $DEFINES $FLAGS -c $in -o $out description = Building Fortran object $out In this case, a script 'f90deps' does the dependencies on the fly. Note that the Fortran_COMPILER rule is now missing any -MMD/-MT stuff. So, to do a similar thing with cmake, one would need to expose the cmake Fortran dependency scanner so that it can be invoked as e.g.: rule Fortran_DEPENDS_SCAN command = cmake scan_dependencies $in $out description = Write Fortran dependencies to $out but I don't know if turning the cmake executable into a build-time dependency scanner is feasible and/or acceptable. Steven -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Fortran support for Ninja generator
On 03/24/2015 05:46 AM, Nils Gladitz wrote: >> One mention was to use the -M* flags supported by gfortran, just as >> with gcc. However, when I tried this approach I found out that the -M* >> flag for gfortran seems buggy, it doesn't output the correct format >> (e.g. the target given with -MT flag is just added instead of >> overwriting the default target). So, my ninja build failed saying the >> depfile format was wrong. Yes, gfortran seems to produce rules with multiple outputs and ninja expects there to be only one output. It looks like gfortran lists the module.mod files produced by a translation unit as outputs and those consumed by a translation unit as dependencies. This is valuable information for ordering compilations, but we need to figure out the right way to get ninja to use the information. In particular, we need to get the information before a rule runs. >> Alternatively someone proposed to have ninja use cmake-generated >> dependencies. > > I am not very familiar with fortran myself but there was this discussion > on the ninja mailing list that implied that this might also require > changes to ninja itself: > > https://groups.google.com/d/msg/ninja-build/b1-AF3pRJuE/NkPDsO0C2IUJ That discussion concludes assuming that CMake scans source files while generating the build files to generate the ordering dependencies. It does not. There could be generated source files or header files that are needed to get the ordering right. In the Makefile generator we have a step to scan dependencies for a target after all its dependencies are finished and its custom commands have executed. This ensures generated files are available. Then CMake puts the dependency scanning results in a place used by the actual compile and link rules. There is absolutely no way to do this without some kind of change to ninja because its build plan graph must be updated dynamically to add Fortran module dependencies as they are discovered. AFAIK the only place ninja has for dynamically updating the build plan is the restat=1 option. IIUC the depfile information is loaded only at the beginning of the build. The "deps=gcc" option causes it to read a depfile immediately after a rule produces it, but I'm not sure whether it only transfers the information to the .ninja_deps file or updates its build graph on the fly. In all of these cases the build plan updates affecting a rule only occur *after* it runs. We cannot pre-generate a depfile because there is not enough information to do so until during the build. I've been thinking about some ideas on how build.ninja rules could express the dynamic scanning and update we need. Rather than posting them now, perhaps the design process would benefit from any independently developed ideas you may have. Thanks, -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Fortran support for Ninja generator
On 3/24/2015 5:46 AM, Nils Gladitz wrote: I am not very familiar with fortran myself but there was this discussion on the ninja mailing list that implied that this might also require changes to ninja itself: https://groups.google.com/d/msg/ninja-build/b1-AF3pRJuE/NkPDsO0C2IUJ Yes, the problem is that to build fortran 95 you have to parse the fortran first and figure out what order to build it, and then do the build. ninja is setup to load the depend tree once when it starts. It would have to be modified to be able to load the depend information dynamically. -Bill -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Fortran support for Ninja generator
On 03/24/2015 10:08 AM, Steven Vancoillie wrote: Dear all, we have a large Fortran project which we build with cmake and I'm interested in making the ninja generator work with Fortran. I found a few short discussions of this on the mailing list, but since then nothing has happened. One mention was to use the -M* flags supported by gfortran, just as with gcc. However, when I tried this approach I found out that the -M* flag for gfortran seems buggy, it doesn't output the correct format (e.g. the target given with -MT flag is just added instead of overwriting the default target). So, my ninja build failed saying the depfile format was wrong. Alternatively someone proposed to have ninja use cmake-generated dependencies. From what I can tell from the ninja documentation, would this mean that the cmake executable needs to generate the depfile instead of the compiler? I'm willing to work on this, but so far my experience with the cmake source code is very rudimentary. If someone can point me in the right direction, or provide some input to get started, this would be greatly appreciated. I am not very familiar with fortran myself but there was this discussion on the ninja mailing list that implied that this might also require changes to ninja itself: https://groups.google.com/d/msg/ninja-build/b1-AF3pRJuE/NkPDsO0C2IUJ Nils -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
[cmake-developers] Fortran support for Ninja generator
Dear all, we have a large Fortran project which we build with cmake and I'm interested in making the ninja generator work with Fortran. I found a few short discussions of this on the mailing list, but since then nothing has happened. One mention was to use the -M* flags supported by gfortran, just as with gcc. However, when I tried this approach I found out that the -M* flag for gfortran seems buggy, it doesn't output the correct format (e.g. the target given with -MT flag is just added instead of overwriting the default target). So, my ninja build failed saying the depfile format was wrong. Alternatively someone proposed to have ninja use cmake-generated dependencies. From what I can tell from the ninja documentation, would this mean that the cmake executable needs to generate the depfile instead of the compiler? I'm willing to work on this, but so far my experience with the cmake source code is very rudimentary. If someone can point me in the right direction, or provide some input to get started, this would be greatly appreciated. Steven -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers