Re: [cmake-developers] Fortran support for Ninja generator
On 03/26/2015 08:53 AM, Brad King wrote: I'm working on a write-up of the whole thing to present to Ninja folks. Once it's drafted I'll post it here for discussion. I've drafted the attached document in Markdown to make it easy to paste into a Github issue or comment for discussion with Ninja upstream. Review on our side should be about whether the proposal is sufficient to satisfy our needs rather than about specific syntax and such. Thanks, -Brad Ninja and Fortran = Ninja has excellent support for implicit dependencies of languages that use the `C` preprocessor. Fortran adds another level of complexity for build systems because its module system adds implicit *outputs* to compilation nodes that are referenced as implicit dependencies of other compilation nodes. These implicit outputs and dependencies may be discovered only during the build process when all (possibly generated) files are available for a given source file to be preprocessed. Below we introduce Fortran modules and their build system requirements. Then we propose new Ninja features to support them. Fortran Modules --- The Fortran module system is defined in section 11.2 of the [Fortran 2008 Standard][ISO/IEC 1539-1:2010], also available as a [Draft PDF][]. [ISO/IEC 1539-1:2010]: http://www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=50459 [Draft PDF]: http://www.j3-fortran.org/doc/year/10/10-007.pdf Each Fortran source file may provide zero or more modules: $ cat provider.F90 MODULE mymod ... END and also consume zero or more modules: $ cat consumer.F90 USE MODULE mymod ... Compilation of a source file consuming a module requires the `.mod` file to be present: $ f95 -c consumer.F90 ... Fatal Error: Can't open module file 'mymod.mod' for reading... Compilation of a source file providing a module generates a `.mod` file for it: $ f95 -c provider.F90 $ ls *.o *.mod provider.o mymod.mod Once the module has been provided then it can be used: $ f95 -c consumer.F90 $ ls *.o *.mod consumer.o provider.o mymod.mod If the source providing a module is modified and recompiled, the module file *may or may not* be updated, depending on whether the interface exposed in the module changes. In general modules used by a source file may be found anywhere in a search path (e.g. `-I` paths). They may be provided by another source in the same build or may be found in external directories existing on the system. The only restriction on module dependencies is that they are acyclic. Manual Module Dependencies -- The above example source files may be built correctly by Ninja using manual specification of the module dependencies. We'll ignore preprocessor implicit dependencies for now: $ cat build.ninja rule FC command = f95 -o $OBJ -c $in restat = $RESTAT build provider.o mymod.mod: FC provider.F90 OBJ = provider.o RESTAT = 1 build consumer.o: FC consumer.F90 | mymod.mod OBJ = consumer.o We use the `OBJ` variable in `FC` because `$out` would contain the module as well as the object file. We use `restat` for the provider because `mymod.mod` may not be updated by the compiler. We can see that the above works in a test session: $ ninja -v consumer.o [1/2] f95 -o provider.o -c provider.F90 [2/2] f95 -o consumer.o -c consumer.F90 $ ninja -v consumer.o ninja: no work to do. $ touch provider.F90 $ ninja -v consumer.o [1/2] f95 -o provider.o -c provider.F90 $ ninja -v consumer.o ninja: no work to do. This example serves as a conceptual model for the basic dependency structure Fortran needs. In practice Fortran source files provide and consume many modules so maintaining the dependencies by hand is impractical. The build system must be able to discover these implicit outputs and implicit dependencies automatically. Automatic Module Dependencies - In order to scan for modules provided or consumed by a source file, it must first be preprocessed. Therefore we split compilation into two steps: 1. A `FP` rule to preprocess a source file. This can use a normal `depfile` for its implicit preprocessing dependencies. We do not need `.mod` files to exist for preprocessing. The `FP` rule will use some kind of `scan` tool to read the preprocessed output and generate dependencies for Ninja to load using a `depgen` feature proposed below. 2. A `FC` rule will compile the already-preprocessed source to produce an object file and possibly some `.mod` files. Implicit outputs and dependencies of compilation nodes using this rule must be added on the fly by reading the `depgen` file generated by preprocessing nodes using the `FP` rule. Our `build.ninja` file now contains: rule FP command = f95 -o $out -E $in -MMD -MT $out scan
Re: [cmake-developers] Fortran support for Ninja generator
On 03/24/2015 10:08 AM, Steven Vancoillie wrote: Dear all, we have a large Fortran project which we build with cmake and I'm interested in making the ninja generator work with Fortran. I found a few short discussions of this on the mailing list, but since then nothing has happened. One mention was to use the -M* flags supported by gfortran, just as with gcc. However, when I tried this approach I found out that the -M* flag for gfortran seems buggy, it doesn't output the correct format (e.g. the target given with -MT flag is just added instead of overwriting the default target). So, my ninja build failed saying the depfile format was wrong. Alternatively someone proposed to have ninja use cmake-generated dependencies. From what I can tell from the ninja documentation, would this mean that the cmake executable needs to generate the depfile instead of the compiler? I'm willing to work on this, but so far my experience with the cmake source code is very rudimentary. If someone can point me in the right direction, or provide some input to get started, this would be greatly appreciated. I am not very familiar with fortran myself but there was this discussion on the ninja mailing list that implied that this might also require changes to ninja itself: https://groups.google.com/d/msg/ninja-build/b1-AF3pRJuE/NkPDsO0C2IUJ Nils -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Fortran support for Ninja generator
On 03/24/2015 05:46 AM, Nils Gladitz wrote: One mention was to use the -M* flags supported by gfortran, just as with gcc. However, when I tried this approach I found out that the -M* flag for gfortran seems buggy, it doesn't output the correct format (e.g. the target given with -MT flag is just added instead of overwriting the default target). So, my ninja build failed saying the depfile format was wrong. Yes, gfortran seems to produce rules with multiple outputs and ninja expects there to be only one output. It looks like gfortran lists the module.mod files produced by a translation unit as outputs and those consumed by a translation unit as dependencies. This is valuable information for ordering compilations, but we need to figure out the right way to get ninja to use the information. In particular, we need to get the information before a rule runs. Alternatively someone proposed to have ninja use cmake-generated dependencies. I am not very familiar with fortran myself but there was this discussion on the ninja mailing list that implied that this might also require changes to ninja itself: https://groups.google.com/d/msg/ninja-build/b1-AF3pRJuE/NkPDsO0C2IUJ That discussion concludes assuming that CMake scans source files while generating the build files to generate the ordering dependencies. It does not. There could be generated source files or header files that are needed to get the ordering right. In the Makefile generator we have a step to scan dependencies for a target after all its dependencies are finished and its custom commands have executed. This ensures generated files are available. Then CMake puts the dependency scanning results in a place used by the actual compile and link rules. There is absolutely no way to do this without some kind of change to ninja because its build plan graph must be updated dynamically to add Fortran module dependencies as they are discovered. AFAIK the only place ninja has for dynamically updating the build plan is the restat=1 option. IIUC the depfile information is loaded only at the beginning of the build. The deps=gcc option causes it to read a depfile immediately after a rule produces it, but I'm not sure whether it only transfers the information to the .ninja_deps file or updates its build graph on the fly. In all of these cases the build plan updates affecting a rule only occur *after* it runs. We cannot pre-generate a depfile because there is not enough information to do so until during the build. I've been thinking about some ideas on how build.ninja rules could express the dynamic scanning and update we need. Rather than posting them now, perhaps the design process would benefit from any independently developed ideas you may have. Thanks, -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
[cmake-developers] Fortran support for Ninja generator
Dear all, we have a large Fortran project which we build with cmake and I'm interested in making the ninja generator work with Fortran. I found a few short discussions of this on the mailing list, but since then nothing has happened. One mention was to use the -M* flags supported by gfortran, just as with gcc. However, when I tried this approach I found out that the -M* flag for gfortran seems buggy, it doesn't output the correct format (e.g. the target given with -MT flag is just added instead of overwriting the default target). So, my ninja build failed saying the depfile format was wrong. Alternatively someone proposed to have ninja use cmake-generated dependencies. From what I can tell from the ninja documentation, would this mean that the cmake executable needs to generate the depfile instead of the compiler? I'm willing to work on this, but so far my experience with the cmake source code is very rudimentary. If someone can point me in the right direction, or provide some input to get started, this would be greatly appreciated. Steven -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Fortran support for Ninja generator
On 3/24/2015 5:46 AM, Nils Gladitz wrote: I am not very familiar with fortran myself but there was this discussion on the ninja mailing list that implied that this might also require changes to ninja itself: https://groups.google.com/d/msg/ninja-build/b1-AF3pRJuE/NkPDsO0C2IUJ Yes, the problem is that to build fortran 95 you have to parse the fortran first and figure out what order to build it, and then do the build. ninja is setup to load the depend tree once when it starts. It would have to be modified to be able to load the depend information dynamically. -Bill -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
