[Bug fortran/40958] module files too large
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Francois-Xavier Coudert fxcoudert at gcc dot gnu.org changed: What|Removed |Added Status|NEW |RESOLVED CC||fxcoudert at gcc dot gnu.org Resolution|--- |FIXED --- Comment #21 from Francois-Xavier Coudert fxcoudert at gcc dot gnu.org --- I suggest we close this PR, as situation has significantly improved. Please open new PR(s) for specific issues related to module size.
[Bug fortran/40958] module files too large
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 --- Comment #19 from kargl at gcc dot gnu.org --- Author: kargl Date: Fri Jun 5 16:54:53 2015 New Revision: 224159 URL: https://gcc.gnu.org/viewcvs?rev=224159root=gccview=rev Log: 2015-06-03 Russell Whitesides russell...@gmail.com Steven G. Kargl ka...@gcc.gnu.org PR fortran/40958 PR fortran/60780 PR fortran/66377 * module.c (load_equiv): Add check for loading duplicate EQUIVALENCEs from different modules. Eliminate the pruning of unused equivalence-objects 2015-06-03 Steven G. Kargl ka...@gcc.gnu.org PR fortran/66377 gfortran.dg/equiv_9.f90: New test. Added: trunk/gcc/testsuite/gfortran.dg/equiv_9.f90 Modified: trunk/gcc/fortran/ChangeLog trunk/gcc/fortran/module.c trunk/gcc/testsuite/ChangeLog
[Bug fortran/40958] module files too large
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 --- Comment #20 from kargl at gcc dot gnu.org --- Author: kargl Date: Fri Jun 5 20:40:35 2015 New Revision: 224171 URL: https://gcc.gnu.org/viewcvs?rev=224171root=gccview=rev Log: 2015-06-03 Russell Whitesides russell...@gmail.com Steven G. Kargl ka...@gcc.gnu.org PR fortran/40958 PR fortran/60780 PR fortran/66377 * module.c (load_equiv): Add check for loading duplicate EQUIVALENCEs from different modules. Eliminate the pruning of unused equivalence-objects 2015-06-03 Steven G. Kargl ka...@gcc.gnu.org PR fortran/66377 gfortran.dg/equiv_9.f90: New test. Added: branches/gcc-5-branch/gcc/testsuite/gfortran.dg/equiv_9.f90 Modified: branches/gcc-5-branch/gcc/fortran/ChangeLog branches/gcc-5-branch/gcc/fortran/module.c branches/gcc-5-branch/gcc/testsuite/ChangeLog
[Bug fortran/40958] module files too large
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 --- Comment #18 from russelldub at gmail dot com --- Created attachment 33703 -- https://gcc.gnu.org/bugzilla/attachment.cgi?id=33703action=edit Proposed patch to fix module equivalence duplicates Here is a proposed fix for the problem related to equivalence statements in nested/recursive use (see also PR 60780). I've run 'make check-fortran' on rev. 216017 with and without this patch had a FAIL on gfortran.dg/typebound_operator_3.f03 (as discussed in PR 63404) both ways.
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Dominique d'Humieres dominiq at lps dot ens.fr changed: What|Removed |Added Status|WAITING |NEW --- Comment #17 from Dominique d'Humieres dominiq at lps dot ens.fr --- However, the fundamental(?) issue of module sizes growing exponentially with deep module hierarchies still remains. The solution to that is to not include transitive dependencies, which in turn would require a module cache for good performance. Whether that is worth doing, and who is willing and able to do it, is unclear. See pr60780 for an example.
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Dominique d'Humieres dominiq at lps dot ens.fr changed: What|Removed |Added Status|REOPENED|WAITING --- Comment #15 from Dominique d'Humieres dominiq at lps dot ens.fr --- However, the fundamental(?) issue of module sizes growing exponentially with deep module hierarchies still remains. The solution to that is to not include transitive dependencies, which in turn would require a module cache for good performance. Whether that is worth doing, and who is willing and able to do it, is unclear. Would not it be simpler to tell the users what they should do to avoid this issue? If yes, what would be the basic rules?
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Harald Anlauf anlauf at gmx dot de changed: What|Removed |Added CC||anlauf at gmx dot de --- Comment #16 from Harald Anlauf anlauf at gmx dot de --- (In reply to Dominique d'Humieres from comment #15) However, the fundamental(?) issue of module sizes growing exponentially with deep module hierarchies still remains. The solution to that is to not include transitive dependencies, which in turn would require a module cache for good performance. Whether that is worth doing, and who is willing and able to do it, is unclear. Would not it be simpler to tell the users what they should do to avoid this issue? If yes, what would be the basic rules? I doubt that this is the right answer. The user wants to write maintainable and portable code. The paradigm of object-oriented programming will more often lead to deeper module hierarchies than simple code. You'd had a hard time to tell users that gfortran requires to flatten those hierarchies when other compilers don't (assuming that the others perform acceptably).
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 --- Comment #14 from Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch --- (In reply to Janne Blomqvist from comment #13) I believe a lot of progress has been made indeed. However, the fundamental(?) issue of module sizes growing exponentially with deep module hierarchies still remains. The solution to that is to not include transitive dependencies, which in turn would require a module cache for good performance. Whether that is worth doing, and who is willing and able to do it, is unclear. note that there could also be disadvantages for that solution. For example, dependencies for a given .F would be more difficult to find (i.e. not just the USE statements). I'm not sure what implications that would have e.g. for 'smart' recompilation (i.e. based on time stamps of .mod). The module cache would also not work very well for the (common, I guess) case of having a single module per file, and all USE statements on top. It might thus be that the current state is the sweet spot.
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Dominique d'Humieres dominiq at lps dot ens.fr changed: What|Removed |Added Status|NEW |WAITING --- Comment #12 from Dominique d'Humieres dominiq at lps dot ens.fr --- Joost, is it fixed after revision 198023? If yes, could you close the PR as FIXED?
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Janne Blomqvist jb at gcc dot gnu.org changed: What|Removed |Added Status|WAITING |REOPENED --- Comment #13 from Janne Blomqvist jb at gcc dot gnu.org --- (In reply to Dominique d'Humieres from comment #12) Joost, is it fixed after revision 198023? If yes, could you close the PR as FIXED? With the introduction of the temp buffer for parsing modules the excessive lseek() calls have been eliminated, and with the introduction of compressed module files the total size of module files on disk have been reduced by an order of magnitude for large projects. However, the fundamental(?) issue of module sizes growing exponentially with deep module hierarchies still remains. The solution to that is to not include transitive dependencies, which in turn would require a module cache for good performance. Whether that is worth doing, and who is willing and able to do it, is unclear.
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Tobias Burnus burnus at gcc dot gnu.org changed: What|Removed |Added CC||burnus at gcc dot gnu.org --- Comment #9 from Tobias Burnus burnus at gcc dot gnu.org 2013-04-17 13:50:31 UTC --- Author: jb Date: Tue Mar 26 22:08:17 2013 New Revision: 197124 URL: http://gcc.gnu.org/viewcvs?rev=197124root=gccview=rev Log: PR 25708 Use a temporary buffer when parsing module files. 2013-03-27 Janne Blomqvist j...@gcc.gnu.org PR fortran/25708 * module.c (module_locus): Use long for position. (module_content): New variable. (module_pos): Likewise. (prev_character): Remove. (bad_module): Free data instead of closing mod file. (set_module_locus): Use module_pos. (get_module_locus): Likewise. (module_char): use buffer rather than stdio file. (module_unget_char): Likewise. (read_module_to_tmpbuf): New function. (gfc_use_module): Call read_module_to_tmpbuf. Modified: trunk/gcc/fortran/ChangeLog trunk/gcc/fortran/module.c
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 --- Comment #10 from Tobias Burnus burnus at gcc dot gnu.org 2013-04-17 13:50:58 UTC --- Author: jb Date: Wed Apr 17 10:19:40 2013 New Revision: 198023 URL: http://gcc.gnu.org/viewcvs?rev=198023root=gccview=rev Log: PR 40958 Compress module files with zlib. frontend ChangeLog: 2013-04-17 Janne Blomqvist j...@gcc.gnu.org PR fortran/40958 * scanner.h: New file. * Make-lang.in: Dependencies on scanner.h. * scanner.c (gfc_directorylist): Move to scanner.h. * module.c: Don't include md5.h, include scanner.h and zlib.h. (MOD_VERSION): Add comment about backwards compatibility. (module_fp): Change type to gzFile. (ctx): Remove. (gzopen_included_file_1): New function. (gzopen_included_file): New function. (gzopen_intrinsic_module): New function. (write_char): Use gzputc. (read_crc32_from_module_file): New function. (read_md5_from_module_file): Remove. (gfc_dump_module): Use gz* functions instead of stdio, check gzip crc32 instead of md5. (read_module_to_tmpbuf): Use gz* functions instead of stdio. (gfc_use_module): Use gz* functions. testsuite ChangeLog: 2013-04-17 Janne Blomqvist j...@gcc.gnu.org PR fortran/40958 * lib/gcc-dg.exp (scan-module): Uncompress module file before scanning. * gfortran.dg/module_md5_1.f90: Remove. Added: trunk/gcc/fortran/scanner.h Removed: trunk/gcc/testsuite/gfortran.dg/module_md5_1.f90 Modified: trunk/gcc/fortran/ChangeLog trunk/gcc/fortran/Make-lang.in trunk/gcc/fortran/module.c trunk/gcc/fortran/scanner.c trunk/gcc/testsuite/ChangeLog trunk/gcc/testsuite/lib/gcc-dg.exp
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 --- Comment #11 from Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch 2013-04-17 19:36:45 UTC --- With these patches in, parallel compilation of multi-file cp2k becomes significantly faster. Time for a full build goes from 70s to 50s. I think that in a parallel build the IO bottleneck (bandwidth) was significant, while this is now much improved. The effect will likely be even larger on mounted filesystems.
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Janne Blomqvist jb at gcc dot gnu.org changed: What|Removed |Added URL||http://gcc.gnu.org/ml/gcc-p ||atches/2013-04/msg00554.htm ||l CC||jb at gcc dot gnu.org --- Comment #8 from Janne Blomqvist jb at gcc dot gnu.org 2013-04-09 18:35:46 UTC --- Patch for compressing module files with zlib at http://gcc.gnu.org/ml/gcc-patches/2013-04/msg00554.html .
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch changed: What|Removed |Added CC||Joost.VandeVondele at mat ||dot ethz.ch --- Comment #7 from Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch 2013-03-29 10:17:20 UTC --- see part 2/3 in the message here: http://gcc.gnu.org/ml/fortran/2013-03/msg00125.html
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 --- Comment #6 from Thomas Koenig tkoenig at gcc dot gnu.org 2011-11-29 17:49:31 UTC --- Author: tkoenig Date: Tue Nov 29 17:49:24 2011 New Revision: 181810 URL: http://gcc.gnu.org/viewcvs?root=gccview=revrev=181810 Log: 2011-11-29 Thomas Koenig tkoe...@gcc.gnu.org PR fortran/40958 * module.c (prev_module_line): New variable. (prev_module_column): New variable. (prev_character): New variable. (module_char): Update the new variables. (module_unget_char): New function. (parse_string): Use module_unget_char. (parse_integer): Likewise. (parse_name): Likewise. Modified: trunk/gcc/fortran/ChangeLog trunk/gcc/fortran/module.c
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 --- Comment #4 from Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch 2011-11-28 14:24:02 UTC --- Just for reference, compiling CP2K_2009-05-01.f90 results in 684 modules, stracing yields something like 12000 calls to open, and 148'847'399 calls to lseek. Clearly anything reducing the number of seeks is likely to have a good impact on compile time. For this particular case, caching modules would help a lot as well. However, our usual pattern is to have a single module per file, and all use statements at the top of the module. Caching would be of little help for this style. An efficient encoding of the information in the module would help. The idea of writing the module compressed, and decompressing it as a big string to memory for reading and parsing, seems appealing to me. Concerning a change of format, it would be important to keep one of gfortran's nice features, that is, the ability to use the modification time of the .mod files to avoid recompilation cascades. If .mod files would contain a reference to other .mod files (instead of containing the info directly), this property might be at risk.
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch changed: What|Removed |Added CC||tkoenig at gcc dot gnu.org --- Comment #5 from Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch 2011-11-28 17:05:05 UTC --- (In reply to comment #4) Just for reference, compiling CP2K_2009-05-01.f90 results in 684 modules, stracing yields something like 12000 calls to open, and 148'847'399 calls to lseek. With Thomas patch (http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958) the number of seeks drops to 19'557'182, which quite an improvement. In the trace output, there are still very long sequences of identical lseek, without any other intermediate call.
[Bug fortran/40958] module files too large
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958 Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch changed: What|Removed |Added CC||blomqvist.janne at gmail ||dot com --- Comment #3 from Joost VandeVondele Joost.VandeVondele at mat dot ethz.ch 2011-11-25 17:57:06 UTC --- A link to the single file CP2K version mentioned below http://dl.dropbox.com/u/40478020/CP2K_2009-05-01.f90.gz gfortran -c CP2K_2009-05-01.f90 needs about 5m38s on my machine, 122Mb of mod files with current 4.7.
[Bug fortran/40958] module files too large
--- Comment #1 from steven at gcc dot gnu dot org 2009-08-04 09:10 --- Yup -- steven at gcc dot gnu dot org changed: What|Removed |Added Status|UNCONFIRMED |NEW Ever Confirmed|0 |1 Last reconfirmed|-00-00 00:00:00 |2009-08-04 09:10:35 date|| http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958
[Bug fortran/40958] module files too large
--- Comment #2 from jv244 at cam dot ac dot uk 2009-08-04 10:13 --- It looks like also module.c contains minit calls that need to be modified. But for a 50% savings in module size, that seems still a localized effort -- http://gcc.gnu.org/bugzilla/show_bug.cgi?id=40958
