[fpc-devel] Messages overhead
IMO compiler messages slow down compilation a lot. When messages are not really output (prevented by message level...), then a lot of procedure calls and string operations can be avoided, when the message preparation is bypassed at all. verbose.Message1 ff. could check the level (and exit accordingly) prior to any further work. MayBeLoadMessageFile could be moved into Msg2Comment in most cases, and again a quick check (or inline) can prevent calling this subroutine at all. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op Wed, 14 Jul 2010, schreef Hans-Peter Diettrich: IMO compiler messages slow down compilation a lot. How do you know this, did you benchmark or is it just your opinion? When messages are not really output (prevented by message level...), then a lot of procedure calls and string operations can be avoided, when the message preparation is bypassed at all. verbose.Message1 ff. could check the level (and exit accordingly) prior to any further work. MayBeLoadMessageFile could be moved into Msg2Comment in most cases, and again a quick check (or inline) can prevent calling this subroutine at all. Sounds reasonable to me, but wether it will speed up compilation is another question. Anyway, simply write a patch, could be a nice excercise to get started with compiler development. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Daniël Mantione schrieb: IMO compiler messages slow down compilation a lot. How do you know this, did you benchmark or is it just your opinion? Common knowledge, proved by experience. [...] Sounds reasonable to me, but wether it will speed up compilation is another question. Anyway, simply write a patch, It's not a matter of an "simple" patch :-( The message ID has to be re-encoded in a way, that allows to determine e.g. the message level immediately. All this currently is done only after text substitution, by inspection of the returned string. could be a nice excercise to get started with compiler development. How is message encoding and composition related to compiler development? DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Hans-Peter Diettrich wrote: Daniël Mantione schrieb: IMO compiler messages slow down compilation a lot. How do you know this, did you benchmark or is it just your opinion? Common knowledge, proved by experience. I had benchmarked that, and submitted some patching some (already long) time ago. Those patches inserted calls to CheckVerbosity before blocks of messages of rarely used verbosity, like V_Tried. This way it was possible to get rid of majority of rarely seen messages, while struggling with more common verbosities seems impractical (those are few anyway). Message processing indeed includes some overhead, but this is not #1 bottleneck to worry about. Maybe #10 or so. Most processing is done in shortstrings, which is fast. Regards, Sergei ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Hans-Peter Diettrich schrieb: > Daniël Mantione schrieb: > >>> IMO compiler messages slow down compilation a lot. >> >> How do you know this, did you benchmark or is it just your opinion? > > Common knowledge, proved by experience. Please profile the compiler first, people did this already. > The message ID has to be re-encoded in a way, that allows to determine > e.g. the message level immediately. All this currently is done only > after text substitution, by inspection of the returned string. This cannot be done because people should have the chance to use custom error files to override message verbosities. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op 2010-07-14 17:17, Daniël Mantione het geskryf: >> IMO compiler messages slow down compilation a lot. > > How do you know this, did you benchmark or is it just your opinion? A few weeks ago, I benchmarked that too (well similar). I did a comparison between Kylix 3 and FPC 2.4.x. When I disabled all compiler message output, FPC compiled the test apps a bit faster that with -va parameter. It wasn't an exhaustive test by any means, but there was a difference. So I would imagine Hans has a point, that removing any calls to message output would reduce compilation even more. Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op 2010-07-15 06:26, Sergei Gorelkin het geskryf: > > Message processing indeed includes some overhead, but this is not #1 > bottleneck to worry about. > Maybe #10 or so. Well, any optimization is better than none. Irrespective of where you rank it, one has to start somewhere. Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Graeme Geldenhuys wrote on Thu, 15 Jul 2010: Well, any optimization is better than none. That's only true if it does not impact clarity or maintainability. Otherwise there is a tradeoff to be made in favour of one or the other. Jonas This message was sent using IMP, the Internet Messaging Program. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op 2010-07-15 14:18, Jonas Maebe het geskryf: > That's only true if it does not impact clarity or maintainability. > Otherwise there is a tradeoff to be made in favour of one or the other. True, but I don't think we are talking about optimization like changing Object Pascal code to ASM. The original poster (as far as I can tell) simply meant that checking for verbosity or whatever before generating message output. So message calls are only generated/used when really needed. This seems like a simple enough optimization without impacting clarity. Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
On Thu, 15 Jul 2010, Graeme Geldenhuys wrote: Op 2010-07-14 17:17, Daniël Mantione het geskryf: IMO compiler messages slow down compilation a lot. How do you know this, did you benchmark or is it just your opinion? A few weeks ago, I benchmarked that too (well similar). I did a comparison between Kylix 3 and FPC 2.4.x. When I disabled all compiler message output, FPC compiled the test apps a bit faster that with -va parameter. It wasn't an exhaustive test by any means, but there was a difference. So I would imagine Hans has a point, that removing any calls to message output would reduce compilation even more. I think that it is more the actual syscalls to write info which make the difference than the preparation of the syscall. If you wrote output to the terminal, then that is an additional slowing-down factor. The only way to test the impact of message preparation is to uncomment the actual write call, compile with -va and then compile again (clean) with -v0. Michael.___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
On Thu, 15 Jul 2010, Michael Van Canneyt wrote: On Thu, 15 Jul 2010, Graeme Geldenhuys wrote: Op 2010-07-14 17:17, Daniël Mantione het geskryf: IMO compiler messages slow down compilation a lot. How do you know this, did you benchmark or is it just your opinion? A few weeks ago, I benchmarked that too (well similar). I did a comparison between Kylix 3 and FPC 2.4.x. When I disabled all compiler message output, FPC compiled the test apps a bit faster that with -va parameter. It wasn't an exhaustive test by any means, but there was a difference. So I would imagine Hans has a point, that removing any calls to message output would reduce compilation even more. I think that it is more the actual syscalls to write info which make the difference than the preparation of the syscall. If you wrote output to the terminal, then that is an additional slowing-down factor. The only way to test the impact of message preparation is to uncomment the That should of course be 'comment out'... Michael. actual write call, compile with -va and then compile again (clean) with -v0. Michael.___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Jonas Maebe schrieb: > Graeme Geldenhuys wrote on Thu, 15 Jul 2010: > >> Well, any optimization is better than none. And before doing a lot of guess work: it is already known that e.g. fillchar is a bottleneck (probably not only for the compiler). So better mail the fastcode coders if we get permission to integrated the fastcode fillchar routines into fpc. This is not that hard, just look how this is down for move. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Florian Klaempfl schrieb: IMO compiler messages slow down compilation a lot. How do you know this, did you benchmark or is it just your opinion? Common knowledge, proved by experience. Please profile the compiler first, people did this already. This is what I'm going to do now. Does there exist some profiling code already, or do I have to reinvent the wheel? The message ID has to be re-encoded in a way, that allows to determine e.g. the message level immediately. All this currently is done only after text substitution, by inspection of the returned string. This cannot be done because people should have the chance to use custom error files to override message verbosities. Couldn't this be delegated to an custom message handler? DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Sergei Gorelkin schrieb: I had benchmarked that, and submitted some patching some (already long) time ago. Those patches inserted calls to CheckVerbosity before blocks of messages of rarely used verbosity, like V_Tried. But the patches have been rejected? For what reason? This way it was possible to get rid of majority of rarely seen messages, while struggling with more common verbosities seems impractical (those are few anyway). That's just what I also had in mind. Message processing indeed includes some overhead, but this is not #1 bottleneck to worry about. Maybe #10 or so. Most processing is done in shortstrings, which is fast. Perhaps I'm too much concentrated on an general parser. In an compiler (and for OPL) the parser overhead may be not so high. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op Thu, 15 Jul 2010, schreef Hans-Peter Diettrich: This is what I'm going to do now. Does there exist some profiling code already, or do I have to reinvent the wheel? No, simply use the -pg compiler option, the use gprof. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
On Thu, July 15, 2010 14:06, Graeme Geldenhuys wrote: > Op 2010-07-15 06:26, Sergei Gorelkin het geskryf: >> >> Message processing indeed includes some overhead, but this is not #1 >> bottleneck to worry about. >> Maybe #10 or so. > > Well, any optimization is better than none. Irrespective of where you rank > it, one has to start somewhere. Have I misunderstood something, or is this optimization really just about the fact that message loading is not necessary if the compiler is requested to output no messages at all? Even if this (probably extremely rare!) case happens, the difference must be completely negligible unless you perform very many compilations of simple and short source files without further dependencies (which doesn't sound like a typical use case to me ;-) ), right? Tomas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
On Thu, July 15, 2010 15:57, Hans-Peter Diettrich wrote: > Florian Klaempfl schrieb: . . >>> The message ID has to be re-encoded in a way, that allows to determine >>> e.g. the message level immediately. All this currently is done only >>> after text substitution, by inspection of the returned string. >> >> This cannot be done because people should have the chance to use custom >> error files to override message verbosities. > > Couldn't this be delegated to an custom message handler? Not sure if I understand here. Do you mean that instead of changing few letters in the message file such people would need to write their own message handler and recompile the compiler for using it? Tomas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Hans-Peter Diettrich wrote: But the patches have been rejected? For what reason? They have been applied in revision 9297. Regards, Sergei ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Tomas Hajny пишет: On Thu, July 15, 2010 14:06, Graeme Geldenhuys wrote: Op 2010-07-15 06:26, Sergei Gorelkin het geskryf: Message processing indeed includes some overhead, but this is not #1 bottleneck to worry about. Maybe #10 or so. Well, any optimization is better than none. Irrespective of where you rank it, one has to start somewhere. Have I misunderstood something, or is this optimization really just about the fact that message loading is not necessary if the compiler is requested to output no messages at all? Even if this (probably extremely rare!) case happens, the difference must be completely negligible unless you perform very many compilations of simple and short source files without further dependencies (which doesn't sound like a typical use case to me ;-) ), right? The issue was that, whenever compiler needs to output a message, it: - loads a messagefile (once in a session) - looks up the message by number - performs the parameter substitution (this involves AnsiStrings, so is somewhat heavy) - only then checks if it should really print the resulting string. The most amount of messages come from unit search; the system units are loaded every time, so you always have about several thousands messages loaded and discarded. This was taking a noticeable amount of executed CPU instructions (profiled with Valgrind). With the patch applied in r9297, I was able to cut the total number of executed instructions down by 20%, but that gave no increase in perceived speed of compilation. So I decided not to put much more effort to this issue. Although, modifying the messaging system in a way when CheckVerbosity() is called as soon as possible (before parameter substitution and other processing) could be beneficial. Regards, Sergei ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Hans-Peter Diettrich schrieb: > Florian Klaempfl schrieb: > > IMO compiler messages slow down compilation a lot. How do you know this, did you benchmark or is it just your opinion? >>> Common knowledge, proved by experience. >> >> Please profile the compiler first, people did this already. > > This is what I'm going to do now. > > Does there exist some profiling code already, or do I have to reinvent > the wheel? I usually use callgrind together with kcachegrind (code needs to be compiled with -gv). ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Tomas Hajny schrieb: This cannot be done because people should have the chance to use custom error files to override message verbosities. Couldn't this be delegated to an custom message handler? Not sure if I understand here. Do you mean that instead of changing few letters in the message file such people would need to write their own message handler and recompile the compiler for using it? This would be one solution. Another one would interpret (such) message files once, on load, to extract the required information from them. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Sergei Gorelkin schrieb: The issue was that, whenever compiler needs to output a message, it: - loads a messagefile (once in a session) - looks up the message by number - performs the parameter substitution (this involves AnsiStrings, so is somewhat heavy) - only then checks if it should really print the resulting string. The most amount of messages come from unit search; the system units are loaded every time, so you always have about several thousands messages loaded and discarded. This was taking a noticeable amount of executed CPU instructions (profiled with Valgrind). This is just what I concluded from a quick glance at the implementation. With the patch applied in r9297, I was able to cut the total number of executed instructions down by 20%, but that gave no increase in perceived speed of compilation. So I decided not to put much more effort to this issue. Good to know :-) I think that it's time to put aside old experience, and replace it by up-to-date performance considerations. And since FPC seems to be optimized and tested very well, even if it doesn't look so at the first glance, I better concentrate on other tasks. Although, modifying the messaging system in a way when CheckVerbosity() is called as soon as possible (before parameter substitution and other processing) could be beneficial. Then at least one can be sure that the mentioned overhead can not occur any more. DoDi ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
On 15 July 2010 17:31, Tomas Hajny wrote: > requested to output no messages at all? Even if this (probably extremely > rare!) case happens, the difference must be completely negligible unless > you perform very many compilations of simple and short source files Our one project alone has over 250 units/forms with complex unit dependencies. This excludes 2 large external frameworks (also implemented in Object Pascal) that we pull in as well during compilation - we depend on them for certain functionality. I also tend to always build _all_ units, because I have been bitten enough times with source code changes which didn't trigger unit recompiles for some reason, and then waist my time debugging something that doesn't actually need debugging. So yes, anything to improve compiler speed is always welcomed by me. I miss the speed of Kylix or Delphi. And before somebody jumps on me, yes I know FPC has a different design because it supports multiple platforms and architectures - maintainability is more important to the core team that speed. -- Regards, - Graeme - ___ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/ ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op Thu, 15 Jul 2010, schreef Graeme Geldenhuys: maintainability is more important to the core team that speed. No. That is doesn't do justice to all the effort that is put into performance optimization. It's not about maintainability being more important. It is about making the right trade-offs between: - Compiler speed - Compiler memory usage - Generated code quality - Compiler portability - And indeed compiler maintainability. Compiler speed can loose from maintainability, but it can also loose from code quality; the performance of your application is probably also worth a lot to you. Nevertheless contest the idea that FPC is a slow compiler, I have put a lot of effort in optimizing compiler speed over the years. I work with many compilers daily, including GCC, Pathscale, Intel, Portland Group. FPC wins from all of these compilers by orders of magnitude. Last week I compiled OpenFOAM, a fluid dynamics software written in C++, with the Intel compiler. It took 9 hours for 110 megabytes of source code. FPC compiles such an amount of code in a few minutes... The fact it can do that can be attributed to the Pascal unit system (compared to include headers), but just as much to the choice of smart algorithms and datastrures and a tons of local code optimizations that were coded over many years. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Hans-Peter Diettrich schrieb: > > I think that it's time to put aside old experience, and replace it by > up-to-date performance considerations. And since FPC seems to be > optimized and tested very well, even if it doesn't look so at the first > glance, I better concentrate on other tasks. > One of the bottlenecks the common user encounters, is unit loading: especially projects like the lazarus suffer from the time spent into unit loading while I suspect that it narrows down also to procedures like fillchar which consume a lot of time. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op 2010-07-15 23:50, Daniël Mantione het geskryf: > No. That is doesn't do justice to all the effort that is put into > performance optimization. It's not about maintainability being more > important. > > It is about making the right trade-offs between: > - Compiler speed > - Compiler memory usage > - Generated code quality > - Compiler portability > - And indeed compiler maintainability. This is kind of what I meant, I just put it in much simpler terms. Sorry if my earlier statement caused offence. > Nevertheless contest the idea that FPC is a slow compiler, I have put a I never said FPC is slow, just that it is "slower" than Kylix and Delphi - but I also tried to explained why it is slower. I do know and acknoledge that FPC is MUCH faster than many other language compilers out there. A testament to the great work you guys hove done. Regards, - Graeme - -- fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/ ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Daniël Mantione wrote: Compiler speed can loose from maintainability, but it can also loose from code quality; the performance of your application is probably also worth a lot to you. Does FPC compile significantly faster when using -O- then? Micha ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op Fri, 16 Jul 2010, schreef Micha Nelissen: Daniël Mantione wrote: Compiler speed can loose from maintainability, but it can also loose from code quality; the performance of your application is probably also worth a lot to you. Does FPC compile significantly faster when using -O- then? That is irrelevant; for example the register allocator made the compiler slower (initially even 3 times, but we managed to reduce it to a few ten %), but had great effects on code quality. The most expensive optimization you can effect with -O is the assembler optimizer, this is noticable in compilation speed. However, debug info is for example something that affects speed more. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Daniël Mantione wrote: Op Fri, 16 Jul 2010, schreef Micha Nelissen: Does FPC compile significantly faster when using -O- then? That is irrelevant; for example the register allocator made the compiler slower (initially even 3 times, but we managed to reduce it to a few ten %), but had great effects on code quality. What do you mean with code quality then? I thought performance of the generated code. Micha ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
Re: [fpc-devel] Messages overhead
Op Fri, 16 Jul 2010, schreef Micha Nelissen: Daniël Mantione wrote: Op Fri, 16 Jul 2010, schreef Micha Nelissen: Does FPC compile significantly faster when using -O- then? That is irrelevant; for example the register allocator made the compiler slower (initially even 3 times, but we managed to reduce it to a few ten %), but had great effects on code quality. What do you mean with code quality then? I thought performance of the generated code. Then we are thinking the same :) The register allocator sacrified compiler speed for code quality. 2.0 generates much better code than 1.0. Also there were lots of bugs due to runing out of registers before, the register allocator dealt with that problem once and forever, so you can also say we sacrified compiler speed for compiler reliability. Daniël___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/mailman/listinfo/fpc-devel
