[fpc-devel] Why/how does the compiler have a non-trivial number of memory leaks after over two decades of development?
I was feeling inquisitive today, so I added
'SetHeapTraceOutput('log.trc');" to the first line of pp.pas and built
myself a debug copy of the compiler using Lazarus. After building the
following very simple program:
program Test2;
{$mode objfpc}{$H+}
procedure TestProc(const S: String);
begin
WriteLn('The test string is: ', S);
end;
begin
TestProc('Hello, world!');
end.
I was honestly surprised to find that there was already an unfreed block:
140145 memory blocks allocated : 23669524/23770688
140144 memory blocks freed : 23669468/23770632
1 unfreed memory blocks : 56
True heap size : 1114112 (192 used in System startup)
True free heap : 1113696
Should be : 1113736
Call trace for block $5E642220 size 56
$0001DB6B
$0001000A35ED TNODE__ALLOCOPTINFO, line 973 of node.pas
$000100164889 SETEXECUTIONWEIGHT, line 366 of optutils.pas
$0001000C2873 FOREACHNODESTATIC, line 321 of nutils.pas
$0001000C2DAA PROCESS_CHILDREN, line 308 of nutils.pas
$0001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
$0001000C2DF1 PROCESS_CHILDREN, line 309 of nutils.pas
$0001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
So then I tried something just a little bit more complex:
program Test;
procedure TestProc(const S: String);
begin
WriteLn('The test string is: ', S);
end;
generic procedure GenericTestProc(const S: T);
begin
WriteLn('The test value is: ', S);
end;
begin
TestProc('Hello, world!');
specialize GenericTestProc(15);
end.
And unfortunately things got a lot worse very quickly:
141240 memory blocks allocated : 23794575/23896072
141219 memory blocks freed : 23791692/23893176
21 unfreed memory blocks : 2883
True heap size : 2195456 (192 used in System startup)
True free heap : 2188992
Should be : 2189680
Call trace for block $7CECC6A0 size 56
$0001DB6B
$0001000A35ED TNODE__ALLOCOPTINFO, line 973 of node.pas
$000100164889 SETEXECUTIONWEIGHT, line 366 of optutils.pas
$0001000C2873 FOREACHNODESTATIC, line 321 of nutils.pas
$0001000C2DAA PROCESS_CHILDREN, line 308 of nutils.pas
$0001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
$0001000C2DF1 PROCESS_CHILDREN, line 309 of nutils.pas
$0001000C28BF FOREACHNODESTATIC, line 336 of nutils.pas
Call trace for block $7CF57310 size 39
$0001DA72
$000180C3
$00018FAB
$000188B4
$0001000D3950 PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 394
of pgenutil.pas
$0001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$00010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$0001001887ED FACTOR, line 3390 of pexpr.pas
Call trace for block $7CF57490 size 46
$0001DA72
$000180C3
$00019052
$0001842D
$0001000D38A0 PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 391
of pgenutil.pas
$0001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$00010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$0001001887ED FACTOR, line 3390 of pexpr.pas
Call trace for block $7CEE93F0 size 128
$000100015F5B
$0001DB4B
$0001000282D5 TFPLIST__SETCAPACITY, line 757 of cclasses.pas
$0001000287C3 TFPLIST__EXPAND, line 846 of cclasses.pas
$00010002845C TFPLIST__ADD, line 784 of cclasses.pas
$0001000D368C PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 374
of pgenutil.pas
$0001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$00010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
Call trace for block $7CEEA950 size 128
$000100015F5B
$0001DB4B
$0001000282D5 TFPLIST__SETCAPACITY, line 757 of cclasses.pas
$0001000287C3 TFPLIST__EXPAND, line 846 of cclasses.pas
$00010002845C TFPLIST__ADD, line 784 of cclasses.pas
$0001000D35FF PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 366
of pgenutil.pas
$0001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$00010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
Call trace for block $7CF34A60 size 12
$0001DB6B
$0001000D35DD PARSE_GENERIC_SPECIALIZATION_TYPES_INTERNAL, line 364
of pgenutil.pas
$0001000D3F1E GENERATE_SPECIALIZATION_PHASE1, line 581 of
pgenutil.pas
$00010018A6B7 FACTOR_READ_ID, line 2850 of pexpr.pas
$0001001887ED FACTOR, line 3390 of pexpr.pas
$00010018BA4A SUB_EXPR, line 4184 of pexpr.pas
$00010018BA72 SUB_EXPR, line 4189 of pexpr.pas
$00010018BA72 SUB_EXPR, line 4189 of pexpr.pas
Call trace for block $7CF57550 size 24
$0001DA72
$0001BF6A
$0001BE05
$00010014E0A5 TSPECIALIZATIONCONTEXT__CREATE, line 62 of
pgentype.pas
$0001000D3EBB GENERATE_SPECIALIZATION_PHASE1, line 578 of
pgenutil.pas
$00010018A6B7 FACT
Re: [fpc-devel] Why/how does the compiler have a non-trivial number of memory leaks after over two decades of development?
On Sun, 29 Jul 2018, Ben Grasset wrote:
I was feeling inquisitive today, so I added
'SetHeapTraceOutput('log.trc');" to the first line of pp.pas and built
myself a debug copy of the compiler using Lazarus. After building the
following very simple program:
I tried other, more complex things after this, some of which are simply too
long to even be reasonably posted on the mailing list.
This raises a couple of big questions:
Considering that FPC provides optional built-in memory tracking that will
take you directly to the exact source line number of a leak, the fact that
there's so many either means people have simply not been doing leak checks
on the compiler at all, or have just been actively ignoring them. Neither
of those are "good news", but which is it?
Actively ignoring. This is a known issue, and also one of the reasons why the
compiler is not published as a library.
Michael.
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon 30/07/18 01:58 , Ben Grasset operato...@gmail.com sent: > I was even able to successfully actually fix some of the leaks, but others > boiled down to a "node" being created as a local variable in the middle of > some 1000+ line method, getting assigned to the "left" or "right" property > of some other globally visible node, and then never being freed. > To add to that, some of then actually *could not* be freed even when I > tried to do so without raising an untraceable access violation later in the > compilation process, also, which means the compiler is basically knowingly > relying on undebuggable undefined behaviour in multiple places that could > easily be affected or altered by anything else anywhere else in the > codebase at any time. > The way-too-short, highly undescriptive naming of variables doesn't > exactly help, either! "hp" and "p" are not good variable names. They really > aren't! > > Overall, I'm not trying to put blame on anyone in particular here, but as > someone who loves using FPC and would like to see it continue to grow, it > would be nice if we could collectively address some of these basic, > fundamental issues (which I'd be happy to help with myself, of course.) > I too love Free Pascal and seek to improve it. If you've found a fix, by all means submit it as a patch in the bug tracker. I've noticed that the compiler doesn't use try...finally blocks to help with freeing blocks. I'm not sure why this is the case, but might be speed related. More complex refactoring might require deeper planning and approval from Florian. For example, the node construction relies a lot on global variables (especially current_procinfo) that could possibly use some improvement. I think a big issue with refactoring is the amount of testing required because of the numerous target platforms. Don't be disheartened though. With enough eyes, all bugs are shallow! Gareth aka. Kit ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, J. Gareth Moreton wrote: On Mon 30/07/18 01:58 , Ben Grasset operato...@gmail.com sent: I was even able to successfully actually fix some of the leaks, but others boiled down to a "node" being created as a local variable in the middle of some 1000+ line method, getting assigned to the "left" or "right" property of some other globally visible node, and then never being freed. To add to that, some of then actually *could not* be freed even when I tried to do so without raising an untraceable access violation later in the compilation process, also, which means the compiler is basically knowingly relying on undebuggable undefined behaviour in multiple places that could easily be affected or altered by anything else anywhere else in the codebase at any time. The way-too-short, highly undescriptive naming of variables doesn't exactly help, either! "hp" and "p" are not good variable names. They really aren't! Overall, I'm not trying to put blame on anyone in particular here, but as someone who loves using FPC and would like to see it continue to grow, it would be nice if we could collectively address some of these basic, fundamental issues (which I'd be happy to help with myself, of course.) I too love Free Pascal and seek to improve it. If you've found a fix, by all means submit it as a patch in the bug tracker. I've noticed that the compiler doesn't use try...finally blocks to help with freeing blocks. I'm not sure why this is the case, but might be speed related. Yes. try/finally blocks cause *tremendous* slowdown. Michael.___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
J. Gareth Moreton schrieb am Mo., 30. Juli 2018, 13:31: > I've noticed that the compiler doesn't use > try...finally blocks to help with freeing > blocks. I'm not sure why this is the case, > but might be speed related. > Correct. Even implicit try-finally frame generation is disabled for the compiler source. Regards, Sven > ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel < fpc-devel@lists.freepascal.org> wrote: > J. Gareth Moreton schrieb am Mo., 30. Juli > 2018, 13:31: > >> I've noticed that the compiler doesn't use >> try...finally blocks to help with freeing >> blocks. I'm not sure why this is the case, >> but might be speed related. >> > > Correct. Even implicit try-finally frame generation is disabled for the > compiler source. > Is performance more important than being correct? :| regards, Marcos Douglas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: > Is performance more important than being correct? :| In this project, the answer is always taken to be yes. -- Regards, Martok ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, Marcos Douglas B. Santos wrote: On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel < fpc-devel@lists.freepascal.org> wrote: J. Gareth Moreton schrieb am Mo., 30. Juli 2018, 13:31: I've noticed that the compiler doesn't use try...finally blocks to help with freeing blocks. I'm not sure why this is the case, but might be speed related. Correct. Even implicit try-finally frame generation is disabled for the compiler source. Is performance more important than being correct? :| You've been around long enough to know that complaints about the speed of the compiler is something that pops up on a regular basis. Also, and this is important: the compiler was developed for TP. It had no concept of exceptions. Everything was done using records and pointers. Many people will probably be surprised, but the use of try/finally is not required for an application: Gradually, some parts of the compiler were switched to classes, but even then, without exceptions: As long as you do not use the sysutils unit, (which was the case for many years for the compiler) exceptions will not be raised, and so there is simply no need for try/finally blocks. Probably it would be better to remove the dependency on sysutils, to keep the compiler free of try/finally blocks. But the compiler devs will be able to provide a more correct&detailed picture than this. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
From: fpc-devel On Behalf Of Marcos
Douglas B. Santos
Sent: Monday, 30 July 2018 22:25
To: FPC developers' list
Subject: Re: [fpc-devel] Why/how does the compiler have a non-trivial number
ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel
mailto:fpc-devel@lists.freepascal.org> > wrote:
J. Gareth Moreton mailto:gar...@moreton-family.com>
> schrieb am Mo., 30. Juli 2018, 13:31:
I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.
Correct. Even implicit try-finally frame generation is disabled for the
compiler source.
Is performance more important than being correct? :|
Freeing memory is irrelevant if your process is going to exit anyway right
after freeing it. The OS doesn’t care about any memory your process has
allocated at that time. All pages go back into the available pool.
Depending on how the allocation/free behaviour of the compiler looks like, but
it might even make sense to use a memory manager that doesn’t even know how to
free memory.
(I’m using windows terminology here, but I’m sure equivalent can be found on
other platforms)
Reserve a large block of address space.
Install a vectored exception handler that reacts to an access violation that
falls inside the reserved block by committing the memory between start and
error address + 1MB (or more or less).
Naïvely and without error checking, Alloc gets reduced to:
var
CurrentPtr: PByte;
function Alloc(Size: Integer):Pointer;
begin
Size := (Size + 3) and not 3; //adjust depending on what allocation alignment
is desired;
If Size > 0 then begin
Result := CurrentPtr;
CurrentPtr := CurrentPtr + Size;
end else
Result := nil;
end;
procedure Free(p: Pointer);
begin
{do nothing}
end;
___
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018 14:36:10 +0200 (CEST) Michael Van Canneyt wrote: >[...] > Many people will probably be surprised, but the use of try/finally > is not required for an application: > > Gradually, some parts of the compiler were switched to classes, but even > then, without exceptions: As long as you do not use the sysutils unit, > (which was the case for many years for the compiler) exceptions will > not be raised, and so there is simply no need for try/finally blocks. Many people will probably surprised, that try/finally/except works without sysutils and try/except does not need "Exception". It works with any TObject. > Probably it would be better to remove the dependency on sysutils, to keep > the compiler free of try/finally blocks. Sysutils has nothing to do with creating exception frames. Mattias ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, Mattias Gaertner wrote: On Mon, 30 Jul 2018 14:36:10 +0200 (CEST) Michael Van Canneyt wrote: [...] Many people will probably be surprised, but the use of try/finally is not required for an application: Gradually, some parts of the compiler were switched to classes, but even then, without exceptions: As long as you do not use the sysutils unit, (which was the case for many years for the compiler) exceptions will not be raised, and so there is simply no need for try/finally blocks. Many people will probably surprised, that try/finally/except works without sysutils and try/except does not need "Exception". It works with any TObject. Probably it would be better to remove the dependency on sysutils, to keep the compiler free of try/finally blocks. Sysutils has nothing to do with creating exception frames. It does: if you know the program will never raise exceptions, it makes no sense to install try/finally blocks. Runtime errors are converted to exceptions by the sysutils unit. By not including it, you know you will just get runtime errors, and hence you don't need try..finally. Obviously provided you don't install another mechanism that does this and don't raise exceptions manually, which - AFAIK - is the case in the compiler... Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
I would say that that's a little naïve and dangerous to think like that. Sure, Windows might have the means to clean up memory after an application terminates, but not all platforms have such heap deallocation features (e.g. pure DOS, where certain procedures and interrupts remain in memory even after the application terminates... so-called memory-resident programs). Even if all blocks are freed when the compiler finishes its work, memory leaks may still cause problems with sufficiently large projects that are being compiled in one go. The more leaks you have, the more likely you're going to hit an out of memory situation, especially as you can't really control the target platforms in regards to how they have their physical and virtual memory and page files configured. A lot of the issues do come about from the fact that Free Pascal has a lot of legacy code and started life as a 16-bit Turbo Pascal project (it was first made because Turbo Pascal was being discontinued so Borland could focus on Windows development, and it didn't have 32-bit support). Free Pascal is advanced enough now that a lot of the features could be rewritten to use more robust and self-contained features, but it's going to be a long job. I've made some suggestions to major improvements, especially to speed or reducing the size of the compiler's binary (and the size of the binaries that it generates), but Florian has turned down some of the patches either because it's too experimental (my Deep Optimizer prototype) or overcomplicates the codes (I replaced a large case block for the x86 peephole optimiser with a custom-made binary search system, which was slightly faster but wasn't obvious in its function and would silently break if it wasn't sorted properly). I would say that trying to minimise the reliance on global variables would be a big improvement to the node builder, because that would allow it to be multi-threaded. *makes note of a possible future area of research!* Gareth aka. Kit On Mon 30/07/18 13:40 , "Thorsten Engler" thorsten.eng...@gmx.net sent:... Freeing memory is irrelevant if your process is going to exit anyway right after freeing it. The OS doesn’t care about any memory your process has allocated at that time. All pages go back into the available pool. ... ___ fpc-devel maillist - fpc-devel@lists.freepascal.org [1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] mailto:fpc-devel@lists.freepascal.org [2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, J. Gareth Moreton wrote: I would say that that's a little naïve and dangerous to think like that. Sure, Windows might have the means to clean up memory after an application terminates, but not all platforms have such heap deallocation features (e.g. pure DOS, where certain procedures and interrupts remain in memory even after the application terminates... so-called memory-resident programs). The compiler is not such program. You obviously need to choose for which programs you do this. But having the memory freed properly would also enable the use of the compiler as a DLL in e.g. Lazarus. Michael.___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
Hi, On Mon, 30 Jul 2018, J. Gareth Moreton wrote: > I would say that that's a little naïve and dangerous to think like that. > Sure, Windows might have the means to clean up memory after an > application terminates, but not all platforms have such heap > deallocation features (e.g. pure DOS, where certain procedures and > interrupts remain in memory even after the application terminates... > so-called memory-resident programs). While I generally agree, in practice it's usually a non-issue. Pure DOS, as in the 16bit target is not supported to run the compiler, only via GO32V2, and having DPMI TSRs is... well... :) On Amiga and similar systems, AROS, MorphOS for example, which indeed does not free up the memory after a program terminates, I implemented a memory pool in the FPC RTL, so all memory is freed anyway during the program's exit. Any application which wants to go beyond this behavior for whatever of its own reasons is free to use the OS allocation functions directly, or implement its own memory manager. The same with open files on this platform, actually, which wouldn't be closed without the RTL taking care of them. And I also have to manage our own stack, and lets not even mention threading... :) The sideeffect is, that this approach silently "fix" the compiler's own leaks too, so FPC remains useable on these systems. In general it is much safer anyway, because expecting all the broken code out there to get fixed for such systems is sadly just naive. Having this said, anyone who still fixes leaks win the compiler will be my hero! :) Charlie___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivialnumberofmemory leaks after over two decades of development?
Yeah, I realised after I said that that keeping programs resident is very different from a memory leak! Still, it's still very good practice to do your own clean-up, especially as you described below. Embedding a compiler DLL into Lazarus would allow for much more interaction. Gareth aka. Kit On Mon 30/07/18 14:57 , Michael Van Canneyt mich...@freepascal.org sent: On Mon, 30 Jul 2018, J. Gareth Moreton wrote: > I would say that that's a little naïve and dangerous to think like that. > Sure, Windows might have the means to clean up memory after an application > terminates, but not all platforms have such heap deallocation features > (e.g. pure DOS, where certain procedures and interrupts remain in memory > even after the application terminates... so-called memory-resident > programs). The compiler is not such program. You obviously need to choose for which programs you do this. But having the memory freed properly would also enable the use of the compiler as a DLL in e.g. Lazarus. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org [1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] mailto:fpc-devel@lists.freepascal.org [2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivialnumberofmemory leaks after over two decades of development?
Being your hero is a good enough reason for me! One little thing that's driven me, even though it's not really an issue, is the size of the EXE files that are produced, so I seek to find any little way to shrink them down without compromising performance in the final binary. What I like about this project is that we all have our little motivators and objectives, and we're free to explore and research. I had an idea to potentially improve the compiler speed on modern systems, but requires a lot of refactoring and I'm currently got my attention focused on researching pure function support. One drawback with any kind of memory management code is that it can put a lot of space and speed overhead on your project, although it is very useful from a debugging point of view. I've programmed my own ones in the past for C-based test projects that are meant to run for 24 hours sometimes, mostly in response to frequent crashes. It's proven very useful in catching memory leaks and identifying where they are so the memory can be freed properly. So, long story short... if you find memory that isn't freed, free it (once you check it's not used afterwards), even if the operating system or RTL cleans it up for you. Gareth aka. Kit On Mon 30/07/18 15:05 , "Karoly Balogh (Charlie/SGR)" char...@scenergy.dfmk.hu sent: Hi, On Mon, 30 Jul 2018, J. Gareth Moreton wrote: > I would say that that's a little naïve and dangerous to think like that. > Sure, Windows might have the means to clean up memory after an > application terminates, but not all platforms have such heap > deallocation features (e.g. pure DOS, where certain procedures and > interrupts remain in memory even after the application terminates... > so-called memory-resident programs). While I generally agree, in practice it's usually a non-issue. Pure DOS, as in the 16bit target is not supported to run the compiler, only via GO32V2, and having DPMI TSRs is... well... :) On Amiga and similar systems, AROS, MorphOS for example, which indeed does not free up the memory after a program terminates, I implemented a memory pool in the FPC RTL, so all memory is freed anyway during the program's exit. Any application which wants to go beyond this behavior for whatever of its own reasons is free to use the OS allocation functions directly, or implement its own memory manager. The same with open files on this platform, actually, which wouldn't be closed without the RTL taking care of them. And I also have to manage our own stack, and lets not even mention threading... :) The sideeffect is, that this approach silently "fix" the compiler's own leaks too, so FPC remains useable on these systems. In general it is much safer anyway, because expecting all the broken code out there to get fixed for such systems is sadly just naive. Having this said, anyone who still fixes leaks win the compiler will be my hero! :) Charlie ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
Michael Van Canneyt schrieb am Mo., 30. Juli 2018, 15:30: > Obviously provided you don't install another mechanism that does this and > don't raise exceptions manually, which - AFAIK - is the case in the > compiler... > The compiler does use exceptions when the compilation needs to be aborted e.g. when too many errors occurred or an internal error is encountered. Regards, Sven > ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, Sven Barth via fpc-devel wrote: Michael Van Canneyt schrieb am Mo., 30. Juli 2018, 15:30: Obviously provided you don't install another mechanism that does this and don't raise exceptions manually, which - AFAIK - is the case in the compiler... The compiler does use exceptions when the compilation needs to be aborted e.g. when too many errors occurred or an internal error is encountered. This is bad news, because it in effect means that the try/finally is necessary if people want to fix memleaks :( This will considerably slow down the compiler. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: > Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: >> Is performance more important than being correct? :| > In this project, the answer is always taken to be yes. > To hopefully offer some explanation for Mr. Santos - if I can compile and run the program 5 times over the course of 20 minutes, that is certainly better than running it and testing it once over the same amount of time, no? It might be hard to imagine FPC taking that much longer than it does currently but ~30min for a large program is the standard with other compilers. I very much enjoy the speed of FPC. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, 30 Jul 2018, R0b0t1 wrote: On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: Is performance more important than being correct? :| In this project, the answer is always taken to be yes. To hopefully offer some explanation for Mr. Santos - if I can compile and run the program 5 times over the course of 20 minutes, that is certainly better than running it and testing it once over the same amount of time, no? It might be hard to imagine FPC taking that much longer than it does currently but ~30min for a large program is the standard with other compilers. I very much enjoy the speed of FPC. 30 *Minutes*, is this real ? Hm. I just complained to a component provider because Delphi takes 25 seconds to compile my program with the way they set up their paths... Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
Not necessarily. The Internal Errors are effectively assertions that should never be triggered. True, cleaning up the memory is a bonus, but here it's safe to assume that the internal state of the compiler is bad in some way. Throwing an exception upon too many errors might be problematic because there's no telling where it might trigger, but otherwise, the compiler abort tends to occur after it's finished attempting to compile its current node trees, so most memory should be freed. Still, it's something one might want to research to see if the exception handler can check and free some global structures. Gareth aka. Kit On Mon 30/07/18 16:24 , "Michael Van Canneyt" mich...@freepascal.org sent: On Mon, 30 Jul 2018, Sven Barth via fpc-devel wrote: > Michael Van Canneyt schrieb am Mo., 30. Juli 2018, > 15:30: > >> Obviously provided you don't install another mechanism that does this and >> don't raise exceptions manually, which - AFAIK - is the case in the >> compiler... >> > > The compiler does use exceptions when the compilation needs to be aborted > e.g. when too many errors occurred or an internal error is encountered. This is bad news, because it in effect means that the try/finally is necessary if people want to fix memleaks :( This will considerably slow down the compiler. Michael. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [1]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 10:35 AM, Michael Van Canneyt wrote: > > > On Mon, 30 Jul 2018, R0b0t1 wrote: > >> On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: >>> >>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: Is performance more important than being correct? :| >>> >>> In this project, the answer is always taken to be yes. >>> >> >> To hopefully offer some explanation for Mr. Santos - if I can compile >> and run the program 5 times over the course of 20 minutes, that is >> certainly better than running it and testing it once over the same >> amount of time, no? >> >> It might be hard to imagine FPC taking that much longer than it does >> currently but ~30min for a large program is the standard with other >> compilers. I very much enjoy the speed of FPC. > > > 30 *Minutes*, is this real ? > > Hm. I just complained to a component provider because Delphi takes 25 > seconds to compile my program with the way they set up their paths... > This is real.[1] I've experienced it myself and you can see some good discussions on https://news.ycombinator.com with a search. FPC/Lazarus has made some waves there in the last few years. FPC does have quite a bit of a lead it could shed in an attempt to be "more correct" but I would caution against this. If at every point in a software stack someone says "modern computers are just so fast, we have a little bit of wiggle room" you end up with something slower than molasses. Perfection requires effort at every single step. A good case study on this is keyboard latency in modern systems.[2] Cheers, R0b0t1 [1]: The most extreme comparison was closer to being something like a couple of minutes vs. a couple of hours, but I can't source that right now. Most people think it's a lie; granted, some of it is likely due to the excellent incremental compilation FPC has by default. [2]: https://danluu.com/keyboard-latency/ ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number of memory leaks after over two decades of development?
Am 30.07.2018 um 02:58 schrieb Ben Grasset: This raises a couple of big questions: Considering that FPC provides optional built-in memory tracking that will take you directly to the exact source line number of a leak, the fact that there's so many either means people have simply not been doing leak checks on the compiler at all, or have just been actively ignoring them. Neither of those are "good news", but which is it? Secondly, are there any plans to try to clean up and refactor the compiler codebase in general? There are a lot of plans. But a compiler is an incredible complex piece of software, imo not comparable with other software, rewriting some parts might be useful but reducing complexity is normally not possible. After diving in to take a look at where some of these leaks originated, it's not at all surprising they exist. The entire compiler essentially seems to revolve around "nodes" and various subclasses of them, that are created in too many different places and ways to count, and cannot be tracked in any meaningful way. This kind of tree design is perfectly valid and relatively common, but the way FPC does it leaves more than a few things to be desired. I was even able to successfully actually fix some of the leaks, but others boiled down to a "node" being created as a local variable in the middle of some 1000+ line method, getting assigned to the "left" or "right" property of some other globally visible node, and then never being freed. I am aware of this, but I consider it as minor. The only way to get around this would be add to each node an owner field and set it to current_procinfo and track all nodes in current_procinfo and clean them together with the current_procinfo. Question is: what do we gain? The compiler consumes more memory because of the tracking and it is slower because of the tracking. Cleaner code? Well, the OS cleans up memory anyways when the process exits. I were even thinking along lines Torsten mentioned (not to free any memory, but let the OS do its job) as memory is during compilation often not a problem, cpu speed is. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 10:24 AM, Michael Van Canneyt wrote: > > > On Mon, 30 Jul 2018, Sven Barth via fpc-devel wrote: > >> Michael Van Canneyt schrieb am Mo., 30. Juli >> 2018, >> 15:30: >> >>> Obviously provided you don't install another mechanism that does this and >>> don't raise exceptions manually, which - AFAIK - is the case in the >>> compiler... >>> >> >> The compiler does use exceptions when the compilation needs to be aborted >> e.g. when too many errors occurred or an internal error is encountered. > > > This is bad news, because it in effect means that the try/finally is > necessary > if people want to fix memleaks :( > > This will considerably slow down the compiler. > I would very much like to see the compiler usable as a dynamic library. Compartmentalizing the bulk of the code in this manner might aid readability and comprehension even if it is not immediately wise to actually use the shared object in a third party application due to memory leaks. I wouldn't mind running some static analysis on FPC in an attempt to get rid of memory leaks. It's fairly mechanical and probably a good introduction to the codebase. This page[1] is promising as it indicates valgrind should work well on FPC binaries (I didn't have much a reason to expect otherwise, but you never know). Is there any reason this hasn't been done? What could raise statements be replaced with if they must be replaced? Cheers, R0b0t1 [1]: http://wiki.lazarus.freepascal.org/Profiling ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote: > On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: >> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: >>> Is performance more important than being correct? :| >> In this project, the answer is always taken to be yes. >> > > To hopefully offer some explanation for Mr. Santos - if I can compile > and run the program 5 times over the course of 20 minutes, that is > certainly better than running it and testing it once over the same > amount of time, no? > > It might be hard to imagine FPC taking that much longer than it does > currently but ~30min for a large program is the standard with other > compilers. I very much enjoy the speed of FPC. All us like fast compiling... However, IMHO, a compiler should use the same good practices that others programs must use. If it cannot use them because it's very slow, what is the matter about good practices? "Do what I'm telling you to do, but don't do what I do" ? :) best regards, Marcos Douglas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
Compilers are one of those programs that fall under a very special, very exclusive set of projects where a lot of the standard rules may not apply, but if they can be followed where possible, brilliant. Sometimes, increasing complexity and maintainability in the name of speed is not a solution, as I found out with my binary search implementation. However, a way around that is to see if we can program such a feature into the compiler itself if it detects it can gain a speed boost from it (in this situation, a case block with 36 values to choose from, but where each branch calls an identically-structured procedure... some of the values call the same routine, but otherwise the routines have the same parameters and return types). If you can find a way to fix a memory leak without sacrificing performance too significantly, by all means submit a patch for review. If I had my way, if I wanted to make the compiler as fast as possible, I'd just drop into assembly language for the most critical parts, but that's a big no-no in this particular project, not because of how hard it is to maintain, debug and explain, but because of cross compilation issues. You should be able to create a compiler that targets any platform of your choice. I think the only time you can use assembly language is with some of the internal compiler routines like the Frac function. Gareth aka. Kit On Mon 30/07/18 17:42 , "Marcos Douglas B. Santos" m...@delfire.net sent: On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote: > On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: >> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: >>> Is performance more important than being correct? :| >> In this project, the answer is always taken to be yes. >> > > To hopefully offer some explanation for Mr. Santos - if I can compile > and run the program 5 times over the course of 20 minutes, that is > certainly better than running it and testing it once over the same > amount of time, no? > > It might be hard to imagine FPC taking that much longer than it does > currently but ~30min for a large program is the standard with other > compilers. I very much enjoy the speed of FPC. All us like fast compiling... However, IMHO, a compiler should use the same good practices that others programs must use. If it cannot use them because it's very slow, what is the matter about good practices? "Do what I'm telling you to do, but don't do what I do" ? :) best regards, Marcos Douglas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org [3] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [4]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] mailto:r03...@gmail.com [2] mailto:list...@martoks-place.de [3] mailto:fpc-devel@lists.freepascal.org [4] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
I thought I'd add a point about compiling and memory. On the Raspberry Pi, I know for a fact that building the compiler can fail after a while due to insufficient memory. When this happens the problem with no memory doesn't present itself as such. Rather you're left with an error indicating a problem with the source. I've gotten around this problem by making the compiler without a desktop environment running and re configuring the swap size. Might this problem be related in some way to the discussion in this thread? ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos wrote: > On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote: >> On Mon, Jul 30, 2018 at 7:32 AM, Martok wrote: >>> Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: Is performance more important than being correct? :| >>> In this project, the answer is always taken to be yes. >>> >> >> To hopefully offer some explanation for Mr. Santos - if I can compile >> and run the program 5 times over the course of 20 minutes, that is >> certainly better than running it and testing it once over the same >> amount of time, no? >> >> It might be hard to imagine FPC taking that much longer than it does >> currently but ~30min for a large program is the standard with other >> compilers. I very much enjoy the speed of FPC. > > All us like fast compiling... > However, IMHO, a compiler should use the same good practices that > others programs must use. > If it cannot use them because it's very slow, what is the matter about > good practices? > > "Do what I'm telling you to do, but don't do what I do" ? :) > If a program isn't long running I see programmers tend to not care about memory management, nor do I make a habit of caring about it myself. I find it hard to see anything wrong with this. As someone has mentioned this can fail on smaller computers, but how much RAM is too little? SBCs now typically come with 1-4GB of memory. Cheers, R0b0t1 ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On 07/30/2018 11:35 AM, Michael Van Canneyt wrote: On Mon, 30 Jul 2018, R0b0t1 wrote: It might be hard to imagine FPC taking that much longer than it does currently but ~30min for a large program is the standard with other compilers. I very much enjoy the speed of FPC. > 30 *Minutes*, is this real ? yep, for sufficiently large projects... when we compile OSG (OpenSceneGraph) over here (using -j 8) it takes 20+ minutes for the first time or if we reconfigure from debug to release mode or the other way around... now, after the first time we compile it, the next times takes mere 10's of seconds... then we move on to SceneGear and FlightGear when we're building that whole project which uses OSG... 30-45 minutes for a complete, from the ground up builds are not uncommon... it takes me back to the days of 30 years ago when you'd start the (borland/turbo pascal) compile process and go get some c0ffee ;) note: the machines these builds are being done on have 16Gig RAM, 1Tb HD, and a 4Ghz 8-core AMD FX-8350 CPU... -- NOTE: No off-list assistance is given without prior approval. *Please keep mailing list traffic on the list unless* *a signed and pre-paid contract is in effect with us.* ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivialnumberofmemory leaks after over two decades of development?
It might be a problem because it's a false error and not exactly clean. Granted, testing such things is not that straightforward. Platforms such as Windows do have a feature where you can tell it to randomly fail memory allocations after about 8 minutes of start time (so the machine can fully boot up) - the trouble is that graphical elements on the desktop stop being rendered properly, among other failures, and you can't predict where an allocation will fail in the program. I think you can put in exceptions so allocations don't fail for Lazarus or the Command Prompt, for example. The trouble is, these allocation failures are designed for C programs, where "malloc" returns NULL if memory isn't allocated. Free Pascal's GetMem raises an exception instead. Granted, it shouldn't be too much of a struggle to implement a custom memory manager for testing and to cause random (or deterministic) allocation failures to see how cleanly the situation is handled. It could be a fun test suite! It is times like these that give weight to being careful with memory allocations. I started programming in the mid-90s where memory was still quite tight (namely, a computer running Windows 95 with only 8 MB of RAM!) While computers are very fast and spacious today, the same standards still apply in places, notably in tight loops and functions that get executed very frequently. Gareth aka. Kit On Mon 30/07/18 19:24 , "Anthony Walter" sys...@gmail.com sent: I thought I'd add a point about compiling and memory. On the Raspberry Pi, I know for a fact that building the compiler can fail after a while due to insufficient memory. When this happens the problem with no memory doesn't present itself as such. Rather you're left with an error indicating a problem with the source. I've gotten around this problem by making the compiler without a desktop environment running and re configuring the swap size. Might this problem be related in some way to the discussion in this thread? ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [1]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
Correct me if I'm mistaken, but I believe one shortcoming with the Free Pascal Compiler, despite its speed, is that it isn't multi-threaded. It can't compile multiple independent units at once, and probably won't be able to for a while due to the use of global variables in the node builder. When it comes to building Lazarus and the compiler, this is gotten around by the Makefiles running several instances of the compiler on different processes, building independent packages that can later be linked. I personally think that if the compiler can be made multi-threaded, then projects like OpenSceneGraph might see a huge improvement in compile times if the compiler is smart enough to identify units that don't depend on each other and compile them in parallel. I'm not sure how cross-compatible it can be made, since different platforms approach multithreading in different ways and I've noticed that the compiler tends to avoid using SysUtils and Classes and the like (so no using TThread for now). This is getting a bit off-subject from fixing memory leaks though. Gareth aka. Kit On Mon 30/07/18 19:39 , wkitt...@windstream.net sent: On 07/30/2018 11:35 AM, Michael Van Canneyt wrote: > On Mon, 30 Jul 2018, R0b0t1 wrote: >> It might be hard to imagine FPC taking that much longer than it does >> currently but ~30min for a large program is the standard with other >> compilers. I very much enjoy the speed of FPC. > > 30 *Minutes*, is this real ? yep, for sufficiently large projects... when we compile OSG (OpenSceneGraph) over here (using -j 8) it takes 20+ minutes for the first time or if we reconfigure from debug to release mode or the other way around... now, after the first time we compile it, the next times takes mere 10's of seconds... then we move on to SceneGear and FlightGear when we're building that whole project which uses OSG... 30-45 minutes for a complete, from the ground up builds are not uncommon... it takes me back to the days of 30 years ago when you'd start the (borland/turbo pascal) compile process and go get some c0ffee ;) note: the machines these builds are being done on have 16Gig RAM, 1Tb HD, and a 4Ghz 8-core AMD FX-8350 CPU... -- NOTE: No off-list assistance is given without prior approval. *Please keep mailing list traffic on the list unless* *a signed and pre-paid contract is in effect with us.* ___ fpc-devel maillist - fpc-devel@lists.freepascal.org [1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] mailto:fpc-devel@lists.freepascal.org [2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 3:31 PM, R0b0t1 wrote: > On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos > wrote: >> On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote: >> >> [...] > > If a program isn't long running I see programmers tend to not care > about memory management, nor do I make a habit of caring about it > myself. I find it hard to see anything wrong with this. > > As someone has mentioned this can fail on smaller computers, but how > much RAM is too little? SBCs now typically come with 1-4GB of memory. I'm sorry, but we work with a language that expected programmers should manage their objects. The language even provides a clean way (try-finall) to do that. I use try-finally even to write a snippet to show an example of code... IMO, that must be a habit in such languages. regards, Marcos Douglas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
As mentioned before, the main problem with try...finally blocks is that they incur a massive speed penalty, so they're not things you want to enter and exit on a frequent basis. Whether that means we need to work on the compiler so it reduces this penalty on such blocks (such as internally removing the construct and copying the code in the 'finally' section into a local exception handler) I'm not sure. I do agree though that we should manage memory with care, especially as Pascal gives us such fine control over it (compared to, say, C++'s RAII feature which is a little nebulous at times with its garbage collection). I think first though we need to fix any obvious memory leaks, namely objects and blocks of memory that we forget to free, then build up from there. At least that will help reduce the risk of an out of memory situation during normal operation. Gareth aka. Kit On Mon 30/07/18 21:27 , "Marcos Douglas B. Santos" m...@delfire.net sent: On Mon, Jul 30, 2018 at 3:31 PM, R0b0t1 wrote: > On Mon, Jul 30, 2018 at 11:42 AM, Marcos Douglas B. Santos > wrote: >> On Mon, Jul 30, 2018 at 12:29 PM, R0b0t1 wrote: >> >> [...] > > If a program isn't long running I see programmers tend to not care > about memory management, nor do I make a habit of caring about it > myself. I find it hard to see anything wrong with this. > > As someone has mentioned this can fail on smaller computers, but how > much RAM is too little? SBCs now typically come with 1-4GB of memory. I'm sorry, but we work with a language that expected programmers should manage their objects. The language even provides a clean way (try-finall) to do that. I use try-finally even to write a snippet to show an example of code... IMO, that must be a habit in such languages. regards, Marcos Douglas ___ fpc-devel maillist - fpc-devel@lists.freepascal.org [4] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [5]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] mailto:r03...@gmail.com [2] mailto:m...@delfire.net [3] mailto:r03...@gmail.com [4] mailto:fpc-devel@lists.freepascal.org [5] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
Am 30.07.2018 um 19:55 schrieb J. Gareth Moreton: Correct me if I'm mistaken, but I believe one shortcoming with the Free Pascal Compiler, despite its speed, is that it isn't multi-threaded. It can't compile multiple independent units at once, and probably won't be able to for a while due to the use of global variables in the node builder. When it comes to building Lazarus and the compiler, this is gotten around by the Makefiles running several instances of the compiler on different processes, building independent packages that can later be linked. I personally think that if the compiler can be made multi-threaded, then projects like OpenSceneGraph might see a huge improvement in compile times if the compiler is smart enough to identify units that don't depend on each other and compile them in parallel. I'm not sure how cross-compatible it can be made, since different platforms approach multithreading in different ways and I've noticed that the compiler tends to avoid using SysUtils and Classes and the like (so no using TThread for now). TThread is not a necessity. It's functionality and the synchronization primitives are based on what TThreadManager provides. This manager resides in the System unit and is accessible through functions like BeginThread and such. Please note however that this might need an external thread manager on certain platforms (e.g. Unix-like systems) and the idea on those platforms is for the compiler *not* to rely on the C-library. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivialnumberofmemory leaks after over two decades of development?
Am 30.07.2018 um 19:49 schrieb J. Gareth Moreton: The trouble is, these allocation failures are designed for C programs, where "malloc" returns NULL if memory isn't allocated. Free Pascal's GetMem raises an exception instead. Granted, it shouldn't be too much of a struggle to implement a custom memory manager for testing and to cause random (or deterministic) allocation failures to see how cleanly the situation is handled. It could be a fun test suite! The default heap of FPC's RTL respects the value of ReturnNilIfGrowHeapFails (which by default is False). Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivialnumberofmemory leaks after over two decades of development?
Well, if a certain platform relies on an external library, multi-threading could be disabled on that platform through a flag of some kind, at least until something can be programmed for that platform (or more conservatively, only enabled if it's known to work without any hitches). Just an idea. I can't remember off-hand a good enumeration set to include such a flag in though. If multi-threading is not supported, then it can just run single-threaded as it does now. Gareth aka. Kit On Mon 30/07/18 22:08 , Sven Barth via fpc-devel fpc-devel@lists.freepascal.org sent: Am 30.07.2018 um 19:55 schrieb J. Gareth Moreton: Correct me if I'm mistaken, but I believe one shortcoming with the Free Pascal Compiler, despite its speed, is that it isn't multi-threaded. It can't compile multiple independent units at once, and probably won't be able to for a while due to the use of global variables in the node builder. When it comes to building Lazarus and the compiler, this is gotten around by the Makefiles running several instances of the compiler on different processes, building independent packages that can later be linked. I personally think that if the compiler can be made multi-threaded, then projects like OpenSceneGraph might see a huge improvement in compile times if the compiler is smart enough to identify units that don't depend on each other and compile them in parallel. I'm not sure how cross-compatible it can be made, since different platforms approach multithreading in different ways and I've noticed that the compiler tends to avoid using SysUtils and Classes and the like (so no using TThread for now). TThread is not a necessity. It's functionality and the synchronization primitives are based on what TThreadManager provides. This manager resides in the System unit and is accessible through functions like BeginThread and such. Please note however that this might need an external thread manager on certain platforms (e.g. Unix-like systems) and the idea on those platforms is for the compiler *not* to rely on the C-library. Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org [1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] mailto:fpc-devel@lists.freepascal.org [2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have anon-trivialnumberofmemory leaks after over two decades of development?
Apologies if these things seem trivial. I probably need to read the documentation cover to cover to learn about these global state variables. I am a bit surprised that hitting out of memory exceptions causes incorrect error messages to appear though. Gareth aka. Kit On Mon 30/07/18 22:10 , Sven Barth via fpc-devel fpc-devel@lists.freepascal.org sent: Am 30.07.2018 um 19:49 schrieb J. Gareth Moreton: The trouble is, these allocation failures are designed for C programs, where "malloc" returns NULL if memory isn't allocated. Free Pascal's GetMem raises an exception instead. Granted, it shouldn't be too much of a struggle to implement a custom memory manager for testing and to cause random (or deterministic) allocation failures to see how cleanly the situation is handled. It could be a fun test suite! The default heap of FPC's RTL respects the value of ReturnNilIfGrowHeapFails (which by default is False). Regards, Sven ___ fpc-devel maillist - fpc-devel@lists.freepascal.org [1] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel [2]">http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel Links: -- [1] mailto:fpc-devel@lists.freepascal.org [2] http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have a non-trivial numberofmemory leaks after over two decades of development?
On Mon, Jul 30, 2018 at 2:40 PM, J. Gareth Moreton wrote: > As mentioned before, the main problem with try...finally blocks is that they > incur a massive speed penalty, so they're not things you want to enter and > exit on a frequent basis. Whether that means we need to work on the compiler > so it reduces this penalty on such blocks (such as internally removing the > construct and copying the code in the 'finally' section into a local > exception handler) I'm not sure. > > I do agree though that we should manage memory with care, especially as > Pascal gives us such fine control over it (compared to, say, C++'s RAII > feature which is a little nebulous at times with its garbage collection). I > think first though we need to fix any obvious memory leaks, namely objects > and blocks of memory that we forget to free, then build up from there. At > least that will help reduce the risk of an out of memory situation during > normal operation. > > Gareth aka. Kit > Adding locally allocated structures to a global list is a pretty gnarly issue that will take some time to solve. The easier ones are simply using labels to easily manage exit conditions. It looks a lot nicer than nested if statements. The issue with such code is it is fairly boring to write, so I understand if someone didn't do it initially. ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
Re: [fpc-devel] Why/how does the compiler have anon-trivialnumberofmemory leaks after over two decades of development?
J. Gareth Moreton schrieb am Mo., 30. Juli 2018, 23:20: > Apologies if these things seem trivial. I probably need to read the > documentation cover to cover to learn about these global state variables. > > I am a bit surprised that hitting out of memory exceptions causes > incorrect error messages to appear though. > Well, going out of memory is a bit of a nasty situation and normally not really checked for anyway. Regards, Sven > ___ fpc-devel maillist - fpc-devel@lists.freepascal.org http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-devel
