Memory leak in event loop inside a thread?

Update: after manually clearing dispatcher's `callbacks` and `timers` and manually calling a GC the memory is properly freed, even when the `cycle` coroutine is still pending. So the question is: why do I have to do this manually? Shouldn't this be done automatically when the thread finishes? T

Memory leak in event loop inside a thread?

Another piece of info: replacing `std/asyncdispatch` with `chronos` fixes the problem in this case. But the problem returns with the following changes: proc cycler(x: ref MyObj) {.async.} = while true: x.value[] += 1 echo x.value[] await sleepAsync(200)

Memory leak in event loop inside a thread?

The code below leaks memory and doesn't print "memory deallocated" (so the object's destructor isn't called). If the `cycler` coroutine isn't created, no leak occurs. Am I doing something wrong, or is there a memory manager bug at play? Tried both `2.0.8` and `2.1.9` nim versions. Compiled with

Spot the memory leak when using Chronos and Chronos/threadsync.ThreadSignalPtr

First up, thanks a ton for making me aware of "onThreadDestruction", I was very much not aware and am a huge fan of more explicit hooks like that. I'll include that going forward for sure. Next up, regarding the ThreadDispatcher: > Looking at the log you provided in the GitHub issue, the intern

Spot the memory leak when using Chronos and Chronos/threadsync.ThreadSignalPtr

cleanupThread() Run In the case shown above, the cycle collector is run _before_ `x` goes out of scope. When `x` goes out of scope, the referenced cell will be registered as a potential cycle root (which allocates memory for ORC's root list), and you'll thus get a m

Spot the memory leak when using Chronos and Chronos/threadsync.ThreadSignalPtr

I'm writing this here as well as on the issue tracker () in the hopes that maybe somebody with more experience can see the problem. So I'm playing around with a multithreading lib that has a while-loop going on another thread that has an asyn

Have I written a silly program or is there a "controlled" memory leak?

Thank you everyone for your input, leading to solve the issue. Although, it still remains a mystery for me that why the memory utilization never exceeded `107MB` and somehow we always "reset" back to `1MB`.

Have I written a silly program or is there a "controlled" memory leak?

also compiler instrumentation is your friend use it with --debugger:native

Have I written a silly program or is there a "controlled" memory leak?

valgrind is your friend.

Have I written a silly program or is there a "controlled" memory leak?

This did the trick! Thanks.

Have I written a silly program or is there a "controlled" memory leak?

Try with `--threads:on` instead, even though you’re not using threads. If I recall correctly, it can impact ORC behavior in some ways.

Have I written a silly program or is there a "controlled" memory leak?

Missing db.close() in the "Rows are already written to DB" return

Have I written a silly program or is there a "controlled" memory leak?

According to my knowledge, "memory leak" means the memory is allocated and is not freed. Leaked memory does not go back until the program is terminated. If there's a memory leak, the amount of used memory will increase by each iteration (See the example in my last post). It will

Have I written a silly program or is there a "controlled" memory leak?

It always goes to `107MB` and back to `1MB`. No one restarts the program. There is no manual intervention at all. Each loop takes a few milliseconds so this memory usage does not make sense. I have added the memory report log to the repository under the name of `memreport.log`

Have I written a silly program or is there a "controlled" memory leak?

Hi Araq, I have added the nginx log, the executable and memory report log to the same repository.

Have I written a silly program or is there a "controlled" memory leak?

I have not tried with `-d:useMalloc` since I did not know about it. However, I have added the memory log report file, one of our nginx logs and the executable to the same repository.

Have I written a silly program or is there a "controlled" memory leak?

> > database OT for the question, as you use the SQLite library you don't need to open and close the 'connection' to it every time. You can open it once an leave it open until the program finishes, the close it. `setControlCHook()`

Have I written a silly program or is there a "controlled" memory leak?

Very hard to tell without actually seeing the program. Have you tried running it with `-d:useMalloc`? By default Nim has an internal memory pool. So when it frees things it doesn't necessarily free it back to the OS right away.

Have I written a silly program or is there a "controlled" memory leak?

Please provide a file with test data, it's interesting.

Have I written a silly program or is there a "controlled" memory leak?

It always go to 107MB and back to 1MB? Never exceeds 107MB? When it goes back, always to 1MB and not 2MB? This happens automatically? If so, I think there's no memory leak. If there is a memory leak, wouldn't it be something like `1MB -> 2MB -> ... 107MB -> 2MB -> 3MB

Have I written a silly program or is there a "controlled" memory leak?

Hi, I have written a small program called nginwho for my company that parses nginx logs and stores them in a database. Here is the link to the repo: The program is compiled using `-d:release --mm:orc --threads:off` flags and Nim version 2.0.2:

Understanding an address sanitizer message for a memory leak caused by global dispatcher (?)

As a current workaround, as I just found out, you can apparently just nil the dispatcher yourself before the threads join: `getGlobalDispatcher(nil)` That gets rid of all leaks related to the dispatcher that I had so far.

Understanding an address sanitizer message for a memory leak caused by global dispatcher (?)

Thank you very much for the insight! I have subsequently opened up an issue regarding this as suggested:

Understanding an address sanitizer message for a memory leak caused by global dispatcher (?)

p{.threadvar.}: owned PDispatcher ## Global dispatcher Run It's a thread-local variable. > Or rather, gut feeling says that this is address sanitizer not seeing the > destruction of the global dispatcher and thus falsely claiming a memory leak. It is a memory leak, if the gDisp t

Understanding an address sanitizer message for a memory leak caused by global dispatcher (?)

never understood what the supposed solution is. Like a memory leak reported by heaptrack from `readLine`. That one just makes me assume that heaptrack is faulty because I trust the output is getting accurately collected. I'm developing similar vibes for address sanitizers that they contain a dec

Understanding an address sanitizer message for a memory leak caused by global dispatcher (?)

=address examples/ex_stdinput.nim` Output The biggest memory leak Indirect leak of 57352 byte(s) in 1 object(s) allocated from: #0 0x55c1436e1de1 in __interceptor_calloc (/home/philipp/.cache/nim/ex_stdinput_r/ex_stdinput_DD41B6791D32A0B925C6208AC76777EEE30185AD+0x116de1

-d:useMalloc fixes memory leak, but I don't know why

At some point I'll create a minimal program that reproduces the issue. But I'll also retest with each new version.

-d:useMalloc fixes memory leak, but I don't know why

Well ARC/ORC is responsible for calling `free` (with or without `-d:useMalloc`) and since it doesn't leak with `-d:useMalloc` it's likely Nim allocator's fault. The allocator got an overhaul for its multi threading support so it's probably related.

-d:useMalloc fixes memory leak, but I don't know why

The thread count of the process matches what I'd expect. But I think that there is likely an ARC/ORC bug that will probably be fixed at some point. I'll just use -d:useMalloc for now, until I take a closer look again.

-d:useMalloc fixes memory leak, but I don't know why

> I know, I just thought that perhaps someone had experienced the same issue > and would recognize it >quickly. The issue Treeform mentioned comes close to > something like that. I'll see what I can do to >create a minimal reproducible > program for the issue. ORC/ARC seem to have problems with

-d:useMalloc fixes memory leak, but I don't know why

I know, I just thought that perhaps someone had experienced the same issue and would recognize it quickly. The issue Treeform mentioned comes close to something like that. I'll see what I can do to create a minimal reproducible program for the issue.

-d:useMalloc fixes memory leak, but I don't know why

That's bad but unless you share some more information with the rest or even try yourself to diagnose the problem, you cant expect for people to guess your issue. Its impractical and a waste of time. I am saying it in good faith. Regards.

-d:useMalloc fixes memory leak, but I don't know why

My progress goes into GBs of memory use until the OOM kills it. So I don't think it's that issue which is only a few MB in difference.

-d:useMalloc fixes memory leak, but I don't know why

You might be running into this: Nim has its own free pool that OS does not know about. OS might think memory is increasing but its just the internal free pool getting bigger.

-d:useMalloc fixes memory leak, but I don't know why

Just to clarify, the leak is fixed without running under valgrind.

-d:useMalloc fixes memory leak, but I don't know why

If I understand correct, valgrind works with malloc only

-d:useMalloc fixes memory leak, but I don't know why

The program I'm running has a thread per websocket. Running `valgrind --tool=massif` showed the websockets were using most of the memory. The websocket library I use is [niv](https://github.com/niv/websocket.nim). This is with the ORC mm. When I use -d:useMalloc as a Nim compile flag, the leak

Foreign thread callbacks causes memory leak

> Try -d:useMalloc. Yes, that works without any problems or leaks.

Foreign thread callbacks causes memory leak

Try `-d:useMalloc`.

Foreign thread callbacks causes memory leak

I'm using callbacks at several places in the code. The callbacks are being run in threads created by a runtime system outside the nim runtime which has no clue about them. The callbacks first calls a nim raw function trampoline which then calls a stored nim closure. When using the allocator in

Memory leak fixes for db_postgres.nim

This fix will be backported, thank you!

Memory leak fixes for db_postgres.nim

I've fixed some memory leaks in db_postgres.nim. I had a long running service that kept running out of memory, and I noticed several Postgres processes were at their maintenance_work_mem at the same time (I think this is because of single-row mode). The pull request has just been merged:

Memory leak when embedding nimscript

This is a nimscripter issue, as such should've been posted there, but I'm guessing something is not GC'd with the interpreter which means a `dealloc` or similar call is required.

Memory leak when embedding nimscript

Hello, I am using Nimscripter to add scripting capabilities to my program. I am using it customize the behavior of my program as per the requirement of different clients. However, I have noticed that it leaks memory whenever a script is run. Program crashes once a day because system runs out of

Memory Leak with --gc:orc and Nimble Decimal Library

I've already found the actual problem (ARC not calling the destructor), see . As a workaround you can change nim-decimal yourself so it uses an object and defines a destructor on it.

Memory Leak with --gc:orc and Nimble Decimal Library

Great to hear you found the issue. Wonderful to have such a quick response!

Memory Leak with --gc:orc and Nimble Decimal Library

A much simpler way to check if memory leaks with `orc` is by using `-d:useMalloc` together with `valgrind`: import std/sugar import decimal/decimal proc main() = let constDcm = newDecimal("0.15") var newRange = collect(for x in 0..<5: constDcm + 1)

Memory Leak with --gc:orc and Nimble Decimal Library

Thanks for the attention to my issue and recommendations on valgrind use. I experiment with valgrind.

Memory Leak with --gc:orc and Nimble Decimal Library

when defined(nimTypeNames): dumpNumberOfInstances() echo() echo "\n\n## No Memory Leak ##\n\n" when isMainModule: main() Run

Memory leak

I'm not sure if this is valid any longer

Memory leak

Thank you for sharing the solution to this issue!

Memory leak

Thank you for sharing! It solved the memory leak for me! It is a halpful thread.

Memory leak

Thank you for sharing! It solved the memory leak for me! It is a helpful thread.

Memory leak

Well, listen to me please! fix the ram

How to use memory leak detection tools on Windows with --gc:arc and -d:useMalloc?

Thanks @Araq. Actually the warnings only seem to appear when the program is exited abnormally, so doesn't seem to be anything to worry about.

How to use memory leak detection tools on Windows with --gc:arc and -d:useMalloc?

> Is there something blocking these tools from detecting the free call done by > --gc:arc? Not that I know of but we allocate a single OutOfMem exception somewhere that is not freed.

How to use memory leak detection tools on Windows with --gc:arc and -d:useMalloc?

Hey guys, I'm using `--gc:arc` and `-d:useMalloc`, but I'm getting memory leak warnings with `crtdbg.h`: proc setCrtDbgMemFlags() {.header:"#include \n#include " importcpp:"_CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK

Memory leak with ARC and malloc when using references and casts

The reason that it doesn't leak without `-d:useMalloc` is because nim holds the memory in case it needs to be reallocated to the leak may not be as easily detectable

Memory leak with ARC and malloc when using references and casts

There is also the .cursor pragma, maybe you can avoid the table altogether somehow?

Memory leak with ARC and malloc when using references and casts

For completeness I'll mention how I solved the issue (at least how I _believe_ I solved the issue). For that I will shortly describe what I actually wanted to achieve. I wanted to create a small entity component manager. For that I need a type `TypeVector`, that can do something like this:

Memory leak with ARC and malloc when using references and casts

My guess: In this expandarc output, `:tmpD_1` is a temporary variable introduced to hold the `new seq[Component]` call. It it then in effect cast _and copied into r, that increments the underlying reference counter, both ``r`` and ``:tmpD_1`` shares a reference to the same buffer but with differ

Memory leak with ARC and malloc when using references and casts

the expandarc looks the same when compiler with or without `-d:useMalloc`, and as the version without the malloc doesn't leak, I assumed that there can't be a missing destroyer. To be honest, I don't quite understand the output: --expandArc: test var r :tmpD

Memory leak with ARC and malloc when using references and casts

My point is to inspect the output of expandarc and try to find if a destructor call is missing.

Memory leak with ARC and malloc when using references and casts

> as the issue happens only with --gc:arc -d:useMalloc The point is that valgrind can see the leak only with --gc:arc -d:useMalloc. See my book:

Memory leak with ARC and malloc when using references and casts

`--expandarc:test`doesn't help here, I believe, as the issue happens only with `--gc:arc -d:useMalloc` but not with only `--gc:arc`.

Memory leak with ARC and malloc when using references and casts

Best way to identify the leak is --expandarc:test. Then open a new issue on GitHub.

Memory leak with ARC and malloc when using references and casts

I fixed the issues with the first example. This still has the same leak issue: type ComponentC = ref object c: int func newComponentC(): ComponentC = result = new ComponentC result[].c = 150 proc test() = var r: ref seq[int8]

Memory leak with ARC and malloc when using references and casts

I tried again with the latest nightly, and it still behaves the same.

Memory leak with ARC and malloc when using references and casts

I don't know about the leak, but shouldn't this code not work? `ref ComponentC` becomes `ref ref int`, no? I assume this is what you meant to write: type ComponentC = ref object c: int func newComponentC(): ComponentC = result = new ComponentC result[

Memory leak with ARC and malloc when using references and casts

Fixed in devel right?

Memory leak with ARC and malloc when using references and casts

The following example doesn't leak when compiled with `nim c --gc:arc main.nim && valgrind --leak-check=full ./main`, but it does leak 16 bytes when compiled with `nim c --gc:arc -d:useMalloc main.nim && valgrind --leak-check=full ./main` type ComponentC = ref object c: int

Guidance on Isolating Memory Leak

Thanks! I had misinterpreted that system modules were auto-imported in nim. The change does fix the memory leak when using the default refc gc. For --gc:{orc|arc}, the app still leaks memory at a far faster rate than originally with --gc:refc.

Guidance on Isolating Memory Leak

yes, int system/strs_v2.nim the cstrToStr makes a fresh string. 2) if you don't want to import system/ansi_c ? then just copy the definition to your own file.

Guidance on Isolating Memory Leak

The decimal maintainers surmise the leak is from not freeing the cstring returned by the underlying mpd_decimal library function mpd_to_sci(). This seems to be in the following code snippet: proc `$`*(s: DecimalType): string = ## Convert DecimalType to string result = $

Guidance on Isolating Memory Leak

@araq: > Compile with `--gc:orc`... Or `--gc:arc` and keep comparing using `getOccupiedMem()` in strategic places can work too; however, be aware that `getOccupiedMem()` doesn't currently work with `--threads:on` combined with Arc/Orc as per issue #18494, always reporting zero with these combi

Guidance on Isolating Memory Leak

Probably the decimal package.

Guidance on Isolating Memory Leak

The modules dependency was a red herring. I managed to produce a trivial testcase that reproduces the leak when using nimble "decimal [0.0.2]" for numeric values: while true: var totStr = "" for item in seqOfDecimals: totStr &= $item

Guidance on Isolating Memory Leak

Thanks for the suggestions. I'm on the trail of the leak and cutting down the application to a small(er) testcase. The issue seems related to a proc call across modules on a ref object with embedded ref objects.

Guidance on Isolating Memory Leak

FYI, I've had the best luck finding memory leaks using Valgrind (. Requires the use of `-d:useMalloc` and I suggest combining with ARC (if you can) or ORC.

Guidance on Isolating Memory Leak

We use -d:nimTypeNames together with to track memory usage over time, per type - it's quite useful for a first hint at what's wrong - here's a sample of what this looks like for a long-running process: [https://metrics.status.im/d/pgeNfj2Wz23/nimbus-fl

Guidance on Isolating Memory Leak

With the old default GCs you can do this: * Compile with -d:nimTypeNames * At strategic places in your program call dumpNumberOfInstances to see what the GC heap contains. This is mostly useful to find logical leaks where data structures simply keep growing and since they stay alive, the

Guidance on Isolating Memory Leak

Compile with `--gc:orc -d:useMalloc` and/or `--gc:boehm` and see if it makes a difference. If the results don't change, you need to review your code and your used dependencies and watch out for data structures that keep growing, for example some "history" like objects with a `seq` inside that st

Guidance on Isolating Memory Leak

Hello, I have an application ported from Python that leaks memory. As a first step to confirm a memory leak, a call to GC_getStatistics() and mratsim's getrusage() procedures is made every 100,000 json messages processed by the application. The output shows an incremental increase i

Forcing a memory leak in Nim

according to the docs, useMalloc >only works with gc:none and with --newruntime. which is a bit out of date, perhaps, but doublechecking the source we have, in mmdisp.nim: elif (defined(nogc) or defined(gcDestructors)) and defined(useMalloc): include system / mm / malloc

Forcing a memory leak in Nim

It turns out that `valgrind` can only function if it could replaces libc's `malloc` calls (see this [FAQ](https://www.valgrind.org/docs/manual/faq.html#faq.hiddenbug)). The page explicitly said that static linking and/or custom memory allocators are not supported by valgrind. This should answer

Forcing a memory leak in Nim

mory allocator (like `malloc`) and does not perform any automated memory collection. > which is why I expect a memory leak when I call alloc without a corresponding > dealloc when I'm in --gc:refc mode. I actually expect this as well. At the very least it should still say that the

Forcing a memory leak in Nim

The thing is that even when manually allocating stuff you use Nim's TLSF allocator unless you switch to -d:useMalloc

Forcing a memory leak in Nim

** get a memory leak. > I do not understand what specifically you do not understand I want to understand _why_ the above program does not produce a memory leak. Even if garbage collection is turned on, that memory should be mine to handle manually because I marked the variable as a `ptr`. Since I

Forcing a memory leak in Nim

useMalloc also works with ORC or refc for that matter

Forcing a memory leak in Nim

I think useMalloc is necessary for valgrind to detect the leak, otherwise it is hidden by Nims TLSF allocation, and useMalloc is only compatible with gcNone or gcArc

Forcing a memory leak in Nim

First you may try to put the Nim code into a main proc, then it will become closer to the C program. And maybe compile with -d:useMalloc and maybe also with --gc:arc.

Forcing a memory leak in Nim

the fact that this doesn't compile with gc:none and -d:useMalloc is a regression you could submit as an issue on github. you can, however use gc:arc and -d:useMalloc, which then leaks as expected.

Forcing a memory leak in Nim

Just `var mem = create int` should do.

Forcing a memory leak in Nim

I am trying to force Nim to leak memory to better understand the memory model. >From my understanding, the following two programs are identical. #include int main() { int *x = malloc(sizeof(int)); x = 0; } Run var mem: ptr int = c