@araq:
> and after the thread dies, everything is freed.
That's really the problem. I am using `threadpool`, which creates 48 threads
when the program starts. Limiting the number of threads seems to cause some to
die after usage, but I don't understand what that code is doing. Are you
suggesti
> That's simple, secure, and extremely efficient. It's difficult to do in Nim
> (or in Rust, last I checked) because I cannot provide my own memory manager
> for portions of the code.
No, it's simple to do in Nim, the allocator is thread local anyway, so in your
worker thread call `GC_disable`
**cdunn2001:** _I guess it has 2 main drawbacks:_
1. You cannot cast from/to something in GCed memory. You'd have to copy the
contents. My problem here is that I have only a pretty vague idea of what
you're doing, so I'm flying blind and am trying to guess what may help you.
2. It's done thi
> It's difficult to do in Nim (or in Rust, last I checked) because I cannot
> provide my own memory manager for portions of the code.
Are you aware that in Nim you can turn off GC with `--gc:none` and for manual
allocation you can use many alloc families such as
[allocShared0](https://nim-lang.
You `MemBlock` idea is interesting. I guess it has 2 main drawbacks:
1. I don't know how to cast a `seq[]` or `string` into a portion of a
`RawArray`. I guess I could map `ptr Object`, for some unsafe operations, but a
big reason why I'm using Nim instead of C++ is the strong bounds-checking.
> > cdunn2001: I still would like to see freelist sizes.
>
> I'm not sure what you mean by that. Large blocks in the allocator (type
> BigChunk) are kept in a single list and are allocated using first-fit. Unless
> you are also allocating a lot of small objects, the amount of free and used
> me
For what it's worth, I threw together a basic implementation of what I wrote
above:
# Free Public License 1.0.0
#
# Permission to use, copy, modify, and/or distribute this software for any
# purpose with or without fee is hereby granted.
#
# THE SOFTWARE IS PROVID
markandsweep wasted even more memory, and it provided no memprofile info.
With boehm, I saw a crash on a nil "threadvar", which brings me back to my need
for clarification on threadvar persistence. I fixed that crash by using
`--tlsemulation:on`, as suggested. boehm then wasted about the same as
**cdunn2001:** _"Pathological"? If dealloc() fragments a large block, then
repeated allocation of large blocks would definitely cause memory use to
explode._
I was talking about something that sounded like unused space going up while
allocated memory remained constant. That would be a pathologi
I have some time this weekend to experiment with boehm and markandsweep.
> It wouldn't be surprising if total heap space (live objects + free space)
> went up as a result of fragmentation, but free space explosion would mean
> that you've encountered a pathological case.
"Pathological"? If deal
I mentioned the NUMA problems because you said you were experiencing poor
performance running on 48 cores, which might be caused by memory locality
issues. There's little you can do within Nim (other than through OS-specific
system calls). You probably want to limit execution to just the cores o
Thanks for the ideas. I did not get much info from running boehm. When I have
more time, I'll try markandsweep, and also tlsemulation.
> It's often best to limit the number of threads to the number of cores on a
> processor...
I don't know how to do that. Nim's threadpool does not offer a way.
First, some notes:
1. Linux defaults to `--tlsemulation:off`, macOS defaults to
`--tlsemulation:on`. I'm not sure how that would affect the behavior you
mention, though.
2. Any computer with 48 cores is going to be a NUMA architecture, so if your
code runs very slow, that might be the reaso
Ok. I have a sort-of repro:
*
[https://gist.github.com/pb-cdunn/9ac52d96d1791cbb2412e76865748a13](https://gist.github.com/pb-cdunn/9ac52d96d1791cbb2412e76865748a13)
When I run that in [https://glot.io/new/nim](https://glot.io/new/nim), things
seem fine. And they are fine on my Mac. (I'll do
First, yes, that's one of the problems with `spawn`: I don't know what thread
I'm on. That limits the value of _threadpool_.
But my crashes are gone when I stop using thread-local `nre.Regex`. Now I
create it locally always. A bit expensive, but safe. I've also dropped my other
thread-local var
Maybe something like this? I haven't played much with threads in Nim yet:
[https://github.com/nim-lang/Nim/blob/4f062c3be08fa2bc3e167e1a6b9842c92bc8c8f7/tests/threads/tonthreadcreation.nim](https://github.com/nim-lang/Nim/blob/4f062c3be08fa2bc3e167e1a6b9842c92bc8c8f7/tests/threads/tonthreadcreatio
Good idea. But clang didn't help.
In order to end the crashes, I had to stop relying on persistence. I simply
release (set to nil) the top variable at the end of each spawned call.
The performance on Linux is terrible, possibly related to allocations. I'll
post if I discover anything interestin
What compiler are you using on both Mac and Linux. If your using clang on Mac
and GCC on Linux, try compiling it on Linux using clang.
I need to use some per-thread, pre-allocated data-structures. The point is to
avoid re-creating large arrays except just once, the first time each thread
uses one.
With Python+C multiprocessing, I simply use a global. Each process has its own
global, and Python manages a pool of processes which
19 matches
Mail list logo
