Re: Is this use of Nim locks correct?
To be honest I don't know I never used volatile, but if your use case was multi-threading, I would follow Intel advice and find another portable way to sync memory/have memory fences.
Re: Send data structures between threads?
In case it helps, my shared memory, Garbage Collected, data structure (and 64 byte aligned) is this: const FORCE_ALIGN = 64 type BlasBufferArray[T] = object dataRef: ref[ptr T] data*: ptr UncheckedArray[T] len*: int proc deallocBlasBufferArray[T](dataRef: ref[ptr T]) = if not dataRef[].isNil: deallocShared(dataRef[]) dataRef[] = nil proc newBlasBuffer[T](size: int): BlasBufferArray[T] = ## Create a heap array aligned with FORCE_ALIGN new(result.dataRef, deallocBlasBufferArray) # Allocate memory, we will move the pointer, if it does not fall at a modulo FORCE_ALIGN boundary let address = cast[ByteAddress](allocShared0(sizeof(T) * size + FORCE_ALIGN - 1)) result.dataRef[] = cast[ptr T](address) result.len = size if (address and (FORCE_ALIGN - 1)) == 0: result.data = cast[ptr UncheckedArray[T]](address) else: let offset = FORCE_ALIGN - (address and (FORCE_ALIGN - 1)) let data_start = cast[ptr UncheckedArray[T]](address +% offset) result.data = data_start I use it for shared-memory parallelism via OpenMP [here](https://github.com/mratsim/Arraymancer/blob/master/src/tensor/fallback/blas_l3_gemm_data_structure.nim#L30-L34). To use it with the convenient array indexing syntax, data is a `ptr UncheckedArray[T]` instead of just ptr T. dataRef is just there to make the object GC-managed, I don't use it otherwise.
Re: Send data structures between threads?
Hi @peheje. I'm really not the expert on this, as I'm a newbie at Nim myself, but I'm _pretty sure_ I read that everything you send over a Channel is _copied_. The refs are local to the thread, so if you send data with ref, both the data, and what is pointed by refs, gets cloned "magically". This is why I'm looking into making my own "shared heap" replacements for seq/array, and eventually sets and tables (although, of course, I would rather use an _existing_ "shared heap" implementation of those structures, if one is available in some nimble package).
Re: Is this use of Nim locks correct?
@mratsim Does that mean I shouldn't use [https://github.com/nim-lang/Nim/blob/master/lib/pure/volatile.nim](https://github.com/nim-lang/Nim/blob/master/lib/pure/volatile.nim) ? @mikra How would you "share" something between threads without globals? Globals are generally considered "evil", but atm I would not know of a better way to do this in Nim... Create your data in the main thread before creating "worker threads", and pass a pointer to your data at thread creation maybe?
Re: Send data structures between threads?
Anything new on this? I'm trying to implement a simple genetic algorithm where each thread shall have access to a shared seq[seq[int]], its easy to partition the data into chunks so the threads know which items to work on, but Nim seems IMPLICITLY copy the data, whether I try channels, parallel statements, spawn statements or even "bare" threads. Am I doing something wrong? I thought that because the object is a ref object only the ref: the address, would be copied? Code here: [https://github.com/peheje/nim_genetic/blob/master/mutable_state.nim](https://github.com/peheje/nim_genetic/blob/master/mutable_state.nim) I appreciate language features like parallel: with spawn does some magic in the background implicitly but I think being too creative might take many programmers by "unpleasent" surprise.
Re: Is this use of Nim locks correct?
@Monster sorry for my late response; I see rayman22201 already answered your question. I think it's a good habit to declare the globals immediately. My code was just to experiment with Locks and Conditions; I personally avoid globals. @rayman22201 It was fun-stuff I have to work now with databases, but also locking stuff and dealing with deadlocks @mratsim thank you for your interesting additions
Re: How to get real time of parallel code
The Unix `time` command [https://en.wikipedia.org/wiki/Time_%28Unix%29](https://en.wikipedia.org/wiki/Time_%28Unix%29) [https://www.lifewire.com/command-return-time-command-4054237](https://www.lifewire.com/command-return-time-command-4054237) returns three (3) variants: `real`, `usr`, `sys`. Nim's `cpuTime()` corresponds to `usr` (total cpu time), whereas `epochTime()` corresponds to `real` (clock time). I would like to recommend that `realTime()` be included in Nim's `time` module to make it shadow that component of the Unix `time` command, which will make it's function and use case much clearer to users. This may only need to create `realTime()` as an alias of `epochTime()`.
Re: Wrapping cpp type with integer parameters
Can't think of a solution except patching the compiler.
Re: How do you keep your motivation on your open-source projects
Slightly off topic, but when I run into compiler limitations/bugs all the time,
I fix the compiler.
Some random remarks about keeping motivation up:
* Our brains evolved in order to control our bodies. A brain is not designed
for heavy thought based processes, these get exhausting quickly. In other
words, there is no alternative to physical activities like **sports**.
* Solve concrete problems, not the imagined problems of other programmers.
"Oh, this vector class only works for float triples, that's not extensible! --
_Shrug_, it's good enough for Unreal Engine 4,
[https://docs.unrealengine.com/latest/INT/API/Runtime/Core/Math/FVector/index.html](https://docs.unrealengine.com/latest/INT/API/Runtime/Core/Math/FVector/index.html)
" (Probably not the best example... )
* Take shortcuts, the perfect thing takes an eternity to develop and once
you're done, you might notice you solved the wrong problem. ("This new GC
algorithm is independent of heap sizes and adheres to deadlines, muhaha it is
perfect!" \--- years later --- "I don't really want to write pointer soup
programs that are so complex they require a GC")
* Accept it's hard work that can burn you out, regardless of the used
technologies. There is a reason people get paid for programming.
Re: How to get real time of parallel code
Yes, that did it. This is a good use case example to distinguish in `time` docs between the use of `cpuTime()` and `epochTime()`. The given example for `cpuTime()` assumes single threaded use and not the effect of multi processor/threads use. [https://nim-lang.org/docs/times.html](https://nim-lang.org/docs/times.html)
How do you keep your motivation on your open-source projects
I've been working on Arraymancer for 6 months now with ups and downs. This is my first big project (and actual the first time I'm developing "for real") and I would like to know how do you keep your energy for the long run. In my case, I seem to alternate between various modes: * "The algorithm/macros rabbit hole": There is some shiny thing I want to implement well, from scratch, and I spent hours trying different stuff, for example using Nim macros to replicate Numpy-like multi-dimensional slicing or implementing state-of-the-art parallel matrix multiplication. * "The benchmark-er": Let's do benchmark of that new shiny toy, Anandtech-style, and see how many `genericReset` I trigger per second. * "The lazy wrapper-er": Wow, the rabbit hole led somewhere, now I'm exhausted and I want "easy" stuff, let's wrap a library, say from Nvidia. * "The showman": Usually at one point, I feel like I need to release and show something so I go deep into "examples" and "documentation". And then there are some times when: * I just don't want to do anything (took 1.5 months off code in July) * Nim seems to have an axe to grind with you, looking at you `static[T]` and `C++ codegen` So what keep you going ?
Re: using if expressions
And another: let channels = 7 let format = if channels == 1: 1 elif channels == 3: 2 elif channels == 4: 3 else: (block: echo "Texture Format Unknown, assuming RGB" 4 ) echo format
Re: Cannot get name of type using typedesc
I think that the run button should stay hidden by default in top right corner until your mouse hovers the code block, because most code posted right now were not submitted to be run and is wasting three lines of text for no reason. Ideally the output area would appear once the output from run execution is received.
Re: Is this use of Nim locks correct?
@Rayman2220 @wizzardx I had shared memory issue in OpenMP at one point in Arraymancer (solved since then). I just checked volatile and from this [Intel post](https://software.intel.com/en-us/blogs/2007/11/30/volatile-almost-useless-for-multi-threaded-programming), it says that there is no guarantee to have a memory fence, it just so happens that for x86 and Itanium volatile implies memory fences. Consequently Intel removed all volatile variables from their TBB (Intel Threads Building Blocks, their task parallelism library). So volatile in C is good for interrupts/hardware communication when memory can be written suddenly by the hardware but not suitable for multithreading.
Re: How to get real time of parallel code
Try _epochTime()_.
Re: Is this use of Nim locks correct?
Ah, also a reminder - try nimgrep in addition to regular grep: [https://nim-lang.org/docs/nimgrep.html](https://nim-lang.org/docs/nimgrep.html) Nim's style insensitivity means that you may miss a few useful things if you just use the regular grep.
Re: testing private methods in external module
Thank you for the responses and code, I'm learning a lot about the macro
system. Nice idea to use a pragma.
Testing the macro I was getting the compiler error:
t.nim(11, 20) Error: invalid indentation
I used treeRepr in the macro to debug the problem. It turned out I was putting
the pragma in the wrong place.
wrong:
proc main {.tpub.} (p: int): string =
right:
proc main(p: int): string {.tpub.} =
If you use the first way, a proc node is passed to the macro as expected but it
does not contain any parameters.
Re: testing private methods in external module
Argh indeed.
Re: Is this use of Nim locks correct?
> so I'm not generally a programming newbie. I don't think anyone meant to imply that you were a newbie! I apologize if I came off condescending. I specialized in concurrency and multi-threading at university, and you would be surprised at how many many people would show me code in their hot path that had lots of loops and allocations, and then wonder why their code was still slow even though they _"parallelized"_ it. These were fairly competent programmers otherwise. * * * > For example, is it OK to directly initialize a global variable guarded with a > lock, as I did, or do I have to leave it uninitialized(nil), and get a lock > just to initialize it? That is an interesting question. This is what the nim manual says about it: [https://nim-lang.org/docs/manual.html#guards-and-locks](https://nim-lang.org/docs/manual.html#guards-and-locks) "Top level accesses to gdata are always allowed so that it can be initialized conveniently. It is assumed (but not enforced) that every top level statement is executed before any concurrent action happens." I think that means that the answer is no, you don't have to take a lock to initialize the global variable. YMMV. * * * > I don't use any "refs" in myList(). Perfect. Based on your post I was pretty sure you did the right thing, I just wanted to be clear and cover all the bases. * * * > my assumption is that this code will only be called when initializing the > modules If it's not in the hot path, then it's good enough * * * > I assume the Nim library hashtables only work on local heap Yeah, that's a safe assumption. Nim doesn't have any good lock free or concurrent data structures built in atm. Thankfully Nim is great at interfacing with C lol. * * * > my grep-foo is strong This should hopefully become less of an issue as Nim matures. All aboard the better and moar documentation train!
Re: Nim T-shirts and mugs now available
Done and done
Nim T-shirts and mugs now available
Hope you guys like these. First iteration so definitely room for improvement. Let me know your thoughts [https://teespring.com/nimlang1](https://teespring.com/nimlang1) [https://teespring.com/nimlang2](https://teespring.com/nimlang2) [https://teespring.com/nimlang_mug](https://teespring.com/nimlang_mug)
Re: testing private methods in external module
@mratsim `export` is about forwarding an imported symbol, it cannot be used to export a local symbol.
