Faster Euclidean algorithm

u > v: swap(u, v) v = v - u if v == 0: return u shl shift Run gist: <https://gist.github.com/mratsim/afb82da37287fb79dcd6825acec23f65> gcd in stdlib: 982 micro second gcdLAR:1195 micro second gcdLAR2: 16

JIT compiling to AMD GPUs

See I develop on RDNA3 so anything between Radeon HD 7790 (2014) to today.

JIT compiling to AMD GPUs

In my latest PR I managed to JIT compile from LLVM IR to AMD GPU: <https://github.com/mratsim/constantine/pull/453> Now the question becomes can e do Nim -> AMD GPU (or Nvidia as AMD toolchain supports both). As we have a LLVM-IR => AMD (and LLVM-IR => Nvidia) we can us

Requirements for an imperative language amenable to sql-like optimizations

es of Polyhedral optimization. Now I'm only talking in automated loop nests optimizations, you have plenty of reference paper on that in Arraymancer for optimizing image processing or tensor operations (PolyMage, Tiramisu, ...): <https://github.com/mratsim/Arraymancer/issues/347#issuecomme

nim simd (avx2) How to get going?

either you use unaligned loads and sequences or you use your own `ptr UncheckedArray` and you align manually and then you can use aligned loads. Either use posix_memalign or windows equivalent <https://github.com/mratsim/constantine/blob/478c19e/constantine/platforms/allocs.nim#L43-

The lowest possible module(s) of graphics in Nim?

Out of the OpenGL centric framework you have also webview: import webview let w = newWebview() # or you can use create() w.title = "Basic Example" # or use setTitle() w.size = (480, 320) # or setSize() w.html = "Thanks fo

Small libraries you'd like to see ported to Nim?

I think a bittorrent library would be a nice effort that is easy to communicate around: * everyone is aware of Bittorrent * it's fairly low-level so no need to build tons of library around (say GUI) * it can be useful * lots of reference implementation The only issue is that you likely

Heap vs stack & value-type vs reference-type

> Are objects allocated on the heap or the stack. type Foo = object # the stack type Bar = ref object # the heap, managed by Nim type Baz = ptr object # the heap, managed by yourself Run And even if you declare an object on the stack you can allocate it on the h

modular KEM based double ratchet for Nim!

Super cool, I've always wanted to implement the Signal Protocol in Nim. > In my humble opinion, the core cryptographic library for nim should have at > least one cryptographic primitive in each category: * PRP - N/A * PRNG: <https://github.com/mratsim/constantine/blob/v0.1

Questions on the current stage of concepts

Quoting myself on July 1st: > And I think it's quite important to document and give examples of "new > concepts" > If we want to have > a strong ecosystem that

Nimble incorrect package structure warning: What is going to be?

Nop that warning should disappear. I've said so countless times on the ~3 RFCs on package structures.

Problem with a highly recursive function

release is needed otherwise GCC/Clang don't enable tail-call optimization (TCO). Your alternatives is either to rewrite the function to be non-recursive, or use a trampoline or find the GCC/Clang flag that triggers TCO.

Releasing Constantine v0.1.0 - A modular cryptography stack for blockchain and proof systems

For those interested in Nim performance vs C, Rust, Go implementations of the same algo, I have made a comprehensive post at

Releasing Constantine v0.1.0 - A modular cryptography stack for blockchain and proof systems

I am very proud to release the very first version of [Constantine](https://github.com/mratsim/constantine), a high-performance modular cryptography stack for blockchains and proof systems. I thank Ethereum Protocol Fellowship Program and Status for sponsoring the part-time work of 2 fellows on

Nim version 2.0.8 released

> tl;dr on Apple's docs: it seems to be as simple as: lipo -create -output > ; has this been > attempted? I actually do this for my library: <https://github.com/mratsim/constantine/blob/ad325cf/constantine.nimble#L221-L226> elif defined(maco

ARC vs refc

> That is only true if you increment len prior to initializing memory on thread > 1. Which I guess may happen due to re-ordering during optimization. Not only, if you're out of buffer space the second thread might hold a reference to the old buffer, and in between the read, thread 1 copies and f

ARC vs refc

> Here is an example scenario where this property is useful: > \- write into a > container on thread 1 (only append/add data e.g. I/O loop) > \- iterate > (read) over the container in thread 2 (e.g. UI thread) Use threadsafe containers or locks. You have a race condition here where if an item i

Nim ask for the Aptos Blockchain, nimAptos

NOT be used for cryptographic keys. Either use: * a hex parser supplied by libsodium if any * or nimcrypto's: <https://github.com/cheatfate/nimcrypto/blob/71bca15/nimcrypto/utils.nim#L58-L59> * or Constantine's (disclaimer, I'm the author): <https://github.com/mrat

Design Q: Why was result chosen? Why not re-use return?

something something about distinct keyword used for 2 different concepts ...

Can I use OpenACC pragma in nim?

Feel free to use the following PRs as inspiration of where to start: * *

Passing an optional proc is not working

> However, should the compiler not be able to infer the Foofunc signature > without the user specifying it? Alas, I agree. It's very annoying to work with high-order functions due to the extra ceremony.

Introducing an async library inspired by Go in Nim

@Alogani > Indeed moving a started coroutine can be (is certainly?) unsafe, because it > can contains GC memory, which will be hard to track. You can't move a started coroutine without preemption. You need the coroutine cooperation, i.e. it should have suspension points where its state it saved

Introducing an async library inspired by Go in Nim

regarding allocation and you wouldn't need this: <https://github.com/status-im/nim-taskpools/blob/d4c4313/taskpools/tasks.nim#L58-L62> The current Weave-IO is at the moment just a copy paste of Constantine's threadpool: <https://github.com/mratsim/constantine/tree/master/c

Introducing an async library inspired by Go in Nim

ost, but only stackful coroutines allows to suspend and resume execution > in arbitrary depthness if I am not wrong. You're not wrong but this is not a problem, I mention it there: <https://github.com/mratsim/weave-io-research/blob/master/design/design_1_coroutines.md> > Coroutines com

Introducing an async library inspired by Go in Nim

> Looks nice and has good ideas. But what's the benefit of this over just using > threads? Cooperative scheduling is bug-prone much like multi-threading is > IMHO without its performance benefits. Cooperative scheduling/fibers and threads are orthogonal. Threads are handled at the scheduler lev

Run a proc stored in an object in another thread with a threadpool

I have no doc in weave-io because I didn't change the internals to optimize for IO 😅 Do you have code? You say nimcall but it;s type Actor = ref object field1: int field2: int command: proc(x,y: int) Run instead of type Actor = ref obje

Rust interop/FFI experiments?

My cryptographic library accelerates Rust: * <https://github.com/mratsim/constantine/tree/9fa6c1c/constantine-rust> In particular this: <https://github.com/mratsim/constantine/blob/12f5686/constantine-rust/constantine-halo2-zal/src/lib.rs> Implements the traits: <htt

Fused multiply-add instruction

In general, for high performance computing, you might as well use SIMD directly for performance: see `fmadd` <https://github.com/mratsim/Arraymancer/blob/7d6d21c/src/arraymancer/laser/primitives/matrix_multiplication/gemm_ukernel_avx_fma.nim#L27> ukernel_gen

ref types and the align pragma

> C and C++ have aligned_alloc for a while now, which should work everywhere as > long as the compiler is somewhat up to date (C11/C++17). Just be aware of the different argument order on Windows. These are my aligned_alloc primitives: <https://github.com/mratsim/weav

Is it impossible to declare an _empty enum_ under quote in order to populate it later in a macro?

It's not hard to create the enum without quote do: <https://github.com/mratsim/trace-of-radiance/blob/master/trace_of_radiance/support/emulate_classes_with_ADTs.nim#L94C7-L132> # var enumValues = nnkEnumTy.newTree() enumValues.add newEmptyNode()

How to use Weave isReady?

You can try `weave-io` just in case. It supports runtime num_threads. Here is how to do a reduction: <https://github.com/mratsim/weave-io/blob/master/examples/e04_parallel_reduce.nim#L23> You can do per-task memory alloc/dealloc in the prologue and epilogue. There won't be more in-f

How to use Weave isReady?

> # Where is the setWeaveNumThreads(...) API?? putEnv before import weave? There is none. > My use case is having thousands of tasks, using the multiprocessor like a > worker pool to churn through jobs, but each running task has a huge amount of > memory and context data structures that take up

Is `concept` similar to `interface` / `trait`?

ctions: <https://github.com/mratsim/constantine/blob/58d8d2c/constantine/hashes.nim#L17-L38> type CryptoHash* = concept h, var ctx, type H ## Interface of a cryptographic hash function ## ## - digestSizeInBytes is the hash output s

Argon2 in Pure Nim.

You're creating `timeCost *syncPoint*parallelism` threads. Creating a thread is very costly and need to be amortized, a threadpool reuses them. You can do at least 5 xor-shift-rotate-add while waiting for the OS to create a _[single](https://forum.nim-lang.org/postActivity.xml#single) thread

Struggling to understand use of ptr object Types

A `ptr object` needs to be allocated first with `alloc`, `alloc0` and then initialized, like in C. There is no convenience for constructor-like procs, it's an escape hatch for low-level programming and if you need that escape hatch it's likely that even if the stdlib tried to provide sugar for y

Nim safety features like Zig & Rust?

If you overengineer for generality (allocators to cover each use-cases), you're not specializing, on the contrary. And you're going straight into architecture astronauting land and Java's Bean Factory Handler Manager ... *

Nim safety features like Zig & Rust?

I wrote in 2020 an article on Nim safety features for security auditors.

Argon2 in Pure Nim.

Quick comments after skimming. * First of all, good job debugging cryptography is quite annoying. * You will likely want to switch to a threadpool, you are creating way too many threads here, it's slow and will oversubscribed your system:

Exponentially-slow compile times with deeply nested iterators

> What do you mean by linked lists of moves and neighbors? I thought iterators > work on control flow, not allocations. It's not about low-level allocation / control-flow but high level algorithm and data structures. In go neighboring stones live or die together, or are captured together, in a

Exponentially-slow compile times with deeply nested iterators

https://gist.github.com/gcr/e0ae41005c8724bbedf15045e6e9b09a#file-main-nim-L108-L123> It's quite similar to Go and you might want to read on the CFG theory (Common Fate Graphs). Tracking those makes captures O(1) instead of having to process a long linked-list of moves and all their neighbors. <https

TCC on ARM(el) backend problem

Someone needs to implement compare-exchange on ARM, which likely needs to map to LL/SC:

A little guidance on threading needed.

Run You need a threadpool that supports data parallelism / parallel for. You can get away with OpenMP by using the `||` operator <https://nim-lang.org/docs/system.html#%7C%7C.i%2CS%2CT%2Cstaticstring> Using <https://github.com/mratsim/laser/blob/master/laser/openmp.nim

Cannot capture a Table[string, string]

Closures like this make my eyes bleed.

A little guidance on threading needed.

You'll have to tell us the algorithm you're trying to implement or context because: 1. If your threads only rarely need concurrent access to the array, you'll be fine, but you might as well use `std/atomics` `fetchAdd` and skip locks. 2. If you have constant concurrent access to the array, t

A cost model for Nim v2

> That made me curious on your perspective on Result types ala Rust (I think Go > has that idea as well but I haven't written a single line of code in it, so > no clue). For attribution, Haskell's `Maybe` and `Either` type and C# popularizing them via their own `Maybe` Monad inspired Rust. And

cleanly terminating a thread if it takes too long

Re `pthread_cancel`. See the doc: 1. It's useless if the code you run does not have cancellation points. 2. It does not reclaim resources (memory, database handles, files, ...) unless you provide cancellation clean-up handlers

Understanding an address sanitizer message for a memory leak caused by global dispatcher (?)

> Does anybody actually use address-sanitizers or valgrind or heaptrack and > eradicates potential memory leaks to the point that nothing shows up while > they run? Yes, I use sanitizers when developing: * Weave * nim-taskpools * Constantine and Constantine's threadpool However: * i

some MM modes leads to SIGSEGV

Clang + `--passC:-fsanitizer=address --passL:-fsanitizer=address` works fine on Windows.

why object variants not support same field name ?

It does if you have the same type type CommandId = enum cmdSearch cmdDownload type Command = object case id: CommandId of cmdSearch, cmdDownload: opts: int Run Otherwise, there are several RFCs related to variants, sum types and

Arraymancer - 2023-12-31 - Overview of longstanding missing features

effort that would > require? Do those libraries change a lot over time? The hardest BLAS function is already implemented and competitive: * OpenMP version: <https://github.com/mratsim/laser/tree/master/laser/primitives/matrix_multiplication> * Weave version: <https://github.com/m

ThreadButler - Multithreading with long-running threads that act as "servers"

ed some until I know enough whether that direction is promising or not: <https://github.com/mratsim/weave/tree/v0.1.0/experiments>

ThreadButler - Multithreading with long-running threads that act as "servers"

difference between Linux selectors/epoll vs io-uring (or it's also called push vs pull). There is a lot of litterature on design tradeoffs if you use those terms. I have a protocol description/implementation that can turn polling `isReady` (or pulling) into pushing: * protocol: &l

Arraymancer - 2023-12-31 - Overview of longstanding missing features

Hello everyone, It's been a while since I worked on Arraymancer, first of all thanks to all who contributed and maintained the library, for example to make it Nim v2 compatible. I've collated the big items I think are critical for the library in this issue: <https://githu

specify lib dir by `--passL` can not work

Can you try with either `--passL:"-L YOURPATH"` or `--clibdir:YOURPATH`. Seems likely that your environment has trouble converting -Le:/ into an actual path. Your path is neither e:used by normal windows or /e/path used by cygwin or gitbash

SIGSEGV: Illegal storage access. (Attempt to read from nil?) in coroutines

`std/coro` actually does stackful coroutines, i.e. fibers. A fiber needs to create it's own stack and switch to it. Nim threads use 2MB stacks, not sure what's the stack size of `std/coro` but it also needs to be sizeable. See also "Fibers under the magnifying glass":

SIGSEGV: Illegal storage access. (Attempt to read from nil?) in coroutines

Also every major language has abandoned fibers / green-threads / stackful coroutines for their standard library. * Go: * Java in 0.1 * Rust:

Parallelism in Nim: trying to call proc with my argument

> About edit: i experimented with for loops. We measure time for running > without threading and with that. Just for that cause i used CpuTime() :) If you have 8 threads and each spend a second to do something, the CPU time is 8s. What you're interested in is the wall time / elapsed time, use `

ref types and the align pragma

> If that's not possible, then maybe the align pragma should also apply to > types (currently it doesn't)? e.g. You can open a RFC for this, this should be possible as `alignas` can also work at the type level: > BTW, I found another thread f

ref types and the align pragma

I don't see a bug? The code is doing exactly what you asked. type MyArr = ref array[4, int] MyObj = ref object MyArrObj = ref object arr {.align(256).}: array[4, int] Run There is no constraint on `MyArr` and `MyObj` type at all.

Hello `nph`, an opinionated source code formatter for Nim

Does it work like `cargo fmt` for imports? If they are in adjacent lines, they are sorted alphabetically. If there is a blank line in-between, it creates a separate group for sorting? I use this in Rust projects to differentiate between `crate` imports, std/core imports and dependencies import.

Direct C/C++ interop without generating wrappers?

> Is it possible to pass paths (and use the "/" operator) to the compile > pragma? Is it possible to use strformat on those pragmas and on the .passC > pragma? I do pass paths in that example, and even const strings and globs. const rel_path = "./arcade_learning_environment/src/"

Direct C/C++ interop without generating wrappers?

ps://github.com/status-im/nim-secp256k1/blob/7246d91/secp256k1/abi.nim#L21-L23> * <https://github.com/arnetheduck/nim-sqlite3-abi/blob/362e1bd/sqlite3_abi/sqlite3_gen.nim#L29> You can even replace CMake with Nim: * <https://github.com/mratsim/agent-smith/blob/a2d9251/third_party/

How to correctly use nim doc when a project has unimported modules

RST->Json has been implemented in the compiler by @haxscramper here There is a large squashed PR with more work but can't review it:

scinim - how to contribute a package

Post in discord #science, with your Github handle, provided you're known in Nim community (which is the case) we'll add you to the org.

How to correctly use nim doc when a project has unimported modules

> I think I need to use docroot here, but I really do not wish to because this > eventually also needs to run on a github workflow which will have different > absolute paths to the project than I do. Welcome to nimble complete lunacy. Anyway I may have stumbled on similar issues on Arraymancer

error occurred when add thread to seq

A Thread cannot be copied only moved. It's a resource. > My guess is that the Thread object is allocated on the stack, but does not > support copy, which means that its lifetime cannot be safely expanded. Not, it just cannot be copied, you have to go through the OS to create/destroy it.

SIGSEGV: Illegal storage access. (Attempt to read from nil?) in coroutines

I don't think raw coroutines are copyable or movable anyway, they would need to be created in-place. And they are incompatible with GCs.

Capture atomics in closure

eed to capture the address of the var and the lock, and it would be even more verbose if `Matrix` was a seq, we would need to capture M[0, 0].addr as a workaround. <https://github.com/mratsim/weave/blob/b6255afa5816ee431dbf2f59cc6bc605d8d657b8/benchmarks/logsumexp/weave_logsumexp.nim#L242-L265&g

Can Nim interact with the hardware at a lower level than the C programming language?

> had checked out constantine for encrypting passwords. At the moment (and likely for a long long time) I don't support this. Use a password manager like Keepass that was audited and supports Argon2 or at the very least bcrypt or scrypt. OpenSSL also supports Argon2, but good luck with finding

Can Nim interact with the hardware at a lower level than the C programming language?

I have a wealth of libraries that Go very low-level, through C or syscalls or direct assembly. * <https://github.com/mratsim/photon-jit>. This is a proof-of-concept JIT assembler. Low-level includes mem-maping executable OS pages to put JIT-ted code in it: * <https://github.co

Hello `nph`, an opinionated source code formatter for Nim

> I think this is technically wrong, the only place where you can use both is > in parameter declaration lists and you can always use , here. nimpretty uses > what is used as the first occurence in the source code in order to decide but > it's not necessary as , is always valid. And in the gene

Package-Rot as growth hindrance

> Something that's more alarming to me is the lack of "versioned" packages, > only ~850 packages have a single version git tag associated with them. Iirc, Nimble has or had weird interactions between git tags and nimble versioning that forced updating tags to install the latest which at on point

Nim raw syscalls

Additionally you can add the `{.discardable.}` pragma if you want using the return value or explicitly discarding it to not be mandatory.

How to make a generic type parameter be late bound for evaluating proc parameter types?

usually `mixin` is my go-to but you never know when early symbol resolution can be delayed, and here it's not a symbol that we want to keep open.

Infiltrating Rust with Nim packages

Reporting back, Infiltration successful, we can now accelerate slow Rust with fast Nim ;) * Description: <https://github.com/mratsim/constantine/blob/master/docs/zk_accel_layer.md> * nimble needs to check the `OUT_DIR` env variable to put the libraries in the correct place:

What would I lose if I turn of garbage collection?

This is probably the largest Nim library written with manual memory management: * <https://github.com/mratsim/constantine> * See allocators: <https://github.com/mratsim/constantine/blob/0a17002/constantine/platforms/allocs.nim#L72-L134> That said for core cryptography, it

Capture atomics in closure

Atomics means you can update them concurrently (solving race conditions) but that's orthogonal to ownership (solving data races). You need to capture its address

Object variant - returning different types

If the return type is the same for all variants, I have a macro for this error, scroll to the bottom for a minimal example: <https://github.com/mratsim/trace-of-radiance/blob/master/trace_of_radiance/support/emulate_classes_with_ADTs.nim> If the return type is not the same you need to p

Nim Slack Community

Does Slack still has a 10k message limit? Would be very annoying to take a lot of time to answer something, and then the answer disappears after 3 weeks because of too much activity. Also in general what kind of audience do you want that would use Slack that doe snot use IRC, Matrix, Discord or

no simple way to import files from dir recusively?

> but i was also wondering why we have to import our own project's files at all > to begin with. explicit > implicit especially for maintenance and refactoring.

Why is building the community and ecosystem such a struggle?

Assume you have money, where would you spend it first to have the maximum impact?

related_post_gen benchmark

> I believe that's because currently the main thread participates as a worker, > so if the main thread is still working, no new work is started. The commit is here

threaded updates to sequence in objects?

`syncRoot` is not needed here because the first thing `exit(Weave)` does is `syncRoot` <https://github.com/mratsim/weave/blob/7682784/weave/runtime.nim#L178-L184> However, usually threadpools are started and exited at the very start and very end of the program because creating threads i

threaded updates to sequence in objects?

Here is the fixed Weave version: import weave type Count = object n: int v: int nseq: seq[int] func initCount(n, v: int): Count {.inline.} = #new(result) result.n = n result.v = v result.nseq = newSeq[int](n)

Alternative to powmod in Nim

Thanks :) On that front, Constantine as being integrated to Google continuous fuzzing framework OSS-fuzz in It's quite complete as it's fuzzed for both 32-bit and 64-bit.

Best way to architect a Nim dll/so that is used by other Nim dll/so/executables

Use the same `b_types.nim` file for types and import it in both?

Alternative to powmod in Nim

Following my last PR, I'm now **faster than GMP, even without assembly** on those small-medium-sized bigints. (with Clang, GCC codegen for bigints is meh)

ugly code or concepts?

You might want to have a look at various ways to implement reverse-mode autodifferentiation: 1\. Via callbacks, which is what you use: * <https://github.com/mratsim/nim-rmad/blob/2e9d0e1/src/math_ops.nim#L46-L48> * <https://github.com/explosion/thinc/blob/cf51ac5/thinc/layers/a

Using Concepts for abstracting display graphics

Are you sure the concept should be at the "Display" level and not at the "Color" level? For color conversions I've implemented a RGB concept here <https://github.com/mratsim/trace-of-radiance/blob/e928285/trace_of_radiance/io/color_conversions.nim#L77-L82>

And the fastest language to print hello world is ...

I've outlined 8 steps to be unbeatable speedwise on secp256k1 benchmark, a conservative expected speedup would be 2x over Rust implementation (current top) but I wouldn't be surprised by a 3x. Outline: <https://github.com/mratsim/constantine/issues/285>

And the fastest language to print hello world is ...

If someone wants to be coached into learning low-level cryptography to smoke the secp256k1 benchmark with Constantine (<https://github.com/mratsim/constantine>), there's quite a few things that can be done, and you start with a good baseline and comprehensive tests: For 2000

Details about procedures and their pointers

Use `cast`

Infiltrating Rust with Nim packages

Even simpler for me, I just need to call `nimble bindings` to delegate all the building to nimble: <https://github.com/mratsim/constantine/blob/977b6ee/constantine.nimble#L212-L230>

Infiltrating Rust with Nim packages

I'd like to publish a Rust frontend to a Nim library that works as if native when use from `cargo build` and `cargo run`. This means at one point, the `build.rs` script should call the Nim compiler. I'm wondering if there is an example of that, or even better a package like `cc` which abstract

Automatic Differentiation/Differentiable Programming Library

For ML (reverse-mode autodiff): * <https://github.com/can-lehmann/exprgrad> can differentiate any expression * <https://github.com/mratsim/arraymancer> can differentiate preconfigured layers And actually autodiff was my very first foray into Nim: <https://github.com/m

Malebolgia & Pixy problem: expression has no address

To work with images easiest would be to use a threadpool that supports parallelFor, see Weave: <https://github.com/mratsim/weave/blob/7682784/demos/raytracing/smallpt.nim#L271-L274> <https://github.com/mratsim/trace-of-radiance/blob/e928285c/trace_of_radiance/render.nim#L49-L68>

How to make custom linear containers automatically converted to OpenArray, like SEQ or Array

`Slice` is a bad name, it's already used for ranges. Use `Span` or `View`

nim documention

Use Dash/Zeal -

Using asm or intrinsic C functions from Nim

> I am looking into using the PDEP/PEXT machine instructions as part of Nim > proc:s and it seems to me that the asm keyword requires that you somehow know > which registers contains which variable. No you can use GCC/Clang extended ASM. <https://github.com/mratsim/finite

Inferring type of zero-sized collection from usage

This sounds like a feature that would complexify and slow down the compiler for no real use-case.

  1   2   3   4   5   6   >