Yesterday when testing performance and memory on my web framework
[KommandKit](https://git.termer.net/termer/KommandKit), I realized something
very shocking -- when compiling with `--mm:refc`, the server stopped leaking
memory and stayed at a steady ~340MiB RSS with 10k connections going max speed.
Furthermore, the server was getting about 30% better performance, if not more,
and had lower latency.
I saw a similar pattern when compiling some of my other web projects with
`refc` instead of `orc`. This is extremely surprising to me, seeing as ORC is
supposed to improve memory usage and performance, yet I'm seeing a massive
penalty in both categories.
Combine this with ORC-related crashes that aren't present in `refc` and the
general instability I'm seeing with it, I believe ORC simply isn't
production-ready. The worst effects are on multithreaded asynchronous code,
which sounds isolated, but this makes Nim basically unusable for networking
applications under ORC.
To illustrate my point, here are two benchmarks I did on the same application.
`nimble -d:release --mm:refc build`
~/ ❯❯❯ bombardier -c 1000 -d 1m -t 10m http://localhost:8070/
Bombarding http://localhost:8070/ for 1m0s using 1000 connection(s)
[===================================================] 1m0s
Done!
Statistics Avg Stdev Max
Reqs/sec 107543.33 6650.47 138119.83
Latency 9.30ms 2.29ms 323.97ms
HTTP codes:
1xx - 0, 2xx - 0, 3xx - 0, 4xx - 6450428, 5xx - 0
others - 0
Throughput: 34.14MB/s
# Highest RSS: ~1,348MiB
Run
`nimble -d:release --mm:orc build`
~/ ❯❯❯ bombardier -c 1000 -d 1m -t 10m http://localhost:8070/
Bombarding http://localhost:8070/ for 1m0s using 1000 connection(s)
[===================================================] 1m0s
Done!
Statistics Avg Stdev Max
Reqs/sec 5876.87 2818.37 19515.32
Latency 170.05ms 150.70ms 4.66s
HTTP codes:
1xx - 0, 2xx - 0, 3xx - 0, 4xx - 353185, 5xx - 0
others - 0
Throughput: 1.87MB/s
# Highest RSS: ~71MiB
Run
This program is running 12 instances of AsyncHttpServer on 12 threads, which
returns a static JSON response.
The performance and memory difference is astounding, and overwhelmingly
condemns ORC. With that in mind, I think ORC is basically unusable in its
current state. I don't care if this supposedly only affects async code, or
asyncnet, or threading, or whatever; this represents serious flaws in ORC that
are not present in `refc`, and I doubt those bugs will only affect the areas
I'm working in.
With all of that said, this makes recommending Nim to others for anything
async/threading/network related impossible. Also, with Nim 2.0 coming up, I
don't want to lock myself into `refc` just to write software that works
properly, because everything else will be moving forward with ORC. That in and
of itself is going to be a huge disaster if there isn't an extremely focused
effort on making ORC work properly. Up until now, I honestly thought that
AsyncHttpServer just sucked really bad, but that seems to be the case only
because I was running it under ORC.
In its current form, ORC is **unusable**. The only benefits of using ORC are
the ones that come from ARC, in which case, you should use ARC and be very very
careful that your code doesn't contain cycles.
