@arnetheduck: > > enabled by a flag such as our --threads:on > > Global flags like these create dialects and don't work for composing > libraries in general which stifles development - this is by and large a > broken model that even C is trying to move away from (who remembers the mess > of linking single-vs-multithreaded C std libs?)
Then I guess we are currently a slave to our past, as one can't currently do anything with multi-threading without `--threads:on`, which automatically imports the `threads` and `channels_builtin` libraries and does all kinds of things internally, mostly to enable using `allocShared` instead of just `alloc` types of things, but also not allowing some libraries such as `locks`/`rlocks`, etc. unless threading is on. But you are right that it is a poor/broken model that has likely held Nim back by at least five years as to being a complete, modern, multi-threading language. Of course it is now recognized that using a thread local garbage collector and "handling" inter-thread passing of parameters/messages through all kinds of "compiler magic" cruft with all sorts of barriers `when/else` areas in the code base to try to ensure the right set of functions/options get enabled when they need to be means that it some combination is missed, then all kinds of weird errors can arise. That is why we are moving to ARC/ORC as our main memory management model, but all those legacy conditions are still in the code. I just encountered one of these, even using ARC/ORC: usually we can get memory stats with `getOccupiedMem()`, `getTotalMem()`, and `getFreeMem()` and these still work with ARC/ORC; however with threading turned on, these are still there but return zeros as it is expected that one would use `getOccupiedSharedMem()`, `getTotalsharedMem()`, and `getFreeSharedMem()`. The trouble is, as these latter functions are behind a `when` barrier to be only compiled when threading is enabled and documentation is generated without threading, the source document comments don't show up anywhere other than in the source code. It is also a design error to expect that "special" functions be used for particular conditions such as when threading is enabled, as the functionality should just automatically be included in the original functions when called with the same name, so code "just works" whether threading is on or not. As per your point. > In general, with regards to thread pools, what's really cool is when a > language supports the primitives necessary to build thread pools outside of > the language and the thread pool doesn't have to be distributed with the > language itself - the primitives are unlikely to change, while thread pools > continue to become better, inevitably making the old ones obsolete - it would > be interesting to hear what _minimal support you need from the language to > build your thread pool - then everyone can choose whether to use a Future[T] > based pool or not depending on the use case. In our Nim, all of the threading primitives are in the `threads` module, and in my mind that is all that should be required to build a standard library thread pool module or an alternative. If the primitives in the `threads` module are incomplete, then its API should be enhanced to cover what is required. I agree with you that almost everything should be able to be implemented as external modules, as I think @araq does too. The only "base" module that is absolutely essential in multi-threading is the low-level `threads` module, as the new (better once it is fixed) `channels` module is already a separate import just as the `threadpool`/new `threadpools` modules always have been separate modules. The older `threadpool` module required all kinds of (now being depreciated) "compiler magic" in order to work, but that no longer needs to be used as, in an effort to unify the compiler checks for what types are supported for transfer/moving inter-thread (no copying allowed), the new `std/isolation` module is now available. Any new library that uses the contained `Isolated[T]` type as the type for inter-thread transfers will have the guarantee that the parameter type has been moved (the new `channels` module already uses that), as should the new `std/threadpools` module use it as well. The problem still is that the automatically imported `threads` module still does "compiler magic" cruft including deep copies as it deems necessary for parameters to thread `proc`'s/`func`'s, which (I suppose) "compiler magic" has been modified so that it supports the types used by ARC/ORC. I suppose if we could turn back the clock and ignore that Nim ever was a garbage collected language, then the `threads` module would be a separate independently imported module and an application would be multi-threaded when this module was imported (whether directly or indirectly) without any external flags needing to be set. Then, if one wants to use channels, they can import the `std/channels` module or use an alternative, and if one wants a thread pool, they can import the `std/threadpools` module or use an alternative to that, too. In my provisional implementation of a `std/threadpools` module, I don't use anything other than the current low-level `threads` module (the main lack being a way to kill/abort/force an exception for a OS thread, meaning that a "non well behaved task" can take over the OS thread without ever returning it to be available to handle other thread pool threads and without being able to force kill it from inside Nim); in order to make it easy to implement; I also use the existing (fixed) `std/channels` along with the `std/isolation` module it requires anyway. If the `threads` primitives were just a little more complete, I don't see that any other primitives would be required **as long as we are talking just basic thread pools and not "green threads"/co-routines/all the variations in between, which would require "compiler magic" to be competitive with such as Go lang has**. By the definition that "compiler threading modes are bad", then the whole importing of special modules enabled by a command line flag was wrong. So in the language design, I guess the open question is **" Can we change so that we don't need compiler flags to turn on threading by Nim version 2.0?, at least when ARC/ORC are used"**
