Thanks for the reply; that cleared up quite a few things for me. > **We 're getting a new channel implementation.**
I must apologize in my mentioning an issue I had with the use of `Channel[T]` with `Isolated[T]`: I didn't notice that there is a `std/channels` library that is meant to be used with `isolated` with `--gc:arc/orc`, which of course works without the issue I was having. Again, just a lack in the documentations which I am preparing to fix. **Is the new `std/channels` the new implementation?** So I've been trying out the new `std/channels` and see that it's a lot cleaner to use than the old `system.channels` in being able to be passed other than by pointers. My only negative observation is the hard limit on the maximum number of channels as in the `nimChannelCacheSize` constant, which just like for the `MaxDistinguishedThread` in `threadpool` can be worked around just by patching as necessary, but still... > They don't: (Chapel doesn't do disjoint checking, etc.) I didn't think it would, but thanks for looking it up for me which also shows you are familiar with the language. I've been playing with Chapel a bit lately to learn more about how HPC languages deal with multi-processing. Don't really like it that much as compared to Nim (compiles about five times slower, compulsory curly braces and semi-colons, OOP, no closures, and I think not type conversion complete in type promotions/reductions for non-primitive generic classes/records, etc.); but the multi-processing bits work pretty well, which of course they must as that's the reason for its existence. > You don't need a disjoint checker, you can declare an alternative parallel > construct to be unsafe. And then it can be done as a library. So there's no reason that Nim can't do the same, which would be quite a bit easier than the current experimental `parallel` implementation requiring "compiler magic". > > Is it on the back burner for now? > > Yeah. I, for one, have never even tried it, as things like `parallel`/`forall`, etc. are mostly useful for big array calculations in parallel where one doesn't care in what order the computations are run, and those still need to be carefully designed so that the individual `spawn`'s are big enough to be worth the cost while small enough to be able to spawn thousands to millions of threads. If I had a project that required something like that, I think I would just use @mratsim's Weave library that offers work grouping/stealing, etc. facilities so as to be more in control than anything that could be provided by single simple constructs like these. Not that these constructs aren't/couldn't be used, but the work to use them properly might be more than just using @mratsim's library. Thus, I think `parallel` should be on the long term experimental back burner just as you say, with maybe a new place holder built with a macro in **the `std/threadpool` library that should replace `system.threadpool`** in the same way and for the same use that `std/channels` replaces `system.channels`, **which new library I don 't think should be on a back burner as we should be able to use "green" thread pool threads with the new channels library without having to build a thread pool ourselves.** Now having tried the new `std/channels` library and seeing that the new `FlowVar[T]` can be nothing but a synonym for a new unbuffered `Channel[T]` with all the "compiler magic" applied in the implementation of `Isolated[T]` which it uses, it doesn't look so hard to write a `std/threadpool`. This would follow the API of the current `threadpool` library but not necessarily exactly, just as the new `std/channels` breaks the API of the old library in some minor ways. In order to avoid "compiler magic" other than as already supplied by `Isolated[T]`, the `spawn` functionality would be provided by templates/macros, as could the stub for the eventually more complete `parallel`. I've written a quicky Proof of Concept and it seems to work, with likely some details to be worked out. As you say, the great thing about it is it's a library built on top of the other core libraries so doesn't need any "compiler magic" help at all other than as that already exists in term rewriting template/macro facilities. I think I'll give it a shot in writing the full thing and see what you think. My early tests seem to indicate comparable performance to doing the same thing in other languages as in calling "green threads" from a thread pool and passing messages between them - spawning a green thread to pass a single simple integer message runs at a rate in the order of a hundred thousand a second. GoLang is faster in that it uses "compiler magic" to inject the "green thread" code directly into the code flow, but we don't like that method much and the factor of about ten improvement isn't going to affect most uses of it (by us in Nim).
