Re: First time using Parallel
On Sunday, 26 December 2021 at 06:10:03 UTC, Era Scarecrow wrote:
This is curious. I was up for trying to parallelize my code,
specifically having a block of code calculate some polynomials
(*Related to Reed Solomon stuff*). So I cracked open
std.parallel and looked over how I would manage this all.
To my surprise I found ParallelForEach, which gives the
example of:
```d
foreach(value; taskPool.parallel(range) ){code}
```
Since my code doesn't require any memory management, shared
resources or race conditions (*other than stdout*), I plugged
in an iota and gave it a go. To my amazement no compiling
issues, and all my cores are in heavy use and it's outputting
results!
Now said results are out of order (*and early results are
garbage from stdout*), but I'd included a bitwidth comment so
sorting should be easy.
```d
0x3,/*7*/
0x11, /*9*/
0x9,/*10*/
0x1D, /*8*/
0x5,/*11*/
0x3,/*15*/
0x53, /*12*/
0x1B, /*13*/
0x2B, /*14*/
```
etc etc.
Previously years ago I remember having to make a struct and
then having to pass a function and a bunch of stuff from within
the struct, often breaking and being hard to get to even work
so I didn't hardly touch this stuff. This is making outputting
data MUCH faster and so easily; Well at least on a beefy
computer and not just some chromebook I'm programming on so it
can all be on the go.
So I suppose, is there anything I need to know? About shared
resources or how to wait until all threads are done?
Parallel programming is one of the deepest rabbit holes you can
actually get to use in practice. Your question at the moment
doesn't really have much context to it so it's difficult to
suggest where you should go directly.
I would start by removing the use of stdout in your loop kernel -
I'm not familiar with what you are calculating, but if you can
basically have the (parallel) loop operate from (say) one array
directly into another then you can get extremely good parallel
scaling with almost no effort.
Not using in the actual loop should make the code faster even
without threads because having a function call in the hot code
will mean compilers optimizer will give up on certain
transformations - i.e. do all the work as compactly as possible
then output the data in one step at the end.
Re: First time using Parallel
On 27/12/2021 12:10 AM, max haughton wrote: I would start by removing the use of stdout in your loop kernel - I'm not familiar with what you are calculating, but if you can basically have the (parallel) loop operate from (say) one array directly into another then you can get extremely good parallel scaling with almost no effort. Not using in the actual loop should make the code faster even without threads because having a function call in the hot code will mean compilers optimizer will give up on certain transformations - i.e. do all the work as compactly as possible then output the data in one step at the end. It'll speed it up significantly. Standard IO has locks in it. So you end up with all calculations grinding to a half waiting for another thread to finish doing something.
Re: How to gets multi results using tuple in D?
On Thursday, 23 December 2021 at 08:56:36 UTC, WebFreak001 wrote:
On Thursday, 23 December 2021 at 08:33:17 UTC, zoujiaqing wrote:
C++ Code:
```cpp
std::tuple DoIt()
{
return {false, 0, 0, "Hello"};
}
auto [r1, r2, r3, r4] = DoIt();
if (r1 == false)
```
D Code:
```D
Tuple!(bool, int, int, string) DoIt()
{
return [false, 1, 1, "Hello"];
}
auto result = DoIt();
auto r1= result[0];
auto r2= result[1];
auto r3= result[2];
auto r3= result[3];
if (r1 == false)
```
D requires more lines of code.
I think this is the best thing you can do:
```d
bool r1;
int r2, r3;
string r4;
AliasSeq!(r1, r2, r3, r4) = DoIt();
```
https://forum.dlang.org/thread/kmugmwmduxeoyfffo...@forum.dlang.org
Thanks! It's not concise!
Re: First time using Parallel
On Sunday, 26 December 2021 at 06:10:03 UTC, Era Scarecrow wrote:
[...]
```d
foreach(value; taskPool.parallel(range) ){code}
```
[...]
Now said results are out of order
[...]
So I suppose, is there anything I need to know? About shared
resources or how to wait until all threads are done?
Have a look at `taskPool.workerLocalStorage`. I learned about
this in [a post by data
pulverizer](https://forum.dlang.org/post/ddgxqoitxoaljfwnl...@forum.dlang.org), who gives this example (slightly modified):
```d
import std.traits : isFloatingPoint;
auto dot(T)(T[] x, T[] y) if (isFloatingPoint!T)
in (y.length == x.length)
{
import std.range : iota;
import std.parallelism : parallel, taskPool;
auto sums = taskPool.workerLocalStorage(0.0L);
foreach (i; parallel(iota(x.length)))
sums.get += x[i] * y[i];
T result = 0.0;
foreach (threadResult; sums.toRange)
result += threadResult;
return result;
}
void main()
{
double[] x = [1, 2, 3, 4, 5];
double[] y = [6, 7, 8, 9, 10];
assert(dot(x, y) == 130);
}
```
(https://run.dlang.io/is/Ia8A0k)
So if you use `workerLocalStorage` to give each thread an
`appender!string` to write output to, and afterwards write those
to `stdout`, you'll get your output in order without sorting.
--Bastiaan.
Re: First time using Parallel
On Sunday, 26 December 2021 at 11:24:54 UTC, rikki cattermole
wrote:
I would start by removing the use of stdout in your loop kernel
- I'm not familiar with what you are calculating, but if you
can basically have the (parallel) loop operate from (say) one
array directly into another then you can get extremely good
parallel scaling with almost no effort.
I'm basically generating a default list of LFSR's for my Reed
Solomon codes. LFSR can be used in Pseudo random numbers, but in
this case it's to build a Galois field for Error Correction.
Using it is simple, you need to know a binary number that when
xored when a 1 bit exits the range, will result in the maximum
numbers (*excluding zero*). So if we do 4 bits (xor of 3) you'd
get:
```
0 0001 -- initial
0 0010
0 0100
0 1000
1 0011 <-
0 0110
0 1100
1 1011 <- 1000
1 0101 <- 0110
0 1010
1 0111 <- 0100
0 1110
1 <- 1100
1 1101 <- 1110
1 1001 <- 1010
1 0001 <- 0010 -- back to our initial value
```
As such the bulk of the work is done in this function. Other
functions leading to this mostly figure out what value should be
according to some rules i set before trying to work (*quite a few
only need 2 bits on*).
```d
bool testfunc(ulong value, ulong bitswide) {
ulong cnt=1, lfsr=2,up=1UL
Re: First time using Parallel
On Sunday, 26 December 2021 at 15:20:09 UTC, Bastiaan Veelo wrote: So if you use `workerLocalStorage` to give each thread an `appender!string` to write output to, and afterwards write those to `stdout`, you'll get your output in order without sorting. Scratch that, I misunderstood the example. It doesn't solve ordering. The example works because order does not matter for addition. Sorry for spreading wrong information. -- Bastiaan.
Re: Double bracket "{{" for scoping static foreach is no longer part of D
On Wednesday, 22 December 2021 at 16:30:06 UTC, data pulverizer
wrote:
On Wednesday, 22 December 2021 at 16:10:42 UTC, Adam D Ruppe
wrote:
So OUTSIDE a function, static foreach() {{ }} is illegal
because a plain {} is illegal outside a function.
But INSIDE a function, static foreach() {{ }} is legal, but it
isn't magic about static foreach - it is just a body with its
optional {} present as well as a scope statement inside.
Just seen this. Thanks - I should have been more patient.
I thought the {{ }} was mostly related to static if, namely that
when you do static if, the block contents is added in scope; So
if you needed a scope you'd do the second bracket as the
outer/first one is stripped out.
I need to once again re-familiarize myself more with D. It's
been too long.
Re: First time using Parallel
On Sunday, 26 December 2021 at 15:36:54 UTC, Bastiaan Veelo wrote: On Sunday, 26 December 2021 at 15:20:09 UTC, Bastiaan Veelo wrote: So if you use `workerLocalStorage` ... you'll get your output in order without sorting. Scratch that, I misunderstood the example. It doesn't solve ordering. The example works because order does not matter for addition. Sorry for spreading wrong information. Maybe. I did notice that the early stuff a bunch of output was getting mixed up; ``` 0x 0x 0x 0x 0x 0x 0x 0x35/*33, /*, /*, /*115, /*, /*3, / *9, /*3410*/*/ ``` Which i assume it's doing several small write calls and different threads are acting at the same time. So if I do an appender string and then outputted the string as a single bock that would likely go away; Though it wouldn't help with ordering. **IF** I didn't have to wait so long to get results and wanted them all at once in order, I would write the results to the offsets of an array and then output it all at once at the end (*and since they'd have their own offset to write to you don't need to lock*).
How to print unicode characters (no library)?
Hi! I'm trying to print some Unicode characters using UTF-8
(char), UTF-16 (wchar) and UTF-32 (dchar). I want to do this
without using any library by using the "write" system call
directly with 64-bit Linux. Only the UTF-8 solution seems to be
working as expected. The other solutions will not print the
unicode characters (I'm using an emoji in my case for example).
Another thing I noticed is the size of the strings. From what I
know (and tell me if I'm mistaken), UTF-16 and UTF-32 have fixed
size lengths for their characters. UTF-16 uses 2 bytes (16 bits)
and UTF-32 uses 4 bytes (32 bits) without treating any character
specially. This doesn't seem to be the case for me however.
Consider my code:
```
import core.stdc.stdio;
void exit(ulong code) {
asm {
"syscall"
: : "a" (60), "D" (code);
}
}
void write(T)(int fd, const T buf, ulong len) {
asm {
"syscall"
: : "a" (1), "D" (1), "S" (buf), "d" (len)
: "memory", "rcx";
}
}
extern (C) void main() {
string utf8s = "Hello 😂\n";
write(1, utf8s.ptr, utf8s.length);
wstring utf16s = "Hello 😂\n"w;
write(1, utf16s.ptr, utf16s.length * 2);
dstring utf32s = "Hello 😂\n"d;
write(1, utf32s.ptr, utf32s.length * 4);
printf("\nutf8s.length = %lu\nutf16s.length =
%lu\nutf32s.length = %lu\n",
utf8s.length, utf16s.length, utf32s.length);
exit(0);
}
```
And its output:
```
Hello 😂
Hello =��
Hello �
utf8s.length = 11
utf16s.length = 9
utf32s.length = 8
```
Now the UTF-8 string will report 11 characters and print them
normally. So it treats every character that is 127 or less as if
it was an ascii character and uses 1-byte for it. Characters
above that range, are either a 2-byte or 4-byte unicode
characters. So it works as I expected based on what I've read/saw
for UTF-8 (now I understand why everyone loves it, lol :P)!
Now what about the other two? I was expecting UTF-16 to report 16
characters and UTF-32 to report 32 characters. Also why the
characters are not shown as expected? Isn't the "write" system
call just writing a sequence of characters without caring which
they are? So if I just give it the right length, shouldn't it
just work? I'm pretty much sure that this is not as I expect it
and it doesn't work like that. Anyone has an idea?
Re: How to print unicode characters (no library)?
On Sunday, 26 December 2021 at 20:50:39 UTC, rempas wrote: I want to do this without using any library by using the "write" system call directly with 64-bit Linux. write just transfers a sequence of bytes. It doesn't know nor care what they represent - that's for the receiving end to figure out. know (and tell me if I'm mistaken), UTF-16 and UTF-32 have fixed size lengths for their characters. You are mistaken. There's several exceptions, utf-16 can come in pairs, and even utf-32 has multiple "characters" that combine onto one thing on screen. I prefer to think of a string as a little virtual machine that can be run to produce output rather than actually being "characters". Even with plain ascii, consider the backspace "character" - it is more an instruction to go back than it is a thing that is displayed on its own. Now the UTF-8 string will report 11 characters and print them normally. This is because the *receiving program* treats them as utf-8 and runs it accordingly. Not all terminals will necessarily do this, and programs you pipe to can do it very differently. Now what about the other two? I was expecting UTF-16 to report 16 characters and UTF-32 to report 32 characters. The [w|d|]string.length function returns the number of elements in there, which is bytes for string, 16 bit elements for wstring (so bytes / 2), or 32 bit elements for dstring (so bytes / 4). This is not necessarily related to the number of characters displayed. Isn't the "write" system call just writing a sequence of characters without caring which they are? yes, it just passes bytes through. It doesn't know they are supposed to be characters...
Re: How to print unicode characters (no library)?
On Sunday, 26 December 2021 at 21:22:42 UTC, Adam Ruppe wrote: On Sunday, 26 December 2021 at 20:50:39 UTC, rempas wrote: [...] write just transfers a sequence of bytes. It doesn't know nor care what they represent - that's for the receiving end to figure out. [...] You are mistaken. There's several exceptions, utf-16 can come in pairs, and even utf-32 has multiple "characters" that combine onto one thing on screen. I prefer to think of a string as a little virtual machine that can be run to produce output rather than actually being "characters". Even with plain ascii, consider the backspace "character" - it is more an instruction to go back than it is a thing that is displayed on its own. [...] This is because the *receiving program* treats them as utf-8 and runs it accordingly. Not all terminals will necessarily do this, and programs you pipe to can do it very differently. [...] The [w|d|]string.length function returns the number of elements in there, which is bytes for string, 16 bit elements for wstring (so bytes / 2), or 32 bit elements for dstring (so bytes / 4). This is not necessarily related to the number of characters displayed. [...] yes, it just passes bytes through. It doesn't know they are supposed to be characters... I think that mental model is pretty good actually. Maybe a more specific idea exists, but this virtual machine concept does actually explain to the new programmer to expect dragons - or at least that the days of plain ASCII are long gone (and never happened, e.g. backspace as you say)
Re: How to print unicode characters (no library)?
On Sun, Dec 26, 2021 at 11:45:25PM +, max haughton via Digitalmars-d-learn wrote: [...] > I think that mental model is pretty good actually. Maybe a more > specific idea exists, but this virtual machine concept does actually > explain to the new programmer to expect dragons - or at least that the > days of plain ASCII are long gone (and never happened, e.g. backspace > as you say) In some Unix terminals, backspace + '_' causes a character to be underlined. So it's really a mini VM, not just pure data. So yeah, the good ole ASCII days never happened. :-D T -- This sentence is false.
Re: How to print unicode characters (no library)?
On Sunday, 26 December 2021 at 21:22:42 UTC, Adam Ruppe wrote: write just transfers a sequence of bytes. It doesn't know nor care what they represent - that's for the receiving end to figure out. Oh, so it was as I expected :P You are mistaken. There's several exceptions, utf-16 can come in pairs, and even utf-32 has multiple "characters" that combine onto one thing on screen. Oh yeah. About that, I wasn't given a demonstration of how it works so I forgot about it. I saw that in Unicode you can combine some code points to get different results but I never saw how that happens in practice. If you combine two code points, you get another different graph. So yeah that one thing I don't understand... I prefer to think of a string as a little virtual machine that can be run to produce output rather than actually being "characters". Even with plain ascii, consider the backspace "character" - it is more an instruction to go back than it is a thing that is displayed on its own. Yes, that's a great way of seeing it. I suppose that this all happens under the hood and it is OS specific so why have to know how the OS we are working with works under the hood to fully understand how this happens. Also the idea of some "characters" been "instructions" is very interesting. Now from what I've seen, non-printable characters are always instructions (except for the "space" character) so another way to think about this is by thinking that every character can have one instruction and this is either to get written (displayed) in the file or to do another modification in the text but without getting displayed itself as a character. Of course, I don't suppose that's what happening under the hood but it's an interesting way of describe it. This is because the *receiving program* treats them as utf-8 and runs it accordingly. Not all terminals will necessarily do this, and programs you pipe to can do it very differently. That's pretty interesting actually. Terminals (and don't forget shells) are programs themselves so they choose the encoding themselves. However, do you know what we do from cross compatibility then? Because this sounds like a HUGE mess real world applications The [w|d|]string.length function returns the number of elements in there, which is bytes for string, 16 bit elements for wstring (so bytes / 2), or 32 bit elements for dstring (so bytes / 4). This is not necessarily related to the number of characters displayed. I don't understand that. Based on your calculations, the results should have been different. Also how are the numbers fixed? Like you said the amount of bytes of each encoding is not always standard for every character. Even if they were fixed this means 2-bytes for each UTF-16 character and 4-bytes for each UTF-32 character so still the numbers doesn't make sense to me. So still the number of the "length" property should have been the same for every encoding or at least for UTF-16 and UTF-32. So are the sizes of every character fixed or not? Damn you guys should got paid for the help you are giving in this forum
Re: How to print unicode characters (no library)?
On Sunday, 26 December 2021 at 23:57:47 UTC, H. S. Teoh wrote: In some Unix terminals, backspace + '_' causes a character to be underlined. So it's really a mini VM, not just pure data. So yeah, the good ole ASCII days never happened. :-D T How can you do that? I'm trying to print the codes for them but it doesn't work. Or you cannot choose to have this behavior and there are only some terminals that support this?
