Re: Reading binary streams with decoding to Unicode

2018-10-16 Thread Steven Schveighoffer via Digitalmars-d-learn
On 10/16/18 11:42 AM, Vinay Sajip wrote: On Monday, 15 October 2018 at 22:49:31 UTC, Nicholas Wilson wrote: Oh, sorry I missed that. Take a look at https://github.com/schveiguy/iopipe Great, thanks. Let me know if anything doesn't work there. The text processing is pretty robust, but haven'

Re: Reading binary streams with decoding to Unicode

2018-10-16 Thread Vinay Sajip via Digitalmars-d-learn
On Monday, 15 October 2018 at 22:49:31 UTC, Nicholas Wilson wrote: Oh, sorry I missed that. Take a look at https://github.com/schveiguy/iopipe Great, thanks.

Re: Reading binary streams with decoding to Unicode

2018-10-15 Thread Nicholas Wilson via Digitalmars-d-learn
On Monday, 15 October 2018 at 21:48:05 UTC, Vinay Sajip wrote: On Monday, 15 October 2018 at 19:56:22 UTC, Nicholas Wilson wrote: import std.file : readText; import std.uni : byCodePoint, byGrapheme; // or import std.utf : byCodeUnit, byChar /*utf8*/, byWchar /*utf16*/, byDchar /*utf32*/, byUT

Re: Reading binary streams with decoding to Unicode

2018-10-15 Thread Vinay Sajip via Digitalmars-d-learn
On Monday, 15 October 2018 at 19:56:22 UTC, Nicholas Wilson wrote: import std.file : readText; import std.uni : byCodePoint, byGrapheme; // or import std.utf : byCodeUnit, byChar /*utf8*/, byWchar /*utf16*/, byDchar /*utf32*/, byUTF /*utf8(?)*/; string a = readText("foo"); foreach(cp; a.byCo

Re: Reading binary streams with decoding to Unicode

2018-10-15 Thread Nicholas Wilson via Digitalmars-d-learn
On Monday, 15 October 2018 at 18:57:19 UTC, Vinay Sajip wrote: On Monday, 15 October 2018 at 17:55:34 UTC, Dukc wrote: This is done automatically for character arrays, which includes strings. wchar arrays wil iterate by UTF-16, and dchar arrays by UTF-32. If you have a byte/ubyte array you kno

Re: Reading binary streams with decoding to Unicode

2018-10-15 Thread Vinay Sajip via Digitalmars-d-learn
On Monday, 15 October 2018 at 17:55:34 UTC, Dukc wrote: This is done automatically for character arrays, which includes strings. wchar arrays wil iterate by UTF-16, and dchar arrays by UTF-32. If you have a byte/ubyte array you know to be unicode-encoded, convert it to char[] to iterate by code

Re: Reading binary streams with decoding to Unicode

2018-10-15 Thread Dukc via Digitalmars-d-learn
On Monday, 15 October 2018 at 10:49:49 UTC, Vinay Sajip wrote: Is there a standardised way of reading over buffered binary streams (at least strings, files, and sockets) where you can layer a decoder on top, so you get a character stream you can read one Unicode char at a time? Initially UTF-8,

Reading binary streams with decoding to Unicode

2018-10-15 Thread Vinay Sajip via Digitalmars-d-learn
Is there a standardised way of reading over buffered binary streams (at least strings, files, and sockets) where you can layer a decoder on top, so you get a character stream you can read one Unicode char at a time? Initially UTF-8, but later also other encodings. I see that std.stream was depr