Re: Reading unicode chars..
On Wed, 03 Sep 2014 15:41:59 + seany via Digitalmars-d-learn wrote: `foreach (dchar aaa; aa)`... signature.asc Description: PGP signature
Re: Reading unicode chars..
On Tuesday, 2 September 2014 at 21:13:04 UTC, monarch_dodra wrote: What's wrong with reading line by line, but processing the characters in said lines 1 by 1? That works "out of the box". import std.stdio; import std.conv; import core.vararg; void main() { string aa = "abc „"; foreach (aaa; aa) writeln(aaa); } output : a b c � � � Linux 64 bit netrunner, console font set to Raleway,, then to Quivira - both supports the character in discussion.
Re: Reading unicode chars..
On Wednesday, 3 September 2014 at 13:54:30 UTC, Ali Çehreli wrote: On 09/03/2014 01:21 AM, "Marc Schütz" " wrote: 1) Adding attributes to function calls which I know some are unsafe (see assumeHasAttribs() below). For example, I don't think getc() should be pure. (?) Also, how could all of its functions be nothrow? Is byDchar() is asking too much of its users? https://github.com/D-Programming-Language/phobos/pull/2483 Yay! Makes so much sense... :) I was going to suggest to the OP exactly what you suggested, but I thought I'd better check whether it actually works. Turns out it didn't :-P
Re: Reading unicode chars..
On 09/03/2014 01:21 AM, "Marc Schütz" " wrote: 1) Adding attributes to function calls which I know some are unsafe (see assumeHasAttribs() below). For example, I don't think getc() should be pure. (?) Also, how could all of its functions be nothrow? Is byDchar() is asking too much of its users? https://github.com/D-Programming-Language/phobos/pull/2483 Yay! Makes so much sense... :) Ali
Re: Reading unicode chars..
On Tuesday, 2 September 2014 at 23:20:38 UTC, Ali Çehreli wrote: On 09/02/2014 02:13 PM, monarch_dodra wrote: > I'd suggest you create a range out of your std.stream.File, which reads > it byte by byte. I was in the process of doing just that. > Then, you pass it to the "byDchar()" range, which will > auto decode those characters. If you really want to do it "character by > character". I first started writing my own byDchar but then used std.utf.byDchar as you suggest. However, I had to resort to 1) Adding attributes to function calls which I know some are unsafe (see assumeHasAttribs() below). For example, I don't think getc() should be pure. (?) Also, how could all of its functions be nothrow? Is byDchar() is asking too much of its users? https://github.com/D-Programming-Language/phobos/pull/2483
Re: Reading unicode chars..
On 09/02/2014 02:13 PM, monarch_dodra wrote: > I'd suggest you create a range out of your std.stream.File, which reads > it byte by byte. I was in the process of doing just that. > Then, you pass it to the "byDchar()" range, which will > auto decode those characters. If you really want to do it "character by > character". I first started writing my own byDchar but then used std.utf.byDchar as you suggest. However, I had to resort to 1) Adding attributes to function calls which I know some are unsafe (see assumeHasAttribs() below). For example, I don't think getc() should be pure. (?) Also, how could all of its functions be nothrow? Is byDchar() is asking too much of its users? 2) I also had to make StreamRange a template just to get attribute inference from the compiler. import std.stdio; import std.stream; import std.utf; import std.traits; auto assumeHasAttribs(T)(T t) pure if (isFunctionPointer!T || isDelegate!T) { enum attrs = functionAttributes!T | FunctionAttribute.pure_ | FunctionAttribute.nogc | FunctionAttribute.nothrow_; return cast(SetFunctionAttributes!(T, functionLinkage!T, attrs)) t; } /* This is a template just to take advantage of compiler's attribute * inference. */ struct StreamRange() { std.stream.File f; char c; this(std.stream.File f) { this.f = f; prime(); } private void prime() { if (!empty()) { c = assumeHasAttribs(&(f.getc))(); } } @property bool empty() const { return assumeHasAttribs(&(f.eof))(); } @property char front() const { return c; } void popFront() { prime(); } } auto streamRange()(std.stream.File file) { return StreamRange!()(file); } void main() { string fileName = "unicode_test_file"; doWrite(fileName); doRead(fileName); } void doWrite(string fileName) { auto file = std.stdio.File(fileName, "w"); file.writeln("abcçd𝔸e„f"); } void doRead(string fileName) { auto range = byDchar(streamRange(new std.stream.File(fileName, FileMode.In))); foreach (c; range) { writeln(c); } } > What's wrong with reading line by line, but processing the characters in > said lines 1 by 1? That works "out of the box". Agreed. Ali
Re: Reading unicode chars..
On Tuesday, 2 September 2014 at 18:30:55 UTC, seany wrote: Your example reads the file by lines, i need to get them by chars. If you are intent on reading the stream character (or wcharacter) 1 by 1, then you will have to decode them manually, as there is no "getcd". Unfortunately, the "newer" std.stdio module does not really provide facilities for such unitary reads. I'd suggest you create a range out of your std.stream.File, which reads it byte by byte. Then, you pass it to the "byDchar()" range, which will auto decode those characters. If you really want to do it "character by character". What's wrong with reading line by line, but processing the characters in said lines 1 by 1? That works "out of the box".
Re: Reading unicode chars..
Your example reads the file by lines, i need to get them by chars.
Re: Reading unicode chars..
On Tuesday, 2 September 2014 at 18:22:54 UTC, Ali Çehreli wrote: That would happen when you you treat the chars on the input and individual dchars. That is precisely where the problem is. If you use the character in a file, and then open it as a stream, then use File.getc() or file.getcw() until EOF is reached, then you get this prblem. I want to read the file char by char, and problem is i dont know where this char will appear, meaning I dont know where i have to treat multiple dchars, read by getc() or getcw() as a single char.
Re: Reading unicode chars..
On 09/02/2014 11:11 AM, seany wrote: > But try to capture „ the low quotation mark, appearing in the > All-purpose punctuations plane of unicode, with \u201e - I worte I am > having problems with \u1FFF and up. You are doing it differently. Can you show us a minimal example? Otherwise, there is nothing special about „. Continuing with my example, just change one line and it still works: file.writeln("abcçd„ef"); > This particular symbol, is seen as a dchar array "\x1e\x20" - so two > dchars That would happen when you you treat the chars on the input and individual dchars. Those two chars must be decoded as a single dchar. My example has shown two different ways of doing it. :) > using wchar returns the same result Same issue: You are treating individual chars as two individual wchars. Ali
Re: Reading unicode chars..
Linux 64 bit, D2, phobos only.
Re: Reading unicode chars..
Hi Ali, i know this example from your book. But try to capture „ the low quotation mark, appearing in the All-purpose punctuations plane of unicode, with \u201e - I worte I am having problems with \u1FFF and up. This particular symbol, is seen as a dchar array "\x1e\x20" - so two dchars, using wchar returns the same result, when I directly profide the symbol to the code. SO I was thinking of using two dchars, and printing the dstring, the problem then is that I do not know beforehand if a particular character read out of the file is a pair of dchars, or a single dchar. And yes, it was stream.getcw, sorry, not file.getcw(). Indeed. Reading this character from a file (using a while loop until EOF) produces an â and an unknown charcter given by a question mark in a white polygon.
Re: Reading unicode chars..
On Tuesday, 2 September 2014 at 17:10:57 UTC, Ali Çehreli wrote: 1) To avoid a common gotcha, note that 'line' is reused at every iteration here. You must make copies of portions of it if you need to. Ali I don't know if you are aware, but "byLineCopy" was recently introduced. It will be available in 2.067. Just spreading info.
Re: Reading unicode chars..
On 09/02/2014 07:06 AM, seany wrote: How do I read unicode chars that has code points \u1FFF and higher from a file? file.getcw() reads only part of the char, and D identifies this character as an array of three or four characters. Importing std.uni does not change the behavior. Thank you. One way is to use std.stdio.File just like you would use stdin and stdout: import std.stdio; void main() { string fileName = "unicode_test_file"; doWrite(fileName); doRead(fileName); } void doWrite(string fileName) { auto file = File(fileName, "w"); file.writeln("abcçdef"); } void doRead(string fileName) { auto file = File(fileName, "r"); foreach (line; file.byLine) {// (1) foreach (dchar c; line) {// (2) writeln(c); } import std.range; foreach (c; line.stride(1)) {// (3) writeln(c); } } } Notes: 1) To avoid a common gotcha, note that 'line' is reused at every iteration here. You must make copies of portions of it if you need to. 2) dchar is important there 3) Any algorithms that turns a string to a range does expose decoded dchars. Here, I used stride. Ali
Re: Reading unicode chars..
On Tuesday, 2 September 2014 at 14:06:04 UTC, seany wrote: How do I read unicode chars that has code points \u1FFF and higher from a file? file.getcw() reads only part of the char, and D identifies this character as an array of three or four characters. Importing std.uni does not change the behavior. Thank you. Maybe someone else here will recognize this, but for me you'd need to supply more information. Std.file doesn't have getcw, I see one in std.stream which has an "outdated" warning and that getcw is documented as implementation-specific. So what platform are you on? Better yet, can you make a small code sample that shows what you're seeing?
Reading unicode chars..
How do I read unicode chars that has code points \u1FFF and higher from a file? file.getcw() reads only part of the char, and D identifies this character as an array of three or four characters. Importing std.uni does not change the behavior. Thank you.