This frustrated me in Ruby unicode too....
Typically i/o is the ultimate in "untrusted and untrustworthy" sources,
coming usually from systems beyond my control.
Likely to be corrupted, or maliciously crafted, or defective...
Unfortunately not all sequences of bytes are valid UTF8.
Thus inevitably in every collection of inputs there are always going to be
around 1 in a million codepoints resulting in an UTFException thrown.
Alas, I always have to do Regex matches on the other 999999 valid
codepoints.....
Is there a standard recipe in stdio for squashing bad codepoints to some
default?
These days memory is very much larger than most files I want to scan.
So if I was doing this in C I would typically mmap the file PROT_READ |
PROT_WRITE and MAP_PRIVATE then run down the file squashing bad codepoints
and then run down it again matching patterns.
In Ruby I have a horridly inefficient utility....
def IO.read_utf_8(file)
read(file,:external_encoding=>'ASCII-8BIT').encode('UTF-8',:undef=>:replace)
end
What is the idiomatic D solution to this conundrum?
--
John Carter
Phone : (64)(3) 358 6639
Tait Electronics
PO Box 1645 Christchurch
New Zealand
--
------------------------------
This email, including any attachments, is only for the intended recipient.
It is subject to copyright, is confidential and may be the subject of legal
or other privilege, none of which is waived or lost by reason of this
transmission.
If you are not an intended recipient, you may not use, disseminate,
distribute or reproduce such email, any attachments, or any part thereof.
If you have received a message in error, please notify the sender
immediately and erase all copies of the message and any attachments.
Unfortunately, we cannot warrant that the email has not been altered or
corrupted during transmission nor can we guarantee that any email or any
attachments are free from computer viruses or other conditions which may
damage or interfere with recipient data, hardware or software. The
recipient relies upon its own procedures and assumes all risk of use and of
opening any attachments.
------------------------------
