Unexpected behaviour using remove on char[]

[IGotD- via Digitalmars-d-learn]

I have a part in my code that use remove

buffer.remove(tuple(0, size));

with

char[] buffer

What I discovered is that remove doesn't really remove size 
number of bytes but also removed entire multibyte characters and 
consider that one step. The result was of course that I got out 
of bounds exceptions as it went past the end.


When I changed char[] to ubyte[] my code started to work 
correctly again.


According to the documentation a char is an "unsigned 8 bit 
(UTF-8 code unit)" so you really believe you are working on 
bytes. I presume that under the hood there are range iterators at 
work and those work multibyte characters. However you can iterate 
over one byte characters as well as an option and you don't know 
what happens underneath.


I'm a bit confused, when should I expect that the primitives work 
with single versus multibyte chars in array operations?

Re: Unexpected behaviour using remove on char[]

[Ali Çehreli via Digitalmars-d-learn]

On 10/25/20 12:29 PM, IGotD- wrote:

What I discovered is that remove doesn't really remove size number of 
bytes but also removed entire multibyte characters and consider that one 
step. The result was of course that I got out of bounds exceptions as it 
went past the end.


This is the infamous "auto decode" at work.


When I changed char[] to ubyte[] my code started to work correctly again.


Instead of changing the type, you can temporarily treat them as ubyte[] 
with std.string.representation:


import std.stdio;
import std.string;

void print(R)(R range) {
  writefln!"%-(%s, %)"(range);
}

void main() {
  auto s = "abcçd".dup;
  print(s);
  print(s.representation);
}

Prints:

a, b, c, ç, d
97, 98, 99, 195, 167, 100

Ali