Re: Providing implicit conversion of - memory-safety
On Monday, 22 January 2024 at 19:11:50 UTC, Siarhei Siamashka wrote: On Monday, 22 January 2024 at 16:39:10 UTC, Nick Treleaven wrote: Memory safety issues are a worse class of bug than arithmetic bugs. The latter are reproducible if you feed them the same input. Memory safety bugs are reproducible with the tools like `valgrind`. Not necessarily, valgrind can execute programs too slowly for human input, so anything that relies on timing is difficult to reproduce. It also uses far more memory, it could be too much memory for the system. Whereas arithmetic overflow bugs are a real PITA to debug. Assuming that the incorrect results are even noticed. You're talking about debugging, whereas I'm saying you often don't even have a chance to *notice* memory-safety bugs, because they might not even occur on the development system, only on the production system. And even if you know there's a memory-safety problem, you can't easily narrow down where it is (without language support for memory-safety). With arithmetic problems it's far easier to narrow down which code is causing them. But I'm strongly in favour of catching any bugs at compile-time (and have been since before I discovered D). I just object to anyone trying to downgrade the importance of automated memory-safety checking.
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 12:34:38 UTC, Nick Treleaven wrote:
But I'm strongly in favour of catching any bugs at compile-time
(and have been since before I discovered D). I just object to
anyone trying to downgrade the importance of automated
memory-safety checking.
I'm not downgrading the importance of memory safety. All I'm
saying is that you can't sell D as a safe language if has bugs
like this.
Here's a reduced version of one of the most bizarre bugs I've
dealt with in any language. The only reason I didn't move on to
another language was because I was too busy at the time.
The code allows for initial values if the index is less than 0,
otherwise it returns the element.
```
import std;
double value(T)(T index, double * x) {
if (index - 5 < 0) {
return 0.0;
} else {
return x[index-5];
}
}
void main() {
double[] v = [1.1, 2.2, 3.3];
// Works
writeln(value(3, v.ptr));
// Lucky: program segfaults
writeln(value(v.length, v.ptr));
}
```
I noticed this behavior only because the program crashes. Once I
figured out what was going on, I realized that the thousands of
lines of code I had already written needed to be checked and
possibly rewritten. If only I had a compiler to do that for me.
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 17:54:25 UTC, bachmeier wrote:
On Tuesday, 23 January 2024 at 12:34:38 UTC, Nick Treleaven
wrote:
But I'm strongly in favour of catching any bugs at
compile-time (and have been since before I discovered D). I
just object to anyone trying to downgrade the importance of
automated memory-safety checking.
I'm not downgrading the importance of memory safety. All I'm
saying is that you can't sell D as a safe language if has bugs
like this.
Here's a reduced version of one of the most bizarre bugs I've
dealt with in any language. The only reason I didn't move on to
another language was because I was too busy at the time.
The code allows for initial values if the index is less than 0,
otherwise it returns the element.
```
import std;
double value(T)(T index, double * x) {
if (index - 5 < 0) {
return 0.0;
} else {
return x[index-5];
}
}
void main() {
double[] v = [1.1, 2.2, 3.3];
// Works
writeln(value(3, v.ptr));
// Lucky: program segfaults
writeln(value(v.length, v.ptr));
}
```
I noticed this behavior only because the program crashes. Once
I figured out what was going on, I realized that the thousands
of lines of code I had already written needed to be checked and
possibly rewritten. If only I had a compiler to do that for me.
This code seems to be doing everything it can to run into
undefined behaviour, though?
Why is `index` of a type T that has no requirements at all (when
the implementation quite clearly wants `size_t`, or at least an
unsigned numerical value)? Why is it using a pointer for x when
clearly you intend to use it as a slice? You probably have
context that I don't, but I would never expect this sort of code
to be anywhere near @safe :D
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 19:27:26 UTC, Renato wrote:
Here's a reduced version of one of the most bizarre bugs I've
dealt with in any language. The only reason I didn't move on
to another language was because I was too busy at the time.
The code allows for initial values if the index is less than
0, otherwise it returns the element.
```
import std;
double value(T)(T index, double * x) {
if (index - 5 < 0) {
return 0.0;
} else {
return x[index-5];
}
}
void main() {
double[] v = [1.1, 2.2, 3.3];
// Works
writeln(value(3, v.ptr));
// Lucky: program segfaults
writeln(value(v.length, v.ptr));
}
```
I noticed this behavior only because the program crashes. Once
I figured out what was going on, I realized that the thousands
of lines of code I had already written needed to be checked
and possibly rewritten. If only I had a compiler to do that
for me.
This code seems to be doing everything it can to run into
undefined behaviour, though?
Why is `index` of a type T that has no requirements at all
(when the implementation quite clearly wants `size_t`, or at
least an unsigned numerical value)? Why is it using a pointer
for x when clearly you intend to use it as a slice? You
probably have context that I don't, but I would never expect
this sort of code to be anywhere near @safe :D
There are two things things that cause the problem. One is the
use of a template and the other is passing an unsigned type. The
reason the first parameter uses a template is because there are a
lot of types I could send as the first argument, and for some of
them there was a transformation of index (for instance, you can
pass a date as a long[2], or you can pass another type and pull
out the length, that sort of thing). It's using a pointer because
I was working with a C library, and that's how the data is stored
and passed around.
The data is time series. If after the transformations the index
is less than zero, it returns 0.0, which is used for all
pre-sample values. If it's non-negative, return the element at
that position.
One of the nice things about D is the ability to write this kind
of code in such a natural and (I thought) intuitive style. I
really like the way all this comes together. There's really no
way that code should have been able to do anything wrong. What's
terribly frustrating is that the compiler had full knowledge of
what was happening, but by choice it didn't say anything, even
though D is supposed to prevent these things that happen in C.
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 21:18:53 UTC, bachmeier wrote:
There are two things things that cause the problem. One is the
use of a template and the other is passing an unsigned type.
The reason the first parameter uses a template is because there
are a lot of types I could send as the first argument, and for
some of them there was a transformation of index (for instance,
you can pass a date as a long[2], or you can pass another type
and pull out the length, that sort of thing). It's using a
pointer because I was working with a C library, and that's how
the data is stored and passed around.
The data is time series. If after the transformations the index
is less than zero, it returns 0.0, which is used for all
pre-sample values. If it's non-negative, return the element at
that position.
One of the nice things about D is the ability to write this
kind of code in such a natural and (I thought) intuitive style.
I really like the way all this comes together. There's really
no way that code should have been able to do anything wrong.
What's terribly frustrating is that the compiler had full
knowledge of what was happening, but by choice it didn't say
anything, even though D is supposed to prevent these things
that happen in C.
While I can understand your frustration, it seems to me D is not
to blame in this instance because the code is quite patently
using unsafe constructs (D does not claim to be fully safe).
Would something like this work?
```d
double value(T)(T index, double* x) if (is(T : size_t))
{
if (index < 5 || x == null)
{
return 0.0;
}
else
{
return x[index - 5];
}
}
void main()
{
import std.stdio;
import std.range : iota;
double[] ds = [1, 2, 3, 4, 5, 6];
ubyte b = 1;
foreach (_; iota(12))
{
writeln(value(b++, ds.ptr));
}
}
```
This will still read rubbish if the index goes past the actual
array (because I assume you can't get the exact length from the C
code? If you can, you should pass that in and do the bounds check
yourself) but there's no unsigned type mistakes (notice that it's
almost always a mistake to subract from any unsigned type - D
scanner correctly warns about that).
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 21:40:46 UTC, Renato wrote: While I can understand your frustration, it seems to me D is not to blame in this instance because the code is quite patently using unsafe constructs I wouldn't blame bachmeier, because many reduced testcases distilled from the real code tend to look nonsensical. The arithmetic overflows, silent undesirable signed/unsigned casts and other pitfalls happen in the `@safe` code too. The use of pointers and other unsafe constructs in the provided testcase is a red herring.
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 17:54:25 UTC, bachmeier wrote:
Here's a reduced version of one of the most bizarre bugs I've
dealt with in any language. The only reason I didn't move on to
another language was because I was too busy at the time.
The code allows for initial values if the index is less than 0,
otherwise it returns the element.
```
import std;
double value(T)(T index, double * x) {
if (index - 5 < 0) {
return 0.0;
} else {
return x[index-5];
}
}
void main() {
double[] v = [1.1, 2.2, 3.3];
// Works
writeln(value(3, v.ptr));
// Lucky: program segfaults
writeln(value(v.length, v.ptr));
}
```
I noticed this behavior only because the program crashes. Once
I figured out what was going on, I realized that the thousands
of lines of code I had already written needed to be checked and
possibly rewritten. If only I had a compiler to do that for me.
How did you make it correct?
Write 2 different versions for `signed` and `unsigned` types?
Or could you utilize `core.checkedint` somehow for checking
overflow?
```d
double value(T)(T index, double * x) {
bool overflow;
subu(index, 5, overflow);
if (overflow) {
return 0.0;
} else {
return x[index-5];
}
}
```
This is probably only correct for `unsigned` types.
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 21:40:46 UTC, Renato wrote:
While I can understand your frustration, it seems to me D is
not to blame in this instance because the code is quite
patently using unsafe constructs (D does not claim to be fully
safe).
It pretends to be safe. Consider this:
```
void main() {
long y = int.max + 1;
writeln(y); // -2147483648
long y2 = int.max;
writeln(y2 + 1); // 2147483648
int y3 = y; // Won't compile
}
```
It can only be described as a mess of inconsistency. `int y3 =
y;` should be an error and it is. `int.max + 1` silently turning
into a negative value is frankly insane because it's the same
problem that a few lines below won't compile.
Would something like this work?
```d
double value(T)(T index, double* x) if (is(T : size_t))
```
There's no way to add a template constraint. Many different
types, most of which I defined myself, could be sent as an
argument.
that it's almost always a mistake to subract from any unsigned
type - D scanner correctly warns about that).
It's the inconsistency that's the problem. You have to program as
if the compiler doesn't catch anything - sometimes it throws
errors, sometimes it lets stuff through because maybe that's what
you want. `int y3 = y` in the code above is not necessarily an
error.
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 23:40:55 UTC, Danilo wrote:
On Tuesday, 23 January 2024 at 17:54:25 UTC, bachmeier wrote:
Here's a reduced version of one of the most bizarre bugs I've
dealt with in any language. The only reason I didn't move on
to another language was because I was too busy at the time.
The code allows for initial values if the index is less than
0, otherwise it returns the element.
```
import std;
double value(T)(T index, double * x) {
if (index - 5 < 0) {
return 0.0;
} else {
return x[index-5];
}
}
void main() {
double[] v = [1.1, 2.2, 3.3];
// Works
writeln(value(3, v.ptr));
// Lucky: program segfaults
writeln(value(v.length, v.ptr));
}
```
I noticed this behavior only because the program crashes. Once
I figured out what was going on, I realized that the thousands
of lines of code I had already written needed to be checked
and possibly rewritten. If only I had a compiler to do that
for me.
How did you make it correct?
The fix is very easy once you realize what's going on. index is
ulong, so index - 5 is ulong (even though it doesn't make any
sense). All you have to do is change index to index.to!long and
the problem is solved.
Re: Providing implicit conversion of - memory-safety
On Tuesday, 23 January 2024 at 23:40:55 UTC, Danilo wrote:
How did you make it correct?
Write 2 different versions for `signed` and `unsigned` types?
Or could you utilize `core.checkedint` somehow for checking
overflow?
```d
double value(T)(T index, double * x) {
bool overflow;
subu(index, 5, overflow);
if (overflow) {
return 0.0;
} else {
return x[index-5];
}
}
```
This is probably only correct for `unsigned` types.
When you have a variable with a "potentially" unsigned type, you
must not subtract from it unless you're sure the result is not
going negative. The fixed code only subtracts 5 from `index`
after checking that `index >= 5`, so it is always safe.
Your previous code was trying to do the same thing incorrectly
because it just subtracted 5 **first**. This is analogous to
checking pointers for null before using them.
The type parameter restriction was not necessary, but it was
added because the code is assuming that the type can be coerced
to size_t, as it's being used as an index - so it's a good idea
to make that part of the template's "signature"... even without
the type limitation, your code wouldn't compile if this was not
the case (but your error message will probably be much worse).
Re: Providing implicit conversion of - memory-safety
On Wednesday, 24 January 2024 at 00:34:19 UTC, bachmeier wrote:
On Tuesday, 23 January 2024 at 21:40:46 UTC, Renato wrote:
While I can understand your frustration, it seems to me D is
not to blame in this instance because the code is quite
patently using unsafe constructs (D does not claim to be fully
safe).
It pretends to be safe. Consider this:
```
void main() {
long y = int.max + 1;
writeln(y); // -2147483648
long y2 = int.max;
writeln(y2 + 1); // 2147483648
int y3 = y; // Won't compile
}
```
It can only be described as a mess of inconsistency. `int y3 =
y;` should be an error and it is. `int.max + 1` silently
turning into a negative value is frankly insane because it's
the same problem that a few lines below won't compile.
Would something like this work?
```d
double value(T)(T index, double* x) if (is(T : size_t))
```
There's no way to add a template constraint. Many different
types, most of which I defined myself, could be sent as an
argument.
that it's almost always a mistake to subract from any unsigned
type - D scanner correctly warns about that).
It's the inconsistency that's the problem. You have to program
as if the compiler doesn't catch anything - sometimes it throws
errors, sometimes it lets stuff through because maybe that's
what you want. `int y3 = y` in the code above is not
necessarily an error.
For the record, even Rust allows you to subtract from an unsigned
type, but it warns you about it and it fails at runtime due to
the subtraction overflowing (which I believe Rust only checks in
debug mode - in release mode I believe it would behave like D
does in this case, but I didn't verify that).
Here's an example program that compiles:
```rust
fn action(n: usize, arr: &[i64]) -> i64 {
if n - 5 < 0 {
0
} else {
arr[n - 5]
}
}
fn main() {
let arr: [i64; 6] = [1,2,3,4,5,6];
println!("{}", action(4, &arr));
}
```
Compiling and running it:
```rust
warning: comparison is useless due to type limits
--> src/main.rs:3:8
|
3 | if n - 5 < 0 {
|^
|
= note: `#[warn(unused_comparisons)]` on by default
warning: `playground` (bin "playground") generated 1 warning
Finished dev [unoptimized + debuginfo] target(s) in 0.49s
Running `target/debug/playground`
thread 'main' panicked at src/main.rs:3:8:
attempt to subtract with overflow
note: run with `RUST_BACKTRACE=1` environment variable to display
a backtrace
```
I believe that DScanner also warns about the OP's code (I see
this warning all the time in my D code)... but again, if you want
to subtract a number from an unsigned typed variable, you should
absolutely check first that variable is `>=` that number, in Rust
or D or any other language.
If you have "widespread" arithmetics which may overflow,
something like https://dlang.org/phobos/core_checkedint.html is
useful, yes, but in this case it's overkill.
Some languages, like Pony, have dedicated operators for "safe
arithmetics" (because they're much slower and are only rarely
strictly needed):
```
// unsigned wrap-around on overflow
U32.max_value() + 1 == 0
// unsafe operator (undefined behaviour, like with C operators)
U32.max_value() +~ 1 // could be anything!
// safe operator (throws on overflow)
U32.max_value() +? 1
```
Re: Providing implicit conversion of - memory-safety
On Wednesday, 24 January 2024 at 09:28:57 UTC, Renato wrote: If you have "widespread" arithmetics which may overflow, something like https://dlang.org/phobos/core_checkedint.html is useful, yes, but in this case it's overkill. To make use of this, one needs to already anticipate an arithmetic overflow bug at some precise location in the code. But this defeats the purpose. Both array bounds checks and arithmetic overflow checks are useful when the compiler can perform these checks globally for the whole code. To discover bugs even in the parts of code, where they were not anticipated.
