Re: nogc v0.5.0 - DIP1008 works!
On 29.05.19 18:37, Valeriy Fedotov wrote: In my point of view @trusted means "I use pointer-related operations correctly. Also I am using all @system interfaces correctly". @trusted means: "This function is as safe as an @safe function. But the compiler doesn't verify the implementation." Whether you use anything correctly only matters when it affects safety. And "being safe" means that the function doesn't exhibit undefined behavior (when called with valid inputs), and that it doesn't lead to undefined behavior in @safe code that is executed later on (e.g., an @trusted function must not return an invalid pointer). [...] If a user supplies mallocator that is not correct, there are two possibilities: - Allocator is buggy. Nothing to do with @trusted code. - Allocator do not conforms to allocator interface. User has broken the contract. Nothing to do with @trusted code. In my example, UnsafeAllocator is correct. It's not buggy, and it conforms to the allocator interface (which doesn't require @safe methods). Still the program exhibits memory corruption. So there must be something else that's wrong. The problem is that I'm setting UnsafeAllocator.instance.i to bad values which leads to out-of-bounds accesses. I do that in the @safe `main`. You might say: "Don't do that then. It's your own fault if you do that. Your mistake is not the library's fault." But D's @safe is supposed catch exactly that kind of mistake. The point of @safe is that I can make all the mistakes in the world in @safe code (except typing out "@trusted") without ever seeing memory corruption. I think we should keep in mind not only technical aspects of @trusted and @system, but this contract too. No. A contract that can be broken in @safe code cannot be relied upon in @trusted code. This is fundamental to D's safety system.
Re: nogc v0.5.0 - DIP1008 works!
On Monday, 27 May 2019 at 14:26:16 UTC, ag0aep6g wrote: Oh, yeah. Getting @trusted right is hard. Getting it right when user-provided types are involved is extra hard, because you can't even trust fundamental operations like assignment or copying. In my point of view @trusted means "I use pointer-related operations correctly. Also I am using all @system interfaces correctly". The code in question uses allocator interface correctly. User of this code has a contract to supply allocator that conforms to this interface. If a user supplies mallocator that is not correct, there are two possibilities: - Allocator is buggy. Nothing to do with @trusted code. - Allocator do not conforms to allocator interface. User has broken the contract. Nothing to do with @trusted code. I think we should keep in mind not only technical aspects of @trusted and @system, but this contract too.
Re: nogc v0.5.0 - DIP1008 works!
On 27.05.19 15:34, Atila Neves wrote: It's ugly but would work. Right now I don't think I can do any better than to follow your suggestion, but I predict many beard-stroking walks for me along Lake Geneva in the near future. Oh, yeah. Getting @trusted right is hard. Getting it right when user-provided types are involved is extra hard, because you can't even trust fundamental operations like assignment or copying. Please note that the allocator stuff is just one of the three violations I had pointed out. You've already pushed a fix for the unsafe .stringz, but you haven't addressed unsafe copy constructors yet. And my list wasn't meant to be complete. There may be other safety holes I didn't notice.
Re: nogc v0.5.0 - DIP1008 works!
On Monday, 27 May 2019 at 10:31:10 UTC, ag0aep6g wrote: On 27.05.19 12:06, Atila Neves wrote: No, and I guess it can't. I'm trying to figure out what the implications are. Can Vector only be @safe for Mallocator? Is it possible to write a @safe Vector at all without having to force the allocator to be @safe? For @safe allocators, Vector can be @safe. For specific @system allocators, like Mallocator, you can make special @trusted cases in Vector. For generic @system allocators, Vector cannot be @safe (or @trusted). It's ugly but would work. Right now I don't think I can do any better than to follow your suggestion, but I predict many beard-stroking walks for me along Lake Geneva in the near future. I'd be nice if I could detect at compile-time that it's not just Mallocator but an allocator that's built using it as well (e.g. FallBackAllocator).
Re: nogc v0.5.0 - DIP1008 works!
On 27.05.19 12:06, Atila Neves wrote: No, and I guess it can't. I'm trying to figure out what the implications are. Can Vector only be @safe for Mallocator? Is it possible to write a @safe Vector at all without having to force the allocator to be @safe? For @safe allocators, Vector can be @safe. For specific @system allocators, like Mallocator, you can make special @trusted cases in Vector. For generic @system allocators, Vector cannot be @safe (or @trusted).
Re: nogc v0.5.0 - DIP1008 works!
On Monday, 27 May 2019 at 10:01:15 UTC, Atila Neves wrote: On Monday, 27 May 2019 at 09:07:48 UTC, Paolo Invernizzi wrote: On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote: On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote: Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with @trusted, and unless someone looks at their code nobody will be the wiser. ... and a @safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more? I'm not convinced either - I'm having a dialogue to figure out potential issues. :-) My nice-try to reduce the problem is: trusted code block needs to by "manually verified" for safety by humans, so it should be "@safe pure", aka, if you can't perform the analysis looking only at the statements in the trusted block, that can't be marked trusted.
Re: nogc v0.5.0 - DIP1008 works!
On Monday, 27 May 2019 at 09:48:27 UTC, ag0aep6g wrote: On 27.05.19 10:54, Atila Neves wrote: I don't see the problem here. This example would throw RangeError at runtime instead of actually overwriting memory unless bounds checking is turned off. No, it doesn't. It's a complete, runnable example. You can try it at home. It does overwrite `foo` and `bar`. It does not throw a RangeError. You're right - I should have run it first. Yes, you can use @trusted to use Mallocator safely. And your code (probably) does that. But the allocator in my example isn't Mallocator, it's UnsafeAllocator. Your code doesn't use that one safely. No, and I guess it can't. I'm trying to figure out what the implications are. Can Vector only be @safe for Mallocator? Is it possible to write a @safe Vector at all without having to force the allocator to be @safe? In this thread, you're the author of that 3rd party library. You've got the bad @trusted functions that lead to memory corruption. I'm the guy who looked at it, noticed the problem, and tells you. Thanks for bringing it up.
Re: nogc v0.5.0 - DIP1008 works!
On Monday, 27 May 2019 at 09:07:48 UTC, Paolo Invernizzi wrote: On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote: On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote: Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with @trusted, and unless someone looks at their code nobody will be the wiser. ... and a @safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more? I'm not convinced either - I'm having a dialogue to figure out potential issues.
Re: nogc v0.5.0 - DIP1008 works!
On 27.05.19 10:54, Atila Neves wrote: I don't see the problem here. This example would throw RangeError at runtime instead of actually overwriting memory unless bounds checking is turned off. No, it doesn't. It's a complete, runnable example. You can try it at home. It does overwrite `foo` and `bar`. It does not throw a RangeError. The other issue is that Mallocator has a @safe allocate function and a @system deallocate function since it's up to the user of the interface to supply a slice that was actually malloc'ed. It's clear that this interface is one that can be used @safely (and is by automem.vector.Vector). Likewise, reallocating is @system because there might be references to the old pointer, but it makes sense that a @trusted block could exist where the reviewer makes sure that there's only ever one reference to the allocated memory. Yes, you can use @trusted to use Mallocator safely. And your code (probably) does that. But the allocator in my example isn't Mallocator, it's UnsafeAllocator. Your code doesn't use that one safely. Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with @trusted, and unless someone looks at their code nobody will be the wiser. In this thread, you're the author of that 3rd party library. You've got the bad @trusted functions that lead to memory corruption. I'm the guy who looked at it, noticed the problem, and tells you.
Re: nogc v0.5.0 - DIP1008 works!
On Monday, 27 May 2019 at 08:54:45 UTC, Atila Neves wrote: On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote: Then there's the fact that if a 3rd party library really does want to corrupt memory they can just tag all their functions with @trusted, and unless someone looks at their code nobody will be the wiser. ... and a @safe conscious programmer will not touch that library ever with a 5 five meters pole. I'm still not convinced that trusted code should accept generic system code ... can you elaborate more? Thanks, Paolo
Re: nogc v0.5.0 - DIP1008 works!
On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:
On 24.05.19 18:19, Atila Neves wrote:
On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:
[...]
My `puts`s might not do any harm, but they could just as well
be buffer overflows.
Could you please give an example of how @system allocator code
could do that?
Sure. You just write beyond some buffer instead of calling
`puts`:
char[3] buf;
char[3] foo = "foo";
char[3] bar = "bar";
struct UnsafeAllocator
{
import std.experimental.allocator.mallocator: Mallocator;
static instance = UnsafeAllocator.init;
size_t i;
void deallocate(void[] bytes) @nogc @system
{
buf.ptr[i .. i + 3] = '!';
Mallocator.instance.deallocate(bytes);
}
void[] allocate(size_t sz) @nogc @system
{
buf.ptr[i .. i + 3] = '!';
return Mallocator.instance.allocate(sz);
}
}
void main() @safe @nogc
{
{
import nogc: BUFFER_SIZE, text;
UnsafeAllocator.instance.i = 8;
/* greater than buf.length, whoops */
auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42);
assert(foo == "foo"); /* fails */
UnsafeAllocator.instance.i = 16;
/* also greater than buf.length, whoops again */
}
assert(bar == "bar"); /* fails */
}
You just can't trust user-provided @system code. It doesn't
matter if it's allocator code or whatever.
I don't see the problem here. This example would throw RangeError
at runtime instead of actually overwriting memory unless bounds
checking is turned off.
The other issue is that Mallocator has a @safe allocate function
and a @system deallocate function since it's up to the user of
the interface to supply a slice that was actually malloc'ed. It's
clear that this interface is one that can be used @safely (and is
by automem.vector.Vector). Likewise, reallocating is @system
because there might be references to the old pointer, but it
makes sense that a @trusted block could exist where the reviewer
makes sure that there's only ever one reference to the allocated
memory.
Then there's the fact that if a 3rd party library really does
want to corrupt memory they can just tag all their functions with
@trusted, and unless someone looks at their code nobody will be
the wiser.
Re: nogc v0.5.0 - DIP1008 works!
On Friday, 24 May 2019 at 11:41:12 UTC, Atila Neves wrote: I'd been holding off on announcing this until DIP1008 actually got implemented, and now it has: https://code.dlang.org/packages/nogc This dub package has a @nogc version of `std.conv.text` (which probably isn't as good yet) that, instead of returning a `string` returns an `automem.vector.Vector` of char. This handles managing memory allocation for the exception message itself in `NoGcException`, which does what it says on the tin. Confused? Here's some code: Ok, so exceptions don't rely on the GC anymore. That's super cool. However, they are still classes. So does that mean they also need RTTI (i.e. TypeInfo)? BetterC builds and some of use trying to use D in a pay-as-you-go fashion intentionally eliminate RTTI from the runtime. Is there any way we can take this a bit further to no longer require RTTI? Do exceptions even necessarily need to be classes? Thanks, Mike
Re: nogc v0.5.0 - DIP1008 works!
On Friday, 24 May 2019 at 16:51:11 UTC, ag0aep6g wrote:
On 24.05.19 18:19, Atila Neves wrote:
On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:
[...]
My `puts`s might not do any harm, but they could just as well
be buffer overflows.
Could you please give an example of how @system allocator code
could do that?
Sure. You just write beyond some buffer instead of calling
`puts`:
char[3] buf;
char[3] foo = "foo";
char[3] bar = "bar";
struct UnsafeAllocator
{
import std.experimental.allocator.mallocator: Mallocator;
static instance = UnsafeAllocator.init;
size_t i;
void deallocate(void[] bytes) @nogc @system
{
buf.ptr[i .. i + 3] = '!';
Mallocator.instance.deallocate(bytes);
}
void[] allocate(size_t sz) @nogc @system
{
buf.ptr[i .. i + 3] = '!';
return Mallocator.instance.allocate(sz);
}
}
void main() @safe @nogc
{
{
import nogc: BUFFER_SIZE, text;
UnsafeAllocator.instance.i = 8;
/* greater than buf.length, whoops */
auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42);
assert(foo == "foo"); /* fails */
UnsafeAllocator.instance.i = 16;
/* also greater than buf.length, whoops again */
}
assert(bar == "bar"); /* fails */
}
You just can't trust user-provided @system code. It doesn't
matter if it's allocator code or whatever.
That's right. If you are wrapping code that is provided by a
third party, you should not mark any code as @trusted that makes
calls to the third party library. By doing this, you are saying
"any third party code I call is memory safe (source: dude just
trust me)". That may work in the case where this third party code
is set in stone and has been hand-audited by either you or the
maintainers (ideally both), but you're accepting any
implementation through a template argument. Doing this is
extremely dangerous, because you're making memory safety promises
about every single Allocator implementation in existence, in the
present AND for the future.
What you have to do is leave the functions that make these calls
unmarked (no @system, @trusted OR @safe), and allow the compiler
to infer it based on the whether the third party implementation
is @system/@trusted/@safe. That's the only sane way I can think
of to do this.
Re: nogc v0.5.0 - DIP1008 works!
On 24.05.19 18:51, ag0aep6g wrote: char[3] buf; [...] buf.ptr[i .. i + 3] = '!'; [...] UnsafeAllocator.instance.i = 8; /* greater than buf.length, whoops */ Actually, anything greater than zero is a whoops, of course.
Re: nogc v0.5.0 - DIP1008 works!
On 24.05.19 18:19, Atila Neves wrote:
On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote:
[...]
My `puts`s might not do any harm, but they could just as well be
buffer overflows.
Could you please give an example of how @system allocator code could do
that?
Sure. You just write beyond some buffer instead of calling `puts`:
char[3] buf;
char[3] foo = "foo";
char[3] bar = "bar";
struct UnsafeAllocator
{
import std.experimental.allocator.mallocator: Mallocator;
static instance = UnsafeAllocator.init;
size_t i;
void deallocate(void[] bytes) @nogc @system
{
buf.ptr[i .. i + 3] = '!';
Mallocator.instance.deallocate(bytes);
}
void[] allocate(size_t sz) @nogc @system
{
buf.ptr[i .. i + 3] = '!';
return Mallocator.instance.allocate(sz);
}
}
void main() @safe @nogc
{
{
import nogc: BUFFER_SIZE, text;
UnsafeAllocator.instance.i = 8;
/* greater than buf.length, whoops */
auto t = text!(BUFFER_SIZE, UnsafeAllocator)(42);
assert(foo == "foo"); /* fails */
UnsafeAllocator.instance.i = 16;
/* also greater than buf.length, whoops again */
}
assert(bar == "bar"); /* fails */
}
You just can't trust user-provided @system code. It doesn't matter if
it's allocator code or whatever.
Re: nogc v0.5.0 - DIP1008 works!
On Friday, 24 May 2019 at 13:30:05 UTC, ag0aep6g wrote: On Friday, 24 May 2019 at 13:13:12 UTC, Atila Neves wrote: Thanks for this. I think the only violation is calling `stringz` on `Z`, and that was due to a poorly designed DbI check on being able to call `stringz`. Allocating generally isn't @system, and freeing is ok to trust since vector is taking care of it for us. I've pushed a fix. I think you're missing the point. When your function is marked as @safe or @trusted, then any execution of a user-provided @system function is a safety violation. My `puts`s might not do any harm, but they could just as well be buffer overflows. Could you please give an example of how @system allocator code could do that?
Re: nogc v0.5.0 - DIP1008 works!
On Friday, 24 May 2019 at 13:13:12 UTC, Atila Neves wrote: Thanks for this. I think the only violation is calling `stringz` on `Z`, and that was due to a poorly designed DbI check on being able to call `stringz`. Allocating generally isn't @system, and freeing is ok to trust since vector is taking care of it for us. I've pushed a fix. I think you're missing the point. When your function is marked as @safe or @trusted, then any execution of a user-provided @system function is a safety violation. My `puts`s might not do any harm, but they could just as well be buffer overflows.
Re: nogc v0.5.0 - DIP1008 works!
On Friday, 24 May 2019 at 12:32:45 UTC, ag0aep6g wrote:
On 24.05.19 13:41, Atila Neves wrote:
[...]
You've got safety violations:
/+ dub.sdl:
name "test"
dependency "nogc" version="~>0.5.0"
+/
import core.stdc.stdio: puts;
struct S1
{
S2 s2;
this(ref const S1 src) const @nogc @system { this.s2 =
src.s2; }
}
struct S2
{
this(ref const S2 src) const @nogc @system { puts("@system
1"); }
}
struct Z
{
char* stringz() const @nogc @system
{
puts("@system 2");
return null;
}
}
struct UnsafeAllocator
{
import std.experimental.allocator.mallocator: Mallocator;
enum instance = UnsafeAllocator.init;
void deallocate(void[] bytes) @nogc @system
{
puts("@system 3");
Mallocator.instance.deallocate(bytes);
}
void[] allocate(size_t sz) @nogc @system
{
puts("@system 4");
return Mallocator.instance.allocate(sz);
}
}
void main() @safe @nogc
{
import nogc: BUFFER_SIZE, text;
S1 a;
Z* z;
auto t = text!(BUFFER_SIZE, UnsafeAllocator)(a, z);
}
All of the `puts` lines are executed. That should not be
possible in @safe code. You're applying @trusted too liberally.
Thanks for this. I think the only violation is calling `stringz`
on `Z`, and that was due to a poorly designed DbI check on being
able to call `stringz`. Allocating generally isn't @system, and
freeing is ok to trust since vector is taking care of it for us.
I've pushed a fix.
Re: nogc v0.5.0 - DIP1008 works!
On 24.05.19 13:41, Atila Neves wrote:
I'd been holding off on announcing this until DIP1008 actually got
implemented, and now it has:
https://code.dlang.org/packages/nogc
You've got safety violations:
/+ dub.sdl:
name "test"
dependency "nogc" version="~>0.5.0"
+/
import core.stdc.stdio: puts;
struct S1
{
S2 s2;
this(ref const S1 src) const @nogc @system { this.s2 = src.s2; }
}
struct S2
{
this(ref const S2 src) const @nogc @system { puts("@system 1"); }
}
struct Z
{
char* stringz() const @nogc @system
{
puts("@system 2");
return null;
}
}
struct UnsafeAllocator
{
import std.experimental.allocator.mallocator: Mallocator;
enum instance = UnsafeAllocator.init;
void deallocate(void[] bytes) @nogc @system
{
puts("@system 3");
Mallocator.instance.deallocate(bytes);
}
void[] allocate(size_t sz) @nogc @system
{
puts("@system 4");
return Mallocator.instance.allocate(sz);
}
}
void main() @safe @nogc
{
import nogc: BUFFER_SIZE, text;
S1 a;
Z* z;
auto t = text!(BUFFER_SIZE, UnsafeAllocator)(a, z);
}
All of the `puts` lines are executed. That should not be possible in
@safe code. You're applying @trusted too liberally.
Re: nogc v0.5.0 - DIP1008 works!
On Friday, 24 May 2019 at 11:41:12 UTC, Atila Neves wrote: I'd been holding off on announcing this until DIP1008 actually got implemented, and now it has: [...] Awesome!! Now we just need to get to compile Druntime and Phobos with -preview=dip1008, s.t. we can enable it by default :) See e.g. https://github.com/dlang/dmd/pull/8508
nogc v0.5.0 - DIP1008 works!
I'd been holding off on announcing this until DIP1008 actually
got implemented, and now it has:
https://code.dlang.org/packages/nogc
This dub package has a @nogc version of `std.conv.text` (which
probably isn't as good yet) that, instead of returning a `string`
returns an `automem.vector.Vector` of char. This handles managing
memory allocation for the exception message itself in
`NoGcException`, which does what it says on the tin. Confused?
Here's some code:
@safe @nogc unittest {
import nogc;
import std.algorithm: equal;
int a = 1, b = 2;
try
enforce(a == b, a, " was not equal to ", b);
catch(NoGcException e) {
assert(equal(e.msg, "1 was not equal to 2"));
}
try
throw new NoGcException(42, " foobar ", 33.3);
catch(NoGcException e) {
assert(equal(e.msg, "42 foobar 33.30"));
assert(e.file == __FILE__);
assert(e.line == __LINE__ - 4);
}
}
It doesn't leak memory either, as proved by ldc's asan.
