Re: @safe and null dereferencing
On Thursday, 27 July 2017 at 23:52:27 UTC, H. S. Teoh wrote: On Thu, Jul 27, 2017 at 09:32:12PM +, Moritz Maxeiner via Digitalmars-d wrote: [...] Yes, and therefore "you already have much bigger things to worry about than D services hanging". That you're ignorant of the compromise does not negate the fact that you do have bigger things to worry about, you're just blissfully unaware of them. :-P Until the good stuff hits the proverbial fan, of course. Sure, but this kind of subversive attack *keeps* you *the defender) ignorant, i.e. you are unlikely to notice that you have bigger issues to worry about.
Re: @safe and null dereferencing
On Thu, Jul 27, 2017 at 09:32:12PM +, Moritz Maxeiner via Digitalmars-d wrote: > On Thursday, 27 July 2017 at 20:48:51 UTC, H. S. Teoh wrote: [...] > > If someone malicious has root access to your server, you already > > have much bigger things to worry about than D services hanging. :-D > > That depends on how valuable you are as a target, how hard it was to > gain root access, and what the attacker's intentions are. If you are > a high value target for which root access was hard to get, the > attacker is unlikely to risk detection by doing things that someone > (or an IDS) will categorize as an attack; the attacker is much more > likely to try and subvert the system without being detected; see for > example how Stuxnet was used to slowly damage centrifuge machines. Yes, and therefore "you already have much bigger things to worry about than D services hanging". That you're ignorant of the compromise does not negate the fact that you do have bigger things to worry about, you're just blissfully unaware of them. :-P Until the good stuff hits the proverbial fan, of course. T -- Tell me and I forget. Teach me and I remember. Involve me and I understand. -- Benjamin Franklin
Re: @safe and null dereferencing
On Thursday, July 27, 2017 13:48:51 H. S. Teoh via Digitalmars-d wrote: > On Thu, Jul 27, 2017 at 07:50:52PM +, Moritz Maxeiner via Digitalmars- d wrote: > > On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote: > [...] > > > > I see no problem whatsoever requiring that the platform segfaults > > > when you dereference null. Anything even vaguely modern will do > > > that. Adding extra null checks is therefore redundant and > > > complicates the compiler for no gain whatsoever. > > > > Except that when someone gets (root) access to any modern Linux > > servers running D services he now has an easy way to create a denial > > of service attack the owner of the server won't easily be able to find > > the cause of, because pretty much everything *looks* right, except > > that somehow the D services hang. > > If someone malicious has root access to your server, you already have > much bigger things to worry about than D services hanging. :-D Agreed. And Safe D has never made any promises about denial of service attacks and whatnot, let alone preventing things going wrong if someone has root access. If you don't want segfaulting to open a window for someone to hit you with a DoS attack, then don't dereference null pointers, and if you don't want someone to do nasty things to your server that would require them to be root, then do the appropriate things to protect your machine so that they don't have root. We can _always_ find ways that a badly written program can have issues with DoS attacks or have trouble if someone malicious has access to the machine that it's running on. @safe is about guaranteeing memory safety, not about stopping people from screwing you over when you write bad code or fail to protect your computer from attacks. - Jonathan M Davis
Re: @safe and null dereferencing
Am Thu, 27 Jul 2017 17:59:41 + schrieb Adrian Matoga : > On Thursday, 27 July 2017 at 17:43:17 UTC, H. S. Teoh wrote: > > On Thu, Jul 27, 2017 at 05:33:22PM +, Adrian Matoga via > > Digitalmars-d wrote: [...] > >> Why can't we just make the compiler insert null checks in > >> @safe code? > > > > Because not inserting null checks is a sacred cow we inherited > > from the C/C++ days of POOP (premature optimization oriented > > programming), and we are loathe to slaughter it. :-P We > > should seriously take some measurements of this in a large D > > project to determine whether or not inserting null checks > > actually makes a significant difference in performance. > > That's exactly what I thought. A typical non-synthetic worst case candidate should be in the tests that would invoke a lot of null checks. (Could be a function call at first to count checks per run of executable and pick a good project.) -- Marco
Re: @safe and null dereferencing
On Thursday, 27 July 2017 at 20:48:51 UTC, H. S. Teoh wrote: On Thu, Jul 27, 2017 at 07:50:52PM +, Moritz Maxeiner via Digitalmars-d wrote: On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote: [...] > I see no problem whatsoever requiring that the platform > segfaults when you dereference null. Anything even vaguely > modern will do that. Adding extra null checks is therefore > redundant and complicates the compiler for no gain > whatsoever. Except that when someone gets (root) access to any modern Linux servers running D services he now has an easy way to create a denial of service attack the owner of the server won't easily be able to find the cause of, because pretty much everything *looks* right, except that somehow the D services hang. If someone malicious has root access to your server, you already have much bigger things to worry about than D services hanging. :-D That depends on how valuable you are as a target, how hard it was to gain root access, and what the attacker's intentions are. If you are a high value target for which root access was hard to get, the attacker is unlikely to risk detection by doing things that someone (or an IDS) will categorize as an attack; the attacker is much more likely to try and subvert the system without being detected; see for example how Stuxnet was used to slowly damage centrifuge machines.
Re: @safe and null dereferencing
On Thu, Jul 27, 2017 at 07:50:52PM +, Moritz Maxeiner via Digitalmars-d wrote: > On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote: [...] > > I see no problem whatsoever requiring that the platform segfaults > > when you dereference null. Anything even vaguely modern will do > > that. Adding extra null checks is therefore redundant and > > complicates the compiler for no gain whatsoever. > > Except that when someone gets (root) access to any modern Linux > servers running D services he now has an easy way to create a denial > of service attack the owner of the server won't easily be able to find > the cause of, because pretty much everything *looks* right, except > that somehow the D services hang. If someone malicious has root access to your server, you already have much bigger things to worry about than D services hanging. :-D T -- Don't get stuck in a closet---wear yourself out.
Re: @safe and null dereferencing
On Thursday, 27 July 2017 at 20:09:46 UTC, Steven Schveighoffer wrote: Well, let's not forget that the services should not be dereferencing null. It's still a bug in the code. Of course, but statistically speaking, all software is buggy so it's not an unreasonable assumption on the attackers part that there is at least one null dereference in complex server code that will eventually trigger. It just may result in something other than a process exit. Which is really bad for process supervision, because it'll likely not detect a problem and not kill+respawn the service. I bet if you lowered that limit, you would cause all sorts of trouble, not just in D safe code. Imagine, any function that returns null specifically to mean an error, now may return it casually as the address of a valid item! You are going to screw up all checks for null! Indeed, but atm I was only concerned about the implications for D @safe code.
Re: @safe and null dereferencing
On 7/27/17 3:50 PM, Moritz Maxeiner wrote: On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote: On Thursday, July 27, 2017 11:03:02 Steven Schveighoffer via Digitalmars-d wrote: A possibility: "@safe D does not support platforms or processes where dereferencing a null pointer does not crash the program. In such situations, dereferencing null is not defined, and @safe code will not prevent this from happening." In terms of not marking C/C++ code safe, I am not convinced we need to go that far, but it's not as horrible a prospect as having to unmark D @safe code that might dereference null. I see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever. Except that when someone gets (root) access to any modern Linux servers running D services he now has an easy way to create a denial of service attack the owner of the server won't easily be able to find the cause of, because pretty much everything *looks* right, except that somehow the D services hang. Well, let's not forget that the services should not be dereferencing null. It's still a bug in the code. It just may result in something other than a process exit. I bet if you lowered that limit, you would cause all sorts of trouble, not just in D safe code. Imagine, any function that returns null specifically to mean an error, now may return it casually as the address of a valid item! You are going to screw up all checks for null! -Steve
Re: @safe and null dereferencing
On Thursday, 27 July 2017 at 18:46:16 UTC, Jonathan M Davis wrote: On Thursday, July 27, 2017 11:03:02 Steven Schveighoffer via Digitalmars-d wrote: A possibility: "@safe D does not support platforms or processes where dereferencing a null pointer does not crash the program. In such situations, dereferencing null is not defined, and @safe code will not prevent this from happening." In terms of not marking C/C++ code safe, I am not convinced we need to go that far, but it's not as horrible a prospect as having to unmark D @safe code that might dereference null. I see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever. Except that when someone gets (root) access to any modern Linux servers running D services he now has an easy way to create a denial of service attack the owner of the server won't easily be able to find the cause of, because pretty much everything *looks* right, except that somehow the D services hang.
Re: @safe and null dereferencing
On Thursday, 27 July 2017 at 17:52:09 UTC, H. S. Teoh wrote: On Thu, Jul 27, 2017 at 11:03:02AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...] However, there do exist places where dereferencing null may NOT cause a segmentation fault. For example, see this post by Moritz Maxeiner: https://forum.dlang.org/post/udkdqogtrvanhbotd...@forum.dlang.org In such cases, the compiled program can have no knowledge that the zero page is mapped somehow. There is no way to prevent it, or guarantee it during compilation. [...] There is one flaw with Moritz's example: if the zero page is mapped somehow, that means 0 is potentially a valid address of a variable, and therefore checking for null is basically not only useless but wrong: a null check of the address of this variable will fail, yet the pointer is actually pointing at a valid address that just happens to be 0. IOW, if the zero page is mapped, we're *already* screwed anyway, might as well just give up now. The point of the example was to show that exploiting the "null dereferences segfault" assumption on a modern Linux system to create completely unexpected behaviour (in the case I showed fgetc is going to make the process hang -> denial of service with hard to detect cause) and break any D program's @safe correctness is almost trivial.
Re: @safe and null dereferencing
On 7/27/17 2:46 PM, Jonathan M Davis via Digitalmars-d wrote: However, one issue that has been brought up from time to time and AFAIK has never really been addressed is that apparently if an object is large enough, when you access one of its members when the pointer is null, you won't get a segfault (I think that it was something like if the object was greater than a page in size). So, as I understand it, ludicrously large objects _could_ result in @safety problems with null pointers. This would not happen in normal code, but it can happen. And if we want @safe to make the guarantees that it claims, we really should either disallow such objects or insert null checks for them. For smaller objects though, what's the point? It buys us nothing if the hardware is already doing it, and the only hardware that wouldn't do it should be too old to matter at this point. Yes: https://issues.dlang.org/show_bug.cgi?id=5176 There is a way to "fix" this: any time you access an object field that goes outside the page size, do a null check on the base pointer. -Steve
Re: @safe and null dereferencing
On Thursday, July 27, 2017 11:03:02 Steven Schveighoffer via Digitalmars-d wrote: > A possibility: > > "@safe D does not support platforms or processes where dereferencing a > null pointer does not crash the program. In such situations, > dereferencing null is not defined, and @safe code will not prevent this > from happening." > > In terms of not marking C/C++ code safe, I am not convinced we need to > go that far, but it's not as horrible a prospect as having to unmark D > @safe code that might dereference null. I see no problem whatsoever requiring that the platform segfaults when you dereference null. Anything even vaguely modern will do that. Adding extra null checks is therefore redundant and complicates the compiler for no gain whatsoever. However, one issue that has been brought up from time to time and AFAIK has never really been addressed is that apparently if an object is large enough, when you access one of its members when the pointer is null, you won't get a segfault (I think that it was something like if the object was greater than a page in size). So, as I understand it, ludicrously large objects _could_ result in @safety problems with null pointers. This would not happen in normal code, but it can happen. And if we want @safe to make the guarantees that it claims, we really should either disallow such objects or insert null checks for them. For smaller objects though, what's the point? It buys us nothing if the hardware is already doing it, and the only hardware that wouldn't do it should be too old to matter at this point. So, I say that we need to deal with the problem with ludicrously large objects, but beyond that, we should just change the spec, because inserting the checks buys us nothing. - Jonathan M Davis
Re: @safe and null dereferencing
On 7/27/17 2:09 PM, Steven Schveighoffer wrote: there's nothing in the spec to require it. And it does seem apparent that we handle this situation. that we *should* handle this situation. -Steve
Re: @safe and null dereferencing
On 7/27/17 1:52 PM, H. S. Teoh via Digitalmars-d wrote: On Thu, Jul 27, 2017 at 11:03:02AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...] However, there do exist places where dereferencing null may NOT cause a segmentation fault. For example, see this post by Moritz Maxeiner: https://forum.dlang.org/post/udkdqogtrvanhbotd...@forum.dlang.org In such cases, the compiled program can have no knowledge that the zero page is mapped somehow. There is no way to prevent it, or guarantee it during compilation. [...] There is one flaw with Moritz's example: if the zero page is mapped somehow, that means 0 is potentially a valid address of a variable, and therefore checking for null is basically not only useless but wrong: a null check of the address of this variable will fail, yet the pointer is actually pointing at a valid address that just happens to be 0. IOW, if the zero page is mapped, we're *already* screwed anyway, might as well just give up now. Very true. You wouldn't want to store anything there as any @safe code could easily get a pointer to that data at any time! Either way, the guarantees of @safe go out the window if dereferencing null is not a crashing error. -Steve
Re: @safe and null dereferencing
On 7/27/17 1:33 PM, Adrian Matoga wrote: Why can't we just make the compiler insert null checks in @safe code? We can afford bounds checking even in @system -O -release. C++ can afford null check upon executing an std::function. The pointer would most likely be in a register anyway, and the conditional branch would almost always not be taken, so the cost of that check would be barely measurable. Moreover, the compiler can elide the check e.g. if the access via pointer is made in a loop in which the pointer doesn't change. And if you prove that this tiny little check ruins performance of your code, there's @trusted to help you. The rationale from Walter has always been that the hardware is already doing this for us. I was always under the assumption that D only supported environments/systems where this happens. But technically there's nothing in the spec to require it. And it does seem apparent that we handle this situation. This question/query is asking whether we should amend the spec with (what I think is) Walter's view, or if we should change the compiler to insert the checks. -Steve
Re: @safe and null dereferencing
On Thu, Jul 27, 2017 at 11:03:02AM -0400, Steven Schveighoffer via Digitalmars-d wrote: [...] > However, there do exist places where dereferencing null may NOT cause > a segmentation fault. For example, see this post by Moritz Maxeiner: > https://forum.dlang.org/post/udkdqogtrvanhbotd...@forum.dlang.org > > In such cases, the compiled program can have no knowledge that the > zero page is mapped somehow. There is no way to prevent it, or > guarantee it during compilation. [...] There is one flaw with Moritz's example: if the zero page is mapped somehow, that means 0 is potentially a valid address of a variable, and therefore checking for null is basically not only useless but wrong: a null check of the address of this variable will fail, yet the pointer is actually pointing at a valid address that just happens to be 0. IOW, if the zero page is mapped, we're *already* screwed anyway, might as well just give up now. One workaround for this is to redefine a null pointer as size_t.max (i.e., all bits set) instead of 0. It's far less likely for a valid address to be size_t.max than for 0 to be a valid address in a system where the zero page is mappable (due to alignment issues, the only possibility is if you have a ubyte* pointing to data stored at the address size_t.max, whereas address 0 can be a valid address for any data type). However, this will break basically *all* code out there in C/C++/D land, so I don't see it ever happening in this lifetime. T -- Those who've learned LaTeX swear by it. Those who are learning LaTeX swear at it. -- Pete Bleackley
Re: @safe and null dereferencing
On Thursday, 27 July 2017 at 17:43:17 UTC, H. S. Teoh wrote: On Thu, Jul 27, 2017 at 05:33:22PM +, Adrian Matoga via Digitalmars-d wrote: [...] Why can't we just make the compiler insert null checks in @safe code? Because not inserting null checks is a sacred cow we inherited from the C/C++ days of POOP (premature optimization oriented programming), and we are loathe to slaughter it. :-P We should seriously take some measurements of this in a large D project to determine whether or not inserting null checks actually makes a significant difference in performance. That's exactly what I thought.
Re: @safe and null dereferencing
On Thu, Jul 27, 2017 at 05:33:22PM +, Adrian Matoga via Digitalmars-d wrote: [...] > Why can't we just make the compiler insert null checks in @safe code? Because not inserting null checks is a sacred cow we inherited from the C/C++ days of POOP (premature optimization oriented programming), and we are loathe to slaughter it. :-P We should seriously take some measurements of this in a large D project to determine whether or not inserting null checks actually makes a significant difference in performance. > We can afford bounds checking even in @system -O -release. C++ can > afford null check upon executing an std::function. The pointer would > most likely be in a register anyway, and the conditional branch would > almost always not be taken, so the cost of that check would be barely > measurable. Moreover, the compiler can elide the check e.g. if the > access via pointer is made in a loop in which the pointer doesn't > change. And if you prove that this tiny little check ruins performance > of your code, there's @trusted to help you. The compiler can (and should, if it doesn't already) also propagate non-nullness (ala VRP) as part of its dataflow analysis, so that once a pointer has been established to be non-null, all subsequent checks of that pointer can be elided (until the next assignment to the pointer, of course). T -- Public parking: euphemism for paid parking. -- Flora
Re: @safe and null dereferencing
On Thursday, 27 July 2017 at 15:03:02 UTC, Steven Schveighoffer
wrote:
Inside the thread for adding @safe/@trusted attributes to OS
functions, it has come to light that @safe has conflicting
rules.
For the definition of safe, it says:
"Safe functions are functions that are statically checked to
exhibit no possibility of undefined behavior."
In the definition of @trusted, it says:
"Trusted functions are guaranteed by the programmer to not
exhibit any undefined behavior if called by a safe function."
Yet, safe functions allow dereferencing of null pointers.
Example:
void foo() @safe
{
int *x;
*x = 5;
}
There are various places on the forum where Walter argues that
null pointer dereferencing should cause a segmentation fault
(or crash) and is checked by the hardware/OS. Therefore,
checking for null pointers before any dereferencing would be a
waste of cycles.
However, there do exist places where dereferencing null may NOT
cause a segmentation fault. For example, see this post by
Moritz Maxeiner:
https://forum.dlang.org/post/udkdqogtrvanhbotd...@forum.dlang.org
In such cases, the compiled program can have no knowledge that
the zero page is mapped somehow. There is no way to prevent it,
or guarantee it during compilation.
It's also worth noting that C/C++ identifies null dereferencing
as undefined behavior. So if we are being completely pedantic,
we could say that no C/C++ code could be marked safe if there
is a possibility that a null pointer would be dereferenced.
The way I see it, we have 2 options. First, we can disallow
null pointer dereferencing in @safe code. This would be hugely
disruptive. We may not have to instrument all @safe code with
null checks, we could do it with flow analysis, and assuming
that all pointers passed into a @safe function are not null.
But it would likely disallow a lot of existing @safe code.
The other option is to explicitly state what happens in such
cases. I would opt for this second option, as the likelihood of
these situations is very low.
If we were to update the spec to take this into account, how
would it look?
A possibility:
"@safe D does not support platforms or processes where
dereferencing a null pointer does not crash the program. In
such situations, dereferencing null is not defined, and @safe
code will not prevent this from happening."
In terms of not marking C/C++ code safe, I am not convinced we
need to go that far, but it's not as horrible a prospect as
having to unmark D @safe code that might dereference null.
Thoughts?
-Steve
Why can't we just make the compiler insert null checks in @safe
code? We can afford bounds checking even in @system -O -release.
C++ can afford null check upon executing an std::function. The
pointer would most likely be in a register anyway, and the
conditional branch would almost always not be taken, so the cost
of that check would be barely measurable. Moreover, the compiler
can elide the check e.g. if the access via pointer is made in a
loop in which the pointer doesn't change. And if you prove that
this tiny little check ruins performance of your code, there's
@trusted to help you.
