Re: Bug with writeln?
On Sunday, September 9, 2018 8:30:12 AM MDT Saurabh Das via Digitalmars-d- learn wrote: > Thank you for explaining all this. > > It is frustrating because the behaviour is very counterintuitive. > > I will use a workaround for now. Ranges are fantastic, and the basic concept is solid, but a number of the implementation details really weren't worked out well enough in the beginning, which means that there's a lot of stuff that works reasonably well with ranges but falls apart on some level in certain circumstances - and what happens when ranges get copied is a big part of that. On some level, it's what we get for being the first language to really implement ranges as part of its standard library. As I understand it, Andrei didn't create the concept, but AFAIK, it wasn't implemented on any real scale by anyone prior to them being added to Phobos. And when you're the first to implement something, you often screw it up on some level. With many things, D was able to learn from C++ and improve, but this is an area where we got to make new mistakes. The result is something that's extremely useful and largely works but which has some problematic corner cases that really shouldn't be there. - Jonathan M Davis
Re: Bug with writeln?
Thank you for explaining all this. It is frustrating because the behaviour is very counterintuitive. I will use a workaround for now. Saurabh
Re: Bug with writeln?
On Thursday, September 6, 2018 1:05:03 PM MDT Steven Schveighoffer via Digitalmars-d-learn wrote: > On 9/6/18 2:52 PM, Jonathan M Davis wrote: > > On Thursday, September 6, 2018 12:21:24 PM MDT Steven Schveighoffer via > > > > Digitalmars-d-learn wrote: > >> On 9/6/18 12:55 PM, Jonathan M Davis wrote: > >>> It's not a bug in writeln. Any time that a range is copied, you must > >>> not > >>> do _anything_ else with the original unless copying it is equivalent > >>> to > >>> calling save on it, because the semantics of copying a range are > >>> unspecified. They vary wildly depending on the range type (e.g. > >>> copying > >>> a dynamic array is equivalent to calling save, but copying a class > >>> reference is not). When you pass the range to writeln, you must > >>> assumed > >>> that it may have been consumed. And since you have range of ranges, > >>> you > >>> must assume that the ranges that are contained may have been consumed. > >>> If you want to pass them to writeln and then do anything else with > >>> them, then you'll need to call save on every range involved (which is > >>> a > >>> bit of a pain with a range of ranges, but it's necessary all the > >>> same). > >> > >> This is not necessarily true. It depends how the sub-ranges are > >> returned. > >> > >> The bug is that formattedWrite takes ranges sometimes by ref, sometimes > >> not. > >> > >> formattedWrite should call save on a forward range whenever it makes a > >> copy, and it doesn't. > >> > >> Case in point, it doesn't matter if you call writeln(b.save), the same > >> thing happens. > > > > That's still not a bug in formattedWrite. save only duplicates the > > outer-most range. And since writeln will ultimately iterate through the > > inner ranges - which weren't saved - you end up with them being > > consumed. > > That is the bug -- formattedWrite should save all the inner ranges > (writeln calls formattedWrite, and lets it do all the work). To not do > so leaves it open to problems such as consuming the sub ranges. > > I can't imagine that anyone would expect or desire the current behavior. It's exactly what you're going to get in all cases if the ranges aren't forward ranges, and it's what you have to do in general when passing ranges of ranges to functions if you want to be able to continue to use any of the ranges involved after passing them to the function. Changing formattedWrite to work around it is only a workaround for this paricular case. It's still a bug in general - though given that this would be one of the more common cases, working around it in this particular case may be worth it. It's still a workaround though and not something that can be relied on in with range-based code in general - especially when most range-based code isn't written to care about what the element types are and copies elements around all the time. > Ironically, when that bug is fixed, you *don't* have to call save on the > outer range! Except you do, because it's passed by value. If it's a dynamic array, then you're fine, since copying saves, but in the general case, you still do. > > When you're passing a range of ranges to a function, you need to > > recursively save them if you don't want the inner ranges in the > > original range to be consumed. Regardless of what formattedWrite does, > > it's a general issue with any function that you pass a range of ranges. > > It comes right back to the same issue of the semantics of copying > > ranges being unspecified and that you therefore must always use save on > > any ranges involved if you want to then use those ranges after having > > passed them to a function or copy them doing anything else. It's that > > much more annoying when you're dealing with a range of ranges rather > > than a range of something else, but the issue is the same. > It's only a problem if the subranges are returned by reference. If they > aren't, then no save is required (because they are already copies). The > fix in this case is to make a copy if possible (using save as expected). > > I think the save semantics have to be one of the worst designs in D. On that we can definitely agree. I'm strongly of the opinion that it should have been required that forward ranges be dynamic arrays or structs (no classes allowed) and that it be required that they have a postblit / copy constructor if the default copy wasn't equivalent to save. If you wanted a class that was a forward range, you would then have to wrap it in a struct with the appropriate postblit / copy constructor. That way, copying a forward range would _always_ be saving it. The harder question is what to then do with basic input ranges. Having them share code with forward ranges is often useful but also frequently a disaster, and to really be correct, they would need to either be full-on reference types or always passed around by reference. Allowing partial reference types is a total disaster when you're allowed to copy the range. Requiring that they be classes would
Re: Bug with writeln?
On 9/6/18 2:52 PM, Jonathan M Davis wrote: On Thursday, September 6, 2018 12:21:24 PM MDT Steven Schveighoffer via Digitalmars-d-learn wrote: On 9/6/18 12:55 PM, Jonathan M Davis wrote: It's not a bug in writeln. Any time that a range is copied, you must not do _anything_ else with the original unless copying it is equivalent to calling save on it, because the semantics of copying a range are unspecified. They vary wildly depending on the range type (e.g. copying a dynamic array is equivalent to calling save, but copying a class reference is not). When you pass the range to writeln, you must assumed that it may have been consumed. And since you have range of ranges, you must assume that the ranges that are contained may have been consumed. If you want to pass them to writeln and then do anything else with them, then you'll need to call save on every range involved (which is a bit of a pain with a range of ranges, but it's necessary all the same). This is not necessarily true. It depends how the sub-ranges are returned. The bug is that formattedWrite takes ranges sometimes by ref, sometimes not. formattedWrite should call save on a forward range whenever it makes a copy, and it doesn't. Case in point, it doesn't matter if you call writeln(b.save), the same thing happens. That's still not a bug in formattedWrite. save only duplicates the outer-most range. And since writeln will ultimately iterate through the inner ranges - which weren't saved - you end up with them being consumed. That is the bug -- formattedWrite should save all the inner ranges (writeln calls formattedWrite, and lets it do all the work). To not do so leaves it open to problems such as consuming the sub ranges. I can't imagine that anyone would expect or desire the current behavior. Ironically, when that bug is fixed, you *don't* have to call save on the outer range! When you're passing a range of ranges to a function, you need to recursively save them if you don't want the inner ranges in the original range to be consumed. Regardless of what formattedWrite does, it's a general issue with any function that you pass a range of ranges. It comes right back to the same issue of the semantics of copying ranges being unspecified and that you therefore must always use save on any ranges involved if you want to then use those ranges after having passed them to a function or copy them doing anything else. It's that much more annoying when you're dealing with a range of ranges rather than a range of something else, but the issue is the same. It's only a problem if the subranges are returned by reference. If they aren't, then no save is required (because they are already copies). The fix in this case is to make a copy if possible (using save as expected). I think the save semantics have to be one of the worst designs in D. -Steve
Re: Bug with writeln?
On Thursday, September 6, 2018 12:21:24 PM MDT Steven Schveighoffer via
Digitalmars-d-learn wrote:
> On 9/6/18 12:55 PM, Jonathan M Davis wrote:
> > On Thursday, September 6, 2018 2:40:08 AM MDT Saurabh Das via
> > Digitalmars-d->
> > learn wrote:
> >> Is this a bug with writeln?
> >>
> >> void main()
> >> {
> >>
> >> import std.stdio, std.range, std.algorithm;
> >>
> >> auto a1 = sort([1,3,5,4,2]);
> >> auto a2 = sort([9,8,9]);
> >> auto a3 = sort([5,4,5,4]);
> >>
> >> pragma(msg, typeof(a1));
> >> pragma(msg, typeof(a2));
> >> pragma(msg, typeof(a3));
> >>
> >> auto b = [a1, a2, a3];
> >> pragma(msg, typeof(b));
> >>
> >> writeln("b:");
> >> writeln(b);
> >> writeln(b); // <-- this one prints incorrectly
> >>
> >> writeln("a:");
> >> writeln(a1);
> >> writeln(a2);
> >> writeln(a3);
> >>
> >> }
> >>
> >> Output
> >> ==
> >>
> >> SortedRange!(int[], "a < b")
> >> SortedRange!(int[], "a < b")
> >> SortedRange!(int[], "a < b")
> >> SortedRange!(int[], "a < b")[]
> >> b:
> >> [[1, 2, 3, 4, 5], [8, 9, 9], [4, 4, 5, 5]]
> >> [[], [], []]
> >> a:
> >> [1, 2, 3, 4, 5]
> >> [8, 9, 9]
> >> [4, 4, 5, 5]
> >>
> >> The issue goes away if I cast 'b' to const before writeln. I
> >> think it is a bug, but maybe I am missing something?
> >
> > It's not a bug in writeln. Any time that a range is copied, you must not
> > do _anything_ else with the original unless copying it is equivalent to
> > calling save on it, because the semantics of copying a range are
> > unspecified. They vary wildly depending on the range type (e.g. copying
> > a dynamic array is equivalent to calling save, but copying a class
> > reference is not). When you pass the range to writeln, you must assumed
> > that it may have been consumed. And since you have range of ranges, you
> > must assume that the ranges that are contained may have been consumed.
> > If you want to pass them to writeln and then do anything else with
> > them, then you'll need to call save on every range involved (which is a
> > bit of a pain with a range of ranges, but it's necessary all the same).
>
> This is not necessarily true. It depends how the sub-ranges are returned.
>
> The bug is that formattedWrite takes ranges sometimes by ref, sometimes
> not.
>
> formattedWrite should call save on a forward range whenever it makes a
> copy, and it doesn't.
>
> Case in point, it doesn't matter if you call writeln(b.save), the same
> thing happens.
That's still not a bug in formattedWrite. save only duplicates the
outer-most range. And since writeln will ultimately iterate through the
inner ranges - which weren't saved - you end up with them being consumed.
When you're passing a range of ranges to a function, you need to recursively
save them if you don't want the inner ranges in the original range to be
consumed. Regardless of what formattedWrite does, it's a general issue with
any function that you pass a range of ranges. It comes right back to the
same issue of the semantics of copying ranges being unspecified and that you
therefore must always use save on any ranges involved if you want to then
use those ranges after having passed them to a function or copy them doing
anything else. It's that much more annoying when you're dealing with a range
of ranges rather than a range of something else, but the issue is the same.
- Jonathan M Davis
Re: Bug with writeln?
On 9/6/18 12:55 PM, Jonathan M Davis wrote:
On Thursday, September 6, 2018 2:40:08 AM MDT Saurabh Das via Digitalmars-d-
learn wrote:
Is this a bug with writeln?
void main()
{
import std.stdio, std.range, std.algorithm;
auto a1 = sort([1,3,5,4,2]);
auto a2 = sort([9,8,9]);
auto a3 = sort([5,4,5,4]);
pragma(msg, typeof(a1));
pragma(msg, typeof(a2));
pragma(msg, typeof(a3));
auto b = [a1, a2, a3];
pragma(msg, typeof(b));
writeln("b:");
writeln(b);
writeln(b); // <-- this one prints incorrectly
writeln("a:");
writeln(a1);
writeln(a2);
writeln(a3);
}
Output
==
SortedRange!(int[], "a < b")
SortedRange!(int[], "a < b")
SortedRange!(int[], "a < b")
SortedRange!(int[], "a < b")[]
b:
[[1, 2, 3, 4, 5], [8, 9, 9], [4, 4, 5, 5]]
[[], [], []]
a:
[1, 2, 3, 4, 5]
[8, 9, 9]
[4, 4, 5, 5]
The issue goes away if I cast 'b' to const before writeln. I
think it is a bug, but maybe I am missing something?
It's not a bug in writeln. Any time that a range is copied, you must not do
_anything_ else with the original unless copying it is equivalent to calling
save on it, because the semantics of copying a range are unspecified. They
vary wildly depending on the range type (e.g. copying a dynamic array is
equivalent to calling save, but copying a class reference is not). When you
pass the range to writeln, you must assumed that it may have been consumed.
And since you have range of ranges, you must assume that the ranges that are
contained may have been consumed. If you want to pass them to writeln and
then do anything else with them, then you'll need to call save on every
range involved (which is a bit of a pain with a range of ranges, but it's
necessary all the same).
This is not necessarily true. It depends how the sub-ranges are returned.
The bug is that formattedWrite takes ranges sometimes by ref, sometimes not.
formattedWrite should call save on a forward range whenever it makes a
copy, and it doesn't.
Case in point, it doesn't matter if you call writeln(b.save), the same
thing happens.
-Steve
Re: Bug with writeln?
On Thursday, September 6, 2018 2:40:08 AM MDT Saurabh Das via Digitalmars-d-
learn wrote:
> Is this a bug with writeln?
>
> void main()
> {
> import std.stdio, std.range, std.algorithm;
>
> auto a1 = sort([1,3,5,4,2]);
> auto a2 = sort([9,8,9]);
> auto a3 = sort([5,4,5,4]);
>
> pragma(msg, typeof(a1));
> pragma(msg, typeof(a2));
> pragma(msg, typeof(a3));
>
> auto b = [a1, a2, a3];
> pragma(msg, typeof(b));
>
> writeln("b:");
> writeln(b);
> writeln(b); // <-- this one prints incorrectly
>
> writeln("a:");
> writeln(a1);
> writeln(a2);
> writeln(a3);
>
> }
>
> Output
> ==
>
> SortedRange!(int[], "a < b")
> SortedRange!(int[], "a < b")
> SortedRange!(int[], "a < b")
> SortedRange!(int[], "a < b")[]
> b:
> [[1, 2, 3, 4, 5], [8, 9, 9], [4, 4, 5, 5]]
> [[], [], []]
> a:
> [1, 2, 3, 4, 5]
> [8, 9, 9]
> [4, 4, 5, 5]
>
> The issue goes away if I cast 'b' to const before writeln. I
> think it is a bug, but maybe I am missing something?
It's not a bug in writeln. Any time that a range is copied, you must not do
_anything_ else with the original unless copying it is equivalent to calling
save on it, because the semantics of copying a range are unspecified. They
vary wildly depending on the range type (e.g. copying a dynamic array is
equivalent to calling save, but copying a class reference is not). When you
pass the range to writeln, you must assumed that it may have been consumed.
And since you have range of ranges, you must assume that the ranges that are
contained may have been consumed. If you want to pass them to writeln and
then do anything else with them, then you'll need to call save on every
range involved (which is a bit of a pain with a range of ranges, but it's
necessary all the same).
In many cases, you can get away with passing a range to a function or use it
with foreach and then continue to use it after that, but that's only because
copying those ranges is equivalent to calling save on them. It doesn't work
if any of the ranges involved aren't saved when they're copied, and it
doesn't work in generic code, because you have no clue whether the ranges
that are going to be used with that code are going to be saved when they're
copied.
- Jonathan M Davis
Re: Bug with writeln?
On Thursday, 6 September 2018 at 09:06:21 UTC, Simen Kjærås wrote: On Thursday, 6 September 2018 at 08:40:08 UTC, Saurabh Das wrote: Is this a bug with writeln? Yup. What happens is writeln destructively iterates over b[i]. Since b[i] is a forward range, this shouldn't be done destructively. Instead, a copy should be made using b[i].save, somewhere deep in Phobos' bowels. For now, here's a workaround: writeln(b.dup); writeln(b); // Look ma, it works twice! .dup creates a copy of the array, and the destructive iteration happens on the copy instead. Issue: https://issues.dlang.org/show_bug.cgi?id=19229 PR: https://github.com/dlang/phobos/pull/6695
Re: Bug with writeln?
On Thursday, 6 September 2018 at 08:40:08 UTC, Saurabh Das wrote: Is this a bug with writeln? Yup. What happens is writeln destructively iterates over b[i]. Since b[i] is a forward range, this shouldn't be done destructively. Instead, a copy should be made using b[i].save, somewhere deep in Phobos' bowels. For now, here's a workaround: writeln(b.dup); writeln(b); // Look ma, it works twice! .dup creates a copy of the array, and the destructive iteration happens on the copy instead. -- Simen
