I have been reading previous threads, the original bug request, and exploring
the javadoc and implementation of toList() on Stream in JDK 16. I don’t want to
waste time rehashing previous discussions, but I want to understand and
prioritize the motivation of this change, and propose what I believe is a safer
alternative name for this method based on the current implementation:
Stream.toUnmodifiableList().
Big +1 to "let's not rehash previous discussions, but help us understand
the motivation." Stewarding the core libraries is a complex task, and
there are rarely hard-and-fast rules for doing the Right Thing.
Your question seems to have two main aspects:
- Why this method, why not others, and why now
- Why take such a strong anti-mutability position with this method
The desire for a Stream::toList method has a long history; when we first
did streams, it was one of the first convenience methods to be
"requested". We resisted then, for good reasons, but we knew this saga
was not over.
"Convenience" methods are a constant challenge in the JDK. On the one
hand, they are, well, convenient, and we want Java to be easy and
pleasant to program in. On the other, the number of potentially-useful
imaginable convenience methods is infinite, and the widespread
perception is that they are so easy, that all that is needed is for
someone to propose the idea. (The (admittedly soft) criteria we use for
judging whether a convenience method meets the bar is an interesting
one, which we can have separately.)
There are basically two stable points with respect to convenience
methods in API design; zero tolerance, and "don't worry, be happy". In
the former, the methods of an API are like a basis (ideally, an
orthonormal one) of a vector space; the minimum number of API points
from which you can derive all possible usages. At the other extreme,
every reasonable combination of methods gets its own special form of
expression. Of course, both are extremes (Stream::count and
IntStream::sum are conveniences for reduce, and even Haskell's Monad has
multiple ways to represent bind), but APIs tend to align themselves in
one direction or another. And, as the JDK APIs go, Streams treats
sparsity and orthogonality as virtues to be striven for.
Eclipse Collections chooses a different (and also valid!) philosophy:
completeness, and it walks the walk. (Having 81 (template-generated)
implementations of HashMap is proof.) Similarly, Tagir's StreamEx is an
example of an extension to Stream that takes the other approach. And
both are great! But also, they are not how the JDK rolls. Which is
fine; it's a big ecosystem, and there's room for multiple philosophies,
and each can find its fans and detractors.
The calls for a convenience for Stream::toList have come pretty much
continuously since we first resisted it (but, we knew even then that if
we had a lifetime budget for just one convenience method, it would end
up being toList.) We knew then that there would be questions to ask
about what the ideal dial settings would be for toList, and were not yet
ready to confront the question, nor did we want to add fuel to the
demands for more convenience methods ("No toSet? Inconsistent!")
When an API is new, and all things are possible, we tend to be in
"imagine everything we could put into it" mode, and streams was no
different. It is wise to resist this temptation -- and maybe even
over-rotate in the other direction -- to allow for some time for the
spirit of what you've built to make itself clear; even creators are not
always immediately clear on the nuances of their creation. So we tried
hard to resist the calls for unnecessary methods, knowing that they
could always be added, but not taken away, and also, allowing for the
true gaps to emerge from usage. (The first method to be added,
takeWhile(), was the very opposite of a convenience; it represented a
reasonable use case that the original design didn't support.)
So, why toList now? Well, a number of reasons. Collecting to a list is
one of the most common terminal operations, so any small irritant (like
a clumsy locution) adds up. And, as has been pointed out, it can be
more efficient if it is brought into the stream core rather than held at
arm's length through Collector. So if we're going to compromise our
principles in one place, after thinking about it for a long time, this
seemed a worthy candidate. (And still, we hesitate, because we knew it
would be firing the starting gun for the "But where's toSet?" arguments.)
So yes, there are lots of good reasons to continue to Just Say No to
conveniences, but, there are also reasonable times to make exceptions --
especially when it is not purely about convenience. And, data suggests
that toList is 5-10x more popular than the next most popular collector,
so there's a clear argument to say that toList is pretty special, and we
can stop there.
List is a mutable interface.
This is true to an extent (though even the specification of List makes
it clear the mutative methods are strictly optional), but even if it
were absolutely true, I am still not sure how relevant it is to what
streams should do. When I wrote Collectors::toList, ArrayList was
indeed the obvious default implementation choice -- but it was also
obviously not a very good choice. We didn't have an efficient
unmodifiable collection at the time, and wrapping with unmodifiableList
seemed like taxing a lot of well-behaved users for the would-be sins of
the few. But if we had efficient unmodifiable collections then, I would
absolutely, positively have made that choice.
Streams is an API that takes functional principles to heart, sometimes
even in ways that are uncomfortable to Java developers. (For example, it
imposes constraints on the lambdas we pass to its methods, which are the
Java analogues of purity and side-effect freedom -- which are not
necessarily familiar constraints.) Data structures are about managing
and organizing data in memory, but streams are about capturing and
composing behavior, not data. (Obviously, streams consume and produce
data at their extreme points, but it tries to make the fewest possible
assumptions about the form that data takes.) Where Stream meets List,
Stream is allowed to have an opinion about what kinds of lists it likes
better, and an unmodifiable list seems far more in the spirit of
Streams. And of course, collect(toCollection(f)) lets you collect to
whatever sort of collection you like.
A convention was established in 2014 with Collectors.toList() returning a
mutable List (ArrayList).
I am having a hard time expressing just how much I disagree with the
sentiment behind this claim. I knew, when I was writing
Collectors::toList, that I would someday be having this discussion; my
best efforts to head this discussion off were memorialized in the
specification for Collectors::toList:
There are no guarantees on the type, mutability, serializability, or
thread-safety of the|List|returned; if more control over the
returned|List|is required, use|toCollection(Supplier)|
<https://docs.oracle.com/javase/8/docs/api/java/util/stream/Collectors.html#toCollection-java.util.function.Supplier->.
I'd hope this would be interpreted as: "Dear developer who assumes that,
just because this returns ArrayList today, that somehow it is reasonable
to assume toList will always return an ArrayList: you are wrong, and I
hope you have the good sense to never make this argument out loud."
The reason that this "reasonable-seeming" assumption -- that what the
first implementation does is reasonable to take as normative, even when
the spec says otherwise -- is so toxic, is that it cripples the ability
of the platform to evolve. There's a reason we write specifications for
APIs; because implementations are intrinsically accidental and
contextual, and context changes out from under us. Even when writing
it, I was aware of the degree to which programmers would be
overwhelmingly tempted (despite what I hoped was their better judgment)
to count on the mutability of the returned list if that is what they
wanted. Saying `toCollection(ArrayList::new)`, which guarantees exactly
the characteristics such users would want, is Just Not That Hard. Sure,
saying toList() is easier, but the tradeoff there is you accept whatever
(compliant) List the library wants to serve up, and the library gets
some say in what that is, and which might even vary from tuesday to
wednesday. A toList() method should try to balance the competing
concerns for what is the most reasonable default, and when the JDK
improves in a way that shifts that balance, or the context shifts, the
JDK should be able to improve with it.
So, this "establish a convention" claim is dangerous because it pushes
us towards the assumption that everything the JDK does, even the things
it *clearly specifies as implementation details that might change*, can
never change. Which means we would have to be *even more deliberate*
about anything we do, which means the rate at which we can move forward
is *even slower*.
But, you are making an even stronger claim than that! We're not trying
to change the implementation of Collectors::toList (which the spec makes
clear should might happen.) We're adding _another_ method with that
name, somewhere else. Which makes the above argument even more
dangerous -- essentially, it says "don't use a word in any API ever,
unless you are prepared to interpret it exactly the same way in all
future contexts." Surely, you see how this doesn't lead to a world we
want to live in.
So, what should `Stream::toList` mean? it should mean: return whatever
kind of list that Streams thinks is the best all-around default
implementation to use, based on the best understanding of what typical
users want. This involves balancing a lot of things, and that balance
can move over time.
We could call this toUnmodifiableList, and there's surely a certain
logic to that. But, this is likely to have unintended consequences.
First, the fact that the name is fussier makes it even less attractive
as a convenience, which is an argument to not do it at all. Users who
mostly count characters (which is sadly common) would be more likely to
continue to use collect(toList()), even if the new method is better in
multiple ways. If we have Stream::toUnmodifiableList, it is *even more
likely* to generate demands for other toXxxList conveniences. Worse, it
would likely generate arguments for a toList that works the same as
collect(toList()) -- which takes an existing "accidental mutability"
problem and guarantees that problem into the infinite future. It's bad
enough that collect(toList()) yields a mutable list -- it would be even
worse for Stream::toList to do the same. Most users don't need
mutability, and are better off not getting it if they don't need it;
they should ask for it if they need it.
[1] Example usages of Eclipse Collections toList:
// toList result is mutable for all of these usages with Eclipse Collections
List list1 = mutableSet.toList();
List list2 = mutableSet.asLazy().toList();
List list3 = mutableSet.asParallel(Executors.newWorkStealingPool(),
10).toList();
List list4 = mutableSet.stream().collect(Collectors.toList());
List list5 = mutableSet.stream().collect(Collectors2.toList());
These are nice, but there's a subtle difference here that is salient.
Eclipse Collections attempts to integrate data management and behavioral
composition into a single library. This is a fine goal, but it does
mean that the behavioral methods have more responsibility to fit with
the data-management side of the story.
Streams took an almost opposite interpretation -- one reason NOT to do a
Stream::toList method was that it overly coupled Streams to
Collections. Laundering stream-to-List via a specific collector (which
is clearly more of a "plug in" than core functionality) seemed
preferable. We chose more of an arms-length relationship between Stream
and Collections. Again, different philosophies. (Adding Stream::toList
goes back on that a bit, after thinking about it for a bunch of years,
and deciding it was OK in this case.)
The primary cost here is a seeming "inconsistency", because people have
been able to convince themselves that `toList()` means "to ArrayList",
and now, there will be cases where that is not true. Given the choice
between catering to explicitly wrong assumptions (the spec even says
"don't make this assumption"!), and improving the platform over time, I
choose the latter. Consistency is a good baseline goal, but
consistencies can be taken to foolish extremes.
