Does anyone have context on jdk.httpserver?
Hi all, Elliot[1] and I have been digging into the HTTP(S) server implementation provided by the jdk.httpserver module. It hasn't taken long to notice that the provided implementation is...lacking. Both in performance, as it places extremely low and errors out on benchmarks [2][3], and in polish, as doing relatively simple tasks like printing out the result of HttpExchange#getRequestHeaders() does not actually include the headers in the output. But, the actual API isn't wretched. The overall design (with a Filter chain and a mutable "out" object for responses) is reminiscent of Servlets and there is an SPI hook to provide custom server implementations. And unlike the jwebserver tool, which is built on this, there isn't a "for education purposes only" sign anywhere and it is part of an exported and supposedly supported module. All that is to ask - does anyone on this mailing list have any historical context on this? Why was it added to the JDK, what were the goals of its implementation, are there any records of the decision process behind its design? Separately is there any appetite for improving the performance of the built-in server implementation? [1]: https://github.com/ebarlas/microhttp [2]: https://www.techempower.com/benchmarks/#hw=ph=plaintext=data-r22 [3]: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Java/httpserver
Should we actually make Connection a TemplateProcessor?
One of the examples in the String Template JEPs, and a stated motivating
factor behind its design, is SQL. Template processors are objects so that
use cases like constructing SQL statements aren't injection prone.
The questions I want to pose are:
* Should java.sql provide an implementation of TemplateProcessor
* Where should that implementation live?
* What, exactly, should be the translation strategy.
The reason I think this isn't an obvious yes and ask that last question is
this.
Say this is some user's code.
try (var conn = ds.getConnection();
var stmt = conn.prepareStatement("""
SELECT user.name
WHERE user.height > ? AND user.width < ?
""")) {
stmt.setInt(1, height);
stmt.setInt(2, width);
var rs = stmt.executeQuery();
process(rs);
}
The transliteration to string templates would be something like
try (var conn = ds.getConnection();
var stmt = conn."""
SELECT user.name
WHERE user.height > \{height} AND user.width < \{width}
""")) {
var rs = stmt.executeQuery();
process(rs);
}
Whether Connection implements TemplateProcessor directly or its something
that you wrap a connection with is somewhat immaterial.
How should we handle "partial templating"?
try (var conn = ds.getConnection();
var stmt = conn."""
SELECT user.name
WHERE user.height > ? AND user.width < \{width}
""")) {
var rs = stmt.executeQuery();
rs.setInt(1, height);
process(rs);
}
Or
try (var conn = ds.getConnection();
var stmt = conn."""
SELECT user.name
WHERE user.height > \{height} AND user.width < ?
""")) {
var rs = stmt.executeQuery();
rs.setInt(2, width);
process(rs);
}
Is replacing every substitution with ? and calling set* is enough? How
could it be known, without parsing the specific sql dialect, what index to
use for the parameter?
try (var conn = ds.getConnection();
var stmt = conn."""
SELECT user.name
WHERE user.name <> '???'
AND user.height > ?
AND user.width < \{width}
""")) {
var rs = stmt.executeQuery();
rs.setInt(1, height);
process(rs);
}
(this seems more library design than language design, hence I sent it here.
I can forward to amber-dev if that is better)
Re: The introduction of Sequenced collections is not a source compatible change
r own collections (or type system around collections in case
> of Kotlin), so knowing the real impact of this change is hard here.
>
> The problem of using a corpus experiment is that the corpus may not
> represent the current state of the Java ecosystem, or at least the one that
> may be impacted.
>
> The problem with the corpus experiment is also that you need to be aware
> that most moden open source projects use "--release" flag, so you have to
> patch it away from the build system.
>
> In my case, on my own repositories (public and private), i had only one
> occurrence of the issue in the main source codes because many of those
> repositories are not using 'var' or even the stream API but on the corpus
> of the unit tests we give to students to check their implementations,
> little less than a third of those JUnit classes had source compatibility
> issues because those tests are using 'var' and different collections
> heavily.
>
> And the situation is a little worst than that because in between now and
> the time people will use Java 21, a lot of codes will be written using Java
> 11 and 17 and may found incompatible later.
>
> A source incompatibility issue is not a big deal, as said in this thread,
> most of the time, explicitly fixing the type argument instead of inferring
> it make the code compile again.
> So the house is not burning, but we should raise awareness of this issue
> given that it may have a bigger impact than other source incompatible
> changes that occur previously.
>
> Rémi
>
> --
>
> *From: *"joe darcy"
> *To: *"Ethan McCue" , "Raffaello
> Giulietti"
>
> *Cc: *"Remi Forax" , "Stuart
> Marks" ,
> "core-libs-dev"
>
> *Sent: *Friday, May 5, 2023 4:38:16 AM
> *Subject: *Re: The introduction of Sequenced collections is not a source
> compatible change
>
> A few comments on the general compatibility policy for the JDK.
> Compatibility is looked after by the Compatibility and Specification Review
> (CSR) process ( Compatibility & Specification Review). Summarizing the
> approach,
>
> The general compatibility policy for exported APIs implemented in the JDK
> is:
>
> * Don't break binary compatibility (as defined in the Java Language
> Specification) without sufficient cause.
> * Avoid introducing source incompatibilities.
> * Manage behavioral compatibility changes.
>
> https://wiki.openjdk.org/display/csr/Main
>
> None of binary, source, and behavioral compatibly are absolutes and
> judgement is used to assess the cost/benefits of changes. For example,
> strict source compatibility would preclude, say, introducing new public
> types in the java.lang package since the implicit import of types in
> java.lang could conflict with a same-named type *-imported from another
> package.
>
> When a proposed change is estimated to be sufficiently disruptive, we
> conduct a corpus experiment to evaluate the impact on the change on many
> public Java libraries. Back in Project Coin in JDK 7, that basic approach
> was used to help quantify various language design choices and the
> infrastructure to run such experiments has been built-out in the subsequent
> releases.
>
> HTH,
>
> -Joe
> CSR Group Lead
> On 5/4/2023 6:32 AM, Ethan McCue wrote:
>
> I guess this a good time to ask, ignoring the benefit part of a cost
> benefit analysis, what mechanisms do we have to measure the number of
> codebases relying on type inference this will break?
>
> Iirc Adoptium built/ran the unit tests of a bunch of public repos, but
> it's also a bit shocking if the jtreg suite had nothing for this.
>
> On Thu, May 4, 2023, 9:27 AM Raffaello Giulietti <
> raffaello.giulie...@oracle.com> wrote:
>
>> Without changing the semantics at all, you could also write
>>
>> final List> list =
>> Stream.>of(nestedDequeue, nestedList).toList();
>>
>> to "help" type inference.
>>
>>
>>
>>
>> On 2023-05-03 15:12, fo...@univ-mlv.fr wrote:
>> > Another example sent to me by a fellow French guy,
>> >
>> > final Deque nestedDequeue = new ArrayDeque<>();
>> > nestedDequeue.addFirst("C");
>> > nestedDequeue.addFirst("B");
>> > nestedDequeue.addFirst("A");
>> >
>> > final List nestedList = new ArrayList<>();
>> > nestedList.add("D");
>> > nestedList.add("E");
>> > nestedList.add("F");
>> >
>> > final List> list = Stream.of(nestedD
Re: The introduction of Sequenced collections is not a source compatible change
For those who are tracking but didn't click through to find it, here was
the writeup of the compatibility analysis for sequenced collections as of
9/20/22.
https://bugs.openjdk.org/secure/attachment/100811/compatibility-20220920.html
Bug page:
https://bugs.openjdk.org/browse/JDK-8266572
On Fri, May 5, 2023 at 12:08 AM Joseph D. Darcy
wrote:
> PS And as a general policy, over the releases we've done more to
> preserve source compatibility when possible for language changes.
>
> For example, there were many source compatibility breakages when
> "assert" was added as a keyword back in JDK 1.4; the current keyword
> management policies (https://openjdk.org/jeps/8223002), including
> contextual keywords (JLS §3.9) and hyphenated-keywords, mitigate the
> impact of analogous changes today.
>
> -Joe
>
> On 5/4/2023 7:47 PM, - wrote:
> > In addition, in the CSR of sequenced collection, it already
> > anticipated some other form of source incompatibility:
> > If a class implements List and Deque at the same time, the return type
> > of reversed() must extend both interfaces as well.
> >
> > This alone would be a greater source incompatibility than this type
> > interference already; however, both are binarily compatible:
> > Java doesn't care about generics at runtime, and for default
> > reversed() overrides, reversed ()Ljava/util/List; and reversed
> > ()Ljava/util/Deque; are distinct methods; code calling reversed via
> > List or Deque interfaces will get the reversed version of respective
> > interfaces, which are functionally correct as well.
> >
> > I personally would consider this type inference usage moot. It is same
> > as adding another method addAll(List) when there is already
> > addAll(String[]): existing addAll(null) calls will break, but this is
> > not a valid argument against adding a List variant of addAll.
> >
> > On Thu, May 4, 2023 at 9:38 PM Joseph D. Darcy
> wrote:
> >> A few comments on the general compatibility policy for the JDK.
> Compatibility is looked after by the Compatibility and Specification Review
> (CSR) process ( Compatibility & Specification Review). Summarizing the
> approach,
> >>
> >> The general compatibility policy for exported APIs implemented in the
> JDK is:
> >>
> >> * Don't break binary compatibility (as defined in the Java
> Language Specification) without sufficient cause.
> >> * Avoid introducing source incompatibilities.
> >> * Manage behavioral compatibility changes.
> >>
> >> https://wiki.openjdk.org/display/csr/Main
> >>
> >> None of binary, source, and behavioral compatibly are absolutes and
> judgement is used to assess the cost/benefits of changes. For example,
> strict source compatibility would preclude, say, introducing new public
> types in the java.lang package since the implicit import of types in
> java.lang could conflict with a same-named type *-imported from another
> package.
> >>
> >> When a proposed change is estimated to be sufficiently disruptive, we
> conduct a corpus experiment to evaluate the impact on the change on many
> public Java libraries. Back in Project Coin in JDK 7, that basic approach
> was used to help quantify various language design choices and the
> infrastructure to run such experiments has been built-out in the subsequent
> releases.
> >>
> >> HTH,
> >>
> >> -Joe
> >> CSR Group Lead
> >>
> >> On 5/4/2023 6:32 AM, Ethan McCue wrote:
> >>
> >> I guess this a good time to ask, ignoring the benefit part of a cost
> benefit analysis, what mechanisms do we have to measure the number of
> codebases relying on type inference this will break?
> >>
> >> Iirc Adoptium built/ran the unit tests of a bunch of public repos, but
> it's also a bit shocking if the jtreg suite had nothing for this.
> >>
> >> On Thu, May 4, 2023, 9:27 AM Raffaello Giulietti <
> raffaello.giulie...@oracle.com> wrote:
> >>> Without changing the semantics at all, you could also write
> >>>
> >>> final List> list =
> >>> Stream.>of(nestedDequeue, nestedList).toList();
> >>>
> >>> to "help" type inference.
> >>>
> >>>
> >>>
> >>>
> >>> On 2023-05-03 15:12, fo...@univ-mlv.fr wrote:
> >>>> Another example sent to me by a fellow French guy,
> >>>>
> >>>> final Deque nestedDequeue = new ArrayDeque<>();
> >>>> nestedDequeue.addFirst("C");
> >&
Re: The introduction of Sequenced collections is not a source compatible change
I guess this a good time to ask, ignoring the benefit part of a cost
benefit analysis, what mechanisms do we have to measure the number of
codebases relying on type inference this will break?
Iirc Adoptium built/ran the unit tests of a bunch of public repos, but it's
also a bit shocking if the jtreg suite had nothing for this.
On Thu, May 4, 2023, 9:27 AM Raffaello Giulietti <
raffaello.giulie...@oracle.com> wrote:
> Without changing the semantics at all, you could also write
>
> final List> list =
> Stream.>of(nestedDequeue, nestedList).toList();
>
> to "help" type inference.
>
>
>
>
> On 2023-05-03 15:12, fo...@univ-mlv.fr wrote:
> > Another example sent to me by a fellow French guy,
> >
> > final Deque nestedDequeue = new ArrayDeque<>();
> > nestedDequeue.addFirst("C");
> > nestedDequeue.addFirst("B");
> > nestedDequeue.addFirst("A");
> >
> > final List nestedList = new ArrayList<>();
> > nestedList.add("D");
> > nestedList.add("E");
> > nestedList.add("F");
> >
> > final List> list = Stream.of(nestedDequeue,
> nestedList).toList();
> >
> > This one is cool because no 'var' is involved and using
> collect(Collectors.toList()) instead of toList() solves the inference
> problem.
> >
> > Rémi
> >
> > - Original Message -
> >> From: "Stuart Marks"
> >> To: "Remi Forax"
> >> Cc: "core-libs-dev"
> >> Sent: Tuesday, May 2, 2023 2:44:28 AM
> >> Subject: Re: The introduction of Sequenced collections is not a source
> compatible change
> >
> >> Hi Rémi,
> >>
> >> Thanks for trying out the latest build!
> >>
> >> I'll make sure this gets mentioned in the release note for Sequenced
> >> Collections.
> >> We'll also raise this issue when we talk about this feature in the
> Quality
> >> Outreach
> >> program.
> >>
> >> s'marks
> >>
> >> On 4/29/23 3:46 AM, Remi Forax wrote:
> >>> I've several repositories that now fails to compile with the latest
> jdk21, which
> >>> introduces sequence collections.
> >>>
> >>> The introduction of a common supertype to existing collections is
> *not* a source
> >>> compatible change because of type inference.
> >>>
> >>> Here is a simplified example:
> >>>
> >>> public static void m(List>>
> factories) {
> >>> }
> >>>
> >>> public static void main(String[] args) {
> >>> Supplier> supplier1 =
> LinkedHashMap::new;
> >>> Supplier> supplier2 = TreeMap::new;
> >>> var factories = List.of(supplier1, supplier2);
> >>> m(factories);
> >>> }
> >>>
> >>>
> >>> This example compiles fine with Java 20 but report an error with Java
> 21:
> >>> SequencedCollectionBug.java:28: error: method m in class
> SequencedCollectionBug
> >>> cannot be applied to given types;
> >>> m(factories);
> >>> ^
> >>> required: List>>
> >>> found:List>>
> >>> reason: argument mismatch; List SequencedMap>>
> >>> cannot be converted to List>>
> >>>
> >>>
> >>>
> >>> Apart from the example above, most of the failures I see are in the
> unit tests
> >>> provided to the students, because we are using a lot of 'var' in them
> so they
> >>> work whatever the name of the types chosen by the students.
> >>>
> >>> Discussing with a colleague, we also believe that this bug is not
> limited to
> >>> Java, existing Kotlin codes will also fail to compile due to this bug.
> >>>
> >>> Regards,
> >>> Rémi
>
Re: JEP-198 - Lets start talking about JSON
Link to the proxy which I forgot to include
https://gist.github.com/bowbahdoe/eb29d172351162408eab5e4ee9d84fec
On Tue, Feb 28, 2023 at 12:16 PM Ethan McCue wrote:
> As an update to my character arc, I documented and wrote up an explanation
> for the prototype library I was working on.[1]
>
> And I've gotten a good deal of feedback on reddit[2] and in private.
>
> I think its relevant to the conversation here in the sense of
>
> - There are more of rzwitserloot's objections to read on the general
> concept JSON as a built in.[3]
> - There are a lot of well reasoned objections to the manner in which I am
> interpreting a JSON tree, as well
> as objections to the usage of a tree as the core. JEP 198's current
> writeup (which I know is subject to a rewrite/retraction)
> presumes that an immutable tree would be the core data structure.
> - The peanut gallery might be interested in a "base" to implement whatever
> their take on an API should be.
>
> For that last category, I have a method-handle proxy written up for those
> who want to try the "push parser into a pull parser"
> transformation I alluded to in my first email of this thread.
>
> [1]: https://mccue.dev/pages/2-26-23-json
> [2]:
> https://www.reddit.com/r/java/comments/11cyoh1/please_try_my_json_library/
> [3]: Including one that reddit took down, but can be seen through reveddit
> https://www.reveddit.com/y/rzwitserloot/?after=t1_jacpsj6=1=new=t1_jaa3x0q_status=all
>
> On Fri, Dec 16, 2022 at 6:23 PM Ethan McCue wrote:
>
>> Sidenote about "Project Galahad" - I know Graal uses json for a few
>> things including a reflection-config.json. Food for thought.
>>
>> > the java.util.log experiment shows that trying to ‘core-librarize’
>> needs that the community at large already fulfills with third party deps
>> isn’t a good move,
>>
>> I, personally, do not have much historical context for java.util.log.
>> What feels distinct about providing a JSON api is that
>> logging is an implicitly global thing. If a JSON api doesn't fill all
>> ecosystem niches, multiple can be used alongside
>> each other.
>>
>> > The root issue with JSON is that you just can’t tell how to interpret
>> any given JSON token
>>
>> The point where this could be an issue is numbers. Once something is
>> identified as a number we can
>>
>> 1. Parse it immediately. Using a long and falling back to a BigInteger.
>> For decimals its harder to know
>> whether to use a double or BigDecimal internally. In the library I've
>> been copy pasting from to build
>> a prototype that last one is an explicit option and it defaults to
>> doubles for the whole parse.
>> 2. Store the string and parse it upon request. We can still model it as a
>> Json.Number, but the
>> work of interpreting is deferred.
>>
>> But in general, making a tree of json values doesn't particularly affect
>> our ability to interpret it
>> in a certain way. That interpretation is just positional. That's just as
>> true as when making assertions
>> in the form of class structure and field types as it is when making
>> assertions in the form of code.[2]
>>
>> record Thing(Instant a) {}
>>
>> // vs.
>>
>> Decoder.field(json, "a", a -> Instant.ofEpochSecond(Decoder.long_(a)))
>>
>> If anything, using a named type as a lookup key for a deserialization
>> function is the less obvious
>> way to do this.
>>
>> > I’m not sure how to square this circle
>> > I don’t like the idea of shipping a non-data-binding JSON API in the
>> core libs.
>>
>> I think the way to cube this rhombus is to find ways to like the idea of
>> a non-data-binding JSON API. ¯\_(ツ)_/¯
>>
>> My personal journey with that is reaching its terminus here I think.
>>
>> Look on the bright side though - there are legit upsides to explicit tree
>> plucking!
>>
>> Yeah, the friction per field is slightly higher, but the relative
>> friction of custom types, or multiple construction methods for a
>> particular type, or maintaining compatibility with
>> legacy representations, or even just handling a top level list of things
>> - its much lower.
>>
>> And all that complexity - that an instant is made by looking for a long
>> or that it is parsed from a string in a
>> particular format - it lives in Java code you can see, touch, feel and
>> taste.
>>
>> I know "nobody does this"[2] but it's not that bad, actually.
>>
>> [1]: I do apologize for the code sketches consistentl
Re: JEP-198 - Lets start talking about JSON
As an update to my character arc, I documented and wrote up an explanation
for the prototype library I was working on.[1]
And I've gotten a good deal of feedback on reddit[2] and in private.
I think its relevant to the conversation here in the sense of
- There are more of rzwitserloot's objections to read on the general
concept JSON as a built in.[3]
- There are a lot of well reasoned objections to the manner in which I am
interpreting a JSON tree, as well
as objections to the usage of a tree as the core. JEP 198's current writeup
(which I know is subject to a rewrite/retraction)
presumes that an immutable tree would be the core data structure.
- The peanut gallery might be interested in a "base" to implement whatever
their take on an API should be.
For that last category, I have a method-handle proxy written up for those
who want to try the "push parser into a pull parser"
transformation I alluded to in my first email of this thread.
[1]: https://mccue.dev/pages/2-26-23-json
[2]:
https://www.reddit.com/r/java/comments/11cyoh1/please_try_my_json_library/
[3]: Including one that reddit took down, but can be seen through reveddit
https://www.reveddit.com/y/rzwitserloot/?after=t1_jacpsj6=1=new=t1_jaa3x0q_status=all
On Fri, Dec 16, 2022 at 6:23 PM Ethan McCue wrote:
> Sidenote about "Project Galahad" - I know Graal uses json for a few things
> including a reflection-config.json. Food for thought.
>
> > the java.util.log experiment shows that trying to ‘core-librarize’ needs
> that the community at large already fulfills with third party deps isn’t a
> good move,
>
> I, personally, do not have much historical context for java.util.log. What
> feels distinct about providing a JSON api is that
> logging is an implicitly global thing. If a JSON api doesn't fill all
> ecosystem niches, multiple can be used alongside
> each other.
>
> > The root issue with JSON is that you just can’t tell how to interpret
> any given JSON token
>
> The point where this could be an issue is numbers. Once something is
> identified as a number we can
>
> 1. Parse it immediately. Using a long and falling back to a BigInteger.
> For decimals its harder to know
> whether to use a double or BigDecimal internally. In the library I've been
> copy pasting from to build
> a prototype that last one is an explicit option and it defaults to doubles
> for the whole parse.
> 2. Store the string and parse it upon request. We can still model it as a
> Json.Number, but the
> work of interpreting is deferred.
>
> But in general, making a tree of json values doesn't particularly affect
> our ability to interpret it
> in a certain way. That interpretation is just positional. That's just as
> true as when making assertions
> in the form of class structure and field types as it is when making
> assertions in the form of code.[2]
>
> record Thing(Instant a) {}
>
> // vs.
>
> Decoder.field(json, "a", a -> Instant.ofEpochSecond(Decoder.long_(a)))
>
> If anything, using a named type as a lookup key for a deserialization
> function is the less obvious
> way to do this.
>
> > I’m not sure how to square this circle
> > I don’t like the idea of shipping a non-data-binding JSON API in the
> core libs.
>
> I think the way to cube this rhombus is to find ways to like the idea of a
> non-data-binding JSON API. ¯\_(ツ)_/¯
>
> My personal journey with that is reaching its terminus here I think.
>
> Look on the bright side though - there are legit upsides to explicit tree
> plucking!
>
> Yeah, the friction per field is slightly higher, but the relative
> friction of custom types, or multiple construction methods for a
> particular type, or maintaining compatibility with
> legacy representations, or even just handling a top level list of things -
> its much lower.
>
> And all that complexity - that an instant is made by looking for a long or
> that it is parsed from a string in a
> particular format - it lives in Java code you can see, touch, feel and
> taste.
>
> I know "nobody does this"[2] but it's not that bad, actually.
>
> [1]: I do apologize for the code sketches consistently being "what I think
> an interaction with a tree api should look like."
> That is what I have been thinking about for a while so it's hard to resist.
> [2]: https://youtu.be/dOgfWXw9VrI?t=1225
>
> On Thu, Dec 15, 2022 at 6:34 PM Ethan McCue wrote:
>
>> > are pure JSON parsers really the go-to for most people?
>>
>> Depends on what you mean by JSON parsers and it depends on what you mean
>> by people.
>>
>> To the best of my knowledge, both python and Javascript do not include
>> streaming, databinding, or path navigation
Re: JEP-198 - Lets start talking about JSON
Sidenote about "Project Galahad" - I know Graal uses json for a few things
including a reflection-config.json. Food for thought.
> the java.util.log experiment shows that trying to ‘core-librarize’ needs
that the community at large already fulfills with third party deps isn’t a
good move,
I, personally, do not have much historical context for java.util.log. What
feels distinct about providing a JSON api is that
logging is an implicitly global thing. If a JSON api doesn't fill all
ecosystem niches, multiple can be used alongside
each other.
> The root issue with JSON is that you just can’t tell how to interpret any
given JSON token
The point where this could be an issue is numbers. Once something is
identified as a number we can
1. Parse it immediately. Using a long and falling back to a BigInteger. For
decimals its harder to know
whether to use a double or BigDecimal internally. In the library I've been
copy pasting from to build
a prototype that last one is an explicit option and it defaults to doubles
for the whole parse.
2. Store the string and parse it upon request. We can still model it as a
Json.Number, but the
work of interpreting is deferred.
But in general, making a tree of json values doesn't particularly affect
our ability to interpret it
in a certain way. That interpretation is just positional. That's just as
true as when making assertions
in the form of class structure and field types as it is when making
assertions in the form of code.[2]
record Thing(Instant a) {}
// vs.
Decoder.field(json, "a", a -> Instant.ofEpochSecond(Decoder.long_(a)))
If anything, using a named type as a lookup key for a deserialization
function is the less obvious
way to do this.
> I’m not sure how to square this circle
> I don’t like the idea of shipping a non-data-binding JSON API in the core
libs.
I think the way to cube this rhombus is to find ways to like the idea of a
non-data-binding JSON API. ¯\_(ツ)_/¯
My personal journey with that is reaching its terminus here I think.
Look on the bright side though - there are legit upsides to explicit tree
plucking!
Yeah, the friction per field is slightly higher, but the relative
friction of custom types, or multiple construction methods for a particular
type, or maintaining compatibility with
legacy representations, or even just handling a top level list of things -
its much lower.
And all that complexity - that an instant is made by looking for a long or
that it is parsed from a string in a
particular format - it lives in Java code you can see, touch, feel and
taste.
I know "nobody does this"[2] but it's not that bad, actually.
[1]: I do apologize for the code sketches consistently being "what I think
an interaction with a tree api should look like."
That is what I have been thinking about for a while so it's hard to resist.
[2]: https://youtu.be/dOgfWXw9VrI?t=1225
On Thu, Dec 15, 2022 at 6:34 PM Ethan McCue wrote:
> > are pure JSON parsers really the go-to for most people?
>
> Depends on what you mean by JSON parsers and it depends on what you mean
> by people.
>
> To the best of my knowledge, both python and Javascript do not include
> streaming, databinding, or path navigation capabilities in their json
> parsers.
>
>
> On Thu, Dec 15, 2022 at 6:26 PM Ethan McCue wrote:
>
>> > The 95%+ use case for working with JSON for your average java coder is
>> best done with data binding.
>>
>> To be brave yet controversial: I'm not sure this is neccesarily true.
>>
>> I will elaborate and respond to the other points after a hot cocoa, but
>> the last point is part of why I think that tree-crawling needs _something_
>> better as an API to fit the bill.
>>
>> With my sketch that set of requirements would be represented as
>>
>> record Thing(
>> List xs
>> ) {
>> static Thing fromJson(Json json)
>> var defaultList = List.of(0L);
>> return new Thing(Decoder.optionalNullableField(
>> json
>> "xs",
>> Decoder.oneOf(
>> Decoder.array(Decoder.oneOf(
>> x -> Long.parseLong(Decoder.string(x)),
>> Decoder::long
>> ))
>> Decoder.null_(defaultList),
>> x -> List.of(Decoder.long_(x))
>> ),
>> defaultList
>> ));
>> )
>> }
>>
>> Which isn't amazing at first glance, but also
>>
>>{}
>>{"xs": null}
>>{"xs": 5}
>>{"xs": [5]} {"xs": ["5"]}
>>{"xs": [1, &quo
Re: JEP-198 - Lets start talking about JSON
> are pure JSON parsers really the go-to for most people?
Depends on what you mean by JSON parsers and it depends on what you mean by
people.
To the best of my knowledge, both python and Javascript do not include
streaming, databinding, or path navigation capabilities in their json
parsers.
On Thu, Dec 15, 2022 at 6:26 PM Ethan McCue wrote:
> > The 95%+ use case for working with JSON for your average java coder is
> best done with data binding.
>
> To be brave yet controversial: I'm not sure this is neccesarily true.
>
> I will elaborate and respond to the other points after a hot cocoa, but
> the last point is part of why I think that tree-crawling needs _something_
> better as an API to fit the bill.
>
> With my sketch that set of requirements would be represented as
>
> record Thing(
> List xs
> ) {
> static Thing fromJson(Json json)
> var defaultList = List.of(0L);
> return new Thing(Decoder.optionalNullableField(
> json
> "xs",
> Decoder.oneOf(
> Decoder.array(Decoder.oneOf(
> x -> Long.parseLong(Decoder.string(x)),
> Decoder::long
> ))
> Decoder.null_(defaultList),
> x -> List.of(Decoder.long_(x))
> ),
> defaultList
> ));
> )
> }
>
> Which isn't amazing at first glance, but also
>
>{}
>{"xs": null}
>{"xs": 5}
>{"xs": [5]} {"xs": ["5"]}
>{"xs": [1, "2", "3"]}
>
> these are some wildly varied structures. You could make a solid argument
> that something which silently treats these all the same is
> a bad API for all the reasons you would consider it a good one.
>
> On Thu, Dec 15, 2022 at 6:18 PM Johannes Lichtenberger <
> lichtenberger.johan...@gmail.com> wrote:
>
>> I'll have to read the whole thing, but are pure JSON parsers really the
>> go-to for most people? I'm a big advocate of providing also something
>> similar to XPath/XQuery and that's IMHO JSONiq (90% XQuery). I might be
>> biased, of course, as I'm working on Brackit[1] in my spare time (which is
>> also a query compiler and intended to be used with proven optimizations by
>> document stores / JSON stores), but also can be used as an in-memory query
>> engine.
>>
>> kind regards
>> Johannes
>>
>> [1] https://github.com/sirixdb/brackit
>>
>> Am Do., 15. Dez. 2022 um 23:03 Uhr schrieb Reinier Zwitserloot <
>> rein...@zwitserloot.com>:
>>
>>> A recent Advent-of-Code puzzle also made me double check the support of
>>> JSON in the java core libs and it is indeed a curious situation that the
>>> java core libs don’t cater to it particularly well.
>>>
>>> However, I’m not seeing an easy way forward to try to close this hole in
>>> the core library offerings.
>>>
>>> If you need to stream huge swaths of JSON, generally there’s a clear
>>> unit size that you can just databind. Something like:
>>>
>>> String jsonStr = """ { "version": 5, "data": [
>>> -- 1 million relatively small records in this list --
>>> ] } """;
>>>
>>>
>>> The usual swath of JSON parsers tend to support this (giving you a
>>> stream of java instances created by databinding those small records one by
>>> one), or if not, the best move forward is presumably to file a pull request
>>> with those projects; the java.util.log experiment shows that trying to
>>> ‘core-librarize’ needs that the community at large already fulfills with
>>> third party deps isn’t a good move, especially if the core library variant
>>> tries to oversimplify to avoid the trap of being too opinionated (which
>>> core libs shouldn’t be). In other words, the need for ’stream this JSON for
>>> me’ style APIs is even more exotic that Ethan is suggesting.
>>>
>>> I see a fundamental problem here:
>>>
>>>
>>>- The 95%+ use case for working with JSON for your average java
>>>coder is best done with data binding.
>>>- core libs doesn’t want to provide it, partly because it’s got a
>>>large design space, partly because the field’s already covered by GSON
>>> and
>>>Jackson-json; java.util.log proves this doesn’t work. At least, I gather
>>>that’s what Ethan thinks and I
Re: JEP-198 - Lets start talking about JSON
le library (JSR310’s story, more or less). JSON parsing would >>qualify as ‘well-established’ (GSON and Jackson) and ‘large design space’ >>as Ethan pointed out. >>- Given that 99% of java projects, even really simple ones, start >>with maven/gradle and a list of deps, is that really a problem? >> >> >> I’m honestly not sure what the right answer is. On one hand, the npm >> ecosystem seems to be doing very well even though their ‘batteries >> included’ situation is an utter shambles. Then again, the notion that your >> average nodejs project includes 10x+ more dependencies than other languages >> is likely a significant part of the security clown fiesta going on over >> there as far as 3rd party deps is concerned, so by no means should java >> just blindly emulate their solutions. >> >> I don’t like the idea of shipping a non-data-binding JSON API in the core >> libs. The root issue with JSON is that you just can’t tell how to interpret >> any given JSON token, because that’s not how JSON is used in practice. What >> does 5 mean? Could be that I’m to take that as an int, or as a double, >> or perhaps even as a j.t.Instant (epoch-millis), and defaulting >> behaviour (similar to j.u.Map’s .getOrDefault is *very* convenient to >> parse most JSON out there in the real world - omitting k/v pairs whose >> value is still on default is very common). That’s what makes those databind >> libraries so enticing: Instead of trying to pattern match my way into this >> behaviour: >> >> >>- If the element isn’t there at all or null, give me a list-of-longs >>with a single 0 in it. >>- If the element is a number, make me a list-of-longs with 1 value in >>it, that is that number, as long. >>- If the element is a string, parse it into a long, then get me a >>list with this one long value (because IEEE double rules mean sometimes >> you >>have to put these things in string form or they get mangled by javascript- >>eval style parsers). >> >> >> And yet the above is quite common, and can easily be done by a >> databinder, which sees you want a List for a field whose default >> value is List.of(1L), and, armed with that knowledge, can transit the >> JSON into java in that way. >> >> You don’t *need* databinding to cater to this idea: You could for >> example have a jsonNode.asLong(123) method that would parse a string if >> need be, even. But this has nothing to do with pattern matching either. >> >> --Reinier Zwitserloot >> >> >> On 15 Dec 2022 at 21:30:17, Ethan McCue wrote: >> >>> I'm writing this to drive some forward motion and to nerd-snipe those >>> who know better than I do into putting their thoughts into words. >>> >>> There are three ways to process JSON[1] >>> - Streaming (Push or Pull) >>> - Traversing a Tree (Realized or Lazy) >>> - Declarative Databind (N ways) >>> >>> Of these, JEP-198 explicitly ruled out providing "JAXB style type safe >>> data binding." >>> >>> No justification is given, but if I had to insert my own: mapping the >>> Json model to/from the Java/JVM object model is a cursed combo of >>> - Huge possible design space >>> - Unpalatably large surface for backwards compatibility >>> - Serialization! Boo![2] >>> >>> So for an artifact like the JDK, it probably doesn't make sense to >>> include. That tracks. >>> It won't make everyone happy, people like databind APIs, but it tracks. >>> >>> So for the "read flow" these are the things to figure out. >>> >>> | Should Provide? | Intended User(s) | >>> +-+--+ >>> Streaming Push | | | >>> +-+--+ >>> Streaming Pull | | | >>> +-+--+ >>> Realized Tree | | | >>> +-+--+ >>> Lazy Tree | | | >>> +-+--+ >>> >>> At which point, we should talk about what "meets needs of Java >>> developers using JSON" implies. >>> >>> JSON is ubiquitous. Most kinds of software us schmucks write could have >>> a reason to interact with it. &g
JEP-198 - Lets start talking about JSON
I'm writing this to drive some forward motion and to nerd-snipe those who
know better than I do into putting their thoughts into words.
There are three ways to process JSON[1]
- Streaming (Push or Pull)
- Traversing a Tree (Realized or Lazy)
- Declarative Databind (N ways)
Of these, JEP-198 explicitly ruled out providing "JAXB style type safe data
binding."
No justification is given, but if I had to insert my own: mapping the Json
model to/from the Java/JVM object model is a cursed combo of
- Huge possible design space
- Unpalatably large surface for backwards compatibility
- Serialization! Boo![2]
So for an artifact like the JDK, it probably doesn't make sense to include.
That tracks.
It won't make everyone happy, people like databind APIs, but it tracks.
So for the "read flow" these are the things to figure out.
| Should Provide? | Intended User(s) |
+-+--+
Streaming Push | | |
+-+--+
Streaming Pull | | |
+-+--+
Realized Tree | | |
+-+--+
Lazy Tree | | |
+-+--+
At which point, we should talk about what "meets needs of Java developers
using JSON" implies.
JSON is ubiquitous. Most kinds of software us schmucks write could have a
reason to interact with it.
The full set of "user personas" therefore aren't practical for me to talk
about.[3]
JSON documents, however, are not so varied.
- There are small ones (1-10kb)
- There are medium ones (10-1000kb)
- There are big ones (1000kb-???)
- There are shallow ones
- There are deep ones
So that feels like an easier direction to talk about it from.
This repo[4] has some convenient toy examples of how some of those APIs
look in libraries
in the ecosystem. Specifically the Streaming Pull and Realized Tree models.
User r = new User();
while (true) {
JsonToken token = reader.peek();
switch (token) {
case BEGIN_OBJECT:
reader.beginObject();
break;
case END_OBJECT:
reader.endObject();
return r;
case NAME:
String fieldname = reader.nextName();
switch (fieldname) {
case "id":
r.setId(reader.nextString());
break;
case "index":
r.setIndex(reader.nextInt());
break;
...
case "friends":
r.setFriends(new ArrayList<>());
Friend f = null;
carryOn = true;
while (carryOn) {
token = reader.peek();
switch (token) {
case BEGIN_ARRAY:
reader.beginArray();
break;
case END_ARRAY:
reader.endArray();
carryOn = false;
break;
case BEGIN_OBJECT:
reader.beginObject();
f = new Friend();
break;
case END_OBJECT:
reader.endObject();
r.getFriends().add(f);
break;
case NAME:
String fn = reader.nextName();
switch (fn) {
case "id":
f.setId(reader.nextString());
break;
case "name":
f.setName(reader.nextString());
break;
}
break;
}
}
break;
}
}
I think its not hard to argue that the streaming apis are brutalist. The
above is Gson, but Jackson, moshi, etc
seem at least morally equivalent.
Its hard to write, hard to write *correctly*, and theres is a curious
protensity towards pairing it
with anemic, mutable models.
That being said, it handles big
Re: Boilerplate to add a new module
Subtract one negative from that last clause On Wed, Dec 14, 2022 at 12:02 PM Ethan McCue wrote: > Thank you! > > And yeah - I'll send a separate email opining on the directions that can > go. > > The features that should be incorporated/designed for (value > classes/string templates/pattern matching) are more clearly in view than > they were three years ago when last it was brought up and certainly more > real than in 2014 when the JEP was made - so I don't think this isn't a > horrible time to start some movement. > > On Wed, Dec 14, 2022 at 3:14 AM Alan Bateman > wrote: > >> On 14/12/2022 03:36, Ethan McCue wrote: >> > Hey all, >> > >> > I'm doodling on JEP-198, and I was wondering if anyone had a line on >> > the boilerplate needed to add a new module to the build. >> > >> > Just adding >> > >> > src/ >> > java.json/ >> > share/ >> > classes/ >> > module-info.java >> > java/ >> > json/ >> > Json.java >> > ... >> > >> > was enough to get it to show up under the output when I run "make >> > images", but I can't shake the feeling that I'm missing something. >> >> Modules with java.* classes must be mapped to the boot or platform class >> loader. That mapping is defined in make/conf/module-loader-map.conf. >> >> java.se is the aggregator module for the APIs for ava SE platform. >> You'll see it has a `requires transitive` for each of the standard >> modules. >> >> That should be enough to get your doodle going. As per previous mails, >> it's very likely that JEP-198 will be withdrawn or replaced with a new >> JEP, when the time comes. >> >> -Alan. >> >
Re: Boilerplate to add a new module
Thank you! And yeah - I'll send a separate email opining on the directions that can go. The features that should be incorporated/designed for (value classes/string templates/pattern matching) are more clearly in view than they were three years ago when last it was brought up and certainly more real than in 2014 when the JEP was made - so I don't think this isn't a horrible time to start some movement. On Wed, Dec 14, 2022 at 3:14 AM Alan Bateman wrote: > On 14/12/2022 03:36, Ethan McCue wrote: > > Hey all, > > > > I'm doodling on JEP-198, and I was wondering if anyone had a line on > > the boilerplate needed to add a new module to the build. > > > > Just adding > > > > src/ > > java.json/ > > share/ > > classes/ > > module-info.java > > java/ > > json/ > > Json.java > > ... > > > > was enough to get it to show up under the output when I run "make > > images", but I can't shake the feeling that I'm missing something. > > Modules with java.* classes must be mapped to the boot or platform class > loader. That mapping is defined in make/conf/module-loader-map.conf. > > java.se is the aggregator module for the APIs for ava SE platform. > You'll see it has a `requires transitive` for each of the standard modules. > > That should be enough to get your doodle going. As per previous mails, > it's very likely that JEP-198 will be withdrawn or replaced with a new > JEP, when the time comes. > > -Alan. >
Boilerplate to add a new module
Hey all, I'm doodling on JEP-198, and I was wondering if anyone had a line on the boilerplate needed to add a new module to the build. Just adding src/ java.json/ share/ classes/ module-info.java java/ json/ Json.java ... was enough to get it to show up under the output when I run "make images", but I can't shake the feeling that I'm missing something. (jtreg I assume has some extra setup? Anything else?)
Re: What is meant by "document context" in JEP 198?
That tracks. The component I, personally, might want JSON available internally for is an artifact resolution API [1]. Gradle's module metadata is out there in the world and could be worth consuming [2]. That's neither here nor there though. I'd say that the JEP as written could use some rework even before its "time has come". * Goals like "useful subset for Java ME" don't feel relevant anymore. * Unstated goals that seem implicit in how folks have talked about potential APIs, like "improve over JSR-353" or "obviate ecosystem desire for JAXB type data binding" * "Goals" that seem derived more from the properties of whatever prototype implementation the JEP had than a core "user story". A split token/event/immutable tree API is just one possible solution to problems like big documents and letting power users avoid allocations. [1]: https://mail.openjdk.org/pipermail/core-libs-dev/2022-September/094231.html [2]: https://docs.gradle.org/current/userguide/publishing_gradle_module_metadata.html On Fri, Nov 25, 2022 at 9:08 AM Alan Bateman wrote: > On 25/11/2022 13:27, Ethan McCue wrote: > > ...huh > > >> The recent removal of Nashorn has indicated several places were > >> javascript were used in the JDK mainly due to it's built-in support for > >> parsing JSON > None of those seem like they would have been using nashorn in the past, so i > wonder what this is referring to. > > > The quoted text seems to be from a mail that Magnus Ihse Bursie sent to > the discuss list in 2020 [1]. It's the same date that JDK-8242934 [1] was > created so it may be that the mail was (at least partly) prompted by the > need to change the test for`jfr print --json`. The test for that feature > used to use Nashorn and had to be changed to use a JSON parser in the test > suite. The JSON parser was subsequently moved to > test/lib/jdk/test/lib/json/JSONValue.java so it could be used by other > tests. > > -Alan > > [1] https://mail.openjdk.org/pipermail/discuss/2020-April/005398.html > [2] https://bugs.openjdk.org/browse/JDK-8242934 >
Re: What is meant by "document context" in JEP 198?
...huh >> The recent removal of Nashorn has indicated several places were >> javascript were used in the JDK mainly due to it's built-in support for >> parsing JSON None of those seem like they would have been using nashorn in the past, so i wonder what this is referring to. On Fri, Nov 25, 2022 at 2:52 AM Alan Bateman wrote: > On 25/11/2022 02:04, Ethan McCue wrote: > > I suppose while I'm asking questions - what exactly are the parts of > > the JDK making use of ad-hoc json? Maybe we could ship *something* as > > a jdk.internal module for those use cases? > > > JEP 198 pre-dates records, sealed classes, pattern matching, ... and I > will assume will re-drafted or replaced when the time comes. > > Right now, I don't think there are too many places in the JDK that > interact with JSON. The `jfr` tool can print the records from a JFR > recording as a JSON document. The HotSpotDiagnosMXBean API and `jcmd` > tool can generate a thread dump as a JSON document. I think the only > parsing at this time is the HotSpot compiler control (JEP 165) where > directives file is a subset of JSON but that is implemented in C++ in > libjvm and probably not what you are looking for. > > There are number of places in the JDK that read configuration and it > might be that some of these could consume configuration in JSON in the > future. > > -Alan >
Re: What is meant by "document context" in JEP 198?
I suppose while I'm asking questions - what exactly are the parts of the
JDK making use of ad-hoc json? Maybe we could ship *something* as a
jdk.internal module for those use cases?
On Thu, Nov 24, 2022 at 8:55 PM Ethan McCue wrote:
> I'm reading JEP 198 and sketching out what an implementation could look
> like pursuant to this old conversation.
>
> https://mail.openjdk.org/pipermail/discuss/2020-April/005401.html
>
> My biggest question right now is what does the JEP mean exactly by
> "document context"
>
>
>- Parsing APIs which allow a choice of parsing token stream, event
>(includes document hierarchy context) stream, or immutable tree
>representation views of JSON documents and data streams.
>
>
>
> So token stream I understand as something akin to
>
> sealed interface Token {
> record StartArray() implements Token {}
> record EndArray() implements Token {}
> record StartObject() implements Token {}
> record EndObject() implements Token {}
> record Number(Json.Number value) implements Token {}
> record String(Json.String value) implements Token {}
> record True() implements Token {}
> record False() implements Token {}
> record Null() implements Token {}
> }
>
> Which matches up with the model in
> https://fasterxml.github.io/jackson-core/javadoc/2.7/com/fasterxml/jackson/core/JsonToken.html
>
> And an immutable tree representation as akin to
>
> sealed interface Json {
> sealed interface Array extends Json, List ...
> sealed interface Object extends Json, Map ...
> sealed abstract class Number extends java.lang.Number implements Json
> ...
> sealed interface String extends Json, CharSequence ...
> sealed interface Boolean ...
> sealed interface Null extends Json ...
> }
>
> Which, ignoring the open questions of
> * Does the immutable tree need to be eagerly realized?
> * Do we need to wait for valhalla to land
> * Do we need to wait for full pattern matching to land
>
> (because they make me sad)
>
> I'm still unsure what information needs to be included in an "Event"
> stream that would constitute "document context". Is it pointers to parent
> collections? The current "Path"?
>
> sealed interface PathElement {
> record Field(String fieldName) implements PathElement {}
> record Index(int index) implements PathElement {}
> }
>
>
>
>
What is meant by "document context" in JEP 198?
I'm reading JEP 198 and sketching out what an implementation could look
like pursuant to this old conversation.
https://mail.openjdk.org/pipermail/discuss/2020-April/005401.html
My biggest question right now is what does the JEP mean exactly by
"document context"
- Parsing APIs which allow a choice of parsing token stream, event
(includes document hierarchy context) stream, or immutable tree
representation views of JSON documents and data streams.
So token stream I understand as something akin to
sealed interface Token {
record StartArray() implements Token {}
record EndArray() implements Token {}
record StartObject() implements Token {}
record EndObject() implements Token {}
record Number(Json.Number value) implements Token {}
record String(Json.String value) implements Token {}
record True() implements Token {}
record False() implements Token {}
record Null() implements Token {}
}
Which matches up with the model in
https://fasterxml.github.io/jackson-core/javadoc/2.7/com/fasterxml/jackson/core/JsonToken.html
And an immutable tree representation as akin to
sealed interface Json {
sealed interface Array extends Json, List ...
sealed interface Object extends Json, Map ...
sealed abstract class Number extends java.lang.Number implements Json
...
sealed interface String extends Json, CharSequence ...
sealed interface Boolean ...
sealed interface Null extends Json ...
}
Which, ignoring the open questions of
* Does the immutable tree need to be eagerly realized?
* Do we need to wait for valhalla to land
* Do we need to wait for full pattern matching to land
(because they make me sad)
I'm still unsure what information needs to be included in an "Event" stream
that would constitute "document context". Is it pointers to parent
collections? The current "Path"?
sealed interface PathElement {
record Field(String fieldName) implements PathElement {}
record Index(int index) implements PathElement {}
}
Re: What are the policies for internal modules?
(always so hard to remember to reply all) On Thu, Nov 24, 2022 at 8:26 PM Ethan McCue wrote: > The use here is giving help to the user on > misspelled classes/methods/fields like "new ArayList". > > For now we've just copy pasted the class since that seems the least > architecturally painful. (And it's not clear that text distance is the full > story with regards to this sort of analysis, so tbd if that stays in its > current form) > > On Mon, Nov 21, 2022 at 3:19 AM Alan Bateman > wrote: > >> On 21/11/2022 01:02, Ethan McCue wrote: >> > There are some modules like jdk.internal.le which contain no >> > publicly exported packages. In some of the experimentation we are >> > doing, We want to use the >> > jdk.internal.org.jline.utils.Levenshtien class from jdk.compiler. >> > >> > Mechanically, we can do this without creating any new modules by >> > adding a qualified export of the utils package to jdk.compiler. We >> > could also make a brand new, tiny pointless module called >> > jdk.internal.levenshtien. That would be the "cleanest" approach, but >> > only if we didn't consider the existence of the internal modules to be >> > public API. >> > >> > So that's my general question - what is the bar for making a new >> > internal module, and is the set of internal modules subject to >> > backwards compatibility concerns? >> >> There aren't many compatibility constraints on jdk.internal modules. >> Assume they can change or be removed in any release. It's important they >> don't export any packages to all modules, otherwise they cease to be >> "internal". Also if they provide services then the name of the module >> may be something that users of jlink may become dependent on, so we have >> to be careful there. >> >> A general point is that we don't want the JDK to backslide to a big ball >> of mud. We put a lot of effort several years ago to de-tangle the >> libraries so it would be disappointing see jdk.compiler, with no >> interest in line editing, add a dependency on jdk.internal.le so that it >> can make use of one of utility classes. So I don't think we should do >> that. >> >> Can you say a bit more about what you are doing? Is this just a method >> to compute the Levenshtein distance between two strings? I assume you >> can just implement that in jdk.compiler without touching the module >> graph. It might be useful to get some sense on how common fuzzy matching >> is in the eco system to see if there is any merit to exposing APIs for >> this. >> >> -Alan >> >>
What are the policies for internal modules?
There are some modules like jdk.internal.le which contain no publicly exported packages. In some of the experimentation we are doing, We want to use the jdk.internal.org.jline.utils.Levenshtien class from jdk.compiler. Mechanically, we can do this without creating any new modules by adding a qualified export of the utils package to jdk.compiler. We could also make a brand new, tiny pointless module called jdk.internal.levenshtien. That would be the "cleanest" approach, but only if we didn't consider the existence of the internal modules to be public API. So that's my general question - what is the bar for making a new internal module, and is the set of internal modules subject to backwards compatibility concerns?
Re: Task for an aspiring JDK contributor
The problem is that would necessitate a new public class that is *only*
used as a return value. Since it would then come up with auto complete for
people searching for ArrayList, that's non trivial.
Deprecated also isn't the right concept. Mechanically it puts a line
through a method invocation, but this is a more specific sort of
disclaimer. Like a @Unsupported.
There is a whole separate universe of pitches for more standard
annotations, but efforts like that would probably have to start in the
community like jspecify is attempting for nullness annotations.
On Fri, Nov 18, 2022, 12:47 PM Andreas Røsdal
wrote:
> How about this as a proposed solution to avoid
> UnsupportedOperationException:
> 1. Renamed the internal ArrayList to ArrayWrappingList
> 2. Override add() in ArrayWrappingList and make it deprecated to warn
> people from using it.
>
>
> On Fri, Nov 18, 2022 at 6:23 PM Ethan McCue wrote:
>
>> I think there is potentially actionable feedback in that the exception
>> thrown when users bump into this limitation kinda sucks
>>
>> jshell> Arrays.asList(1, 2, 3).add(4);
>> | Exception java.lang.UnsupportedOperationException
>> |at AbstractList.add (AbstractList.java:155)
>> |at AbstractList.add (AbstractList.java:113)
>> |at (#8:1)
>>
>> jshell> try { Arrays.asList(1, 2, 3).add(4);} catch (Exception e) {
>> System.out.println(e.getMessage());}
>> null
>>
>> I think there would be value in overriding "add" and other such
>> operations to provide enough context for users to *understand* that
>> * they did an unsupported operation
>> * the list given by `Arrays.asList` was the cause
>>
>> If I had to guess, that's the core of the frustration. The process to get
>> from "not understanding what went wrong" -> "understanding what went wrong"
>> -> "knowing how to fix it" is high.
>>
>> The design problem is how much context can be conveyed in an exception
>> message/stack trace.
>>
>> There is a similar conversation to be had for the collections returned by
>> List.of() and similar.
>>
>> jshell> List.of().add(1)
>> | Exception java.lang.UnsupportedOperationException
>> |at ImmutableCollections.uoe (ImmutableCollections.java:142)
>> |at ImmutableCollections$AbstractImmutableCollection.add
>> (ImmutableCollections.java:147)
>> |at (#6:1)
>>
>> jshell> try { List.of(1, 2, 3).add(4);} catch (Exception e) {
>> System.out.println(e.getMessage());}
>> null
>>
>> There is a clue in the stack trace here though for List.of() with the
>> "ImmutableCollections" calls. Maybe if we took two moves
>>
>> 1. Renamed the internal ArrayList to something like ArrayWrappingList
>> 2. Overrode add
>>
>> then the stack trace could be enough (or better than the status quo)
>>
>> jshell> Arrays.asList(1, 2, 3).add(4);
>> | Exception java.lang.UnsupportedOperationException
>> |at ArrayWrappingList.add (ArrayWrappingList.java:155)
>> |at ArrayWrappingList.add (ArrayWrappingList.java:113)
>> |at (#8:1)
>>
>> On Fri, Nov 18, 2022 at 12:14 PM Andreas Røsdal
>> wrote:
>>
>>> `new ArrayList<>(Arrays.asList(array))` is quite complex syntax to
>>> convert an array to an java.util.ArrayList,
>>> so a suggestion could be to add a new method to Arrays to convert an
>>> array to a normal java.util.ArrayList which is modifiable.
>>>
>>> Are there any low-hanging-fruit issues in core-libs-dev in
>>> bugs.openjdk.org that you are aware of that
>>> you would like me to help you implement?
>>>
>>> Thanks for considering my request.
>>>
>>> Regards,
>>> Andreas
>>>
>>>
>>>
>>>
>>> On Fri, Nov 18, 2022 at 5:51 PM Daniel Fuchs
>>> wrote:
>>>
>>>> Hi Andreas,
>>>>
>>>> First of all, congratulations for seeking advice before working on
>>>> a PR. This is exactly how first contributions should start. Thank
>>>> you for that!
>>>>
>>>> Given the area in which you intended to work however, `core-libs-dev`
>>>> might have been a better list than `discuss` to start from.
>>>>
>>>> With regard to the meat of the issue however, and as noted by Ethan,
>>>> Arrays.asList() behaves as intended, and changing that would be a
>>>> major incompatible change, as many users of the API expect the list
>>>> returned by Arra
Re: Task for an aspiring JDK contributor
I think there is potentially actionable feedback in that the exception
thrown when users bump into this limitation kinda sucks
jshell> Arrays.asList(1, 2, 3).add(4);
| Exception java.lang.UnsupportedOperationException
|at AbstractList.add (AbstractList.java:155)
|at AbstractList.add (AbstractList.java:113)
|at (#8:1)
jshell> try { Arrays.asList(1, 2, 3).add(4);} catch (Exception e) {
System.out.println(e.getMessage());}
null
I think there would be value in overriding "add" and other such operations
to provide enough context for users to *understand* that
* they did an unsupported operation
* the list given by `Arrays.asList` was the cause
If I had to guess, that's the core of the frustration. The process to get
from "not understanding what went wrong" -> "understanding what went wrong"
-> "knowing how to fix it" is high.
The design problem is how much context can be conveyed in an exception
message/stack trace.
There is a similar conversation to be had for the collections returned by
List.of() and similar.
jshell> List.of().add(1)
| Exception java.lang.UnsupportedOperationException
|at ImmutableCollections.uoe (ImmutableCollections.java:142)
|at ImmutableCollections$AbstractImmutableCollection.add
(ImmutableCollections.java:147)
|at (#6:1)
jshell> try { List.of(1, 2, 3).add(4);} catch (Exception e) {
System.out.println(e.getMessage());}
null
There is a clue in the stack trace here though for List.of() with the
"ImmutableCollections" calls. Maybe if we took two moves
1. Renamed the internal ArrayList to something like ArrayWrappingList
2. Overrode add
then the stack trace could be enough (or better than the status quo)
jshell> Arrays.asList(1, 2, 3).add(4);
| Exception java.lang.UnsupportedOperationException
|at ArrayWrappingList.add (ArrayWrappingList.java:155)
|at ArrayWrappingList.add (ArrayWrappingList.java:113)
|at (#8:1)
On Fri, Nov 18, 2022 at 12:14 PM Andreas Røsdal
wrote:
> `new ArrayList<>(Arrays.asList(array))` is quite complex syntax to
> convert an array to an java.util.ArrayList,
> so a suggestion could be to add a new method to Arrays to convert an array
> to a normal java.util.ArrayList which is modifiable.
>
> Are there any low-hanging-fruit issues in core-libs-dev in
> bugs.openjdk.org that you are aware of that
> you would like me to help you implement?
>
> Thanks for considering my request.
>
> Regards,
> Andreas
>
>
>
>
> On Fri, Nov 18, 2022 at 5:51 PM Daniel Fuchs
> wrote:
>
>> Hi Andreas,
>>
>> First of all, congratulations for seeking advice before working on
>> a PR. This is exactly how first contributions should start. Thank
>> you for that!
>>
>> Given the area in which you intended to work however, `core-libs-dev`
>> might have been a better list than `discuss` to start from.
>>
>> With regard to the meat of the issue however, and as noted by Ethan,
>> Arrays.asList() behaves as intended, and changing that would be a
>> major incompatible change, as many users of the API expect the list
>> returned by Arrays.asList to be immutable (and depend on it).
>> It is not possible nor desirable to change that.
>>
>> As for your observation, I believe that:
>>
>>`new ArrayList<>(Arrays.asList(array))`
>>
>> will get you what you want.
>>
>> best regards,
>>
>> -- daniel
>>
>> On 18/11/2022 16:29, Andreas Røsdal wrote:
>> > Yes, the exception comes when adding objects to the returned list. So I
>> > would like a convenient way to use Arrays to convert an array to a
>> > normal modifiable java.util.ArrayList, instead of this AbstractList.
>> >
>> >
>> > On Fri, Nov 18, 2022 at 5:23 PM Ethan McCue > > <mailto:et...@mccue.dev>> wrote:
>> >
>> > What situation were you encountering the exception? Was it when
>> > trying to add to the returned list?
>> >
>> > If so, that's expected. Arrays.asList only wraps an underlying
>> > array, it can't grow it. By that measure List.of() is even more
>> > unintuitive because you can't set anything.
>> >
>> > On Fri, Nov 18, 2022, 11:06 AM Andreas Røsdal
>> > mailto:andreas.ros...@gmail.com>> wrote:
>> >
>> > Hello!
>> >
>> > I am an aspiring JDK contributor, having used Java in my work as
>> > a developer.
>> >
>> > I was recently surprised by an Exception thrown when using the
>> > java.util.Arrays.asList() method
>> > so I would like to propose
Re: Task for an aspiring JDK contributor
new ArrayList<>(Arrays.asList(theArray)); I'm not intending to be pithy - just this is one of the "known deficiencies" of the collections framework. If you look hard enough you can find some spitballing I did on how to address it in the core-libs mailing list (which would be the place to go for this sort of stuff - cc-ing). The internal class names aren't trivial to fix either because someone somewhere has serialized a java.util.Arrays$ArrayList using ObjectOutputStream and wants to read it back out at some point. Not impossible to address, just tricky. What is the situation where you encountered the internal class name? It could be worth it to make it more clear in whatever that debug scenario is. --- Other than that, to contribute code to the JDK you need to sign the OCA. It will take a bit so you should get that out of the way. On Fri, Nov 18, 2022 at 11:29 AM Andreas Røsdal wrote: > Yes, the exception comes when adding objects to the returned list. So I > would like a convenient way to use Arrays to convert an array to a normal > modifiable java.util.ArrayList, instead of this AbstractList. > > > On Fri, Nov 18, 2022 at 5:23 PM Ethan McCue wrote: > >> What situation were you encountering the exception? Was it when trying to >> add to the returned list? >> >> If so, that's expected. Arrays.asList only wraps an underlying array, it >> can't grow it. By that measure List.of() is even more unintuitive because >> you can't set anything. >> >> On Fri, Nov 18, 2022, 11:06 AM Andreas Røsdal >> wrote: >> >>> Hello! >>> >>> I am an aspiring JDK contributor, having used Java in my work as a >>> developer. >>> >>> I was recently surprised by an Exception thrown when using the >>> java.util.Arrays.asList() method >>> so I would like to propose some improvements to this API. In particular: >>> - When using java.util.Arrays.asList() then AbstractList is throwing >>> UnsupportedOperationException >>> which is not user-friendly or intuitive. >>> - java.util.Arrays.asList() returning a private class called ArrayList, >>> which is not the usual java.util.ArrayList, so it's not user-friendly or >>> intuitive. >>> >>> Since this would be my first contribution to the JDK, and the goal is to >>> complete the contribution with accepted pull request and get the Arrays API >>> improved, what would the first step to making this first improvement to the >>> JDK be? >>> >>> I would also like to share a link to an open source project I've been >>> working on: >>> https://github.com/fciv-net/fciv-net >>> >>> Thank you! >>> >>> Regards, >>> Andreas R. >>> >>
Re: Standard Artifact Resolution API
Okay so from what I can parse there are different streams of work here 1. The JDK could be well served by a "standard" build tool. I don't think this is controversial, but I'm nowhere near qualified enough to say what the "right" choice is. Maybe it is just taking gradle, who knows. What I do posit, though, is that the problem space is underexplored. I will claim this is partially because things like dependency resolution have been out of reach of the kinds of low stakes amateur exploration that drives innovation. The Node ecosystem churns through a build system a season, but they are at least actively exploring options - y'know? Let's call this hypothetical tool "jproject". 2. The JDK has the power to standardize improvements to dependency metadata and resolution The gradle team has done what they think is a good extension of the classifier system. There is some other analogous work in the rust ecosystem around features that could be interesting to incorporate. Maybe even ABI-level tool enforced minimum semver bumps like Elm, where we make something similar to Scala's Tasty metadata. Of all the players in the ecosystem, the JDK probably has the most power to make a given answer "happen", and being comfortable that the right answer has been reached may or may not be a blocker for "jproject" being real. 3. A standard dependency resolution api In order to successfully drive improvements to dependency resolution the JDK needs a tool, but I don't think that needs to be dependent on either knowing the "best-path-forward" metadata or deciding on a JDK build tool. There is enough prior work in tools like coursier (https://get-coursier.io/), shrinkwrap resolver (https://github.com/shrinkwrap/resolver), maven aether ( https://maven.apache.org/aether.html), and tools deps ( https://github.com/clojure/tools.deps.alpha) to at least start to talk about what a "jresolve" cli tool or "java.util.resolve"' api could look like. (apologies if that is only mildly coherent) On Sat, Sep 10, 2022 at 12:19 PM Scott Palmer wrote: > After thinking about it a bit more, I agree that a canonical dependency > format wouldn’t be much good without the tool to go with it. In my > particular case, I have builds that incorporate native code and Java. I > use Gradle partly because it handles both (though it could do a better job > with C and cross-compiling), and also because it can handle complex build > logic for putting the pieces together. One thing I dislike about tools > such as Rusts’ cargo or Node’s npm, is that they seem to make the > assumption that my project is based on a single language. That’s one reason > I don’t use Maven - it assumes you have a vanilla Java project and is > painful to use for anything beyond that. Any tool integrated into the JDK > should be able to build the JDK itself, including all the native bits, and > dealing with the multiple platforms and conditions. There should be no > need for external scripts to hack it together and very few external tools. > E.g. a C/C++ compiler would be needed to b build the JDK, but not > bash/Cygwin. > > I’ve often thought, why don’t they just use Gradle for building the JDK? > It seems to me that it would be so much simpler (and faster), but I can > imagine the mess of build-logic history that would need to be untangled. > I haven’t looked into alternatives like Bazel, as Gradle is working well > for me, but in the design of any tool for the JDK all of these other tools > should be looked at. Perhaps in the end the JDK could simply settle on one > of the modern tools already available and integrate or endorse it? That to > me would make more sense than building yet another tool ( > https://xkcd.com/927/). For example, if the JDK were to adopt Gradle as > the tool to use to build the JDK, it could de-facto become the “JDK build > tool”. Other projects formerly part of the JDK, like JavaFX, are already > using Gradle (though not as effectively as they could). > > Anyway, that’s just my 2 cents on the subject. > > Regards, > > Scott > > > On Sep 9, 2022, at 1:31 PM, Ethan McCue wrote: > > > We already have Ant+Ivy, Maven, Gradle and they have significantly > different philosophies such that agreeing on a single dependency management > tool may be too ambitious. > > Maybe it would be useful to dig into what those different philosophies are? > > > I suggest looking at what Gradle has done in this area. > > It would be a reasonable goal for Java to have a canonical format (like > Rust’s ‘cargo’ format) for external dependencies that addressed all the > issues and tools could use it and benefit from the potential cross-tool > compatibility. > > I don't disagree per-se, but without an actual tool the JDK doesn'
Re: Standard Artifact Resolution API
> We already have Ant+Ivy, Maven, Gradle and they have significantly different philosophies such that agreeing on a single dependency management tool may be too ambitious. Maybe it would be useful to dig into what those different philosophies are? > I suggest looking at what Gradle has done in this area. > It would be a reasonable goal for Java to have a canonical format (like Rust’s ‘cargo’ format) for external dependencies that addressed all the issues and tools could use it and benefit from the potential cross-tool compatibility. I don't disagree per-se, but without an actual tool the JDK doesn't exactly have much leverage to drive adoption of whatever dependency-metadata.[xml|json|toml|tar.gz] would address all the issues. It also would still need to handle "the world as is" for published artifacts. > agreeing on a single dependency management tool may be too ambitious. Maybe, but an uncharitable combination of accepting both the statements "it's nearly impossible to write a modern application without external dependencies" and "the jdk does not provide the ability to resolve external dependencies" is "it's nearly impossible to write a modern application with just the jdk" Which is at least a tad unfortunate. On Fri, Sep 9, 2022 at 12:22 PM Scott Palmer wrote: > I suggest looking at what Gradle has done in this area. For example they > found that the POM format didn’t have information required to properly > identify variants so they added additional metadata. (See: > https://docs.gradle.org/current/userguide/variant_model.html) > > It would be a reasonable goal for Java to have a canonical format (like > Rust’s ‘cargo’ format) for external dependencies that addressed all the > issues and tools could use it and benefit from from the potential > cross-tool compatibility. I think however that the focus would be on the > repository format & metadata, not the tool. We already have Ant+Ivy, > Maven, Gradle and they have significantly different philosophies such that > agreeing on a single dependency management tool may be too ambitious. > > Scott > > > On Sep 9, 2022, at 9:59 AM, Ethan McCue wrote: > > This email is mostly to test the waters, I expect some hostility. > > Say, as a premise, that an API for resolving external dependencies was > something that the JDK concerned itself with providing. > > I think the foundation for that is there - the POM v4 format is more or > less around forever, maven style repositories aren't going anywhere, and > it's nearly impossible to write a modern application without some degree of > dependence on code that was written by other people. > > What properties would folks want from such an API? What blockers would > there be (technical/social)? Any other initial thoughts? > > >
Standard Artifact Resolution API
This email is mostly to test the waters, I expect some hostility. Say, as a premise, that an API for resolving external dependencies was something that the JDK concerned itself with providing. I think the foundation for that is there - the POM v4 format is more or less around forever, maven style repositories aren't going anywhere, and it's nearly impossible to write a modern application without some degree of dependence on code that was written by other people. What properties would folks want from such an API? What blockers would there be (technical/social)? Any other initial thoughts?
Re: Proposal: Collection mutability marker interfaces
If the collections would decide whether or not to copy, I don't think just
requesting an immutable reference would be enough.
static List listCopy(Collection coll) {
if (coll instanceof List12 || (coll instanceof ListN && !
((ListN)coll).allowNulls)) {
return (List)coll;
} else {
return (List)List.of(coll.toArray()); // implicit nullcheck
of coll
}
}
The two things that List.copyOf needs to know are that the list is
immutable, but also that it isn't a variant that might contain a null.
So maybe instead of
List y = x.immutableCopy();
It could be appropriate to use the spliterator approach and request a copy
which has certain characteristics.
static List listCopy(Collection coll) {
if (coll instanceof List list) {
return list.copyWhere(EnumSet.of(IMMUTABLE, DISALLOW_NULLS));
} else {
return (List)List.of(coll.toArray()); // implicit nullcheck
of coll
}
}
but that leaves open whether you would want to request the *presence* of
capabilities or the *absence* of them.
Maybe
List.of().copyWhere();
Could be defined to give a list where it is immutable and nulls aren't
allowed. And then
List.of(1, 2, 3).copyWhere(EnumSet.of(ADDABLE, NULLS_ALLOWED));
gives you a mutable copy where nulls are allowed.
This still does presume that making a copy if a capability isn't present is
the only use of knowing the capabilities - which from the conversation so
far isn't that unrealistic
On Fri, Aug 26, 2022 at 11:20 AM John Hendrikx
wrote:
>
> On 24/08/2022 15:38, Ethan McCue wrote:
>
> A use case that doesn't cover is adding to a collection.
>
> Say as part of a method's contract you state that you take ownership of a
> List. You aren't going to copy even if the list is mutable.
>
> Later on, you may want to add to the list. Add is supported on ArrayList
> so you don't need to copy and replace your reference, but you would if the
> list you were given was made with List.of or Arrays.asList
>
> I don't think this is a common enough use case that should be catered
> for. It might be better handled with concurrent lists instead.
>
> The most common use case by far is wanting to make sure a collection
> you've received is not going to be modified while you are working with it.
> I don't think another proposal which does cover the most common cases
> should be dismissed out of hand because it doesn't support a rather rare
> use case.
> --John
>
>
>
> On Wed, Aug 24, 2022, 8:13 AM John Hendrikx
> wrote:
>
>> Would it be an option to not make the receiver responsible for the
>> decision whether to make a copy or not? Instead put this burden (using
>> default methods) on the various collections?
>>
>> If List/Set/Map had a method like this:
>>
>> List immutableCopy(); // returns a (shallow) immutable copy if
>> list is mutable (basically always copies, unless proven otherwise)
>>
>> Paired with methods on Collections to prevent collections from being
>> modified:
>>
>> Collections.immutableList(List)
>>
>> This wrapper is similar to `unmodifiableList` except it implements
>> `immutableCopy` as `return this`.
>>
>> Then for the various scenario's, where `x` is an untrusted source of List
>> with unknown status:
>>
>> // Create a defensive copy; result is a private list that cannot be
>> modified:
>>
>> List y = x.immutableCopy();
>>
>> // Create a defensive copy for sharing, promising it won't ever
>> change:
>>
>> List y = Collections.immutableList(x.immutableCopy());
>>
>> // Create a defensive copy for mutating:
>>
>> List y = new ArrayList<>(x); // same as always
>>
>> // Create a mutable copy, modify it, then expose as immutable:
>>
>> List y = new ArrayList<>(x); // same as always
>>
>> y.add( );
>>
>> List z = Collections.immutableList(y);
>>
>> y = null; // we promise `z` won't change again by clearing the only
>> path to mutating it!
>>
>> The advantage would be that this information isn't part of the type
>> system where it can easily get lost. The actual implementation knows best
>> whether a copy must be made or not.
>>
>> Of course, the immutableList wrapper can be used incorrectly and the
>> promise here can be broken by keeping a reference to the original (mutable)
>> list, but I think that's an acceptable trade-off.
>>
>> --John
>>
>> PS. Chosen names are just for illustration; there is some discussion as
>> what "unmodifiable" vs "immutable" means in the
Re: Proposal: Collection mutability marker interfaces
All trivial collection operations scream stream api and all stream api
operations imply a full copy or at least a full scan.
> so having trouble to write this kind of code is more a feature than an
issue :)
I love all Java code equally
On Wed, Aug 24, 2022 at 11:21 AM wrote:
>
>
> --
>
> *From: *"Ethan McCue"
> *To: *"Remi Forax"
> *Cc: *"John Hendrikx" , "core-libs-dev" <
> core-libs-dev@openjdk.org>
> *Sent: *Wednesday, August 24, 2022 4:27:01 PM
> *Subject: *Re: Proposal: Collection mutability marker interfaces
>
> > so it's perhaps better to call add() inside a try/catch on
> UnsupportedOperationException.
>
> As much as sin is wrong, sketching out what that might look like...
> forgive the toyness of the example
>
>
>
>
>
>
> VS
>
>
>
> final class Ex {
> private Ex() {}
>
> /*
> * Adds the odd numbers from 1 to 10 to the List then makes all the
> odds even.
> *
> * Takes ownership of the list, avoids making copies if it doesn't
> need to
> */
> static List addOdds(List l) {
> for (int i = 1; i <= 10; i++) {
> try {
> l.add(i);
> } catch (UnsupportedOperationException e) {
> l = new ArrayList<>(l);
>
>
> i -= 1; // restart with an ArrayList
>
>
> }
> }
>
> for (int i = 0; i < l.size(); i++) {
> if (l.get(i) % 2 == 1) {
> try {
> l.set(i, l.get(i) + 1);
> } catch (UnsupportedOperationException e) {
> l = new ArrayList<>(l);
> }
> }
> }
> }
> }
>
>
> as Roger said, there is no way in Java to know if the caller has not kept
> a reference (unlike Rust),
> so having trouble to write this kind of code is more a feature than an
> issue :)
>
> This kind of examples scream the Stream API, which has the correct
> semantics
> IntStream.rangeClosed(1, 10).map(i -> i % 2 == 0? i + 1:
> i).boxed().toList()
>
> Rémi
>
>
>
>
> On Wed, Aug 24, 2022 at 10:03 AM Remi Forax wrote:
>
>>
>>
>> --
>>
>> *From: *"Ethan McCue"
>> *To: *"John Hendrikx"
>> *Cc: *"core-libs-dev"
>> *Sent: *Wednesday, August 24, 2022 3:38:26 PM
>> *Subject: *Re: Proposal: Collection mutability marker interfaces
>>
>> A use case that doesn't cover is adding to a collection.
>>
>> Say as part of a method's contract you state that you take ownership of a
>> List. You aren't going to copy even if the list is mutable.
>>
>> Later on, you may want to add to the list. Add is supported on ArrayList
>> so you don't need to copy and replace your reference, but you would if the
>> list you were given was made with List.of or Arrays.asList
>>
>>
>> You can ask if the spliterator considers the collection as immutable or
>> not,
>> list.spliterator().hasCharacteristics(Spliterator.IMMUTABLE)
>>
>> sadly, List.of()/Map.of() does not report the spliterator characteristics
>> correctly (the spliterator implementations are inherited from
>> AbstracList/AbstractMap).
>>
>> so it's perhaps better to call add() inside a try/catch on
>> UnsupportedOperationException.
>>
>> Rémi
>>
>>
>> On Wed, Aug 24, 2022, 8:13 AM John Hendrikx
>> wrote:
>>
>>> Would it be an option to not make the receiver responsible for the
>>> decision whether to make a copy or not? Instead put this burden (using
>>> default methods) on the various collections?
>>>
>>> If List/Set/Map had a method like this:
>>>
>>> List immutableCopy(); // returns a (shallow) immutable copy if
>>> list is mutable (basically always copies, unless proven otherwise)
>>>
>>> Paired with methods on Collections to prevent collections from being
>>> modified:
>>>
>>> Collections.immutableList(List)
>>>
>>> This wrapper is similar to `unmodifiableList` except it implements
>>> `immutableCopy` as `return this`.
>>>
>>> Then for the various scenario's, where `x` is an untrusted source of
>>> List with unknown status:
>>>
>>> // Create a defensive copy; result is a private list that cannot be
>>> modified:
>>>
>>> List y = x.immutableCopy();
>>>
>>> // Create a defensive copy for sharing, pr
Re: Proposal: Collection mutability marker interfaces
> Safety-wise, taking transferring ownership is fraught True, or at the very least a true-ism we generally accept. But that's still a deliberate choice to make that makes a performance tradeoff. The risk of misuse is proportional always to the exposure of and audience of the api. On Wed, Aug 24, 2022 at 10:28 AM Roger Riggs wrote: > Hi, > > Safety-wise, taking transferring ownership is fraught, the new owner can't > be certain that the original owner hasn't kept a reference to it or to its > implementation and might be mucking around with it behind the new owners > back. > > Its cleaner to design the APIs to be defensive, either the API should > handle the List creation itself (and only expose immutable Lists) or make > defensive copies before using or saving it. > > $0.02, Roger > > > On 8/24/22 9:38 AM, Ethan McCue wrote: > > A use case that doesn't cover is adding to a collection. > > Say as part of a method's contract you state that you take ownership of a > List. You aren't going to copy even if the list is mutable. > > Later on, you may want to add to the list. Add is supported on ArrayList > so you don't need to copy and replace your reference, but you would if the > list you were given was made with List.of or Arrays.asList > > On Wed, Aug 24, 2022, 8:13 AM John Hendrikx > wrote: > >> Would it be an option to not make the receiver responsible for the >> decision whether to make a copy or not? Instead put this burden (using >> default methods) on the various collections? >> >> If List/Set/Map had a method like this: >> >> List immutableCopy(); // returns a (shallow) immutable copy if >> list is mutable (basically always copies, unless proven otherwise) >> >> Paired with methods on Collections to prevent collections from being >> modified: >> >> Collections.immutableList(List) >> >> This wrapper is similar to `unmodifiableList` except it implements >> `immutableCopy` as `return this`. >> >> Then for the various scenario's, where `x` is an untrusted source of List >> with unknown status: >> >> // Create a defensive copy; result is a private list that cannot be >> modified: >> >> List y = x.immutableCopy(); >> >> // Create a defensive copy for sharing, promising it won't ever >> change: >> >> List y = Collections.immutableList(x.immutableCopy()); >> >> // Create a defensive copy for mutating: >> >> List y = new ArrayList<>(x); // same as always >> >> // Create a mutable copy, modify it, then expose as immutable: >> >> List y = new ArrayList<>(x); // same as always >> >> y.add( ); >> >> List z = Collections.immutableList(y); >> >> y = null; // we promise `z` won't change again by clearing the only >> path to mutating it! >> >> The advantage would be that this information isn't part of the type >> system where it can easily get lost. The actual implementation knows best >> whether a copy must be made or not. >> >> Of course, the immutableList wrapper can be used incorrectly and the >> promise here can be broken by keeping a reference to the original (mutable) >> list, but I think that's an acceptable trade-off. >> >> --John >> >> PS. Chosen names are just for illustration; there is some discussion as >> what "unmodifiable" vs "immutable" means in the context of collections that >> may contain elements that are mutable. In this post, immutable refers to >> shallow immutability . >> On 24/08/2022 03:24, Ethan McCue wrote: >> >> Ah, I'm an idiot. >> >> There is still a proposal here somewhere...maybe. right now non jdk lists >> can't participate in the special casing? >> >> On Tue, Aug 23, 2022, 9:00 PM Paul Sandoz wrote: >> >>> List.copyOf already does what you want. >>> >>> >>> https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/List.java#L1068 >>> >>> https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/ImmutableCollections.java#L168 >>> >>> Paul. >>> >>> > On Aug 23, 2022, at 4:49 PM, Ethan McCue wrote: >>> > >>> > Hi all, >>> > >>> > I am running into an issue with the collections framework where I have >>> to choose between good semantics for users and performance. >>> > >>> > Specifically I am taking a java.util.List from my users and I need to >>> choose to either &g
Re: Proposal: Collection mutability marker interfaces
> so it's perhaps better to call add() inside a try/catch on
UnsupportedOperationException.
As much as sin is wrong, sketching out what that might look like... forgive
the toyness of the example
final class Ex {
private Ex() {}
/*
* Adds the odd numbers from 1 to 10 to the List then makes all the
odds even.
*
* Takes ownership of the list, avoids making copies if it doesn't need
to
*/
static List addOdds(List l) {
for (int i = 1; i <= 10; i++) {
l.add(i);
}
for (int i = 0; i < l.size(); i++) {
if (l.get(i) % 2 == 1) {
l.set(i, l.get(i) + 1);
}
}
}
}
VS
final class Ex {
private Ex() {}
/*
* Adds the odd numbers from 1 to 10 to the List then makes all the
odds even.
*
* Takes ownership of the list, avoids making copies if it doesn't need
to
*/
static List addOdds(List l) {
for (int i = 1; i <= 10; i++) {
try {
l.add(i);
} catch (UnsupportedOperationException e) {
l = new ArrayList<>(l);
}
}
for (int i = 0; i < l.size(); i++) {
if (l.get(i) % 2 == 1) {
try {
l.set(i, l.get(i) + 1);
} catch (UnsupportedOperationException e) {
l = new ArrayList<>(l);
}
}
}
}
}
VS
final class Ex {
private Ex() {}
/*
* Adds the odd numbers from 1 to 10 to the List then makes all the
odds even.
*
* Takes ownership of the list, avoids making copies if it doesn't need
to
*/
static List addOdds(List l) {
if (!(l instanceof Settable && l instanceof Addable)) {
l = new ArrayList<>(l);
}
for (int i = 1; i <= 10; i++) {
l.add(i);
}
for (int i = 0; i < l.size(); i++) {
if (l.get(i) % 2 == 1) {
l.set(i, l.get(i) + 1);
}
}
}
}
On Wed, Aug 24, 2022 at 10:03 AM Remi Forax wrote:
>
>
> --
>
> *From: *"Ethan McCue"
> *To: *"John Hendrikx"
> *Cc: *"core-libs-dev"
> *Sent: *Wednesday, August 24, 2022 3:38:26 PM
> *Subject: *Re: Proposal: Collection mutability marker interfaces
>
> A use case that doesn't cover is adding to a collection.
>
> Say as part of a method's contract you state that you take ownership of a
> List. You aren't going to copy even if the list is mutable.
>
> Later on, you may want to add to the list. Add is supported on ArrayList
> so you don't need to copy and replace your reference, but you would if the
> list you were given was made with List.of or Arrays.asList
>
>
> You can ask if the spliterator considers the collection as immutable or
> not,
> list.spliterator().hasCharacteristics(Spliterator.IMMUTABLE)
>
> sadly, List.of()/Map.of() does not report the spliterator characteristics
> correctly (the spliterator implementations are inherited from
> AbstracList/AbstractMap).
>
> so it's perhaps better to call add() inside a try/catch on
> UnsupportedOperationException.
>
> Rémi
>
>
> On Wed, Aug 24, 2022, 8:13 AM John Hendrikx
> wrote:
>
>> Would it be an option to not make the receiver responsible for the
>> decision whether to make a copy or not? Instead put this burden (using
>> default methods) on the various collections?
>>
>> If List/Set/Map had a method like this:
>>
>> List immutableCopy(); // returns a (shallow) immutable copy if
>> list is mutable (basically always copies, unless proven otherwise)
>>
>> Paired with methods on Collections to prevent collections from being
>> modified:
>>
>> Collections.immutableList(List)
>>
>> This wrapper is similar to `unmodifiableList` except it implements
>> `immutableCopy` as `return this`.
>>
>> Then for the various scenario's, where `x` is an untrusted source of List
>> with unknown status:
>>
>> // Create a defensive copy; result is a private list that cannot be
>> modified:
>>
>> List y = x.immutableCopy();
>>
>> // Create a defensive copy for sharing, promising it won't ever
>> change:
>>
>> List y = Collections.immutableList(x.immutableCopy());
>>
>> // Create a defensive copy for mutating:
>>
>> List y = new ArrayList<>(x); // same as always
>>
>> // Create a mutable copy, modify it, then expose as immutable:
>>
>> List y = new ArrayList<>(x); // same as always
>>
>> y.add( );
>>
>> List z = Colle
Re: Proposal: Collection mutability marker interfaces
A use case that doesn't cover is adding to a collection. Say as part of a method's contract you state that you take ownership of a List. You aren't going to copy even if the list is mutable. Later on, you may want to add to the list. Add is supported on ArrayList so you don't need to copy and replace your reference, but you would if the list you were given was made with List.of or Arrays.asList On Wed, Aug 24, 2022, 8:13 AM John Hendrikx wrote: > Would it be an option to not make the receiver responsible for the > decision whether to make a copy or not? Instead put this burden (using > default methods) on the various collections? > > If List/Set/Map had a method like this: > > List immutableCopy(); // returns a (shallow) immutable copy if > list is mutable (basically always copies, unless proven otherwise) > > Paired with methods on Collections to prevent collections from being > modified: > > Collections.immutableList(List) > > This wrapper is similar to `unmodifiableList` except it implements > `immutableCopy` as `return this`. > > Then for the various scenario's, where `x` is an untrusted source of List > with unknown status: > > // Create a defensive copy; result is a private list that cannot be > modified: > > List y = x.immutableCopy(); > > // Create a defensive copy for sharing, promising it won't ever > change: > > List y = Collections.immutableList(x.immutableCopy()); > > // Create a defensive copy for mutating: > > List y = new ArrayList<>(x); // same as always > > // Create a mutable copy, modify it, then expose as immutable: > > List y = new ArrayList<>(x); // same as always > > y.add( ); > > List z = Collections.immutableList(y); > > y = null; // we promise `z` won't change again by clearing the only > path to mutating it! > > The advantage would be that this information isn't part of the type system > where it can easily get lost. The actual implementation knows best whether > a copy must be made or not. > > Of course, the immutableList wrapper can be used incorrectly and the > promise here can be broken by keeping a reference to the original (mutable) > list, but I think that's an acceptable trade-off. > > --John > > PS. Chosen names are just for illustration; there is some discussion as > what "unmodifiable" vs "immutable" means in the context of collections that > may contain elements that are mutable. In this post, immutable refers to > shallow immutability . > On 24/08/2022 03:24, Ethan McCue wrote: > > Ah, I'm an idiot. > > There is still a proposal here somewhere...maybe. right now non jdk lists > can't participate in the special casing? > > On Tue, Aug 23, 2022, 9:00 PM Paul Sandoz wrote: > >> List.copyOf already does what you want. >> >> >> https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/List.java#L1068 >> >> https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/ImmutableCollections.java#L168 >> >> Paul. >> >> > On Aug 23, 2022, at 4:49 PM, Ethan McCue wrote: >> > >> > Hi all, >> > >> > I am running into an issue with the collections framework where I have >> to choose between good semantics for users and performance. >> > >> > Specifically I am taking a java.util.List from my users and I need to >> choose to either >> > * Not defensively copy and expose a potential footgun when I pass that >> List to another thread >> > * Defensively copy and make my users pay an unnecessary runtime cost. >> > >> > What I would really want, in a nutshell, is for List.copyOf to be a >> no-op when used on lists made with List.of(). >> > >> > Below the line is a pitch I wrote up on reddit 7 months ago for a >> mechanism I think could accomplish that. My goal is to share the idea a bit >> more widely and to this specific audience to get feedback. >> > >> > >> https://www.reddit.com/r/java/comments/sf8qrv/comment/hv8or92/?utm_source=share_medium=web2x=3 >> > >> > Important also for context is Ron Pressler's comment above. >> > -- >> > >> > What if the collections api added more marker interfaces like >> RandomAccess? >> > >> > It's already a common thing for codebases to make explicit null checks >> at error boundaries because the type system can't encode null | >> List. >> > >> > This feels like a similar problem. >> > If you have a List in the type system then you do
Re: Proposal: Collection mutability marker interfaces
Ah, I'm an idiot.
There is still a proposal here somewhere...maybe. right now non jdk lists
can't participate in the special casing?
On Tue, Aug 23, 2022, 9:00 PM Paul Sandoz wrote:
> List.copyOf already does what you want.
>
>
> https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/List.java#L1068
>
> https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/ImmutableCollections.java#L168
>
> Paul.
>
> > On Aug 23, 2022, at 4:49 PM, Ethan McCue wrote:
> >
> > Hi all,
> >
> > I am running into an issue with the collections framework where I have
> to choose between good semantics for users and performance.
> >
> > Specifically I am taking a java.util.List from my users and I need to
> choose to either
> > * Not defensively copy and expose a potential footgun when I pass that
> List to another thread
> > * Defensively copy and make my users pay an unnecessary runtime cost.
> >
> > What I would really want, in a nutshell, is for List.copyOf to be a
> no-op when used on lists made with List.of().
> >
> > Below the line is a pitch I wrote up on reddit 7 months ago for a
> mechanism I think could accomplish that. My goal is to share the idea a bit
> more widely and to this specific audience to get feedback.
> >
> >
> https://www.reddit.com/r/java/comments/sf8qrv/comment/hv8or92/?utm_source=share_medium=web2x=3
> >
> > Important also for context is Ron Pressler's comment above.
> > --
> >
> > What if the collections api added more marker interfaces like
> RandomAccess?
> >
> > It's already a common thing for codebases to make explicit null checks
> at error boundaries because the type system can't encode null |
> List.
> >
> > This feels like a similar problem.
> > If you have a List in the type system then you don't know for sure
> you can call any methods on it until you check that its not null. In the
> same way, there is a set of methods that you don't know at the
> type/interface level if you are allowed to call.
> >
> > If the List is actually a __
> > Then you can definitely call
> > And you know other reference holders might call
> > And you can confirm its this case by
> > null
> > no methods
> > no methods
> > list == null
> > List.of(...)
> > get, size
> > get, size
> > ???
> > Collections.unmodifiableList(...)
> > get, size
> > get, size, add, set
> > ???
> > Arrays.asList(...)
> > get, size, set
> > get, size, set
> > ???
> > new ArrayList<>()
> > get, size, add, set
> > get, size, add, set
> > ???
> > While yes, there is no feasible way to encode these things in the type
> system. Its not impossible to encode it at runtime though.
> > interface FullyImmutable {
> > // So you know the existence of this implies the absence
> > // of the others
> > default Void cantIntersect() { return null; }
> > }
> >
> > interace MutationCapability {
> > default String cantIntersect() { return ""; }
> > }
> >
> > interface Addable extends MutationCapability {}
> > interface Settable extends MutationCapability {}
> >
> > If the List is actually a __
> > Then you can definitely call
> > And you know other reference holders might call
> > And you can confirm its this case by
> > null
> > no methods
> > no methods
> > list == null
> > List.of(...)
> > get, size
> > get, size
> > instanceof FullyImmutable
> > Collections.unmodifiableList(...)
> > get, size
> > get, size, add, set
> > !(instanceof Addable) && !(instanceof Settable)
> > Arrays.asList(...)
> > get, size, set
> > get, size, set
> > instanceof Settable
> > new ArrayList<>()
> > get, size, add, set
> > get, size, add, set
> > instanceof Settable && instanceof Addable
> > In the same way a RandomAccess check let's implementations decide
> whether they want to try an alternative algorithm or crash, some marker
> "capability" interfaces would let users of a collection decide if they want
> to clone what they are given before working on it.
> >
> >
> > --
> >
> > So the applicability of this would be that the list returned by List.of
> could implement FullyImmutable, signifying that there is no caller which
> might have a mutable handle on the collection. Then List.of could check for
> this interface and skip a copy.
> >
> >
>
>
Proposal: Collection mutability marker interfaces
Hi all,
I am running into an issue with the collections framework where I have to
choose between good semantics for users and performance.
Specifically I am taking a java.util.List from my users and I need to
choose to either
* Not defensively copy and expose a potential footgun when I pass that List
to another thread
* Defensively copy and make my users pay an unnecessary runtime cost.
What I would really want, in a nutshell, is for List.copyOf to be a no-op
when used on lists made with List.of().
Below the line is a pitch I wrote up on reddit 7 months ago for a mechanism
I think could accomplish that. My goal is to share the idea a bit more
widely and to this specific audience to get feedback.
https://www.reddit.com/r/java/comments/sf8qrv/comment/hv8or92/?utm_source=share_medium=web2x=3
Important also for context is Ron Pressler's comment above.
--
What if the collections api added more marker interfaces like RandomAccess?
It's already a common thing for codebases to make explicit null checks at
error boundaries because the type system can't encode null | List.
This feels like a similar problem.
If you have a List in the type system then you don't know for sure you
can call any methods on it until you check that its not null. In the same
way, there is a set of methods that you don't know at the type/interface
level if you are allowed to call.
If the List is actually a __
Then you can definitely call
And you know other reference holders might call
And you can confirm its this case by
null
no methods
no methods
list == null
List.of(...)
get, size
get, size
???
Collections.unmodifiableList(...)
get, size
get, size, add, set
???
Arrays.asList(...)
get, size, set
get, size, set
???
new ArrayList<>()
get, size, add, set
get, size, add, set
???
While yes, there is no feasible way to encode these things in the type
system. Its not impossible to encode it at runtime though.
interface FullyImmutable {
// So you know the existence of this implies the absence
// of the others
default Void cantIntersect() { return null; }
}
interace MutationCapability {
default String cantIntersect() { return ""; }
}
interface Addable extends MutationCapability {}
interface Settable extends MutationCapability {}
If the List is actually a __
Then you can definitely call
And you know other reference holders might call
And you can confirm its this case by
null
no methods
no methods
list == null
List.of(...)
get, size
get, size
instanceof FullyImmutable
Collections.unmodifiableList(...)
get, size
get, size, add, set
!(instanceof Addable) && !(instanceof Settable)
Arrays.asList(...)
get, size, set
get, size, set
instanceof Settable
new ArrayList<>()
get, size, add, set
get, size, add, set
instanceof Settable && instanceof Addable
In the same way a RandomAccess check let's implementations decide whether
they want to try an alternative algorithm or crash, some marker
"capability" interfaces would let users of a collection decide if they want
to clone what they are given before working on it.
--
So the applicability of this would be that the list returned by List.of
could implement FullyImmutable, signifying that there is no caller which
might have a mutable handle on the collection. Then List.of could check for
this interface and skip a copy.
